Python Project Structure

As the python project I've been working on gets bigger and bigger I've been seeking to setup the projects structure nicely for source code and tests. I already knew about nose but what I mainly wanted was a definitive directory structure - "best practice" type stuff. The most helpful thing I found was a blog post which I've replicated here.

Do:

• name the directory something related to your project. For example, if your project is named "Twisted", name the top-level directory for its source files Twisted. When you do releases, you should include a version number suffix: Twisted-2.5.
• create a directory Twisted/bin and put your executables there, if you have any. Don't give them a .py extension, even if they are Python source files. Don't put any code in them except an import of and call to a main function defined somewhere else in your projects.
• If your project is expressible as a single Python source file, then put it into the directory and name it something related to your project. For example, Twisted/twisted.py. If you need multiple source files, create a package instead (Twisted/twisted/, with an empty Twisted/twisted/__init__.py) and place your source files in it. For example, Twisted/twisted/internet.py.
• put your unit tests in a sub-package of your package (note - this means that the single Python source file option above was a trick - you always need at least one other file for your unit tests). For example, Twisted/twisted/test/. Of course, make it a package with Twisted/twisted/test/__init__.py. Place tests in files like Twisted/twisted/test/test_internet.py.
• add Twisted/README and Twisted/setup.py to explain and install your software, respectively, if you're feeling nice.

Don't:

• put your source in a directory called src or lib. This makes it hard to run without installing.
• put your tests outside of your Python project. This makes it hard to run the tests against an installed version.
• create a package that only has a __init__.py and then put all your code into __init__.py. Just make a module instead of a package, it's simpler.
• try to come up with magical hacks to make Python able to import your module or package without having the user add the directory containing it to their import path (either via PYTHONPATH or some other mechanism). You will not correctly handle all cases and users will get angry at you when your software doesn't work in their environment.

I found the above very helpful in organising my code as well as two other important things I've found.

• For tests which need to see your source code, steer away from using relative imports and instead put your project on the PYTHONPATH. My thinking is that anywhere the project will be used it will need to be properly installed (i.e. on the PYTONPATH) so that's how it should work normally. You can use virtualenv if you don't want to clutter your site-packages

• I've had to change my thinking from Java/C# and start to accept that multiple classes in one file is OK (C# will actually let you do this too, and Java too apparently). With that in mind, I keep classes which are functionally similar in a module and when that module starts to try and do too much I create a folder with submodules. So from the above examples Twisted/twisted.py and Twisted/test/test_* is fine for a relatively simple twisted.py (maybe 3 or 4 classes) but once the library starts to grow I'd consider breaking it up at Twisted/twisted/thispart.py and Twisted/twisted/thatpart.py

In all of this it was helpful to browse the twisted source and see how that was laid out.

Changing keymapping in Ubuntu

I recently needed to remap the 2nd enter key on my macbook so that I would actually have an insert key (what were apple thinking?). Here's how I did it:

• xev | grep keycode (run this and press they key you want to map, this will help you determine its keycode)
• in a file put "keycode xxx = MyKey" in my case I had "keycode 104 = Insert"
• xmodmap keymapfile (the key should work straight away after this)
• to make the change permanent create ~/.xmodmap with all the key mappings you want and then go to System ▸ Preferences ▸ Sessions, click the Add button, fill in the Name and Description fields and put the following into the Command field:

xmodmap ~/.xmodmap

Creating a Truecrypt NTFS volume in both Ubuntu and OSX

I wanted to encrypt my external disk and I wanted it to work under Linux, Windows and OSX (read and write for all). To my mind the two obvious options were dmcrypt and truecrypt. Dmcrypt works nicely for my laptop but as far as I could find it only had one windows client (FreeOTFE) and no OSX client which pretty much left me with Truecrypt. I also needed a filesystem that worked well on all three OSs so NTFS was the best option largely due to ntfs-3g.

I ended up creating the encrypted volume under both Linux and OSX during some troubleshooting so here's how I did it in both OSs. Note: you only need to create the encrypted once in either Linux or OSX

The steps for Linux (Ubuntu 8.10):

• Install Truecrypt (grab the .deb and install that)
• truecrypt -t -c /dev/sdc (create the encrypted volume - choose "None" for the file system)
• truecrypt -t --filesystem=none /dev/sdc (mount the volume)
  
• mkfs.ntfs -f -L Cams_1Tb /dev/mapper/truecrypt1 (format the volume as ntfs)
  

Then you should be able to mount the volume either via the gui or the command line and have it automatically mount without problem

The steps for OS X (Leopard):

• Install Truecrypt
• Install ntfs-3g for mac
• I also installed MacFuse 2.0 at one stage following on from some forums posts but this may not be necessary (Truecrypt 6.1a installs MacFuse 1.6 or 1.7)
• /Applications/TrueCrypt.app/Contents/MacOS/TrueCrypt -t --filesystem=fat -c /dev/rdisk2 (fat seemed to work better than no file system at all)
• Mount the Volume in Truecrypt and look at the "Volume Properties" on the newly attached volume. Take note of the "Virtual Device" in my case this was /dev/disk5
• sudo diskutil eraseVolume NTFS-3G Cams_1Tb /dev/disk5 (convert the FAT volume to NTFS)
  

From here on end mounting and dismounting the device in Truecrypt should work a charm.