Adding SSH host key to known_hosts

$ sudo sh -c "ssh-keyscan -H -t rsa,dsa hostname >> /etc/ssh/ssh_known_hosts"

-H Hash all hostnames and addresses in the output.  Hashed names may be used normally by ssh and sshd, but they do not reveal identifying information should the file's contents be disclosed.

-t type Specifies the type of the key to fetch from the scanned hosts.  The possible values are “rsa1” for protocol version 1 and “dsa”, “ecdsa” or “rsa” for protocol version 2.  Multiple values may be specified by separating them with commas.  The default is “rsa”.

Optimising Linux for minimal writes - useful for an SSD

Tip 1: Use ext2 instead of ext3/ext4.

ext3 and ext4 are journaled filesystems and so maintain a log of all filesystem changes (which can be used for recovery if need be).  If you're using a laptop chances are sudden power failure isn't really a threat so you'll lose less by not having a journaled filesystem - if you're using a desktop your choice to use a non-journaled filesystem should be an informed one.

Tip 2: Don't update file access times.

Modify your /etc/fstab adding noatime to the options for mounted drives.  This will stop writing to the file system every time a file is accessed.  (This is also used to improve disk I/O performance for critical applications - git does this for example, though they may have switched to relatime).

Tip 3: Mount "non-essential" write-heavy directories as tmpfs.

Update /etc/fstab to mount /tmp, /var/tmp and /var/log as tmps:

#              

tmpfs           /tmp            tmpfs   defaults        0       0
tmpfs           /var/tmp        tmpfs   defaults        0       0
tmpfs           /var/log        tmpfs   defaults        0       0

Unfortunately when /var/log is now mounted it won't have any of the necessary directories so we'll need to create them each time on boot.  Update /etc/rc.local with the following snippet just before the "exit 0" line.

for dir in apparmor apt ConsoleKit cups dist-upgrade fsck gdm installer libvirt news samba speech-dispatcher unattended-upgrades; do
if [ ! -e /var/log/$dir ] ; then
mkdir /var/log/$dir
fi
done

You can get the list of directories you need by running cd /var/log; ls -d */

These tips can be found repeated on the web but I gathered most of my info from here.  That page also lists some further optimisations for the kernel to take full advantage of an SSD's speed.

Mounting an encrypted logical volume

I found this useful when trying to do some recovery work on my computer which uses dm-crypt for full disk encryption.  The following was done using the Ubuntu Maverick (10.10) livecd.

Mount the encrypted partition

root@ubuntu:/home/ubuntu# sudo cryptsetup luksOpen /dev/sda4 data

Enter passphrase for /dev/sda4:

root@ubuntu:/home/ubuntu# ls /dev/mapper

control  data

Mount the logical volume

root@ubuntu:/home/ubuntu# aptitude install lvm2

root@ubuntu:/home/ubuntu# vgscan

  Reading all physical volumes.  This may take a while...

  Found volume group "vg0" using metadata type lvm2

root@ubuntu:/home/ubuntu# vgchange -ay vg0

  2 logical volume(s) in volume group "vg0" now active

root@ubuntu:/home/ubuntu# mkdir /mnt/tmp

root@ubuntu:/home/ubuntu# sudo mount /dev/vg0/root /mnt/tmp

Linux: Add existing user to existing group

I seem to need to look this up at least once a month....


usermod -a -G ftp tony # add tony to the ftp group

Install dependencies with dpkg (kinda)

dpkg -i /tmp/package.deb # Try to install package, fail, but generate list of unresolved dependencies
apt-get -f --force-yes --yes install # Resolve generated dependencies
dpkg -i /tmp/package.deb # Install package (with dependencies now met)

Delete blank lines in vim

Thanks to stackoverflow for this one.

:g/^$/d will delete all blank lines in a file

Why? Because :g will execute a command on lines which match the regex and :d deletes :)

Put the results of a command in the current vim buffer

:r will read a file into the current buffer in vim
:! will execute a command
You can use them together to read the results of a command and put them in the current buffer
e.g. :r ! ls -1 /home/user/directory | sort -r

Update Ubuntu Release from the command line

sudo apt-get install update-manager-core # may/may not be necessary
sudo do-release-upgrade


If there's no new release found and you think there should be, check /etc/update-manager/release-upgrades and change Prompt=lts to Prompt=normal if necessary

Reset Moinmoin password

I always forget this

moin account resetpw --name=username password

Postgres Recovery

Postgres killed itself during a disk failure and while later trying to start it up I got messages along the lines of:

database system was interrupted; last known up at ...
database system was not properly shut down; automatic recovery in progress
redo starts at 309/3BA1EB48
record with zero length at 309/3C9F8ED8
redo done at 309/3C9F8EA8
last completed transaction was at log time ....
could not fdatasync log file 777, segment 60: Input/output error
startup process (PID 23142) was terminated by signal 6: Aborted
aborting startup due to startup process failure

On further investigation it sound like the transaction log was corrupted. This can be fixed with pg_resetxlog. This will clear the write ahead log and may result in some data loss or loss of integrity but when nothing else works it's a lifesaver. The documentation describes some follow up steps to ensure the integrity of data after postgres is starting properly.

You can do a dry run:
sudo -u postgres /usr/lib/postgresql/8.4/bin/pg_resetxlog -n /var/lib/postgresql/8.4/main
and if that indicates a new segment and there's no other option then you might as well reset with:
sudo -u postgres /usr/lib/postgresql/8.4/bin/pg_resetxlog /var/lib/postgresql/8.4/main
or
sudo -u postgres /usr/lib/postgresql/8.4/bin/pg_resetxlog -f /var/lib/postgresql/8.4/main

Pretty Print XML in Python

From the command line:

python -c "import xml.dom.minidom; xml = xml.dom.minidom.parse('myxmldoc.xml'); print xml.toprettyxml()"

Mounting an encrypted LUKS LVM volume from a live CD

Get Luks and dm-crypt running on the live disk:

sudo apt-get install lvm2 cryptsetup

sudo modprobe dm-crypt

sudo cryptsetup luksOpen /dev/sda4 crypt1

If you LVM setup then you need to continue with:

sudo vgscan --mknodes

sudo vgchange -ay

Take note of the volume group name. 'vg0' in my case.

Mount the disk:

sudo mkdir /mnt/disk

sudo mount /dev/vg0/root /mnt/disk

Adapted rom: http://www.ubuntugeek.com/rescue-an-encrypted-luks-lvm-volume.html

Move Lucid min, max, close buttons to the right

gconftool-2 --set /apps/metacity/general/button_layout --type string menu:minimize,maximize,close

Inotify

Inotify is awesome, I've used the python bindings before to good effect but just today I needed to monitor a directory for whenever a file was created.

Pretty easy:
sudo aptitude install inotify-tools
inotifywait -m -r --format '%f' -e CREATE data/

Clock Screensaver for Ubuntu

You can use the GLText screensaver to display the time whenever you lock your screen:

In the file /usr/share/applications/screensavers/gltext.desktop change

Exec=gltext -root
to
Exec=gltext -root -front -text '%l:%M:%S %p'

ipython and virtualenv

When recently working with virtualenv recently ipython wasn't picking up the correct sites packages directory. Anyhow this post cleared everything up.


In summary...

create ~/.ipython/virtualenv.py
import site
from os import environ
from os.path import join
from sys import version_info

if 'VIRTUAL_ENV' in environ:
    virtual_env = join(environ.get('VIRTUAL_ENV'),
                       'lib',
                       'python%d.%d' % version_info[:2],
                       'site-packages')
    site.addsitedir(virtual_env)
    print 'VIRTUAL_ENV ->', virtual_env
    del virtual_env
del site, environ, join, version_info

add this to ~/.ipython/ipy_user_conf.py
def main():
    execf('~/.ipython/virtualenv.py')

Python unique time string

Because this is always handy and I always seem to forget:

>>> import time
>>> str(time.time()).split('.')[0]

Useful for generating filenames based on the time etc.

Postfix Sink

I'm not too familiar with postfix but recently I needed an SMTP server that would catch all mail (act as an open relay) and deliver none.

I did this in Ubuntu Jaunty with the following two lines added to /etc/postfix/main.cf:
relay_domains = mydomain.com # accept all mail destined for mydomain.com
defer_transports = smtp # don't deliver anything that comes in via SMTP

Strictly speaking this isn't an open relay (we only relay to one domain). There's more restrictions around relaying as described in The Book of Postfix.

From here you can view the queue using postqueue -p, dump the contents of a message using postcat -q [id] > file.mail and then finally delete the message using sudo postsuper -d [id]

Python Project Structure

As the python project I've been working on gets bigger and bigger I've been seeking to setup the projects structure nicely for source code and tests. I already knew about nose but what I mainly wanted was a definitive directory structure - "best practice" type stuff. The most helpful thing I found was a blog post which I've replicated here.

Do:

• name the directory something related to your project. For example, if your project is named "Twisted", name the top-level directory for its source files Twisted. When you do releases, you should include a version number suffix: Twisted-2.5.
• create a directory Twisted/bin and put your executables there, if you have any. Don't give them a .py extension, even if they are Python source files. Don't put any code in them except an import of and call to a main function defined somewhere else in your projects.
• If your project is expressible as a single Python source file, then put it into the directory and name it something related to your project. For example, Twisted/twisted.py. If you need multiple source files, create a package instead (Twisted/twisted/, with an empty Twisted/twisted/__init__.py) and place your source files in it. For example, Twisted/twisted/internet.py.
• put your unit tests in a sub-package of your package (note - this means that the single Python source file option above was a trick - you always need at least one other file for your unit tests). For example, Twisted/twisted/test/. Of course, make it a package with Twisted/twisted/test/__init__.py. Place tests in files like Twisted/twisted/test/test_internet.py.
• add Twisted/README and Twisted/setup.py to explain and install your software, respectively, if you're feeling nice.

Don't:

• put your source in a directory called src or lib. This makes it hard to run without installing.
• put your tests outside of your Python project. This makes it hard to run the tests against an installed version.
• create a package that only has a __init__.py and then put all your code into __init__.py. Just make a module instead of a package, it's simpler.
• try to come up with magical hacks to make Python able to import your module or package without having the user add the directory containing it to their import path (either via PYTHONPATH or some other mechanism). You will not correctly handle all cases and users will get angry at you when your software doesn't work in their environment.

I found the above very helpful in organising my code as well as two other important things I've found.

• For tests which need to see your source code, steer away from using relative imports and instead put your project on the PYTHONPATH. My thinking is that anywhere the project will be used it will need to be properly installed (i.e. on the PYTONPATH) so that's how it should work normally. You can use virtualenv if you don't want to clutter your site-packages

• I've had to change my thinking from Java/C# and start to accept that multiple classes in one file is OK (C# will actually let you do this too, and Java too apparently). With that in mind, I keep classes which are functionally similar in a module and when that module starts to try and do too much I create a folder with submodules. So from the above examples Twisted/twisted.py and Twisted/test/test_* is fine for a relatively simple twisted.py (maybe 3 or 4 classes) but once the library starts to grow I'd consider breaking it up at Twisted/twisted/thispart.py and Twisted/twisted/thatpart.py

In all of this it was helpful to browse the twisted source and see how that was laid out.

Changing keymapping in Ubuntu

I recently needed to remap the 2nd enter key on my macbook so that I would actually have an insert key (what were apple thinking?). Here's how I did it:

• xev | grep keycode (run this and press they key you want to map, this will help you determine its keycode)
• in a file put "keycode xxx = MyKey" in my case I had "keycode 104 = Insert"
• xmodmap keymapfile (the key should work straight away after this)
• to make the change permanent create ~/.xmodmap with all the key mappings you want and then go to System ▸ Preferences ▸ Sessions, click the Add button, fill in the Name and Description fields and put the following into the Command field:

xmodmap ~/.xmodmap