Last year BoingBoing linked to a video featuring delta waves produced by the idling engine of an ice breaker in the arctic. I found it to be a useful tool, so downloaded it for offline access. Later, I decided I wanted the audio on my phone. The video is a 10 hour loop, resulting in too large a file for mobile storage. To turn it into something reasonable for a phone, I used ffmpeg to extract the audio, chop it down to 3 hours, and add a 10 second fade on either end.
Once every year or two I need to send a fax. Never receive, just send. Usually for something involving the finance industry. Twilio makes this about as painless as it can be in the 21st century.
Unfortunately the Twilio Fax API doesn’t allow you to post the document to it directly, so the first step is to get the PDF online somewhere. After that, it can be faxed via curl.
This queues up the document to be sent, which usually takes a couple minutes. Somewhere in the response will be a URL that looks like https://fax.twilio.com/v1/Faxes/$GIBBERISH. After a few minutes, this URL can be used to check the status.
If the status is queued, processing or sending, check back in a few minutes. If it is delivered, you’re all done and can delete the uploaded PDF. If the status is something else, you probably need to try again. Perhaps ask the recipient to sign out of AOL and hang-up their modem so that their fax machine can accept your call.
The first password manager I ever used was a simple text file encrypted with GnuPG. When I needed a password I would decrypt the file, read it in Vim, and copy the required entry to the system clipboard. This system didn’t last. At the time I wasn’t using GnuPG for much else, and this was in the very beginning of my Vim days, when the program seemed cumbersome and daunting. I shortly moved to other, purpose-built password managers.
After some experimentation I landed on KeePassX, which I used for a number of years. Some time ago I decided that I wanted to move to a command-line solution. KeePassX and a web browser were the only graphical applications that I was using with any regularity. I could see no need for a password manager to have a graphical interface, and the GUI’s dependency on a mouse decreased my productivity. After a cursory look at the available choices I landed right back where I started all those years ago: Vim and GnuPG.
These days Vim is my most used program outside of a web browser and I use GnuPG daily for handling the majority of my encryption needs. My greater familiarity with both of these tools is one of the reasons I’ve been successful with the system this time around. I believe the other reason is my more systematic approach.
Structure
The power of this system comes from its simplicity: passwords are stored in plain text files that have been encrypted with GnuPG. Every platform out there has some implementation of the PGP protocol, so the files can easily be decrypted anywhere. After they’ve been decrypted, there’s no fancy file formats to deal with. It’s all just text, which can be manipulated with a plethora of powerful tools. I favor reading the text in Vim, but any text editor will do the job.
All passwords are stored within a directory called ~/pw. Within this directory are multiple files. Each of these files can be thought of as a separate password database. I store bank information in financial.gpg. Login information for various shopping websites are in ecommerce.gpg. My email credentials are in email.gpg. All of these entries could very well be stored in a single file, but breaking it out into multiple files allows me some measure of access control.
Access
I regularly use two computers: my laptop at home and a desktop machine at work. I trust my laptop. It has my GnuPG key on it and it should have access to all password database files. I do not place complete trust in my machine at work. I don’t trust it enough to give it access to my GnuPG key, and as such I have a different GnuPG key on that machine that I use for encryption at work.
Having passwords segregated into multiple database files allows me to encrypt the different files to different keys. Every file is encrypted to my primary GnuPG key, but only some are encrypted with my work key. Login credentials needed for work are encrypted to the work key. I have no need to login to my bank accounts at work, and it wouldn’t be prudent to do so on a machine that I do not fully trust, so the financial.gpg file is not encrypted to my work key. If someone compromises my work computer, they still will be no closer to accessing my banking credentials.
Git
The ~/pw directory is a git repository. This gives me version control on all of my passwords. If I accidentally delete an entry I can always get it back. It also provides syncing and redundant storage without depending on a third-party like Dropbox.
Keys
An advantage of using a directory full of encrypted files as my password manager is that I’m not limited to only storing usernames and passwords. Any file can be added to the repository. I keep keys for backups, SSH keys, and SSL keys (all of which have been encrypted with my GnuPG key) in the directory. This gives me one location for all of my authentication credentials, which simplifies the locating and backing up of these important files.
Markup
Each file is structured with Vim folds and indentation. There are various ways for Vim to fold text. I use markers, sticking with the default {{{/}}} characters. A typical password entry will look like this:
Each file is full of entries like this. Certain entries are grouped together within other folds for organization. Certain entries may have comments so that I have a record of the false personally identifiable information the service requested when I registered.
12345678
Super Ecommerce{{{
user: foobar
pass: g0d
Comments{{{
birthday: 1/1/1911
first car: delorean
}}}
}}}
Following a consistent structure like this makes the file easier to navigate and allows for the possibility of the file being parsed by a script. The fold markers come into play with my Vim configuration.
Vim
I use Vim with the vim-gnupg plugin. This makes editing of encrypted files seamless. When opening existing files, the contents are decrypted. When opening new files, the plugin asks which recipients the file should be encrypted to. When a file is open, leaking the clear text is avoided by disabling viminfo, swapfile, and undofile. I run gpg-agent so that my passphrase is remembered for a short period of time after I use it. This makes it easy and secure to work with (and create) the encrypted files with Vim. I define a few extra options in my vimrc to facilitate working with passwords.
""""""""""""""""""""" GnuPG Extensions """""""""""""""""""""" Tell the GnuPG plugin to armor new files.letg:GPGPreferArmor=1" Tell the GnuPG plugin to sign new files.letg:GPGPreferSign=1
augroup GnuPGExtra
" Set extra file options.
autocmd BufReadCmd,FileReadCmd *.\(gpg\|asc\|pgp\)call SetGPGOptions()" Automatically close unmodified files after inactivity.
autocmd CursorHold *.\(gpg\|asc\|pgp\) quit
augroup END
function SetGPGOptions()" Set updatetime to 1 minute.setupdatetime=60000" Fold at markers.setfoldmethod=marker
" Automatically close all folds.setfoldclose=all" Only open folds with insert commands.setfoldopen=insert
endfunction
The first two options simply tell vim-gnupg to always ASCII-armor and sign new files. These have nothing particular to do with password management, but are good practices for all encrypted files.
The first autocmd calls a function which holds the options that I wanted applied to my password files. I have these options apply to all encrypted files, although they’re intended primarily for use when Vim is acting as my password manager.
Folding
The primary shortcoming with using an encrypted text file as a password database is the lack of protection against shoulder-surfing. After the file has been decrypted and opened, anyone standing behind you can look over your shoulder and view all the entries. This is solved with folds and is what most of these extra options address.
I set foldmethod to marker so that Vim knows to look for all the {{{/}}} characters and use them to build the folds. Then I set foldclose to all. This closes all folds unless the cursor is in them. This way only one fold can be open at a time – or, to put it another way, only one password entry is ever visible at once.
The final fold option instructs Vim when it is allowed to open folds. Folds can always be opened manually, but by default Vim will also open them for many other cases: if you navigate to a fold, jump to a mark within a fold or search for a pattern within a fold, they will open. By setting foldopen to insert I instruct Vim that the only time it should automatically open a fold is if my cursor is in a fold and I change to insert mode. The effect of this is that when I open a file, all folds are closed by default. I can navigate through the file, search and jump through matches, all without opening any of the folds and inadvertently exposing the passwords on my screen. The fold will open if I change to insert mode within it, but it is difficult to do that by mistake.
I have my spacebar setup to toggle folds within Vim. After I have navigated to the desired entry, I can simply whack the spacebar to open it and copy the credential that I need to the system clipboard. At that point I can whack the spacebar again to close the fold, or I can quit Vim. Or I can simply wait.
Locking
The other special option I set is updatetime. Vim uses this option to determine when it should write swap files for crash recovery. Since vim-gnupg disables swap files for decrypted files, this has no effect. I use it for something else.
In the second autocmd I tell Vim to close itself on CursorHold. CursorHold is triggered whenever no key has been pressed for the time specified by updatetime. So the effect of this is that my password files are automatically closed after 1 minute of inactivity. This is similar to KeePassX’s behaviour of “locking the workspace” after a set period of inactivity.
Clipboard
To easily copy a credential to the system clipboard from Vim I have two shortcuts mapped.
12345
" Yank WORD to system clipboard in normal mode
nmap <leader>y "+yE
" Yank selection to system clipboard in visual mode
vmap <leader>y "+y
Selections are “owned” by an application, and disappear when that application (e.g., Vim) exits, thus losing the data, whereas cut-buffers, are stored within the X-server itself and remain until written over or the X-server exits (e.g., upon logging out).
The result is that I can copy a username or password by placing the cursor on its first character and hitting <leader>y. I can paste the credential wherever it is needed. After I close Vim, or after Vim closes itself after 1 minute of inactivity, the credential is removed from the clipboard. This replicates KeePassX’s behaviour of clearing the clipboard so many seconds after a username or password has been copied.
Generation
Passwords should be long and unique. To satisfy this any password manager needs some sort of password generator. Vim provides this with its ability to call and read external commands I can tell Vim to call the standard-issue pwgen program to generate a secure 24-character password utilizing special characters and insert the output at the cursor, like this:
1
:r!pwgen -sy241
Backups
The ~/pw directory is backed up in the same way as most other things on my hard drive: to Tarsnap via Tarsnapper, to an external drive via rsnapshot and cryptshot, rsync to a mirror drive. The issue with these standard backups is that they’re all encrypted and the keys to decrypt them are stored in the password manager. If I loose ~/pw I’ll have plenty of backups around, but none that I can actually access. I address this problem with regular backups to optical media.
At the beginning of every month I burn the password directory to two CDs. One copy is stored at home and the other at an off-site location. I began these optical media backups in December, so I currently have two sets consisting of five discs each. Any one of these discs will provide me with the keys I need to access a backup made with one of the more frequent methods.
Of course, all the files being burned to these discs are still encrypted with my GnuPG key. If I loose that key or passphrase I will have no way to decrypt any of these files. Protecting one’s GnuPG key is another problem entirely. I’ve taken steps that make me feel confident in my ability to always be able to recover a copy of my key, but none that I’m comfortable discussing publicly.
# Set the password database directory.PASSDIR=~/pw
# Create or edit password databases.
pw(){cd"$PASSDIR"if[ ! -z "$1"];then$EDITOR$(buildfile "$1")cd"$OLDPWD"fi}
This allows me to easily open any password file from wherever I am in the filesystem without specifying the full path. These two commands are equivalent, but the one utilizing pw() requires fewer keystrokes:
12
$ vim ~/pw/financial.gpg
$ pw financial
The function changes to the password directory before opening the file so that while I’m in Vim I can drop down to a shell with :sh and already be in the proper directory to manipulate the files. After I close Vim the function returns me to the previous working directory.
This still required a few more keystrokes than I like, so I configured my shell to perform autocompletion in the directory. If financial.gpg is the only file in the directory beginning with an “f”, typing pw f<tab> is all that is required to open the file.
Simplicity
This setup provides simplicity, power, and portability. It uses the same tools that I already employ in my daily life, and does not require the use of the mouse or any graphical windows. I’ve been happily utilizing it for about 6 months now.
Initially I had thought I would supplement the setup with a script that would search the databases for a desired entry, using some combination of grep, awk and cut, and then copy it to my clipboard via xsel. As it turns out, I haven’t felt the desire to do this. Simply opening the file in Vim, searching for the desired entry, opening the fold and copying the credential to the system clipboard is quick enough. The whole process, absent of typing in my passphrase, takes me only a couple of seconds.
Resources
I’m certainly not the first to come up with the idea of managing password with Vim. These resources were particularly useful to me when I was researching the possibilities:
As a long-time user of Unix-like systems, I prefer to do as much work in the command-line as possible. I store data in plain text whenever appropriate. I edit in vim and take advantage of the pipeline to manipulate the data with powerful tools like awk and grep.
Notes are one such instance where plain text makes sense. All of my notes – which are scratch-pads for ideas, reference material, logs, and whatnot – are kept as individual text files in the directory ~/documents/notes. The entire ~/documents directory is synced between my laptop and my work computer, and of course it is backed up with tarsnap.
When I want to read, edit or create a note, my habit is simply to open the file in vim.
1
$ vim ~/documents/notes/todo.txt
When I want to view a list of my notes, I can just ls the directory. I pass along the -t and -r flags. The first flag sorts the files by modification date, newest first. The second flag reverses the order. The result is that the most recently modified files end up at the bottom of the list, nearest the prompt. This allows me to quickly see which notes I have recently created or changed. These notes are generally active – they’re the ones I’m currently doing something with, so they’re the ones I want to see. Using ls to see which files have been most recently modified is incredibly useful, and a behaviour that I use often enough to have created an alias for it.
The first function, n(), takes the name of the note as an argument – minus the file extension – and opens it.
123
function n {
nano ~/Dropbox/Notes/$1.txt
}
I liked the idea. It would allow me to open a note from anywhere in the filesystem without specifying the full path. After changing the editor and the path, I could open the same note as before with far fewer keystrokes.
1
$ n todo
I needed to make a few changes to the function to increase its flexibility.
First, the extension. Most of my notes have .txt extensions. Some have a .gpg extension.1 Some have no extension. I didn’t want to force the .txt extension in the function.
If I specified a file extension, that extension should be used. If I failed to specify the extension, I wanted the function to open the file as I specified it only if it existed. Otherwise, I wanted it to look for that file with a .gpg extension and open that if it was found. As a last resort, I wanted it to open the file with a .txt extension regardless of whether it existed or not. I implemented this behaviour in a separate function, buildfile(), so that I could take advantage of it wherever I wanted.
1 2 3 4 5 6 7 8 9101112131415161718192021
# Take a text file and build it with an extension, if needed.# Prefer gpg extension over txt.
buildfile(){# If an extension was given, use it.if[["$1"== *.* ]];thenecho"$1"# If no extension was given...else# ... try the file without any extension.if[ -e "$1"];thenecho"$1"# ... try the file with a gpg extension.elif[ -e "$1".gpg ];thenecho"$1".gpg
# ... use a txt extension.elseecho"$1".txt
fifi}
I then rewrote the original note function to take advantage of this.
1 2 3 4 5 6 7 8 910111213
# Set the note directory.NOTEDIR=~/documents/notes
# Create or edit notes.
n(){# If no note was given, list the notes.if[ -z "$1"];then
lt "$NOTEDIR"# If a note was given, open it.else$EDITOR$(buildfile "$NOTEDIR"/"$1")fi}
Now I can edit the note ~/documents/notes/todo.txt by simply passing along the filename without an extension.
1
$ n todo
If I want to specify the extension, that will work too.
1
$ n todo.txt
If I have a note called ~/documents/notes/world-domination.gpg, I can edit that without specifying the extension.
1
$ n world-domination
If I have a note called ~/documents/notes/readme, I can edit that and the function will respect the lack of an extension.
1
$ n readme
Finally, if I want to create a note, I can just specify the name of the note and a file with that name will be created with a .txt extension.
The other change I made was to make the function print a reverse-chronologically sorted list of my notes if it was called with no arguments. This allows me to view my notes by typing a single character.
1
$ n
The original blog post also included a function ns() for searching notes by their title.
123
function ns {
ls -c ~/Dropbox/Notes | grep $1}
I thought this was a good idea, but I considered the behaviour to be finding a note rather than searching a note. I renamed the function to reflect its behaviour, took advantage of my ls alias, and made the search case-insensitive. I also modified it so that if no argument was given, it would simply print an ordered list of the notes, just like n().
12345678
# Find a note by title.
nf(){if[ -z "$1"];then
lt "$NOTEDIR"else
lt "$NOTEDIR"| grep -i $1fi}
I thought it would be nice to also have a quick way to search within notes. That was accomplished with ns(), the simplest function of the trio.
12
# Search within notes.
ns(){cd$NOTEDIR; grep -rin $1;cd"$OLDPWD";}
I chose to change into the note directory before searching so that the results do not have the full path prefixed to the filename. Thanks to the first function I don’t need to specify the full path of the note to open it, and this makes the output easier to read.
123
$ ns "take over"
world-domination.txt:1:This is my plan to take over the world.
$ n world-domination
To finish it up, I added completion to zsh so that I can tab-complete filenames when using n.
12
# Set autocompletion for notes.
compctl -W $NOTEDIR -f n
If you’re interested in seeing some of the other way that I personalize my working environment, all of my dotfiles are on GitHub. The note functions are in shellrc, which is my shell-agnostic configuration file that I source from both zshrc and bashrc2.
Notes
↵ I use vim with the gnupg.vim plugin for seamless editing of PGP-encrypted files.
↵ I prefer zsh, but I do still find myself in bash on some machines. I find it prudent to maintain configurations for both shells. There's a lot of cross-over between them and I like to stick to the DRY principle.
A laptop presents some problems for reliably backing up data. Unlike a server, the laptop may not always be turned on. When it is on, it may not be connected to the backup medium. If you’re doing online backups, the laptop may be offline. If you’re backing up to an external drive, the drive may not be plugged in. To address these issues I wrote a shell script called backitup.sh.
The Problem
Let’s say you want to backup a laptop to an external USB drive once per day with cryptshot.
You could add a cron entry to call cryptshot.sh at a certain time every day. What if the laptop isn’t turned on? What if the drive isn’t connected? In either case the backup will not be completed. The machine will then wait a full 24 hours before even attempting the backup again. This could easily result in weeks passing without a successful backup.
If you’re using anacron, or one of its derivatives, things get slightly better. Instead of specifying a time to call cryptshot.sh, you set the cron interval to @daily. If the machine is turned off at whatever time anacron is setup to execute @daily scripts, all of the commands will simply be executed the next time the machine boots. But that still doesn’t solve the problem of the drive not being plugged in.
The Solution
backitup.sh attempts to perform a backup if a certain amount of time has passed. It monitors for a report of successful completion of the backup. Once configured, you no longer call the backup program directly. Instead, you call backitup.sh. It then decides whether or not to actually execute the backup.
How it works
The script is configured with the backup program that should be executed, the period for which you want to complete backups, and the location of a file that holds the timestamp of the last successful backup. It can be configured either by modifying the variables at the top of the script, or by passing in command-line arguments.
$ backitup.sh -h
Usage: backitup.sh [OPTION...]
Note that any command line arguments overwrite variables defined in the source.
Options:
-p the period for which backups should attempt to be executed
(integer seconds or 'DAILY', 'WEEKLY' or 'MONTHLY')
-b the backup command to execute; note that this should be quoted if it contains a space
-l the location of the file that holds the timestamp of the last successful backup.
-n the command to be executed if the above file does not exist
When the script executes, it reads the timestamp contained in the last-run file. This is then compared to the user-specified period. If the difference between the timestamp and the current time is greater than the period, backitup.sh calls the backup program. If the difference between the stored timestamp and the current time is less than the requested period, the script simply exits without running the backup program.
After the backup program completes, the script looks at the returned exit code. If the exit code is 0, the backup was completed successfully, and the timestamp in the last-run file is replaced with the current time. If the backup program returns a non-zero exit code, no changes are made to the last-run file. In this case, the result is that the next time backitup.sh is called it will once again attempt to execute the backup program.
The period can either be specified in seconds or with the strings DAILY, WEEKLY or MONTHLY. The behaviour of DAILY differs from 86400 (24-hours in seconds). With the latter configuration, the backup program will only attempt to execute once per 24-hour period. If DAILY is specified, the backup may be completed successfully at, for example, 23:30 one day and again at 00:15 the following day.
Use
You still want to backup a laptop to an external USB drive once per day with cryptshot. Rather than calling cryptshot.sh, you call backitup.sh.
Tell the script that you wish to complete daily backups, and then use cron to call the script more frequently than the desired backup period. For my local backups, I call backitup.sh every hour.
The default period of backitup.sh is DAILY, so in this case I don’t have to provide a period of my own. But I also do weekly and monthly backups, so I need two more entries to execute cryptshot with those periods.
All three of these entries are executed hourly, which means that at the top of every hour, my laptop attempts to back itself up. As long as the USB drive is plugged in during one of those hours, the backup will complete. If cryptshot is executed, but fails, another attempt will be made the next hour. Daily backups will only be successfully completed, at most, once per day; weekly backups, once per week; and monthly backups, once per month. This setup works well for me, but if you want a higher assurance that your daily backups will be completed every day you could change the cron interval to */5 * * * *, which will result in cron executing backitup.sh every 5 minutes.
What if you want to perform daily online backups with Tarsnapper?
At the top of every hour your laptop will attempt to run Tarsnap via Tarsnapper. If the laptop is offline, it will try again the following hour. If Tarsnap begins but you go offline before it can complete, the backup will be resumed the following hour.
The script can of course be called with something other than cron. Put it in your ~/.profile and have you backups attempt to execute every time you login. Add it to your network manager and have your online backups attempt to execute every time you get online. If you’re using something like udev, have your local backups attempt to execute every time your USB drive is plugged in.
The Special Case
The final configuration option of backitup.sh represents a special case. If the script runs and it can’t find the specified file, the default behaviour is to assume that this is the first time it has ever run: it creates the file and executes the backup. That is what most users will want, but this behaviour can be changed.
When I first wrote backitup.sh it was to help manage backups of my Dropbox folder. Dropbox doesn’t provide support client-side encryption, which means users need to handle encryption themselves. The most common way to do this is to create an encfs file-system or two and place those within the Dropbox directory. That’s the way I use Dropbox.
I wanted to backup all the data stored in Dropbox with Tarsnap. Unlike Dropbox, Tarsnap does do client-side encryption, so when I backup my Dropbox folder, I don’t want to actually backup the encrypted contents of the folder – I want to backup the decrypted contents. That allows me to take better advantage of Tarsnap’s deduplication and it makes restoring backups much simpler. Rather than comparing inodes and restoring a file using an encrypted filename like 6,8xHZgiIGN0vbDTBGw6w3lf/1nvj1,SSuiYY0qoYh-of5YX8 I can just restore documents/todo.txt.
If my encfs filesystem mount point is ~/documents, I can configure Tarsnapper to create an archive of that directory, but if for some reason the filesystem is not mounted when Tarsnapper is called, I would be making a backup of an empty directory. That’s a waste of time. The solution is to tell backitup.sh to put the last-run file inside the encfs filesystem. If it can’t find the file, that means that the filesystem isn’t mounted. If that’s the case, I tell it to call the script I use to automatically mount the encfs filesystem (which, the way I have it setup, requires no interaction from me).
backitup.sh solves all of my backup scheduling problems. I only call backup programs directly if I want to make an on-demand backup. All of my automated backups go through backitup.sh. If you’re interested in the script, you can download it directly from GitHub. You can clone my entire backups repository if you’re also interested in the other scripts I’ve written to manage different aspects of backing up data.
Earlier this year I switched from Duplicity to rsnapshot for my local backups. Duplicity uses a full + incremental backup schema: the first time a backup is executed, all files are copied to the backup medium. Successive backups copy only the deltas of changed objects. Over time this results in a chain of deltas that need to be replayed when restoring from a backup. If a single delta is somehow corrupted, the whole chain is broke. To minimize the chances of this happening, the common practice is to complete a new full backup every so often – I usually do a full backup every 3 or 4 weeks. Completing a full backup takes time when you’re backing up hundreds of gigabytes, even over USB 3.0. It also takes up disk space. I keep around two full backups when using Duplicity, which means I’m using a little over twice as much space on the backup medium as what I’m backing up.
The backup schema that rsnapshot uses is different. The first time it runs, it completes a full backup. Each time after that, it completes what could be considered a “full” backup, but unchanged files are not copied over. Instead, rsnapshot simply hard links to the previously copied file. If you modify very large files regularly, this model may be inefficient, but for me – and I think for most users – it’s great. Backups are speedy, disk space usage on the backup medium isn’t too much more than the data being backed up, and I have multiple full backups that I can restore from.
The great strength of Duplicity – and the great weakness of rsnapshot – is encryption. Duplicity uses GnuPG to encrypt backups, which makes it one of the few solutions appropriate for remote backups. In contrast, rsnapshot does no encryption. That makes it completely inappropriate for remote backups, but the shortcoming can be worked around when backing up locally.
My local backups are done to an external, USB hard drive. Encrypting the drive is simple with LUKS and dm-crypt. For example, to encrypt /dev/sdb:
At this point, the drive will be encrypted with a passphrase. To make it easier to mount programatically, I also add a key file full of some random data generated from /dev/urandom.
There are still a few considerations to address before backups to this encrypted drive can be completed automatically with no user interaction. Since the target is a USB drive and the source is a laptop, there’s a good chance that the drive won’t be plugged in when the scheduler kicks in the backup program. If it is plugged in, the drive needs to be decrypted before calling rsnapshot to do its thing. I wrote a wrapper script called cryptshot to address these issues.
Cryptshot is configured with the UUID of the target drive and the key file used to decrypt the drive. When it is executed, the first thing it does is look to see if the UUID exists. If it does, that means the drive is plugged in and accessible. The script then decrypts the drive with the specified key file and mounts it. Finally, rsnapshot is called to execute the backup as usual. Any argument passed to cryptshot is passed along to rsnapshot. What that means is that cryptshot becomes a drop-in replacement for encrypted, rsnapshot backups. Where I previously called rsnapshot daily, I now call cryptshot daily. Everything after that point just works, with no interaction needed from me.
If you’re interested in cryptshot, you can download it directly from GitHub. The script could easily be modified to execute a backup program other than rsnapshot. You can clone my entire backups repository if you’re also interested in the other scripts I’ve written to manage different aspects of backing up data.
Three years ago I wrote a guide to building a VPS web server for serving sites in a PHP environment. That setup served me well for some time, but most of the sites I run now – including this one – are now written in Python. Earlier this year I built another web server to reflect this. It’s similar to before; I still use Ubuntu and I still like to serve pages with nginx. But PHP has been replaced with Python, and many of the packages used to build the environment have changed as a result. As with the last time, I decided to compile my notes into a guide, both for my own reference and in case anyone else would like to duplicate it. So far, the server has proven to be fast and efficient. It serves Python using uWSGI, uses a PostgreSQL database, and includes a simple mail server provided by Postfix. I think it’s a good setup for serving simple Django-based websites.
Basic Setup
As with last time, I recommend following Slicehost’s basic server setup article. It discusses user administration, SSH security, and firewalls. I no longer use Slicehost as my VPS provider, but I find that Slicehost’s articles provide an excellent base regardless of the host.
Packages
Packages should upgraded immediately to address any known security vulnerabilities.
12
$ sudo apt-get update
$ sudo apt-get upgrade
After the repositories have been updated, I install some essential packages.
Build-essential includes necessary tools to compile programs. I am incapable of using a computer that does not have screen on it, so that gets installed too. The third package, dnsutils, is optional, but includes dig which is useful for troubleshooting DNS issues.
DenyHosts
Slicehost’s setup article recommends turning off password authentication in SSH, forcing users to login with keys only. I use keys whenever I can, but I appreciate the option of being able to login to my server from any computer, when I may or may not have my SSH key with me. So I leave password authentication enabled. This presents the possibility of brute-force attacks. Enter DenyHosts. DenyHosts, which I have discussed previously attempts to protect against SSH attacks by banning hosts after a certain number of failed login attempts. When password authentication is enabled, running DenyHosts is a smart move.
12
$ sudo apt-get install denyhosts
$ sudo vim /etc/denyhosts.conf
Personalize the Environment
I use update-alternatives to set my default editor to vim.
1
$ sudo update-alternatives --config editor
All of my personal configuration files are kept in a github repository. I’ll check out that repository into ~/src and install the files.
Later on I’ll be installing some Python development packages. One of them creates a directory called /usr/lib/pymodules/python2.6/.path, which sets off a warning in chkrootkit. Part of chkrootkit’s desgin philosophy is to not include any whitelists: if chkrootkit finds something that it doesn’t like, you’re going to hear about it. I have cron run chkrootkit nightly and I want to receive any warnings, but I don’t want to receive the same false positive every morning in my inbox.
The solution is to create a file that contains chkrootkit’s warning. I call that file whitelist and store it in the same directory as chkrootkit. When chkrootkit is run, any output is redirected to a file. That file is compared to the whitelist using diff and the output of that – if any – is then read. At the end, the file containing chkrootkit’s output is deleted so that the working directory is ready for the next run. The effect is that I only hear warnings from chkrootkit that I have not explicit whitelisted. All of this can be accomplished in a single crontab entry.
The script that my cronjob runs is slightly different from the one demonstrated in the Slicehost article. Their script executes a few commands, groups the output together, and sends it to mail to email the system administrator. This results in daily emails, regardless of whether rkhunter finds any warnings or not. My script is simpler and does not result in so many messages.
The version check and update commands both have the -q switch, which disables any output – I don’t care to know whether rkhunter updated itself or not. The final line actually executes the scan. Notice that there’s no reference to mail. This script does not send any messages. The reason for that is that rkhunter itself provides the mail functionality. Inside of /etc/rkhunter.conf there is a MAIL-ON-WARNING variable. As long as the machine has an smtp server on it (which I’ll get to later in this guide), simply filling in this variable will result in any warnings being emailed to the system administrator.
Web Server
With the basics complete, it’s time to start serving something! In my previous article I covered serving a PHP-based Wordpress site via FastCGI and nginx. This time around the stack will be different: nginx, uWSGI, Python, and Django.
A few basic packages will help flesh out the server’s Python development environment:
Installing uWSGI is a simple matter of compiling it and moving the resulting binary into one of your system’s bin directories.
123456
$ cd ~/src/
$ wget http://projects.unbit.it/downloads/uwsgi-0.9.8.tar.gz
$ tar xvzf ~/uwsgi-0.9.8.tar.gz
$ cd uwsgi-0.9.8/
$ make -f Makefile.Py26
$ sudo cp uwsgi /usr/local/sbin
nginx
The nginx package in Ubuntu’s official repositories is always notoriously outdated. It used to be you had to compile the server from source, but there is now an Ubuntu PPA for the latest stable versions. As described by the nginx wiki, all that is needed is to add the PPA to your sources.list and apt-get away!
If you do Python development and haven’t heard of virtualenv, it is well worth reading up on. It allows the user to create an isolated, virtual Python environment for each project. This helps immensely when developing (or serving) multiple projects on a single machine. Needless to say, I consider it to be a required package.
Install
I’ll be installing virtualenv and virtualenvwrapper (a set of scripts to facilitate working with virtual environments). I also prefer pip over easy_install for managing Python packages.
Virtual environments can be stored wherever you fancy. For now, I keep them in a hidden folder in my home directory. For these examples, I’ll setup an environment called myproject.
Notice the --no-site-packages switch. That tells virtualenv to create this environment without any of the Python packages already installed, creating a completely fresh, clean environment. The --distribute switch causes the new virtual environment to be setup with distribute, a replacement for the old and rather broken setuptools.
All that’s needed to get virtualenvwrapper up and running is to add two lines to your .bashrc and re-source the file.
123456
$ vim ~/.bashrc
exportWORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh
$ . ~/.bashrc
We can now use commands like workon to ease the process of activating a certain environment.
I’ll go ahead and install yolk in the environment to help manage packages.
123
$ workon myproject
$ pip install yolk
$ yolk -l
The last command will cause yolk to list all packages installed in the environment. Try deactivating the environment and then running yolk again.
123
$ deactivate
$ yolk -l
yolk: command not found
‘yolk’ wasn’t found, because it was only installed within the virtual environment. Neat!
Install Django
Finally, it’s time to install Django! The process is simple enough.
12
$ workon myproject
$ pip install django
And that’s it!
The Python Imaging Library is likely to be needed for any Django project. I installed it in the beginning of this section, but because I used the --no-site-packages when creating my virtual environment, it is not available for use within the project. To fix that, I’ll just link the package in. I also previously installed psyopg2, which Python will need to communicate with my PostgreSQL database, so I’ll link that in as well. psyopg2 depends on mx, which was also previously installed but still must be made available in the environment.
While I’m still in the virtual environment, I’ll go ahead and create a new Django project. The project will have the same name as the environment: myproject. For this tutorial, I’ll stick with the precedence set by the Slicehost tutorials and use demo as the name of both my user and group on the server.
I like to keep my sites in the /srv/ directory. I structure them so that the code that runs the site, any public files, logs, and backups are all stored in separate sub-directories.
123456789
$ cd /srv/
$ sudo mkdir -p myproject.com/{code,public,logs,backup}
$ sudo mkdir -p myproject.com/public/{media,static}
$ sudo chown -R demo:demo myproject.com
$ cd myproject.com
$ sudo chown -R :www-data logs public
$ sudo chmod -R g+w logs public
$ cd code/
$ django-admin.py startproject myproject
Notice that the logs and public directories were chowned to the www-data group. That is the name of the user and group that nginx will run as. The web server will need permissions to write to those locations.
Save Requirements
With the environment setup and all the necessary packages installed, now is a good time to tell pip to freeze all the packages and their versions. I keep this file in a deploy folder in my project.
Now that I have something to serve, I’ll configure uWSGI to serve it. The first step is to create a configuration file for the project. I call mine wsgi.py and store it in /srv/myproject.com/code/myproject/. It appends the current directory to the Python path, specifies the Django settings file for the project, and registers the WSGI handler.
With that done, the next step is to decide how uWSGI should be run. I’m going to use Ubuntu’s upstart to supervise the service. I keep the upstart script in my project’s deploy/ directory.
1 2 3 4 5 6 7 8 9101112131415161718
$ vim /srv/myproject.com/code/deploy/uwsgi.conf
description "uWSGI server for My Project"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
exec /usr/local/sbin/uwsgi \
--home /home/demo/.virtualenvs/myproject/ \
--socket /var/run/myproject.com.sock \
--chmod-socket \
--pythonpath /srv/myproject.com/code/ \
--module myproject.wsgi \
--process 2\
--harakiri 30\
--master \
--logto /srv/myproject.com/logs/uwsgi.log
Sadly, upstart doesn’t seem to recognize links. Rather than linking the config file into /etc/init/, I have to copy it.
Nginx’s configuration is pretty straight-forward. If you’ve never configured the server before, Slicehost’s articles can set you down the right path. My own nginx config looks something like this:
I keep the virtual host config for my project inside the project’s code/deploy/ directory. A basic virtual host for a Django project would looks like this:
To install and enable the virtual host, I’ll link the configuration file first to the nginx sites-available directory, and then link that link to the sites-enabled directory.
Django has a very good built-in cache framework. I like to take advantage of it with a memory-based backend: namely, memcached. It’s fast, efficient, and easy to setup.
All that’s needed is to install memcached on the server, followed by the Python API python-memcached.
The default configuration file in Ubuntu lives at /etc/memcached.conf. I usually stick with the defaults, but sometimes end up changing the port that memchached runs on or the amount of memory it is allowed to use.
I maintain a configuration file for each of the domains being served by the machine. The file for a domain lives in – you guessed it – the associated project’s deploy/ folder. Each contains two entries: one for the nginx virtual host and one for the uWSGI instance. The reason for this is that each config block needs a postrotate section to restart the associated server after the logs have been rotated. I don’t want nginx to be restarted everytime a uWSGI log is rotated, and I don’t want uWSGI restarted everytime an nginx log is rotated.
A web server isn’t much use without a database these days. I use PostgreSQL.
Install
1
$ sudo apt-get install postgresql
Configure
PostgreSQL has some unique terminology and ways of doing things. When I first set it up for the first time, having coming from a MySQL background, not everything was completely straightforward. As usual, Slicehost has a number of articles that will provide a foundation.
In the /etc/postgresql/8.4/main/postgresql.conf file, I uncomment the following two lines:
track_counts = on
autovacuum = on
Then restart the database server.
1
$ sudo /etc/init.d/postgresql-8.4 restart
After that I’ll change the password for the postgres user and the postgres database role.
If I’m restoring a previous database from a backup, now would be the time to import the backup.
1
$ psql -U myproject < myproject.postgresql
And now Django should be able to connect!
The basic server is setup and secure. Django, uWSGI, nginx and PostgreSQL are all running and getting along swimmingly. At this point, many people would be done, but I also like to have a minimal mail server.
Mail Server
Most of my domains use Google Apps, so I don’t need a full-blown mail server. I do want programs and scripts to be able to send mail, and I prefer not to do so through an external SMTP server – I’d rather just deal with having sendmail running on my own box. And I do have a few domains that do not use Google Apps. They have one or two aliases associated with them, so the server needs to receive messages for those domains and forward them off to an external address. If any of this sounds vaguely familiar, it’s because it’s the same thing I detailed last time. My setup now is the same as then, so I won’t repeat any of it here.
I use Git to keep track of the code for all my projects. (If you’re new to Git, you ought to skim the Git Reference or Everyday GIT With 20 Commands Or So). To manage websites, I create a repository of the directory with the code that runs the site (in this case, /srv/myproject.com/code/) and another empty, bare repository to work as a hub. With a post-update and post-commit, the end result is an excellent web workflow:
A copy of the hub can be checked out on a local machine for development. Whenever a change is committed, a simple git push will push the code to the web server and automatically make it live.
Changes can be made on the server in the actual live website directory. (This is not a best practice, but I do it more often than I should probably admit.) Whenever a change is committed, it is automatically pushed to the hub, so that a simple git pull is all that’s needed on the development machine to update its repository.
A more detailed explanation of this workflow is at Joe Maller’s blog.
To start, I need to create a repository for the new project I created in this tutorial. And, since this is a new server, I need to give Git my name and email address to record with every commit.
Before adding the files, I create a .gitignore file in the repository root to tell Git to ignore compiled Python files.
$ vim .gitignore
*.pyc
Now I add all the files to the repository, confirm that it worked, and commit the files.
123
$ git add .
$ git status -s
$ git commit -m "Initial commit of myproject.com"
I create the bare hub directory directly along side the projects code/.
1234
$ cd ../
$ mkdir hub.git
$ cd hub.git
$ git --bare init
With the hub created, I need to add it as the remote for the main repository and push the master branch to it.
1234
$ cd ../code
$ git remote add hub /srv/myproject.com/hub.git
$ git remote show hub
$ git push hub master
Now the hub needs a post-update script so that every time something is pushed to it, that change is automagically pulled into the live website directory.
And the live website directory requires a post-commit script so that every time something is committed inside of it, that change is automagically pushed to the hub.
1 2 3 4 5 6 7 8 910
$ vim /srv/myproject.com/code/.git/hooks/post-commit
#!/bin/shechoecho"**** pushing changes to Hub"echo
git push hub
$ chmod +x /srv/myproject.com/code/.git/hooks/post-commit
All that’s left is to check out the hub onto the development machine – my laptop, in this case!
To test things out, we can add a file to the repository on the development machine.
12345
$ cd code/
$ touch test
$ git add test
$ git commit -m "A test"
$ git push
Now go back to the server, and the file should be there! To test things the other way around, I’ll delete the file from the live repository.
12345
$ cd /srv/myproject.com/code/
$ ls
myproject test
$ git rm test
$ git commit -m "Removing the test file"
And once again to the development machine:
123
$ git pull
$ ls
deploy myproject
No more test! It’s pretty dandy.
Restoring
If I was building a new server and restoring a project from an old server, I would simply mirror the old hub and then clone that in the live directory.
Prior to building this server, I was new to a lot of this – particularly, uWSGI and virtualenv. The following tutorials helped me a good deal in putting together the perfect setup for my needs.
Slicehost has an excellent article repository, containing guides on a number of subjects. After building a fresh Slice, you should first follow Part 1 and Part 2 of Slicehost’s basic setup articles.
I use slightly different coloring in my bash prompt, so, rather than what Slicehost suggests in their article, I add the following to ~/.bashrc:
This is a good time to protect SSH by installing DenyHosts, which I discuss here:
1
$ sudo aptitude install denyhosts
Ubuntu’s default text editor is nano, which I abhor. Real men use vim. Ubuntu comes with a slimmed down version of vim, but you’ll probably want the full version:
1
$ sudo aptitude install vim
To change the global default editor variable, execute the following and select the editor of your choice:
1
$ sudo update-alternatives --config editor
This is also a perfect time to install GNU Screen.
# Print a pretty line at the bottom of the screen
hardstatus alwayslastline
hardstatus string '%{= kG}[ %{G}%H %{g}][%= %{=kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{Y}%Y-%m-%d %{W}%c %{g}]'# Nobody likes startup messages
startup_message off
# Turn visual bell on and set the message to display for only a fraction of a second
vbell on
vbellwait .3
# Set default shell title to blank
shelltitle ''# Gimme my scrollback!
defscrollback 5000# Change command character to backtick
escape ``# Stop programs (like vim) from leaving their contents# in the window after they exit
altscreen on
# Default screens
screen -t shell 0
I prefer to have my bash profile setup to connect me to Screen as soon as I login. If there are no running sessions, it will create one. If there is a current session, it will disconnect the session from wherever it is connected and connect it to my login. When I disconnect from screen, it automatically logs me out. To achieve this, I add the following to ~/.bashrc:
12345
# If possible, reattach to an existing session and detach that session# elsewhere. If not possible, create a new session.if[ -z "$STY"];thenexec screen -dR
fi
