pig-monkey.com - python

Tarsnapper: Managing Tarsnap Backups

2012-09-16T00:00:00-07:00

Tarsnap bills itself as “online backups for the truly paranoid”. I began using the service last January. It fast became my preferred way to backup to the cloud. It stores data on Amazon S3 and costs $0.30 per GB per month for storage and $0.30 per GB for bandwidth. Those prices are higher than just using Amazon S3 directly, but Tarsnap implements some impressive data de-duplication and compression that results in the service costing very little. For example, I currently have 67 different archives stored in Tarsnap from my laptop. They total 46GB in size. De-duplicated that comes out to 1.9GB. After compression, I only pay to store 1.4GB. Peanuts.

Of course, the primary requirement for any online backup service is encryption. Tarsnap delivers. And, most importantly, the Tarsnap client is open-source, so the claims of encryption can actually be verified by the user. The majority of for-profit, online backup services out there fail on this critical point.

So Tarsnap is amazing and you should use it. The client follows the Unix philosophy: “do one thing and do it well”. It’s basically like tar. It can create archives, read the contents of an archive, extract archives, and delete archives. For someone coming from an application like Duplicity, the disadvantage to the Tarsnap client is that it doesn’t include any way to automatically manage backups. You can’t tell Tarsnap how many copies of a backup you wish to keep, or how long backups should be allowed to age before deletion.

Thanks to the de-duplication and compression, there’s not a great economic incentive to not keep old backups around. It likely won’t cost you that much extra. But I like to keep things clean and minimal. If I haven’t used an online backup in 4 weeks, I generally consider it stale and have no further use for it.

To manage my Tarsnap backups, I wrote a Python script called Tarsnapper. The primary intent was to create a script that would automatically delete old archives. It does this by accepting a maximum age from the user. Whenever Tarsnapper runs, it gets a list of all Tarsnap archives. The timestamp is parsed out from the list and any archive that has a timestamp greater than the maximum allowed age is deleted. This is seamless, and means I never need to manually intervene to clean my archives.

Tarsnapper also provides some help for creating Tarsnap archives. It allows the user to define any number of named archives and the directories that those archives should contain. On my laptop I have four different directories that I backup with Tarsnap, three of them in one archive and the last in another archive. Tarsnapper knows about this, so whenever I want to backup to Tarsnap I just call a single command.

Tarsnapper also can automatically add a suffix to the end of each archive name. This makes it easier to know which archive is which when you are looking at a list. By default, the suffix is the current date and time.

Configuring Tarsnapper can be done either directly by changing the variables at the top of the script, or by creating a configuration file named tarsnapper.conf in your home directory. The config file on my laptop looks like this:

[Settings]
tarsnap: /usr/bin/tarsnap

[Archives]
nous-cloud: /home/pigmonkey/work /home/pigmonkey/documents /home/pigmonkey/vault/
nous-config: /home/pigmonkey/.config

There is also support for command-line arguments to specify the location of the configuration file to use, to delete old archives and exit without creating new archives, and to execute only a single named-archive rather than all of those that you may have defined.

$ tarsnapper.py --help
usage: tarsnapper.py [-h] [-c CONFIG] [-a ARCHIVE] [-r]

A Python script to manage Tarsnap archives.

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG, --config CONFIG
                        Specify the configuration file to use.
  -a ARCHIVE, --archive ARCHIVE
                        Specify a named archive to execute.
  -r, --remove          Remove archives old archives and exit.

It makes using a great service very simple. My backups can all be executed simply by a single call to Tarsnapper. Stale archives are deleted, saving me precious picodollars. I use this system on my laptop, as well as multiple servers. If you’re interested in it, Tarsnapper can be downloaded directly from GitHub. You can clone my entire backups repository if you’re also interested in the other scripts I’ve written to manage different aspects of backing up data.

An Ubuntu VPS for Django

2011-07-19T00:00:00-07:00

Three years ago I wrote a guide to building a VPS web server for serving sites in a PHP environment. That setup served me well for some time, but most of the sites I run now – including this one – are now written in Python. Earlier this year I built another web server to reflect this. It’s similar to before; I still use Ubuntu and I still like to serve pages with nginx. But PHP has been replaced with Python, and many of the packages used to build the environment have changed as a result. As with the last time, I decided to compile my notes into a guide, both for my own reference and in case anyone else would like to duplicate it. So far, the server has proven to be fast and efficient. It serves Python using uWSGI, uses a PostgreSQL database, and includes a simple mail server provided by Postfix. I think it’s a good setup for serving simple Django-based websites.

Basic Setup

As with last time, I recommend following Slicehost’s basic server setup article. It discusses user administration, SSH security, and firewalls. I no longer use Slicehost as my VPS provider, but I find that Slicehost’s articles provide an excellent base regardless of the host.

Packages

Packages should upgraded immediately to address any known security vulnerabilities.

$ sudo apt-get update
$ sudo apt-get upgrade

After the repositories have been updated, I install some essential packages.

1	`$ sudo apt-get install build-essential screen dnsutils`

Build-essential includes necessary tools to compile programs. I am incapable of using a computer that does not have screen on it, so that gets installed too. The third package, dnsutils, is optional, but includes dig which is useful for troubleshooting DNS issues.

DenyHosts

Slicehost’s setup article recommends turning off password authentication in SSH, forcing users to login with keys only. I use keys whenever I can, but I appreciate the option of being able to login to my server from any computer, when I may or may not have my SSH key with me. So I leave password authentication enabled. This presents the possibility of brute-force attacks. Enter DenyHosts. DenyHosts, which I have discussed previously attempts to protect against SSH attacks by banning hosts after a certain number of failed login attempts. When password authentication is enabled, running DenyHosts is a smart move.

$ sudo apt-get install denyhosts
$ sudo vim /etc/denyhosts.conf

Personalize the Environment

I use update-alternatives to set my default editor to vim.

1	`$ sudo update-alternatives --config editor`

All of my personal configuration files are kept in a github repository. I’ll check out that repository into ~/src and install the files.

$ mkdir ~/src
$ cd ~/src
$ sudo apt-get install git-core
$ git clone git://github.com/pigmonkey/dotfiles.git
$ ln -s ~/src/dotfiles/bash_profile ~/.bash_profile
$ ln -s ~/src/dotfiles/bashrc ~/.bashrc
$ ln -s ~/src/dotfiles/bash_aliases ~/.bash_aliases
$ ln -s ~/src/dotfiles/bash_colors ~/.bash_colors
$ ln -s ~/src/dotfiles/vimrc ~/.vimrc
$ ln -s ~/src/dotfiles/vim ~/.vim
$ ln -s ~/src/dotfiles/screenrc ~/.screenrc
$ source ~/.bash_profile

Time

The next step is to set the server’s timezone and sync the clock with NTP.

$ sudo dpkg-reconfigure tzdata
$ sudo apt-get install ntp

Rootkits

There’s no reason not to run both chkrootkit and rkhunter to check for rootkits and related vulnerabilities.

chrkrootkit

Slicehost has an excellent article for setting up and using chkrootkit.

Later on I’ll be installing some Python development packages. One of them creates a directory called /usr/lib/pymodules/python2.6/.path, which sets off a warning in chkrootkit. Part of chkrootkit’s desgin philosophy is to not include any whitelists: if chkrootkit finds something that it doesn’t like, you’re going to hear about it. I have cron run chkrootkit nightly and I want to receive any warnings, but I don’t want to receive the same false positive every morning in my inbox.

The solution is to create a file that contains chkrootkit’s warning. I call that file whitelist and store it in the same directory as chkrootkit. When chkrootkit is run, any output is redirected to a file. That file is compared to the whitelist using diff and the output of that – if any – is then read. At the end, the file containing chkrootkit’s output is deleted so that the working directory is ready for the next run. The effect is that I only hear warnings from chkrootkit that I have not explicit whitelisted. All of this can be accomplished in a single crontab entry.

0 3 * * * (cd /home/demo/src/chkrootkit-0.49; ./chkrootkit -q > message 2>&1; diff -w whitelist message; rm -f message)

rkhunter

I’m sure it doesn’t surprise you that I’m going to recommend reading Slicehost’s article on rkhunter.

Unlike chkrootkit, rkhunter does allow for whitelists. On a clean Ubuntu 10.04 system, I find that I need to whitelist a few items in the config.

$ sudo vim /etc/rkhunter.conf

SCRIPTWHITELIST="/usr/sbin/adduser /usr/bin/ldd /bin/which"
ALLOWHIDDENDIR="/dev/.udev /dev/.initramfs"
APP_WHITELIST="openssl:0.9.8k gpg:1.4.10"

$ sudo /usr/local/bin/rkhunter --propupd

The script that my cronjob runs is slightly different from the one demonstrated in the Slicehost article. Their script executes a few commands, groups the output together, and sends it to mail to email the system administrator. This results in daily emails, regardless of whether rkhunter finds any warnings or not. My script is simpler and does not result in so many messages.

#!/bin/sh
/usr/local/bin/rkhunter --versioncheck -q
/usr/local/bin/rkhunter --update -q
/usr/local/bin/rkhunter --cronjob --report-warnings-only

The version check and update commands both have the -q switch, which disables any output – I don’t care to know whether rkhunter updated itself or not. The final line actually executes the scan. Notice that there’s no reference to mail. This script does not send any messages. The reason for that is that rkhunter itself provides the mail functionality. Inside of /etc/rkhunter.conf there is a MAIL-ON-WARNING variable. As long as the machine has an smtp server on it (which I’ll get to later in this guide), simply filling in this variable will result in any warnings being emailed to the system administrator.

Web Server

With the basics complete, it’s time to start serving something! In my previous article I covered serving a PHP-based Wordpress site via FastCGI and nginx. This time around the stack will be different: nginx, uWSGI, Python, and Django.

A few basic packages will help flesh out the server’s Python development environment:

$ sudo apt-get install python-psycopg2 python-setuptools python2.6-dev psmisc python-imaging locate python-dateutil libxml2-dev python-software-properties

uWSGI

Installing uWSGI is a simple matter of compiling it and moving the resulting binary into one of your system’s bin directories.

$ cd ~/src/
$ wget http://projects.unbit.it/downloads/uwsgi-0.9.8.tar.gz
$ tar xvzf ~/uwsgi-0.9.8.tar.gz
$ cd uwsgi-0.9.8/
$ make -f Makefile.Py26
$ sudo cp uwsgi /usr/local/sbin

nginx

The nginx package in Ubuntu’s official repositories is always notoriously outdated. It used to be you had to compile the server from source, but there is now an Ubuntu PPA for the latest stable versions. As described by the nginx wiki, all that is needed is to add the PPA to your sources.list and apt-get away!

$ sudo add-apt-repository ppa:nginx/stable
$ sudo apt-get update 
$ sudo apt-get install nginx

Python and Django

If you do Python development and haven’t heard of virtualenv, it is well worth reading up on. It allows the user to create an isolated, virtual Python environment for each project. This helps immensely when developing (or serving) multiple projects on a single machine. Needless to say, I consider it to be a required package.

Install

I’ll be installing virtualenv and virtualenvwrapper (a set of scripts to facilitate working with virtual environments). I also prefer pip over easy_install for managing Python packages.

$ sudo easy_install -U pip 
$ sudo pip install virtualenv
$ sudo pip install virtualenvwrapper

Setup a virtual environment

Virtual environments can be stored wherever you fancy. For now, I keep them in a hidden folder in my home directory. For these examples, I’ll setup an environment called myproject.

$ mkdir ~/.virtualenvs
$ cd ~/.virtualenvs
$ virtualenv --no-site-packages --distribute myproject

Notice the --no-site-packages switch. That tells virtualenv to create this environment without any of the Python packages already installed, creating a completely fresh, clean environment. The --distribute switch causes the new virtual environment to be setup with distribute, a replacement for the old and rather broken setuptools.

All that’s needed to get virtualenvwrapper up and running is to add two lines to your .bashrc and re-source the file.

$ vim ~/.bashrc

export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh

$ . ~/.bashrc

We can now use commands like workon to ease the process of activating a certain environment.

I’ll go ahead and install yolk in the environment to help manage packages.

$ workon myproject
$ pip install yolk
$ yolk -l

The last command will cause yolk to list all packages installed in the environment. Try deactivating the environment and then running yolk again.

$ deactivate
$ yolk -l
yolk: command not found

‘yolk’ wasn’t found, because it was only installed within the virtual environment. Neat!

Install Django

Finally, it’s time to install Django! The process is simple enough.

$ workon myproject
$ pip install django

And that’s it!

The Python Imaging Library is likely to be needed for any Django project. I installed it in the beginning of this section, but because I used the --no-site-packages when creating my virtual environment, it is not available for use within the project. To fix that, I’ll just link the package in. I also previously installed psyopg2, which Python will need to communicate with my PostgreSQL database, so I’ll link that in as well. psyopg2 depends on mx, which was also previously installed but still must be made available in the environment.

$ cdsitepackages
$ ln -s /usr/lib/python2.6/dist-packages/PIL
$ ln -s /usr/lib/python2.6/dist-packages/PIL.pth
$ ln -s /usr/lib/python2.6/dist-packages/psycopg2
$ ln -s /usr/lib/python2.6/dist-packages/mx

That wasn’t too painful!

Create a Django project

While I’m still in the virtual environment, I’ll go ahead and create a new Django project. The project will have the same name as the environment: myproject. For this tutorial, I’ll stick with the precedence set by the Slicehost tutorials and use demo as the name of both my user and group on the server.

I like to keep my sites in the /srv/ directory. I structure them so that the code that runs the site, any public files, logs, and backups are all stored in separate sub-directories.

$ cd /srv/
$ sudo mkdir -p myproject.com/{code,public,logs,backup}
$ sudo mkdir -p myproject.com/public/{media,static}
$ sudo chown -R demo:demo myproject.com
$ cd myproject.com
$ sudo chown -R :www-data logs public
$ sudo chmod -R g+w logs public
$ cd code/
$ django-admin.py startproject myproject

Notice that the logs and public directories were chowned to the www-data group. That is the name of the user and group that nginx will run as. The web server will need permissions to write to those locations.

Save Requirements

With the environment setup and all the necessary packages installed, now is a good time to tell pip to freeze all the packages and their versions. I keep this file in a deploy folder in my project.

$ mkdir /srv/myproject.com/code/deploy
$ pip freeze > /srv/myproject.com/code/deploy/requirements.txt

Now, if I needed to recreate the virtual environment somewhere else, I could just tell pip to install all the packages from that file.

1	`$ pip install -r /srv/myproject.com/code/deploy/requirements.txt`

Configure uWSGI

Now that I have something to serve, I’ll configure uWSGI to serve it. The first step is to create a configuration file for the project. I call mine wsgi.py and store it in /srv/myproject.com/code/myproject/. It appends the current directory to the Python path, specifies the Django settings file for the project, and registers the WSGI handler.

import sys
import os

sys.path.append(os.path.abspath(os.path.dirname(__file__)))
os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings'

import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()

With that done, the next step is to decide how uWSGI should be run. I’m going to use Ubuntu’s upstart to supervise the service. I keep the upstart script in my project’s deploy/ directory.

$ vim /srv/myproject.com/code/deploy/uwsgi.conf

description "uWSGI server for My Project"

start on runlevel [2345]
stop on runlevel [!2345]

respawn
exec /usr/local/sbin/uwsgi \
--home /home/demo/.virtualenvs/myproject/ \
--socket /var/run/myproject.com.sock \
--chmod-socket \
--pythonpath /srv/myproject.com/code/ \
--module myproject.wsgi \
--process 2 \
--harakiri 30 \
--master \
--logto /srv/myproject.com/logs/uwsgi.log

Sadly, upstart doesn’t seem to recognize links. Rather than linking the config file into /etc/init/, I have to copy it.

$ sudo cp /srv/myproject.com/code/deploy/uwsgi.conf /etc/init/uwsgi-myproject.conf

Configure nginx

Nginx’s configuration is pretty straight-forward. If you’ve never configured the server before, Slicehost’s articles can set you down the right path. My own nginx config looks something like this:

user www-data www-data;
worker_processes 4;
pid /var/run/nginx.pid;

events {
        worker_connections 768;
        use epoll;
}

http {

    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 30;
    types_hash_max_size 2048;
    # server_tokens off;

    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # Logging Settings
    ##

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    ##
    # Gzip Settings
    ##

    gzip on;
    gzip_disable "msie6";
    gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
    gzip_proxied any;
    gzip_comp_level 2;

    # gzip_vary on;
    # gzip_buffers 16 8k;
    # gzip_http_version 1.1;

    ##
    # Virtual Host Configs
    ##

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

I keep the virtual host config for my project inside the project’s code/deploy/ directory. A basic virtual host for a Django project would looks like this:

server {
    listen 80;
    server_name www.myproject.com;
    rewrite ^/(.*) http://myproject.com/$1 permanent;
}

server {
    listen 80;
    server_name myproject.com;

    access_log /srv/myproject.com/logs/access.log;
    error_log /srv/myproject.com/logs/error.log;

    location /media {
        root /srv/myproject.com/public/;
    }

    location /static {
        root /srv/myproject.com/public/;
    }

    location / {
        uwsgi_pass unix:///var/run/myproject.com.sock;
        include uwsgi_params;
    }
}

To install and enable the virtual host, I’ll link the configuration file first to the nginx sites-available directory, and then link that link to the sites-enabled directory.

$ sudo ln -s /srv/myproject.com/code/deploy/nginx.conf /etc/nginx/sites-available/myproject.com
$ sudo ln -s /etc/nginx/sites-available/myproject.com /etc/nginx/sites-enabled/myproject.com

SSL

If you need to encrypt communications, Linode has a tutorial on using both self-signed certificates and commercial certificates with nginx.

Fire it Up

Nginx should be set to talk to uWSGI, which should be set to talk to the Django project. Time for a test run!

$ sudo start uwsgi-myproject
$ sudo /etc/init.d/nginx start

memcached

Django has a very good built-in cache framework. I like to take advantage of it with a memory-based backend: namely, memcached. It’s fast, efficient, and easy to setup.

All that’s needed is to install memcached on the server, followed by the Python API python-memcached.

$ sudo apt-get install memcached
$ workon myproject
$ pip install python-memcached

The default configuration file in Ubuntu lives at /etc/memcached.conf. I usually stick with the defaults, but sometimes end up changing the port that memchached runs on or the amount of memory it is allowed to use.

logrotate

With the web server more-or-less complete, I like to setup logrotate to manage the logs in my project’s directory. Once again, Slicehost has an excellent introduction to logrotate and an example config for virtual hosts.

I maintain a configuration file for each of the domains being served by the machine. The file for a domain lives in – you guessed it – the associated project’s deploy/ folder. Each contains two entries: one for the nginx virtual host and one for the uWSGI instance. The reason for this is that each config block needs a postrotate section to restart the associated server after the logs have been rotated. I don’t want nginx to be restarted everytime a uWSGI log is rotated, and I don’t want uWSGI restarted everytime an nginx log is rotated.

$ vim /srv/myproject.com/code/deploy/logrotate

/srv/myproject.com/logs/access.log /srv/myproject.com/logs/error.log {
    rotate 14
    daily
    compress
    delaycompress
    sharedscripts
    postrotate
        [ ! -f /var/run/nginx.pid ] || kill -USR1 `cat /var/run/nginx.pid`
    endscript
}

/srv/myproject.com/logs/uwsgi.log {
    rotate 14
    daily
    compress
    delaycompress
    postrotate
       restart --quiet uwsgi-myproject
    endscript
}

This file is linked in to the /etc/logrotate.d/ directory. Logrotate will automatically include any file in that directory inside its configuration.

$ sudo ln -s /srv/myproject.com/code/deploy/logrotate /etc/logrotate.d/myproject

Database Server

A web server isn’t much use without a database these days. I use PostgreSQL.

Install

1	`$ sudo apt-get install postgresql`

Configure

PostgreSQL has some unique terminology and ways of doing things. When I first set it up for the first time, having coming from a MySQL background, not everything was completely straightforward. As usual, Slicehost has a number of articles that will provide a foundation.

In the /etc/postgresql/8.4/main/postgresql.conf file, I uncomment the following two lines:

track_counts = on
autovacuum = on

Then restart the database server.

1	`$ sudo /etc/init.d/postgresql-8.4 restart`

After that I’ll change the password for the postgres user and the postgres database role.

$ sudo passwd postgres
$ sudo -u postgres psql
\password postgres
\q

To allow local socket connections to the database using passwords, I open up /etc/postgresql/8.4/main/pg_hba.conf and find the following line:

local   all         all                               ident

Which I then change to:

local   all         all                               md5

After which another restart is in order.

1	`$ sudo /etc/init.d/postgresql-8.4 restart`

Create a database

The next step is to create a user (or role, in PostgreSQL’s parlance) and database for the project. I use the same name for both.

$ sudo -u postgres createuser -PE myproject
$ sudo -u postgres createdb -O myproject myproject

After that, I should be able to connect.

1	`$ psql -U myproject`

Import the Database

If I’m restoring a previous database from a backup, now would be the time to import the backup.

1	`$ psql -U myproject < myproject.postgresql`

And now Django should be able to connect!

The basic server is setup and secure. Django, uWSGI, nginx and PostgreSQL are all running and getting along swimmingly. At this point, many people would be done, but I also like to have a minimal mail server.

Mail Server

Most of my domains use Google Apps, so I don’t need a full-blown mail server. I do want programs and scripts to be able to send mail, and I prefer not to do so through an external SMTP server – I’d rather just deal with having sendmail running on my own box. And I do have a few domains that do not use Google Apps. They have one or two aliases associated with them, so the server needs to receive messages for those domains and forward them off to an external address. If any of this sounds vaguely familiar, it’s because it’s the same thing I detailed last time. My setup now is the same as then, so I won’t repeat any of it here.

For a more detailed explanation of running Postfix, you can read the Slicehost articles.

A Note on Git

I use Git to keep track of the code for all my projects. (If you’re new to Git, you ought to skim the Git Reference or Everyday GIT With 20 Commands Or So). To manage websites, I create a repository of the directory with the code that runs the site (in this case, /srv/myproject.com/code/) and another empty, bare repository to work as a hub. With a post-update and post-commit, the end result is an excellent web workflow:

A copy of the hub can be checked out on a local machine for development. Whenever a change is committed, a simple git push will push the code to the web server and automatically make it live.
Changes can be made on the server in the actual live website directory. (This is not a best practice, but I do it more often than I should probably admit.) Whenever a change is committed, it is automatically pushed to the hub, so that a simple git pull is all that’s needed on the development machine to update its repository.

A more detailed explanation of this workflow is at Joe Maller’s blog.

To start, I need to create a repository for the new project I created in this tutorial. And, since this is a new server, I need to give Git my name and email address to record with every commit.

$ git config --global user.name "Pig Monkey"
$ git config --global user.email "pm@pig-monkey.com"
$ cd /srv/myproject.com/code/
$ git init

Before adding the files, I create a .gitignore file in the repository root to tell Git to ignore compiled Python files.

$ vim .gitignore

*.pyc

Now I add all the files to the repository, confirm that it worked, and commit the files.

$ git add .
$ git status -s
$ git commit -m "Initial commit of myproject.com"

I create the bare hub directory directly along side the projects code/.

$ cd ../
$ mkdir hub.git
$ cd hub.git
$ git --bare init

With the hub created, I need to add it as the remote for the main repository and push the master branch to it.

$ cd ../code
$ git remote add hub /srv/myproject.com/hub.git
$ git remote show hub
$ git push hub master

Now the hub needs a post-update script so that every time something is pushed to it, that change is automagically pulled into the live website directory.

$ vim /srv/myproject.com/hub.git/hooks/post-update

#!/bin/sh
echo
echo "**** Pulling changes into live"
echo

cd /srv/myproject.com/code || exit
unset GIT_DIR
git pull hub master

exec git-update-server-info

$ chmod +x /srv/myproject.com/hub.git/hooks/post-update

And the live website directory requires a post-commit script so that every time something is committed inside of it, that change is automagically pushed to the hub.

$ vim /srv/myproject.com/code/.git/hooks/post-commit

#!/bin/sh
echo
echo "**** pushing changes to Hub"
echo

git push hub

$ chmod +x /srv/myproject.com/code/.git/hooks/post-commit

All that’s left is to check out the hub onto the development machine – my laptop, in this case!

$ mkdir ~/work/myproject/
$ cd ~/work/myproject/
$ git clone ssh://myserver.com/srv/myproject.com/hub.git code

To test things out, we can add a file to the repository on the development machine.

$ cd code/
$ touch test
$ git add test
$ git commit -m "A test"
$ git push

Now go back to the server, and the file should be there! To test things the other way around, I’ll delete the file from the live repository.

$ cd /srv/myproject.com/code/
$ ls
myproject test
$ git rm test
$ git commit -m "Removing the test file"

And once again to the development machine:

$ git pull
$ ls
deploy  myproject

No more test! It’s pretty dandy.

Restoring

If I was building a new server and restoring a project from an old server, I would simply mirror the old hub and then clone that in the live directory.

$ cd /srv/myproject.com/
$ git clone --mirror ssh://myoldserver.com/srv/myproject.com/hub.git
$ git clone hub.git code/

Resources

Prior to building this server, I was new to a lot of this – particularly, uWSGI and virtualenv. The following tutorials helped me a good deal in putting together the perfect setup for my needs.

A Primer on virtualenv by Chris Scott
A web-focused Git workflow by Joe Maller
Deployment with uWSGI and nginx by Zachary Voase
Django on uWSGI and Nginx by Brandon Konkle
Django, Nginx and uWSGI in production by Jeremy Bowers
Notes on using pip and virtualenv with Django by Eliot
Presentation: pip and virtualenv by Rich Leland
Provisioning a new Ubuntu server for Django by Brandon Konkle
Running Django with Nginx and uWSGI by Simon Westphahl
virtualenvwrapper Command Reference
Working with virtualenv by Arthur Koziel

A Move to Django

2011-06-11T00:00:00-07:00

You may not notice much, but this blog has been completely rewritten.

I started developing in Django last winter and quickly became smitten with both the Django framework and the Python. Most of the coding I’ve done this year has been in Python. Naturally, I had thoughts of moving this website from Wordpress over to a Django-based blog.

For a while I did nothing about it. Then I had another project come up that required some basic blog functionality be added to a Django-based site. A blog is – or, at least, can be – a fairly simple affair, but before writing my own I decided to look around and see what else was out there. There’s a number of Django-based blogs floating around (Kevin Fricovsky has a list), but few of them jumped out at me. Most were not actively developed and depended on too many stale packages for my taste, or they just had a feature set that I didn’t like.

Out of all of them, two presented themselves as possibilities: Mingus (written by the previously mentioned Kevin) and Nathan Borror’s django-basic-apps. Mingus tries to be a full-featured blogging application and was much too complex for the simple project I was then working on. But the blog application in django-basic-apps (a fork of which provides Mingus with its core blog functionality) looked like it would fit the bill. As the name implies, it is meant to be a very basic blog. I dived in to the code I discovered that, with a few modifications, it would do what I needed.

So I finished that project. But now having messed with blogging in Django I was more motivated to get started on rewriting my own site. I took another look at Mingus. Although it was too complex for the previous project, the features it provides are very similar to the features I wanted for this website. I looked at and thought about Mingus for a time, repeatedly turning it down and then coming back to it. The question centered around the project’s staleness more than anything else. Currently, Mingus is built for Django 1.1. That’s an old version. As of this writing, the current version is 1.3. Many improvements have been made in Django since 1.1 and I was not too keen to forgo them and run an old piece of code. Mingus is under active development, and will be updated for Django 1.3, but it’s a hobby-project, so the work is understandably slow.

In the end, I decided that the best thing to do was go my own route, but take some pointers and inspiration from Mingus. I would make my own fork of django-basic-apps, using that blog as the basis, and build a system on top of that. I created my fork last month and have been steadily plodding away on it in my free time. Over the course of the development I created a few simple applications to complement the core blog, and contributed code to another project.

It’s not quite done – there’s still a few things I want to improve – but it’s good enough to launch. (If you notice any kinks, let me know.) I’m quite pleased with it.

This is a notable occasion. I’ve been using Wordpress since before it was Wordpress, but it is time to move on. (Wordpress is a fork of an old piece of code called b2/cafelog. My database tables have been rocking the b2 prefix since 2002.)

As you’ve no doubt noticed, the look of the site hasn’t changed much. I tweaked a few things here and there, but for the most part just recreated the same template as what I had written for Wordpress. I am planning on a redesign eventually. For now, I wanted to spend my time developing the actual blog rather than screwing with CSS.

So, there you have it. Everything is open source. Download it, fork it, hack it (and don’t forget to send your code changes back my way). Let me know what you think. Build your own blog with it! (There’s even a script to import data from Wordpress.) I think it’s pretty sweet. The only thing lacking is documentation, and that’s my next goal.

Disqus

The biggest change for the user is probably the comments, which are now powered by Disqus. Consider it a trial. I’ve seen Disqus popping up on a number of sites the past year or so. At first it annoyed me, mostly because I use NoScript and did not want to enable JavaScript for another domain just to comment on a site. But after I got over that I found that Disqus wasn’t too bad. As a user I found it to be on par with the standard comment systems provided by Wordpress, Blogger, and the like. The extra features don’t appeal to me. But as an administrator, Disqus appeals to me more because it means that I no longer have to manage comments myself! And as a developer, I’m attracted to some of the things that Disqus has done (they’re a Python shop, and run on top of Django) and their open source contributions.

So I’m giving it a shot. Disqus will happily export comments, so if I (or you) decide that I don’t like it, it will be easy to move to another system.

Markdown

One final note: I like Markdown. That might be an understatement.

I first starting using Markdown on GitHub, which I signed up for about the same time I started with Django and Python. After learning the syntax and playing with it for a few weeks, I discovered that I had a very hard time writing prose in anything else. In fact, the desire to write blog posts in Markdown was probably the biggest factor that influenced me to get off my butt and move away from Wordpress.

So, I incorporated Markdown into the blog. But rather than just making the blog Markdown-only, I took a hint from Mingus and included django-markup, which supports rendering in many lightweight markup languages.

Because I’m still new to Markdown and occasionally cannot remember the correct syntax, I wanted to include some version of WMD. WMD is a What You See Is What You Mean editor for Markdown, a sort of alternative to WYSIWYG editors like TinyMCE. (It is my believe that WYSIWYG editors are one of the worst things to happen to the Internet.) All WMD consists of is a JavaScript library. The original was written by a guy named John Fraser, who was abducted by aliens some time in 2008. Since his disappearance from the interwebs, WMD has been forked countless times. I looked around at a few found a version that I was happy with (which happens to be a fork of a fork of a fork of a fork), and rolled it into a reusable app. While I was at it, I made some visual changes to the editing area for the post body. The result is an attractive post editing area that is simple to use and produces clean code. I think it is much better than what is offered by Wordpress.

Wishlist

2011-01-10T00:00:00-08:00

Wishlist is a Django application for creating wishlists.

I used Amazon Wishlist for a number of years, but my paranoia finally caught up to me and I decided that I didn’t need to give Amazon that much more information about my interests.

I tried a few substitutes and found that my requirements for a wishlist were less than common. I don’t often use wishlists in the usual way of asking people for gifts on special occasions. Instead, I use wishlists privately to keep track of items that I wish to purchase myself. It helps me to determine savings goals, to track books that I want to read, etc. As such, I usually do not want items on my wishlist to be publicly viewable.

Out of the substitutes I tried, Wishlistr was undoubtedly the best, but there were some aspects of it that I didn’t like. After using it for a while, I decided to write my own app.

As with Gear Tracker, Wishlist is open source. You can get the code on GitHub.

Gear Tracker

2010-12-24T00:00:00-08:00

Gear Tracker is an open-source inventory system for wilderness travel gear.

When I first bought my scale, I started a spreadsheet containing the weights of various pieces of gear. It seemed like a good idea – I knew I wanted some sort of database to store my measured weights and other notes in – but I never got around to updating it. Data in a spreadsheet is too static. You can’t do much with it. I think that characteristic contributed to my disinterest with the spreadsheet.

So for a while now I’ve had the idea of writing a web application to track my gear. Over the past week, I finally got around to doing it.

Gear Tracker is built on Django, a web application framework. (If you’re not familiar with Django, and you have anything to do with making websites, it’s probably worth your time to learn a thing or two.)

Gear

Gear Tracker’s primary purpose is to track gear.

Each item has a weight and acquisition date associated with it. It can be categorized, tagged, and related to other items. There are fields to input size, a link to the manufacturer’s page, a link to a review, and to upload an image. A text area allows the user to store any notes related to the item.

Items can be archived. This provides a way to not list gear that the user no longer owns, but to keep it in the database for future reference of its weight or other attributes.

Weights

Weights are always input in grams.

The metric system makes the most sense and is the easiest to work with. An item’s weight can be displayed in grams or, if the item weighs more than 1,000 grams, kilograms. But because some of us are crippled and still like to see imperial weights, Gear Tracker can also display the item’s weight in ounces or pounds.

Gear Lists

Gear Tracker can also generate gear lists.

One of the things that has prevented me from doing many gear lists in the past is that they’re a pain in the rear to create. It takes a while to manually write out every item of gear that I take on a trip. If I want to add the weight of each item – well, that’s asking too much! It’s not worth it.

Now, making gear lists is easy. Gear Tracker already has detailed knowledge about each piece of gear. All it takes to create a gear list is to select the item, specify how many of that item I took, and whether the item was packed or carried. The result is an organized, detailed gear list for every trip. Total weights are calculated, of course, and can be output in either metric or imperial units.

Private Gear Lists

Gear lists can be made private.

I generally create gear lists when I’m packing before a trip. But I don’t like to publish the lists until I actually return from the trip and also have a report and photos for people to peruse. So, Gear Tracker allows a gear list to be marked as private.

Download It, Hack It, Use It

I’m running Gear Tracker at /gear, but if you want to grab your own copy and run it yourself, you can! I’ve open-sourced the code under a BSD-license. You can find it at GitHub.