LUKS Header Backup

I’d neglected backup LUKS headers until Gwern’s data loss postmortem last year. After reading his post I dumped the headers of the drives I had accessible, but I never got around to performing the task on my less frequently accessed drives. Last month I had trouble mounting one of those drives. It turned out I was simply using the wrong passphrase, but the experience prompted me to make sure I had completed the header backup procedure for all drives.

I dump the header to memory using the procedure from the Arch wiki. This is probably unnecessary, but only takes a few extra steps. The header is stored in my password store, which is obsessively backed up.

$ sudo mkdir /mnt/tmp
$ sudo mount ramfs /mnt/tmp -t ramfs
$ sudo cryptsetup luksHeaderBackup /dev/sdc --header-backup-file /mnt/tmp/dump
$ sudo chown pigmonkey:pigmonkey /mnt/tmp/dump
$ pass insert -m crypt/luksheader/themisto < /mnt/tmp/dump
$ sudo umount /mnt/tmp
$ sudo rmdir /mnt/tmp

Flattening Water Stones

I’ve been sharpening my knives on the same Japanese water stones for a dozen years now. Despite my best intentions, I do not always use the full length of the stones. Somewhere in the back of my mind I have always been concerned about dishing the stones. Last month I took a sharpening class at Bernal Cutlery, which was the first hands-on instruction I’ve ever had in the subject. One of the things I learned was that there are other stones that can be used to flatten sharpening stones.

After the class I purchased a 95-micron DMT Dia-Flat Lapping Plate. It only took 30 seconds or so for it to flatten my water stones. Either it works extremely well or my stones were not as dished as I thought they were. After using it, there was a very obvious improvement in how the stones sharpened. More than I would expect just from flattening them. It makes me think that perhaps the pores of the stones had been clogged from years of use, which was addressed by removing the top layer of material with the lapping plate.


The lapping plate is certainly not cheap. I’m sure that they do not last forever for professional sharpeners, but given how frequently I use my stones I think the lapping plate falls into the buy-it-once-for-life category. It has extended the life of my water stones, which I think makes it a justified expense.

This post was published on . It was tagged with knife.

An Inbox for Taskwarrior

My experience with all task managements systems — whether software or otherwise — is that the more you put into them, the more useful they become. Not only adding as many tasks as possible, however small they may be, but also enriching the tasks with as much metadata as possible. When I began using taskwarrior, one of the problems I encountered was how to address this effectively.

Throughout the day I’ll be working on something when I receive an unrelated request. I want to log those requests so that I remember them and eventually complete them, but I don’t want to break from whatever I’m currently doing and take the time to mark these tasks up with the full metadata they eventually need. Context switching is expensive.

To address this, I’ve introduced the idea of a task inbox. I have an alias to add a task to taskwarrior with a due date of tomorrow and a tag of inbox.

alias ti='task add due:tomorrow tag:inbox'

This allows me to very quickly add a task without needing to think about it.

$ ti do something important

Each morning I run task ls. The tasks which were previously added to my inbox are at the top, overdue with a high priority. At this point I’ll modify each of them, removing the inbox tag, setting a real due date, and assigning them to a project. If they are more complex, I may also add annotations or notes, or build the task out with dependencies. If the task is simple — something that may only take a minute or two — I’ll just complete it immediately and mark it as done without bothering to remove the inbox tag.

This alias lowers the barrier of entry enough that I am likely to log even the smallest of tasks, while the inbox concept provides a framework for me to later make the tasks rich in a way that allows me to take advantage of the power that taskwarrior provides.

This post was published on . It was tagged with linux.

Borg Assimilation

For years the core of my backup strategy has been rsnapshot via cryptshot to various external drives for local backups, and Tarsnap for remote backups.

Tarsnap, however, can be slow. It tends to take somewhere between 15 to 20 minutes to create my dozen or so archives, even if little has changed since the last run. My impression is that this is simply due to the number of archives I have stored and the number of files I ask it to archive. Once it has decided what to do, the time spent transferring data is negligible. I run Tarsnap hourly. Twenty minutes out of every hour seems like a lot of time spent Tarsnapping.

I’ve eyed Borg for a while (and before that, Attic), but avoided using it due to the rapid development of its earlier days. While activity is nice, too many changes too close together do not create a reassuring image of a backup project. Borg seems to have stabilized now and has a large enough user base that I feel comfortable with it. About a month ago, I began using it to backup my laptop to rsync.net.

Initially I played with borgmatic to perform and maintain the backups. Unfortunately it seems to have issues with signal handling, which caused me to end up with annoying lock files left over from interrupted backups. Borg itself has good documentation and is easy to use, and I think it is useful to build familiarity with the program itself instead of only interacting with it through something else. So I did away with borgmatic and wrote a small bash script to handle my use case.

Creating the backups is simple enough. Borg disables compression by default, but after a little experimentation I found that LZ4 seemed to be a decent compromise between compression and performance.

Pruning backups is equally easy. I knew I wanted to match roughly what I had with Tarsnap: hourly backups for a day or so, daily backups for a week or so, then a month or two of weekly backups, and finally a year or so of monthly backups.

My only hesitation was in how to maintain the health of the backups. Borg provides the convenient borg check command, which is able to verify the consistency of both a repository and the archives themselves. Unsurprisingly, this is a slow process. I didn’t want to run it with my hourly backups. Daily, or perhaps even weekly, seemed more reasonable, but I did want to make sure that both checks were completed successfully with some frequency. Luckily this is just the problem that I wrote backitup to solve.

Because the consistency checks take a while and consume some resources, I thought it would also be a good idea to avoid performing them when I’m running on battery. Giving backitup the ability to detect if the machine is on battery or AC power was a simple hack. The script now features the -a switch to specify that the program should only be executed when on AC power.

My completed Borg wrapper is thus:

export BORG_PASSPHRASE='supers3cr3t'
export BORG_REPO='borg-rsync:borg/nous'
export BORG_REMOTE_PATH='borg1'

# Create backups
echo "Creating backups..."
borg create --verbose --stats --compression=lz4             \
    --exclude ~/projects/foo/bar/baz                        \
    --exclude ~/projects/xyz/bigfatbinaries                 \
    ::'{hostname}-{user}-{utcnow:%Y-%m-%dT%H:%M:%S}'        \
    ~/documents                                             \
    ~/projects                                              \
    ~/mail                                                  \
    # ...etc

# Prune backups
echo "Pruning backups..."
borg prune --verbose --list --prefix '{hostname}-{user}-'    \
    --keep-within=2d                                         \
    --keep-daily=14                                          \
    --keep-weekly=8                                          \
    --keep-monthly=12                                        \

# Check backups
echo "Checking repository..."
backitup -a                                             \
    -p 172800                                           \
    -l ~/.borg_check-repo.lastrun                       \
    -b "borg check --verbose --repository-only"         \
echo "Checking archives..."
backitup -a                                             \
    -p 259200                                           \
    -l ~/.borg_check-arch.lastrun                       \
    -b "borg check --verbose --archives-only --last 24" \

This is executed by a systemd service.

Description=Borg Backup



The service is called hourly by a systemd timer.

Description=Borg Backup Timer



I don’t enable the timer directly, but add it to /usr/local/etc/trusted_units so that nmtrust activates it when I’m connected to trusted networks.

$ echo "borg.timer,user:pigmonkey" >> /usr/local/etc/trusted_units

I’ve been running this for about a month now and have been pleased with the results. It averages about 30 seconds to create the backups every hour, and another 30 seconds or so to prune the old ones. As with Tarsnap, deduplication is great.

                       Original size      Compressed size    Deduplicated size
This archive:               19.87 GB             18.41 GB             10.21 MB
All archives:              836.02 GB            773.35 GB             19.32 GB
                       Unique chunks         Total chunks
Chunk index:                  371527             14704634

The most recent repository consistency check took about 30 minutes, but only runs every 172800 seconds, or once every other day. The most recent archive consistency check took about 40 minutes, but only runs every 259200 seconds, or once per 3 days. I’m not sure that those schedules are the best option for the consistency checks. I may tweak their frequencies, but because I know they will only be executed when I am on a trusted network and AC power, I’m less concerned about the length of time.

With Borg running hourly, I’ve reduced Tarsnap to run only once per day. Time will tell if Borg will slow as the number of stored archives increase, but for now running Borg hourly and Tarsnap daily seems like a great setup. Tarsnap and Borg both target the same files (with a few exceptions). Tarsnap runs in the AWS us-east-1 region. I’ve always kept my rsync.net account in their Zurich datacenter. This provides the kind of redundancy that lets me rest easy.

Contrary to what you might expect given the number of blog posts on the subject, I actually spend close to no time worrying about data loss in my day to day life, thanks to stuff like this. An ounce of prevention, and all that. (Maybe a few kilograms of prevention in my case.)

I’ve been happily using my AquaRain filter for a little short of a decade now.

My only complaint about the system is that the filter elements degrade slowly enough that I rarely notice the decreased flow. Cleaning and assessing the health of the elements (which is done by measuring their circumference with the provided tool) should happen periodically, but it isn’t the type of thing I’ll ever think to do myself. As with my water rotation, I let taskwarrior solve the problem for me.

$ task add project:waterstorage due:2017-07-01 recur:6months wait:due-7days clean and assess aquarain filter

This post was published on . It was tagged with micro, water.

Water Rotation

I use four WaterBricks for water storage at home, and for the occasional vehicle-borne excursions. They’re simple to store in small areas, stack securely, and are easy to pour from with the spigot assembly. I prefer them over the more common Scepter Water Canisters. The 3.5 gallon capacity of the WaterBricks is in the sweet spot of being able to hold a lot of water, but isn’t so heavy that life sucks when you need to haul them around.

I took one of the WaterBricks on this year’s ARRL Field Day last month. This was the first time this particular WaterBrick had been opened in three years. The water tasted fine, albeit with a plasticy flavor that wasn’t surprising, but storing water for this length of time seems at best excessive and at worse negligent. I took this as an opportunity to implement a rotation schedule.

Each of the WaterBricks is now labelled. They are grouped in to two 12-month rotation periods, each six months apart. This provides an opportunity to not only change the water, but also bleach and dry the inside of the containers to discourage any growth. By performing the rotation six months apart, I can be assured of always having two full WaterBricks on hand.

By scheduling the rotation in taskwarrior I never have to think about it.

$ task add project:waterstorage due:2017-06-01 recur:yearly wait:due-7days rotate waterbrick alpha
$ task add project:waterstorage due:2017-06-01 recur:yearly wait:due-7days rotate waterbrick bravo
$ task add project:waterstorage due:2017-12-01 recur:yearly wait:due-7days rotate waterbrick charlie
$ task add project:waterstorage due:2017-12-01 recur:yearly wait:due-7days rotate waterbrick delta

I use 28 drops of Aquamira chlorine dioxide per WaterBrick, although I’m not sure how necessary that is now with the rotation schedule.

This post was published on . It was tagged with water.