As I was migrating my online backups to Restic, I was concurrently building a system that would allow me to use Restic to complete full-disk backups to external USB drives, thus replacing my use of cryptshot and rsnapshot. I give you Resnap.

For me, the main appeal of rsnapshot has always been that the resulting backups are just directories full of files. When it comes time to restore, no special tooling is required for access. Just copy the files over using rsync or cp or whatever tickles your fancy. If you can read the disk, you can recover data. Most competing systems create backups in a way that you also need the backup tool for recovery. This introduces fragility into the setup. My experience has been that most people do not think or care about this, but I have spent many years thinking about how not to lose data. I have certain requirements for backup systems. “Robustness” is a big one.

Restic does fall into the second category of tool. If you have a copy of a Restic repository but the Restic project has imploded and you do not have or cannot execute the tool, you’re in for a bad time. But the reason Restic has been on my radar since the project first stated a decade ago (it takes me a while to adopt new backup software – also I recall it took them a while to implement compression) is that it is compiled to a single, statically linked Go binary. This means you can easily store the tool itself alongside (not within) your backups. You’re not going to do that with a sprawling Python project like Borg, which is why I continued to use rsnapshot for my local full-disk backups even after I adopted Borg for online backups.

But with Restic you can just cp the binary to the backup drive. And that’s what I do. Resnap also provides resnap-restore, a simple wrapper script that makes it easy to perform the restic restore ... operation during full-disk recovery. This too is stored on the backup drive alongside the binary.

What this means is that you can use Resnap to create a full-disk backup to an external drive and then bury that drive in your backyard for 25 years (properly sealed). After you dig it up, as long as you have the passphrase, can read the drive’s filesystem (ext4 for me), and can execute a binary compiled for whatever architecture you were using 25 years ago (x86-64 for me) – both highly likely given the commonality of those systems – you are going to have no problem restoring that data (subject to the physics of spinning rust). An unlikely scenario perhaps, but that’s sort of the data backup baseline we plan for here at pig-monkey.com.

After mulling it over for many cycles, I finally decided to migrate my online backups to Restic. As is my wont, I have published my solution as Restash so that members of the Pig Monkey Data Backups Fan Club can be like me.

The restash script will look pretty familiar to anyone who has been using my old Borg wrapper script. It is mostly the same basic structure, with Borg logic replaced with Restic logic, and two other significant differences.

Previously I ran the Borg wrapper hourly via a systemd timer, and used backitup to run the verification checks (and compacting) less frequently. Now the script has subcommands, and I use different systemd units to call the different functions on different schedules – backups more frequently, verification and pruning less frequently.

Previously I achieved redundancy by using Borg to backup hourly to my rsync.net account, while Tarsnap ran daily backups of a smaller subset of the same data. Now I’m using Restic to backup hourly to my rsync.net account, and also using Restic to backup the same data daily to a Backblaze B2 bucket. I’ve been an rsync.net customer since 2014 and I think highly of their service, but the B2 bucket is cheap insurance that helps me sleep better.

In the example config you will see that the SFTP_HOST is simply restic. That refers to an entry in my ~/.ssh/config, which looks something like this:

Host restic
    Hostname abc123.rsync.net
    User abc123
    IdentityFile ~/.ssh/passphraseless-key
    IdentitiesOnly yes
    BatchMode yes
    ServerAliveInterval 60
    ServerAliveCountMax 3

The only other significant change from the old Borg wrapper script is that I broke out some of the config options, backup targets, and excludes to separate files to make it easier for you, my adoring public, to reuse the tool.

As hinted at in the README, I still use nmtrust to only execute backups on trusted networks. Don’t be a data litterer.

Continuing with the theme of replacing drives, recently I decided to preemptively replace one of the external drives that I backup to via rsnapshot – or, more specifically, via cryptshot. The drive was functioning nominally, but its date of manufacture was 2014. That’s way too long to trust spinning rust.

rsnapshot implements deduplication via hard links. Were I to just rsync the contents of the old drive to the new drive without any special consideration for the links, it would dereference the links, copying them over as separate files. This would cause the size of the backups to balloon past the capacity of the drive. Rsync provides the --hard-links flag to address this, but I’ve heard some stories about this failing to act as expected when the source directory has a large number of hard links (for some unknown definition of “large”). I’ve been rsnapshotting since 2012 (after a pause sometime after 2006, apparently) and feel safe assuming that my rsnapshot repository does have a “large” number of hard links.

I also do not really care about syncing. The destination is completely empty. There’s no file comparison that needs to happen. I don’t need to the ability to pause partway through the transfer and resume later. Rsync is my default solution for pushing files around, but in this case it is not really needed. I only want to mirror the contents of the old drive onto the new drive, exactly as they exist on the old drive. So I avoided the problem all together and just copied the partition via dd.

Both drives are encrypted with LUKS, so first I decrypt them. Importantly, I do not mount either decrypted partition. I don’t want to risk any modifications being made to either while the copy is ongoing.

$ sudo cryptsetup luksOpen /dev/sda old
$ sudo cryptsetup luksOpen /dev/sdb new

Then I copy the old partition to the new one.

$ sudo dd if=/dev/mapper/old of=/dev/mapper/new bs=32M status=progress

My new drive is the same size as my old drive, so after dd finished I was done. If the sizes differed I would need to use resize2fs to resize the partition on the new drive.

If I was replacing the old drive not just because it was old and I was ageist, but because I thought it may be corrupted, I would probably do this with GNU ddrescue rather than plain old dd. (Though, realistically, if that was the case I’d probably just copy the contents of my other rsnapshot target drive to the new drive, and replace the corrupt drive with that. Multiple backup mediums make life easier.)

As previously mentioned, I use myrepos to keep local copies of useful code repositories. While working with backups yesterday I noticed that this directory had gotten quite large. I realized that in the 8 years that I’ve been using this system, I’ve never once run git gc in any of the repos.

Fortunately this is the sort of thing that myrepos makes simple – even providing it as an example on its homepage. I added two new lines to the [DEFAULT] section of my ~/library/src/myrepos.conf file: one telling it that it can run 3 parallel jobs, and one teaching it how to run git gc.

[DEFAULT]
skip = [ "$1" = update ] && ! hours_since "$1" 24
jobs = 3
git_gc = git gc "$@"

That allowed me to use my existing lmr alias to clean up all the git repositories. The software knows which repositories are git, and only attempts to run the command in those.

$ lmr gc

After completing this process – which burned through a lot of CPU – my ~/library/src directory dropped from 70 GB to 15 GB.

So that helped.

I noticed that my Borg directory on The Cloud was 239 GB. This struck me as problematic, as I could see in my local logs that Borg itself reported the deduplicated size of all archives to be 86 GB.

A web search revealed borg compact, which apparently I have been meant to run manually since 2019. Oops. After compacting, the directory dropped from 239 GB to 81 GB.

My borg wrapper script now looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

#!/bin/sh
source ~/.keys/borg.sh
export BORG_REPO='borg-rsync:borg/nous'
export BORG_REMOTE_PATH='borg1'

# Create backups
echo "Creating backups..."
borg create --verbose --stats --compression=lz4             \
    --exclude ~/projects/foo/bar/baz                        \
    --exclude ~/projects/xyz/bigfatbinaries                 \
    ::'{hostname}-{user}-{utcnow:%Y-%m-%dT%H:%M:%S}'        \
    ~/documents                                             \
    ~/projects                                              \
    ~/mail                                                  \
    # ...etc

# Prune
echo "Pruning backups..."
borg prune --verbose --list --glob-archives '{hostname}-{user}-*'   \
    --keep-within=1d                                                \
    --keep-daily=14                                                 \
    --keep-weekly=8                                                 \
    --keep-monthly=12                                               \

# Compact
echo "Compacting repository..."
backitup                                \
    -p 604800                           \
    -l ~/.borg_compact-repo.lastrun     \
    -b "borg compact --verbose"         \

# Check
echo "Checking repository..."
backitup -a                                                         \
    -p 172800                                                       \
    -l ~/.borg_check-repo.lastrun                                   \
    -b "borg check --verbose --repository-only --max-duration=1200" \

echo "Checking archives..."
backitup -a                                             \
    -p 259200                                           \
    -l ~/.borg_check-arch.lastrun                       \
    -b "borg check --verbose --archives-only --last 18" \

Other than the addition of a weekly compact, my setup is the same as it ever was.