You are currently viewing all posts tagged with linux.

toolchain, linux, shell, annex

Music Organization with Beets

I organize my music with Beets.

Beets imports music into my library, warns me if I’m missing tracks, identifies tracks based on their accoustic fingerprint, scrubs extraneous metadata, fetches and stores album art, cleans genres, fetches lyrics, and – most importantly – fetches metadata from MusicBrainz. After some basic configuration, all of this happens automatically when I import new files into my library.

After the files have been imported, beets makes it easy to query my library based on any of the clean, consistent, high quality, crowd-sourced metadata.

$ beet stats genre:ambient
Tracks: 649
Total time: 2.7 days
Approximate total size: 22.4 GiB
Artists: 76
Albums: 53
Album artists: 34

$ beet ls -a 'added:2019-07-01..'
Deathcount in Silicon Valley - Acheron
Dlareme - Compass
The Higher Intelligence Agency & Biosphere - Polar Sequences
JK/47 - Tokyo Empires
Matt Morton - Apollo 11 Soundtrack

$ beet ls -ap albumartist:joplin
/home/pigmonkey/library/audio/music/Janis Joplin/Full Tilt Boogie
/home/pigmonkey/library/audio/music/Janis Joplin/I Got Dem Ol' Kozmic Blues Again Mama!

As regular readers will have surmised, the files themselves are stored in git-annex.

toolchain, linux, shell

Terminal Countdown

Termdown is a program that provides a countdown timer and stopwatch in the terminal. It uses FIGlet for its display. Its most attractive feature, I think, is the ability to support arbitrary script execution.

I use it most often as a countdown timer. One of my frequent applications is as a meditation timer. For this I want a 11 minute timer, with an alert at 10.5 minutes, 60 seconds, and 1 second. This gives me a 10 minute session with 30 seconds preparation and 30 seconds to return. Termdown makes this easy.

$ termdown --exec-cmd "case {0} in 630|30) mpv ~/library/audio/sounds/bell.mp3;; 1) mpv ~/library/audio/sounds/ring.mp3;; esac" 11m
toolchain, linux, shell

An Offline Lexicon

dictd is a dictionary database server and client. It can be used to lookup word definitions over a network. I don’t use it for that. I use the program to provide an offline dictionary. Depending on a network connection, web browser and third-party websites just to define a word strikes me as dumb.

To make this go, dictionary files must be installed. I use the GNU Collaborative International Dictionary of English (GCIDE), WordNet, and the Moby Thesaurus. The GCIDE is derived from Noah Webster’s famous American dictionary. WordNet is a more modern (one might say “dry”) resource. The Moby Thesaurus is a public domain thesaurus originally built by Grady Ward. Between these three sources I can have a pretty good grasp on the English language. No network connectivity required.

I use a shell alias to always pipe the definitions through less.

def () {
    dict $1 | less
}
backups, crypto, linux, annex

Optical Backups of Financial Archives

Every year I burn an optical archive of my financial documents, similar to how (and why) I create optical backups of photos. I schedule this financial archive for the spring, after the previous year’s taxes have been submitted and accepted. Taskwarrior solves the problem of remembering to complete the archive.

$ task add project:finance due:2019-04-30 recur:yearly wait:due-4weeks "burn optical financial archive with parity"

The archive includes two git-annex repositories.

The first is my ledger repository. Ledger is the double-entry accounting system I began using in 2012 to record the movement of every penny that crosses one of my bank accounts (small cash transactions, less than about $20, are usually-but-not-always except from being recorded). In addition to the plain-text ledger files, this repository also holds PDF or JPG images of receipts.

The second repository holds my tax information. Each tax year gets a ctmg container which contains any documents used to complete my tax returns, the returns themselves, and any notifications of those returns being accepted.

The yearly optical archive that I create holds the entirety of these two repositories – not just the information from the previous year – so really each disc only needs to have a shelf life of 12 months. Keeping the older discs around just provides redundancy for prior years.

Creating the Archive

The process of creating the archive is very similar to the process I outlined six years ago for the photo archives.

The two repositories, combined, are about 2GB (most of that is the directory of receipts from the ledger repository). I burn these to a 25GB BD-R disc, so file size is not a concern. I’ll tar them, but skip any compression, which would just add extra complexity for no gain.

$ mkdir ~/tmp/archive
$ cd ~/library
$ tar cvf ~/tmp/archive/ledger.tar ledger
$ tar cvf ~/tmp/archive/tax.tar tax

The ledger archive will get signed and encrypted with my PGP key. The contents of the tax repository are already encrypted, so I’ll skip encryption and just sign the archive. I like using detached signatures for this.

$ cd ~/tmp/archive
$ gpg -e -r peter@havenaut.net -o ledger.tar.gpg ledger.tar
$ gpg -bo ledger.tar.gpg.sig ledger.tar.gpg
$ gpg -bo tax.tar.sig tax.tar
$ rm ledger.tar

Previously, when creating optical photo archives, I used DVDisaster to create the disc image with parity. DVDisaster no longer exists. The code can still be found, and the program still works, but nobody is developing it and it doesn’t even an official web presence. This makes me uncomfortable for a tool that is part of my long-term archiving plans. As a result, I’ve moved back to using Parchive for parity. Parchive also does not have much in the way of active development around it, but it is still maintained, has been around for a long period of time, is still used by a wide community, and will probably continue to exist as long as people share files on less-than-perfectly-reliable mediums.

As previously mentioned, I’m not worried about the storage space for these files, so I tell par2create to create PAR2 files with 30% redundancy. I suppose I could go even higher, but 30% seems like a good number. By default this process will be allowed to use 16MB of memory, which is cute, but RAM is cheap and I usually have enough to spare so I’ll give it permission to use up to 8GB.

$ par2create -r30 -m8000 recovery.par2 *

Next I’ll use hashdeep to generate message digests for all the files in the archive.

$ hashdeep * > hashes

At this point all the file processing is completed. I’ll put a blank disc in my burner (a Pioneer BDR-XD05B) and burn the directory using growisofs.

$ growisofs -Z /dev/sr0 -V "Finances 2019" -r *

Verification

The final step is to verify the disc. I have a few options on this front. These are the same steps I’d take years down the road if I actually needed to recover data from the archive.

I can use the previous hashes to find any files that do not match, which is a quick way to identify bit rot.

$ hashdeep -x -k hashes *.{gpg,tar,sig,par2}

I can check the integrity of the PGP signatures.

$ gpg --verify tax.tar.gpg{.sig,}
$ gpg --verify tax.tar{.sig,}

I can use the PAR2 files to verify the original data files.

$ par2 verify recovery.par2
code, shell, linux

GOESImage

GOESImage is a bash script which downloads the latest imagery from the NOAA Geostationary Operational Environment Satellites and sets it as the desktop background via feh. If you don’t use feh, it should be easy to plug GOESImage into any desktop background control program.

GOESImage Example

I wrote GOESImage after using himawaripy for a few years, which is a program that provides imagery of the Asia-Pacific region from the Himawari 8 Japanese weather satellite. I like seeing the Earth, and I’ve found that real time imagery of my location is actually useful for identifying the approach of large-scale weather systems. NOAA’s nighttime multispectral infrared coloring is pretty neat, too.

annex, linux, backups

Archiving Bookmarks

I signed-up for Pinboard in 2014. It provides everything I need from a bookmarking service, which is mostly, you know, bookmarking. I pay for the archival account, meaning that Pinboard downloads a copy of everything I bookmark and provides me with full-text search. I find this useful and well worth the $25 yearly fee, but Pinboard’s archive is only part of the solution. I also need an offline copy of my bookmarks.

Pinboard provides an API that makes it easy to acquire a list of bookmarks. I have a small shell script which pulls down a JSON-formatted list of my bookmarks and adds the file to git-annex. This is controlled via a systemd service and timer, which wraps the script in backitup to ensure daily dumps. The systemd timer itself is controlled by nmtrust, so that it only runs when I am connected to a trusted network.

This provides data portability, ensuring that I could import my tagged URLs to another bookmarking service if I ever found something better than Pinboard (unlikely, competing with Pinboard is futile). But I also want a locally archived copy of the pages themselves, which Pinboard does not offer through the API. I carry very much about being able to work offline. The usefulness of a computer is directly propertional to the amount of data that is accessible without a network connection.

To address this I use bookmark-archiver, a Python script which reads URLs from a variety of input files, including Pinboard’s JSON dumps. It archives each URL via wget, generates a screenshot and PDF via headless Chromium, and submits the URL to the Internet Archive (with WARC hopefully on the way). It will then generate an HTML index page, allowing the archives to be easily browsed. When I want to browse the archive, I simply change into the directory and use python -m http.server to serve the bookmarks at localhost:8000. Once downloaded locally, the archives are of course backed up, via the usual suspects like borg and cryptshot.

The archiver is configured via environment variables. I configure my preferences and point the program at the Pinboard JSON dump in my annex via a shell script (creatively also named bookmark-archiver). This wrapper script is called by the previous script which dumps the JSON from Pinboard.

The result of all of this is that every day I get a fresh dump of all my bookmarks, each URL is archived locally in multiple formats, and the archive enters into my normal backup queue. Link rot may defeat the Supreme Court, but between this and my automated repository tracking I have a pretty good system for backing up useful pieces of other people’s data.

linux, crypto

PGP Key Renewal

Last year I demonstrated setting up the USB Armory for PGP key management. The two management operations I perform on the Armory are key signing and key renewal. I set my keys to expire each year, so that each year I need to confirm that I am not dead, still control the keys, and still consider them trustworthy.

After booting up the Armory, I first verify that NTP is disabled and set the current UTC date and time. Time is critical for any cryptography operations, and the Armory has no battery to maintain a clock.

$ timedatectl set-ntp false
$ timedatectl set-time "yyyy-mm-dd hh:mm:ss"

My keys are stored on an encrypted microSD card, which I mount and decrypt.

$ mkdir /mnt/sdcard
$ cryptsetup luksOpen /dev/sda sdcrypt
$ mount /dev/mapper/sdcrypt /mnt/sdcard

Next I’ll export a few environment variables to make things less redundant later on.

$ export YEAR=$(date +%Y)
$ export PREVYEAR=$(($YEAR-1))
$ export GNUPGHOME="/mnt/sdcard/gpg/$YEAR-renewal/.gnupg"
$ export KEYID="0x70B220FF8D2ACF29"

I perform each renewal in a directory specific to the current year, but the GNUPGHOME directory I set for this year’s renewal doesn’t exist yet. Better create it.

$ mkdir -p $GNUPGHOME
$ chmod 700 $GNUPGHOME

I keep a copy of my gpg.conf on the microSD card. That needs to be copied in to the new directory, and I’ll need to tell GnuPG what pinentry program to use.

$ cp /mnt/sdcard/gpg/gpg.conf $GNUPGHOME
$ echo "pinentry-program /usr/bin/pinentry-curses" > $GNUPGHOME/gpg-agent.conf

After renewing the master key and subkey the previous year, I exported them. I’ll now import those backups from the previous year.

$ gpg --import /mnt/sdcard/gpg/$PREVYEAR-renewal/backup/peter\@havenaut.net.master.gpg-key
$ gpg --import /mnt/sdcard/gpg/$PREVYEAR-renewal/backup/peter\@havenaut.net.subkeys.gpg-key

When performing the actual renewal, I’ll set the expiration to 13 months. This needs to be done for the master key, the signing subkey, the encryption subkey, and the authentication subkey.

$ gpg --edit-key $KEYID
trust
5
expire
13m
y
key 1
key 2
key 3
expire
y
13m
y
save

That’s the renewal. I’ll list the keys to make sure they look as expected.

$ gpg --list-keys

Before moving the subkeys to my Yubikey, I back everything up. This will be what I import the following year.

$ mkdir /mnt/sdcard/gpg/$YEAR-renewal/backup
$ gpg --armor --export-secret-keys $KEYID > /mnt/sdcard/gpg/$YEAR-renewal/backup/peter\@havenaut.net.master.gpg-key
$ gpg --armor --export-secret-subkeys $KEYID > /mnt/sdcard/gpg/$YEAR-renewal/backup/peter\@havenaut.net.subkeys.gpg-key

Now I can insert my Yubikey, struggle to remember the admin PIN I set on it, and move over the subkeys.

$ gpg --edit-key $KEYID
toggle
key 1 # signature
keytocard
1
key 1
key 2 # encryption
keytocard
2
key 2
key 3 # authentication
keytocard
3
save

When I list the secret keys, I expect them to all be stubs (showing as ssb>).

$ gpg --list-secret-keys

Of course, for this to be useful I need to export my renewed public key and copy it to some place where it can be brought to a networked machine for dissemination.

$ gpg --armor --export $KEYID > /mnt/sdcard/gpg/$YEAR-renewal/peter\@havenaut.net.public.gpg-key
$ mkdir /mnt/usb
$ mount /dev/sdb1 /mnt/usb
$ cp /mnt/sdcard/gpg/$YEAR-renewal/peter\@havenaut.net.public.gpg-key /mnt/usb/

That’s it. Clean up, shutdown, and lock the Armory up until next year.

$ umount /mnt/usb
$ umount /mnt/sdcard
$ cryptsetup luksClose sdcrypt
$ systemctl poweroff
backups, crypto, linux

LUKS Header Backup

I’d neglected backup LUKS headers until Gwern’s data loss postmortem last year. After reading his post I dumped the headers of the drives I had accessible, but I never got around to performing the task on my less frequently accessed drives. Last month I had trouble mounting one of those drives. It turned out I was simply using the wrong passphrase, but the experience prompted me to make sure I had completed the header backup procedure for all drives.

I dump the header to memory using the procedure from the Arch wiki. This is probably unnecessary, but only takes a few extra steps. The header is stored in my password store, which is obsessively backed up.

$ sudo mkdir /mnt/tmp
$ sudo mount ramfs /mnt/tmp -t ramfs
$ sudo cryptsetup luksHeaderBackup /dev/sdc --header-backup-file /mnt/tmp/dump
$ sudo chown pigmonkey:pigmonkey /mnt/tmp/dump
$ pass insert -m crypt/luksheader/themisto < /mnt/tmp/dump
$ sudo umount /mnt/tmp
$ sudo rmdir /mnt/tmp