pig-monkey.com - linux

The Things I Do for Time

2024-02-14T00:00:00-08:00

I am a believer in the sacred word as defined in ISO 8601, and the later revelations such as RFC 3339. Numerical dates should be formatted as YYYY-MM-DD. Hours should be written in 24-hour time. I will die on this hill.

Since time immemorial, this has been accomplished on Linux systems by setting LC_TIME to the en_DK locale. More specifically, the git history for glibc shows that en_DK was added (with ISO 8601 date formatting) by Ulrich Drepper on 1997-03-05.

A few years ago, this stopped working in Firefox. Instead Firefox started to think that numerical dates were supposed to be formatted as DD/MM/YYYY, which is at least as asinine as the typical American MM-DD-YYYY format. I finally got fed up with this and decided to investigate.

The best discussion of the issue is in Thunderbird bug 1426907. Here I learned that the problem is caused by Thunderbird (and by extension Firefox) no longer respecting glibc locales. Mozilla software simply takes the name of the system locale, ignores its definition, and looks up formatting in the Unicode CLDR. The CLDR has redefined en_DK to use DD/MM/YYYY¹.

The hack to address the problem was also documented in the Thunderbird bug report. The CLDR includes a definition for en_SE which uses YYYY-MM-DD² and 24-hour time. (It also separates the time from the date with a comma, which is weird, but Sweden is weird, so I’ll allow it.) There is no en_SE locale in glibc. But it can be created by linking to the en_DK locale. This new locale can then be used for LC_TIME.

$ sudo ln -s /usr/share/i18n/locales/en_DK /usr/share/i18n/locales/en_SE
$ echo 'en_SE.UTF-8 UTF-8' | sudo tee -a /etc/locale.gen
$ sudo locale-gen
$ sed -i 's/^LC_TIME=.*/LC_TIME=en_SE.UTF-8/' /etc/locale.conf

Now anything that respects glibc locales will effectively use en_DK, albeit under a different name. Anything that uses CLDR will just see that it is supposed to use a locale named en_SE, which still results in sane formatting. Thus one can use HTML date input fields without going crazy.

Notes

↵ The Unicode specification defines this pattern as "dd/MM/y", which is rather unintuitive, but worth including here for search engines.
↵ The Unicode specification defines this pattern as "y-MM-DD".

Redswitch

2021-10-17T00:00:00-07:00

Redshift is a program that adjusts the color temperature of the screen based on time and location. It can automatically fetch one’s location via GeoClue. I’ve used it for years. It works most of the time. But, more often than I’d like, it fails to fetch my location from GeoClue. When this happens, I find GeoClue impossible to debug. Redshift does not cache location information, so when it fails to fetch my location the result is an eye-meltingly bright screen at night. To address this, I wrote a small shell script to avoid GeoClue entirely.

Redswitch fetches the current location via the Mozilla Location Service (using GeoClue’s API key, which may go away). The result is stored and compared against the previous location to determine if the device has moved. If a change in location is detected, Redshift is killed and relaunched with the new location (this will result in a noticeable flash, but there seems to be no alternative since Redshift cannot reload its settings while running). If Redshift is not running, it is launched. If no change in location is detected and Redshift is already running, nothing happens. Because the location information is stored, this can safely be used to launch Redshift when the machine is offline (or when the Mozilla Location Service API is down or rate-limited).

My laptop does not experience frequent, drastic changes in location. I find that having the script automatically execute once upon login is adequate for my needs. If you’re jetting around the world, you could periodically execute the script via cron or a systemd timer.

This solves all my problems with Redshift. I can go back to forgetting about its existence, which is my goal for software of this sort.

Searching Books

2020-06-11T00:00:00-07:00

ripgrep-all is a small wrapper around ripgrep that adds support for additional file formats.

I discovered it while looking for a program that would allow me to search my e-book library without needing to open individual books and search their contents via Calibre. ripgrep-all accomplishes this by using Pandoc to convert files to plain text and then running ripgrep on the output. One of the numerous formats supported by Pandoc is EPUB, which is the format I use to store books.

Running Pandoc on every book in my library to extract its text can take some time, but ripgrep-all caches the extracted text so that subsequent runs are similar in speed to simply searching plain text – which is blazing fast thanks to ripgrep’s speed. It takes around two seconds to search 1,706 books.

$ time(rga -li 'pandemic' ~/library/books/ | wc -l)
33

real    0m1.225s
user    0m2.458s
sys     0m1.759s

I published my script for creating optical backups.

2020-04-23T00:00:00-07:00

Optician archives a directory, optionally encrypts it, records the integrity of all the things, and burns it to disc. I created it last year after writing about the steps I took to create optical backups of financial archives. Since then I’ve used it to create my monthly password database backups, yearly e-book library backups, and this year’s annual financial backup.

Personal Information Management

2019-12-14T00:00:00-08:00

pimutils is a collection of software for personal information management. The core piece is vdirsyncer, which synchronizes calendars and contacts between the local filesystem and CalDav and CardDAV servers. Calendars may then be interacted with via khal, and contacts via khard. There’s not much to say about these three programs, other than they all just work. Having offline access to my calendars and contacts is critical, as is the ability to synchronize that data across machines.

Khard integrates easily with mutt to provide autocomplete when composing emails. I find its interface for creating, editing and reading contacts to be intuitive. It can also output a calendar of birthdays, which can then be imported into khal.

Khal’s interface for adding new calendar events is much simpler and quicker than all the mousing required by GUI calendar programs.

$ khal new 2019-11-16 21:30 5h Alessandro Cortini at Public Works :: 161 Erie St

There are times when a more complex user interface makes calendaring tasks easier. For this Khal offers the interactive option, which provides a TUI for creating, editing and reading events.

Khal can also import iCalendar files, which is a simple way of getting existing events into my world.

$ khal import invite.ics

Vdirsyncer has maintenance problems that may call its future into question, but the whole point of modular tools that operate on open data formats is that they are replaceable.

I have a simple and often used script which calls khal calendar and task list (the latter command being taskwarrior), answering the question: what am I supposed to be doing right now?

Terminal Calculations

2019-12-12T00:00:00-08:00

Qalculate! is a well known GTK-based GUI calculator. For years I ignored it because I failed to realize that it included a terminal interface, qalc. Since learning about qalc last year it has become my go-to calculator. It supports all the same features as the GUI, including RPN and unit conversions. I primarily use GNU Units for unit wrangling, but being able to perform unit conversions within my calculator is sometimes useful.

$ qalc
> 1EUR to USD
It has been 20 day(s) since the exchange rates last were updated
Do you wish to update the exchange rates now? y

  1 * euro = approx. $1.1137000

> 32oC to oF

  32 * celsius = 89.6 oF

The RPN mode is not quite as intuitive as a purpose built RPN calculator like Orpie, but it is adequate for my uses. My most frequent use of RPN mode is totaling a long list of numbers without bothering with all those tedious + symbols.

> rpn on
> stack
The RPN stack is empty
> 85

  85 = 85

> 42

  42 = 42

> 198

  198 = 198

> 5

  5 = 5

> 659

  659 = 659

> stack

  1:    659
  2:    5
  3:    198
  4:    42
  5:    85

> total

  total([659, 5, 198, 42, 85]) = 989

> stack

  1:    989

Also provided are some basic statistics functions that can help save time.

> mean(2,12,5,3,1)
  mean([2, 12, 5, 3, 1]) = 4.6

And of course there are the varaibles and constants you would expect

> 12+3*8)/2
  (12 + (3 * 8)) / 2 = 18
> ans*pi
  ans * pi = 56.548668

I reach for qalc more frequently than alternative calculators like bc, insect, or the Python shell.

Mutt is my mail user agent of choice.

2019-07-31T00:00:00-07:00

More specifically, Neomutt. Mutt ICS is a python script which takes an iCalendar file and outputs the contents in a human friendly format. I use it in my mailcap so that I can see details of calendar attachments when reading email. It’s a simple script that improves my email workflow.

Without an OCR layer, PDF files are of limited use.

2019-07-30T00:00:00-07:00

OCRmyPDF is a tool that applies optical character recognition to PDFs. It uses Tesseract to perform the OCR, and unpaper to clean, deskew and optimize the input files. It outputs PDF/A files, optimized for long-term storage. This isn’t a tool I use frequently, but it is one I greatly appreciate having when I need it. If you ever find yourself scanning or photographing documents, you want OCRmyPDF.

Date Manipulation

2019-07-29T00:00:00-07:00

Dateutils is a collection of tools for the quick manipulation of dates. The tool I use most frequently is datediff. This program answers questions like: “How many days has it been since a date?” or “How many days are left in summer?”

$ datediff 2019-03-21 now
131
$ datediff now 2019-09-23
55

My second most frequently used program is dateadd, which is used to add a duration to a date. It can answer questions like: “What will the date be in 3 weeks?”

$ dateadd now +3w
2019-08-20T02:02:23

The tools are much more powerful than these examples, but hardly a week goes by when I don’t use datediff or dateadd for simple tasks like this.

Unit Wrangling

2019-07-24T00:00:00-07:00

I use GNU Units to convert measurements.

The program knows about many obscure and antiquated units, but I mostly use it for boring things like converting currencies and between metric and imperial units. It can be used directly from the command line, or via a prompted interactive mode.

$ units 57EUR USD
        * 63.526262
        / 0.015741521

$ units
Currency exchange rates from FloatRates (USD base) on 2019-07-24
3460 units, 109 prefixes, 109 nonlinear units

You have: 16 floz
You want: ml
        * 473.17647
        / 0.0021133764
You have: tempC(30)
You want: tempF
        86

GNU Units is picky about its unit definitions, and they are case sensitive. For example, it knows what USD is, but usd is undefined. It supports tab completion of units in interactive mode, which can be helpful. It knows the difference between a US fluid ounce and a British fluid ounce.

$ units "1 usfloz" ml
        * 29.57353
        / 0.033814023

$ units "1 brfloz" ml
        * 28.413063
        / 0.03519508

The unit definitions are stored at /usr/share/units/definitions.units. Occasionally I’ll need to peruse through this file to find the correct formatting for the unit I’m interested in. Sometimes when doing this I’ll run into one of the more obscure definitions, such as beespace. Apparently this unit is used in beekeeping when designing hive boxes. It is described in the definition file thusly: “Bees will fill any space that is smaller than the bee space and leave open spaces that are larger. The size of the space varies with species.”

$ units 12inches beespace
        * 48
        / 0.020833333

Every so often you need to know how many Earth days are in one Martian year. With GNU Units that information is a few keystrokes away.

$ units 1marsyear days
        * 686.97959
        / 0.0014556473

Currency definitions are stored in /var/lib/units/currency.units. They are updated using the units_cur program. In the past I would update currencies whenever I needed them, but recently I setup a systemd timer to update these definitions roughly once per day (depending on network connectivity). This provides me with conversion rates that are current enough for my own use, which I can take advantage of even when offline, and does not require me to let a third party know which currencies or quantities I am interested in.

Astute readers will have noted that I am big on this offline computing thing.

Undertime

2019-07-23T00:00:00-07:00

Undertime is a simple program that assists in coordinating events across time zones. It prints a table of your system’s local time zone, along with other any other specified zones. The output is colorized based on the start and end hour of the working day. If you want to talk to someone in Paris tomorrow, and you want the conversation to happen at an hour that is reasonable for both parties, Undertime can help.

I often find myself converting between local time and UTC. Usually this happens when working with system logs. If I have a specific date and time I want to translate, I’ll use date.

# Convert a time from PDT to UTC:
$ env TZ="UTC" date -d "2016-03-25T11:33 PDT"
# Convert a time from UTC to local:
$ date -d '2016-03-24T12:00 UTC'

If I’m not looking to convert an exact time, but just want to answer a more generalized question like “Approximately when was 14:00 UTC?” without doing the mental math, I find that Undertime is the quickest solution.

$ undertime UTC
╔═══════╦═══════╗
║  PDT  ║  UTC  ║
╠═══════╬═══════╣
║ 00:00 ║ 07:00 ║
║ 01:00 ║ 08:00 ║
║ 02:00 ║ 09:00 ║
║ 03:00 ║ 10:00 ║
║ 04:00 ║ 11:00 ║
║ 05:00 ║ 12:00 ║
║ 06:00 ║ 13:00 ║
║ 07:00 ║ 14:00 ║
║ 08:00 ║ 15:00 ║
║ 09:00 ║ 16:00 ║
║ 10:00 ║ 17:00 ║
║ 11:00 ║ 18:00 ║
║ 12:00 ║ 19:00 ║
║ 13:00 ║ 20:00 ║
║ 14:00 ║ 21:00 ║
║ 15:00 ║ 22:00 ║
║ 16:00 ║ 23:00 ║
║ 17:00 ║ 00:00 ║
║ 18:00 ║ 01:00 ║
║ 19:00 ║ 02:00 ║
║ 19:04 ║ 02:04 ║
║ 20:00 ║ 03:00 ║
║ 21:00 ║ 04:00 ║
║ 22:00 ║ 05:00 ║
║ 23:00 ║ 06:00 ║
╚═══════╩═══════╝
Table generated for time: 2019-07-23 19:04:00-07:00

Music Organization with Beets

2019-07-22T00:00:00-07:00

I organize my music with Beets.

Beets imports music into my library, warns me if I’m missing tracks, identifies tracks based on their accoustic fingerprint, scrubs extraneous metadata, fetches and stores album art, cleans genres, fetches lyrics, and – most importantly – fetches metadata from MusicBrainz. After some basic configuration, all of this happens automatically when I import new files into my library.

After the files have been imported, beets makes it easy to query my library based on any of the clean, consistent, high quality, crowd-sourced metadata.

$ beet stats genre:ambient
Tracks: 649
Total time: 2.7 days
Approximate total size: 22.4 GiB
Artists: 76
Albums: 53
Album artists: 34

$ beet ls -a 'added:2019-07-01..'
Deathcount in Silicon Valley - Acheron
Dlareme - Compass
The Higher Intelligence Agency & Biosphere - Polar Sequences
JK/47 - Tokyo Empires
Matt Morton - Apollo 11 Soundtrack

$ beet ls -ap albumartist:joplin
/home/pigmonkey/library/audio/music/Janis Joplin/Full Tilt Boogie
/home/pigmonkey/library/audio/music/Janis Joplin/I Got Dem Ol' Kozmic Blues Again Mama!

As regular readers will have surmised, the files themselves are stored in git-annex.

Terminal Countdown

2019-07-20T00:00:00-07:00

Termdown is a program that provides a countdown timer and stopwatch in the terminal. It uses FIGlet for its display. Its most attractive feature, I think, is the ability to support arbitrary script execution.

I use it most often as a countdown timer. One of my frequent applications is as a meditation timer. For this I want a 11 minute timer, with an alert at 10.5 minutes, 60 seconds, and 1 second. This gives me a 10 minute session with 30 seconds preparation and 30 seconds to return. Termdown makes this easy.

$ termdown --exec-cmd "case {0} in 630|30) mpv ~/library/audio/sounds/bell.mp3;; 1) mpv ~/library/audio/sounds/ring.mp3;; esac" 11m

An Offline Lexicon

2019-07-19T00:00:00-07:00

dictd is a dictionary database server and client. It can be used to lookup word definitions over a network. I don’t use it for that. I use the program to provide an offline dictionary. Depending on a network connection, web browser and third-party websites just to define a word strikes me as dumb.

To make this go, dictionary files must be installed. I use the GNU Collaborative International Dictionary of English (GCIDE), WordNet, and the Moby Thesaurus. The GCIDE is derived from Noah Webster’s famous American dictionary. WordNet is a more modern (one might say “dry”) resource. The Moby Thesaurus is a public domain thesaurus originally built by Grady Ward. Between these three sources I can have a pretty good grasp on the English language. No network connectivity required.

I use a shell alias to always pipe the definitions through less.

def () {
    dict $1 | less
}

Optical Backups of Financial Archives

2019-06-29T00:00:00-07:00

Every year I burn an optical archive of my financial documents, similar to how (and why) I create optical backups of photos. I schedule this financial archive for the spring, after the previous year’s taxes have been submitted and accepted. Taskwarrior solves the problem of remembering to complete the archive.

$ task add project:finance due:2019-04-30 recur:yearly wait:due-4weeks "burn optical financial archive with parity"

The archive includes two git-annex repositories.

The first is my ledger repository. Ledger is the double-entry accounting system I began using in 2012 to record the movement of every penny that crosses one of my bank accounts (small cash transactions, less than about $20, are usually-but-not-always except from being recorded). In addition to the plain-text ledger files, this repository also holds PDF or JPG images of receipts.

The second repository holds my tax information. Each tax year gets a ctmg container which contains any documents used to complete my tax returns, the returns themselves, and any notifications of those returns being accepted.

The yearly optical archive that I create holds the entirety of these two repositories – not just the information from the previous year – so really each disc only needs to have a shelf life of 12 months. Keeping the older discs around just provides redundancy for prior years.

Creating the Archive

The process of creating the archive is very similar to the process I outlined six years ago for the photo archives.

The two repositories, combined, are about 2GB (most of that is the directory of receipts from the ledger repository). I burn these to a 25GB BD-R disc, so file size is not a concern. I’ll tar them, but skip any compression, which would just add extra complexity for no gain.

$ mkdir ~/tmp/archive
$ cd ~/library
$ tar cvf ~/tmp/archive/ledger.tar ledger
$ tar cvf ~/tmp/archive/tax.tar tax

The ledger archive will get signed and encrypted with my PGP key. The contents of the tax repository are already encrypted, so I’ll skip encryption and just sign the archive. I like using detached signatures for this.

$ cd ~/tmp/archive
$ gpg -e -r peter@havenaut.net -o ledger.tar.gpg ledger.tar
$ gpg -bo ledger.tar.gpg.sig ledger.tar.gpg
$ gpg -bo tax.tar.sig tax.tar
$ rm ledger.tar

Previously, when creating optical photo archives, I used DVDisaster to create the disc image with parity. DVDisaster no longer exists. The code can still be found, and the program still works, but nobody is developing it and it doesn’t even an official web presence. This makes me uncomfortable for a tool that is part of my long-term archiving plans. As a result, I’ve moved back to using Parchive for parity. Parchive also does not have much in the way of active development around it, but it is still maintained, has been around for a long period of time, is still used by a wide community, and will probably continue to exist as long as people share files on less-than-perfectly-reliable mediums.

As previously mentioned, I’m not worried about the storage space for these files, so I tell par2create to create PAR2 files with 30% redundancy. I suppose I could go even higher, but 30% seems like a good number. By default this process will be allowed to use 16MB of memory, which is cute, but RAM is cheap and I usually have enough to spare so I’ll give it permission to use up to 8GB.

$ par2create -r30 -m8000 recovery.par2 *

Next I’ll use hashdeep to generate message digests for all the files in the archive.

$ hashdeep * > hashes

At this point all the file processing is completed. I’ll put a blank disc in my burner (a Pioneer BDR-XD05B) and burn the directory using growisofs.

$ growisofs -Z /dev/sr0 -V "Finances 2019" -r *

Verification

The final step is to verify the disc. I have a few options on this front. These are the same steps I’d take years down the road if I actually needed to recover data from the archive.

I can use the previous hashes to find any files that do not match, which is a quick way to identify bit rot.

$ hashdeep -x -k hashes *.{gpg,tar,sig,par2}

I can check the integrity of the PGP signatures.

$ gpg --verify tax.tar.gpg{.sig,}
$ gpg --verify tax.tar{.sig,}

I can use the PAR2 files to verify the original data files.

$ par2 verify recovery.par2

GOESImage

2019-06-17T00:00:00-07:00

GOESImage is a bash script which downloads the latest imagery from the NOAA Geostationary Operational Environment Satellites and sets it as the desktop background via feh. If you don’t use feh, it should be easy to plug GOESImage into any desktop background control program.

I wrote GOESImage after using himawaripy for a few years, which is a program that provides imagery of the Asia-Pacific region from the Himawari 8 Japanese weather satellite. I like seeing the Earth, and I’ve found that real time imagery of my location is actually useful for identifying the approach of large-scale weather systems. NOAA’s nighttime multispectral infrared coloring is pretty neat, too.