pig-monkey.com - toolchain

Redswitch

2021-10-17T00:00:00-07:00

Redshift is a program that adjusts the color temperature of the screen based on time and location. It can automatically fetch one’s location via GeoClue. I’ve used it for years. It works most of the time. But, more often than I’d like, it fails to fetch my location from GeoClue. When this happens, I find GeoClue impossible to debug. Redshift does not cache location information, so when it fails to fetch my location the result is an eye-meltingly bright screen at night. To address this, I wrote a small shell script to avoid GeoClue entirely.

Redswitch fetches the current location via the Mozilla Location Service (using GeoClue’s API key, which may go away). The result is stored and compared against the previous location to determine if the device has moved. If a change in location is detected, Redshift is killed and relaunched with the new location (this will result in a noticeable flash, but there seems to be no alternative since Redshift cannot reload its settings while running). If Redshift is not running, it is launched. If no change in location is detected and Redshift is already running, nothing happens. Because the location information is stored, this can safely be used to launch Redshift when the machine is offline (or when the Mozilla Location Service API is down or rate-limited).

My laptop does not experience frequent, drastic changes in location. I find that having the script automatically execute once upon login is adequate for my needs. If you’re jetting around the world, you could periodically execute the script via cron or a systemd timer.

This solves all my problems with Redshift. I can go back to forgetting about its existence, which is my goal for software of this sort.

Browser Extensions

2021-10-01T00:00:00-07:00

I try to keep the number of browser extensions I use to a minimum. The following are what I find necessary in Firefox.

ClearURLS

ClearURLs removes extra cruft from URLs. I don’t really a problem with things like UTM parameters. Such things seem reasonable to me. But, more broadly, digital advertising has proved itself hostile to my interests, so I choose to be hostile right back.

Cookie AutoDelete

Cookie AutoDelete deletes cookies after a tab is closed or the domain changes. I whitelist cookies for some of the services I run, like my RSS reader, but every other cookie gets deleted 10 seconds after I leave the site. The extension can also manage other data stores, like IndexedDB and Local Storage.

Feed Preview

Feed Preview adds an icon to the address bar when a page includes an RSS or Atom feed in its header. This used to be built in to Firefox, but for some inexplicable reason they removed it some years ago now. Removing the icon broke one of the core ways that I use a web browser. As the name suggests, the extension can also render a preview of the feed. I don’t use it for that. I just want my icon back.

Firefox Multi-Account Containers

Firefox Multi-Account Containers is a Mozilla provided extension to create different containers and assign domains to them. In modern web browser parlance, a container means isolated storage. So a cookie in container A is not visible within container B, and vice versa.

Temporary Containers

Temporary Containers is the real workhorse of my containment strategy. It generates a new, temporary container for every domain. It automatically deletes the containers it generates 5 minutes after the last tab in that container is closed. This effectively isolates all domains from one another.

History Cleaner

History Cleaner deletes browser history that is older than 200 days. History is useful, but if I haven’t visited a URL in more than 200 days, I probably no longer care about. Having all that cruft automatically cleaned out makes it easier to find what I’m looking for in the remaining history, and speeds up autocomplete in the address bar.

Redirector

Redirector lets you create pattern-based URL redirects. I use it to redirect Reddit URLs to Teddit, Twitter URLs to Nitter, and Wikipedia mobile URLs to the normal Wikipedia site.

Stylus

Stylus allows custom CSS to be applied to websites. I use it to make websites less eye-burningly-bright. Dark Reader is another solution to this problem, but I found it to be somewhat resource intensive. Stylus lets me darken websites with no performance penalty.

Tree Style Tab

Tree Style Tab moves tabs from the default horizontal bar across the top of the browser chrome to a vertical sidebar, and allows the tabs to be placed into a nested tree-like hierarchy. In a recent-ish version of Firefox, Mozilla uglified the default horizontal tab bar. This was what finally pushed me into adopting tree style tabs. It took me a couple weeks to get used to it, but now I’m a convert. I wouldn’t want to use a browser without it. Unfortunately, the extension does seem to have a performance penalty. Not so much during normal use, but it definitely increases the time required to launch the browser. To me, it is worth it.

uBlock Origin

uBlock Origin blocks advertisements, malware, and other waste. This extension should need no introduction. The modern web is unusable without it. Until recently I used this in combination with uMatrix. I removed uMatrix when it was abandoned by the author, but was pleasantly surprised to find that current versions of uBlock by itself satisfies my needs in this department.

User-Agent Switcher

User-Agent Switcher allows the user-agent string to be changed. It seems odd that the user would need an extension to change the user-agent string in their user agent, but here we are. I mostly use this for testing things.

Vim Vixen

Vim Vixen allows the browser to be controlled using vim-like keys. Back in those halcyon days before Mozilla broke their extension system, I switched between two extensions called Vimperator and Pentadactyl to accomplish this. Those were both complete extensions that were able to improve every interaction point with the browser. Vim Vixen is an inferior experience, but seems to be the best current solution. It’s mostly alright.

Wallabagger

Wallabagger lets me save articles to my Wallabag instance with a single click.

Web Archives

Web Archives allows web pages to be looked up in various archives. I just use it for quick access to the Internet Archive’s Wayback Machine.

Organizing Ledger

2020-08-03T00:00:00-07:00

Ledger is a double-entry accounting system that stores data in plain text. I began using it in 2012. Almost every dollar that has passed through my world since then is tracked by Ledger.¹

Ledger is not the only plain text accounting system out there. It has inspired others, such as hledger and beancount. I began with Ledger for lack of a compelling argument in favor of the alternatives. After close to a decade of use, my only regret is that I didn’t start using earlier.

My Ledger repository is stored at ~/library/ledger. This repository contains a data directory, which includes yearly Ledger journal files such as data/2019.ldg and data/2020.ldg. Ledger files don’t necessarily need to be split at all, but I like having one file per year. In January, after I clear the last transaction from the previous year, I know the year is locked and the file never gets touched again (unless I go back in to rejigger my account structure).

The root of the directory has a .ledger file which includes all of these data files, plus a special journal file with periodic transactions that I sometimes use for budgeting. My ~/.ledgerrc file tells Ledger to use the .ledger file as the primary journal, which has the effect of including all the yearly files.

$ cat ~/.ledgerrc
--file ~/library/ledger/.ledger
--date-format=%Y-%m-%d

$ cat ~/library/ledger/.ledger
include data/periodic.ldg
include data/2012.ldg
include data/2013.ldg
include data/2014.ldg
include data/2015.ldg
include data/2016.ldg
include data/2017.ldg
include data/2018.ldg
include data/2019.ldg
include data/2020.ldg

Ledger’s include format does support globbing (ie include data/*.ldg) but the ordering of the transactions can get weird, so I prefer to be explicit.

The repository also contains receipts in the receipts directory, invoices in the invoices directory, scans of checks (remember those?) in the checks directory, and CSV dumps from banks in the dump directory.

$ tree -d ~/library/ledger
/home/pigmonkey/library/ledger
├── checks
├── data
├── dump
├── invoices
└── receipts

5 directories

The repository is managed using a mix of vanilla git and git-annex.² It is important to me that the Ledger journal files in the data directory are stored directly in git. I want the ability to diff changes before committing them, and to be able to pull the history of those files. Every other file I want stored in git-annex. I don’t care about the history of files like PDF receipts. They never change. In fact, I want to make them read-only so I can’t accidentally change them. I want encrypted versions of them distributed to my numerous special remotes for safekeeping, and someday I may even want to drop old receipts or invoices from my local store so that they don’t take up disk space until I actually need to read them. That sounds like asking a lot, but git-annex magically solves all the problems with its largefiles configuration option.

$ cat ~/library/ledger/.gitattributes
*.ldg annex.largefiles=nothing

This tells git-annex that any file ending with *.ldg should not be treated as a “large file”, which means it should be added directly to git. Any other file should be added to git-annex and locked, making it read-only. Having this configured means that I can just blindly git annex add . or git add . within the repository and git-annex will always do the right thing.

I don’t run the git-annex assistant in this repository because I don’t want any automatic commits. Like a traditional git repository, I only commit changes to Ledger’s journal files after reviewing the diffs, and I want those commits to have meaningful messages.

Notes

↵ I do not always track miscellaneous cash transactions less than $20. If a thing costs more than that, it is worth tracking, regardless of what it is or how it was purchased. If it costs less than that, and it isn't part of a meaningful expense account, I'll probably let laziness win out. If I buy a $8 sandwich for lunch with cash, it'll get logged, because I care about tracking dining expenses. If I buy a $1 pencil erasure, I probably won't log it, because it isn't part of an account worth considering.
↵ I bet you saw that coming.

Searching Books

2020-06-11T00:00:00-07:00

ripgrep-all is a small wrapper around ripgrep that adds support for additional file formats.

I discovered it while looking for a program that would allow me to search my e-book library without needing to open individual books and search their contents via Calibre. ripgrep-all accomplishes this by using Pandoc to convert files to plain text and then running ripgrep on the output. One of the numerous formats supported by Pandoc is EPUB, which is the format I use to store books.

Running Pandoc on every book in my library to extract its text can take some time, but ripgrep-all caches the extracted text so that subsequent runs are similar in speed to simply searching plain text – which is blazing fast thanks to ripgrep’s speed. It takes around two seconds to search 1,706 books.

$ time(rga -li 'pandemic' ~/library/books/ | wc -l)
33

real    0m1.225s
user    0m2.458s
sys     0m1.759s

Monitoring Legible News

2020-01-31T00:00:00-08:00

I was sent a link to Legible News last November by someone who had read my post on the now-defunct Breaking News. Legible News is a website that simply scrapes headlines from Wikipedia’s Current Events once per day and presents them in a legible format. This seems like a simple thing, but is far beyond the capabilities of most news organizations today.

Legible News provides no update notification mechanism. I addressed this by plugging it into my urlwatch system. Initially this presented two problems: the email notification included the HTML markup, which I didn’t care about, and it included both the old and new content of every changed line – effectively sending me the news from today and yesterday.

The first problem was easily solved by using the html2text filter provided by urlwatch. This strips out all markup, which is what I thought I wanted. I ran this for a bit before deciding that I did want the output to contain links. What I really wanted was some sort of html2markdown filter.

I also realized I did not just want to be sent new lines, but every line anytime there was a change. If the news yesterday included a section titled “Armed conflicts and attacks”, and the news today included a section with the same title, I wanted that in my output despite it not having changed.

I solved both of these problems using the diff_tool argument of urlwatch. This allows the user to pass in a special tool to replace the default use of diff to generate the notification output. The tool will be called with two arguments: the filename of the previously downloaded version of the URL and the filename of the current version. I wrote a simple script called html2markdown.sh which ignores the first argument and simply passes the second argument to Pandoc for formatting.

#!/bin/sh

pandoc --from html \
--to markdown_strict \
--reference-links \
--reference-location=block \
$2

This script is used as the diff_tool in the urlwatch job definition.

kind: url
name: Legible News
url: https://legiblenews.com/
diff_tool: /home/pigmonkey/bin/html2markdown.sh

The result is the latest version of Legible News, nicely converted to Markdown, delivered to my inbox every day. The output would be even better if Legible News used semantic markup – specifically heading elements – but it is perfectly serviceable as is.

After I built this I discovered that somebody had created an RSS feed for Legible News using a service called Feed43.

I use FeedIron to repair neutered RSS feeds.

2019-12-17T00:00:00-08:00

FeedIron is a plugin for my feed reader, Tiny Tiny RSS. It takes broken, partial feeds and extracts the full article content, allowing me to read the article in my feed reader the way god intended. The plugin can be configured to extract content using a number of filters. I find that using the xpath filter to specify an element on the page like div[@class='entry-content'] corrects most neutered feeds.

I use urlwatch to monitor the global information super highway.

2019-12-15T00:00:00-08:00

urlwatch is a simple program that monitors a list of URLs and sends an alert when it detects a change. It can be configured to only look for changes within certain HTML elements, or to grep for certain strings. I configure it to send me the changes via email. As with RSS-Bridge, this tool is part of my strategy to liberate content from toxic silos and Make the Internet Great Again™.

Personal Information Management

2019-12-14T00:00:00-08:00

pimutils is a collection of software for personal information management. The core piece is vdirsyncer, which synchronizes calendars and contacts between the local filesystem and CalDav and CardDAV servers. Calendars may then be interacted with via khal, and contacts via khard. There’s not much to say about these three programs, other than they all just work. Having offline access to my calendars and contacts is critical, as is the ability to synchronize that data across machines.

Khard integrates easily with mutt to provide autocomplete when composing emails. I find its interface for creating, editing and reading contacts to be intuitive. It can also output a calendar of birthdays, which can then be imported into khal.

Khal’s interface for adding new calendar events is much simpler and quicker than all the mousing required by GUI calendar programs.

$ khal new 2019-11-16 21:30 5h Alessandro Cortini at Public Works :: 161 Erie St

There are times when a more complex user interface makes calendaring tasks easier. For this Khal offers the interactive option, which provides a TUI for creating, editing and reading events.

Khal can also import iCalendar files, which is a simple way of getting existing events into my world.

$ khal import invite.ics

Vdirsyncer has maintenance problems that may call its future into question, but the whole point of modular tools that operate on open data formats is that they are replaceable.

I have a simple and often used script which calls khal calendar and task list (the latter command being taskwarrior), answering the question: what am I supposed to be doing right now?

Terminal Calculations

2019-12-12T00:00:00-08:00

Qalculate! is a well known GTK-based GUI calculator. For years I ignored it because I failed to realize that it included a terminal interface, qalc. Since learning about qalc last year it has become my go-to calculator. It supports all the same features as the GUI, including RPN and unit conversions. I primarily use GNU Units for unit wrangling, but being able to perform unit conversions within my calculator is sometimes useful.

$ qalc
> 1EUR to USD
It has been 20 day(s) since the exchange rates last were updated
Do you wish to update the exchange rates now? y

  1 * euro = approx. $1.1137000

> 32oC to oF

  32 * celsius = 89.6 oF

The RPN mode is not quite as intuitive as a purpose built RPN calculator like Orpie, but it is adequate for my uses. My most frequent use of RPN mode is totaling a long list of numbers without bothering with all those tedious + symbols.

> rpn on
> stack
The RPN stack is empty
> 85

  85 = 85

> 42

  42 = 42

> 198

  198 = 198

> 5

  5 = 5

> 659

  659 = 659

> stack

  1:    659
  2:    5
  3:    198
  4:    42
  5:    85

> total

  total([659, 5, 198, 42, 85]) = 989

> stack

  1:    989

Also provided are some basic statistics functions that can help save time.

> mean(2,12,5,3,1)
  mean([2, 12, 5, 3, 1]) = 4.6

And of course there are the varaibles and constants you would expect

> 12+3*8)/2
  (12 + (3 * 8)) / 2 = 18
> ans*pi
  ans * pi = 56.548668

I reach for qalc more frequently than alternative calculators like bc, insect, or the Python shell.

I use Blokada to reduce the amount of advertisements on my telephone.

2019-11-04T00:00:00-08:00

Blokada registers itself as a VPN service on the phone so that it can intercept all network traffic. It then downloads filter lists to route the domains of known advertisers, trackers, etc to a black hole, exactly like what I do on my real computer with hostsctl. For me it has had no noticeable impact on battery life. I have found it especially useful when travelling internationally and purchasing cellular plans with small data caps. The only disadvantage I have found is that Blokada must be disabled when I want to connect to a real VPN via WireGuard or OpenVPN.

Blokada must be installed via F-Droid (or directly through the APK) because Google frowns upon blocking advertisements (but at least Google allows you to install software on your telephone outside of their walled garden, unlike their competitor).

I use RSS-Bridge to stitch together the balkanized web.

2019-10-14T00:00:00-07:00

RSS-Bridge is an open-source project that liberates content from toxic walled gardens, allowing it to be shared and syndicated in my feed reader. The project can generate RSS or Atom feeds for a number of sites. It let’s me pretend that we live in a better time.

Mutt is my mail user agent of choice.

2019-07-31T00:00:00-07:00

More specifically, Neomutt. Mutt ICS is a python script which takes an iCalendar file and outputs the contents in a human friendly format. I use it in my mailcap so that I can see details of calendar attachments when reading email. It’s a simple script that improves my email workflow.

Without an OCR layer, PDF files are of limited use.

2019-07-30T00:00:00-07:00

OCRmyPDF is a tool that applies optical character recognition to PDFs. It uses Tesseract to perform the OCR, and unpaper to clean, deskew and optimize the input files. It outputs PDF/A files, optimized for long-term storage. This isn’t a tool I use frequently, but it is one I greatly appreciate having when I need it. If you ever find yourself scanning or photographing documents, you want OCRmyPDF.

Date Manipulation

2019-07-29T00:00:00-07:00

Dateutils is a collection of tools for the quick manipulation of dates. The tool I use most frequently is datediff. This program answers questions like: “How many days has it been since a date?” or “How many days are left in summer?”

$ datediff 2019-03-21 now
131
$ datediff now 2019-09-23
55

My second most frequently used program is dateadd, which is used to add a duration to a date. It can answer questions like: “What will the date be in 3 weeks?”

$ dateadd now +3w
2019-08-20T02:02:23

The tools are much more powerful than these examples, but hardly a week goes by when I don’t use datediff or dateadd for simple tasks like this.

Unit Wrangling

2019-07-24T00:00:00-07:00

I use GNU Units to convert measurements.

The program knows about many obscure and antiquated units, but I mostly use it for boring things like converting currencies and between metric and imperial units. It can be used directly from the command line, or via a prompted interactive mode.

$ units 57EUR USD
        * 63.526262
        / 0.015741521

$ units
Currency exchange rates from FloatRates (USD base) on 2019-07-24
3460 units, 109 prefixes, 109 nonlinear units

You have: 16 floz
You want: ml
        * 473.17647
        / 0.0021133764
You have: tempC(30)
You want: tempF
        86

GNU Units is picky about its unit definitions, and they are case sensitive. For example, it knows what USD is, but usd is undefined. It supports tab completion of units in interactive mode, which can be helpful. It knows the difference between a US fluid ounce and a British fluid ounce.

$ units "1 usfloz" ml
        * 29.57353
        / 0.033814023

$ units "1 brfloz" ml
        * 28.413063
        / 0.03519508

The unit definitions are stored at /usr/share/units/definitions.units. Occasionally I’ll need to peruse through this file to find the correct formatting for the unit I’m interested in. Sometimes when doing this I’ll run into one of the more obscure definitions, such as beespace. Apparently this unit is used in beekeeping when designing hive boxes. It is described in the definition file thusly: “Bees will fill any space that is smaller than the bee space and leave open spaces that are larger. The size of the space varies with species.”

$ units 12inches beespace
        * 48
        / 0.020833333

Every so often you need to know how many Earth days are in one Martian year. With GNU Units that information is a few keystrokes away.

$ units 1marsyear days
        * 686.97959
        / 0.0014556473

Currency definitions are stored in /var/lib/units/currency.units. They are updated using the units_cur program. In the past I would update currencies whenever I needed them, but recently I setup a systemd timer to update these definitions roughly once per day (depending on network connectivity). This provides me with conversion rates that are current enough for my own use, which I can take advantage of even when offline, and does not require me to let a third party know which currencies or quantities I am interested in.

Astute readers will have noted that I am big on this offline computing thing.

Undertime

2019-07-23T00:00:00-07:00

Undertime is a simple program that assists in coordinating events across time zones. It prints a table of your system’s local time zone, along with other any other specified zones. The output is colorized based on the start and end hour of the working day. If you want to talk to someone in Paris tomorrow, and you want the conversation to happen at an hour that is reasonable for both parties, Undertime can help.

I often find myself converting between local time and UTC. Usually this happens when working with system logs. If I have a specific date and time I want to translate, I’ll use date.

# Convert a time from PDT to UTC:
$ env TZ="UTC" date -d "2016-03-25T11:33 PDT"
# Convert a time from UTC to local:
$ date -d '2016-03-24T12:00 UTC'

If I’m not looking to convert an exact time, but just want to answer a more generalized question like “Approximately when was 14:00 UTC?” without doing the mental math, I find that Undertime is the quickest solution.

$ undertime UTC
╔═══════╦═══════╗
║  PDT  ║  UTC  ║
╠═══════╬═══════╣
║ 00:00 ║ 07:00 ║
║ 01:00 ║ 08:00 ║
║ 02:00 ║ 09:00 ║
║ 03:00 ║ 10:00 ║
║ 04:00 ║ 11:00 ║
║ 05:00 ║ 12:00 ║
║ 06:00 ║ 13:00 ║
║ 07:00 ║ 14:00 ║
║ 08:00 ║ 15:00 ║
║ 09:00 ║ 16:00 ║
║ 10:00 ║ 17:00 ║
║ 11:00 ║ 18:00 ║
║ 12:00 ║ 19:00 ║
║ 13:00 ║ 20:00 ║
║ 14:00 ║ 21:00 ║
║ 15:00 ║ 22:00 ║
║ 16:00 ║ 23:00 ║
║ 17:00 ║ 00:00 ║
║ 18:00 ║ 01:00 ║
║ 19:00 ║ 02:00 ║
║ 19:04 ║ 02:04 ║
║ 20:00 ║ 03:00 ║
║ 21:00 ║ 04:00 ║
║ 22:00 ║ 05:00 ║
║ 23:00 ║ 06:00 ║
╚═══════╩═══════╝
Table generated for time: 2019-07-23 19:04:00-07:00