pig-monkey.com - osint

Monitoring Legible News

2020-01-31T00:00:00-08:00

I was sent a link to Legible News last November by someone who had read my post on the now-defunct Breaking News. Legible News is a website that simply scrapes headlines from Wikipedia’s Current Events once per day and presents them in a legible format. This seems like a simple thing, but is far beyond the capabilities of most news organizations today.

Legible News provides no update notification mechanism. I addressed this by plugging it into my urlwatch system. Initially this presented two problems: the email notification included the HTML markup, which I didn’t care about, and it included both the old and new content of every changed line – effectively sending me the news from today and yesterday.

The first problem was easily solved by using the html2text filter provided by urlwatch. This strips out all markup, which is what I thought I wanted. I ran this for a bit before deciding that I did want the output to contain links. What I really wanted was some sort of html2markdown filter.

I also realized I did not just want to be sent new lines, but every line anytime there was a change. If the news yesterday included a section titled “Armed conflicts and attacks”, and the news today included a section with the same title, I wanted that in my output despite it not having changed.

I solved both of these problems using the diff_tool argument of urlwatch. This allows the user to pass in a special tool to replace the default use of diff to generate the notification output. The tool will be called with two arguments: the filename of the previously downloaded version of the URL and the filename of the current version. I wrote a simple script called html2markdown.sh which ignores the first argument and simply passes the second argument to Pandoc for formatting.

#!/bin/sh

pandoc --from html \
--to markdown_strict \
--reference-links \
--reference-location=block \
$2

This script is used as the diff_tool in the urlwatch job definition.

kind: url
name: Legible News
url: https://legiblenews.com/
diff_tool: /home/pigmonkey/bin/html2markdown.sh

The result is the latest version of Legible News, nicely converted to Markdown, delivered to my inbox every day. The output would be even better if Legible News used semantic markup – specifically heading elements – but it is perfectly serviceable as is.

After I built this I discovered that somebody had created an RSS feed for Legible News using a service called Feed43.

I don't know anything about, or have much interest in, high-frequency trading.

2019-01-06T00:00:00-08:00

But some of the technology behind it is fascinating. This past summer the Sniper in Mahwah blog published a four part series investigating the use of shortwave radio as a low latency link in high-frequency trading. I’d call it the best piece of hacker-tourism since Mother Earth Mother Board, but I think it’s probably the only piece of hacker-tourism since Mother Earth Mother Board. It doesn’t have much competition.

I track disasters via RSOE EDIS.

2018-12-01T00:00:00-08:00

Operated by the Hungarian National Association of Radio Distress-Signalling and Infocommunications (RSOE), the Emergency and Disaster Information Service (EDIS) collects disaster information from around the globe, and disseminates it using the Common Alerting Protocol. They offer the data via web map, email, RSS, and Android application (an API is also available, though keys are apparently restricted to government organizations). There is a wide range of “disasters” included, but the normalized protocol supports filtering the events based on criteria such as scope, severity, and urgency. I use the Android application, configured to only show emergency-level alerts.

For earthquakes, I supplement RSOE EDIS with the USGS Earthquake Notification Service, which provides regional-based subscriptions. I subscribe to email alerts for earthquakes greater than 6.0 for all of the US, and greater than 4.0 for my local area.

Breaking News

2016-07-22T00:00:00-07:00

In the past I’ve struggled to find a way to be alerted to breaking news, were “breaking news” is defined as things that matter¹.

For some time, Scanner Radio Pro was my preferred solution. The Android application uses Broadcastify to stream feeds of police and EMS radio (among other sources). While a police scanner is always a good thing to have, the application is relevant here because of its notification support. It will push notifications when a channel has a certain number of users listening to it. The application default threshold is 3500 listeners for all channels, but it also allows you to configure per-channel thresholds. I set the threshold to 200 listeners for San Francisco police and EMS channels. I also have the threshold set to 200 for any station within 50 miles of my current location. Those two are redundant when I’m in San Francisco, but it means that if I’m travelling I’ll receive alerts for wherever I am, and I’ll be able to stay up to date on what’s happening back in the city.

The alerts simply tell me that a large number of people are listening to a certain channel, which indicates that something is probably going on in that location. It doesn’t tell me what is happening, but prompts me to figure that out for myself. When the shooting started in Dallas a couple weeks ago, I was alerted immediately that something was happening due to the unusually high number of people listening to the Dallas PD channel. When I saw that, I performed a quick search for Dallas news and discovered the reason.

I’ve had good success with these alerts. It has worked well for shootings like in Dallas. The alerts kept me informed of disruptions in the area this past winter when the Super Bowl was in town. Unfortunately, most of the Broadcastify feeds and listeners are US-focused, so it tends not to help with events outside of the country. There were no alerts for the attack in Nice, France.

More recently I’ve begun supplementing Scanner Radio Pro with the conveniently named Breaking News. They are basically a modern wire service, collating data from traditional news sources, social media channels like Twitter, and direct user submissions. Human editors manage the service, which does seem to take care of some of the cruft. Events are grouped into topics, and topics can be muted or subscribed to. Their mobile application supports push notifications for global major stories, as well as stories based on geographic proximity to the device’s current location.

I’ve noticed that the application does have an impact on my battery (even with the “battery saver” option enabled), but I think it’s worth it for the service provided. They provide alerts for a wider breadth of topics than the police scanner, and provide immediate context for the alert. If they provide an alert for an event that I don’t care about, I can mute the topic and never hear about it again. It’s rare that I need to do this, as they seem judicious in their use of push notifications.

Scanner Radio Pro and the Breaking News application live on the home screen of my phone in a directory labelled “Intelligence”. Along with applications like Flightradar24 and Marine Traffic (like Flightradar24 but for boats), they help provide context for and build awareness of the world around me.

Notes

↵ Not celebrities. Not sports.

Russia's latest spyplane is broadcasting its flight home from Syria.

2016-02-29T00:00:00-08:00

The Aviationist pointed out that the plane had its ADS-B transponder on, allowing it to be tracked on Flightradar24. The Tu-214R also broadcast its deployment to Syria on the 15th.

An older article on The Aviationist discusses the impact of ADS-B and MLAT tracking on military craft. Somewhat related is a recent Defense One article on using commercial satellite imagery to identify airstrip expansion.

Watch the Sky

2016-02-04T00:00:00-08:00

I learned about Flighradar24 last year. They offer a live flight radar, using data provided by ADS-B on top of Google Maps.

Immediately after the San Bernadino shooting in December, a Cessna 182 with the registration number N404KR flew circles around the area for hours. This plane is registered to OBR Leasing, which is one of the shell companies the FBI uses for surveillance. The ability to observe these types of flights is interesting, and sometimes useful.

The airspace above San Francisco is usually fairly dead. From what I’ve seen, most commercial jets heading in and out of SFO or OAK avoid flying directly over the city. This week I’ve been watching the radar more frequently than usual. With the Super Bowl in town, I assumed there would be more interesting traffic. I use the radar to look for suspicious flights, which to me means smaller craft with circular paths.

N2462G has what must be a mind-numbingly boring job. Every day, the Cessna 206 takes off from the Concord area (possibly Buchanan) and flies circles around the Embarcadero and Treasure Island. The aircraft is registered to an aerial sports broadcasting company, so I assume they’re just shooting footage of Super Bowl City.

More interesting are the government flights. Today during lunch I watched N6241L. The Beech Super King took off from Mountain View, flew up the coast, and made circles around the Golden Gate. A quick search showed this to be registered to the Department of Homeland Security. The radar isn’t limited to fixed wing craft. N3933A, a Eurocopter Squirrel also registered to DHS, was flying down the coast at about the same time.

There are of course flights that don’t show up on the radar, but the service provides an idea of what’s going on in the sky. I think the Android application is a good buy. Combined with a police scanner, you can begin to turn a smartphone into a mobile intelligence gathering toolkit. It is useful for locating excitement, whether you’re trying to avoid it, get into it, or just be aware of it.