I began using Joplin for keeping notes a few months ago; it may have been Mischa of OpenBSD Amsterdam who recommended it, but in any case he raves about it. I’ll admit it’s the note-taking solution I like best so far, and I’ve been through a number of “solutions”.

screenshot of Joplin on macOS

Joplin has three features which convinced me to adopt it:

  • WebDAV as one of the supported cloud services
  • Clients for macOS, iOS, and Android
  • Markdown syntax for the notes

In addition to different methods for importing and exporting data (I wouldn’t want to lock myself in, would I now), Joplin has an API. It is enabled by launching Joplin’s Web Clipper service via its preferences and listens for HTTP requests on port 41184 by default. The preference page also lists the token required for accessing the API.

Tweets in Joplin

For a long time I’ve wanted to be able to collect specific tweets in my notes, and I thought I’d use the API to do so. The Python program can surely be improved upon, but it works well for what I want it to do.

I give it a URL to an individual tweet (it could also go through my favorites, but I don’t necessarily want only favorites), and the utility uses Tweepy to grab the status’ text, downloads its images and uploads these as so-called resources to Joplin, and then creates a new note. The Markdown of the note has image links to Joplin’s resources.

The screenshots show the result: above on macOS, below on iOS.

a Joplin note on an iPhone

Joplin’s been very reliable so far, and I hope that doesn’t change with me battering it this way. Updating Joplin is a bit cumbersome on macOS because I have to download and install a .dmg which I then allow macOS to launch, but I can live with that.

notes and api :: 09 Oct 2020 :: e-mail

I miss travel, and I miss flying. At the end of March Lufthansa canceled two transatlantic flights I had booked, and that was that; no more flying for me (or for almost anybody else). Is that a reason I’m currently interested in IATA airport codes? Maybe it’s the desire to fly to far-away places.

I began using a list of six thousand airport codes when I developed the first Ansible training, using them as fodder in an application students have to set up, and I decided to use the same data set in an exercise for an advanced Ansible training we’re developing.

The problem with the data I had is that I was about 97.2% sure that I was permitted to use it, but there wasn’t a really clear license associated with it. I’ve meanwhile found OurAirports; they provide a public domain data set, and on the weekend I had a bit of fun with it.

First I wrote an HTTP airport data server which will be running on lab machines. Students will be tasked with the assembly of an Ansible lookup plugin to obtain data from that server for use in templates. (I should probably clarify that my trainings are designed to run without a connection to the Internet – there have been a number of occasions when I’ve had to give trainings in environments which just do not permit their users to use such resources, hence the local service.)

   vars:
      fra: "{{ lookup('airport', 'fra') }}"
   tasks:
      - debug: var=fra
TASK [debug] ************************************************************************
ok: [localhost] => {
    "fra": {
        "cc": "DE",
        "city": "Frankfurt am Main",
        "iata": "FRA",
        "id": "2212",
        "lat": "50.033333",
        "lon": "8.570556",
        "name": "Frankfurt am Main Airport",
        "osm": "https://openstreetmap.org/?mlat=50.033333&mlon=8.570556&zoom=12",
        "type": "large_airport"
    }
}

Just put it in the DNS!

I thought it’d be amusing to provide the public domain data via the DNS, so I did just that. Each airport has a couple of TXT and a LOC record associated with it. The domain name is the 3-letter IATA code:

$ dig +short BCN.air.jpmens.net TXT
"cc:ES; m:Barcelona; t:large, n:Barcelona International Airport"
$ dig +short BCN.air.jpmens.net LOC
41 17 49.560 N 2 4 42.456 E 4.00m 1m 10000m 10m

we also provide a URI which loads an OpenStreetmap.org map to the correct location, and if it’s in the source data, a URI to the Wikipedia page:

$ dig +short CDG.air.jpmens.net URI
10 1 "https://openstreetmap.org/?mlat=49.012798&mlon=2.55&zoom=12"
10 2 "https://en.wikipedia.org/wiki/Charles_de_Gaulle_Airport"

You can obtain the IATA codes indexed by city, providing you spell it as OurAirports has, and in addition to IDNA names I’ve un-unicoded the names, so Münster becomes “munster”, and Tromsø becomes “tromso”. Query the domain for TXT records for the city:

$ dig +short PARIS.air.jpmens.net TXT
"PHT," "PRX," "LBG," "CDG," "ORY"

I generate a file with resource records I then $INCLUDE into a zone master file, and I’m making this zone data file available in this repository for self-hosting. Do tell me if you use it, and I’ll add a pointer to your DNS server if you like.

I assume the data is relatively static (who is building airports nowadays – even BER has been completed but marked “closed”), but if we should notice it’s very volatile we can easily add dynamic DNS updates to update the DNS on the fly.

Whether this is useful is in the eye of the beholder. I can say, that I quite frequently use my country-code lookup service in the DNS.

$ dig +short JP.cc.jpmens.net TXT
"JAPAN"

So, have I now finally earned myself this mug?

put it in the DNS mug

As usual, do talk to me if you have ideas for improvement.

Updates

  • Standing of the shoulders of a giant, I’ve added support for IDNA in the city to IATA mappings; use a client which has IDNA support to query for “düsseldorf”
  • Mentioned on Stéphane Bortzmeyer’s on peut tout mettre dans le DNS, même les codes postaux.
  • Oli Schacher whipped up a service which provides the locations of postal codes in Switzerland via the DNS in the zone zipdns.ch, e.g. via the looking glass: zipdns via the looking glass
  • Inspired by Stéphane’s and Oli’s work on zip codes, I’ve set up an additional zone with forward/reverse lookups of zip codes in Germany (Postleitzahlen), based on publicly available data.
  $ dig PASSAU.zipde.jpmens.net TXT
  ;; ANSWER SECTION:
  PASSAU.zipde.jpmens.net. 604800	IN	TXT	"94036"
  PASSAU.zipde.jpmens.net. 604800	IN	TXT	"94034"
  PASSAU.zipde.jpmens.net. 604800	IN	TXT	"94032"

  $ dig 94034.zipde.jpmens.net TXT
  ;; ANSWER SECTION:
  94034.zipde.jpmens.net.	604800	IN	TXT	"Passau"
  
  • Florian blames me became inspired by what we all did here, and he built a zone which currently holds 200,000 entries in PowerDNS with UN/LOCODE. Florian and his colleagues use these codes to name anycast instances of k-root as IATA codes are too coarse for their purposes.
  $ dig +short nl-ams.locode.sha256.net LOC
  52 24 0.000 N 4 49 0.000 E 0.00m 1m 10000m 10m

  $ dig +short nl-ams.locode.sha256.net TXT
  "Amsterdam, NL"
  

All these zones we speak of here are DNSSEC signed.

DNS and ansible :: 04 Oct 2020 :: e-mail

When we originally contributed the {{ ansible_managed }} feature to the Ansible project it was quite rudimentary: the intention was to have a token which would be templated out to ensure people looking at the remote file would keep their paws off it, warning against manual modifications to the file which would be overwritten at the next playbook run. We added more features, and I think we finalized the variable almost exactly six years ago.

ansible_managed        | {{ ansible_managed }}
template_host          | {{ template_host }}
template_uid           | {{ template_uid }}
template_path          | {{ template_path }}

The output from this template was something like

ansible_managed        | Ansible managed: /tmp/a/input.j2 modified on 2020-09-29 14:12:44 by jpm on rabbit.ww.mens.de
template_host          | rabbit.ww.mens.de
template_uid           | jpm
template_path          | /tmp/a/input.j2

The {{ ansible_managed }} variable was (and still is) configurable via the ansible.cfg mechanisms, and I recall spending quite some time on the time stamp feature, i.e. the modification timestamp of the source template on the controller. It was meant to change on the target only when the source template was modified. All was fine and dandy. I thought.

Years later somebody noticed that there are cases, particularly when using git, when the timestamp of the source changes involuntarily. This, in my opinion unfortunately, caused the project to change the default handling of {{ ansible_managed }} to henceforth output the string Ansible managed only – rather boring. :-)

As mentioned above, we originally made the string configurable, so it’s easy enough to adapt the value to almost anything users want. Here’s the top of my .ansible.cfg showing how the source template file’s modification timestamp is formatted with strftime(3) tokens:

[defaults]
nocows = 1
ansible_managed = Ansible managed: {file} modified on %Y-%m-%d %H:%M:%S by {uid} on {host}

The other tokens are {file} which contains the full path to the source template, {host} which is the controller’s hostname, and {uid}, the owner name of the source template.

There might well be problems with this format on some VCS or remote file system, and quite a few issues have been reported, so configure it as you wish. A different language? Why not?

ansible_managed = Cambiado por {uid}

Meanwhile certain deprecations reportedly having been suggested by developers include

  • {host} should expand to the inventory hostname rather than the controller hostname
  • {file} should expand to the destination file rather than the template file.

both of which quite definitely are not the intended behaviour, on the contrary: if I’m looking at the templated file on the remote node, I know on which host I am, and I know the name of the file! I want to know where the file came from, which is why we designed it thusly.

I still believe {{ ansible_managed }} is a good idea, and as it’s configurable, we can all use it as we wish, and I for one, hope the variable won’t be removed.

Happy templating!

Updates

Somebody asked on Mastodon how they could add the last git commit to this variable. It’s not possible to append to {{ ansible_managed }} without modifying the code, but a lookup plugin in the template does the trick:

# {{ ansible_managed }}
# commit: {{ lookup('pipe', 'git rev-parse --short HEAD') }}
ansible :: 29 Sep 2020 :: e-mail

When I began teaching Unix around anno 1986, the Unix ls(1) utility printed a list of files to stdout, one per line. It was trivial to explain the concept of not requiring a program which tells you how many files you have in a directory:

you have a program which outputs one file name per line, and you have another program which counts lines of input; if you combine the output of the first with the input of the second via a pipe you get the information you want

This simple idea became a bit more cumbersome to explain when ls changed its default behaviour to output file names in multi-column format: students understandably didn’t immediately realize that ls was playing tricks with isatty(3) when they invoked the program in a shell. Of course not, how should they?

Sure, we then showed the effect with ls -1 at a terminal, and demonstrated with ls | cat, etc., but it just wasn’t the same. (At the time I changed from using ls to who(1) – easy to do because these were multi-user systems, and the examples made sense.)

Why do I mention this?

Today in some versions of Linux ls puts single quotes around file names which contain white space likely in order to have those paths easier to copy and paste, but it does so only if !isatty().

I’m not a great fan of changing a program’s well-known behaviour, specially in a case such as with GNU ls which already uses an environment variable for coloring output; it would likely have been easy to augment that for the file name quoting.

It used to be simpler to teach. (But I don’t really teach Unix beginners any more.) :-)

Updates

  • I learn from Quoting File names, that this behavior was introduced in 2016 already; I’d not noticed, probably because I typically don’t embed spaces in filenames.
  • Quite a few interesting comments on Hacker news.
  • Somebody also submitted this to Lobste.rs.

Unix :: 27 Sep 2020 :: e-mail

I can no longer sync iOS’ Contacts with my macOS Catalina’s Finder (the iOS sync portion of iTunes is now built into the Finder in macOS Catalina); the OS insists I’ve iCloud configured for Contacts which I do not. I’ve gone through all the steps Apple recommends, done the upgrades and the reboots, but there’s nothing doing. All the swearing and threatening of moving to a different operating system aren’t really helping. This started sometime after I wrote about setting up macOS to “dial” a number using a shell script, but it cannot be related.

I was spilling my sorrows on Christoph who simply said he avoids all those issues by using CardDAV. I slapped my forehead: I’ve been using CalDAV for years, for synchronizing two calendars across devices: my own calendar across two Macs, an iPad, and an iPhone, and the family calendar across the family’s devices. How could I have forgotten about CardDAV?

Baïkal has served us well for many years (the modification timestamp on the configuration reads August 2013). Before looking at using it for CardDAV I thought I’d put it on my little FreeBSD system in a BastilleBSD jail. The calendar migration was easy enough but produced a litre or two of adrenalin when I watched one calendar after the other disappear from iCal. Anyway, I got the data imported and our small family was back in sync.

I then created a Baïkal address book for myself, set up macOS and an iPhone to use that, and created a pseudo person’s entry on the Mac and another on iOS, and experimented a bit with how long it takes either side to sync, etc. As soon as I was satisfied I copy/pasted all contacts from macOS’ internal address book to its CardDAV store. When my contacts showed up on the iPhone, I knew I was half in business. After an additional backup I cleared out all the local contacts on the Mac.

Android

I recently had to purchase a new old phone for testing OwnTracks on Android and settled for a used Samsung Galaxy S8, a device which vastly exceeded my expectations. A really nice piece of kit, if only it weren’t for Android … #halfkidding

I went the whole hog and decided to see how well I could work with the S8 and purchased a copy of DAVx5 which is a really good and well thought-out bit of software. There’s an Open Source but I decided to pay for the software – developers must live off something other than applause and fresh air.

screenshot of contact on Galaxy S8

I can create, edit, and delete contacts on the S8 and DAVx5 will synchronize them back via CardDAV to my server. Just the way it ought to be.

CLI FreeBSD / Linux

I mostly use Mutt as an email client which means I want to be able to integrate whichever solution I come up with with these tools. A quick search led me to install vdirsyncer which enables me to synchronize my CardDAV server with a local directory of vCard files on the file system. If I change any of these files the utility can then synchronize the changes back to the server and, using a post_hook script I write, can version control the vCard files it receives on the file system. (Source code: vdirsyncer.) The program’s configuration file reminds me of OfflineIMAP’s and its operating principles are similar: I configure a local and a remote store which are kept in sync.

$ export REQUESTS_CA_BUNDLE=mensCA.crt
$ vdirsyncer discover
$ vdirsyncer sync

Now that I have a set of vCard files in a directory, I can use khard, an address book for the Unix command line, which reads, creates, and modifies these files, and which can also be integrated with other programs, suitable for my use with Mutt.

$ khard new ..
$ khard edit ..

$ vdirsyncer -v WARNING sync
$ khard list jolie
Address book: jpm-ab
Index    Name           Phone                          Email
1        Jolie, Jane    work, pref: +49 555 6302547    work, pref: jane.jolie@example.org

$ khard show jolie
Name: Jane Jolie
Full name: Jane Jolie
Address book: jpm-ab
General:
    Birthday: 1969-07-10
    Nickname: jj
Phone
    work, pref: +49 555 6302547
E-Mail
    work, pref: jane.jolie@example.org
Private:
    OwnTracks: owntracks/jjolie/s8
Miscellaneous
    UID: 9245cfe6-256f-40d3-aabf-b3bab4601273
    Webpage:
        - _$!<HomePage>!$_: jj.example.org
    Note: Actress

Khard is just an example; any program able to read or manipulate vCard files would be suitable.

mcds

mcds, written by Timothy Brown, is a Mutt CardDAV search program – a C program which uses a combination of curl and libxml2 to query a CardDAV server to obtain mail addresses from the vCards. The program uses a simple configuration file in which I can specify the URL to my server, a username and whether I want to use a ~/.netrc containing the clear-text password for the CardDAV endpoint. Alternatively I can configure it to use a GPG-encryped password file. And oh, the glory: mcds has a man page!

$ cat ~/.mcdsrc
url = https://example.org/dav.php/addressbooks/jpm/default/
verify = yes
username = jpm
password_file = /Users/jpm/.mcds-pw.gpg

$ mcds jane

jane.jolie@example.org  Jane Jolie

mcds and khard are quite different: the former queries an online CardDAV server for email addresses to present to Mutt, whereas the latter is more of an all-purpose address book (of sorts) for the command line, but it needs an additional utility (vdirsync) to get the data from and to my CardDAV server.

As I’m away from home and thus from my CardDAV server a lot I’ve settled for khard et.al., but who knows: I may have to rewrite the “away from home” portion of this sentence, and I’d then prefer mcds.

Thunderbird

Thunderbird isn’t a client I use, but if I moved to FreeBSD or one of its desktop siblings, I likely would (in addition to Mutt), so I thought I’d do you the favor and see how well it supports CardDAV. It doesn’t really, because it lacks built-in support for CardDAV, but I found TBsync which, with Dav-4-TbSync, allow me to synchronize my CardDAV and CalDAV stores to it. I tested this on FreeBSD, and it seems to work well enough at a first glance.

Thunderbird addressbook entry for Jane

I can edit address book entries on Thunderbird and thanks to a builtin scheduler can automatically (or on-demand) synchronize changes back to the server. TbSync is open source and there’s a short getting started page.

Two people suggested I try CardBookAddon, a new Thunderbird address book based on the CardDAV and vCard standards, which I did, also on FreeBSD. The addon has very many options, and I had trouble finding where to configure my CardDAV server; I resorted to reading manual which often helps ;)

screenshot of Thunderbird's CardBook

Nice about CardBookAddon (source code) is that it integrates remote addressbooks and local ones. I’m thankful for the recommendation: this is likely the addon I would choose to use if I used Thunderbird.

CardDAVMATE

CardDAVMATE is an Open Source CardDAV web client which is nice to use for editing some of the more esoteric (read: not available in the GUIs) vCard attributes, in particular those for spouse and the social profiles.

screenshot of Jane's card in carddavmate

The client runs in the browser as a JavaScript and jQuery application. Other than the CardDAV server proper, no server is required. It made a good impression during my initial testing, but it mucks with the base64-encoding of contact images in such a way as that other clients (Android, khard) no longer recognize the pictures.

Further reading

Ends

The Coronavirus isolation and my speaking more with friends inspired this blog post. Thank you, Christoph, for the heads up.

carddav, addressbook, and cli :: 12 Apr 2020 :: e-mail

Other recent entries