RFC 4034 defines key tag as an identifier with which a DNSKEY RR containing the public key that a validator can use to verify the signature, but over time I have used the terms key tag and key ID interchangeably, without really knowing where they came from.

small poll on the Fediverse

In a small poll on the Fediverse, done mostly jokingly, 35% of those who understood the question answered key ID, and the next morning a question arose:

How come “key tag” didn’t win? Isn’t that the terminology from the RFCs?

It is, but it’s complicated, as section 5.4 shows:

section 5.4 uses both keytag and keyid

I didn’t know the answer. The BIND source code contains roughly 150 mentions of “key tag” and 500 references to “key ID” (with and without a space, and case-insensitive).

I ask Evan, who knows the answer (of course):

the text formatting of DNSKEY records in BIND is based on earlier code that was written for KEY records, five years before 4034. The “key id” string came from there. (Commit 0e93f65e103c, if you’re interested.)

I suspect it was a shortening of “key identifier”, which is used in RFC 2535 in the example SIG records - though, interestingly, that also uses “key tag” in the text.

I don’t know if it was deliberate or an oversight to leave it unchanged for DNSKEY.

Yet another bit of history clarified. Like the semicolon in zone master files.

dnssec :: 18 Nov 2022 :: e-mail

It’s a wee ironic that a day after I tell a group of students that DNSSEC is very stable today compared to what it was like years ago (grandpa telling tales at the fireplace), I get a mail from the .CH registry:

email from registry

My first thought was “they must be mistaken; I have only one dummy zone there”, and my second though was “wow, that’s pretty neat that they monitor customer zones!”. Still convinced it must be a case of mixed up identity, I click on the link which takes me to

dns health report

OK, that’s definitely my zone; it’s the one I set up for testing DNSSEC provisioning automation with CDS/CDNSKEY.

I query the NS RRset for the domain and get a … SERVFAIL. What?!

Next up: DNSviz, of course, and there’s no doubt: a cryptographic signature on the RRSIG of the NS RRset does not compute:

crypto kaputt on NS RRSIG

I’m getting a bit nervous because the third day of training commences in an hour, but let me see what I see. Two of the four NS are under my control, so I force a transfer from the primary and notify all secondaries. The SOA serials match across the board. I contact the operator of the other two secondaries and request him to force transfer; no change – the kaputtness remains.

There’s no way for me to force re-signing of the zone, so I simply don’t know how to “fix” that one signature. I copy the zone file aside on primary and secondary, and then un-sign the primary zone, bump the serial, reload. This will now of course completely break validation as there still is a DS in the parent zone, but kaputt is kaputt and I can’t kaputt it further.

I bump the SOA serial again and sign the zone and see that the new zone is now on all servers.

Green. Relief.

dnsviz reports green

post mortem

My training ends at 21:00 UTC, and I can’t switch off – I need to know what happened.

Using ldns-verify-zone on the kaputt zone, I see it too reports errors:

% ldns-verify-zone -V5 tcp53.ch.kaputt > /dev/null
Error: Bogus DNSSEC signature for tcp53.ch.	NS
tcp53.ch.	3600	IN	NS	ns1.dnspartner.de.
tcp53.ch.	3600	IN	NS	ns2.dnspartner.de.
tcp53.ch.	3600	IN	NS	lumpy.jpmens.net.
tcp53.ch.	3600	IN	NS	woozle.jpmens.net.
tcp53.ch.	3600	IN	RRSIG	NS 13 2 3600 20221120102241 20221021101736 44444 tcp53.ch. TPfWAl4GffhEcyX50bZ4z43dtsrjL3dj/i+sSAMnJPXTuYmMCtQvLM8Hr/TbracpOPjymPrgvSQ+8wfBLeZgxw==

There were errors in the zone

I then load the zone into a fresh server, and see what delv has to say:

% delv +vtrace +root=tcp53.ch +multiline +trust +rrcomments +crypto +rtrace -d 99 -a tcp53.keys  @::1 tcp53.ch NS
;; validating tcp53.ch/NS: verify rdataset (keyid=44444): RRSIG failed to verify
;; validating tcp53.ch/NS: failed to verify rdataset
;; validating tcp53.ch/NS: verify failure: success
;; validating tcp53.ch/NS: no valid signature found

OK, I know of the failure, I was hoping for a few more details.

It occurred to me to use Perl Net::DNS to see if I could obtain more details. I asked for a bit of help, and Oli Schacher came to the rescue. First I verify that the current zone is OK so I use the current NS RRset and its RRSIG:

#!/usr/bin/env perl

use strict;
use Net::DNS;
use Net::DNS::SEC;

my $dnskey = "tcp53.ch. 3600 IN DNSKEY  256 3 13 egTvRrsMdaMjapWI4pC2M5dq6s0W6gpsLT4LwiwXvYs66CqPu+N+JgbO kLVIAwm8PGnPDEIDcAcHViYSvFbHpg==";

# current one: verifies
my $rrsig = "tcp53.ch. 3600 IN RRSIG NS 13 2 3600 20221122031533 20221109133528 44444 tcp53.ch. xxW7tx5fIMUiIOIYrjfCq4h/T28rLlR6NSa0NOZC5NFalz/ShKPkpL3K KgsBKjTD0lleKUd5cqGCtyM4vIFm1Q==";

# this one is kaputt
# $rrsig = "tcp53.ch. 3600 IN RRSIG NS 13 2 3600 20221120102241 20221021101736 44444 tcp53.ch. TPfWAl4GffhEcyX50bZ4z43dtsrjL3dj/i+sSAMnJPXTuYmMCtQvLM8Hr/TbracpOPjymPrgvSQ+8wfBLeZgxw==";

my @data = ();
push(@data, Net::DNS::RR->new("tcp53.ch.  3600 IN NS woozle.jpmens.net."));
push(@data, Net::DNS::RR->new("tcp53.ch.  3600 IN NS ns2.dnspartner.de."));
push(@data, Net::DNS::RR->new("tcp53.ch.  3600 IN NS lumpy.jpmens.net."));
push(@data, Net::DNS::RR->new("tcp53.ch.  3600 IN NS ns1.dnspartner.de."));

my $dnskeyrr = Net::DNS::RR->new($dnskey);

my $nssig = Net::DNS::RR->new($rrsig);

my $v = $nssig->verify( [ @data ], [ $dnskeyrr ]);
print "verifies\n" if $v or die $nssig->vrfyerrstr;
% ./sig.pl

I then run the program with the kaputt RRSIG, and voila, we’re on the right track:

% ./sig.pl
key 44444: signature verification failed at ./sig.pl line 28.

I say “on the right track”, but I’m not really – I’ve simply verified what the registry’s email informed me of that morning.

But why is this signature broken?

Oli checks the logs from their scanner tool that actually caused this email to see if they have more details from the Extended DNS Error, but it also just reports “(DNSSEC Bogus)”.

It’s now quite late, and Oli thinks the kaputt RRSIG is probably due to bit flip, cosmic rays, a bug in the signer, or the Pentium FDIV, and I am starting to think he’s right about that…

Time to sleep.

I rise, get coffee, and look through the repository which gets a copy of all zone activity upon notify.

git diff of the zone

The (now broken) signature was introduced on 2022-10-21. Then, on 2022-10-31T11:21:05, I replaced one of the nameservers, retiring “kanga” and introducing “woozle”, and due to cosmic whatever, the signature was not updated. (It must be a bug, but how on earth do I report that?!)

Can I prove that’s the reason? Yes, I can. In our Perl program, if I s/woozle/kanga/ in the NS RRset the (reportedly broken) RRSIG validates!

I deduce it’s a problem in the signer, but have no idea why it occurred. However, my biggest question is: why did this take so long to be noticed?

While getting the data for this blogticle, I note that DNSviz noticed the failure on 2022-10-31 15:13:10Z which was the time at which the zone went insecure because of the NS RRset change. (But that’s a story for another day.)

insecure and broken

I was lucky that this is currently a toy zone.

And kudos to our friends at SWITCH for the excellent service; thank you!


I went for groceries and a spot of lunch. This topic didn’t leave me alone, and I didn’t really believe in signer bugs or Oli’s cosmic rays. Upon returning to my desk I checked the dynamic DNS update logs:

% grep tcp53\.ch update.log
22-Sep-2021 06:24:38.475 ::1#59860/key local-ddns: updating zone 'tcp53.ch/IN': adding an RR at 'tcp53.ch' RP . jpm.people.dnslab.org.
25-Sep-2021 18:44:43.001 ::1#54888/key local-ddns: updating zone 'tcp53.ch/IN': adding an RR at 'tcp53.ch' CDS 0 0 0 00
31-Oct-2022 17:02:14.689 ::1#62762/key <redacted> updating zone 'tcp53.ch/IN': adding an RR at '<redacted>.tcp53.ch' NS <redacted>.
31-Oct-2022 17:07:48.056 2a03:b0c0:3:d0::1453:6001#45137/key <redacted> updating zone 'tcp53.ch/IN': adding an RR at '<redacted>.tcp53.ch' DS 1722 13 2 AD....

I bet you see it coming…

I serve a few zones for myself, some of my projects, and a few other people, and most of my zones are signed; other people’s are signed only if they ask me to. (There’s no specific reason for me not to sign them by default.)

Anywow, the plain (i.e. unsigned) zone files live in a separate directory, and (you do see it coming, dontcha?) upon changing the NS RRset, and being a lazy bum, I “automated” the task:

% for z in *; do sed -i -e s,kanga,woozle, $z; done

Do you want to guess which other zone file was in this directory? Yes.

This should have become apparent when the simple change of a single NS RR in our Perl program suddenly validated.

So, the very good news is that I can now finally put this to rest, and the even better news is that Oli was wrong with his FDIV bug and cosmic whatnots. On the flip side, this is embarrassing, but not a fraction as embarrassing as the cake fiasco as this didn’t impact anybody other than me and my pride.

Guillaume-J puts it very nicely:

it’s not a DNSSEC error per se: let’s say that signing your zone helped noticing the failure in your “manual” zone edition process :-)

dnssec :: 10 Nov 2022 :: e-mail

There are two TV series adaptations of Johannes Mario Simmel’s “Es muss nicht immer Kaviar sein”: the first with O.W. Fischer in 1961, and the second with Siegfried Rauch in 1977. I think I saw the second, but I digress: the name of the book occurred to me when thinking about this blogticle.

NLnet Labs make DNS software (and more), and amongst their best-known utilities are probably the recursive Unbound server and the authoritative NSD server, one which was originally written to drive a root DNS server.

I’m occasionally asked whether NSD can be used to serve a DNSSEC-signed zone (the answer is ‘yes’), but NSD isn’t a signer: it requires other utilities to actually sign zones which it can then serve. I’ve occasionally mentioned ldns, specifically when I wrote about using a SmartCard-HSM for DNSSEC and again when I wrote about DNSSEC signing with an offline KSK.

But let’s look at the task of signing one or more zones and serving them with NLnet Labs programs.


We start with an unsigned zone file, here for the domain example.aa (aa is a user-assigned code element.)

$TTL 3600
@	SOA	nsd  jp  1 3H 1H 1W 1H
	NS      nsd
nsd	A
	AAAA	2001:0db8:0:0:0202:b3ff:fe1a:8329

I’d like to sign this zone with a single Combined Signing Key (CSK), also called a Single Signing Key (SSK), and I chose algorithm ECDSA Curve P-256 with SHA-256 (number 13) for the key, and I specify the key will have flags 257 on it (KSK), though that is not strictly necessary:

$ ldns-keygen -a13 -k example.aa

$ ls -l K*
-rw-r--r--  1 jpm  staff   94 Nov  6 13:16 Kexample.aa.+013+22967.ds
-rw-r--r--  1 jpm  staff  153 Nov  6 13:16 Kexample.aa.+013+22967.key
-rw-------  1 jpm  staff  114 Nov  6 13:16 Kexample.aa.+013+22967.private

The key filenames begin with a capital K (for “key”), are followed by the zone name (“example.aa”, the key algorithm (13), the key ID or key tag, and the file extension:

  • the file containing the Delegation Signer record(s)
  • the public key (later queryable via the DNSKEY resource record)
  • the private key

I keep the private key safely (ldns has removed group and other permissions, and I tend to chmod 400 the file to ensure I don’t overwrite it accidentally).

The *.ds file contains the Delegation Signer (DS) record which must be added to the parent zone. I can reproduce this file at will with a different ldns utility, and with the -n option I decide whether I want the DS printed to stdout or not:

$ ldns-key2ds -2 Kexample.aa.+013+22967.key
$ cat Kexample.aa.+013+22967.ds
example.aa.	3600	IN	DS	22967 13 2 cfb29a4218cf608d93c08c0d14c3e315b9ab85797bf20269b820ed3a2c1d8f47

$ ldns-key2ds -n -2 Kexample.aa.+013+22967.key
example.aa.	3600	IN	DS	22967 13 2 cfb29a4218cf608d93c08c0d14c3e315b9ab85797bf20269b820ed3a2c1d8f47

Having generated the required key(s), I can now sign the zone. I specify -u to have the SOA serial number set to the Unix epoch time, the -o origin (i.e. zone name) of the zone I’m signing, and the key filename of the key ldns should use to sign the zone (without its file extension).

$ ldns-signzone -u \
                -o example.aa \
		example.aa \

$ ls -l *.signed
-rw-r--r--  1 jpm  staff  1762 Nov  6 13:22 example.aa.signed

ldns-signzone uses NSEC by default. Use -n to have it create NSEC3 records compliant with RFC 9276 — SHA-1, no extra iteration, empty salt and opt-out not set.

The ldns-read-zone utility is very practical. I can use it to strip DNSSEC data from a zone file – something I do when I want to concentrate on the actual zone data. Here I’ve used it to print out just one record type.

$ ldns-read-zone -E SOA example.aa.signed
example.aa.	3600	IN	SOA	nsd.example.aa. jp.example.aa. 1667737361 10800 3600 604800 3600

Finally, to ensure the signed zone is valid, I can use another utility to verify that, though I prefer using something written by a different developers to ensure they agree that the zone is valid.

$ ldns-verify-zone -V4 example.aa.signed
Zone is verified and complete

$ validns -p all example.aa.signed
example.aa.signed:3: there should be at least two NS records per name
example.aa.signed:5: No KSK found

validns is a bit of an older utility but still quite useful. Here it complains that my zone has only one NS record (correct, it does), and that no KSK was found. This latter has to do with validns not realizing that a zone can be signed with a single key. I ignore this warning because I know the zone is otherwise valid.


I configure NSD to load my zone from the *.signed file and serve it:

    name: "example.aa"
    zonefile: "example.aa.signed"
    provide-xfr: NOKEY

After reloading the server I query it and see the signatures (I use +nocrypto to omit the dreary base64-encoded stuff):

$ dig @::1 example.aa SOA +dnssec +multi +nocrypto +norec
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65367
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

; EDNS: version: 0, flags: do; udp: 1232
;example.aa.		IN SOA

example.aa.		3600 IN	SOA nsd.example.aa. jp.example.aa. (
				1667737361 ; serial
				10800      ; refresh (3 hours)
				3600       ; retry (1 hour)
				604800     ; expire (1 week)
				3600       ; minimum (1 hour)
example.aa.		3600 IN	RRSIG SOA 13 2 3600 (
				20221204122241 20221106122241 22967 example.aa.
				[omitted] )

;; Query time: 0 msec
;; SERVER: ::1#53(::1)
;; WHEN: Sun Nov 06 12:37:16 UTC 2022
;; MSG SIZE  rcvd: 188

Signatures (RRSIG) have a default validity of 30 days, but I can easily change that. I’ve created eitime which prints a time stamp in UTC YYYYMMDDHHMMSS for RRSIG expiration/inception times and makes my life easier:

$ ldns-signzone -e $(eitime -z 180) ...

Don’t forget to re-sign and reload the zone in a timely manner before its signatures expire. Cron is your friend.


There’s one thing missing in order to complete the chain of trust to my zone: submit the DS record to the parent, and all I can do here, unfortunately, is to say “that’s your problem”. The method by which you do this varies widely.

Regarding caviar: signing with ldns and serving with NSD might well be very appealing to you, and I know people for whom this is true.

further reading

nsd, ldns, and dnssec :: 06 Nov 2022 :: e-mail

Ever since thinking “that’s quite nifty”, I have the devil of a time spelling ntfy without transposing the ‘t’ and the ‘f’, but it’s called notify so the ‘t’ comes before the ‘f’. :-)

With ntfy I can send push notifications to a phone or desktop via a simple HTTP-based pub/sub service. Philipp C. Heckel created ntfy and open-sourced its components.

ntfy is the backend server (written in Golang), Android and iOS apps, and a desktop UI which runs in a Web browser. There’s also a command-line client – the same binary as the server.

Messages are published via either HTTP PUT or POST, and something as simple as this suffices to transmit a message. Note that there’s no access control on the public server so use your own server with access control or chose a topic which is hard to guess.

$ curl -d "Hola mundo" ntfy.sh/my-topic

Subscribing via the API is equally easy:

$ curl -s ntfy.sh/admin-alerts/json
{"id":"7osug0iTf31n","time":1667123207,"event":"message","topic":"admin-alerts","message":"deployment on alice is complete. 🐄","priority":1,"tags":["heavy_check_mark"]}

All manner of features can be added to a message: priorities, tags, title, attachments, and messages can even contain an action to send a HTTP request from the client when an action button is tapped.

The CLI can also publish and subscribe, and a nifty feature is its ability to launch a command for each message received:

$ ntfy publish -p high --tags warning admin-alerts hola mundo
$ ntfy subscribe admin-alerts /usr/bin/myprog

The program gets environment variables passed to it:

m=hola mundo
message=hola mundo
NTFY_MESSAGE=hola mundo
NTFY_RAW={"id":"f9T2bupNLjjR","time":1667153184,"event":"message","topic":"admin-alerts","message":"hola mundo","priority":4,"tags":["warning"]}
raw={"id":"f9T2bupNLjjR","time":1667153184,"event":"message","topic":"admin-alerts","message":"hola mundo","priority":4,"tags":["warning"]}

(I later found the documentation on the client.)

ntfy on iOS


I was thinking about Ansible action plugins the other day and decided to implement one which notifies via ntfy. Why an action plugin and not a module? Because I’d expect notification to be emitted from the Ansible controller (also possible with a module delegated to it) so that’s how I did it. (Let me know if you prefer a module; we could do both.)

- ntfy:
     msg: "deployment on {{ inventory_hostname }} is complete. 🐄"

ansible-ntfy notifies ntfy.sh by default with a topic configured in a play var. Topic, URL, and additional publishing features can be configured in a dict passed as attrs:

- ntfy:
     topic: "admin-alerts"
     url: "https://nfty.sh"
     msg: "that's a wrap"
        tags: [ rotating_light, heavy_check_mark ]
        priority: 4
           - action: view
             label: "Open Mastodon"
             url: "https://mastodon.social/@jpmens"

Aside from this Ansible module, there are lots of existing integrations.

ntfy is nifty. :-)

push, ansible, and notification :: 30 Oct 2022 :: e-mail

Here I am again with Ansible’s local facts, but this time on Windows nodes, where the local facts behave differently.

Local fact files on Unix/Linux must be named *.fact irrespective of whether they contain INI or JSON or whether they are executable programs which emit JSON, and they are searched for in a default fact_path which is typically /etc/ansible/facts.d (or /usr/local/etc/ansible/facts.d on BSD systems).

There’s no default fact directory for Windows nodes, but we specify one with the fact_path parameter to the setup module or in an ansible.cfg file:

nocows = 1
fact_path = c:/users/r1/facts

That directory must exist and it MUST contain PowerShell (*.ps1) or JSON (*.json) files. The PowerShell scripts output objects which Ansible formats as JSON:

    username = 'janej'
    lat = 48.856826
    lon = 2.292713

When Ansible’s setup module runs, we obtain the following fact from the Windows node:

"ansible_os_family": "Windows",
"ansible_osm": {
  "lat": 48.856826,
  "lon": 2.292713,
  "username": "janej"

Note that Ansible places these objects alongside other Ansible facts, whereas on Unix, if I have a JSON file /etc/ansible/facts.d/osm.fact, Ansible returns osm nested under ansible_local:

"ansible_os_family": "Darwin",
"ansible_local": {
    "osm": {
        "lat": 48.856826,
        "lon": 2.292713,
        "username": "janej"


On Unix/Linux *.fact files may be executable and return JSON, so I install my program as /etc/ansible/facts.d/hungry.fact

$ /etc/ansible/facts.d/hungry.fact
{"id":4,"meal":"dinner","dish":"tikka masala"}

The utility is executed during setup and the result is:

"ansible_local": {
    "hungry": {
        "dish": "tikka masala",
        "id": 4,
        "meal": "dinner"
    "osm": {
        "lat": 48.856826,
        "lon": 2.292713,
        "username": "janej"

On Windows nodes fact files MUST be *.ps1 so I figure out^W^W google how to invoke a program from a PowerShell script and produce this masterpiece in a file called currybeer.ps1:

Invoke-Expression "& `"C:\Users\r1\facts\hungry.exe`"  /run /exit /SilentMode"

Sadly, that doesn’t do what I thought it would: the PowerShell script is returning a string:

"ansible_currybeer": "{\"id\":4,\"meal\":\"dinner\",\"dish\":\"tikka masala\"}"

A bit more figuring out later, I add | ConvertFrom-Json to the .ps1 script, and Ansible correctly interprets its output:

"ansible_currybeer": {
  "dish": "tikka masala",
  "id": 4,
  "meal": "dinner"
"ansible_osm": {
  "lat": 48.856826,
  "lon": 2.292713,
  "username": "janej"

I can now use these facts like any others:

- hosts: win
  gather_facts: true
  - debug: msg="{{ ansible_currybeer.dish }} for {{ ansible_currybeer.meal }}"

and look forward to this evening:

TASK [debug] *************************
ok: [] => {
    "msg": "tikka masala for dinner"

I don’t know why Ansible nests local facts differently on Windows nodes, and I don’t see an advantage in doing so.

And if you need inspiration, we’re collecting ideas for using local facts

ansible and windows :: 28 Oct 2022 :: e-mail

Other recent entries