When Michael asked what the leading P and PT entries in front of some times in named configuration options are, I recognized them from working with OpenDNSSEC: they are ISO 8601 duration and time period specifications.

I knew some named.conf options supported these, but I could neither find a definitive list nor an exact description of <duration> in the Bv9ARM, but Evan, as he typically can do, documented it all in a single tweet.

Yes, a duration can be specified in either format. If it starts with ‘P’ then it’s parsed as an ISO 8601 period, otherwise as a TTL.

"grep cfg_type_duration lib/isccfg/namedconf.c" will tell you which options this applies to.

Here’s the alphabetically-sorted list of options as from just a moment ago, obtained from the latest commit:

dnskey-ttl
fstrm-set-reopen-interval
interface-interval
lame-ttl
max-cache-ttl
max-ncache-ttl
max-policy-ttl
max-stale-ttl
max-zone-ttl
min-cache-ttl
min-ncache-ttl
min-update-interval
nta-lifetime
nta-recheck
parent-ds-ttl
parent-propagation-delay
parent-registration-delay
publish-safety
purge-keys
retire-safety
servfail-ttl
signatures-refresh
signatures-validity
signatures-validity-dnskey
stale-answer-ttl
stale-refresh-time
zone-propagation-delay

(I hope “just a moment ago” is a valid time specification ..) Update: it isn’t ;-)

bind :: 09 Sep 2021 :: e-mail

Many readers know that I’m involved in our open source OwnTracks offerings – apps which permit you to keep track of your own location, and you might also know that Christoph and I created OwnTracks.de to provide affordable devices and services for tracking vehicles. It’s a very modest business, too much so considering the time and effort we invest, but we enjoy it and are quite involved. Our customers particularly appreciate owning the data their vehicles produce and having a partner willing to go the extra mile to provide specialized service.

A few years ago we were asked by a client whether it would be possible that they are informed when a tracked vehicle stops at a fueling station. They wanted to be able to cross check the fueling of their tour buses with chits delivered by drivers at the end of the tour.

In theory this is pretty easy: we know where a vehicle is located so all we have to do is to check whether there’s a fueling station at that particular location. In essence what we’ll do is pre-compute where the stations are and then, for each position reported by a vehicle, verify it maps to one of the station locations.

where are the fueling stations?

The first order of the day was to obtain data on geographic coordinates of fueling stations. For Germany this is quite easy, for example the 14000+ stations by Tankerkoenig, but what’s with the rest of Europe? We wanted to be sure to provide adequate information for the continent, as most of our clientele travels internationally. I was pointed at the data provided by OpenStreetMap, and long may they live! (And yes, we donate a modest amount to them yearly.)

The first task was to understand the data. GeoFabrik offers smaller extracts, and we started with the manageable 36MB file of Hamburg, but quickly moved to using the full Europe dump.

The OpenStreetMap PBF (Protocolbuffer Binary Format) files are about half the size of the XML files which is reason enough to use them. They contain data composed of nodes, ways, and relations. A single place (e.g. amenity=hospital or amenity=fuel) can be specified as a <node/> or a <way/> or a <relation/>. It took me a bit to understand how this all works, and I decided that in order to find all petrol stations, I would extract all nodes and ways with amenity=fuel (recently adding charging_station to the selection; there are many more features) and somehow store these in a CDB. We thought “constant” database because new fueling stations would likely not appear frequently and considered rebuilding this dataset maybe every six months.

The following examples use a charging station on the Autobahn A7 at Malsfeld, Germany. You can partially follow along by right-clicking on the map and selecting Query features. The node describing this charging station is marked with an orange circle.

OSM node

We obtain these specific nodes by running osmium (the fastest utility I’ve found) to create an OSM XML output. (As reference, osmium on europe-latest.osm.pbf runs 12 minutes whereas osmosis runs 190 minutes.)

$ osmium tags-filter europe-latest.osm.pbf nw/amenity=fuel,charging_station -o europe.merged.osm

The result is an OSM XML file with the nodes we selected. Here’s one, and what we most need are the lat and lon elements of the <node/>:

<node id="3263128134" version="4" timestamp="2016-10-28T19:32:21Z" lat="51.0872282" lon="9.4851045">
  <tag k="car" v="yes"/>
  <tag k="name" v="Tesla Supercharger Malsfeld"/>
  <tag k="access" v="customers"/>
  <tag k="amenity" v="charging_station"/>
  <tag k="bicycle" v="no"/>
  <tag k="capacity" v="8"/>
  <tag k="operator" v="Tesla Motors Inc."/>
  <tag k="opening_hours" v="24/7"/>
  <tag k="socket:tesla_supercharger" v="6"/>
</node>

We then use a custom utility which reads the XML and, for each <node/> therein, creates a JSON object with tags copied from the OSM node data, the calculated geohash, and an address field obtained from data graciously given to us by our friends at OpenCageData.

{
  "nodeid": "3263128134",
  "lat": 51.0872282,
  "lon": 9.4851045,
  "ghash": "u1ndxu4u",
  "addr": "Autohof Malsfeld, Dr.-Reimer-Straße 2, 34323 Elfershausen, Germany",
  "tags": {
    "car": "yes",
    "name": "Tesla Supercharger Malsfeld",
    "access": "customers",
    "amenity": "charging_station",
    "bicycle": "no",
    "capacity": "8",
    "operator": "Tesla Motors Inc.",
    "opening_hours": "24/7",
    "socket:tesla_supercharger": "6"
  }
}

So what we now have is a CDB indexed by an 8-character geohash with the above JSON as the data. The 340,036 records in the database require 108MB of space.

regarding geohashes

What’s with the geohash? A geohash is a method of expressing a location anywhere in the world using a short alphanumeric string, with greater precision using longer strings. As can be seen on the very practical geohashes page, geohashes identify rectangular cells; the 8 characters we use here define an area of roughly 38m x 19m.

So how to we determine that a vehicle is at a fueling station?

On each ignition event received from Traccar we convert the latitude, longitude coordinates of a vehicle’s position to an 8 character geohash and check in the CDB whether the key exists. If it does we’re at a station; if the key doesn’t exist, we’re not.

That sounds easy, and it turns out it was too simplistic. Recall our geohash maps to a small rectangle of roughly 38mx19m, a very small area for a fueling station, and most are much larger than that, so what we do is check all the neighbours of the geohash which we receive from a vehicle position. In doing so we hope to cover an area which encompasses the actual fueling area, ignoring parking, etc. This isn’t fool proof, but it’s close enough for the requirements.

visualizing geohashes

From the above image we see that a vehicle is located at coordinates 51.0873396,9.48528275 which translates to geohash u1ndxu4v. We also see that the area delimited by the geohash (the blueish rectangle) is not the fueling (here: charging) station but that it’s north of it. So in checking all the neighbors of the geohash we stab around and see if the fueling/charging station is at one of those positions, and if so we have a hit. (Alternatively we could have used a shorter geohash string (which maps to a larger rectangle), but we found that was too coarse: we had too many false positives.)

This small custom utility shows what’s happening: we invoke it with our coordinates, and it prints out the corresponding geohash and each of its neighbors. It obtains a hit on the S (southern) neighbor (u1ndxu4u) containing the charging station.

$ geoh 51.0873396,9.48528275
51.087370,9.485149
u1ndxu4v **
u1ndxu4y N
u1ndxu5n NE
u1ndxu5j E
u1ndxu5h SE
u1ndxu4u S  {"nodeid": "3263128134", "lat": 51.0872282, "lon": 9.4851045, "ghash": "u1ndxu4u", "addr": "Autohof Malsfeld, Dr.-Reimer-Stra\u00dfe 2, 34323 Elfershausen, Germany", "tags": {"car": "yes", "name": "Tesla Supercharger Malsfeld", "access": "customers", "amenity": "charging_station", "bicycle": "no", "capacity": "8", "operator": "Tesla Motors Inc.", "opening_hours": "24/7", "socket:tesla_supercharger": "6"}}
u1ndxu4s SW
u1ndxu4t W
u1ndxu4w NW

If a query in the database for the geohash or any of its neighbors finds a record, we consider the vehicle is at a fueling / charging station and report a “hit”.

As demonstrated, we’re actually a bit off mark, but we can a) live with the inaccuracy at the moment, and b) in order to be as precise as possible we use these small geohashes (i.e. long strings).

Here’s an example where that counts: we detected the vehicle at the blue pin, i.e. the actual center of the petrol station is west of the vehicle.

West of direct match

and in the following example we matched the center of the station with the first geohash the software checked.

direct match on station

This final example shows nicely how important it is to “reach around” to see if we’re still at the station; the geohash ending in y8 covers only a fraction of the huge station of the Aire de Wasserbillig in Luxemburg, but the enclosing way would be too large an area.

Wasserbillig

publishing data

Once a hit is obtained, we publish an enriched payload over MQTT, and this is consumed by subscribers which record the fact in a database, or alert in some way. (Who knows, maybe one day we’ll wire up an MQTT version of Jeff Geerling’s Bell Slapper to hear a ding whenever a vehicle stops at a fueling station.)

{
  "_type": "fuelstop",
  "_position_id": 5016671,
  "_uuid": "2498c02e-f885-4921-bcbb-c5c3dffeee21",
  "_neighbor": "S",
  "tst": 1624542898,
  "ignition": false,
  "addr": "Autohof Malsfeld, Dr.-Reimer-Str 2, 34323 Elfershausen, Germany",
  "lat": 51.0873396,
  "lon": 9.48528275,
  "osm_url": "https://www.openstreetmap.org/?mlat=51.087340&mlon=9.485283&zoom=18",
  "loc": "51.08733960,9.48528275",
  "ghash": "u1ndxu4v",
  "name": "AA-BB-CC",
  "u": "owntracks/qtripp/123456789244442",
  "station": "Tesla Supercharger Malsfeld",
  "amenity": "charging_station",
  "nodeid": "3263128134"
}

OwnTracks.de customers appreciate that we are small enough to actually enjoy thinking about and implementing this kind of feature, and more importantly we realize that this wouldn’t be possible without the amazing work done by OpenStreetMap and the thousands of contributors to its incredibly rich data set.

owntracks, openstreetmap, and data :: 24 Jun 2021 :: e-mail

When documenting my experiences using a SmartCard-HSM for DNSSEC I linked to a post by Remy van Elst in which he discusses using a CardContact SmartCard-HSM with SSH, and I thought I’d try that, focussing on using EC keys.

SmartCard-HSM in keyboard

The six year-old HSM I have has support for 2048 bit only RSA keys which is enough reason to attempt using EC keys, but as Remy pointed out when he wrote the article in 2016, OpenSSH had no PKCS#11 support for them then.

It turns out my client of choice has OpenSSH_8.1p1 which isn’t recent enough either, so I install portable OpenSSH version 8.6p1.

$ ./configure --prefix=/tmp/ssh \
              --with-ssl-dir=/usr/local/Cellar/openssl\@1.1/1.1.1i/
...
OpenSSH has been configured with the following options:
...
                   PKCS#11 support: yes
                  U2F/FIDO support: yes
...

I then create an EC key pair on the HSM:

$ pkcs11-tool --pin 123456 --keypairgen --key-type EC:prime256v1 --label SSH-key-s00
Using slot 0 with a present token (0x0)
Key pair generated:
Private Key Object; EC
  label:      SSH-key-s00
  ID:         6423210e6bde55c0c4b8ce6d6eafb98b6bdd2a4c
  Usage:      sign, derive
  Access:     none
Public Key Object; EC  EC_POINT 256 bits
  EC_POINT:   044104accb84a58d36204a25f14bf76c5c967de7c48b93831a86dd370d6fd9eb16d88d5f02fdec36ea6006c4ca21cc704997a5886133f8bf1e8acdbd34223456f4f482
  EC_PARAMS:  06082a8648ce3d030107
  label:      SSH-key-s00
  ID:         6423210e6bde55c0c4b8ce6d6eafb98b6bdd2a4c
  Usage:      verify, derive
  Access:     none

and extract the SSH public key for this object; nice that the key label is added to the output:

$ pkcs15-tool --read-ssh-key 6423210e6bde55c0c4b8ce6d6eafb98b6bdd2a4c
Using reader with a card: Identive SCT3522CC token
ecdsa-sha2-nistp256 AAAAE2VjZHNhLX[..]hM/i/HorNvTQiNFb09II= SSH-key-s00

The public SSH key I place into an authorized_keys file on the target system. (I do this manually now but will typically use ssh-copy-id to do so.)

$ cat .ssh/authorized_keys
ecdsa-sha2-nistp256 AAAAE2VjZHNhLX[..]hM/i/HorNvTQiNFb09II= SSH-key-s00

Back on my client, I attempt to connect using the newly-built SSH client:

$ /tmp/ssh/bin/ssh -v -o "PKCS11Provider /usr/local/Cellar/opensc/0.21.0/lib/opensc-pkcs11.so"  192.168.33.15
OpenSSH_8.6p1, OpenSSL 1.1.1i  8 Dec 2020
debug1: Connecting to 192.168.33.15 [192.168.33.15] port 22.
debug1: Connection established.
debug1: provider /usr/local/Cellar/opensc/0.21.0/lib/opensc-pkcs11.so: manufacturerID <OpenSC Project> cryptokiVersion 2.20 libraryDescription <OpenSC smartcard framework> libraryVersion 0.21
debug1: provider /usr/local/Cellar/opensc/0.21.0/lib/opensc-pkcs11.so slot 0: label <SmartCard-HSM (User... (UserPIN> manufacturerID <www.CardContact.de> model <PKCS#15 emulate> serial <DECC0500007> flags 0x40d
..
debug1: Local version string SSH-2.0-OpenSSH_8.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9p1 Debian-10+deb10u2
..
debug1: Will attempt key: rs001 RSA SHA256:B0KDkBLyWjkyfGS3X0AhBdHgFboXSWTpzbG5AP2qcjI token
..
debug1: Will attempt key: SSH-key-s00 ECDSA SHA256:/uGTFUy11/xvMbsp0n2GSw7xqrK0+HV+sCALVNjBRkw token
..
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: SSH-key-s00 ECDSA SHA256:/uGTFUy11/xvMbsp0n2GSw7xqrK0+HV+sCALVNjBRkw token
debug1: Server accepts key: SSH-key-s00 ECDSA SHA256:/uGTFUy11/xvMbsp0n2GSw7xqrK0+HV+sCALVNjBRkw token
Enter PIN for 'SmartCard-HSM (User... (UserPIN':
debug1: pkcs11_check_obj_bool_attrib: provider 0x7fc8a48047c0 slot 0 object 140499730079136: attrib 514 = 0
..
debug1: Authentication succeeded (publickey).
Authenticated to 192.168.33.15 ([192.168.33.15]:22).
..
You have new mail.

That looks good to me, so I launch an SSH agent so as to not have to enter the HSM PIN whenever I connect a machine.

$ eval $(/tmp/ssh/bin/ssh-agent -P "/usr/local/Cellar/opensc/0.21.0/lib/*")
Agent pid 72428

$ /tmp/ssh/bin/ssh-add -s /usr/local/lib/opensc-pkcs11.so
Enter passphrase for PKCS#11:
Card added: /usr/local/lib/opensc-pkcs11.so

Note how I use -P to whitelist the paths for PKCS#11 shared libraries that may be added using the -s option to ssh-add. This option expects the real directory and not a symlink to it.

$ /tmp/ssh/bin/ssh-add -l
2048 SHA256:B0KDkBLyWjkyfGS3X0AhBdHgFboXSWTpzbG5AP2qcjI rs001 (RSA)
256 SHA256:/uGTFUy11/xvMbsp0n2GSw7xqrK0+HV+sCALVNjBRkw SSH-key-s00 (ECDSA)

And the rest, as they say, should be obvious: I can SSH into systems without having to specify the SmartCard-HSM PIN as the key is now managed by the agent.

$ /tmp/ssh/bin/ssh -v 192.168.33.15
..
debug1: Server accepts key: SSH-key-s00 ECDSA SHA256:/uGTFUy11/xvMbsp0n2GSw7xqrK0+HV+sCALVNjBRkw agent
..

If I unplug the HSM from USB the SSH agent loses connection and will not be able to provide the key to SSH clients. Plugging the token back in doesn’t do anything here; I have to kill and restart the agent and add the keys back into it.

I’m grateful to Remy for having inspired me to do this.

ssh, security, and hsm :: 16 Jun 2021 :: e-mail

I’ve yet to come across a diagram showing a DNSSEC chain of trust which I actually like. It took me a long time to design one of my own for my post on CDS/DNSKEY records, but I feel that doesn’t clearly convey the chain I mean.

This is my latest creation:

a diagram showing chain of trust

The text on the blue connectors is a bit small in this rendition (it says “refers to”) – I might have to tweak that. The blue is designed to suggest a link (as on a Web page), which I can use to explain kaputt.

I’d like to thank Horia, Ingo, and Florian for their feedback and suggestions.

dnssec :: 09 Jun 2021 :: e-mail

I’ve been thinking a bit about affordable hardware security modules (HSM); affordable is relative so I’ll start with “inexpensive”, and I recall I had mixed experiences with a very small HSM I purchased six years ago. I am curious to see whether I’m now able to wire up the CardContact SmartCard-HSM to some Open Source DNS servers and whether the experience has improved. TL;DR: I can and it has.

SmartCard-HSM in a Thinkpad

With private (preferably ECDSA) keys generated on the HSM I’d like to be able to accomplish the following tasks:

  1. Manually sign zones with dnssec-signzone and BIND inline signing. I expect BIND’s new dnssec-policy will not work as named would have to create keys on the HSM; the policy currently supports key-directory only
  2. Use Knot DNS to automatically sign zones, at least using “manual” policy, ideally with “automatic” key management in which the server maintains signing keys
  3. Use LDNS’s ldns-signzone utility to manually sign a zone

All this is possible using keys stored on the file system, but having private keys on a hardware security module makes for a very aggravating experience when attempting to steal them. Look at the photo: the private keys are on that USB token; breaking into the server over the network isn’t enough – the device has to be physically abstracted.

Using BIND utilities

There exists an article on PKCS#11 in BIND9 (recommended reading), but I’m grateful that ISC’s Ondřej Surý points me in the preferred direction:

as of this moment, the OpenSSL engine_pkcs11 is a preferred way, so any HSM that’s well supported by OpenSC project should be ok.

Applications which need to use a smart card library do so using an interface of which there are, most unfortunately, several different implementations. One of these is PKCS#11 which exists on most operating systems; PKCS#11 is an API for accessing cryptographic hardware such as HSMs. OpenSC implements this interface as a smart card library on multiple platforms.

Quoting from libp11:

engine_pkcs11 tries to fit the PKCS#11 API within the engine API of OpenSSL. That is, it provides a gateway between PKCS#11 modules and the OpenSSL engine API. One has to register the engine with OpenSSL and one has to provide the path to the PKCS#11 module which should be gatewayed to.

Lots of words. Luckily I understand a few of the two-syllable ones, so I feel I can get started.

Software installation

I begin with a fresh installation of Fedora 34, and since I don’t need to compile BIND myself for native PKCS#11, I choose to install a 9.16.16 package by ISC. Other system packages I install are:

  • opensc for the pkcs11-tool utility
  • ldns-utils for the LDNS utilities
  • knot, available in the current version 3.0.6 in the Fedora repository

I also need the libp11 PKCS#11 wrapper library which will interface between, e.g. BIND and the actual HSM driver, so I clone its repository. Ondřej recommends I use libp11 0.4.12 which was slated for release this year. It’s not yet been released hence I’m building from source. It contains many improvements to engine_pkcs11 sponsored by ISC.

% ./configure --with-enginesdir=/opt/bind9/engines --prefix=/opt
% make install
% ls -l /opt/bind9/engines/*.so
lrwxrwxrwx. 1 root root      9 Jun  3 14:32 /opt/bind9/engines/libpkcs11.so -> pkcs11.so
-rwxr-xr-x. 1 root root 364784 Jun  3 14:32 /opt/bind9/engines/pkcs11.so

Libp11’s documentation says I might need to configure the location of the modules in an OpenSSL config file, but later I re-read to see that in systems with p11-kit-proxy engine_pkcs11 has access to all the configured PKCS#11 modules. I see p11-kit{,-trust,-server} are installed, so this is likely the reason why BIND’s utilities and LDNS can access the SmartCard-HSM without my setting $OPENSSL_CONF. Be that as it may, this is what I add to /opt/bind9/etc/openssl.cnf, but it’t not being used:

[openssl_init]
engines=engine_section

[engine_section]
pkcs11 = pkcs11_section

[pkcs11_section]
engine_id = pkcs11
dynamic_path = /opt/bind9/engines/pkcs11.so
MODULE_PATH = /opt/lib/libsc-hsm-pkcs11.so
init = 0

(I’ve decided I won’t try to force remove p11-*.rpm ;-)

Working with the card

I insert the CardContact SmartCard-HSM and use the opensc-tool to verify the HSM device is accessible:

[10091.366761] usb 2-1.2: new full-speed USB device number 5 using ehci-pci
[10091.451190] usb 2-1.2: New USB device found, idVendor=04e6, idProduct=5817, bcdDevice= 1.00
[10091.451208] usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=5
[10091.451215] usb 2-1.2: Product: SCT3522CC token
[10091.451219] usb 2-1.2: Manufacturer: Identive
[10091.451223] usb 2-1.2: SerialNumber: 21121350600105
% opensc-tool -l
# Detected readers (pcsc)
Nr.  Card  Features  Name
0    Yes             Identive SCT3522CC token [CCID Interface] (21121350600105) 00 00

I had already initialized the card, so I begin by creating a key pair on the card.

% pkcs11-tool --module /usr/lib64/opensc-pkcs11.so \
              --login \
	      --keypairgen \
	      --key-type EC:prime256v1 \
	      --label ta-example.com
Using slot 0 with a present token (0x1)
Logging in to "SmartCard-HSM".
Please enter User PIN:
Key pair generated:
Private Key Object; EC
  label:      ta-example.com
  ID:         04
  Usage:      sign
  Access:     sensitive, always sensitive, never extractable, local
Public Key Object; EC  EC_POINT 256 bits
  EC_POINT:   0441047301f6b89a2468db4aaa57678481eb787aa88c33a255f7ae3c171ec3df4943621d9d4173043a737fa81bd654b1f84cf0e73115387356fa456477e7c6c0a01ad3
  EC_PARAMS:  06082a8648ce3d030107
  label:      ta-example.com
  ID:         04
  Usage:      encrypt, verify
  Access:     local

Then I need to generate the pair of key files which reference the key object stored on our HSM. The private key file will be used for DNSSEC signing of our zone as if it were a conventional key on the file system (i.e. one created with dnssec-keygen). The key material is stored on the HSM (and we cannot extract it), and the actual signing takes place there as we’ll vividly see when we time the signing of a zone with 1000 records in it.

I use the PIN here directly on the command line, but it can also be ferreted away into a file using pin-source. That adds just a level of indirection as the file must of course be readable by whichever utility (e.g. named) will be using its content.

% dnssec-keyfromlabel -E pkcs11 \
                      -a 13 \
		      -l "object=ta-example.com;pin-value=123456" \
		      -f KSK example.com
Kexample.com.+013+34583

% ls -l Kexample*
-rw-r--r--  1 root root  341 Jun  4 12:39 Kexample.com.+013+34583.key
-rw-------  1 root root  236 Jun  4 12:39 Kexample.com.+013+34583.private

% cat Kexample.com.+013+34583.private
Private-key-format: v1.3
Algorithm: 13 (ECDSAP256SHA256)
PrivateKey:
Engine: cGtjczExAA==
Label: cGtjczExOm9iamVjdD10YS1leGFtcGxlLmNvbTtwaW4tdmFsdWU9MTIzNDU2AA==
Created: 20210604103940
Publish: 20210604103940
Activate: 20210604103940

The two base64-encoded values we see in the private key file are “pkcs11” and “pkcs11:object=ta-example.com;pin-value=123456” for Engine and Label respectively. Our signing utilities and named will need to access the HSM and need the PIN to do so, which is why it’s stored in the .private file.

Manual signing with dnssec-signzone

We now have a single ECDSA key pair (algorithm 13) which we’ll use to sign the zone.

% dnssec-signzone -E pkcs11 -S -3 - -H 0 -A -x -z -t example.com
Fetching example.com/ECDSAP256SHA256/34583 (KSK) from key repository.
Verifying the zone using the following algorithms:
- ECDSAP256SHA256
Zone fully signed:
Algorithm: ECDSAP256SHA256: KSKs: 1 active, 0 stand-by, 0 revoked
                            ZSKs: 0 active, 0 present, 0 revoked
example.com.signed
Signatures generated:                     1883
Signatures retained:                         0
Signatures dropped:                          0
Signatures successfully verified:            0
Signatures unsuccessfully verified:          0
Signing time in seconds:               291.264
Signatures per second:                   6.464
Runtime in seconds:                    293.640

Six signatures per second is slow, but this particular HSM costs €60.00 only.

In addition to the existing KSK (on the HSM) I create a ZSK on the file system and have all the records in the zone signed with it except for the DNSKEY RRset which will be signed with the KSK on the HSM. This is bound to be much faster.

% dnssec-keygen -a 13 example.com
Generating key pair.
Kexample.com.+013+28096

% cat Kexample.com.+013+28096.private
Private-key-format: v1.3
Algorithm: 13 (ECDSAP256SHA256)
PrivateKey: MJQosqFwHt2ijSVJC/PoSD6qg9u/oeIgjGBaKrWNwoY=
Created: 20210604105112
Publish: 20210604105112
Activate: 20210604105112

% dnssec-signzone -E pkcs11 -S -3 - -H 0 -A -x -t example.com
Fetching example.com/ECDSAP256SHA256/34583 (KSK) from key repository.
Fetching example.com/ECDSAP256SHA256/28096 (ZSK) from key repository.
Verifying the zone using the following algorithms:
- ECDSAP256SHA256
Zone fully signed:
Algorithm: ECDSAP256SHA256: KSKs: 1 active, 0 stand-by, 0 revoked
                            ZSKs: 1 active, 0 present, 0 revoked
example.com.signed
Signatures generated:                     1883
Signatures retained:                         0
Signatures dropped:                          0
Signatures successfully verified:            0
Signatures unsuccessfully verified:          0
Signing time in seconds:                 0.317
Signatures per second:                5939.931
Runtime in seconds:                      2.381

It is. Much faster. :-) (Note the information regarding fetching keys from the repository: the KSK is on the HSM and the ZSK on the file system.)

I’m a great believer in CSK/SSK (Combined- or Single Signing Key), but I will change my mind if I have to use an inexpensive HSM: using split keys (ZSK on the file system, KSK on the HSM) is definitely and understandably a lot faster.

Manual signing with ldns-signzone

An alternative to using BIND command-line utilities exists in the form of LDNS, a programming library created by NLnetLabs. LDNS provides some interesting and practical command-line tools such as is drill, a dig lookalike specifically designed to work with DNSSEC, and ldns-signzone which, well, signs zones. This program accomplishes what dnssec-signzone does, and I can use the zone file it signs as input to, say, NSD. (I’m looking forward to the new NSD version with CREDNS functionality.) If ldns_signzone is compiled with engine support I can use it with the HSM. (For example on OpenBSD, the program doesn’t support engines.)

ldns-signzone and NSD

With option -K I specify the KSK to use as an algorithm mnemonic or number (e.g. 13) followed by a comma, and the PKCS#11 key identifier, here the literal “label_” concatenated with the label of the key on the HSM as created above. The program signs our zone with the KSK only (as in our first example earlier) and does the job in the same time, the bottleneck being our slow HSM.

$ time ldns-signzone -E pkcs11 \
        -o example.com \
        -f example.com.signed.ldns \
        -K ECDSAP256SHA256,label_ta-example.com \
        example.com
Engine key id: label_ta-example.com, algo 13
Enter PKCS#11 token PIN for SmartCard-HSM (UserPIN):

real    4m57.318s
user    0m0.833s
sys     0m0.515s

To automate signing with ldns-signzone I can specify the PIN and the key to use in a PKCS#11 URI. Like in the second example with dnssec-signzone above, I additionally create an on-filesystem ZSK with which to sign the zone.

$ ldns-keygen -a 13 example.com
Kexample.com.+013+61121

$ printf 123456 > mypin
$ time ldns-signzone -E pkcs11 \
      -o example.com \
      -K 'ECDSAP256SHA256,pkcs11:object=ta-example.com;pin-source=mypin' \
      example.com \
      Kexample.com.+013+61121
Engine key id: pkcs11:object=ta-example.com;pin-source=mypin, algo 13

real    0m2.365s
user    0m0.218s
sys     0m0.029s

Both zone signing utilities accomplish the task in just over two seconds: the ZSK on the file system signs all records in the zone and the KSK in the HSM the DNSKEY RRset.

Enough of the manual signing.

BIND inline signing

With BIND 9.9, ISC introduced a new “inline-signing” option which allows for transparent signing of zones loaded or transferred to the server. named creates a signed version of the zone which answers all queries and transfer requests without altering the original unsigned version. This method can work well for DNS administrators who wish to continue editing their zone files without “seeing” the DNSSEC data in them, and it also works very well for dynamically updatable zones.

BIND inline signing

We now configure BIND to use inline signing and maintain DNSSEC automatically for us. I’ll start off with a zone containing just a few records and a configuration that enables me to update the zone dynamically. (I want to see the green LED on the SmartCard-HSM blink every time I do a dynamic update… :-)

In named.conf I define the zone:

zone "example.com" IN {
    type primary;
    file "example.com";
    inline-signing yes;
    auto-dnssec maintain;

    update-policy {
            grant local-ddns zonesub ANY;
            grant "jpm" zonesub ANY;
    };

    key-directory "keys";
};

I then launch named in the foreground:

% named -E pkcs11 -g
starting BIND 9.16.16 (Stable Release) <id:0c314d8>
...
zone example.com/IN (unsigned): loaded serial 1
all zones loaded
running
zone example.com/IN (signed): loaded serial 1
zone example.com/IN (signed): receive_secure_serial: unchanged
zone example.com/IN (signed): sending notifies (serial 1)
zone example.com/IN (signed): reconfiguring zone keys
Fetching example.com/ECDSAP256SHA256/34583 (KSK) from key repository.
DNSKEY example.com/ECDSAP256SHA256/34583 (KSK) is now published
DNSKEY example.com/ECDSAP256SHA256/34583 (KSK) is now active
zone example.com/IN (signed): next key event: 04-Jun-2021 14:00:06.698
zone example.com/IN (signed): sending notifies (serial 3)

That looks very good. BIND has accessed the HSM and has signed the zone, and I see the DNSKEY RRset.

$ dig @::1 example.com DNSKEY +dnssec +nocrypto

;; ANSWER SECTION:
example.com.            86400   IN      DNSKEY  257 3 13 [key id = 34583]
example.com.            86400   IN      RRSIG   DNSKEY 13 2 86400 20210704110005 20210604100006 34583 example.com. [omitted]

And behold the #blinkenlights when I submit a dynamic update to the zone. (You can’t see the blinking – you’ll have to trust me.)

client .. 127.0.0.1#47646/key jpm: updating zone 'example.com/IN': adding an RR at 'jp.example.com' TXT "2021-06-04T13:05:14"
zone example.com/IN (signed): serial 4 (unsigned 2)

Knot DNS

Knot DNS is a very capable authoritative server which manages keys in a key store; it’s also known to work with several PKCS#11 devices. Knot is capable of performing fully automatic signing with a Key And Signing Policy (KASP) and can maintain signing keys.

knot signing with SmartCard-HSM

I’m aware that Knot uses GnuTLS so I use the latter’s p11tool to obtain the PKCS#11 URI for the HSM, and don’t ask me why it knows, but it’s correctly identified the device; probably some PC/SD magic?

% p11tool --list-tokens
...
Token 2:
        URL: pkcs11:model=PKCS%2315%20emulated;manufacturer=www.CardContact.de;serial=DECC0100509;token=SmartCard-HSM%20%28UserPIN%29
        Label: SmartCard-HSM (UserPIN)
        Type: Hardware token
        Flags: RNG, Requires login
        Manufacturer: www.CardContact.de
        Model: PKCS#15 emulated
...

Configuring that URI into Knot’s keystore configuration caused Knot to report DNSSEC, failed to initialize (PKCS #11 token not available) even when I added pin-value, so I simplified it to pin-value only. Upon startup, the server rightly reports no configured keys, and since it cannot sign the zone without them, rightly refuses to serve the zone at all.

2021-06-06T10:43:40+0200 error: [example.com.] DNSSEC, no keys are available
2021-06-06T10:43:40+0200 error: [example.com.] DNSSEC, failed to load keys (no keys for signing)
2021-06-06T10:43:40+0200 error: [example.com.] zone event 'load' failed (no keys for signing)

So far so good, and I see from a change in the green LED (what would we do without #blinkenlights) that the HSM has been accessed.

The configuration I’m using for this test now looks like this:

server:
    rundir: "/run/knot"
    user: root:root
    listen: [ 127.0.0.1@53, ::1@53 ]
log:
  - target: "/etc/knot/k.log"
    any: info
database:
    storage: "/var/lib/knot"

keystore:
   - id: smartcardHSM
     backend: pkcs11
     config: "pkcs11:pin-value=123456 /usr/lib64/opensc-pkcs11.so"
     # note the space before path to ^^ module

key:
  - id: jpm
    algorithm: hmac-sha256
    secret: L4HAwrawtduzDlkq29jbprfxnOh+0hr8DLPEYNiwZM8=

acl:
  - id: xfr
    address: 127.0.0.1
    action: transfer

  - id: upd
    key: jpm
    action: update
    
template:
  - id: default
    storage: "/var/lib/knot"
    file: "%s.zone"

policy:
  - id: manualHSM
    keystore: smartcardHSM
    single-type-signing: on
    manual: on

zone:
  - domain: example.com
    dnssec-signing: on
    dnssec-policy: manualHSM
    acl: [ xfr, upd ]

I now “import” the key from the HSM into Knot’s keystore. It’s called importing but it’s basically just associating the HSM key with an object in the keystore a bit like how BIND’s utilities do. Knot’s keymgr utility requires a key_id (not a key label which we’ve used so far) with which to import the key. I obtain this using p11tool (it’s the same ID which was reported earlier when I generated the key).

% p11tool \
     --set-pin=123456 \
     --list-all \
     'pkcs11:model=PKCS%2315%20emulated;manufacturer=www.CardContact.de;serial=DECC0100509;token=SmartCard-HSM%20%28UserPIN%29'
...
Object 3:
        URL: pkcs11:model=PKCS%2315%20emulated;manufacturer=www.CardContact.de;serial=DECC0100509;token=SmartCard-HSM%20%28UserPIN%29;id=%04;object=ta-example.com;type=public
        Type: Public key (EC/ECDSA-SECP256R1)
        Label: ta-example.com
        ID: 04
...

Now I attempt to import said key and describe the key to Knot so that it knows how to handle it and what to do with it.

% keymgr example.com import-pkcs11 04 algorithm=ECDSAP256SHA256 ksk=yes zsk=yes
04
OK

% knotc reload
Reloaded

and behold more #blinkenlights as Knot begins to sign the zone. (I’m using a single signing key so I expect this to take approximately as long as the *signzone utilities needed.)

2021-06-06T10:51:22+0200 info: configuration reloaded
2021-06-06T10:51:22+0200 info: [example.com.] zone file parsed, serial 8
2021-06-06T10:51:22+0200 info: [example.com.] DNSSEC, key, tag 34583, algorithm ECDSAP256SHA256, CSK, public, active
2021-06-06T10:51:22+0200 info: [example.com.] DNSSEC, signing started
2021-06-06T10:56:12+0200 info: [example.com.] DNSSEC, successfully signed
2021-06-06T10:56:12+0200 info: [example.com.] loaded, serial none -> 8 -> 9, 328474 bytes
% keymgr example.com ds
example.com. DS 34583 13 2 cf4b311003e58abc776723158f8406c1af4f4f2e1ec6f7092f6cffb93a292589
example.com. DS 34583 13 4 52c057fa00cf0ee35a5e16ed2a0f0317b02479ac5a2382b0e7750d79e47b4954b6068de603a7b93351d8fb65cacf8d49

I verify by dynamically updating the zone that it’s being signed by the CSK. I could also have used kzonesign to sign the zone from the CLI.

Knot creates keys on the HSM

I use keymgr to create a key on the SmartCard-HSM, and verify that it is indeed on the device:

% keymgr example.com generate algorithm=ECDSAP256SHA256 ksk=yes
144821ef5ae749180c5299376c27f8154f16a5d6

% p11tool --set-pin=123456 --list-all 'pkcs11:serial=DECC0100509'
...
Object 5:
        URL: pkcs11:model=PKCS%2315%20emulated;manufacturer=www.CardContact.de;serial=DECC0100509;token=SmartCard-HSM%20%28UserPIN%29;id=%14%48%21%EF%5A%E7%49%18%0C%52%99%37%6C%27%F8%15%4F%16%A5%D6;type=public
        Type: Public key (EC/ECDSA-SECP256R1)
        Label:
        ID: 14:48:21:ef:5a:e7:49:18:0c:52:99:37:6c:27:f8:15:4f:16:a5:d6
...

Sadly I’ve not found an incantation of p11tool options which would permit me to set the key label. I realize Knot doesn’t need it, but I find the label documents what the key is for.

So, can I convince Knot to create and maintain keys on the HSM automatically? I configure a new zone with a new policy (and crazy timers, I know, but I want changes to occur quickly).

policy:
  - id: autoHSM
    keystore: smartcardHSM
    single-type-signing: off
    manual: off
    algorithm: ecdsap256sha256
    zone-max-ttl: 60
    dnskey-ttl: 60
    propagation-delay: 60
    zsk-lifetime: 300

zone:
  - domain: example.net
    dnssec-signing: on
    dnssec-policy: autoHSM
    acl: [ xfr, upd ]

I launch the server afresh and see it begins by generating a KSK and a ZSK and henceforth neatly rolls the ZSK every few minutes.

2021-06-07T08:30:11+0200 info: [example.net.] DNSSEC, ZSK rollover started
2021-06-07T08:30:11+0200 info: [example.net.] DNSSEC, key, tag 16833, algorithm ECDSAP256SHA256, KSK, public, active
2021-06-07T08:30:11+0200 info: [example.net.] DNSSEC, key, tag  1808, algorithm ECDSAP256SHA256, public
2021-06-07T08:30:11+0200 info: [example.net.] DNSSEC, key, tag 19569, algorithm ECDSAP256SHA256, public, active


2021-06-07T08:37:11+0200 info: [example.net.] DNSSEC, ZSK rollover started
2021-06-07T08:37:11+0200 info: [example.net.] DNSSEC, key, tag 16833, algorithm ECDSAP256SHA256, KSK, public, active
2021-06-07T08:37:11+0200 info: [example.net.] DNSSEC, key, tag 54994, algorithm ECDSAP256SHA256, public
2021-06-07T08:37:11+0200 info: [example.net.] DNSSEC, key, tag  1808, algorithm ECDSAP256SHA256, public, active

After two ZSK rolls, I see a diagnostic error repeating itself DNSSEC re-sign' failed (signing error). More or less simultaneously I become curious as to whether the key_id (the CKA_ID attribute is intended as a means of distinguishing multiple public-key/private-key pairs) can be calculated, and I chase after the key creation. I get lucky with pkcs11_generate_key() and see the CKA_ID is a random value. Per chance I notice that in case of missing randomness DNSSEC_KEY_GENERATE_ERROR is returned. I wonder whether that’s the issue, or is this too much of a coincidence? Maybe knot. ;-)

$ grep -r DNSSEC_KEY_GENERATE_ERROR .
./libdnssec/error.c:    { DNSSEC_KEY_GENERATE_ERROR,    "key generation error" },
./libdnssec/error.h:    DNSSEC_KEY_GENERATE_ERROR,
./libdnssec/keystore/pkcs11.c:          return DNSSEC_KEY_GENERATE_ERROR;
./libdnssec/keystore/pkcs8.c:           return DNSSEC_KEY_GENERATE_ERROR;

On a hunch, and in spite of working on actual hardware and not on a virtualized system, I install haveged assuming that GnuTLS’ gnutls_rnd() functions will read from /dev/urandom. The situation is better but not good: Knot manages first seven then five ZSK rollovers and goes into the key generation error dance.

Thanks to the beauty of Open Source, I have a binary with an fprintf() lurking in it:

2021-06-07T12:36:17+0200 info: [example.net.] DNSSEC, signing zone
**** JP: gnutls_pkcs11_privkey_generate3: 0xFFFFFECC
2021-06-07T12:36:17+0200 warning: [example.net.] DNSSEC, key rollover, action generate (key generation error)
2021-06-07T12:36:17+0200 error: [example.net.] DNSSEC, failed to initialize (key generation error)

I decide it’s time to open an issue but also arrive at the conclusion it’s not Knot’s fault but likely a glitch on the device and close the ticket the next day. The firmware on my SmartCard-HSM is that with which it was delivered six years ago.

Update: I perform a firmware update on the SmartCard-HSM, a process which works very smoothly on my Mac. I’m disappointed that the HSM now doesn’t work at all. Thanks to Andreas Schwier who informs me I’ve been mixing up drivers, it’s now working. I quote:

You seem to be mixing two different PKCS#11 modules, the one from OpenSC and our own from sc-hsm-embedded (the former is opensc-pkcs11.so, the later libsc-hsm-pkcs11.so). That should be no issue, as both use the same encoding for meta-data - but differences may be subtle.

% rm /opt/lib/libsc-hsm-pkcs11.so
% find find / -name opensc-pkcs11.so
/usr/lib64/opensc-pkcs11.so
/usr/lib64/pkcs11/opensc-pkcs11.so

% cmp $(find / -name opensc-pkcs11.so)
%

After proceeding as Andreas suggests, I rerun my tests and Knot rolls one key after the other every seven minutes over the course of almost three days.

% grep -c ZSK.rollover.started knot.log
557

Issue solved, and ticket updated to indicate so.

Verifying signed zones

I used three different checkers to validate each of the signed zones that were produced: validns, dnssec-verify, and ldns-verify. If the signer didn’t write out a file proper, I transferred the zone into a file and checked that.

All three are happy with the signed zones when they’re signed with KSK/ZSK, but there is a bug in validns: it fails the check and complains with an inaccurate diagnostic when the zone is signed with a CSK.

$ ldns-verify-zone example.com.signed
Zone is verified and complete

$ dnssec-verify -x -o example.com example.com.signed
Loading zone 'example.com' from file 'example.com.signed'
Verifying the zone using the following algorithms: ECDSAP256SHA256.
Zone fully signed:
Algorithm: ECDSAP256SHA256: KSKs: 1 active, 0 stand-by, 0 revoked
                            ZSKs: 0 active, 0 present, 0 revoked

$ validns -z example.com -p all example.com.signed
example.com.signed:8: No KSK found
$ echo $?
1

Line 8 of that file has a DNSKEY resource record with flags 257. At best a missing ZSK should be reported.

Summary

That went much more smoothly than expected, but I did disable SELinux because I didn’t want to waste more time than I already had getting named to be permitted to access the USB device. Long story short: fully disabled SELinux configuration and BIND and Knot DNS running as root. (If you feel the need to volunteer dictating to me what I must type to get these working with SELinux and smart card access, you know where to find me. :-)

The CardContact SmartCard-HSM is a lightweight hardware security module which comes in different form factors and protects your RSA and ECC keys, and it’s one which can definitely be used with BIND, Knot, NSD (with, say, ldns-signzone) and, as discussed previously, also with PowerDNS. It is actually a quite capable device: it provides optional key backup and restore which permits setting up multiple cards for disaster recovery or for key escrow schemes, where multiple key custodians can be nominated for protecting key imports and exports.

A new version of this device with support for larger keys (RSA 4096 bit, ECC 521 bit, and AES with 128, 196 and 256 bit) and key domains – a group of SmartCard-HSMs that can share the same cryptographic material – was released in 2019.

It seems that the Nitrokey HSM 2 is in fact a SmartCard-HSM.

Two new SmartCard-HSM or Nitrokey HSM2 would be tempting to play with key export/imports…

Further reading

dnssec, dns, and hsm :: 04 Jun 2021 :: e-mail

Other recent entries