I’ve been thinking a bit about affordable hardware security modules (HSM); affordable is relative so I’ll start with “inexpensive”, and I recall I had mixed experiences with a very small HSM I purchased six years ago. I am curious to see whether I’m now able to wire up the CardContact SmartCard-HSM to some Open Source DNS servers and whether the experience has improved. TL;DR: I can and it has.

SmartCard-HSM in a Thinkpad

With private (preferably ECDSA) keys generated on the HSM I’d like to be able to accomplish the following tasks:

  1. Manually sign zones with dnssec-signzone and BIND inline signing. I expect BIND’s new dnssec-policy will not work as named would have to create keys on the HSM; the policy currently supports key-directory only
  2. Use Knot DNS to automatically sign zones, at least using “manual” policy, ideally with “automatic” key management in which the server maintains signing keys
  3. Use LDNS’s ldns-signzone utility to manually sign a zone

All this is possible using keys stored on the file system, but having private keys on a hardware security module makes for a very aggravating experience when attempting to steal them. Look at the photo: the private keys are on that USB token; breaking into the server over the network isn’t enough – the device has to be physically abstracted.

Using BIND utilities

There exists an article on PKCS#11 in BIND9 (recommended reading), but I’m grateful that ISC’s Ondřej Surý points me in the preferred direction:

as of this moment, the OpenSSL engine_pkcs11 is a preferred way, so any HSM that’s well supported by OpenSC project should be ok.

Applications which need to use a smart card library do so using an interface of which there are, most unfortunately, several different implementations. One of these is PKCS#11 which exists on most operating systems; PKCS#11 is an API for accessing cryptographic hardware such as HSMs. OpenSC implements this interface as a smart card library on multiple platforms.

Quoting from libp11:

engine_pkcs11 tries to fit the PKCS#11 API within the engine API of OpenSSL. That is, it provides a gateway between PKCS#11 modules and the OpenSSL engine API. One has to register the engine with OpenSSL and one has to provide the path to the PKCS#11 module which should be gatewayed to.

Lots of words. Luckily I understand a few of the two-syllable ones, so I feel I can get started.

Software installation

I begin with a fresh installation of Fedora 34, and since I don’t need to compile BIND myself for native PKCS#11, I choose to install a 9.16.16 package by ISC. Other system packages I install are:

  • opensc for the pkcs11-tool utility
  • ldns-utils for the LDNS utilities
  • knot, available in the current version 3.0.6 in the Fedora repository

I also need the libp11 PKCS#11 wrapper library which will interface between, e.g. BIND and the actual HSM driver, so I clone its repository. Ondřej recommends I use libp11 0.4.12 which was slated for release this year. It’s not yet been released hence I’m building from source. It contains many improvements to engine_pkcs11 sponsored by ISC.

% ./configure --with-enginesdir=/opt/bind9/engines --prefix=/opt
% make install
% ls -l /opt/bind9/engines/*.so
lrwxrwxrwx. 1 root root      9 Jun  3 14:32 /opt/bind9/engines/libpkcs11.so -> pkcs11.so
-rwxr-xr-x. 1 root root 364784 Jun  3 14:32 /opt/bind9/engines/pkcs11.so

Libp11’s documentation says I might need to configure the location of the modules in an OpenSSL config file, but later I re-read to see that in systems with p11-kit-proxy engine_pkcs11 has access to all the configured PKCS#11 modules. I see p11-kit{,-trust,-server} are installed, so this is likely the reason why BIND’s utilities and LDNS can access the SmartCard-HSM without my setting $OPENSSL_CONF. Be that as it may, this is what I add to /opt/bind9/etc/openssl.cnf, but it’t not being used:

[openssl_init]
engines=engine_section

[engine_section]
pkcs11 = pkcs11_section

[pkcs11_section]
engine_id = pkcs11
dynamic_path = /opt/bind9/engines/pkcs11.so
MODULE_PATH = /opt/lib/libsc-hsm-pkcs11.so
init = 0

(I’ve decided I won’t try to force remove p11-*.rpm ;-)

Working with the card

I insert the CardContact SmartCard-HSM and use the opensc-tool to verify the HSM device is accessible:

[10091.366761] usb 2-1.2: new full-speed USB device number 5 using ehci-pci
[10091.451190] usb 2-1.2: New USB device found, idVendor=04e6, idProduct=5817, bcdDevice= 1.00
[10091.451208] usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=5
[10091.451215] usb 2-1.2: Product: SCT3522CC token
[10091.451219] usb 2-1.2: Manufacturer: Identive
[10091.451223] usb 2-1.2: SerialNumber: 21121350600105
% opensc-tool -l
# Detected readers (pcsc)
Nr.  Card  Features  Name
0    Yes             Identive SCT3522CC token [CCID Interface] (21121350600105) 00 00

I had already initialized the card, so I begin by creating a key pair on the card.

% pkcs11-tool --module /usr/lib64/opensc-pkcs11.so \
              --login \
	      --keypairgen \
	      --key-type EC:prime256v1 \
	      --label ta-example.com
Using slot 0 with a present token (0x1)
Logging in to "SmartCard-HSM".
Please enter User PIN:
Key pair generated:
Private Key Object; EC
  label:      ta-example.com
  ID:         04
  Usage:      sign
  Access:     sensitive, always sensitive, never extractable, local
Public Key Object; EC  EC_POINT 256 bits
  EC_POINT:   0441047301f6b89a2468db4aaa57678481eb787aa88c33a255f7ae3c171ec3df4943621d9d4173043a737fa81bd654b1f84cf0e73115387356fa456477e7c6c0a01ad3
  EC_PARAMS:  06082a8648ce3d030107
  label:      ta-example.com
  ID:         04
  Usage:      encrypt, verify
  Access:     local

Then I need to generate the pair of key files which reference the key object stored on our HSM. The private key file will be used for DNSSEC signing of our zone as if it were a conventional key on the file system (i.e. one created with dnssec-keygen). The key material is stored on the HSM (and we cannot extract it), and the actual signing takes place there as we’ll vividly see when we time the signing of a zone with 1000 records in it.

I use the PIN here directly on the command line, but it can also be ferreted away into a file using pin-source. That adds just a level of indirection as the file must of course be readable by whichever utility (e.g. named) will be using its content.

% dnssec-keyfromlabel -E pkcs11 \
                      -a 13 \
		      -l "object=ta-example.com;pin-value=123456" \
		      -f KSK example.com
Kexample.com.+013+34583

% ls -l Kexample*
-rw-r--r--  1 root root  341 Jun  4 12:39 Kexample.com.+013+34583.key
-rw-------  1 root root  236 Jun  4 12:39 Kexample.com.+013+34583.private

% cat Kexample.com.+013+34583.private
Private-key-format: v1.3
Algorithm: 13 (ECDSAP256SHA256)
PrivateKey:
Engine: cGtjczExAA==
Label: cGtjczExOm9iamVjdD10YS1leGFtcGxlLmNvbTtwaW4tdmFsdWU9MTIzNDU2AA==
Created: 20210604103940
Publish: 20210604103940
Activate: 20210604103940

The two base64-encoded values we see in the private key file are “pkcs11” and “pkcs11:object=ta-example.com;pin-value=123456” for Engine and Label respectively. Our signing utilities and named will need to access the HSM and need the PIN to do so, which is why it’s stored in the .private file.

Manual signing with dnssec-signzone

We now have a single ECDSA key pair (algorithm 13) which we’ll use to sign the zone.

% dnssec-signzone -E pkcs11 -S -3 - -H 0 -A -x -z -t example.com
Fetching example.com/ECDSAP256SHA256/34583 (KSK) from key repository.
Verifying the zone using the following algorithms:
- ECDSAP256SHA256
Zone fully signed:
Algorithm: ECDSAP256SHA256: KSKs: 1 active, 0 stand-by, 0 revoked
                            ZSKs: 0 active, 0 present, 0 revoked
example.com.signed
Signatures generated:                     1883
Signatures retained:                         0
Signatures dropped:                          0
Signatures successfully verified:            0
Signatures unsuccessfully verified:          0
Signing time in seconds:               291.264
Signatures per second:                   6.464
Runtime in seconds:                    293.640

Six signatures per second is slow, but this particular HSM costs €60.00 only.

In addition to the existing KSK (on the HSM) I create a ZSK on the file system and have all the records in the zone signed with it except for the DNSKEY RRset which will be signed with the KSK on the HSM. This is bound to be much faster.

% dnssec-keygen -a 13 example.com
Generating key pair.
Kexample.com.+013+28096

% cat Kexample.com.+013+28096.private
Private-key-format: v1.3
Algorithm: 13 (ECDSAP256SHA256)
PrivateKey: MJQosqFwHt2ijSVJC/PoSD6qg9u/oeIgjGBaKrWNwoY=
Created: 20210604105112
Publish: 20210604105112
Activate: 20210604105112

% dnssec-signzone -E pkcs11 -S -3 - -H 0 -A -x -t example.com
Fetching example.com/ECDSAP256SHA256/34583 (KSK) from key repository.
Fetching example.com/ECDSAP256SHA256/28096 (ZSK) from key repository.
Verifying the zone using the following algorithms:
- ECDSAP256SHA256
Zone fully signed:
Algorithm: ECDSAP256SHA256: KSKs: 1 active, 0 stand-by, 0 revoked
                            ZSKs: 1 active, 0 present, 0 revoked
example.com.signed
Signatures generated:                     1883
Signatures retained:                         0
Signatures dropped:                          0
Signatures successfully verified:            0
Signatures unsuccessfully verified:          0
Signing time in seconds:                 0.317
Signatures per second:                5939.931
Runtime in seconds:                      2.381

It is. Much faster. :-) (Note the information regarding fetching keys from the repository: the KSK is on the HSM and the ZSK on the file system.)

I’m a great believer in CSK/SSK (Combined- or Single Signing Key), but I will change my mind if I have to use an inexpensive HSM: using split keys (ZSK on the file system, KSK on the HSM) is definitely and understandably a lot faster.

Manual signing with ldns-signzone

An alternative to using BIND command-line utilities exists in the form of LDNS, a programming library created by NLnetLabs. LDNS provides some interesting and practical command-line tools such as is drill, a dig lookalike specifically designed to work with DNSSEC, and ldns-signzone which, well, signs zones. This program accomplishes what dnssec-signzone does, and I can use the zone file it signs as input to, say, NSD. (I’m looking forward to the new NSD version with CREDNS functionality.) If ldns_signzone is compiled with engine support I can use it with the HSM. (For example on OpenBSD, the program doesn’t support engines.)

ldns-signzone and NSD

With option -K I specify the KSK to use as an algorithm mnemonic or number (e.g. 13) followed by a comma, and the PKCS#11 key identifier, here the literal “label_” concatenated with the label of the key on the HSM as created above. The program signs our zone with the KSK only (as in our first example earlier) and does the job in the same time, the bottleneck being our slow HSM.

$ time ldns-signzone -E pkcs11 \
        -o example.com \
        -f example.com.signed.ldns \
        -K ECDSAP256SHA256,label_ta-example.com \
        example.com
Engine key id: label_ta-example.com, algo 13
Enter PKCS#11 token PIN for SmartCard-HSM (UserPIN):

real    4m57.318s
user    0m0.833s
sys     0m0.515s

To automate signing with ldns-signzone I can specify the PIN and the key to use in a PKCS#11 URI. Like in the second example with dnssec-signzone above, I additionally create an on-filesystem ZSK with which to sign the zone.

$ ldns-keygen -a 13 example.com
Kexample.com.+013+61121

$ printf 123456 > mypin
$ time ldns-signzone -E pkcs11 \
      -o example.com \
      -K 'ECDSAP256SHA256,pkcs11:object=ta-example.com;pin-source=mypin' \
      example.com \
      Kexample.com.+013+61121
Engine key id: pkcs11:object=ta-example.com;pin-source=mypin, algo 13

real    0m2.365s
user    0m0.218s
sys     0m0.029s

Both zone signing utilities accomplish the task in just over two seconds: the ZSK on the file system signs all records in the zone and the KSK in the HSM the DNSKEY RRset.

Enough of the manual signing.

BIND inline signing

With BIND 9.9, ISC introduced a new “inline-signing” option which allows for transparent signing of zones loaded or transferred to the server. named creates a signed version of the zone which answers all queries and transfer requests without altering the original unsigned version. This method can work well for DNS administrators who wish to continue editing their zone files without “seeing” the DNSSEC data in them, and it also works very well for dynamically updatable zones.

BIND inline signing

We now configure BIND to use inline signing and maintain DNSSEC automatically for us. I’ll start off with a zone containing just a few records and a configuration that enables me to update the zone dynamically. (I want to see the green LED on the SmartCard-HSM blink every time I do a dynamic update… :-)

In named.conf I define the zone:

zone "example.com" IN {
    type primary;
    file "example.com";
    inline-signing yes;
    auto-dnssec maintain;

    update-policy {
            grant local-ddns zonesub ANY;
            grant "jpm" zonesub ANY;
    };

    key-directory "keys";
};

I then launch named in the foreground:

% named -E pkcs11 -g
starting BIND 9.16.16 (Stable Release) <id:0c314d8>
...
zone example.com/IN (unsigned): loaded serial 1
all zones loaded
running
zone example.com/IN (signed): loaded serial 1
zone example.com/IN (signed): receive_secure_serial: unchanged
zone example.com/IN (signed): sending notifies (serial 1)
zone example.com/IN (signed): reconfiguring zone keys
Fetching example.com/ECDSAP256SHA256/34583 (KSK) from key repository.
DNSKEY example.com/ECDSAP256SHA256/34583 (KSK) is now published
DNSKEY example.com/ECDSAP256SHA256/34583 (KSK) is now active
zone example.com/IN (signed): next key event: 04-Jun-2021 14:00:06.698
zone example.com/IN (signed): sending notifies (serial 3)

That looks very good. BIND has accessed the HSM and has signed the zone, and I see the DNSKEY RRset.

$ dig @::1 example.com DNSKEY +dnssec +nocrypto

;; ANSWER SECTION:
example.com.            86400   IN      DNSKEY  257 3 13 [key id = 34583]
example.com.            86400   IN      RRSIG   DNSKEY 13 2 86400 20210704110005 20210604100006 34583 example.com. [omitted]

And behold the #blinkenlights when I submit a dynamic update to the zone. (You can’t see the blinking – you’ll have to trust me.)

client .. 127.0.0.1#47646/key jpm: updating zone 'example.com/IN': adding an RR at 'jp.example.com' TXT "2021-06-04T13:05:14"
zone example.com/IN (signed): serial 4 (unsigned 2)

Knot DNS

Knot DNS is a very capable authoritative server which manages keys in a key store; it’s also known to work with several PKCS#11 devices. Knot is capable of performing fully automatic signing with a Key And Signing Policy (KASP) and can maintain signing keys.

knot signing with SmartCard-HSM

I’m aware that Knot uses GnuTLS so I use the latter’s p11tool to obtain the PKCS#11 URI for the HSM, and don’t ask me why it knows, but it’s correctly identified the device; probably some PC/SD magic?

% p11tool --list-tokens
...
Token 2:
        URL: pkcs11:model=PKCS%2315%20emulated;manufacturer=www.CardContact.de;serial=DECC0100509;token=SmartCard-HSM%20%28UserPIN%29
        Label: SmartCard-HSM (UserPIN)
        Type: Hardware token
        Flags: RNG, Requires login
        Manufacturer: www.CardContact.de
        Model: PKCS#15 emulated
...

Configuring that URI into Knot’s keystore configuration caused Knot to report DNSSEC, failed to initialize (PKCS #11 token not available) even when I added pin-value, so I simplified it to pin-value only. Upon startup, the server rightly reports no configured keys, and since it cannot sign the zone without them, rightly refuses to serve the zone at all.

2021-06-06T10:43:40+0200 error: [example.com.] DNSSEC, no keys are available
2021-06-06T10:43:40+0200 error: [example.com.] DNSSEC, failed to load keys (no keys for signing)
2021-06-06T10:43:40+0200 error: [example.com.] zone event 'load' failed (no keys for signing)

So far so good, and I see from a change in the green LED (what would we do without #blinkenlights) that the HSM has been accessed.

The configuration I’m using for this test now looks like this:

server:
    rundir: "/run/knot"
    user: root:root
    listen: [ 127.0.0.1@53, ::1@53 ]
log:
  - target: "/etc/knot/k.log"
    any: info
database:
    storage: "/var/lib/knot"

keystore:
   - id: smartcardHSM
     backend: pkcs11
     config: "pkcs11:pin-value=123456 /usr/lib64/opensc-pkcs11.so"
     # note the space before path to ^^ module

key:
  - id: jpm
    algorithm: hmac-sha256
    secret: L4HAwrawtduzDlkq29jbprfxnOh+0hr8DLPEYNiwZM8=

acl:
  - id: xfr
    address: 127.0.0.1
    action: transfer

  - id: upd
    key: jpm
    action: update
    
template:
  - id: default
    storage: "/var/lib/knot"
    file: "%s.zone"

policy:
  - id: manualHSM
    keystore: smartcardHSM
    single-type-signing: on
    manual: on

zone:
  - domain: example.com
    dnssec-signing: on
    dnssec-policy: manualHSM
    acl: [ xfr, upd ]

I now “import” the key from the HSM into Knot’s keystore. It’s called importing but it’s basically just associating the HSM key with an object in the keystore a bit like how BIND’s utilities do. Knot’s keymgr utility requires a key_id (not a key label which we’ve used so far) with which to import the key. I obtain this using p11tool (it’s the same ID which was reported earlier when I generated the key).

% p11tool \
     --set-pin=123456 \
     --list-all \
     'pkcs11:model=PKCS%2315%20emulated;manufacturer=www.CardContact.de;serial=DECC0100509;token=SmartCard-HSM%20%28UserPIN%29'
...
Object 3:
        URL: pkcs11:model=PKCS%2315%20emulated;manufacturer=www.CardContact.de;serial=DECC0100509;token=SmartCard-HSM%20%28UserPIN%29;id=%04;object=ta-example.com;type=public
        Type: Public key (EC/ECDSA-SECP256R1)
        Label: ta-example.com
        ID: 04
...

Now I attempt to import said key and describe the key to Knot so that it knows how to handle it and what to do with it.

% keymgr example.com import-pkcs11 04 algorithm=ECDSAP256SHA256 ksk=yes zsk=yes
04
OK

% knotc reload
Reloaded

and behold more #blinkenlights as Knot begins to sign the zone. (I’m using a single signing key so I expect this to take approximately as long as the *signzone utilities needed.)

2021-06-06T10:51:22+0200 info: configuration reloaded
2021-06-06T10:51:22+0200 info: [example.com.] zone file parsed, serial 8
2021-06-06T10:51:22+0200 info: [example.com.] DNSSEC, key, tag 34583, algorithm ECDSAP256SHA256, CSK, public, active
2021-06-06T10:51:22+0200 info: [example.com.] DNSSEC, signing started
2021-06-06T10:56:12+0200 info: [example.com.] DNSSEC, successfully signed
2021-06-06T10:56:12+0200 info: [example.com.] loaded, serial none -> 8 -> 9, 328474 bytes
% keymgr example.com ds
example.com. DS 34583 13 2 cf4b311003e58abc776723158f8406c1af4f4f2e1ec6f7092f6cffb93a292589
example.com. DS 34583 13 4 52c057fa00cf0ee35a5e16ed2a0f0317b02479ac5a2382b0e7750d79e47b4954b6068de603a7b93351d8fb65cacf8d49

I verify by dynamically updating the zone that it’s being signed by the CSK. I could also have used kzonesign to sign the zone from the CLI.

Knot creates keys on the HSM

I use keymgr to create a key on the SmartCard-HSM, and verify that it is indeed on the device:

% keymgr example.com generate algorithm=ECDSAP256SHA256 ksk=yes
144821ef5ae749180c5299376c27f8154f16a5d6

% p11tool --set-pin=123456 --list-all 'pkcs11:serial=DECC0100509'
...
Object 5:
        URL: pkcs11:model=PKCS%2315%20emulated;manufacturer=www.CardContact.de;serial=DECC0100509;token=SmartCard-HSM%20%28UserPIN%29;id=%14%48%21%EF%5A%E7%49%18%0C%52%99%37%6C%27%F8%15%4F%16%A5%D6;type=public
        Type: Public key (EC/ECDSA-SECP256R1)
        Label:
        ID: 14:48:21:ef:5a:e7:49:18:0c:52:99:37:6c:27:f8:15:4f:16:a5:d6
...

Sadly I’ve not found an incantation of p11tool options which would permit me to set the key label. I realize Knot doesn’t need it, but I find the label documents what the key is for.

So, can I convince Knot to create and maintain keys on the HSM automatically? I configure a new zone with a new policy (and crazy timers, I know, but I want changes to occur quickly).

policy:
  - id: autoHSM
    keystore: smartcardHSM
    single-type-signing: off
    manual: off
    algorithm: ecdsap256sha256
    zone-max-ttl: 60
    dnskey-ttl: 60
    propagation-delay: 60
    zsk-lifetime: 300

zone:
  - domain: example.net
    dnssec-signing: on
    dnssec-policy: autoHSM
    acl: [ xfr, upd ]

I launch the server afresh and see it begins by generating a KSK and a ZSK and henceforth neatly rolls the ZSK every few minutes.

2021-06-07T08:30:11+0200 info: [example.net.] DNSSEC, ZSK rollover started
2021-06-07T08:30:11+0200 info: [example.net.] DNSSEC, key, tag 16833, algorithm ECDSAP256SHA256, KSK, public, active
2021-06-07T08:30:11+0200 info: [example.net.] DNSSEC, key, tag  1808, algorithm ECDSAP256SHA256, public
2021-06-07T08:30:11+0200 info: [example.net.] DNSSEC, key, tag 19569, algorithm ECDSAP256SHA256, public, active


2021-06-07T08:37:11+0200 info: [example.net.] DNSSEC, ZSK rollover started
2021-06-07T08:37:11+0200 info: [example.net.] DNSSEC, key, tag 16833, algorithm ECDSAP256SHA256, KSK, public, active
2021-06-07T08:37:11+0200 info: [example.net.] DNSSEC, key, tag 54994, algorithm ECDSAP256SHA256, public
2021-06-07T08:37:11+0200 info: [example.net.] DNSSEC, key, tag  1808, algorithm ECDSAP256SHA256, public, active

After two ZSK rolls, I see a diagnostic error repeating itself DNSSEC re-sign' failed (signing error). More or less simultaneously I become curious as to whether the key_id (the CKA_ID attribute is intended as a means of distinguishing multiple public-key/private-key pairs) can be calculated, and I chase after the key creation. I get lucky with pkcs11_generate_key() and see the CKA_ID is a random value. Per chance I notice that in case of missing randomness DNSSEC_KEY_GENERATE_ERROR is returned. I wonder whether that’s the issue, or is this too much of a coincidence? Maybe knot. ;-)

$ grep -r DNSSEC_KEY_GENERATE_ERROR .
./libdnssec/error.c:    { DNSSEC_KEY_GENERATE_ERROR,    "key generation error" },
./libdnssec/error.h:    DNSSEC_KEY_GENERATE_ERROR,
./libdnssec/keystore/pkcs11.c:          return DNSSEC_KEY_GENERATE_ERROR;
./libdnssec/keystore/pkcs8.c:           return DNSSEC_KEY_GENERATE_ERROR;

On a hunch, and in spite of working on actual hardware and not on a virtualized system, I install haveged assuming that GnuTLS’ gnutls_rnd() functions will read from /dev/urandom. The situation is better but not good: Knot manages first seven then five ZSK rollovers and goes into the key generation error dance.

Thanks to the beauty of Open Source, I have a binary with an fprintf() lurking in it:

2021-06-07T12:36:17+0200 info: [example.net.] DNSSEC, signing zone
**** JP: gnutls_pkcs11_privkey_generate3: 0xFFFFFECC
2021-06-07T12:36:17+0200 warning: [example.net.] DNSSEC, key rollover, action generate (key generation error)
2021-06-07T12:36:17+0200 error: [example.net.] DNSSEC, failed to initialize (key generation error)

I decide it’s time to open an issue but also arrive at the conclusion it’s not Knot’s fault but likely a glitch on the device and close the ticket the next day. The firmware on my SmartCard-HSM is that with which it was delivered six years ago.

Update: I perform a firmware update on the SmartCard-HSM, a process which works very smoothly on my Mac. I’m disappointed that the HSM now doesn’t work at all. Thanks to Andreas Schwier who informs me I’ve been mixing up drivers, it’s now working. I quote:

You seem to be mixing two different PKCS#11 modules, the one from OpenSC and our own from sc-hsm-embedded (the former is opensc-pkcs11.so, the later libsc-hsm-pkcs11.so). That should be no issue, as both use the same encoding for meta-data - but differences may be subtle.

% rm /opt/lib/libsc-hsm-pkcs11.so
% find find / -name opensc-pkcs11.so
/usr/lib64/opensc-pkcs11.so
/usr/lib64/pkcs11/opensc-pkcs11.so

% cmp $(find / -name opensc-pkcs11.so)
%

After proceeding as Andreas suggests, I rerun my tests and Knot rolls one key after the other every seven minutes over the course of almost three days.

% grep -c ZSK.rollover.started knot.log
557

Issue solved, and ticket updated to indicate so.

Verifying signed zones

I used three different checkers to validate each of the signed zones that were produced: validns, dnssec-verify, and ldns-verify. If the signer didn’t write out a file proper, I transferred the zone into a file and checked that.

All three are happy with the signed zones when they’re signed with KSK/ZSK, but there is a bug in validns: it fails the check and complains with an inaccurate diagnostic when the zone is signed with a CSK.

$ ldns-verify-zone example.com.signed
Zone is verified and complete

$ dnssec-verify -x -o example.com example.com.signed
Loading zone 'example.com' from file 'example.com.signed'
Verifying the zone using the following algorithms: ECDSAP256SHA256.
Zone fully signed:
Algorithm: ECDSAP256SHA256: KSKs: 1 active, 0 stand-by, 0 revoked
                            ZSKs: 0 active, 0 present, 0 revoked

$ validns -z example.com -p all example.com.signed
example.com.signed:8: No KSK found
$ echo $?
1

Line 8 of that file has a DNSKEY resource record with flags 257. At best a missing ZSK should be reported.

Summary

That went much more smoothly than expected, but I did disable SELinux because I didn’t want to waste more time than I already had getting named to be permitted to access the USB device. Long story short: fully disabled SELinux configuration and BIND and Knot DNS running as root. (If you feel the need to volunteer dictating to me what I must type to get these working with SELinux and smart card access, you know where to find me. :-)

The CardContact SmartCard-HSM is a lightweight hardware security module which comes in different form factors and protects your RSA and ECC keys, and it’s one which can definitely be used with BIND, Knot, NSD (with, say, ldns-signzone) and, as discussed previously, also with PowerDNS. It is actually a quite capable device: it provides optional key backup and restore which permits setting up multiple cards for disaster recovery or for key escrow schemes, where multiple key custodians can be nominated for protecting key imports and exports.

A new version of this device with support for larger keys (RSA 4096 bit, ECC 521 bit, and AES with 128, 196 and 256 bit) and key domains – a group of SmartCard-HSMs that can share the same cryptographic material – was released in 2019.

It seems that the Nitrokey HSM 2 is in fact a SmartCard-HSM.

Two new SmartCard-HSM or Nitrokey HSM2 would be tempting to play with key export/imports…

Further reading

dnssec, dns, and hsm :: 04 Jun 2021 :: e-mail