I was asked to assist in debugging a strange issue involving a BIND resolver: seemingly correlating with an upgrade to Debian 10 a while ago, the chaps were reporting that their 9.11.5 BIND resolvers where responding with impossible TTLs on NOERROR/NODATA responses. My answer: nope – can’t happen.

Spoiler: it can.

Let’s look at an SOA record. It’s incomplete, but the important bits are showing (and for those in the know, the negative TTL here is also 3600). You’re welcome to follow along – example.com is delegated. :-)

Querying the authority server for the SOA shows

;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 1

example.com.   3600  IN   SOA

Now, when querying the resolver for this owner with a non-existent type, SRV, say, I should get a NOERROR/NODATA response, so here it is:

;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

example.com.            10800   IN      SOA

Whazzat! Where does the TTL of 10800 come from!?! I blamed everything and the kitchen sink. The network, the authority server, the selection of menus in the company restaurant (even though I’m at home) – you name it, I blamed it, but I honestly didn’t believe BIND was doing this. It must be that hand-rolled ancient Perl-based database-driven unmaintained authoritative server from which they’re feeding it records … I even blamed AppArmor so that was disabled as well.

In cases like these I remove as much complexity as possible, and thankfully the chaps I was working with were game to attempting anything; they’d been working at this problem for this for a while. I should add that I’d spent an hour in the morning attempting but failing at reproducing the issue on multiple versions of BIND.

First things first, let’s start with the following configuration file: (I can almost hear Evan tell me “that’s too verbose!” :-)

options {
        directory ".";

This configuration also surfaced the 10800 TTL. Hmm.

We started off by capping max-ncache-ttl; I choose idiotic numbers I will recognize. Let’s set that to 37.

example.com.            37      IN      SOA

OK, that looks sane. Let’s up it a bit to, say, max-ncache-ttl 12341234; even though it “will be silently truncated to 7 days if set to a greater value”.

example.com.            604800  IN      SOA

DAFUQ? The behavior of that TTL increasing this way doesn’t make sense to me at all.

Well into the morning, people were getting hungry so we took a 30 minute break, and I got a cup of coffee and came to the conclusion the cause must be a combination of strange defaults. On the way back to my desk it occurred to me what the issue might be, and a moment later I was able to reproduce the 10800 TTL on my own test install of 9.11.5, and then I “fixed” it. Here’s how:

options {
        directory ".";
	dnssec-validation auto;

In 9.11.5 dnssec-validation defaults to yes, and the Bv9ARM clearly says

If set to yes, DNSSEC validation is enabled, but a trust anchor must be manually configured using a trusted-keys or managed-keys statement. The default is yes.

A trust anchor was not configured, neither manually nor at all. As they say in Turkey “trust anchor yok”. My chaps had noticed the option now defaulting to yes, but they’d ignored it during the update thinking “we’ll check that later when all quiets down, let’s concentrate on the important things first”. Ouch. That caused the pain. (If you’re reading this, beware: in future versions the semantics of this option change.)

So why does the auto fix it? Quoting from the ARM again:

If set to auto, DNSSEC validation is enabled, and a default trust anchor for the DNS root zone is used.

All’s well which ends well.

I’d normally submit a bug report, but I’ve decided it’s probably not worthwhile; it’s basically a configuration error, although it wasn’t logged as such. (If somebody at ISC sees this and wants me to, they know where to find me. :-)

OK, so I lied: we were at it for 16800 seconds, and it was a very pleasant debugging / hair-pulling session with fun and very capable chaps. I cannot tell you why this TTL business is important because you could deduce a company name from that, and I don’t divulge those. Unless we’re out having a drink. ;-)

DNS :: 21 Apr 2021 :: e-mail

I tend to overlook this frequently, so as a note to self, it’s possible to store arbitrary passwords or secrets in the macOS keychain, so as to use them from, say, Shell scripts.

From the command line I add an item named myk01 which will contain generic password with

$ security add-generic-password -a jpm -s myk01 -j 'this is for the program xyz' -w
password data for new item:
retype password for new item:

The secret is in the keychain, as this command tells me:

$ security find-generic-password -a jpm -s myk01
keychain: "/Users/jpm/Library/Keychains/login.keychain-db"
version: 512
class: "genp"
    0x00000007 <blob>="myk01"
    0x00000008 <blob>=<NULL>
    "cdat"<timedate>=0x32303231303431383039313632375A00  "20210418091627Z\000"
    "icmt"<blob>="this is for the program xyz"
    "mdat"<timedate>=0x32303231303431383039313632375A00  "20210418091627Z\000"

macOS keychain

And I can obtain just the password by adding a -w to the command:

$ security find-generic-password -a jpm -s myk01 -w

And that latter format is what I will use to obtain a password for this particular command:


export PASS=$(security find-generic-password -a $(whoami) -s myk01 -w)


Useful on occasion.


I want to access such a generic password from a Go program, which I do with go-keychain. I test with a “personalized” example from the README:

package main

import (

func main() {
	q := keychain.NewItem()


	res, err := keychain.QueryItem(q)
	if err != nil {
	} else if len(res) != 1 {
		fmt.Println("Not found")
	} else {
		password := string(res[0].Data)
		fmt.Printf("Password is: %T: %v\n", password, password)
{map[acct:jpm class:140735646167104 m_Limit:140735646169888 r_Data:true svce:myk01]}
Password is: string: sekr1t

Further reading

  • Roberto points us to a bit more high-level method using envchain, with support for macOS keychain or with D-Bus secret service.

shell and macos :: 18 Apr 2021 :: e-mail

I understand the classic UNIX Magic poster by Gary Overacre was distributed at past USENIX conferences, and I’ve known of it for years but now in confinement, I’ve decided I want one to hang in my office.

Unix Magic

The poster features a white bearded wizard with UNIX related objects around him, for example a spool of thread, a boot, a fork, pipes, and a bunch of containers labeled troff, awk, diff, uucp, make, null, and there’s even a C container and cracked B container. And some oregano:

The oregano is reputedly referring to an incident in which one of the original folks involved with BSD was hassled for coming across the Canadian/U.S. border with a bag of what was assumed to be an illegal substance, and turned out to be oregano.

How to get a poster which was drawn in 1986? There are lots of bad photos of the poster floating around, but certainly nothing good enough to turn into a poster of any size.

Via Twitter, Jaume points to a 32MB PNG (3675 x 5475) which I download and convert to TIFF. I then ask around a bit and find a shop willing to print that onto a material which looks exactly like what I want.

image detail

The material is called Aluminium Dibond which, and I quote, makes for a “matte, reflection-free print using a state-of-the-art, 7-color printing method. White and brighter parts of the image have a subtle sheen. With the aluminium Dibond backing, this print combines high picture quality and perfect durability”. Quite a mouthful, but the result, which I got in two days, is astonishing.

I love the material and how almost luxurious it looks and feels even though the colors are “matte” as described for this material. The result is not as vibrant or shiny as the image at the top of this page, but the advantage is no reflections when the light shines on it either.

the side

I was a bit skeptical and shy of spending EUR 80 on the picture, but I can say without reservation, that I’m very happy with what Whitewall has made for me, and thanks to Jaume for the file!



So, thanks to Jaco I’ve been able to get in touch with Gary Overacre’s spouse. They are willing to sell original posters, signed by Gary. There are about 30 “Unix Feuds” and 65 “Unix Magic” posters left (“it’s hard to get an accurate count as they’re lying flat in a drawer and they don’t want to wrinkle them” :-) The cost of each poster is USD 250.00 (two-hundred and fifty USD) plus packing and shipping. I have told them I’d be willing to collect and forward email addresses to them. So, in order to simplify this for the Overacres, I propose you think about whether you want either or both of the posters, and send me (Jan-Piet) an email at jp()mens.de with your order. Specify “Feuds” and/or “Magic”, your full name and a short sentence that you commit to ordering. I will collect the first 30 addresses for “Feud” and the first 65 for “Magic” and send them off to the Overacres, and the rest is between you and them.

As soon as this last paragraph is stricken through, all posters will have been accounted for.

unix :: 09 Apr 2021 :: e-mail

It’s early mornings here, and I’ve not [yet] opened a bottle, and to my knowledge I’ve not caught the pest, but the words in the title are related to each other. How?

white dig towel

If you look closely at the dig syntax on that oh-so lovely towel ISC have in their merchandise shop (click on the towel) you’ll notice an optional %comment at the end of the command. I remembered that from way back but also remembered that I never actually bothered to find out what that even did.

Winfried asked and I woke up our friends at ISC to find out. Turns out the percent sign really does introduce a comment and has been in the code since BIND 8.

The syntax on the towel still is in the fabulous BIND 9 ARM but isn’t explained, and dig(1) man pages no longer even bother mentioning it.

From a current dig.c:

rc = argc;
rv = argv;
for (rc--, rv++; rc > 0; rc--, rv++) {
	debug("main parsing %s", rv[0]);
	if (strncmp(rv[0], "%", 1) == 0) {

(Let’s ignore the renaming of argc and argv (saving a few characters on each), and let’s also ignore using strncmp to check the first character of a variable, ok?)

So what does it do? Nothing. It is a comment and inhibits further parsing of the command line:

$ dig jpmens.net %rest-is-ignored +dnssec

The +dnssec flag is unused.

The assumption is that it could be used in batch files, maybe before the advent of shell scripts (which have their own comment sign), or (gasp!) in digrc files.

And seeing you made it this far, you might also be interested to learn why zone master files use a semicolon for comments.

dns :: 31 Mar 2021 :: e-mail

There’s no end to the fun that was made of me today for touching the K-word. Be that as it may, I have to do this if I want to continue giving AWX/Tower trainings, and in order to do that I need AWX to use an SSH jump host to get to nodes. The reasons for that lie hidden in here.

This post is going to be a quick and dirty collection of how I solved the particular issue I requested help on, and the last thing you want to do is to ask me for help on Kubernetes & co. It took me several hours to solve this problem.

What’s the problem? I need to deploy an SSH key and an SSH conf file into the containers (or are those pods?) the AWX task (awx-task) processes are running in.

In order to accomplish this, I finally was able to

  1. Create kubernetes secrets from files
  2. Deploy (I hope that’s the right term) AWX configured to use those secrets to create read-only volumes in minikube.

Creating the secrets is easy:

$ kubectl create secret generic awx-jp01 \
	--from-file=keyfile=$HOME/umleit \
secret/awx-jp01 created

$ kubectl describe secrets awx-jp01
Name:         awx-jp01
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

conf:     221 bytes
keyfile:  444 bytes

Then comes the operator (?) deployment (?) file. (You’re noticing I don’t know the proper terminology, and I’m not quite sure I really want to know it…)

apiVersion: awx.ansible.com/v1beta1
kind: AWX
  name: awx
  tower_ingress_type: Ingress
  tower_hostname: awx.example.com
  tower_admin_user: admin
  tower_admin_email: admin@example.com
  tower_admin_password_secret: changeme
  tower_task_extra_volume_mounts: |
     - name: "rootsshdir"
       mountPath: "/var/lib/awx/.ssh"
       readOnly: true
  tower_extra_volumes: |
     - name: "rootsshdir"
          secretName: "awx-jp01"
          - key: keyfile
            path: "id_ed25519"
          - key: conf
            path: "config"

The magic is in tower_extra_volumes which sources the secrets created earlier, and tower_task_extra_volume_mounts which creates the actual bind-mount from the secrets. I’ve not found a stich of documentation on this; either term produces two google hits; quite the record. </sarcasm>

So, I then apply this configuration, and watch how the new pod gets deployed. By the way, if you’re doing this, there are “ALOT” of situations in which the config is accepted but nothing happens.

$ minikube kubectl apply -- -f ~jpm/myawx.yml
awx.awx.ansible.com/awx configured

$ kubectl get pods
NAME                           READY   STATUS        RESTARTS   AGE
awx-555d75485d-nbzf5           4/4     Running       0          5s
awx-86b468c746-sbqdc           0/4     Terminating   0          17m
awx-operator-57bcb58f5-7crxs   1/1     Running       0          106m
awx-postgres-0                 1/1     Running       0          104m

And then I login to the pod itself and see my files:

$ kubectl exec -c awx-task awx-555d75485d-nbzf5 -i -t -- bash -o vi
bash-4.4$ cd /var/lib/awx/.ssh
bash-4.4$ ls -l
total 0
lrwxrwxrwx 1 root root 25 Mar 25 17:24 config -> ..data/config
lrwxrwxrwx 1 root root 17 Mar 25 17:24 id_ed25519 -> ..data/id_ed25519

bash-4.4$ ls -lL
-rw-r--r-- 1 root root 221 Mar 25 17:24 config
-rw-r--r-- 1 root root 444 Mar 25 17:24 id_ed25519

bash-4.4$ ssh -l ansible data.user01.example.com uname -a
Failed to add the host to the list of known hosts (/var/lib/awx/.ssh/known_hosts).
Password for ansible@data.USER01.example.com:
FreeBSD data.USER01.example.com 12.2-RELEASE-p1 FreeBSD 12.2-RELEASE-p1 GENERIC  amd64

And now I have definitely deserved a drink. Oh, and have you seen the new “look” of AWX?


This works well from the command line as demonstrated above, but from within AWX it doesn’t. The only hint I’ve so far found is in the Ansible Runner documentation:

Ansible Runner will automatically bind mount your local ssh agent UNIX-domain socket (SSH_AUTH_SOCK) into the container runtime. However, this does not work if files in your ~/.ssh/ directory happen to be symlinked to another directory that is also not mounted into the container runtime

The files are indeed symlinks. Does this mean game over?

Another update

I’ve learned that I can create actual directories and files to avoid the symlinks, but AWX doesn’t use the configuration I drop into /var/lib/awx/.ssh so I’m a bit at wit’s end:

  tower_task_extra_volume_mounts: |
     - name: "sshconfig"
       mountPath: "/var/lib/awx/.ssh/config"
       subPath: "config"
       readOnly: true
     - name: "sshkey"
       mountPath: "/var/lib/awx/.ssh/id_ed25519"
       subPath: "id_ed25519"
       readOnly: true
  tower_extra_volumes: |
     - name: "sshconfig"
          secretName: "awx-jp01"
          - key: conf
            path: "config"
     - name: "sshkey"
          secretName: "awx-jp01"
          - key: keyfile
            path: "id_ed25519"

The SSH configuration is neither taken from /var/lib/awx/.ssh nor from /root/.ssh, and I’m not able to inject a file into /etc/ssh/ssh_config.d/xxx.conf as Kubernetes turns that into a directory, and I cannot overwrite /etc/ssh/ssh_config because K8s complains the container won’t start as a directory is trying to overwrite a file (I assume this is related to this “file injection”), so I’m actually capitulating at this point.

Unless somebody has a proven solution, I give up. It’s been a long time since I’ve given up in the face of a bit of software…

Another another update

This isn’t giving me peace. I set up another set of machines and an AWX to find out as which user AWX is actually executing my play / tasks, and spying on it with a local command id, I see uid=1000(runner) which is not the uid=1000(awx) I was expecting, so this is all occurring on awx-ee (execution environment). I had checked that, but maybe used the wrong home directory? So, let’s try


The runner is launching ansible as uid=1000 with a home directory of /home/runner in a temporary directory, so why isn’t our SSH config being used?

It’s not using /runner/env/envvars (which it probably should be?) If I launch ansible on awx-ee I see it’s not picking up an ansible config file (neither in ~/.ansible.cfg nor in ~/ansible.cfg), but I can provoke it to by exporting ANSIBLE_CONFIG. This is better than a treasure hunt not.

ansible :: 25 Mar 2021 :: e-mail

Other recent entries