I understand the classic UNIX Magic poster by Gary Overacre was distributed at past USENIX conferences, and I’ve known of it for years but now in confinement, I’ve decided I want one to hang in my office.

Unix Magic

The poster features a white bearded wizard with UNIX related objects around him, for example a spool of thread, a boot, a fork, pipes, and a bunch of containers labeled troff, awk, diff, uucp, make, null, and there’s even a C container and cracked B container. And some oregano:

The oregano is reputedly referring to an incident in which one of the original folks involved with BSD was hassled for coming across the Canadian/U.S. border with a bag of what was assumed to be an illegal substance, and turned out to be oregano.

How to get a poster which was drawn in 1986? There are lots of bad photos of the poster floating around, but certainly nothing good enough to turn into a poster of any size.

Via Twitter, Jaume points to a 32MB PNG (3675 x 5475) which I download and convert to TIFF. I then ask around a bit and find a shop willing to print that onto a material which looks exactly like what I want.

image detail

The material is called Aluminium Dibond which, and I quote, makes for a “matte, reflection-free print using a state-of-the-art, 7-color printing method. White and brighter parts of the image have a subtle sheen. With the aluminium Dibond backing, this print combines high picture quality and perfect durability”. Quite a mouthful, but the result, which I got in two days, is astonishing.

the side

I was a bit skeptical and shy of spending EUR 80 on the picture, but I can say without reservation, that I’m very happy with what Whitewall has made for me, and thanks to Jaume for the file!

unix :: 09 Apr 2021 :: e-mail

It’s early mornings here, and I’ve not [yet] opened a bottle, and to my knowledge I’ve not caught the pest, but the words in the title are related to each other. How?

white dig towel

If you look closely at the dig syntax on that oh-so lovely towel ISC have in their merchandise shop (click on the towel) you’ll notice an optional %comment at the end of the command. I remembered that from way back but also remembered that I never actually bothered to find out what that even did.

Winfried asked and I woke up our friends at ISC to find out. Turns out the percent sign really does introduce a comment and has been in the code since BIND 8.

The syntax on the towel still is in the fabulous BIND 9 ARM but isn’t explained, and dig(1) man pages no longer even bother mentioning it.

From a current dig.c:

rc = argc;
rv = argv;
for (rc--, rv++; rc > 0; rc--, rv++) {
	debug("main parsing %s", rv[0]);
	if (strncmp(rv[0], "%", 1) == 0) {

(Let’s ignore the renaming of argc and argv (saving a few characters on each), and let’s also ignore using strncmp to check the first character of a variable, ok?)

So what does it do? Nothing. It is a comment and inhibits further parsing of the command line:

$ dig jpmens.net %rest-is-ignored +dnssec

The +dnssec flag is unused.

The assumption is that it could be used in batch files, maybe before the advent of shell scripts (which have their own comment sign), or (gasp!) in digrc files.

And seeing you made it this far, you might also be interested to learn why zone master files use a semicolon for comments.

dns :: 31 Mar 2021 :: e-mail

There’s no end to the fun that was made of me today for touching the K-word. Be that as it may, I have to do this if I want to continue giving AWX/Tower trainings, and in order to do that I need AWX to use an SSH jump host to get to nodes. The reasons for that lie hidden in here.

This post is going to be a quick and dirty collection of how I solved the particular issue I requested help on, and the last thing you want to do is to ask me for help on Kubernetes & co. It took me several hours to solve this problem.

What’s the problem? I need to deploy an SSH key and an SSH conf file into the containers (or are those pods?) the AWX task (awx-task) processes are running in.

In order to accomplish this, I finally was able to

  1. Create kubernetes secrets from files
  2. Deploy (I hope that’s the right term) AWX configured to use those secrets to create read-only volumes in minikube.

Creating the secrets is easy:

$ kubectl create secret generic awx-jp01 \
	--from-file=keyfile=$HOME/umleit \
secret/awx-jp01 created

$ kubectl describe secrets awx-jp01
Name:         awx-jp01
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

conf:     221 bytes
keyfile:  444 bytes

Then comes the operator (?) deployment (?) file. (You’re noticing I don’t know the proper terminology, and I’m not quite sure I really want to know it…)

apiVersion: awx.ansible.com/v1beta1
kind: AWX
  name: awx
  tower_ingress_type: Ingress
  tower_hostname: awx.example.com
  tower_admin_user: admin
  tower_admin_email: admin@example.com
  tower_admin_password_secret: changeme
  tower_task_extra_volume_mounts: |
     - name: "rootsshdir"
       mountPath: "/var/lib/awx/.ssh"
       readOnly: true
  tower_extra_volumes: |
     - name: "rootsshdir"
          secretName: "awx-jp01"
          - key: keyfile
            path: "id_ed25519"
          - key: conf
            path: "config"

The magic is in tower_extra_volumes which sources the secrets created earlier, and tower_task_extra_volume_mounts which creates the actual bind-mount from the secrets. I’ve not found a stich of documentation on this; either term produces two google hits; quite the record. </sarcasm>

So, I then apply this configuration, and watch how the new pod gets deployed. By the way, if you’re doing this, there are “ALOT” of situations in which the config is accepted but nothing happens.

$ minikube kubectl apply -- -f ~jpm/myawx.yml
awx.awx.ansible.com/awx configured

$ kubectl get pods
NAME                           READY   STATUS        RESTARTS   AGE
awx-555d75485d-nbzf5           4/4     Running       0          5s
awx-86b468c746-sbqdc           0/4     Terminating   0          17m
awx-operator-57bcb58f5-7crxs   1/1     Running       0          106m
awx-postgres-0                 1/1     Running       0          104m

And then I login to the pod itself and see my files:

$ kubectl exec -c awx-task awx-555d75485d-nbzf5 -i -t -- bash -o vi
bash-4.4$ cd /var/lib/awx/.ssh
bash-4.4$ ls -l
total 0
lrwxrwxrwx 1 root root 25 Mar 25 17:24 config -> ..data/config
lrwxrwxrwx 1 root root 17 Mar 25 17:24 id_ed25519 -> ..data/id_ed25519

bash-4.4$ ls -lL
-rw-r--r-- 1 root root 221 Mar 25 17:24 config
-rw-r--r-- 1 root root 444 Mar 25 17:24 id_ed25519

bash-4.4$ ssh -l ansible data.user01.example.com uname -a
Failed to add the host to the list of known hosts (/var/lib/awx/.ssh/known_hosts).
Password for ansible@data.USER01.example.com:
FreeBSD data.USER01.example.com 12.2-RELEASE-p1 FreeBSD 12.2-RELEASE-p1 GENERIC  amd64

And now I have definitely deserved a drink. Oh, and have you seen the new “look” of AWX?


This works well from the command line as demonstrated above, but from within AWX it doesn’t. The only hint I’ve so far found is in the Ansible Runner documentation:

Ansible Runner will automatically bind mount your local ssh agent UNIX-domain socket (SSH_AUTH_SOCK) into the container runtime. However, this does not work if files in your ~/.ssh/ directory happen to be symlinked to another directory that is also not mounted into the container runtime

The files are indeed symlinks. Does this mean game over?

Another update

I’ve learned that I can create actual directories and files to avoid the symlinks, but AWX doesn’t use the configuration I drop into /var/lib/awx/.ssh so I’m a bit at wit’s end:

  tower_task_extra_volume_mounts: |
     - name: "sshconfig"
       mountPath: "/var/lib/awx/.ssh/config"
       subPath: "config"
       readOnly: true
     - name: "sshkey"
       mountPath: "/var/lib/awx/.ssh/id_ed25519"
       subPath: "id_ed25519"
       readOnly: true
  tower_extra_volumes: |
     - name: "sshconfig"
          secretName: "awx-jp01"
          - key: conf
            path: "config"
     - name: "sshkey"
          secretName: "awx-jp01"
          - key: keyfile
            path: "id_ed25519"

The SSH configuration is neither taken from /var/lib/awx/.ssh nor from /root/.ssh, and I’m not able to inject a file into /etc/ssh/ssh_config.d/xxx.conf as Kubernetes turns that into a directory, and I cannot overwrite /etc/ssh/ssh_config because K8s complains the container won’t start as a directory is trying to overwrite a file (I assume this is related to this “file injection”), so I’m actually capitulating at this point.

Unless somebody has a proven solution, I give up. It’s been a long time since I’ve given up in the face of a bit of software…

Another another update

This isn’t giving me peace. I set up another set of machines and an AWX to find out as which user AWX is actually executing my play / tasks, and spying on it with a local command id, I see uid=1000(runner) which is not the uid=1000(awx) I was expecting, so this is all occurring on awx-ee (execution environment). I had checked that, but maybe used the wrong home directory? So, let’s try


The runner is launching ansible as uid=1000 with a home directory of /home/runner in a temporary directory, so why isn’t our SSH config being used?

It’s not using /runner/env/envvars (which it probably should be?) If I launch ansible on awx-ee I see it’s not picking up an ansible config file (neither in ~/.ansible.cfg nor in ~/ansible.cfg), but I can provoke it to by exporting ANSIBLE_CONFIG. This is better than a treasure hunt not.

ansible :: 25 Mar 2021 :: e-mail

Paternoster, named after the lift, enables me to quite easily create a “program” which uses the Ansible API to actually invoke and run a playbook, mostly hiding this fact from the user. I’ve known of Paternoster since at least March 2018 when I PRed one of my infamous one-letter fixes and have since wanted to blog about it. Here goes.

$ pip install paternoster ansible

The executable playbook I create has a hash bang which points to Paternoster. It can be specified in a sudoers file, say, to give it specific privileges. Other than that, the playbook in my “program” contains two plays: the first defines Paternoster parameters which become program options, the second and subsequent are the typical Ansible plays which Paternoster invokes with variables.

The host paternoster doesn’t actually exist in the inventory – it’s a magic word which triggers Paternoster to set up its parameters.

#!/usr/bin/env paternoster

- hosts: paternoster
      description: Create a user
        - name: uname
          help: Username to create
          prompt: true
          type: paternoster.types.restricted_str
              regex: "^[A-Z]+[0-9]$"
              minlen: 2
              maxlen: 5
- hosts: localhost
  connection: local
    - name: Create user
          msg: "Will create user "

Parameters are represented by a dictionary which I found quite self-explanatory. They are passed to Ansible prefixed with param_, so my uname parameter becomes the variable param_uname in Ansible. Standard Python types can be used for describing variables (e.g. int), but Paternoster provides a few others in order to enforce, say, string validation.

For example, paternoster.types.restricted_str can be validated with a regex or have certain character classes, a minimum and maximum lengths. The domain type is checked for a DNS domain, the url type for a URL, etc.

Parameters can also be prompted for, and as with Ansible’s vars_prompt, if the parameter’s value is passed on the CLI as an option Paternoster doesn’t prompt for it but does validate the value according to my specification.

Assuming the above script is called mkuser, the following could occur:

$ mkuser -h
usage: mkuser [-h] [--uname UNAME] [-v]

Create a user

optional arguments:
  -h, --help     show this help message and exit
  --uname UNAME  Username to create
  -v, --verbose  run with a lot of debugging output

$ mkuser 
Uname: jane8
usage: mkuser [-h] [--uname UNAME] [-v]
mkuser: error: argument --uname: invalid string value: 'jane8'

$ mkuser --uname JANE8
Will create user JANE8

The first invocation shows help with the descriptions I specify in the first play. The second shows how I’m prompted for a variable and how validation (regex, lengths) fails, and the third then finally works because it’s upper-case only, the right length and ends in a digit.

Paternoster’s check_user verifies that the user running the script is the one specified, which allows me to have programs only a particular user can execute, and if I set become_user, Paternoster uses sudo(1) to execute the playbook as the specified user. If set, success_msg is printed on successful exit as a confirmation to the user, untemplated.

What I like about Paternoster is how it basically “hides” that an Ansible playbook is being invoked, and that it wraps that into a utility which I can use on the command line as any other, options and all. And being Ansible means I can run it here have something actually done on remote systems. Users who wish to see what’s happening, can use -v on the program to show the typical playbook output.

Paternoster was created by Michael “luto” Lutonsky to supplant a bunch of shell scripts at Uberspace, the hoster which also hosts this site. (I could rave about them but the simple fact that I’ve never complained about them speaks volumes!) Uberspace sponsored the development of the open code, and one day I will rave about them.

Further reading

ansible :: 19 Mar 2021 :: e-mail

If you’ve used Ansible, you’re likely very familiar with this default output when a playbook runs a few tasks: the green indicates “ok” and unchanged, and the yellow indicates that a task has reported a change on the remote node.

Basic Ansible output

When we invoke ansible-playbook with a -v or two, the output becomes more verbose, and simultaneously quite difficult to read:

Debugging output

Serge tweeted something last night which I first looked at on a small mobile screen and didn’t really “see” as being interesting, but when I gave it another look this morning it made a lot of sense, and I want to demonstrate what that does.

Serge's tweet

I typically use a shell-script “wrapper” to invoke Ansible playbooks, and I keep that ./a shell script alongside the playbook. As “lazy” as I am, I typically alias a to ./a to further simplify my life. (And while we’re at it, I am not a friend of hash bangs and YAML files made executable because YAML isn’t executable.)

From today, as per what Serge suggests, my wrapper script looks like this:


echo -n "$@" | grep -q -- "-v" && export ANSIBLE_STDOUT_CALLBACK=yaml

ansible-playbook test.yml "$@"

What that does is to detect I’ve invoked my wrapper with one or more -v (the grep is quiet) and then sets the stdout yaml callback , which produces much more readable output:

Debugging output with YAML

Thank you, Serge.

ansible :: 12 Mar 2021 :: e-mail

Other recent entries