Wherever possible I use vi or a newer incantation of it called vim, but I try to stick to low common denominators so that my editing is “portable” across platforms.

I learned vi the hard way, on a terminal for which (thankfully) no cursors or function keys were defined in termcap/terminfo. This forced me to spend a quarter of an hour practicing and learning that l or <space> moves the cursor to the right and j moves it down. (And that J is something completely different.)

The worst part of my learning experience was the bell: I couldn’t turn it off on that terminal, and it drove me crazy the first day. I quickly understood command mode and text enter mode in order to avoid the beep, and I’ve not looked back.

Why “thankfully” no cursor block? The . command works properly, movement (65l or 7W) becomes easy. Finding a $ in the line becomes f$ or reverse F$, and performing a bunch of changes (but not global search/replace) becomes /search, then a change, say, cwblablaESC, then n next and n next, and finally . to repeat whichever change I did last (yes, also deletes, or appends, or inserts: any change).

I think for close on twenty years I had a .exrc which consisted of two lines:

:set ai
:ab KK /*x */ESCFxs

A few years ago I went all out when I started doing Python, and my .exrc became a .vimrc which still contains those two lines plus

filetype plugin on
" au BufRead,BufNewFile *.py set expandtab
" au BufRead,BufNewFile *.py set tabstop=4
" au BufRead,BufNewFile *.py set backspace=indent,eol,start
" autocmd BufRead,BufNewFile *.py set smartindent cinwords=if,elif,else,for,while,try,except,finally,def,class

There have been times when I’ve been a bit jealous of people who master Emacs: the fact that they can, from their editor, read mail, post news (Usenet news; you probably don’t know what that is), and do all sorts of crazy things tempted me to learn it, and I have done a bit of that: I tend to write up notes in Org mode, but I’ll confess that muscle-memory for vi commands is an order of magnitude more developed than it is for emacs.

I’m writing this to drive home that whatever editor a person uses is fine, as long as that person utilises the editor properly, and if you’re good with Microsoft’s VS-Code, might be in awe at what you can do with it. (You need a graphical terminal; I don’t.)

I give lots of trainings, and I almost literally cry when I see people stumbling in vi, hitting ESCape three times after adding a bit of text, adding a character by inserting, then ESCaping, then moving back a key deleting the superfluous character – it’s horrid. It’s almost as bad as watching people go up 17 times in their shell history to find .. ls. (There have been cases of me honestly yelling at people for doing that, but I trust they’ve forgiven me meanwhile.)

Use a text-editing utility that you can handle and like to use. Unix has lots of editors. I’m not going to say ed(1), but there’s Nano and Pico and mg, and loads of them. Use one which is convenient and simple to wield. Become proficient in editing. Please.

If you don’t have the time or the patience to learn vi or emacs, don’t; you’ll find I’ll respect you more if you stick to notepad.exe or whichever. :-)

Way back then, I also learned how to exit vi, and that was not using :wq! because that broke how make works and incremental backups again picked up files which didn’t have their content modified. If you use vi, learn about ZZ or :x and save on a key press… It’s all about being efficient when editing text, and :wq still does things you need to be aware of.

I wrote a 700-page book using vi. True story.

utility :: 15 Jun 2019 :: e-mail

I attended BSDCan 2019 in Ottawa, Canada this year. I’m sadly no longer particularly familiar with the BSD world because most of my work for customers revolves around Linux, but there was an exception recently, and I hope that won’t be the last.

The event began with the disappointing news that the full-day OpenBSD porting workshop was cancelled due to force majeure.

I had signed up for Peter Hessler’s BGP for non-experts tutorial on the second day because of the explicit “non-experts” in the title. I’ve used ping a couple of times, so I thought I’d be fine. :-) Not only could I follow, but I actually enjoyed it and learned a lot. One technical detail about the infrastructure used in the presentation impressed me: Peter’s OpenBSD laptop runs around 60 VMM machines onto which we all logged on for lab experiments; Peter has a slide deck describing parts of the setup.

The conference proper was on the third and fourth days, and I attended a number of talks.

  • In building an accessible OpenBSD laptop Stefan Sperling speaks of how he proposes OpenBSD for his friend Maurice, an actor who survived a brain hemorrhage. In between recounting the Maurice’s impediments, Stefan details some of the software and hardware issues he experienced on building the laptop and some solutions. At the end of the talk, Stefan set up a live video conference with Maurice which was cute. All in all I had the impression that Stefan went to a huge amount of work and stumbled over lots of software problems for this particular task.

  • unwind(8) by Florian Obser interested me because it proposes a solution which will offer road warriors and most workstations DNSSEC resolution directly on the device. I haven’t had the opportunity of using unwind productively yet, but I get the impression it does a bit more than DNSSEC Trigger does in attempting to provide seamless DNSSEC resolution to the workstation.

  • The talk Diving and OpenBSD given by Kristaps Dzonsons was brilliant. He’s a good entertainer, and he explained how he uses specific programs on OpenBSD to process pictures and other data from multiple underwater computers and cameras. A very entertaining presentation with sensational underwater pictures projected onto the big screen.

  • In OpenBSD network-booted Workstations, Jan Klemkow explained how and with which methods they have OpenBSD boot on up to 250 workstations. I enjoyed the topic because I used to do a quite a bit of remote booting back in the days when we did that. I actually thought it’s a foregone technology, but it appears it isn’t; good.

I took a break and skipped the last slot of the day, preferring some peace and quiet and the chance to let the day’s impressions settle. That evening I went out for a rather mediocre Thai meal with half a dozen people, but we had fun making fun of the restaurant, and after the last beer for me at the Royal Oak, I took a bow.

Day two, for me, consisted of these talks:

  • Bob Beck did a loud and clear (and I mean this very positively: being hard of hearing, loud and intelligible audio is important to me, and BOB IS VERY LOUD :-) presentation on Unveil in OpenBSD in which he explained the intricacies of the unveil(2) system call which basically “hides” portions of the file system. I think I now know everything I ever wanted to know about that; very well done. unveil(2) is a system call other Unixes and Linux would profit from (as with pledge(2) which is likely much more complicated to implement). Both of these make me as a programmer think about what I believe my program should be permitted to do / see and have the program crash (in the case of pledge) or file operations fail (for unveil) if the program attempts to exceed the self-imposed limitations.

  • syspatch(8) is used to fetch, apply, and revert binary OpenBSD patches, and Antoine Jacoutot gave a very good and comprehensive overview of what the utility does, how it works, and how patches are actually created on OpenBSD. Of particular interest to me was his “war story” on how he came about to make this happen. (Real life stories are what I most enjoy hearing about at conferences.)

  • I listened to Philipp Buehler speak about adding OpenBSD VMM support to “packer” which is an open source tool for creating identical machine images for multiple platforms from a single source. I’m not familiar enough with packer to fully grasp what he’s doing but it sounded sound.

  • I had a very entertaining breakfast with Ollivier (who shot the conference photos) and Aaron Poffenberger, and one of the talks I’d marked as “must see” was Aaron’s Road Warrior Disaster Recovery, and I was not disappointed. Aaron covered various strategies available to us to ensure our laptops are secure, synchronized, backed up and ready to recover. Lots of food for thought for me. I gave it a rating of “my vote for Best Talk at BSDCan 2019”.

As to my own talk, I had a hell of a time with my demo: the university of Ottawa prohibits WiFi routers, so the demonstration I wanted to give with equipment brought across the Atlantic was in danger. With the help of a few people who lent me bits and pieces (thank you all, again, for switch and cables) I was able to convert most of my demo hardware to cabled access. Simply the Wemos-D1 had to remain on WiFi so I risked it, plugging it in a mere minute before I needed it and crossing fingers it wouldn’t be remotely de-authed. Unfortunately I couldn’t get the Raspi with the #blinkenlights to work; very sad.

official feedback

Even so, i got good feedback for my own talk, MQTT for system administrators (and for the IoT) which came across with quite a large audience and many interested and interesting questions at the end.

The final and closing talk at BSDCan2019 was moderated by Dan Langille, and it was hugely amusing. I’ll say just one thing: an auction for a good cause. :-) This was followed by the closing social event which gave me a chance to chat to a number of interesting people. It also gave me the chance to receive quite a bit of praise for my MQTT talk.

I’d like to thank the BSDCan committee for inviting me to speak and attend the conference; you may do that again. :-)

bsd and conference :: 22 May 2019 :: e-mail

My LOADays experience started with a discussion-filled evening during and after the speakers’ dinner on Friday, with the “after” portion taking place in the hotel bar. I think I stumbled into bed after midnight.

Up early the Saturday morning to drive Pieter and Andrea to the venue in time for the lovely breakfast they serve speakers there. (If you need a reason to speak at LOADays: the thin, fluffy, tasty omelettes are hard to beat, pun not intended.)

Robert then accompanied me to the room reserved for a bit of a special event: when we last met, I promised him I’d do a special introductory DNS talk for his apprentices, a dozen of which attended bravely sitting up in the front row. I hope I didn’t speak too fast, and that whatever I said made half sense: it’s difficult to distill all I have to say about the topic into a single hour, particularly to young people possibly not accustomed to a lot of technical info. Even so, I had the impression that a few of the attendees were really interested.

and there were books

As a special feature, I schlepped two cases of my book along and gave each of the students a complimentary copy. If nothing else, I told them, they could look at the nice diagrams in it. :-) I was touched upon being given a cellophane-wrapped basket containing a selection of Belgian beer. Those of you who know me know that I’m not terribly partial to it, but I will make a point of explicitly tasting each and every one of them. Here they are, all neatly lined up before I stow them in the fridge.

belgian beer

After the DNS intro (which we’d opened to the “general public”) and after a bit of a break I took, I listened to the tail end of Bert Van Vreckem’s talk on Leveling up your shell scripting skills which had a few interesting tidbits along with some rather impossible ones in my opinion: the example of idempotence on creating a Unix user was “cute” but it doesn’t work the way he showed us, because he completely ignored changing gecos field, shell, or home directory (that requires config management).

Next was Pieter Lexis whom I consider to be a particularly fun speaker; his Running containers and Operating System images with systemd-nspawn sparked quite a bit of joy with me except for his use of “dark mode” on screenshots. (That happened a lot this year; I think nobody listens to me: I yell that out clearly every year several times :-) I’ll study the screenshots quietly directly from Pieter’s slide deck. I’ve always thought it’s a shame systemd-nspawn isn’t spoken about more, and Pieter drove that point home again: it’s really worth looking into, particularly as it’s really well integrated into systemd, whether you like it or don’t.

Pieter Hollants (yes, another Pieter: and there was a Peter too, but he didn’t speak much :-), gave an overview of how he didn’t become an electronics engineer which was fun, and he brought hardware along. The talk saddened me a little bit because it drove home how playing with toys like those is becoming more and more difficult for me with eye-sight failing and hands shaking. His slides are here.

There was a bit of a “Zabbix Track”, if I may call it that: Patrik Uytterhoeven talked about Low-Level Zabbix Discovery for JMX, and I also had a Zabbix talk on loadable C modules and Low-Level Discovery. I had the impression attendees weren’t really thrilled to hear about all of that. I did.

Andrea Tosatto, a very good presenter and colleage of Pieter and Peter, gave an excellent talk on scaling Ansible. He’s fun, he’s fast, but to be honest, he could have simply used my slide deck and saved a lot of time. I’m just kidding of course. His presentation has numbers and it’s well worth your time

Before the traditional open-air pizza (omg I thought it was soo cold outside!), there was a “PowerPoint karaoke” which was hilarious, but it was also too long for me. Shoutout from me to Jeroen Baten and Pieter Lexis for being fantastic spontaneous speakers!

I had to leave that evening due to “management” having made alternative engagements and so missed out on the Sunday breakfast and talks. As such, I’m already looking forward to LOADays 2020. :-)

conference :: 05 May 2019 :: e-mail

I like having tmux or screen start automatically when I connect to a machine, and if possible I like having the previous detached session restored. I think it was this post which inspired me at the time, at least it sounds familiar.

Be that as it may, there are a couple of things which I’ve modified:

  1. on machines which periodically clear out /tmp I want tmux’ Unix domain socket directory moved to a more permanent location, so I set TMUX_TMPDIR accordingly.
  2. When using rsync -e ssh to a machine, a pty is not requested, and so I circumvent the tmux magic.

Here’s the tail of my ~/.profile:

if [[ -n "$SSH_TTY" ]]; then
	export TMUX_TMPDIR=$HOME/tmp
        sname="ssh-$(hostname | cut -d. -f1)"
        if [[ -z "$TMUX" ]] && [ "$SSH_CONNECTION" != "" ]; then
            exec tmux new-session -A -s $sname

So when I login via SSH tmux is started with a new session or, if the session exists, it attaches to that existing session. The login shell execs tmux, so I’m logged out automatically when I detach from tmux.

I’d have liked to exec the tmux but I can’t do it with the ||. Thanks to Daniel Néri and Ivan Tomica for the idea on using the -A flag.

Thanks to Raf Czlonka, we get a POSIX-compliant version which ought to work for most shells, and it does away with mixing POSIX and non-POSIX test, uses hostname -s which should work on most systems, and simplifies the if:

test -n "$SSH_TTY" && {
        export TMUX_TMPDIR
        test -d $TMUX_TMPDIR || mkdir -p $TMUX_TMPDIR
	test -z "$TMUX" -a -n "$SSH_CONNECTION" &&
                exec tmux new-session -A -s "ssh-$(hostname -s)"
tmux and SSH :: 28 Apr 2019 :: e-mail

Let’s assume you want to access a monitoring host from an Ansible play which is launched by Ansible Tower/AWX, and let’s further assume that you require credentials with which to do so. The trivial demonstration play will be this one:

- name: Zshow me your variables
  hosts: all
  - name: Which is the zurl?
    debug: msg="it is {{ zurl }}"

  - name: And the password?
    debug: msg="the secret is {{ zpass }}"

You could well use Ansible’s extra_vars with which to do so, something along the lines of

zurl: http://zabbix.example.net/zabbix
zuser: zabbix
zpass: password

This means, though, that whoever can access the Tower/AWX job template description will see the values (in particular the password) in clear text. Furthermore, logs, of job template runs will also show them, and you really don’t want that to happen, do you? You don’t!

extra vars are visible at any time in job logs

Ansible Tower/AWX have somthing I particularly enjoy (and I spend some time explaining it in the trainings I give): it’s called “custom credential types”.

I can create a new “type” of credential, which AWX/Tower presents to the user as though it were built-in. Furthermore, AWX/Tower secure the data therein by encrypting fields marked secret into the backend database, just like they do for other types of credentials such as machine or AWS credentials.

I first create a new type by defining the fields it will have and how these will be passed to my playbook. The former is called input configuration and the latter injector configuration.


Both these configurations are specified as YAML or as JSON, and ought to be pretty self-explanatory; note the secret attribute on my zpass field which ensures its value cannot be seen on data entry or later, when the credential is opened.

- id: zurl
  type: string
  label: Zurl
- id: zuser
  type: string
  label: Zusername
- id: zpass
  type: string
  secret: true
  label: Zpassword
- zpass
- zuser
- zurl

There’s also a multiple choice field which can be quite practical, though it cannot be fed from a URL, unfortunately. (But thanks to the AWX API, we can pump data into that field from outside; I digress.)

The injector configuration defines how these variables should be passed to our Play. We can use extra vars as I’ve shown, but AWX/Tower can also create INI files for us and drop those in a temporary location on the controller. I find extra vars easier to handle (and to explain) and they should suffice for most use-cases.

As soon as the credential type is defined, I can go ahead and create a credential of that type:

I create a credential of our new type

Now that I have a credential with the data we want in it, I can assign that to our job template, and I do that just like assigning any other credential type:

assign credential to job template

When I now launch my job template (note that there are no extra_vars in the template definition), the play runs as I want it to:

the play runs

Tower/AWX takes particular care of these values: they don’t appear in logs (unless I’m careless enough to provoke that as in this example where I output the values), and they cannot be overwritten, so if I erroneously create an extra var called zurl, say, AWX will not clobber the one from the credential type. (I would try to apply naming conventions so that that can’t happen by mistake, though.)

I used Zabbix in this example for two reasons: first of all I use it myself (did I tell you I intend to give a talk about Zabbix loadable C modules at the upcoming LOADays conference?), and then because I recently stumbled across a similar example on a Red Hat site which upset me because it showed credentials in extra variables; that should be a no-no.

ansible, credentials, and awx :: 16 Apr 2019 :: e-mail

Other recent entries