I frequently present during conferences or when giving trainings, and during the course of several years I’ve been very satisfied with a laser pointer which fits in my hand comfortably and can page-forward and page-backwards. The only issue I had with the device was a press on the laser in Terminal produced ugly escape sequences, but I killed those with Karabiner-Elements years ago.

What more could I want?

Well, I’m being confronted more and more frequently with TV panels instead of projectors, and the laser pointer doesn’t reflect off the panels.

I think it was Gerrit who, last year, showed me an equivalent solution, but it was then Bernd who, during a small demonstration, convinced me to opt for a Logitech Spotlight, which I did. I was afraid the software would suck, but it’s ok other than consuming quite a bit of CPU (5%) even when the Spotlight isn’t plugged in.


The device is pleasant to hold and the buttons are fine. I’ve configured it for a “laser highlight” only because using multiple different software pointers was confusing. Viewers also like the new pointer, and in particular, somebody who has a red/green deficiency convinced me to change the pointer’s color to a blueish, and he was then very happy!

blue laser

There’s one thing which is naturally unnatural: as a speaker, when I’m about to point at something with a real laser, the hand goes to the right direction before pressing the switch; this means the light comes on pretty much exactly where I want it. Using the Spotlight, this doesn’t work because only the software knows where the marker was last positioned, so I have to first press the light on and then drag the marker to its intended destination, wobbling and all. It takes a bit of getting used to, but I’ll manage.

I decided to solve the dilemma easily: I pack both, and if I’m given a projector with a laser-compatible screen I’ll use the laser and for a TV panel I’ll use the Spotlight.

presentation :: 04 Aug 2019 :: e-mail

I tell people that my first contribution to the Ansible project was killing off the cows, but while researching for this post, I see memory fails me: the first written record of me contributing to Ansible is dated a bit earlier, on September 4th, 2012, but that doesn’t matter. By the way, it was quite a bit later that the cow thing became configurable. Be that as it may, it’s been almost seven years – a long time and a good time.

I’ve contributed a number of modules to Ansible, quite a few lookup plugins, documentation patches, and the piece de resistance was the documentation (a.k.a. ansible-doc) system which is still used today. I recall sitting on the floor in a conference center in Boston getting ready to push what I lovingly called a “jumbo patch” which would target each and every module: I had Michael’s ok that he wouldn’t commit anything for 24h so that my patch would apply cleanly. It was very exciting for me, having to wield a new utility (git) and hoping I wouldn’t mess anything up. A lot of adrenalin flowed. Good times.

Ansible’s grown enormously since then, and I’ve not always been able to pay attention to it as much as I’d have liked to, but I’ve used it and have also had the pleasure of training a large number of people in using Ansible. In the course of the last two or three years, I’ve met two hundred people interested in how Ansible works. That is a lot.

During that time I’ve occasionally commented on a pull request, responded to an issue, and I’ve even submitted at least one new lookup plugin for consideration: it’s the lmdb lookup plugin.

I’m telling you this story to set a bit of background for what follows and to demonstrate that Ansible is a project I’ve been very fond of.

If you look at that last link, you’ll see I submitted it two years ago, in May 2017. Since then, I’ve had to rebase the plugin once because of changes which made it unmergeable. Judging by labels added (by a human?) the plugin affects Ansible version 2.4 (current is 2.8.1), and it needs all sorts of things like a maintainer (huh?), a revision, tests, and whatnot. Does that mean I should leave it as is? Does that mean I should just close the PR and forget about it? I don’t know. I do know one thing: I will not rebase it again, and the powers that be must just as well close the PR as I’ve completely lost interest in it.


Ansible is a hugely popular project; it’s approaching 38K stars at the time of this writing, and it has 16K forks. I don’t know how many people Red Hat has working for Ansible, but I guess there to be several committers, and I know some of them by name and a couple personally.

In my opinion they’re being inundated.

At this time Ansible has 3900 open issues, and 1900 open pull requests. The oldest open PR is dated March 2015, and the oldest open issue is two months older. Never ever will it be possible to process all that, if only because the code has changed so much meanwhile that most of the PRs can likely no longer be applied cleanly, and I would think a lot of the issues are no longer even valid. What I also expect is that many (possibly first-time) contributors have meanwhile lost interest in their contribution just as I’ve done for the lmdb lookup plugin.

I was at a customer site recently a fortnight ago, and we had a devil of a time because it appeared that the archive module was omitting files from an archive. Hard to believe, I know, but an empty file gintonic in a directory-to-be archived, never appeared on the target hosts. What?!

After a lot of testing, and upon beginning the tedious process of submitting a bug report, I discovered this bug was already reported in January 2018. 1.5 years ago. This module error is not just some cosmetic thing of a feature request. It is a bug which causes data loss, and somebody added a waiting_on_contributor label to it, and I am starting to wonder how long the wait is going to be, particularly since a kind newcomer provided a fix in February of this year. (It also saddens me that this contributor may have been frightened off by being requested to provide integration tests for the fix.) To be fair, the archive module is a module maintained by the community which its documentation clearly states; not being a core module, the Ansible project proper is not responsible for it. So Ansible includes batteries but some batteries are not be as powerful as others?

This is but one example which I use as it’s the one which happened most recently to me.

In the course of the past year I’ve spoken to a small handful of Red Hat people about this. Unofficially I see a lot of nodding, and I see some organization which has taken place in the project (repositories, communities, etc.), but I don’t see PRs and issues diminishing in great numbers which could, of course, be due to an ever increasing number of submissions.

I proposed a dramatic and drastic solution: close the 4000 issues, close the 1900 PRs, and start over. Explain why this is being done. (It’ll be less painful to have a week of bad press than permanent bad press.) Have people begin to raise issues and PRs again. Brutally close whatever makes no sense. Merge the PRs quickly after a group of peers have OK’d them. (e.g. 3 peers apply an OK, the PR gets merged, somewhat how the OpenBSD project does it, I think). If an issue remains unchanged for, say, 3 months, or a PR remains without activity for that time, it’s marked as stale and will be closed after a short grace period. Ansible is an Open Source project backed by a strong company. I for one would much prefer to be told “no, we don’t want that” than to be left hanging and guessing.

Ansible users are telling me their trust in Ansible is diminishing: every release brings breakage (e.g. for Solaris, OpenBSD, a lot of non-Linux). PRs are not being merged in a timely fashion, and issues are not being solved. (Here’s an example of an issue addressed to me, and I have clearly ignored it.)

This makes me very sad.

Ansible Communities was an attempt by Dag Wieers to give people a trusted process and some privileges to work independently on sets of.modules, but it’s not got a lot of traction, unfortunately.

The new Ansible Collections, proposed in Ansible 2.8, could alleviate the situation with modules by moving the modules to Galaxy and to people’s repositories. This would mean that Ansible in future might just be a core with a minimal set of modules. My worry is that the number of disparate versions of a particular module will explode: contributors will go away, there’ll be no central module repository and that can mean dozens of versions of a particular module, each will different features and/or patches. For example, searching Galaxy for apache server filtered by Role produces 869 results, not all of which really are an Apache server, and there are 10 roles to install an Mosquitto broker; that’s a lot of choice… And many users I speak to in Europe are quite uninterested in Galaxy; are you really willing to pull configuration code you didn’t write to blindly deploy onto your servers? I smell lots of cake ;-)

On the plus side, moving most of the modules out of Ansible will make core developers happier, as they don’t feel responsible for the dozens of contributed modules anyway. All the more reason to clear out the large number of issues and PRs.

Let’s see what happens.

Update: Ansible responds to my blog post: Thoughts on restructuring the Ansible project.

ansible :: 21 Jun 2019 :: e-mail

Wherever possible I use vi or a newer incantation of it called vim, but I try to stick to low common denominators so that my editing is “portable” across platforms.

I learned vi the hard way, on a terminal for which (thankfully) no cursors or function keys were defined in termcap/terminfo. This forced me to spend a quarter of an hour practicing and learning that l or <space> moves the cursor to the right and j moves it down. (And that J is something completely different.)

The worst part of my learning experience was the bell: I couldn’t turn it off on that terminal, and it drove me crazy the first day. I quickly understood command mode and text enter mode in order to avoid the beep, and I’ve not looked back.

Why “thankfully” no cursor block? The . command works properly, movement (65l or 7W) becomes easy. Finding a $ in the line becomes f$ or reverse F$, and performing a bunch of changes (but not global search/replace) becomes /search, then a change, say, cwblablaESC, then n next and n next, and finally . to repeat whichever change I did last (yes, also deletes, or appends, or inserts: any change).

I think for close on twenty years I had a .exrc which consisted of two lines:

:set ai
:ab KK /*x */ESCFxs

A few years ago I went all out when I started doing Python, and my .exrc became a .vimrc which still contains those two lines plus

filetype plugin on
" au BufRead,BufNewFile *.py set expandtab
" au BufRead,BufNewFile *.py set tabstop=4
" au BufRead,BufNewFile *.py set backspace=indent,eol,start
" autocmd BufRead,BufNewFile *.py set smartindent cinwords=if,elif,else,for,while,try,except,finally,def,class

There have been times when I’ve been a bit jealous of people who master Emacs: the fact that they can, from their editor, read mail, post news (Usenet news; you probably don’t know what that is), and do all sorts of crazy things tempted me to learn it, and I have done a bit of that: I tend to write up notes in Org mode, but I’ll confess that muscle-memory for vi commands is an order of magnitude more developed than it is for emacs.

I’m writing this to drive home that whatever editor a person uses is fine, as long as that person utilises the editor properly, and if you’re good with Microsoft’s VS-Code, I might be in awe at what you can do with it. (Although you need a graphical terminal; I don’t.)

I give lots of trainings, and I almost literally cry when I see people stumbling in vi, hitting ESCape three times after adding a bit of text, adding a character by inserting, then ESCaping, then moving back a key deleting the superfluous character – it’s horrid. It’s almost as bad as watching people go up 17 times in their shell history to find .. ls. (There have been cases of me honestly yelling at people for doing that, but I fear they’ve forgiven me meanwhile.)

Use a text-editing utility that you can handle and like to use. Unix has lots of editors. I’m not going to say ed(1), but there’s Nano and Pico and mg, and loads of them. Use one which is convenient and simple to wield. Become proficient in editing. Please.

If you don’t have the time or the patience to learn vi or emacs, don’t; you’ll find I’ll respect you more if you stick to notepad.exe, nano, or whichever. :-)

Way back then, I also learned how to exit vi, and that was not using :wq! because that broke how make works and incremental backups again picked up files which didn’t have their content modified. If you use vi, learn about ZZ or :x and save on a key press… It’s all about being efficient when editing text, and :wq still does things you need to be aware of.

I wrote a 700-page book using vi. True story.

utility :: 15 Jun 2019 :: e-mail

I attended BSDCan 2019 in Ottawa, Canada this year. I’m sadly no longer particularly familiar with the BSD world because most of my work for customers revolves around Linux, but there was an exception recently, and I hope that won’t be the last.

The event began with the disappointing news that the full-day OpenBSD porting workshop was cancelled due to force majeure.

I had signed up for Peter Hessler’s BGP for non-experts tutorial on the second day because of the explicit “non-experts” in the title. I’ve used ping a couple of times, so I thought I’d be fine. :-) Not only could I follow, but I actually enjoyed it and learned a lot. One technical detail about the infrastructure used in the presentation impressed me: Peter’s OpenBSD laptop runs around 60 VMM machines onto which we all logged on for lab experiments; Peter has a slide deck describing parts of the setup.

The conference proper was on the third and fourth days, and I attended a number of talks.

  • In building an accessible OpenBSD laptop Stefan Sperling speaks of how he proposes OpenBSD for his friend Maurice, an actor who survived a brain hemorrhage. In between recounting the Maurice’s impediments, Stefan details some of the software and hardware issues he experienced on building the laptop and some solutions. At the end of the talk, Stefan set up a live video conference with Maurice which was cute. All in all I had the impression that Stefan went to a huge amount of work and stumbled over lots of software problems for this particular task.

  • unwind(8) by Florian Obser interested me because it proposes a solution which will offer road warriors and most workstations DNSSEC resolution directly on the device. I haven’t had the opportunity of using unwind productively yet, but I get the impression it does a bit more than DNSSEC Trigger does in attempting to provide seamless DNSSEC resolution to the workstation.

  • The talk Diving and OpenBSD given by Kristaps Dzonsons was brilliant. He’s a good entertainer, and he explained how he uses specific programs on OpenBSD to process pictures and other data from multiple underwater computers and cameras. A very entertaining presentation with sensational underwater pictures projected onto the big screen.

  • In OpenBSD network-booted Workstations, Jan Klemkow explained how and with which methods they have OpenBSD boot on up to 250 workstations. I enjoyed the topic because I used to do a quite a bit of remote booting back in the days when we did that. I actually thought it’s a foregone technology, but it appears it isn’t; good.

I took a break and skipped the last slot of the day, preferring some peace and quiet and the chance to let the day’s impressions settle. That evening I went out for a rather mediocre Thai meal with half a dozen people, but we had fun making fun of the restaurant, and after the last beer for me at the Royal Oak, I took a bow.

Day two, for me, consisted of these talks:

  • Bob Beck did a loud and clear (and I mean this very positively: being hard of hearing, loud and intelligible audio is important to me, and BOB IS VERY LOUD :-) presentation on Unveil in OpenBSD in which he explained the intricacies of the unveil(2) system call which basically “hides” portions of the file system. I think I now know everything I ever wanted to know about that; very well done. unveil(2) is a system call other Unixes and Linux would profit from (as with pledge(2) which is likely much more complicated to implement). Both of these make me as a programmer think about what I believe my program should be permitted to do / see and have the program crash (in the case of pledge) or file operations fail (for unveil) if the program attempts to exceed the self-imposed limitations.

  • syspatch(8) is used to fetch, apply, and revert binary OpenBSD patches, and Antoine Jacoutot gave a very good and comprehensive overview of what the utility does, how it works, and how patches are actually created on OpenBSD. Of particular interest to me was his “war story” on how he came about to make this happen. (Real life stories are what I most enjoy hearing about at conferences.)

  • I listened to Philipp Buehler speak about adding OpenBSD VMM support to “packer” which is an open source tool for creating identical machine images for multiple platforms from a single source. I’m not familiar enough with packer to fully grasp what he’s doing but it sounded sound.

  • I had a very entertaining breakfast with Ollivier (who shot the conference photos) and Aaron Poffenberger, and one of the talks I’d marked as “must see” was Aaron’s Road Warrior Disaster Recovery, and I was not disappointed. Aaron covered various strategies available to us to ensure our laptops are secure, synchronized, backed up and ready to recover. Lots of food for thought for me. I gave it a rating of “my vote for Best Talk at BSDCan 2019”.

As to my own talk, I had a hell of a time with my demo: the university of Ottawa prohibits WiFi routers, so the demonstration I wanted to give with equipment brought across the Atlantic was in danger. With the help of a few people who lent me bits and pieces (thank you all, again, for switch and cables) I was able to convert most of my demo hardware to cabled access. Simply the Wemos-D1 had to remain on WiFi so I risked it, plugging it in a mere minute before I needed it and crossing fingers it wouldn’t be remotely de-authed. Unfortunately I couldn’t get the Raspi with the #blinkenlights to work; very sad.

official feedback

Even so, i got good feedback for my own talk, MQTT for system administrators (and for the IoT) which came across with quite a large audience and many interested and interesting questions at the end.

The final and closing talk at BSDCan2019 was moderated by Dan Langille, and it was hugely amusing. I’ll say just one thing: an auction for a good cause. :-) This was followed by the closing social event which gave me a chance to chat to a number of interesting people. It also gave me the chance to receive quite a bit of praise for my MQTT talk.

I’d like to thank the BSDCan committee for inviting me to speak and attend the conference; you may do that again. :-)

bsd and conference :: 22 May 2019 :: e-mail

My LOADays experience started with a discussion-filled evening during and after the speakers’ dinner on Friday, with the “after” portion taking place in the hotel bar. I think I stumbled into bed after midnight.

Up early the Saturday morning to drive Pieter and Andrea to the venue in time for the lovely breakfast they serve speakers there. (If you need a reason to speak at LOADays: the thin, fluffy, tasty omelettes are hard to beat, pun not intended.)

Robert then accompanied me to the room reserved for a bit of a special event: when we last met, I promised him I’d do a special introductory DNS talk for his apprentices, a dozen of which attended bravely sitting up in the front row. I hope I didn’t speak too fast, and that whatever I said made half sense: it’s difficult to distill all I have to say about the topic into a single hour, particularly to young people possibly not accustomed to a lot of technical info. Even so, I had the impression that a few of the attendees were really interested.

and there were books

As a special feature, I schlepped two cases of my book along and gave each of the students a complimentary copy. If nothing else, I told them, they could look at the nice diagrams in it. :-) I was touched upon being given a cellophane-wrapped basket containing a selection of Belgian beer. Those of you who know me know that I’m not terribly partial to it, but I will make a point of explicitly tasting each and every one of them. Here they are, all neatly lined up before I stow them in the fridge.

belgian beer

After the DNS intro (which we’d opened to the “general public”) and after a bit of a break I took, I listened to the tail end of Bert Van Vreckem’s talk on Leveling up your shell scripting skills which had a few interesting tidbits along with some rather impossible ones in my opinion: the example of idempotence on creating a Unix user was “cute” but it doesn’t work the way he showed us, because he completely ignored changing gecos field, shell, or home directory (that requires config management).

Next was Pieter Lexis whom I consider to be a particularly fun speaker; his Running containers and Operating System images with systemd-nspawn sparked quite a bit of joy with me except for his use of “dark mode” on screenshots. (That happened a lot this year; I think nobody listens to me: I yell that out clearly every year several times :-) I’ll study the screenshots quietly directly from Pieter’s slide deck. I’ve always thought it’s a shame systemd-nspawn isn’t spoken about more, and Pieter drove that point home again: it’s really worth looking into, particularly as it’s really well integrated into systemd, whether you like it or don’t.

Pieter Hollants (yes, another Pieter: and there was a Peter too, but he didn’t speak much :-), gave an overview of how he didn’t become an electronics engineer which was fun, and he brought hardware along. The talk saddened me a little bit because it drove home how playing with toys like those is becoming more and more difficult for me with eye-sight failing and hands shaking. His slides are here.

There was a bit of a “Zabbix Track”, if I may call it that: Patrik Uytterhoeven talked about Low-Level Zabbix Discovery for JMX, and I also had a Zabbix talk on loadable C modules and Low-Level Discovery. I had the impression attendees weren’t really thrilled to hear about all of that. I did.

Andrea Tosatto, a very good presenter and colleage of Pieter and Peter, gave an excellent talk on scaling Ansible. He’s fun, he’s fast, but to be honest, he could have simply used my slide deck and saved a lot of time. I’m just kidding of course. His presentation has numbers and it’s well worth your time

Before the traditional open-air pizza (omg I thought it was soo cold outside!), there was a “PowerPoint karaoke” which was hilarious, but it was also too long for me. Shoutout from me to Jeroen Baten and Pieter Lexis for being fantastic spontaneous speakers!

I had to leave that evening due to “management” having made alternative engagements and so missed out on the Sunday breakfast and talks. As such, I’m already looking forward to LOADays 2020. :-)

conference :: 05 May 2019 :: e-mail

Other recent entries