So, recruiters, you’re looking for somebody with ten (10) years of Ansible experience? Here I am. On 2012-06-06, ten years to the day, I wrote my first blog post on Ansible.

I’m certainly not the only one who’s worked with the Ansible software so long, and there are people who’ve done a lot more for its community, but I thought I’d share some of the things I did or experienced along the way.

The first written record on Github of me contributing to Ansible is dated July 20th, 2012. (I may have had previous interactions on the Ansible mailing list but it’s impossible for me to search through that.) I provided the world with a few small modules and lookup plugins, but my pièce de résistance has always been the documentation generator: modules (and later plugins) have their documentation inline, and this is extracted on-the-fly with ansible-doc(1). The official module documentation pages, e.g. get_url are generated from the same source. Back then, running ansible-doc -l to list modules and their single-line description was fast, but Ansible meanwhile has over 6,500 modules and having to instantiate them all to access the DOCUMENTATION variable hurts. I work around this deficiency by recommending ansible-doc -l > doc and grep(1) through that.

Michael's tweet about ansible-doc

Contributing modules used to be fun, but hearing years later that ini_file doesn’t work when a section is empty or whatnot, just makes me yell “well then just don’t leave it empty!” into the void. Maintainers know what suffering is, and I’ve had a minuscule amount of it. On the other hand it’s a confirmation that the code is being used. (Nobody’s ever complained about my mqtt notification module…)

There have been ups and downs. One of the “ups” was when Michael invited me to the very first Ansible Fest in Boston in 2013. There I also had the distinct pleasure of meeting Seth Vidal with whom I discussed creating local facts. Sadly, Seth passed shortly thereafter – a loss to the community.

Another definitive “up” for me is how I won the argument against Michael that module docs should have formatting in them. A very simply C(constant) type syntax was invented, which he didn’t want, but I won. I’ve dined out on that story and strongly hope he has no hard feelings about it. :-)

I’ve previously written about how I care about Ansible, and I still do. I greatly enjoy teaching students about Ansible and how to use it, and while I actively stopped doing so for Tower/AWX in 2020, several hundred people have meanwhile listened to me be mostly positive about the rest, and we went a step further and created a more advanced course.

One of my infamous “downs” was the sudo cake fiasco which gathered some notoriety and has even found mention in books. In terms of embarrassing mistakes in my career, it certainly takes the cake. Today, still, after countless hours of therapy I use the cake metaphor to teach students that we have to respect config management and that running modules with root privileges on hundreds of machines can be exciting. An upside of this fiasco: it lead to Brian (who read the post back then) suggesting template validation, meanwhile also available on a bunch of other modules, and while I like saying validation is for wimps, students do always realize I’m poking fun at myself in saying so!

sudo cake

I regret having had to abandon work I began on an Ansible booklet which could have become a good bit of (printed) documentation.

draft Ansible booklet

I couldn’t find the time to complete the effort and then forgot about it. Others have done stellar work on documenting Ansible: Jeff Geerling with his book Ansible for DevOps and Lorin Hochstein, René Moser, and Bas Meijer with their book Ansible Up & Running which I’ve had the honour of reviewing for the 2nd and 3rd editions.

I’m no fan of cowsay and coined the configuration NOCOWS=1, but I did write up a cow and bull story explaining their Ansible-related background. In my Ansible trainings I install a self-written cow server called cattled for an exercise and hope that will aid in reinstating me amongst cow lovers. Apropos cows: I’m a bit chuffed to report I’m the person who convinced O’Reilly to turn the cow around on the cover of “Up and Running”’s 3rd edition so that it matches the direction in which cowsay cows look.

Ansible Up and Running covers 2nd and 3rd ed.

We invented the {{ ansible_managed }} variable for templates back in 2012 as a mechanism with which I can mark a file as being managed somewhere else, hopefully preventing administrators in editing this file. The default value has always been configurable and was broken during some misunderstanding. Anyway, it can still be configured. Ten years later I stumble across official documentation which demonstrates how to couple that with {{ comment }} to make it output over several lines. The times I’ve wanted that … It turns out the idea was part of the PR introducing the comment filter, so thank you Jiri!

It’s been a bit of fun.

I decided to ask my followers whether I have somehow influenced/improved/deteriorated their lives in some way with my Ansible work, and these are some (partially abbreviated) responses I got:

  • “I started automating stuff with Ansible because of your blogposts years ago” – Hagen
  • “You made my life as admin easier. And while I can only write little things in Ansible, it really helps me.” joanna
  • “I’m learning about your contributions every week. Did not know that you invented ansible_managed or the dig-lookup. So thank you!” – FiLiS
  • “Since you’re responsible for ANSIBLE_NOCOWS=1, I’d say it did, yea.” Armin
  • “I first heard about Ansible through one of your first blog posts.” Serge
  • ”.. your blog posts at the time really showed how easy it was to get started ..” Vincent
  • “There are only a few devs who I can almost guarantee will show up in my Google results when deep into a weird Ansible playbook… and you’re one of them.” Jeff
  • “Your impact is obvious to anyone who looks.” Greg
  • “Your blog posts and presentations were definitely inspiring.” – René
  • “Your talk about Ansible at OSDC years ago got us to give it a try at Sipgate (and move away from puppet). Today, Ansible powers large parts of our deployments and system/network infrastructure.” Rudolph
  • “Somewhere early 2012 you tweeted about Ansible [..] which turned out to have a massive impact on my working and personal life which I never could have imagined.” - Ton (in a long email)
  • “i think it was your tweets and blogging about ansible that made me aware of it, and it was just what i needed (just use ssh, agentless, with check and diff modes). … ansible was soon adopted by my colleagues and it became the standard config management tool in cambridge university information services” Tony

Sadly some people didn’t properly read my query (I think) and just sent a few dozen “Ansible sucks” and “fuck YAML” my way, so my expectations in that regard were also met.

It gives me a warm, fuzzy feeling that you might in some part of the world be using some small portion of Ansible in which I muddled, back in the time. Enjoy it. And what really gives me warm feelings is that along the way I’ve met a number of people (in real life) whom I enjoy encountering when I can.

And why did I even glance at something as unfinished as Ansible was at the time, those ten years ago? I worked in a project in which I wanted to automate deployment of a number of DNS servers, and I disliked the tool I had to use so intensely, that I grasped at any straw I could find. That’s my story, and I’ve no regrets.

ansible and anniversary :: 06 Jun 2022 :: e-mail

I created msoak as a utility to subscribe to multiple MQTT brokers simultaneously and consume any number of topics; it soaks up what they have to offer and prints the topic/payload to stdout. I pipe that output to tinylog for storage, compression, and file rotation. I’ve been using this setup to log from a bunch of brokers ever since msoak was born two years ago.

Today, while looking at previously-logged data in preparation for consumption of future similar data, I discovered to my horror, that tinylog(8) truncates lines at what I first assumed was 1024 but actually is 1000 bytes.

I simply hadn’t noticed because most of the payloads we save are less than 1000 bytes. It’s my fault, as it’s documented, but I’d overlooked that sentence.

tinylog ignores empty lines, truncates lines longer than 1000 characters

Within minutes I found the definition and I’m now good. (80k is really sufficient for the moment.)

--- perp-2.07/perp/tinylog.c	2013-01-11 16:10:32.000000000 +0100
+++ perp-2.07-jp/perp/tinylog.c	2022-05-25 17:53:35.000000000 +0200
@@ -96,7 +96,7 @@
 /* default maximum size for log file (bytes): */
 #define CURRENT_MAX  100000
 /* maximum size for line in log file (bytes): */
-#define LOGLINE_MAX    1000
+#define LOGLINE_MAX    80000
 /* pause on exceptional error (billionths of second): */
 #define EPAUSE  555444321UL

They say a backup is only as good as its recovery. Very true.

toolbox :: 25 May 2022 :: e-mail

I need a flipchart or a whiteboard while teaching, but during online sessions I resort to using a terminal and a text editor and quickly got tired of having to swap sharing different windows in Zoom, Big Blue Button, & co. (I share individual windows, not the whole screen.) I share a window of a Web browser with my presentation in one tab and an open terminal using gotty in the second, and can flip between the two with CMD-1 / CMD-2.

screenshot of browser with two tabs

I stumbled over gotty some years ago, and have used it to enable users to login to systems if they can’t SSH in (think corporate firewalls). It works well and is reliable. During the new setup I’m doing for training machines I need one gotty per user which is easily accomplished using distinct TCP port numbers. My first attempt in launching them was by templating out an rc.local script, but I wanted something more elegant.

I recalled that systemd services can take an argument via the service@argument syntax which fits in well with my user numbering, and after a bit of experimentation came up with a solution which works well.

[Unit]
Description=GoTTY Web Terminal %i
After=network.target

[Service]
User=user%i
Group=user%i

WorkingDirectory=/home/user%i
ExecStart=/usr/local/sbin/gotty -p 91%i tmux new-session -A -s user%i

[Install]
WantedBy=multi-user.target

The unit file’s WorkingDirectory is needed or gotty starts in /, and the tmux invocation is to automatically start or attach to tmux in an SSH session. Users can SSH into the machine and/or use gotty and see the same screen. More importantly, I can, once given consent by a student, login and see their screen, help with error messages, give tips, etc.

Each user’s home directory has a ~/.gotty preferences file containing

address = "127.0.0.1"
credential = "username:password"
title_format = "Ansible:username"
permit_write = true
preferences {
    font_size = 20
    background_color = "rgb(255, 255, 255)"
    foreground_color = "rgb(0, 0, 0)"
}

The required number of gotty sessions are launched during setup of the machine with Ansible. (The sequence lookup produces a list of numbers 00, 01, 02 based on the number of users I expect to welcome.)

- name: "Gotty: start gotty services"
  systemd: name="gotty@{{ item }}" enabled=true state=started
  with_sequence: '{{ nusers }}'

This is the resulting process list:

$ ps aux|grep g[o]tty
user01      1209  ... /usr/local/sbin/gotty -p 9101 tmux new-session -A -s user01
user00      1244  ... /usr/local/sbin/gotty -p 9100 tmux new-session -A -s user00

Each gotty listens on the loopback interface. A templated-out configuration lets a TLS-protected nginx reverse proxy talk to the gottys.

location /tty00/ {
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $remote_addr;
	proxy_set_header Host $http_host;
	rewrite ^/tty00/?$ / break;
	rewrite ^/tty00/(.*)$ /$1 break;
	proxy_pass http://127.0.0.1:9100;
	proxy_http_version 1.1;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection "upgrade";
}

It’s only just now while writing this, that I realize GoTTY, which means “tty thing written in Go” actually reminds me of getty(8). :-)

Updates

  • There’s a newer and maintained version at https://github.com/sorenisanerd/gotty. The only way I can get it to understand my font/colour preferences is if I configure term = "hterm", but that, precisely, is what this fork appears to be dropping in favor of xterm.

terminals :: 03 May 2022 :: e-mail

I avoid using the cloud in situations where it’s not really necessary which is why I had the “whole” infrastructure with me during Ansible trainings: no dependencies on the Internet not working at a customer site, firewall rules forbidding access to resources, etc.

During the second half of 2019 I dreamed up and created an Ansible training environment based on FreeBSD and spent quite a bit of time getting it just right. TL;DR: it was a laptop with FreeBSD and a number of jails on it. Students would each have three “machines” from and onto which they could perform deployments in labs during the training. The setup contained a package repository and a PIP repo for select installs required during labs, a local DNS server, etc.; everything was self contained. Perfect for me, and after its first productive use I reported how enthusiastic I was that nobody had noticed anything untoward:

They looked at me a bit curiously, and when I told them they’d all been working together on one laptop, I saw the odd jaw open.

You might have noticed that last report is dated beginning of the pandemic: February 2020.

As you can imagine, all planned events for that year (and the next, and the next, but I digress) were cancelled, postponed, or requested as “online” versions. How to solve the problem of my portable data center?

Use the cloud, they said.

I found Digital Ocean offered FreeBSD as one of their Droplet operating systems, and I was quite quickly able to convert my setup to one of their Droplets. A few hours before a training, I would launch my magical shell scripts (sue me!), and behold I’d have a for me perfect setup for doing the labs. Once the training over, I would destroy the Droplet and reiterate at the next occasion.

Life was good. The setup performed well, and I tweaked and changed things to suit me better, but I was happy with how it all fitted together, and I don’t recall having heard a complaint during a training. (Once, a student said: “this isn’t Linux, is it?”). Then came the time when I learned that FreeBSD 12.2 was to become EOL. I contacted Digital Ocean to ask about FreeBSD’s future on their platform: hmm, nothing specific could yet be said. They were discussing it internally and couldn’t comment at this time. I thought that ominous.

Just after 12.2 EOL in March of this year it turns out that Digital Ocean are dropping support for FreeBSD entirely. FiLiS reports about Digital Ocean saying:

the action plan is to retire FreeBSD versions through the UI starting June 2022 and through the API starting August 2022.

Damnation.

I became aware that it is possible to upload FreeBSD custom images to Digital Ocean, and a number of people responded to tweets etc. telling me likewise, but I simply am not interested in hacking that. First of all because I don’t care to actually do the work and have to maintain the lot, and secondly because storing images isn’t free of charge. It’s not a large amount of money, but there’s the odd gin&tonic to be gained from not doing so. But honestly, why should I do this if alternative operating systems are maintained? I don’t see a benefit for myself.

The good folk at BastilleBSD pointed me at Vultr who actually do have a current (13.0) version of FreeBSD, but unfortunately it’s UFS file system only, and I need ZFS because of my iocage use. I’ve asked Vultr, and they’ve logged a feature request to their development team, but it’s unknown whether it will happen and in which time frame, and that’s a problem: if Digital Ocean really kill FreeBSD in June/August, then I’m in dire need of a working solution.

I probably could, providing enough time and effort, change, replace, rip out, re-insert, and whatever, for example replace iocage by BastilleBSD, but I just don’t feel like it any more. I’ve reached an age at which it’s just no fun. I need something with a “best before” date further in the future. Preparing for a course the evening before only to realize that stuff doesn’t work and I have to stay up for hours so that a dozen students don’t realize it didn’t work is just no longer my cup of tea. (This occurred to me near Utrecht a few months ago: at midnight I set up the servers only to have the system fail because a change to the way pkg bootstrapping worked was sprung upon me. Not fun.)

At the moment I don’t see an alternative to changing my setup. I will likely use individual Droplets, virtual machines, instances, or whatever you call them, launching them, getting a few things prepared and being done with it. I’m a perfectionist during trainings, and labs etc. have to be just so.

And to be quite clear, this is certainly not a reason for me to stop using Digital Ocean’s offerings, on the contrary. They’ve been good for me and will hopefully continue to be so. It’s not as though they’ll be losing a huge amount of money from my not using an 8GB machine once in a while, right? FreeBSD is likely not big business for them and hence they drop it. C’est la vie. We will gladly continue using their services for our Linux-based DNS & DNSSEC trainings.

I’ll get over it and solve the problem, but judging from what I’ve looked at the past 24 hours it won’t be using FreeBSD.

Further reading

ansible, freebsd, and linux :: 28 Apr 2022 :: e-mail

I’m regularly made fun of when teaching DNSSEC because I tell people I use a “napkin” when creating DNSSEC keys to jot down the key tags (or key IDs), and it’s true: also during trainings I have the “napkin” – to be precise it’s a sheet of A4 paper on which I note modifications to schedule, timezones, whether I still owe answers to questions, and of course, the key IDs of DNSSEC keys.

Here’s a partial scan of last week’s napkin:

the last napkin I used for DNSSEC signings

I don’t mind students’ grins and typically laugh last when they say they’re getting errors (e.g. during manual signing), whereupon I can victoriously respond: “if you’d used a napkin to make a note of which key tag is KSK and which is ZSK that wouldn’t have occurred!” :-)

I wonder whether I ought to start a napkin business and what the design should look like …

dnssec :: 24 Jan 2022 :: e-mail

Other recent entries