The idea for an Ansible reference (or cheat) sheet was reborn last week at the Linuxhotel; a few students who knew they would be receiving a tmux mug as a gift when leaving asked why we don’t give out Ansible mugs.

One person approached me and said mugs are impractical: most people already have their favorite beverage mug, mugs have little space for information, and they typically find their way into the back of some kitchen cabinet. Why don’t we consider a nicely-printed DIN A3 sheet on good (plastified) paper? I thought that a good idea.

Ansible cheat sheet

And so began the work of gathering the information I wanted to present and packing it onto a double-sided A3 page. I’d have very much liked to add details for our advanced course, but there’s simply not enough space on these two pages. (If your eye sight is very good you might be able to read it when printed on A4.)

So, here’s the reference card, and please let me know if you have feedback.

ansible and documentation :: 24 Jul 2022 :: e-mail

My request was pretty easy, I thought, when I asked the person to give me a list of student names and their email addresses so I could send out invitations.

I got a spreadsheet:


Five empty rows. Eight rows with an additional third column containing a first name which is already in column A. Two additional rows with some blurb and a given name, but as there’s no email address associated, I can’t use the entry.

Mind-boggling. Even more so when you know what this IT person’s title is.

people :: 19 Jul 2022 :: e-mail

So, recruiters, you’re looking for somebody with ten (10) years of Ansible experience? Here I am. On 2012-06-06, ten years to the day, I wrote my first blog post on Ansible.

I’m certainly not the only one who’s worked with the Ansible software so long, and there are people who’ve done a lot more for its community, but I thought I’d share some of the things I did or experienced along the way.

The first written record on Github of me contributing to Ansible is dated July 20th, 2012. (I may have had previous interactions on the Ansible mailing list but it’s impossible for me to search through that.) I provided the world with a few small modules and lookup plugins, but my pièce de résistance has always been the documentation generator: modules (and later plugins) have their documentation inline, and this is extracted on-the-fly with ansible-doc(1). The official module documentation pages, e.g. get_url are generated from the same source. Back then, running ansible-doc -l to list modules and their single-line description was fast, but Ansible meanwhile has over 6,500 modules and having to instantiate them all to access the DOCUMENTATION variable hurts. I work around this deficiency by recommending ansible-doc -l > doc and grep(1) through that.

Michael's tweet about ansible-doc

Contributing modules used to be fun, but hearing years later that ini_file doesn’t work when a section is empty or whatnot, just makes me yell “well then just don’t leave it empty!” into the void. Maintainers know what suffering is, and I’ve had a minuscule amount of it. On the other hand it’s a confirmation that the code is being used. (Nobody’s ever complained about my mqtt notification module…)

There have been ups and downs. One of the “ups” was when Michael invited me to the very first Ansible Fest in Boston in 2013. There I also had the distinct pleasure of meeting Seth Vidal with whom I discussed creating local facts. Sadly, Seth passed shortly thereafter – a loss to the community.

Another definitive “up” for me is how I won the argument against Michael that module docs should have formatting in them. A very simply C(constant) type syntax was invented, which he didn’t want, but I won. I’ve dined out on that story and strongly hope he has no hard feelings about it. :-)

I’ve previously written about how I care about Ansible, and I still do. I greatly enjoy teaching students about Ansible and how to use it, and while I actively stopped doing so for Tower/AWX in 2020, several hundred people have meanwhile listened to me be mostly positive about the rest, and we went a step further and created a more advanced course.

One of my infamous “downs” was the sudo cake fiasco which gathered some notoriety and has even found mention in books. In terms of embarrassing mistakes in my career, it certainly takes the cake. Today, still, after countless hours of therapy I use the cake metaphor to teach students that we have to respect config management and that running modules with root privileges on hundreds of machines can be exciting. An upside of this fiasco: it lead to Brian (who read the post back then) suggesting template validation, meanwhile also available on a bunch of other modules, and while I like saying validation is for wimps, students do always realize I’m poking fun at myself in saying so!

sudo cake

I regret having had to abandon work I began on an Ansible booklet which could have become a good bit of (printed) documentation.

draft Ansible booklet

I couldn’t find the time to complete the effort and then forgot about it. Others have done stellar work on documenting Ansible: Jeff Geerling with his book Ansible for DevOps and Lorin Hochstein, René Moser, and Bas Meijer with their book Ansible Up & Running which I’ve had the honour of reviewing for the 2nd and 3rd editions.

I’m no fan of cowsay and coined the configuration NOCOWS=1, but I did write up a cow and bull story explaining their Ansible-related background. In my Ansible trainings I install a self-written cow server called cattled for an exercise and hope that will aid in reinstating me amongst cow lovers. Apropos cows: I’m a bit chuffed to report I’m the person who convinced O’Reilly to turn the cow around on the cover of “Up and Running”’s 3rd edition so that it matches the direction in which cowsay cows look.

Ansible Up and Running covers 2nd and 3rd ed.

We invented the {{ ansible_managed }} variable for templates back in 2012 as a mechanism with which I can mark a file as being managed somewhere else, hopefully preventing administrators in editing this file. The default value has always been configurable and was broken during some misunderstanding. Anyway, it can still be configured. Ten years later I stumble across official documentation which demonstrates how to couple that with {{ comment }} to make it output over several lines. The times I’ve wanted that … It turns out the idea was part of the PR introducing the comment filter, so thank you Jiri!

It’s been a bit of fun.

I decided to ask my followers whether I have somehow influenced/improved/deteriorated their lives in some way with my Ansible work, and these are some (partially abbreviated) responses I got:

  • “I started automating stuff with Ansible because of your blogposts years ago” – Hagen
  • “You made my life as admin easier. And while I can only write little things in Ansible, it really helps me.” joanna
  • “I’m learning about your contributions every week. Did not know that you invented ansible_managed or the dig-lookup. So thank you!” – FiLiS
  • “Since you’re responsible for ANSIBLE_NOCOWS=1, I’d say it did, yea.” Armin
  • “I first heard about Ansible through one of your first blog posts.” Serge
  • ”.. your blog posts at the time really showed how easy it was to get started ..” Vincent
  • “There are only a few devs who I can almost guarantee will show up in my Google results when deep into a weird Ansible playbook… and you’re one of them.” Jeff
  • “Your impact is obvious to anyone who looks.” Greg
  • “Your blog posts and presentations were definitely inspiring.” – René
  • “Your talk about Ansible at OSDC years ago got us to give it a try at Sipgate (and move away from puppet). Today, Ansible powers large parts of our deployments and system/network infrastructure.” Rudolph
  • “Somewhere early 2012 you tweeted about Ansible [..] which turned out to have a massive impact on my working and personal life which I never could have imagined.” - Ton (in a long email)
  • “i think it was your tweets and blogging about ansible that made me aware of it, and it was just what i needed (just use ssh, agentless, with check and diff modes). … ansible was soon adopted by my colleagues and it became the standard config management tool in cambridge university information services” Tony

Sadly some people didn’t properly read my query (I think) and just sent a few dozen “Ansible sucks” and “fuck YAML” my way, so my expectations in that regard were also met.

It gives me a warm, fuzzy feeling that you might in some part of the world be using some small portion of Ansible in which I muddled, back in the time. Enjoy it. And what really gives me warm feelings is that along the way I’ve met a number of people (in real life) whom I enjoy encountering when I can.

And why did I even glance at something as unfinished as Ansible was at the time, those ten years ago? I worked in a project in which I wanted to automate deployment of a number of DNS servers, and I disliked the tool I had to use so intensely, that I grasped at any straw I could find. That’s my story, and I’ve no regrets.

ansible and anniversary :: 06 Jun 2022 :: e-mail

I created msoak as a utility to subscribe to multiple MQTT brokers simultaneously and consume any number of topics; it soaks up what they have to offer and prints the topic/payload to stdout. I pipe that output to tinylog for storage, compression, and file rotation. I’ve been using this setup to log from a bunch of brokers ever since msoak was born two years ago.

Today, while looking at previously-logged data in preparation for consumption of future similar data, I discovered to my horror, that tinylog(8) truncates lines at what I first assumed was 1024 but actually is 1000 bytes.

I simply hadn’t noticed because most of the payloads we save are less than 1000 bytes. It’s my fault, as it’s documented, but I’d overlooked that sentence.

tinylog ignores empty lines, truncates lines longer than 1000 characters

Within minutes I found the definition and I’m now good. (80k is really sufficient for the moment.)

--- perp-2.07/perp/tinylog.c	2013-01-11 16:10:32.000000000 +0100
+++ perp-2.07-jp/perp/tinylog.c	2022-05-25 17:53:35.000000000 +0200
@@ -96,7 +96,7 @@
 /* default maximum size for log file (bytes): */
 #define CURRENT_MAX  100000
 /* maximum size for line in log file (bytes): */
-#define LOGLINE_MAX    1000
+#define LOGLINE_MAX    80000
 /* pause on exceptional error (billionths of second): */
 #define EPAUSE  555444321UL

They say a backup is only as good as its recovery. Very true.

toolbox :: 25 May 2022 :: e-mail

I need a flipchart or a whiteboard while teaching, but during online sessions I resort to using a terminal and a text editor and quickly got tired of having to swap sharing different windows in Zoom, Big Blue Button, & co. (I share individual windows, not the whole screen.) I share a window of a Web browser with my presentation in one tab and an open terminal using gotty in the second, and can flip between the two with CMD-1 / CMD-2.

screenshot of browser with two tabs

I stumbled over gotty some years ago, and have used it to enable users to login to systems if they can’t SSH in (think corporate firewalls). It works well and is reliable. During the new setup I’m doing for training machines I need one gotty per user which is easily accomplished using distinct TCP port numbers. My first attempt in launching them was by templating out an rc.local script, but I wanted something more elegant.

I recalled that systemd services can take an argument via the service@argument syntax which fits in well with my user numbering, and after a bit of experimentation came up with a solution which works well.

Description=GoTTY Web Terminal %i


ExecStart=/usr/local/sbin/gotty -p 91%i tmux new-session -A -s user%i


The unit file’s WorkingDirectory is needed or gotty starts in /, and the tmux invocation is to automatically start or attach to tmux in an SSH session. Users can SSH into the machine and/or use gotty and see the same screen. More importantly, I can, once given consent by a student, login and see their screen, help with error messages, give tips, etc.

Each user’s home directory has a ~/.gotty preferences file containing

address = ""
credential = "username:password"
title_format = "Ansible:username"
permit_write = true
preferences {
    font_size = 20
    background_color = "rgb(255, 255, 255)"
    foreground_color = "rgb(0, 0, 0)"

The required number of gotty sessions are launched during setup of the machine with Ansible. (The sequence lookup produces a list of numbers 00, 01, 02 based on the number of users I expect to welcome.)

- name: "Gotty: start gotty services"
  systemd: name="gotty@{{ item }}" enabled=true state=started
  with_sequence: '{{ nusers }}'

This is the resulting process list:

$ ps aux|grep g[o]tty
user01      1209  ... /usr/local/sbin/gotty -p 9101 tmux new-session -A -s user01
user00      1244  ... /usr/local/sbin/gotty -p 9100 tmux new-session -A -s user00

Each gotty listens on the loopback interface. A templated-out configuration lets a TLS-protected nginx reverse proxy talk to the gottys.

location /tty00/ {
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $remote_addr;
	proxy_set_header Host $http_host;
	rewrite ^/tty00/?$ / break;
	rewrite ^/tty00/(.*)$ /$1 break;
	proxy_http_version 1.1;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection "upgrade";

It’s only just now while writing this, that I realize GoTTY, which means “tty thing written in Go” actually reminds me of getty(8). :-)


  • There’s a newer and maintained version at The only way I can get it to understand my font/colour preferences is if I configure term = "hterm", but that, precisely, is what this fork appears to be dropping in favor of xterm.

terminals :: 03 May 2022 :: e-mail

Other recent entries