For a long time I’ve been annoyed that VirtualBox issues a different DHCP lease to a virtual machine guest after a while, and there doesn’t seem to be any way to change that within VirtualBox itself. I asked on Twitter, and got a response which trigerred an aha!

virtualbox dhcp disable

I disabled VirtualBox’ DHCP for the interface I’m interested in, and configured a dnsmasq on my notebook.


Minutes later I saw this on the console upon launching dnsmasq and then a VirtualBox guest I already had:

dnsmasq: started, version 2.78 cachesize 150
dnsmasq: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack no-ipset auth no-DNSSEC loop-detect no-inotify
dnsmasq: setting --bind-interfaces option because of OS limitations
dnsmasq-dhcp: DHCP, IP range --, lease time 12h
dnsmasq: reading /etc/resolv.conf
dnsmasq: using nameserver
dnsmasq: using nameserver
dnsmasq: read /etc/hosts - 3 addresses

dnsmasq-dhcp: DHCPDISCOVER(vboxnet0) 08:00:27:7d:aa:db
dnsmasq-dhcp: DHCPOFFER(vboxnet0) 08:00:27:7d:aa:db
dnsmasq-dhcp: DHCPREQUEST(vboxnet0) 08:00:27:7d:aa:db
dnsmasq-dhcp: DHCPACK(vboxnet0) 08:00:27:7d:aa:db cen7pdns

Lovely. It works.

View Comments :: VirtualBox and DHCP :: 07 Mar 2018 :: e-mail

After three (or was it almost four?) years of using Slack, we took the plunge and set up our own instance of Mattermost. The reasons for doing this include wanting more control over our data and wanting an unlimited history which Slack, as a hosted service, offers only to paying customers. This is more than fair enough – it’s not their fault that we’re too stingy. Apropos stingy: Mattermost exists in three editions – we chose the Team Edition; guess why.

Web UI

If you know how to wield Slack, you know how to use Mattermost. (Oh, please don’t mind the awful colors above – this is my test installation and I need severe optical distinction in order to not mistake the installations.) There are few differences, if any. Mattermost’s use of Markdown appears to be more comprehensive, in particular because its Webhooks support Markdown.

Installing Mattermost is easy thanks to the good documentation they provide which explains, step by step, what I have to do to install Mattermost on a machine. I chose to use PostgreSQL because I recall having read that’s Mattermost’s primary database candidate, and because it allows me to use PostgreSQL – a reason sufficient on its own. I selected to loosely follow advice given regarding location of config.json as that seemed a sensible thing to do. (config.json is Mattermost’s central configuration file which is reloaded on change.) If Ansible’s your drug, Pieter Lexis created an Ansible role for installing Mattermost on Debian/Unbuntu, and there’s playbook which does that and more also.

I can create as many teams as I want on a server, and each team can have as many channels as I want. In the Team Edition users authenticate with password and I enforced e-mail verification. (Other editions offer 2FA and LDAP.)

Mattermost users can upload files (images, code snippets, etc.) which have to be stored somewhere. By default a configurable directory on the local file system is used, but Mattermost’s system console allows me to configure Amazon S3 storage such as Minio.

Webhooks, API, Websocket, CLI, etc.

One of the things I like most about Slack are its integrations, and low and behold, Mattermost has these as well. Incoming and outgoing Webhooks as well as slash commands. Lovely.

Also very powerful is the Websocket API; there’s a Python3 driver which works very well, and next to that is Mattermost’s API with which I can create users, get their details, enumerate posts, create posts, etc. The following example using curl and jo shows how I can add a post from the command line:

json=$(jo channel=cartoons1 \
          username="my-script" \
          icon_url="" \
          text='Ha, this is _just_  an example using `curl`, :tada:')

curl -H 'Content-Type: application/json' \
     --data "$json" \

posted with curl

(For a much more flexible solution see 42wim’s mattertee.)

Programs which use Mattermost’s API must authenticate to the service and they can do so with either session tokens that expire and Personal Access Tokens which I create in my account preferences which don’t expire until I revoke them. Additionally Mattermost can act as an OAuth 2.0 provider.

Masses of messages

One of the channels we have is reserved for Nagios/Icinga-type notifications. One thing I wanted to be able to do is to delete and purge those messages; I don’t see why I need to know weeks later that something was offline for a moment. However, if I delete a message, either interactively or via the API, Mattermost soft deletes it; the message is marked as deleted with a time stamp, but it remains in the database.

So I went in search of an API to physically remove these messages, but it doesn’t exist. The solution? Use, say, the API to find the posts I want to remove, “delete” them using said API, and then use an SQL DELETE to purge:

DELETE FROM posts WHERE deleteat <> 0;

Mattermost at a console

Mattermost has a Web UI and some mobile and fat clients, but what does a person do with just a terminal at his/her disposal? Use either matterhorn or Irssi or your favorite IRC client with matterircd.

seed in Matterhorn

What you see in the screenshot above is matterhorn showing what the first Web UI screenshot shows. The program has some really cool features including scripts – just shell scripts which are given the text I enter on stdin and the stdout they produce is posted. Matterircd on the other hand is an IRC to Mattermost gateway written also by 42wim: you connect it to your remote Mattermost installation and talk to it via your IRC client.


Do I regret leaving Slack? Not really, even though their mobile apps are quite a bit more polished than Mattermost’s are – a result of development effort obviously. I now get a warm and quite fuzzy feeling knowing that we have control over our data, how we back it up, and what we do with it. And I’m confident that (other than the NSA) no third party has it.

Apropos 3rd party: while it’s possible to access dozens (or hundreds?) of integrations using an external service called Zapier we will not as that defeats the purpose of wanting to be the sole owners of said data. Similarly we’ve been discussing mobile notifications for which we could either set up mobile push or do it ourselves; we haven’t finally decided yet.

Do it yourself push

Do it ourselves, you ask? Yes, that’s possible by creating a notification endpoint which Mattermost uses whenever it’s about to notify a mobile device. The post I created with a shell script earlier, pushes to this example MQTT:

$ mosquitto_sub -v -t 'mm/#'
mm/_notif my-script in cartoons: Ha, this is _just_  an example using `curl`, :tada:

The way this happens is that Mattermost notifies a Web service I create which obtains the message and disposes of it in any way I want:

#!/usr/bin/env python

from bottle import run, request, post
import json
import paho.mqtt.publish as paho

__author__    = 'Jan-Piet Mens <jp()>'

def post1():

    data = json.loads(

    paho.single("mm/_notif", data['message'],

run(host='', port=8864)

If you’re not interested in mobile push, there’s always e-mail: when users are away or offline they can choose to be notified of new content by e-mail.

If I’d known about Mattermost before, I’d have migrated earlier.

View Comments :: Social :: 30 Jan 2018 :: e-mail

Gogs and Gitea are Open Source, single-binary Go implementations of a Github-like git repository hosting system which I can use privately or in teams. Both are light on system requirements, contrary to a pretty large and quite popular Open Source Java implementation of a similar utility.

Gitea is a fork of Gogs, and it’s very difficult to say which is better: Gogs is, at times, a bit more Github-lookalike I’d say, whereas Gitea’s look is a bit fresher and feels “younger”. Gitea brings everything in a single binary whereas Gogs has a number of support files (CSS, JavaScript, templates) it requires. (This is advantageous because I can change or replace templates if desired. Fabian reminds me that it’s possible to customize Gitea as well.) Both projects appear to be alive and kicking in spite of rumors that Gogs had been abandoned.

gogs installer

Installation basically consists of ensuring the appropriate binary is launched. I then launch a Web browser and point it at the port number indicated on the console ( by default) and answer a few questions. The responses are written into an INI-type configuration file (app.ini) which can be pre-populated of course and looks like this:

DB_TYPE  = sqlite3
HOST     =
NAME     = gogs
USER     = root
SSL_MODE = disable
PATH     = data/gogs.db

ROOT = data/repos

DOMAIN           = localhost
HTTP_PORT        = 4000
ROOT_URL         = http://localhost:4000/
DISABLE_SSH      = false
SSH_PORT         = 4022
OFFLINE_MODE     = true

A cursory search will also provide plenty of resources for installing either with, say, Ansible, if that’s your preferred method. Alternatively, both programs support installation with docker.

Both programs support different backend databases (SQLite3, PostgreSQL, MySQL) and SMTP, PAM, or LDAP authentication. They offer git over HTTP and have an optional embedded SSH server for git over SSH. (Just like in Github or Gitlab, I upload one or more SSH keys which Gogs/Gitea use for authentication.) While there exists a list of Gitea features and Gogs features these lists are difficult to compare.

Both programs have a CLI albeit with slightly differing commands. The CLI is used for backups, user creation, and other administrative commands.

$ gogs admin create-user --name jane --password sekrit --email
New user 'jane' has been successfully created!

I initially chose to use Gitea but thought I’d ask yesterday in a poll: 50/50 is about the reponse I get. A friendly user wrote privately and said:

I’ve been running Gogs for ~20 active users on a low-end VPS alongside other services for a year now. The cli is quite simple and works well, especially backups. Manual upgrades via git went well too 6 month ago.

Both utilities have a dump respectively backup CLI command to create a backup of their data in a ZIP file.

I find Gogs documentation more comprehensive (and Gitea’s sometimes links to it). Featurewise, both are more or less on par, at least in terms of visible features. Both are Open Source, and both projects have over 500 open issues in their trackers and several dozen open pull requests.

Gogs and Gitea can import projects, so I used one of my Github-hosted repositories as a source to produce the following screen shots.





A bit of both

Both programs display commits like Github does and have a unified diff and a split (side-by-side) diff view. In Gitea the knobs are located as I know them from Github, but that doesn’t mean Gogs’ knob placement isn’t better. Interestingly, repository settings and other pages in Gogs are styled more similarly to Github than in Gitea. So again, six of one and half a dozen of the other. Both allow import of existing repositories (as I did above), though just the repository is imported: neither the issues nor the pull requests, at least not from Github.

issue tracker gitea

Both have an issue tracker with github-flavored Markdown support, file attachments, etc. There are slight cosmetic differences but nothing drastic that I can see. Both support Git hooks, Webhooks and deployment keys (and I do prefer the page layout that Gogs offers in the “sub pages” such as settings).

I’ve chosen to use Gitea, but as I’ve said: it’s a hard toss.

Can I have an SCM (Source Code Management) update trigger the launch of an AWX job? The answer is yes, and it’s one of the interesting things I can do to remote-control AWX.


What I need is some way to invoke a program as soon as I commit something. Subversion, git, etc. support hooks, but cloud-based repository systems (Github, Bitbucket) also support Webhooks which I’ll use here.

In this example I’m using gitea which calls itself a painless self-hosted Git service; it’s all of that and more – it’s one of those gorgeous single-binary Go programs which I’m liking more and more. (I showed you another recently – CoreDNS.) Gitea is very much like using Github, but you host it yourself, and it’s trivial to get started with it.

Within gitea, I configure an outgoing Webhook:

Gitea with a Webhook

From now on, as soon as this repository gets a commit pushed to it, the specified URL will be invoked

On the other side, I run a lightweight, configurable utility (again in Go), called adnanh/webhook. This listens for HTTP payloads, extracts JSON from them, and it can invoke a program of my choice to react to the hook. This could be any kind of HTTP endpoint which reacts to a Webhook POST, but I chose this for simplicity.

I configure webhook to run with the following configuration, which will extract the repository’s name and the secret specified in the hook invocation from the incoming payload (here is the full payload sent by gitea).

    "id": "awx-atest",
    "execute-command": "/Users/jpm/bin/",
    "command-working-directory": "/tmp/",
    "pass-arguments-to-command": [
        "source": "payload",
        "name": "repository.full_name"
        "source": "payload",
        "name": "secret"

I launch webhook and watch what happens when I commit and push to the repository:

./webhook -hooks hooks.json -verbose
[webhook] 2017/10/23 18:17:07 version 2.6.5 starting
[webhook] 2017/10/23 18:17:07 setting up os signal watcher
[webhook] 2017/10/23 18:17:07 attempting to load hooks from hooks.json
[webhook] 2017/10/23 18:17:07 os signal watcher ready
[webhook] 2017/10/23 18:17:07 found 1 hook(s) in file
[webhook] 2017/10/23 18:17:07 	loaded: awx-atest
[webhook] 2017/10/23 18:17:07 serving hooks on{id}
[webhook] 2017/10/23 18:17:09 incoming HTTP request from [::1]:54005
[webhook] 2017/10/23 18:17:09 awx-atest got matched
[webhook] 2017/10/23 18:17:09 awx-atest hook triggered successfully
[webhook] 2017/10/23 18:17:09 200 | 388.746µs | localhost:9000 | POST /hooks/awx-atest
[webhook] 2017/10/23 18:17:09 executing /Users/jpm/bin/ with arguments ["/Users/jpm/bin/" "jpm/atest" "none-of-your-business"] and environment [] using /tmp/ as cwd
[webhook] 2017/10/23 18:17:10 command output: {"job":331,"ignored_fields":{},...
[webhook] 2017/10/23 18:17:10 finished handling awx-atest

The truncated output in the second to last line is the JSON returned from the AWX job launch which happens in the script:




if [ "$secret" == "none-of-your-business" ]; then
    curl -qs \
        -d '{"extra_vars":{"newpoem":"hello good world"}}' \
        -H "Content-type: application/json" \
        -u admin:password  \

All this is obviously just an example. Refine to your taste (and add lots of error-handling!)

From AWX

Whilst on the topic of Webhooks: AWX can trigger an arbitrary Webhook as a notification; these are invoked on success or on failure (as desired), and produce a payload which looks like this:

  "created_by": "admin",
  "credential": "jp-ed25519",
  "extra_vars": "{}",
  "finished": "2017-10-24T06:05:09.626734+00:00",
  "friendly_name": "Job",
  "hosts": {
    "alice": {
      "changed": 0,
      "dark": 0,
      "failed": false,
      "failures": 0,
      "ok": 2,
      "processed": 1,
      "skipped": 0
  "id": 335,
  "inventory": "j1",
  "limit": "",
  "name": "pi1",
  "playbook": "ping.yml",
  "project": "p1",
  "started": "2017-10-24T06:04:54.127124+00:00",
  "status": "successful",
  "traceback": "",
  "url": "https://towerhost/#/jobs/335"

The next step is to take bits of the payload to indicate success or failure on your monitoring blinkenlights.

View Comments :: Ansible and AWX :: 23 Oct 2017 :: e-mail

I believe there’s a document floating around somewhere in which is written that “JP Mens brought Ansible to Europe in 2012” or something to similar effect. Whilst I think that may be a tad exaggerated it is true that I did a few conferences and talks during which I enthusiastically spoke about the then new kid on the block. I’m recounting this anecdote because something similar may happen with Ansible AWX. I’ll be talking about AWX to anybody who wants to listen.

Ansible AWX is the upstream project which holds the code which at some point in time, and I guess periodically, turns into Ansible Tower. It’s been a long time coming, but Ansible has now open sourced AWX, and I’ll tell you two things:

  1. I wouldn’t want to have to use AWX and forgo the command line (but I know how to overcome the angst)
  2. I know a lot of people have been waiting for this to happen

Forget about my first point: that’s possibly just I, but I do mean it: Ansible without ansible-playbook on the CLI, seeing stdout move past, etc. wouldn’t feel right to me.

I’ve been kicking AWX’ tires quite a bit for several days, and I’ll say one thing: it really is very capable, and I will be recommending organizations take a closer look at it. If you know Tower you know AWX, but there are many who don’t know Tower.

Let me start with a few things I dislike, because it’s quite a short list:

  • documentation is basically what’s available for Ansible Tower, but some bits in that are just not available in AWX, or at least I cannot find them (e.g. What we need are docs for things like management, backups, etc. But that’ll hopefully be written in the course of time
  • installation is supported for either docker, OpenShift or Minishift. That’s it. (I had a bit of difficulty wrapping my head around the *shifts, but I got along with the docker install.)
  • the UI needs a huge screen to be usable and occasionally feels sluggish (possibly due to delayed reaction due to background architecture)

Now for the things which I like in AWX:

  • the API, the API, and the API. Honestly, these guys got most of this very right. All we see in the UI is available in the API. tower-cli is also very good
  • the UI which updates via Websockets
  • multiple authentication backends. (I’ve tested TACACS+ and LDAP; both work). Even so, AWX supports local users (yes, which can also be created via the API); there’s also Github, Google, and whatnot
  • some of the terminology is a bit funny, but I quickly got the gist of it, and it makes sense (project, jobs, templates, etc)
  • inventories. Lots of them. Dynamic, static, internal, from SCM.
  • SCM all over. AWX is basically something you can replace and it obtains all it needs from external sources (SCM and PostgreSQL)
  • Role Based Access Control for those who need it. Works pretty well. Give access to a template and user inherits required access to credentials, inventory, etc.
  • Credentials store. Hugely useful.
  • Webhooks (outgoing) as well as API trigger from incoming hooks. That’s how I’d use AWX to avoid having to click in the UI
  • Workflows. Neat. Like a mini CI/CD thing.
  • external logging (ELK, Splunk.etc.) though what I see going out in the logs is meh
  • Notifications galore. Why wasn’t my mqtt notifier implemented? :-)
  • Clustering and High-Availability.

This isn’t an introduction to AWX. It’s more me wanting to whet your appetite. I’ll be speaking about AWX very soon, and I’m already working on an AWX training. At the first talk which I’ll give in the Netherlands, at the NLUUG I’ll be diving into as good an overview as I can give in 45 minutes. With screen-shots & things.

Further reading:

View Comments :: Ansible :: 20 Oct 2017 :: e-mail

Other recent entries