Ben and Alexander recently twisted my arm until I promised to create something which would consume Mosquitto's $SYS/broker/# topic branch and write those values to collectd. You will know that that topic branch emits broker statistics via MQTT:

$SYS/broker/bytes/received 79665672
$SYS/broker/bytes/sent 27887950
$SYS/broker/load/messages/received/1min 70.10

My first thought was: let me add a service to mqttwarn for that (if you don't know mqttwarn you may be interested in these two articles introducing it), but for the systems it was to run on, it was to be a standalone thing in C. The first cut of the program was operational within a couple of hours: it is hooked into collectd via collectd's exec plugin which launches a long-lived process and reads metrics it issues from stdin.

It then occurred to me I could also handle JSON payloads easily enough, extracting an element's value from the JSON to use as a number. One thing led to another, and I then wanted elements from the JSON payload to be interpolated into the metric names collectd is given, so I added that as well. The result is a rather fast minimal mqttwarn which hands metrics to collectd.

An ini-type configuration file provides the settings required to run mqttcollect, instead of a dozen command-line options. Hostname, port, CA certificate file for TLS, TLS-PSK are all supported. The nodename given to collectd is configurable (it defaults to the short uname) as is an optional prefix which is, well, prefixed to a metric to differentiate instances of a plugin.

[defaults]
host = localhost
port = 1883
username = jane
password = s1c#ret
; psk_key =
; psk_identity =
; ca_file =
; nodename = foob
; progname = mqttcollect
; prefix   = PREFIX

You configure any number of topics which mqttcollect will subscribe to, and you specify the type of metric as well as whether or not the MQTT topic should be translated before being handed off. There are three possibilities:

  1. No translation. E.g a topic temperature/arduino is passed through unchanged.
  2. Rewrite a topic (e.g. temperature/arduino) to hot/in/here.
  3. Rewrite with JSON interpolation.

The first two mechanisms are described in the mqttcollect.ini.example file we provide. The third is a bit more difficult to grasp, so let me elaborate.

metric configuration

Assume for instance I subscribe to the wildcarded topic branch arduino/temp/+ and that the payload of messages which are published on that branch contains JSON with a celsius and fahrenheit temperatures as well as the name of the room in which the temperature was measured.

{"fahrenheit": 53.26, "celsius": 11.81, "room": "kitchen"}

I can have mqttcollect use each of the elements in the JSON (e.g. celsius, room, etc.) to construct the name of a metric. So, if I configure, say, a metric name of heat.{room} and, as shown above, the JSON payload has a { "room" : "kitchen" } in it, the metric name will be rewritten to heat.kitchen.

You'll notice the < character followed by a word; this indicates that mqttcollect should retrieve the value of the metric from said JSON element. So, for example, <celsius means: the value of the metric should be taken from the payload's { "celsius" : 11.81 } element. Likewise, <fahrenheit would consume that value, but who'd want that? ;-)

Putting this together, if I configure mqttcollect with this section:

[arduino/temp/+]
gauge = heat.{room}<celsius

mqttcollect will rewrite the metric name from the JSON payload and obtain the value for the metric from the same JSON payload, handing collectd the following line:

PUTVAL tiggr/mqttcollect/gauge-heat.kitchen 1431548550:11.81

A practical example we're using this for is for our OwnTracks Greenwich devices which publish JSON payloads like this example (shortened for clarity):

{
    "_type": "location",
    "alt": 53,
    "tid": "BB",
    "vel": 62
}

mqttcollect will, when configured as below, produce three metrics per message it receives.

[owntracks/+/+]
gauge = vehicle/{tid}/speed<vel
gauge = vehicle/{tid}/altitude<alt
counter = vehicle/{tid}/odometer<trip
PUTVAL tiggr/mqttcollect/gauge-vehicle/BB/speed 1431543655:62.00
PUTVAL tiggr/mqttcollect/gauge-vehicle/BB/altitude 1431543655:53.00
PUTVAL tiggr/mqttcollect/counter-vehicle/BB/odometer 1431543655:672798.00

This lets us produce nice graphs with all sorts of useless information, such as at which altitude vehicles are driving at using InfluxDB and Grafana.

Viewed in Grafana

I've put the source code to mqttcollect up on this repository.

View Comments :: monitoring and MQTT :: 15 May 2015 :: e-mail

When we started using Slack in our small team I was sold on the spot, and I particularly enjoy being able to build integrations which, well, integrate seamlessly into Slack: type a command into a Slack channel, and get a response right there, without leaving Slack.

I was thinking about how to go about getting a bit of our OwnTracks into Slack. Wouldn't it be neat if you could tell with a single command where a team mate is? Sure, we could glance at a map or use one of the other utilities that the OwnTracks project provides, but let's see if we can do better.

Slack command in action

If you've been following a bit of what we do with OwnTracks, you'll know privacy is a big word for us, and as such we recommend you use your own MQTT broker for OwnTracks. I have a dozen friends and family members using my broker with carefully designed ACLs so that only consenting parties can see each others' location. I don't want to open up MQTT publishes from my broker to the rest of the world, but I want my Slack team to be able to determine where I am so they know if I can be disturbed. (I trust these guys.)

OwnTracks maps

What I'm going to show you is how we can use a combination of OwnTracks, mqttwarn (which I introduced here), and a Slack slash command to produce a useful utility you can use in Slack as well. (If you're more of an IRC user, I'm sure you can use some of these techniques to write a bot which will do similarly, and do tell me about how you solved it!) The difference between a slash command in Slack and other integrations such as Webhooks is that only the person who issued the command sees the response, so it doesn't clutter the channel.

what we build

My smart phone publishes a location update via MQTT to my broker, from which mqttwarn picks it up, reformats the payload, and performs a HTTP POST to an external system on which a Slack integration can obtain the data.

The payload published by OwnTracks over MQTT is JSON which contains the latitude, longitude, altitude, etc. of the smart phone, as well as a time stamp, and a tracker-ID called TID which I configure on the device itself. The payload looks like this (I've removed some of the elements to keep the example brief):

{
  "_type": "location",
  "tid": "jJ",
  "tst": 1430493519,
  "lon": "2.295134",
  "lat": "48.858334"
}

mqttwarn subscribes to MQTT messages published at the topic used by my phone, and determines whether the message qualifies to be handled. The filter function can drop messages I'm not interested in handling even though they're published to the same topic (e.g. lwt messages when the broker detects the device has temporarily gone offline).

I also provide a custom function in slackfuncs.py called slack_whereis() which will extract latitude and longitude from the OwnTracks payload and perform a reverse-geo lookup.

[defaults]
functions = 'slackfuncs'
launch   = log, http

[config:http]
timeout = 60
targets = {
   'slack_whereis' : ['post', 'https://example.org/whereis/user/jjolie', None, None],
  }

[owntracks-Slack]
topic = owntracks/jjolie/+
filter = slack_whereis_filter()
alldata = slack_whereis()
targets = http:slack_whereis
format = reported last at {_dthhmm} from {geo}

So, if OwnTracks publishes the JSON shown above to the topic owntracks/jjolie/nex, say, mqttwarn, which is subscribed to that topic, will receive and handle the message and, via the slack_whereis() function, will add a geo element which mqttwarn can use in format when finally sending the message to the configured target.

In this case, I use the http target to POST data to a remote endpoint where the association (Jane is at this place) is stored.

The HTTP endpoint is a small Bottle app which accepts these location update POSTs on the one hand, and which is an endpoint for a Slack command on the other.

whereis.py

Slack slash commands

There's pretty little that the guys over at SlackHQ forgot to implement, and one of my favorites is

/remind me in 3 days to tell bbucks to push changes to master soon

reminder

Slack allows me to add my own custom slash commands which I configure in Slack integrations. What I'm going to do here is to create a /whereis command so that my mates can see where I am (and maybe I can see where they are).

custom command

Adding a custom command requires two steps:

  1. add the Slack integration for the custom command
  2. create the code which will actually run the command

Adding the command integration is easy: choose the name of the command (/whereis), the HTTP endpoint, and a usage hint for the user. I make a note of the token which I can use in my code to ensure only clients which know the token can query my service.

When I invoke a custom command, Slack fires off a POST (or optional GET) request to the endpoint we configure. The example I show you here implements a simple solution for the task: use the value passed to /whereis to find information about a particular user and return a one-line text result which Slack displays au-lieu of the command I entered. (I run this particular app under nginx and UWSGI.)

#!/usr/bin/python
# Jan-Piet Mens, May 2015.  Slack slash command. (/whereis user)

import bottle   # pip install bottle
from persist import PersistentDict # http://code.activestate.com/recipes/576642/

botname = 'owntracks'

path = '/home/jpm/slack-whereis/db/userlist.json'
userlist = PersistentDict(path, 'c', format='json')

app = application = bottle.Bottle()

@app.route('/', method='POST')
def slack_post():
    body = bottle.request.body.read()

    token           = request.forms.get('token')          # yJxxxxxxxxxxxxxxxxxxxxxx
    team_id         = request.forms.get('team_id')        # T00000000
    team_domain     = request.forms.get('team_domain')    # example
    service_id      = request.forms.get('service_id')     # 0123456789
    channel_id      = request.forms.get('channel_id')     # C01234567
    channel_name    = request.forms.get('channel_name')   # general
    timestamp       = request.forms.get('timestamp')      # 1428242917.000011
    user_id         = request.forms.get('user_id')        # U10101010
    user_name       = request.forms.get('user_name')      # jpmens
    text            = request.forms.get('text')           # <free form>
    trigger_words   = request.forms.get('trigger_words')

    if token != '1xyfx098xelxRxk913x01234':  # integration token
        return "NOPE"

    # text contains the username (or it is empty)
    who = text.lower().rstrip()
    if who == "" or who is None:
        return "Who?"

    if who in userlist:
        response = "%s %s" % (who, userlist.get(who, 'dunno'))
    else:
        response = "I haven't a clue where %s is." % (who)

    return response

# curl -d 'is lying on the beach...' https://example.org/whereis/user/jjolie

@app.route('/user/<username>', method='POST')
def user(username):
    text = bottle.request.body.read()

    userlist[username] = text
    userlist.sync()

    return ""

if __name__ == '__main__':
    bottle.run(app, host='0.0.0.0', port=80)

If I'm very concerned about privacy, I can add a bit of haversine magic or use the OwnTracks waypoints feature on the location reported by OwnTracks to keep things anonymous even from trusted team mates:

at home

For example, I can hide at which restaurant customer I am, etc.

If this has inspired you to build something with OwnTracks, we'd love to know; talk to us.

View Comments :: OwnTracks and Slack :: 02 May 2015 :: e-mail

It must have been over a year ago that somebody mentioned Jenkins is not just a tool for developers; system administrators can also put it to good use. I recall glancing at it and subsequently forgot about it. Anyway, I'm probably the very last person on Earth to learn this.

During Loadays in Antwerp last weekend, Fabian Arrotin mentioned this again, and I convinced him (in exchange of a beverage or two at the hotel bar) to show me. He demonstrated how he schedules Ansible jobs to run from within Jenkins, and the coin clicked as I watched what he did.

Job configuration

In the course of today's lunch break, I set up Jenkins on my laptop, installed a couple of plugins and everything I tried just worked. Fabian showed me the color plugin and mentioned the console column plugin which allows me to open the last console output at the click of a button from the dashboard.

Dashboard

Within a very few minutes I was kicking Ansible playbook-runs by checking something into a git repository; a commit hook kicks off a Jenkins build.

Console output

I think for things like Ansible, scheduled "builds" (think cron) will be tremendously useful, in particular because I can browse through prior build history etc. Within the same lunch break I had the MQTT sending me notifications of failed builds via mqttwarn.

I'm getting addicted in spite of its UI, and I can't wait for the weekend to read more about what Jenkins can do.

View Comments :: Sysadmin, Ansible, and Jenkins :: 16 Apr 2015 :: e-mail

One day after giving a one-hour presentation on what Ansible is capable of, "colleagues" flocked into my office and wanted to see stuff happen, so I showed them each a few odds and ends, in particular how Ansible can template out configuration files. I don't think I exaggerate when I say that I think I saw tears of joy come to somebody's eyes. Lovely. Anyhow, just a few days later, I was asked to find a solution for managing the creation (and destruction) of a potential boatload of DNS zones on a rather large number of PowerDNS servers.

I whipped up an Ansible module to create, delete, and list master or slave zones on authoritative PowerDNS servers with enabled REST API.

Unfortunately I had to resort to using urllib2 instead of Requests because I must not touch (i.e. install anything on) these machines. Thanks to James' comment below, I use Ansible's built-in fetch_url(). The pdns_zone module is very new, but it seems to do its job.

Create a master zone

In order to create a master zone, I invoke the module like this:

- hosts:
  - t1.prox
  gather_facts: False
  tasks:
  - action: pdns_zone name="ansi.test" action=master
            soa="ns.example.net hostmaster.example.com 1 1800 900 604800 3602"
            nsset="ns.example.net,ns.example.org"

The API then adds the following records to the records table:

mysql> SELECT * FROM records WHERE domain_id = (SELECT id FROM domains WHERE name = 'ansi.test');
+-------+-----------+-----------+------+--------------------------------------------------------------+-------+------+-------------+----------+-----------+------+
| id    | domain_id | name      | type | content                                                      | ttl   | prio | change_date | disabled | ordername | auth |
+-------+-----------+-----------+------+--------------------------------------------------------------+-------+------+-------------+----------+-----------+------+
| 16280 |        50 | ansi.test | SOA  | ns.example.net hostmaster.example.com 1 1800 900 604800 3602 | 86400 |    0 |        NULL |        0 | NULL      |    1 |
| 16281 |        50 | ansi.test | NS   | ns.example.net                                               | 86400 |    0 |        NULL |        0 | NULL      |    1 |
| 16282 |        50 | ansi.test | NS   | ns.example.org                                               | 86400 |    0 |        NULL |        0 | NULL      |    1 |
+-------+-----------+-----------+------+--------------------------------------------------------------+-------+------+-------------+----------+-----------+------+
3 rows in set (0.00 sec)

I can specify options to control how the module connects to the API, but by default it obtains these settings from the pdns.conf file. (See the module documentation.) Simultaenously, the comments table is also modified via the API (even though I'm still not quite understanding the use of this; maybe somebody can help me see that):

mysql> SELECT * FROM comments WHERE domain_id = (SELECT id FROM domains WHERE name = 'ansi.test');
+----+-----------+-----------+------+-------------+---------+-----------------+
| id | domain_id | name      | type | modified_at | account | comment         |
+----+-----------+-----------+------+-------------+---------+-----------------+
| 27 |        50 | ansi.test | SOA  |  1429114613 |         | Ansible-managed |
+----+-----------+-----------+------+-------------+---------+-----------------+

Peter gave me an interesting use-case for the per/RRset comments in PowerDNS: people can add, say, issue-tracking numbers to the records' comment in order to document how a record came to exist respectively why it was updated. It's an interesting use-case, but it doesn't cater for deletions... ;-)

Create a slave zone

Setting up a slave zone is very similar; the API modifies the domains table and, as shown above, the comments table.

- name: Create slave zone
  action: pdns_zone zone="example.org"
          action=slave
          masters="127.0.0.2:5301"
mysql> SELECT * FROM domains WHERE name = 'example.org';
+----+-------------+-----------------+------------+-------+-----------------+---------+
| id | name        | master          | last_check | type  | notified_serial | account |
+----+-------------+-----------------+------------+-------+-----------------+---------+
| 51 | example.org | 127.0.0.2:5301  |       NULL | SLAVE |            NULL | NULL    |
+----+-------------+-----------------+------------+-------+-----------------+---------+

Deleting a zone requires specifying action=delete, and it's removed from the back-end database. In the case of deletion of a master zone, all records are purged with the zone proper.

List zones

We can use the module to enumerate zones and their types (a.k.a. "kind"). As a special case, when we list zones, we can specify a shell-like glob which will match on names of zones. Consider this Ansible playbook and the associated template:

- hosts:
  - t1.prox
  vars:
  gather_facts: False
  tasks:
  - name: List existing .org zones
    action: pdns_zone action=list zone=*.org
    register: zl

  - name: Create report
    local_action: template src=a-zlist.j2 dest=/tmp/rep.out
{% for z in zl.zones | sort(attribute='name') %}
{{ "%-20s %-10s %s"|format(z.name, z.kind, z.serial) }}
{% endfor %}

The output produced looks like this:

e5.org               master     2015012203
example.org          slave      0

I think the list function is very practical as it allows me to connect to an authoritative server via SSH to enumerate zones, then turn around towards a second authoritative slave server (also via SSH) and create corresponding slave zones. (This is what you'd probably typically do with the PowerDNS superslave capability.)

pdns_zone

The diagram illustrates this: from our management console, we use Ansible via SSH to connect to one server, and use the obtained list of zones to create, via Ansible and the same module of course, appropriate slave zones on a second server. (If this doesn't make terribly much sense to you, you have my full understanding; trust me: it must be done this way in this particular case, if only because the machines have SSH access only.)

- hosts:
  - t1.prox
  gather_facts: True
  tasks:
  - name: List existing zones on main PowerDNS server
    action: pdns_zone action=list zone=*.org
    register: zl

- hosts:
  - deb.prox
  gather_facts: False
  tasks:
  - name: Create slave zones on secondary PowerDNS server
    action: pdns_zone zone={{item.name}}
                action=slave
                masters="{{ hostvars['t1.prox'].ansible_default_ipv4.address }}:5301"
                api_host=127.0.0.1
                api_port=8083
                api_key="ohoh"
    with_items: hostvars['t1.prox'].zl.zones

The JSON which is returned in the list command looks like this, with kind forced to lower case:

{
  "zones": [
    {
      "serial": 2015012203,
      "name": "e5.org",
      "kind": "master"
    },
    {
      "serial": 0,
      "name": "example.org",
      "kind": "slave"
    }
  ]
}

Now, if only the PowerDNS BIND back-end could be driven thusly, hint, hint ;-)

If this has piqued your interest, I've made the code and a few examples available in the pdns_zone module repository.

View Comments :: PowerDNS and Ansible :: 15 Apr 2015 :: e-mail

I've been doing a lot of work with and testing of the PowerDNS authoritative DNS server lately, and I must say I quickly tire at having to create new zones in its MySQL back-end database. Yes, I can and do use the PowerDNS API or nsedit for that as well as trivial shell scripts, but I remain an aficionado of command-line utilities such as cp and vi for zone file maintenance.

For some reason, and even though I describe that first in my book, I've been neglecting PowerDNS' so-called bind back-end (a misnomer in my opinion; it could equally well have been called the nsd back-end :-). Configured to use the bind back-end, PowerDNS reads zone master files directly off the file system without requiring a heavy-duty relational database system.

PowerDNS with the bind back-end runs in one of two modes:

  1. a hybrid mode in which it stores DNSSEC-related configuration in a separate back-end (e.g. MySQL or PostgreSQL)
  2. a non-hybrid mode in which PowerDNS uses a compiled-in version of SQLite in which to store DNSSEC-related configuration and metadata.

I will discuss this second form as it avoids large "moving parts", i.e. we don't require a relational database alongside the DNS server.

PowerDNS bind back-end

Let's assume the following configuration in /etc/powerdns/pdns.conf:

launch=bind
master=yes
slave=yes
security-poll-suffix=
bind-dnssec-db=/etc/powerdns/bind/dnssec.db
bind-config=/etc/powerdns/bind/named.conf
bind-check-interval=600

When PowerDNS launches, it checks its configuration and loads the zones enumerated in bind-config from master zone files just like BIND and NSD do. It obtains the names and types of the zones it should serve from a named.conf - like file, but it requires only a minuscule subset of BIND's directives. Basically the following suffices to configure one master and one slave zone.

options {
    directory "/etc/powerdns/bind";
};

zone "example.aa" IN {
    type master;
    file "example.aa";
};

zone "ww.mens.de" IN {
    type slave;
    masters { 192.168.1.10; };
    file "ww.mens.de";
};

PowerDNS starts up very quickly even with a very large number of zones. Slave zones are transferred in, but the file specified for the slave zone must exist and be writeable for PowerDNS or it won't transfer (AXFR) the zone to disk but a misleading diagnostic message is logged at first try.

Furthermore, there is no need to reload the server when a zone is added or removed: simply change the file pointed to by bind-config, and PowerDNS will pick this up every bind-check-interval seconds or explictly when you invoke

pdns_control rediscover

One really brilliant feature of the PowerDNS bind back-end is, I can edit one of the zone master files on disk (vi, etc.), and PowerDNS will pick up that change within a second without me having to do anything. (It checks the file's modification time once per second when queried for a zone and reloads it on change.)

PowerDNS has to store DNSSEC-related data and metadata for a zone somewhere; zone master files don't cater for that. In particular the keys (respectively pointers to keys on a HSM) must be available, and the server uses the configured bind-dnssec-db for doing that. This database contains the domainmetadata, cryptokeys, and tsigkeys tables. In other words, if I create DNSSEC keys and associate those with a zone, PowerDNS looks for that data in this database which we create before first launching PowerDNS:

pdnssec create-bind-db /etc/powerdns/bind/dnssec.db

I'm fond of hand-crafting zone master files, but there are cases in which automation is necessary. Unfortunately, the bind back-end has neither support for RFC 2136 dynamic updates (even though PowerDNS has this for some back-ends), nor does it support the REST API. (I thought I could lure one of the developers to bite, but my feeble attempt at an April fool's joke wasn't taken seriously ... even though I think having both these features in the bind back-end a very good idea. ;-)

In short: on the plus side:

  • Fewer moving parts (no relational database management system)
  • Fast
  • Zone master files are immediately reloaded when modified
  • DNSSEC support
  • Did I say "fast"?

On the minus side:

  • Neither the PowerDNS REST API (which is built-in to the server so it could be utilized) nor, and this is very minus, RFC 2136 updates which the server is also capable of doing.
  • Incoming AXFR fails if the zone file doesn't exist; the server could, permissions permitting, create this itself.

All in all, this combination could become one of my favorites...

View Comments :: DNS, PowerDNS, and BIND :: 02 Apr 2015 :: e-mail

Other recent entries