Jan-Piet Mens

A few notification modules for Ansible

2013-05-08T20:23:45+02:00

Brian Coca recently submitted a jabber module for use with Ansible. This module allows an Ansible playbook to connect to an XMPP service and just push out a message, such as "all's well" or "I'm done", when, say, a Playbook completes. In addition to the `mail' module (created by Dag Wieers and heavily improved upon by yours truly :-), I thought I'd add two more.

The first is a module which fires off a message to an IRC channel.

That was posted to the channel by this play:

---
- hosts: all
  tasks:
  - local_action: irc port=6669 
         channel="#t1"
         msg="Server {{ ansible_hostname }} completed at {{ ansible_date_time.iso8601 }}"
         color=green
         nick=ansibleIRC

The second (very similar) module I created works similarly, but notifies via MQTT. MQTT is the new hotness in the Internet of Things (IoT), and we talked about it here recently. Using this module, Ansible can "notify" by publishing an arbitrary payload to an MQTT topic.

That subscription was published to by this play:

---
- hosts: all
  tasks:
  - local_action: mqtt
                  server=localhost
                  port=1883 
                  topic=service/ansible/{{ ansible_hostname }}
                  payload="Server {{ ansible_hostname }} done at {{ ansible_date_time.iso8601 }}"
                  qos=0
                  retain=false
                  client_id=ans001

(Publishing topics to an MQTT broker would also be a neat way of watching Ansible at work with a callback; maybe another time.)

Live blinkenlights dashboard with SVG

2013-04-22T09:56:52+02:00

Blinking dashboards turn me on. Really. I love seeing things move and blink.

Andrew Hazelden's post on monitoring a solar hot water system using a dashboard created in SVG made a deep impression when I saw it, and I've been meaning to experiment with that. As I've been doing things with MQTT, I thought I'd finally give it a try, swiping bits of code created by Robert Hekkers.

What I have is an SVG file which can be updated in real-time using socket.io. A Web browser loads an HTML resource from a Web server. A bit of JavaScript in the page connects to a socket.io process and awaits data. On the other side, we push data into that socket, and the JavaScript tweaks the SVG to make things do "blinkenlights".

The SVG is scalable (the 'S' in SVG'), so I can zoom into it:

I drew the SVG with Inkscape, using its "XML Editor" to change object identifiers to something that makes sense to me. I'll be using the second level of the MQTT topic (e.g. svgdash/smtp0) to name the SVG object.

Furthermore, in the XML, I specify a type for the object: "num" will be a numeric value, and "slider" is a bar with a maximum width of "maxw", which will be displayed to show a particular level. The following example shows a "slider" for an object called "smtp0" (the left-most SMTP server in the SVG). Note the "maxw" and "type" elements I added manually

    <rect
       style="fill:#e0e5de;....."
       id="smtp0"
       width="35.75985"
       height="10.547414"
       x="213.35257"
       y="640.42432"
       inkscape:label="#rect3549"
       type="slider"
       maxw="35" />

An MQTT client publishes to a specific topic, with a JSON payload containing something like this:

{
 "msg" : "ok",
 "status" : 0,
 "level" : 4193
}

The "msg" is a string which should probably be displayed on the SVG object. (Here I show it on the top left.) "status" is a Nagios/Icinga-type status (OK, WARNING, CRITICAL). "level" is used to fill objects of type "slider" or to post literally as text on objects of type "num". You get the idea. I hope.

If I add a component to the network, I simply choose a new topic, update the SVG, refresh the Web page, and that's it. Nice. Of course, if I were half a designer/developer, I could use something like the beautiful Raphaël to do this "electrically".

The code I have is up for grabs, so do feel free to improve on this. Enjoy!

Update: watch the blinkenlights

Curl test program for iOS

2013-04-13T18:36:09+02:00

iCurlHTTP is a nifty little program for iOS, which will shoot off an HTTP GET or HEAD request.

Nice for testing.

via Daniel Stenberg

Managing SSH Keys for Automated Access

2013-04-10T16:35:32+02:00

You are a UNIX/Linux administrator, and you use SSH? Print this out, get yourself a nice drink, make yourself comfortable, and read Managing SSH Keys for Automated Access - Current Recommended Practice.

Do it now.

Egosurfing

2013-03-22T17:50:44+01:00

He's talking about this book. :)

JSON in BIND 9's statistics-server

2013-03-18T09:56:27+01:00

I recently showed a customer who makes heavy use of the BIND name server, that he could access BIND's statistics via the statistics-server. We weren't really amused when we noticed that producing the XML on a server with a lot of zones took several minutes and delivered a whopping fifty megs of data.

It occurred to me that this could be streamlined (produce less data and consume less time) if BIND would output its information as JSON. I proposed the idea and a patch and received very positive feedback from BIND users (a lot of +1s in my inbox) and also from ISC.

Evan Hunt talked to me, re-implemented my patch correctly, and added some very good ideas. And I have some extremely good news: BIND's statistics server will provide JSON (in addition to optional XML) starting with BIND 9.10!

What follows is a preview (careful! This could change until the release) of what the JSON output looks like.

The plan is to have different URIs which will provide just the information I'm interested in. This is different to the XML statistics, which are output as one huge data set. The URIs currently implemented (in the version I've seen) are

/json
/json/server
/json/zones
/json/tasks
/json/net
/json/mem

So, for example, the /json URI will print everything:

{
  "json-stats-version":"1.0",
  "boot-time":"2013-03-18T08:53:32Z",
  "current-time":"2013-03-18T09:09:58Z",
  "opcodes":{
    "QUERY":1
  },
  "qtypes":{
    "A":1
  },
  "nsstats":{
    "Requestv4":1,
    "Response":1,
    "QrySuccess":1,
    "QryNoauthAns":1,
    "QryRecursion":1
  },
  "zonestats":{
    "SOAOutv4":30,
    "AXFRReqv4":2,
    "XfrFail":2
  },
  "views":{
    "_default":{
      "zones":[
        {
          "name":"b.aa",
          "class":"IN",
[...]

The /json/zones URI will produce the list of views and zones only:

{
  "json-stats-version":"1.0",
  "boot-time":"2013-03-18T08:53:32Z",
  "current-time":"2013-03-18T09:14:23Z",
  "views":{
    "_default":{
      "zones":[
        {
          "name":"b.aa",
          "class":"IN",
          "serial":1
        },
[...]

etc. I think this is really a useful addition to the statistics-server, because parsing JSON is much faster than parsing XML.

As I mentioned, this may still be a moving target, but I thought you might want to know what's on the horizon for BIND9's statistics server.

Knot-DNS: Dynamic Updates and a bit of RRL

2013-03-12T08:25:02+01:00

A year has passed since I last looked at the Knot DNS server, so it's time to re-visit it; a lot has happened, and there are two features I want to single out: support for RFC 2136 (dynamic DNS updates) and a bit of RRL (Response Rate Limiting).

Dynamic DNS

Knot protects Dynamic DNS Updates to master zones with TSIG keys I create and copy into knot.conf:

...
keys {
  jpkey hmac-md5 "U/B55N6G9M8=";
}
...
remotes {
  tiggr { address 172.16.153.1@53; key jpkey; }
  anyclient { address 0.0.0.0/0;  key jpkey; }
}
...
zones {
  example.com {
    file "example.com.zone";
    update-in tiggr;
    zonefile-sync 5m;
  }
...
}

The TSIG key (named jpkey here) is associated with a remote server (or update client) and I specify the name of the remote which is allowed to update a zone with the update-in (i.e. allow IN-coming updates) statement within the zone statement.

Updates received to a master zone will be handled by the server and written out to the source zone file when the zonefile-sync period kicks in. Knot will forward updates received for a slave zone to the zone's primary master server, which I specify in an xfr-in directive.

The server logs update requests, as follows:

09:01:19.704047+01:00 UPDATE of 'example.com.' from '172.16.153.1@52480' key 'jpkey.': Started.
09:01:19.704361+01:00 UPDATE of 'example.com.' from '172.16.153.1@52480' key 'jpkey.':: Finished.

There are currently a number of limitations regarding DNS updates on DNSSEC-signed zones (in particular: Knot can't re-sign records), so keep an eye on the documentation!

Response Rate Limiting

Running dnsperf against my test installation (running in my portable data center, so please don't pay too much attention to the actual numbers), I obtain the following results (output abbreviated):

  ...
  Run time limit:       30 seconds
  Ran through file:     88305 times
  Queries sent:         88305 queries
  Queries completed:    88305 queries
  Queries lost:         0 queries

  Queries per second:   2943.478807 qps

Latency   Success  Fail |
<    0.100s 88305     0 |############################################################
...

As of version 1.2.0, response rate limiting is compiled into Knot, but it is disabled by default. I enable it by configuring the rate-limit option in the system section, for example like this:

system {
  storage "/etc/knot";
  user: "root";

  rate-limit 100;
  rate-limit-slip 2;
}

After restarting Knot with RRL enabled as above, I run dnsperf again to obtain these quite different numbers:

[Timeout] Query timed out: msg id 2042
[...]
[Timeout] Query timed out: msg id 3236

  Run time limit:       30 seconds
  Ran through file:     3236 times
  Queries sent:         3237 queries
  Queries completed:    3118 queries
  Queries lost:         119 queries

  Queries per second:   91.900425 qps

Latency   Success  Fail |
<    0.100s  3118     0 |############################################################
...

Looking only at Queries per second and Queries completed, we see Knot has indeed limited the rate of responses, and the server logs the fact that it is engaging RRL mode:

08:23:16.774694+01:00 [notice] Address '172.16.153.1' leaves rate-limiting (class 'POSITIVE').
08:23:16.877744+01:00 [notice] Address '172.16.153.1' enters rate-limiting (class 'POSITIVE').

Keep an eye on the mailing list for more things to come.

Visualizing energy consumption with MQTT

2013-03-10T18:21:39+01:00

The two spikes you see in the graph below were produced during lunch (a very good Carbonara sauce on bavette) and later while making popcorn.

I've fallen for MQTT and Mosquitto, and I thought it would be practical to grab the current electricity consumption from the Youless and hand it around via MQTT. Unfortunately, Youless doesn't (yet) support MQTT, so I had to obtain the current consumption it with a small Python program which polls the Youless and PUBlishes the current value to my MQTT broker:

#!/usr/bin/env python

import requests
import json
import mosquitto
import time, sys

URL = "http://192.168.1.9/a?f=j"

def energy(url):

    pwr = 0
    status = 0

    r = requests.get(url)
    if r.status_code != 200:
        return dict(status=1)

    js = json.loads(r.text)
    if 'pwr' in js:
        pwr = js['pwr']

    return dict(pwr=pwr, status=0)

if __name__ == '__main__':

    mqttc = mosquitto.Mosquitto()
    mqttc.connect("hippo", 1883, 60)

    e = energy(URL)
    while e['status'] == 0:
        print e['pwr']

        try:

            string = json.dumps(e)
            string = e['pwr']
            mqttc.publish('hw/youless', string , 0)
            mqttc.loop()

            time.sleep(15)
            e = energy(URL)
        except KeyboardInterrupt:
            sys.exit(1)
        except:
            raise

This works well, but I wanted to graph the data (and I still haven't had the time to properly set up Graphite hereabouts, but it's coming: here a small example to whet your appetite :-)

Cosm, the artist formerly know as "Pachube" provides a beta MQTT service which allows me to publish data into a feed. (The Mosquitto site has a good quick-start guide.)

Nick O’Leary had been there and done that so I followed his recommendation and added a bridge to Mosquitto which translates the topic name I used and bridges that into the Cosm data-stream. This is my Mosquitto configuration:

connection pachube_yl
address api.cosm.com:1883
clientid n001
username 000000000000000000000000000000000000000000000000
topic "" out 0 hw/youless /v2/feeds/118350/datastreams/0.csv

As you can see from the screen-shot above, it works like a charm. (I must say though, that the Cosm's MQTT service has been a bit flaky from here: connections sometimes take ages to make; at other times, the data just doesn't arrive there.)

I've already contacted the guys at Youless to suggest it might well be worth their while to implement optional MQTT support in their firmware. I'm keeping my fingers crossed.

Processing BIND's XML statistics with XSL

2013-03-01T13:37:57+01:00

When last discussing using the XML produced by BIND's statistics server, I showed you how I created a C program to parse the XML, but it didn't occur to me to use XSL and the xsltproc utility to do so.

BIND9 old XML format

A kind reader added a comment to that post a few moments ago, and I took the liberty of streamlining his code a bit to produce this:

<xsl:stylesheet version="1.0" xmlns="http://www.w3.org/1999/xhtml" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<!--
  ** XSLT to parse BIND9 XML from the statistics-server (old format)
  ** inspired by a comment by Andrii Shynkarchuk on 
  ** http://jpmens.net/2010/10/21/using-binds-statistics-server-to-list-zones-and-axfr-the-list/
  **
  ** Set the view name you're interested in where it says "_default"
  ** (This could be accessed with an XSLT param or so.)
  -->
<xsl:output method="text" />
  <xsl:strip-space elements="*"/>

  <xsl:template match="isc/bind/statistics">
    <xsl:apply-templates select="views/view">
    </xsl:apply-templates></xsl:template>
      <xsl:template match="views/view">
      <xsl:choose>
    <xsl:when test="name='_default'">
        <xsl:for-each select="zones/zone">
            <xsl:value-of select="serial" />
            <xsl:text> </xsl:text> 
            <!-- zone name only (w/o class/view) -->
            <xsl:value-of select="substring-before(name, '/')" />
            <xsl:text>&#xa;</xsl:text> <!-- newline -->
        </xsl:for-each>

    </xsl:when>
      </xsl:choose>
    </xsl:template>
</xsl:stylesheet>

If I run the result through xsltproc on a 240MB XML file containing 136,000 zones, I get results in 8 seconds on my laptop, which is pretty good.

curl -qs http://172.16.153.101:8053/ |
    xsltproc bconvert.xsl -
2012011108 example.com
0 0.in-addr.arpa
0 1.0.0.127.in-addr.arpa
0 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa
2011080707 example.org
97 big.aa
...

BIND9 newstats XML format

And this works for me for the "newstats" format, introduced experimentally in 9.9.3:

<xsl:stylesheet version="1.0" xmlns="http://www.w3.org/1999/xhtml" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<!--
  ** XSLT to parse BIND9 (version3) XML from the statistics-server (new format)
  ** inspired by a comment by Andrii Shynkarchuk on 
  ** http://jpmens.net/2010/10/21/using-binds-statistics-server-to-list-zones-and-axfr-the-list/
  **
  ** Set the view name you're interested in where it says "_default"
  ** (This could be accessed with an XSLT param or so.)
  -->
<xsl:output method="text" />
  <xsl:strip-space elements="*"/>

<xsl:template match="statistics"> 
    <xsl:apply-templates select="views/view">
    </xsl:apply-templates></xsl:template>
      <xsl:template match="views/view">
        <xsl:for-each select="zones/zone">
            <xsl:value-of select="serial" />
            <xsl:text> </xsl:text> 
            <xsl:value-of select="@name" />
            <xsl:text>&#xa;</xsl:text> <!-- newline -->
        </xsl:for-each>
</xsl:template> 
</xsl:stylesheet>

Thank you, Andrii. :)

Lots of messages: MQTT, Pub/Sub, and the Mosquitto broker

2013-02-25T11:28:37+01:00

Publish/Subscribe queues are fun and useful. I first learned about them when tinkering with Redis a while back. One big drawback of Redis' Pub/Sub is that Redis project refuses to add some form of transport layer security, which means anything and everything is transferred into and out of Redis unencrypted.

I recently discovered MQTT, the Message Queue Telemetry Transport protocol, which is

is a lightweight broker-based publish/subscribe messaging protocol designed to be open, simple, lightweight and easy to implement.

The MQTT protocol provides one-to-may message distribution over TCP/IP with a small transport overhead, and a method to notify parties of abnormal disconnection of a client.

I think of a Pub/Sub message queue as "Twitter for my network". Clients subscribe to be notified of incoming messages pertaining to specific topics, and other clients publish on those topics. A topic (think of it as a kind of channel) classifies messages. For example, I could have topics called nagios/mta, nagios/disk, test/jp/private, etc. Clients can subscribe to any number of topics, and may include wild-cards when subscribing (e.g. nagios/#). In the context of MQTT, messages are blobs of opaque data (UTF-8, i.e. binary safe) with a maximum size of 256MB.

MQTT can be used in low-bandwidth or unreliable network environments, on embedded devices (e.g. on an mbed with the mbed MQTT library, on an Arduino with the Arduino client for MQTT), from Lua, Python, Perl, etc.

What I find particularly interesting with this technology, is that it enables me to employ a single (secured) transport mechanism for all sorts of data. I don't have to pierce holes in a firewall to specifically allow this and that; a single TCP port suffices to transport all sorts of (smallish) stuff.

Mosquitto is an open source MQTT broker, which I chose over IBM's RSMB because source code of the latter is not freely available. Mosquitto is easy to install and deploy, and its documentation is more than adequate.

The Mosquitto broker supports TLS out of the box, and provides authentication either via username/password, pre-shared keys or TLS client certificates. Furthermore, Mosquitto has a simple ACL by which the broker administrator can configure which clients may access which topics. (Clients identify themselves by a name they specify upon connecting.)

A small Python program connects to the broker and subscribes to a few topics:

#!/usr/bin/env python

import mosquitto

def on_message(mosq, obj, msg):
    print "%-20s %d %s" % (msg.topic, msg.qos, msg.payload)

    mosq.publish('pong', "Thanks", 0)

def on_publish(mosq, obj, mid):
    pass

cli = mosquitto.Mosquitto()
cli.on_message = on_message
cli.on_publish = on_publish

cli.tls_set('root.ca',
    certfile='c1.crt',
    keyfile='c1.key')

cli.connect("hippo", 1883, 60)

cli.subscribe("dns/all", 0)
cli.subscribe("nagios/#", 0)

while cli.loop() == 0:
    pass

If I run that on a client, the program will sit there waiting for messages published to the dns/all or nagios/# topics. In a different window, I can use the mosquitto_pub utility to fire off a message:

mosquitto_pub -d -h hippo.ww.mens.de \
    --cafile root.ca \
    --cert c1.crt \
    --key c1.key \
    -q 0 \                             # quality-of-service
    -t nagios/n1 \                     # topic
    -m 'mail server 001 is broke'      # message

Whereupon the Python program displays:

nagios/n1 0 mail server 001 is broke

(You'll have noticed that the Python program publishes a "Thanks" on the pong topic; clients subscribed to that will receive it.)

Clients can optionally set a Will (in Python before the connect() call). This "last will and testament" (so to speak) is published by the broker when a client disconnects unexpectedly. This can be useful to notify of particular clients' deaths. In other words, if a client issues the following request before connecting, and it dies at some later point, the broker will publish the payload on this client's behalf.

cli.will_set('clients/w1', payload="Adios!", qos=0)

Bridge

Mosquitto can be configured as a so-called "bridge". I could imagine this being useful in, say, different data centers.

In a bridge configuration, Mosquitto is configured to pass certain topics in certain directions. For example, I could configure a bridge to notify a "central" broker for messages of topic #/important. A bridge connects to a broker like any other client and subscribes to topics which are then "imported" into the bridge; clients connected to said bridge can thus be notified for particular subscriptions from the "main" broker.

Mosquitto periodically publishes statistics which interested parties can subscribe to, e.g. for monitoring purposes.

$SYS/broker/version mosquitto version 1.1
$SYS/broker/clients/total 368
$SYS/broker/clients/active 91
$SYS/broker/clients/inactive 277
$SYS/broker/clients/maximum 368
$SYS/broker/messages/received 13358099
$SYS/broker/messages/sent 16381123
$SYS/broker/messages/dropped 414180
$SYS/broker/messages/stored 10806
$SYS/broker/messages/sent 16381123
$SYS/broker/messages/sent 16381123
$SYS/broker/bytes/received 761223497
$SYS/broker/bytes/sent 476065843
$SYS/broker/load/bytes/sent/1min 28745.93
$SYS/broker/load/bytes/sent/5min 15418.24
$SYS/broker/load/bytes/sent/15min 6980.69
[...]

The Mosquitto project has a test server you can use if you don't want to set up your own (just launch mosquitto_sub at it), and there's a nice-looking Web-based interface you can use (on your public MQTT server) at http://mqtt.io.

Wishlist

There are a couple of things which would be neat to see implemented:

Nagios and Icinga typically "go out" and get statii of services, unless I configure them to accept passive checks. How about a plugin or component for Nagios and Icinga which subscribes to specific topics? That would allow for blindingly fast notification of problems within the network, without having to await a check interval? (It's been a while since I did something with these monitoring processes, so this may be stupid, but it sounds good to me ...) A notification could be wrapped into a JSON object containing a Nagios exit code and the reason.
Logstash should support MQTT as an input filter. That would allow me to use this message broker for shipping logs. (Logstash already supports AMQP and ZeroMQ.) On the other hand, I could easily create a Mosquitto to Redis "converter", and use Redis as input to Logstash.

Does anybody feel like doing that? :-)