Tony Finch has created a gem of a utility called nsnotifyd. It's a teeny-tiny DNS "server" which sits around and listens for DNS NOTIFY messages which are sent by authority servers when they instruct their slaves that the zone has been updated and they should re-transfer (AXFR / IXFR) them. As soon as nsnotifyd receives a NOTIFY, it executes a shell script you provide. (This is a very welcome alternative to doing it in Perl, as I did when I wanted to be notified of new and changed KSK in a zone.)

The script you provide (Tony has a few examples in the repository) is passed the name of the notified zone, its SOA serial number and the address of the master authority server which emitted the NOTIFY. Whether or not you actually use that data is up to you, of course. nsnotifyd can be used for all sorts of things:

  • Creating backups (discussed below). (You could even rsync zone files to a remote location.)
  • Alerting humans or machines on zone changes.
  • Taking snapshots of PowerDNS databases on zone changes.
  • Read the zone, create reverse-DNS entries in another zone. Tony provides a script which helps with that.
  • Poor-man's bump-in-the-wire DNSSEC signer; see repository.

So, let's assume I wish to keep a Git repository with changes to a few zones. I create a repository, add empty zone files, and create the shell script nsnotifyd will be invoking.

#!/bin/sh

export GIT_AUTHOR_NAME='nsnotifyd'
export GIT_AUTHOR_EMAIL='nsnotifyd@example.com'
export GIT_COMMITTER_NAME=$GIT_AUTHOR_NAME
export GIT_COMMITTER_EMAIL=$GIT_AUTHOR_EMAIL

(
        cd /etc/nsd/backup || exit 2
        zone=$1 serial=$2 master=$3
        case $zone in
        (.) zone=root
        esac
        dig +noall +answer +onesoa +multiline @$master $zone axfr >$zone
        git commit -q -m "$zone IN SOA $serial" $zone
) 2>&1 |
logger -p daemon.notice -t $0 -s

I add an explicit notify statement to the zone stanza in NSD, and enable transfers from the address nsnotifyd runs on.

zone:
        name: "example.org"
        zonefile: "example.org.zone"
        allow-notify: 192.168.1.110 NOKEY
        request-xfr: 192.168.1.110 NOKEY
        provide-xfr: 192.168.1.110 NOKEY
        provide-xfr: 127.0.0.1 NOKEY
        notify: 127.0.0.1@5353 NOKEY

I then launch nsnotifyd which daemonizes itself.

$ nsnotifyd -4 -a 127.0.0.1 -p 5353 -s 127.0.0.1 /etc/nsd/nsnotifyd/jp2git.sh example.org

As soon as nsnotifyd starts up, it queries the SOA record of the specified zones, typically by querying the host's resolver. (I overrode that with the -s option to force it to query a particular name server.) It then sits back and waits until refresh time has elapsed, whereupon it will check whether the serial number has changed; if so, it launches our script.

nsnotifyd[5117]: example.org refresh at 2015-06-16 15:22:12 +0000

I then issue a notify using nsd notify (or Gavin Brown's pnotify). Following along in the file syslog logs to, we see:

nsnotifyd[5117]: example.org notify from 127.0.0.1/50028
nsnotifyd[5117]: example.org IN SOA 1434350317 unchanged
nsnotifyd[5117]: example.org refresh at 2015-06-16 15:24:45 +0000

I then actually update the zone. NSD sends out its NOTIFY, and nsnotifyd reacts:

nsnotifyd[5117]: example.org notify from 127.0.0.1/45669
nsnotifyd[5117]: example.org IN SOA 1434350318 updated; running /etc/nsd/nsnotifyd/jp2git.sh
nsnotifyd[5117]: example.org refresh at 2015-06-16 15:25:43 +0000

and our repository looks like this:

commit 23278840888ea80785e5c6efef8bdc3c5ec73842
Author: nsnotifyd <nsnotifyd@example.com>
Date:   Tue Jun 16 07:25:43 2015 +0000

    example.org IN SOA 1434350318

nsnotifyd is perfect for keeping backups of zones, particularly if you update them dynamically.

Upon arriving at our Frankfurt location, I see from the monitor panel at the reception that Jane is in the office, so I'll pop in to ask a question.

office panel

We've been able to do this for quite a while now with OwnTracks, but due to GPS location as reported by mobile phones being what it is, the accuracy is sometimes off by up to a thousand meters, and we wanted to be able allow you to pinpoint people more precisely. Instead of relying solely on the location your smart phone thinks you are at, small, and relatively inexpensive iBeacons can pinpoint you down to a few meters. OwnTracks for iOS has had support for iBeacons for a few releases now, and it works very reliably.

Blukii SmartBeacon/H

Pictured above is a blukii SmartBeacon /H which comes in many different form factors, battery or mains powered, small or large, etc.

Beacons use Bluetooth low energy to transmit a UUID (typically modifiable) together with user-defined major and minor numbers, and these allow us to identify, say, a particular room in a building or even a specific corner of a room. The major number can be used, say, to identify an office building, whereas you'd configure a beacon's minor number to identify a room within that building. Alternatively, if you don't want to bother with identifying rooms, you can e.g. set all beacons to have the same major and minor numbers.

Let's assume the office we're discussing has a few beacons. Let's further assume we do not want to track people within a particular room; instead we just want to capture whether an employee is in this particular office building. We can configure all beacons with the same UUID, and we will ignore the major and minor numbers. (How a beacon gets it's UUID, major, and minor set depends on the product.)

We define a UUID, say, DEADBEEF-ABBA-CDEF-1001-000000000001 which we assign to all beacons, and configure them accordingly. (This is what the Blukii utility (Windows only) looks like.)

Blukii configuration utility

What we then do is configure a waypoint within the iOS OwnTracks app. The values for latitude/longitude are irrelevant. What is important is the UUID separated from the name of the beacon (I chose *Main@WestWing here) by a colon. A beacon's major and minor are optionally concatenated to that string, also colon-separated.

OwnTracks beacon settings

Instead of painstakingly configuring this on the device proper, I prepare a small file called office.otrw (the .otrw extension is important), with the following JSON payload:

{
  "waypoints" : [
    {
      "tst" : 1432817332,
      "lat" : 52.0,
      "_type" : "waypoint",
      "lon" : 13.0,
      "rad" : 0,
      "desc" : "*Main@WestWing:DEADBEEF-ABBA-CDEF-1001-000000000001:0001"
    }
  ],
  "_type" : "waypoints"
}

I then either place that file on a Web server, or e-mail it as attachment to my colleagues who open that on their OwnTracks device, and presto: the device has the beacon monitoring regions configured.

From this point on, OwnTracks monitors all beacons with that particular UUID, and it will publish an enter or leave event whenever the device gets within range of a beacon or leaves it. Additionally, the device shows the event with a local iOS notification.

iBeacon notification on iPhone

We publish these events as JSON via MQTT to the MQTT broker the device is connected to, and from there, you consume the message and do as you please.

{
    "_type": "transition",
    "acc": 65,
    "desc": "*Main@WestWing",
    "event": "enter",
    "lat": 2.2222,
    "lon": 1.1111,
    "tid": "jp",
    "tst": 1433342520,
    "wtst": 1432817332
}

For instance, send an e-mail when a particular person leaves the building, publish a list of people who remain in the building on a monitor (as above), etc.

To summarize: OwnTracks can monitor beacons by configuring it either with:

  • a UUID only, in which case the app would report any beacon with that UUID, irrespective of its major/minor numbers.
  • a UUID with a major number: the app would report iBeacons with the specified UUID and exactly that major number
  • all three: the UUID, the major and minor numbers, in which case the app reports events on precisely that beacon.

The app recognizes the beacon typically within 10 seconds, which is a typical beacon-publishing frequency, and this is very good for presence detection.

View Comments :: OwnTracks and iBeacons :: 03 Jun 2015 :: e-mail

Ben and Alexander recently twisted my arm until I promised to create something which would consume Mosquitto's $SYS/broker/# topic branch and write those values to collectd. You will know that that topic branch emits broker statistics via MQTT:

$SYS/broker/bytes/received 79665672
$SYS/broker/bytes/sent 27887950
$SYS/broker/load/messages/received/1min 70.10

My first thought was: let me add a service to mqttwarn for that (if you don't know mqttwarn you may be interested in these two articles introducing it), but for the systems it was to run on, it was to be a standalone thing in C. The first cut of the program was operational within a couple of hours: it is hooked into collectd via collectd's exec plugin which launches a long-lived process and reads metrics it issues from stdin.

It then occurred to me I could also handle JSON payloads easily enough, extracting an element's value from the JSON to use as a number. One thing led to another, and I then wanted elements from the JSON payload to be interpolated into the metric names collectd is given, so I added that as well. The result is a rather fast minimal mqttwarn which hands metrics to collectd.

An ini-type configuration file provides the settings required to run mqttcollect, instead of a dozen command-line options. Hostname, port, CA certificate file for TLS, TLS-PSK are all supported. The nodename given to collectd is configurable (it defaults to the short uname) as is an optional prefix which is, well, prefixed to a metric to differentiate instances of a plugin.

[defaults]
host = localhost
port = 1883
username = jane
password = s1c#ret
; psk_key =
; psk_identity =
; ca_file =
; nodename = foob
; progname = mqttcollect
; prefix   = PREFIX

You configure any number of topics which mqttcollect will subscribe to, and you specify the type of metric as well as whether or not the MQTT topic should be translated before being handed off. There are three possibilities:

  1. No translation. E.g a topic temperature/arduino is passed through unchanged.
  2. Rewrite a topic (e.g. temperature/arduino) to hot/in/here.
  3. Rewrite with JSON interpolation.

The first two mechanisms are described in the mqttcollect.ini.example file we provide. The third is a bit more difficult to grasp, so let me elaborate.

metric configuration

Assume for instance I subscribe to the wildcarded topic branch arduino/temp/+ and that the payload of messages which are published on that branch contains JSON with a celsius and fahrenheit temperatures as well as the name of the room in which the temperature was measured.

{"fahrenheit": 53.26, "celsius": 11.81, "room": "kitchen"}

I can have mqttcollect use each of the elements in the JSON (e.g. celsius, room, etc.) to construct the name of a metric. So, if I configure, say, a metric name of heat.{room} and, as shown above, the JSON payload has a { "room" : "kitchen" } in it, the metric name will be rewritten to heat.kitchen.

You'll notice the < character followed by a word; this indicates that mqttcollect should retrieve the value of the metric from said JSON element. So, for example, <celsius means: the value of the metric should be taken from the payload's { "celsius" : 11.81 } element. Likewise, <fahrenheit would consume that value, but who'd want that? ;-)

Putting this together, if I configure mqttcollect with this section:

[arduino/temp/+]
gauge = heat.{room}<celsius

mqttcollect will rewrite the metric name from the JSON payload and obtain the value for the metric from the same JSON payload, handing collectd the following line:

PUTVAL tiggr/mqttcollect/gauge-heat.kitchen 1431548550:11.81

A practical example we're using this for is for our OwnTracks Greenwich devices which publish JSON payloads like this example (shortened for clarity):

{
    "_type": "location",
    "alt": 53,
    "tid": "BB",
    "vel": 62
}

mqttcollect will, when configured as below, produce three metrics per message it receives.

[owntracks/+/+]
gauge = vehicle/{tid}/speed<vel
gauge = vehicle/{tid}/altitude<alt
counter = vehicle/{tid}/odometer<trip
PUTVAL tiggr/mqttcollect/gauge-vehicle/BB/speed 1431543655:62.00
PUTVAL tiggr/mqttcollect/gauge-vehicle/BB/altitude 1431543655:53.00
PUTVAL tiggr/mqttcollect/counter-vehicle/BB/odometer 1431543655:672798.00

This lets us produce nice graphs with all sorts of useless information, such as at which altitude vehicles are driving at using InfluxDB and Grafana.

Viewed in Grafana

I've put the source code to mqttcollect up on this repository.

View Comments :: monitoring and MQTT :: 15 May 2015 :: e-mail

When we started using Slack in our small team I was sold on the spot, and I particularly enjoy being able to build integrations which, well, integrate seamlessly into Slack: type a command into a Slack channel, and get a response right there, without leaving Slack.

I was thinking about how to go about getting a bit of our OwnTracks into Slack. Wouldn't it be neat if you could tell with a single command where a team mate is? Sure, we could glance at a map or use one of the other utilities that the OwnTracks project provides, but let's see if we can do better.

Slack command in action

If you've been following a bit of what we do with OwnTracks, you'll know privacy is a big word for us, and as such we recommend you use your own MQTT broker for OwnTracks. I have a dozen friends and family members using my broker with carefully designed ACLs so that only consenting parties can see each others' location. I don't want to open up MQTT publishes from my broker to the rest of the world, but I want my Slack team to be able to determine where I am so they know if I can be disturbed. (I trust these guys.)

OwnTracks maps

What I'm going to show you is how we can use a combination of OwnTracks, mqttwarn (which I introduced here), and a Slack slash command to produce a useful utility you can use in Slack as well. (If you're more of an IRC user, I'm sure you can use some of these techniques to write a bot which will do similarly, and do tell me about how you solved it!) The difference between a slash command in Slack and other integrations such as Webhooks is that only the person who issued the command sees the response, so it doesn't clutter the channel.

what we build

My smart phone publishes a location update via MQTT to my broker, from which mqttwarn picks it up, reformats the payload, and performs a HTTP POST to an external system on which a Slack integration can obtain the data.

The payload published by OwnTracks over MQTT is JSON which contains the latitude, longitude, altitude, etc. of the smart phone, as well as a time stamp, and a tracker-ID called TID which I configure on the device itself. The payload looks like this (I've removed some of the elements to keep the example brief):

{
  "_type": "location",
  "tid": "jJ",
  "tst": 1430493519,
  "lon": "2.295134",
  "lat": "48.858334"
}

mqttwarn subscribes to MQTT messages published at the topic used by my phone, and determines whether the message qualifies to be handled. The filter function can drop messages I'm not interested in handling even though they're published to the same topic (e.g. lwt messages when the broker detects the device has temporarily gone offline).

I also provide a custom function in slackfuncs.py called slack_whereis() which will extract latitude and longitude from the OwnTracks payload and perform a reverse-geo lookup.

[defaults]
functions = 'slackfuncs'
launch   = log, http

[config:http]
timeout = 60
targets = {
   'slack_whereis' : ['post', 'https://example.org/whereis/user/jjolie', None, None],
  }

[owntracks-Slack]
topic = owntracks/jjolie/+
filter = slack_whereis_filter()
alldata = slack_whereis()
targets = http:slack_whereis
format = reported last at {_dthhmm} from {geo}

So, if OwnTracks publishes the JSON shown above to the topic owntracks/jjolie/nex, say, mqttwarn, which is subscribed to that topic, will receive and handle the message and, via the slack_whereis() function, will add a geo element which mqttwarn can use in format when finally sending the message to the configured target.

In this case, I use the http target to POST data to a remote endpoint where the association (Jane is at this place) is stored.

The HTTP endpoint is a small Bottle app which accepts these location update POSTs on the one hand, and which is an endpoint for a Slack command on the other.

whereis.py

Slack slash commands

There's pretty little that the guys over at SlackHQ forgot to implement, and one of my favorites is

/remind me in 3 days to tell bbucks to push changes to master soon

reminder

Slack allows me to add my own custom slash commands which I configure in Slack integrations. What I'm going to do here is to create a /whereis command so that my mates can see where I am (and maybe I can see where they are).

custom command

Adding a custom command requires two steps:

  1. add the Slack integration for the custom command
  2. create the code which will actually run the command

Adding the command integration is easy: choose the name of the command (/whereis), the HTTP endpoint, and a usage hint for the user. I make a note of the token which I can use in my code to ensure only clients which know the token can query my service.

When I invoke a custom command, Slack fires off a POST (or optional GET) request to the endpoint we configure. The example I show you here implements a simple solution for the task: use the value passed to /whereis to find information about a particular user and return a one-line text result which Slack displays au-lieu of the command I entered. (I run this particular app under nginx and UWSGI.)

#!/usr/bin/python
# Jan-Piet Mens, May 2015.  Slack slash command. (/whereis user)

import bottle   # pip install bottle
from persist import PersistentDict # http://code.activestate.com/recipes/576642/

botname = 'owntracks'

path = '/home/jpm/slack-whereis/db/userlist.json'
userlist = PersistentDict(path, 'c', format='json')

app = application = bottle.Bottle()

@app.route('/', method='POST')
def slack_post():
    body = bottle.request.body.read()

    token           = request.forms.get('token')          # yJxxxxxxxxxxxxxxxxxxxxxx
    team_id         = request.forms.get('team_id')        # T00000000
    team_domain     = request.forms.get('team_domain')    # example
    service_id      = request.forms.get('service_id')     # 0123456789
    channel_id      = request.forms.get('channel_id')     # C01234567
    channel_name    = request.forms.get('channel_name')   # general
    timestamp       = request.forms.get('timestamp')      # 1428242917.000011
    user_id         = request.forms.get('user_id')        # U10101010
    user_name       = request.forms.get('user_name')      # jpmens
    text            = request.forms.get('text')           # <free form>
    trigger_words   = request.forms.get('trigger_words')

    if token != '1xyfx098xelxRxk913x01234':  # integration token
        return "NOPE"

    # text contains the username (or it is empty)
    who = text.lower().rstrip()
    if who == "" or who is None:
        return "Who?"

    if who in userlist:
        response = "%s %s" % (who, userlist.get(who, 'dunno'))
    else:
        response = "I haven't a clue where %s is." % (who)

    return response

# curl -d 'is lying on the beach...' https://example.org/whereis/user/jjolie

@app.route('/user/<username>', method='POST')
def user(username):
    text = bottle.request.body.read()

    userlist[username] = text
    userlist.sync()

    return ""

if __name__ == '__main__':
    bottle.run(app, host='0.0.0.0', port=80)

If I'm very concerned about privacy, I can add a bit of haversine magic or use the OwnTracks waypoints feature on the location reported by OwnTracks to keep things anonymous even from trusted team mates:

at home

For example, I can hide at which restaurant customer I am, etc.

If this has inspired you to build something with OwnTracks, we'd love to know; talk to us.

View Comments :: OwnTracks and Slack :: 02 May 2015 :: e-mail

It must have been over a year ago that somebody mentioned Jenkins is not just a tool for developers; system administrators can also put it to good use. I recall glancing at it and subsequently forgot about it. Anyway, I'm probably the very last person on Earth to learn this.

During Loadays in Antwerp last weekend, Fabian Arrotin mentioned this again, and I convinced him (in exchange of a beverage or two at the hotel bar) to show me. He demonstrated how he schedules Ansible jobs to run from within Jenkins, and the coin clicked as I watched what he did.

Job configuration

In the course of today's lunch break, I set up Jenkins on my laptop, installed a couple of plugins and everything I tried just worked. Fabian showed me the color plugin and mentioned the console column plugin which allows me to open the last console output at the click of a button from the dashboard.

Dashboard

Within a very few minutes I was kicking Ansible playbook-runs by checking something into a git repository; a commit hook kicks off a Jenkins build.

Console output

I think for things like Ansible, scheduled "builds" (think cron) will be tremendously useful, in particular because I can browse through prior build history etc. Within the same lunch break I had the MQTT sending me notifications of failed builds via mqttwarn.

I'm getting addicted in spite of its UI, and I can't wait for the weekend to read more about what Jenkins can do.

View Comments :: Sysadmin, Ansible, and Jenkins :: 16 Apr 2015 :: e-mail

Other recent entries