When Peter asked me the other day whether I could recommend some stack/platform/whatever for storing/viewing metrics, my first answer was Graphite, but it then quickly occurred to me that I myself never really got around to finishing that project. A Graphite installation is a bit messy, and I don't find it trivial to support.

I had previously stumbled over InfluxDB, and I liked the look of it because

  • it's a single binary
  • it has okay-ish documentation
  • it worked for me out of the box

Reasons enough for me to have a closer look.

I could write whole books about what I don't know about InfluxDB, and I have no idea whatsoever how it'll fare up to really heavy 10,000-server-type of usage, but I don't really care: my use-case is to get some decent time-series database set up for a SOHO environment (storing a few sensor datapoints, electricty consumption, and things like that), in addition to being able to easily render some nice-looking graphs, and all that without spending hours setting it up and maintaining it. (Also, my experience with Xiveley, the artist previously known as Cosm, the artist previously known as Pachube was only okay-ish (sometimes), and it's in a cloud, but I digress.)

Whatever it is I use should be easy. Period. It must have a decent-enough API, be supported by Python, and I want to be up and running (and hopefully not look back) within an hour or two. Maximum three.

InfluxDB seems to be a good match. It is based on LevelDB (I know somebody I admire who'd tear my head off for even using that word...), and it runs on all relevant platforms I can think of. It has a built-in administration panel which is good enough, API support with HTTP Basic authentication (or parametrized auth), and it's extensible: in addition to an HTTP API, I can configure InfluxDB for Graphite carbon support, which means I can submit data via a number different methods, and adding further listeners is possible and looks pretty straight forward for somebody in the know of Go. (I can immediately think of a listener I'd like to see; talk to me, do it, and make me happy. :-)

InfluxDB can also query Graphite data if that's your cuppa and/or if you already have a big dataset you want to use for it.

First off, the Carbon support made me add a carbon plugin to mqttwarn, and I used that to feed in a set of test data to InfluxDB to then glance at it via the built-in Web thing that InfluxDB has.

Using the InfluxDB API, I can then, say, retrieve values via HTTP:

[
    {
        "columns": [
            "time", 
            "sequence_number", 
            "value"
        ], 
        "name": "c.youless", 
        "points": [
            [
                1405011574000, 
                1, 
                1029
            ], 
            [
                1405011544000, 
                1, 
                2334
            ]
        ]
    }
]

InfluxDB's Web console is useful but a bit rudimentary. Enter Grafana which is based on or at least heavily inspired by ElasicSearch's Kibana.

I set that up, and the bit that took longest in this whole exercise was to find the right button to click in order to have Grafana read from InfluxDB. (I had already configured it to do so, I just couldn't find the data source button, so to speak.)

The result is rather nice, I find, even though I haven't been able to switch to a white background...

Whilst possibly not as flexible as Graphite, I'm certainly going to install this pair of utilities at home, and let it run for a while; it appears to be just what this doctor ordered for himself.

View Comments :: MQTT, metrics, graphite, and carbon :: 10 Jul 2014 :: e-mail

When I wrote about the HiveMQ MQTT broker I said it had a killer-feature: support for Websockets. I'll admit to have been pestering Roger Light, creator of Mosquitto, so much that to shut me up (I think) he's been adding support for Websockets into the Mosquitto broker!

This makes using MQTT even more fun.

Mosquitto and Websockets

Websocket support must be explicitly enabled when building Mosquitto, and building requires libwebsockets. Once enabled, we can add listeners with the websocket protocol. For example, this mosquitto.conf provides a default MQTT listener on port 1883, and two Websocket listeners, one with TLS, one without:

autosave_interval 1800
persistence true
persistence_file m2.db
persistence_location /var/mosquitto/
connection_messages true
log_timestamp true

acl_file /etc/mosquitto/jp.acl
password_file /etc/mosquitto/jp.pw

listener 1883

listener 9001 127.0.0.1
protocol websockets

listener 9002 127.0.0.1
protocol websockets
cafile /etc/mosquitto/tls/ca.crt
certfile /etc/mosquitto/tls/tiggr.ww.mens.de.crt
keyfile /etc/mosquitto/tls/tiggr.ww.mens.de.key

(In case you're interested, I create the X.509 certificates for TLS using generate-CA.sh from the OwnTracks project.)

Mosquitto ACLs work for Websockets just as they do for MQTT publishes & subscribes; when accessing the broker via Websockets a different TCP transport channel is used -- everthing else remains the same.

Using the Paho MQTT JavaScript client, we can create something like this which shows messages published to the Mosquitto broker as soon as they're received.

<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <title>Mosquitto Websockets</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <script src="mqttws31.js" type="text/javascript"></script>
    <script src="jquery.min.js" type="text/javascript"></script>
    <script src="config.js" type="text/javascript"></script>

    <script type="text/javascript">
    var mqtt;
    var reconnectTimeout = 2000;

    function MQTTconnect() {
        mqtt = new Messaging.Client(
                        host,
                        port,
                        "web_" + parseInt(Math.random() * 100,
                        10));
        var options = {
            timeout: 3,
            useSSL: useTLS,
            cleanSession: cleansession,
            onSuccess: onConnect,
            onFailure: function (message) {
                $('#status').val("Connection failed: " + message.errorMessage + "Retrying");
                setTimeout(MQTTconnect, reconnectTimeout);
            }
        };

        mqtt.onConnectionLost = onConnectionLost;
        mqtt.onMessageArrived = onMessageArrived;

        if (username != null) {
            options.userName = username;
            options.password = password;
        }
        console.log("Host="+ host + ", port=" + port + " TLS = " + useTLS + " username=" + username + " password=" + password);
        mqtt.connect(options);
    }

    function onConnect() {
        $('#status').val('Connected to ' + host + ':' + port);
        // Connection succeeded; subscribe to our topic
        mqtt.subscribe(topic, {qos: 0});
        $('#topic').val(topic);
    }

    function onConnectionLost(response) {
        setTimeout(MQTTconnect, reconnectTimeout);
        $('#status').val("connection lost: " + responseObject.errorMessage + ". Reconnecting");

    };

    function onMessageArrived(message) {

        var topic = message.destinationName;
        var payload = message.payloadString;

        $('#ws').prepend('<li>' + topic + ' = ' + payload + '</li>');
    };


    $(document).ready(function() {
        MQTTconnect();
    });

    </script>
  </head>
  <body>
    <h1>Mosquitto Websockets</h1>
    <div>
        <div>Subscribed to <input type='text' id='topic' disabled />
        Status: <input type='text' id='status' size="80" disabled /></div>

        <ul id='ws' style="font-family: 'Courier New', Courier, monospace;"></ul>
    </div>
  </body>
</html>

I've put the source to this simple example on Github, and I've verified it works with Safari, Chrome, and FireFox.

Demo

Instead of writing your own Web application, you can easily use the mqtt.io or the HiveMQ Websocket clients; configure them to use the host name and TCP port number (from the listener directive) of your Mosquitto broker.

Mosquitto with Websocket support should be with you very soon, and I hope you'll agree with me, that this is very good news. :-)

The idea for an application on a smartphone which publishes your location via MQTT was born a few weeks prior to the first commit to the Android app. My first notes are dated June 16th, and shortly thereafter I spoke to Alexander about the idea of the app which would turn into what we now call OwnTracks. Just after first writing about it, Christoph created the iOS app.

The principle is simple enough: an app on an iOS or Android device transmits the device's location in a JSON payload via MQTT to a broker I operate and from there, I can do whatever I wish with the data.

Andy's tweet

We started this project under what we thought was a very techy-sounding and very sexy name MQTTitude, but it was unpronounceable; friends thought we were being rude when we tried to say the word, and since we couldn't agree on giving the name a few more t_s to make it really impossible, we decided to re-brand and chose OwnTracks. The name should associate tracks as in "location tracking", and own_ as in "you own the data" or "your own data".

Initially, the JSON payload contained a time-stamp, and decimal latitude and longitude. These were augmented with the device's battery level (for monitoring, you know) and positional accuracy. JSON makes it easy to extract the data you want from the payload in almost any programming or scripting language, and it enables us to add elements to the object without breaking your application. (You are not using regular expressions to access payload data, are you?!)

{
    "_type": "location", 
    "acc": "141", 
    "batt": "27", 
    "lat": "48.858334", 
    "lon": "2.295134", 
    "tst": "1402507486"
}

Early on, we had a lot of ideas pour in, and people started doing fun things with OwnTracks. Marshall T. Rose added support for OwnTracks into The Thing System which greatly pleased us, and Ben Jones wanted us to add support for waypoints and geo-fences which he used to implemented a binding for openHAB to control "stuff" upon entering or leaving the premises.

If I recall correctly, we then added support for what we call "Friends and Family": people who share an MQTT broker can 'see' each other, and the user-interface displays icons on a list or on a map. Friends optionally get to see me entering/leaving waypoints, if I share them.

Screenshots iOS/Android

Bryan Boyd literally took OwnTracks for a run when he live-tweeted a marathon using a bunch of technology together with our app as a GPS capture device, and others have put OwnTracks to good use in other ways, using it to determine colleagues' whereabouts, track and display delayed trains, and all sorts of other things.

While we do try and keep the apps' capabilities in sync with eachother (iOS / Android), Christoph tends to implement features on iOS as soon as we breathe an idea to him, which is why the iOS app is slightly ahead feature-wise. For example, on iOS we now have iBeacon support as well as support for the iPhone 5 pedometer function, and I'll tell you something: testing that has literally been exhausting for me: all that walking ... The way it works is, OwnTracks keeps track of the number of steps and I can request it reports the steps taken since midnight:

{"_type":"steps", "tst":1401966882, "steps":3269}

OwnTracks has really become international, and I think the user farthest away from us here in Germany is in Tasmania. (The only reason I know is because I've seen an OwnTracks JSON payload which was handed over for debugging purposes; remember: it's your data; we don't see it if you don't show us, and that's good! :-) Update: Matt reminds me that I'm quite wrong: Ben's furthest away, in New Zealand. :-)

OwnTracks in Tasmania

Many thanks to everybody who has contributed code, ideas, time for testing, documentation, and everything else we've needed to have over one thousand people now use our Open Source apps!

View Comments :: OwnTracks, location, and MQTT :: 16 Jun 2014 :: e-mail

Stefano Costa sent me a Choral Greenwich for evaluation. I've had my eyes on that or something similar for a while now because my brother in law has been fascinated by OwnTracks, and he's interested in tracking a number of tour buses they operate.

Choral Greenwich

The Greenwich is a lovely little device which comes preloaded with a demo-application which Choral calls OrionMQTT -- an application which periodically reports the device's location over MQTT. (The too-shiny coin on the photo above is a 1 Euro piece.)

Some notable features of the Greenwich:

  • Powered by 8-32 V DC
  • Li-Ion battery with internal charger
  • GSM/GPRS chip
  • Accelerometer
  • Digital inputs and outputs
  • UART serial
  • Optional CAN bus
  • Operating temperature: -20 +60 °C
  • Java programmable

That last bit is what held me off a little because of my ignorance of all things Java; I was hoping to be able to easily modify the code (which will soon be open sourced by Choral). Instead, and for the time being, I'm just re-publishing the data in a format suitable for OwnTracks. Choral provides an SDK for programming the device, but I haven't looked at that yet.

I was given the advanced Greenwich which has all the trimmings on it, including a D-SUB9 through which I configured the device. After booting, I can use #commands to configure it; authenticate, set up GPRS connectivity, tracking interval, and start tracking:

#PWD 1234567890

#CPRSCFG internet.apn,TCP,192.168.1.22,1883

#TRKTIM 120s

#TRK ON

Other commands such as #PUBTOPIC allow me to configure the MQTT topic to which Greenwich publishes, and #SETID (that's my 41 below) configures a device identifier which can be made to be meaningful for your application (e.g. vehicle number).

After that, it's basically just a matter of carrying the light little box to the vehicle and installing it somewhere (e.g. in the glove box or boot). Mine is provisionally installed, and I hooked it up to an always-on cigarette-lighter in the back of the car.

The Greenwich publishes a CSV-like string which contains a header ($CHX) and a configurable ID (here: 41). The device also reports date (YY) and time obtained via GPS, position as well as a battery level, travelled distance in metres, and a few other interesting values such as altitude and speed over ground.

$CHX,41,A,09,140613,133708,5057.66047,N,00842.97512,E,16.10,13.440,240.4,102780,4.4V,E,08,00,00000000*2D

CSV

The temporary republisher I have extracts the elements from the CSV and produces OwnTracks-compatible JSON payloads.

{
    "alt": "240.4", 
    "batt": "4.4V E", 
    "distance": "102780", 
    "lat": "50.961008", 
    "lon": "8.716252", 
    "sat": "09", 
    "setid": "41", 
    "tst": "1402666628", 
    "vel": "24.89"
}

In addition to periodic publishing over MQTT, I can send an SMS to the device with the string gps in it, and it responds via SMS, which is rather neat. The software can also be rebooted via SMS, and we can send it a park message which will alert (via SMS) if the vehicle is moved after it receives the park.

The Greenwich I have has been behaving very well; a watchdog on it automatically restarts it if something goes wrong. For example, I drove through an area with zero GPRS coverage yesterday afternoon for about 10 kilometers, and I noticed the LEDs flashing to indicate a reboot; after a few minutes (my logs tell me later), the device continued reporting. (A reboot resets the travelled distance, of course.)

This is a track I recorded yesterday (via the OwnTracks m2s back-end), with the device reporting every 30 seconds.

Driving with Greenwich

and if I zoom in a bit we still see the quite fine granularity of the reports:

Zoom in

Before we can take this to production, there are definitely features missing in the software, and I'm talking to Choral about this as we speak (the people at Choral have been extremely helpful, and they're very responsive):

  • TLS is a must-have; currently the connection to the MQTT broker is in plain text.
  • Authentication is also a must-have, though this should be no great issue as Greenwich uses the Paho MQTT client for Java
  • I'd very much like to see a #TRKDIST command with which I can define a distance (in meters) after which a location is published; this would allow lowering the required GPRS data volume.

The Choral Greenwich is a really neat little box, and it's performing very well.

View Comments :: MQTT, OwnTracks, and Location :: 14 Jun 2014 :: e-mail

Dinner at the NLUUG conference a few weeks ago was rather good: I chose carpaccio and a Black Angus and placed myself at the end of the long table sitting between Jos and Raymond, with Bill opposite us to the left.

The carpaccio had just been served when Raymond, who had listened to me speak about MQTT, said: "may I ask you a question?" To cut a long story short, I had plenty to think about on the subsequent late-night drive home. (And this post should have possibly been titled "Zabbix, meats, and MQTT". :-)

Raymond explained to me that Zabbix transports monitoring data insecurely and whether I thought MQTT would be a solution because it has TLS and authentication built into it? In spite of my knowing just about as much about Zabbix as about rearing Angus cattle, I said yes. And it is.

This is what we want to accomplish:

  • when a host comes on-line it should register itself as such in Zabbix.
  • a host should be able to report any number of metrics via MQTT which are made available as Zabbix items.
  • MQTT's Last Will and Testament (LWT) will be leveraged to provide food for Zabbix's triggers. In other words, if the monitored host dies, the MQTT broker will publish its demise on its behalf.
  • all data must be encrypted in transit. (A basic feature of MQTT.)
  • clients should be able to authenticate before being able to submit metrics. (This too is a basic MQTT feature.)
  • new items published to Zabbix should optionally be created on the fly

So, assume for a moment, what I'm calling the "Zabbix MQTT agent" comes up, it would publish messages such as these:

zabbix/clients/www 1
zabbix/item/www/ambient/temp 27.8
zabbix/item/www/disk/sda/blocks.free 9999

(Whether Zabbix keys are slash-separated to accommodate MQTT topic name structures or whether they're dot-separated to adhere to Zabbix naming conventions is up to discussion.)

I started studying as much as time permitted, and I found several potential entry points for data into Zabbix. (If you are a Zabbix expert, take what follows with many pinches of salt (remember this is all very new to me), and if you aren't you won't notice mistakes I make...)

  1. User parameters
  2. Loadable modules written in C
  3. External checks
  4. Trapper items
  5. The Zabbix REST API
  6. Low-Level Discovery (LLD)

User parameters are scripts which get invoked by Zabbix Agent on demand, which is unsuitable for what we want to do. Loadable modules provide an Agent with data, but it's not possible to have the module submit new keys at will: the agent has to be restarted, so this is out of the question, and external checks are similarly not suitable as they are a one-shot kind of thing where the Zabbix agent forks/execs; nice for ad-hoc queries, but unsuited to what we intend doing.

What I then attempted was to create a trapper item with the API and feed metrics to it with the zabbix_sender protocol. That worked nicely if I either waited a minute between those two or reduced the CacheUpdateFrequency configuration setting, but it didn't seem like the right way to do it.

Raymond taught me about Low-Level Discovery (LLD) and he implemented some prototypical shell scripts. After looking at those, and asking a few questions, the light went on. The manual page for LLD says:

Low-level discovery provides a way to automatically create items, triggers, and graphs for different entities on a computer. For instance, Zabbix can automatically start monitoring file systems or network interfaces on your machine, without the need to create items for each file system or network interface manually. Additionally it is possible to configure Zabbix to remove unneeded entities automatically based on actual results of periodically performed discovery.

and that does indeed sound like what the doctor ordered.

Zabbix-MQTTwarn Architecture

I have added support for Zabbix to mqttwarn, and it works like this. Assume we have a Zabbix MQTT agent (there's an example in the mqttwarn repository) which publishes to the following topics:

zabbix/clients/jog09 1
zabbix/item/jog09/system.cpu.load 8
zabbix/item/jog09/time.stamp 2014-05-27 11:39:36
...
zabbix/clients/jog09 0

As soon as mqttwarn sees a new client come up (zabbix/clients/jog09 with a payload of 1) it sends a low-level discovery request to the Zabbix trapper, which creates the host from a template, and instantiates the template's items.

The 1 is used to populate what I call a "status key" which can be used by Zabbix to trigger up/down alarms.

From that moment onwards, MQTT publishes to the zabbix/item/ topic branch with a client hostname and a Zabbix key contained in the topic name cause the items to be populated with the corresponding values.

The configuration for mqttwarn is simple enough, but we do need a transformation function (see the documentation) for it in order to extract the client and the Zabbix item's key name from the MQTT topic.

[defaults]
functions = 'myfuncs'

launch  = zabbix

[config:zabbix]
targets = {
            # Trapper address   port   
    't1'  : [ '172.16.153.110', 10051 ],
  }

; These branches could be handled by "zabbix/#", but I'm splitting
; them to avoid, say, "zabbix/alerts" being handled.

[zabbix/clients/+]
alldata = ZabbixData()
targets = zabbix:t1

[zabbix/item/#]
alldata = ZabbixData()
targets = zabbix:t1

Similarly to what I did when I asked How do your servers talk to you?, I can configure Zabbix to send alerts via MQTT with an external media script, but I haven't yet found a way to create an action in a template. Anybody?

This is but a beginning, but I believe it has potential. Tell us what you think.

View Comments :: monitoring, MQTT, and Zabbix :: 27 May 2014 :: e-mail

Other recent entries