A 10-minute session with a gin/tonic at a bar in Berlin earlier this year seems to have sufficed for me to convince Lennart Koopmann to add an MQTT input to Graylog2. (Zoom into the picture in that tweet :-)) The damage a single drink can do ...

A new version of Graylog2 was released yesterday. One of its new features is support for loadable plugins, so I built the graylog2-mqtt-input plugin, and dropped it into Graylog2's plugin/ directory before starting graylog2-server.

The first thing I then did was to add the MQTT plugin as an input using Graylog2's Web interface:

Add MQTT as input

plugin config

I like that the developers have thought of allowing me to specify multiple topics Graylog2 will subscribe to. As soon as that was complete, the plugin was launched and I saw the connection in my broker's log.

running input

The MQTT input plugin expects messages to be in GELF. In particular, the mandatory host and short_message elements must be present or the message isn't processed.

Using this simple program as an example (without the sleep() in it), I obtained a throughput of just over 950 messages/second in Graylog2 on my portable datacenter. To be quite explicit: the container in which I tested this is running on my laptop and has ElasticSearch, Graylog2-server, and Graylog2-Web, all in a meager 768 MB RAM!

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import paho.mqtt.client as paho   # pip install paho-mqtt
import time
import socket
import random
import json
import os

hostname = socket.gethostname()

clientid = 'jp-%s-%s' % (hostname, os.getpid())

mqttc = paho.Client(clientid, clean_session=False, userdata=None)
mqttc.connect("localhost", 1883, 60)

while True:
    temp = random.randint(10, 33)
    text = "Temperature %s" % temp
    data = {
        'host'      : hostname,
        'short_message' : text,
        'temperature'   : temp,
    }

    mqttc.publish('gl/temp', json.dumps(data))
    time.sleep(1)

The program publishes messages to the topic gl/temp which Graylog2 subscribes to, and these messages are added to the index. Selecting one of them demonstrates how Graylog2 adds a timestamp as well as the MQTT topic the message was published to.

temp details

Because of the GELF requirements, without modifying the modus operandi (i.e. the Java code) of the plugin, we can't just publish MQTT payloads which contain, say, an integer temperature or something similar. We'll typically have to create a republisher which subscribes to the topics you want logged, transforms the payload into GELF and republishes messages to the topic Graylog2 is subscribed to. One possible candidate for doing this may be mqttwarn, if you want something quick and dirty. Here is an example taking OwnTracks data and splitting it up for the MQTT plugin.

OwnTracks pubs

All in all, this is looking very good. I have only had the combination running for a couple of hours, but it's working well so far.

Thank you very much, Lennart and the Graylog2 team for doing this.

View Comments :: MQTT and logging :: 24 Sep 2014 :: e-mail

My hands are almost bleeding from plugging and unplugging Molex connectors from the back of 26 Choral Greenwich devices, but it's been fun. (Those damn things are almost impossible to take out, which is good; they don't fall out by themselves!) I've told you about the Choral Greenwich before, but a lot has happened in the last three months.

Greenwich

First and foremost, my unstoppable friend Christoph, who also made OwnTracks for iOS and MQTT Inspector for iOS, has completely revamped the software onboard this vehicle and asset tracking device. With some advice from our friends at Choral, and a boatload of testing by both of us, we've transformed the software into something which is ready to roll and, in fact, has already been running live on six vehicles during the last couple of months without a hitch.

In particular, what we lovingly call the OwnTracks Edition of the software has the following features:

  • Publishing of location updates over MQTT with TLS
  • Device configuration via MQTT or SMS
  • Message payloads compatible with OwnTracks' JSON which means you can use our OwnTracks apps to display location data on a smartphone
  • Watchdog resets the device if it becomes unresponsive
  • Over the Air (OTA) upgrades
  • GPIO (input and output) controllable over MQTT
  • Alarming if device is moved

and many more features which make this a very versatile and low-cost device. In particular, I'm proud of being able to trigger an OTA upgrade directly over MQTT, which, a) saves on the cost of an SMS (relatively expensive with M2M SIM data plans), and b) allows us to seamlessly push software updates out into devices installed in the field. While OTA upgrades may sound expensive, the M2M data plan we're using translates a full upgrade into 0.0416 €, which is almost not worth speaking of.

As for the JSON payloads, we've added a few elements. First and foremost, a so-called tid which we use also in the new iOS app to display what we call an "automatic face", if you don't want to associate a photo from your address book with an OwnTracks MQTT topic. Next up are cog (course over ground), vel (speed in KM/h), dist (distance covered in meters from the previoius location published), and trip which shows the total distance covered since reset (and yes: you can zero this counter via MQTT).

{
    "_type": "location", 
    "alt": 497, 
    "cog": 223, 
    "dist": 1572, 
    "lat": "47.453453", 
    "lon": "8.703986", 
    "t": "t", 
    "tid": "AP", 
    "trip": 119779, 
    "tst": "1411120145", 
    "vel": 98
}

In fact we've already got our first "customer", namely my brother in law, who's ordered said 26 devices to put on tour buses they operate. He's been so impressed with the first four, that he's decided to put one in each of their vehicles.

Initial provisioning of naked Greenwich devices has been a bit of a pain, but we've now solved that rather nicely: a small program creates and sends the PDU SMS for the OTA upgrade, and off we go. Here's a photo of my workspace during the last 20 hours or so.

Provisioning Greenwich

We're also hard at work on an appropriate back-end which will collect and visualize data. This is almost ready and will be contributed to the OwnTracks project in due course. In addition to being able to display historic routes using GeoJSON we generate from the database, we're making heavy use of "live" updates, which are provided by Mosquitto with Websockets.

table view

So, if you're a fan of MQTT, and you're looking for a very neat device for tracking the position of your vehicles, talk to me.

View Comments :: MQTT, GPS, and vehicle :: 18 Sep 2014 :: e-mail

I met Holger Koch at a conference in April this year. Holger works for DB Systel, the IT company of the Deutsche Bahn, and he has had the guts to convince the organization to host an internal "Open Source Workshop". Knowing a bit of how large organizations "operate", and believe me: the Deutsch Bahn is large, I'm sure that task was gargantuan.

I had the honour to be invited to speak there, and I gladly accepted the invitation. (I spoke for an hour of what Ansible is and how it has evolved in the course of the last year or so.) And this was an invitation: travel (by first-class train :-) ) and very comfortable accomodation were provided for.

To be honest I wouldn't typically report this; there are many conferences which focus on Open Source topics, and some of my readers go to many more of those than I do. What makes this event noteworthy, is that this was a company-internal show: all participants were employees of the Deutsche Bahn or its assets. Promoting Open Source within an organization in that form is quite special, at least I've yet to hear about that being done at that kind of scale.

During his ending keynote, Holger spoke of why people and particularly companies should promote and contribute to Open Source. Contributions can take the form of donations or code contributions, and he gave several examples. One of the examples Holger brought back to memory was how the OpenSSL project, one of the pillars of security on the Internet, had received just a few thousand dollars in donations before the HeartBleed bug. (That has changed since then.) A few voices in the audience suggested promoting Open Source within organizations, as Holger has done, is also a great way to contribute, and I fully agree. It would be great to see much more of this done, and it isn't difficult to do. What's also very important: it doesn't cost a lot; certainly a few work hours must be sacrificed, but on the other hand, the added value for companies is certainly higher motivation of employees. I think this model would work on a larger scale.

That got me thinking about whether I, as an individual, actually contribute to Open Source, and if so, how. I suddenly felt like a bit of a leach because I use a lot of Open Source compontents (utilities, operating systems, programming languages, etc.)

I have no doubt you'll correct me if you think I'm wrong, but I believe I do contribute, at least to projects which interest me:

  • I do quite a bit of testing and try to report bugs, to the best of my ability. (If you've ever done a bit of this, you know it can be very time consuming.)
  • I contribute little bits and pieces of code and/or documentation.
  • I take the time (and typically pay for my own travel and accomodation expenses) to speak at conferences about particular Open Source topics.

In the course of the last few years, I've spoken maybe two dozen times about DNS, DNSSEC, Ansible, MQTT, and a few other topics I can't recall off the top of my head.

Does that count as contributing to the Open Source community? I hope it does.

When Peter asked me the other day whether I could recommend some stack/platform/whatever for storing/viewing metrics, my first answer was Graphite, but it then quickly occurred to me that I myself never really got around to finishing that project. A Graphite installation is a bit messy, and I don't find it trivial to support.

I had previously stumbled over InfluxDB, and I liked the look of it because

  • it's a single binary
  • it has okay-ish documentation
  • it worked for me out of the box

Reasons enough for me to have a closer look.

I could write whole books about what I don't know about InfluxDB, and I have no idea whatsoever how it'll fare up to really heavy 10,000-server-type of usage, but I don't really care: my use-case is to get some decent time-series database set up for a SOHO environment (storing a few sensor datapoints, electricty consumption, and things like that), in addition to being able to easily render some nice-looking graphs, and all that without spending hours setting it up and maintaining it. (Also, my experience with Xiveley, the artist previously known as Cosm, the artist previously known as Pachube was only okay-ish (sometimes), and it's in a cloud, but I digress.)

Whatever it is I use should be easy. Period. It must have a decent-enough API, be supported by Python, and I want to be up and running (and hopefully not look back) within an hour or two. Maximum three.

InfluxDB seems to be a good match. It is based on LevelDB (I know somebody I admire who'd tear my head off for even using that word...), and it runs on all relevant platforms I can think of. It has a built-in administration panel which is good enough, API support with HTTP Basic authentication (or parametrized auth), and it's extensible: in addition to an HTTP API, I can configure InfluxDB for Graphite carbon support, which means I can submit data via a number different methods, and adding further listeners is possible and looks pretty straight forward for somebody in the know of Go. (I can immediately think of a listener I'd like to see; talk to me, do it, and make me happy. :-)

InfluxDB can also query Graphite data if that's your cuppa and/or if you already have a big dataset you want to use for it.

First off, the Carbon support made me add a carbon plugin to mqttwarn, and I used that to feed in a set of test data to InfluxDB to then glance at it via the built-in Web thing that InfluxDB has.

Using the InfluxDB API, I can then, say, retrieve values via HTTP:

[
    {
        "columns": [
            "time", 
            "sequence_number", 
            "value"
        ], 
        "name": "c.youless", 
        "points": [
            [
                1405011574000, 
                1, 
                1029
            ], 
            [
                1405011544000, 
                1, 
                2334
            ]
        ]
    }
]

InfluxDB's Web console is useful but a bit rudimentary. Enter Grafana which is based on or at least heavily inspired by ElasicSearch's Kibana.

I set that up, and the bit that took longest in this whole exercise was to find the right button to click in order to have Grafana read from InfluxDB. (I had already configured it to do so, I just couldn't find the data source button, so to speak.)

The result is rather nice, I find, even though I haven't been able to switch to a white background...

Whilst possibly not as flexible as Graphite, I'm certainly going to install this pair of utilities at home, and let it run for a while; it appears to be just what this doctor ordered for himself.

View Comments :: MQTT, metrics, graphite, and carbon :: 10 Jul 2014 :: e-mail

When I wrote about the HiveMQ MQTT broker I said it had a killer-feature: support for Websockets. I'll admit to have been pestering Roger Light, creator of Mosquitto, so much that to shut me up (I think) he's been adding support for Websockets into the Mosquitto broker!

This makes using MQTT even more fun.

Mosquitto and Websockets

Websocket support must be explicitly enabled when building Mosquitto, and building requires libwebsockets. Once enabled, we can add listeners with the websocket protocol. For example, this mosquitto.conf provides a default MQTT listener on port 1883, and two Websocket listeners, one with TLS, one without:

autosave_interval 1800
persistence true
persistence_file m2.db
persistence_location /var/mosquitto/
connection_messages true
log_timestamp true

acl_file /etc/mosquitto/jp.acl
password_file /etc/mosquitto/jp.pw

listener 1883

listener 9001 127.0.0.1
protocol websockets

listener 9002 127.0.0.1
protocol websockets
cafile /etc/mosquitto/tls/ca.crt
certfile /etc/mosquitto/tls/tiggr.ww.mens.de.crt
keyfile /etc/mosquitto/tls/tiggr.ww.mens.de.key

(In case you're interested, I create the X.509 certificates for TLS using generate-CA.sh from the OwnTracks project.)

Mosquitto ACLs work for Websockets just as they do for MQTT publishes & subscribes; when accessing the broker via Websockets a different TCP transport channel is used -- everthing else remains the same.

Using the Paho MQTT JavaScript client, we can create something like this which shows messages published to the Mosquitto broker as soon as they're received.

<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <title>Mosquitto Websockets</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <script src="mqttws31.js" type="text/javascript"></script>
    <script src="jquery.min.js" type="text/javascript"></script>
    <script src="config.js" type="text/javascript"></script>

    <script type="text/javascript">
    var mqtt;
    var reconnectTimeout = 2000;

    function MQTTconnect() {
        mqtt = new Messaging.Client(
                        host,
                        port,
                        "web_" + parseInt(Math.random() * 100,
                        10));
        var options = {
            timeout: 3,
            useSSL: useTLS,
            cleanSession: cleansession,
            onSuccess: onConnect,
            onFailure: function (message) {
                $('#status').val("Connection failed: " + message.errorMessage + "Retrying");
                setTimeout(MQTTconnect, reconnectTimeout);
            }
        };

        mqtt.onConnectionLost = onConnectionLost;
        mqtt.onMessageArrived = onMessageArrived;

        if (username != null) {
            options.userName = username;
            options.password = password;
        }
        console.log("Host="+ host + ", port=" + port + " TLS = " + useTLS + " username=" + username + " password=" + password);
        mqtt.connect(options);
    }

    function onConnect() {
        $('#status').val('Connected to ' + host + ':' + port);
        // Connection succeeded; subscribe to our topic
        mqtt.subscribe(topic, {qos: 0});
        $('#topic').val(topic);
    }

    function onConnectionLost(response) {
        setTimeout(MQTTconnect, reconnectTimeout);
        $('#status').val("connection lost: " + responseObject.errorMessage + ". Reconnecting");

    };

    function onMessageArrived(message) {

        var topic = message.destinationName;
        var payload = message.payloadString;

        $('#ws').prepend('<li>' + topic + ' = ' + payload + '</li>');
    };


    $(document).ready(function() {
        MQTTconnect();
    });

    </script>
  </head>
  <body>
    <h1>Mosquitto Websockets</h1>
    <div>
        <div>Subscribed to <input type='text' id='topic' disabled />
        Status: <input type='text' id='status' size="80" disabled /></div>

        <ul id='ws' style="font-family: 'Courier New', Courier, monospace;"></ul>
    </div>
  </body>
</html>

I've put the source to this simple example on Github, and I've verified it works with Safari, Chrome, and FireFox.

Demo

Instead of writing your own Web application, you can easily use the mqtt.io or the HiveMQ Websocket clients; configure them to use the host name and TCP port number (from the listener directive) of your Mosquitto broker.

Mosquitto with Websocket support should be with you very soon, and I hope you'll agree with me, that this is very good news. :-)

Other recent entries