A 10-minute session with a gin/tonic at a bar in Berlin earlier this year seems to have sufficed for me to convince Lennart Koopmann to add an MQTT input to Graylog2. (Zoom into the picture in that tweet :-)) The damage a single drink can do …

A new version of Graylog2 was released yesterday. One of its new features is support for loadable plugins, so I built the graylog2-mqtt-input plugin, and dropped it into Graylog2’s plugin/ directory before starting graylog2-server (version 0.90.1 with ElasticSearch 0.90.10).

The first thing I then did was to add the MQTT plugin as an input using Graylog2’s Web interface:

I like that the developers have thought of allowing me to specify multiple topics Graylog2 will subscribe to. As soon as that was complete, the plugin was launched and I saw the connection in my broker’s log.

The MQTT input plugin expects messages to be in GELF. In particular, the mandatory host and short_message elements must be present or the message isn’t processed.

Using this simple program as an example (without the sleep() in it), I obtained a throughput of just over 950 messages/second in Graylog2 on my portable datacenter. To be quite explicit: the container in which I tested this is running on my laptop and has ElasticSearch, Graylog2-server, and Graylog2-Web, all in a meager 768 MB RAM!

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import paho.mqtt.client as paho   # pip install paho-mqtt
import time
import socket
import random
import json
import os

hostname = socket.gethostname()

clientid = 'jp-%s-%s' % (hostname, os.getpid())

mqttc = paho.Client(clientid, clean_session=False, userdata=None)
mqttc.connect("localhost", 1883, 60)

while True:
    temp = random.randint(10, 33)
    text = "Temperature %s" % temp
    data = {
        'host'      : hostname,
        'short_message' : text,
        'temperature'   : temp,
    }

    mqttc.publish('gl/temp', json.dumps(data))
    time.sleep(1)

The program publishes messages to the topic gl/temp which Graylog2 subscribes to, and these messages are added to the index. Selecting one of them demonstrates how Graylog2 adds a timestamp as well as the MQTT topic the message was published to.

Because of the GELF requirements, without modifying the modus operandi (i.e. the Java code) of the plugin, we can’t just publish MQTT payloads which contain, say, an integer temperature or something similar. We’ll typically have to create a republisher which subscribes to the topics you want logged, transforms the payload into GELF and republishes messages to the topic Graylog2 is subscribed to. One possible candidate for doing this may be mqttwarn, if you want something quick and dirty. Here is an example taking OwnTracks data and splitting it up for the MQTT plugin.

All in all, this is looking very good. I have only had the combination running for a couple of hours, but it’s working well so far.

Thank you very much, Lennart and the Graylog2 team for doing this.