Slack calls itself a platform for team communication and it's all of that, and if you forgive my enthusiasm, a shit-load more. It has changed the way our tiny team works quite radically; for one, we've abandoned e-mail. I won't go into all that Slack offers (their documentation is extensive and very good, and there are tons of introductory articles), but I will tell you what we do with it, other than just simple "chat".

BTW, Slack is free of charge until you get addicted to it. If you're like me, you'll actually want to pay for it. :-)

Aside from simple searchable chats, as I mentioned earlier, Slack allows us to create any number of so-called "channels", a little bit like an IRC channel. Team members can be invited to channels to talk about particular topics. A two-person team doesn't need many of those, but larger organizations will want to separate topics.

Where Slack gets really interesting is in the integrations it offers; a seemingly endless list of services which allow us to get notifications, information, etc. into Slack. If you like a wee bit of hacking, there's an API and what they term incoming and outgoing "WebHooks" with which we can send data into and get data out of Slack respectively.

For example, for our Greenwich devices we've created a WebHook which allows us to query a database of information which is kept updated via MQTT from the devices.

In other words, I can just enter something like versions or howis XX into the particular Slack channel we created for it, and Slack goes out to a service we created which returns the information into that channel.

We also get notifications in specific channels from external processes. For example, a Nagios integration brings alerts into one channel. Another channel displays warnings we emit via MQTT and get into Slack with the help of mqttwarn (for which I created a slack plugin).

I can't tell you how incredibly cool this is, particularly since it also works mobile with the respective iOS and Android apps. Trust me on this: you want to use Slack.

View Comments :: Productivity :: 23 Oct 2014 :: e-mail

Lunchtime humor

(A lunchtime inspiration by a party of 4)

A 10-minute session with a gin/tonic at a bar in Berlin earlier this year seems to have sufficed for me to convince Lennart Koopmann to add an MQTT input to Graylog2. (Zoom into the picture in that tweet :-)) The damage a single drink can do ...

A new version of Graylog2 was released yesterday. One of its new features is support for loadable plugins, so I built the graylog2-mqtt-input plugin, and dropped it into Graylog2's plugin/ directory before starting graylog2-server.

The first thing I then did was to add the MQTT plugin as an input using Graylog2's Web interface:

Add MQTT as input

plugin config

I like that the developers have thought of allowing me to specify multiple topics Graylog2 will subscribe to. As soon as that was complete, the plugin was launched and I saw the connection in my broker's log.

running input

The MQTT input plugin expects messages to be in GELF. In particular, the mandatory host and short_message elements must be present or the message isn't processed.

Using this simple program as an example (without the sleep() in it), I obtained a throughput of just over 950 messages/second in Graylog2 on my portable datacenter. To be quite explicit: the container in which I tested this is running on my laptop and has ElasticSearch, Graylog2-server, and Graylog2-Web, all in a meager 768 MB RAM!

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import paho.mqtt.client as paho   # pip install paho-mqtt
import time
import socket
import random
import json
import os

hostname = socket.gethostname()

clientid = 'jp-%s-%s' % (hostname, os.getpid())

mqttc = paho.Client(clientid, clean_session=False, userdata=None)
mqttc.connect("localhost", 1883, 60)

while True:
    temp = random.randint(10, 33)
    text = "Temperature %s" % temp
    data = {
        'host'      : hostname,
        'short_message' : text,
        'temperature'   : temp,
    }

    mqttc.publish('gl/temp', json.dumps(data))
    time.sleep(1)

The program publishes messages to the topic gl/temp which Graylog2 subscribes to, and these messages are added to the index. Selecting one of them demonstrates how Graylog2 adds a timestamp as well as the MQTT topic the message was published to.

temp details

Because of the GELF requirements, without modifying the modus operandi (i.e. the Java code) of the plugin, we can't just publish MQTT payloads which contain, say, an integer temperature or something similar. We'll typically have to create a republisher which subscribes to the topics you want logged, transforms the payload into GELF and republishes messages to the topic Graylog2 is subscribed to. One possible candidate for doing this may be mqttwarn, if you want something quick and dirty. Here is an example taking OwnTracks data and splitting it up for the MQTT plugin.

OwnTracks pubs

All in all, this is looking very good. I have only had the combination running for a couple of hours, but it's working well so far.

Thank you very much, Lennart and the Graylog2 team for doing this.

View Comments :: MQTT and logging :: 24 Sep 2014 :: e-mail

My hands are almost bleeding from plugging and unplugging Molex connectors from the back of 26 Choral Greenwich devices, but it's been fun. (Those damn things are almost impossible to take out, which is good; they don't fall out by themselves!) I've told you about the Choral Greenwich before, but a lot has happened in the last three months.

Greenwich

First and foremost, my unstoppable friend Christoph, who also made OwnTracks for iOS and MQTT Inspector for iOS, has completely revamped the software onboard this vehicle and asset tracking device. With some advice from our friends at Choral, and a boatload of testing by both of us, we've transformed the software into something which is ready to roll and, in fact, has already been running live on six vehicles during the last couple of months without a hitch.

In particular, what we lovingly call the OwnTracks Edition of the software has the following features:

  • Publishing of location updates over MQTT with TLS
  • Device configuration via MQTT or SMS
  • Message payloads compatible with OwnTracks' JSON which means you can use our OwnTracks apps to display location data on a smartphone
  • Watchdog resets the device if it becomes unresponsive
  • Over the Air (OTA) upgrades
  • GPIO (input and output) controllable over MQTT
  • Alarming if device is moved

and many more features which make this a very versatile and affordable device. In particular, I'm proud of being able to trigger an OTA upgrade directly over MQTT, which, a) saves on the cost of an SMS (relatively expensive with M2M SIM data plans), and b) allows us to seamlessly push software updates out into devices installed in the field. While OTA upgrades may sound expensive, the M2M data plan we're using translates a full upgrade into 0.0416 €, which is almost not worth speaking of.

As for the JSON payloads, we've added a few elements. First and foremost, a so-called tid which we use also in the new iOS app to display what we call an "automatic face", if you don't want to associate a photo from your address book with an OwnTracks MQTT topic. Next up are cog (course over ground), vel (speed in KM/h), dist (distance covered in meters from the previous location published), and trip which shows the total distance covered since reset (and yes: you can zero this counter via MQTT).

{
    "_type": "location", 
    "alt": 497, 
    "cog": 223, 
    "dist": 1572, 
    "lat": "47.453453", 
    "lon": "8.703986", 
    "t": "t", 
    "tid": "AP", 
    "trip": 119779, 
    "tst": "1411120145", 
    "vel": 98
}

In fact we've already got our first "customer", namely my brother in law, who's ordered said 26 devices to put on tour buses they operate. He's been so impressed with the first four, that he's decided to put one in each of their vehicles.

Initial provisioning of naked Greenwich devices has been a bit of a pain, but we've now solved that rather nicely: a small program creates and sends the PDU SMS for the OTA upgrade, and off we go. Here's a photo of my workspace during the last 20 hours or so.

Provisioning Greenwich

We're also hard at work on an appropriate back-end which will collect and visualize data. This is almost ready and will be contributed to the OwnTracks project in due course. In addition to being able to display historic routes using GeoJSON we generate from the database, we're making heavy use of "live" updates, which are provided by Mosquitto with Websockets.

table view

So, if you're a fan of MQTT, and you're looking for a very neat device for tracking the position of your vehicles, talk to me.

View Comments :: MQTT, GPS, and vehicle :: 18 Sep 2014 :: e-mail

I met Holger Koch at a conference in April this year. Holger works for DB Systel, the IT company of the Deutsche Bahn, and he has had the guts to convince the organization to host an internal "Open Source Workshop". Knowing a bit of how large organizations "operate", and believe me: the Deutsch Bahn is large, I'm sure that task was gargantuan.

I had the honour to be invited to speak there, and I gladly accepted the invitation. (I spoke for an hour of what Ansible is and how it has evolved in the course of the last year or so.) And this was an invitation: travel (by first-class train :-) ) and very comfortable accomodation were provided for.

To be honest I wouldn't typically report this; there are many conferences which focus on Open Source topics, and some of my readers go to many more of those than I do. What makes this event noteworthy, is that this was a company-internal show: all participants were employees of the Deutsche Bahn or its assets. Promoting Open Source within an organization in that form is quite special, at least I've yet to hear about that being done at that kind of scale.

During his ending keynote, Holger spoke of why people and particularly companies should promote and contribute to Open Source. Contributions can take the form of donations or code contributions, and he gave several examples. One of the examples Holger brought back to memory was how the OpenSSL project, one of the pillars of security on the Internet, had received just a few thousand dollars in donations before the HeartBleed bug. (That has changed since then.) A few voices in the audience suggested promoting Open Source within organizations, as Holger has done, is also a great way to contribute, and I fully agree. It would be great to see much more of this done, and it isn't difficult to do. What's also very important: it doesn't cost a lot; certainly a few work hours must be sacrificed, but on the other hand, the added value for companies is certainly higher motivation of employees. I think this model would work on a larger scale.

That got me thinking about whether I, as an individual, actually contribute to Open Source, and if so, how. I suddenly felt like a bit of a leach because I use a lot of Open Source compontents (utilities, operating systems, programming languages, etc.)

I have no doubt you'll correct me if you think I'm wrong, but I believe I do contribute, at least to projects which interest me:

  • I do quite a bit of testing and try to report bugs, to the best of my ability. (If you've ever done a bit of this, you know it can be very time consuming.)
  • I contribute little bits and pieces of code and/or documentation.
  • I take the time (and typically pay for my own travel and accomodation expenses) to speak at conferences about particular Open Source topics.

In the course of the last few years, I've spoken maybe two dozen times about DNS, DNSSEC, Ansible, MQTT, and a few other topics I can't recall off the top of my head.

Does that count as contributing to the Open Source community? I hope it does.

Other recent entries