I’ve never used Yahoo! Pipes, but I understand it’s a construction kit with which I can build things which take their input from “other things” (i.e. Internet resources). It’s probably grand, but as I just said: I’ve never used it.

For some time now, I’ve been very enthusiastic with all things related to MQTT as faithful readers know. Most of the time I have programs which need to produce messages to be sent over MQTT or, conversely, I have consumers of MQTT messages with payloads that need transforming or need to be stored somewhere. The outcome of this is a pretty large collection of little programs, each of which do a dedicated task. One example, is OwnTracks’s m2s program. It subscribes to a topic branch, and for each message, decodes the payload, does a couple of (optional) HTTP lookups, and writes the result into a relational database.

All in all, I’m pleased with the way it works, but I’ve always thought it must be possible to do this kind of message manipulation / transformation somewhat easier. In other words, without having to write custom programs, ensure they’re running, etc.

There is a way, using Node-RED.

Node-RED console

Node-RED is a visual tool for wiring the Internet of Things. Node-RED, created by Nick O’Leary, is built upon Node.JS. It has a Web-based rapid-prototyping panel (see screen shot above), and it offers a number of building blocks I click and wire together. The building blocks are arranged into groups such as:

  • Inputs (e.g. mqtt, http, tcp, udp)
  • Outputs (e.g. mqtt, http, notify (a.k.a. Growl), tcp)
  • Functions (e.g. template, switch, delay)
  • Storage (e.g. mysql, file, redis)

What I then do, is to drag the building blocks I want onto the work area and wire those together. If necessary, and it will sometimes be, I add function blocks which contain JavaScript with which I manipulate data which flows into and out of the function, and said function blocks can have more than one output (see my transform payload function above).

It is fascinating! (I don’t use the a-word.)

The flow shown in the screen shot above works as follows:

  1. It subscribes to an MQTT topic branch of nt/+.
  2. Arriving messages flow into the topicswitch, which checks the topic name. Messages published on topic nt/notify are passed to notif (Growl, in green on the top right), messages published on topic nt/data are passed to the cclookup.
  3. cclookup is an http request node. It does an HTTP POST of the payload in the message, and reads the returned HTTP body (here in JSON), which is handed off to
  4. transform payload, a function with two outputs: 1) mysql, and 2) mqtt.

When everything is wired up, I hit the big red “Deploy” button on the UI, and off Node-RED goes.

Let me show you what happens when I publish the following message with

$ cat /tmp/in
{ "cc" : "NL", "tst" : 123 }

$ mosquitto_pub -t nt/data -f /tmp/in

A running subscription to the same broker will show me:

$ mosquitto_sub -v -t 'nt/#'
nt/data { "cc" : "NL", "tst" : 123 }
nt/NL Netherlands

The first message (nt/data) comes from the “manual” publish with mosquitto_pub. The second (nt/NL) is the result of the Node-RED “piping” to an HTTP POST, obtaining that result and re-publishing to a different topic (and you’ll believe me when I say the output was also stored in MySQL thanks to it being piped to mystore).

a function in the editor

This really is brilliant.

I’m not yet sure how much I’ll be using Node-RED, but it will certainly be of tremendous use to people who don’t want to have to write custom code (other than in JavaScript, that is).

MQTT and NodeJS :: 05 Dec 2013 :: e-mail