You know the drill: you provision a server with a master zone, and then you have to hop over to all secondary servers and add the slave zone to their configuration. You probably do that with some form of automation, or you use something slightly convoluted like what we've discussed previously in automatic provisioning of slave DNS servers. If your master and slave servers are BIND, you're in luck: catalog zones will automate this for you within the BIND code itself: there'll no longer be a need for "hacking" this to accomplish automatic provisioning of slave BIND servers.

Catalog zones are scheduled for release in BIND 9.11, but Evan Hunt graciously let me have a peek at a preview of the code, and I must warn you: first, this is a work in progres and hasn't been released yet, and second (and more importantly) what I'm going to show you is the little I know of catalog zones. Be warned on both accounts!

Catalog zones are losely based on an (expired) Internet-Draft called draft-muks-dnsop-dns-catalog-zones-00 which describes a method for automatic synchronization of zones among primary and secondary servers. The way this is works is that a zone contains the names (and optional metadata) describing which zones are to be used on slave (secondary) servers; these zone names in a catalog are called member zones. The secondary server obtains a copy of this catalog (via zone transfer (AXFR/IXFR)) and uses it to update the internal catalog of zones it ought to have; once it detects a change, it adds and slaves the member zone or deletes it, depending on whether the member zone was added or removed.

More concisely: we add one or more member zones to the catalog zone which is transferred to its slaves where it triggers the creation (or removal) of the member zones.

Catalog zones

Let's see how this works in practice with a preliminary copy of BIND which supports this.

Primary and secondary servers

On my primary test server, I add the following to named.conf:

options {
        directory "/etc/namedb";
        server-id "authoritative";
        allow-new-zones yes;

zone "" {
        type master;
        file "";
        also-notify {; };
        notify explicit;

Pay attention to the allow-new-zones directive. The catalog zone proper is mostly empty: mname, rname and NS are irrelevant, and the relative version label specifies the catalog zone format version:

@ 3600 SOA . . 1 86400 3600 86400 3600
@ 3600 IN NS nop.
version IN TXT "1"

On a secondary ( I have the following configuration:

options {
        directory "/etc/namedb";
        masterfile-format text;
        zone-statistics yes;
        server-id "slave";

        allow-new-zones yes;
        catalog-zones {
                zone "" masters {; };

zone "" {
        type slave;
        file "";
        masters {; };

Here we also add allow-new-zones so that the secondary can create member zones on the fly, and we use the new catalog-zones stanza to define the primary server which holds our catalog. The masters statement in the catalog-zones stanza defines the default masters for its member zones.

Adding a new member zone

On the primary server I add a new zone, either dynamically with rndc addzone or manually; I'll use the former:

$ rndc addzone '{type master; file "";};'
$ dig +short @localhost soa
localhost. root.localhost. 22 10800 3600 604800 3600

From this point onwards, my primary server is ready to serve this master zone, but I want the secondary to serve the zone as well. Here's what I do:

  1. I create a hash of a value. Basically any value should do, but it must currently be a valid BIND "nzf" hash. (The pre-alpha code I have does a little oops if it isn't -- I'm reporting that as we speak.) ISC is considering loosening this rule to allow me to use any label I wish to, but for interoperability with possible future implementations which comply with the draft, the hash is preferable.
  2. I then add a record to the catalog zone in which I name the member zone, bump its serial and reload the zone.
#!/usr/bin/env python

import hashlib
import sys

print hashlib.sha1([1]).to_wire()).hexdigest()

I run the above (thanks Witold!) to create the hash:

$ ./

I now use that hash for step 2 and add a record containing the member zone:

@ 3600 SOA . . 2 86400 3600 86400 3600
@ 3600 IN NS nop.

47ac1a4d93b61fffdb4762c18c9e7d1a6b046d33.zones           PTR

When I reload named on the primary (rndc reload), I watch the transfer logs; they look very promising:

xfer-out: info: client ( transfer of '': AXFR-style IXFR started (serial 2)
xfer-out: info: client ( transfer of '': AXFR-style IXFR ended
xfer-out: info: client ( transfer of '': AXFR started (serial 22)
xfer-out: info: client ( transfer of '': AXFR ended

Note how first the catalog zone was transferred, and then our new zone.

So, what's on the secondary?

$ ls -l /etc/namedb
-rw-r--r-- 1 root root 2389 May 24 17:22 bind.keys
-rw-r--r-- 1 root root  327 May 24 19:33
-rw-r--r-- 1 root root  316 May 24 19:33
-rw-r--r-- 1 root root 2048 May 24 18:05 named.conf
-rw-r--r-- 1 root root   91 May 24 17:25 rndc.key

The __catz__* file contains a copy of the slaved zone. These files are currently all stored in named's configured directory.

If I delete a member zone record from the catalog the process is reversed, and the secondaries remove the zone completely:

catz: updating catalog zone '' with serial 3
catz: deleting zone '' from catalog '' - success
catz: catz_delzone_taskaction: zone '' deleted


The BIND test suite hints at having the possibility of setting metadata on a zone and transferring with the catalog, for example different master servers for a particular zone. According to the ARM, this is currently the only metadata implemented in BIND for catalog zones.

This allows us to add a member zone to the catalog, which is configured with different master servers. So, on my primary server, I can add the following to the catalog: the first record specifies a member zone, and the second record the master(s) for that zone.

45414dc99ae20a84507692de578c0dd9d2522134.zones           IN PTR
masters.45414dc99ae20a84507692de578c0dd9d2522134.zones   IN A

When the secondaries get the catalog, this member zone is created and configured to obtain a copy of its data from the specified master server.


It goes without saying (so why am I?) that the catalog zone can be dynamically updated of course; this makes provisioning easy and ensures the SOA serial number is bumped.

We should probably all wish for other Open Source DNS server implementations to embrace catalog zones and add support for them in order to increase interoperability between DNS server software brands.

View Comments :: BIND and DNS :: 24 May 2016 :: e-mail

I took the plunge and decided to "design" and create a small PCB intended to keep together neatly the bits I want on a couple of sensors.


Writing a book was, for me, a lot more work than designing a small printed circuit board (PCB), but even so, the design took a few hours to create, particularly since I wanted to squeeze everything as closely together as possible to save cost on the manufacturing. The book-writing gave me a lot more satisfaction, but I'll admit to being tickled when I received a thin envelope with two very small PCBs in it.


I suppose most of you will snort at seeing that, as it's basically just a holder for a few integrated parts and not a real board design, but that doesn't bother me at all; snort as much as you like.

I created these using Fritzing and used their (not inexpensive) Fritzing Fab PCB factory to produce them. They're located near Berlin, so I was hoping for quick turnaround which I got. The PCB has a Wemos-D1 ESP8266 on it with a DS18B20 digital temperature sensor and a digital Lux meter thing which flips when luminosity goes under or above a threshhold which is set on it. And the PCB works.


The firmware running on the Wemos-D1 mini is based on Homie because I've had very good experiences with it, and it's rock stable.

There will probably be a second iteration of the Homie/LT PCB. For one, the big fat red LED will go away (yes, I was told it'd be idiotic, but I wanted blinkenlights!), and we'll probably add a simple LDR instead of the digital light sensor as the latter is very tall and gives the whole thing a rather ungainly look.


Apropos Homie: I decided to create a small Python utility called homie-ota which comes with a really retro-style looking, vintage Web interface upon which you can see an inventory of Homie devices in your network, upload new firmware binary files and issue an OTA (Over the Air) request via MQTT to a device for it to come and get a new firmware. I've been laughed at for the look-and-feel of the Web pages, but Ben has nevertheless been very supportive and has provided a ton of features and patches.


Relay board

Another thing which is running Homie in the house is a small WiFi IoT Relay board which I had to try: it costs $6 including enclosure and postage & handling, and it switches two 220V outputs.


Nathan Chantrell wrote an in-depth review about this and a second module while the Electrodragons were en-route to me. To cut a long story short, I don't regret this purchase, and the devices appear to be safe enough for the load I'm running them on (a lightbulb or two).


The small relay board I described above is well suited to be integrated into my OpenHAB installation which I do via MQTT of course. This items file picks up one of the relays and allows me to switch it:

Group homie "Homie"

String homie_name { mqtt="<[hippo:devices/dr03/$name:state:REGEX((.*))]" }
Switch homie_online { mqtt="<[hippo:devices/dr03/$online:state:JS(homie-tf.js)"}

Switch  homie_relay
    (Relays) { 

Switch homie_relay_setter {

String homie_fwname {
String homie_fwversion { 

Number homie_signal {

String homie_fwname_fwversion

As it's all MQTT-controlled, I can switch a relay either by pressing the switch in the OpenHAB UI or with an MQTT publish:

mosquitto_pub -t 'devices/dr03/relay1/on/set' -m true


It's a huge bit of fun to play with these ESP8266 modules, and at their extremely low cost, there's not much you can do wrong.

View Comments :: automation and MQTT :: 02 May 2016 :: e-mail

Is it possible to create a minature and very inexpensive GPS online and offline tracker which is OwnTracks-compatible? Yes, it is possible.

During a bout of craze^W inspiration, I obtained a u-blox Neo 6M GPS module (EUR 9.30) and connected it to an Wemos-D1 mini ESP8266 module (EUR 3.35). Within the hour, I was happily publishing OwnTracks JSON location messages to a local broker over WiFi. The device is so small I named it Pico. (The rubber bands you see in the photo below are part of the cough design cough to make it vehicle-ready, and the LED is simply for additional blinkenlights of course. The black box is a USB battery pack.)


The idea would have been to use mobile phone WiFi tethering or a MiFi to take the result along in a vehicle, but that would have been a bit boring. I learned during my experience with Homie that the ESP8266 has an SPIFFS file system, and you can probably guess what's going to happen.

When the Pico loses connection to its MQTT broker or disconnects from WiFi, it continues in offline mode, recording to the file system. Once a minute it attempts to reconnect, and if it succeeds, it publishes what it's collected so far to the configured MQTT broker. This appears to work quite reliably.


The only slight handicap in Pico is, that the PubSubClient MQTT library has a default maximum MQTT message size of 128 octets which is too little for the information we want in OwnTracks JSON. (We don't just publish lat and lon, but also velocity, altitude, distance travelled, and a few other bits.) We could easily change definition in PubSubClient, but I opted in the interest of portability and in the interest of maximizing the number of offline location publishes to use OwnTracks CSV which we developed for our Greenwich devices, and which is already fully supported by the Recorder. This means, with a 3MB file system on the ESP8266 we can store in excess of 60,000 points.

The CSV content may look a bit strange because we hexalize (is that a word?) the time stamp, unfloat (is that also a word?) lat and lon (we save the space of two decimal points) and do a few other optimizations (in the interest of conserving valuable GSM data).

CSV format

The Pico publishes (or serializes to the file system if it's offline) a location when it detects it's moved mindist meters (100 by default) and 5 seconds interval have elapsed. When it's stationary, it'll publish a ping location (that's what we call it) once an hour, and on booting it publishes a first location to tell us it's alive.

A challenge would now be to take a EUR 4.40 GSM/GPRS module and allow the Pico to be a fully capable online tracker which publishes MQTT over the Internet via GPRS. Any takers?

View Comments :: GPS, MQTT, and OwnTracks :: 16 Apr 2016 :: e-mail

Two of the features which most impressed me of the Particle Photon were its built-in WiFi, and the possibility of updating the device over the air (OTA). However, using a Photon as, say, a simple temperature sensor is quite expensive at over EUR 23.00 when purchased here.

Dirt cheap in comparison is one of the ESP family of microcontrollers and development boards, and I ordered a few so-called NodeMCU dev kits (right in photo at EUR 3.89 a piece) and Wemos D1 (center, at EUR 3.63 per unit). I have not made a mistake on the price tags; they even include postage and handling from China. All we need is a fortnight's patience. (And while waiting you find that the adorable Wemos-D1 has a boat-load of shields which you should also have ordered ... :-)

Wemos-D1 and NodeMCU

Both of these are based on the ESP8266, and as such they have on-board WiFi. The NodeMCU can be flashed with firmware which supports the Lua programming language (or even BASIC I hear). I've settled on using an environment I'm more or less comfortable with, namely the Arduino for ESP8266 firmware which is trivial to install through the Arduino IDE.

These little microcontrollers just work, but what is a bit of a chore is having to invent boilerplate code which enables reconnection, reboot on error, etc.

I stumbled over a project called homie-esp8266 created by Marvin Roger. It's basically a small framework which couples the ESP8266 to MQTT and provides automatic connection/reconnection handling, JSON configuration, an optional Web UI to configure the microcontroller which starts up in Access Point mode initially, and Over-The-Air support for updating the code I create for the ESP8266. Yes: OTA support, and TLS.

Martin has obviously carefully thought about how a device should interact with its environment, and the code reflects this. So, as an example, when one of my sensors launches, it will publish these retained MQTT messages:

sensors/homie/cf3a07e0/$nodes light:light,temperature:temperature,relay:relay
sensors/homie/cf3a07e0/$online true
sensors/homie/cf3a07e0/$name Livingroom bookshelf
sensors/homie/cf3a07e0/$fwname h-sensor
sensors/homie/cf3a07e0/$fwversion 1.0.1
sensors/homie/cf3a07e0/temperature/unit c
sensors/homie/cf3a07e0/$signal 96
sensors/homie/cf3a07e0/temperature/degrees 23.38

$nodes inform the consumer about the type of endpoints I've made available on the device. $online indicates whether the device is currently alive, $name is a description I initially assign to the device, $signal is an indication of WiFi coverage as seen by the microcontroller, etc. Then the device starts publishing a temperature and whatever other sensor data I wish to make available from the device.

The homie-esp8266 framework also listens for messages: for example, I can switch a relay I have connected to a Wemos-D1 with the following command:

mosquitto_pub -t 'sensors/homie/cf3a07e0/relay/on/set' -m true

The code I create for the device is invoked when Homie receives above publish, and it can then do something, e.g. switch the relay on, and instruct Homie to note that the action occurred, whereupon Homie will publish a retained value indicating the current state to (note the missing /set at the end of this topic):

sensors/homie/cf3a07e0/relay/on true

Using the homie-esp8266 framework makes this trivial to do:

#include <Homie.h>

#define RELAY_PIN 16

HomieNode relayNode("relay", "relay");

bool relayHandler(String message)
    if (message == "true") {
        digitalWrite(RELAY_PIN, HIGH);
        Homie.setNodeProperty(relayNode, "on", "true");
    } else if (message == "false") {
        digitalWrite(RELAY_PIN, LOW);
        Homie.setNodeProperty(relayNode, "on", "false");
    } else {
        return false;

    return true;

void setup()
    pinMode(RELAY_PIN, OUTPUT);
    digitalWrite(RELAY_PIN, LOW);

    Homie.setFirmware("h-sensor", "1.0.1");
    relayNode.subscribe("on", relayHandler);

void loop()

I mentioned above that Homie, can be configured via a Web-based interface when a device initially starts (to be quite clear: the site in the screenshot justs hosts the HTML to talk to your Homie ESP; the device itself doesn't talk to the cloud):


Once that is done, the device connects to the WiFi network you specify during configuration, and starts operating on that. Alternatively, we can upload a JSON configuration file to the flash file system on the device or create that programatically (which is what I did, using a small sketch). The great advantage of storing the configuration separately from the actual Homie firmware is I can reflash (for example OTA) a different sketch onto the device and it will use the previous configuration settings (e.g. for MQTT broker, port, etc.).

OTA updates are triggered by publishing a version number to the $ota endpoint; Homie then verifies the version it's received is higher than the one it's running, and instructs the ESP to boot and get the update via HTTP/HTTPS. (How this works is rather well explained here.) I think it's important to mention that the homie-esp8266 framework is cloudless; it doesn't require a cloud service, and it talks exactly to the servers you configure it to speak to.

Homie's documentation is good and plentiful, and there are examples to get us started quickly. Marvin has been tremendously helpful, and he very quickly implemented the possibility to define a base_topic for MQTT publishes when I asked him to.

A stamp-sized chip with on-board WiFi and 4MB of flash which speaks MQTT over TLS and can be updated OTA for under EUR 4.00. Exciting times.

Further reading:

View Comments :: MQTT and IoT :: 22 Mar 2016 :: e-mail

I tired of mangling shell scripts to produce JSON. You've likely seen this somewhere:

echo '{"name":"Jane"}'

It gets merrier if an element contains an environment variable: open double, close single, add variable, open single, blergh.

A here! script will do it as will a a printf(1), but neither much improve legibility:

printf '{"name": "%s"}\n' "Jane"


Enter jo:

$ jo name=Jane

The idea occurred to me late at night, and I don't know why this has taken so long to happen:

$ jo time=$(date +%s) dir=$HOME

Bam! Jo tries to be clever about types and knows null, booleans, strings and numbers. It does arrays, and it pretty-prints on demand:

$ jo -p -a spring summer winter

Inspired by a comment on HN, I added another hack: if a key's value begins with an opening brace ({) or a bracket ([]) we attempt to decode JSON from it; this allows jo to add objects or arrays (use -a!) to itself. Watch:

$ jo -p name=JP object=$(jo fruit=Orange point=$(jo x=10 y=20) number=17) sunday=false
   "name": "JP",
   "object": {
      "fruit": "Orange",
      "point": {
         "x": 10,
         "y": 20
      "number": 17
   "sunday": false

jo also supports nested types natively:

$ jo -p number=17 pass=true geo[lon]=88 geo[cc]=ES point[]=1 point[]=2 geo[lat]=123.45
   "number": 17,
   "pass": true,
   "geo": {
      "lon": 88,
      "cc": "ES",
      "lat": 123.45
   "point": [

Why did I do this? I need lots of JSON for testing OwnTracks, and this just looks better in scripts.

$ jo _type=location \
   cog=$((RANDOM % 360)) \
   t=u \
   lat=48.85833 \
   lon=2.29513 \
   acc=5 \
   tid=JJ \
   tst=$(date +%s) | mosquitto_pub -t owntracks/jjolie/test -l

A suggestion by Andrew Bibby brought along the possibility of obtaining JSON element values from files, so I added @file to read a file (sans traling newline or carriage return) into a value as well as something which I use a lot, convert binary files to base64, using the %file syntax:

$ jo _type=card name="Vanessa" face=%vanessa.png
{"_type":"card","name":"Vanessa","face":"iVBORw0KGgoAAA ... QmCC"}

And it has a man page. Go get it. Jo!


View Comments :: Toolbox and JSON :: 05 Mar 2016 :: e-mail

Other recent entries