Is it possible to create a minature and very inexpensive GPS online and offline tracker which is OwnTracks-compatible? Yes, it is possible.

During a bout of craze^W inspiration, I obtained a u-blox Neo 6M GPS module (EUR 9.30) and connected it to an Wemos-D1 mini ESP8266 module (EUR 3.35). Within the hour, I was happily publishing OwnTracks JSON location messages to a local broker over WiFi. The device is so small I named it Pico. (The rubber bands you see in the photo below are part of the cough design cough to make it vehicle-ready, and the LED is simply for additional blinkenlights of course. The black box is a USB battery pack.)

Pico

The idea would have been to use mobile phone WiFi tethering or a MiFi to take the result along in a vehicle, but that would have been a bit boring. I learned during my experience with Homie that the ESP8266 has an SPIFFS file system, and you can probably guess what's going to happen.

When the Pico loses connection to its MQTT broker or disconnects from WiFi, it continues in offline mode, recording to the file system. Once a minute it attempts to reconnect, and if it succeeds, it publishes what it's collected so far to the configured MQTT broker. This appears to work quite reliably.

map

The only slight handicap in Pico is, that the PubSubClient MQTT library has a default maximum MQTT message size of 128 octets which is too little for the information we want in OwnTracks JSON. (We don't just publish lat and lon, but also velocity, altitude, distance travelled, and a few other bits.) We could easily change definition in PubSubClient, but I opted in the interest of portability and in the interest of maximizing the number of offline location publishes to use OwnTracks CSV which we developed for our Greenwich devices, and which is already fully supported by the Recorder. This means, with a 3MB file system on the ESP8266 we can store in excess of 60,000 points.

The CSV content may look a bit strange because we hexalize (is that a word?) the time stamp, unfloat (is that also a word?) lat and lon (we save the space of two decimal points) and do a few other optimizations (in the interest of conserving valuable GSM data).

CSV format

The Pico publishes (or serializes to the file system if it's offline) a location when it detects it's moved mindist meters (100 by default) and 5 seconds interval have elapsed. When it's stationary, it'll publish a ping location (that's what we call it) once an hour, and on booting it publishes a first location to tell us it's alive.

A challenge would now be to take a EUR 4.40 GSM/GPRS module and allow the Pico to be a fully capable online tracker which publishes MQTT over the Internet via GPRS. Any takers?

View Comments :: GPS, MQTT, and OwnTracks :: 16 Apr 2016 :: e-mail

Two of the features which most impressed me of the Particle Photon were its built-in WiFi, and the possibility of updating the device over the air (OTA). However, using a Photon as, say, a simple temperature sensor is quite expensive at over EUR 23.00 when purchased here.

Dirt cheap in comparison is one of the ESP family of microcontrollers and development boards, and I ordered a few so-called NodeMCU dev kits (right in photo at EUR 3.89 a piece) and Wemos D1 (center, at EUR 3.63 per unit). I have not made a mistake on the price tags; they even include postage and handling from China. All we need is a fortnight's patience. (And while waiting you find that the adorable Wemos-D1 has a boat-load of shields which you should also have ordered ... :-)

Wemos-D1 and NodeMCU

Both of these are based on the ESP8266, and as such they have on-board WiFi. The NodeMCU can be flashed with firmware which supports the Lua programming language (or even BASIC I hear). I've settled on using an environment I'm more or less comfortable with, namely the Arduino for ESP8266 firmware which is trivial to install through the Arduino IDE.

These little microcontrollers just work, but what is a bit of a chore is having to invent boilerplate code which enables reconnection, reboot on error, etc.

I stumbled over a project called homie-esp8266 created by Marvin Roger. It's basically a small framework which couples the ESP8266 to MQTT and provides automatic connection/reconnection handling, JSON configuration, an optional Web UI to configure the microcontroller which starts up in Access Point mode initially, and Over-The-Air support for updating the code I create for the ESP8266. Yes: OTA support, and TLS.

Martin has obviously carefully thought about how a device should interact with its environment, and the code reflects this. So, as an example, when one of my sensors launches, it will publish these retained MQTT messages:

sensors/homie/cf3a07e0/$nodes light:light,temperature:temperature,relay:relay
sensors/homie/cf3a07e0/$online true
sensors/homie/cf3a07e0/$name Livingroom bookshelf
sensors/homie/cf3a07e0/$localip 192.168.1.192
sensors/homie/cf3a07e0/$fwname h-sensor
sensors/homie/cf3a07e0/$fwversion 1.0.1
sensors/homie/cf3a07e0/temperature/unit c
sensors/homie/cf3a07e0/$signal 96
sensors/homie/cf3a07e0/temperature/degrees 23.38

$nodes inform the consumer about the type of endpoints I've made available on the device. $online indicates whether the device is currently alive, $name is a description I initially assign to the device, $signal is an indication of WiFi coverage as seen by the microcontroller, etc. Then the device starts publishing a temperature and whatever other sensor data I wish to make available from the device.

The homie-esp8266 framework also listens for messages: for example, I can switch a relay I have connected to a Wemos-D1 with the following command:

mosquitto_pub -t 'sensors/homie/cf3a07e0/relay/on/set' -m true

The code I create for the device is invoked when Homie receives above publish, and it can then do something, e.g. switch the relay on, and instruct Homie to note that the action occurred, whereupon Homie will publish a retained value indicating the current state to (note the missing /set at the end of this topic):

sensors/homie/cf3a07e0/relay/on true

Using the homie-esp8266 framework makes this trivial to do:

#include <Homie.h>

#define RELAY_PIN 16

HomieNode relayNode("relay", "relay");

bool relayHandler(String message)
{
    if (message == "true") {
        digitalWrite(RELAY_PIN, HIGH);
        Homie.setNodeProperty(relayNode, "on", "true");
    } else if (message == "false") {
        digitalWrite(RELAY_PIN, LOW);
        Homie.setNodeProperty(relayNode, "on", "false");
    } else {
        return false;
    }

    return true;
}

void setup()
{
    pinMode(RELAY_PIN, OUTPUT);
    digitalWrite(RELAY_PIN, LOW);

    Homie.setFirmware("h-sensor", "1.0.1");
    relayNode.subscribe("on", relayHandler);
    Homie.registerNode(relayNode);
    Homie.setup();
}

void loop()
{
    Homie.loop();
}

I mentioned above that Homie, can be configured via a Web-based interface when a device initially starts (to be quite clear: the site in the screenshot justs hosts the HTML to talk to your Homie ESP; the device itself doesn't talk to the cloud):

configurator

Once that is done, the device connects to the WiFi network you specify during configuration, and starts operating on that. Alternatively, we can upload a JSON configuration file to the flash file system on the device or create that programatically (which is what I did, using a small sketch). The great advantage of storing the configuration separately from the actual Homie firmware is I can reflash (for example OTA) a different sketch onto the device and it will use the previous configuration settings (e.g. for MQTT broker, port, etc.).

OTA updates are triggered by publishing a version number to the $ota endpoint; Homie then verifies the version it's received is higher than the one it's running, and instructs the ESP to boot and get the update via HTTP/HTTPS. (How this works is rather well explained here.) I think it's important to mention that the homie-esp8266 framework is cloudless; it doesn't require a cloud service, and it talks exactly to the servers you configure it to speak to.

Homie's documentation is good and plentiful, and there are examples to get us started quickly. Marvin has been tremendously helpful, and he very quickly implemented the possibility to define a base_topic for MQTT publishes when I asked him to.

A stamp-sized chip with on-board WiFi and 4MB of flash which speaks MQTT over TLS and can be updated OTA for under EUR 4.00. Exciting times.

Further reading:

  • Ben found another firmware for ESPs called ESPEasy. I gave it a glance, and it looks quite capable, but it appears to be geared to people who want a bit more of a toolbox on their ESP.
View Comments :: MQTT and IoT :: 22 Mar 2016 :: e-mail

I tired of mangling shell scripts to produce JSON. You've likely seen this somewhere:

echo '{"name":"Jane"}'

It gets merrier if an element contains an environment variable: open double, close single, add variable, open single, blergh.

A here! script will do it as will a a printf(1), but neither much improve legibility:

printf '{"name": "%s"}\n' "Jane"

Logo

Enter jo:

$ jo name=Jane
{"name":"Jane"}

The idea occurred to me late at night, and I don't know why this has taken so long to happen:

$ jo time=$(date +%s) dir=$HOME
{"time":1457195712,"dir":"/Users/jpm"}

Bam! Jo tries to be clever about types and knows null, booleans, strings and numbers. It does arrays, and it pretty-prints on demand:

$ jo -p -a spring summer winter
[
   "spring",
   "summer",
   "winter"
]

Inspired by a comment on HN, I added another hack: if a key's value begins with an opening brace ({) or a bracket ([]) we attempt to decode JSON from it; this allows jo to add objects or arrays (use -a!) to itself. Watch:

$ jo -p name=JP object=$(jo fruit=Orange point=$(jo x=10 y=20) number=17) sunday=false
{
   "name": "JP",
   "object": {
      "fruit": "Orange",
      "point": {
         "x": 10,
         "y": 20
      },
      "number": 17
   },
   "sunday": false
}

jo also supports nested types natively:

$ jo -p number=17 pass=true geo[lon]=88 geo[cc]=ES point[]=1 point[]=2 geo[lat]=123.45
{
   "number": 17,
   "pass": true,
   "geo": {
      "lon": 88,
      "cc": "ES",
      "lat": 123.45
   },
   "point": [
      1,
      2
   ]
}

Why did I do this? I need lots of JSON for testing OwnTracks, and this just looks better in scripts.

$ jo _type=location \
   cog=$((RANDOM % 360)) \
   t=u \
   lat=48.85833 \
   lon=2.29513 \
   acc=5 \
   tid=JJ \
   tst=$(date +%s) | mosquitto_pub -t owntracks/jjolie/test -l

A suggestion by Andrew Bibby brought along the possibility of obtaining JSON element values from files, so I added @file to read a file (sans traling newline or carriage return) into a value as well as something which I use a lot, convert binary files to base64, using the %file syntax:

$ jo _type=card name="Vanessa" face=%vanessa.png
{"_type":"card","name":"Vanessa","face":"iVBORw0KGgoAAA ... QmCC"}

And it has a man page. Go get it. Jo!

Updates:

View Comments :: Toolbox and JSON :: 05 Mar 2016 :: e-mail

Christoph and I have been working on automating a few things at openBerlin, a Cisco innovation center. In particular, they have office spaces with banks of lights in them but no wall switches or controls. Instead, a non-optimal Web page allows people to change the brightness and colour of the lamps in the offices, assuming they know which office they're sitting in and which lamp is "theirs". As openBerlin has a number of iBeacons located around the office, we thought we could pimp OwnTracks in such a way as that a person could whip out their smartphone and, at the click of a button, switch the lamp they are sitting under. To cut a long story short, it works.

What we actually do is related to iBeacons in the office in that a beacon transition event (e.g. "JP is entering the proximity of beacon #47") allows us to know which beacon a user is closest to, and as there are lots of beacons, we can associate a beacon with a lamp, and therefore know where somebody is sitting. So far so good, but how do we provide a "click here to control your lamp"?

featured content

The OwnTracks iOS app has a new tab which is shown when it receives a particular MQTT message of type cmd.

{"action": "action", "content": "Click here: http://example.org/lamp/1", "_type": "cmd"}

That tab disappears if content is omitted, hence with an enter event we show the tab, and with a leave event we can hide the tab.

We were then able to load a page which allows a user to click to be directed to a page to switch or dim the lamp they are closest to. (In the openBerlin infrastructure, this means talking to a REST api which actually then control the lights.)

When we told Ben about this, he asked how we talk to openHAB. I was a little stumped because this has nothing to do with openHAB, but I then thought: hey, why not? We discussed this a bit and Ben suggested creating individual openHAB sitemaps with one switch each and then load those as pages.

openhab sitemap

Within minutes, Christoph had the OwnTracks app using the alternate url instead of content, and that allowed us to load a page into the application's tab:

{"action": "action", "url": "http://ohab.ww.mens.de:8080/openhab.app?sitemap=zn", "_type": "cmd"}

One thing led to another, and my resident openHAB consultant tought me a few new tricks, in particular, that I can configure a switch to have multiple bindings on it, for example an actual Zwave switch which can be triggered with an MQTT publish to lights/zn/cmd and which retains the switch's current status at the topic lights/zn/status:

Switch Zn "Zn plug" (Lights,gSunset) {
   zwave="14:command=switch_binary",
   mqtt="<[localmosquitto:lights/zn/cmd:command:default]",
   mqtt=">[mqretain:lights/zn/status:state:*:default]"
}

So, an MQTT publish will switch the lamp on or off:

mosquitto_pub -t lights/zn/cmd -m ON

The result is a jQuery mobile page loaded by OwnTracks which speaks MQTT over Websockets to the MQTT broker openHAB is configured to use.

Light control

This new OwnTracks feature works well, and I could imagine people will find plenty of uses for it with or without iBeacons.

Getting back to the beginning of this story, what we're really doing is throwing technology at the fact that the building's architect omitted wall switches; that is something I wouldn't have done.

View Comments :: OwnTracks, iBeacons, and automation :: 15 Feb 2016 :: e-mail

Having a few temperature sensors at my disposal, I made a big mistake: I compared the values they're emitting.

Sensor values

From left to right we have here, in an area of 20 cm2, the following sensors:

Humidity is even more off than temperature:

humidity

The TI SensorTag CC2650 currently doesn't feel like reporting humidity, so there!

Who actually calibrates these things? But much more importantly: how high is the temperature in my office?

View Comments :: sensors :: 02 Feb 2016 :: e-mail

Other recent entries