Dinner at the NLUUG conference a few weeks ago was rather good: I chose carpaccio and a Black Angus and placed myself at the end of the long table sitting between Jos and Raymond, with Bill opposite us to the left.
The carpaccio had just been served when Raymond, who had listened to me speak about MQTT, said: “may I ask you a question?” To cut a long story short, I had plenty to think about on the subsequent late-night drive home. (And this post should have possibly been titled “Zabbix, meats, and MQTT”. :-)
Raymond explained to me that Zabbix transports monitoring data insecurely and whether I thought MQTT would be a solution because it has TLS and authentication built into it? In spite of my knowing just about as much about Zabbix as about rearing Angus cattle, I said yes. And it is.
This is what we want to accomplish:
- when a host comes on-line it should register itself as such in Zabbix.
- a host should be able to report any number of metrics via MQTT which are made available as Zabbix items.
- MQTT’s Last Will and Testament (LWT) will be leveraged to provide food for Zabbix’s triggers. In other words, if the monitored host dies, the MQTT broker will publish its demise on its behalf.
- all data must be encrypted in transit. (A basic feature of MQTT.)
- clients should be able to authenticate before being able to submit metrics. (This too is a basic MQTT feature.)
- new items published to Zabbix should optionally be created on the fly
So, assume for a moment, what I’m calling the “Zabbix MQTT agent” comes up, it would publish messages such as these:
(Whether Zabbix keys are slash-separated to accommodate MQTT topic name structures or whether they’re dot-separated to adhere to Zabbix naming conventions is up to discussion.)
I started studying as much as time permitted, and I found several potential entry points for data into Zabbix. (If you are a Zabbix expert, take what follows with many pinches of salt (remember this is all very new to me), and if you aren’t you won’t notice mistakes I make…)
- User parameters
- Loadable modules written in C
- External checks
- Trapper items
- The Zabbix REST API
- Low-Level Discovery (LLD)
User parameters are scripts which get invoked by Zabbix Agent on demand, which is unsuitable for what we want to do. Loadable modules provide an Agent with data, but it’s not possible to have the module submit new keys at will: the agent has to be restarted, so this is out of the question, and external checks are similarly not suitable as they are a one-shot kind of thing where the Zabbix agent forks/execs; nice for ad-hoc queries, but unsuited to what we intend doing.
What I then attempted was to create a trapper item with the API and feed metrics to it with the zabbix_sender protocol. That worked nicely if I either waited a minute between those two or reduced the
CacheUpdateFrequency configuration setting, but it didn’t seem like the right way to do it.
Raymond taught me about Low-Level Discovery (LLD) and he implemented some prototypical shell scripts. After looking at those, and asking a few questions, the light went on. The manual page for LLD says:
Low-level discovery provides a way to automatically create items, triggers, and graphs for different entities on a computer. For instance, Zabbix can automatically start monitoring file systems or network interfaces on your machine, without the need to create items for each file system or network interface manually. Additionally it is possible to configure Zabbix to remove unneeded entities automatically based on actual results of periodically performed discovery.
and that does indeed sound like what the doctor ordered.
As soon as mqttwarn sees a new client come up (
zabbix/clients/jog09 with a payload of
it sends a low-level discovery request to the Zabbix trapper, which creates the host from
a template, and instantiates the template’s items.
1 is used to populate what I call a “status key” which can be used by Zabbix
to trigger up/down alarms.
From that moment onwards, MQTT publishes to the
zabbix/item/ topic branch with
a client hostname and a Zabbix key contained in the topic name cause the items
to be populated with the corresponding values.
The configuration for mqttwarn is simple enough, but we do need a transformation function (see the documentation) for it in order to extract the client and the Zabbix item’s key name from the MQTT topic.
Similarly to what I did when I asked How do your servers talk to you?, I can configure Zabbix to send alerts via MQTT with an external media script, but I haven’t yet found a way to create an action in a template. Anybody?
This is but a beginning, but I believe it has potential. Tell us what you think.