It’s a bit of a (cough) bother to install, but it’s certainly worth doing: Graphite is a real-time, scalable graphing system which basically allows me to collect statistics (counters, gauges, timers) and, once these numbers are in Graphite’s database, I have access to a dashboard which lets me view (almost in real-time) graphs of these counters.

I’ve been experimenting a bit with it, and, well it’s pretty wow. Let me first show you a result, obtained by plugging a bit of Lua into the PowerDNS Recursor server. Details below, but suffice to say that PowerDNS now fires off a small UDP datagram (more on that in a moment) upon being queried, and the following diagram shows the throughput in queries per second.

PDNS recursor

It took me half a minute to compose that layout by pressing a few buttons on Graphite’s Web interface. :-)

Let me take a step back. Or two.

Graphite is a collection of Python modules and programs which provide a back-end data storage engine called Whisper which is basically a glorified RRD. Whisper is fed by Carbon, the data collector. Carbon listens on TCP port 2003 (and optionally UDP) for lines of statistics which it stores in Whisper and which it provides to the Graphite Web interface. (Carbon in fact is composed of three distinct processes, but that’s irrelevant to this discussion.) On top of Graphite I run Apache or a WSGI framework to provide Web access to Graphite.

So far so good? Great. It gets worse. :-)

To streamline delivery of metrics to Carbon, Etsy created StatsD, a simple daemon which accepts UDP datagrams containing one or more metrics I want recorded. It aggregates these metrics up every (configurable) 10 seconds and pushes them to Carbon for delivery to Whisper. Basically, StatsD allows me to fire off lightweight UDP datagrams (fire & forget) and accumulates these for me before handing them to Graphite for drawing. Etsy’s StatsD has become so tremendously popular, that there are StatsD implementaions in umpteen languages available.

The way I picture the whole architecture is like this:

Graphite/StatsD architecture

  • The probes and the app I write. The former fire off UDP datagrams to StatsD and the latter connect to Carbon via TCP. I use either one or the other or both, depending on how I want to provide metrics to Graphite’s Carbon.
  • StatsD is one of the UDP collectors.
  • Apache or a WSGI framework serves up what Graphite renders
  • All other components are part of Graphite

Once Graphite’s components are installed and running, I connect via Web browser to the Graphite host on port 8085 and can start clicking on metrics that show up in a tree on the left of the screen. Alternatively, I can use Graphite’s dashboard on which I can click my way into the statistics I want to visualize (here the same PowerDNS queries and the CPU utilization of my workstation). There’s also a really cool Web-based CLI interface [sic] with which I can create diagrams etc.

Graphite dashboard

I can also directly link to specific graphs, for with a URL like this

and I obtain a CSV of the raw data by tacking a parameter onto the URL:


Going back to the first diagram (PowerDNS Recursor statistics), that’s implemented with a Lua as one of the little probes (one the little gray boxes on the left of the diagram above). The script fires a UDP datagram to a listening StatsD server which hands the lot off to Graphite. This is the script:

require "luarocks.require"

st = statsd('', 8125) -- address,port of StatsD server

-- note: influenced by Recursor caching!
function preresolve( ip, domain, qtype )
    ret = {}
    st:recordCount('pdns.query', 1, 1)  -- statds receives: pdns.query:1|c|@1
    return -1, ret

That about covers it, I think.

One more thing: I’ve been talking about Logstash recently and it comes prepared to work with Graphite or StatsD with its statsd and graphite outputs.

Further reading:

Logging and Statistics :: 23 Aug 2012 :: e-mail