There’s a rather busy Graylog installation next door which dissects messages given it via a few syslog inputs and pushes these into a stream to be viewed in a Graylog dashboard. In particular, it attempts to determine a user connecting via POP3 or IMAP, and this has been working rather well.
I was given the task to think about how we could store a per/day count of unique users, and I started dabbling about with the stream settings, to find I could export a stream via GELF, so I did that.
The GELF forwarder pushes each message to a small Python program (prototype production version below) which sees this after unpacking the GELF:
The small utility opens a UDP datagram port and waits for messages to flow in, unpacking the GELF from each and processing the data.
This has been running for a couple of weeks and appears to be quite reliable, even with the volume of messages we’re seeing.
At the end of the day we can read the incremented values which are stored in redis.