I’m writing this because Michael recently tweeted a link I looked at. I shivered so hard that I want to, without shaming or blaming, attempt to improve on what I saw.

The original poster was using a PAM (Pluggable Authentication Module) to notify (via Pushover) that a login had happened on a machine. Not a bad idea at all, and I learned about pam_exec. However the notification dance was being done with a shell script which invoked curl to do the protocol thing and Pushover’s credentials were in the shell script. Those things are what caused some ice-cold monster to walk down my spine for some reasons:

• hard-coded credentials for a service like Pushover in a bunch of scripts on a whole number of sytems? That’s pretty awful, nay, horrid, unless you can at the very least manage those systems automatically with, say, Ansible
• the script I saw used curl to connect to a service. That’s fine and dandy, but what if the service isn’t available? SSH will hang. You don’t want SSH to hang on you. Well you might not mind, but I mind a lot.
• In order to avoid that, the script backgrounded that curl portion with nohup. That could mean DDoSsing your own box by having a while number of backgrounded processes hanging around. I’ve seen it happen.

What the original poster wanted was an alert of an SSH login to his machines. I think I can solve that with a few bits and pieces of code. Admittedly there may be a larger total of moving parts, but because of how I use them, we’re centralizing those parts, and I’m going to keep the PAM configuration (and the original alert of an SSH login) to an absolute minimum.

Here’s what’s going to happen:

1. A very small compiled C program is going to produce a UDP datagram informing of a login. That datagram carries a short payload with the username of the person logging in, the source address, etc. That UDP datagram could possibly get lost (tough luck), but it’s not likely to in our LAN. Furthermore, if the UDP listener isn’t there, the login isn’t at all hindered or delayed; users won’t notice which is fine. For the record, if this program dies (even with a signal 11) it won’t hurt; PAM will silently ignore it which is exactly what I want to happen.
2. A UDP listener gets the login datagrams and processes them. How it does so is up to you. I’ll have them published to MQTT.
3. If you’ve been here before, you know what’ll then happen. :-) What I do here is to use what I once called twitter for my network: MQTT.

So, fifty-odd lines of C give the original version of the hare utility which is what we’ll be invoking from PAM, and the way we invoke this is by configuring it in /etc/pam.d/sshd (tested on CentOS Linux and on FreeBSD 10.2) as

session    optional     pam_exec.so /usr/local/sbin/hare 192.168.1.131

(Hare you ask? I believe Karanbir once told me many, many years ago, that the CentOS project had a Nabaztag rabbit which alerts them of logins. I loved that story, and this system I describe here is dedicated to that rabbit; I’m calling it hare. (Or is it because I saw something hairy? :-))

So whenever somebody uses the SSH subsystem (ssh, scp, sftp) our hare program will talk to its daemon, hared, on the specified server. This hared is so small I have it right here:

Update: meanwhile it’s turned into a Pip-installable package, and Juzam’s made a fully-compatible Go version of the daemon :-)

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import paho.mqtt.publish as mqtt
import socket
import json

__author__    = 'Jan-Piet Mens <jp()mens.de>'

server_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server_socket.bind(('', 8035))

while True:
    message, address = server_socket.recvfrom(1024)
    host, port = address

    ''' ensure we have JSON '''
    js = json.dumps(json.loads(message))

    mqtt.single("logging/hare", js, hostname='127.0.0.1')

All it does is wait for UDP datagrams, and it then packages each up as a JSON string which it publishes to MQTT. Update: due to lots of reasons, I’ve made the code configurable etc. and it is now here and installable via Pypi.

$ mosquitto_sub -v -t 'logging/hare'
logging/hare {"tst": 1522152142, "hostname": "zabb01", "user": "jjolie", "service": "sshd", "rhost": "192.168.1.130"}
logging/hare {"tty": "ttyv0", "service": "login", "hostname": "fbsd103", "user": "root", "tst": 1522161689, "rhost": "<unknown>"}

$ cat /var/log/ssh-logins.log
login via sshd by jjolie on zabb01 from 192.168.1.130 at 2018-03-27 14:02:22
login via login by root on zabb01 from <unknown> at 2018-03-27 14:07:45

In order to actually process those messages we have mqttwarn which is configured to produce a file with logins (as shown above; note how the login service shows up as well as I’ve also configured that to use hare in PAM), can notify via Pushover (as shown at top), and can easily be used to use any number of the meanwhile 65 services to process these messages. It’s also mqttwarn which translates the epoch timestamp from the JSON payload into a human-readable time.

Here’s the configuration I’m using:

[defaults]
launch	 = log, pushover, file, smtp
functions = 'pamfuncs.py'

[config:file]
append_newline = True
overwrite = False
targets = {
   'mylog'     : ['/var/log/ssh-logins.log'],
  }

[config:pushover]
targets = {
    'pam'       : ['xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', 'yyyyyyyyyyyyyyyyyyyyyyyyyyyyyy'],
  }

[config:smtp]
server  =  'localhost:25'
sender  =  "MQTTwarn <jpm@localhost>"
username  =  None
password  =  None
starttls  =  False
targets = {
    'admins'    : [ 'rooters@ww.mens.de' ],
    }

[logging/hare]
targets = pushover:pam, file:mylog, smtp:admins
alldata = moredata()
title: SSH login on {hostname}
format = login via {service} by {user} on {hostname} from {rhost} at {tstamp}

It’s easy for us to add additional targets without touching PAM configuration on any machine, and without changing our hare system: we add a target to mqttwarn, and there you go: you have mail:

Tue, 27 Mar 2018 14:02:22 +0200
From: MQTTwarn <jpm@localhost>
To: rooters@ww.mens.de
Subject: SSH login on zabb01
X-Mailer: mqttwarn

login via sshd by jjolie on zabb01 from 192.168.1.130 at 2018-03-27 14:02:22

So what are the benefits or pitfalls of doing it the way I did?

• Credentials for the Pushover service are securely stashed on a single machine, the machine which runs mqttwarn. If we want to or have to change them, e.g. because they got compromised, we do so centrally.
• All nodes to which people login talk to a single machine in my network. It is from that one system, again the one running mqttwarn, that we access the external service.
• If we want to rip out Pushover and alert/notify with some other service, or if we want to add some additional type of notification we can do this centrally.
• Above all, we ensure that the SSH login service on the nodes will not hang or somehow be delayed.
• One (slight) disadvantage is that we have to create architecture-dependent versions of hare. We have to distribute/install those, but that we easily do with our configuration managment system.

I explain how a lot of the mqttwarn stuff works in how do your servers talk to you?. I hope some of these technologies make the lives of your systems safer and your life easier.