I’m writing this because Michael recently tweeted a link I looked at. I shivered so hard that I want to, without shaming or blaming, attempt to improve on what I saw.

The original poster was using a PAM (Pluggable Authentication Module) to notify (via Pushover) that a login had happened on a machine. Not a bad idea at all, and I learned about pam_exec. However the notification dance was being done with a shell script which invoked curl to do the protocol thing and Pushover’s credentials were in the shell script. Those things are what caused some ice-cold monster to walk down my spine for some reasons:

  • hard-coded credentials for a service like Pushover in a bunch of scripts on a whole number of sytems? That’s pretty awful, nay, horrid, unless you can at the very least manage those systems automatically with, say, Ansible
  • the script I saw used curl to connect to a service. That’s fine and dandy, but what if the service isn’t available? SSH will hang. You don’t want SSH to hang on you. Well you might not mind, but I mind a lot.
  • In order to avoid that, the script backgrounded that curl portion with nohup. That could mean DDoSsing your own box by having a while number of backgrounded processes hanging around. I’ve seen it happen.

pushover screenshot

What the original poster wanted was an alert of an SSH login to his machines. I think I can solve that with a few bits and pieces of code. Admittedly there may be a larger total of moving parts, but because of how I use them, we’re centralizing those parts, and I’m going to keep the PAM configuration (and the original alert of an SSH login) to an absolute minimum.

Here’s what’s going to happen:

  1. A very small compiled C program is going to produce a UDP datagram informing of a login. That datagram carries a short payload with the username of the person logging in, the source address, etc. That UDP datagram could possibly get lost (tough luck), but it’s not likely to in our LAN. Furthermore, if the UDP listener isn’t there, the login isn’t at all hindered or delayed; users won’t notice which is fine. For the record, if this program dies (even with a signal 11) it won’t hurt; PAM will silently ignore it which is exactly what I want to happen.
  2. A UDP listener gets the login datagrams and processes them. How it does so is up to you. I’ll have them published to MQTT.
  3. If you’ve been here before, you know what’ll then happen. :-) What I do here is to use what I once called twitter for my network: MQTT.

So, fifty-odd lines of C give the original version of the hare utility which is what we’ll be invoking from PAM, and the way we invoke this is by configuring it in /etc/pam.d/sshd (tested on CentOS Linux and on FreeBSD 10.2) as

session    optional     pam_exec.so /usr/local/sbin/hare 192.168.1.131

(Hare you ask? I believe Karanbir once told me many, many years ago, that the CentOS project had a Nabaztag rabbit which alerts them of logins. I loved that story, and this system I describe here is dedicated to that rabbit; I’m calling it hare. (Or is it because I saw something hairy? :-))

So whenever somebody uses the SSH subsystem (ssh, scp, sftp) our hare program will talk to its daemon, hared, on the specified server. This hared is so small I have it right here:

Update: meanwhile it’s turned into a Pip-installable package, and Juzam’s made a fully-compatible Go version of the daemon :-)

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import paho.mqtt.publish as mqtt
import socket
import json

__author__    = 'Jan-Piet Mens <jp()mens.de>'

server_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server_socket.bind(('', 8035))

while True:
    message, address = server_socket.recvfrom(1024)
    host, port = address

    ''' ensure we have JSON '''
    js = json.dumps(json.loads(message))

    mqtt.single("logging/hare", js, hostname='127.0.0.1')

All it does is wait for UDP datagrams, and it then packages each up as a JSON string which it publishes to MQTT. Update: due to lots of reasons, I’ve made the code configurable etc. and it is now here and installable via Pypi.

$ mosquitto_sub -v -t 'logging/hare'
logging/hare {"tst": 1522152142, "hostname": "zabb01", "user": "jjolie", "service": "sshd", "rhost": "192.168.1.130"}
logging/hare {"tty": "ttyv0", "service": "login", "hostname": "fbsd103", "user": "root", "tst": 1522161689, "rhost": "<unknown>"}
$ cat /var/log/ssh-logins.log
login via sshd by jjolie on zabb01 from 192.168.1.130 at 2018-03-27 14:02:22
login via login by root on zabb01 from <unknown> at 2018-03-27 14:07:45

In order to actually process those messages we have mqttwarn which is configured to produce a file with logins (as shown above; note how the login service shows up as well as I’ve also configured that to use hare in PAM), can notify via Pushover (as shown at top), and can easily be used to use any number of the meanwhile 65 services to process these messages. It’s also mqttwarn which translates the epoch timestamp from the JSON payload into a human-readable time.

Here’s the configuration I’m using:

[defaults]
launch	 = log, pushover, file, smtp
functions = 'pamfuncs.py'

[config:file]
append_newline = True
overwrite = False
targets = {
   'mylog'     : ['/var/log/ssh-logins.log'],
  }

[config:pushover]
targets = {
    'pam'       : ['xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', 'yyyyyyyyyyyyyyyyyyyyyyyyyyyyyy'],
  }

[config:smtp]
server  =  'localhost:25'
sender  =  "MQTTwarn <jpm@localhost>"
username  =  None
password  =  None
starttls  =  False
targets = {
    'admins'    : [ 'rooters@ww.mens.de' ],
    }

[logging/hare]
targets = pushover:pam, file:mylog, smtp:admins
alldata = moredata()
title: SSH login on {hostname}
format = login via {service} by {user} on {hostname} from {rhost} at {tstamp}

It’s easy for us to add additional targets without touching PAM configuration on any machine, and without changing our hare system: we add a target to mqttwarn, and there you go: you have mail:

Tue, 27 Mar 2018 14:02:22 +0200
From: MQTTwarn <jpm@localhost>
To: rooters@ww.mens.de
Subject: SSH login on zabb01
X-Mailer: mqttwarn

login via sshd by jjolie on zabb01 from 192.168.1.130 at 2018-03-27 14:02:22

So what are the benefits or pitfalls of doing it the way I did?

  • Credentials for the Pushover service are securely stashed on a single machine, the machine which runs mqttwarn. If we want to or have to change them, e.g. because they got compromised, we do so centrally.
  • All nodes to which people login talk to a single machine in my network. It is from that one system, again the one running mqttwarn, that we access the external service.
  • If we want to rip out Pushover and alert/notify with some other service, or if we want to add some additional type of notification we can do this centrally.
  • Above all, we ensure that the SSH login service on the nodes will not hang or somehow be delayed.
  • One (slight) disadvantage is that we have to create architecture-dependent versions of hare. We have to distribute/install those, but that we easily do with our configuration managment system.

I explain how a lot of the mqttwarn stuff works in how do your servers talk to you?. I hope some of these technologies make the lives of your systems safer and your life easier.

SSH, MQTT, daemons, and rabbit :: 25 Mar 2018 :: e-mail

For a long time I’ve been annoyed that VirtualBox issues a different DHCP lease to a virtual machine guest after a while, and there doesn’t seem to be any way to change that within VirtualBox itself. I asked on Twitter, and got a response which trigerred an aha!

virtualbox dhcp disable

I disabled VirtualBox’ DHCP for the interface I’m interested in, and configured a dnsmasq on my notebook.

port=53
interface=vboxnet0
dhcp-range=192.168.56.101,192.168.56.254,255.255.255.0,12h
dhcp-option=option:dns-server,192.168.56.1
dhcp-leasefile=/usr/local/var/lib/misc/dnsmasq/dnsmasq.leases

Minutes later I saw this on the console upon launching dnsmasq and then a VirtualBox guest I already had:

dnsmasq: started, version 2.78 cachesize 150
dnsmasq: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack no-ipset auth no-DNSSEC loop-detect no-inotify
dnsmasq: setting --bind-interfaces option because of OS limitations
dnsmasq-dhcp: DHCP, IP range 192.168.56.101 -- 192.168.56.254, lease time 12h
dnsmasq: reading /etc/resolv.conf
dnsmasq: using nameserver 192.168.1.82#53
dnsmasq: using nameserver 192.168.1.113#53
dnsmasq: read /etc/hosts - 3 addresses


dnsmasq-dhcp: DHCPDISCOVER(vboxnet0) 192.168.56.102 08:00:27:7d:aa:db
dnsmasq-dhcp: DHCPOFFER(vboxnet0) 192.168.56.102 08:00:27:7d:aa:db
dnsmasq-dhcp: DHCPREQUEST(vboxnet0) 192.168.56.102 08:00:27:7d:aa:db
dnsmasq-dhcp: DHCPACK(vboxnet0) 192.168.56.102 08:00:27:7d:aa:db cen7pdns

Lovely. It works.

VirtualBox and DHCP :: 07 Mar 2018 :: e-mail

After three (or was it almost four?) years of using Slack, we took the plunge and set up our own instance of Mattermost. The reasons for doing this include wanting more control over our data and wanting an unlimited history which Slack, as a hosted service, offers only to paying customers. This is more than fair enough – it’s not their fault that we’re too stingy. Apropos stingy: Mattermost exists in three editions – we chose the Team Edition; guess why.

Web UI

If you know how to wield Slack, you know how to use Mattermost. (Oh, please don’t mind the awful colors above – this is my test installation and I need severe optical distinction in order to not mistake the installations.) There are few differences, if any. Mattermost’s use of Markdown appears to be more comprehensive, in particular because its Webhooks support Markdown.

Installing Mattermost is easy thanks to the good documentation they provide which explains, step by step, what I have to do to install Mattermost on a machine. I chose to use PostgreSQL because I recall having read that’s Mattermost’s primary database candidate, and because it allows me to use PostgreSQL – a reason sufficient on its own. I selected to loosely follow advice given regarding location of config.json as that seemed a sensible thing to do. (config.json is Mattermost’s central configuration file which is reloaded on change.) If Ansible’s your drug, Pieter Lexis created an Ansible role for installing Mattermost on Debian/Unbuntu, and there’s playbook which does that and more also.

I can create as many teams as I want on a server, and each team can have as many channels as I want. In the Team Edition users authenticate with password and I enforced e-mail verification. (Other editions offer 2FA and LDAP.)

Mattermost users can upload files (images, code snippets, etc.) which have to be stored somewhere. By default a configurable directory on the local file system is used, but Mattermost’s system console allows me to configure Amazon S3 storage such as Minio.

Webhooks, API, Websocket, CLI, etc.

One of the things I like most about Slack are its integrations, and low and behold, Mattermost has these as well. Incoming and outgoing Webhooks as well as slash commands. Lovely.

Also very powerful is the Websocket API; there’s a Python3 driver which works very well, and next to that is Mattermost’s API with which I can create users, get their details, enumerate posts, create posts, etc. The following example using curl and jo shows how I can add a post from the command line:

json=$(jo channel=cartoons1 \
          username="my-script" \
          icon_url="http://192.168.1.130/~jpm/monster.png" \
          text='Ha, this is _just_  an example using `curl`, :tada:')

curl -H 'Content-Type: application/json' \
     --data "$json" \
     http://192.168.1.189:8065/hooks/s63wiwe4wjgwfxmuh7jqiujcyo

posted with curl

(For a much more flexible solution see 42wim’s mattertee.)

Programs which use Mattermost’s API must authenticate to the service and they can do so with either session tokens that expire and Personal Access Tokens which I create in my account preferences which don’t expire until I revoke them. Additionally Mattermost can act as an OAuth 2.0 provider.

Masses of messages

One of the channels we have is reserved for Nagios/Icinga-type notifications. One thing I wanted to be able to do is to delete and purge those messages; I don’t see why I need to know weeks later that something was offline for a moment. However, if I delete a message, either interactively or via the API, Mattermost soft deletes it; the message is marked as deleted with a time stamp, but it remains in the database.

So I went in search of an API to physically remove these messages, but it doesn’t exist. The solution? Use, say, the API to find the posts I want to remove, “delete” them using said API, and then use an SQL DELETE to purge:

DELETE FROM posts WHERE deleteat <> 0;

Mattermost at a console

Mattermost has a Web UI and some mobile and fat clients, but what does a person do with just a terminal at his/her disposal? Use either matterhorn or Irssi or your favorite IRC client with matterircd.

seed in Matterhorn

What you see in the screenshot above is matterhorn showing what the first Web UI screenshot shows. The program has some really cool features including scripts – just shell scripts which are given the text I enter on stdin and the stdout they produce is posted. Matterircd on the other hand is an IRC to Mattermost gateway written also by 42wim: you connect it to your remote Mattermost installation and talk to it via your IRC client.

Regrets?

Do I regret leaving Slack? Not really, even though their mobile apps are quite a bit more polished than Mattermost’s are – a result of development effort obviously. I now get a warm and quite fuzzy feeling knowing that we have control over our data, how we back it up, and what we do with it. And I’m confident that (other than the NSA) no third party has it.

Apropos 3rd party: while it’s possible to access dozens (or hundreds?) of integrations using an external service called Zapier we will not as that defeats the purpose of wanting to be the sole owners of said data. Similarly we’ve been discussing mobile notifications for which we could either set up mobile push or do it ourselves; we haven’t finally decided yet.

Do it yourself push

Do it ourselves, you ask? Yes, that’s possible by creating a notification endpoint which Mattermost uses whenever it’s about to notify a mobile device. The post I created with a shell script earlier, pushes to this example MQTT:

$ mosquitto_sub -v -t 'mm/#'
mm/_notif my-script in cartoons: Ha, this is _just_  an example using `curl`, :tada:

The way this happens is that Mattermost notifies a Web service I create which obtains the message and disposes of it in any way I want:

#!/usr/bin/env python

from bottle import run, request, post
import json
import paho.mqtt.publish as paho

__author__    = 'Jan-Piet Mens <jp()mens.de>'

@post('/test/api/v1/send_push')
def post1():

    data = json.loads(request.body.read())

    paho.single("mm/_notif", data['message'],
        client_id="jp-mm-notif")

run(host='192.168.1.130', port=8864)

If you’re not interested in mobile push, there’s always e-mail: when users are away or offline they can choose to be notified of new content by e-mail.

If I’d known about Mattermost before, I’d have migrated earlier.

Social :: 30 Jan 2018 :: e-mail

Gogs and Gitea are Open Source, single-binary Go implementations of a Github-like git repository hosting system which I can use privately or in teams. Both are light on system requirements, contrary to a pretty large and quite popular Open Source Java implementation of a similar utility.

Gitea is a fork of Gogs, and it’s very difficult to say which is better: Gogs is, at times, a bit more Github-lookalike I’d say, whereas Gitea’s look is a bit fresher and feels “younger”. Gitea brings everything in a single binary whereas Gogs has a number of support files (CSS, JavaScript, templates) it requires. (This is advantageous because I can change or replace templates if desired. Fabian reminds me that it’s possible to customize Gitea as well.) Both projects appear to be alive and kicking in spite of rumors that Gogs had been abandoned.

gogs installer

Installation basically consists of ensuring the appropriate binary is launched. I then launch a Web browser and point it at the port number indicated on the console (http://0.0.0.0:3000 by default) and answer a few questions. The responses are written into an INI-type configuration file (app.ini) which can be pre-populated of course and looks like this:

[database]
DB_TYPE  = sqlite3
HOST     = 127.0.0.1:3306
NAME     = gogs
USER     = root
PASSWD   =
SSL_MODE = disable
PATH     = data/gogs.db

[repository]
ROOT = data/repos

[server]
DOMAIN           = localhost
HTTP_PORT        = 4000
ROOT_URL         = http://localhost:4000/
DISABLE_SSH      = false
SSH_PORT         = 4022
START_SSH_SERVER = false
OFFLINE_MODE     = true
...

A cursory search will also provide plenty of resources for installing either with, say, Ansible, if that’s your preferred method. Alternatively, both programs support installation with docker.

Both programs support different backend databases (SQLite3, PostgreSQL, MySQL) and SMTP, PAM, or LDAP authentication. They offer git over HTTP and have an optional embedded SSH server for git over SSH. (Just like in Github or Gitlab, I upload one or more SSH keys which Gogs/Gitea use for authentication.) While there exists a list of Gitea features and Gogs features these lists are difficult to compare.

Both programs have a CLI albeit with slightly differing commands. The CLI is used for backups, user creation, and other administrative commands.

$ gogs admin create-user --name jane --password sekrit --email jane@example.com
New user 'jane' has been successfully created!

I initially chose to use Gitea but thought I’d ask yesterday in a poll: 50/50 is about the reponse I get. A friendly user wrote privately and said:

I’ve been running Gogs for ~20 active users on a low-end VPS alongside other services for a year now. The cli is quite simple and works well, especially backups. Manual upgrades via git went well too 6 month ago.

Both utilities have a dump respectively backup CLI command to create a backup of their data in a ZIP file.

I find Gogs documentation more comprehensive (and Gitea’s sometimes links to it). Featurewise, both are more or less on par, at least in terms of visible features. Both are Open Source, and both projects have over 500 open issues in their trackers and several dozen open pull requests.

Gogs and Gitea can import projects, so I used one of my Github-hosted repositories as a source to produce the following screen shots.

gogs

gogs

gitea

gitea

A bit of both

Both programs display commits like Github does and have a unified diff and a split (side-by-side) diff view. In Gitea the knobs are located as I know them from Github, but that doesn’t mean Gogs’ knob placement isn’t better. Interestingly, repository settings and other pages in Gogs are styled more similarly to Github than in Gitea. So again, six of one and half a dozen of the other. Both allow import of existing repositories (as I did above), though just the repository is imported: neither the issues nor the pull requests, at least not from Github.

issue tracker gitea

Both have an issue tracker with github-flavored Markdown support, file attachments, etc. There are slight cosmetic differences but nothing drastic that I can see. Both support Git hooks, Webhooks and deployment keys (and I do prefer the page layout that Gogs offers in the “sub pages” such as settings).

I’ve chosen to use Gitea, but as I’ve said: it’s a hard toss.

git :: 25 Oct 2017 :: e-mail

Can I have an SCM (Source Code Management) update trigger the launch of an AWX job? The answer is yes, and it’s one of the interesting things I can do to remote-control AWX.

hooks

What I need is some way to invoke a program as soon as I commit something. Subversion, git, etc. support hooks, but cloud-based repository systems (Github, Bitbucket) also support Webhooks which I’ll use here.

In this example I’m using gitea which calls itself a painless self-hosted Git service; it’s all of that and more – it’s one of those gorgeous single-binary Go programs which I’m liking more and more. (I showed you another recently – CoreDNS.) Gitea is very much like using Github, but you host it yourself, and it’s trivial to get started with it.

Within gitea, I configure an outgoing Webhook:

Gitea with a Webhook

From now on, as soon as this repository gets a commit pushed to it, the specified URL will be invoked

On the other side, I run a lightweight, configurable utility (again in Go), called adnanh/webhook. This listens for HTTP payloads, extracts JSON from them, and it can invoke a program of my choice to react to the hook. This could be any kind of HTTP endpoint which reacts to a Webhook POST, but I chose this for simplicity.

I configure webhook to run with the following configuration, which will extract the repository’s name and the secret specified in the hook invocation from the incoming payload (here is the full payload sent by gitea).

[
  {
    "id": "awx-atest",
    "execute-command": "/Users/jpm/bin/awx-hook.sh",
    "command-working-directory": "/tmp/",
    "pass-arguments-to-command": [
      {
        "source": "payload",
        "name": "repository.full_name"
      },
      {
        "source": "payload",
        "name": "secret"
      }
    ]
  }
]

I launch webhook and watch what happens when I commit and push to the repository:

./webhook -hooks hooks.json -verbose
[webhook] 2017/10/23 18:17:07 version 2.6.5 starting
[webhook] 2017/10/23 18:17:07 setting up os signal watcher
[webhook] 2017/10/23 18:17:07 attempting to load hooks from hooks.json
[webhook] 2017/10/23 18:17:07 os signal watcher ready
[webhook] 2017/10/23 18:17:07 found 1 hook(s) in file
[webhook] 2017/10/23 18:17:07 	loaded: awx-atest
[webhook] 2017/10/23 18:17:07 serving hooks on http://0.0.0.0:9000/hooks/{id}
[webhook] 2017/10/23 18:17:09 incoming HTTP request from [::1]:54005
[webhook] 2017/10/23 18:17:09 awx-atest got matched
[webhook] 2017/10/23 18:17:09 awx-atest hook triggered successfully
[webhook] 2017/10/23 18:17:09 200 | 388.746µs | localhost:9000 | POST /hooks/awx-atest
[webhook] 2017/10/23 18:17:09 executing /Users/jpm/bin/awx-hook.sh with arguments ["/Users/jpm/bin/awx-hook.sh" "jpm/atest" "none-of-your-business"] and environment [] using /tmp/ as cwd
[webhook] 2017/10/23 18:17:10 command output: {"job":331,"ignored_fields":{},...
[webhook] 2017/10/23 18:17:10 finished handling awx-atest

The truncated output in the second to last line is the JSON returned from the AWX job launch which happens in the awx-hook.sh script:

#!/bin/sh

mysecret="none-of-your-business"

repo="$1"
secret="$2"

if [ "$secret" == "none-of-your-business" ]; then
    curl -qs \
        -d '{"extra_vars":{"newpoem":"hello good world"}}' \
        -H "Content-type: application/json" \
        -u admin:password  \
        "http://192.168.1.210/api/v2/job_templates/${repo}/launch/"
fi

All this is obviously just an example. Refine to your taste (and add lots of error-handling!)

From AWX

Whilst on the topic of Webhooks: AWX can trigger an arbitrary Webhook as a notification; these are invoked on success or on failure (as desired), and produce a payload which looks like this:

{
  "created_by": "admin",
  "credential": "jp-ed25519",
  "extra_vars": "{}",
  "finished": "2017-10-24T06:05:09.626734+00:00",
  "friendly_name": "Job",
  "hosts": {
    "alice": {
      "changed": 0,
      "dark": 0,
      "failed": false,
      "failures": 0,
      "ok": 2,
      "processed": 1,
      "skipped": 0
    }
  },
  "id": 335,
  "inventory": "j1",
  "limit": "",
  "name": "pi1",
  "playbook": "ping.yml",
  "project": "p1",
  "started": "2017-10-24T06:04:54.127124+00:00",
  "status": "successful",
  "traceback": "",
  "url": "https://towerhost/#/jobs/335"
}

The next step is to take bits of the payload to indicate success or failure on your monitoring blinkenlights.

Ansible and AWX :: 23 Oct 2017 :: e-mail

Other recent entries