I typically have a number of smallish programs I want launched as daemons. Getting those set up nicely typically requires root privileges, startup-scripts, etc. which I find cumbersome. Instead of /etc/init.d and friends, I use a lovely program called Supervisord. I think of Supervisord as my personal init: it launches programs, monitors them for failure, logs their output, and restarts them. Furthermore, it is trivial to set up new “daemons”, either manually or via, say Ansible (which even has a module specifically for Supervisord).

The Supervisord package has Web and XML-RPC interfaces which can be disabled if you don’t need them.

Supervisord Web interface

After installing Supervisord (as root), I can choose to configure it in such a way as that it can be controlled by a non-privileged user (the configuration below does that: Supervisord will make my user owner of its socket, which allows me to control it without using sudo. (As you know, using sudo can be hazardous to my health … ;-)

file=/tmp/supervisor.sock   ; (the path to the socket file)
chmod=0700                  ; socket file mode (default 0700)
chown=jpm:jpm               ; socket file uid:gid owner


logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB        ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10           ; (num of main logfile rotation backups;default 10)
loglevel=info                ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false               ; (start in foreground if true;default false)
minfds=1024                  ; (min. avail startup file descriptors;default 1024)
minprocs=200                 ; (min. avail process descriptors;default 200)
user=jpm                     ; (default is current user, required if root)
directory=/tmp               ; (default is not to cd during start)

supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket

files = /etc/supervisord.d/*.ini

Once Supervisord is running, I can create a small configuration file (ini -style) for a new program I want to run under its control. Here’s an example for mqttwarn:

directory = /home/jpm/services/mqttwarn
command = /home/jpm/services/mqttwarn/mqttwarn.py
user = jpm
environment= MQTTWARNINI="/home/jpm/Auto/services/mqttwarn/mqttwarn.ini"

If supervisord is running as root (which it typically does, but need not), I can configure the user under which my program is launched, set up specific environment variables, define expected exit status codes, whether or not to restart the program on exit, stdout redirection, logfile rotation, and a slew of other things.

I then use supervisorctl to “talk” to the supervisord process, and tell it to re-read its configuration and launch or stop processes depending on the state of the configuration:

$ cp xx.ini /etc/supervisord.d/mqttwarn.ini
$ supervisorctl reread
mqttwarn: available
$ supervisorctl update
mqttwarn: added process group
$ supervisorctl status
WSS                              RUNNING    pid 7330, uptime 0:08:40
mqttwarn                    RUNNING    pid 8331, uptime 0:00:12
yamlwatch                        RUNNING    pid 7329, uptime 0:08:40

Programs can be stopped or restarted. reread reloads the configuration without stopping or starting processes, and update will relaunch processes with changed configurations. So, if you want to pick up changes to config files with minimal disruptions, use reread followed by update.

If Supervisord’s XML-RPC interface is enabled, I can use the interface to check on Supervisord with, say, Nagios or Icinga. For a very simple example, this Python script connects to Supervisord and obtains information on whether WSS is running or not:

#!/usr/bin/env python

import xmlrpclib
import sys

# Simple Nagios/Icinga check for a process under Supervisord.
# Requires XML-RPC interface (inet) enabled

__author__    = 'Jan-Piet Mens <jpmens()gmail.com>'

supervisorduri = 'http://localhost:9003/RPC2'
procname = 'WSS'

OK = 0
stati = {
    0 : 'OK',
    1 : 'WARNING',
    2 : 'CRITICAL',
    3 : 'UNKNOWN' }
status = OK
statustext = None

server = xmlrpclib.Server(supervisorduri)

    info = server.supervisor.getProcessInfo(procname)
except xmlrpclib.Fault:
    status = CRITICAL
    statustext = "supervisor: process %s not found" % procname
except Exception, e:
    status = CRITICAL
    statustext = "supervisor: process %s: %s" % (procname, str(e))

state = info['statename']

if state != 'RUNNING':
    status = CRITICAL

if statustext is None:
    statustext = '%s supervisor for %s: %s' % (stati[status], procname,  info['description'])

print statustext

The above program produces output like this:

OK supervisor for WSS: pid 10088, uptime 0:07:17

Supervisord is highly recommended; I rely on it on my workstation and on servers.


init and Python :: 13 Feb 2014 :: e-mail