The back-end program we’ve created for the OwnTracks apps is finding a modest amount of appreciation, but in spite of relatively simple building instructions, I’ve been wanting to provide ready-to-run packages for a while now. (Not to say that many people have requested such packages.) I bit the bullet, and I’m happy to say I think I’m doing this like all the cool kids do it. Sort of. I hope.

A while back I created a Homebrew formula for the Recorder. A client suggested a Docker image, so I did that too. Whenever I create a release, I update both and push them into their appropriate locations. About a week ago, I read that Docker hub allows automatic building, so I switched to that and had a builder kicked by a Github notification thing. And now packages. Sigh.

Without Jordan Sissel’s fpm I wouldn’t have done this. My copy of Maximum RPM (I think that was the name) collects dust somewhere in the basement; I read it many many years ago and can’t be bothered to study that again. Furthermore, I knew I’d have to dig into the Debian way of packaging and flatly refuse to learn how to build a .deb the way it probably should be done. fpm to the rescue, wrapped in a little shell script. Life is good.

Then of course I need hosts on which to build said packages. I wanted to offer Debian 8, Raspbian, and Centos 7. A short while later, I could ssh into these systems, launch make and, ta-da!, had three package files. Which need to be signed. Sigh. Another few shell scripts wrapped rpm --sign and reprepro, pushed the files here and there, and an rsync then finally dropped them in our repository. It worked, and I took copious notes on which machine to do what after which other step. You know how it is. (Ansible wouldn’t have helped me much I decided.)

After several years of hearing how people automate things, I thought: no, this can’t be it. Let me try and do this properly; it’ll hurt, but the effort should be worth it. I re-read the notes I took about Jenkins and read most carefully what Mark Maas had to say about Rundeck. In for a penny, in for a pound, why not? Moments later, (java -jar rundeck-launcher.jar) I had Rundeck’s Web interface up and running and looked around a bit. It appeared simple, but how to add hosts (nodes)?

<?xml version="1.0" encoding="UTF-8"?>
<project>
  <node name="ot-make-cen7"
    description="Centos7 builder"
    tags=""
    hostname="ot-make-cen7"
    osArch="x86_64"
    osFamily="unix"
    osName="centos"
    osVersion="7.3"
    username="jpm"/>
</project>

Aha: an editor on the enterprisey-looking XML file resources.xml fixed that, and few minutes after downloading Rundeck I was launching remote tasks. Very nice indeed. The Web interface is good and snappy, and it has an API. Of course it has an API.

So, I created a job to pull our code, run make and fpm, and drop the resulting package file on shared storage. Hey, this is very productive! Rundeck talks to my servers over SSH (and I recommend you allow its variables to pass through. I then quickly write another job to sign the packages, but this job never stops running … What’s the problem? Ah, the Rundeck debug log shows "Enter passphrase:". Signing means GnuPG keys, and these have a passphrase. Duh. OK, I’m quite sure the world has solved this problem before I have to. It seems that lots (most?) people “solve” this by running expect as a wrapper around the signing program to have it “key in” the clear-text passphrase. I’m aghast, and it seems rpm has trouble with gpg-agent… So, if the cool kids do it, I must also:

#!/usr/bin/expect

## Usage: ./rpmsign.exp *.rpm
#  requires: $GPGPASSPHRASE in environment
#  hacked together by JPM from http://aaronhawley.livejournal.com/10615.html and
#  from @roidela_____ ;-)

global env

spawn rpmsign "--addsign" {*}$argv
expect "Enter pass phrase: "
send $env(GPGPASSPHRASE)
send \r
expect eof

Anyway, where to put the passphrase? Rundeck uses something they call “key storage”.

Basically these are files on the Rundeck server’s file system which contain my clear-text secrets which Rundeck can pass into a job via, say, an environment variable. This is not what I’d call “safe storage”, but I’ll let it pass for the moment. (The reason I’ll let is pass is that all this is happening locally on my Mac and not in the cloud, so I feel relatively safe – famous last words…)

I now have two Rundeck jobs: one to get and compile code, and the other to sign packages. (I typically learn things in small steps to get to know how things work.) What I could now do is to merge those two jobs into one, but Rundeck can do that for me with a “workflow” job, in which I tell it to run this job, then that. Perfect. Rundeck also exports and imports job descriptions in XML or YAML:

- description: Pull git repo and make
  executionEnabled: true
  id: d5c0f354-5c85-4181-9791-ab170f1a6612
  loglevel: INFO
  name: make Recorder packages
  nodefilters:
    dispatch:
      excludePrecedence: true
      keepgoing: true
      rankOrder: ascending
      threadcount: 1
    filter: ot-make-cen7 ot-make-deb8 ot-raspi
  nodesSelectedByDefault: true
  scheduleEnabled: true
  sequence:
    commands:
    - description: pullmake
      script: |
        rm -rf recorder.git
        git clone https://github.com/owntracks/recorder.git recorder.git
        cd recorder.git
        cp etc/@node.os-name@/config.mk.in config.mk
        make
        sh etc/@node.os-name@/fpm-make.sh
    keepgoing: false
    strategy: node-first
  uuid: d5c0f354-5c85-4181-9791-ab170f1a6612

This is looking very good, even though I’m still unhappy with the GPG passphrase thing. What I’m probably going to do is to have a passphrase-less key: whether I have that or the passphrase in clear text is probably six of one or half-a-dozen of the other.

Hey, can I have Rundeck tell my colleagues when a new package is ready? Yes, with notifications directly into our Slack channel.

So, my workflow now goes like this: when I want to build packages, I kick my local Rundeck into action via its API:

#!/bin/sh

token=QeAJ32IWc1qYb0uuMzuEdOErCarm7Mqo
jobid=ec5bcb05-fe01-43bd-9bf9-240f15d18875

curl --header "X-Rundeck-Auth-Token:$token" http://localhost:4440/api/1/job/$jobid/run

Rundeck then launches the build on the (currently) three machines, takes the packages, signs them, and pushes the result out to our public repository.

As soon as that’s done, the last Rundeck job is to kick the Docker hub builder into action so that it builds our OwnTracks Recorder Docker container. (I’m aware a git push to Github could trigger Rundeck, but recall my installation is not publically reachable which is why I kick it into life manually.)

I knew it would be a lot of work, and I guessed the investment would amortize itself quickly; I was right. Automation is always a good investment.

May I now, finally, consider myself grown up?