It’s been six years since I first looked at Gitea, and I have only now begun looking at implementing continuous integration (CI) with it.
Gitea’s Actions didn’t convince me when I toyed with them, so I looked further and found Woodpecker-CI which I set up using this guide after removing the actual Gitea container because I already have that running as a service. Woodpecker-CI’s documentation is plentiful and quite understandable, but I missed an introduction on why to use docker-compose and not the binaries. After toying around I decided to actually do the Docker dance as I’d be needing that for the Woodpecker agents anyway.
Setting up a pipeline was easy enough, and my current playground looks a bit like this:
$cat .woodpecker.yml
steps:
build something:
image: alpine
commands:
- echo "Here we go!"
- apk add --no-cache build-base gcc
#...
notify via MQTT:
image: example.org/jpm/wpmqtt
settings:
payload: "${CI_COMMIT_AUTHOR} on ${CI_REPO_NAME} is done with ${CI_COMMIT_SHA}"
secrets: [ mqtt_host, mqtt_topic ]
$woodpecker-cli lint
✅ Config is valid
I can create secrets (e.g. mqtt_host above) which are associated with my repository and are passed into pipeline steps and plugins. Very useful for hiding passwords and other, well, secrets, also multi-line secrets such as SSH keys. I create secrets via the UI or via woodpecker-cli. (I had the devil of a time finding the WOODPECKER_TOKEN required for this program and couldn’t for the life of me find it in the docs, but it turns out it’s trivial: in the Woodpecker UI I click on my user avatar and find the two shell export statements required.)
I discovered Woodpecker-CI is extensible via plugins and couldn’t resist. For the plugin proper, I create the program (here a simple shell script). The variables will be populated from secrets and from command parameters (see .woodpecker.yml above):
I recently spent an inordinate amount of time trying to debug why a curl-initiated Webhook POST to AWX was being rejected with the lame message
{"detail":"A server error has occurred."}
In spite of configuring debug logging and log forwarding from AWX, I couldn’t figure out what was wrong. My assumption was the body of the post was missing something. I looked at the source code of the api view controller and still didn’t figure it out and basically gave up after an hour. Actual webhooks posted from Gitea worked (when configured in AWX as Github), but my simple curl invocation wouldn’t. (Remind me to rave about how I like Gitea and Forgejo.)
Ton then showed me how to view what’s going on in the AWX web task:
$kubectl get pods -n awx | grep awx-web
awx-web-6b6bddcf69-75jdp 3/3 Running
$kubectl logs -n awx pod/awx-web-6b6bddcf69-75jdp -c awx-web -f...
2023-09-13 10:58:59,846 ERROR [6e4b76bd6879443f9fbf29a65ccda3a7] django.request Internal Server Error: /api/v2/job_templates/9/github/
Traceback (most recent call last):
...
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/api/views/webhooks.py", line 185, in get_signature
hash_alg, signature = header_sig.split('=')
ValueError: not enough values to unpack (expected 2, got 1)
...
I quickly recognized where I’d gone wrong: I omitted the = sign between the string sha1 and the actual digest. Github now prefers SHA256 (which AWX doesn’t support here) and its header is different.
During a lunch break in Munich last week, Michael mentioned Ansible rulebooks, and I realized I had not taken the time to look into them.
Rulebooks are the system by which Ansible is told which events to use in Event-Driven Ansible. They are written in YAML and contain three main components: sources which define the event sources to be used, rules which define conditionals matched from sources, and actions which trigger what should occur when a condition is met.
Here’s a small example I’ve cobbled together to test Event-Driven Ansible (EDA).
-name:Rulebook to do somethinghosts:localhostgather_facts:falsesources:-ansible.eda.webhook:host:127.0.0.1port:6000-ansible.eda.file_watch:path:"files/"recursive:falseignore_regexes:['.*\.o']rules:-name:Launch playbook on start cmdcondition:event.payload.cmd == 'start'action:run_playbook:name:jp01.ymlextra_vars:dessert:"{{event.payload.data}}"home:"{{HOME}}"# from environmentperson:"{{person}}"# from vars-name:trigger on rangecondition:event.change == 'created'# 'modified'action:run_module:name:copymodule_args:src:"{{event.src_path}}"dest:"/tmp/files"
My rulebook defines two sources: the first listens to HTTP webhooks on port 6000, and the second watches (requires pip install watchdog) a directory for new files.
Then I define two rules:
the first matches the cmd element in the HTTP payload to the word start and performs an action on match. The action run_playbook launches the specified playbook using the inventory we give ansible-rulebook.
the second matches when the event indicates a file has changed in the directory and invokes an Ansible module (copy) to copy new discovered file to a particular destination.
I launch the rulebook:
$ansible-rulebook -i inventory \-r rulebook.yml \--vars v.yml \-E HOME \--print-events
I then POST a webhook and create a new file in the watched directory:
Internally ansible-rulebook uses the Java Drools rule engine and ansible-runner to actually invoke Ansible content. Rulebooks can gather facts which can be used in conditions. ansible-rulebook appears to ignore ansible.cfg as far as I can tell, and it requires me passing it an inventory explicitly. As shown in my example above, I can pass variables to it (from JSON or YAML files), and have it import environment variables – both useful for introducing API keys and whatnot.
I’ve only just begun toying with Event-Driven Ansible (and have reported an issue with watch_file which may not even be one), but it appears to work quite well so far.
I think this will be most interesting when interfacing with, say, repository pushes etc. which can then trigger Ansible playbook runs.
I mentioned back in 2010 that my first impression of OpenDNSSEC wasn’t bad, but I changed my mind when I had to maintain three large installations a few years later and pledged I wouldn’t do so again.
It’s seldom I speak negatively about a particular piece of software; if a program is good I will say so, but otherwise I’ll simply not mention it. I’m making an exception, in spite of OpenDNSSEC having become an NLnet Labs project – the brilliant people who bring us top-notch DNS servers NSD and Unbound.
Note: I originally forgot to mention my notes pertain to a much older, no-longer maintained version of OpenDNSSEC. The software has had significant changes done to it meanwhile, so my complaints may well no longer be valid. Furthermore, I inaccurately stated SoftHSM is an NLnet Labs project: it is an independent project with its own project owners and developers.
Here were four paragraphs detailing the issues we had with the program, but: water under the bridge. I will dwell a bit on the documentation though. OpenDNSSEC’s documentation has gone from one ill-maintained wiki to the next only to meanwhile have even lost all images; I’ve lost the love to report it. For SoftHSM the situation is even worse – what’s termed ‘documentation’ is pretty useless. Just go ahead and compare NSD’s or Unbound’s documentation to that of OpenDNSSEC or SoftHSM. Apropos SoftHSM: during an unrelated experiment about a year ago, I dropped 1,000 keys into a SoftHSM key store which resulted in 655,138 stat(2) system calls when signing a three-record zone; just look at the increasing time required to generate keys. I couldn’t even be bothered to report this as a bug, nor this. I’m sorry.
My time at the company was over and I stopped consulting for them so I wasn’t directly involved, thankfully, but tangentially got to hear about countless OpenDNSSEC enforcer failures, signers which crash, and even one or two unscheduled rollovers occurring on the spur of the moment.
I know there are some people who use OpenDNSSEC for a zone or two, and I know it’s used by some TLDs (who typically have relatively few but large zones), but my experience with the software over the years has been suboptimal, and I have once or twice reccommended against using it.
Roughly two years ago my customer asked me to return as they were having issues with the signing system, but I refused: I wouldn’t touch the existing installation but would gladly return to implement a new system.
“New” was easier said than done because an eventual migration came with one absolute blocking feature: the keys used for DNSSEC KSKs are stored on a slew of Thales/Entrust HSMs and an absolute prerequisite for migration was that KSKs were to be reused and not roll until ordered to.
After a bit of experimentation and different tests with BIND (for which DNSSEC Policy was on the horizon) and Knot-DNS, it turned out the latter had no apparent issues talking PKCS#11 to the installed HSMs, so the future of the project was clear: Knot-DNS would become the signer.
Knot-DNS, like BIND can be a bump-in-the-wire signer, which is how we use it. Knot implements a Key And Signing Policy (KASP) system such as that which was used in OpenDNSSEC, and we were able to configure the new signer according to our requirements, so we were basically good to go.
It took a while for the project to actually kick off, enterprise being enterprise, but we’ve meanwhile successfully migrated two of three largish OpenDNSSEC installations to Knot-DNS. (Just in time BTW, because the last OpenDNSSEC installation is logging that keys can no longer be created … I’m crossing my fingers it’ll hold out another fortnight.)
Knot’s logs are descriptive and are easy to read (important when I’m trying to find out what the server is doing), and it’s documentation is good. The Knot developers react quickly and helpfully on the mailing list, and are open to new ideas:
code I contributed to have keymgr produce JSON was merged in short order
a request for a CKA_LABEL to be added to keys generated via PKCS#11 was implemented in short time
And now? I’m satisfied. The new Knot signer is performing well, and the lack of surprises to date is refreshing. It is of course early days, but I’m confident for the future.
While explaining Ansible’s local facts to students last week, I was asked whether it’s possible to have encrypted facts on the node which get decrypted on use when read by the Ansible controller. The use-case would be some smallish data which is encrypted at rest, “invisible” to curious people on the node, but usable on the management server.
It might be feasible to to place Vault-encrypted (have you seen nanvault?) data on the node and unvaulting it (that’s verb I invented) on the controller (wasn’t there a filter for doing that..?), but the first thing which came to mind was to use age and to create a small filter for it, particularly as this student stayed on for the advanced Ansible course in which we discuss filter creation amongst other things.
age calls itself a “simple, modern and secure encryption tool (and Go library) with small explicit keys, no config options, and UNIX-style composeability”. I’m quite fond of the concept and the utility and use a pair of Yubikeys on which I keep some of its identities (secret keys).
In order to use age we require a key pair; the public key (recipient in age terminology) is written alongside the secret identity into the key file, and it’s printed to stderr to be copy/pasted directly from the console. As is typical, the secret key must be kept secret. (Here I display one for illustration.)
$age-keygen -o cow.key
Public key: age1mkmc34wqy8tdda58077cm2p0eg3xedg4g8dk8sqwwczxl69gyvnqq84pha
$cat cow.key
#created: 2023-06-01T09:56:06+02:00
#public key: age1mkmc34wqy8tdda58077cm2p0eg3xedg4g8dk8sqwwczxl69gyvnqq84pha
AGE-SECRET-KEY-1TU78S7WPWN78XHFKZD368Q4AGHRF6498E4H8ELWXZ5VURK7GZM9Q2WCPQX
$age-keygen -o ansible.key # create another because it's fun and they're cheapPublic key: age19q8pzgaxq2uynsrp3dluxv5apxmqym2pyldwpkp4s30qf4vfzqrsvvjzjv
I then encrypt the data I wish to protect to one or more public keys, first using ASCII armour, and then by base64-encoding the binary encrypted data, showing the two distinct output encodings which my Ansible filter (below) will support.
$echo"The quick brown fox" |
age -r "age1mkmc34wqy8tdda58077cm2p0eg3xedg4g8dk8sqwwczxl69gyvnqq84pha" -a
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB4Z3VVZDlnQmhHTG8zREJk
VUJ5eHNVLzZ6WFlhVXFkNzEvaHN3QnFkdGlFCm00QnZlcktveXo0bDdzVmZXM0Iz
QjA1SWVhVEY4dThsU2k5SStPS1dJaGcKLS0tIEdyRTduOFAvN0xHZTNMQi9RZnMz
cUNWVHJCV293ajhRSW5NUUptTC9ndmMKI5DvYVXGNT/I+5FBZ1saqSwac9ObmYZd
vK/exrZMqlXVwlsXXavxAPA8R9se6vpMYkkI7w==
-----END AGE ENCRYPTED FILE-----
$echo"Pack my box with five dozen" |
age -r "age19q8pzgaxq2uynsrp3dluxv5apxmqym2pyldwpkp4s30qf4vfzqrsvvjzjv" |
base64
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpQ21sTVN4ZDVkVEkzRWNCSkhYOGJaM3laOXlaRU5nNit3QmtDZkJUem00Cm1CdUFVSFVUNVEyVEY0S011YnJxQWdmYnljU1ZMb3VPTDN3ZzdBOUNRd0UKLS0tIERMeUdENUxFUVIvSUNxeXU5RmFYRDFRaWZpeEhjZXZDOVA2Q1F6K1VrTnMKmTG2+GrnkCezjKkTC3skuQGGEfleHh/PGfYxp0spYgITPNPXZs43cFECnyupz039QQtxljh1HBpggSe+
I ensure the local fact files, which must be named *.fact on Unix/Linux, are placed on the node I wish them to be on. The first in INI format, the second in JSON:
$cat /etc/ansible/facts.d/pangram.fact
[p1]
short=YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpQ21sTVN4ZDVkVEkzRWNCSkhYOGJaM3laOXlaRU5nNit3QmtDZkJUem00Cm1CdUFVSFVUNVEyVEY0S011YnJxQWdmYnljU1ZMb3VPTDN3ZzdBOUNRd0UKLS0tIERMeUdENUxFUVIvSUNxeXU5RmFYRDFRaWZpeEhjZXZDOVA2Q1F6K1VrTnMKmTG2+GrnkCezjKkTC3skuQGGEfleHh/PGfYxp0spYgITPNPXZs43cFECnyupz039QQtxljh1HBpggSe+
$cat /etc/ansible/facts.d/other.fact
{
"armored": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB4Z3VVZDlnQmhHTG8zREJk\nVUJ5eHNVLzZ6WFlhVXFkNzEvaHN3QnFkdGlFCm00QnZlcktveXo0bDdzVmZXM0Iz\nQjA1SWVhVEY4dThsU2k5SStPS1dJaGcKLS0tIEdyRTduOFAvN0xHZTNMQi9RZnMz\ncUNWVHJCV293ajhRSW5NUUptTC9ndmMKI5DvYVXGNT/I+5FBZ1saqSwac9ObmYZd\nvK/exrZMqlXVwlsXXavxAPA8R9se6vpMYkkI7w==\n-----END AGE ENCRYPTED FILE-----\n"
}
This is definitely a case in which I’d like Ansible to have support for YAML fact files (quite easily implemented as a custom executable fact – yq –tojson or Python come to mind), as the following looks much more elegant:
armored:|-----BEGIN AGE ENCRYPTED FILE-----YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB4Z3VVZDlnQmhHTG8zREJkVUJ5eHNVLzZ6WFlhVXFkNzEvaHN3QnFkdGlFCm00QnZlcktveXo0bDdzVmZXM0IzQjA1SWVhVEY4dThsU2k5SStPS1dJaGcKLS0tIEdyRTduOFAvN0xHZTNMQi9RZnMzcUNWVHJCV293ajhRSW5NUUptTC9ndmMKI5DvYVXGNT/I+5FBZ1saqSwac9ObmYZdvK/exrZMqlXVwlsXXavxAPA8R9se6vpMYkkI7w==-----END AGE ENCRYPTED FILE-----
Now we have age-encrypted data on a controlled node. Where the data was encrypted is unimportant; what matters is that it is located on the node and can be read by the controlling node during a fact-gathering dance. When Ansible obtains the local facts from a node, it will see the likes of this:
"ansible_local":{"other":{"armored":"-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB4Z3VVZDlnQmhHTG8zREJk\nVUJ5eHNVLzZ6WFlhVXFkNzEvaHN3QnFkdGlFCm00QnZlcktveXo0bDdzVmZXM0Iz\nQjA1SWVhVEY4dThsU2k5SStPS1dJaGcKLS0tIEdyRTduOFAvN0xHZTNMQi9RZnMz\ncUNWVHJCV293ajhRSW5NUUptTC9ndmMKI5DvYVXGNT/I+5FBZ1saqSwac9ObmYZd\nvK/exrZMqlXVwlsXXavxAPA8R9se6vpMYkkI7w==\n-----END AGE ENCRYPTED FILE-----\n"},"pangram":{"p1":{"short":"YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpQ21sTVN4ZDVkVEkzRWNCSkhYOGJaM3laOXlaRU5nNit3QmtDZkJUem00Cm1CdUFVSFVUNVEyVEY0S011YnJxQWdmYnljU1ZMb3VPTDN3ZzdBOUNRd0UKLS0tIERMeUdENUxFUVIvSUNxeXU5RmFYRDFRaWZpeEhjZXZDOVA2Q1F6K1VrTnMKmTG2+GrnkCezjKkTC3skuQGGEfleHh/PGfYxp0spYgITPNPXZs43cFECnyupz039QQtxljh1HBpggSe+"}}},"ansible_machine":"arm64"
Back on the Ansible controller machine, the playbook will use the content of the two facts which it will decrypt using an age identity; in the first instance using a default key file named ansible.key, and in the second the specified secret key, with both files located on the file system of the controller node (filters run in the templating engine which is invoked on the controller):
-hosts:alicetasks:-name:Decrypt the age-encrypted and base64-encoded pangramdebug:msg:"{{ansible_local.pangram.p1.short|age_d}}liquorjugs"-name:Decrypt the age-encrypted and ASCII-armoured 2nd pangramdebug:msg:"{{ansible_local.other.armored|age_d('cow.key')}}"-name:Use age command to encrypt the current date string ...shell:cmd:"age-e-a-rage19q8pzgaxq2uynsrp3dluxv5apxmqym2pyldwpkp4s30qf4vfzqrsvvjzjv<(date)"executable:/bin/bash# yuckregister:c-name:... and decrypt it using our filterdebug:msg:"{{c.stdout|age_d}}"
The output should be predictable:
PLAY [alice] *******************************************************
TASK [Gathering Facts] *********************************************
ok: [alice]
TASK [Decrypt the age-encrypted and base64-encoded pangram] ********
ok: [alice] => {
"msg": "Pack my box with five dozen liquor jugs"
}
TASK [Decrypt the age-encrypted and ASCII-armoured 2nd pangram] ****
ok: [alice] => {
"msg": "The quick brown fox"
}
TASK [Use age command to encrypt the current date string ...] ******
changed: [alice]
TASK [... and decrypt it using our filter] *************************
ok: [alice] => {
"msg": "Fri Jun 2 12:55:17 UTC 2023"
}
For decryption the filter requires a path to a file containing identities (i.e. one or more secret keys). While this file could, in theory, be Ansible-vaulted, the age CLI sensibly has no provision for passing the identity on the CLI so Ansible would have to unvault the file before giving to age – probably quite unwise to do.
age can encrypt data to a set of recipients, making it possible to decrypt the data with distinct identities keys, say, when more than one Ansible controller access nodes and should use individual identities for decryption. age’s keys can also be password-protected, but I feel that would be overkill for this task.
On the train-ride back from Berlin today, I decided encryption might also be interesting, so the filter now has that too:
vars:recipient:"age19q8pzgaxq2uynsrp3dluxv5apxmqym2pyldwpkp4s30qf4vfzqrsvvjzjv"tasks:-name:Encrypt to age with base64 encoding ..set_fact:secret:"{{'Moo🐄'|age_e(recipient)|b64encode}}"armored:"{{'moremoo🐮'|age_e(recipient,true)}}"-name:.. and decrypt using age_ddebug:msg:"{{secret|age_d}}"-name:.. and decrypt the armored value using age_ddebug:msg:"{{armored|age_d}}"
And that produces this output:
TASK [Encrypt to age with base64 encoding ..] **************
ok: [localhost]
TASK [.. and decrypt using age_d] **************************
ok: [localhost] => {
"msg": "Moo 🐄"
}
TASK [.. and decrypt the armored value using age_d] ********
ok: [localhost] => {
"msg": "more moo 🐮"
}
I’m quite sure my filter’s Python code can be improved upon, so you know what to do: here it is.
