If you've been here previously, you may know about the sudo cake I had baked and the reason I had it baked. Current events make me want to point out again how important it is to validate files you push out to machines with Ansible. Case in point is a shop which brought down a group of DNS servers by pushing out a broken configuration file.

Consider this Ansible playbook, and in particular, the template action:

---
- hosts: bindservers
  user: root
  vars:
  - checkconf: /usr/sbin/named-checkconf
  tasks:
  - name: Install and validate named.conf
    action: template src=named.conf.j2 dest=/etc/named.conf validate='{{ checkconf }} %s'

The validate option (also available for the copy command) specifies a validation command which will be run when Ansible creates (i.e. copies) the template to the target machine, but before that is moved into it's final dest_ination. If this command fails (a single %s is replaced by the filename) the whole task fails, which is good: the dest_ination file is not modified. Here's an example of the task failing:

TASK: [Install and validate named.conf] ***************************************
failed: [ns03] => {"failed": true}
msg: failed to validate: rc:1 error:

FATAL: all hosts have already failed -- aborting

and here is the task passing validation:

TASK: [Install and validate named.conf] ***************************************
changed: [ns03]

In this particular example I'm lucky because the BIND nameserver package actually has a program which can perform a validation check on its own configuration. (Just as sudo can with its visudo program, as I learned the hard way ...) If that weren't possible, I might want to create some script which does it for me and install that as an early Ansible task.

This is just a friendly reminder.

View Comments :: Ansible :: 26 Feb 2015 :: e-mail

Whilst glancing through somebody's slide deck recently, I noticed a reference to the DNSTXT lookup-plugin for Ansible which I wrote in 2012, and I caught myself smiling. Then, just a day ago, I saw a request on the Ansible mailing-list where somebody asked how to do DNS lookups via Ansible. Serge responded along the lines of "use dig(1) with the pipe lookup plugin", which is probably good enough for most cases.

Even so, in a moment of weakness, I thought: na, let's do better. That, however, turned out to be a bit of a long story, because I found an issue with the way lookup plugins return lists to Jinja2 templates. Be that as it may, Brian Coca of Ansible has merged a fix which allows me to present you with the first version of a new Ansible lookup plugin which I'm naming dig. :-)

Assume the following playbook to demonstrate dig in a template:

---
- hosts:
  - localhost
  connection: local
  gather_facts: False
  tasks:
  - local_action: template src=dns.j2 dest=/tmp/dns.out
  - action: debug msg="GOOG == {{ item.1 }}"
    with_indexed_items: "{{ lookup('dig', 'google.com./A', wantlist=True) }}"

and this template:

# 01) -- My external IP address is: {{ lookup('dig', '@ns1.google.com  o-o.myaddr.l.google.com/TXT') }}

# 02) --- iis.se/DNSKEY
{% for dnskey in lookup('dig', 'iis.se./DNSKEY', wantlist=True) -%}
{{ dnskey }}
{% endfor %}

# 03) --- isc.org/TXT
{% for rr in lookup('dig', 'isc.org./TXT', wantlist=True) -%}
{{ rr }}
{% endfor %}

# 04) --- iis.se/DS
{% for rr in lookup('dig', 'iis.se./DS', wantlist=True) -%}
{{ rr }}
{% endfor %}

# 05) --- google.com/A
{% for rr in lookup('dig', 'google.com./A', wantlist=True) -%}
{{ rr }}
{% endfor %}

# 06) --- google.com   --- specials
First  Goog record: {{ lookup('dig', 'google.com', wantlist=True)[0] }}
Random Goog record: {{ lookup('dig', 'google.com', wantlist=True) | random }}

# 07) --- www.kame.net/AAAA
{% for rr in lookup('dig', 'www.kame.net/AAAA', wantlist=True) -%}
{{ rr }}
{% endfor %}

# 08a) --- nl.cc.jpmens.net/TXT
{% for rr in lookup('dig', 'nl.cc.jpmens.net./TXT', wantlist=True) -%}
{{ rr | capitalize }}
{% endfor %}
# 08b) --- nl.cc.jpmens.net qtype=txt
{% for rr in lookup('dig', 'nl.cc.jpmens.net. qtype=txt', wantlist=True) -%}
{{ rr | capitalize }}
{% endfor %}

# 09) --- denic.de/NSEC3PARAM
{% for rr in lookup('dig', 'denic.de./NSEC3PARAM', wantlist=True) -%}
{{ rr | upper}}
{% endfor %}

# 10) --- kdkdkd.iis.se/A   [NXDOMAIN expected]
{% for rr in lookup('dig', 'kdkdkd.iis.se.', wantlist=True) -%}
{{ rr }}
{% endfor %}

# 11) --- mary.jpmens.org/TXT
{% for rr in lookup('dig', 'mary.jpmens.org/TXT', wantlist=True) -%}
[{{ rr | upper }}]
{% endfor %}

# 12) -- 8.8.8.8/PTR
{% for rr in lookup('dig', '8.8.8.8/PTR', wantlist=True) -%}
{{ rr }}
{% endfor %}

# 13) -- 4.4.8.8.in-addr.arpa./PTR
{% for rr in lookup('dig', '4.4.8.8.in-addr.arpa./PTR', wantlist=True) -%}
{{ rr }}
{% endfor %}

# 14) -- @192.168.1.114 p01.aa/TXT    [query specific server]
{% for rr in lookup('dig', '@192.168.1.114 p01.aa/TXT', wantlist=True) %}
{{ rr }}
{% endfor %}
{% for rr in lookup('dig', 'p01.aa/TXT @192.168.1.114 ', wantlist=True) %}
{{ rr }}
{% endfor %}

# 15) -- @192.168.1.114 p01.aa/MX   [NODATA expected]
{% for rr in lookup('dig', '@192.168.1.114 p01.aa/MX', wantlist=True) %}
--->{{ rr }}<---
{% endfor %}


# 16) -- heise.de/MX (just hostname of first returned MX)
{#
    lookup returns list.
    take first element [0]
    split that into list (priority, hostname)
    grab hostname [1]
#}
{{ lookup('dig', 'heise.de/MX', wantlist=True)[0].split()[1] }}


# 17) -- PTR of ansible_default_ipv4
{{ lookup('dig', ansible_default_ipv4.address + '/PTR') }}

# 18) -- ***** dict: iis.se. qtype=A flat=0
{% set data = lookup('dig', 'iis.se. qtype=a flat=0', wantlist=True)[0] -%}
Domain...: {{ data.owner }}
Type.....: {{ data.type }}
Address..: {{ data.address }}
TTL......: {{ data.ttl }}

# 19) -- **** dict: iis.se qtype=MX flat=0
{% set data = lookup('dig', 'iis.se. qtype=mx flat=0', wantlist=True) -%}
{{ data | pprint }}

# 20) -- **** dict: ubu.jpmens.org qtype=SSHFP flat=0
{% set data = lookup('dig', 'ubu.jpmens.org. qtype=sshfp flat=0', wantlist=True) -%}
{{ data | pprint }}

# 21) -- **** dict: 8.8.8.8/PTR flat=0
{{ lookup('dig', '8.8.8.8/ptr flat=0', wantlist=True) | pprint }}

# 22) -- **** dict: statdns.net LOC
{{ lookup('dig', 'statdns.net./LOC flat=1', wantlist=True) | pprint }}
{{ lookup('dig', 'statdns.net/loc flat=0', wantlist=True) | pprint }}

When I run the playbook, the following output is produced:

# 01) -- My external IP address is: 174.0.23.4

# 02) --- iis.se/DNSKEY
257 3 5 AwEAAcq5u+qe5VibnyvSnGU20panweAk 2QxflGVuVQhzQABQV4SIdAQs+LNVHF61 lcxe504jhPmjeQ656X6t+dHpRz1DdPO/ ukcIITjIRoJHqS+XXyL6gUluZoDU+K6v pxkGJx5m5n4boRTKCTUAR/9rw2+IQRRT tb6nBwsC3pmf9IlJQjQMb1cQTb0UO7fY gXDZIYVul2LwGpKRrMJ6Ul1nepkSxTMw Q4H9iKE9FhqPeIpzU9dnXGtJ+ZCx9tWS Z9VsSLWBJtUwoE6ZfIoF1ioqqxfGl9JV 1/6GkDxo3pMN2edhkp8aqoo/R+mrJYi0 vE8jbXvhZ12151DywuSxbGjAlxk=
256 3 5 BQEAAAAB9AdrxXeW/9/peShyQ3c2Pjuh mmzOtVIuBPiNS9hcAFv42yRWU3Fyk04U JrU5XLlNcznV+8nc/3WcgAe2qlRR0Wma C5CyiaLX8zdd19JSmrDXMb33gNUCc3wk vt6YPxf+y+ionxKoGJaPXw/bNmnkIHNa gGEo5jFxrdS8NRIQx6s=

# 03) --- isc.org/TXT
v=spf1 a mx ip4:204.152.184.0/21 ip4:149.20.0.0/16 ip6:2001:04F8::0/32 ip6:2001:500:60::65/128 ~all
$Id: isc.org,v 1.1971 2015-02-25 18:38:13 jtl Exp $

# 04) --- iis.se/DS
18937 5 1 10dd1efdc7841abfdf630c8bb37153724d70830a
18937 5 2 b5c422428dea4137fbf15e1049a48d27fa5eade64d2ec9f3b58a994a6abde543

# 05) --- google.com/A
74.125.136.100
74.125.136.139
74.125.136.102
74.125.136.138
74.125.136.101
74.125.136.113

# 06) --- google.com   --- specials
First  Goog record: 74.125.136.100
Random Goog record: 74.125.136.138

# 07) --- www.kame.net/AAAA
2001:200:dff:fff1:216:3eff:feb1:44d7

# 08a) --- nl.cc.jpmens.net/TXT
Netherlands
# 08b) --- nl.cc.jpmens.net qtype=txt
Netherlands

# 09) --- denic.de/NSEC3PARAM
1 0 15 DE1C

# 10) --- kdkdkd.iis.se/A   [NXDOMAIN expected]
NXDOMAIN

# 11) --- mary.jpmens.org/TXT
[HIS FLEECE WAS WHITE AS SNOW,]
[AND EVERYWHERE THAT MARY WENT,]
[THE LAMB WAS SURE TO GO.]
[HE FOLLOWED HER TO SCHOOL ONE DAY," "WHICH WAS AGAINST THE RULE,]
[MARY HAD A LITTLE LAMB,]
[IT MADE THE CHILDREN LAUGH AND PLAY" "TO SEE A LAMB AT SCHOOL.]

# 12) -- 8.8.8.8/PTR
google-public-dns-a.google.com.

# 13) -- 4.4.8.8.in-addr.arpa./PTR
google-public-dns-b.google.com.

# 14) -- @192.168.1.114 p01.aa/TXT    [query specific server]
Hola patatin
Hola patatin

# 15) -- @192.168.1.114 p01.aa/MX   [NODATA expected]
---><---


# 16) -- heise.de/MX (just hostname of first returned MX)
relay.heise.de.


# 17) -- PTR of ansible_default_ipv4
tiggr.ww.mens.de.

# 18) -- ***** dict: iis.se. qtype=A flat=0
Domain...: iis.se.
Type.....: A
Address..: 91.226.36.46
TTL......: 17

# 19) -- **** dict: iis.se qtype=MX flat=0
[{'exchange': 'mx2.iis.se.',
  'owner': 'iis.se.',
  'preference': 5,
  'ttl': 17,
  'type': 'MX'},
 {'exchange': 'mx1.iis.se.',
  'owner': 'iis.se.',
  'preference': 5,
  'ttl': 17,
  'type': 'MX'}]

# 20) -- **** dict: ubu.jpmens.org qtype=SSHFP flat=0
[{'algorithm': 1,
  'fingerprint': '7e7a55cea3b8e15528665a6781ca7c35190cf0eb',
  'fp_type': 1,
  'owner': 'ubu.jpmens.org.',
  'ttl': 77,
  'type': 'SSHFP'},
 {'algorithm': 2,
  'fingerprint': 'cc17f14da60cf38e809fe58b10d0f22680d59d08',
  'fp_type': 1,
  'owner': 'ubu.jpmens.org.',
  'ttl': 77,
  'type': 'SSHFP'}]

# 21) -- **** dict: 8.8.8.8/PTR flat=0
[{'owner': '8.8.8.8.in-addr.arpa.',
  'target': 'google-public-dns-a.google.com.',
  'ttl': 23925,
  'type': 'PTR'}]

# 22) -- **** dict: statdns.net LOC
['52 22 23.000 N 4 53 32.000 E -2.00m 0.00m 10000.00m 10.00m']
[{'altitude': -200.0,
  'horizontal_precision': 1000000.0,
  'latitude': (52, 22, 23, 0),
  'longitude': (4, 53, 32, 0),
  'owner': 'statdns.net.',
  'size': 0.0,
  'ttl': 227,
  'type': 'LOC',
  'vertical_precision': 1000.0}]

Some highlights:

  • Specify a name to query for an A record by default. (I'll write in my will that one of my grandchildren should submit a pull-request to change that to AAAA as soon as it's mainstream. :-)
  • Specify domain.name/QTYPE to query for that type.
  • Alternatively, specify domain.name qtype=QTYPE to query for that type.
  • Use 192.168.1.2/PTR (or 192.168.1.2 qtype=PTR) as a shorthand to 2.1.168.192.in-addr.arpa/PTR. (Think dig -x.)
  • Add @nameserver to use that particular resolver. nameserver may also be a comma-separated list of addresses or hostnames, whereby hostnames are resolved by the systems' default resolver.
  • TXT records are stripped of leading and trailing quotes. This produces strange results at times.
  • Specify flat=0 to obtain results as a list of dicts (currently A, AAAA, DS, SOA, DNSKEY, LOC, MX and a few others.)

I'm not sure whether I should submit this as a PR to the Ansible project. I have submitted this as a pull-request to Ansible core, and it was merged on 2015-02-27.

View Comments :: Ansible and DNS :: 20 Feb 2015 :: e-mail

The good people of HiveMQ are running a series of articles about MQTT on their blog.

Good reading.

The Knot DNS server is an authoritative DNS server. We've spoken about it before when I introduced it to you it almost 3 years ago and again when I discussed how Knot does dynamic DNS updates and RRL. A lot of Internet time has elapsed since then, and a lot of code has been added to Knot, so it's high time for me to revisit it.

Knot now supports DNSSEC signing of authoritative zones. However, it doesn't provide the utilities with which we create keys, so we use dnssec-keygen (from BIND) or ldns-keygen (from LDNS) to do so. All keys for a zone must be kept in their individual directory which is configured with dnssec-keydir in Knot. In the following zone stanza of my knot.conf I chose to keep the zone and its keys in their own directory (storage).

k03.aa {
  storage "/usr/local/etc/knot/k03";
  dnssec-keydir ".";
  file "k03.aa";
  dnssec-enable on;
  signature-lifetime 1d;  # default: 30d (1d for experiments!)
  serial-policy increment;  # or unixtime
  update-in local;
}

The serial-policy setting specifies how the SOA serial number will be changed after a dynamic DNS update; I prefer a simple + 1 so I use increment. signature-lifetime specifies how long the DNSSEC signatures (RRSIG) should be valid for, and dnssec-enable obviously enables signing.

After creating keys in my storage directory, I can use knotc to reconfigure Knot, and this is what I see in the logs:

info: [k03.aa] DNSSEC, signing started
info: [k03.aa] DNSSEC, loaded key 32514, file 'Kk03.aa.+012+32514.private', KSK, active, public
info: [k03.aa] DNSSEC, loaded key 16266, file 'Kk03.aa.+012+16266.private', ZSK, active, public
info: [k03.aa] DNSSEC, successfully signed
info: [k03.aa] DNSSEC, next signing on 2015-02-07T15:16:01
info: [k03.aa] loaded, serial 0 -> 12
info: [k03.aa] zone file updated, serial 11 -> 12

The serial number transitions look a bit strange, but I interpret them as zone loaded (0 -> 12, which should be 11 actually) and then signatures and DNSKEY as well as NSEC data added (11 -> 12).

Will it blend?

;; ANSWER SECTION:
k03.aa.                 3600 IN SOA k03.aa. jpmens.gmail.com. (
                                12         ; serial
                                21600      ; refresh (6 hours)
                                3600       ; retry (1 hour)
                                604800     ; expire (1 week)
                                600        ; minimum (10 minutes)
                                )
k03.aa.                 3600 IN RRSIG SOA 12 2 3600 20150207174001 (
                                20150206174001 16266 k03.aa.
                                gVTOAXQRDrIL+yeJrK5+T7xemET7A3B0BmBxA/CwsD/x
                                jgfTPZKbV+f98yp98UqwGn5gHOXtbeduL88lWvGPXQ== )

The signature was produced by the key with keytab 16266 (the ZSK). so that looks ok. Actually, I then copied the trust anchor (KSK) to an Unbound server, reloaded that, and it validated the data produced by my Knot server. Also, as Knot will thankfully sign dynamic updates on the fly, any changes I make to the zone via dynamic DNS updates are validatable as soon as the TTL expires.

If the keys I give to Knot are NSEC3-capable it will add NSEC3 denial records to the zone; this is also controlled by the NSEC3PARAM record if present in the original unsigned zone file.

It's a pity Knot overwrites the original unsigned zone file with the signed version. Also, it's not possible to sign zones which are slaved in, making a bump-in-the-wire signer impossible.

Knot's "DNS control utility", knotc shows me the signing status of the zones (the zone name is separated from the rest of the line with a TAB and other wite space are spaces)

$ knotc zonestatus
aa.     type=slave | serial=2015020603 | refresh in 0h29m1s | DNSSEC signing disabled
k03.aa. type=master | serial=14 | DNSSEC resign in 21h32m59s | automatic DNSSEC, resigning at: 2015-02-08T04:38:31
p01.aa. type=slave | serial=2015020514 | refresh in 0h29m1s | DNSSEC signing disabled
example.com.    type=master | serial=2010111213 |  idle | DNSSEC signing disabled

Synthetic records

A feature some people want is the ability for an authoritative server to synthesize answers to queries, a bit like BIND's $GENERATE does. Knot has support for this in form of a query module which you enable on a per/zone basis.

Let's assume we wish to provide responses for an address range 10.1.2/24, I can configure the following in a zone stanza:

example.com {
    file "/usr/local/etc/knot/example.com.zone";
    query_module {
        #                     prefix  TTL     CIDR
        synth_record "forward dyn-    180     10.1.2.0/24";
    }
}

Clients which query names such as dyn-10-1-2-89.example.com (address portions separated by dashes) get the following authoritative response:

$ dig dyn-10-1-2-89.example.com

;; ANSWER SECTION:
dyn-10-1-2-89.example.com. 180  IN      A       10.1.2.89

Synthesized responses are possible for reverse zones as well. Note, however, that currently these records are not signed in DNSSEC-enabled zones.

The documentation provided by the Knot DNS project is adequate, and will help you get started with Knot.

View Comments :: DNSSEC and DNS :: 06 Feb 2015 :: e-mail

One of the first steps in an Ansible playbook run (unless you explicitly disable it) is the gathering of facts via the setup module. These facts are collected on each machine and were kept in memory for the duration of the playbook run before being destroyed. This meant, that a task wanting to reference a host variable from a different machine would have to talk to that machine at least once in the playbook in order for Ansible to have access to its facts, which in turn sometimes means talking to hosts although we just need a teeny weeny bit of information from that host.

One interesting feature of Ansible version 1.8 is called "fact caching". It allows us to build a cache of all facts for all hosts Ansible talks to. This cache will be populated with all facts for hosts for which the setup module (i.e. gather_facts) runs. Optional expiry of cached entries as well as enabling the cache itself is controlled by settings in ansible.cfg:

fact_caching = redis
fact_caching_timeout = 3600
fact_caching_connection = localhost:6379:0

By default, fact_caching is set to memory. Configuring it as above, makes Ansible use a Redis instance (on the local machine) as its cache. The timeout specifies when individual Redis keys (i.e. facts on a per/machine basis) will expire. Setting this value to 0 effectively disables expiry, and a positive value is a TTL in seconds.

The following small experiment will run over 246 machines.

---
- hosts:
  - mygroup
  gather_facts: True
  tasks:
  - action: debug msg="memfree = {{ ansible_memfree_mb }}"
PLAY [mygroup] *****************************************************************

GATHERING FACTS ***************************************************************
ok: [www01]
...
TASK: [debug msg="memfree = {{ ansible_memfree_mb }}"] ************************
ok: [www01] => {
    "msg": "memfree = 7811"
}
...

Running my sample playbook gathers all facts on each run. This playbook took just over a minute to run (1m11). So, after the run, what's in Redis?

127.0.0.1:6379> keys *
1) "ansible_cache_keys"
2) "JPM"
3) "ansible_factswww01"
...

Each of the keys in Redis contains a JSON string value -- the list of all facts collected by Ansible. Let's have a look:

#!/usr/bin/env python

import redis
import json

r = redis.StrictRedis(host='localhost', port=6379, db=0)
key = "ansible_facts" + "www01"
val = r.get(key)

data = json.loads(val)
print data['ansible_memfree_mb']  # => 7811

If I configure gather_facts = False, the setup module is not invoked in the playbook, and Ansible accesses the cache to obtain facts. Note, of course, that the value of each fact variable will be that which was previously cached. Also, because the fact gathering doesn't take place, the playbook runs a bit faster (which may be negligible depending on what tasks it's set to accomplish). In this particular case, the play ran in just under a minute (0m50) -- a slight speedup.

A second caching mechanism exists at the time of this writing: it's called jsonfile, and it allows me to use a directory of JSON files as the cache; expiry is supported as for Redis even though the JSON file remains on disk after it's expired (the file's mtime is used to calculate expiry). If I alter the caching configuration in ansible.cfg, I can activate it:

fact_caching = jsonfile
fact_caching_connection = /tmp/mycachedir

The "connection" setting must point to a writeable directory in which a file containing facts in JSON format for each host are stored. A memcached plugin for the cache also exists. Any playbook which gathers facts effectively populates the cache for the machines it speaks to.

The following playbook doesn't talk to the www01 machine, but it can access that machine's facts from the cache. (The city fact isn't default in Ansible: I set this up using facts.d.)

---
- hosts:
  - ldap21
  gather_facts: False
  tasks:
  - action: debug msg="City of www01 = {{ hostvars['www01'].ansible_local.system.location.city }}"
PLAY [ldap21] **************************************************************

TASK: [debug msg="City of www01 = {{ hostvars['www01'].ansible_local.system.location.city }}"] ***
ok: [ldap21] => {
    "msg": "City of www01 = Boston"
}

As soon as a cache entry expires these fact variables will be undefined, and the play will fail.

Populating or rejuvenating the facts cache is trivial: I'll be running the following playbook periodically in accordance with the cache timeout I've configured:

---
- hosts:
  - all
  gather_facts: True

In case of doubt, clear the cache by invoking ansible-playbook with the --flush-cache option.

View Comments :: Ansible, Redis, and JSON :: 29 Jan 2015 :: e-mail

Other recent entries