DNS zones are typically served by more than one server. One of these is called the master or primary and the ‘copies’ are called slave or secondary servers. The slave servers obtain zone data via a process called zone transfer (AXFR) or incremental zone transfer (IXFR). (And I refuse to abstain from using the terms master/slave.)
When we provision a master server with a new zone we must update all slave servers and inform them of the existence of this new zone and which addresses the master servers for the zone have. Unless you are using PowerDNS with MySQL or PostgreSQL replication (which is off topic for this discussion) this is a procedure that is normally done manually. By manually I mean there is no standard procedure for doing this.
A typical way of provisioning slave servers with the names of zones they should slave is to obtain (out-of-band somehow) a list of zones, add them to the slave’s server configuration and tell the slave to “go get ‘em”. One such proposed method is called dper: the DNS Peering Protocol which has an extra XML configuration that needs to be transported (somehow) to the slaves. If BIND is your master, you could use BIND’s statistics server and, as Tony points out in the comments below, Paul Vixie’s metazones solve the “transport” of a zone list as well. Other typical ways include writing a small program to slurp through the provisioning system, dump a list of zones, etc.
There are some obvious problems with this:
- There typically exists a time lag between creating the master zone and having it replicated to the slaves because the slaves need to know about the new zones.
- The out-of-band method of transporting a list (or a full configuration) for slave servers is non-standard. Some people query, say, the database back-end when using a database as provisioning system, others create a list and pull it from the slaves via HTTP, yet others rsync it over, etc. The possibilities are almost endless.
When operating as a master and slave combination, PowerDNS has a pretty unique feature for its slave servers which is called ‘Superslave’. Quoting myself (again) from Chapter 6 of my book:
A unique feature of PowerDNS is that you can configure it to accept notifications from specified “trustworthy” master servers for which it does not yet carry slave zones, and have it create the slave zone(s), and provision them from the master via incoming zone transfers, all automatically. In this role, the server is said to be a Superslave. (By contrast, a normal slave will accept/action NOTIFYs only if you have manually pre-configured it as a slave for the zone. You have to create a zone on the slave, giving it the name of the zone, and create an NS record in the zone pointing to this slave itself.)
This effectively allows PowerDNS to provision its slave servers automatically.
This isn’t just a theoretical question. I’m currently working with a customer who has a large amount of zones which are pretty volatile (i.e. zones come and go). They have a hidden master server (PowerDNS, but it could easily have been of a different brand) and BIND slave servers (to be augmented by a couple of NSD slave servers shortly). The issues we’re seeing here (irrespective of the brand of the hidden master server) are:
- A new zone is provisioned on the master, whereupon PowerDNS sends out a NOTIFY to its slaves.
- The slaves don’t yet know anything about the existence of this new zone, so they ignore the NOTIFY.
- Meanwhile, the provisioning system creates an include file for BIND’s
named.confand reloads the server, during which the server is “deaf” for queries (also for the next bunch of NOTIFY messages the master may send out due to new zones being provisioned).
- The server is now ready, and named goes and transfers (AXFR) the new zones at which point they are available to be queried.
Even by shortening the frequency of the periodical re-provisioning of named (step 3. above), there is always some time lag (sometimes several minutes) until the new zone(s) is (are) available on the BIND slave servers.
I’ve given this some thought, particularly in view of the fact that BIND has addzone and NSD4 also has support for addzone, allowing both brands of servers to configure new zones on-the-fly. (While there is an initiative to create some form of control protocol (e.g. Requirements for the Nameserver Communication protocol) I’m not aware of something that can be used out of the box.)
The method I propose, and for which I include a prototypical proof of concept (which is working very nicely in my portable data center) can be used to provision BIND slaves, and NSD slaves from any server which is capable of sending a DNS NOTIFY to its slaves.
A small utility called metaslave runs on the slave servers alongside BIND or NSD and, on a different port number, listens for NOTIFY requests. As soon as it receives a NOTIFY, it checks a local database (simplistically implemented as a file on the file system in this PoC) to see whether it knows of the zone. If it doesn’t, metaslave launches an external command to add the zone to the particular brand of name server. (Instead of using a database, I can check existence of the zone file proper, which ought to be safe enough; this does mean ensuring all slave servers have the same “formula” for contructing paths to zone files.)
My proof-of-concept metaslave uses Net::DNS which, if you recall, was augmented with a NotifyHandler for me back when I wrote the book. Note also, that this is currently very simplistic and needs quite a bit of refinement, which I’ll probably do soon-ish.
The (hidden) master servers have to be able to NOTIFY a separate server:
- PowerDNS has
ALSO_NOTIFYin its domainmetadata table with support for specifying an alternative port number (see example below).
- NSD has
notify(with optional TSIG keys), and a port number can be added to the address with
- BIND has
also-notifywith an optional port setting.
As an example, consider the following SQL I inject into my PowerDNS server:
The last two INSERT statements populate the domainmetadata table and set up the additional notification to the metaslave program. After a few seconds, I see the following logged by PowerDNS (I’ve numbered the lines to discuss them.)
NOTIFYs (2. and 3.) are sent by PowerDNS to the NS RRset (the set of NS records),
whereas 4. and 5. (note the :5353) are those from
ALSO-NOTIFY. The former
are negatively acknowledged in lines 6. through 9. because the slave server don’t know
about the b.aa zone (yet). The actual zone transfers (AXFR) from both servers
follow suit and complete within a second.
A great advantage in adding zones dynamically, at least to BIND, is we keep the time frame in which named is “deaf” to queries very short.
While we’re able to provision new zones on slave servers this way, and I’m confident
this is a viable method, we cannot decomission removed zones: the DNS protocol doesn’t
REMOVE-THIS-ZONE querytype; we continue to have to do that “manually”.
Or do we? Read on!
Removing zones from slave servers
Tony Finch mentioned metazones again this morning, and this got me thinking I could use them to automatically remove zones from slaves when they’re deleted on the master server.
Recall that the particular master I’m talking about is an authoritative PowerDNS master with a database back-end. (For a different brand of master, BIND, say, we’d have to modify a number of details: I would probably create a dynamically updatable metazone and script dynamic updates when adding or removing zones from the master.)
I’m creating a master zone called
meta.meta which will store a metazone (of sorts).
This is what it looks like in the records table
When a zone is deleted from the domains database table, a trigger adds a record to the metazone by an INSERT into the records table.
Suppose I delete a zone
example.com, the records database table then looks like
The database trigger added a new TXT record with rdata containing the original
PowerDNS domain_id (for debugging) and a UTC timestamp. Simultaneously, the trigger
also updated the SOA serial number of the
meta.meta zone, which will cause the zone’s
slaves to be notified whereupon they will transfer it. So far, so good.
A program on the slave servers can then periodically (via cron for example) perform
a zone transfer (from 127.0.0.1) to obtain the
meta.meta zone and remove zones
from the slave server configurations as well as moving the slave zone files into a
backup area. Basically the program looks like this:
When I run it on one of my slaves it reports:
Which is, indeed, the zone we deleted a few moments ago on the PowerDNS master.
To cater for the possibility that a zone is removed from the master DNS server and it is added at a later time (yes, that happens), I have a second trigger which cleans out the record of a previously “deleted” domain from the metazone:
I think the only thing missing is a periodic cleanup of the
on the hidden master; a trivial task.