A Kerberos Key Distribution Center (KDC) stores user accounts (principals) and
their keys (derived from a user’s password with the
string2key() function) in its
database. These keys are used as part of the authentication process to
verify whether a user is
authenticated when he or she accesses a server or service. Similarly, hosts and
services in a Kerberized environment have a host or service principal – an
account and a bunch of keys also in the KDC database.
If keys associated with network services had passwords, a service couldn’t start up without administrative intervention by a human to enter the password. This means that service principals typically have randomly generated keys, and the network service has to have a copy of its keys on the system. Such keys are typically stored in files on the file system called keytabs and keytabs can contain more than one key. For example, the Kerberos keytab on my workstation contains the following keys:
A keytab is typically created (or keys added to it) by an administrator logging
in to the system on which the keytab is to be maintained, invoking the Kerberos
administration utility (
kadmin) over the network, authenticating on as an authorized
principal, generating a key and
adding that key to a new or existing keytab. (Alternatively, a keytab can be created
on a foreign machine and transported securely, e.g. via SSH.) That process looks like this:
kadmin interface, I create a new principal
blog/demokey@MENS.DE in the
KDC’s database, and then export that principal’s keys into the specified keytab. Fine and
dandy if I do that once in a while, but I wouldn’t like to have to do that for a few dozen
or hundred machines deployed in a large environment. And how would we automatically
and securely (!) provision keytabs on machines that are automatically deployed?
(I hate to say this, but I’ve seen shell scripts which invoke
kadmin -w with the
hard-coded password of a principal allowed to create keys contained in them … (sigh);
don’t do that!)
I digress a bit but I can “see” the principal created above looking at the LDAP back-end in which my KDC’s database is stored; the LDAP entry looks like this:
So, how do we deploy keytabs automatically onto nodes? With Wallet.
Wallet, created by Russ Allbery, is a system for managing authorization and retrieval of secure data, Secure data can be any file (e.g. configuration files which contain clear-text passwords) and keytabs.
Wallet is a client/server program:
- The wallet client is a small C program which talks to the wallet back-end over remctl. Wallet uses remctl for authenticated and encrypted communication between a Wallet client and its server.
- The wallet back-end is a Perl program which checks authorization, documents an audit trail of operations performed by the client, and actually serves data (either static files out of a file bucket, or keytabs).
After obtaining and installing Wallet with the usual
./configure && make install
magic, I’m ready to configure it. First I add the following two lines to
remctld if necessary.
The Wallet back-end requires a database in which ACLs and the audit trail are stored.
Either of MySQL or SQLite can be used: I chose the latter.
To configure the Wallet back-end I set up a file called
wallet.conf, which must be
Please ignore the commented bits and the
default_owner sub for the time
being; I’ll get to those in a bit.
I then initialize Wallet’s database, specifying the Kerberos principal of an administrator. (Because I chose SQLite, I don’t have to create a database – Wallet does that for me now.)
I can now test Wallet. As a normal user, I try the following:
I can see my wallet client is speaking to the back-end, so we can continue from here.
At this point I add the following snippet to
krb5.conf to avoid having to specify
the Wallet-server’s name on each invocation of
wallet. (Note: this name can also
be compiled-in to the client.)
Let’s create a keytab:
That seems to have worked (and it has: I can see the new principal using kadmin.local directly), but if I now try and retrieve that same keytab, I get an error:
I found this difficult to understand, but Russ explained it to me a bit: by default, objects can be created by anybody on the ADMIN ACL. The ADMIN ACL was created automatically when we initialized the Wallet database, and the Kerberos principal we specified during initialization was added to the ADMIN ACL. Objects are created without ownership, but as ACLs are checked upon retrieving objects, we have to set an owner to an object before it can be retrieved:
So, that seems to be working, but how can I use Wallet for automation?
I explained above, that a principal and its keys are contained in a keytab, and that this keytab can be used to authenticate to a Kerberos service. This is precisely what we’ll do to authenticate a Wallet client to the Wallet server in order to obtain a host principal for the node we’re setting up.
The diagram shows what we do:
- I create a service principal named
service/wallet@MENS.DE, and export that to a keytab which is copied onto all nodes as part of the basic operating-system installation. The “copying” can be accomplished with whatever method you
- At any convenient moment the node has both a correct time set and network-connectivity
(to the KDC and Wallet servers), the wallet client tries to obtain a Kerberos
host principal (
host/mailserver.example.com) for the node it’s running on and stores that in the default keytab at
- The Wallet server does the brunt of the work, generating the keytab and providing other “secret” configuration data.
The service principal created in step 1 must be able to create additional host
principals (and change their keys) so, I add that to the kadmin’s
I discussed above, that the Wallet back-end uses ACLs to determine which Kerberos
entity (i.e. principal) is allowed to do what with respect to Wallet objects. Since
I want keytabs created by the node to be retrievable by the node, I create an ACL
KTabs and add the service
service/wallet principal to it.
Now comes a little bit of magic: when Wallet autocreates the host principal, it
checks whether a function (sub) called
default_owner exists in the server’s
wallet.conf (see above). This sub is used to set default ownership of
objects which don’t yet have an owner.
Because the principal creating the keytab (i.e. the object) is a non-ADMIN
service/wallet@MENS.DE is not in the ADMIN ACL) the sub invoked. It
should return a list containing the name of the ACL to use (
KTabs) and an
array of elements identifying the type (
krb5) and identifier
service/wallet@MENS.DE) for the ACL entry. So, after I authenticate as the
service principal using
kinit below, I can get a keytab which is made to
belong to the
KTabs ACL entry. (I may have omitted to mention that an authorized
principal can get a keytab even if it doesn’t exist; it is autocreated if
The service principal contained in our service keytab cannot do much harm; for example, it cannot remove (i.e. destroy) a principal:
So, keytabs are autocreated on the fly, can be downloaded by the special service principal and stored in a keytab of the target node.
Configuration files (a.k.a. other secret data)
Wallet allows me to not only manage keytabs but any other data I
consider worthy of similar protection: files. For example configuration files,
of which many contain clear-text passwords. (Some examples: MySQL’s
Arbitrary files managed by Wallet are stored in a file bucket – a directory
with subdirectories hashed by filename. The bucket-directory name is specified
$FILE_BUCKET, and objects within the file bucket can be
In the following example, I want administrators to be able to add files to the
bucket, and I want my service principal to only be able to read files and not
manipulate them: I create an ACL called
ko-files, create a file object and
set the file object’s ownership to that ACL. Then I add an administrative
root/admin@MENS.DE) to the ACL so that she can actually store
data in the file.
We now create a special get ACL to which we add our service principal so that it can retrieve configuration files. The reason for using a special ACL is to avoid the service principal being able to overwrite or even destroy config files.
Members of the
ko-readers ACL can retrieve the file’s content. For example,
to print the file to stdout, but they can’t do anything else with the object:
Kickstart (a.k.a. deployment)
Now that we have a keytab for a service principal, and Wallet is operational, I can put this to good use in setting up machines to obtain keytabs for host principals automatically, using almost any means at my disposal. I thought I’d show how to do it [with Ansible], but it’s probably easier to show what I did using RHEL/Centos’ Kickstart. Here is my post-install in the kickstart file.
Something similar could be done with, say, SLES installs, in AutoYast’s
post-scripts element of the XML file, and similarly for other platforms. Do
remember, though, that the machine’s clock must be within the clock-skew you allow in
your Kerberos environment, so one of the first things your installation script
should do is to synchronize with NTP.
Wallet has support for Stanford’s NetDB with which it can authorize principals, but since I neither have nor want a full-blown NetDB installation, I created a hack I call tenDB which emulates bits of NetDB sufficiently to please Wallet; tenDB uses LDAP as an authorization database.
- Every time Wallet retrieves (i.e. performs a get on) a keytab, that principal’s
key version number
is incremented (i.e. new keys are generated); while this is perfectly normal,
you may have to clear your credentials-cache to correctly authenticate with
the target node (
If you want to avoid this behavior, Wallet has a special keytab-back-end which retrieves a keytab for an existing principal without changing its current key.