A Kerberos Key Distribution Center (KDC) stores user accounts (principals) and their keys (derived from a user’s password with the string2key() function) in its database. These keys are used as part of the authentication process to verify whether a user is authenticated when he or she accesses a server or service. Similarly, hosts and services in a Kerberized environment have a host or service principal – an account and a bunch of keys also in the KDC database.

If keys associated with network services had passwords, a service couldn’t start up without administrative intervention by a human to enter the password. This means that service principals typically have randomly generated keys, and the network service has to have a copy of its keys on the system. Such keys are typically stored in files on the file system called keytabs and keytabs can contain more than one key. For example, the Kerberos keytab on my workstation contains the following keys:

$ ktutil
ktutil:  rkt /etc/krb5.keytab
ktutil:  l -e
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
   1    2   ldap/jmbp.ww.mens.de@MENS.DE (AES-256 CTS mode with 96-bit SHA-1 HMAC) 
   2    2   ldap/jmbp.ww.mens.de@MENS.DE (AES-128 CTS mode with 96-bit SHA-1 HMAC) 
   3    2   ldap/jmbp.ww.mens.de@MENS.DE (ArcFour with HMAC/md5) 
   4    5   host/jmbp.ww.mens.de@MENS.DE (AES-256 CTS mode with 96-bit SHA-1 HMAC) 
   5    5   host/jmbp.ww.mens.de@MENS.DE (AES-128 CTS mode with 96-bit SHA-1 HMAC) 
   6    5   host/jmbp.ww.mens.de@MENS.DE (ArcFour with HMAC/md5) 
ktutil:  quit

A keytab is typically created (or keys added to it) by an administrator logging in to the system on which the keytab is to be maintained, invoking the Kerberos administration utility (kadmin) over the network, authenticating on as an authorized principal, generating a key and adding that key to a new or existing keytab. (Alternatively, a keytab can be created on a foreign machine and transported securely, e.g. via SSH.) That process looks like this:

$ kadmin -p root/admin
Authenticating as principal root/admin with password.
Password for root/admin@MENS.DE: 
kadmin:  ank -clearpolicy -randkey blog/demokey
Principal "blog/demokey@MENS.DE" created.

kadmin:  ktadd -k /etc/krb5.keytab blog/demokey
Entry for principal blog/demokey with kvno 3, encryption type AES-256 CTS mode with 96-bit SHA-1 HMAC added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal blog/demokey with kvno 3, encryption type AES-128 CTS mode with 96-bit SHA-1 HMAC added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal blog/demokey with kvno 3, encryption type ArcFour with HMAC/md5 added to keytab WRFILE:/etc/krb5.keytab.
kadmin:  quit

$ ls -l /etc/krb5.keytab
-rw-------  1 root  wheel  798 Jun 14 20:09 /tmp/krb5.keytab

Within the kadmin interface, I create a new principal blog/demokey@MENS.DE in the KDC’s database, and then export that principal’s keys into the specified keytab. Fine and dandy if I do that once in a while, but I wouldn’t like to have to do that for a few dozen or hundred machines deployed in a large environment. And how would we automatically and securely (!) provision keytabs on machines that are automatically deployed? (I hate to say this, but I’ve seen shell scripts which invoke kadmin -w with the hard-coded password of a principal allowed to create keys contained in them … (sigh); don’t do that!)

I digress a bit but I can “see” the principal created above looking at the LDAP back-end in which my KDC’s database is stored; the LDAP entry looks like this:

$ ldapsearch -LLL krbPrincipalName=blog/demokey@MENS.DE
dn: krbPrincipalName=blog/demokey@MENS.DE,cn=MENS.DE,ou=kerberos,dc=mens,dc=de
krbPrincipalName: blog/demokey@MENS.DE
objectClass: krbPrincipal
objectClass: krbPrincipalAux
objectClass: krbTicketPolicyAux
krbTicketFlags: 0
krbLoginFailedCount: 0
krbPasswordExpiration: 19700101000000Z
krbLastPwdChange: 20120614180930Z
krbExtraData:: AALaKNpPcm9vdC9hZG1pbkBNRU5TLkRFAA==
krbExtraData:: AAgBAA==

So, how do we deploy keytabs automatically onto nodes? With Wallet.


Wallet, created by Russ Allbery, is a system for managing authorization and retrieval of secure data, Secure data can be any file (e.g. configuration files which contain clear-text passwords) and keytabs.

Wallet architecture

Wallet is a client/server program:

  • The wallet client is a small C program which talks to the wallet back-end over remctl. Wallet uses remctl for authenticated and encrypted communication between a Wallet client and its server.
  • The wallet back-end is a Perl program which checks authorization, documents an audit trail of operations performed by the client, and actually serves data (either static files out of a file bucket, or keytabs).

After obtaining and installing Wallet with the usual ./configure && make install magic, I’m ready to configure it. First I add the following two lines to remctl.conf and reload remctld if necessary.

wallet store /usr/local/sbin/wallet-backend stdin=4 ANYUSER
wallet ALL /usr/local/sbin/wallet-backend ANYUSER

The Wallet back-end requires a database in which ACLs and the audit trail are stored. Either of MySQL or SQLite can be used: I chose the latter. To configure the Wallet back-end I set up a file called wallet.conf, which must be valid Perl:

$DB_DRIVER = 'SQLite';
$DB_INFO = '/etc/wallet/wallet.db';

$KEYTAB_KRBTYPE         = 'MIT';
$KEYTAB_FILE            = '/etc/wallet/srv.wallet.keytab';
$KEYTAB_PRINCIPAL       = 'service/wallet@MENS.DE';
$KEYTAB_REALM           = 'MENS.DE';
$KEYTAB_FLAGS           = '-clearpolicy';
$KEYTAB_HOST            = 'hippo.ww.mens.de';
$KEYTAB_TMP             = '/var/tmp';
$KEYTAB_KADMIN          = '/usr/bin/kadmin';

$FILE_BUCKET            = '/etc/wallet/bucket';
$FILE_MAX_SIZE          = 4096;

# This is for JP's NetDB-shim, a.k.a. tenDB; see
# https://github.com/jpmens/tenDB  for details
#  $NETDB_REALM            = 'MENS.DE';
#  $NETDB_REMCTL_HOST      = 'hippo.ww.mens.de';
#       Kerberos credential-cache required for Wallet
#       to speak  to NetDB over remctl. Can be primed
#       and kept alive with k5start.
#  $NETDB_REMCTL_CACHE     = '/etc/wallet/tenDB.ccache';

sub default_owner {
  my ($type, $name) = @_;

  if ($type eq 'keytab') {
    return ('KTabs', [ 'krb5', 'service/wallet@MENS.DE' ]);
  } else {

Please ignore the commented bits and the default_owner sub for the time being; I’ll get to those in a bit.

I then initialize Wallet’s database, specifying the Kerberos principal of an administrator. (Because I chose SQLite, I don’t have to create a database – Wallet does that for me now.)

# wallet-admin initialize root/admin@MENS.DE

I can now test Wallet. As a normal user, I try the following:

$ wallet -s hippo.ww.mens.de get keytab test
wallet: jpm@MENS.DE not authorized to create keytab:test

I can see my wallet client is speaking to the back-end, so we can continue from here. At this point I add the following snippet to krb5.conf to avoid having to specify the Wallet-server’s name on each invocation of wallet. (Note: this name can also be compiled-in to the client.)

   wallet = {
      wallet_port   = 4373
      wallet_server = hippo.ww.mens.de

Let’s create a keytab:

$ wallet -u root/admin@MENS.DE create keytab host/test1
Password for root/admin@MENS.DE:

That seems to have worked (and it has: I can see the new principal using kadmin.local directly), but if I now try and retrieve that same keytab, I get an error:

$ wallet -u root/admin@MENS.DE get keytab host/test1
Password for root/admin@MENS.DE: 
wallet: root/admin@MENS.DE not authorized to get keytab:host/test1

I found this difficult to understand, but Russ explained it to me a bit: by default, objects can be created by anybody on the ADMIN ACL. The ADMIN ACL was created automatically when we initialized the Wallet database, and the Kerberos principal we specified during initialization was added to the ADMIN ACL. Objects are created without ownership, but as ACLs are checked upon retrieving objects, we have to set an owner to an object before it can be retrieved:

$ kinit root/admin
Password for root/admin@MENS.DE: 
$ wallet create keytab host/test2
$ wallet show keytab host/test2
           Type: keytab
           Name: host/test2
     Created by: root/admin@MENS.DE
   Created from: hippo.ww.mens.de
     Created on: 2012-06-19 15:03:17
$ wallet owner keytab host/test2 ADMIN
$ wallet -f /tmp/test2.keytab get keytab host/test2
$ ktutil
ktutil:  rkt /tmp/test2.keytab
ktutil:  l
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
   1    2                       host/test2@MENS.DE
   2    2                       host/test2@MENS.DE
   3    2                       host/test2@MENS.DE
ktutil:  quit

$ wallet show keytab host/test2
           Type: keytab
           Name: host/test2
          Owner: ADMIN
     Created by: root/admin@MENS.DE
   Created from: hippo.ww.mens.de
     Created on: 2012-06-19 15:04:53
  Downloaded by: root/admin@MENS.DE
Downloaded from: hippo.ww.mens.de
  Downloaded on: 2012-06-19 15:05:30

Members of ACL ADMIN (id: 1) are:
  krb5 root/admin@MENS.DE

So, that seems to be working, but how can I use Wallet for automation?


I explained above, that a principal and its keys are contained in a keytab, and that this keytab can be used to authenticate to a Kerberos service. This is precisely what we’ll do to authenticate a Wallet client to the Wallet server in order to obtain a host principal for the node we’re setting up.

Automatic keytab distribution

The diagram shows what we do:

  1. I create a service principal named service/wallet@MENS.DE, and export that to a keytab which is copied onto all nodes as part of the basic operating-system installation. The “copying” can be accomplished with whatever method you
  2. At any convenient moment the node has both a correct time set and network-connectivity (to the KDC and Wallet servers), the wallet client tries to obtain a Kerberos host principal (host/mailserver.example.com) for the node it’s running on and stores that in the default keytab at /etc/krb5.keytab.
  3. The Wallet server does the brunt of the work, generating the keytab and providing other “secret” configuration data.

The service principal created in step 1 must be able to create additional host principals (and change their keys) so, I add that to the kadmin’s kadm5.acl:

*/admin@MENS.DE *
service/wallet@MENS.DE  *

I discussed above, that the Wallet back-end uses ACLs to determine which Kerberos entity (i.e. principal) is allowed to do what with respect to Wallet objects. Since I want keytabs created by the node to be retrievable by the node, I create an ACL I name KTabs and add the service service/wallet principal to it.

# wallet acl create KTabs
# wallet acl add KTabs krb5 service/wallet@MENS.DE

Now comes a little bit of magic: when Wallet autocreates the host principal, it checks whether a function (sub) called default_owner exists in the server’s wallet.conf (see above). This sub is used to set default ownership of objects which don’t yet have an owner.

Because the principal creating the keytab (i.e. the object) is a non-ADMIN (i.e. service/wallet@MENS.DE is not in the ADMIN ACL) the sub invoked. It should return a list containing the name of the ACL to use (KTabs) and an array of elements identifying the type (krb5) and identifier (service/wallet@MENS.DE) for the ACL entry. So, after I authenticate as the service principal using kinit below, I can get a keytab which is made to belong to the KTabs ACL entry. (I may have omitted to mention that an authorized principal can get a keytab even if it doesn’t exist; it is autocreated if necessary.)

$ kinit -k -t srv.wallet.keytab -p service/wallet@MENS.DE
$ wallet -f /tmp/k4 get keytab nfs/a.4
$ wallet show keytab nfs/a.4
           Type: keytab
           Name: nfs/a.4
          Owner: KTabs
     Created by: service/wallet@MENS.DE
   Created from: hippo.ww.mens.de
     Created on: 2012-06-19 15:57:33
  Downloaded by: service/wallet@MENS.DE
Downloaded from: hippo.ww.mens.de
  Downloaded on: 2012-06-19 15:57:33

Members of ACL KTabs (id: 2) are:
  krb5 service/wallet@MENS.DE

$ ls -l /tmp/k4
-rw------- 1 root root 334 Jun 19 15:57 /tmp/k4

The service principal contained in our service keytab cannot do much harm; for example, it cannot remove (i.e. destroy) a principal:

$ wallet destroy keytab nfs/a.4
wallet: service/wallet@MENS.DE not authorized to destroy keytab:nfs/a.4

So, keytabs are autocreated on the fly, can be downloaded by the special service principal and stored in a keytab of the target node.

Configuration files (a.k.a. other secret data)

Wallet allows me to not only manage keytabs but any other data I consider worthy of similar protection: files. For example configuration files, of which many contain clear-text passwords. (Some examples: MySQL’s my.cnf, NSS LDAP’s ldap.conf, etc.)

Arbitrary files managed by Wallet are stored in a file bucket – a directory with subdirectories hashed by filename. The bucket-directory name is specified in $FILE_BUCKET, and objects within the file bucket can be $FILE_MAX_SIZE large.

In the following example, I want administrators to be able to add files to the bucket, and I want my service principal to only be able to read files and not manipulate them: I create an ACL called ko-files, create a file object and set the file object’s ownership to that ACL. Then I add an administrative principal (root/admin@MENS.DE) to the ACL so that she can actually store data in the file.

$ wallet acl create ko-files
$ wallet create file my.config
$ wallet owner file my.config ko-files
$ wallet acl add ko-files krb5 root/admin@MENS.DE
$ echo "very secret" | wallet store file my.config

We now create a special get ACL to which we add our service principal so that it can retrieve configuration files. The reason for using a special ACL is to avoid the service principal being able to overwrite or even destroy config files.

$ wallet acl create ko-readers
$ wallet acl add ko-readers krb5 service/wallet@MENS.DE
$ wallet setacl file my.config get ko-readers
$ wallet show file my.config
           Type: file
           Name: my.config
          Owner: ko-files
        Get ACL: ko-readers
     Created by: root/admin@MENS.DE
   Created from: hippo.ww.mens.de
     Created on: 2012-06-25 16:24:55
      Stored by: service/wallet@MENS.DE
    Stored from: ::ffff:
      Stored on: 2012-06-25 16:26:44
  Downloaded by: service/wallet@MENS.DE
Downloaded from: ::ffff:
  Downloaded on: 2012-06-25 16:59:06

Members of ACL ko-files (id: 3) are:
  krb5 root/admin@MENS.DE

Members of ACL ko-readers (id: 4) are:
  krb5 service/wallet@MENS.DE

Members of the ko-readers ACL can retrieve the file’s content. For example, to print the file to stdout, but they can’t do anything else with the object:

$ wallet get file my.config
very secret

$ wallet show file my.config
wallet: service/wallet@MENS.DE not authorized to show file:my.config

Kickstart (a.k.a. deployment)

Now that we have a keytab for a service principal, and Wallet is operational, I can put this to good use in setting up machines to obtain keytabs for host principals automatically, using almost any means at my disposal. I thought I’d show how to do it [with Ansible], but it’s probably easier to show what I did using RHEL/Centos’ Kickstart. Here is my post-install in the kickstart file.


# Wallet & remctl. Obtain package (built locally) and install.
# Not setting up a special repo for this, which is why I wget/yum it.

wget -O /tmp/remwallet.rpm http://hippo.ww.mens.de/wallet/remwallet.rpm &&
   yum -q -y install /tmp/remwallet.rpm &&
   rm /tmp/remwallet.rpm
wget -O /etc/krb5.conf http://hippo.ww.mens.de/wallet/krb5.conf

# Retrieve service keytab for wallet and install it

wget -O /etc/srv.wallet.keytab http://hippo.ww.mens.de/wallet/srv.wallet.keytab
chmod 400 /etc/srv.wallet.keytab

# Get our host keytab from Wallet

kinit -k -t /etc/srv.wallet.keytab -p service/wallet@MENS.DE
wallet -f /etc/krb5.keytab get keytab host/c2.ww.mens.de
chmod 600 /etc/krb5.keytab

# Retrieve MySQL config
wallet -f /root/.my.cnf get file my.config

kdestroy	# don't leave cache lying around

Something similar could be done with, say, SLES installs, in AutoYast’s post-scripts element of the XML file, and similarly for other platforms. Do remember, though, that the machine’s clock must be within the clock-skew you allow in your Kerberos environment, so one of the first things your installation script should do is to synchronize with NTP.

Final thoughts

Wallet has support for Stanford’s NetDB with which it can authorize principals, but since I neither have nor want a full-blown NetDB installation, I created a hack I call tenDB which emulates bits of NetDB sufficiently to please Wallet; tenDB uses LDAP as an authorization database.


  • Every time Wallet retrieves (i.e. performs a get on) a keytab, that principal’s key version number is incremented (i.e. new keys are generated); while this is perfectly normal, you may have to clear your credentials-cache to correctly authenticate with the target node (kinit).
  • If you want to avoid this behavior, Wallet has a special keytab-back-end which retrieves a keytab for an existing principal without changing its current key.

Kerberos, keytab, and Ansible :: 26 Jun 2012 :: e-mail