Barbican and Dogtag/IPA

The default back-end plugins that ship with Barbican are not really suitable for a production deployment. The simple storage plugin simply encrypts the secrets using a single symmetric key that is stored in a plain text file, and the snake oil plugin uses self signed certificates.

Over the last 10 years, I have been part of the Dogtag Certificate System team. Dogtag is the upstream project for the Red Hat Certificate System, which has been deployed and battle-hardened in some of the largest PKI deployments in the world. We use it to generate certificates, store private keys (encryption keys), issue certificates to tokens etc. It has also passed Common Criteria, works with the major HSMs, and provides robust  HA and DR solutions.

Over the last few years, we have augmented one of its components, the Key Recovery Agent (KRA) to be able to securely store more than just encryption keys. Now we can store symmetric keys (for volume encryption, for example) and passphrases. This makes the KRA an ideal back-end for Barbican.

Moreover, we have added REST interfaces and python client code to the KRA and CA components. This makes these components ideal to act as backends for Barbican, and a Barbican plugin has been written (mostly by me) to interface with the KRA.

In this posting, I’m going to describe the simple steps needed to integrate
Dogtag with Barbican, so that you can create a production-backend-backed Barbican.

Dogtag and IPA

Over the last few years, we have also integrated Dogtag into FreeIPA (that is Identity Management or IdM). IdM is part of RHEL and includes the Dogtag CA and KRA as key components used for issuing server certificates and providing the back-end store for the IPA Vault respectively.

While it is certainly possible to configure Barbican with a stand-alone Dogtag CA and KRA, there are many benefits that a full IPA instance provides to an Openstack deployment. A lot of those benefits were  explored in some of the demos we created for the Tokyo summit. <links>

In particular, we are able to:

  • Register each of the openstack services and service users in the
    undercloud with IPA providing them with kerberos credentials.
  • Use these credentials to issue SSL certs for haproxy, so that the service endpoints would be secured.
  • Secure communication with the database and the message queues.
  • Use a nova plugin (written by Rich Megginson) to register new VMs to IPA, providing them with an identity and kerberos credentials. This solves the problem that Kevin Fox is trying to solve in (Fox spec).  Certificates for each VM (and any other resources) can be retrieved using kerberos credentials. IPA can also be used to manage the host and provide useful features like access control.
  • IPA can be tied into existing identity stores like Active Directory.

The demos provide some compelling use cases for deploying Barbican
and Openstack with IPA. In this blog, I’ll describe how to configure
Barbican with IPA.

Creating an IPA instance

To use IPA, it is highly recommended that you set up IPA in a separate VM. IPA uses some of the same ports needed by Openstack, including ports 80 and 443 (which might be used by horizon), as well as ports 8080 and 8443 (used by Dogtag). A separate VM just makes things clearer and easier to debug, and keeps a crisp separation of concerns.

Using a container

One of the easiest ways to deploy IPA is through the use of a container. Once the container is created, an configuration and data is stored separately so that the container can be started/stopped or created and recreated without loss of data.

This also allows one to treat the IPA instance essentially like an appliance. Just as you would not necessarily care to know about the internal workings of your stove or fridge – you would not need to know about the internals of your IPA instance. Just turn it on and configure Barbican to work with it.

This is useful also for folks who are not as familiar with – or do not want to run Fedora or RHEL machines. They could, for instance, run IPA on RHEL/Centos/Fedora in a container, and run Barbican on Ubuntu.

  • Install and start docker.
sudo yum install docker
systemctl start docker
  • Pull down the IPA Docker image.
docker pull adelton/freeipa-server
  • Create a directory in which will contain server info (/var/lib/ipa-data for example).  This directory will be used during the initial install, and will be used to save any data needed to restore the container.
  • Create a file: ipa-server-install-options, which will contain command line parameters to the ipa-server-install command that is run after the container is created.  You want at least:

  • Run the container.  This will create the container and will execute ipa-server-install to install IPA and the Dogtag CA, and will start a console in which you are logged in to the server.
    docker run --name freeipa-server-container -ti \
       -h ipa.example.test \
       -v /var/lib/ipa-data:/data:Z freeipa-server
  • The current IPA containers do not yet install the KRA.  This is easy to do though.  While logged onto the ipa container:

Using a separate VM

IPA can easily be installed on a RHEL 7 (Centos 7) or Fedora 22/23 VM. Specify RHEL 7 or Fedora 22/23 to ensure that the right version of Dogtag is installed. You need at least Dogtag 10.2 to ensure that the Dogtag REST API is present.

The steps are:

  • Install a VM with RHEL 7 (Centos 7) or Fedora 23/22.
  • Install the IPA software
sudo yum install freeipa-server (Fedora) OR
sudo yum install ipa-server (RHEL/Centos)
  • Install IPA with the CA
sudo ipa-server-install
  • Install the KRA:
sudo ipa-kra-install

Setting up Barbican to use Dogtag (in IPA)

Setting up Barbican to use Dogtag in IPA is straightforward.

  • Copy over a PEM file containing the certificate and key for the IPA
    CA and KRA agent from the IPA machine. Copy this file to
    /etc/barbican/kra_agent.pem and set the ownership to root:barbican.  On the IPA machine, this file is located at /etc/httpd/alias/kra-agent.pem.  In the docker example above, this file can be found on the host machine at /var/lib/ipa-data/etc/httpd/alias/kra-agent.pem.

If integrating with Dogtag directly, you can create this file by executing the following OpenSSL commands to extract the agent certificate and key from the agent NSS database::

openssl pkcs12 -in ~/.dogtag/pki-tomcat/ca_admin_cert.p12 \ 
 -out ~/admin_cert.pem -nodes

Note: Ideally one should use the Dogtag admin tools to create a new agent specifically for Barbican usage. Better yet, we should use the kerberos credentials provided by IPA to interact with Dogtag and create this user. This requires some changes to Dogtag (kra acls design), which are currently in design and are slated to go out in Dogtag 10.3.

  • On the Barbican machine, install the Dogtag client code::
sudo yum install pki-base

Right now, this yum operation will pull in all the dependencies for both the Python and Java Dogtag clients. There is currently work underway to separate these clients, so that only the Python client would be installed. (Dogtag separation ticket). That work is expected to be completed soon.

For Debian systems, there should be Debian packages. Alternatively, one could try the pip -installable Dogtag client package at .

Also, note that at the time I wrote this, Centos 7 had not yet updated to reflect the release of RHEL 7.2.  This means that Centos 7 pulled in RHEL 7.1 bits which includes pki-base 10.1.  This version does not have the required python client library calls.  We expect Centos 7 to update to pull in Dogtag 10.2 shortly.  In the meantime, Centos 7 builds can be obtained from the IPA copr repo.

  • Modify the /etc/barbican/barbican.conf configuration file to enable the Dogtag plugins for certificate generation and secret storage:
crudini --set /etc/barbican/barbican.conf secretstore enabled_secretstore_plugins dogtag_crypto
crudini --set /etc/barbican/barbican.conf dogtag_plugin dogtag_host ipa_host
crudini --set /etc/barbican/barbican.conf dogtag_plugin dogtag_port 8443
crudini --set /etc/barbican/barbican.conf dogtag_plugin pem_path /etc/barbican/kra-agent.pem
crudini --del --list /etc/barbican/barbican.conf certificate enabled_certificate_plugins
crudini --set /etc/barbican/barbican.conf certificate enabled_certificate_plugins dogtag
systemctl restart openstack-barbican-api.service
The final version of barbican.conf will contain sections that look like :

    namespace = barbican.secretstore.plugin 
    enabled_secretstore_plugins = dogtag_crypto 

    pem_path = /etc/barbican/kra-agent.pem 
    dogtag_host = ipa_host 
    dogtag_port = 8443 
    nss_db_path = '/etc/barbican/alias' 
    nss_db_path_ca = '/etc/barbican/alias-ca' 
    nss_password = 'password123' 
    simple_cmc_profile = 'caOtherCert' 
    ca_expiration_time = 1 
    plugin_working_dir = '/etc/barbican/dogtag' 

    namespace = barbican.certificate.plugin 
    enabled_certificate_plugins = dogtag

Testing Barbican/Dogtag

To test out Barbican/Dogtag, I usually just end up performing the tests
described in the Testing Barbican section above (previous post).  The Dogtag plugin currently does not support subordinate CAs, but that should change soon.

To make sure the certificate and secret storage and retrieval requests
actually being sent to the Dogtag CA and KRA, I usually tail the debug
logs of the CA and KRA while executing the tests.

On the IPA VM,

tail -f /var/log/pki/pki-tomcat/kra/debug 
tail -f /var/log/pki/pki-tomcat/ca/debug

4 thoughts on “Barbican and Dogtag/IPA

  1. Have tested IPA node on RH7 VM and Barbican already running on RDO FC23 .Little bit confused – if Dogtag should be installed on IPA node ?

    On barbican ran #yum install dogtag-pki.noarch . Made all rest of suggested settings .My barbican.conf looks like now :

    namespace = barbican.secretstore.plugin
    enabled_secretstore_plugins = dogtag_crypto

    pem_path = /etc/barbican/kra-agent.pem
    dogtag_host =
    dogtag_port = 8443
    nss_db_path = ‘/etc/barbican/alias’
    nss_db_path_ca = ‘/etc/barbican/alias-ca’
    nss_password = ‘password123’
    simple_cmc_profile = ‘caOtherCert’
    ca_expiration_time = 1
    plugin_working_dir = ‘/etc/barbican/dogtag’

    namespace = barbican.certificate.plugin
    enabled_certificate_plugins = dogtag

    restarted openstack-barbican-api.service and right after that I cannot create a new encrypted volume which was possible before the barbican.conf change.

    2015-12-10 15:39:00.787 10985 ERROR cinder.keymgr.barbican [req-3c84c9c7-2a54-4e14-83a4-23031e93dc79 – – – – -] Error creating key.

    tcpdump doesn’t show any traffic to IPA server .

    also from Barbican

    [root@aiob2 ~(keystone_admin)]# telnet 8443
    Connected to
    Escape character is ‘^]’.

    Thanks .


    1. First off, on the Barbican node, you just need the Dogtag client code, so you need to install pki-base – rather than dogtag-pki (which installs everything).

      That should not cause a problem though, because pki-base should have been installed when you installed dogtag-pki.

      Did you copy over the agent p12 file from the IPA machine and put it in the right place on the barbican machine?

      Do you see that an NSS database has been created under /etc/barbican/alias?
      Do you see that a transport certificate has been retrieved from the ipa machine?
      You can check this with: certutil -L -d /etc/barbican/alias

      If you look at the logs for barbican —
      journalctl -u openstack-barbican-api.service

      you can see what happened during startup, in particular when the dogtag plugin was supposed to be loaded. Search for dogtag .. That should tell you what went wrong. In fact, there is likely to be a stacktrace.

      Finally, the Dogtag client code has a CLI called pki which can be used to connect to Dogtag on the IPA machine. This is delivered by pki-tools. See “man pki” for details. Try connecting to the KRA and doing a list of secrets, or to the CA and getting a list of certs.


  2. I took ca-agent.p12 from IPA and uploaded it on Barbican

    [root@aiob2 ~(keystone_admin)]# pk12util -i ca-agent.p12 -d /etc/barbican/alias/
    Enter Password or Pin for “NSS Certificate DB”:
    Enter Password or Pin for “NSS Certificate DB”:
    Enter password for PKCS12 file:

    Now have this :

    [root@aiob2 ~(keystone_admin)]# certutil -L -d /etc/barbican/alias/

    Certificate Nickname Trust Attributes

    ipa-ca-agent u,u,u

    If the right file you mentioned (agent p12 file ) ?

    When creating a new encrypted volume see the following in the journalctl for openstack-barbican-api.service

    Dec 13 13:19:57 gunicorn[26280]: 2015-12-13 13:19:57.368 26514 ERROR barbican.plugin.util.utils [req-ce33f647-64cd-4859-8ae3-3c3207e66ac6 4592fc3cf3a140ca98c1f2a1685321d5 c38ca2777fe04aa4992a37b804da6b8e – default default] Problem seen creating plugin: ‘dogtag_crypto’


    1. OK – the Dogtag plugin uses python-requests to contact the Dogtag server on the IPA machine. Therefore it expects the agent cert and key to be in a PEM file. In the example above, that PEM file would be located at the configured value:

      pem_path = /etc/barbican/kra-agent.pem

      This PEM file is actually created on the IPA machine during the IPA install (see post for locations), That file should be copied over and the appropriate permissions and ownership should be set.

      The NSS database that is under /etc/barbican/alias is used by the Dogtag plugin to do crypto operations – like wrapping the secrets being passed to the KRA in the KRA transport key. As such, it will contain the KRA transport key which is fetched by the plugin on startup. I suggest the you remove the /etc/barbican/alias directory. It will be recreated by the plugin on initialization. Once running, it should contain the transport cert.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s