Category Archives: Uncategorized

Using Clevis/Tang to do Volume Encryption in OpenStack

Nathaniel McCallum has been working for awhile now on a new mechanism to automate the unlocking of encrypted disks in a data center.  The idea is that as long as the disk remains in the data center – or more specifically, accessible to an encryption server  – on what is presumably a trusted network – the disk can be automatically unlocked.  More importantly though, the disk cannot be unlocked without access to the encryption server.

I say encryption server rather than key server, because the server does not actually store any volume keys.  There is no escrow.

Lets explore a little about how the McCallum Relyea exchange works, and why its so cool.


The math looks complicated, but it really isn’t.  Lets break it down a bit.

In the provisioning step, the server generates a key pair with private key S and public key s.  It then advertises the public key s.  The client also creates a key pair with private key C and public key c.

It then creates a symmetric key K using the server public key s and its own private key C.  Note that this is the same symmetric key that the server would create in the Diffie Hellmann exchange if it knew the client public key.

client: K = sC =  gSC          server: K = cS = gCS  = gSC

In this case, though, the client does not advertise its public key c, which means that only the client can derive K.  It uses K as a KEK for the volume and writes it in one of the LUKS slots.  It then discards K and its private key C — which means that it can no longer derive K – at least without the help of the server.

In the recovery step, the client generates a new key pair and combines that key with the client public key.  Based on what comes back from the server, the client can derive K, because:

K = y – sE = xS – gSE = (c+e)S – gSE = (c +gE)S – gSE = cS +gES -sSE = cS

A couple of notes:

  • During provisioning, only the server public key is needed.  The server does not even need to be online.  You could for instance put the public key in a kickstart file.
  • There is no state on the server.  No keys are transferred.  No escrow.
  • All of the stuff that is transferred (s,x,y) are either public or meaningless to an eavesdropper, so no TLS/encryption of the channel is required.

So what about OpenStack?

This is all great for the data center, where you have a trusted network.  But what about OpenStack.  How can you make an encryption server (Tang server) act like its on a local trusted network?  Nathaniel came up with a suggestion on how to do that too — using VPNs!


So, in this case, the Tang server looks like it is on the local network.  You could even configure the VPN to provide an IP for the Tang server that is on a private subnet in your tenant.  This totally solves the BYOK scenario.

Proof of Concept

In preparation for the OpenStack summit in Sydney, I did a basic proof of concept.  The goal here was to show that a local Tang server could be used to automatically decrypt volumes in an OpenStack deployment.

At this point, Tang/Clevis only supports the encryption of root volumes, so we had to test those volumes only,  I expect Tang/Clevis to be even more useful — and easier to integrate for non-root volumes when that support is added.

Step 1: Set up a Tang server

This is dead easy.

  • Create a Nova instance using a Centos 7.4 or RHEL 7.4 image.
  • Install and start the tang server
    • $ sudo dnf install tang
    • $ sudo systemctl enable --now tangd.socket

Step 2: Create a Provisioned Volume

The steps here are probably a bit more complicated than they need to be.  When I have time, I’ll go back and make them simpler and more consumable – ie. using CLIs and kickstarts., rather than using horizon.

  • Upload a Centos 7.4 ISO to glance.  Be sure to use the ISO type.  I used a minimal ISO. ( )
  • Create an empty volume in cinder.  In this case, I created one that was 10G large.  We’ll call this Vol1.
  • Create an instance in Nova using the ISO as the base image.
  • Attach Vol1 to that image.
  • Connect to the instance console.  The instance should boot into the anaconda install.
  • For the disk partitioning, add Vol1 and create two partitions –
    • A boot partition (/boot) of 256MB).  This partition MUST be unencrypted.
    • The rest as the root volume (/).  Be sure to select this volume to be encrypted and provide a passphrase.
  • Set the root password and complete the installation.
  • Now detach the (now no longer empty) Vol1 from the instance and set it to bootable.
  • Destroy the old instance.
  • Create a new instance with Vol1 as the base image.
  • Once the instance has spawned, log on through the console as the root user.  You will need to type in the LUKS password for the server to come up.  We are now going to install clevis, rebuild the initramfs and provision the volume with the Tang server.
    • ifup eth0
    • yum install wget clevis-dracut
    • change /etc/sysconfig/network-scripts/ifcfg-eth0 to enable the network on boot. (ONBOOT=yes)
    • Rebuild the initramfs:
      • dracut -f
    • Register with Tang:
      • clevis bind luks -d /dev/vda2 tang ‘{“url”:”http://<tang server ip>”}’
    • Reboot

Step 3: Test Away!

At this point, you can reboot the test instance and see that the server will automatically boot and mount the encrypted root volume when the tang server is up.

You can also shut down the Tang server and see that the instance boot process stops at the prompt to provide the LUKS password.  QED




Quick Update: Novajoin in the Undercloud

This is an update to the previous note, to account for changes in novajoin and puppet-novajoin as things have matured more and patches have landed.

In particular,

  • Novajoin has been simplified and caching is no longer done.
  • puppet-novajoin has been added to puppet-nova, rather than as a separate package.
  • puppet-ipa is no longer used.  This was primarily because Openstack would not accept puppet-ipa’s license.  Instead, some of the functionality needed (ability to register as a FreeIPA client) has been re-implemented in puppet-novajoin.
  • triple-o quickstart contains some in-review changes to set the DNS server for the undercloud node.  We want this to be the IPA server.


The basic steps needed are the same as before, albeit achieved slightly differently.  More details are provided in the subsequent details section.

  1. Create a new undercloud image containing a repo file for novajoin and a modified puppet-nova.
  2. Upload the image and checksum file to an accessible location.
  3. Register your undercloud node in IPA.
  4. Check out OOO Quickstart with the relevant patches.
  5. Modify the config file to update your IPA server etc.
  6. Run quickstart!


These are details on each of the steps above:

  1. You can create your own image by running the following script or you can use the image I created here.
  2. Make sure to upload both the image and the md5sum files.
  3. To register the IPA node, log onto the IPA server node and:
    kinit admin
    ipa host-add --password=MySecret --force
  4.   Do the following:
    git clone
    cd tripleo-quickstart
    git fetch refs/changes/72/398772/2 && git checkout FETCH_HEAD
  5.   Modify the file config/general_config/ha_ipa.yml:
    1. Set the proper ipa_domain, ipa_server, ipa_otp, ipa_principal and ipa_password.
    2. Set the IP address of the ipa server for the undercloud_dns_servers.
    3. Set the location of the undercloud image through:
    undercloud_image_url: ''
  6. Run quickstart.
    ./ --config config/general_config/ha_ipa.yml -e undercloud_image_url='' -R master --no-clone

Quick Note: NovaJoin in the Overcloud

In the last post, we discussed how to — starting from quickstart — get an undercloud which was enrolled with FreeIPA and which had the novajoin service up and running.  As part of the work to get that going, we had to create some puppet modules for novajoin.  We can reuse those same puppet modules to help us deploy overcloud controllers which are also enrolled with FreeIPA and on which novajoin is installed and configured.

As before, this is a quick note, so there is stuff here that will likely change as we iterate through this.  All the configuration steps below take place on an undercloud which has been registered with FreeIPA, and on which novajoin is running.  You could, for example, use the methods in the previous section to create this undercloud.

Prepare the Overcloud Image

The overcloud image needs to be customized in a number of ways:

  1. The image needs to have a recent enough cloud-init in order to retrieve the vendor metadata to register with IPA.
  2. By default, package installs are disabled during an overcloud install.  Essentially, any package operations are replaced by a no-op.  This means that all required packages (novajoin in particular) need to be installed in the image ahead of time.

As root, run the following:

cd ~stack
source ./stackrc

cat > novajoin.repo << EOF
name=Copr repo for novajoin owned by rcritten

chmod 777 /home/stack
virt-copy-in -a overcloud-full.qcow2 novajoin.repo /etc/yum.repos.d
virt-customize -a overcloud-full.qcow2 --install
virt-customize -a overcloud-full.qcow2 --install python-novajoin
openstack overcloud image upload --update-existing
chmod 700 /home/stack

Make sure DNS is set correctly on the undercloud

As the stack user,

source ./stackrc

# set nameserver to ipa server 
ID=$(openstack subnet list -f value -c ID)
openstack subnet set ${ID} --dns-nameserver  <ipa_server_address>

Add the relevant puppet modules

We need to add the puppet modules for IPA and novajoin to the overcloud image.  In addition, a new puppet manifest needs to be added to puppet-tripleo to call the novajoin puppet modules.  We’ll pull all these changes in from current gerrit reviews.

Rather than using virt-customize to copy in the relevant files, we will use the swift artifacts mechanism ( to deploy the puppet modules.

# set up puppet artifact mechanism
git config --global ""
git config --global "Ade Lee"

git clone
export PATH="$PATH:/home/stack/tripleo-common/scripts"

mkdir puppet-modules
cd puppet-modules
git clone tripleo

cd tripleo
git fetch \
  refs/changes/88/374288/1 && git cherry-pick FETCH_HEAD
cd ..
git clone ipa
git clone novajoin
cd ~

upload-puppet-modules -d puppet-modules

Get heat-templates

A new profile needs to be added to tripleo-heat-templates for the novajoin service.  This profile then needs to be included as an optional component.

git clone
cd tripleo-heat-templates
git fetch \
  refs/changes/85/374285/2 && git checkout FETCH_HEAD
cd ..

Create Environment Files

We create two environment files – one for joining IPA and one for novajoin.

cat >  /home/stack/tripleo-heat-templates/environments/ipa-join.yaml << EOF
    ipa_hostClass: app_server
    ipa_enroll: True

cat > novajoin.yaml << EOF
    OS::TripleO::Services::Novajoin: ./tripleo-heat-templates/puppet/services/novajoin.yaml
    IpaDomain: ''
    IpaPassword: 'redhat123'
    IpaPrincipal: 'admin'
    IpaServer: '<ipa server hostname>'
    NovaPassword: 'DxpwEd4bXxtCQgPan8QDHQQMT'

Deploy the Overcloud!

openstack overcloud deploy \
  --templates ./tripleo-heat-templates \
   -e ./tripleo-heat-templates/environments/ipa-join.yaml \
   -e novajoin.yaml


Quick Note: IPA Nova Join in the Undercloud

Note: The exact steps in this post have been superseded by this update.

Rob Crittenden has been working on a new nova microservice called novajoin that would instances created by nova to be automatically registered to IPA during cloud-init. (  It works using the newly provided vendor_metadata mechanism (   There is a short description of the design for this in the at the source link. This quick note is about steps that I took to integrate novajoin into an undercloud install as started by quickstart.  In particular, the goal here was to create an undercloud node which is registered to an IPA server, installs novajoin, configures nova to use novajoin and starts the novajoin service.  The undercloud novajoin service could then be used to deploy overcloud nodes that are automatically enrolled as IPA clients.

Create the undercloud image

First, it is necessary to create a new undercloud image.  This is required because the software we need is not yet present in the undercloud image.  The exact steps to create the undercloud image can be found here: In particular, we do the following:

    1. Start with an undercloud image.  In the script, I download from the triple-O master branch.
  • Add some code to /usr/share/instack-undercloud/puppet-stack-config/puppet-stack-config.pp to get the instack puppet code to call the puppet modules in (1) and (2).
  • Add a slightly modified novajoin-install script called ipa-novajoin-install-ipa.  This is a temporary step.  Right now, the novajoin install script does all the openstack configuration, as well as the IPA config.  We want to use the openstack puppet modules to do the required openstack configuration, and have the install script do the IPA configuration steps only.  Rob will re-factor the install scripts shortly and this step will no longer be necessary.

All of the above steps will in time become unnecessary as the relevant modules and code are merged into the Occata code base. The modified undercloud image can be downloaded from here:,

Add the undercloud node to IPA

Before invoking quickstart, we need to register the undercloud node to IPA providing it with an OTP.  On the IPA server, I did:

kinit admin

ipa host-add --password=MySecret --force

Modifying Quickstart

The puppet code that we added in instack-undercloud needs parameters which are provided by hieradata in quickstart-hieradata-overrides.yaml.  We need to make a small change to quickstart to allow it to pass these parameters to the instack-undercloud puppet modules.

We add the following to roles/tripleo/undercloud/templates/quickstart-hieradata-overrides.yaml.j2

{% if undercloud_ipa_client_install is defined %}
 enable_ipa_client_install: true
 ipa_domain: '{{ipa_domain}}'
 ipa_server: '{{ipa_server}}'
 ipa_otp: '{{ipa_otp}}'
 {% endif %}

{% if undercloud_novajoin_install is defined %}
 enable_novajoin_install: true
 nova::api::vendordata_jsonfile_path: '/etc/nova/cloud-config.json'
 nova::api::vendordata_providers: ['StaticJSON', 'DynamicJSON']
 nova::api::vendordata_dynamic_targets: ['join@']
 nova::notification_topics: 'notifications'
 nova::notify_on_state_change: 'vm_state'
 novajoin::api::hostname: "undercloud.%{hiera('ipa_domain')}"
 novajoin::api::ipa_domain: "%{hiera('ipa_domain')}"
 novajoin::api::ipa_password: "%{hiera('ipa_password')}"
 novajoin::api::ipa_principal: "%{hiera('ipa_principal')}"
 novajoin::api::ipa_server: "%{hiera('ipa_server')}"
 novajoin::api::keystone_identity_uri: "%{hiera('keystone_identity_uri')}"
 novajoin::api::keystone_auth_url: "%{hiera('keystone_auth_uri')}"
 novajoin::api::keystone_auth_uri: "%{hiera('keystone_auth_uri')}"
 novajoin::api::nova_password: "%{hiera('nova::keystone::authtoken::password')}"
 novajoin::api::transport_url: "%{hiera('nova::default_transport_url')}"
 ipa_principal: '{{ipa_principal}}'
 ipa_password: '{{ipa_password}}'
 {% endif %}

Then, we can create an environment file that specifies the required parameters (config/general_config/ha_ipa.yml)

undercloud_vcpu: 4

# Create three controller nodes and one compute node.
 - name: control_0
 flavor: control
 - name: control_1
 flavor: control
 - name: control_2
 flavor: control

- name: compute_0
 flavor: compute

# We don't need introspection in a virtual environment (because we are
 # creating all the "hardware" we really know the necessary
 # information).
 step_introspect: false

# Tell tripleo about our environment.
 network_isolation: true
 extra_args: >-
 --control-scale 3 --neutron-network-type vxlan
 --neutron-tunnel-types vxlan
 test_tempest: false
 test_ping: true
 enable_pacemaker: true

#ipa settings
 ipa_domain: ''
 ipa_server: ''
 ipa_otp: 'MySecret'

#novajoin settings
 ipa_principal: 'admin'
 ipa_password: 'password123'

undercloud_novajoin_install: true
 undercloud_ipa_client_install: true

The new stuff is really the part at the bottom – from IPA settings downwards.  As you can see, you just need to pass in the IPA domain, server and OTP.

Run Quickstart

We can now run quickstart as follows:

./ --config config/general_config/ha_ipa.yml -e undercloud_image_url='' -R master --no-clone


Setting up Barbican Behind TLS/SSL

Given that we are passing secrets back and forth between clients and the
Barbican server, it is absolutely imperative that the communications be
encrypted using TLS/SSL.

To be even more assured that the secrets are secure, one could use the
transport key mechanism that had been added to Barbican for use with
the Dogtag plugin. With this mechanism, the secret is encrypted with
a backend transport key that can only be decrypted on the back-end.

This means that secrets are always encrypted – even when there is no
SSL connection, and they are double-encrypted when there is an SSL

We will not focus on transport keys here, but rather on securing the
Barbican endpoint using TLS/SSL using haproxy. Haproxy will serve
https://hostname:9311, and will proxy the requests to the Barbican server
which will be listening on port 9312.

Note that in general, you are going to want to protect all the Openstack endpoints behind haproxy.  In this post, I’m only focusing on barbican.  For instructions on how to set up all the services (including Barbican), I recommend looking at the ansible scripts in rippowam

The steps are as follows:

  • Install haproxy::
sudo yum install haproxy
  • Get SSL certificate for haproxy from IPA using certmonger.  There are many possible ways of doing this.  I’m going to document how to do it using IPA (and registering the Openstack server as a client.  Ultimately, we’re going to end up registering the Openstack services in IPA.
sudo yum install ipa-client
sudo systemctl start certmonger.service 
echo <ipa_admin_password> | kinit admin@<ipa_realm> 
ipa service-add principal=HTTP/`hostname`@<ipa_realm> --force
setenforce 0 
ipa-getcert request -w -f /etc/haproxy/server.crt \
  -k /etc/haproxy/server.key -D "`hostname`" -K HTTP/`hostname`

cat /etc/haproxy/server.crt /etc/haproxy/server.key > /etc/haproxy/cert.pem 
chown haproxy: /etc/haproxy/cert.pem 
chmod 0600 /etc/haproxy/cert.pem
setenforce 1
  • . You could also get the certificates needed by contacting Dogtag directly using the pki CLI, or by using Barbican/Dogtag  to issue a server cert.
    • TODO – show how to use Barbican to get the cert from dogtag
  • Install haproxy config file in /etc/haproxy.cfg.
 # to have these messages end up in /var/log/haproxy.log you will 
 # need to: 
 # 1) configure syslog to accept network log events. This is done 
 # by adding the '-r' option to the SYSLOGD_OPTIONS in 
 # /etc/sysconfig/syslog 
 # 2) configure local2 events to go to the /var/log/haproxy.log 
 # file. A line like the following can be added to 
 # /etc/sysconfig/syslog 
 # local2.* /var/log/haproxy.log 
 log local2
chroot /var/lib/haproxy 
 pidfile /var/run/ 
 maxconn 4000 
 user haproxy 
 group haproxy 
# turn on stats unix socket 
 stats socket /var/lib/haproxy/stats
 # common defaults that all the 'listen' and 'backend' sections will 
 # use if not designated in their block 
 mode http 
 timeout connect 10s 
 timeout client 10s 
 timeout server 10s 
 maxconn 10000 
 balance roundrobin 
 option forwardfor
backend barbican-api 
 server barbican-01 check inter 10s
frontend barbican-api 
 bind ssl crt /etc/haproxy/cert.pem 
 default_backend barbican-api
  • Set up Barbican to bind locally to port 9312.::
crudini --set /etc/barbican/barbican.conf DEFAULT bind_host
crudini --set /etc/barbican/barbican.conf DEFAULT bind_port 9312 
crudini --set /etc/barbican/barbican.conf DEFAULT host_href https://`hostname`:9311
crudini --set /etc/barbican/vassals/barbican-api.ini uwsgi socket :9312 
sed -i 's/bind = '\'''\''/bind = '\'''\''/' /etc/barbican/
systemctl restart openstack-barbican-api.service
systemctl restart haproxy.service