You have secrets in your cluster. Everybody does, it’s a fact of life. Database passwords, API keys, deployment tokens, just to name a few. Secrets are hard to manage, even before you throw in the fact that most of us now are operating in a cloud environment. In software development, the common advice to “never roll your own crypto” tends to get thrown out the window when you’re talking about infrastructure secrets. These home-grown solutions are hard to maintain and rarely audited properly. But the alternatives aren’t great: go without or suffer vendor lock in. And that’s without even mentioning tasks we all need to know we need to do like key rolling. Add containers into the mix and you’ve got a real headache on your hands.

At Cisco, as we developed our opensource microservices platform Mantl, we wanted to address all aspects of the software developers security concerns. Security is still one of the top concerns in cloud today and it should be.  To address security in depth, you need to start at the beginning of the project with security practices and controls in mind. In this post we are looking at secrets. We leveraged Vault, an open-source tool recently released by Hashicorp to manage secret sharing within a datacenter. We’re going to highlight some key features that you’ll want to know about, and then get into how to actually use the software in Mantl. Vault on Mantl means that you can have more secure integration with your cloud resources as well as enhanced operational procedures internally. We think that it sets Mantl apart as a microservices  infrastructure that takes security seriously.

The first, and arguably most useful, feature of Vault is automatic secret rolling. As the administrator, you set up and manage secure backends like PostgreSQL or AWS. Essentially, you give Vault permissions to create and manage users with a specific set of access permissions in these systems, and Vault will take care of issuing and revoking secrets. It has the concept of a lease on secrets, so a client knows how long the secret it has is good for, and when it will need to get a new one.

But secret rolling isn’t very useful without the ability to audit secret issuance and access, and Vault delivers here too. It can write audit logs to a file on the system or to syslog for your log solution to slurp up for later review. And how do you connect with this system? Vault ships a handful of auth backends. For operators, Vault has LDAP, MFA tokens, and Github authentication. For automated consumers, there are the methods you’d expect like TLS certificates and token-based authentication plus “app ID”, a mechanism for new nodes to authenticate with Vault.

You can either download Vault for your platform at the Vault project download page. A note about the download and release process: Hashicorp publishes checksums signed by their GPG key for verification. The codebase itself, besides being open source, is professionally audited by iSec. That is not to say that it’s perfect, but it’s quite a step up from unsigned release binaries on Github. Of course, since Vault is open source you can also choose to compile a release yourself with the instructions in their README.

Let’s get started using Vault. We’ll be running these examples locally, and I’m assuming that the vault tool is in your path. If you’ve just downloaded the release, navigate to the directory and at the command line use ./vault instead. In these examples, $ indicates terminal input, all other lines are output.

Dev Server

We’ll be using Vault’s dev server locally. To start it, run vault server -dev:

$ vault server -dev

==> WARNING: Dev mode is enabled!

In this mode, Vault is completely in-memory and unsealed. Vault is configured to only have a single unseal key. The root token has already been authenticated with the CLI, so you can immediately begin using the Vault CLI.

The only step you need to take is to set the following environment variables:

export VAULT_ADDR=’http://127.0.0.1:8200′
The unseal key and root token are reproduced below in case you want to seal/unseal the Vault or play with authentication.

Unseal Key: 6968dc217d5d7e9af8cba3abb1ca9547a57b68482301306018b1b9b50f640c07 Root Token: 4f02a7b0-93ff-fe5e-dc20-9aa26635cd5e

==> Vault server configuration:

   Log Level: info
        Mlock: supported: false, enabled: false
      Backend: inmem
   Listener 1: tcp (addr: “127.0.0.1:8200”, tls: “disabled”)
      Version: Vault v0.5.2
==> Vault server started! Log data will stream in below:

2016/04/25 10:32:02 [INFO] core: security barrier initialized (shares: 1, threshold 1)

2016/04/25 10:32:02 [INFO] core: post-unseal setup starting

2016/04/25 10:32:02 [INFO] core: mounted backend of type generic at secret/

2016/04/25 10:32:02 [INFO] core: mounted backend of type cubbyhole at cubbyhole/

2016/04/25 10:32:02 [INFO] core: mounted backend of type system at sys/

2016/04/25 10:32:02 [INFO] rollback: starting rollback manager

2016/04/25 10:32:02 [INFO] core: post-unseal setup complete

2016/04/25 10:32:02 [INFO] core: root token generated

2016/04/25 10:32:02 [INFO] core: pre-seal teardown starting

2016/04/25 10:32:02 [INFO] rollback: stopping rollback manager

2016/04/25 10:32:02 [INFO] core: pre-seal teardown complete

2016/04/25 10:32:02 [INFO] core: vault is unsealed

2016/04/25 10:32:02 [INFO] core: post-unseal setup starting

2016/04/25 10:32:02 [INFO] core: mounted backend of type generic at secret/

2016/04/25 10:32:02 [INFO] core: mounted backend of type cubbyhole at cubbyhole/

2016/04/25 10:32:02 [INFO] core: mounted backend of type system at sys/

2016/04/25 10:32:02 [INFO] rollback: starting rollback manager

2016/04/25 10:32:02 [INFO] core: post-unseal setup complete

$ export VAULT_ADDR=’http://127.0.0.1:8200′

Vault shows some basic instructions for getting connected. Write down the unseal key and root token, we’ll need them later. Note that this method of running the server is inappropriate for putting into production since the data is only held locally and in-memory. This dev server also does not encrypt communication between the server and the client (which is why we explictly have to set the protocol to http in VAULT_ADDR.)

Setting and Retrieving a Secret

Once we have the VAULT_ADDR environment variable in place, we can read and write secrets. These secrets operate quite like a K/V store. The key is a slash-separated path which you can use as an identifier. The big difference here is that the value is a mapping of keys and values itself. Let’s demonstrate by writing a key named test/cisco under the default secret namespace used for generic secret storage:

$ vault write secret/test/cisco hello=world

Success! Data written to: secret/test/cisco

That data has been written to Vault, and we can get it out again with the inverse operation, vault read:

$ vault read secret/test/cisco

Key            Value

lease_duration 2592000

Hello          world

The result is a little table (by default for human consumption, but you can also ask for JSON or YAML with -format=.) We’ll come back to the lease duration, but see that our key and value is set exactly as we formatted it on the command line. Since this is a map of data, we can set more keys in the data as well:

$ vault write secret/test/cisco hello=world vault=works

Success! Data written to: secret/test/cisco

$ vault read secret/test/cisco

Key            Value

lease_duration 2592000

Hello          world

vault          works

Sealing and Unsealing

Now that we have some data to work with, let’s demonstrate sealing Vault’s secret store. Vault has two states: sealed and unsealed. When Vault is sealed, no secret material can go in or out, and the master key is revoked. The dev server starts off unsealed, but in production Vault will start sealed. Unsealing Vault requires a quorum of keys (so-called “unseal keys”), which you can distribute around your organization however you see fit.

Since we’re using the dev server, the vault is open right now. Let’s seal it to demonstrate:

$ vault seal

Vault is now sealed.

This is your panic button. If there is a detected intrusion, you can seal Vault right away to limit the damage. Of course, it will render secret backends temporarily inoperable but desperate times call for desperate measures! To demonstrate this, try reading our key from earlier:

$ vault read secret/test/cisco

Error reading secret/test/cisco: Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/test/cisco

Code: 503.

Errors:

* Vault is sealed

Fortunately we can recover from this with the vault unseal command using the unseal key from earlier:

$ vault unseal 6968dc217d5d7e9af8cba3abb1ca9547a57b68482301306018b1b9b50f640c07

Sealed: false

Key Shares: 1

Key Threshold: 1

Unseal Progress: 0

In this case, we can recover with a single unseal key. In production, you will want to have multiple key shards distributed among your trusted operators, a quorum of which will be required to unseal.

Dynamic Secret Backends

Now that we’ve unsealed, let’s put a cherry on top by setting up a dynamic secret backend. We’ll use AWS as an easy example. Using the AWS backend, you can issue users with arbitrary IAM permissions whose lifecycle will be fully managed by Vault. To get started, we need to mount the AWS backend:

$ vault mount aws

Successfully mounted ‘aws’ at ‘aws’!

Next, we’ll need to create an IAM user with access to create and modify users. You can limit this however you feel comfortable. After retrieving the credentials from the AWS management console, we’ll feed them to Vault:

$ vault write aws/config/root \

   access_key=… \

   secret_key=… \

   region=us-east-1

Success! Data written to: aws/config/root

The last step before we can issue new users is setting a policy. You can write your own policies, of course, or you can use a predefined AWS policy. We’ll do that here to give our users read-only access to EC2.

$ vault write aws/roles/readonly arn=arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess

Success! Data written to: aws/roles/readonly

Now to get a new set of credentials, we just need to read them out. Vault will generate new users with appropriate permissions on the fly.

$ vault read aws/creds/readonly

Key             Value

lease_id        aws/creds/readonly/2839a94d-e43c-fd78-6df3-54d463f3e651 lease_duration  2592000

lease_renewable true

access_key      AKIAJAOZWGRYNCIZQTYQ

secret_key      h7fTJRpx5v6tzdW02IV5ruM3BCu3Uh7mIPWs64+T

security_token  <nil>

Success! If you check your AWS console you will now be able to see a new user with the AmazonEC2ReadOnlyAccess policy attached. Note the lease_id and lease_duration parameters above. These are used for key rotation. The lease can be renewed; in this case, for 30 days. However, the default lease duration can be changed, along with the maximum key lifetime.

Of course, these credentials can be revoked at any time with vault revoke and the lease ID:

$ vault revoke aws/creds/readonly/2839a94d-e43c-fd78-6df3-54d463f3e651

Key revoked with ID ‘aws/creds/readonly/2839a94d-e43c-fd78-6df3-54d463f3e651’.

And the user is gone.

Now that we’ve seen how and why to use Vault, how are you going to get it into production? You could try to figure it all out yourself, but it (along with HA mode and some service integration) are built into Cisco’s Mantl. You could be up and running in the cloud of your choice and building applications with Vault within the hour. Give Mantl a try and let us know your thoughts about security concerns in a microservices infrastructure.  Stay tuned for additional enhancements Cisco’s making in microservices infrastructure.