How not to keep secrets where we have to, or why we need Hashicorp Vault

Ask yourself the question - how to store the password from the database used by your service? In a separate repository with secrets? In the application repository? In deployment system (Jenkins, Teamcity, etc)? In the configuration management system? Only on a personal computer? Only on the servers your service is running on? In a certain repository of secrets?
Why think about it? To minimize the security risks of your infrastructure.
We begin the study of the issue with the definition of requirements for the storage of secrets.

Requirements for the storage of infrastructure secrets:

• Tweaking secrets access rules. We give access only to those secrets that are necessary and sufficient to perform the work.
• Access Lifecycle Management. The ability to issue, revoke, set a lifetime and reissue or extend access.
• Audit access to secrets. Each access deed is recorded and will be verified by a third party.
• The minimum perimeter of the attack. The less "smearing" on the system, the better.
• Fault tolerance. No single point of failure.
• Comfortable working with secrets for people and automated systems. Minimum time to learn and implement new tools.

Let's take an example of our requirements for possible solutions:

• Repository storage (any):
  
  • There is no flexible configuration of access to secrets. Access is binary - either there is or not. There are solutions to this problem: use gpg for files. For configuration management systems use Ansible vault, Puppet gpg-hiera, Chef encrypted data bags and so on. Problems begin where automation occurs. An attacker who gains access to the server where the decryption keys are stored (the master server for Puppet \ Chef, the host for automatic launch of Ansible playbooks, the host for automatic launch of tasks that need encrypted data from the repository) gains access to secrets.
  • No support for full access life cycle. You cannot give temporary access. You cannot revoke access to secrets (you can only change access to the repository and the secrets themselves in it). The lack of a regular access update process reduces transparency.
  • No audit.
  • Unpredictable "blurring" of the system. Each appearance of this repository in the infrastructure adds points of attack to you. Whether it’s Jenkins, different chef \ puppet servers for prod / dev / test / uat environments, or the need to run a small script from one server that is stored in this repository - you get risks in exchange for 'convenience' and the ability not to think about security.
• Storage in the Deploy system:
  
  • There is no flexible configuration of access to secrets. A person with access to the deployment system can use secrets in his task - and thus gain access to them. If you use Jenkins, then secrets can be pulled out at the server level .
  • No support for full access life cycle.
• Manual insertion of secrets to the server:
  
  • No support for full access life cycle.
  • No audit. You do not know who and when got access to the secrets.
  • No fault tolerance. If the secret is stored on only one server - the loss of the server threatens to lose the secret.
  • No convenience of work. But this method can be used when you have a few servers and services. With the growth of infrastructure you are waiting for the increased cost of maintenance of the process of managing secrets.

Why Vault?

Vault is a repository of secrets from the recognized "solvers" of problems of modern infrastructure - Hashicorp , the authors of Vagrant, Consul, Terraform, Otto, etc. Secrets are stored in key-value form. Access to the repository is carried out exclusively through the API.
Main features of the Vault:

• All data is stored in an encrypted container. Receiving the container itself does not reveal the data.
• Flexible access policies. You can create as many tokens to access and manage secrets as you need. And to give them the permissions that are necessary and sufficient to perform the work.
• The possibility of listening access to secrets. Each request to the Vault will be logged for subsequent auditing.
• Supports automatic generation of secrets for several popular databases (postgresql, mysql, mssql, cassandra), for rabbitmq, ssh and for aws.
• Support for encrypting and decrypting data without saving it. This may be convenient for transmitting data in encrypted form over unsecured communication channels.
• Full secret life cycle support: creation / revocation / expiration / renewal.
• Uberfitch, the importance of which is difficult to overestimate, is the ability to create your own CA (Certificate Authority) to manage self-signed certificates within your infrastructure.
• Backend / cubbyhole, which allows you to create your own repository of secrets, not even available to other root-tokens.
• Ready modules and plugins for popular configuration management systems.

For us, Vault solves the problem of transferring secrets over unprotected channels, the problem of fault-tolerant storage of secrets, as well as problems of flexible separation and access auditing. Plans to use Vault as your own CA.

Beginning of work

I will not translate official documentation, so here are some links:

• Getting started is available here .
• There is an interactive tutorial.
• Here is another great article.

So, you launched Vault. First of all, we put our key in the repository. The official work process is:

1. For each service we create a separate master token. This token can manage secrets in the container / secret / service_name / and can create other tokens.
2. Switch to this token and create a secret.
3. We release a new token to read this secret.
4. We feed this token to our automation system
5. ...
6. PROFIT !!!
   Further nuances begin. I will analyze them in more detail.

About master token for each service

It turned out to be inconvenient to constantly switch tokens if you are responsible for several services.
Due to the peculiarities of managing access policies (more detailed below), the overhead costs for the initial implementation of Vault in the service workflow have increased.

Pro features of managing access policies

They are described in JSON or HCL format and are recorded in Vault using a request to the API or through cli. For example:

$ vault policy-write policy_name policy_file.hcl 

In order for the master token you created to create new tokens, it must have a policy:

 path "auth/token/create" { policy = "write" } 

A master token can create tokens only with those policies it possesses.
This means that it is not enough for you to issue the service_name_prod_root policy to the token:

 path "secret/service_name/prod/*" { policy = "write" } 

which has full access to secret / service_name / prod / *, but you also need to attend to it, by the service_name_prod_read policy:

 path "secret/service_name/prod/*" { policy = "read" } 

and then you can create tokens for read-only access with the command:

 $ vault token-create -policy=service_name_prod_read 

And there is a nuance: the policies are applied according to the degree of detail. This means that if you make a root policy like this:

 path "secret/service_name/prod/*" { policy = "write" } 

and want to give read access to the service_name_prod_database_read policy:

 path "secret/service_name/prod/database/*" { policy = "read" } 

then you will need to assign the token that manages this service to both of these policies. When you do this, you cannot write to secret / service_name / prod / database / *:

 $ vault write secret/service_name/prod/database/replicas \ secret_key=sha1blabla Error writing data to secret/service_name/prod/database/replicas: Error making API request. URL: PUT https://vault.service.consul:8200/v1/secret/service_name/prod/database/replicas Code: 400. Errors: * permission denied 

You will have to balance each detailed reading policy to prevent this from happening. For example, modify the service_name_prod_root policy:

 path "secret/service_name/prod/*" { policy = "write" } path "secret/service_name/prod/database/*" { policy = "write" } 

This does not make the life of operators easier, so we abandoned this approach and work with secrets only from under root keys.

About accounting and updating tokens

The Vault documentation does not say in vain about the importance of accounting for released tokens.
There is no easy way to know which keys are present in the system. If you created a token and forgot about it, without recording any information, then you will have to wait until it expires.
Root tokens have no lifetime, so they will remain on your system indefinitely. To avoid this situation, to create a new token for testing and verification purposes, specify the excellent -ttl = "1h" flag, which sets the lifetime.
This will allow you to work quietly and not be afraid to increase the number of tokens without control.
There is also a risk not to keep track of the expiration of the token's life and learn about it, for example, during deployment.
This problem can be solved by recording tokens and monitoring the lifetime. But writing tokens somewhere in one place is not safe. Therefore, since Vault 0.5.2, every token after creation returns the accessor parameter, knowing which, you can interact with the token without knowing it (withdraw, update, add policies), for example

 $ vault token-revoke --accessor b30ee2a3-ea4b-9da0-3e5c-4189d375cad9 

Details here .
Maintaining data on issued tokens allows you to quickly audit active accesses in the system and configure monitoring for the token to expire.

About token settings

 $ vault token-create -policy=service_name_prod_root -policy=service_name_prod_read Key Value --- ----- token 82c5fb97-da1b-1d2c-cfd5-23fa1dca7c85 token_accessor dd256e17-b9d9-172d-981b-a70422e12cb8 token_duration 2592000 token_renewable true token_policies [default, service_name_prod_root, service_name_prod_read ] 

Go through the returned parameters:

 token -      Vault. token_accessor - ,       ,    .   :    ,  . token_duration -     . token_renewable -  true,       ,            max-lease-ttl,     30d.  ,          30       30 ,      . token_policies - ,   .    ,       . 

Non-obviousness and usefulness

If the token was the parent for other tokens, then by default, when this token is revoked, all tokens and secrets created by the revoked token are recursively revoked. This can be avoided by specifying the -mode flag. In details

 vault token-revoke -h 

If you use the consul-template for automatic regeneration of configs when changing secrets, keep in mind (and you will not find this in the documentation) that the consul-template polls the change of secret in two cases:

• Start or restart of the consul-template itself.
• Every (TimeToLive secret / 2) seconds after the start of the consul-template.

In order not to enter your working token all the time, you can put it in $ HOME / .vault-token or in the environment variable VAULT_TOKEN. Here it is necessary to make a reservation, that by default I consider the admin workstation protected (encrypted disks, no guest input, auto-blocking after a minute of inactivity). If this is not the case, then it is worth abandoning this idea.

Our work process:

We didn’t have the official process of working with Vault because of the excessive complexity of implementing Vault into service operation. I will share our solutions and answers to your questions during the implementation of Vault.

• Fault tolerance:
  We use Consul in infrastructure, so we chose it as a fault tolerant backend for Vault.
• Number of keys to print the repository:
  Printing is an exclusively manual process, during which an encrypted container with secrets is placed in memory and the master key for decryption is restored. Details here . Making it possible for one person to print the repository is a risk, because if this key or keys are compromised, attackers will have access to the repository (but not yet the secrets). It should be understood that if you do not type the necessary minimum keys for printing the storage, access to the data will be lost. We decided that 4 key holders and 2 keys are enough for us to decrypt.
• Root tokens and their role in the system:
  The official position of Hashicorp is to use these tokens only to manage policies, roles, system settings, and to issue tokens for services. The smaller the root keys in the system, the less likely the keys are compromised. But the fewer such keys in the system, the more everything is tied to certain people who are sick, go on vacation, in general - are not available. We started with 4 root keys for a group of 7 people. This was inconvenient, so we departed from the official process for the first time - we issued root tokens to all operating engineers and stopped creating master tokens to manage service secrets. All operations are performed under root tokens. Only read-only tokens are created.
• Secrets storage scheme:
  For us, I chose the following scheme - keep the secrets of services in / secret / service_name / env. And common infrastructure keys (for example, api-token for jenkins or credentials for accessing the package repository) are stored in / secret / infra / *.

• Naming secrets:

  
  

   $ vault write secret/service_name/prod/database \ base=appname \ login=appname \ password=difficult_password 

  
  

  or for example

  
  

   $ vault write secret/service_name/prod/database/base value=appname $ vault write secret/service_name/prod/database/login value=appname $ vault write secret/service_name/prod/database/password value=difficult_password 

  
  

  Resolve this issue before your list of secrets grows quickly and you begin to fasten various automation tools. We use the first scheme.

  
  
• Keys accounting and monitoring their lifetime:
  By default, the maximum lifetime of a non-root token is 30 days. We keep track of accessor keys for tokens using the repository - and this is inconvenient. But allowed to set up monitoring, which reports the expiration of the keys for 5 days.
  There are plans to start a small service for registering keys and write a small wrapper for automating some actions with tokens (for example, automatic recording of a newly created token to the service).

Let's return to keeping the secret from the database.

We installed and launched Vault, got a root token. What's next?
According to the latest version of the process adopted in my company, it will look like this:
Install the vault .
Specify the location of the Vault:

 $ export VAULT_ADDR='https://vault.service.consul:8200' 

Log in as root token:

 $ vault auth 82c5fb97-da1b-1d2c-cfd5-23fa1dca7c85 

Let's write our secret:

 $ vault write secret/service_name/prod/database base=appname login=appname password=difficult_password 

We write the policy to read in the service_name_prod_read.hcl file:

 path "secret/service_name/prod/database*" { policy = "read" } 

Create a policy in Vault:

 $ vault policy-write service_name_prod_read service_prod_read.hcl 

Generate a token to read:

 $ vault token-create -policy=service_name_prod_read Key Value --- ----- token cb347ae0-9eb4-85d1-c556-df43e82be4b0 token_accessor c8996492-17e3-16a7-2af1-d58598ae10d8 token_duration 2592000 token_renewable true token_policies [default, service_name_prod_read ] 

Let's write token_accessor for the subsequent audit and monitoring.
Check that there is read access:

 $ vault auth cb347ae0-9eb4-85d1-c556-df43e82be4b0 $ vault read secret/service_name/prod/database Key Value lease_duration 2592000 base appname login appname password difficult_password 

Everything works, we are ready to use our token in automation systems. I will give examples for popular systems.

Just curl:

 $ curl \ -H "X-Vault-Token:cb347ae0-9eb4-85d1-c556-df43e82be4b0" \ https://vault.service.consul:8200/v1/secret/service_name/prod/database 

Ansible (we use https://github.com/jhaals/ansible-vault ):

Customize the environment:

 $ export VAULT_ADDR=https://vault.service.consul:8200 $ export VAULT_TOKEN=cb347ae0-9eb4-85d1-c556-df43e82be4b0 

Use variables from Vault in the role:

 database_password: "{{ lookup('vault', 'secret/service_name/prod/database', 'password') }}" 

We make requests to vault only at the level group_vars / group_name. This is convenient and allows you not to look for variables by role.

Chef:

Hashicorp in his blog described several ways to use secrets from Vault in their chef-kukbukah - https://www.hashicorp.com/blog/using-hashicorp-vault-with-chef.html

Puppet:

There is an excellent module for puppet https://github.com/jsok/hiera-vault , from the documentation for which the process of using secrets from Vault is clear.

Consul-template:

You must either have a token and Vault address in environment variables:

 $ export VAULT_ADDR=https://vault.service.consul:8200 $ export VAULT_TOKEN=cb347ae0-9eb4-85d1-c556-df43e82be4b0 

or add lines to config:

 vault { address = "https://vault.service.consul:8200" token = "cb347ae0-9eb4-85d1-c556-df43e82be4b0" renew = true } 

and use the secrets in your templates:

 {{with secret "secret/service_name/prod/database"}}{{.Data.password}}{{end}} 

See readme for more details.

Thanks for your time, I hope this was helpful.
Ready to answer your questions.