Helm secrets - the missing part of Kubernetes

In this article, I will talk about how we manage secrets in BaseCRM's Kubernetes infrastructure.

Our goal is to use the Helm Charts in BaseCRM Kubernetes clusters with minimal effort, involving only the management of parameter values ​​and secrets. There is no official secret management tool in Helm or Kubernetes, and we decided to fix it.

Introduction

Kubernetes is a platform developed by Google with substantial community support to automate the deployment and scaling of containerized applications in a cluster. It is a portable, reliable and self-healing system designed to build next-generation PaaS infrastructures.

On top of Kubernetes we use Helm - the official Kubernetes package manager. With a Charts system with template support, it helps to manage various types of applications running in Kubernetes.

Helm Charts is needed to define, install, and update reusable application templates in a Kubernetes cluster, including the most complex, with multiple dependent subcharts. With Helm Charts, you can easily manage multiple versions of charts, as well as stop them using direct manifests, which are controlled only by copy-paste in Kubernetes stock, which is completely unacceptable for complex environments.

With the help of Helm, we can use parts of chart templates with application charts of any complexity that are reusable for deploying Kubernetes PaaS in several clusters. With CI / CD, we can easily deploy the stack of applications in the Kubernetes cluster, or simply launch Helm manually and close the whole world from scratch.

What problem are we trying to solve?

To ensure the security of the infrastructure, passwords and access tokens are needed. At the same time, the infrastructure should still be easily accessible and distributed according to clear rules. Also, logging of changes, versioning of secrets, logging, and maybe even warnings about changes in the values ​​of secrets will not interfere. At the same time, we want to be able to re-create Kubernetes clusters, while there will be credentials that do not need to be changed.

Some keep secrets in Git for VPN, while getting problems with multiple copies, unencrypted disks and rotation of secrets. Others register it directly in the cluster and lose this data along with the cluster when something happens to it.

On the other hand, it is possible to integrate Kubernetes with the Hashicorp Vault service, which is a repository of secrets. In this case, the cluster, when receiving secrets, Kubernetes will refer to this service.

We want to deploy applications to Kubernetes clusters without using the Vault service, which for security reasons should not be part of the Kubernetes cluster. We need the ability to work locally and do idle runs without having to connect to the Vault infrastructure.

How do we do it?

Since the foundation of Base, we have been using Ansible. We also decided that we would use Helm to deploy Kubernetes clusters (this topic most likely deserves a separate article). Faced with the problem of managing secrets, we came up with the helm-secrets plugin, which was a lot like the Ansible vault. At that time, we used PGP and encrypted the entire secrets file.

It was a simple decision, although from the very beginning it was clear that it was not final. Soon we decided to move to managing each YAML secret separately.

First, we made a list of requirements and conducted a search for tools that would satisfy them. We managed to find Mozilla SOPS , which in its capabilities was suitable as a backend of the secrets management system for our plugin.

All that was left to do was to write a simple layer between the SOPS backend and the Helm-frontend using the first version of helm-secrets.

This is what we did, in the process deciding to share the results of our work with the public. Meet the release of the helm-secrets plugin.

Here is a list of its capabilities (there are even more of them than listed here):

• Simple level of integration and installation with the helm plugin.
• Encryption support for Helm YAML structures.
• Encryption of individual values ​​- this allows diff to be used even with encrypted files.
• For git diff, decryption is provided on the fly.
• For the helm install / upgrade / rollback commands, the helm-wrapper command provides on-the-fly decryption and cleaning.
• Simultaneous use of multiple key management solution secrets such as PGP and AWS KMS is supported.
• For encrypted files, simple addition / removal of keys is supported.
• Work is supported with the permission management system for AWS KMS keys without the need for re-encryption.
• Splitting a directory tree to store secrets files using recursive search in .sops.yaml files.
• Extract individual items from an encrypted file structure.
• Encryption of part of the file.

The system is able to work within the CI / CD, serving several development teams with isolated access to certain subdirectories of the Git repository, which contains secrets in the context of projects / environments / regions for certain Kubernetes clusters. We encrypt secrets using different KMS keys and a PGP master key as a backup.

To automate and speed up the execution of tasks in our internal chart repository, a Makefile is used.

This is how it works.

Helm Installation

Follow the instructions on the helm project page .

Installing the helm-secrets plugin

helm plugin install https://github.com/futuresimple/helm-secrets 

Usage example

The helm-secrets repository provides an example of working with this plugin, as well as the corresponding directory structure for storing the values ​​of the helm charts. For kubernetes charts, a similar structure applies.

For this example, the test.sh file from the helm-secrets repository is used.

 example/helm_vars/ ├── .sops.yaml ├── projectX │ ├── .sops.yaml │ ├── production │ │ └── us-east-1 │ │ └── java-app │ │ ├── secrets.yaml │ │ └── value.yaml │ └── sandbox │ └── us-east-1 │ └── java-app │ ├── secrets.yaml │ └── value.yaml ├── projectY │ ├── .sops.yaml │ ├── production │ │ └── us-east-1 │ │ └── java-app │ │ ├── secrets.yaml │ │ └── value.yaml │ └── sandbox │ └── us-east-1 │ └── java-app │ ├── secrets.yaml │ └── value.yaml ├── secrets.yaml └── values.yaml 

Let's take this example.

• We have two PGP keys (you can also use KMS keys):
  
  • one for projectx;
  • another for projecty.
• All other secrets are managed within the project, and one key is used for each project.
• The keys are isolated from each other, and only the global secrets.yaml file located in the helm_vars root can be decrypted by any of these keys.
• All recursion depth rules for encryption / decryption are in the .sops.yaml files.

Encryption, decryption and even more.

Before encryption example / helm_vars / secrets.yaml

 global_secret: global_bar 

Now encrypt:

 helm-wrapper secrets enc example/helm_vars/secrets.yaml 

As a result, we get a file with unencrypted keys and encrypted values.

After the key section, there is the sops-data needed to encrypt or decrypt the structures of secrets.

There is also metadata of sops version, lastmodified or unencrypted_suffix type. You can read about the format of the sops file on the Mozilla SOPS project website .

 global_secret: ENC[AES256_GCM,data:pTyPdC6YA+z84Q==,iv:aF5hb9CS8Au0B3RWADPtP8fXYzYakU7JJ8ZxzJgHRF0=,tag:c3pCyOf0NpQU7VPL/72XPg==,type:str] sops: .… …. …. unencrypted_suffix: _unencrypted version: 2.0.9 

Now decryption:

 helm-wrapper secrets dec example/helm_vars/secrets.yaml 

And the result:

 global_secret: global_bar 

Now we can work with such files without decryption, for example, when searching for keys in several secrets files. We can even decrypt them on the fly with git diff config, which makes the work even more comfortable, especially compared to applications that do not support this functionality.

We can also manage the secrets permissions more flexibly without recreating them thanks to the more advanced AWS KMS.

Moreover, viewing and editing secrets with simple plugin commands makes daily work easier.

 helm-wrapper secrets view example/helm_vars/secrets.yaml 

After decryption, the secrets file will be output to stdout:

 helm-wrapper secrets edit example/helm_vars/secrets.yaml 

edit command, the decrypted file will open in the editor, and the changes will be automatically saved in encrypted form.

These functions make it much easier to work with encrypted data in almost any scenario.

Helm Usage Example

Let's deploy something in a Kubernetes cluster.

To simplify the process and make it more transparent, we created a helm-wrapper. This bash binding for helm automatically decrypts all the specified secrets and uses the decrypted data to perform the deployment procedure using helm. If an error occurs or the procedure is completed successfully, all temporary data with secrets is deleted.

A real example for an application like helloworld:

 AWS_PROFILE=production helm-secrets upgrade --install --timeout 600 --wait helloworld stable/java-app --kube-context=production --namespace=projectx --set global.app_version=bff8fc4 -f helm_vars/projectx/sandbox/us-east-1/java-app/helloworld/secrets.yaml -f helm_vars/projectx/sandbox/us-east-1/java-app/helloworld/values.yaml -f helm_vars/secrets.yaml -f helm_vars/values.yaml >>>>>> Decrypt Decrypting helm_vars/projectx/sandbox/us-east-1/java-app/helloworld/secrets.yaml >>>>>> Decrypt Decrypting helm_vars/secrets.yaml Release "helloworld" has been upgraded. Happy Helming! LAST DEPLOYED: Fri May 5 13:27:01 2017 NAMESPACE: projectx STATUS: DEPLOYED RESOURCES: ==> extensions/v1beta1/Deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE helloworld 3 3 3 2 1h ==> v1/Secret NAME TYPE DATA AGE helloworld Opaque 10 1h ==> v1/ConfigMap NAME DATA AGE helloworld 2 1h ==> v1/Service NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE helloworld 100.65.221.245 8080/TCP 1h NOTES: Deploy success helloworld-bff8fc4 in namespace projectx >>>>>> Cleanup helm_vars/projectx/sandbox/us-east-1/java-app/helloworld/secrets.yaml.dec helm_vars/secrets.yaml.dec 

As you can see from the listing, there is a special command that runs on the CI side to deploy the application. This example uses our internal universal chart for java applications, which contains a template for the config map, secrets, services, and deployment.

All this is generated from the values ​​in the files specified with the -f option of the helm command. If an encrypted secret is found in such a file, the script will decrypt it on the fly, and helm will not notice anything.

When using AWS KMS, you must specify the name AWS_PROFILE.

If an error occurs, it will be cleaned.

 AWS_PROFILE=production helm-wrapper upgrade --install --timeout 600 --wait helloworld stable/java-app --kube-context=wrongcontext --namespace=projectx --set global.app_version=bff8fc4 -f helm_vars/projectx/sandbox/us-east-1/java-app/helloworld/secrets.yaml -f helm_vars/projectx/sandbox/us-east-1/java-app/helloworld/values.yaml -f helm_vars/secrets.yaml -f helm_vars/values.yaml >>>>>> Decrypt Decrypting helm_vars/projectx/sandbox/us-east-1/java-app/helloworld/secrets.yaml >>>>>> Decrypt Decrypting helm_vars/secrets.yaml Error: could not get kubernetes config for context 'wrongcontext': context "wrongcontext" does not exist >>>>>> Cleanup helm_vars/projectx/sandbox/us-east-1/java-app/helloworld/secrets.yaml.dec helm_vars/secrets.yaml.dec 

This approach to using secrets in helm minimizes effort, while maintaining the maximum level of security.

Safe use with git

With decrypted secrets (decrypted manually), we still have two levels of security: excluding decrypted files using .gitignore and adding hooks that check whether commit files are encrypted using SOPS.

All this is also in the helm-secret documentation and verified by our CI / CD process.

Conclusion

Helm-secrets is only a thin wrapper for the sops backend, and any command of this plugin can be replaced with other tools.

We also use this plugin to distribute secrets through a public git-repository and plan to add to the process of working with secrets integrated into Kubernetes service Vaulting, which will give our solution completeness.

For a small infrastructure, the helm-secrets plugin should be sufficient. We hope you enjoy using it.

If you want to add some functions to the project or just discuss it, go to https://github.com/futuresimple/helm-secrets .

References:

1. Original: HELM SECRETS - A MISSING PIECE IN KUBERNETES .