Backup and Restore Kubernetes resources with Heptio Ark utility

You probably had to restore the Kubernetes cluster after a crash. Did you have a sensible backup strategy that does not require plowing for several days? Yes, you can make backups to the etcd-cluster, but what if only part of the cluster has fallen off or are you using persistent volumes, like AWS EBS?

In such cases, the easiest way is to use the Heptio Ark utility.

With Heptio, you can make backup copies of the entire cluster, individual namespaces or types of resources and make backups on a schedule. For me, the main advantage of Heptio Ark is integration with different cloud service providers, such as AWS, Azure, Google Cloud, etc. So when backing up, it takes snapshots of used persistent volumes.

Let's see how to install this utility and how it makes simple and planned backups, and then restores them.

About backup permanent volumes will be a separate post.

Installation

Installation instructions can be found here: examples / README.md. The process will create several custom resource definitions, RBAC (role-based access control) rules allowing Heptio to backup, and deployment. By default, they are in the heptio-ark namespace.

Important! After a successful installation, you need to configure heptio-ark to tell the server which cloud provider to use and where to store the backups. This is what this configuration looks like:

apiVersion: ark.heptio.com/v1 kind: Config metadata: namespace: heptio-ark name: default backupStorageProvider: name: aws bucket: heptio-backup-bucket config: region: eu-central-1 backupSyncPeriod: 30m gcSyncPeriod: 30m scheduleSyncPeriod: 1m restoreOnlyMode: false 

You can apply it with the command

  kubectl apply -f heptio.yaml 

Now Heptio knows which batch to back up. The backup storage location should be accessible from the heptio-server pods, so you can use an instance profile with access to this batch or Kube2IAM for dynamic instance profiles based on the hearth.

Finally, for backups, scheduling and recovery, you need to download Heptio Ark CLI from the GitHub .

Almost all commands can be executed as custom resource definitions via YAML or JSON.

Backup

In this small example, I created a simple NGINX deployment, and before it a service in the webserver namespace:

 $ kubectl get all NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deploy/nginx 1 1 1 1 28s NAME DESIRED CURRENT READY AGE rs/nginx-66f5756f9b 1 1 1 28s NAME READY STATUS RESTARTS AGE po/nginx-66f5756f9b-c88ck 1/1 Running 0 28s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE svc/nginx ClusterIP 10.32.0.183 <none> 80/TCP 28s 

Let's make a backup from Heptio Ark CLI:

 $ ark backup create nginx-simple --include-namespaces webserver 

This command backs up the webserver namespace only. Without this parameter, Heptio Ark will create a full backup of all resources in the Kubernetes cluster. Backup will take some time. The copy will be saved to the specified bakt in S3 ( heptio-backup-bucket ). To view the status and list of all backups, in the CLI, enter the following command:

 $ ark backup get NAME STATUS CREATED EXPIRES SELECTOR nginx-simple Completed 2018-07-08 17:35:09 +0200 CEST 29d <none> 

As you can see, the backup is made.

Restore backups

Let's remove the webserver inline namespace:

 $ kubectl delete ns heptio-test 

And now we will restore the namespace after “accidental” deletion, and again from Heptio Ark CLI:

 $ ark restore create --from-backup nginx-simple Restore request "nginx-simple-20180708173924" submitted successfully. Run `ark restore describe nginx-simple-20180708173924` for more details. 

You should see that the namespace and all resources (deployment, replica set, sub, and service) are restored:

 $ kubectl get all NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deploy/nginx 1 1 1 1 20s NAME DESIRED CURRENT READY AGE rs/nginx-66f5756f9b 1 1 1 20s NAME READY STATUS RESTARTS AGE po/nginx-66f5756f9b-9mjvg 1/1 Running 0 20s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE svc/nginx ClusterIP 10.32.0.77 <none> 80/TCP 20s 

Backup structure

To view the backup structure, simply upload it from the bake to S3 or enter the Heptio Ark command:

 $ ark backup download nginx-simple Backup nginx-simple has been successfully downloaded to $PWD/nginx-simple-data.tar.gz 

In the webserver.json file of our namespace, we see the usual namespace resource.

 { "apiVersion":"v1", "kind":"Namespace", "metadata": { "annotations": { "kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"name\":\"webserver\",\"namespace\":\"\"}}\n" }, "creationTimestamp":"2018-07-08T15:26:44Z", "name":"webserver", "resourceVersion":"3364", "selfLink":"/api/v1/namespaces/webserver", "uid":"52698ae7-82c3-11e8-8529-0645eb60c3f4" }, "spec": { "finalizers":["kubernetes"] }, "status": { "phase":"Active" } } 

If we do not need a full recovery, we can restore only a part using the Heptio Ark command or go to backup directly and restore this part via kubectl.

 $ ark schedule create nginx-schedule --schedule="* 10 * * *" --include-namespaces webserver Schedule "nginx-schedule" created successfully. 

Run backup on a schedule

Heptio Ark can perform scheduled tasks. We indicate which resources and namespaces should be included in the backup or excluded from it and when it is necessary to perform a backup:

 $ ark schedule create nginx-schedule --schedule="* 10 * * *" --include-namespaces webserver Schedule "nginx-schedule" created successfully. 

In this case, the backup will be created every day at 10 o'clock and include only the webserver namespace. In Heptio Ark CLI, we see that the schedule has appeared and Heptio Ark has already created the first backup:

 $ ark schedule get NAME STATUS CREATED SCHEDULE BACKUP TTL LAST BACKUP SELECTOR nginx-schedule Enabled 2018-07-08 17:49:00 +0200 CEST * 10 * * * 720h0m0s 25s ago <none> $ ~/Downloads/ark backup get NAME STATUS CREATED EXPIRES SELECTOR nginx-schedule-20180708154900 Completed 2018-07-08 17:49:00 +0200 CEST 29d <none> nginx-simple Completed 2018-07-08 17:35:09 +0200 CEST 29d <none> 

Here it is stated that the planned backups are deleted after 720 hours, that is, after 30 days. When you create a backup or schedule, you can specify the lifetime of the copy - TTL. After this period, the backup will be deleted from storage, in our case AWS.