We get acquainted with the alpha version of snapshots volumes in Kubernetes

Note trans. : The original article was recently published on the Kubernetes blog and written by Google and Huawei (Jing Xu, Xing Yang, Saad Ali), whose activities you will certainly see in the GitHub project if you have ever been interested in the K8s features and problems with data storage. Engineers talk about the purpose of volume snapshots, their current capabilities and the basics of working with them.

Kubernetes v1.12 presented an alpha version of snapshot support for volumes. This feature allows you to create and delete snapshots of volumes, as well as create new volumes from snapshots using the “native” means of the system - through the Kubernetes API.
')

What is snapshot?

Many storage systems (like Google Cloud Persistent Disks, Amazon Elastic Block Storage and numerous on-premise storage systems) offer the ability to create snapshots (snapshots) for a persistent volume. A snapshot is a copy of a volume at a specific point in time. It can be used to provision a new volume (already filled with data from snapshot) or restore an existing volume to the previous state (which is represented in snapshot).

Why add snapshots to Kubernetes?

A powerful abstraction is available in the plugin system of the Kubernetes volumes that automates the provisioning, mounting and mounting of block and file storages.

Ensuring all of these capabilities is part of Kubernetes workload portability goals: Kubernetes strives to create a level of abstraction between applications running as distributed systems and underlying clusters in such a way that applications do not depend on the specifics of the cluster on which they are running and does not require application deployment any cluster-specific knowledge.

The Kubernetes Storage SIG team identified the snapshot operations as critical opportunities for many stateful workloads. For example, a database administrator may want to snapshot his database before performing any operation with it.

With the standard way to invoke snapshot operations in Kubernetes API, Kubernetes users can work with them without the need for workarounds (and manually invoking storage-specific operations). Instead, users were given the opportunity to embed snapshot operations into their tools and policies with the calm understanding that everything will work with any Kubernetes clusters, regardless of the underlying storage.

In addition, these Kubernetes primitives function as basic building blocks, opening up possibilities for developing more advanced enterprise-level storage management features — for example, data protection, replication, and migration.

What volume plugins support snapshots in Kubernetes?

Kubernetes supports three types of volume plugins: in-tree, Flex, and CSI. See the Kubernetes Volume Plugin FAQ for details.

Snapshots are supported only for CSI drivers (they are not supported for in-tree or for Flex). To take advantage of this opportunity, make sure that the Kubernetes cluster has a CSI driver deployed that supports snapshots.

By the time of this blog post (October 9, 2018 - approx. Transl. ) , Snapshots are supported by the following CSI drivers:

Support for snapshots for other drivers is in development and should be available soon. More details about CSI and how to deploy CSI drivers are described in the “ Container Storage Interface (CSI) for Kubernetes Goes Beta ” publication (also see our translation of the note “ Understand the Container Storage Interface (in Kubernetes and beyond) ” - approx. Transl. ) .

Kubernetes Snapshot API

To manage snapshots in Kubernetes Volume Snapshots, three new API objects are presented in the same way as in the Kubernetes Persistent Volumes API:

VolumeSnapshot
- Created by the user Kubernetes to request to create a snapshot for the specified volume. Contains information about snapshot operation, such as timestamp snapshot removal and whether it is ready for use.
- Like the PersistentVolumeClaim object, creating and deleting this object represents the user's desire to create or delete a cluster resource (snapshot).
VolumeSnapshotContent
- Created by the CSI driver when the snapshot was successfully created. Contains information about snapshot including its ID.
- Like the PersistentVolume object, it represents the resource already serviced by the cluster (snapshot).
- Like the PersistentVolumeClaim and PersistentVolume objects, when the snapshot is created, the VolumeSnapshotContent object VolumeSnapshotContent bound to the VolumeSnapshot for which it was created (one-to-one mapping is used).
VolumeSnapshotClass
- Set by cluster administrators to describe which snapshots can be created. Includes information about the driver, secrets for accessing snapshots, etc.

It is important to note that - unlike the main Persistent Volume objects in Kubernetes, these snapshots objects are defined as CustomResourceDefinitions (CRDs) . The Kubernetes project is gradually moving away from the previously defined types of resources in the API Server, approaching the model in which the API Server does not depend on API objects. This approach allows you to reuse the API server in other projects (besides Kubernetes), and consumer'y (like the same Kubernetes) can set the types of resources they need as CRD.

CSI drivers supporting snapshots will automatically install the necessary CRDs. The end users of Kubernetes will only need to verify that the CSI driver supporting snapshots is deployed in a cluster.

In addition to these new objects, the already existing PersistentVolumeClaim a new DataSource field:

 type PersistentVolumeClaimSpec struct { AccessModes []PersistentVolumeAccessMode Selector *metav1.LabelSelector Resources ResourceRequirements VolumeName string StorageClassName *string VolumeMode *PersistentVolumeMode DataSource *TypedLocalObjectReference }

This field (in the status of alpha version) allows you to automatically fill it with data from an existing snapshot when creating a new volume.

Kubernetes snapshots requirements

Before using snapshots of volumes in Kubernetes, you must:

make sure that the CSI driver that implements snapshots is deployed and running on the cluster;
enable Kubernetes Volume Snapshotting via a new feature gate (disabled by default for the alpha version):
- set the following flag for the executable API Server file: --feature-gates=VolumeSnapshotDataSource=true

Before creating a snapshot, you must also determine the CSI driver used, which is done by creating a VolumeSnapshotClass object and specifying the CSI driver in the snapshotter field. In the following example with VolumeSnapshotClass this driver is com.example.csi-driver . Each snapshot provider requires at least one VolumeSnapshotClass object. It is also possible to define the default VolumeSnapshotClass for each CSI driver — this is done by setting the snapshot.storage.kubernetes.io/is-default-class: "true" annotation in the class definition:

 apiVersion: snapshot.storage.k8s.io/v1alpha1 kind: VolumeSnapshotClass metadata: name: default-snapclass annotations: snapshot.storage.kubernetes.io/is-default-class: "true" snapshotter: com.example.csi-driver apiVersion: snapshot.storage.k8s.io/v1alpha1 kind: VolumeSnapshotClass metadata: name: csi-snapclass snapshotter: com.example.csi-driver parameters: fakeSnapshotOption: foo csiSnapshotterSecretName: csi-secret csiSnapshotterSecretNamespace: csi-namespace

All required parameters must be set in accordance with the CSI driver documentation. In the example above, the fakeSnapshotOption: foo parameter and all the mentioned secrets will be passed to the CSI driver during the creation and deletion of the snapshot. The CSI external-snapshotter saves the csiSnapshotterSecretName and csiSnapshotterSecretNamespace parameter keys by default.

Finally, before creating a snapshot, you must create a volume through the CSI driver and fill it with the data you want to see there (see this publication for details on how to use CSI volumes).

Creating a new snapshot in Kubernetes

Once the VolumeSnapshotClass object VolumeSnapshotClass defined and there is a volume from which you want to take a snapshot, you can perform this operation by creating a VolumeSnapshot object.

The source for snapshot is determined by two parameters:

kind - PersistentVolumeClaim indicated here;
name is the actual name of the PVC object.

This implies that the namespace of the volume for which snapshot is created is determined by the namespace of the VolumeSnapshot object.

 apiVersion: snapshot.storage.k8s.io/v1alpha1 kind: VolumeSnapshot metadata: name: new-snapshot-demo namespace: demo-namespace spec: snapshotClassName: csi-snapclass source: name: mypvc kind: PersistentVolumeClaim

The VolumeSnapshot specification can VolumeSnapshot a VolumeSnapshotClass , which contains information about which CSI driver will be used to create the snapshot. As previously reported, after creating the VolumeSnapshot object VolumeSnapshot the fakeSnapshotOption: foo parameter and all the mentioned VolumeSnapshotClass secrets are passed to the CSI plugin com.example.csi-driver in the CreateSnapshot call.

In response to such a request, the CSI driver snapshot the volume and then automatically creates a VolumeSnapshotContent object that represents the new snapshot and binds this object to the VolumeSnapshot , making it ready for use. If the CSI driver cannot create snapshots and returns an error, the snapshot controller reports this error in the status of the VolumeSnapshot object and does not make new attempts (this behavior differs from other controllers in Kubernetes - it is implemented in order not to create snapshots in unpredictable times) .

If the snapshot class is not set, external-snapshotter will try to find the default class and use it for the snapshot being created. In this case, the CSI driver pointed to by the snapshotter in the default class must match the CSI driver pointed to by the provisioner in the PVC storage class.

Please note that the alpha release snapshots for Kubernetes does not guarantee consistency. To ensure complete data in the snapshot, it is necessary to prepare the application accordingly (stop the application, freeze the file system, etc.) before removing it.

You can VolumeSnapshot that a VolumeSnapshot object VolumeSnapshot created and associated with a VolumeSnapshotContent by using the kubectl describe volumesnapshot command kubectl describe volumesnapshot :

Ready should be true in Status , which will indicate that the snapshot volume is ready for use.
The Creation Time field shows when the snapshot was actually made.
Restore Size field - the minimum size of the volume to restore snapshot.
The Snapshot Content Name field in the specification points to a VolumeSnapshotContent object created for this snapshot.

Import existing snapshot into Kubernetes

You can import an existing snapshot into Kubernetes by manually creating a VolumeSnapshotContent object that will represent this snapshot. Since VolumeSnapshotContent is an API object that is not bound to a namespace, only a system administrator has permission to create it.

When the VolumeSnapshotContent object VolumeSnapshotContent created, the user can create another object — the VolumeSnapshot — that will point to it. The external-snapshotter controller will mark the snapshot as ready after checking for the existence and correctness of the connection between the VolumeSnapshot and VolumeSnapshotContent . Snapshot is ready for use in Kubernetes when this connection is established.

The VolumeSnapshotContent object should be created with the following fields representing the pre-provisioned snapshot:

csiVolumeSnapshotSource - information identifying snapshot:
- snapshotHandle - name / identifier for snapshot. Obligatory field;
- driver - CSI driver used to work with this volume. Obligatory field. Must match the name of the snapshotter in the controller (snapshot controller);
- creationTime and restoreSize — these are optional for pre-prepared volumes. The external-snapshotter controller will automatically update them after creating the snapshot.
volumeSnapshotRef - a pointer to the VolumeSnapshot object to which this object (ie, VolumeSnapshotContent ) should be VolumeSnapshotContent :
- name and namespace - the name and namespace of the VolumeSnapshot object, the contents of which are bound;
- UID - optional (for pre-prepared volumes) field. The external-snapshotter controller will automatically update this field after binding. If the user specifies this field, you must ensure that it corresponds to the snapshot UID for which the binding occurs. If there is no match, the content is considered irrelevant (orphan-object) and therefore the controller will delete both it and its associated snapshot.
snapshotClassName is an optional field. The external-snapshotter controller will automatically update it after binding.

 apiVersion: snapshot.storage.k8s.io/v1alpha1 kind: VolumeSnapshotContent metadata: name: static-snapshot-content spec: csiVolumeSnapshotSource: driver: com.example.csi-driver snapshotHandle: snapshotcontent-example-id volumeSnapshotRef: kind: VolumeSnapshot name: static-snapshot-demo namespace: demo-namespace

The VolumeSnapshot object must be created so that the user can work with snapshot. In him:

snapshotClassName is the name of the snapshot class of the volume. Optional field. If set, the snapshotter field in the snapshot class must match the name of the snapshot controller. If not set, the controller will search for the default snapshot class;
snapshotContentName is the name of the contents of the snapshot volume. Required field for pre-prepared volumes.

 apiVersion: snapshot.storage.k8s.io/v1alpha1 kind: VolumeSnapshot metadata: name: static-snapshot-demo namespace: demo-namespace spec: snapshotClassName: csi-snapclass snapshotContentName: static-snapshot-content

When these objects are created, the snapshot controller will link them, set the Ready field (in Status ) to True , indicating that the snapshot is ready for use.

Preparing a new snapshot volume in Kubernetes

To create a new volume, pre-filled with data from the snapshot object, use the new dataSource field in PersistentVolumeClaim . It has three parameters:

name - the name of the VolumeSnapshot object that represents the source snapshot;
kind - must be specified as a VolumeSnapshot ;
apiGroup - should be snapshot.storage.k8s.io .

It is assumed that the namespace of the source — the VolumeSnapshot object — corresponds to the namespace of the PersistentVolumeClaim object.

 apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-restore Namespace: demo-namespace spec: storageClassName: csi-storageclass dataSource: name: new-snapshot-demo kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io accessModes: - ReadWriteOnce resources: requests: storage: 1Gi

When the PersistentVolumeClaim object is created, it will provide for the provisioning of a new volume, previously filled with data from the specified snapshot.

How to add snapshots support to my CSI driver if I am a storage developer?

In order to support snapshots, additional features of the controller must be added to the CSI driver: CREATE_DELETE_SNAPSHOT and LIST_SNAPSHOTS , and additional RPC controllers are CreateSnapshot : CreateSnapshot , DeleteSnapshot , ListSnapshots . See the CSI specification for details.

Although Kubernetes provides the most minimal guidelines for packaging and deploying the CSI Volume Driver, there is a recommended mechanism for deploying an arbitrary containerized CSI driver to Kubernetes to simplify this process.

As part of the recommended deployment process, the Kubernetes team suggests using multiple sidecar- (i.e., ancillary) containers, including a sidecar-container with an external-snapshotter .

The external-snapshotter referred to in the API Server for the VolumeSnapshot and VolumeSnapshotContent , invoking the CreateSnapshot and DeleteSnapshot for the CSI endpoint. The CSI external-provisioner sidecar-container has also been updated to support recovery of the volume from snapshot using the new dataSource field in the PVC.

To support snapshot capabilities, storage vendors are recommended to deploy sidecar containers with an external snapshotter in addition to an external provisioner, and place the CSI driver in a StatefulSet , as shown in the diagram below:

In this example, the Deployment presents two sidecar containers, the external provisioner and the external snapshotter, and the CSI drivers are deployed along with the CSI hostpath plugin within the StatefulSet pod. The CSI hostpath is an example of a plugin that is not intended for use in production.

What are the limitations of the alpha version?

The alpha version of the implementation of snapshots in Kubernetes has the following limitations:

It does not support rollback of the existing volume to the previous state represented by snapshot (only provisioning of the new volume from snapshot is supported).
The in-place restore of the existing PersistentVolumeClaim from snapshot is not supported: i.e. The provisioning of a new volume from snapshot works, but not updating an existing PersistentVolumeClaim so that it points to a new volume and the PVC rolls back to an earlier state (only the use of a new volume created from snapshot through the new PV / PVC is supported).
Snapshot consistency guarantees do not go beyond the guarantees provided by the storage system (for example, integrity in a fall).

What's next?

The Kubernetes team plans to bring the implementation of snapshots for CSI to beta in releases 1.13 or 1.14, depending on the feedback received and the adaptation of the technology.

How to find out more details?

See additional snapshots documentation on k8s.io/docs/concepts/storage/volume-snapshots and kubernetes-csi.imtqy.com/docs .