Understand the Container Storage Interface (in Kubernetes and not only)

Note trans. : about the so-called “out-of-tree” Kubernetes (Out-of-Tree CSI Volume Plugins) storage plug-ins, we first talked about in our review of the K8s 1.9 release , where this feature appeared in the alpha version status. The author of the new material, Anoop Vijayan Maniankara (Lead DevOps Engineer of the Finnish company Tuxera), collected key information about ideas and the CSI device, which helps to quickly get acquainted with the new concept, which some of our employees claim will be “next big thing”. For a more detailed and technical study of CSI at the end of the article are useful links, among which I will highlight the presentation of one of the authors of this specification (Jie Yu). But it’s still worth starting with the “big picture” ...

Container Storage Interface (CSI) is an initiative designed to unify the storage interface, such as Ceph, Portworx, NetApp, etc., in container orchestration systems: Kubernetes, Mesos, Docker Swarm, Cloud Foundry, and others. The idea is that the implementation of one CSI by the storage manufacturer is guaranteed to work with all these systems.

')

Image source: CSI report from Jie Yu to CloudNativeCon EU 2018

Please note : this article will only cover dynamic provisioning. Pre-configured volumes and flex volumes go beyond its scope. If you want to better understand what will be discussed, you should first read the Kubernetes documentation . In addition, the article will not be deeply immersed in the details of the implementation of CSI. I will present a high-level overview of CSI and lay the foundation for creating a CSI volume. One last thing: for examples and references to details, the information for Kubernetes is used.

Before diving into the topic, it is also important to know what sidecar-containers are in Kubernetes. They extend the capabilities of the main container ( main ), existing in the same pod, sharing storage and network.

At the time of this writing (August 13, 2018), the CSI components had the following versions:

Before the advent of CSI

The first release of CSI - v0.1 - took place in December 2017. Of course, provisioning for external storage in orchestration systems was possible before it appeared. In the case of Kubernetes, volume plugins answered for storage needs:

As can be seen from the image above, such plugins are part of the core of the orchestration system. Because of this, the following issues mentioned in the CSI architecture document occurred:

plugin volume development is tightly linked to and dependent on Kubernetes releases;
the Kubernetes developers / community are responsible for testing and supporting all plug-ins, instead of just testing and maintaining a stable plug-in API;
Bugs in volume plugins can drop not only the plugin itself, but also the critical components of Kubernetes;
plugins receive full privileges of the components Kubernetes (kubelet and kube-controller-manager);
plugin developers are forced to publish the source code of the plugin and can not choose the path of the binaries.

Understand CSI

Representing CSI, the Kubernetes team has released external components that are not part of the kernel and are designed to interact with other external components sold by manufacturers. Between themselves, they communicate through domain sockets (UNIX domain sockets - approx. Transl.) Using gRPC .

Kubernetes external components

They are fully implemented and maintained by the Kubernetes team and extend the actions of Kubernetes outside Kubernetes. Manufacturers can not worry about the features of their implementation. Consist of three parts:

Driver registrar is a sidecar container that registers a CSI driver with a kubelet and adds a NodeId driver to a node object label with the Kubernetes API. To do this, it interacts with the service of the CSI-driver Identity (for more details about it, see below - approx. Transl.) And call GetNodeId from CSI;
External provisioner - a sidecar-container that monitors PersistentVolumeClaim objects in the Kubernetes API and calls the CreateVolume and DeleteVolume for the driver's endpoint;
External attacher is a sidecar container that monitors the VolumeAttachment objects in the Kubernetes API and invokes the ControllerPublish and ControllerUnpublish commands for the driver's endpoint.

External component from storage / third party manufacturer

Vendor specific implementation. Each manufacturer implements the necessary APIs as part of the gRPC service functions. For example, the implementation of GCE PD or Ceph , etc. They also consist of three components:

CSI Identity - mainly for plugin identification: make sure that it is functioning, return basic information about the plugin;
```
 service Identity { //      rpc GetPluginInfo(GetPluginInfoRequest) returns (GetPluginInfoResponse) {} // ,       Controller rpc GetPluginCapabilities(GetPluginCapabilitiesRequest) returns (GetPluginCapabilitiesResponse) {} //   ,  ,    rpc Probe (ProbeRequest) returns (ProbeResponse) {} } 
```
( kubernetes-csi-identity.proto )

CSI Controller is responsible for controlling and managing volumes: creating, deleting, attaching / detaching, snapshots, etc .;

 service Controller { //  provisioning  rpc CreateVolume (CreateVolumeRequest) returns (CreateVolumeResponse) {} //     rpc DeleteVolume (DeleteVolumeRequest) returns (DeleteVolumeResponse) {} //       rpc ControllerPublishVolume (ControllerPublishVolumeRequest) returns (ControllerPublishVolumeResponse) {} //       rpc ControllerUnpublishVolume (ControllerUnpublishVolumeRequest) returns (ControllerUnpublishVolumeResponse) {} // ,     /    rpc ValidateVolumeCapabilities (ValidateVolumeCapabilitiesRequest) returns (ValidateVolumeCapabilitiesResponse) {} //     rpc ListVolumes (ListVolumesRequest) returns (ListVolumesResponse) {} //       rpc GetCapacity (GetCapacityRequest) returns (GetCapacityResponse) {} // ,      GetCapacity  Snapshotting rpc ControllerGetCapabilities (ControllerGetCapabilitiesRequest) returns (ControllerGetCapabilitiesResponse) {} //   rpc CreateSnapshot (CreateSnapshotRequest) returns (CreateSnapshotResponse) {} //   rpc DeleteSnapshot (DeleteSnapshotRequest) returns (DeleteSnapshotResponse) {} //    rpc ListSnapshots (ListSnapshotsRequest) returns (ListSnapshotsResponse) {} }

( kubernetes-csi-controller.proto )

CSI Node is responsible for controlling the volume actions on the Kubernetes node.

 service Node { //     staging- rpc NodeStageVolume (NodeStageVolumeRequest) returns (NodeStageVolumeResponse) {} //    staging- rpc NodeUnstageVolume (NodeUnstageVolumeRequest) returns (NodeUnstageVolumeResponse) {} //    staging    rpc NodePublishVolume (NodePublishVolumeRequest) returns (NodePublishVolumeResponse) {} //      rpc NodeUnpublishVolume (NodeUnpublishVolumeRequest) returns (NodeUnpublishVolumeResponse) {} //    rpc NodeGetVolumeStats (NodeGetVolumeStatsRequest) returns (NodeGetVolumeStatsResponse) {} //   ID  rpc NodeGetId (NodeGetIdRequest) returns (NodeGetIdResponse) { option deprecated = true; } //   (capabilities)  rpc NodeGetCapabilities (NodeGetCapabilitiesRequest) returns (NodeGetCapabilitiesResponse) {} //   NodeGetId rpc NodeGetInfo (NodeGetInfoRequest) returns (NodeGetInfoResponse) {} }

( kubernetes-csi-node.proto )

Conclusion

The emergence of CSI brought an obvious plus to orchestration systems and storage manufacturers. In addition, well-defined interfaces help the simple implementation and testing of CSI to both developers and future orchestration systems. If you decide to start implementing your CSI after reading this material, a good starting point will be the article “ How to write a Container Storage Interface (CSI) plugin ” from Fatih Arslan.

References to the literature

CSI Specification ;
Sidecar containers at Kubernetes ;
Report on CSI from Jie Yu at KubeCon EU: CloudNativeCon EU 2018 (and here is the video from this performance available - approx. Transl.) ;
CSI Architecture Document ;
Current CSI documentation from Kubernetes ;
Outdated CSI documentation from Kubernetes .