Operators for Kubernetes: how to run stateful applications

The problem of stateful applications in Kubernetes

Configuration, launch and further scaling of applications and services are carried out simply when it comes to cases classified as stateless, i.e. without saving the data. It is convenient to launch such services in Kubernetes, using its standard API, because everything happens out of the box: according to standard configurations, without invoking any specificity and magic.

Simply put, to launch five more copies of the PHP / Ruby / Python backend in a cluster, you only need to raise a new server 5 times and copy the sources. Since both the source code and the init script lie in the image, scaling the stateless application becomes quite elementary. As well known to lovers of containers and microservice architecture, difficulties begin for stateful applications , i.e. with data storage such as databases and caches (MySQL, PostgreSQL, Redis, ElasticSearch, Cassandra ...). This applies both to software that independently implements a quorum cluster (for example, Percona XtraDB and Cassandra), and software that requires separate control utilities (such as Redis, MySQL, PostgreSQL ...).

Difficulties arise because the source code and the launch of the service becomes not enough - you need to perform some more actions. At a minimum, copy the data and / or join the cluster. More specifically, these services require an understanding of how to properly scale, update and reconfigure them without losing data and their temporary unavailability. Accounting for these needs is called “operational knowledge”.

CoreOS operators

In order to “program” operational knowledge, at the end of last year, the CoreOS project introduced a “new software class” for the Kubernetes platform — Operators (Operators, from the English word “operation”, ie, “operation”).
')
Operators, using and extending the basic capabilities of Kubernetes (including StatefulSets , for the difference with which see below), allow DevOps specialists to add operational knowledge to the application code.

The goal of the Operator is to provide the user with an API that allows him to manage the many entities of a stateful application in the Kubernetes cluster, not thinking about what is under his hood (what data and what to do with it, what commands need to be executed to maintain the cluster). In fact, the Operator is designed to maximally simplify work with the application within the cluster, automating the performance of operational tasks that previously had to be solved manually.

How Operators Work

ReplicaSets in Kubernetes allow you to specify the desired number of running hearths, and controllers ensure that their number is maintained (creating and deleting hearths). The Operator works in a similar way, which adds to the standard resource and the Kubernetes controller a set of operational knowledge that allows you to perform additional actions to maintain the required number of application entities.

How is this different from StatefulSets intended for applications requiring the cluster to provide them with stateful resources such as data storage or static IP? For such applications, Operators can use StatefulSets (instead of ReplicaSets ) as a basis, offering additional automation : perform necessary actions in case of crashes, make backups, update configuration, etc.

So how does all this work? The operator is a daemon controller that:

1. Subscribes to the event API in Kubernetes
2. obtains data about the system from it (about its ReplicaSets , Pods , Services , etc.);
3. obtains data about the Third Party Resources (see examples below);
4. reacts to the appearance / change of the Third Party Resources (for example, to change the size, change the version, and so on);
5. responds to changes in the state of the system (about its ReplicaSets , Pods , Services , etc.);
6. the most important thing:
   
   
   1. calls the Kubernetes API to create everything you need (again, your ReplicaSets , Pods , Services ...),
   2. performs some magic (it is possible, for simplicity, to think that the Operator comes into the platforms themselves and calls commands, for example, to join a cluster or to upgrade the data format when updating a version).

In fact, as can be seen from the image, a separate application is simply added to Kubernetes (normal Deployment with ReplicaSet ), which is called the Operator. It lives in the usual pod (usually one and only) and, as a rule, is responsible only for its Namespace . This operator application implements its API - though not directly, but through the Third Party Resources at Kubernetes.

Thus, after we have created an Operator in Namespace , we can add Third Party Resources to it .

Example for etcd (see below for details) :

apiVersion: etcd.coreos.com/v1beta1 kind: Cluster metadata: name: example-etcd-cluster spec: size: 3 version: 3.1.0 

Example for Elasticsearch:

 apiVersion: enterprises.upmc.com/v1 kind: ElasticsearchCluster metadata: name: example-es-cluster spec: client-node-replicas: 3 master-node-replicas: 2 data-node-replicas: 3 zones: - us-east-1c - us-east-1d - us-east-1e data-volume-size: 10Gi java-options: "-Xms1024m -Xmx1024m" snapshot: scheduler-enabled: true bucket-name: elasticsnapshots99 cron-schedule: "@every 2m" storage: type: gp2 storage-class-provisioner: kubernetes.io/aws-ebs 

Requirements for Operators

In CoreOS, they formulated the main patterns obtained by engineers during the work on the Operators. Despite the fact that all Operators are individual (created for a specific application with their own features and needs), their creation should be based on a kind of framework that makes the following requirements:

1. Installation should be done through a single Deployment : kubectl create -f SOME_OPERATOR_URL / deployment.yaml - and do not require additional actions.
2. When installing the Operator in Kubernetes, a new third-party type (ThirdPartyResource) should be created. For launching application instances (cluster instances) and further managing them (updating versions, resizing, etc.) the user will use this type.
3. Whenever possible, you need to use the primitives built into Kubernetes, such as Services and ReplicaSets , in order to use well-tested and understandable code.
4. It requires backward compatibility of Operators and support for old versions of resources created by the user.
5. When removing the Operator, the application itself should continue to function without changes.
6. Users should be able to determine the desired version of the application and perform orchestration of the application version updates. The lack of software updates is a frequent source of operational and security problems, therefore Operators should assist users in this matter.
7. Operators should be tested with a tool like Chaos Monkey, which identifies potential failures in subframes, configurations and networks.

etcd Operator

An example of the implementation of the Operator - etcd Operator, prepared for the day of the announcement of this concept. Cluster configuration etcd can be complicated due to the need to maintain a quorum, the need to reconfigure cluster membership, create backups, etc. For example, scaling a cluster etcd means manually creating a DNS name for a new cluster member, starting a new etcd entity, notifying the cluster about a new member ( etcdctl member add ). In the case of the Operator, it will be enough for the user to change the cluster size - everything else will happen automatically.

And since etcd was also created in CoreOS, it was quite logical to see the appearance of its Operator first. How does he work? The etcd operator logic is defined by three components:

1. Observation (Observe). The operator monitors the cluster status by the Kubernetes API.
2. Analyze Finds the difference between the current status and the desired one (defined by the user configuration).
3. Action (Act). Eliminates differences using the etcd and / or Kubernetes API.

To implement this logic, the Operator has prepared the functions Create / Destroy (creating and deleting cluster members etcd) and Resize (changing the number of cluster members). Checking the correctness of its performance was checked using a utility created in the likeness of Chaos Monkey from Netflix, i.e. killing pods etcd randomly.

For full operation, etcd, the Operator provides additional features: Backup (automatic and invisible for users to create backup copies - it’s enough to determine in the config file how often to do them and how many to keep, and then restore data from them) and Upgrade (updating etcd installations without just me).

How does the work with the operator look like?

 $ kubectl create -f https://coreos.com/operators/etcd/latest/deployment.yaml $ kubectl create -f https://coreos.com/operators/etcd/latest/example-etcd-cluster.yaml $ kubectl get pods NAME READY STATUS RESTARTS AGE etcd-cluster-0000 1/1 Running 0 23s etcd-cluster-0001 1/1 Running 0 16s etcd-cluster-0002 1/1 Running 0 8s etcd-cluster-backup-tool-rhygq 1/1 Running 0 18s 

The current status of etcd Operator is a beta version that requires Kubernetes 1.5.3+ and etcd 3.0+ to work. Source code and documentation (including instructions for use) are available on GitHub .

Another implementation example from CoreOS, Prometheus Operator , has been created, but it is still in alpha version (not all planned functions are implemented).

Status and prospects

5 months have passed since the announcement of the Kubernetes Operators. In the official CoreOS repository, only two implementations are still available (for etcd and Prometheus). Both have not yet reached their stable versions, but commits are observed daily in them.

The developers expect “a future in which users install Postgres Operators, Cassandra Operators or Redis Operators in their Kubernetes clusters and work with scalable entities of these applications as easily as today happens with the replica of stateless web applications”. First Operators from third-party developers really began to appear:

• Elasticsearch Operator from UPMC Enterprises;
• PostgreSQL Operator from Crunchy Data (announced at the end of March 2017.);
• Rook Operator from the authors of Ceph based distributed storage system (Rook is in alpha status);
• Openstack Operators from SAP CCloud.

At the largest European free software conference FOSDEM, which was held in February 2017 in Brussels, Josh Wood from CoreOS announced the Operators in the report (video available on the link!), Which should contribute to the growing popularity of this concept in the wider Open Source community.

PS Thank you for your interest in the article! Subscribe to our hub , not to miss new materials and recipes on DevOps and GNU / Linux system administration - we will publish them regularly!