Orchestration of the CockroachDB DBMS at Kubernetes

Preface of the translator : Literally a week after our publication of acquaintance with CockroachDB , the first final release of this distributed and scalable open source DBMS took place, which marked its official readiness for use in production. And that means it's time to learn how to “prepare” it in the realities of microservices with Kubernetes.

This article shows you how to orchestrate deployments and manage an unsafe 3-node CockroachDB cluster using Kubernetes and its StatefulSet function, which is in beta.

Yes, launching a stateful application like CockroachDB on Kubernetes requires the use of complex system capabilities that are currently supported at the beta level. You can run CockroachDB on Kubernetes for testing and easier, however, the approach described here is intended for deploying the DBMS in production, when the necessary functions in Kubernetes are finally ready for this. (Approx. Translation: about the problem of stateful applications in Kubernetes and one of the approaches to its solution, we told in the material about Kubernetes Operators .)

Please note that an unsecured cluster (i.e., with encryption disabled - approx. Transl.) Is not recommended for production data. The authors promise to update the manual after the deployment of secure clusters is improved.

Step 1. Choosing an environment for deployment

Select the environment in which you will run CockroachDB in Kubernetes: cloudy or local. The instructions given in the article take into account both options, and each of them has its own nuances.
')
So, for a start, it will be useful to familiarize yourself with the terminology associated with Kubernetes:

• minikube [relevant only for local use] - a utility that allows you to run a Kubernetes cluster from a single node inside a virtual machine of your computer;
  
  
• An instance [relevant only for cloud applications] is a physical or virtual machine. In this guide, a script for Kubernetes will be launched from your local computer, which will remotely create four instances of GCE or AWS and join them to a single cluster Kubernetes;
  
  
• pod (under) - a group of one or more Docker containers. In this tutorial, each of them will run on a separate instance and contain one Docker container, which runs a single CockroachDB node. We will start with 3 sweeps and increase their number to 4;
  
  
• StatefulSet is a group of pods considered as stateful units. Each sub has its own network address and always after restarting is tied back to the same permanent storage. StatefulSets is a beta-based feature from Kubernetes 1.5;
  
  
• persistent volume (constant volume) - part of the network storage, i.e. Persistent Disk for GCE and Elastic Block Store for AWS [in the case of cloud application] , or local storage [in the local case] , mounted in under. The lifetime of a permanent volume does not depend on the lifetime of the hearth that uses it, and each CockroachDB node, when restarted, is tied back to its old repository.
  
  
  • Clarification for the cloud: the guide implies that dynamic provisioning is available for volumes. Otherwise, you must manually create persistent volume claims . See the steps necessary for this in minikube.sh .
    
    
  • For local use: when using minikube, persistent volumes are temporary external directories that continue to exist until they are manually removed or the entire Kubernetes cluster is deleted.
    
• persistent volume claim [pertinent to local use only] - when pods are created (one for each CockroachDB node), each of them requests a persistent volume claim to determine the persistent storage for its node.

Step 2. Installing and running Kubernetes

In the cloud

Install and run the Kubernetes cluster from your computer according to the project documentation:

• For GCE instructions, see Running Kubernetes on Google Compute Engine .
• For AWS instructions, see “ Running Kubernetes on AWS EC2 ”.

The central part of this step will be the launch of the Kubernetes script, which will create four instances of GCE or AWS, combining them into a single cluster of Kubernetes - and all this remotely from your computer. Subsequent steps will also be performed from your computer.

Locally

According to the Kubernetes documentation, install minikube and kubectl for your operating system. After that, start the local cluster Kubernetes:

$ minikube start Starting local Kubernetes cluster... Kubectl is now configured to use the cluster. 

Step 3. Launch the CockroachDB Cluster

In the cloud

Use the ready -made cockroachdb-statefulset.yaml file to create the StatefulSet from your computer:

 kubectl create -f https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/cockroachdb-statefulset.yaml 

Translator's note : Before using ready-made cockroachdb-statefulset.yaml from the developers at Cockroach Labs, it is useful to examine its contents. Fortunately, the authors took care of comments that help in understanding what and how unfolds in Kubernetes. And, of course, you can modify it in accordance with your expectations from the experimental or combat operation of CockroachDB.

Locally

Download the minikube.sh script and the cockroachdb-statefulset.yaml configuration file, run the script:

 $ wget https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/minikube.sh $ wget https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/cockroachdb-statefulset.yaml $ sh minikube.sh 

This script automates the process of creating persistent volumes and persistent volume claims. It also executes the kubectl create command with the cockroachdb-statefulset.yaml configuration to create a StatefulSet.

General

Further actions of this step are common for cloud and local applications.

Use the kubectl get command to verify that persistent volumes and corresponding claims were created:

 $ kubectl get persistentvolumes NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE pvc-52f51ecf-8bd5-11e6-a4f4-42010a800002 1Gi RWO Delete Bound default/datadir-cockroachdb-0 26s pvc-52fd3a39-8bd5-11e6-a4f4-42010a800002 1Gi RWO Delete Bound default/datadir-cockroachdb-1 27s pvc-5315efda-8bd5-11e6-a4f4-42010a800002 1Gi RWO Delete Bound default/datadir-cockroachdb-2 27s 

For local volumes, the output of the same command will look like this:

 NAME CAPACITY ACCESSMODES STATUS CLAIM REASON AGE pv0 1Gi RWO Bound default/datadir-cockroachdb-0 27s pv1 1Gi RWO Bound default/datadir-cockroachdb-1 26s pv2 1Gi RWO Bound default/datadir-cockroachdb-2 26s 

Wait a bit and make sure that the three hearths were also successfully created If you still don’t see them, wait and check again:

 $ kubectl get pods NAME READY STATUS RESTARTS AGE cockroachdb-0 1/1 Running 0 2m cockroachdb-1 1/1 Running 0 2m cockroachdb-2 1/1 Running 0 2m 

The StatefulSet configuration configures all the CockroachDB nodes to write to stderr, so if you need access to the subsea logs or nodes for analyzing problems, use the kubectl logs <podname> instead of the “physical” log check on the actual sub.

Step 4. Using the embedded SQL client

Run the embedded SQL client in an interactive temporary (created for one-time use) pod, substituting the host name cockroachdb-public to connect to the CockroachDB cluster:

 $ kubectl run cockroachdb -it --image=cockroachdb/cockroach --rm --restart=Never \ -- sql --insecure --host=cockroachdb-public 

Run the CockroachDB SQL commands (for more information, see the project documentation ):

 CREATE DATABASE bank; CREATE TABLE bank.accounts (id INT PRIMARY KEY, balance DECIMAL); INSERT INTO bank.accounts VALUES (1234, 10000.50); SELECT * FROM bank.accounts; +------+----------+ | id | balance | +------+----------+ | 1234 | 10000.50 | +------+----------+ (1 row) 

When the execution of commands in the SQL client is complete, press <Ctrl> + <d>, <Ctrl> + <c> or enter the \q command to exit and delete the temporary sub.

Step 5. Simulation of a node fall

According to replicas: 3 in the StatefulSet configuration, Kubernetes checks that three sub-sites / nodes are constantly running. If the / node falls, Kubernetes will automatically create a new one with the same network name and persistent storage.

Let's look at it in action:

1. Kill one of the CockroachDB nodes:
   
   

    $ kubectl delete pod cockroachdb-2 pod "cockroachdb-2" deleted 

   
   
2. Check that under restarted:
   
   

    $ kubectl get pod cockroachdb-2 NAME READY STATUS RESTARTS AGE cockroachdb-2 0/1 ContainerCreating 0 3s 

   
   
3. Wait a bit and check that it is ready to work:
   
   

    $ kubectl get pod cockroachdb-2 NAME READY STATUS RESTARTS AGE cockroachdb-2 1/1 Running 0 1m 

Step 6. Cluster scaling

In the cloud

The Kubernetes script created 4 nodes: 1 master and 3 workers (workers). The pods are placed only on the working nodes, so to be sure that you will not have two pods on one node (see recommendations in best practices for production), you need to add a new working node and edit the StatefulSet configuration, adding another one.

1. Add a working node. To do this, resize the Managed Instance Group on the GCE and resize the Auto Scaling Group on the AWS.
2. Use the kubectl scale command to add a sub into the StatefulSet:
   
   

    $ kubectl scale statefulset cockroachdb --replicas=4 statefulset "cockroachdb" scaled 

   
   
3. Make sure the fourth sub has been added:
   
   

    $ kubectl get pods NAME READY STATUS RESTARTS AGE cockroachdb-0 1/1 Running 0 2h cockroachdb-1 1/1 Running 0 2h cockroachdb-2 1/1 Running 0 9m cockroachdb-3 1/1 Running 0 46s 

Locally

To increase the number of pods, it is enough to run the same kubectl scale command, and then check the success of the operation performed ( kubectl get pods ).

Step 7. Stop the cluster

In the cloud

To stop a CockroachDB cluster:

1. Use the kubectl delete command to clear all created resources (including logs and remote persistent volumes):
   
   

    $ kubectl delete pods,statefulsets,services,persistentvolumeclaims,persistentvolumes,poddisruptionbudget \ -l app=cockroachdb 

   
   
2. Run the cluster/kube-down.sh from the kubernetes directory.

Note : if you stop Kubernetes without first deleting resources, the deleted persistent volumes will continue to exist in your cloud.

Locally

• If you plan to start the cluster again, use the minikube stop command. It will turn off the virtual machine with minikube, but will save all the resources created:
  
  

   $ minikube stop Stopping local Kubernetes cluster... Machine stopped. 

  
  Recovery of the cluster in the last state is possible using the minikube start command.
  
  
• If you are not planning further cluster launches, use the minikube delete command. It will turn off and delete the virtual machine with minikube and all created resources (including permanent volumes):
  
  

   $ minikube delete Deleting local Kubernetes cluster... Machine deleted. 

Note : to save the logs, copy them from the stderr of each hearth before removing the cluster and all its resources. To access the stderr stream, use the command kubectl logs <podname> .

Translator's afterword : The material translated here from the Cockroach Labs website (link is provided in the source) corresponds to the document in the markdown markup , updated in the project's Git repository.

We ourselves do not use CockroachDB in production yet, but we look closely at this interesting project and see in it real potential.