Introducing shell-operator: it's even easier to create operators for Kubernetes

Our blog already had articles about the capabilities of operators in Kubernetes and how to write a simple operator yourself . This time we want to bring to your attention our Open Source-solution, which takes the creation of operators to the super-easy level - get to know the shell-operator !

What for?

The idea of a shell-operator is quite simple: subscribe to events from Kubernetes objects, and when you receive these events, start an external program, providing it with information about the event:

')
The need for it has arisen in our country, when, during the operation of clusters, small tasks began to appear that we really wanted to automate in the right way. All these small tasks were solved with the help of simple bash scripts, although, as you know, it is better to write operators on Golang. It is obvious that to invest in the full-scale development of the operator for each such minor task would be inefficient.

Operator in 15 minutes

Consider an example of what can be automated in the Kubernetes cluster and how the shell-operator will help. An example would be the following: replicating the secret for accessing the docker registry.

Pods that use private registry images should include in their manifest a link to the secret with data for accessing the registry. This secret must be created in each namespace before creating the pods. It is quite possible to do this manually, but if we configure dynamic environments, then the namespace for one application will become a lot. And if the applications are also not 2-3 ... the number of secrets becomes very large. And one more thing about secrets: I would like to change the key for access to the registry from time to time. As a result, manual operations as a solution are completely ineffective - you need to automate the creation and updating of secrets.

Simple automation

We write a shell script that runs every N seconds and checks namespaces for the presence of a secret, and if there is no secret, then it is created. The advantage of this solution is that it looks like a shell script in cron - a classic and understandable approach. The downside is that in the interval between its launches a new namespace can be created and for some time it will remain without a secret, which will lead to launch errors for pods.

Automation with shell operator

For our script to work correctly, the classic launch by cron should be replaced by the launch when the namespace add event occurs: in this case, you can create a secret before using it. Let's see how to implement this with a shell-operator.

First, let's sort the script. Scripts in terms of a shell-operator are called hooks. Each hook when launched with the --config flag informs the shell-operator of its binding, i.e. what events it needs to run. In our case, we use onKubernetesEvent :

 #!/bin/bash if [[ $1 == "--config" ]] ; then cat <<EOF { "onKubernetesEvent": [ { "kind": "namespace", "event":["add"] } ]} EOF fi

Here it is described that we are interested in adding events ( add ) of objects of type namespace .

Now we need to add code that will be executed when an event occurs:

 #!/bin/bash if [[ $1 == "--config" ]] ; then #  cat <<EOF { "onKubernetesEvent": [ { "kind": "namespace", "event":["add"] } ]} EOF else # : # ,  namespace  createdNamespace=$(jq -r '.[0].resourceName' $BINDING_CONTEXT_PATH) #      kubectl create -n ${createdNamespace} -f - <<EOF apiVersion: v1 kind: Secret metadata: ... data: ... EOF fi

Fine! It turned out a small, beautiful script. To “revive” it, it remains two steps: prepare the image and run it in a cluster.

Preparing an image with a hook

If you look at the script, you can see that the commands kubectl and jq . This means that the image should have the following things: our hook, a shell-operator who will follow the events and run the hook, as well as the commands used by the hook (kubectl and jq). On hub.docker.com there is already a ready image in which shell-operator, kubectl and jq are packed. It remains to add a simple Dockerfile hook:

 $ cat Dockerfile FROM flant/shell-operator:v1.0.0-beta.1-alpine3.9 ADD namespace-hook.sh /hooks $ docker build -t registry.example.com/my-operator:v1 . $ docker push registry.example.com/my-operator:v1

Running in a cluster

Once again, we look at the hook and this time we will write out what actions and with what objects it performs in the cluster:

Subscribes to namespace creation events
creates a secret in namespaces other than where it is running.

It turns out that the pod in which our image will be launched must have permissions for these actions. This can be done by creating your own ServiceAccount. The resolution should be done in the form of ClusterRole and ClusterRoleBinding, since We are interested in objects from the entire cluster.

The final description in YAML is approximately as follows:

 --- apiVersion: v1 kind: ServiceAccount metadata: name: monitor-namespaces-acc --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: monitor-namespaces rules: - apiGroups: [""] resources: ["namespaces"] verbs: ["get", "watch", "list"] - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list", "create", "patch"] --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: monitor-namespaces roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: monitor-namespaces subjects: - kind: ServiceAccount name: monitor-namespaces-acc namespace: example-monitor-namespaces

You can run the assembled image as a simple Deployment:

 apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-operator spec: template: spec: containers: - name: my-operator image: registry.example.com/my-operator:v1 serviceAccountName: monitor-namespaces-acc

For convenience, a separate namespace is created, where the shell-operator will be launched and the generated manifests are applied:

 $ kubectl create ns example-monitor-namespaces $ kubectl -n example-monitor-namespaces apply -f rbac.yaml $ kubectl -n example-monitor-namespaces apply -f deployment.yaml

That's all: the shell-operator starts up, subscribes to the creation events of namespaces, and will launch the hook when needed.

Thus, a simple shell script has become a real operator for Kubernetes and works as part of the cluster. And all this without the complicated process of developing operators on Golang:

There is another illustration on this subject ...

We will reveal its meaning in more detail in one of the following publications.

Filtration

Tracking objects is good, but it is often necessary to respond to changes in some properties of an object , for example, changes in the number of replicas in the Deployment or changes in the labels of an object.

When an event arrives, the shell-operator receives the JSON manifest of the object. You can highlight in this JSON properties that interest us and run the hook only when they change. For this, the jqFilter field is jqFilter , where you need to specify the jq expression that will be applied to the JSON manifest.

For example, in order to react to changing labels on Deployment objects, you need to filter the labels field from the metadata field. The config will be like this:

 cat <<EOF { "onKubernetesEvent": [ { "kind": "deployment", "event":["update"], "jqFilter": ".metadata.labels" } ]} EOF

This jqFilter expression turns a long Deployment JSON manifest into a short JSON with labels:

A shell-operator will run the hook only in cases where this short JSON changes, and changes in other properties will be ignored.

Hook context

The hook configuration allows you to specify several variants of events - for example, 2 variants for events from Kubernetes and 2 schedules:

 {"onKubernetesEvent":[ {"name":"OnCreatePod", "kind": "pod", "event":["add"] }, {"name":"OnModifiedNamespace", "kind": "namespace", "event":["update"], "jqFilter": ".metadata.labels" } ], "schedule": [ { "name":"every 10 min", "crontab":"* */10 * * * *" }, {"name":"on Mondays at 12:10", "crontab": "* 10 12 * * 1" ]}

A small digression: yes, shell-operator supports crontab-style scripting . More details can be read in the documentation .

To distinguish between, in which the hook was launched, the shell-operator creates a temporary file and passes the hook to it in the BINDING_CONTEXT_TYPE variable. The file contains a JSON description of the reason for starting the hook. For example, every 10 minutes the hook will run with the following contents:

 [{ "binding": "every 10 min"}]

... and on Monday it will start with this:

 [{ "binding": "every 10 min"}, { "binding": "on Mondays at 12:10"}]

For onKubernetesEvent will be more JSON positives because it contains a description of the object:

 [ { "binding": "onCreatePod", "resourceEvent": "add", "resourceKind": "pod", "resourceName": "foo", "resourceNamespace": "bar" } ]

The contents of the fields can be understood from their names, and in more detail - read the documentation . An example of obtaining a resource name from the resourceName field using jq has already been shown in a hook that replicates secrets:

 jq -r '.[0].resourceName' $BINDING_CONTEXT_PATH

Similarly, you can get the rest of the field.

What's next?

In the project repository, in the / examples directory , there are examples of hooks that are ready to run in a cluster. When writing your hooks, you can take them as a basis.

There is support for collecting metrics with Prometheus - the available metrics are written in the METRICS section.

As it is easy to guess the shell-operator is written in Go and distributed under the Open Source-license (Apache 2.0). We will be grateful for any help on the development of the project on GitHub : asterisks, issues, and pull requests.

Opening the veil of secrecy, we also inform that the shell-operator is a small part of our system, which is able to keep the add-ons installed in the Kubernetes cluster up to date and performs various automatic actions. We told more about this system literally on Monday at HighLoad ++ 2019 in St. Petersburg - a video and transcript of this report will be published soon.

We have a plan to open the rest of this system: the addon-operator and our collection of hooks and modules. By the way, addon-operator is already available on GitHub , but its documentation is still on the way. Release of the collection of modules is planned in the summer.

Stay tuned!