How did this sidecar container get here [at Kubernetes]?

Note trans. : With this article written by Scott Rahner - an engineer at Dow Jones, we continue the cycle of numerous materials that are available to tell about how Kubernetes works, how they work, are interconnected, and use its basic components. This time it is a practical note with example code for creating a hook in Kubernetes, demonstrated by the author “under the pretext” of automatic creation of sidecar-containers.

(The author of the photo - Gordon A. Maxwell, found on the Internet.)

When I began to study sidecar-containers and service mesh, I needed to understand how the key mechanism works - automatic insertion of a sidecar-container. Indeed, in the case of using systems like Istio or Consul, when a container with an application is deployed, an Envoy container that is already configured appears suddenly in its pod (a similar situation occurs in Conduit, which we wrote about earlier in the year - approx. Transl.) . What? How? So began my research ...
')
For those who do not know, the sidecar-container is a container that is deployed next to the containers of the application in order to “help” this application in any way. An example of such use is a proxy for managing traffic and terminating TLS sessions, a container for streaming logs and metrics, a container for scanning security problems ... The idea is to isolate various aspects of the entire application from business logic by using separate containers for each functions.

Before continuing, I will state my expectations. The purpose of this article is not to explain the intricacies and scenarios of using Docker, Kubernetes, service meshes, etc., but to clearly demonstrate one powerful approach to expanding the capabilities of these technologies. The article is for those who are already familiar with the use of these technologies or, at least, have read a lot about them. To try the practical part in action, you need a machine with already configured Docker and Kubernetes. The easiest way to do this is https://docs.docker.com/docker-for-windows/kubernetes/ (a Windows instruction that works in Docker for Mac). (Note. Trans .: As an alternative to users of Linux and * nix-systems, we can offer Minikube .)

Overall picture

First, let's get a little closer with Kubernetes:

Kube Arch , licensed under CC BY 4.0

When you are going to enclose something in Kubernetes, you need to send the object to kube-apiserver. Most often this is done by passing arguments or a YAML file to kubectl. In this case, the API server goes through several stages before directly putting the data into etcd and scheduling the corresponding tasks:

This sequence is important to understand how the insertion of sidecar-containers works. In particular, attention should be paid to the Admission Control , under which Kubernetes validates and, if necessary, modifies the objects before storing them (for more about this stage, see the “Admission Control” chapter of this article - approx. Transl.) . Kubernetes also allows you to register webhooks that can perform user-defined validation and changes (mutations) .

However, the process of creating and registering your hooks is not so simple and well documented. I had to spend several days reading and rereading the documentation, as well as analyzing the Istio and Consul code. And when it came to the code for some of the API responses, I spent at least half a day doing random trials and errors.

After the result has been achieved, I think it will be dishonest not to share it with all of you. It is simple and at the same time effective.

Code

The name webhook speaks for itself - this is the HTTP endpoint that implements the API defined in Kubernetes. You are creating an API server that Kubernetes can call before it deals with Deployment. Here I had to deal with difficulties, since only a few examples are available, some of which are just Kubernetes unit tests, others are hidden in the middle of a huge code base ... and all are written in Go. But I chose a more affordable option - Node.js:

const app = express(); app.use(bodyParser.json()); app.post('/mutate', (req, res) => { console.log(req.body) console.log(req.body.request.object) let adminResp = {response:{ allowed: true, patch: Buffer.from("[{ \"op\": \"add\", \"path\": \"/metadata/labels/foo\", \"value\": \"bar\" }]").toString('base64'), patchType: "JSONPatch", }} console.log(adminResp) res.send(adminResp) }) const server = https.createServer(options, app);

( index.js )

The path to the API - in this case, /mutate - can be arbitrary (it should only correspond to YAML transmitted to Kubernetes in the future). It is important for him to see and understand the JSON received from the API server. In this case, we do not pull anything out of JSON, but this may be useful in other scripts. In the above code, we update JSON. Two things are needed for this:

Learn and understand JSON Patch .
Correctly convert JSON Patch expression to a byte array encoded with base64.

Once this is done, all you need to do is send the response to the API server with a very simple object. In this case, we add the label foo=bar any pod that comes to us.

Deployment

Well, we have code that accepts requests from the Kubernetes API server and responds to them, but how can we fix it? And how to make Kubernetes redirect these requests to us? You can deploy such an endpoint anywhere that the Kubernetes API server can reach. The simplest way is to deploy the code into the Kubernetes cluster itself, which we will do in this example. I tried to make the example as simple as possible, so for all actions I use only Docker and kubectl. Let's start by creating a container in which the code will run:

 FROM node:8 USER node WORKDIR /home/node COPY index.js . COPY package.json . RUN npm install #       TLS CMD node index.js

( Dockerfile )

As you can see, everything is very simple. Take the image from the node from the community and drop the code into it. Now you can perform a simple build:

 docker build . -t localserver

The next step is to create the Deployment in Kubernetes:

 apiVersion: apps/v1 kind: Deployment metadata: name: webhook-server spec: replicas: 1 selector: matchLabels: component: webhook-server template: metadata: labels: component: webhook-server spec: containers: - name: webhook-server imagePullPolicy: Never image: localserver

( deployment.yaml )

Noticed how we referred to the newly created image? It could just as well have been a pod, and anything else to which we can connect the service to Kubernetes. Now we define this Service:

 apiVersion: v1 kind: Service metadata: name: webhook-service spec: ports: - port: 443 targetPort: 8443 selector: component: webhook-server

This way, an endpoint with an internal name that indicates our container will appear in Kubernetes. The final step is to inform Kubernetes that we want the API server to call this service when it is ready to make changes (mutations) :

 apiVersion: admissionregistration.k8s.io/v1beta1 kind: MutatingWebhookConfiguration metadata: name: webhook webhooks: - name: webhook-service.default.svc failurePolicy: Fail clientConfig: service: name: webhook-service namespace: default path: "/mutate" #    base64-  rootCA.crt #    `cat rootCA.crt | base64 | tr -d '\n'` #    .  caBundle: "LS==" rules: - operations: [ "CREATE" ] apiGroups: [""] apiVersions: ["v1"] resources: ["pods"]

( hook.yaml )

The name and the path here can be any, but I tried to make them as meaningful as possible. Changing the path will mean the need to modify the corresponding code in JavaScript. Webhook failurePolicy is also failurePolicy - it determines whether the object should persist if the hook returns an error or does not work. We in this case tell Kubernetes not to continue processing. Finally, rules: they will change depending on which API calls you expect actions from Kubernetes. In this case, since we are trying to emulate the insertion of a sidecar container, we need to intercept requests to create a pod.

That's all! So simple ... but what about security? RBAC is one aspect that is not covered in the article. I assume that you run the example in Minikube or in Kubernetes that comes with the Docker for Windows / Mac. However, I will tell about one more necessary element. The Kubernetes API server only accesses HTTPS endpoints, so the application will require SSL certificates. You will also need to tell Kubernetes who is the certification authority for the root certificate.

Tls

For demonstration purposes only (!!!), I added some code to the Dockerfile to create a root CA and use it to sign the certificate:

 RUN openssl genrsa -out rootCA.key 4096 RUN openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.crt \ -subj "/C=US/ST=New Jersey/L=Princeton /O=Dow Jones/OU=PIB/CN=*.default.svc/emailAddress=scott.rahner@dowjones.com" RUN openssl genrsa -out webhook.key 4096 RUN openssl req -new -key webhook.key -out webhook.csr \ -subj "/C=US/ST=New Jersey/L=Princeton /O=Dow Jones/OU=PIB/CN=webhook-service.default.svc/emailAddress=scott.rahner@dowjones.com" RUN openssl x509 -req -in webhook.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out webhook.crt -days 1024 -sha256 RUN cat rootCA.crt | base64 | tr -d '\n'

( Dockerfile )

Note: the last stage - displays a single line with the root CA, encoded in base64. This is exactly what is required for the hook configuration, so in your further tests make sure to copy this line into the caBundle field of the caBundle file. Dockerfile throws certificates directly into WORKDIR , so JavaScript simply takes them from there and uses them for the server:

 const privateKey = fs.readFileSync('webhook.key').toString(); const certificate = fs.readFileSync('webhook.crt').toString(); //… const options = {key: privateKey, cert: certificate}; const server = https.createServer(options, app);

Now the code supports the launch of HTTPS, and also told Kubernetes where to find us and which authentication center to trust. It remains only to enclose all this into a cluster:

 kubectl create -f deployment.yaml kubectl create -f service.yaml kubectl create -f hook.yaml

We summarize

Deployment.yaml runs a container that serves the hook API over HTTPS and returns a JSON Patch to modify the object.
Service.yaml provides endpoint for the container - webhook-service.default.svc .
Hook.yaml tells the API server where to find us: https://webhook-service.default.svc/mutate .

Let's try in business!

Everything is deployed in a cluster - time to try the code in action, which we will do by adding a new pod / Deployment. If everything works correctly, the hook will have to add an additional foo label:

 apiVersion: apps/v1 kind: Deployment metadata: name: test spec: replicas: 1 selector: matchLabels: component: test template: metadata: labels: component: test spec: containers: - name: test image: node:8 command: [ "/bin/sh", "-c", "--" ] args: [ "while true; do sleep 30; done;" ]

( test.yaml )

 kubectl create -f test.yaml

Ok, we saw deployment.apps test created ... but did it work out?

 kubectl describe pods test Name: test-6f79f9f8bd-r7tbd Namespace: default Node: docker-for-desktop/192.168.65.3 Start Time: Sat, 10 Nov 2018 16:08:47 -0500 Labels: component=test foo=bar

Wonderful! Although test.yaml had a single label set ( component ), the resulting pod received two: component and foo .

Homework

But wait! Were we going to use this code to create a sidecar container? I warned that I’ll show you how to add a sidecar ... And now, with the knowledge and code you received: https://github.com/dowjones/k8s-webhook - experiment and dare to make your own sidecar automatically inserted. It's quite simple: you only need to prepare the correct JSON Patch, which will add an additional container in the test Deployment. Happy orchestration!