Kubernetes 1.9: a review of major innovations

The next release of the system Kubernetes, 1.9, should happen this week. According to the current plan , it will happen today (December 13). We already know about the main innovations that this issue will bring: just like last time , there are really a lot of them. We present an overview of the most significant changes that come to Kubernetes with the upcoming release of 1.9.

When writing this article were used:
')

• official draft K8s 1.9 Release Notes ;
• table for Kubernetes Features OSS tracking board developers (note that some of its data disagree with reality, because not all of the features listed there managed to be completed for release) ;
• CHANGELOG-1.9 .

... and, of course, endless issues and pull requests in the project's GitHub repositories.

API

Redesign Event API

The event API architecture (and the EventRecorder library) implements changes aimed at improving in two ways:

1. Reduce the impact of events on overall cluster performance .
2. To make the Event object more structured in order to simplify further work with it (“the first and necessary step for being able to automate its analysis”).

Among the problems solved by these changes:

• Now events send too many messages (for example, the impossibility of scheduling a submission generates an event every few seconds);
• the semantics of events is not always clear (for example, there is no distinction between the reasons for generating an event and the reasons for taking any action related to the event);
• increased load on the cluster due to events in case of any problems (for example, the user has created much more pods than the cluster is able to withstand, which leads to repeated errors in their planning);
• ...

Comparison of old and new Event objects:



In detail, all this is described in design-proposals: events-redesign . The current status is alpha version.

Admission webhooks

A beta version of a mechanism called admission webhooks has been added that allows you to modify and / or accept and reject API requests . It refers to the admission control phase and is triggered at the last request processing steps before they are saved to the database (etcd). At their core, these webhooks are similar to initializers (we wrote about them in the translation article “ What happens in Kubernetes when starting the kubectl run? Part 1 ”) , however:

• apply to all actions (creation, modification, deletion);
• performed synchronously (i.e., long delays are not allowed);
• Objects for which hooks are executed are not available until they are executed.

An example of using admission webhooks from documentation is to check one object field and set a value for its other field. Another example is the automatic addition of a service container (for example, performing a backup) according to the annotation of the sub. For example, we have a sub under MySQL, with one container (in which MySQL itself), and this pod has a special annotation. When we create a request, the request comes to the Kubernetes API, which sends the request to our application via admission webhook , and the application sees the request, sees what is created under, sees its special annotation and does just rewrite - it adds to the second container.

A detailed description of admission webhooks published here .

Workload API

Workload API was introduced in the last Kubernetes release and combined the workload-related interfaces: DaemonSet , Deployment , ReplicaSet , StatefulSet . For them, a separate API group called apps was allocated, and with the release of K8s 1.9, all these changes received a stable status .

The main changes in the Workload API, presented in the releases Kubernetes 1.8 and 1.9, are summarized in the project documentation .

Other

• Condition support has been added to the StatefulSet (and DaemonSet ) state, making them compatible with other core/v1 controllers.
• The --chunk-size={SIZE} flag has --chunk-size={SIZE} added to kubectl get , and the support for restricting data output by the API (API chunking) has generally received beta status.

Storage

Out-of-Tree Volume Plugins (CSI)

The available volume plug-ins are included in Kubernetes: they are called “in-tree”, because all the code is included in the main project repository, and in compiled form they are distributed as part of the K8s binaries. This approach has an obvious disadvantage: the support of third-party repositories by their manufacturers / developers depends on the release cycles of Kubernetes (besides, all the source code must be Open Source). In part, this problem is solved by the Flexvolume plugin, but this mechanism does not guarantee backward compatibility and (when installing the driver) requires deploying its files to specific locations on nodes and wizards.

The new approach is called Out-of-Tree CSI Volume Plugins (CSI is the Container Storage Interface, which defines how container orchestration systems can use third-party storage) . It is designed to implement a full-fledged API in Kubernetes, which allows you to:

• create volumes “out of the tree” (Kubernetes repository);
• do not require the inclusion of compiled code in K8s binary files;
• do not require direct access to Kubernetes machines to deploy these plug-ins (drivers).

The authors proposed the following recommended mechanism for deploying CSI drivers in Kubernetes:



The explanations for this scheme and the details of the project as a whole are published in this document . Implementation status in Kubernetes 1.9 - alpha version.

Running mount inside the deck

The feature called Containerized mounts introduces to Kubernetes the ability to run mount utilities on the sub, and not on the host machine . Thus, the host system can remain a minimal GNU / Linux distribution without third-party software: to create Ceph RBD ( /usr/bin/rbd ), mount GlusterFS ( mount.glusterfs ), etc., and all utilities for working with volumes (operations provision / delete, attach / detach, mount / unmount) are placed in the slots.

Details about this feature are published in design-proposals: containerized-mounter-pod . The current status is alpha version.

Resizing volumes

Basic support for the resize operation for existing PV ( PersistentVolume ) appeared in Kubernetes 1.8. By 1.9, it still did not reach the beta version, however, significant progress is evident: added support for resizing for file systems, and with it for GCE, EBS, Cinder, and also Ceph RBD volumes. A beta version is expected for release 1.10.

Other

• The removal of PersistentVolumeClaims used by any of the pods is now prevented (alpha version).
• The volumeMode and volumeDevice for PV ( PersistentVolume ) options allow you to directly use block raw devices instead of the file system (alpha version).

Network

IPv6 support

Implemented basic IPv6 support, providing "all Kubernetes capabilities on an IPv6 network instead of IPv4." At the moment, this implementation is in the alpha version and includes:

• support for deploying Kubernetes clusters in the “IPv6 only” mode;
• support for deployment of a cluster with IPv6 via kubeadm ;
• support K8s control plane and data plane;
• iptables kube-proxy backend support using ip6tables;
• use of CNI 0.6.0 assembly for the IPv6 network;
• IPv6 support in kube-dns via SRV records
• restrictions: only bridge and local-ipam were checked from CNI plugins; lack of support for HostPorts; The netmask for the sub / cluster must be / 66 or more.

Other

• An experimental mode has been added to kubeadm , in which CoreDNS is used instead of kube-dns . More details about the progress of the project CoreDNS we wrote here .
• The IPVS mode in kube-proxy , first introduced in K8s 1.8, has switched to beta status.

Other components and changes

Scheduler

The cluster resource release mechanism ( pod preemption ) has been improved thanks to the support of PodDisruptionBudget and nominated pods . In addition, the scheduler added support for a new type of queue, based on priorities (the first will be scheduled pods with the highest priority).

When using hostPort , the pod has the opportunity to listen to the same port on different IP addresses.

Authentication and Security

An important innovation from SIG Auth is ClusterRole Aggregation for the ability to add permissions to existing ones, i.e. roles built into RBAC ( admin , edit , view ).

You can also note that:

• Support for */subresource format has been added to RBAC policy rules (for example, */scale will include replicationcontroller/scale ;
• In PodSecurityPolicy , noticeable improvements have been made, including a multiple increase in performance when using a large number of policies in a cluster (authorization now only takes place for valid sub-policies, not all cluster policies) and support for K8s add-ons .

CLI

The feature name kubectl diff speaks for itself: this is a new command that allows you to view the differences between the locally described Kubernetes object and the current state of the real object (in the cluster). The current status is alpha version.

The option --etcd-upgrade added to kubeadm upgrade apply , which performs an upgrade to etcd in sub-versions to a version that is officially supported by the target release Kubernetes (3.1.10 in the case of K8s 1.9).

Kubeadm has added support for Kubelet Dynamic Configuration , which means that it is possible to roll out kubelet configurations to a working cluster (the feature itself first appeared in the previous version of Kubernetes and retains the status of the alpha version with the prospect of increasing to beta in K8s 1.10).

Windows

• Through kubeadm , you can now add work nodes with Windows to the cluster.
• Kubelet now has support for specifying mount paths in Windows format, not just absolute Linux paths, as it was before.

Openstack

Improved integration with the cloud platform:

• Cinder v3 API support added and Cinder version detection fixed;
• OpenStack LBaaS v2 Provider has become customizable;
• OpenStack Octavia v2 (not only Neutron LBaaS v2) can now be used for load balancing;
• extended support for OpenStack security groups.

Other changes

• Validation of third-party resources defined through Custom Resource Definition (CRD) , received the status of a beta version, and also added a sample of the CRD controller.
• A cloud-controller-manager has been added to the hyperkube (a binary file that includes all Kubernetes server components).
• Upgrading cAdvisor to 0.28.1 added support for monitoring containerd.
• AppArmor can now be disabled by setting the profile to unconfined .
• When you start kubelet , the dependency on Docker is no longer checked.
• The format of container logs in CRI has learned to break long lines into several lines, and fluentd has added support for logs in CRI format.

Compatibility

• The supported etcd version is 3.1.10 (in Kubernetes 1.8 was 3.0.17).
• Proven Docker versions range from 1.11.2 to 1.13.1 and 17.03.x.
• Go version - 1.9.2 (instead of 1.8.3), minimum supported - 1.9.1.
• CNI version is 0.6.0.

PS

Read also in our blog:

• “ Four releases 1.0 from CNCF and major announcements about Kubernetes from KubeCon 2017 ”;
• “ Kubernetes 1.8: review of major innovations ”;
• “ Docker 17.06 and Kubernetes 1.7: key innovations ”;
• " Our experience with Kubernetes in small projects " (video of the report, which includes an introduction to the technical device Kubernetes);
• " Infrastructure with Kubernetes as an affordable service ."