Container - on the conveyor: CRI-O is now the default in OpenShift Container Platform 4
The Red Hat OpenShift Container Platform 4 platform allows you to streamline the creation of hosts for container deployment , including in the infrastructure of cloud service providers, on virtualization platforms or in bare-metal systems. In order to create a cloud platform in the full sense, we had to take all the applied elements under tight control and thus increase the reliability of the complex automation process.
The obvious solution was to use Red Hat Enterprise Linux CoreOS (Red Hat Enterprise Linux flavors) and CRI-O as the standard, and here's why ...
Since the topic of navigation is very successful for finding analogies in explaining the work of Kubernetes and containers, we will try to talk about the business problems that CoreOS and CRI-O solve, using the example of Brunel’s invention for the production of rigging blocks . In 1803, Mark Brunel was tasked to manufacture 100,000 rigging blocks for the needs of the growing navy of Great Britain. The rigging unit is a type of rig that is used to fasten ropes to sails. Up until the very beginning of the 19th century, these blocks were made by hand, but Brunel managed to automate production and begin to produce standardized blocks using machine tools. Automation of this process meant that as a result, all the blocks were almost identical, could be easily replaced in the event of a breakdown, and could be manufactured in large quantities.
')
Now imagine that Brunel would have to do this work for 20 different models of ships (Kubernetes versions) and for five different planets with completely different sea currents and winds (cloud providers). In addition, it was required that all ships (clusters of OpenShift), regardless of the planets on which navigation is carried out, from the point of view of the captains (operators, managing the work of clusters) behaved the same way. Continuing the maritime analogy, it is absolutely not important for the captains of the ships which rigging blocks (CRI-O) are used on their ships - the main thing for them is that these blocks be strong and reliable.
OpenShift 4, as a cloud platform, faces a very similar business task. New nodes should be created when the cluster is created, in the event of a failure in one of the nodes, or when the cluster is scaled. When creating and initializing a new node, critical components of the host, including CRI-O, must be configured accordingly. As in any other production at the beginning it is necessary to submit “raw materials”. In the case of ships, metal and wood act as raw materials. However, if you create a host for deploying containers in an OpenShift 4 cluster, you need to have configuration files and servers provided by the API at the entrance. After that, OpenShift will provide the necessary level of automation throughout the entire life cycle, offering the necessary product support for end users and thus paying for investment in the platform.
OpenShift 4 was created in such a way as to ensure the possibility of convenient system updates throughout the entire life cycle of the platform (for versions 4.X) for all major cloud computing providers, virtualization platforms, and even bare metal systems. For this, nodes must be created on the basis of interchangeable elements. When the cluster requires a new version of Kubernetes, it also receives the corresponding version of CRI-O on CoreOS. Since the CRI-O version is tied directly to Kubernetes, all of this greatly simplifies any permutations for the purpose of testing, troubleshooting, or support. In addition, this approach reduces costs for end users and Red Hat.
This is a fundamentally new look at Kubernetes clusters, which lays the foundation for planning new, very useful and attractive features. CRI-O (open container Container Runtime Interface - Open Container Initiative project, abbreviated as CRI-OCI) turned out to be the most successful choice for mass creation of nodes that is necessary for working with OpenShift. CRI-O will replace the previously used Docker engine, offering OpenShift users a cost- effective , stable, simple and boring - yes, you heard right - a boring container engine created specifically to work with Kubernetes.
The world has long been moving to open containers. Whether in Kubernetes, or at lower levels, the development of container standards leads to the emergence of an ecosystem of innovations at every level.
It all started with the creation of the Open Containers Initiative initiative in June 2015 . At this early stage, the specifications for the container image (image) and the runtime environment were formed. This made it possible to ensure that tools can use a single standard of container images and a single format for working with them. Later, distribution specifications were added, which allowed users to easily share container images .
The Kubernetes community then developed a single pluggable interface standard called the Container Runtime Interface (CRI) . Thanks to this, users of Kubernetes were able to connect various engines for working with containers in addition to Docker.
Red Hat and Google engineers saw the need for a container engine on the market that could accept requests from Kubelet via the CRI protocol and presented containers that were compatible with the OCI specifications mentioned above. So OCID appeared . But excuse me, because we said that this material will be dedicated to CRI-O? In fact, it is, just with the release of version 1.0, the project was renamed to CRI-O.
Fig. one.
With the launch of the OpenShift 4 platform, the container engine used in the platform by default was changed, and Docker was replaced by CRI-O, which offered an economical, stable, simple and boring environment to launch the container, which develops in parallel with Kubernetes. This greatly simplifies cluster support and configuration. Configuring the container engine and host, as well as their management, becomes automated within the framework of OpenShift 4.
Stop like that?
That's right, with the advent of OpenShift 4, now there is no longer any need to connect to individual hosts and install the container engine, configure the storage, configure the servers for searching, or configure the network. The OpenShift 4 platform has been completely redesigned to use the Operator Framework not only in terms of end-user applications, but also in terms of basic operations at the platform level, such as deploying images, configuring the system, or installing updates.
Kubernetes has always allowed users to control applications by defining the desired state and using controllers (Controllers) to ensure that the actual state best corresponds to a given state. This approach with the use of a given state and the actual state opens up great opportunities in terms of both development and operations. Developers can determine the desired state, transfer it to the operator in the form of a YAML or JSON file, and then the operator can create the necessary application instance in the operating environment, and the operating state of this instance will fully correspond to the specified one.
Using operators in the platform, OpenShift 4 brings this new paradigm (using the concept of a given and actual state) to the management of RHEL CoreOS and CRI-O. The tasks of configuring and managing versions of the operating system and container engine are automated using the so-called machine configuration operator (Machine Config Operator, MCO) . The MCO greatly simplifies the work of the cluster administrator, essentially automating the final stages of the installation, as well as subsequent post-installation operations (day two operations). All this makes OpenShift 4 a real cloud platform. We will stop on this a bit later.
Users had the opportunity to use the CRI-O engine in the OpenShift platform starting from version 3.7 in the Tech Preview status and from version 3.9 in the Generally Available status (currently supported). In addition, Red Hat massively uses CRI-O to start production workloads in OpenShift Online starting from version 3.10. All this allowed the team working on CRI-O to gain a vast experience of mass container launch on large Kubernetes clusters. To get a basic understanding of how Kubernetes uses CRI-O, let's look at the following illustration, which shows how the architecture works.
Fig. 2. How containers work in a Kubernetes cluster
CRI-O simplifies the creation of new container hosts by synchronizing the entire top level when initializing new nodes, and when new versions of the OpenShift platform are released. Revision of the entire platform allows you to perform transactional updates / rollbacks, and also prevents interlocks in dependencies between the container tail core, the container engine, Kubelets nodes, and the Kubernetes Master master node. With centralized management of all platform components, with version control and management, you can always track a clear path from state A to state B. This simplifies the update process, improves security, improves performance reporting, and helps reduce the cost of updates and installing new versions.
As mentioned earlier, using Machine Config Operator to manage the container host and container engine in OpenShift 4 provides a new level of automation that was not possible on the Kubernetes platform before. To demonstrate the new features, we will show how you could make changes to the crio.conf file. In order not to get confused in terminology, try to concentrate on the results.
First, let's create something called a Container Runtime Config. Consider that this is a resource Kubernetes, which represents the configuration for CRI-O. In reality, this is a specialized version of what is called MachineConfig, which is any configuration deployed on a RHEL CoreOS machine within an OpenShift cluster.
This custom resource, called ContainerRuntimeConfig, was designed to make it easier for cluster administrators to set up CRI-O. This is quite a powerful tool that it can only be applied to specific nodes, depending on the MachineConfigPool settings. Consider this a group of machines that serve the same purpose.
Notice the last two lines that we are going to change in the /etc/crio/crio.conf file. These two lines are very similar to the lines in the crio.conf file, this is:
Conclusion:
Now we will send this file to the Kubernetes cluster and verify that it is actually created. Please note that the work is carried out in the same way as with any other Kubernetes resource:
Conclusion:
After we created the ContainerRuntimeConfig, we need to change one of the MachineConfigPools to make it clear to Kubernetes that we want to apply this configuration to a specific group of machines in the cluster. In this case, we will change the MachineConfigPool for the master nodes:
Conclusion (for clarity, the main essence is left):
At this point, the MCO starts creating a new crio.conf file for the cluster. At the same time, a fully prepared configuration file can be viewed using the Kubernetes API. Remember, ContainerRuntimeConfig is just a specialized version of MachineConfig, so we can see the result by looking at the right lines in MachineConfigs:
Conclusion:
Note that the resulting configuration file for the master nodes was a newer version than the original configuration. To view it, run the following command. In passing, we note that this is probably one of the best single-line scripts in the history of Kubernetes:
Conclusion:
Now make sure that the configuration has been applied to all master nodes. First we get a list of nodes in the cluster:
Now browse the installed file. You will see that the file has been updated with the new values ​​for the pid and debug directives that we specified in the ContainerRuntimeConfig resource. Elegance itself:
Conclusion:
All these changes in the cluster were made even without running SSH. All work was done by referring to the Kuberentes master site. That is, these new parameters were configured only on the master nodes. At the same time, the work nodes did not change, which demonstrates the advantages of the Kubernetes methodology with the use of specified and actual states in relation to hosts of containers and container engines with interchangeable elements.
The above example shows the ability to make changes to a small cluster of OpenShift Container Platform 4 with three working nodes or to a huge production cluster with 3000 nodes. In any case, the amount of work will be the same - and quite small - just configure the file ContainerRuntimeConfig, and change one label in the MachineConfigPool. And you can do this with any version of the OpenShift Container Platform 4.X platform used in Kubernetes during its entire life cycle.
Often, technology companies are developing so fast that we are not able to explain why we choose certain technologies for basic components. Container engines have historically been the component with which users interact directly. Since the popularity of containers naturally began with the advent of container engines, users often show interest in them. This is another reason why Red Hat opted for CRI-O. Containers are evolving, with today's focus on orchestration, and we conclude that CRI-O provides the best possible experience with OpenShift 4.
The obvious solution was to use Red Hat Enterprise Linux CoreOS (Red Hat Enterprise Linux flavors) and CRI-O as the standard, and here's why ...
Since the topic of navigation is very successful for finding analogies in explaining the work of Kubernetes and containers, we will try to talk about the business problems that CoreOS and CRI-O solve, using the example of Brunel’s invention for the production of rigging blocks . In 1803, Mark Brunel was tasked to manufacture 100,000 rigging blocks for the needs of the growing navy of Great Britain. The rigging unit is a type of rig that is used to fasten ropes to sails. Up until the very beginning of the 19th century, these blocks were made by hand, but Brunel managed to automate production and begin to produce standardized blocks using machine tools. Automation of this process meant that as a result, all the blocks were almost identical, could be easily replaced in the event of a breakdown, and could be manufactured in large quantities.
')
Now imagine that Brunel would have to do this work for 20 different models of ships (Kubernetes versions) and for five different planets with completely different sea currents and winds (cloud providers). In addition, it was required that all ships (clusters of OpenShift), regardless of the planets on which navigation is carried out, from the point of view of the captains (operators, managing the work of clusters) behaved the same way. Continuing the maritime analogy, it is absolutely not important for the captains of the ships which rigging blocks (CRI-O) are used on their ships - the main thing for them is that these blocks be strong and reliable.
OpenShift 4, as a cloud platform, faces a very similar business task. New nodes should be created when the cluster is created, in the event of a failure in one of the nodes, or when the cluster is scaled. When creating and initializing a new node, critical components of the host, including CRI-O, must be configured accordingly. As in any other production at the beginning it is necessary to submit “raw materials”. In the case of ships, metal and wood act as raw materials. However, if you create a host for deploying containers in an OpenShift 4 cluster, you need to have configuration files and servers provided by the API at the entrance. After that, OpenShift will provide the necessary level of automation throughout the entire life cycle, offering the necessary product support for end users and thus paying for investment in the platform.
OpenShift 4 was created in such a way as to ensure the possibility of convenient system updates throughout the entire life cycle of the platform (for versions 4.X) for all major cloud computing providers, virtualization platforms, and even bare metal systems. For this, nodes must be created on the basis of interchangeable elements. When the cluster requires a new version of Kubernetes, it also receives the corresponding version of CRI-O on CoreOS. Since the CRI-O version is tied directly to Kubernetes, all of this greatly simplifies any permutations for the purpose of testing, troubleshooting, or support. In addition, this approach reduces costs for end users and Red Hat.
This is a fundamentally new look at Kubernetes clusters, which lays the foundation for planning new, very useful and attractive features. CRI-O (open container Container Runtime Interface - Open Container Initiative project, abbreviated as CRI-OCI) turned out to be the most successful choice for mass creation of nodes that is necessary for working with OpenShift. CRI-O will replace the previously used Docker engine, offering OpenShift users a cost- effective , stable, simple and boring - yes, you heard right - a boring container engine created specifically to work with Kubernetes.
The world of open containers
The world has long been moving to open containers. Whether in Kubernetes, or at lower levels, the development of container standards leads to the emergence of an ecosystem of innovations at every level.
It all started with the creation of the Open Containers Initiative initiative in June 2015 . At this early stage, the specifications for the container image (image) and the runtime environment were formed. This made it possible to ensure that tools can use a single standard of container images and a single format for working with them. Later, distribution specifications were added, which allowed users to easily share container images .
The Kubernetes community then developed a single pluggable interface standard called the Container Runtime Interface (CRI) . Thanks to this, users of Kubernetes were able to connect various engines for working with containers in addition to Docker.
Red Hat and Google engineers saw the need for a container engine on the market that could accept requests from Kubelet via the CRI protocol and presented containers that were compatible with the OCI specifications mentioned above. So OCID appeared . But excuse me, because we said that this material will be dedicated to CRI-O? In fact, it is, just with the release of version 1.0, the project was renamed to CRI-O.
Fig. one.
Innovations with CRI-O and CoreOS
With the launch of the OpenShift 4 platform, the container engine used in the platform by default was changed, and Docker was replaced by CRI-O, which offered an economical, stable, simple and boring environment to launch the container, which develops in parallel with Kubernetes. This greatly simplifies cluster support and configuration. Configuring the container engine and host, as well as their management, becomes automated within the framework of OpenShift 4.
Stop like that?
That's right, with the advent of OpenShift 4, now there is no longer any need to connect to individual hosts and install the container engine, configure the storage, configure the servers for searching, or configure the network. The OpenShift 4 platform has been completely redesigned to use the Operator Framework not only in terms of end-user applications, but also in terms of basic operations at the platform level, such as deploying images, configuring the system, or installing updates.
Kubernetes has always allowed users to control applications by defining the desired state and using controllers (Controllers) to ensure that the actual state best corresponds to a given state. This approach with the use of a given state and the actual state opens up great opportunities in terms of both development and operations. Developers can determine the desired state, transfer it to the operator in the form of a YAML or JSON file, and then the operator can create the necessary application instance in the operating environment, and the operating state of this instance will fully correspond to the specified one.
Using operators in the platform, OpenShift 4 brings this new paradigm (using the concept of a given and actual state) to the management of RHEL CoreOS and CRI-O. The tasks of configuring and managing versions of the operating system and container engine are automated using the so-called machine configuration operator (Machine Config Operator, MCO) . The MCO greatly simplifies the work of the cluster administrator, essentially automating the final stages of the installation, as well as subsequent post-installation operations (day two operations). All this makes OpenShift 4 a real cloud platform. We will stop on this a bit later.
Launch containers
Users had the opportunity to use the CRI-O engine in the OpenShift platform starting from version 3.7 in the Tech Preview status and from version 3.9 in the Generally Available status (currently supported). In addition, Red Hat massively uses CRI-O to start production workloads in OpenShift Online starting from version 3.10. All this allowed the team working on CRI-O to gain a vast experience of mass container launch on large Kubernetes clusters. To get a basic understanding of how Kubernetes uses CRI-O, let's look at the following illustration, which shows how the architecture works.
Fig. 2. How containers work in a Kubernetes cluster
CRI-O simplifies the creation of new container hosts by synchronizing the entire top level when initializing new nodes, and when new versions of the OpenShift platform are released. Revision of the entire platform allows you to perform transactional updates / rollbacks, and also prevents interlocks in dependencies between the container tail core, the container engine, Kubelets nodes, and the Kubernetes Master master node. With centralized management of all platform components, with version control and management, you can always track a clear path from state A to state B. This simplifies the update process, improves security, improves performance reporting, and helps reduce the cost of updates and installing new versions.
Demonstration of the power of interchangeable elements
As mentioned earlier, using Machine Config Operator to manage the container host and container engine in OpenShift 4 provides a new level of automation that was not possible on the Kubernetes platform before. To demonstrate the new features, we will show how you could make changes to the crio.conf file. In order not to get confused in terminology, try to concentrate on the results.
First, let's create something called a Container Runtime Config. Consider that this is a resource Kubernetes, which represents the configuration for CRI-O. In reality, this is a specialized version of what is called MachineConfig, which is any configuration deployed on a RHEL CoreOS machine within an OpenShift cluster.
This custom resource, called ContainerRuntimeConfig, was designed to make it easier for cluster administrators to set up CRI-O. This is quite a powerful tool that it can only be applied to specific nodes, depending on the MachineConfigPool settings. Consider this a group of machines that serve the same purpose.
Notice the last two lines that we are going to change in the /etc/crio/crio.conf file. These two lines are very similar to the lines in the crio.conf file, this is:
vi ContainerRuntimeConfig.yaml Conclusion:
apiVersion: machineconfiguration.openshift.io/v1 kind: ContainerRuntimeConfig metadata: name: set-log-and-pid spec: machineConfigPoolSelector: matchLabels: debug-crio: config-log-and-pid containerRuntimeConfig: pidsLimit: 2048 logLevel: debug Now we will send this file to the Kubernetes cluster and verify that it is actually created. Please note that the work is carried out in the same way as with any other Kubernetes resource:
oc create -f ContainerRuntimeConfig.yaml oc get ContainerRuntimeConfig Conclusion:
NAME AGE set-log-and-pid 22h After we created the ContainerRuntimeConfig, we need to change one of the MachineConfigPools to make it clear to Kubernetes that we want to apply this configuration to a specific group of machines in the cluster. In this case, we will change the MachineConfigPool for the master nodes:
oc edit MachineConfigPool/master Conclusion (for clarity, the main essence is left):
... metadata: creationTimestamp: 2019-04-10T23:42:28Z generation: 1 labels: debug-crio: config-log-and-pid operator.machineconfiguration.openshift.io/required-for-upgrade: "" ... At this point, the MCO starts creating a new crio.conf file for the cluster. At the same time, a fully prepared configuration file can be viewed using the Kubernetes API. Remember, ContainerRuntimeConfig is just a specialized version of MachineConfig, so we can see the result by looking at the right lines in MachineConfigs:
oc get MachineConfigs | grep rendered Conclusion:
rendered-master-c923f24f01a0e38c77a05acfd631910b 4.0.22-201904011459-dirty 2.2.0 16h rendered-master-f722b027a98ac5b8e0b41d71e992f626 4.0.22-201904011459-dirty 2.2.0 4m rendered-worker-9777325797fe7e74c3f2dd11d359bc62 4.0.22-201904011459-dirty 2.2.0 16h Note that the resulting configuration file for the master nodes was a newer version than the original configuration. To view it, run the following command. In passing, we note that this is probably one of the best single-line scripts in the history of Kubernetes:
python3 -c "import sys, urllib.parse; print(urllib.parse.unquote(sys.argv[1]))" $(oc get MachineConfig/rendered-master-f722b027a98ac5b8e0b41d71e992f626 -o YAML | grep -B4 crio.conf | grep source | tail -n 1 | cut -d, -f2) | grep pid Conclusion:
pids_limit = 2048 Now make sure that the configuration has been applied to all master nodes. First we get a list of nodes in the cluster:
oc get node | grep master Output: ip-10-0-135-153.us-east-2.compute.internal Ready master 23h v1.12.4+509916ce1 ip-10-0-154-0.us-east-2.compute.internal Ready master 23h v1.12.4+509916ce1 ip-10-0-166-79.us-east-2.compute.internal Ready master 23h v1.12.4+509916ce1 Now browse the installed file. You will see that the file has been updated with the new values ​​for the pid and debug directives that we specified in the ContainerRuntimeConfig resource. Elegance itself:
oc debug node/ip-10-0-135-153.us-east-2.compute.internal — cat /host/etc/crio/crio.conf | egrep 'debug||pid' Conclusion:
... pids_limit = 2048 ... log_level = "debug" ... All these changes in the cluster were made even without running SSH. All work was done by referring to the Kuberentes master site. That is, these new parameters were configured only on the master nodes. At the same time, the work nodes did not change, which demonstrates the advantages of the Kubernetes methodology with the use of specified and actual states in relation to hosts of containers and container engines with interchangeable elements.
The above example shows the ability to make changes to a small cluster of OpenShift Container Platform 4 with three working nodes or to a huge production cluster with 3000 nodes. In any case, the amount of work will be the same - and quite small - just configure the file ContainerRuntimeConfig, and change one label in the MachineConfigPool. And you can do this with any version of the OpenShift Container Platform 4.X platform used in Kubernetes during its entire life cycle.
Often, technology companies are developing so fast that we are not able to explain why we choose certain technologies for basic components. Container engines have historically been the component with which users interact directly. Since the popularity of containers naturally began with the advent of container engines, users often show interest in them. This is another reason why Red Hat opted for CRI-O. Containers are evolving, with today's focus on orchestration, and we conclude that CRI-O provides the best possible experience with OpenShift 4.
Source: https://habr.com/ru/post/460495/
All Articles