Testing in Openshift: An Introduction

Hello dear members of the IT community. My name is Oleg, I work in a software development company. I do manual and automated testing of Linux and Unix products and I would like to share the positive experience of automated testing in Openshift Origin .

The goals that I pursue are :

1. To inform the Russian-speaking community about the features of working with Openshift Origin in the context of testing.
2. Tell about the advantages and disadvantages of testing in containers.
3. Aggregate and update your knowledge about Kubernetes / Openshift.

All material is presented in three articles :

1. Testing in Openshift: An Introduction
2. Testing in Openshift: Internal cluster structure
3. Testing in Openshift: Automated Testing

Note: I would like to immediately note that the material presented concerns Openshift v3, and not Openshift v2 (when Red Hat has not yet begun to use Kubernetes as the core for its products and services).

Why Openshift Origin was chosen :

It so happened that after being employed at the current place of work, it turned out that there was no automated testing. After working for several months in manual mode, it came to understand the need for two things: a tool for deploying and maintaining test environments, a framework for writing tests.

At the initial stage, an automatically updated Vagrant repository ( VirtualBox ) was organized for deployment and support of environments, which automatically updates and packages various operating systems. This was an aid not only for testing, but also for development, because: the virtual machines were configured according to the developed requirements; virtual machines have been updated, the necessary tools have been preinstalled; there were Vagrant scripts for deploying complex environments. It is worth noting that all Vagrant environments are deployed on the local machines of the development participants, and not in the dedicated IaaS , which was expected to cause performance problems for the workstations.

Time was gained, it became more convenient to work, some standard and predictability appeared, but the main problem remained - long deployment of the tested environments (full virtualization, various Linux distributions, additional services (which participate in testing)). Deployment of the tested media could take tens of minutes, while the test itself took place in a matter of seconds. At the same time, the number of automated tests grew and it took longer to expect results.

The question of organizing a separate IaaS, in which all the testing tasks would occur, became natural, but this approach was not perfect: full virtualization would still be used, building investments in the purchase of hardware would be required to build IaaS (a small fleet of weak workstations was present) .

The next stage was the development of a fast (in the context of the environment performance) CI / CD with new requirements, namely (basic, according to priorities):

1. Open Source solution at no cost to purchase software.
2. Rapid deployment speed of test environments. Connectivity and awareness of the existence of services between environments. Built-in support and control of test runs inside the environment. High density accommodation to meet point 4.
3. An active community, a popular project with a predictable future, owner's reputation, documentation.
4. No additional investment in the hardware component.

I will not bother readers with a detailed retelling of the thorny path of finding suitable products, but I want to acquaint you with the list of reviewed software (February 2016):

1. Replacing full virtualization: Docker , LXC , OpenVZ , kvmtool .
2. Orchestration and cluster management: Kubernetes , Nomad , Openshift , Openstack , Swarm .
3. Jenkins support: Docker plugin , Kubernetes plugin , Openshift plugin

Some tools were poorly managed or required too much support and maintenance, others were poorly integrated with Jenkins and did not provide the necessary functionality, others were just beginning their way. The clear leader was Kubernetes and its derivatives (of which there are not few), but in its pure form the platform from Google was difficult to learn and deploy (the developers make great efforts to simplify the use of this platform).

In your own words about Openshift Origin:

Openshift Origin is primarily an Open Source platform for developing and publishing software, which is based on the Kubernetes container orchestration platform. The platform extends the capabilities of Kubernetes with the help of special objects and mechanisms.

Cluster:

A cluster can be deployed or updated using Ansible scripts. Inside the cluster can be used as a built-in and external Docker register. Cluster nodes can be assigned on the basis of special labels for individual projects. There is a garbage collector and a custom scheduler . A cluster can be deployed inside Openstack and integrated with it in order to automatically scale and provide data storage. Environments can be aware of published services and other nodes using environment variables and DNS names. It is possible to check the availability of services through an HTTP / TCP request and / or through the execution of a command inside the container. Cluster resources can be quotated at the project level (processor, memory, number of objects, etc.). There is an opportunity to monitor the cluster at the level of containers and working nodes.

Data:

Temporary or permanent volumes, which are mounted directly in containers, can be used as data storage methods. Backend for these volumes can be: NFS, GlusterFS, OpenStack Cinder, Ceph RBD, AWS Elastic Block Store (EBS), GCE Persistent Disk, etc. There is a possibility (hostPath) of mounting the local directory of the work node on which the container is running.

Network:

By default, all communications are carried out using Open vSwitch , but there is support for other network solutions through the Kubernetes plugin system. By default, all environments can communicate with each other, but isolation at the project level is possible. DNS resolution of service names is supported using the built-in SkyDNS service. Services published within the cluster can be accessed externally. Iptables assumes all the functionality of isolation and NAT functions.

Security:

Differentiation of user rights based on roles. Isolating containers with SELinux (and not only). Support for secrets that are used to access various resources. All communications between the working nodes of the cluster and the master are carried out through encrypted connections (CA is created, client certificates are issued).

Control:

APIs are available for both Openshift itself and Kubernetes. A cross-platform console client is available . A web-based interface is available, with the help of which users can: work in isolated projects according to their authority, interact with containers running in a cluster, view created objects, track events, etc.

Conclusion:

Despite the abundance of technologies and products that are on the market, finding the right tool is quite difficult. Balancing between the unification and integration of CI / CD processes, given the complexity of maintenance, tracking the chains of the technologies involved, Kubernetes and Openshift Origin were found. Unlike the base platform of the orchestration with Kubernetes containers, Openshift introduces the necessary elements for convenient and efficient work.