Kubernetes success stories in production. Part 1: 4200 pods and TessMaster on eBay

As often happens in the first years of life of infrastructure projects that are rapidly gaining popularity, while many are just eyeing Kubernetes, assessing its capabilities and maturity, others have time to move on, test and run in production (fully or partially), having received their first "adult" operating experience. This article begins a review cycle of examples from world practice of fairly well-known companies using Kubernetes in production.

Note : all the examples talk about using the original Kubernetes upstream distribution, not its derivatives like OpenShift (Red Hat) and Tectonic (CoreOS).
')
Let's start with eBay, whose specialists have been working seriously with Kubernetes for more than 2 years and have made significant progress in this ...

2015: the beginning

So, in 2015, one of the largest online stores on eBay (one of the top 10 global Internet companies in terms of profits) began to introduce Kubernetes. As Ashwin Raveendran from the eBay cloud team told KubeCon 2015, his company, which had more than 150,000 servers at the time, was one of the first public examples of implementing the Kubernetes system on top of the OpenStack cloud-based Open Source platform.

(Slide from Kubernetes on OpenStack at eBay presentation at OpenStack Day Seattle 2015)

The early stages of Kubernetes adaptation in eBay "were not only about switching to containers for deploying applications, but also about changing the life cycle of applications in the company," which was a response to the needs of developers. At the same time, eBay talked about plans to "move to a more flexible deployment model using containers as an executable environment and Kubernetes on top of OpenStack to manage these containers." The Kubernetes installation in eBay received the internal name Tess.IO, but it was not possible to find public information about the features of this fork (?). (There are no reports on any eBay plans to open the Tess.IO code: on the GitHub account of the company there is a fork of the Kubernetes codebase, which has not been updated since 2015.)

2016: first results

A year after the first mention of Tess.IO from the director of the company's cloud infrastructure (Suneet Nandwani), it became known that there were seven Kubernetes clusters in eBay, each of which served approximately 100 physical servers. To work with them, the company prepared a single administrative interface integrated with OpenStack and called TessMaster.

The new eBay project, still published on GitHub only as a small README without any source, was positioned as an alternative to OpenStack Magnum, the official API offered by OpenStack for integrating orchestration tools (Docker Swarm, Kubernetes and Apache Mesos) with a cloud platform. Magnum uses Heat (a key orchestration service in OpenStack to manage the lifecycle of infrastructure and applications) to orchestrate the operating system image and run it on virtual machines or on bare hardware in a cluster configuration.

From the words of Suneet Nandwani in 2016:

“We were very purposeful in adapting Kubernetes. EBay development teams love Docker containers for dev and test environments, and Kubernetes supports Docker and takes control of it. In addition to the dev and test environments, we started running production-applications in containers. Each application is represented by a set of containers that implement various components and dependencies. [..] This year we launched four applications in Kubernetes. We see enormous potential in how applications work and how much flexibility developers have. ”

As of the end of 2016, about 2000 containers in production at eBay were managed with Kubernetes, and this number was promised to increase "to tens of thousands in the near future."

Kubernetes or Apache Mesos?

What makes this case even more interesting: a year and a half before the first news about eBay and Kubernetes, in April 2014, the company's engineers spoke about their approach to continuous integration implemented with the help of direct competitor Kubernetes - Apache Mesos (and also Marathon, ZooKeeper, Jenkins):

Moreover, eBay is still on the current list of Organizations using Mesos . However , the InformationWeek material from December 28, 2016 refers to the words of the same Suneet Nandwani (director of cloud infrastructure and platforms on eBay), who claims that the company chose between the available orchestration solutions in 2015 *. Despite the fact that the developers appreciated the convenience of the containers when testing and delivering code, “launching Docker by itself [within the existing infrastructure] was very difficult” and required convenient maintenance. Docker Swarm at that time was not yet, and the choice between Apache Mesos and Kubernetes fell in favor of the latter, because this system was considered the most mature solution according to the results of internal testing along with Docker containers.

* The article indicates the year 2016, but another Suneet performance (see below) + the profile of one of the Tess.IO developers indicate that the active use of Kubernetes in eBay (the creation of Tess.IO) began all the same earlier, in 2015:

2017: Present and Perspectives

Speaking at the OpenStack Summit in Boston in May 2017, Suneet Nandwani said that Kubernetes in eBay is still running on top of the OpenStack private cloud and currently serves 178 applications in 4,200 subfields.

Note : The entire OpenStack cloud in eBay (as of May 2017) has 167,000 virtual machines and 4,000 applications, serving 95% of the traffic. This is one of the largest OpenStack installations in the world!

In the framework of the same report, an eBay specialist described in more detail the reasons for choosing Kubernetes. Those were called:

focus on applications (app centric) as opposed to (or rather, in addition) infrastructure orientation (in OpenStack);
Open Source (“We have a lot of experience with OpenStack, we also contribute to the development of this project”; “All large platforms that we use should be open”) ;
support for Docker containers (it became an obvious need for developers after the transition to the microservice architecture);
model driven architecture;
declarative approach;
active community;
advanced planning mechanisms;
geographical federation (“Being a large enterprise, we run everything in different data centers”) .

EBay engineers run Kubernetes and stateful , and stateless applications, and among specific applications are: an artificial intelligence platform, solutions for automating network management, ElasticSearch, Apache Kafka, distributed NoSQL solutions.

The main groups of problems that they had to face when introducing and operating Kubernetes are listed on this slide:

Some technical details of how to overcome them in the early stages of Kubernetes introduction can be heard in the 2015 Cloud-Scale Kubernetes at eBay report (Ashwin Raveendran at KubeCon 2015). A more complete form of the solution of some of these problems found in the development of its own product.

Tessmaster

The already mentioned TessMaster solution, created in eBay, is intended to ensure the management of the entire life cycle for Kubernetes clusters hosted by different providers (public / private cloud environments, bare metal). From virtualization (i.e., VMs that form the infrastructure in addition to the iron servers), TessMaster currently supports OpenStack and VirtualBox.

For the end user (DevOps Engineer), TessMaster offers a web interface in which the entire infrastructure is visually presented (Kubernetes clusters). It is divided into regions, availability zones, network circuits (for example, dev and prod). For each cluster, the resources used and available nodes are shown, which can be “entered” in order to see the list of virtual / physical servers and nodes in the cluster ... and literally in a couple of clicks to change the number of nodes (i.e., change the Kubernetes cluster configuration and see visually, “in real time”, how this change is triggered).

The most pleasant thing about TessMaster's presentation is the promise of eBay to publish its development as an Open Source-project “within a couple of months”. Given that the event took place in May, this event should be expected as soon as possible. Then we will learn about other features of this development, hiding behind the remaining tabs of its web interface, and current and potential major users of Kubernetes will get a very interesting tool for convenient operation of large-scale clusters.

PS

Subscribe to our blog in order not to miss the following articles from this cycle, and also pay attention to other publications already available: