Kubernetes success stories in production. Part 2: Concur and SAP
We continue to tell on the example of well-known companies that Kubernetes in production is not only dreams and hopes. This article is again about world-famous techies: SAP and its convergent cloud based on OpenStack and Kubernetes. But let's start with the less well-known Concur, and here's why ...
Concur Technologies is an American company that has existed since 1993 and today has 6,800 employees. Her business is SaaS for travel and cost management services for businesses. In 2010, she managed to get into the hundred best small companies according to Forbes magazine, and according to actual data (2017), about 70% of Forbes 100 and Forbes 500 lists use its services. In September 2014, it became known that SAP buys Concur Technologies for 8.3 billion USD, and the takeover was completed in December of the same year.
I decided to start the article about SAP with Concur, because this company, already a division of SAP, was an excellent example of one of the early serious users of Kubernetes in production. The first information about this appeared in May 2016. At the ContainerDays 2016 event held in Boston at the time, the leading engineer of Dale Ragan gave an interview in which he spoke about the experience of Concur.
')
The use of Kubernetes in the company began with a single piece of infrastructure - for a check management service. At that time, Concur was already working with Amazon Web Services and wanted to run its application both locally (on-premises) and in the cloud. To use EC2 Container Service, which was then in beta testing, they did not want to at Concur because of the emerging dependency on one supplier (Amazon).
One of the noticeable obstacles in the operation of Kubernetes was the lack of support for AWS Availability Zones for NoSQL-storage etcd - the problem was solved by the efforts of the company's engineers and promised to transfer to the Open Source-community. The result of the implementation of Kubernetes, according to the engineer Concur, was the “immutable infrastructure” (immutable infrastructure) - such that when you make changes to it, instead of modifying the existing cluster, a new one is created (and migration takes place there). This allowed them to "feel confident during migrations, when versioning is done for the entire infrastructure, just as new versions of the application are deployed." Prometheus and Grafana were introduced in the company to monitor performance, and the main problem for the Kubernetes project was that they didn’t provide good / detailed documentation for their implementation at Concur.
Speaking about this event in his blog, Dale Ragan presented a general outline of the company's infrastructure in AWS based on Kubernetes:
... and promised to tell more about this project, but for some reason this did not happen.
Despite this, the use of Kubernetes in Concur was actively pursued and developed. In the autumn of the same year, the senior architect Dan Wilson spoke at KubeCon 2016 with a report on scaling microservices using Kubernetes, and a year later at CoreOS Fest 2017 with a report ( PowerPoint presentation ) about using Federated Cluster Selector from Kubernetes 1.7.
In the last report, architect Concur explained the choice of Kubernetes for the following reasons:
Separate remarks are worth the fact that by 2017, Concur fell in love with CoreOS products (although they started with the use of only native Kubernetes components), including explaining this as “the best documentation for Kubernetes”. The architecture of clusters in the company received the following form:
In the same report, Dan Wilson presented one of the Open Source-projects Concur for Kubernetes - kubegowatcher . This is a plug-in template that is designed to run in a container inside Kubernetes and track changes in services, sub-nodes and nodes (with the ability to add your business logic to add / modify / delete events of events). Additionally, I note that the architect Concur makes his edits in the upstream Kubernetes itself.
There is no official confirmation that Concur has had a solid experience with Kubernetes from SAP. However, the general ideas were at least “in the air” - even after the purchase of Concur, the head of SAP expressed enthusiasm for “expanding the SAP cloud portfolio with leading travel and expense management solutions”.
Anyway, the first public mention of Kubernetes for SAP needs is met in the summer of 2016, when cloud infrastructure architect Darren Hague spoke at the DevOps Enterprise Summit 2016 (DOES16) in London with the report " SAP's DevOps Journey: From Building App to Building a Cloud " . The company came to DevOps in 2010–2013 (see the presentation of the same author at FlowCon San Francisco 2013) , and Kubernetes was the next logical step in its evolution.
Representing Kubernetes, Darren conducted an analogy with pets and livestock borrowed from a CERN employee: if the former “have their own names, are unique, receive care with love and restore health in the event of problems,” the latter “get numbers instead of names, are almost identical to each other and they are easily replaced in the case of diseases ” (I’d personally add that I don’t completely share this attitude towards animals, but this goes far beyond the article) . So Kubernetes, in the opinion of the SAP architect, is the ability to "handle containers like cattle, not pets."
One of the peculiarities of using the Kubernetes system in the SAP infrastructure was its use as a foundation, on top of which (that is, in the Docker containers of which) the cloud platform OpenStack was launched. Already in the middle of 2016, the company's task was announced to implement such a new platform in 13 regions and 18 data centers so that it would be used simultaneously “for SAP cloud services and for internal innovations”. Communication vision of the platform was as follows:
The penultimate item deserves special mention: the company's engineers created their own “automation agent” called Arc, which, unfortunately, there are not so many references on the Internet. The existing project on Launchpad describes the development as “a secure framework for remote execution of commands in OpenStack for setting up and running tasks on guest systems in Linux and Windows”, referring to the now missing repository in GitHub .
The statistics on the R & D of the created platform turned out to be interesting: less than 10% of the work on the project are categorized as "new developments", and more than 90% are improvements to the existing Open Source code. In particular, SAP experts have made changes to the upstream of such projects as OpenStack, Kubernetes, Docker and Grafana. (However, according to Stackalytics , SAP is not included in any significant top most active contributors either for OpenStack as a whole, nor for the Kubernetes component in particular - the check was made for OpenStack releases of 2016–2017.)
We will return to the details of this platform, but for now we will follow the chronology. In November of the same year, a report ( PDF ) of two SAP Labs employees on a pilot project to launch containers with enterprise-services to Kubernetes was heard at KubeCon 2016. From it, one could learn about the continuing interest of SAP engineers in translating monolithic applications to the microservice architecture and the available methods for its implementation. The common architecture in SAP is as follows:
A note to this slide states that SAP works with both HashiCorp and the Mesosphere, but the default container implementation chosen is Docker, and a specific report is dedicated to their use with Kubernetes and CoreOS. The result of the SAP Labs research project is that it is not finished at that time: there are a number of difficulties (mainly standardization and integration of various components and processes of the company) that engineers intend to overcome before the practice will be universally adapted. Separate mention in this project is the intention of SAP Labs "to release in the near future, Open Source-tool for managing Kubernetes - a tool that will be useful for developers and operating teams."
In March 2017, the history of the SAP cloud platform based on Kubernetes and OpenStack received a public continuation: the corporate blog announces the SAP Converged Cloud (SAPCC) production-status of the SAP. It is claimed that it is intended only for internal use in the company, but its scale is still impressive: the project combined 23 other cloud solutions used previously. There were so many of them for various reasons: “from fast innovation in different areas of business to the many acquisitions made by SAP in recent years” (companies such as SuccessFactors, Ariba and, of course, the already mentioned Concur, had their clouds at the time of their accession to SAP).
According to the announcement, "OpenStack is deployed 100% containerized using an automated system based on Kubernetes." Some details about the technical features and difficulties of Kubernetes can be found in the report (by the way, this video contains a 4-minute demonstration of how a new data center can be deployed in SAP Converged Cloud) by architect Michael Schmidt at the OpenStack Summit 2016:
The entire CI / CD pipeline to SAP (used by Concourse ) is integrated with containers from the cloud (using OpenStack Kolla ), and the package manager Helm is used to conveniently manage the infrastructure deployed by Kubernetes. GitHub publishes various Helm Charts: for deploying a basic IaaS based on OpenStack , as well as additional ones that are required for SAP Converged Cloud.
To work with SAPCC, a web interface has been created that allows you to manage cloud resources. The speaker emphasizes that in the server management interface, an additional terminal is also offered for executing the usual OpenStack console commands in it:
At the time of the announcement of the SAP Converged Cloud, only one availability zone in Australia was available with plans to add two more European ones for the next 2 weeks and to achieve cloud propagation to 19 data centers in 13 regions by the end of 2018. In the SAPCC project GitHub, you can find other interesting repositories of components used in the infrastructure - for example, operators for managing OpenStack on Kubernetes (we have written in more detail about operators for Kubernetes in this article ) or a utility for checking the network settings of the Kubernetes cluster.
Finally, I note that in the same year, SAP HANA DBMS was certified to run on the Google Cloud Platform (GCP).
Concur
Concur Technologies is an American company that has existed since 1993 and today has 6,800 employees. Her business is SaaS for travel and cost management services for businesses. In 2010, she managed to get into the hundred best small companies according to Forbes magazine, and according to actual data (2017), about 70% of Forbes 100 and Forbes 500 lists use its services. In September 2014, it became known that SAP buys Concur Technologies for 8.3 billion USD, and the takeover was completed in December of the same year.
I decided to start the article about SAP with Concur, because this company, already a division of SAP, was an excellent example of one of the early serious users of Kubernetes in production. The first information about this appeared in May 2016. At the ContainerDays 2016 event held in Boston at the time, the leading engineer of Dale Ragan gave an interview in which he spoke about the experience of Concur.
')
The use of Kubernetes in the company began with a single piece of infrastructure - for a check management service. At that time, Concur was already working with Amazon Web Services and wanted to run its application both locally (on-premises) and in the cloud. To use EC2 Container Service, which was then in beta testing, they did not want to at Concur because of the emerging dependency on one supplier (Amazon).
One of the noticeable obstacles in the operation of Kubernetes was the lack of support for AWS Availability Zones for NoSQL-storage etcd - the problem was solved by the efforts of the company's engineers and promised to transfer to the Open Source-community. The result of the implementation of Kubernetes, according to the engineer Concur, was the “immutable infrastructure” (immutable infrastructure) - such that when you make changes to it, instead of modifying the existing cluster, a new one is created (and migration takes place there). This allowed them to "feel confident during migrations, when versioning is done for the entire infrastructure, just as new versions of the application are deployed." Prometheus and Grafana were introduced in the company to monitor performance, and the main problem for the Kubernetes project was that they didn’t provide good / detailed documentation for their implementation at Concur.
Speaking about this event in his blog, Dale Ragan presented a general outline of the company's infrastructure in AWS based on Kubernetes:
... and promised to tell more about this project, but for some reason this did not happen.
Despite this, the use of Kubernetes in Concur was actively pursued and developed. In the autumn of the same year, the senior architect Dan Wilson spoke at KubeCon 2016 with a report on scaling microservices using Kubernetes, and a year later at CoreOS Fest 2017 with a report ( PowerPoint presentation ) about using Federated Cluster Selector from Kubernetes 1.7.
In the last report, architect Concur explained the choice of Kubernetes for the following reasons:
Separate remarks are worth the fact that by 2017, Concur fell in love with CoreOS products (although they started with the use of only native Kubernetes components), including explaining this as “the best documentation for Kubernetes”. The architecture of clusters in the company received the following form:
In the same report, Dan Wilson presented one of the Open Source-projects Concur for Kubernetes - kubegowatcher . This is a plug-in template that is designed to run in a container inside Kubernetes and track changes in services, sub-nodes and nodes (with the ability to add your business logic to add / modify / delete events of events). Additionally, I note that the architect Concur makes his edits in the upstream Kubernetes itself.
SAP: DevOps, Microservices, Kubernetes (2016)
There is no official confirmation that Concur has had a solid experience with Kubernetes from SAP. However, the general ideas were at least “in the air” - even after the purchase of Concur, the head of SAP expressed enthusiasm for “expanding the SAP cloud portfolio with leading travel and expense management solutions”.
Anyway, the first public mention of Kubernetes for SAP needs is met in the summer of 2016, when cloud infrastructure architect Darren Hague spoke at the DevOps Enterprise Summit 2016 (DOES16) in London with the report " SAP's DevOps Journey: From Building App to Building a Cloud " . The company came to DevOps in 2010–2013 (see the presentation of the same author at FlowCon San Francisco 2013) , and Kubernetes was the next logical step in its evolution.
Representing Kubernetes, Darren conducted an analogy with pets and livestock borrowed from a CERN employee: if the former “have their own names, are unique, receive care with love and restore health in the event of problems,” the latter “get numbers instead of names, are almost identical to each other and they are easily replaced in the case of diseases ” (I’d personally add that I don’t completely share this attitude towards animals, but this goes far beyond the article) . So Kubernetes, in the opinion of the SAP architect, is the ability to "handle containers like cattle, not pets."
One of the peculiarities of using the Kubernetes system in the SAP infrastructure was its use as a foundation, on top of which (that is, in the Docker containers of which) the cloud platform OpenStack was launched. Already in the middle of 2016, the company's task was announced to implement such a new platform in 13 regions and 18 data centers so that it would be used simultaneously “for SAP cloud services and for internal innovations”. Communication vision of the platform was as follows:
The penultimate item deserves special mention: the company's engineers created their own “automation agent” called Arc, which, unfortunately, there are not so many references on the Internet. The existing project on Launchpad describes the development as “a secure framework for remote execution of commands in OpenStack for setting up and running tasks on guest systems in Linux and Windows”, referring to the now missing repository in GitHub .
The statistics on the R & D of the created platform turned out to be interesting: less than 10% of the work on the project are categorized as "new developments", and more than 90% are improvements to the existing Open Source code. In particular, SAP experts have made changes to the upstream of such projects as OpenStack, Kubernetes, Docker and Grafana. (However, according to Stackalytics , SAP is not included in any significant top most active contributors either for OpenStack as a whole, nor for the Kubernetes component in particular - the check was made for OpenStack releases of 2016–2017.)
We will return to the details of this platform, but for now we will follow the chronology. In November of the same year, a report ( PDF ) of two SAP Labs employees on a pilot project to launch containers with enterprise-services to Kubernetes was heard at KubeCon 2016. From it, one could learn about the continuing interest of SAP engineers in translating monolithic applications to the microservice architecture and the available methods for its implementation. The common architecture in SAP is as follows:
A note to this slide states that SAP works with both HashiCorp and the Mesosphere, but the default container implementation chosen is Docker, and a specific report is dedicated to their use with Kubernetes and CoreOS. The result of the SAP Labs research project is that it is not finished at that time: there are a number of difficulties (mainly standardization and integration of various components and processes of the company) that engineers intend to overcome before the practice will be universally adapted. Separate mention in this project is the intention of SAP Labs "to release in the near future, Open Source-tool for managing Kubernetes - a tool that will be useful for developers and operating teams."
SAP: Converged Cloud (2017)
In March 2017, the history of the SAP cloud platform based on Kubernetes and OpenStack received a public continuation: the corporate blog announces the SAP Converged Cloud (SAPCC) production-status of the SAP. It is claimed that it is intended only for internal use in the company, but its scale is still impressive: the project combined 23 other cloud solutions used previously. There were so many of them for various reasons: “from fast innovation in different areas of business to the many acquisitions made by SAP in recent years” (companies such as SuccessFactors, Ariba and, of course, the already mentioned Concur, had their clouds at the time of their accession to SAP).
According to the announcement, "OpenStack is deployed 100% containerized using an automated system based on Kubernetes." Some details about the technical features and difficulties of Kubernetes can be found in the report (by the way, this video contains a 4-minute demonstration of how a new data center can be deployed in SAP Converged Cloud) by architect Michael Schmidt at the OpenStack Summit 2016:
The entire CI / CD pipeline to SAP (used by Concourse ) is integrated with containers from the cloud (using OpenStack Kolla ), and the package manager Helm is used to conveniently manage the infrastructure deployed by Kubernetes. GitHub publishes various Helm Charts: for deploying a basic IaaS based on OpenStack , as well as additional ones that are required for SAP Converged Cloud.
To work with SAPCC, a web interface has been created that allows you to manage cloud resources. The speaker emphasizes that in the server management interface, an additional terminal is also offered for executing the usual OpenStack console commands in it:
At the time of the announcement of the SAP Converged Cloud, only one availability zone in Australia was available with plans to add two more European ones for the next 2 weeks and to achieve cloud propagation to 19 data centers in 13 regions by the end of 2018. In the SAPCC project GitHub, you can find other interesting repositories of components used in the infrastructure - for example, operators for managing OpenStack on Kubernetes (we have written in more detail about operators for Kubernetes in this article ) or a utility for checking the network settings of the Kubernetes cluster.
Finally, I note that in the same year, SAP HANA DBMS was certified to run on the Google Cloud Platform (GCP).
Other articles from the cycle
- “ Kubernetes success stories in production. Part 1: 4,200 pods and TessMaster on eBay . ”
- “ Kubernetes success stories in production. Part 3: GitHub .
- “ Kubernetes success stories in production. Part 4: SoundCloud (by Prometheus) . ”
- “ Kubernetes success stories in production. Part 5: Monzo Digital Bank .
- “ Kubernetes success stories in production. Part 6: BlaBlaCar .
- “ Kubernetes success stories in production. Part 7: BlackRock .
- “ Kubernetes success stories in production. Part 8: Huawei .
- “ Kubernetes success stories in production. Part 9: CERN and 210 K8s clusters. ”
- “ Kubernetes success stories in production. Part 10: Reddit .
Source: https://habr.com/ru/post/334770/
All Articles