HPE invests in containers
Today, many technology companies are paying close attention to container technology. Among them - Google, IBM, Microsoft and of course HPE. Containers allow you to "pack" into one physical server a lot more applications than virtual machines allow you to do. Container technologies are not new, for example, many people know about them from the development of the company Parallels, but they play an increasingly important role in data centers and clouds. About them will be discussed.
A little bit of those containers, which are discussed, but we must also draw attention :)
Like many companies in the world of high technology, the company Hewlett Packard Enterprise is engaged in venture capital investments. One recent example is investment in the developer of market-leading software-defined object storage Scality. Another example of an investment in advanced technology is support for the startup Mesosphere, the developer of the so-called DC / OS.
The DC / OS (Data Center Operating System) operating system, previously known as DCOS, has recently moved to the Open Source category. The San Francisco Mesosphere development company decided to open the source code of its core.
')
Mesosphere developed the commercial version of DCOS a couple of years ago. Based on the Apache Mesos core, DC / OS simplifies the management of heterogeneous workloads on platforms such as Hadoop, Spark, Kafka and Docker. To date, Mesosphere DC / OS has become a collective project involving more than 50 companies, including Autodesk, Canonical, Cisco, Citrix, EMC, HPE, Joyent, Microsoft, NetApp, and Verizon.
Containers and microservices began to gain popularity in connection with the development and deployment of distributed applications. As it turned out, containerization gives a number of advantages. Applications in containers are called native cloud applications.
What is so good about DC / OS? Although containers can achieve good scalability, they have a drawback - it is difficult to achieve tight integration with existing applications, such as databases, data warehouses and platforms for big data. DC / OS is designed to control different types of applications in the same environment. It bridges the gap between traditional business applications and modern cloud applications and provides orchestration.
For example, in DC / OS, you can work with horizontally scalable Web applications in containers that interact with Apache Hadoop or Cassandra clusters, use Marathon to orchestrate container applications, and Chronos to plan long-term tasks. Such a unified environment allows for the implementation of scalable container applications along with traditional ones. This is the main feature of DC / OS.
Although Docker is traditionally used to create and deploy containers, an additional level is needed to orchestrate container applications. Docker Swarm and Kubernetes are such container management solutions at a higher level. Docker Swarm is developed and maintained by Docker, and Kubernetes is the result of translating into Open Source a subset of the data center management tool developed by Google called Borg. Interestingly, DC / OS can manage both Docker Swarm and Kubernetes environments.
The DC / OS operating system spans all servers in a data center or in the cloud and implements a powerful level of resource abstraction. The underlying Apache Mesos DC / OS core for distributed systems includes a scheduler that creates a pool of resources, automatically distributes them and schedules tasks according to requests and policies.
In essence, the data center works like one big computer. The Mesosphere technology can be seen as a kind of brain that automatically distributes resources and data for applications.
DC / OS provides flexible deployment options for big data applications, services, and infrastructure using shared resources. In such a scalable environment, you can run different applications and workloads - from PaaS to big data applications and DBMS. It runs DC / OS in any modern Linux environment, in a private and public cloud, in virtual machines, on x86 bare metal, ensuring efficient use of resources.
Last year, the Cloud Native Computing Foundation (CNCF) consortium was formed under the leadership of Google to promote native cloud applications. Joyent, CoreOS, IBM, VMWare, Cisco, Weaveworks and others became its members. The CNCF Consortium is managed by the Linux Foundation. Mesosphere is one of the founders of this consortium.
Mesosphere did not create a separate DC / OS consortium, but it supports the community of relevant software developers. If the CNCF consortium is more associated with Kubernetes, then the issues of DC / OS development and promotion are on the agenda of the community led by Mesosphere and Microsoft. Experts believe that over time, DC / OS can become a real alternative to Kubernetes.
HPE and Microsoft are Mesosphere's key investors. Their investments amounted to $ 73.5 million, with the leading role played by Hewlett Packard Enterprise. DC / OS technology will play an important role in Microsoft hybrid clouds and may be involved in the future platform for the private Microsoft Azure Stack cloud. Thanks to the Azure Container Service in the public cloud and DC / OS in the private cloud, Microsoft will be able to offer corporate customers the capabilities of a hybrid cloud. In DC / OS enterprise data centers, other technologies can also be used, such as HDInsight in Windows Server.
Developments Mesosphere are reflected in the Microsoft Azure Container Service (ACS). Azure Container Service greatly accelerates software development. Microsoft has made its container orchestration service available at the same time as the Mesosphere DC / OS announcement. ACS is a CaaS service (“containers as a service”), competing with the Google Container Engine and Amazon EC2 Container Service. However, neither Amazon nor Microsoft are members of the Cloud Native Computing Foundation. Amazon CaaS uses a proprietary orchestration mechanism, and Azure Container Service supports Docker Swarm and DC / OS. Kubernetes they do not use.
Verizon uses the Mesosphere DC / OS to manage its data centers, and Apple, Twitter and Airbnb use this technology to work with large amounts of data. For example, Apple used it in its Siri virtual assistant.
Undoubtedly, DC / OS should play an important role in the management of corporate cloud data centers. This explains HP’s investment. Being involved in products such as HP Helion and Microsoft Azure Stack, it should help corporate customers move to the cloud model while maintaining their investments in the data center.
By deploying Apache Spark, Kafka, Cassandra and Zeppelin to DC / OS, you can get an optimized environment for large-scale data processing, the resources of which are simultaneously available for other workloads, for example, for Web servers or Java applications.
What does DC / OS deployment on HPE ProLiant and Cloudline servers do? There are a lot of advantages:
• Scalability, flexibility and automation. The DC / OS scheduler allows you to create a pool of distributed loads (DC / OS services). The tasks of deploying services, scaling them up and managing them are reduced to simple commands, and the provision of security and business continuity is greatly simplified. HPE ProLiant servers have a flexible architecture and an extensible I / O subsystem, so they are well suited for this environment. Hyper-scaled HPE Cloudline servers are specifically designed for providers. They provide for rapid deployment, operational flexibility and low TCO rates.
• Accelerated results from new services. DC / OS allows enterprises to easily deploy and scale services (and several teams can participate in this), use Jenkins integration tools, use object repositories and code control tools.
• Flexible infrastructure management. ProLiant servers support management and software updates such as HPE OneView and Smart Update, integrated remote management and monitoring with HPE iLO. Cloudline servers are designed for open management tools and standard interfaces that easily fit into a heterogeneous environment with platforms from different vendors. HPE is one of the founders of the open Redfish specification for software-defined data centers. This standard is largely based on the implementation of the RESTful API for HPE iLO. The Redfish API can be used with HPE ProLiant and Cloudline servers.
• Moving, integrating, and delivering applications between different environments. HPE adheres to open standards, API-based and Open Source architectures, including Cloud Foundry, DC / OS, and OpenStack. Whatever the cloud, whether private, public or hybrid, it has an extensive ecosystem of hardware, software and services.
• Fast analysis of heterogeneous data. DC / OS supports services such as Kafka, Spark and Cassandra, often used in the Internet of things and big data analytics. This allows enterprises to get a simple and complete solution for big data applications.
In this example, ProLiant DL380 Gen 9 servers with the following configuration were used to orchestrate containers:
The DS / OS logical architecture with Marathon looks like this:
Marathon is a resource scheduler in a cluster. It provides high scalability of applications and services in a DC / OS environment. Marathon runs on top of DC / OS, where you have your own scheduler for starting / stopping tasks or application services. In this case, Marathon performs the functions of a higher level - it carries out monitoring and orchestration.
DC / OS ships with Marathon Failover Service for long-running applications. In this example, it is the one that is used to orchestrate containers — Container Orchestration. The marathon-lb service provides load balancing for Marathon applications. It starts like this:
Now you can use a cluster DC / OS environment with marathon-lb on the DC / OS Public Agent. Marathon applications that are configured for load balancing will see the corresponding addresses and ports. The Marathon user interface looks like this:
DC / OS supports both native Linux containers and Docker containers. The language of the Marathon application specification is the same.
The following Python example shows the configuration of a Marathon application for a Web server inside a native Linux container. This application serves the Corporate Cafeteria Menu of Load Balancer VIP. Used balancer Marathon Load Balancer. As shown below, the definition of a Marathon application is a simple file simple.json.
The application can be launched from the Marathon UI or from the DC / OS CLI command:
This is what the Marathon UI Container Orchestration looks like with the Web Menu application. Application Definition The Corporate Menu contains instructions for connecting this application to the Load Balancer balancer and indicates which TCP port is open for the Web Service.
Now let's test an application that accesses the Load Balancer VIP through the specified service port (10001). To execute the Load Balancer, use the DC / OS Public Agent at 10.250.40.138. Marathon Load Balancer shows the following Container Orchestration user interface of the Web Menu application:
Now you can scale the application to 1000 copies. Marathon will run them, tracking each execution and showing the status in the UI. Instances are distributed across all available DC / OS agents. The load on the DC / OS cluster increases:
Now let's test the Java application using Docker. After running 1000 instances in the Corporate Web Menu, create a second Marathon application. It uses a Docker image that installs the Java Runtime Environment. This application will run the Apache FTP Server. The definition of a Marathon application for a Java application in a Docker container looks like this:
Run the application with the command DC / OS CLI:
After launching a single instance of a Java application for Docker, we get in Marathon UI:
Scale it to 250 copies:
Unlike the Web server example, the Load Balancer balancer is not used for this application. In this case, Mesos-DNS as part of DC / OS is used as a Service Discovery tool. To determine the IP-port involved Marathon UI. For a single instance of a Java application, the Marathon UI shows the following information:
Now we use an FTP client to test one of 250 instances of a Java application:
So, in this example, a large number of application instances were launched in a DC / OS cluster using two different container technologies. This is done in minutes, and scaling applications is a matter of a few seconds.
In addition, a high utilization rate of the DC / OS cluster is achieved, the ROI is improved.
So let's summarize. Datacenter Operating System (DC / OS) is an open platform for running enterprise-class applications based on the Apache Mesos core. The use of DC / OS by corporate customers and cloud providers with HPE ProLiant and Cloudline servers allows you to create a hyper-scalable environment with open source software. Businesses get the flexibility to deploy distributed applications on reliable HPE systems. DC / OS helps to cope with the growing complexity of abstracted (virtualized, containerized, etc.) IT infrastructure, it can orchestrate all types of resources in the data center, including physical and virtual servers.
Containers run at the same level as physical servers. The lack of virtualized hardware and the use of real hardware and drivers provide unrivaled performance. And each container can scale to the resources of the entire physical server.
Virtualization technology at the OS level allows you to achieve the highest density among the available virtualization solutions. It is possible to create and launch hundreds of containers on one standard physical server. Containers use a single OS, which simplifies their support and updating. Applications can also be deployed in a separate environment.
Containers today are often viewed as an alternative to virtualization. In hardware virtualization, the base layer is the hypervisor. It is loaded on the server and provides interaction between its hardware and virtual machines. To provide resources to virtual machines, they are virtualized on the server. Virtual machines run their own copy of the operating system and applications on virtualized hardware.
VMs require significant system resources. Each virtual machine contains not just a copy of the OS, but a virtual copy of all the necessary hardware. This requires CPU and RAM resources. A container for executing a specific application is sufficient operating system, supporting programs and libraries. In practice, this means that the server can support two to three times more applications than in the case of VMs. Are you already using containers in your data center?
A little bit of those containers, which are discussed, but we must also draw attention :)
Like many companies in the world of high technology, the company Hewlett Packard Enterprise is engaged in venture capital investments. One recent example is investment in the developer of market-leading software-defined object storage Scality. Another example of an investment in advanced technology is support for the startup Mesosphere, the developer of the so-called DC / OS.
What is DC / OS and how is it good?
The DC / OS (Data Center Operating System) operating system, previously known as DCOS, has recently moved to the Open Source category. The San Francisco Mesosphere development company decided to open the source code of its core.
')
Mesosphere developed the commercial version of DCOS a couple of years ago. Based on the Apache Mesos core, DC / OS simplifies the management of heterogeneous workloads on platforms such as Hadoop, Spark, Kafka and Docker. To date, Mesosphere DC / OS has become a collective project involving more than 50 companies, including Autodesk, Canonical, Cisco, Citrix, EMC, HPE, Joyent, Microsoft, NetApp, and Verizon.
Containers and microservices began to gain popularity in connection with the development and deployment of distributed applications. As it turned out, containerization gives a number of advantages. Applications in containers are called native cloud applications.
What is so good about DC / OS? Although containers can achieve good scalability, they have a drawback - it is difficult to achieve tight integration with existing applications, such as databases, data warehouses and platforms for big data. DC / OS is designed to control different types of applications in the same environment. It bridges the gap between traditional business applications and modern cloud applications and provides orchestration.
For example, in DC / OS, you can work with horizontally scalable Web applications in containers that interact with Apache Hadoop or Cassandra clusters, use Marathon to orchestrate container applications, and Chronos to plan long-term tasks. Such a unified environment allows for the implementation of scalable container applications along with traditional ones. This is the main feature of DC / OS.
Although Docker is traditionally used to create and deploy containers, an additional level is needed to orchestrate container applications. Docker Swarm and Kubernetes are such container management solutions at a higher level. Docker Swarm is developed and maintained by Docker, and Kubernetes is the result of translating into Open Source a subset of the data center management tool developed by Google called Borg. Interestingly, DC / OS can manage both Docker Swarm and Kubernetes environments.
The DC / OS operating system spans all servers in a data center or in the cloud and implements a powerful level of resource abstraction. The underlying Apache Mesos DC / OS core for distributed systems includes a scheduler that creates a pool of resources, automatically distributes them and schedules tasks according to requests and policies.
In essence, the data center works like one big computer. The Mesosphere technology can be seen as a kind of brain that automatically distributes resources and data for applications.
DC / OS provides flexible deployment options for big data applications, services, and infrastructure using shared resources. In such a scalable environment, you can run different applications and workloads - from PaaS to big data applications and DBMS. It runs DC / OS in any modern Linux environment, in a private and public cloud, in virtual machines, on x86 bare metal, ensuring efficient use of resources.
Industry support
Last year, the Cloud Native Computing Foundation (CNCF) consortium was formed under the leadership of Google to promote native cloud applications. Joyent, CoreOS, IBM, VMWare, Cisco, Weaveworks and others became its members. The CNCF Consortium is managed by the Linux Foundation. Mesosphere is one of the founders of this consortium.
Mesosphere did not create a separate DC / OS consortium, but it supports the community of relevant software developers. If the CNCF consortium is more associated with Kubernetes, then the issues of DC / OS development and promotion are on the agenda of the community led by Mesosphere and Microsoft. Experts believe that over time, DC / OS can become a real alternative to Kubernetes.
HPE and Microsoft are Mesosphere's key investors. Their investments amounted to $ 73.5 million, with the leading role played by Hewlett Packard Enterprise. DC / OS technology will play an important role in Microsoft hybrid clouds and may be involved in the future platform for the private Microsoft Azure Stack cloud. Thanks to the Azure Container Service in the public cloud and DC / OS in the private cloud, Microsoft will be able to offer corporate customers the capabilities of a hybrid cloud. In DC / OS enterprise data centers, other technologies can also be used, such as HDInsight in Windows Server.
Developments Mesosphere are reflected in the Microsoft Azure Container Service (ACS). Azure Container Service greatly accelerates software development. Microsoft has made its container orchestration service available at the same time as the Mesosphere DC / OS announcement. ACS is a CaaS service (“containers as a service”), competing with the Google Container Engine and Amazon EC2 Container Service. However, neither Amazon nor Microsoft are members of the Cloud Native Computing Foundation. Amazon CaaS uses a proprietary orchestration mechanism, and Azure Container Service supports Docker Swarm and DC / OS. Kubernetes they do not use.
Verizon uses the Mesosphere DC / OS to manage its data centers, and Apple, Twitter and Airbnb use this technology to work with large amounts of data. For example, Apple used it in its Siri virtual assistant.
Undoubtedly, DC / OS should play an important role in the management of corporate cloud data centers. This explains HP’s investment. Being involved in products such as HP Helion and Microsoft Azure Stack, it should help corporate customers move to the cloud model while maintaining their investments in the data center.
By deploying Apache Spark, Kafka, Cassandra and Zeppelin to DC / OS, you can get an optimized environment for large-scale data processing, the resources of which are simultaneously available for other workloads, for example, for Web servers or Java applications.
DC / OS on HPE ProLiant and Cloudline servers
What does DC / OS deployment on HPE ProLiant and Cloudline servers do? There are a lot of advantages:
• Scalability, flexibility and automation. The DC / OS scheduler allows you to create a pool of distributed loads (DC / OS services). The tasks of deploying services, scaling them up and managing them are reduced to simple commands, and the provision of security and business continuity is greatly simplified. HPE ProLiant servers have a flexible architecture and an extensible I / O subsystem, so they are well suited for this environment. Hyper-scaled HPE Cloudline servers are specifically designed for providers. They provide for rapid deployment, operational flexibility and low TCO rates.
• Accelerated results from new services. DC / OS allows enterprises to easily deploy and scale services (and several teams can participate in this), use Jenkins integration tools, use object repositories and code control tools.
• Flexible infrastructure management. ProLiant servers support management and software updates such as HPE OneView and Smart Update, integrated remote management and monitoring with HPE iLO. Cloudline servers are designed for open management tools and standard interfaces that easily fit into a heterogeneous environment with platforms from different vendors. HPE is one of the founders of the open Redfish specification for software-defined data centers. This standard is largely based on the implementation of the RESTful API for HPE iLO. The Redfish API can be used with HPE ProLiant and Cloudline servers.
• Moving, integrating, and delivering applications between different environments. HPE adheres to open standards, API-based and Open Source architectures, including Cloud Foundry, DC / OS, and OpenStack. Whatever the cloud, whether private, public or hybrid, it has an extensive ecosystem of hardware, software and services.
• Fast analysis of heterogeneous data. DC / OS supports services such as Kafka, Spark and Cassandra, often used in the Internet of things and big data analytics. This allows enterprises to get a simple and complete solution for big data applications.
Example: Container Orchestration
In this example, ProLiant DL380 Gen 9 servers with the following configuration were used to orchestrate containers:
The DS / OS logical architecture with Marathon looks like this:
Marathon is a resource scheduler in a cluster. It provides high scalability of applications and services in a DC / OS environment. Marathon runs on top of DC / OS, where you have your own scheduler for starting / stopping tasks or application services. In this case, Marathon performs the functions of a higher level - it carries out monitoring and orchestration.
DC / OS ships with Marathon Failover Service for long-running applications. In this example, it is the one that is used to orchestrate containers — Container Orchestration. The marathon-lb service provides load balancing for Marathon applications. It starts like this:
$ dcos package install –yes marathon-lb Now you can use a cluster DC / OS environment with marathon-lb on the DC / OS Public Agent. Marathon applications that are configured for load balancing will see the corresponding addresses and ports. The Marathon user interface looks like this:
DC / OS supports both native Linux containers and Docker containers. The language of the Marathon application specification is the same.
The following Python example shows the configuration of a Marathon application for a Web server inside a native Linux container. This application serves the Corporate Cafeteria Menu of Load Balancer VIP. Used balancer Marathon Load Balancer. As shown below, the definition of a Marathon application is a simple file simple.json.
The application can be launched from the Marathon UI or from the DC / OS CLI command:
$ dcos marathon app add 0c-python-corpmenu-lb.json This is what the Marathon UI Container Orchestration looks like with the Web Menu application. Application Definition The Corporate Menu contains instructions for connecting this application to the Load Balancer balancer and indicates which TCP port is open for the Web Service.
Now let's test an application that accesses the Load Balancer VIP through the specified service port (10001). To execute the Load Balancer, use the DC / OS Public Agent at 10.250.40.138. Marathon Load Balancer shows the following Container Orchestration user interface of the Web Menu application:
Now you can scale the application to 1000 copies. Marathon will run them, tracking each execution and showing the status in the UI. Instances are distributed across all available DC / OS agents. The load on the DC / OS cluster increases:
Now let's test the Java application using Docker. After running 1000 instances in the Corporate Web Menu, create a second Marathon application. It uses a Docker image that installs the Java Runtime Environment. This application will run the Apache FTP Server. The definition of a Marathon application for a Java application in a Docker container looks like this:
Run the application with the command DC / OS CLI:
$ dcos marathon app add 10b-apacheftp-java-docker.json After launching a single instance of a Java application for Docker, we get in Marathon UI:
Scale it to 250 copies:
Unlike the Web server example, the Load Balancer balancer is not used for this application. In this case, Mesos-DNS as part of DC / OS is used as a Service Discovery tool. To determine the IP-port involved Marathon UI. For a single instance of a Java application, the Marathon UI shows the following information:
Now we use an FTP client to test one of 250 instances of a Java application:
Container Orchestration: Results
So, in this example, a large number of application instances were launched in a DC / OS cluster using two different container technologies. This is done in minutes, and scaling applications is a matter of a few seconds.
In addition, a high utilization rate of the DC / OS cluster is achieved, the ROI is improved.
Container Future - Today
So let's summarize. Datacenter Operating System (DC / OS) is an open platform for running enterprise-class applications based on the Apache Mesos core. The use of DC / OS by corporate customers and cloud providers with HPE ProLiant and Cloudline servers allows you to create a hyper-scalable environment with open source software. Businesses get the flexibility to deploy distributed applications on reliable HPE systems. DC / OS helps to cope with the growing complexity of abstracted (virtualized, containerized, etc.) IT infrastructure, it can orchestrate all types of resources in the data center, including physical and virtual servers.
Containers run at the same level as physical servers. The lack of virtualized hardware and the use of real hardware and drivers provide unrivaled performance. And each container can scale to the resources of the entire physical server.
Virtualization technology at the OS level allows you to achieve the highest density among the available virtualization solutions. It is possible to create and launch hundreds of containers on one standard physical server. Containers use a single OS, which simplifies their support and updating. Applications can also be deployed in a separate environment.
Containers today are often viewed as an alternative to virtualization. In hardware virtualization, the base layer is the hypervisor. It is loaded on the server and provides interaction between its hardware and virtual machines. To provide resources to virtual machines, they are virtualized on the server. Virtual machines run their own copy of the operating system and applications on virtualized hardware.
VMs require significant system resources. Each virtual machine contains not just a copy of the OS, but a virtual copy of all the necessary hardware. This requires CPU and RAM resources. A container for executing a specific application is sufficient operating system, supporting programs and libraries. In practice, this means that the server can support two to three times more applications than in the case of VMs. Are you already using containers in your data center?
Source: https://habr.com/ru/post/302892/
All Articles