#DockerCon #DockerCon #DockerCon Production and Persistence for Containers
At the end of June 2015, a conference for Docker developers was held in San Francisco, USA. We bring to your attention the story of EMC engineer Ryan Wollner about how DockerCon 2015 passed, about the get-together and the ecosystem around the conference.
I have been involved in the open source community for more than 5 years, I have been working in network technologies, PaaS infrastructure and containerization technologies. Participation in projects such as OpenFlow / SDN, OpenStack was held with varying success, but Docker objectively collects now the maximum attention from the technical community.
Developers around the world pay a lot of attention to embedded architectures, logging, monitoring, data migration, networking, and running stateful services in containers. Before diving deeper into the topic of persistence volumes and showing the results of the work of my team, I want to highlight several topical topics of the conference and the community as a whole.
')
One of the themes of DockerCon 2015 was the use of Docker in production. Depending on who you ask about your experience with Docker and exactly how it was used, the answers will be different. Production using containers with microservices can be “hell”, as Brian Cantrill put it (if you haven’t seen Brian’s talk about Unix philosophy and debugging at the implementation stage, I highly recommend watching any video recordings of his reports, especially from O'Reilly conference, which was held on the eve of DockerCon). But, on the other hand, it helps to split applications into limited domains with a high degree of controllability, and development teams are able to efficiently carry out the CI / CD debugging / delivery / deployment process. Open source code Netflix on github is a confirmation of this, and Andrian Cockcroft rather deeply delved into the topic. You can also see my post about microservices, which will help you understand the context.
Is Docker good for sales? Below I will describe the moments that you need to know before using Docker in production. I would also like to add that each of the items is a separate big DockerCon theme.
The interaction between the Docker company and the Socketplane startup resulted in the possibility of using Libnetwork technology in projects, which allows you to connect Docker daemons from different hosts, which simplifies the transfer of IP traffic to the container on top of the data link layer of the network. Libnetwork is supported from outside the main Docker daemon, does not depend on the network implementation and, most importantly, hides the implementation details from the user, allowing the entire system to “just work”.
I spoke with Eric Windisch of Docker, who said that the security of the Docker engine had always received key attention, since any vulnerability in it would mean a threat to the container. Large-scale work is underway to improve the code in the security subsystem, as well as other tools based on AppArmor and SELinux are used . It is very curious how the Docker security subsystem will unfold in conjunction with projects such as VMWare Lightwave .
Containers are, of course, cool, but by running a thousand or ten thousand containers at the same time, you realize the need for a tool that would allow you to effectively debug, monitor and manage them. And it seems that we are waiting for a decision. Project Orca, announced by Docker founder Solomon Hicks , aims to connect the Docker Engine, Docker Swarm, GUI, Docker Compose, as well as security, installation, deployment and configuration tools. Of course, it is not necessary to run Docker through Orca, but it would be great to have a way to get all of the above, without losing speed and control. Other tools (loggly, cadvisor , ruzit, datalog, log entries) are competing for a place on this list, and this is just great!
Docker gives the initiative to the community, stating that it wants to be an extensible tool that allows various plugins to work with the network, authentication, data warehouses. Currently there is VMWare Lightwave for authentication and EMC ScaleIO for storage persistence. Not so bad, considering that Docker is just two years old!
EMC and ClusterHQ did a great job together by creating the open source project Flocker . It allows you to manage container volumes and provides data migration and high availability of volumes in situations where you need to restore containerized applications.
Also of great interest at the conference was the announcement of extensions of Docker's native volumes. Despite the experimental status, starting from version 1.7.x, these extensions provide various options for managing services while maintaining internal state. Flocker itself can work with Docker-volume extensions using the flag volume-driver = flocker, if you use the Flocker-Docker plugin .
The result of interaction with ClusterHQ was the publication on GitHub of the source of drivers that allow working with EMC ScaleIO and EMC XtremIO . In the Pivotal lab in San Francisco, we demonstrated the full capabilities of the drivers. As a result, people gathered to watch demos for beer and conversations in an informal setting. This link is a demo of ScaleIO, which we showed at a meeting dedicated to the work of Flocker + ScaleIO, deployed on Amazon AWS using the MEAN-stack of applications that have downloaded data from Twitter and placed in MongoDB. Slides from the meeting are also available here .
Not all DockerCon guests had enough seats, many listeners stood or sat on the floor for the entire session. We hoped that we would have a little more time so that the guys with laptops had the opportunity to hack our demo, but 40 minutes were not enough for us. They did everything they could!
Through a native Go implementation called RexRay , EMC integrates persistence volume management paths, but without the automatic data migration that Flocker produces. The possibilities of RexRay are quite wide, for example, you can use several backends at the same time, say, EC2 EBS + EMC ScaleIO.
In general, persistence and containers continue to be relevant, and for good reason. In the beginning, processes that did not preserve the internal state ( stateless ) and the methodology of the twelve factors were in trend. But all this is divorced from reality. The community understands that states exist, and the launch of stateful processes in containers, for example, databases, is really relevant for the world of microservices. Each container is in some state, even if it is a “stateless” state. For example, there may be states in memory such as executable applications and open sockets with which you need to work in certain cases, for example, during data migrations on the fly.
Data becomes extremely important in a container environment. As the mapping of the container architecture mappings, it becomes necessary to add consistency of data. Integrity and availability will be required for services that maintain internal state. Add to this all more industrial storage. The conference proposed various solutions to this problem, here are some of them:
• Crate.io . Scalable distributed database that is installed on the application server.
• ClusterHQ / Flocker. An open source container volume manager that allows you to containerize databases and other stateful processes, as well as move them without worrying about the backend.
• Red Hat announced the integration of persistent storage for containers using RHS (Red Hat Storage).
• Nutanix spoke about the Volume API, which will help ensure the sustainability of containers on their platform.
• Portworx. It seems this is a new startup that will compete with ClusterHQ. The announcement tells how the “part” of their platform will become open source.
• Kubernetes support for repositories. Kubernetes updated the GitHub documentation , which reflects the ability to use the Google Cloud persistant disk with k8s.
In general, it was a great DockerCon, full of fun events, great guys and innovative technologies. Hope to see you in Barcelona in November !
I have been involved in the open source community for more than 5 years, I have been working in network technologies, PaaS infrastructure and containerization technologies. Participation in projects such as OpenFlow / SDN, OpenStack was held with varying success, but Docker objectively collects now the maximum attention from the technical community.
Developers around the world pay a lot of attention to embedded architectures, logging, monitoring, data migration, networking, and running stateful services in containers. Before diving deeper into the topic of persistence volumes and showing the results of the work of my team, I want to highlight several topical topics of the conference and the community as a whole.
')
One of the themes of DockerCon 2015 was the use of Docker in production. Depending on who you ask about your experience with Docker and exactly how it was used, the answers will be different. Production using containers with microservices can be “hell”, as Brian Cantrill put it (if you haven’t seen Brian’s talk about Unix philosophy and debugging at the implementation stage, I highly recommend watching any video recordings of his reports, especially from O'Reilly conference, which was held on the eve of DockerCon). But, on the other hand, it helps to split applications into limited domains with a high degree of controllability, and development teams are able to efficiently carry out the CI / CD debugging / delivery / deployment process. Open source code Netflix on github is a confirmation of this, and Andrian Cockcroft rather deeply delved into the topic. You can also see my post about microservices, which will help you understand the context.
Is Docker good for sales? Below I will describe the moments that you need to know before using Docker in production. I would also like to add that each of the items is a separate big DockerCon theme.
Network
The interaction between the Docker company and the Socketplane startup resulted in the possibility of using Libnetwork technology in projects, which allows you to connect Docker daemons from different hosts, which simplifies the transfer of IP traffic to the container on top of the data link layer of the network. Libnetwork is supported from outside the main Docker daemon, does not depend on the network implementation and, most importantly, hides the implementation details from the user, allowing the entire system to “just work”.
Security
I spoke with Eric Windisch of Docker, who said that the security of the Docker engine had always received key attention, since any vulnerability in it would mean a threat to the container. Large-scale work is underway to improve the code in the security subsystem, as well as other tools based on AppArmor and SELinux are used . It is very curious how the Docker security subsystem will unfold in conjunction with projects such as VMWare Lightwave .
Logging, monitoring, manageability
Containers are, of course, cool, but by running a thousand or ten thousand containers at the same time, you realize the need for a tool that would allow you to effectively debug, monitor and manage them. And it seems that we are waiting for a decision. Project Orca, announced by Docker founder Solomon Hicks , aims to connect the Docker Engine, Docker Swarm, GUI, Docker Compose, as well as security, installation, deployment and configuration tools. Of course, it is not necessary to run Docker through Orca, but it would be great to have a way to get all of the above, without losing speed and control. Other tools (loggly, cadvisor , ruzit, datalog, log entries) are competing for a place on this list, and this is just great!
Extensibility
Docker gives the initiative to the community, stating that it wants to be an extensible tool that allows various plugins to work with the network, authentication, data warehouses. Currently there is VMWare Lightwave for authentication and EMC ScaleIO for storage persistence. Not so bad, considering that Docker is just two years old!
Persistent-services with preservation of internal state
EMC and ClusterHQ did a great job together by creating the open source project Flocker . It allows you to manage container volumes and provides data migration and high availability of volumes in situations where you need to restore containerized applications.
Also of great interest at the conference was the announcement of extensions of Docker's native volumes. Despite the experimental status, starting from version 1.7.x, these extensions provide various options for managing services while maintaining internal state. Flocker itself can work with Docker-volume extensions using the flag volume-driver = flocker, if you use the Flocker-Docker plugin .
The result of interaction with ClusterHQ was the publication on GitHub of the source of drivers that allow working with EMC ScaleIO and EMC XtremIO . In the Pivotal lab in San Francisco, we demonstrated the full capabilities of the drivers. As a result, people gathered to watch demos for beer and conversations in an informal setting. This link is a demo of ScaleIO, which we showed at a meeting dedicated to the work of Flocker + ScaleIO, deployed on Amazon AWS using the MEAN-stack of applications that have downloaded data from Twitter and placed in MongoDB. Slides from the meeting are also available here .
Not all DockerCon guests had enough seats, many listeners stood or sat on the floor for the entire session. We hoped that we would have a little more time so that the guys with laptops had the opportunity to hack our demo, but 40 minutes were not enough for us. They did everything they could!
Through a native Go implementation called RexRay , EMC integrates persistence volume management paths, but without the automatic data migration that Flocker produces. The possibilities of RexRay are quite wide, for example, you can use several backends at the same time, say, EC2 EBS + EMC ScaleIO.
In general, persistence and containers continue to be relevant, and for good reason. In the beginning, processes that did not preserve the internal state ( stateless ) and the methodology of the twelve factors were in trend. But all this is divorced from reality. The community understands that states exist, and the launch of stateful processes in containers, for example, databases, is really relevant for the world of microservices. Each container is in some state, even if it is a “stateless” state. For example, there may be states in memory such as executable applications and open sockets with which you need to work in certain cases, for example, during data migrations on the fly.
Data becomes extremely important in a container environment. As the mapping of the container architecture mappings, it becomes necessary to add consistency of data. Integrity and availability will be required for services that maintain internal state. Add to this all more industrial storage. The conference proposed various solutions to this problem, here are some of them:
• Crate.io . Scalable distributed database that is installed on the application server.
• ClusterHQ / Flocker. An open source container volume manager that allows you to containerize databases and other stateful processes, as well as move them without worrying about the backend.
• Red Hat announced the integration of persistent storage for containers using RHS (Red Hat Storage).
• Nutanix spoke about the Volume API, which will help ensure the sustainability of containers on their platform.
• Portworx. It seems this is a new startup that will compete with ClusterHQ. The announcement tells how the “part” of their platform will become open source.
• Kubernetes support for repositories. Kubernetes updated the GitHub documentation , which reflects the ability to use the Google Cloud persistant disk with k8s.
In general, it was a great DockerCon, full of fun events, great guys and innovative technologies. Hope to see you in Barcelona in November !
Source: https://habr.com/ru/post/263055/
All Articles