Containerize it! What Fuel is and what it uses for Docker
While patent wars remain a hidden threat to the OpenStack ecosystem, let's talk about the technology that allows you to deploy OpenStack in almost one click. The name of this project was repeatedly met in the posts of our blog, but there was not a single text dedicated to Fuel . Meanwhile, it was this project that significantly simplified the deployment procedure of OpenStack and made the process of further cloud management less time consuming. Of course, you can act in the old manner. Using Fuel is not required to work with OpenStack. However, we believe that if the theater starts with a hanger, then OpenStack starts with Fuel. At least - Mirantis OpenStack (MOS).
Fuel is a tool for deploying OpenStack and then managing the cloud. It is difficult to name a project that would give more informational reasons. Right now we are telling you that Fuel now uses Docker containers, but we already know about the launch of support for plugins. And nevertheless at first we will talk about containers. On the eve of the first Russian Docker-mitap, such a conversation is more than justified.
Those of you who have already tried OpenStack (we are not just telling you what it is, but we hope to convince you to spend time experimenting with technology that two years ago attracted our attention and does not let it go until now) we could make sure that manual installation is a laborious, complicated and, most importantly, fraught with a large number of errors. Fuel is our answer to manual deployment problems. This is an intuitive tool equipped with a web-based user interface, which, in essence, is a control panel for deploying a cloud with a few clicks and further work with it. It is important to note that Fuel simplifies large-scale (to date, tested on the environment of 100 nodes) deployment, testing and support of OpenStack configurations. As one of the OpenStack ecosystem projects, the Fuel project is not related to any particular brand (even ours).
')
We will return to this development more than once. And now let's talk about Docker.
Starting with MOS 5.0, Mirantis staked on Docker in order to implement a simple and reliable transactional update. Fuel: we raise new containers, check, and if something goes wrong, we return the old ones. The speed has come along with the simplicity. Tell about how it was.
The changes we made allowed the user to re-deploy Fuel in less than 30 seconds without using scripts to return the environment to its original state. Now it is possible to make changes, test them promptly and, if necessary, “roll back” repeatedly.
The speed of change was the result of the use of “containers”, the scope of which is, in fact, very wide. The corresponding environment can be created on a laptop with Mac OS X or Windows (using boot2docker), on QA servers under Ubuntu in the cloud, and also on virtual machines in real data centers based on Red Hat Enterprise Linux.
To update the Fuel application in Docker, simply remove the old containers and start new ones, saving 1-2 hours you would need to re-assemble Fuel ISO and test the changes made.
In Mirantis OpenStack 5.1, we reduced the time required to install Fuel on the master node. With a virtualized deployment on an SSD disk, the loading time of the images was reduced twice - from 18 minutes to 9 minutes, and containers - from 9 to 8 minutes. When deployed in the lab (on physical servers), the image loading time in version 5.1 has decreased by 20%.
There was an idea to use temporary file storage (tmpfs) on the master node to reduce installation time, but we revealed its unreliability in environments with insufficient RAM, which slowed down the process. While our main task is to gain time.
But the stability of Fuel in version 5.1 was improved thanks to the dockerctl utility, which tracks the activity of settings for Fuel during the Docker update process, in which hundreds of parameters in 13 containers are transferred. The dockerctl utility is especially important in light of the fact that a large number of matches are established for applications that require 5-6 parameters from our configuration file (astute.yaml) for each container in order to deploy an application.
Instead of etcd or other tools requiring systemd, we used astute.yaml parameters, left a local YAML file, and launched Puppet inside containers to deploy applications on the Fuel master node. We will not necessarily do this in the future, but this method was simpler and less destructive than using other available means. However, the local YAML file format and Puppet had several flaws, for example, they contributed to establishing a large number of connections in Docker, which resulted in large disk resources for each container due to the installation of Puppet and dependent modules. But even so, we use in our work some of the most compact full-featured OS images of Docker. The size of the Ubuntu-based Docker image is huge, whereas the size of our CentOS-based image with Puppet installed is compressed to 39MB.
We have eliminated errors in the settings of Fuel and implemented a rotation of logs to free up disk space. In addition, when setting system requirements for 6.0, we took into account the problem with the logrotate utility, which by default saves the last five archives (this requires a disproportionate disk space, despite the fact that Docker itself needs disk space for log files that can fill all free space and crash the file system). This problem does not have an automatic solution, so even with the log storage improvements made by Mirantis, you still need to monitor the disk capacity, since we cannot predict how much logs you will generate. Despite the fact that logrotate updates the logs every hour, it does not track the free disk space, and the setting that would increase the size of the folder with Docker logs is missing. In fact, the logrotate settings limit the size of a single log file, after which it is archived. Therefore, even with increased parameters, you should allocate 30GB exclusively for logs (when deployed to 20 nodes). But if you turn on debug mode (Debug), large files will be rotated every 10 minutes, not every hour.
Other innovations for Fuel based on Docker in Mirantis OpenStack 6.0 include using CentOS 6.5 as the operating system for the master node.
Since CentOS 6.5 does not have a systemd initialization daemon that would constantly start services, Mirantis uses the Supervisor management utility to launch and monitor Docker containers, as well as launch web applications. “Two in one”, as advertisers like to say.
In addition, we use a simple “try-or-fail” scheme to solve
problems with stopping Docker containers that have no inherited dependency tracking tools. In our solution, if the Docker containers try to start and then stop due to the fact that PostgreSQL or RabbitMQ is not yet running, Supervisor waits for a few seconds and tries to restart the container, which failed to start up, until all the containers will be running. This method is much easier to maintain than a complex sequence of putting containers into action based on priorities or
dependencies. And, while dockerctl has the proper sequence when running all containers, this sequence only applies during initial deployment.
Once again, pay attention to the fact that Fuel is a project of the entire OpenStack ecosystem, so you can use this tool to deploy, test and support your OpenStack platform. Although, of course, we believe that the easiest way is to get it as part of the Mirantis OpenStack distribution . Still, we are responsible for our distribution and we can guarantee that its use will not damage the reputation of the ecosystem as a whole.
Since we ourselves were convinced that Docker is a useful tool, we want to share this knowledge. On February 26, from 6:00 pm to 9:00 pm, Mirantis participates in the first Docker meeting in Moscow and invites those who wish to join. Participation is free. More information about the event can be found on the website of the Russian community OpenStack , and you can register here . For those who wish to participate in the meeting - registration is required.
Fabrizio Soppelsa , an engineer at our Fuel support team , author of the Linux Journal and just an OpenStack enthusiast, will start the meeting. He will formulate a brief description of the Docker technology, describe the advantages of using this powerful platform and what makes it popular among developers.
Next, Matthew Mosesohn is a senior developer at Mirantis. Matthew will talk about the deployment of Docker “Off the Grid”. By the way, it was on his post in our corporate blog dedicated to the transition of Fuel to Docker that we relied in the preparation of this material.
Denis Zaitsev , head of the cloud platform and CDN operations team at Yandex, will tell you about the Docker Registry scaling; about launching Docker applications in the OpenStack-cloud with the help of Murano - Sergey Melikyan , senior developer at Mirantis.
Andrei Vagin , a developer on Parallels’s Linux Kernel team, will complete the mitap. Its theme is “Libcontainer: joining forces under one roof.” Andrei will make a description of the plans and the current status of this task.
You probably have (or appeared) questions. The event is the right place to ask. But if you for some reason do not get to the meeting, then write questions in the comments - we will try to call our experts to answer.
Some common words about Fuel
Fuel is a tool for deploying OpenStack and then managing the cloud. It is difficult to name a project that would give more informational reasons. Right now we are telling you that Fuel now uses Docker containers, but we already know about the launch of support for plugins. And nevertheless at first we will talk about containers. On the eve of the first Russian Docker-mitap, such a conversation is more than justified.
Those of you who have already tried OpenStack (we are not just telling you what it is, but we hope to convince you to spend time experimenting with technology that two years ago attracted our attention and does not let it go until now) we could make sure that manual installation is a laborious, complicated and, most importantly, fraught with a large number of errors. Fuel is our answer to manual deployment problems. This is an intuitive tool equipped with a web-based user interface, which, in essence, is a control panel for deploying a cloud with a few clicks and further work with it. It is important to note that Fuel simplifies large-scale (to date, tested on the environment of 100 nodes) deployment, testing and support of OpenStack configurations. As one of the OpenStack ecosystem projects, the Fuel project is not related to any particular brand (even ours).
')
We will return to this development more than once. And now let's talk about Docker.
Docker: saving time and effort
Starting with MOS 5.0, Mirantis staked on Docker in order to implement a simple and reliable transactional update. Fuel: we raise new containers, check, and if something goes wrong, we return the old ones. The speed has come along with the simplicity. Tell about how it was.
Retrospective
MOS 5.0
The changes we made allowed the user to re-deploy Fuel in less than 30 seconds without using scripts to return the environment to its original state. Now it is possible to make changes, test them promptly and, if necessary, “roll back” repeatedly.
The speed of change was the result of the use of “containers”, the scope of which is, in fact, very wide. The corresponding environment can be created on a laptop with Mac OS X or Windows (using boot2docker), on QA servers under Ubuntu in the cloud, and also on virtual machines in real data centers based on Red Hat Enterprise Linux.
To update the Fuel application in Docker, simply remove the old containers and start new ones, saving 1-2 hours you would need to re-assemble Fuel ISO and test the changes made.
MOS 5.1
In Mirantis OpenStack 5.1, we reduced the time required to install Fuel on the master node. With a virtualized deployment on an SSD disk, the loading time of the images was reduced twice - from 18 minutes to 9 minutes, and containers - from 9 to 8 minutes. When deployed in the lab (on physical servers), the image loading time in version 5.1 has decreased by 20%.
There was an idea to use temporary file storage (tmpfs) on the master node to reduce installation time, but we revealed its unreliability in environments with insufficient RAM, which slowed down the process. While our main task is to gain time.
But the stability of Fuel in version 5.1 was improved thanks to the dockerctl utility, which tracks the activity of settings for Fuel during the Docker update process, in which hundreds of parameters in 13 containers are transferred. The dockerctl utility is especially important in light of the fact that a large number of matches are established for applications that require 5-6 parameters from our configuration file (astute.yaml) for each container in order to deploy an application.
Instead of etcd or other tools requiring systemd, we used astute.yaml parameters, left a local YAML file, and launched Puppet inside containers to deploy applications on the Fuel master node. We will not necessarily do this in the future, but this method was simpler and less destructive than using other available means. However, the local YAML file format and Puppet had several flaws, for example, they contributed to establishing a large number of connections in Docker, which resulted in large disk resources for each container due to the installation of Puppet and dependent modules. But even so, we use in our work some of the most compact full-featured OS images of Docker. The size of the Ubuntu-based Docker image is huge, whereas the size of our CentOS-based image with Puppet installed is compressed to 39MB.
Status quo: MOS 6.0
We have eliminated errors in the settings of Fuel and implemented a rotation of logs to free up disk space. In addition, when setting system requirements for 6.0, we took into account the problem with the logrotate utility, which by default saves the last five archives (this requires a disproportionate disk space, despite the fact that Docker itself needs disk space for log files that can fill all free space and crash the file system). This problem does not have an automatic solution, so even with the log storage improvements made by Mirantis, you still need to monitor the disk capacity, since we cannot predict how much logs you will generate. Despite the fact that logrotate updates the logs every hour, it does not track the free disk space, and the setting that would increase the size of the folder with Docker logs is missing. In fact, the logrotate settings limit the size of a single log file, after which it is archived. Therefore, even with increased parameters, you should allocate 30GB exclusively for logs (when deployed to 20 nodes). But if you turn on debug mode (Debug), large files will be rotated every 10 minutes, not every hour.
Other innovations for Fuel based on Docker in Mirantis OpenStack 6.0 include using CentOS 6.5 as the operating system for the master node.
Since CentOS 6.5 does not have a systemd initialization daemon that would constantly start services, Mirantis uses the Supervisor management utility to launch and monitor Docker containers, as well as launch web applications. “Two in one”, as advertisers like to say.
In addition, we use a simple “try-or-fail” scheme to solve
problems with stopping Docker containers that have no inherited dependency tracking tools. In our solution, if the Docker containers try to start and then stop due to the fact that PostgreSQL or RabbitMQ is not yet running, Supervisor waits for a few seconds and tries to restart the container, which failed to start up, until all the containers will be running. This method is much easier to maintain than a complex sequence of putting containers into action based on priorities or
dependencies. And, while dockerctl has the proper sequence when running all containers, this sequence only applies during initial deployment.
Once again, pay attention to the fact that Fuel is a project of the entire OpenStack ecosystem, so you can use this tool to deploy, test and support your OpenStack platform. Although, of course, we believe that the easiest way is to get it as part of the Mirantis OpenStack distribution . Still, we are responsible for our distribution and we can guarantee that its use will not damage the reputation of the ecosystem as a whole.
First Docker-event in Moscow
Since we ourselves were convinced that Docker is a useful tool, we want to share this knowledge. On February 26, from 6:00 pm to 9:00 pm, Mirantis participates in the first Docker meeting in Moscow and invites those who wish to join. Participation is free. More information about the event can be found on the website of the Russian community OpenStack , and you can register here . For those who wish to participate in the meeting - registration is required.
Fabrizio Soppelsa , an engineer at our Fuel support team , author of the Linux Journal and just an OpenStack enthusiast, will start the meeting. He will formulate a brief description of the Docker technology, describe the advantages of using this powerful platform and what makes it popular among developers.
Next, Matthew Mosesohn is a senior developer at Mirantis. Matthew will talk about the deployment of Docker “Off the Grid”. By the way, it was on his post in our corporate blog dedicated to the transition of Fuel to Docker that we relied in the preparation of this material.
Denis Zaitsev , head of the cloud platform and CDN operations team at Yandex, will tell you about the Docker Registry scaling; about launching Docker applications in the OpenStack-cloud with the help of Murano - Sergey Melikyan , senior developer at Mirantis.
Andrei Vagin , a developer on Parallels’s Linux Kernel team, will complete the mitap. Its theme is “Libcontainer: joining forces under one roof.” Andrei will make a description of the plans and the current status of this task.
You probably have (or appeared) questions. The event is the right place to ask. But if you for some reason do not get to the meeting, then write questions in the comments - we will try to call our experts to answer.
Source: https://habr.com/ru/post/250493/
All Articles