OpenStack large-scale benchmarking: How we tested the Mirantis OpenStack in SoftLayer
Author: Oleg Gelbukh
It seems that one of the main issues of concern to those involved in the cloud computing industry was the question of the readiness of OpenStack for operation at the enterprise level at peak load. Among the common topics of conversation - the stability and performance of cloud services based on OpenStack in scale. But what does “scale” mean? What scale is applicable in the real world?
Our experience with numerous corporate clients of OpenStack shows that the “large private cloud” usually means something that has tens of thousands of virtual machines (VM). Hundreds of users participating in hundreds of projects simultaneously manage these VMs. On such a scale, I think it would be fair to say that the “private cloud”, in fact, becomes a “virtual data processing center”. ')
The virtual data center has a number of significant advantages over the traditional physical one, not least of which is its flexibility. The period between receipt of an order from a client and the readiness of the requested infrastructure is reduced from several days / hours to several minutes / seconds. On the other hand, here you need to be sure of the stability of your IaaS platform under load.
Scaling Mirantis OpenStack based on Softlayer
We decided to evaluate the desired behavior of the Mirantis OpenStack platform under the load expected from it in such a virtual data center. Collaborating with our friends from IBM Softlayer, we created an infrastructure of 350 physical machines located in two locations. To begin with, we wanted to establish a certain baseline under a service level agreement (SLA) for virtual data centers operating on Mirantis OpenStack.
Obviously size does matter. But there were other questions regarding the OpenStack virtual data center in Softlayer, which we found important to answer from the very beginning. For example: • Can a cloud-based virtual data center serve hundreds of requests at the same time?
• Will the increase in the number of virtual servers in the virtual data center affect the user experience?
To answer the first question, we needed to perform a series of tests with an increase in the number of simultaneously sent requests. The target of parallel processing at this stage of benchmarking was set to 500. We approached the second question, specifying it in order to estimate how long it would take to provision the server in our virtual data center, and, more importantly, how long the number of available servers.
Based on the above expectations, we have developed a first set of benchmarks, the purpose of which is to test the infrastructure of Mirantis OpenStack Compute, a key component of the platform. Testing affected not only the Nova API and the scheduler, but also Compute agents that run on the basis of the hypervisor servers.
Since we are performing benchmark testing of launch time, measuring the scalability of the OpenStack Compute service does not require creating any load inside the VM, so you can use the flavor of the OpenStack platform with low resource requirements, in a small way that you can download there or tasks performed on the VM. When using instance types defined for OpenStack by default, the size of the instance does not have a significant effect on the provisioning time. However, we needed to perform testing on real hardware in order to get a clear idea of ​​the capabilities of the Nova Compute agent for handling requests under load.
The basic SLA parameters that we wanted to check are as follows: • Dispersion of the load time of virtual server instances, including the relationship between time and the number of virtual instances in the cloud database;
• The success rate of the boot process for our instances, in other words, how many requests out of 100 will be unfulfilled for one reason or another.
As a target for this benchmark, we set 100,000 virtual instances running in the same OpenStack cluster. With such a resource-undemanding variety, our experiment can be compared with a specialized site for servicing mobile applications. Another possible use case with this workload is a test environment for a mobile application server with several thousand clients running non-independent embedded systems. We assumed that we could understand this with the help of several hundred servers generously provided to us by Softlayer for a limited period of time for use as a test bench.
Setting up a test environment for OpenStack benchmarking
We often hear the question: “How much hardware do you need?”. And often jokingly we answer clients: “More”. But it is clear that the work of hundreds of thousands of VMs requires sufficient hardware potential. The IBM Softlayer company offers quite a wide variety of configurations, and its employees were just like us, curious to see what OpenStack can provide at this level.
To determine the amount of resources required, we first set an instance type, or a variant in terms of OpenStack: 1 virtual processor, 64MB of RAM and 2GB hard disk (which is much less than, for example, the m1.micro AWS instance type). This is quite enough to load the Cirros test image.
Using an instance type with very modest resource requirements does not affect the test result, since for instance varieties / types specified in OpenStack by default, the instance size does not have a significant effect on the timing of provisioning. The reason for this is that these varieties determine only the number of processors, the amount of RAM and the size of files on the disk. These resources are usually provided with virtually zero dispersion depending on the size of the instance, because, unlike the AWS instance, which also provides additional services such as elastic block storage (EBS), OpenStack instances are isolated; the creation of a short-term storage, in fact, is carried out by creating, for example, an empty file. Therefore, we can use a very small instance without fear of distortion of the test results.
Total number of resources: • 100,000 virtual processors (vCPU);
• 6250 Gb of RAM;
• 200TB of disk space.
Therefore, we decided that a representative configuration of a computing server for this test would be a configuration with 4 physical cores (+4 for HT), 16Gb of RAM and 250GB of disk space. Since we are not going to allow any real load on the VM, we can safely redistribute the CPU at a ratio of 1:32 (twice the redistribution coefficient (overcommit rate) set for OpenStack by default) and redistribute the RAM at a ratio of 1: 1.5 , which gives us a density of 250 virtual servers per physical node. With such non-standard redistribution coefficients, we only needed about 400 hardware servers to achieve the target of 100 thousand VM.
Knowing that the maximum load will be on our controllers, we set for them an extended server configuration. Each controller had 128GB of RAM and a solid-state hard disk (SSD) for storing data, which made it possible to speed up the operation of the OpenStack state database based on the controller. An interesting note: due to differences in the configuration of the Compute and Controller nodes, they were provisioned to various SoftLayer data centers with a delay of 40ms between them; it would not be strange to expect that in a real cluster you can have exactly the same distributed configuration. Fortunately, due to the low latency between the data centers of Softlayer, no significant effect of such a distributed architecture was noticed during benchmarking.
OpenStack Details
We used the 4.0 release of the Mirantis OpenStack product , which includes OpenStack 2013.2 (Havana), and deployed it on the servers provided by SoftLayer. To do this, we had to slightly change our deployment infrastructure (Deployment Framework): since we could not unload Mirantis OpenStack ISO into SoftLayer, we simply copied the necessary code using rsync through the standard operating system image. After deployment, the cluster was ready for benchmarking.
Our configuration included only three OpenStack services: Compute (Nova), Image (Glance) and Identity (Keystone). This provided the basic functionality of the Compute service, sufficient for our usage scenarios (testing of the mobile platform).
To build the network, we used the Nova Network service in FlatDHCP mode with the Multi-host feature enabled. In our simple use case (one private network, FlatDHCP network and no floating IP), the functions are exactly the same as in Neutron (the Nova Network component, which was recently restored from “not recommended for use” status and returned to OpenStack upstream ).
When conducting benchmarking, we had to increase the values ​​of some parameters of the default OpenStack configuration. The most important change is the connection pool size for the SQLAlchemy module (the max_pool_size and max_overflow parameters in the nova.conf file). In all other aspects, we used the high-availability standard Mirantis OpenStack architecture, including a database with synchronous replication with multiple hosts (MySQL + Galera), a software load balancer for API services (haproxy) and corosync / pacemaker to provide high availability IP.
Rally as a benchmarking tool
To actually perform the benchmark test, we needed a testing tool; The logical choice was the project Rally in OpenStack . Rally automates test scripts in OpenStack, i.e. allows you to customize the steps you need to perform each time you run a test, specify the number of test runs, and also record the time it took to complete each request. With such functions, we could calculate the parameters that needed to be checked: the variance of the load time and the percentage of successful execution.
We already had experience using Rally based on SoftLayer , which helped us a lot when conducting this benchmarking test. The reference configuration looks like this:
The parameters in this configuration are as follows: • execution - the mode of operation of the benchmarking tool. If continuous, the workflow Rally will issue the next request for returning the result of the previous one (or expiration of the timeout).
• tenants - the number of temporary Identity tenants (or projects, as they are now called in OpenStack), created in the OpenStack Identity service. This parameter reflects the number of projects that can correspond, for example, to the versions of the tested mobile platform.
• users_per_tenant — The number of temporary end users created in each of the above-mentioned temporary projects. Rally streams send real requests on behalf of users randomly selected from this list.
• active_users - the actual number of Rally workflows. All these threads start at the same time and start sending requests. This parameter reflects the number of users working with the cluster (in our case creating instances) at any time.
• times is the total number of queries sent during benchmarking. Based on this value, the success rate is determined.
• image_id and flavor_id are cluster-specific parameters used for launching virtual servers. Rally uses the same image and variation for all servers in the reference iteration.
• nics - the number of networks to which each virtual server will be connected. By default, there is only one such network (this is the default network for the project).
The process of benchmarking
To begin with, we have established a stable rate of simultaneous sending requests. To do this, we created a set of reference configurations that differ only in two parameters - tenants and active_users - from 100 to 500. The number of VMs that had to be started was 5,000 or 25,000. In order to save resources in the initial stages, we started with more a small cluster of 200 Compute servers plus 3 Controller servers. The diagrams below show the dependence of the launch time of a certain virtual server on the number of simultaneous requests that are executed.
Test run 1.1. 100 simultaneous sessions, 5000 VM
Total Success. (%) Min. (sec) Max. (s) Average (s) 90th percentile 95th percentile
A couple of times during benchmarking, we faced limitations, but all of them were eliminated by increasing the size of the pool of connections in sqlalchemy (from 5 to 100 connections) and in haproxy (from 4,000 to 16,000 connections). In addition, several times Pacemaker erroneously detected a malfunction of haproxy. The reason for this was the spent pool of haproxy connections, which caused Pacemaker to switch its virtual IP to another controller node. In both cases, the load balancer did not respond to failover. These problems have been solved by increasing the limiting number of compounds in haproxy.
The next step was to increase the number of VMs in the cloud to a target of 100,000. We set a different set of configurations and run Rally for it under test load with target populations of 10,000, 25,000, 40,000, 50,000, 75,000 and 100,000 concurrent virtual servers. We found that the maximum stable level is 250 parallel queries. At this stage, we added another 150 physical hosts to the cluster, the total number of Compute servers was 350 plus 3 Controller servers.
Based on the results, we made changes to the targets and lowered the target from 100,000 to 75,000 VMs. This allowed us to continue the experiment in a 350 + 3 cluster configuration. At the established level of parallel execution, it took a total of more than 8 hours to deploy 75,000 virtual servers with the selected instance configuration. The diagram below shows the VM startup time as a function of the number of VMs running when running tests.
Test run 1.3. 500 simultaneous sessions, 75,000 VM
Total Success. (%) Min. (sec) Max. (sec) Average (s) 90th percentile 95th percentile
As can be seen from the diagram, with 500 simultaneous connections, the average time per machine was less than 200 seconds, or an average of 250 VM every 98.8 seconds.
Test results of Mirantis OpenStack based on Softlayer
The first stage of benchmarking allowed us to establish a baseline SLA expectations for a virtual data center configured for computing resources (as opposed, for example, to something that is resource intensive in terms of data storage and I / O operations). The Mirantis OpenStack platform is capable of simultaneously processing 500 requests and simultaneously supporting 250 requests to build a collection of 75,000 virtual server instances in one cluster. In the Compute architecture, instances of virtual servers do not assume the constant presence of attached blocks for virtual data storage, and they do not form complex network configurations. Please note that in order to be conservative, we did not endow the cluster with any OpenStack scalability functions, such as cells, a modified scheduler, or dedicated clusters for OpenStack services or platform services (for example, a database or a queue). This benchmarking test is based on the standard Mirantis OpenStack architecture; any of the above modifications would raise performance to an even higher level. An important success factor, of course, was the quality of IBM SoftLayer hardware and network infrastructure for benchmarking. Our results show that the Mirantis OpenStack platform can be used to control the operation of a large virtual data center on SoftLayer standard hardware. Although we did not measure the network parameters to estimate the load on the connections, we did not observe any impact. In addition, the quality of the network connections between the data centers in SoftLayer allowed us to easily build OpenStack clusters with several data centers.
Looking Ahead: Expanding the OpenStack Workload Range
In the next stage of the benchmarking project, we plan to use SoftLayer hardware with benchmark tasks running inside virtual server instances. We plan to set service level thresholds for disk and network I / O based on common workload profiles (web hosting, virtual desktop, build / develop environment), and evaluate the behavior of Mirantis OpenStack in each of these scenarios.