Second billing, marketplace and sandboxes for Big Data: what can test environments in the cloud
Any software development company needs test environments that are close to the production environment. This is especially true for boxed software, which has a long release cycle.
Many problems building test environments solves their placement in the cloud. We will tell you about the testing capabilities on our Mail.Ru Cloud Solutions (MCS) cloud platform. But part of what we tell is true for any cloud.
The complexity of setting up a test environment
Before we talk about the possibilities of test environments in the cloud, we will talk about the difficulties that companies face in the process of software testing.
Different Git branches - different environments
Most software development companies use version control systems. The most common of these is Git , which is used by 87% of developers (a RhodeCode survey on Twitter ).
The best practice in Git are the so-called feature-branches (branches), when for each new functionality a separate branch in the repository is allocated. This approach allows developers not to be “pushed by elbows” and makes changes more independent, but requires the deployment of many dedicated environments for testing individual features.
')
Underutilization of computing resources
25% of physical servers are “zombies” who consume electricity but do nothing useful. And many IT specialists cannot say what 15–30% of the servers installed in their company are doing.
So here. On-premise iron for test environments here is no different from any other and, as a rule, poorly disposed of. And there can be no other way. When you buy a fleet of server hardware, you reload on resources in case of increased consumption and number of test environments. As a result, test environments are idle at night and on weekends “just in case”, and you just pay for the electricity and cooling consumed.
Simple application of virtualization in your private cloud does not solve the problem of inflexible scalability and under-utilization of equipment.
Difficulty setting
The test and development environments are not just a few virtual machines. It typically includes an application server, a database server, plus message queue servers and cache servers.
Developers find it difficult to raise and configure such infrastructure on their own, if they do not have the appropriate expertise, it may take several days. For experienced admins this is easy, but not interesting at all. Plus, bureaucracy is superimposed, and the process of allocating resources to creating a test environment can be very long.
Another problem is that a full-time system administrator often does not have expertise in deploying specific test environments, and the company is forced to attract expensive consultants from outside. An example is a sandbox for big data based on Hadoop, Spark, HDFS or Airflow. Deployment of such an environment takes at least a week for a specialist qualified in the BigData area and not less than a month for an ordinary system administrator. A similar story with Kubernetes clusters: building a test cluster close to the production configuration takes at least a few days.
As a result, the company is forced to turn to a third party - to attract consultants. However, it is quite difficult to find specialists in big data , and administrators of such systems can be counted almost on one hand. Yes, the DevOps engineer can cope with all the tasks for deploying test environments, but he also doesn’t have it in every company, and finding a good expert in this area is no easier than a BigData specialist. In 2016, the job search site indeed.com noted that companies are looking for a DevOps engineer longer than any other specialists.
How the cloud solves these problems
Second billing only for consumed resources and instant scaling
The most obvious advantage of cloud services is the ability to rent the cloud infrastructure of any configuration (one server or a whole cluster) for any period. On our Mail.Ru Cloud Solutions (MCS) platform, billing is per second - you only need to pay for the computing resources that were actually used.
Unlike bare metal, private cloud and most Russian cloud providers, MCS does not rate the resources of stopped virtual machines (RAM, CPU): you only need to pay for the occupied disk space. For example, a test configuration of 20 CPU, 40 GB RAM and 1 TB HDD costs 24,800 â‚˝ / month, and payment for disk space - 7,000 â‚˝ / month. For example, if automation means to stop such an environment at 9:00 pm and deploy at 9:00 am, you will pay $ 15,900 - one and a half times less than with a round-the-clock lease.
In our experience, the total savings from testing in the cloud can reach 60–70% compared to testing in the on-premise infrastructure.
The ability to instantly scale resources in the cloud up and down allows you to quickly deploy environments only at the right time and dispose of leased capacity by almost 100%. The deployment time of the test environment is reduced by several days, or even months, compared to on-premise.
Resources are allocated by the developers themselves.
Using self-service portals, developers can set up test environments in the cloud and allocate resources for them without asking the administrators. Even this simple action can shorten the time-to-market product by several weeks.
API for automatic creation and destruction of test environments
The cloud allows you to create short-lived environments on the model " just in time ". Using the REST API and CLI, you can massively deploy test environments, stop, update, and delete them. In this way, you can configure the full life cycle for test environments and keep hundreds of environments running. Thanks to instant cloud scaling and per-second billing, flexibility and reduced operating costs are achieved.
Here is an example from our experience.
The company is developing software for 150 banks. Each bank has its own branch in Git with additional functions built on top of the boxed solution. The company has to raise two or three parallel working test environments for each client: testing with the current version, testing with the new version, checking for updates from the current to the new one. In total, for 150 clients, it is necessary to deploy and maintain up to 450 test environments at the same time, which is without regard to development environments.
In the private cloud mode (they use virtualization in their own data center) it is almost impossible to work with such a load, since there is often not enough available hardware for parallel work of all media. As a result, developers are waiting for their turn to test and can not quickly check the operation of the new version of the application.
Automated deployment and management of test environments in a public cloud removes the limit in testing speed and ultimately reduces time to market. In addition, cost management in the rental of cloud resources is more predictable, and the costs themselves are lower than when using their own specialized equipment for test benches.
In the private cloud mode (they use virtualization in their own data center) it is almost impossible to work with such a load, since there is often not enough available hardware for parallel work of all media. As a result, developers are waiting for their turn to test and can not quickly check the operation of the new version of the application.
Automated deployment and management of test environments in a public cloud removes the limit in testing speed and ultimately reduces time to market. In addition, cost management in the rental of cloud resources is more predictable, and the costs themselves are lower than when using their own specialized equipment for test benches.
Another advantage of cloud test environments is the integration with CI / CD tools. These solutions (for example, Jenkins) contain plug-ins that allow you to dynamically create test environments in the MCS cloud during the build. The test environment can be activated right before running functional or regression tests. If the tests were successful, the environment will collapse automatically, if something is wrong - the environment will continue, so that developers can reconnect and understand the cause of the regression.
Ready building blocks for test environments
Quickly deploy a test environment to the Mail.Ru Cloud Solutions platform
help PaaS, for example - containers Kubernetes and Databases in the cloud. Kubernetes is developing a service directory with hundreds of applications.
From the catalog, you can deploy ActiveMQ, RabbitMQ, Kafka clusters, log monitoring and analysis systems, various CMS and databases in a few minutes. Unlike the popular Docker Hub , which contains only individual application templates, Kubernetes Service Catalog has high-level templates that simplify integration. From templates, you can deploy pre-configured application components (message queue, service discovery, databases, application servers, CI / CD tools, caching servers, blockchain tools, and more), without having to configure the virtual machines on which they are deployed.
Applications available in the Kubeapps marketplace.
Sandbox for Big Data
On MCS, you can deploy test environments for big data applications. Using the PaaS Big Data service, you can create Hadoop, Spark, HBase and Airflow clusters. The deployment process is fully automated, and it saves weeks of time compared to self-tuning.
An additional advantage here is given by second billing and instant scaling. The ability to create scalable test environments for short periods reduces the company's costs of maintaining an analytical IT infrastructure. Savings can reach 80% compared to on-premise.
Integration with Infrastructure-as-a-Code
Unlike proprietary solutions based on VMWare, the MCS cloud platform is based on OpenStack open source software, which has full integration with various tools: Terraform, Ansible, Puppet, Chef.
Terraform is built in the spirit of Infrastructure-as-Code (IaC), when the process of setting up the infrastructure is designed like writing code and is more familiar to developers. Terraform is difficult to use in a private cloud because it does not have full integration with VMware. In the MCS cloud, companies can use ready-made examples of working with Terraform (they are in this GitHub repository ).
Create a test environment using Terraform.
Remember
- Testing in the cloud is becoming a natural practice in the IT industry and has already been applied by a wide variety of organizations: from state universities to large IT companies .
- Building test environments in the cloud helps to save due to the possibility of instant scaling and automation through APIs, which provide almost 100% utilization of rented infrastructure.
- Additional services on a cloud platform and templates on Kuberntetes containers are ready-made building blocks that do not need to be customized.
- Services that are difficult to set up, such as Big Data processing tools, are easier to rent in the cloud in the form of pre-configured templates. You will save weeks by avoiding diving into settings.
- MCS is built on OpenStack and fully integrated with Terraform and other DevOps tools.
All these tools can be tried for free in the Mail.Ru Cloud Solutions platform. Until the end of November , this link with the ILOVEHABR promo code can add 1000 rubles to the account and test-test-test.
Source: https://habr.com/ru/post/429154/
All Articles