New service: virtual private cloud

The main news of this month: we launched a public testing of the new service - “Virtual Private Cloud”. Any user can get their own cloud infrastructure, flexible, manageable, easily scalable and able to cope with any loads.

We will describe in detail the features and benefits of the new service below.

')

Introduction

We have been working with cloud technologies for a long time and we are closely following the situation on the market of the corresponding services. We paid attention to the following point: practically no one of the Russian providers of cloud services offers solutions that fully meet the requirements of corporate clients.

First of all, we should mention public clouds - perhaps the most common type of service in the area under discussion today. They are well suited for individuals and small companies - in such a cloud, even an inexperienced user can quickly set up a website or launch a small application.

With more complex cloud use cases, problems often arise: for example, the lack of a public API can make it difficult to integrate cloud machines with other applications, and the inability to load your own images leads to the need to manually configure each server, and in some cases does not allow you to use the necessary software.

The scope of public clouds in corporate practice is also very limited: they can be used to create a corporate portal or organize workflow for a small company, but it is difficult to use them as a medium for deploying complex business applications - there are issues related to the division of rights between employees or convenience planning. The situation is quite typical when one employee is the actual owner of the account, another is engaged in planning the use of resources, and the whole department is setting up servers. Access to the cloud at the same time is possible only on behalf of the user to whom the account is registered; For corporate users, this is understandably unacceptable.

As an alternative to a public cloud, a private (private) cloud can be made on the basis of its own or leased infrastructure. Unfortunately, the organization and maintenance of a private cloud usually requires significant financial investments at an early stage and may be ineffective in many situations.

It should also be noted that many of the existing solutions are primarily focused on vertical scaling - manual or automatic increase in the performance of virtual servers as the load increases.

However, if we are talking about very large loads (arising, for example, when working with "heavy" corporate applications), the capabilities of the hardware and software components used in the cloud will not allow to increase the performance of an individual virtual machine indefinitely.

In addition, vertical scaling in the cloud can sometimes be very disadvantageous from a financial point of view: payment on the pay-as-you-go principle (i.e., on actual consumption) with unplanned loads can turn into very unpleasant surprises. For corporate clients, this principle of payment makes it difficult to plan expenses - in most cases it is difficult to predict how much resources will be consumed by a machine operating under real load.

In many situations, horizontal scaling is more effective. In theory, it can provide much higher aggregate performance. However, in practice, such scaling is often difficult due to the complexity of adding components to the infrastructure and the need to carefully plan the application architecture.

An argument in favor of horizontal scaling is the widespread adoption of configuration management systems, such as Puppet or Chef. These tools allow you to almost completely automate the addition of new components to the cluster, thereby increasing its total performance without additional labor costs.

Virtual private cloud

Technologies are developing, the requirements for clouds are growing, but so far no product has appeared on the Russian market in which all current trends would be taken into account. Working on our new service, we tried to focus on the needs of the business and provide maximum flexibility and the ability to build a cloud environment adapted to any task. The new service is called “Virtual Private Cloud”.

In a number of modern cloud services, this service occupies a special place. This is not a public cloud in the traditional sense, but also not a private cloud belonging to one client. We offer for rent an isolated cloud infrastructure in which the user can create virtual machines, connect block devices to them, upload virtual machine images, configure the network topology of any complexity and perform many other operations. The new cloud is managed using an open API that allows you to integrate it with other applications.

Corporate clients will appreciate the flexible system of projects and users, which will allow them to distinguish between access to servers and access to the main account. Our approach to the allocation of resources will allow for a clear planning - the client rents the necessary amount of resources, distributes them among his projects and uses them to create virtual machines and other objects.

A promising option for using the service is reselling. Reseller can create projects and transfer access to their customers. Thus, the reseller client may not even know that his virtual machines are running on the basis of our infrastructure - he works with his project using a separate external control panel.

The service will undoubtedly be attractive for individuals who will be able to use our new service for a wide variety of purposes, ranging from the deployment of environments for running games to software testing.

To implement all the features described, we needed a platform with a public API. Our choice fell on OpenStack - a platform that has long been de facto a standard for building modern cloud services.

OpenStack: quick reference

OpenStack is a collection of open source services for building public and private clouds.

Work on OpenStack began in 2010, when the code of the two platforms was merged: Nebula (the name of the platform created specifically for NASA) and RackSpace CloudFiles (developed by RackSpace). Soon, developers of various Linux distributions began to show interest in the new project: in 2011, OpenStack became the main cloud platform for Ubuntu Server and Ubuntu Enterprise Cloud. In the same year, OpenStack was also used in Debian OC. In 2012, RedHat joined the project and began work on its own distribution.

Today, about 9000 individual developers and 250 organizations, including such famous companies as IBM, EMC, HP, Canonical, Yahoo!, take part in the work on OpenStack projects. other.

OpenStack has a modular architecture and includes the following components (we list only the most important components - in fact, there are many more):

• Nova - a controller for managing virtual machines. It performs such functions as processing requests for the creation of virtual machines, performance monitoring, load distribution on physical machines, response to failures, etc .;
• Neutron is a component for managing networks. With it, users can define networks, subnets and routers and configure the internal topology of the network infrastructure. Neutron supports the floating IP mechanism, which can be dynamically allocated to virtual machines;
• Cinder - block device service. Provides the functionality of virtual machine disks;
• Glance is a component for managing virtual machine images. Glance images are used to quickly and consistently deploy new servers;
• Keystone - authorization module; a key component that provides a single point of user authorization in all services;
• Heat - the module responsible for the orchestration. With it, you can automatically deploy virtual servers and template-based applications;
• Swift is an object data storage module that we have been using for our Cloud Storage service for several years.

The interaction of components can be represented in the form of the following scheme:



Why did OpenStack become, as already mentioned above, the de facto standard for building modern cloud services? The following factors contributed to this:

• Open source code that provides ample opportunities for the refinement and improvement of existing services;
• Flexibility. OpenStack based cloud infrastructure is flexible and manageable. The configured system can withstand any design changes and, if necessary, can be easily reconfigured, for example, if you plan to use another data storage system;
• Extensive integration options. OpenStack is not a closed system and can interact with other products. Available support for various virtualization systems, there are plugins for connecting a variety of storage devices and network equipment.

It is worth noting that all Openstack components have an open API, thanks to which a large number of different scripts and applications are written under the platform, allowing you to automatically deploy entire clusters, configure virtual machines and start services with minimal manual configuration.

How does the new cloud work?

Domains and Projects

Each user of the Virtual Private Cloud service has access to a domain - its own virtual space for creating projects and users. A project in this case is a collection of objects and resources to which a user can have access: virtual machines, disks,
networks and others.

Resources and quotas

At the beginning of work, the owner of the cloud acquires resources (RAM, CPU cores, disk space) and distributes them among his projects, thereby setting limits on the creation of various objects. At any time, you can reallocate resources between projects or abandon some of the resources.

Access control

The owner of a virtual private cloud can create users and set access rights for projects to them. This feature is intended for corporate customers, as well as for resellers who create projects and provide access to them to third-party users on a commercial basis.

For each project, a direct link is created, through which you can access its resources without registering in our control panel. In the near future, we plan to implement the possibility of customization and design changes for the external user panel.

Flexible creation of virtual machines

Our service allows the user to create just such a machine, what he needs, without forcing him to choose from several predefined configurations (flavor), as is often done in clouds implemented on the basis of Openstack. At any time, you can increase each parameter of the virtual machine (for example, the amount of RAM), without touching others.

Network connections

Virtual machines can be connected to the network in various ways. A cloud owner can either rent a dedicated public subnet from us (similar to how it is done for dedicated physical servers), or use the floating IP mechanism.

In the latter case, the machine receives the address from the private network, to which you can assign a public address from the public pool. At any time, you can unbind an IP address from one machine and attach it to another. The conversion of external addresses to internal and vice versa is performed using a virtual router.

Machines can be connected to an isolated local network (if, for example, it is used as a backend, and it does not need direct access to the Internet). If necessary, such a network can be connected to a router and assign external IP addresses to some machines — for example, for ease of maintenance.

Loading images

The new cloud supports a set of images for the most common Linux distributions (such as Ubuntu, Debian, CentOS, OpenSUSE), and Windows Server 2012. In addition to the standard cloud-init agent, the images contain an agent created by us that implements a number of useful functions: password reset upon request from the control panel, static network configuration (in the absence of DHCP), management of SSH keys. In addition, customers can upload their own virtual machine images to the project. Supported image formats used in virtualization systems Virtualbox, KVM, VMware, as well as the Amazon EC2 format.

Hard drives

An arbitrary number of hard drives can be connected to the machine. Disks can be created both empty and on the basis of images. Snapshot technology is supported. If the capabilities of the operating system in the virtual machine allow, the disks can be connected and disconnected on the fly.

Other features

In addition, it supports the ability to add SSH keys, a graphical console, generation of RC files for working with console clients, and others. Many new features are expected in the near future - stay tuned!

Control Panel

An important component of any cloud is the user panel, through which virtual machines and other resources are managed. The community is developing a panel called Horizon - typically, Openstack-based product vendors include it as a primary means of accessing the infrastructure. Implementing our own cloud, we decided to abandon Horizon and created our own panel. What was the reason for this decision?

Those who see Horizon for the first time are unlikely to notice any inconvenience. Disadvantages become noticeable only in the process of work, when the attractiveness of the design fades into the background. The creators of Horizon tried to fit all possible cloud management operations into one panel, and as a result it turned out to be too cumbersome. Some of the necessary functions are often “hidden” too deeply (novice users are unlikely to be able to find them), and this makes working with Horizon quite uncomfortable.

The interface of our panel is organized in such a way that all the necessary functions are always at hand. In addition, we have automated some complex scenarios (such as building a network infrastructure, including creating networks and connecting them to software routers). Thus, those users who do not need all the functionality of the cloud, get an interface that allows you to easily create machines in a dedicated subnet, and more demanding users have full control over the infrastructure.

Another disadvantage of Horizon lies in the fact that it (being created relatively recently - in 2011) is based on a morally obsolete scheme involving the generation of pages on the server side. As an example of the disadvantages of this approach, we can mention that all status updates come in the form of rendered HTML fragments.

We tried to make our panel more productive and easy to use, for which we implemented it on the basis of current technologies - the panel is a single-page web application (Single Page Application) written in JavaScript. This solution allowed for quick interface response, thereby increasing comfort and user experience with the control panel.



Based on the selected set of technologies, we also needed to implement a web-server used for updating the statuses of objects in real time. Thanks to him, we can learn about the changes that have occurred with virtual machines and disks - for example, that the installation of the machine is complete.

Additional server components are also needed to implement user and resource management.

Conclusion

The “Virtual Private Cloud” service provides users with ample opportunities to talk about
which one article is clearly not enough. In the near future, the cycle of publications devoted to OpenStack will continue.

Now the virtual private cloud is in beta testing mode. For further work on its improvement, it is very important for us to get acquainted with the opinions of users. Therefore, we invite you to take part in testing. We have already sent invitations to all our registered customers. A brief guide to the basics of working with our new service is published here .

An application for participation in testing can be sent using the ticket system .

You can notify us of errors and share ideas on improving the Virtual Private Cloud service either in the comments to this post, or by email or via the ticket system.

We will try to take into account all comments, suggestions and suggestions in future work.

Readers who for one reason or another can not leave comments here are invited to our blog .