DBaaS: databases in the cloud

A source

Friends, everyone with the Programmer’s Day yesterday! Life without bugs and beautiful code!)

We continue to analyze the cloud services of Technoserv Cloud and today we will decompose in detail what our cloud database consists of. If you look at the results of a study conducted by IDG Connect for an Oracle order, you will see that DBaaS will soon be the most requested private cloud service. The number of public DBaaS services is also growing.

Reducing costs by consolidating resources, scaling as needed, controlling costs, accessing data from anywhere - all of these are factors that influence the choice in favor of a cloud database. On the cloud services market, its database is offered by its leading players - Amazon Web Services, IBM, Microsoft and Oracle. But there is one problem - they all deploy databases outside of Russia, moreover, not all of them offer service - administration, performance management, round-the-clock technical support (preferably in Russian) - but only a platform.

To respond to this market request, we launched our service and became the only Russian cloud provider working with four main databases under FZ-152 and FZ-242.

DBaaS Market

At first, briefly about the market. If you are already familiar with this information, go straight to the “Four in One” block, but it seemed to us that this data is quite interesting.

According to Technavio's forecast, in the coming years, the global DBaaS market will show exponential growth - more than 65% annually. Instead of investing heavily in hardware platforms, many companies tend to invest in services with weekly, quarterly, or annual subscription fees.

The scope of work to maintain its own numerous databases and servers can be quite serious. Standardization, when everything is in the same environment, moves the process to a higher level and simplifies working with the database. The key word here is “simplifies,” IDC analysts say.

Judging by the results of surveys, relational cloud DBMSs are among the most popular public cloud services. They are used by 35% of respondents, 14% are experimenting, and 12% plan to implement (source - RightScale ).

Moreover, the transition to cloud computing reduces costs by consolidating resources, increasing the efficiency of the IT infrastructure. By consolidating resources, you can also provide customers with additional performance and manageability.

According to Forrester , AWS is the market leader in DBaaS. Amazon Relational Database Service allows you to work with Oracle databases, Microsoft SQL Server, MySQL, MariaDB and PostgreSQL in EC2 environment. Of the 100,000 DBs examined by 2ndWatch, 67% were Amazon RDS.

Growth in global DBaaS market in millions of dollars (according to 451 Research).

Cloud computing not only allows companies to scale as needed, but also helps them manage maintenance costs. The growing popularity of mobile apps is also encouraging companies to use DBaaS: data can be accessed from anywhere. All these factors contribute to the growth of the DBaaS market.

Growth of Oracle DBaaS service in the world.

Analysts Markets & Markets predict that the cloud DBMS / DBaaS market will grow from $ 1.07 billion in 2014 to $ 14.05 billion by 2019 with an annual growth rate (CAGR) of 46%.

What is better, service or its own database?

A cloud database or DBaaS (Database as a Service) is any DBMS that is provided by subscription as a cloud service within the framework of the platform service model. That is, DBaaS is one of the PaaS services. In the case of PaaS, “platform as a service”, the customer receives already installed and configured software for developing and testing or deploying applications. For the customer, a database of the required configuration is created in one of the following options:

• DB without virtualization (on a physical machine)
• DB on virtual machine
• DB as a container in a multi-tenant database

For example, deploying databases in separate virtual machines on a common server platform simplifies migration to the cloud, but leads to additional costs, complicates support for DBMS versions, etc. If DBMSs are working on a pool of physical servers from a single OS, then it is possible to unify the versions, simplify management, make more efficient use of equipment, consider Oracle.

Individual databases are consolidated on physical servers and are grouped in cloud pools. Any server from the pool can host one or more database instances (source - Oracle).

Using DBaaS, the customer can get access to a database of a particular type upon request, quickly deploy the database on the required hardware and software platform (operating system). In this model, the payment may be charged depending on the capacity and other consumed IT resources, as well as the functions and means of database administration. Note that all database functions available locally are also implemented in the cloud.
DBaaS allows, for example, to quickly deploy databases or application servers, to use high-volume, rapid database cloning for development or testing. Databases are literally cloned in seconds using snapshots. After logging in to the technical support panel, everything happens automatically.

Consolidation of resources in the service provider's data center improves IT efficiency, and the use of standard, tested configurations increases reliability. You can also order high-availability or disaster-resistant configurations, use the hybrid model for occasional load increases. DBaaS removes from the customer the problem of deploying and maintaining the DBMS. In addition, you can work with a cloud database at any time, from anywhere in the world and from any application.

Creating database configurations based on standard templates allows you to apply a self-service model. This frees administrators from manually setting up databases in response to each individual request. When the DBaaS environment is ready for use, customers will be able to prepare bases with simple operations without involving DBAs on the resource allocation parameters, setting access restrictions and performing other common tasks.

As practice shows, many DBaaS customers note:

• Reduced total costs.
• Greater independence of business users from IT departments.
• Reducing risks in IT planning scenarios.
• Greater predictability and flexibility.
• Application developers have enough freedom for creativity and innovation.
• DBaaS improves database administrators. They concentrate more on business tasks and less on routine operations.

Differences DBaaS from the traditional approach (source - Oracle).

The main advantages of the cloud database

1. To begin with, not every company can allocate an individual employee to monitor and manage their DBMS. The result - high risks of stopping business processes, interruption of production processes, data loss and other troubles. With the “Cloud Database” service we take care of all these concerns. Advantages of DBaaS:
   • high scalability
   • cost reduction
   • fast provision of services
   • increased reliability and safety.

   
   
2. It is not only safer and safer, but also more profitable. As already noted, the company can significantly save on DBA salaries. Indeed, there are very few qualified specialists in the market, and often the existing ones are overloaded with current tasks. In the cloud, there is no need to take care of the settings of the database server, and the price of support is usually much lower. In addition, our own database specialists are not always able to provide the necessary level of service, and cloud solutions guarantee a high level of accessibility - we monitor around the clock at the platform level and at the DBMS level.
3. In fact, many companies do not have specialists in the database at all. It is one thing to start SQL Server or MySQL on the server and ensure the operation of the application, another is to correct the errors in the database and optimize its functioning. In the case of real problems will have to urgently seek an external specialist. In the cloud, the “bottlenecks” of the database become the headache of the provider. In addition, he may well hire specialists in database and maintain a high level of their skills. And at your service - the first line of support. This is the question of choosing the location of the database - "at home" or in the cloud.
4. Another advantage of the service is fast-deployable ready-made typical configurations that are optimized in performance for typical database use tasks. Creating individual and clustering configurations can take up to three business days.
5. The Cloud Database service uses backup virtual resources, provides availability reports, basic database and OS administration, database backup and restore from backup, storage capacity management. Additional services - monitoring database performance, monitoring business processes, advanced administration and backup with an individual backup schedule and storage depth. Professional services also include auditing, consulting, migration, including conversion and data transfer (with the provision of storage systems). And of course, we provide fault tolerance and increased database availability. After all, the database is one of the most critical services for the enterprise.

Four in one

Go to the description of our service " Cloud Database ". As we have already said, this service can work with four main databases: we provide Microsoft SQL databases ready for work (MS licenses using the SPLA model), PostgreSQL and MySQL, as well as Oracle hosting.

The tariff plan provides for more than 80 options for providing services for various classes of database instances.

The following versions and editions of the DBMS are available to customers:

Features of the "Cloud Database" service

• High performance.
• Optimized in accordance with the recommendations of the developers and the best practices of operating system and database configuration.
• Professional technical support, which employs database integrator experts.
• The service was created taking into account FZ-149, FZ-152 and FZ-242.
• The price offer is on average 35-40% more profitable than Western cloud platforms (taking into account the costs of database administration and operating systems that are included by default in the service).

According to the legislation of the Russian Federation, the main documents defining the requirements for the protection of information are:

• Law FZ-149 “On Information, Information Technologies and Protection of Information”.
• Law FZ-152 “On Personal Data”.
• Orders of FSTEC of Russia No. 17 and No. 21 .
• Law FZ-242 , which specifies the FZ-152 and obliges personal data operators to process and store personal data of Russians using databases located in the territory of the Russian Federation.

By default, our service is provided to customers in accordance with the 3rd and 4th categories of personal data protection ( data security ), and upon request in the certified segment * of the platform, in accordance with all information protection requirements established by the FSTEC of Russia.

* Certificate of compliance with information security requirements

This certificate confirms the security of using our cloud to accommodate the IP of state and commercial organizations that have stringent requirements in terms of information security. We are talking about Russian customers specializing in PD processing: GIS operators, PD systems with a high level of security, for example, medical and insurance companies.

This certificate also eliminates the stage of certification of the customer's IT infrastructure, thereby reducing the amount of time required by 50%, significantly reduces the amount of investment and greatly simplifies the process of certification of IP. This allows you to optimize the financial and time costs of the organization, reducing the cost of building and maintaining its IT infrastructure, building and maintaining an internal information protection system.

Sanctions are also pushing our potential customers to find alternative solutions on the Russian market. Against the background of a high exchange rate and a complex political situation, they are looking for possible ways to minimize risks, and the Cloud Database, as a Russian service offered in Russia at ruble prices, deserves close attention. Cloud services can act as an alternative to the purchase of IT equipment of western vendors, and open source software, on the basis of which the service functions, responds to the course on import substitution. Open Source platforms, such as OpenStack, are one of the key alternatives to proprietary solutions.

Our service includes all options that ensure database availability:

• dynamically expandable computing resources;
• OS and DBMS administration by the provider;
• DBMS monitoring;
• data backup.

And offers the following options for connecting to the database:

• public networks (public Internet).
• secure VPN connection over the Internet (IPSec VPN).
• secure L2 VPN communication channel.
• IP VPN network.
• via the internal network, provided that the application servers are located in the Cloud Technoserv cloud platform (10 Gbps).

Database administration includes the following operations:

• solving incidents in the database;
• installation and configuration of the client and the database software;
• access control to the database;
• backup database;
• software update database;
• monitoring database availability;
• monitoring of the state of the database and backups, periodic testing of backups (1 time per month);
• space management;
• analysis of logs and trace files;
• restore the database from backups in case of failure *.

* If the need to restore the database from a backup copy was caused by incorrect actions of the user, then such works are considered additional and are charged separately.

Administration of operating systems includes the following operations:

• installation and basic configuration of the server operating system;
• preventive maintenance of the server operating system;
• setting up the operating system (during operation);
• solving incidents that occur in the operation of systems, including recovery; health after a failure;
• system update;
• backup system data;
• accessibility monitoring.

To implement the service, we use the OpenStack segment in our cloud platform with the following features:

• OpenStack / KVM virtualization environment;
• data exchange between VMs - up to 10 Gbit / s;
• OpenStack management system (based on the release of Mitaka);
• using the Neutron network virtualization module allows you to implement the functions of NAT, VPNaaS, FWaaS, LBaaS, routing;
• software-defined storage (SDS) - triple backup ensures not only data integrity in case of failure of any disk, node or group of nodes, but also automated recovery of copies on other nodes.

Resources at the OpenStack segment (including redundancy):

• vCPU - 1000 cores.
• vRam - 1400 GB.
• vHDD - 24,000 GB.

What problems we solve?

Our service reduces operational costs, which include among other things the salaries of database administrators (for example, in the Moscow region, the Oracle DBA salary can reach up to 200 tr. Per month), and also reduces the risks associated with the infrastructure on which the databases are deployed. It can be used to launch new or update existing business applications, such as corporate email systems, CRM and HRM systems, accounting, warehousing, financial and analytical software.

It minimizes the risk of unavailability of business applications due to database errors, reduces the likelihood of data loss due to untimely backups, and makes it possible to ensure high availability of the database in the absence or inadequacy of its own resources / expertise. In addition, you can increase the resiliency of critical business systems by using database clustering technology, obtain sufficient performance when there is a shortage of computing resources and throughput of your own equipment, or implement a disaster recovery plan (DRP). The service is also convenient to use backups of data for backup and consistency checking.

For different applications we have typical configurations:

Common options for using DBaaS are creating test environments for functional and load testing, one-time projects with tight deadlines, solving the problem of episodic “peak loads”, for example, generating analytical or financial reports.

Single Database: Backup and Restore

This is the basic version of the service for Oracle, MS SQL Server (2012/2014/2016 Standard Edition), MySQL and PostgreSQL, in which the system is provided with the necessary characteristics. The database instance is located on a virtual machine.
The Commvault backup system backs up database files and transaction logs. They are stored for seven days. During this period, you can restore the database as of the last backup of the log files (at intervals of an hour).

By default, the backup is performed in the following scenario:

• Once a week - full reservation.
• A differential backup is saved every day.
• The transaction log is saved every hour.

Target recovery time (RTO) - up to several hours (depending on database size), target recovery point (RPO) - up to 1 hour. The recovery time of the service in case of loss of the database depends on the size of the database and the intensity of its use. In the event of a hardware failure, the system is automatically replaced within a few minutes.

Architecture: Single Database

Single database - recovery time and potential data loss:

DB increased security

The configuration depends on the type of database:

SQL Server 2012/2014/2016 Standard Edition

Database Mirroring: Two systems are provided with the necessary characteristics. On one of the machines is the main instance, on the second - the backup. Uses Database Mirroring, which has the following advantages:

1. Increases database availability. In high security mode with automatic failover and failure, the backup copy of the database is transferred to the online mode (without data loss). In other modes of operation, the database administrator may enable forced maintenance (with possible data loss) on the backup copy of the database.
2. Enhances data protection. Database mirroring provides complete data redundancy (using high security mode). In addition, the mirroring participant attempts to automatically resolve certain types of errors that may interfere with the reading of the data page. A member who cannot read the page requests a new copy from another member. If this request completes successfully, the unreadable page is replaced by a copy, as a result of which the error is usually fixed.
3. Increases availability of production database during updates. To minimize the downtime of the database participating in mirroring, you can sequentially update the instances of SQL Server hosting the failover partners. This will cause idle time only for a single failover.
   In the event of a failure of the main database, the backup switches to a backup, and the idle time is usually less than a minute.
   To automatically switch to another resource (the process, according to which, when the main server is unavailable, the mirror server assumes the role of the main server and outputs its copy of the database to the network as the main database); It also uses an additional “service” instance of SQL Server - “Witness”, which can be simultaneously used by several systems. The “High Security Mode” is applied, in which the database mirroring session runs synchronously, and it uses a witness server, as well as the main server and the mirror server. The witness witness server does not maintain the database, its only function is to support automatic transition to the backup system.
   As a last line of data protection, Commvault backup is used. In the event of a hardware failure, our system will automatically provide a replacement within a few minutes.

PostgreSQL and MySQL

Database of increased security: there are two systems with the necessary characteristics. On one of the virtual machines is the main instance, on the second - the backup. In the event of a failure of the main database, a switch to the backup occurs. Service recovery time in case of database loss does not exceed 15 minutes. Backup is performed by Commvault.

Architecture: Advanced Security Database

Service availability - 99.95%. RTO - from several minutes to 1 hour, RPO - close or equal to zero.

DB of high availability and security

SQL Server 2012/2014/2016 Enterprise Edition

Always High Availability Groups are used, which provide a wide range of options to improve database availability and resource utilization, and also have the following advantages:

• Support for up to nine availability replicas (default is 2 replicas). A replica is a dedicated instance of the availability group that resides on a separate instance of SQL Server and maintains a local copy of each availability database that belongs to the availability group. Each availability group supports one primary replica and up to eight secondary replicas.
• Support for alternative availability modes: 1) asynchronous commit (disaster recovery solution that works well when availability replicas are distributed and removed from the other); 2) synchronous fixation (preference is given to a high level of availability and data protection over performance by increasing transaction latency). A single availability group can support up to three availability replicas with synchronous commit, including the current primary replica.
• Support of various forms of working out the failure of another availability group: 1) automatic transfer to another resource; 2) the planned transition to another resource manually; 3) forced transition to another resource manually.
• The ability to configure availability replicas to support one or both of the active secondary replica features: 1) read-only access, which allows the use of read-only connections to read databases while running as a secondary replica; 2) performing backup operations for their databases while working as a secondary replica.
• Listener support for each availability group. A listener is a server to which clients can connect to access the database from a primary or secondary replica of an AlwaysOn availability group. The availability group listeners direct incoming connections to the primary replica or to the read-only secondary replica. Listener provides fast application failover after availability group failover.
• Support flexible failover policy to provide greater control over the failover of the availability group.
• Support automatic page recovery to protect against damage.
• Encryption and compression support for secure, high-performance transport.

In the event of a failure of the main database, the system switches to the backup database, and the idle time is usually less than a minute.
Each replica of an availability group is located on a separate node in a Windows Server Failover Cluster (WSFC). Deploying WSFC requires that servers participating in WSFC (also called nodes) are joined to the same domain.
As the last line of data protection, a backup is used, which is performed by Commvault.

Architecture: High Availability and Security Database (1)

DB of high availability and security. RTO - from a few seconds to 1 hour, RPO - is close to or equal to zero.

(1) - This option is in development. Planned date November 2017

How it works?

“Cloud Database” (TS-Cloud.DBaaS) includes:

• Orchestration subsystem that provides service and infrastructure management. It interacts with the self-service portal and uses the TS-Cloud cloud resources to host the VM.
• Self-service portal.
• Backup subsystem based on TS-Cloud.BaaS CommVault Simpana service.
• TS-Cloud cloud platform (OpenStack).
• Zabbix based monitoring subsystem.
• Domain Name System (DNS)

The heat orchestration module, which is part of the OpenStack platform, provides an additional level of abstraction when working with the cloud and saves the attendants from a lot of routine actions. It allocates and configures computing resources for the DBaaS service when the client requests a service through the self-service portal.

TS-Cloud.DBaaS scheme.

After a client requests a service, the portal launches a stack template corresponding to the DBMS selected by the client and sends the necessary service parameters (node ​​name, size of disks to host the database and transaction logs, etc.) to Heat. Heat requests resources for the service from the components of the virtualization platform, creates the VM from the image, connects the necessary additional disks and networks to it, starts the VM. Next, the VM is initialized using Cloud-init. VM service metadata (internal and external network addresses, stack id, etc.) are transmitted to the portal. The self-service portal interacts with Heat through the Heat-API program interface.

Ansible DBaaS , . TS-Cloud .
Ansible REST API, Flansible. Ansible (), .

Ansible- — REST-API ( , , , , ..) . .

Cloud — . DBaaS , .

Heat Ansible . « » DBaaS.

API Heat OpenStack, Heat IP- Ansible .

Heat Ansible , . Zabbix , , .

, . Microsoft Mirroring Always On Availability Groups, Oracle Data Guard, Oracle Golden Gate, Oracle Dbvision. :

• .
• .
• .
• .
• .
• .
• .
• .
• .
• .
• .
• , .

How much is it?

Natural question. Let's count. , . , , , . .

1. PostgreSQL «1: », .

2. MS SQL ERP MS Axapta .

3. Oracle .

On our website, you can calculate the cost of services using an online calculator .

Since the service is new, there will be a lot of promotions and other promo. In the meantime, for the first 10 customers who have declared that they came from Habr, we are migrating data for free.