Cloud storage: you need 100% compatibility with Amazon S3 API

Hello! Today we have decided to raise the topic of the clouds that have already fully entered our lives, and in particular - cloud storage in business. The topic is not new, but we decided to talk about it, because we ourselves had released our own cloud platform as recently as last year, and a couple of days ago we began to officially provide a storage service on it. Why did we decide to launch this particular service, what does it carry for the client, what were they guided by when creating it, which competitors were equal to, and what kind of storage did we get in the end? Welcome below.

Thinking out loud, or prerequisites for creating our repository

Cloud storage has become a well-known service for PC and mobile users. In the past five years, after services such as SkyDrive or Dropbox, they literally appear one after another. As we all know, the main essence of these services is that they allow you to easily and quickly, at any time and from any computer or smartphone, get authorized users access to their own files.

Traditional cloud storage such as Dropbox, Google Drive, Microsoft OneDrive, in international terminology, belong to the class “Enterprise File Sync & Share (EFSS)” - file synchronization and file sharing services. Usually they are used for personal use: files on the computer are synchronized with the cloud, they can be shared.
Developers also know such services as object storage - Amazon S3, Google Cloud Storage, Microsoft Blobs Storage. Highly reliable object storage, accessible via the Internet, is designed to store a huge number of files (billions and trillions), they are scaled to hundreds of petabytes.

It is in the object storages that synchronization and file sharing services keep their files. Object storage stores half of the content on the Internet (images, videos, downloads). The cloud storage of Technoserv Cloud created by us belongs to the class of object storages, but about it a bit later.

Object Storage

The primary object storage access protocol is a protocol created by Amazon Web Services for AWS S3 (Simple Storage Service). S3 API is a set of commands that "understands" the repository and performs certain actions in response. The completeness of the S3 API support and defines the functionality of the object storage compared to Amazon S3.
In fact, half of the services on the Internet store data in Amazon S3. In Russia, not so many object storage services are presented (see Table 1).

Table 1 . Synchronization and file sharing services and object storages.

Both types of services are in growing demand.

Public Storage Services

Public storage services today attract the attention of not only small and medium-sized businesses, but also large enterprises and organizations implementing a hybrid cloud model. This is facilitated by the dynamics of the development of the Russian cloud market as a whole. According to Forrester, by 2020 the volume of clouds in the Russian Federation will reach 48 billion rubles. According to the forecast of iKS Consulting, its growth rates will significantly outpace the growth of the domestic IT market (see Figure 1).

Fig. 1. The Russian cloud market is developing at a faster pace.

About the requirements and benefits of cloud storage for customers

Of course, there are special requirements for corporate repositories, because the data are often the basis of business. If the storage service is not reliable enough, if it is not possible to provide constant access, then the company's activities may be at risk. Therefore, the key factors for cloud storage should be not only the cost of the service, but also its reliability, security, speed of access to data in the cloud.

Cloud storage involves storing data on multiple servers distributed across a network (geographically distant from each other or not) provided for the use of clients. The internal structure of such servers to the client, as a rule, is not visible: the data is stored and processed in the so-called cloud.

The cloud storage for business offered today allows you to store data fairly reliably and inexpensively, provide access to them regardless of location, and easily scale resources. This service provides a significant reduction in the cost of building and maintaining IT infrastructure and will help increase the flexibility of business processes (see Table 2).

Currently, cloud storage services are offered by the largest Russian hosting providers, system integrators and telecom operators. At first glance, they differ little from each other: a user-configurable folder and file structure, the ability to configure access, a similar interface. However, their internal architecture and proposed features usually have significant differences . That is what we decided to bet on.

The result - our cloud storage Technoserv Cloud

So, after thinking about all the above topics, we decided to launch a cloud storage service based on the cloud platform we already have. We’ll tell you what happened, how it differs from others.

Like many similar services, it is charged on the basis of actual use or available storage capacity. What can boast of what has been achieved - the cost of storing data here is lower than that of Russian counterparts, and downloading them is much more profitable than offers from foreign suppliers. You can use the necessary resources via the Internet, a dedicated communication channel, or from the Technoserv Cloud virtual data center .

Fig. 2. The service is powered by Technoserv Cloud , accessible via the Internet, dedicated communication channels or from the Technoserv Cloud virtual data center.

Our storage can be used, for example, in the following scenarios:

● Backup storage.
● Placing web site content and letters (photos, videos, files for download).
● Storage of surveillance system records.
● Storage of archives (to comply with the requirements of the law).
● Storage and distribution of media content (video, music).
● Storage for workflow systems.

And also for any tasks for which companies usually choose Amazon S3 services (see Figure 3).

Fig. 3. Typical tasks for cloud storage Technoserv Cloud.

The uniqueness of the solution - 100% guaranteed compatibility with Amazon S3 API

Object stores represented in Russia, as a rule, are created on the basis of Ceph or Swift solutions with connectors for the S3 API and do not provide full support for the S3 API. For users, this may mean:

1. Application compatibility problem (the application is written with the full set of S3 commands, and the storage, for example, supports half).
2. Limited functionality.

Therefore, when developing our object cloud storage, we paid particular attention to the completeness of S3 API support, that is, to compatibility and functionality of the solution.

Full compatibility with the Amazon Simple Storage Service (S3) API is one of the main features of the cloud storage solution Technoserv Cloud . Support for standards for connecting to cloud storage (cloud API) greatly simplifies the integration of customer systems with the cloud platform (see Figure 4).

Full-featured S3 API support ensures compatibility with various software. S3 API is supported by a large number of applications, in particular, by almost all backup systems (except Veeam) and content management systems (for example, 1C-Bitrix or Wordpress). A set of application development tools for any platforms and in any languages ​​is also available.

Meanwhile, many large-scale data storage systems in the cloud work through S3 connectors and support only part of the operations, which creates risks to the health of the entire system. For example, solutions based on CEPH with an S3 connector, have "medium compatibility" for operations.

Fig. 4. Since the storage of Technoserv Cloud is 100% protocol compatible with AWS S3, the “move” from S3 to this platform is very simple. It is enough in the application to replace the server address and enter a new key.

Technoserv Cloud cloud storage core is not only built using S3 technologies, but also supports the full range of S3 API commands, including advanced ones, such as ACL (access rights) management, V4SIG authentication (modern Amazon authorization protocol), full version control, multi-threaded download and other

So, if solutions based on CEPH offer limited support for S3 API commands, then the solution from Technoserv supports almost the entire “corporate set”. This means a minimum of problems for the customer in organizing backups or deploying other applications, high compatibility with the Amazon cloud now and in the future.

Gateway to connect to the repository from any application

To access the cloud storage, the Internet or a secure L2 VPN channel is used, connecting the Cloud Technoserv Cloud to the client’s office or data center. Another option we worked out is to connect to the storage from the virtual data center Technoserv Cloud.

To connect to the repository of applications that do not support S3, for example, Veeam Backup & Replication, we provide gateways. Through the gateway, you can connect to the cloud using NFS, CIFS (SMB) or FTP protocols. The gateway is a virtual machine hosted on the customer’s network (VMware or Microsoft Hyper-V), which looks like a network folder / FTP server and translates NFS / CIFS / SFTP to S3 protocol.

The procedure of connecting to our cloud storage also does not require long and complicated settings of the firewall on the client side. To do this, use the standard 443 TCP port HTTPS.

Integration with any backup systems, file management and versioning

A popular service today is backup to the cloud. Most backup systems can be connected to our cloud storage via the S3 API (see Figure 5); for the rest, an NFS gateway is used. Supported and work with built-in backup applications. In this case, the cloud acts as an alternative to tape storages, but with quick access to information.

Fig. 5. Cloud Storage Technoserv Cloud supports Acronis Backup, Arcserve Backup, CommVault Simpana, IBM Spectrum Protect, Rubric, Veeam Backup & Replication, Veritas / Symantec NetBackup and Backup Exec backup systems, etc. Free NFS is provided for Acronis and Veeam products \ CIFS gateway.

S3 API for backup support and popular content management systems (CMS), for example, 1C-Bitrix and Wordpress. Even if the system does not explicitly support S3, as a rule, you can use the curl utility for backup.

The repository provides for the ability to customize corporate information storage policies, as well as versioning. It is very convenient. For example, if a company leads a long-term project, collectively discusses a working paper, or develops a database, then the entire history of data changes is retained, which makes it possible to return to an earlier version.

Versioning can be used to save, retrieve and restore previous versions of objects stored in the cloud. The “depth” of version storage is determined by the customer. This is a great insurance against user errors. You can share objects or even whole "baskets" for other users. With the help of ACL access rights are configured.

The management portal allows the administrator to create user accounts and assign rights, manage user quotas, view statistics, and users to manage files via the Internet, create / cancel S3 API keys used in applications, grant access rights / publish their baskets and files.

About the reliability and security of our cloud

We have been working on this important topic for a long time. The distinguishing features of Technoserv Cloud are high reliability and safety. The storage reliability is “eight nines” (99.999999%), which is about 100 times higher than with classic storage with RAID 6 (see Figure 6).

Fig. 6. The basis of the cloud solution "Technoserv" - software-defined storage (SDS), in which there is no single point of failure and provides full redundancy of all components.

To ensure redundancy during storage, each file is divided into a specific number of parts. The default is Erasure Coding 5 + 3. This means that any object (file) is distributed to eight nodes (servers), and any five parts are enough to preserve the integrity of the file (see Fig. 7). Thus, the data remains available even with the complete simultaneous loss of any three servers. EC 5 + 3 provides reliability of 99.999999% (8 nines). EC 4 + 4 options are also possible, or any replication from 2 to N.

Fig. 7. Use Erasure Coding to increase storage reliability.

Finally, Technoserv Cloud’s cloud storage is located on a high-availability site — in DataPro’s Tier III data center. This data center has two certificates of the Uptime Institute - “Desing” and “Constructed Facility”. In Russia, only four data centers have similar certification.

The Tier III level allows for repair work (including the replacement of system components, the addition and removal of failed equipment) without stopping the data center. Engineering systems are once reserved, there are several power distribution and cooling channels, two independent power inputs from different substations, its own transformer substation, two independent FOCL routes.

Guaranteed SLA provides for availability of the service at the level of 99.95%, that is, no more than 21 minutes of downtime per month. Plus, round-the-clock technical support with qualified engineers.

It also provides for redundancy of the network architecture (see Figure 8) and server components. Power supplies, processors, network interfaces, OS disks, storage devices (according to the SDS architecture) are reserved.

Fig. 8. Redundant network architecture for high availability.

We also paid special attention to cloud storage security. Encryption during transmission and storage of data (with the ability to encrypt on the storage side with the client key), detailed access rights, logging of all file operations ensure reliable protection of client information. In addition, for the purpose of information security, regular personnel monitoring is carried out. Employees of the Cloud Platform Technoserv Cloud Platform are thoroughly tested before being hired (including a polygraph test). This service can be used for the most important tasks.

Usage scenarios

There are a variety of scenarios for the use of this cloud storage. For example, archiving, file storage and collaboration with them. The flexible scalability of the cloud environment with its high reliability, the absence of overpayment for unused resources are the clear benefits of this model.

Amazon S3 is an unlimited capacity object storage with a simple web interface. What is it usually used for? Tasks can be very different. For example, as the main storage for cloud applications, big data storage, as a data repository for analytics and, of course, for backup and disaster recovery.

The logical question is: “So why not take advantage of the“ real ”Amazon S3? What are the advantages of the Technoserv Cloud storage? ”There are data storage requirements in Russia. In accordance with 152- “About personal data”, some data should be stored within the country. In addition, there are applications that require low latency when accessing data. Our storage is located closer to the Russian client, which allows us to get a lower delay. And these benefits are not exhausted (see Table 3).

Another common task is hosting static website content. The engine and texts of the site can be placed at the hosting provider, and photos, videos, files for download - in the object cloud storage. This reduces the load on the site, speeds up the loading of static content and copes with peak loads. Note that many CMS initially support S3, for example, the popular 1C-Bitrix and Wordpress. But this is not at all an obligatory requirement, since files from the Technoserv repository are accessible via the usual hyperlinks.

In principle, such storage will also be an ideal “data warehouse”: it can be used for any data that is usually stored once and may be needed in the future, whether it be legal documents, tax records or medical records, digital assets and media files, various documents.

For example, broadcasting and media companies need large repositories for media materials. The volume of materials is growing exponentially, traditional solutions are becoming too expensive and not scalable enough. For broadcasters, studios, advertising agencies, post-production companies, new formats, such as 4K, panoramic photos, high-resolution video at 60 frames / second, represent serious challenges, and cloud storage can be the answer.

Summarizing

The offer of cloud services in the Russian market and the number of ready-made solutions are constantly growing, related services are being developed. Advanced cloud technologies are becoming available to Russian customers. Now these offerings have added our cloud storage service Technoserv, compatible with Amazon S3 and built on the same principles. It became the development of the competence of the company, which provides the Virtual Data Center service to a number of clients, and expanded the range of cloud services available to Russian companies. We hope that he has a great future.