Amazon S3 and all-all-all: choose object storage
The world's best-known object storage is Amazon S3 (Simple Storage Service), Google Cloud Storage and Microsoft Blobs Storage. They are reliable, accessible via the Internet and scale to hundreds of petabytes. The most popular way to access object storage is Amazon's S3 RESTful API. There is an alternative, OpenStack Swift - supports both APIs and even has its own (like Microsoft or Google, for example). Additional access is provided via other file and block protocols like WebDAV, NFS and FTP, but in the context of this article we are testing only HTTPS access through various APIs.
For comparison, take the domestic public object storage. There are not so many of them, and we will consider the most notable players on the Russian market: Mail.Ru Cloud Solutions, servers.ru, Selectel, Technoserv and Rostelecom.
The storage offerings of the big western three can be divided into three parts: storage volumes, operations (requests), traffic and additional services - like Amazon S3 Storage Management, Amazon S3 Transfer Acceleration and inter-regional replication. Detailed prices are available on Amazon S3 (prices without taxes), Microsoft Blobs Storage ( outbound traffic ) and Google Cloud Storage . We are interested in standard storages with "hot" data - for rarely used data and archives, many providers offer cheaper options, which we do not touch on in this review.
The main components of the cost of services will be storage volumes, as well as outgoing traffic outside the cloud provider network. Let's try to evaluate them and for convenience we will convert prices into rubles at the rate of the Central Bank of the Russian Federation on January 13, 2018
')
Domestic companies have price lists that are very diverse (see help), for convenience we will recalculate them with the limitations that the big western trio use. All prices are in rubles with 18% VAT.
Prices are not very obvious, moreover, outgoing traffic to the Internet outside the provider’s network plays a noticeable (often the main) role in the cost of services, so we will try to estimate the cost of services using the example of some project-based object storage. Suppose we want to create a service and a client mobile application for it. The question of storing static and user data immediately arises. There is an opinion that using object storages for statics is costly due to large volumes of outgoing traffic - there are separate providers that allow you to create a content delivery and distribution network (CDN) with caching servers all over the world. If you have a large project with a huge number of users, this may make sense. Small startups can even keep statics in object storage, but since this is all individual, we will not consider such a scenario. Much more interesting user data. Due to constant availability, scalability and ability to withstand high loads, object storage is great for user files of any format: photos, documents, video clips, etc. This is especially noticeable in small projects - several tens of thousands of files and several thousand requests per day will cost you literally a matter of dollars a month. Let's now move on to the calculations. Let the volume of user data be quite tangible (for other options, there is still traditional hosting) and amounts to 1 TB - these are about 5 million photos or 50 thousand small videos. Let our conditional project generate outgoing traffic of 1 TB per month. Let's calculate how much it will cost us.
Operations we have taken out of the brackets, because at the initial stage it is problematic to understand their number - in any case, this is not the main item of expenditure. It immediately becomes noticeable that large western providers have very expensive outgoing traffic at similar prices for storage volumes. At Amazon, for example, it is $ 0.09 per GB (within 10 TB per month) or $ 90 for our conditional project - a very expensive pleasure. Google is even more expensive - $ 0.12 per gigabyte. A good reason to choose a domestic service provider.
Mail.ru Cloud Solutions is developing in the same way as Western services. In addition to Hotbox, customers are offered Icebox storage for rarely used data. S3-compatible API allows you to connect to the repository using a variety of different applications and use connectors to different programming languages.
The servers.ru repository uses OpenStack Swift (S3 API is not supported). The provider does not offer inexpensive tariff plans for rarely used data and archives - in fact, this is a good option for hosting, allowing, for example, to store static data of sites and solve other similar problems. For corporate virtual infrastructure, the list of services is still poor. There is information that the provider builds private clouds for clients and, in particular, makes dedicated ceph installations there. This site is not reported.
In Selectel everything is the same: access via OpenStack Swift (S3 API does not support storage) and the absence of separate tariff plans for rarely used data, although a very tasty price smooths this moment. The main application, as in the previous case, is hosting static data, websites, etc.
The company Technoserv has launched a full-fledged corporate object storage for business and may well argue with Mail.Ru Cloud Solutions. Declared full compatibility with the S3 API and there is a separate rate for storing "cold" data. There is one drawback - the provider does not work with individuals.
Hope of the national cloud platform - Rostelecom uses the Hitachi Content Platform and supports the S3 API, the OpenStack Swift and some REST API. There are no separate tariff plans for rarely used data from Rostelecom, and the configuration of the storage and access to it, how to put it more softly, is not obvious. For example - the article [http://help.cloud.rt.ru/virthran/start/instrukcyas3browver] about access through the S3 Browser - the need to manually encode the login in base64 and read the md5 hash for a password pleases inexpressibly. Rostelecom does not work with individuals either - all cloud services of the company are available only to business and government agencies.
Alas, we cannot run programs on the object storage side, therefore client-server bandwidth measurement (iperf, etc.) cannot be used. We will ping our storages with hundreds of small (64 bytes) and hundreds of large (1 kilobyte) packets from the external to all experimental virtual machines running in one of the Russian data centers. Packages do not fight (0% loss), but the response time is different. In domestic services, it is much smaller, as expected - when exchanging large batches of small files, this will play its role. Ping Technoserv is not a technical possibility, the provider filters ICMP.
The data centers of our services are located in Moscow, the big western troika in the USA.
To test the performance from Russia, we look at the speed of downloading files to the storage and the speed of downloading from an external to all experimental machines (the same as in previous experience). For the first test, take a large file, and for the second - a bunch of small files in bulk. We summarize all the results in a diagram and take a look at the result. Of course, we will work through the REST API, which supports one or another service - with the various file and block protocols that we are not interested in.
For clients from Russia, our services are clearly faster than the American Google and Microsoft, but Amazon is quite capable of arguing with them and even seriously outperforming the download speed of a large file. Although such a file is downloaded from domestic repositories significantly faster. With a set of small files, the picture is completely different - domestic services seriously overtake foreign ones, which was expected. Selectel and servers.ru public repositories showed an interesting feature: the content is downloaded from them much faster than it is downloaded (other services have the opposite). This once again confirms that these services are optimized for the distribution of static content and are good for a variety of hosting, but less suitable for corporate IT infrastructure.
Domestic providers compete with the big Western troika. We tested public data stores for hot data, and having data centers in Russia (read, in Moscow) helps them a lot. Although Amazon is able to deliver high speeds on large (1.4 GB) files, our services run much faster with bundles of smaller ones, which is important: the main application scenarios still require working with a large number of small files. The prices are not so simple - there are options that are much cheaper than foreign analogues, and there are more expensive (the spread is quite large). At servers.ru and Selectel there are no inexpensive tariff plans for rarely used data. This will become a problem if you are going to deploy a virtual IT infrastructure of the company on their site (I would not want to pay for the storage of backups and archives in full). But they very quickly give static content and are good for hosting. Rostelecom's services are complicated to set up, but its list of supported APIs is the widest and the speed of storage is impressive. Although a separate tariff for rarely used data is not there either. For corporate use, the offers of Mail.ru and Technoserv are the most interesting - here you can get support for the S3 API, and separate rates for rarely used data, and decent access speeds. In general, there is plenty to choose from.
Another plus of Russian services is the absence of conflicts with the law (we all remember about working with personal data). Even collecting personal information about users of the site, you can inadvertently get a serious fine if the information is stored outside the country. Here, the leaders of the world market will not be able to oppose anything to local players (Amazon, Google and Microsoft have no data centers in Russia).
For comparison, take the domestic public object storage. There are not so many of them, and we will consider the most notable players on the Russian market: Mail.Ru Cloud Solutions, servers.ru, Selectel, Technoserv and Rostelecom.
Prices
The storage offerings of the big western three can be divided into three parts: storage volumes, operations (requests), traffic and additional services - like Amazon S3 Storage Management, Amazon S3 Transfer Acceleration and inter-regional replication. Detailed prices are available on Amazon S3 (prices without taxes), Microsoft Blobs Storage ( outbound traffic ) and Google Cloud Storage . We are interested in standard storages with "hot" data - for rarely used data and archives, many providers offer cheaper options, which we do not touch on in this review.
The main components of the cost of services will be storage volumes, as well as outgoing traffic outside the cloud provider network. Let's try to evaluate them and for convenience we will convert prices into rubles at the rate of the Central Bank of the Russian Federation on January 13, 2018
')
Domestic companies have price lists that are very diverse (see help), for convenience we will recalculate them with the limitations that the big western trio use. All prices are in rubles with 18% VAT.
reference
Domestic providers mainly use a similar pricing scheme. Hotbox storage of Hotbox in Mail.ru Cloud Solutions will cost the customer 2.5 rubles per gigabyte per month. It also pays for outgoing traffic and requests - everything is standard. The prices of servers.ru (Moscow data center, we do not consider others) are almost similar: 2.5 rubles / GB per month for the first terabyte, 2.30 rubles / GB - the next 50 TB, 2.10 rubles / GB - the next 100 TB and 1.90 RUB / GB - the next 150 TB (prices include VAT and there is only additional traffic paid for - there’s nothing in the price list about the operations). At Selectel storage up to 1 TB per month is 1.40 rubles / GB per month, up to 10 TB - 1.30 rubles / GB, up to 100 TB - 1.20 rubles / GB, more than 100 TB - 1 rub / GB (prices including VAT 18%). On Technoserv Cloud site, the price list could not be found, but the calculator shows approximately the same prices as Selectel - 1.40 rubles / GB per month, but for some reason without lowering the price with an increase in volume and excluding VAT. Rostelecom uses a slightly different pricing methodology - each container costs 10 rubles per day (approximately 300 rubles per month), and 10 GB of disk space costs 1 ruble per day - ~ 30 rubles per month or ~ 3 rubles per month for 1 GB. Separately, you will have to pay support for versioning (0.50 rubles per day for 10 GB) and support for searching metadata - 0.25 rubles for 10 GB per day (all prices in the price list excluding VAT). But about the traffic in the price list does not say anything, but for Rostelecom, this is absolutely not surprising.
Spherical application in vacuum
Prices are not very obvious, moreover, outgoing traffic to the Internet outside the provider’s network plays a noticeable (often the main) role in the cost of services, so we will try to estimate the cost of services using the example of some project-based object storage. Suppose we want to create a service and a client mobile application for it. The question of storing static and user data immediately arises. There is an opinion that using object storages for statics is costly due to large volumes of outgoing traffic - there are separate providers that allow you to create a content delivery and distribution network (CDN) with caching servers all over the world. If you have a large project with a huge number of users, this may make sense. Small startups can even keep statics in object storage, but since this is all individual, we will not consider such a scenario. Much more interesting user data. Due to constant availability, scalability and ability to withstand high loads, object storage is great for user files of any format: photos, documents, video clips, etc. This is especially noticeable in small projects - several tens of thousands of files and several thousand requests per day will cost you literally a matter of dollars a month. Let's now move on to the calculations. Let the volume of user data be quite tangible (for other options, there is still traditional hosting) and amounts to 1 TB - these are about 5 million photos or 50 thousand small videos. Let our conditional project generate outgoing traffic of 1 TB per month. Let's calculate how much it will cost us.
Operations we have taken out of the brackets, because at the initial stage it is problematic to understand their number - in any case, this is not the main item of expenditure. It immediately becomes noticeable that large western providers have very expensive outgoing traffic at similar prices for storage volumes. At Amazon, for example, it is $ 0.09 per GB (within 10 TB per month) or $ 90 for our conditional project - a very expensive pleasure. Google is even more expensive - $ 0.12 per gigabyte. A good reason to choose a domestic service provider.
Features of domestic providers
Mail.ru Cloud Solutions is developing in the same way as Western services. In addition to Hotbox, customers are offered Icebox storage for rarely used data. S3-compatible API allows you to connect to the repository using a variety of different applications and use connectors to different programming languages.
The servers.ru repository uses OpenStack Swift (S3 API is not supported). The provider does not offer inexpensive tariff plans for rarely used data and archives - in fact, this is a good option for hosting, allowing, for example, to store static data of sites and solve other similar problems. For corporate virtual infrastructure, the list of services is still poor. There is information that the provider builds private clouds for clients and, in particular, makes dedicated ceph installations there. This site is not reported.
In Selectel everything is the same: access via OpenStack Swift (S3 API does not support storage) and the absence of separate tariff plans for rarely used data, although a very tasty price smooths this moment. The main application, as in the previous case, is hosting static data, websites, etc.
The company Technoserv has launched a full-fledged corporate object storage for business and may well argue with Mail.Ru Cloud Solutions. Declared full compatibility with the S3 API and there is a separate rate for storing "cold" data. There is one drawback - the provider does not work with individuals.
Hope of the national cloud platform - Rostelecom uses the Hitachi Content Platform and supports the S3 API, the OpenStack Swift and some REST API. There are no separate tariff plans for rarely used data from Rostelecom, and the configuration of the storage and access to it, how to put it more softly, is not obvious. For example - the article [http://help.cloud.rt.ru/virthran/start/instrukcyas3browver] about access through the S3 Browser - the need to manually encode the login in base64 and read the md5 hash for a password pleases inexpressibly. Rostelecom does not work with individuals either - all cloud services of the company are available only to business and government agencies.
Ping from Russia
Alas, we cannot run programs on the object storage side, therefore client-server bandwidth measurement (iperf, etc.) cannot be used. We will ping our storages with hundreds of small (64 bytes) and hundreds of large (1 kilobyte) packets from the external to all experimental virtual machines running in one of the Russian data centers. Packages do not fight (0% loss), but the response time is different. In domestic services, it is much smaller, as expected - when exchanging large batches of small files, this will play its role. Ping Technoserv is not a technical possibility, the provider filters ICMP.
The data centers of our services are located in Moscow, the big western troika in the USA.
Load tests
To test the performance from Russia, we look at the speed of downloading files to the storage and the speed of downloading from an external to all experimental machines (the same as in previous experience). For the first test, take a large file, and for the second - a bunch of small files in bulk. We summarize all the results in a diagram and take a look at the result. Of course, we will work through the REST API, which supports one or another service - with the various file and block protocols that we are not interested in.
For clients from Russia, our services are clearly faster than the American Google and Microsoft, but Amazon is quite capable of arguing with them and even seriously outperforming the download speed of a large file. Although such a file is downloaded from domestic repositories significantly faster. With a set of small files, the picture is completely different - domestic services seriously overtake foreign ones, which was expected. Selectel and servers.ru public repositories showed an interesting feature: the content is downloaded from them much faster than it is downloaded (other services have the opposite). This once again confirms that these services are optimized for the distribution of static content and are good for a variety of hosting, but less suitable for corporate IT infrastructure.
Results
Domestic providers compete with the big Western troika. We tested public data stores for hot data, and having data centers in Russia (read, in Moscow) helps them a lot. Although Amazon is able to deliver high speeds on large (1.4 GB) files, our services run much faster with bundles of smaller ones, which is important: the main application scenarios still require working with a large number of small files. The prices are not so simple - there are options that are much cheaper than foreign analogues, and there are more expensive (the spread is quite large). At servers.ru and Selectel there are no inexpensive tariff plans for rarely used data. This will become a problem if you are going to deploy a virtual IT infrastructure of the company on their site (I would not want to pay for the storage of backups and archives in full). But they very quickly give static content and are good for hosting. Rostelecom's services are complicated to set up, but its list of supported APIs is the widest and the speed of storage is impressive. Although a separate tariff for rarely used data is not there either. For corporate use, the offers of Mail.ru and Technoserv are the most interesting - here you can get support for the S3 API, and separate rates for rarely used data, and decent access speeds. In general, there is plenty to choose from.
Another plus of Russian services is the absence of conflicts with the law (we all remember about working with personal data). Even collecting personal information about users of the site, you can inadvertently get a serious fine if the information is stored outside the country. Here, the leaders of the world market will not be able to oppose anything to local players (Amazon, Google and Microsoft have no data centers in Russia).
Source: https://habr.com/ru/post/348542/
All Articles