Cloud file storage support in 1C-Bitrix: Site Management 11
With this article we are opening a whole cycle of posts devoted to new products implemented in the new, 11th version of “1C-Bitrix: Site Management” .
One of the most interesting features that appeared in the new release is support for cloud storage.
')
In this post we will tell you what it is, what kind of repositories are, how they work, and how you can use them with great benefit on your website.
First, let's look at any new or already working web project?
What does its owner strive for? With the exception of highly specialized resources - the growth of the site, the expansion of its audience, the increase in the profit (monetary - in an explicit form, or "profit by attention"), which brings the project. A good and understandable goal for any online store, information resource, social network, etc.
Such a goal by itself implies several smaller tasks that need to be learned in the most efficient way to solve:
You can invent several bicycles to solve all these problems: put several servers, instead of one, to ensure reliability; permanently upgrade iron as it grows; render statics on a separate file server ...
The solution, it seems to us, is not very flexible.
It is much more convenient, and most importantly - more effective (in our opinion) - to learn how to work in a cloud infrastructure, use its services, scale and be ready to develop not just a site, but a real cloud service. ( We have already written about the advantages of the “cloud” compared to the traditional hosting on Habré , we invite all those who are not indifferent to the topic) to the discussion).
That is why in the 11th version of “1C-Bitrix Site Management” a module appeared to support the most common cloud storages .
What is it - cloud storage?
If you try to describe literally in two words: for a user, this is a large-large server for storing static content that can be distributed very quickly via HTTP. :) From the inside (from the point of view of the provider) - everything is “slightly” more complicated: all data in the repository is replicated at several points, which ensures its reliability; There is an API for working with files in the repository, etc. (For some features, we will write further in more detail.)
The idea of the cloud storage module in our product is very simple - any files that the system user can work with can be absolutely transparent for him to be stored not on the local server of the website, but in the cloud storage.
What problems do we solve by transferring files to the "cloud"?
1. Reducing the cost of operation
In the general case, the cost of placing files in the cloud will be lower than using a file server of a similar size. If you carry out independent calculations, do not forget to calculate the cost of the server (you can’t use a “bare” disk;)), organization of backups, traffic, etc.
2. We can use it in conjunction with CDN to speed up content delivery.
Almost all cloud storage providers offer customers a CDN (Content Delivery Network or Content Distribution Network) - a geographically distributed network infrastructure that allows you to optimize content delivery and distribution to end users on the Internet, Wikipedia ).
3. We reduce the load on web-sites
Your application servers can only run scripts and deliver dynamic content to users. Web server processes are not busy with static return (especially true for video and distribution kits). Plus reduces the load on the disk.
4. Using a centralized repository, we solve the problem of synchronizing content between multiple web sites.
If you use a load balancer and several web servers, you need to solve the problem of synchronizing content between them. Options - either periodically start synchronization on local storage (rsync, csync2), which is quite problematic when working with large amounts of data; or use one or another centralized repository. For example, cloudy.
5. Accelerate rendering pages in the browser
If all site images are sent from the main domain (for example, www.site.ru ), then almost all browsers by default open a limited number of connections to a single domain (most often - 5).
Therefore, even if the speed of your Internet channel allows you to receive content from the site very quickly, you will be limited to the number of connections. Visually, the page in the browser will not be drawn too quickly.
If the pictures are placed on a separate domain relative to the site itself - this speeds up the download of all the necessary content.
Cloud storage providers
We support Amazon S3 cloud storage, Google Storage, Microsoft's Windows Azure Storage, RackSpace, OpenStack.
Almost all of them offer approximately similar functionality (from the user's point of view):
However, of course, have their own characteristics.
Amazon Simple Storage Service (Amazon S3)
The service implemented within the framework of the whole complex of cloud services from Amazon - Amazon Web Services (AWS).
Can be shared with another Amazon service - CloudFront - Amazon's own CDN
Files from S3 can be distributed not only via HTTP, but also via BitTorrent protocol.
An interesting feature - you can use a special type of storage - Reduced Redundancy Storage (RRS). In this case, less reliability of data is guaranteed (compared to standard storage), however, users are offered a lower cost of data storage. Can be used for renewable content.
Prices (for standard storage; vary by region):
Google Cloud Storage
The storage implemented by Google, mainly for data storage when working with Google App Engine (but, of course, can be used in other applications).
Content can be delivered via Google CDN.
Until the end of 2011, there is an opportunity to use the free trial offer (the size of the storage is up to 5 GB, there are limits on traffic and the number of requests). A good offer for those who want to just try and evaluate the service.
Prices (vary by region):
Windows Azure Storage
Storage created by Microsoft companies in the framework of the development of the Windows Azure cloud platform.
Have your own CDN. Additional services are interesting (for example, Table Service, Queue Service).
Users can use the free trial for the entire Windows Azure platform as a whole, including by accessing the storage (trial limits - 20 GB of storage, 50,000 transactions).
Prices (vary by region):
Rackspace Cloud Files
Rackspace is one of the world's largest cloud providers. Which among other services offers its customers and cloud storage.
Files can be distributed through the CDN, which is organized in partnership with Akamai .
Prices:
Already mentioned earlier, Rackspace and NASA were developing the Nebula cloud platform.
In mid-2010, it was decided to open this platform and create a new project on its basis - OpenStack.
Currently, OpenStack is a whole complex of software that anyone can use to create their own cloud. One of the components of OpenStack - OpenStack Swift - a complex for organizing file storage.
The project is supported by very many companies (Citrix, Dell, AMD, Intel, etc.)
Universal support of the OpenStack Swift API at the 1C-Bitrix platform level allows any hoster to implement its own file storage based on OpenStack - and any owner of the website created on Bitrix can immediately use it (without additional costs for integration and learning a new specific API) .
The first (and, as far as we know, the only ones so far) in Russia, own cloud-based file storage based on OpenStack implemented Clodo.ru .
Some technical features of the implementation of the storage are described in the blog Clodo.ru on Habré .
As a result, you can buy a hosting (virtual machine) from Clodo under “1C-Bitrix”, immediately with it - a license for the product, and then connect the cloud storage to the project. Great package deal for customers !
We urge Russian hosting providers to develop their own cloud solutions!
We hope that the emergence of such solutions will give impetus to the development of cloud infrastructure and CDN-networks in Russia.
How does all this work in the 1C-Bitrix platform?
One or several storages can be connected to any site operating on 1C-Bitrix. This is done in the administrative interface - this is done simply, just specify a few parameters:
In the simplest version, you can simply transfer all project files to the cloud by selecting the desired item in the context menu:
In the future, all new files (photos in the photo gallery, pictures of product descriptions in the online store, user avatars in social networks, etc.) will automatically be uploaded to the cloud, and links to them will be automatically generated in the right way.
If you want more flexibility in managing files, you can use the system of “filter rules” implemented in the platform.
Rules are configured:
For example, you can connect two different repositories and for each of them set up your own filter rules. Suppose that all files over 100 MB are moved to the Google Storage cloud, and all videos are moved to Amazon S3. Depending on the profitability of the providers' offers, you can change your rules by “pouring” data into other “cloud” folders. Or completely turn off the storage, which became expensive or inconvenient.
A little bit of "insides" for developers
Most importantly, we consider the possibility of “transparent” work with the “cloud” very important for customers. That is why the cloud storage support module is included in all product editions, starting with “Start”.
At first glance, the implementation of support for the “clouds” in the platform may seem rather simple. However, our developers had to modify about 30 modules in the system! All of them somehow work with files. And for a user who decides to “move to the cloud,” everything should be transparent.
API for developers working with files remained the same transparent (you can read in detail in Maxim Smirnov’s blog ).
And this means that any developer, working with the platform API, can fully use the full functionality of cloud storage.
* * *
The 1C-Bitrix platform has become even more flexible for both end users and developers. There are new tools for scaling sites and effectively solving other problems.
We strive to make 1C-Bitrix a platform not only for creating websites, but also for developing cloud web services, large scalable web projects hosted in the cloud.
One of the most interesting features that appeared in the new release is support for cloud storage.
')
In this post we will tell you what it is, what kind of repositories are, how they work, and how you can use them with great benefit on your website.
First, let's look at any new or already working web project?
What does its owner strive for? With the exception of highly specialized resources - the growth of the site, the expansion of its audience, the increase in the profit (monetary - in an explicit form, or "profit by attention"), which brings the project. A good and understandable goal for any online store, information resource, social network, etc.
Such a goal by itself implies several smaller tasks that need to be learned in the most efficient way to solve:
- Minimizing operating costs and reducing financial risks at the start of the project
- Scaling with increasing load and reverse scaling
- Reliability - ensuring SLA, with which, perhaps, a different level of SLA for different categories of customers
- Fast return of dynamic and static content
You can invent several bicycles to solve all these problems: put several servers, instead of one, to ensure reliability; permanently upgrade iron as it grows; render statics on a separate file server ...
The solution, it seems to us, is not very flexible.
It is much more convenient, and most importantly - more effective (in our opinion) - to learn how to work in a cloud infrastructure, use its services, scale and be ready to develop not just a site, but a real cloud service. ( We have already written about the advantages of the “cloud” compared to the traditional hosting on Habré , we invite all those who are not indifferent to the topic) to the discussion).
That is why in the 11th version of “1C-Bitrix Site Management” a module appeared to support the most common cloud storages .
What is it - cloud storage?
If you try to describe literally in two words: for a user, this is a large-large server for storing static content that can be distributed very quickly via HTTP. :) From the inside (from the point of view of the provider) - everything is “slightly” more complicated: all data in the repository is replicated at several points, which ensures its reliability; There is an API for working with files in the repository, etc. (For some features, we will write further in more detail.)
The idea of the cloud storage module in our product is very simple - any files that the system user can work with can be absolutely transparent for him to be stored not on the local server of the website, but in the cloud storage.
What problems do we solve by transferring files to the "cloud"?
1. Reducing the cost of operation
In the general case, the cost of placing files in the cloud will be lower than using a file server of a similar size. If you carry out independent calculations, do not forget to calculate the cost of the server (you can’t use a “bare” disk;)), organization of backups, traffic, etc.
2. We can use it in conjunction with CDN to speed up content delivery.
Almost all cloud storage providers offer customers a CDN (Content Delivery Network or Content Distribution Network) - a geographically distributed network infrastructure that allows you to optimize content delivery and distribution to end users on the Internet, Wikipedia ).
3. We reduce the load on web-sites
Your application servers can only run scripts and deliver dynamic content to users. Web server processes are not busy with static return (especially true for video and distribution kits). Plus reduces the load on the disk.
4. Using a centralized repository, we solve the problem of synchronizing content between multiple web sites.
If you use a load balancer and several web servers, you need to solve the problem of synchronizing content between them. Options - either periodically start synchronization on local storage (rsync, csync2), which is quite problematic when working with large amounts of data; or use one or another centralized repository. For example, cloudy.
5. Accelerate rendering pages in the browser
If all site images are sent from the main domain (for example, www.site.ru ), then almost all browsers by default open a limited number of connections to a single domain (most often - 5).
Therefore, even if the speed of your Internet channel allows you to receive content from the site very quickly, you will be limited to the number of connections. Visually, the page in the browser will not be drawn too quickly.
If the pictures are placed on a separate domain relative to the site itself - this speeds up the download of all the necessary content.
Cloud storage providers
We support Amazon S3 cloud storage, Google Storage, Microsoft's Windows Azure Storage, RackSpace, OpenStack.
Almost all of them offer approximately similar functionality (from the user's point of view):
- Any number of objects (most often - up to several TB each)
- The possibility of placing in different data centers (regions)
- Object Grouping
- Authorization and ACL mechanisms
- REST and SOAP interfaces for working with objects
- HTTP Direct File Upload
- High availability
- Low price
- Access via external tools (FUSE, clients)
However, of course, have their own characteristics.
Amazon Simple Storage Service (Amazon S3)
The service implemented within the framework of the whole complex of cloud services from Amazon - Amazon Web Services (AWS).
Can be shared with another Amazon service - CloudFront - Amazon's own CDN
Files from S3 can be distributed not only via HTTP, but also via BitTorrent protocol.
An interesting feature - you can use a special type of storage - Reduced Redundancy Storage (RRS). In this case, less reliability of data is guaranteed (compared to standard storage), however, users are offered a lower cost of data storage. Can be used for renewable content.
Prices (for standard storage; vary by region):
- Storage - 1 GB (up to 1 TB) $ 0.14 / month.
- $ 0.01 for 1000 PUT, COPY, POST or LIST requests
- $ 0.01 for 10,000 GET requests
- Traffic - 1 GB (up to 10 TB) $ 0.12 (the first 1 GB is free)
Google Cloud Storage
The storage implemented by Google, mainly for data storage when working with Google App Engine (but, of course, can be used in other applications).
Content can be delivered via Google CDN.
Until the end of 2011, there is an opportunity to use the free trial offer (the size of the storage is up to 5 GB, there are limits on traffic and the number of requests). A good offer for those who want to just try and evaluate the service.
Prices (vary by region):
- Storage - 1 GB (up to 1 TB) $ 0.13 / month.
- $ 0.01 for 1000 PUT, COPY, POST or LIST requests
- $ 0.01 for 10,000 GET requests
- Traffic - 1 GB (up to 1 TB) $ 0.12
Windows Azure Storage
Storage created by Microsoft companies in the framework of the development of the Windows Azure cloud platform.
Have your own CDN. Additional services are interesting (for example, Table Service, Queue Service).
Users can use the free trial for the entire Windows Azure platform as a whole, including by accessing the storage (trial limits - 20 GB of storage, 50,000 transactions).
Prices (vary by region):
- Storage - 1 GB $ 0.15 / month
- $ 0.01 for 10,000 requests
- Traffic - 1 GB $ 0.15
Rackspace Cloud Files
Rackspace is one of the world's largest cloud providers. Which among other services offers its customers and cloud storage.
Files can be distributed through the CDN, which is organized in partnership with Akamai .
Prices:
- Storage - 1 GB $ 0.15 / month
- Requests for files less than 250 KB, as well as HEAD, GET, DELETE - for free
- Traffic - 1 GB $ 0.18
Already mentioned earlier, Rackspace and NASA were developing the Nebula cloud platform.
In mid-2010, it was decided to open this platform and create a new project on its basis - OpenStack.
Currently, OpenStack is a whole complex of software that anyone can use to create their own cloud. One of the components of OpenStack - OpenStack Swift - a complex for organizing file storage.
The project is supported by very many companies (Citrix, Dell, AMD, Intel, etc.)
Universal support of the OpenStack Swift API at the 1C-Bitrix platform level allows any hoster to implement its own file storage based on OpenStack - and any owner of the website created on Bitrix can immediately use it (without additional costs for integration and learning a new specific API) .
The first (and, as far as we know, the only ones so far) in Russia, own cloud-based file storage based on OpenStack implemented Clodo.ru .
Some technical features of the implementation of the storage are described in the blog Clodo.ru on Habré .
As a result, you can buy a hosting (virtual machine) from Clodo under “1C-Bitrix”, immediately with it - a license for the product, and then connect the cloud storage to the project. Great package deal for customers !
We urge Russian hosting providers to develop their own cloud solutions!
We hope that the emergence of such solutions will give impetus to the development of cloud infrastructure and CDN-networks in Russia.
How does all this work in the 1C-Bitrix platform?
One or several storages can be connected to any site operating on 1C-Bitrix. This is done in the administrative interface - this is done simply, just specify a few parameters:
In the simplest version, you can simply transfer all project files to the cloud by selecting the desired item in the context menu:
In the future, all new files (photos in the photo gallery, pictures of product descriptions in the online store, user avatars in social networks, etc.) will automatically be uploaded to the cloud, and links to them will be automatically generated in the right way.
If you want more flexibility in managing files, you can use the system of “filter rules” implemented in the platform.
Rules are configured:
- by system modules
- by file extensions
- by file size
For example, you can connect two different repositories and for each of them set up your own filter rules. Suppose that all files over 100 MB are moved to the Google Storage cloud, and all videos are moved to Amazon S3. Depending on the profitability of the providers' offers, you can change your rules by “pouring” data into other “cloud” folders. Or completely turn off the storage, which became expensive or inconvenient.
A little bit of "insides" for developers
Most importantly, we consider the possibility of “transparent” work with the “cloud” very important for customers. That is why the cloud storage support module is included in all product editions, starting with “Start”.
At first glance, the implementation of support for the “clouds” in the platform may seem rather simple. However, our developers had to modify about 30 modules in the system! All of them somehow work with files. And for a user who decides to “move to the cloud,” everything should be transparent.
API for developers working with files remained the same transparent (you can read in detail in Maxim Smirnov’s blog ).
And this means that any developer, working with the platform API, can fully use the full functionality of cloud storage.
* * *
The 1C-Bitrix platform has become even more flexible for both end users and developers. There are new tools for scaling sites and effectively solving other problems.
We strive to make 1C-Bitrix a platform not only for creating websites, but also for developing cloud web services, large scalable web projects hosted in the cloud.
Source: https://habr.com/ru/post/130595/
All Articles