How to compact up to 90% storage of backups in object storage

Our Turkish clients asked us to configure the backup for the data center correctly. We are doing similar projects in Russia, but it was here that the story was more about researching how best to do it.

Given: there is a local S3 storage, there is Veritas NetBackup, which has acquired new advanced functionality for moving data to object stores now with deduplication support, and there is a problem with free space in this local storage.

Objective: to make everything so that the backup storage process is fast and cheap.
')
Actually, before that, in S3 everything was just files, and these were complete casts of critical data center machines. That is not so that it is very optimized, but everything worked at the start. Now it's time to figure it out and do it right.

In the picture, what we have come to:



As you can see, the first backup was done slowly (70 Mb / s), and subsequent backups of the same systems were much faster.

Actually, further a little more details about what features are there.

What is the problem

Customers want to backup as often as possible and store as cheaply as possible. It is cheap to store them best in S3 type object storages, because they are the cheapest at the price of maintenance per megabyte from where you can roll back up in a reasonable amount of time. When there are a lot of backups, it becomes not very cheap, because most of the storage is occupied by copies of the same data. In the case of HaaS, Turkish colleagues can compact the storage by about 80-90%. It is clear that this applies specifically to their specifics, but I would definitely count on at least 50% of the dedup.

To solve the problem, the main vendors have long made gateways on Amazon S3. All of their methods are compatible with local S3 if they support the Amazon API. In the Turkish data center, backup is done in our S3, as well as in the T-III “Compressor” in Russia, since such a scheme of work proved to be good with us.

And our S3 is fully compatible with backup methods in Amazon S3. That is, all the backup tools that support these methods allow you to copy everything to such a storage “out of the box”.

Veritas NetBackup made a CloudCatalyst feature:



That is, between the machines that need to be backed up, and the gateway there is an intermediate Linux server through which the backup traffic from the CPC agents passes and deduplication is done on the fly before transferring them to S3. If before there were 30 backups of 20 GB each with compression, now (due to the similarity of the machines) they have become 90% smaller in volume. The deduplication engine is used the same as when stored on regular disks using Netbackup.

Here's what happens before the staging server:



We tested and concluded that when implemented in our data centers, this saves space in S3 storages for us and for customers. As the owner of commercial data centers, of course, we charge for the occupied volume, but still it is very profitable for us too - because we start earning at more scalable places in software, and not on iron rental. Well, this is a reduction in internal costs.


The accelerator allows you to reduce traffic from agents, because only data changes are transmitted, that is, even full backups do not pour in whole, since the media server collects subsequent full backups from incremental backups.

The intermediate server has its own repository where it writes a "cache" of data and holds the base for deduplication.

In full architecture, it looks like this:

1. The master server manages the configuration, updates, and more, and is located in the cloud.
2. The media server (intermediate * nix machine) should be located closest to the redundant systems in terms of network availability. Here backups are deduplicated from all redundant machines.
3. There are agents on the redundant machines that generally send to the media server only what is not in its storage.

It all starts with a full scan - this is a full-fledged full backup. At this point, the media server takes everything, deduplicates it and transfers it to S3. The speed to the media server is low, from it - higher. The main limitation is the processing power of the server.

The following backups are made complete from the point of view of all systems, but in reality it is something like synthetic full backups. That is, the actual transfer and recording to the media server is only those data blocks that have not yet been seen in VM backups before. And the transfer and recording to S3 is only those data blocks whose hash is not in the deduplication database of the media server. If in simpler words - that there were no VMs in any backup before.

When restoring, the media server requests the necessary deduplicated objects from S3, rehydrates them and passes them to the CPC agents, i.e. it is necessary to take into account the volume of traffic during the restore, which will be equal to the real volume of data being restored.

Here's what it looks like:




The integrity of the data is ensured by the protection of S3 itself - there is good redundancy for protection against hardware failures such as a dead hard drive spindle.

The media server needs 4 terabytes of cache - this is Veritas’s recommendation for the minimum size. Better more, but we did just that.

Total

When a partner threw 20 GB into our S3, we stored 60 GB, because we provide three-time geo-reservation of data. Now the traffic is much less, which is good for the channel, and for storage charging.

In this case, the routes are closed past the "big Internet", but you can drive traffic through VPN L2 via the Internet, but it’s better to set the media server to the provider’s entrance.

If you are interested in learning about these features in our Russian data centers or if you have questions about how to implement it, ask in the comments or ekorotkikh@croc.ru mail.