s3ql - file system based on cloud storage

Description

With S3QL, you can create a file system based on Selectel Storage cloud storage, which can be mounted in any modern version of Linux, FreeBSD or Mac OS X.

Opportunities

Transparency

S3QL is almost indistinguishable from the local file system. It supports hardlinks, symlinks, standard system permissions, extended attributes, and files up to 2TB in size.

Dynamic size

The size of the file system is not limited, it increases when data is saved and decreases after deletion. The cached data is stored on the local disk.
')

Compression

Before saving to cloud storage, data can be compressed (lzma, bzip2, zlib is supported).

Encryption

After compression (but before loading into the repository), the data can be encrypted with the AES encryption algorithm with a 256-bit key. An additional SHA256 HMAC checksum is also used to protect data from external manipulation.

Data deduplication

If several files have the same content, then duplicate data will be recorded only once, which allows you to efficiently store a large amount of similar data, with a minimal amount of space. This functionality works both on whole files and on data blocks that are repeated in different files.

Copy-on-write / snapshotting

S3QL can quickly copy entire directories without using additional disk space. Only when making changes to one of the copies, the modified data will take up additional space.

Data protection against change

You can prohibit changing data for directories, for example, to store backups that will be protected from accidental deletion.

High performance regardless of network latency

The operations associated with changing the structure (creating directories, moving files and changing rights) are performed very quickly, as they do not require network interaction.

Support for slow communication channels

S3QL splits the contents of files into small blocks and caches them locally, which reduces the number of network calls to read and write data, and also reduces the amount of data transferred, since only the changed data blocks are transmitted over the network.

Installation

The s3ql package is available in the Debian (sid) and Ubuntu repositories, the installation of which comes down to the usual run of apt-get install s3ql.
If you do not have this package in the repository or you are not satisfied with its version, you can always install it manually using the instructions .

Creating a file system

mkfs.s3ql swift://auth.selcdn.ru/container
By default, all inodes are divided into blocks of 10MB, this parameter can be set when creating the file system - max-obj-size. After creating the file system, this parameter cannot be changed.
Also by default, when creating a file system, you will need to enter a password to encrypt data, but if you do not need encryption, you can specify the option --plain and then the data will not be encrypted, which can increase file system performance.

Mounting

Connecting cloud storage is very simple:
mount.s3ql swift://auth.selcdn.ru/container /mnt/container

Compression

When mounting a file system, you can specify the necessary data compression algorithm, but you have to understand that the stronger the compression, the greater the load on the processor and the lower the speed when saving data. Compression algorithms relate to each other as follows: lzma is slow, but has better compression relative to bzip2, and bzip2, in turn, has better compression relative to zlib, but slower. For maximum file system performance, it is better to choose a compression algorithm based on network bandwidth.

Caching

S3QL supports local file system data cache for faster access. The cache is based on data blocks, so it is possible that only parts of the files are in the cache.

Additional commands

s3qladm - used to change the password for encryption, update the file system, delete all data and download metadata from the repository;
fsck.s3ql - sometimes after the loss of the network connection to the storage or after the fall of the s3ql itself, it is necessary to check the file system, without which there is no possibility of further mounting;
s3qlcp - allows replicating directories within a single s3ql file system;
s3qlctrl - to control the current settings of the mounted file system, for example, cache size and logging level, and also allows you to force flushcache and load meta data into the repository;
s3qllock - makes folders unchangeable;
s3qlrm - recursively deletes files and directories on the s3ql file system, it can also delete directories that are marked as immutable;
s3qlstat — Displays statistics for a mounted file system.

Unmounting

umount.s3ql /mnt/container
Only the user who mounted it or root can unmount a file system. When unmounting, additional data is loaded into the storage, which have not yet been recorded - the command is blocked until the download is complete.

Total

pros

• client-side encryption, which provides storage of fully encrypted data in the storage;
• using data deduplication, you can reduce the cost of storing data in the cloud relative to the traditional approach;
• Caching data reduces the cost of outgoing traffic for storage.

disadvantages

• There is no possibility of multiple mounting (simultaneously on different machines).

PS For more detailed acquaintance with s3ql you can use its documentation.

The source of the article .