Cloud.Mail.Ru + EncFS without local file storage

Synopsis

It's no secret that many of us have 1 TB of free space on the MRU-Cloud since its beta test. The volume is decent by today's standards - but what to do with it? Photos and your video there just do not really want to upload - hacking accounts occur often, and in any case - the cloud is a cloud, and you can not discount the simple fact that any cloud storage belongs to a commercial company in whose interests to use it for own benefit. So, we need an additional layer of protection, for example, EncFS. We look through Habr and githab, like there is a decision to encrypt data in your cloud. But there is an unobvious, but very important inconvenience, which is not mentioned in the original article - in order to synchronize, you need to locally store 600GB of encrypted photos . For modest local repositories, in which even unencrypted 600GB photos barely fit, this is too much.

Idea

Therefore, an ambitious plan was conceived - to write a FUSE-file system for the Cloud, and already mount EncFS on top of it (for better security, generate a GPG key and use ecryptfs, because modern computer capacities allow you to hack EncFS). In some way, this will be similar to how using the MEGA cloud we drag a file into the browser window — E2E encryption is applied and the file is sent to the cloud, without occupying anything locally. It was decided to call the name "MARC-FS" - MAail.Ru Cloud FileSystem. With its help, we move away from the indicated problem of local storage of encrypted data - by copying the data into the EncFS directory, we will automatically encrypt it and send it to the cloud. In fact, a lower level, we will call a rather long chain of transformations of the form cp -> kernel FUSE module -> userspace libfuse hook -> EncFS -> FUSE kernel -> libfuse hook -> MARC-FS -> kernel network stack, but a simplified scheme of work would look something like this:

+---------------+ | | | Local data | | | | /media/DATA/ | | | +---------------+ | | | | +-------v-------+ | | | EncFS | | | | ~/remote-enc | | | +---------------+ | | FUSE | encryption | +-------v-------+ | | | MARC-FS | | | | ~/remote | | | +---------------+ | | FUSE | networking | +-------v-------+ | | | | | cloud.mail.ru | | | | | +---------------+ 

Implementation

The plan is ambitious, but in fact I had to face many limitations, both from the FUSE and the core, and from the Cloud. The most probably disturbing thing is that data deduplication is used in the Cloud. That is, files that have the same hashes and sizes are stored in some common pool, and users after downloading simply add a hardlink to them. Understandably, your encrypted files will be unique, because sewn with your key, in connection with which there is a fear as if Mail.ru did not pay attention to such personal clouds and did not make a complaint about this. Hopefully, there were not very many OS X / Linux users who dared to even test Mail.ru for beta.

Some other interesting details:

• The cloud does not support downloading from Transfer-Encoding: chunked , and in the FUSE interface there is no way to know in advance the size of the file before it is written. It is necessary to first load the file into memory, and only then, knowing the size, send over the network.
• EncFS requires random read / write for its work, because it splits files into pieces, which it encrypts later. For network file systems (NFS, for example), this is fear and horror, because the protocols do not contain instructions like "write 4096 bytes at this offset from the beginning of the file." Therefore, for EncFS to work correctly, it is necessary to keep the file in memory while working with it (Somewhat later, I added the ability to specify the cache directory in which intermediate data will be stored, for more details, see README.md in the repository - author's note).
• The connection to the cloud is established via https, and this, suddenly, adds a rather large overhead memory to initialize the SSL engine. Moreover, it is not entirely clear how to deal with this, and other network file systems also face this .
• Although the size of the storage and terabytes, the Cloud has a limit on the size of the file uploaded by a single request - 2 GB, and only paid services make it possible to upload more. It is suitable for a library of photos or music, but it’s impossible to upload your collection of HD movies, so later I added a transparent split-file — if you upload or download files larger than 2GB, the file system will break or paste them itself.
• MacOS uses file descriptor multiplexing, which is its difference from Linux, so I had to link the project not with the system libfuse present, but with osxfuse using Homebrew Cask. It contains appropriate workarounds. Also, of the possible problems, I can note that the mounting of one FUSE file system to another on MacOS is disabled by default due to problems with recursions, but this is bypassed using the manual mounting options.

As a result (and maybe, nevertheless) it turned out to be a viable option, with the help of which it was possible to flood the entire collection of music and photos and anime to the cloud in the form of encrypted files. The result of the work on MARC-FS is presented here , there you can also find instructions on how to configure the bundle with EncFS. Work is underway and the front of improvements is wide, pull requests and suggestions are, of course, welcome.

An example of an encrypted directory:

Locally decrypted:

On the cloud:

A couple of margin notes

Could not find any description of API Clouds, I had to pull out from related articles and projects. But we must pay tribute to Mail.Ru - even when working in 25 threads and hundreds of getattr / read / write requests per second only occasionally, 500 Internal Server error skips in response. I don't see him so often, but if someone from the Clouds team reads this article and needs details, I can pull out a few of these reports.

libfuse when writing the file system, and in general the architecture itself FUSE reminded of its own concept of userspace client-server drivers, for example in GNU / Hurd or Plan 9 From Bell Labs . It's nice to see that interesting ideas that were abandoned with the loss of relevance of these systems, have received a new life.

Multithreading for the FS had to be invented from scratch and on the knee, so it was an interesting challenge. There is no clear certainty that everything turned out at its best (and without bugs), but it works - and it works pretty quickly. Unless there are doubts about how the Cloud itself will react to the simultaneous uploading and downloading of files.

It should be warned that I did not test MARC-FS on Mac OS X in detail. My friend and I achieved a stable build and copied music to the terminal a couple of times, but no more. I do not have Apple hardware at hand and I can not effectively test the work of the file system, so if someone can support the work of the file system on this platform, I will be glad.

Disclaimers

I publish the code and this article in the hope that they can be useful to someone, but I do not provide guarantees of presentation or suitability for use. The internal API of the Cloud can change at any second, and the opportunity to end up with a trough is rather high. Use at your own risk.

I know that in the Mail.Ru license agreement there are lines about the prohibition of automatic interaction tools, but none of the legal_dep could understand my question in letters and objectively clarify the situation. In the end, we all use email clients that interact in the same way programmatically with Mail.ru. I was redirected twice to D. A., who honestly replied that legal advice was not his path.

Links

• Code
• Readme

PS Before I am warned about the possible consequences of writing file systems - I already know .