File storage system with deduplication between users
This project began with a very simple idea, about which, I wrote a year ago in the pages of habrabrabr. That's when I wondered why all file storage services are so greedy? Why do they earn that several different users store the same file? Why do we have to pay twice, thrice, and sometimes hundreds of times more than it actually costs? In my opinion this is unfair.
The second problem with all paid file storage services is subscription fees. We are forced to constantly pay for the storage of our files. If you do not pay, the files will be deleted. More like extortion than a convenient service, especially in the light of this study on the cost of storing data. Therefore, I decided that it was time to create a project that was devoid of these shortcomings.
So there was an idea, and what came out of it can be read under habrakat.
MD5 File is a personal file storage system with deduplication. While others earn money by storing multiple identical files with different users, the service suggests sharing the cost of storing a file among all users. Thus, permanent file storage can be practically worthless for the user, and the maximum storage cost does not exceed $ 1 per gigabyte. This is a one-time fee. No subscription fees are charged.
')
When a file is loaded, the system checks if there are other users who use the same file and calculates the cost of storage. For example, if 10 users store the same 10 gigabyte file, then in fact, each of them will only use 1 gigabyte of their space. As the number of users increases, the system automatically recalculates the space occupied by the file and the free space of all users of the file increases.
I really want the project not to become “another file storage system”, because besides simply uploading your files, especially for a long time, you need to have a convenient search and systematization system. Personally, I don’t really like the system with folders, as it doesn’t allow you to simultaneously add a file to several different groups, so I decided that it was necessary to implement tag support. That is why the service supports a very flexible search system that allows you to set, for example, such rules:
The same system is implemented in webdav, i.e. You can mount a webdav disk and make links to it by tags, for example, \ md5file \ storage \ paris \ children or \ md5file \ storage \ family \ children in any combination. Currently, the webdav service is in beta-readonly mode at
Link to project: MD5 File
PS Unfortunately, registration via social networks is not yet supported, due to the fact that a password is required for WebDav. In the future we plan to implement such an authorization.
The second problem with all paid file storage services is subscription fees. We are forced to constantly pay for the storage of our files. If you do not pay, the files will be deleted. More like extortion than a convenient service, especially in the light of this study on the cost of storing data. Therefore, I decided that it was time to create a project that was devoid of these shortcomings.
So there was an idea, and what came out of it can be read under habrakat.
MD5 File is a personal file storage system with deduplication. While others earn money by storing multiple identical files with different users, the service suggests sharing the cost of storing a file among all users. Thus, permanent file storage can be practically worthless for the user, and the maximum storage cost does not exceed $ 1 per gigabyte. This is a one-time fee. No subscription fees are charged.
')
How it works.
When a file is loaded, the system checks if there are other users who use the same file and calculates the cost of storage. For example, if 10 users store the same 10 gigabyte file, then in fact, each of them will only use 1 gigabyte of their space. As the number of users increases, the system automatically recalculates the space occupied by the file and the free space of all users of the file increases.
I really want the project not to become “another file storage system”, because besides simply uploading your files, especially for a long time, you need to have a convenient search and systematization system. Personally, I don’t really like the system with folders, as it doesn’t allow you to simultaneously add a file to several different groups, so I decided that it was necessary to implement tag support. That is why the service supports a very flexible search system that allows you to set, for example, such rules:
,!- searches for all files with a tag or name paris , but without a tag or name wife,!tag:,name=2012- searches for all files with a tag or name paris , but without the tag i , whose name has the text 2012|tag:,ext:pdf- searches for all files with a tag or name paris or with a tag wife and pdf extension
The same system is implemented in webdav, i.e. You can mount a webdav disk and make links to it by tags, for example, \ md5file \ storage \ paris \ children or \ md5file \ storage \ family \ children in any combination. Currently, the webdav service is in beta-readonly mode at
webdav.md5file.com/storageWhat are your plans for the future?
- Download files from torrents and HTTP / FTP links
- WebDav (full support)
- File sharing between users
- Integration with other file storage services
- Instant file download (if available on the server)
Link to project: MD5 File
PS Unfortunately, registration via social networks is not yet supported, due to the fact that a password is required for WebDav. In the future we plan to implement such an authorization.
Source: https://habr.com/ru/post/138080/
All Articles