Telegram. Unlimited network drive. Free
Hi, Habr.
Surely many people thought about it, maybe someone has this idea in TODO. She lay with me for about a year, but still managed to implement it in the form of a working prototype.
Implementation details and link to the repository under the cut.
')
The idea to create a VFS based on Telegram appeared while working on AudioTubeBot and VideoTubeBot - after finding a way to get around the 50Mb limit on downloading files for bots - I studied the source code of Telegram - I discovered a lot of interesting things. For example, the download speed is limited mostly by the client itself, and not by the server.
As a result, I increased the download speed to Telegram cloud up to ~ 240 Mb / s per session (it has been working steadily for over a year), which is 15 times faster than the official client.
The logical step was to use unlimited file storage with a high download speed as a network drive — surely many people do, but the problem is that telegram clients are inconvenient to use for searching and organizing files. In addition - there is a limit of 1.5 GB per file size.
To solve these problems, I first decided to make a patch of the Linux kernel and make my virtual file system that would allow us to conveniently group the files uploaded to Telegram, while using only metadata.
In the process of studying the issue, it turned out that the patch kernel is not necessarily because Linux has a special FUSE kernel module that allows developers to create new types of file systems that can be mounted by users without privileges. For the module, there are wrappers for almost all popular languages.
To access Telegram, I use the Telethon library version 0.19.1 with a modified multi-threaded file download (TelegramClientX).
Not to write a file system from scratch - dedupfs in python is taken as a basis, using fuse and sqlite to store metadata that splits files into blocks, considers sha1 hash and reuses identical blocks - you just need to replace the methods for loading data into the storage with your own.
By combining all this, a working prototype of tgcloud has turned out, so far without cosmic velocities, but still a worker, I hope to involve the community in the development. The downloaded files are divided into blocks of 10MB, sha1 is calculated and loaded into SavedMessages, the hash is saved in the local database and the file is found when the block is read. The blocks are glued together in the correct order and a stream of bytes is issued — as if you are reading them from the disk.
.
— , .. — « », — — , , — — . — , , 90% , VFS — .
— , — , Telegram — « — .. ..»
DEDIC.SH Intel Scalable 2019 :
Xeon Silver 4214 — 15210 /
— !
— VDS 150 / — , !
Surely many people thought about it, maybe someone has this idea in TODO. She lay with me for about a year, but still managed to implement it in the form of a working prototype.
TgCloud:
- Virtual file system with open source g **** code.
- On the local disk - only metadata: names, size, folder structure, etc.
- Data is stored in Telegram and downloaded only when working with a file.
- The size and type of files is not limited, can be used with any OS.
Implementation details and link to the repository under the cut.
')
Idea
The idea to create a VFS based on Telegram appeared while working on AudioTubeBot and VideoTubeBot - after finding a way to get around the 50Mb limit on downloading files for bots - I studied the source code of Telegram - I discovered a lot of interesting things. For example, the download speed is limited mostly by the client itself, and not by the server.
As a result, I increased the download speed to Telegram cloud up to ~ 240 Mb / s per session (it has been working steadily for over a year), which is 15 times faster than the official client.
The logical step was to use unlimited file storage with a high download speed as a network drive — surely many people do, but the problem is that telegram clients are inconvenient to use for searching and organizing files. In addition - there is a limit of 1.5 GB per file size.
FUSE
To solve these problems, I first decided to make a patch of the Linux kernel and make my virtual file system that would allow us to conveniently group the files uploaded to Telegram, while using only metadata.
In the process of studying the issue, it turned out that the patch kernel is not necessarily because Linux has a special FUSE kernel module that allows developers to create new types of file systems that can be mounted by users without privileges. For the module, there are wrappers for almost all popular languages.
Telethon
To access Telegram, I use the Telethon library version 0.19.1 with a modified multi-threaded file download (TelegramClientX).
Vfs
Not to write a file system from scratch - dedupfs in python is taken as a basis, using fuse and sqlite to store metadata that splits files into blocks, considers sha1 hash and reuses identical blocks - you just need to replace the methods for loading data into the storage with your own.
Tgcloud
By combining all this, a working prototype of tgcloud has turned out, so far without cosmic velocities, but still a worker, I hope to involve the community in the development. The downloaded files are divided into blocks of 10MB, sha1 is calculated and loaded into SavedMessages, the hash is saved in the local database and the file is found when the block is read. The blocks are glued together in the correct order and a stream of bytes is issued — as if you are reading them from the disk.
To run the code
- You need to get api_hash and api_id on my.telegram.org and insert this data along with the phone number of your account in tg_access.py
- Create a session by running from the project folder
python3.6 download_service.py sudo yum install python-fusemkdir storage- VFS:
python dedupfs/dedupfs.py -df --block-size 10240000 storage/python dedupfs/dedupfs.py --block-size 10240000 storage/ - , filebrowser
.
P.S. ( )
— , .. — « », — — , , — — . — , , 90% , VFS — .
— , — , Telegram — « — .. ..»
Telegram
GitHub
DEDIC.SH Intel Scalable 2019 :
- 2x Xeon Silver 4214 — 24
- 2x Xeon Gold 5218 — 32
- 2x Xeon Gold 6240 — 36 .
Xeon Silver 4214 — 15210 /
— !
— VDS 150 / — , !
Source: https://habr.com/ru/post/456290/
All Articles