Rclone: ​​rsync for clouds

In previous publications, we have repeatedly told about useful utilities for working with our cloud storage. Today we will talk about one more interesting, easy to use and - not afraid of this word - unique tool. Meet rclone . The developers describe it with a brief and succinct phrase "rsync for cloud storage."

The main function of rclone is to synchronize data in the storage and on the local machine. The utility will undoubtedly be useful for a wide range of cloud storage users. It can be used both for backup and for working with static sites ...
')
Rclone also has options that no other tool of the same kind has. More about all this we will discuss below.

Installation and initial setup

The undoubted advantage and undoubted advantage of rclone over other products of the same plan is the support of many operating systems: Linux, Windows, MacOS, Solaris, FreeBSD, OpenBSD, NetBSD and Plan 9.
Links for downloading packages for all the mentioned operating systems can be found on the download page .

We will describe the features of c rclone on the Linux OS material. To install, we need to download the required package, and then execute:

$ unzip rclone-current-linux-amd64.zip $ cd rclone-current-linux-arm $ sudo cp rclone /usr/sbin/ $ sudo chown root:root /usr/sbin/rclone $ sudo chmod 755 /usr/sbin/rclone $ sudo mkdir -p /usr/local/share/man/man1 $ sudo cp rclone.1 /usr/local/share/man/man1/ $ sudo mandb 

After the installation is complete, configure rclone to work with the Selectel cloud storage:

 $ rclone config 

The following dialog will appear on the console:

 No remotes found - make a new one
 n) New remote
 q) Quit config
 n / q> 

Select n and press Enter. Next we will need to specify the name of the connection to the remote repository:

  name>

Specify any name (for example, Selectel) and go to the next item:

 1 / Amazon Cloud Drive
    \ "amazon cloud drive"
  2 / Amazon S3 (also Dreamhost, Ceph)
    \ "s3"
  3 / Backblaze B2
    \ "b2"
  4 / Dropbox
    \ "dropbox"
  5 / Google Cloud Storage (this is not Google Drive)
    \ "google cloud storage"
  6 / Google Drive
    \ "drive"
  7 / Hubic
    \ "hubic"
  8 / Local Disk
    \ "local"
  9 / Microsoft OneDrive
    \ "onedrive"
 10 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
    \ "swift"
 11 / Yandex Disk
    \ "yandex"

Select the number 10 (swift) and press Enter. After that, the program will ask for a username and password:

 Authentication URL for server.
 Choose a number from below, or type in your own value
  1 / Rackspace US
    \ "https://auth.api.rackspacecloud.com/v1.0"
  2 / Rackspace UK
    \ "https://lon.auth.api.rackspacecloud.com/v1.0"
  3 / Rackspace v2
    \ "https://identity.api.rackspacecloud.com/v2.0"
  4 / Memset Memstore UK
    \ "https://auth.storage.memset.com/v1.0"
  5 / Memset Memstore UK v2
    \ "https://auth.storage.memset.com/v2.0"
  6 / OVH
    \ "https://auth.cloud.ovh.net/v2.0"

Our storage is not in the list, so we’ll specify the address manually:

 auth> https://auth.selcdn.ru/v1.0

The following two points (tenant and region) are optional and can be skipped. In the last question of the dialogue, we will be asked to check all the settings again:

 Remote config
 --------------------
 [selectel]
 user = your_username
 key = your_password
 auth = https://auth.selcdn.ru/v1.0
 tenant = 
 region = 
 --------------------
 y) Yes this is OK
 e) Edit this remote
 d) Delete this remote
 y / e / d> 

If everything is correct, select the option y and press Enter.

Command Examples

The command syntax for working with the repository is simple:

 #      $ rclone lsd selectel: #    $ rclone mkdir selectel:[ ] #      $ rclone ls selectel:[ ] #        rclone copy /home/local/directory #         $ rclone sync /home/local/directory selectel:[ ] #          $ rclone selectel:[ ] sync /home/local/directory 

When performing copy and synchronization operations, rclone checks all files by date and time of change or md5-sum. The files that have been modified are transferred from the source directory to the destination directory.
We will not describe all the commands in detail in this article: you can read about everything in the official documentation . Also, a brief help can be obtained using the command:

 $ rclone --help 

Most of the functions of rclone are the same as those of other tools for working with cloud storage. But it has one unique function that none of the tools we know of has: data transfer from one cloud storage to another.

Consider the following practical example: we have a folder with photos on Google Docs, and its contents need to be transferred to our cloud storage. Using rclone is very simple. Create a new connection; In the list of available cloud storage, select Google Drive. After that, we will be asked to specify two parameters: client_id and client_secret. In response to the relevant questions, we do not enter anything and just press Enter.

Next we will be asked the following question:

 Remote config
 Use auto config?
  * Say Y if not sure
  * Say N if you didn’t work
 y) Yes
 n) No
 y / n>

Select the answer is "no" (n). Rclone will generate a link to get the code:

 If your browser doesn't open automatically, go to the following link: https://accounts.google.com/o/oauth2/auth?client_id=202264815644.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg% 3Aoauth% 3A2. 0% 3Aoob & response_type = code & scope = https% 3A% 2F% 2Fwww.googleapis.com% 2Fauth% 2Fdrive & state = ac901aefe97aff8ce65fe593060d0b0c
 Log in and authorize rclone for access

Open this link in the browser and give rclone permission to access the files.

After this, the Google Drive API will return the code to be inserted in response to the question:

 Enter verification code>

That's all. Connection to Google Drive is configured, and you can start copying:

 $ rclone copy [ ]:[ ] [selectel]:[ ] 

The task of copying files rclone copes quite quickly: we copied a folder with photos of 1.8 GB in 1 minute 55 seconds.

During the experiments, we also found out that rclone also copies GoogleDocs text documents without any problems, converting them into the docx format.

Loading large objects

Let's check the speed of rclone in one more test: we will try to load a large object into the storage - more than 20 GB. Files up to 20 GB are uploaded to the storage using standard commands. The procedure for downloading larger files is different: the file is divided into segments, which are loaded into a separate container.

Rclone by default divides such files into segments of 5GB each. If necessary, the segment size can be changed using the option -swift-chunk-size. We tried to load a 25 GB file into the storage. Rclone handled this task in 11 minutes 14 seconds. The result, as we see, is quite good.

Conclusion

Rclone is quite an interesting and promising tool that can be recommended for use. If it seems to you that we have forgotten to tell something, let us know about it, and we will definitely expand our review.

And if you are already familiar with rclone - share your experience in the comments.
If for some reason you can not leave comments here - welcome to our blog .