The console version of Cyberduck: working with cloud storage

In previous publications, we have repeatedly told about utilities that can be used to work with our cloud storage ( 1 and 2 ). In one of the articles, we already mentioned the Cyberduck utility, a convenient file manager for MacOS, Linux and Windows, that works with the FTP, SFTP, WebDAV, OpenStack Swift and AmazonS3 protocols. From our official website, you can also download a profile through which Cyberduck is automatically configured to work with our cloud storage.

Users are mostly familiar with the graphical version of Cyberduck. A console version has just appeared. In this article we will talk about its capabilities and show how it can be used to work with the repository.

general information

')
The console version of Cyberduck works with all major operating systems - MacOS, Windows and Linux. The program can be used as an FTP and SFTP client, as well as for working with various cloud storage services.
The documentation for the console version of Cyberduck is published on the official website, but, unfortunately, most of the important functions are not described in such detail as we would like. Below we will talk about what operations with our repository can be performed using it, and also give detailed explanations of the syntax of the main commands.

Installation

Mac OS

The console version of Cyberduck for MacOS is installed using the Homebrew package manager:

 $ brew install duck

Linux

In this article, we describe the installation procedure for Ubuntu 14.04; users of other distributions refer to the official documentation .

To install the console version of Cyberduck, we first add the corresponding repository:

 $ echo 'deb https://s3.amazonaws.com/repo.deb.cyberduck.io nightly main'> / etc / apt / sources.list

 $ echo 'deb https://s3.amazonaws.com/repo.deb.cyberduck.io stable main'> / etc / apt / sources.list

Then add the key:

  $ sudo apt-key adv - keyserver keyserver.ubuntu.com --recv-keys FE7097963FEFBE72

and execute commands

 $ sudo apt-get update
 $ sudo apt-get install duck

Basic cloud storage operations

All commands for working with cloud storage are as follows:

 $ duck - <argument> swift: // <username> @ auth.selcdn.ru / <object path> -p <password>

After entering the command, the program will ask for the name of the account owner (Tenant). In response, you need to specify the user under which account access to the repository is performed. The dialogue mode can be turned off altogether - the -q option is used for this.

Getting a list of files in a container

To get a list of files stored in a container, use the -l (or −−list) option:

 $ duck -l swift: //username@auth.selcdn.ru/ <container path> -p <password>

 Listing directory images ...
 1.jpg
 2.jpg
 3.png

File download

To download a file from the repository, use the command:

 $ duck -d swift: //username@auth.selcdn.ru/ <file path> <file name> -p <password>

Opening a file for editing on a local machine

Using the console version of CyberDuck, you can open files for editing on a local machine; upon completion of editing, the updated (with all changes) version of the file will be downloaded to the repository. For this, the ––edit option is used:

 $ duck --edit swift: // <username> @ auth.selcdn.ru / <file path> -p <password>

The file will be opened in the application used in the system for this file type; downloading a modified version starts automatically.
This Cyberduck feature will be especially interesting for users hosting static sites in our storage. For example, to quickly edit the text on the site, it is enough to run the above command, make changes to the necessary files and save them.

Loading object into storage

The load operation is performed using the command:

 $ duck --upload swift: //username@auth.selcdn.ru <full path to object in storage> <path to object on local machine> -p <password>

Pay special attention to the fact that when loading an object you need to specify the full path to the storage location of the object itself. For example, if we want to save the file myimage.png in the images container, then the path to it must be specified as follows: / images / myimage.png.

Large (larger than 2GB) Cyberduck objects are loaded into the storage in parts.

Object Versions and Backup

The console version of Cyberduck is a handy tool for backing up and archiving data. Consider these functions in more detail on specific practical examples.

Imagine that we have a directory on the local machine, the contents of which need to be periodically copied to the cloud storage. To do this, a special script was written and a Cron task was added, which sends a backup copy to the vault every day at a specified time.
The script looks like this:

 #! / bin / bash
 SWIFT_USERNAME = username
 SWIFT PASSWORD = password to access the repository
 SWIFT_AUTH_URL = auth.selcdn.ru
 BACKUP_PATH = path to the backup storage location
 LOCAL_PATH = folder path on local machine

 duck --upload swift: // $ SWIFT_USERNAME @ $ SWIFT_AUTHURL / $ BACKUP_PATH / $ LOCAL_PATH --existing rename --password $ SWIFT_PASSWORD -q

Note the syntax of the duck command. In the given example, the key is used --existing, which indicates what to do with the files already in the repository. The rename option renames an existing backup, adding time and date to its name.

Differential backups can also be performed using cyberduck. The compare option is used for this:

 $ duck --upload swift: //username@auth.selcdn.ru <full path to the object in the repository> <path to the file on the local machine> --existing compare -p <password>

When executing the above command, the program compares the loadable backup with the one already available in size, change date and checksum. If the parameters are different, then the old version will be deleted, and the new version will be loaded into the repository.

When using the skip option in the repository, only new files will be uploaded (those that appeared in the folder on the local machine after the previous download). Already existing files will not be uploaded, even if they have been changed on the local machine.

Finally, the overwrite option will simply remove the existing backup from the repository and load a new one.

Synchronization of local files with files in storage

File synchronization is a process in which two directories, one of which is located on the local machine, and the other in the repository, will contain the same set of files in the same version with the most recent change date. If any files on the local machine have been changed, added or deleted, the same files will be changed, added or deleted in the repository, and vice versa.

Synchronization is performed using the command:

 $ duck --synchronize swift: // <username@auth.selcdn.ru> / <path to the folder in the repository> <path to the folder on the local machine>

Using the synchronization function, you can keep backup copies of data from the local machine up to date in the storage.
Here is an example of a simple script:

 # bin / bash
 SWIFT_USERNAME = username
 SWIFT PASSWORD = password to access the repository
 SWIFT_AUTH_URL = auth.selcdn.ru
 BACKUP_PATH = path to the backup storage location
 LOCAL_PATH = folder path on local machine

 duck --synchronize swift: // $ SWIFT_USERNAME @ SWIFT_AUTHURL / $ BACKUP_PATH $ LOCAL_PATH --password $ SWIFT_PASSWORD -q

It is enough to add the corresponding task to cron - and the data will be automatically synchronized with the specified periodicity.

This feature will be useful for those who host static sites in the repository. To update the site, it is enough to make the appropriate changes to the files on the local machine, and then execute the synchronization command.

Copying files

To copy a file from one container to another, use the command:

 $ duck --copy swift: // <username@auth.selcdn.ru> / <full path to file> <username@auth.selcdn.ru> / <path to new storage location> -p <password>

Option -v

To display information on all HTTP requests made when performing operations with the repository, as well as on responses to them, the -v (or —verbose) option is used. This helps to understand how third-party applications interact with the repository.

Conclusion

The console version of Cyberduck is a handy tool for working with cloud storage, which has many features. The emergence of such a tool should please users of Windows. The fact is that for Windows, but recently there were no console programs for working with cloud storages based on OpenStack Swift at all, and for these purposes we had to use FTP-clients, which is not always convenient.
We hope that you will appreciate the console version of Cyberduck will be used in its daily practice.

Readers who for one reason or another cannot post comments here are invited to our blog .