Cloud storage: new API functions

Recently, we talked about the transition to a new platform , thanks to which we were able to improve the work of cloud storage. We described in detail how we refined the logic and architecture of the repository and rewrote some of the components on Go, so that everything began to work much faster and more stable than before.

At the same time, we’ve told far from everything: during the work on the API, we launched several new features that we hope will be useful for you.

Authentication

Our repository fully supports the OpenStack Swift API , for which there are many third-party clients.
')
For compatibility with these clients, support for Swift v1 authentication has been implemented, and Keystone v2 and v3 authentication options have been limitedly implemented. Thus, three methods of authentication are now available in our API.
The first is almost the same as what was used before (only the authorization URI has changed to better conform to the standard):

$ curl -i https://api.selcdn.ru/auth/v1.0 -H "X-Auth-User:[ ]" -H "X-Auth-Key:[] 

In the second method (Keystone v2 type authentication), a POST request is used, not a GET; The username and password are passed in the JSON array:

 $ curl -i -X POST https://api.selcdn.ru/v2.0/tokens -H 'Content-type: application/json' -d '{"auth": {"passwordCredentials": {"username": [ ], "password": []}}}' 

The response to such an API request will also be returned in JSON format.

Authentication according to Keysone v3 looks like this:

 $ curl -i https://api.selcdn.ru/v3/auth/tokens -XPOST -d '{"auth": { "identity": { "methods": ["password"], "password": { "user": { "id": [ ], "password": []}}}}}' 

We draw your attention to the fact that the Keystone API in the repository is not fully implemented: only login and password authentication functions work. No extensions of the Keystone protocol are supported by us.

The base address of the repository has also changed. Now it looks like this: api.selcdn.ru/v1/SEL_XXX (instead of XXX, you must, of course, substitute the user ID).

Another change related to the authentication system is a new mechanism for temporary tokens. Now it is possible to generate additional tokens with a limited validity period, providing access only to a strictly defined container. Something that reminds Security Service in Amazon S3 :

 $ curl -i -X GET https://api.selcdn.ru/v1/temptokens -H "X-Auth-Token: [  ]" -H "X-Container:[]" -H "X-ExpireAfter: 8600" 

The valid token of the main user is transferred in the X-Auth-Token header, the container for which the token will be valid is indicated in the X-Container, and the duration of this token in seconds is indicated in the X-Expire-After.

Work with files

The new API expanded file handling capabilities. So, archives in tar, .tar.gz, gzip formats can be unpacked in the background after the file has been completely downloaded to the storage. The previous version of unpacking suggested unpacking archives on the fly, i.e. it was necessary to maintain a connection with our servers until the requests to create all the files from the downloaded archive are executed. And when the connection was broken, you had to download everything again.

Now everything is much easier. If, for example, you need to upload an archive containing several thousand files to the repository, this can be done using a PUT request to the container address with the optional query parameter extract-archive-v2. After the archive has been completely sent, code 201 is immediately given and background unpacking is started on the storage side:

 curl -i -X PUT https://api.selcdn.ru/v1/SEL_XXX/[ ]/?extract-archive-v2=tar.bz2 -H "X-Auth-Token: []" -T "photos.tar.bz2" HTTP/1.1 100 Continue HTTP/1.1 201 Created Access-Control-Expose-Headers: X-Backend-Timestamp, Etag, Last-Modified, X-Object-Manifest, X-Timestamp Content-Length: 0 Content-Type: text/html Etag: a1adb438cb26e91228870158a2062ef2 Extract-Id: 6a62579d-9ee2-2a32-26a4-207d5a47af2a 

Notice the Extract-Id header present in the response. With its help you can find out at what stage the unpacking operation is:

 $ curl -i https://api.selcdn.ru/v1/extract-archive/[Extract-Id] -H "X-Auth-Token: []" HTTP/1.1 200 OK Number Files Created: 1185 Response Body: Response Status: 201 Created Errors: 

In our case, the response with code 200 indicates that the unpack operation was completed successfully. The Number Files Created parameter indicates the number of files extracted from the archive.

New features

Many new features have appeared in the API, in particular, the management of users, domains, certificates and working with CDNs. We will tell about them in more detail below.

user management

Previously, adding and deleting additional users was possible only through the control panel. Now it can be done through standard HTTP requests to the API.

You can view the list of users as follows:

 $ curl -i -XGET https://api.selcd.ru/v1/users[?format=json] -H "X-Auth-Token: []" 

Creating a new user is carried out using the query:

 $ curl -i -XPUT https://api.selcd.ru/v1/users/new_user -H "X-Auth-Token: []" -H "X-Auth-Key: 123456" -H "X-User-ACL-Containers-W: container1, container2, container3" -H "X-User-ACL-Containers-R: container4" -H "X-User-Store-Password: yes" -H "X-User-Active: on" 

We have already received requests to expand rights management for users. We have fulfilled these requests, and this can be seen from the example of the request just cited: pay attention to the X-User-ACL-Containers-W and X-User-ACL-Containers-R headers. The first indicates the containers in which the new user can write information, and the second indicates the containers, access to which will be opened only for reading.

The X-Auth-Key header indicates the password of the user being created. X-User-Store-Password is equivalent to the “Store password on Selectel servers” option in the control panel. The X-User-Active header with a value of on indicates that the new user will be active.

You can change the data of an already created user like this:

 $ curl -i -XPOST https://api.selcd.ru/v1/users/new_user -H "X-Auth-Token: []" -H "X-Auth-Key: 123456" -H "X-User-ACL-Containers-W: container1, container4" -H "X-User-ACL-Containers-R: container4" -H "X-User-Store-Password: yes" -H "X-User-Active: on" 

Deleting a user is done using a DELETE request:

 $ curl -i -XDELETE https://selcdn.ru/users/[ ] -H "X-Auth-Token: []" 

For the primary user, only the password can be changed. This is done using a POST request with an X-Auth-Key header sent to api.selcd.ru/v1/users [primary user name].

Domain Management

Previously, it was also possible to work with domains exclusively through a graphical interface. Now all operations can be performed through the API.

So, for example, you can view a list of all domains associated with containers:

 $ curl -i https://api.selcdn.ru/v1/domains -H "X-Auth-Token: []" 

You can attach a domain using a POST request to the corresponding container with the X-Add-Container-Domains header:

 $ curl -i https://api.selcdn.ru/v1/SEL_XXX/{container_name} -XPOST -H "X-Add-Container-Domains: domain1.ru" -H "X-Auth-Token: []" 

To delete a domain, a similar request is used, only the domain name is passed in the X-Remove-Container-Domains header:

 $ curl -i https://api.selcdn.ru/v1/SEL_XXX/{container_name} -XPOST -H "X-Remove-Container-Domains: domain1.ru" -H "X-Auth-Token: []" 

To completely change the list of domains linked to the container, you need to perform a POST request to the address of this container and pass the updated list in the X-Container-Domains header:

 $ curl -i https://api.selcdn.ru/SEL_XXX/[ ] -H "X-Auth-Token: []" -XPOST -H "X-ontainer-Domains: domain1.ru domain2.ru" 

If several domains are attached to the container and they need to be deleted, then in the X-Container-Domains header you can simply pass the number 0:

 $ curl -i https://.selcdn.ru/[ ] -H "X-Auth-Token: []" -XPOST -H "X-ontainer-Domains: 0" 

Request logs

Through the API, you can request logs of network access to user containers for a specified period:

 $ curl -i -XPOST https://api.selcdn.ru/v1/logs -H "X-Auth-Token: []" -H "X-Start-Time: 2016-02-02 09:00:00" -H "X-End-Time: 2016-05-02 12:00:00" -H "X-Limit: 10000" 

The headings X-Start-Time and X-End-Time, as it is not difficult to guess, indicate the beginning and end of the period.

The optional X-Limit header indicates the maximum number of log entries that will be issued in this request (by default, you will receive all the logs for the specified time interval).

Upon completion of the sample, a private logs container will be automatically created in the storage, where the logs will be placed.

SSL certificate management

Another new feature is the management of SSL certificates. You can work with certificates both through a graphical interface and through an API.

So, for example, you can view a list of all certificates attached to containers:

 $ curl -i https://api.selcdn.ru/v1/ssl -XGET -H "X-Auth-Token: []" 

To view information about the certificate, use the following query:

 $ curl -i https://api.selcdn.ru/v1/ssl/{cert_name} -XGET -H "X-Auth-Token: []" 

The request to add a certificate looks like this:

 $ curl -i https://api.selcdn.ru/v1/ssl/{cert_name} -XPUT -H "X-Auth-Token: []" 

The certificate name ({cert_name}) is transmitted in the format 001_cert1, where the number is the user login and the second component is an arbitrary name. Certificate names cannot duplicate each other. The certificate itself and the private key are transmitted in the request body as a single file.

To delete a certificate, you need to send a DELETE request of the form:

 $ curl -I https://api.selcdn.ru/v1/ssl/001_cert1 -H "X-Auth-Token: []" -XDELETE 

Work with CDN

Appeared in the API and the ability to work with CDN. You can send requests to clear the cache (previously this could be done only through a graphical interface), as well as view the status of the execution of these requests.

Clearing the CDN cache is done using a PURGE query:

 $ curl -i -X PURGE https://api.selcdn.ru/v1/cdn -H "X-Auth-Token: []" -d 'https://api.selcdn.ru/v1/SEL_XXX/container1/file1\nhttps://XXX.selcdn.ru/container1/file1' 

Now we use akamai ccu v3 with fast purge, and after sending such a request, the cache should be cleared for about 5 seconds. The response to the request looks like this:

 {"estimatedSeconds": 5, "purgeId": "0bb4d6fa-4ce4-11e6-b75e-9fdde8cfe544", "supportId": "17PY1468845301403318-210404544", "httpStatus": 201, "detail": "Request accepted"} 

Conclusion

In this article, we talked about new cloud storage API features. Soon work on improving the storage will continue. And if you have your own wishes for improving and expanding the functionality - speak out in the comments to this post. We will take note of the most interesting ideas and try to implement them.