Your Cocaine. Yandex cloud platform

We already told on Habré about the cloud infrastructure of Yandex. Today came the turn from words to action - we want to show in steps how to deploy your own cloud on Elliptics and Cocaine.

Scheme

Let's take a look at installing a small cloud in which you can run a test application using flask .

This cloudlet consists of the following elements:

• cocaine-runtime , which runs applications in Docker;
• Docker-registry for storing images of applications;
• Elliptics as a distributed application repository, as well as a cloud configuration;
• aggregating node cocaine-runtime - a single point of entry into the cloud for the client cocaine code;
• HTTP frontend as an alternative way to access applications.

At each stage we will carry out checks that test the success of the stage.
')


As you can see, we will need 5 (possible virtual) machines with a kernel of at least 3.8 to support Docker . Required packages can be found in the repository .

5 minutes on elliptics

I will have to briefly describe how to deploy the Elliptics installation from one machine to demonstrate the work of our cluster. Immediately, I note that this can in no way be considered as a guide for deploying this Elliptics for combat use. This is sure to tell you the guys who are engaged in the development of Elliptics. Also information can be found in the documentation .

In a couple of commands, it looks like this:

 sudo apt-get install elliptics=2.24.14.31 elliptics-client=2.24.14.31 mkdir /tmp/history/ && mkdir /tmp/root cp /usr/share/doc/elliptics/examples/ioserv.conf ./tst_ioserv.conf 

Further, in tst_ioserv.conf we will have to change literally 3 lines. So:

 group = 1 addr = localhost:1025:2-0 192.168.50.201:1025:2-1 //    IP indexes_shard_count = 16 

After this, run:

 dnet_ioserv -c tst_ioserv.conf 

Setup cocaine-runtime + Docker

Let's start the implementation of our mini-cloud with the configuration of machines that directly run the code of our application. The core of the installation will definitely be cocaine-runtime . You also need to install Docker itself and cocaine-plugin to work with Docker .

I’ll clarify that although our goal is to run the application in a docker container, we will also start the application as a normal process.

Containerization is a blessing. It solves environmental problems for the application:

• dependences are always with us;
• no conflicts between dependencies of two applications;
• invariance of the environment in dev, test, prod environments.

Running without a container is applicable and convenient when there are no dependencies (for example, an application for go) or technical ability to use containers (for example, the kernel is lower than the required version and cannot be updated).

After connecting the repository, install the necessary packages:

 sudo apt-get install cocaine-runtime libcocaine-core2 libcocaine-plugin-docker2 libcocaine-plugin-elliptics=2.24.14.31 elliptics-client=2.24.14.31 

To manage the applications, we need the cocaine-tool utility. The easiest way to install it is from PyPI , the standard repository of Python packages.

It is necessary to set the binding in Python for msgpack .

 sudo apt-get install msgpack-python 

And then cocaine-tools .

 sudo pip install cocaine-tools 

Docker is installed according to the description in the official documentation in any convenient way. The following path is used:

 curl -s https://get.docker.io/ubuntu/ | sudo sh 

After successful installation of packages, cocaine-runtime starts with the default configuration, which we need to change.
Stop the cocaine-runtime and proceed to the configuration modification.

 sudo service cocaine-runtime stop 

The default configuration file is located in /etc/cocaine/cocaine-default.conf and looks like this:

 { "version": 2, "paths": { "plugins": "/usr/lib/cocaine", "runtime": "/var/run/cocaine" }, "services": { "logging": { "type": "logging" }, "storage": { "type": "storage", "args": { "backend": "core" } }, "node": { "type": "node", "args": { "runlist": "default" } } }, "storages": { "core": { "type": "files", "args": { "path": "/var/lib/cocaine" } }, "cache": { "type": "files", "args": { "path": "/var/cache/cocaine" } } }, "loggers": { "core": { "type": "syslog", "args": { "identity": "cocaine", "verbosity": "info" } } } } 

Make the configuration cloudy. To do this, you need to configure 2 entities:

1. Configure work with distributed storage (in our example - Elliptics ). The thing is, cocaine-runtime reads from the storage a list of running applications, application launch profiles, application manifests. If you use the out-of-the-box file storage, you will have to maintain the consistency of this data on each node, which is extremely inconvenient. In the case of a distributed stack, this issue is resolved by itself.
2. To force runtime to inform the information on itself to aggregating cloud nodes.

For experiments, it is convenient to modify the copy of the provided config “by default” (for example, /etc/cocaine/cocaine-cloud.conf ).

To force cocaine-runtime to inform aggregating nodes about applications and services running on it, by adding the network section to the configuration at the same nesting level with services .

  "network" : { "group": "224.168.2.9" }, "services": { ... } 

The only group parameter describes the address of the multicast group to which alerts will be sent.
The storage configuration is performed by modifying the services / storage and storages / core sections:

 "storage": { "type": "elliptics" }, "storages" : { "core": { "type": "elliptics", "args": { "nodes" : { "192.168.50.201" : 1025 }, "io-thread-num" : 8, "wait-timeout" : 30, "check-timeout" : 60, "net-thread-num" : 8, "groups" : [1], "verbosity" : 2 } } } 

Implementing storage on Elliptics technology is provided by the appropriate plug-in. By the way, all the plugins from our packages are placed in / usr / lib / cocaine, where cocaine-runtime is looking for them by default.

Now let's make cocaine start with this config by default. Create
/etc/default/cocaine-runtime :
CONFIG_PATH = "/ etc / cocaine / cocaine-cloud.conf"

Run cocaine-runtime with the new config, make sure that the process works.


After starting on port 10053, the service locator of this node will wait for requests about the availability of the application. This is a kind of DNS for cloud applications and services, that is, by the name of the service or application you will know where to send the requests so that they are processed.

You can request information about applications on this node using the command cocaine-tool info . The output of this command does not shake the imagination, but remember it - it will change very soon.

Recall that in addition to cocaine-runtime and plug-ins, we installed Docker . Check that it also started successfully:


We assume that similar actions will be performed on the second machine. In total, we have 2 cars running cocaine-runtime that look into shared storage - that's great!

At this point, it is already possible to run applications. This, of course, is not a cloud at all, but still. Roll up the sleeves, let's start.

First of all, we need to deliver the application code to the cocaine storage. This problem is solved with ease using the cocaine-tool .

Clone the repository with an example and go into the directory.

 git clone git@github.com:cocaine/cocaine-framework-python.git -b v0.11 cd cocaine-framework-python/examples/flask/ 

You may notice that in addition to the application code, manifest.json is located here:

 { "slave": "main.py" } 

The main purpose of this file is to tell the platform what to run. You can also pass the environment .

To upload the application code that we want to run as a process to the cloud, run the following command:

 cocaine-tool app upload --name example 

cocaine-tool will pack your application to the archive and upload it to the cloud storage. And then we look at the list of downloaded applications.



Here it is, our application! The main thing is not to forget to install flask , which is required by our application. Here is the disadvantage of running without a container.

 sudo apt-get install python-flask 

Next, you need to configure the resources allocated for the application, the type of isolation, the number of workers and so on. For this purpose, serve profiles. A profile can be associated with multiple applications.

The simplest profile is empty JSON {} . In this case, the default settings are used.

A description of all available options can be found in the description . A more interesting example will be considered later when we run the application in a container.

Let's name our profile default and upload it to the cloud storage. Check that it boots correctly.


Finally, the time has come when you can pull the switch and the application will start:

 cocaine-tool app start --name example --profile default 

Now the output of the cocaine-tool info command has changed. It shows which applications are running on this node and statistics about applications.

Let's try to send a message to our application. For this purpose, such a small python script is sufficient:

 #!/usr/bin/env python from cocaine.services import Service app = Service("example") print(app.enqueue("write", "DATA").get()) print(app.enqueue("read", "DATA").get()) 

The application is still running only on one node, but you can go to the second one and run it there. Now you can continue to build the infrastructure for applications in Docker .

Docker-registry

Docker-registry is your private repository of images (containers) of applications. You can read more about it in the docker official documentation .

Docker-registry supports many backends for storing images. The assortment starts with writing the images to the local file system and extends to storing the images in Elliptics .

The easiest way to run the Docker registry is to use Docker. To do this, we need to put Docker (see above) and give the command to run the desired image.

 sudo docker run -p 5000:5000 registry 

The image of the registry for the first time will be loaded from the repository, cached and the next time it will run quickly.

After starting the Docker-registry will listen to port 5000 . Ping it:

 curl "http://192.168.50.4:5000/_ping" 

This completes the setup.

Build and Deploy Container Applications

The application application in the case of a container will differ little from a non-container one. The most important difference is that Dockerfile is required to build the image. In essence, it consists of a set of shell commands that must be executed to form the internal state of the image. After that, the image is poured in the Docker-registry.

Then any Docker that wants to run the container can download an image from the registry. Dockerfile syntax, available commands can be found here. Instead of a Dockerfile, you can provide a Chef recipe or Puppet manifest that will be executed inside the container during assembly.

The profile for this application must specify the Docker isolation type. To do this, create a new profile.

 { "queue-limit": 1000, "pool-limit": 10, "isolate": { "type": "docker", "args": { "memory_limit": 1000000000, "endpoint": "unix:///var/run/docker.sock", "registry": "registry.cloud.net:5000", "cpu_shares": 0 } }, "concurrency": 200 } 

We have placed the profile in the file named docker-profile.json, we will load it with the name docker-profile

 cocaine-tool profile upload --name docker-profile --profile=docker-profile.json 

In this example, we have a ready Dockerfile, and the application is downloaded to the cloud as follows:

 sudo cocaine-tool app upload --docker=unix:///var/run/docker.sock --registry=registry.cloud.net:5000 --manifest manifest-docker.json --name example-docker --timeout 20000 

This is not an instant process, it all depends on the size of your image and network. But as a result, you should see a similar inscription. Next, the application starts.


One moment is lost sight of what applications cocaine-runtime starts at startup and how it finds out about it. For this there is a runlist . In essence, this associative array of applications and their profiles, which cocane-runtime reads at startup. It is located, as well as profiles, in storage . Since the sets of running applications on different nodes of the cloud may differ, then the runlists on them may be different. The ranlist name is specified in the cocaine-runtime config in the runlist parameter of the node service.

Configuring the aggregation node

In essence, this part of the configuration is very similar to the item “Configuring cocaine-runtime”. This is natural, since cocaine-runtime acts as an aggregating node. All the work on balancing in it is performed by the gateway plugin .

We have two implementations of this plugin. The first is called adhoc . It is available out of the box and does not require the installation of additional packages. The second implementation we use in combat is called ipvs . As the name implies, it uses the same technology. Adhoc is reasonable to use when it is not possible to use ipvs . And of course, you can write your own implementation of this plugin to use the technology you like to balance it.

From words to deeds. Install the packages:

 sudo apt-get install cocaine-runtime libcocaine-core2 libcocaine-plugin-ipvs2 libcocaine-plugin-elliptics=2.24.14.31 elliptics-client=2.24.14.31 

If it is not possible to use ipvs , then there is no need to install the last of the above packages. The differences in the base configuration will be minor and will be listed separately.

To make runtime an aggregating node will allow adding a network section with a subsection of the gateway . We assume that, as before, we correct the copy of the “default” config. The storage service is configured in the same way as the previous item - it will be needed soon.

  "network": { "group": "224.168.2.9", "gateway": { "type": "ipvs" // adhoc } }, "services"... 

After restarting sudo service cocane-runtime restart should start the process:

 cocaine /usr/bin/cocaine-runtime --daemonize --configuration /etc/cocaine/cocaine-gateway.conf --pidfile /var/run/cocaine/runtime.pid 

In fact, the gateway takes a number of parameters (you can read about them in the documentation).

Let's open the log (by default it is written to syslog). You may notice that the connection was registered two nodes, and then one of them turned off. You can do similar operations with your backends and make sure discovery works.

Cocaine-native-proxy

Cocaine-native-proxy allows you to go to cloud applications via HTTP. Installing and configuring it is extremely simple.

 apt-get install cocaine-native-proxy 

Then we need to specify a list of aggregating nodes in the locators section. In our case, such a node is one. Strictly speaking, it is possible to specify the addresses of locators and nodes that do not work in the aggregation mode. This is bad for the reason that such a node knows nothing about services and applications on other nodes, which means you will not have access to them. A more detailed description of the parameters can be found at github.com/cocaine/cocaine-native-proxy/blob/master/README.md

 { "endpoints": [ "0.0.0.0:8080" ], "backlog": 2048, "threads": 2, "application": { "locators": ["192.168.50.103:10053"], "service_pool": 5, "reconnect_timeout": 180, "request_timeout": 5 } } 

The point of entry into the cloud in our installation is a machine with an aggregate node. Despite the fact that the HTTP interface is brought out through cocaine-native-proxy , in fact, this proxy also uses an aggregation node. We pointed it out in the config in the locators section. Let's try to call our applications in two ways. The HTTP client will use the http event handler, and from the Python client code, we will handle the write and read events.

The client code now indicates the address of the entry point to the cloud. In my case - 192.168.50.103.

 #!/usr/bin/env python from cocaine.services import Service app = Service("example", host="192.168.50.103") print(app.enqueue("write", "DATA").get()) print(app.enqueue("read", "DATA").get()) 

As an event handler for the http event, an application is called on flask. Currently, HTTP-proxy supports two ways of determining which application to send which event to. The first method assumes that the passed URL is patterned: ///tail?arg=1&args=2. URL . /tail?arg=1&args=2 . .

X-Cocaine-Service, X-Cocaine-Event . nginx .

, :
curl "http://localhost:8080/read" -H "X-Cocaine-Service: example" -H "X-Cocaine-Event: http" curl "http://localhost:8080/example/http/read"



. , . , . . . , , (, 5%). . . , , .

. example , exampleGroup . example 1000 . .
service/locator: adding group 'exampleGroup'


exampleGroup . , , app.py hello:
@app.route('/') def hello(name=None): return "HELLO! I'm version #2"

, example2 .



curl "http://localhost:8080/exampleGroup/http/"

, . example2 0 , HTTP-proxy . , HTTP-proxy . , .


, , . , Cocaine , c, . . ///tail?arg=1&args=2. URL . /tail?arg=1&args=2 . .

X-Cocaine-Service, X-Cocaine-Event . nginx .

, :
curl "http://localhost:8080/read" -H "X-Cocaine-Service: example" -H "X-Cocaine-Event: http" curl "http://localhost:8080/example/http/read"



. , . , . . . , , (, 5%). . . , , .

. example , exampleGroup . example 1000 . .
service/locator: adding group 'exampleGroup'


exampleGroup . , , app.py hello:
@app.route('/') def hello(name=None): return "HELLO! I'm version #2"

, example2 .



curl "http://localhost:8080/exampleGroup/http/"

, . example2 0 , HTTP-proxy . , HTTP-proxy . , .


, , . , Cocaine , c, . .

 ///tail?arg=1&args=2.  URL   .       /tail?arg=1&args=2 .      . 
 
          X-Cocaine-Service, X-Cocaine-Event .      nginx . 
 
  ,       : 
 curl "http://localhost:8080/read" -H "X-Cocaine-Service: example" -H "X-Cocaine-Event: http" curl "http://localhost:8080/example/http/read" 
 
     
 
               .    ,               .           ,    .          .    .  ,       ,       (, 5%).     .        .        ,       ,       . 
 
    .       example ,  exampleGroup .     example   1000 .      .        
 service/locator: adding group 'exampleGroup' 
  
 
           exampleGroup .    ,    ,   app.py    hello: 
 @app.route('/') def hello(name=None): return "HELLO! I'm version #2" 
 
           ,   example2 . 
  
 
    
 curl "http://localhost:8080/exampleGroup/http/" 
 
   ,      .    example2  0 ,      HTTP-proxy     .   ,   HTTP-proxy    .   ,   . 
 
   
     ,     ,    .        ,     Cocaine   ,    c,   .      . 

///tail?arg=1&args=2. URL . /tail?arg=1&args=2 . .

X-Cocaine-Service, X-Cocaine-Event . nginx .

, :
curl "http://localhost:8080/read" -H "X-Cocaine-Service: example" -H "X-Cocaine-Event: http" curl "http://localhost:8080/example/http/read"



. , . , . . . , , (, 5%). . . , , .

. example , exampleGroup . example 1000 . .
service/locator: adding group 'exampleGroup'


exampleGroup . , , app.py hello:
@app.route('/') def hello(name=None): return "HELLO! I'm version #2"

, example2 .



curl "http://localhost:8080/exampleGroup/http/"

, . example2 0 , HTTP-proxy . , HTTP-proxy . , .


, , . , Cocaine , c, . .

///tail?arg=1&args=2. URL . /tail?arg=1&args=2 . .

X-Cocaine-Service, X-Cocaine-Event . nginx .

, :
curl "http://localhost:8080/read" -H "X-Cocaine-Service: example" -H "X-Cocaine-Event: http" curl "http://localhost:8080/example/http/read"



. , . , . . . , , (, 5%). . . , , .

. example , exampleGroup . example 1000 . .
service/locator: adding group 'exampleGroup'


exampleGroup . , , app.py hello:
@app.route('/') def hello(name=None): return "HELLO! I'm version #2"

, example2 .



curl "http://localhost:8080/exampleGroup/http/"

, . example2 0 , HTTP-proxy . , HTTP-proxy . , .


, , . , Cocaine , c, . .

///tail?arg=1&args=2. URL . /tail?arg=1&args=2 . .

X-Cocaine-Service, X-Cocaine-Event . nginx .

, :
curl "http://localhost:8080/read" -H "X-Cocaine-Service: example" -H "X-Cocaine-Event: http" curl "http://localhost:8080/example/http/read"



. , . , . . . , , (, 5%). . . , , .

. example , exampleGroup . example 1000 . .
service/locator: adding group 'exampleGroup'


exampleGroup . , , app.py hello:
@app.route('/') def hello(name=None): return "HELLO! I'm version #2"

, example2 .



curl "http://localhost:8080/exampleGroup/http/"

, . example2 0 , HTTP-proxy . , HTTP-proxy . , .


, , . , Cocaine , c, . .

 ///tail?arg=1&args=2.  URL   .       /tail?arg=1&args=2 .      . 
 
          X-Cocaine-Service, X-Cocaine-Event .      nginx . 
 
  ,       : 
 curl "http://localhost:8080/read" -H "X-Cocaine-Service: example" -H "X-Cocaine-Event: http" curl "http://localhost:8080/example/http/read" 
 
     
 
               .    ,               .           ,    .          .    .  ,       ,       (, 5%).     .        .        ,       ,       . 
 
    .       example ,  exampleGroup .     example   1000 .      .        
 service/locator: adding group 'exampleGroup' 
  
 
           exampleGroup .    ,    ,   app.py    hello: 
 @app.route('/') def hello(name=None): return "HELLO! I'm version #2" 
 
           ,   example2 . 
  
 
    
 curl "http://localhost:8080/exampleGroup/http/" 
 
   ,      .    example2  0 ,      HTTP-proxy     .   ,   HTTP-proxy    .   ,   . 
 
   
     ,     ,    .        ,     Cocaine   ,    c,   .      . 

///tail?arg=1&args=2. URL . /tail?arg=1&args=2 . .

X-Cocaine-Service, X-Cocaine-Event . nginx .

, :
curl "http://localhost:8080/read" -H "X-Cocaine-Service: example" -H "X-Cocaine-Event: http" curl "http://localhost:8080/example/http/read"



. , . , . . . , , (, 5%). . . , , .

. example , exampleGroup . example 1000 . .
service/locator: adding group 'exampleGroup'


exampleGroup . , , app.py hello:
@app.route('/') def hello(name=None): return "HELLO! I'm version #2"

, example2 .



curl "http://localhost:8080/exampleGroup/http/"

, . example2 0 , HTTP-proxy . , HTTP-proxy . , .


, , . , Cocaine , c, . .

 ///tail?arg=1&args=2.  URL   .       /tail?arg=1&args=2 .      . 
 
          X-Cocaine-Service, X-Cocaine-Event .      nginx . 
 
  ,       : 
 curl "http://localhost:8080/read" -H "X-Cocaine-Service: example" -H "X-Cocaine-Event: http" curl "http://localhost:8080/example/http/read" 
 
     
 
               .    ,               .           ,    .          .    .  ,       ,       (, 5%).     .        .        ,       ,       . 
 
    .       example ,  exampleGroup .     example   1000 .      .        
 service/locator: adding group 'exampleGroup' 
  
 
           exampleGroup .    ,    ,   app.py    hello: 
 @app.route('/') def hello(name=None): return "HELLO! I'm version #2" 
 
           ,   example2 . 
  
 
    
 curl "http://localhost:8080/exampleGroup/http/" 
 
   ,      .    example2  0 ,      HTTP-proxy     .   ,   HTTP-proxy    .   ,   . 
 
   
     ,     ,    .        ,     Cocaine   ,    c,   .      . 

///tail?arg=1&args=2. URL . /tail?arg=1&args=2 . .

X-Cocaine-Service, X-Cocaine-Event . nginx .

, :
curl "http://localhost:8080/read" -H "X-Cocaine-Service: example" -H "X-Cocaine-Event: http" curl "http://localhost:8080/example/http/read"



. , . , . . . , , (, 5%). . . , , .

. example , exampleGroup . example 1000 . .
service/locator: adding group 'exampleGroup'


exampleGroup . , , app.py hello:
@app.route('/') def hello(name=None): return "HELLO! I'm version #2"

, example2 .



curl "http://localhost:8080/exampleGroup/http/"

, . example2 0 , HTTP-proxy . , HTTP-proxy . , .


, , . , Cocaine , c, . .

 ///tail?arg=1&args=2.  URL   .       /tail?arg=1&args=2 .      . 
 
          X-Cocaine-Service, X-Cocaine-Event .      nginx . 
 
  ,       : 
 curl "http://localhost:8080/read" -H "X-Cocaine-Service: example" -H "X-Cocaine-Event: http" curl "http://localhost:8080/example/http/read" 
 
     
 
               .    ,               .           ,    .          .    .  ,       ,       (, 5%).     .        .        ,       ,       . 
 
    .       example ,  exampleGroup .     example   1000 .      .        
 service/locator: adding group 'exampleGroup' 
  
 
           exampleGroup .    ,    ,   app.py    hello: 
 @app.route('/') def hello(name=None): return "HELLO! I'm version #2" 
 
           ,   example2 . 
  
 
    
 curl "http://localhost:8080/exampleGroup/http/" 
 
   ,      .    example2  0 ,      HTTP-proxy     .   ,   HTTP-proxy    .   ,   . 
 
   
     ,     ,    .        ,     Cocaine   ,    c,   .      . 

///tail?arg=1&args=2. URL . /tail?arg=1&args=2 . .

X-Cocaine-Service, X-Cocaine-Event . nginx .

, :
curl "http://localhost:8080/read" -H "X-Cocaine-Service: example" -H "X-Cocaine-Event: http" curl "http://localhost:8080/example/http/read"



. , . , . . . , , (, 5%). . . , , .

. example , exampleGroup . example 1000 . .
service/locator: adding group 'exampleGroup'


exampleGroup . , , app.py hello:
@app.route('/') def hello(name=None): return "HELLO! I'm version #2"

, example2 .



curl "http://localhost:8080/exampleGroup/http/"

, . example2 0 , HTTP-proxy . , HTTP-proxy . , .


, , . , Cocaine , c, . .

///tail?arg=1&args=2. URL . /tail?arg=1&args=2 . .

X-Cocaine-Service, X-Cocaine-Event . nginx .

, :
curl "http://localhost:8080/read" -H "X-Cocaine-Service: example" -H "X-Cocaine-Event: http" curl "http://localhost:8080/example/http/read"



. , . , . . . , , (, 5%). . . , , .

. example , exampleGroup . example 1000 . .
service/locator: adding group 'exampleGroup'


exampleGroup . , , app.py hello:
@app.route('/') def hello(name=None): return "HELLO! I'm version #2"

, example2 .



curl "http://localhost:8080/exampleGroup/http/"

, . example2 0 , HTTP-proxy . , HTTP-proxy . , .


, , . , Cocaine , c, . .

///tail?arg=1&args=2. URL . /tail?arg=1&args=2 . .

X-Cocaine-Service, X-Cocaine-Event . nginx .

, :
curl "http://localhost:8080/read" -H "X-Cocaine-Service: example" -H "X-Cocaine-Event: http" curl "http://localhost:8080/example/http/read"



. , . , . . . , , (, 5%). . . , , .

. example , exampleGroup . example 1000 . .
service/locator: adding group 'exampleGroup'


exampleGroup . , , app.py hello:
@app.route('/') def hello(name=None): return "HELLO! I'm version #2"

, example2 .



curl "http://localhost:8080/exampleGroup/http/"

, . example2 0 , HTTP-proxy . , HTTP-proxy . , .


, , . , Cocaine , c, . .