Yandex cloud platform: Cocaine in action
We have already told you what Cocaine is and how to deploy it “at home”. Today we will talk about how to use its infrastructure at the programmer level. By the way, on April 26 at 14:00 a meeting will take place in the Moscow office of Yandex, where you can talk to us live - the team that makes Cocaine. Come, but do not forget to register .
So, from today's post you will learn:
')
Let us begin our immersion in the programmer’s cocaine days.
This text is quite voluminous and very technical. It is understood that you have read our previous posts, and you have access to the cocaine cloud, on which machines the required libraries and programs are installed. And that you are quite familiar with asynchronous programming and you are not afraid of the words: event loop, callback and cortutina. For each example, the requirements will be listed under the cut or explicitly. And yes, for the repetition of everything told, it is not necessary to have a cluster of 50 cars. One laptop with Mac OS was enough for me.
Let's start our review by loading into the cloud an already configured ready application, the power of which knows no bounds, echo . The application is written in Python and represents a typical Hello World for testing the correctness of new frameworks.
Virtually the entire article will use the cocaine-tool console utilities, which themselves depend on the installed cocaine-framework-python. In the magical world of Python, these libraries are set by the command:
Download the application to the cloud, go to the directory with the application and run the following command:
This is important: before downloading, do not forget to make the input point in your application executable. In our case:
The result will be displayed:
To run it, we need a ready profile. If you have Cocaine configured to use Docker as an isolation system, you can use the profile provided in the previous article . If Cocaine is configured for normal process spawn, just use the profile I prepared (profile.json):
You can load a profile with a similar command:
The result of the work should be something like this:
Having a ready profile, it remains only to launch the recently downloaded application:
If something like this is displayed in the console, then everything was done successfully and our application is now
part of the cloud.
Let's go back to the example. Do not really delve into the code yet. Soon we will write a similar, but more functional application, but this time
already with explanations. The only thing this application does is respond to an event named ping with the same
the very message that was sent to him.
Now our task is to write the code that forms the request and accepts the answer from this application. It is worth noting that cocaine applications can be accessed both via HTTP and directly from client code. We will look at the first method a little later; the second method involves the use of one of the frameworks provided. To warm up, we will write the code on Python using the pythonic framework that was supposed to be bundled with the cocaine-tool .
After importing the necessary libraries and classes, we simply create a cocaine service object using the library class
This is important: when the python framework was written, the Asyncio library was not yet completely ready, and it was necessary to choose between Twisted and Tornado. The choice fell on the Tornado mainly because of its more modern look and functionality. Now the full rewriting of the pitonach framework on asyncio, or rather on its backport for the second Python - Trollius, is in full swing.
The sequence of calls can be represented as follows:
In the cloud, the application will go like this:
The mysterious decorator
Check the written code:
It is worth noting that in frameworks for dynamic languages, such as Python is, the set of available methods is obtained dynamically while connecting the object of the
Surely you immediately had questions:
responds to erroneous requests. For example, we set as the name of the application, the transmitted
echo , and some name of a nonexistent application. Let's say 42 , that is:
I sincerely refuse that you do not have such an application in the cloud. In this case, Cocaine will return an error, and the framework will throw out
an exception.
It is worth noting that frameworks are absolutely not obliged to throw exceptions for errors. The frameworks are specially created to provide the user with a native Cocaine interface for the language, which means that if the considered language is the most preferred (or the only, in the Go case) method of returning an error, for example, returning an error code, then it will be used.
Similarly, you can try to request the processing of a non-existent event, pass an incorrect number of arguments, or make a mistake in their types.
Let's stop for a moment and look under the hood of our cloud platform. How does Cocaine know exactly which
request came to him and how to handle it? In other words, let's briefly review the protocol through which Cocaine communicates with customers.
For each call, Cocaine opens a typed channel that allows you to exchange messages with the application. Messages can be of three types:
Chunk is a regular message, information that the client wants to send to the application, or vice versa. In the event of any error, the application (or Cocaine itself, if it can recognize the error in advance) responds with the Error type, carrying some code and a human-readable description of the error that has occurred. Finally, each channel must be closed by a message of type Choke . Cocaine guarantees that after such a message there will be no other messages.
Why am I doing this? Under the hood, each framework implements this protocol (and a bit more), masking it under function calls
or methods, the usual formation of the message and, on the other hand, carrying out dispatching, unpacking the received
messages and sending them back to the client.
Since the protocol is streaming, and the last message is always
return an unknown amount of
return.
Let's modify our application so that it returns not one Chunk, but several. Say two. Consider the code of our super application, from which the logger was dropped:
As you can see, practically the same processing pattern that we wrote in the client code is used. For each incoming event named
Accordingly, in order for our application to respond with several Chunks, it is enough to call this method again. For example:
How to check it? Well, first of all, the application should be reloaded and restarted:
This is important: if the modified application stubbornly does not want to be updated and works as before, even after reloading,
check the cache settings in the config cocaine-runtime (cache section).
After that, we will restart our client, and he will give out ... the same thing as before. This is normal behavior. Let me remind you that we have a stream protocol, and we read exactly one Chunk from the channel. To make it work, change the
If you try to consider another chunk as a framework, a special ChokeEvent
the framework is dependent (in others it may not be, and the notification of a closed channel is carried out in a different way).
Now that we have reviewed the basic work with the application and the python framework, let's consider the reasons that led us to dwell on the choice of an asynchronous model for working with code.
To do this, consider Cocaine and its surrounding frameworks in more detail. As noted in our first article, the basic model underlying Cocaine is the model of actors. The frameworks, in turn, are nothing more than wrappers over RPC calls to Cocaine and processing its responses. Here, we requested something from the
In such cases, two common approaches are used. In the first of them, each request is processed in a separate thread, and control is returned to the code mainly in the waiting method of the issued future, in the second case, the event-handling cycle and cooperative multitasking are used (callbacks, coroutines, fiberes — not important). But again, in the case of processing thousands of simultaneous requests - which is quite a realistic situation for high-load systems - the first method gives a significant overhead in performance.
That is why all cocaine frameworks implement primarily an asynchronous way of interacting with services. Some also implement the synchronous method in addition, but not all.
After we poured the Cocaine application, it became part of the infrastructure , or unified . Now you can access it from any convenient language for development and perform all the same actions discussed above, but without restricting yourself in choosing development paradigms and libraries.
We demonstrate this by writing clients for several frameworks in various languages. In this case, we will test the original version of the echo application.
A similar client in C ++ will look like this:
You need to compile as follows (in the case of using clang):
If you are using GCC, then simply replace clang ++ with g ++, and remove the need to use libc ++ instead of libstdc ++ by removing
I will not particularly comment on the code; in more detail about using the native framework you can read here .
In the case of Ruby, this is completely simple We use the wonderful
use Ruby Fiber and write asynchronous code as if it were synchronous. No callbacks and futures!
As can be seen from the examples, in contrast to CORBA, Cocaine does not need to determine in advance the IDL of applications and services used. In the case of statically typed languages, for ease of use and type checking, you can write stubs for each service, but usually their implementation does not take up very much space. If a language has at least some normal reflex, then there is even no need to write an implementation — only interfaces are sufficient. In the case of dynamically typed languages, you can generally do without stubs and create a service API dynamically, obtaining it after determining the basic information about it during the work of the Locator method
Spreading the writing of client code for cocaine applications, I propose to go to the other side of the cloud and consider the rules by which the applications themselves and the means for writing them are written. In fact, there is not much difference between the client code and the application code - you will see soon.
Each of the existing frameworks has tools for writing all sorts of applications that will be multiplied by Cocaine under load. We will use the API API again for the warm-up.
As an example, we will write an application that will generate QR codes of a certain size upon request. To avoid being bored, the application will also cache the largest images in the
In such a simple way, you can build a real web of requests between small but interlinked applications. In this case, the question of saving resources is very acute. Remember we considered why all the frameworks are built on an asynchronous event-oriented model? It is the saving of resources that prompted us to make this choice.
Returning to our example, we will give our application an event called
This is important: when writing this example, we will use the qrcode python library. It depends on the well-known
I will cite immediately the code of the entire application, and then analyze each line in detail, which can cause even the slightest questions:
With the line
configured in the config cocaine-runtime in the second article. By default it will be file storage. In this case, it doesn’t matter to us at all, since we use the cloud infrastructure provided to us.
Now pay attention to the last lines:
With this code, we create a worker object that will connect to Cocaine behind the scenes, as well as some other actions. Then we start the event processing loop, registering the
Exactly two parameters are passed to each such function:
We read the data from the socket and unpack it. Why make an extra packing, why can't you just write something like:
After unpacking the necessary arguments, the logic of the application itself follows. Wrapping it in a large try / except block, we guarantee that all exceptions thrown will be caught not by the framework, but by us, and will have the error codes and messages we need.
This is important: if we do not catch the exception ourselves, the framework will do it.
After checking the size of the picture we have branching. If the size is small enough, then we will generate the QR code again, otherwise with the lines:
We form a key and check if there is such a document in our repository with such a key. If not, then the method
This is important: the
when our expected event has arrived. The rest of the events are still processed. ,
Storage,
, , .
:
, :
? ,
. ,
, . - , , HTTP.
: .
:
:
API , .
:
? , - , http-, . , , . cocaine-proxy — , HTTP, .
: production
, , . ? .
URI . :
. URI :
. , - :
:
, , , . , .
, Cocaine .
ab . : , , 100000 32 ( , ) , Cocaine .
, ,
.
:
Cocaine , - :
, . . «state» . . «slaves» , , Cocaine ( ). «queue» ( , ) . , . , «sessions» , , . , , . — .
! ,
, , . 100000 32 .
, 6, 10. , 7 . , Cocaine , .
, , . - —
Cocaine.
No problem! , — ( ) , .
Go, , . , , . , , .
Go — . Have fun!
, ! Cocaine .
Cocaine , . , . , , Cocaine Runtime, . , , , . Cocaine.
, . Cocaine, , . , , , , .
So, from today's post you will learn:
')
- how to write applications;
- how to use applications and services natively using the provided frameworks;
- how to change the application so that it responds to http, as well as how to test these applications using Cocaine http proxy;
- how to write your own service.
Let us begin our immersion in the programmer’s cocaine days.
This text is quite voluminous and very technical. It is understood that you have read our previous posts, and you have access to the cocaine cloud, on which machines the required libraries and programs are installed. And that you are quite familiar with asynchronous programming and you are not afraid of the words: event loop, callback and cortutina. For each example, the requirements will be listed under the cut or explicitly. And yes, for the repetition of everything told, it is not necessary to have a cluster of 50 cars. One laptop with Mac OS was enough for me.
Part 1. We work with Cocaine
Let's start our review by loading into the cloud an already configured ready application, the power of which knows no bounds, echo . The application is written in Python and represents a typical Hello World for testing the correctness of new frameworks.
How to download an application
Virtually the entire article will use the cocaine-tool console utilities, which themselves depend on the installed cocaine-framework-python. In the magical world of Python, these libraries are set by the command:
pip install cocaine-tools Download the application to the cloud, go to the directory with the application and run the following command:
cocaine-tool app upload This is important: before downloading, do not forget to make the input point in your application executable. In our case:
chmod u+x echo.py The result will be displayed:
To run it, we need a ready profile. If you have Cocaine configured to use Docker as an isolation system, you can use the profile provided in the previous article . If Cocaine is configured for normal process spawn, just use the profile I prepared (profile.json):
{ "isolate": { "args": { "spool": "/var/spool/cocaine" }, "type": "process" } } You can load a profile with a similar command:
cocaine-tool profile upload --name profile@test --profile ./profile.json The result of the work should be something like this:
Having a ready profile, it remains only to launch the recently downloaded application:
cocaine-tool app start --name echo --profile profile@test If something like this is displayed in the console, then everything was done successfully and our application is now
part of the cloud.
Let's go back to the example. Do not really delve into the code yet. Soon we will write a similar, but more functional application, but this time
already with explanations. The only thing this application does is respond to an event named ping with the same
the very message that was sent to him.
Now our task is to write the code that forms the request and accepts the answer from this application. It is worth noting that cocaine applications can be accessed both via HTTP and directly from client code. We will look at the first method a little later; the second method involves the use of one of the frameworks provided. To warm up, we will write the code on Python using the pythonic framework that was supposed to be bundled with the cocaine-tool .
from cocaine.futures import chain from cocaine.services import Service from tornado.ioloop import IOLoop # Alias for more readability. asynchronous = chain.source if __name__ == '__main__': io_loop = IOLoop.current() service = Service('echo') @asynchronous def invoke(message): result = yield service.enqueue('ping', message) print(result) invoke('Hello World!') invoke('Hello again!') io_loop.start() After importing the necessary libraries and classes, we simply create a cocaine service object using the library class
Service , passing the application name (echo in our case) as a parameter. This object is a mapping of the methods and events of the cloud service / application to the class methods in the language used. Our work with it is to asynchronously call the enqueue method provided by all applications, passing in the name of the event as the first parameter and the remaining parameters required by the application (all of a sudden); and then just asynchronously wait for a response. To do this, we get the object of the event loop using the Tornado library, start it and wait. When the answer is received, the cycle does not stop explicitly.This is important: when the python framework was written, the Asyncio library was not yet completely ready, and it was necessary to choose between Twisted and Tornado. The choice fell on the Tornado mainly because of its more modern look and functionality. Now the full rewriting of the pitonach framework on asyncio, or rather on its backport for the second Python - Trollius, is in full swing.
The sequence of calls can be represented as follows:
In the cloud, the application will go like this:
The mysterious decorator
@asynchronous additional actions to the decorated function, namely:- adds its call to the event loop;
- if the function is korutina, it controls its asynchronous promotion process.
Check the written code:
It is worth noting that in frameworks for dynamic languages, such as Python is, the set of available methods is obtained dynamically while connecting the object of the
Service class to Cocaine. In other languages, one has to write one or other stubs, which will be shown later.Surely you immediately had questions:
- Why so hard?
- Why was the asynchronous model chosen?
responds to erroneous requests. For example, we set as the name of the application, the transmitted
Service object is notecho , and some name of a nonexistent application. Let's say 42 , that is:
echo = Service('42') I sincerely refuse that you do not have such an application in the cloud. In this case, Cocaine will return an error, and the framework will throw out
an exception.
It is worth noting that frameworks are absolutely not obliged to throw exceptions for errors. The frameworks are specially created to provide the user with a native Cocaine interface for the language, which means that if the considered language is the most preferred (or the only, in the Go case) method of returning an error, for example, returning an error code, then it will be used.
Similarly, you can try to request the processing of a non-existent event, pass an incorrect number of arguments, or make a mistake in their types.
Cocaine protocol
Let's stop for a moment and look under the hood of our cloud platform. How does Cocaine know exactly which
request came to him and how to handle it? In other words, let's briefly review the protocol through which Cocaine communicates with customers.
For each call, Cocaine opens a typed channel that allows you to exchange messages with the application. Messages can be of three types:
0, Chunk ::= <Tuple> 1, Error ::= (<Number>, <String>) 2, Choke ::= () Chunk is a regular message, information that the client wants to send to the application, or vice versa. In the event of any error, the application (or Cocaine itself, if it can recognize the error in advance) responds with the Error type, carrying some code and a human-readable description of the error that has occurred. Finally, each channel must be closed by a message of type Choke . Cocaine guarantees that after such a message there will be no other messages.
Why am I doing this? Under the hood, each framework implements this protocol (and a bit more), masking it under function calls
or methods, the usual formation of the message and, on the other hand, carrying out dispatching, unpacking the received
messages and sending them back to the client.
Since the protocol is streaming, and the last message is always
Choke , strictly speaking, applications canreturn an unknown amount of
Chunk 's, thereby organizing a real stream processing of data and their asynchronousreturn.
Going back to the example
Let's modify our application so that it returns not one Chunk, but several. Say two. Consider the code of our super application, from which the logger was dropped:
from cocaine.worker import Worker</python> def echo(request, response): message = yield request.read() response.write(message) response.close() W = Worker() W.run({ 'ping': echo, }) As you can see, practically the same processing pattern that we wrote in the client code is used. For each incoming event named
ping , the echo function is called, which takes the already prepared request and response objects as arguments. By calling the request.read() method, we asynchronously read one Chunk of the request (in our case, it is strictly one), and we respond using the response.write(...) method.Accordingly, in order for our application to respond with several Chunks, it is enough to call this method again. For example:
def echo(request, response): message = yield request.read() response.write(message) response.write('Additional message') response.close() How to check it? Well, first of all, the application should be reloaded and restarted:
cocaine-tool app upload && cocaine-tool app restart --name echo --profile profile@test This is important: if the modified application stubbornly does not want to be updated and works as before, even after reloading,
check the cache settings in the config cocaine-runtime (cache section).
After that, we will restart our client, and he will give out ... the same thing as before. This is normal behavior. Let me remind you that we have a stream protocol, and we read exactly one Chunk from the channel. To make it work, change the
invoke function as follows: @asynchronous def invoke(message): result = yield service.enqueue('ping', message) another_result = yield print(result, another_result) io_loop.stop() If you try to consider another chunk as a framework, a special ChokeEvent
ChokeEvent will be thrown, which againthe framework is dependent (in others it may not be, and the notification of a closed channel is carried out in a different way).
Asynchrony everywhere
Now that we have reviewed the basic work with the application and the python framework, let's consider the reasons that led us to dwell on the choice of an asynchronous model for working with code.
To do this, consider Cocaine and its surrounding frameworks in more detail. As noted in our first article, the basic model underlying Cocaine is the model of actors. The frameworks, in turn, are nothing more than wrappers over RPC calls to Cocaine and processing its responses. Here, we requested something from the
Service object. What is going on with him? After encoding the message and sending it, it waits. Waits for a response, waits for an error, waits for the timeout to occur. All these expectations are normal IO operations that do not waste processor resources. So why give it one more time to cool, if you can give more work? Imagine that you are making requests to some application or service not from the client code, but from another application . It is clear that if this application makes stop the world on each such request, it will lead to the rapid spawning of such applications, which subsequently will simply be a waste of system resources.In such cases, two common approaches are used. In the first of them, each request is processed in a separate thread, and control is returned to the code mainly in the waiting method of the issued future, in the second case, the event-handling cycle and cooperative multitasking are used (callbacks, coroutines, fiberes — not important). But again, in the case of processing thousands of simultaneous requests - which is quite a realistic situation for high-load systems - the first method gives a significant overhead in performance.
That is why all cocaine frameworks implement primarily an asynchronous way of interacting with services. Some also implement the synchronous method in addition, but not all.
But what about the other languages?
After we poured the Cocaine application, it became part of the infrastructure , or unified . Now you can access it from any convenient language for development and perform all the same actions discussed above, but without restricting yourself in choosing development paradigms and libraries.
We demonstrate this by writing clients for several frameworks in various languages. In this case, we will test the original version of the echo application.
C ++
A similar client in C ++ will look like this:
#include <cocaine/framework/services/app.hpp> #include <cocaine/framework/services/storage.hpp> #include <iostream> #include <condition_variable> #include <mutex> #include <atomic> namespace cf = cocaine::framework; int main(int argc, char** argv) { auto manager = cf::service_manager_t::create(cf::service_manager_t::endpoint_t("localhost", 10053)); // Get application service object. auto app = manager->get_service<cf::app_service_t>("echo"); std::atomic<int> counter(0); std::condition_variable cv; // Call application. auto g1 = app->enqueue("ping", "Hello from C++"); auto g2 = app->enqueue("ping", "Hello again!"); auto handler = [&counter, &cv](cf::generator<std::string>& g) { counter++; try { // Always packed data. std::cout << "result: " << g.next() << std::endl; } catch (const std::exception& e) { std::cout << "error: " << e.what() << std::endl; } cv.notify_all(); }; g1.then(handler); g2.then(handler); std::mutex m; std::unique_lock<std::mutex> guard(m); while (counter < 2) { cv.wait(guard); } return 0; } You need to compile as follows (in the case of using clang):
clang++ -std=c++11 -stdlib=libc++ -lcocaine-core -lcocaine-framework -lmsgpack -lboost_system -lev ../src/clients/echo-client.cpp If you are using GCC, then simply replace clang ++ with g ++, and remove the need to use libc ++ instead of libstdc ++ by removing
-stdlib=libc++ .I will not particularly comment on the code; in more detail about using the native framework you can read here .
Ruby
In the case of Ruby, this is completely simple We use the wonderful
eventmachine framework, which provides the opportunityuse Ruby Fiber and write asynchronous code as if it were synchronous. No callbacks and futures!
require 'cocaine' require 'cocaine/synchrony/service' require 'em-synchrony' class EchoClient EM.synchrony do results = [] service = Cocaine::Synchrony::Service.new 'echo' channel = service.enqueue('ping', 'Hello from Ruby!') channel.each do |result| results.push result end puts results EM.stop end end As can be seen from the examples, in contrast to CORBA, Cocaine does not need to determine in advance the IDL of applications and services used. In the case of statically typed languages, for ease of use and type checking, you can write stubs for each service, but usually their implementation does not take up very much space. If a language has at least some normal reflex, then there is even no need to write an implementation — only interfaces are sufficient. In the case of dynamically typed languages, you can generally do without stubs and create a service API dynamically, obtaining it after determining the basic information about it during the work of the Locator method
resolve .Part 2. Workers
Spreading the writing of client code for cocaine applications, I propose to go to the other side of the cloud and consider the rules by which the applications themselves and the means for writing them are written. In fact, there is not much difference between the client code and the application code - you will see soon.
Each of the existing frameworks has tools for writing all sorts of applications that will be multiplied by Cocaine under load. We will use the API API again for the warm-up.
As an example, we will write an application that will generate QR codes of a certain size upon request. To avoid being bored, the application will also cache the largest images in the
Storage service, using secondary requests from the application back to Cocaine.In such a simple way, you can build a real web of requests between small but interlinked applications. In this case, the question of saving resources is very acute. Remember we considered why all the frameworks are built on an asynchronous event-oriented model? It is the saving of resources that prompted us to make this choice.
Returning to our example, we will give our application an event called
generate . It will return the generated QR code in binary form as some sort of byte array (well, how else?), Which we will feed to a special object in order to convert them into a real picture.This is important: when writing this example, we will use the qrcode python library. It depends on the well-known
PIL or pillow , which in turn requires many binary dependencies, such as libpng. Make sure you have them, and then set qrcode using the same pip : >pip install qrcode I will cite immediately the code of the entire application, and then analyze each line in detail, which can cause even the slightest questions:
#!/usr/bin/env python import StringIO import msgpack import qrcode from cocaine.exceptions import ServiceError from cocaine.decorators import http from cocaine.logging import Logger from cocaine.services import Service from cocaine.worker import Worker storage = Service('storage') def generate_qr(message, size=10): if size <= 0 or size >= 20: raise ValueError('size argument must fit in [0; 20]') out = StringIO.StringIO() img = qrcode.make(message, box_size=size) img.save(out, 'png') return out.getvalue() def generate(request, response): rq = yield request.read() message, size = msgpack.loads(rq) try: if size < 10: data = generate_qr(message, size) else: key = '{0}size={1}'.format(message, size) try: data = yield storage.read('qr-codes', key) except ServiceError: data = generate_qr(message, size) yield storage.write('qr-codes', key, data) response.write(data) except Exception as err: response.error(1, str(err)) finally: response.close() w = Worker() w.run({ 'generate': generate }) With the line
storage = Service('storage') we declare a storage service global for the worker, which, let me remind you, weconfigured in the config cocaine-runtime in the second article. By default it will be file storage. In this case, it doesn’t matter to us at all, since we use the cloud infrastructure provided to us.
Now pay attention to the last lines:
w = Worker() w.run({ 'generate': generate }) With this code, we create a worker object that will connect to Cocaine behind the scenes, as well as some other actions. Then we start the event processing loop, registering the
generate event handler along the way. Now any invoke message with the correct parameters will be redirected to the specified function. Consider it in more detail.Exactly two parameters are passed to each such function:
- prepared request object that contains a single
readmethod for deferred reading of data sent
from socket; - response object, which is a channel to which you can write (remember the description of the protocol) all
listed message types (Chunk,Error,Choke).
rq = yield request.read() message, size = msgpack.loads(rq) We read the data from the socket and unpack it. Why make an extra packing, why can't you just write something like:
message, size = yield request.read() ? Do not forget that we have to unpack some structure from raw bytes. Last time it was a string, this time a tuple. Need some kind of coder / decoder. For this purpose it may be useful even json , at least msgpack , even protobuf . We chose msgpack in this example, because its implementations exist for all popular languages and it is fast.After unpacking the necessary arguments, the logic of the application itself follows. Wrapping it in a large try / except block, we guarantee that all exceptions thrown will be caught not by the framework, but by us, and will have the error codes and messages we need.
This is important: if we do not catch the exception ourselves, the framework will do it.
After checking the size of the picture we have branching. If the size is small enough, then we will generate the QR code again, otherwise with the lines:
key = '{0}size={1}'.format(message, size) try: data = yield storage.read('qr-codes', key) except ServiceError: data = generate_qr(message, size) yield storage.write('qr-codes', key, data) We form a key and check if there is such a document in our repository with such a key. If not, then the method
storage.read('qr-codes', key) throw an exception. In this case, we still generate the image and save it.This is important: the
read and write operations are performed asynchronously. Every time when we in our Corout run into the keyword yield , control returns to the event loop. It returns back onlywhen our expected event has arrived. The rest of the events are still processed. ,
Storage,
read . ., , .
finally . , , , .:
, :
? ,
echo . Cocaine, cocaine-tool : cd src/qr && cocaine-tool app upload && cocaine-tool app start --name qr --profile profile@test . ,
echo : import StringIO import msgpack from PIL import Image from tornado.ioloop import IOLoop from cocaine.futures import chain from cocaine.services import Service # Alias for more readability. asynchronous = chain.source if __name__ == '__main__': io_loop = IOLoop.current() service = Service('qr') @asynchronous def invoke(message): try: result = yield service.enqueue('generate', msgpack.dumps([message, 10])) print('Result:', result) out = StringIO.StringIO() out.write(result) out.seek(0) img = Image.open(out) img.save('qr.png', 'png') except Exception as err: print('Error: ', err) finally: io_loop.stop() invoke('What is best in life? To crush your enemies, see them driven before you, and to hear the lamentation of ' 'their women.') io_loop.start() invoke , . , , PIL ., . - , , HTTP.
: .
:
@http . ( ) request response . - read , , , cocaine.decorators.http._HTTPRequest , — , , .. ( response ) .:
>#!/usr/bin/env python import StringIO import msgpack import qrcode from cocaine.exceptions import ServiceError from cocaine.decorators import http from cocaine.logging import Logger from cocaine.services import Service from cocaine.worker import Worker storage = Service('storage') log = Logger() def generate_qr(message, size=10): if size <= 0 or size >= 20: raise ValueError('size argument must fit in [0; 20]') out = StringIO.StringIO() img = qrcode.make(message, box_size=size) img.save(out, 'png') return out.getvalue() @http def generate(request, response): request = yield request.read() try: message = request.request['message'] size = int(request.request.get('size', 10)) if size < 10: data = generate_qr(message, size) else: key = '{0}size={1}'.format(message, size) try: data = yield storage.read('qr-codes', key) except ServiceError: data = generate_qr(message, size) yield storage.write('qr-codes', key, data) response.write_head(200, [('Content-type', 'image/png')]) response.write(data) except KeyError: response.write_head(400, [('Content-type', 'text/plain')]) response.write('Query field "message" is required') except Exception as err: response.write_head(400, [('Content-type', 'text/plain')]) response.write(str(err)) finally: response.close() w = Worker() w.run({ 'generate-http': generate }) API , .
:
generate-http — .? , - , http-, . , , . cocaine-proxy — , HTTP, .
cocaine-native-proxy . , - , cocaine-tool , , Python, : cocaine-tool proxy start --count=32 count — , .: production
cocaine-native-proxy . , ., , . ? .
URI . :
<APP>/<METHOD>[?<Args>] .X-Cocaine-Service X-Cocaine-Event .. URI :
/qr/generate-http?message=Hello%20World!&size=10 .. , - :
:
cocaine-proxy (, Storage ) — ., , , . , .
, Cocaine .
ab . : , , 100000 32 ( , ) , Cocaine .
, ,
.
cocaine-runtime ,:
cocaine-tool info Cocaine , - :
, . . «state» . . «slaves» , , Cocaine ( ). «queue» ( , ) . , . , «sessions» , , . , , . — .
! ,
cocaine-runtime cocaine-tool info : ab -n 100000 -c 32 'localhost:8080/qr/generate-http?message=Hello%20World!&size=10' , , . 100000 32 .
cocaine-runtime , 6 QR-, cocaine-runtime : [Tue Apr 8 15:36:18 2014] [INFO] app/qr: enlarging the pool from 5 to 6 slaves cocaine-tool info - :, 6, 10. , 7 . , Cocaine , .
, , . - —
SIGTERM , .Cocaine.
Python!
No problem! , — ( ) , .
Go, , . , , . , , .
Go — . Have fun!
>package main import ( "net/http" "code.google.com/p/rsc/qr" "github.com/ugorji/go/codec" "github.com/cocaine/cocaine-framework-go/cocaine" ) //msgpack specific var ( mh codec.MsgpackHandle h = &mh ) var ( OkHeaders cocaine.Headers = cocaine.Headers{[2]string{"Content-type", "image/png"}} ErrorHeaders cocaine.Headers = cocaine.Headers{[2]string{"Content-type", "text/plain"}} storage *cocaine.Service ) const ( cacheNamepspace = "qr-code" cacheTag = "qr-tag" ) func qenerate(text string) (png []byte, err error) { res := <-storage.Call("read", cacheNamepspace, text) if res.Err() == nil { err = res.Extract(&png) return } c, err := qr.Encode(text, qr.L) if err != nil { return } png = c.PNG() <-storage.Call("write", cacheNamepspace, text, string(png), []string{cacheTag}) return } func on_generate(request *cocaine.Request, response *cocaine.Response) { defer response.Close() inc := <-request.Read() var task struct { Text string Size int } err := codec.NewDecoderBytes(inc, h).Decode(&task) if err != nil { response.ErrorMsg(-100, err.Error()) return } png, err := qenerate(task.Text) if err != nil { response.ErrorMsg(-200, err.Error()) return } response.Write(png) } func on_http_generate(request *cocaine.Request, response *cocaine.Response) { defer response.Close() r, err := cocaine.UnpackProxyRequest(<-request.Read()) if err != nil { response.ErrorMsg(-200, err.Error()) return } message := r.FormValue("message") if len(message) == 0 { response.Write(cocaine.WriteHead(http.StatusBadRequest, ErrorHeaders)) response.Write("Missing argument `message`") return } png, err := qenerate(message) if err != nil { response.Write(cocaine.WriteHead(http.StatusInternalServerError, ErrorHeaders)) response.Write("Unable to generate QR") return } response.Write(cocaine.WriteHead(http.StatusOK, OkHeaders)) response.Write(png) } func main() { binds := map[string]cocaine.EventHandler{ "generate": on_generate, "generate-http": on_http_generate, } Worker, err := cocaine.NewWorker() if err != nil { panic(err) } storage, err = cocaine.NewService("storage") if err != nil { panic(err) } Worker.Loop(binds) } , ! Cocaine .
Instead of conclusion
Cocaine , . , . , , Cocaine Runtime, . , , , . Cocaine.
, . Cocaine, , . , , , , .
Source: https://habr.com/ru/post/220243/
All Articles