ZeroMQ. Chapter 1: Getting Started
Hello!
I want to start a free translation of the book “ZeroMQ.Use ZeroMQ”. I am sure that many will want to deal with this interesting library.
Content
Welcome to ZeroMQ! This chapter is an introduction to ZeroMQ and gives the reader a general idea of what the message queue system is and, most importantly, what ZeroMQ is. In this chapter we will talk about the following topics:
')
People throughout their lives socially interact with each other. Programs behave in this way. The program must communicate with another program, since we live in the world of the Internet. We have UDP, TCP, HTTP, IPX, WebSocket and other protocols for connecting applications.
Nevertheless, approaches of such a low level complicate our lives, we need something simpler and faster. High-level abstractions sacrifice speed and flexibility, while dealing directly with low-level details is not easy. ZeroMQ shows what the output is, giving us the convenience of using high-level methods at the speed of a low-level approach.
Before we begin to deal with ZeroMQ, let's first look at the general concept of message queuing.
The message queue, or technically, FIFO (First In First Out), is one of the main and well-studied data structures. There are various queue implementations, such as a priority queue or a two-way queue, which have different properties, but the general idea is that data is added to the queue when it arrives or the caller is ready.
However, the message queue provides guarantees that the message will be delivered no matter what happens. A message queue allows asynchronous communication between loosely coupled components, and also provides a strict queue sequence. In the event of a shortage of resources, which prevents you from immediately processing the sent data, you can put them in a message queue on the server that will store the data until the client is ready.
Message queue plays an important role in scaling distributed systems, as it supports asynchronous communications. We give a brief information about the difference between synchronous and asynchronous systems.
In conventional synchronous systems, tasks are processed one at a time. A task is considered not processed until its processing is completed. This is the easiest way to organize work.
We can also inject threads into this system. In this case, the processing of each task would be performed in parallel.
In a multi-threaded model, threads are controlled by the operating system itself on a single processor or multiple processors / cores.
Asynchronous I / O (AIO) allows the program to continue execution when processing I / O requests. AIO is required in real-time applications. With AIO, we can handle several tasks in one thread.
This is a traditional way of programming, after the process begins, we wait for it to complete. The disadvantage of this approach is that it blocks the execution of the program while the task is being processed. But there is another AIO approach. In AIO, a task that is independent of the process is still ongoing. In the second chapter, we will take a closer look at AIO and use it in ZeroMQ.
You may wonder why you should use a message queue instead of processing all threads for a single-thread or multi-thread approach. Let's look at a situation where you have a web application similar to Google Images, in which you allow users to enter some URLs. Once they submit the form, your application displays all the images for this URL. However:
In this case, you know that you need to add these URLs to the queue and process them. So, you need a message queuing system.
Up to this point we have considered what a message queue is, which led us to the goal of this book, that is, to ZeroMQ. Society ZeroMQ is defined as "sockets on steroids." Formally, ZeroMQ is defined as a message library that helps developers create distributed and parallel applications.
The first thing we need to learn about ZeroMQ is that it is not a traditional message queuing system, such as ActiveMQ, WebSphereMQ, or RabbitMQ. ZeroMQ is different. She gives us the tools to create your own message queuing system. This is a library.
It works on various architectures from ARM to Itanium and is supported in more than 20 programming languages.
ZeroMQ is simple. We can do some asynchronous I / O operations; ZeroMQ can also queue messages from an I / O stream into a message queue. I / O streams in ZeroMQ work asynchronously when communicating with network traffic, so they can do the rest for you. If you have previously worked with sockets, then you should know that this is not very easy. However, ZeroMQ makes it much easier to work with them.
ZeroMQ is fast. The Second Life website managed to get 13.4 microseconds of continuous latency and up to 4,100,000 messages per second. ZeroMQ can use a multicast transport protocol, which is an effective method for transmitting data in different directions.
So, it's time to start writing code, after we have analyzed what the message queue is and what ZeroMQ is. And of course, we will start with the famous “Hello World” program.
Let's consider the situation when we have a server and a client. The server responds with
Below is the server code that sends the
Below is the client code that sends a
We have the first basic architecture: request-response, as shown in the following diagram:
Let's take a closer look at the code in order to understand how it works.
First we create the context and socket. The
The server connects the
This simple program “hello world” shows us an example of using the first request-response pattern.
We use a request-response pattern (pattern) to send a message from a client to one or more servers and get a response to each sent message. Most likely, this is the easiest way to use ZeroMQ. Requests for answers must be strictly in order.
Below is the response of the request-response pattern:
The server uses the
This book is completely devoted to queues. You may be surprised when you find out what we mean when we talk about
To understand how it works, let's say that the flows in the previous figure send 16, 2, 6 and 8 packets per second, respectively, but only 12 packets per second can be processed at the output. In this case, we could transmit 4 packets per second, but Stream 2 only transmits 2 packets per second. The fair-queue rules are that there can be no free exits unless all exits are free at the same time. Thus, it is possible to allow Thread 2 to transmit its 2 packets per second and divide the remaining 10 between the remaining flows.
This incoming message routing strategy uses
Below is the request part of the request-response pattern:
The client uses
This part combines the request and response sections and shows how the answer to someone’s request is and how it is answered.
The client sends a message to the server using
After sending the message, the client waits to receive a response. This is done with
Each programming language has its own approach to handling strings. In Erlang, there are no strings at all (there they are represented as lists of characters). In the C programming language, lines with a zero at the end. Strings in C are basically character arrays, where '\ 0' means the end of the string. Errors with strings are common and the result of many vulnerabilities.
According to Miller et al. (1995), 65 percent of Unix vulnerabilities are due to errors when working with strings, such as zero bytes or buffer overflows, so string handling in C must be done carefully.
When using ZeroMQ, it falls on your shoulders so that the message is reliably formatted, so that other applications can read it. ZeroMQ knows only the size of the message and only.
Using different programming languages in one application is usually the case. If applications written in a programming language that does not add a zero byte at the end of the line should somehow communicate with applications written in C, then you will get strange results.
You can send a message, such as world, as in our zero byte example, as follows:
But in Erlang you would send the same message as follows:
Suppose that our client, written in C, connects to the ZeroMQ service, written in Erlang, and we send the message
Lines in ZeroMQ are fixed in length and sent without a null byte. Thus, the ZeroMQ string is transmitted as several bytes (the string itself in this example), as well as the length.
It is very useful to know which version of ZeroMQ you are using. In some cases, to avoid unwanted surprises, you need to know the exact version. For example, there are some differences between ZeroMQ 2.x and ZeroMQ 3.x, such as using obsolete methods; therefore, if you know the exact version of ZeroMQ that is installed on your computer, then you can avoid using outdated methods.
In this chapter, we looked at what a message queue is and also made a small introduction to ZeroMQ. We considered how ZeroMQ processes strings and the first pattern (request-response). And also wrote the first application “hello world”.
Resources for this article you can download link
Thank you all for your attention!
I want to start a free translation of the book “ZeroMQ.Use ZeroMQ”. I am sure that many will want to deal with this interesting library.
Content
- Chapter 1: Getting Started
- Chapter 2: Introduction to Sockets
- Chapter 3: Using Socket Topology
- Chapter 4: Additional Patterns
Welcome to ZeroMQ! This chapter is an introduction to ZeroMQ and gives the reader a general idea of what the message queue system is and, most importantly, what ZeroMQ is. In this chapter we will talk about the following topics:
- An overview of what a message queue is
- Why use ZeroMQ and what makes it different from other message queue technologies
- Basics of client / server architecture
- Consider the first pattern: request-response
- How can we handle strings in C
- Checking installed ZeroMQ versions
')
Start
People throughout their lives socially interact with each other. Programs behave in this way. The program must communicate with another program, since we live in the world of the Internet. We have UDP, TCP, HTTP, IPX, WebSocket and other protocols for connecting applications.
Nevertheless, approaches of such a low level complicate our lives, we need something simpler and faster. High-level abstractions sacrifice speed and flexibility, while dealing directly with low-level details is not easy. ZeroMQ shows what the output is, giving us the convenience of using high-level methods at the speed of a low-level approach.
Before we begin to deal with ZeroMQ, let's first look at the general concept of message queuing.
Message queue
The message queue, or technically, FIFO (First In First Out), is one of the main and well-studied data structures. There are various queue implementations, such as a priority queue or a two-way queue, which have different properties, but the general idea is that data is added to the queue when it arrives or the caller is ready.
However, the message queue provides guarantees that the message will be delivered no matter what happens. A message queue allows asynchronous communication between loosely coupled components, and also provides a strict queue sequence. In the event of a shortage of resources, which prevents you from immediately processing the sent data, you can put them in a message queue on the server that will store the data until the client is ready.
Message queue plays an important role in scaling distributed systems, as it supports asynchronous communications. We give a brief information about the difference between synchronous and asynchronous systems.
In conventional synchronous systems, tasks are processed one at a time. A task is considered not processed until its processing is completed. This is the easiest way to organize work.
We can also inject threads into this system. In this case, the processing of each task would be performed in parallel.
In a multi-threaded model, threads are controlled by the operating system itself on a single processor or multiple processors / cores.
Asynchronous I / O (AIO) allows the program to continue execution when processing I / O requests. AIO is required in real-time applications. With AIO, we can handle several tasks in one thread.
This is a traditional way of programming, after the process begins, we wait for it to complete. The disadvantage of this approach is that it blocks the execution of the program while the task is being processed. But there is another AIO approach. In AIO, a task that is independent of the process is still ongoing. In the second chapter, we will take a closer look at AIO and use it in ZeroMQ.
You may wonder why you should use a message queue instead of processing all threads for a single-thread or multi-thread approach. Let's look at a situation where you have a web application similar to Google Images, in which you allow users to enter some URLs. Once they submit the form, your application displays all the images for this URL. However:
- If you use a single-line queue, in the case of a large number of users, the program will not be able to process data for all URLs.
- If you are using a multi-thread approach for a queue, then your application may be subject to a DDoS attack.
- You lose data for all URLs in case of a hardware failure.
In this case, you know that you need to add these URLs to the queue and process them. So, you need a message queuing system.
Introduction to ZeroMQ
Up to this point we have considered what a message queue is, which led us to the goal of this book, that is, to ZeroMQ. Society ZeroMQ is defined as "sockets on steroids." Formally, ZeroMQ is defined as a message library that helps developers create distributed and parallel applications.
The first thing we need to learn about ZeroMQ is that it is not a traditional message queuing system, such as ActiveMQ, WebSphereMQ, or RabbitMQ. ZeroMQ is different. She gives us the tools to create your own message queuing system. This is a library.
It works on various architectures from ARM to Itanium and is supported in more than 20 programming languages.
Simplicity
ZeroMQ is simple. We can do some asynchronous I / O operations; ZeroMQ can also queue messages from an I / O stream into a message queue. I / O streams in ZeroMQ work asynchronously when communicating with network traffic, so they can do the rest for you. If you have previously worked with sockets, then you should know that this is not very easy. However, ZeroMQ makes it much easier to work with them.
Performance
ZeroMQ is fast. The Second Life website managed to get 13.4 microseconds of continuous latency and up to 4,100,000 messages per second. ZeroMQ can use a multicast transport protocol, which is an effective method for transmitting data in different directions.
Hello world
So, it's time to start writing code, after we have analyzed what the message queue is and what ZeroMQ is. And of course, we will start with the famous “Hello World” program.
Let's consider the situation when we have a server and a client. The server responds with
world whenever it receives a hello message from the client. The server works on port 4040, and the client, accordingly, sends messages to the same port.Below is the server code that sends the
world message to the client: #include <string.h> #include <stdio.h> #include <unistd.h> #include "zmq.h" int main (int argc, char const *argv[]) { void* context = zmq_ctx_new(); void* respond = zmq_socket(context, ZMQ_REP); zmq_bind(respond, "tcp://*:4040"); printf("Starting…\n"); for(;;) { zmq_msg_t request; zmq_msg_init(&request); zmq_msg_recv(&request, respond, 0); printf("Received: hello\n"); zmq_msg_close(&request); sleep(1); // sleep one second zmq_msg_t reply; zmq_msg_init_size(&reply, strlen("world")); memcpy(zmq_msg_data(&reply), "world", 5); zmq_msg_send(&reply, respond, 0); zmq_msg_close(&reply); } zmq_close(respond); zmq_ctx_destroy(context); return 0; } Below is the client code that sends a
hello message to the server: #include <string.h> #include <stdio.h> #include <unistd.h> #include "zmq.h" int main (int argc, char const *argv[]) { void* context = zmq_ctx_new(); printf("Client Starting….\n"); void* request = zmq_socket(context, ZMQ_REQ); zmq_connect(request, "tcp://localhost:4040"); int count = 0; for(;;) { zmq_msg_t req; zmq_msg_init_size(&req, strlen("hello")); memcpy(zmq_msg_data(&req), "hello", 5); printf("Sending: hello - %d\n", count); zmq_msg_send(&req, request, 0); zmq_msg_close(&req); zmq_msg_t reply; zmq_msg_init(&reply); zmq_msg_recv(&reply, request, 0); printf("Received: hello - %d\n", count); zmq_msg_close(&reply); count++; } // We never get here though. zmq_close(request); zmq_ctx_destroy(context); return 0; } We have the first basic architecture: request-response, as shown in the following diagram:
Let's take a closer look at the code in order to understand how it works.
First we create the context and socket. The
zmq_ctx_new() method creates a new context. It is thread safe, so the context can be used in multiple threads.zmq_socket(2) creates a new socket in a specific context. ZeroMQ sockets are not thread-safe, so they should only be used in the thread in which they were created. Traditional sockets are synchronous, while ZeroMQ sockets have the ability to create one queue on the client side, and another on the server side, asynchronously controlling the request-response pattern. ZeroMQ automatically organizes connection setup, reconnection, disconnection and content delivery. In Chapter 3, we will examine in more detail the difference between traditional sockets and ZeroMQ sockets.The server connects the
ZMQ_REP socket and port 4040 and starts to wait for requests and responds every time it receives a message.This simple program “hello world” shows us an example of using the first request-response pattern.
Request-response pattern
We use a request-response pattern (pattern) to send a message from a client to one or more servers and get a response to each sent message. Most likely, this is the easiest way to use ZeroMQ. Requests for answers must be strictly in order.
Answer
Below is the response of the request-response pattern:
void* context = zmq_ctx_new(); void* respond = zmq_socket(context, ZMQ_REP); zmq_bind(respond, "tcp://*:4040"); The server uses the
ZMQ_REP socket to receive messages from the Internet and send responses to clients. For routing incoming messages from ZMQ_REP , the fair-queue strategy is valid, and for outgoing messages - last-peer .Fair queue strategy
This book is completely devoted to queues. You may be surprised when you find out what we mean when we talk about
fair-queue strategy. This algorithm for planning and allocating resources is fair by definition.To understand how it works, let's say that the flows in the previous figure send 16, 2, 6 and 8 packets per second, respectively, but only 12 packets per second can be processed at the output. In this case, we could transmit 4 packets per second, but Stream 2 only transmits 2 packets per second. The fair-queue rules are that there can be no free exits unless all exits are free at the same time. Thus, it is possible to allow Thread 2 to transmit its 2 packets per second and divide the remaining 10 between the remaining flows.
This incoming message routing strategy uses
ZMQ_REP . Cyclic planning is the easiest way to implement a fair queue strategy, which is also used in ZeroMQ.Request
Below is the request part of the request-response pattern:
void* context = zmq_ctx_new(); printf("Client Starting….\n"); void* request = zmq_socket(context, ZMQ_REQ); zmq_connect(request, "tcp://localhost:4040"); The client uses
ZMQ_REQ to send messages and receive responses from the server. All messages are sent using a round-robin routing strategy. Inbound routing strategy is last-peer .ZMQ_REQ does not throw out any messages. If there are no services available to send a message or all services are busy, then everything is sent to the zmq_send(3) operation, which will be blocked until one of the servers becomes available to send the message. ZMQ_REQ compatible with ZMQ_REP and ZMQ_ROUTER . In chapter 4, we look at ZMQ_ROUTER .Posting a message
This part combines the request and response sections and shows how the answer to someone’s request is and how it is answered.
printf("Sending: hello - %d\n", count); zmq_msg_send(&req, request, 0); zmq_msg_close(&req); The client sends a message to the server using
zmq_msg_send(3) . This is the next message and sends it to the socket. int zmq_send_msg(zmq_msg_t *msg, void *socket, int flags) zmq_msg_send takes three parameters, namely the message, the socket and the flag:- The message parameter is reset to zero at the time of the request, so if you want to send a message to several sockets, you must copy it.
- A successful
zmq_msg_send()request does not indicate if the message was sent over the network. - The flag parameter can be either
ZMQ_DONTWAITorZMQ_SNDMORE. ZMQ_DONTWAITZMQ_SNDMORE. ZMQ_DONTWAITindicates that the message should be sent asynchronously.ZMQ_SNDMOREindicates that the message is composite and the rest of the message is in transit.
After sending the message, the client waits to receive a response. This is done with
zmq_msg_recv(3) . zmq_msg_recv(&reply, request, 0); printf("Received: hello - %d\n", count); zmq_msg_close(&reply); zmq_msg_recv(3) receives part of the message from the socket, as specified in the socket parameter, and saves the response in the message parameter. int zmq_msg_recv (zmq_msg_t *msg, void *socket, int flags) zmq_msg_recv takes three parameters, namely the message, the socket, and the flags.- Previously received messages (if any) are considered invalid.
- The flag parameter can be
ZMQ_DONTWAIT, which indicates that the operation should be performed asynchronously
Working with strings in C
Each programming language has its own approach to handling strings. In Erlang, there are no strings at all (there they are represented as lists of characters). In the C programming language, lines with a zero at the end. Strings in C are basically character arrays, where '\ 0' means the end of the string. Errors with strings are common and the result of many vulnerabilities.
According to Miller et al. (1995), 65 percent of Unix vulnerabilities are due to errors when working with strings, such as zero bytes or buffer overflows, so string handling in C must be done carefully.
When using ZeroMQ, it falls on your shoulders so that the message is reliably formatted, so that other applications can read it. ZeroMQ knows only the size of the message and only.
Using different programming languages in one application is usually the case. If applications written in a programming language that does not add a zero byte at the end of the line should somehow communicate with applications written in C, then you will get strange results.
You can send a message, such as world, as in our zero byte example, as follows:
zmq_msg_init_data_(&request, "world", 6, NULL, NULL); But in Erlang you would send the same message as follows:
erlzmq:send(Request, <<"world">>) Suppose that our client, written in C, connects to the ZeroMQ service, written in Erlang, and we send the message
world this service. In this case, Erlang will display the world . If we send a message with zero byte, then Erlang will print the following [119,111,114,108,100,0] . Instead of the desired string, we got a list containing some numbers, these are ASCII codes. However, it is no longer interpreted as a string.Lines in ZeroMQ are fixed in length and sent without a null byte. Thus, the ZeroMQ string is transmitted as several bytes (the string itself in this example), as well as the length.
Check ZeroMQ version
It is very useful to know which version of ZeroMQ you are using. In some cases, to avoid unwanted surprises, you need to know the exact version. For example, there are some differences between ZeroMQ 2.x and ZeroMQ 3.x, such as using obsolete methods; therefore, if you know the exact version of ZeroMQ that is installed on your computer, then you can avoid using outdated methods.
#include <stdio.h> #include "zmq.h" int main (int argc, char const *argv[]) { int major, minor, patch; zmq_version(&major, &minor, &patch); printf("Installed ZeroMQ version: %d.%d.%d\n", major, minor, patch); return 0; } Summary
In this chapter, we looked at what a message queue is and also made a small introduction to ZeroMQ. We considered how ZeroMQ processes strings and the first pattern (request-response). And also wrote the first application “hello world”.
Resources for this article you can download link
Thank you all for your attention!
Source: https://habr.com/ru/post/198578/
All Articles