Messaging in Microsoft Azure, or How to communicate in the clouds
My acquaintance with the Azure cloud began during my practice at DataArt. In the third month of training, my colleague Anton came to my mentor Dima and said that he needed someone for a small demonstration project. So we met.
Anton - a man enthusiastic and very active, knows how to inspire ideas. The practice ended a long time ago, after some time, Anton broke up with DataArt, but continues to be actively interested in and promote Microsoft Azure, often consults our colleagues on Azure technology and offers to participate in various thematic events. In particular, he holds various seminars, trainings, meetings dedicated to this cloud, and not long ago, the first Ukrainian Azure Community in our country was created in Kiev.
This is a group of like-minded people who are actively interested in technology, share their knowledge and communicate on the subject of Azure. Once a month or two meetings are held, each devoted to a separate topic. Often these meetings take place just at the DataArt office in Kiev.
')
The theme of the meeting, where, in particular, I spoke, chose to exchange messages using different Azure services. We heard about Azure Storage Queues, Azure Service Bus Queues, Azure Service Bus Topics & Subscriptions.
I will share the technical part. There are three types of messaging models.
Fig. one
In the case of model A, the message is sent directly from the sender to the recipient. This is the simplest model, but it has flaws:
- it is difficult to scale (yes, yes, these are clouds!);
- it is easy to lose messages (for example, the recipient goes offline);
- difficult to implement retry policy.
Because of these shortcomings, models B are mainly used in Azure (the message gets into a kind of broker, the recipient "picks up" it from there) and B (the message gets into the broker, and the broker sends it to the recipient).
This is the most common FIFO queue. It corresponds to model B.
Within one Azure account, you can create an arbitrary number of queues with unique names. The queue name must be url-friendly. Each queue can contain an arbitrary number of messages, but the size of a single message is limited to 64 KB. That is, the queue is not intended for exchanging data, but only for exchanging tasks.
(A question to think: what if the message for some reason does not fit in 64K? Think in the context of Azure.)
Fig. 2
Sending a message to the queue looks like this:
The capacity of Azure to send to the queue - up to 20 thousand messages per second.
The peculiarity of this queue is a specific timeout (by default 30 seconds) for processing a single message. That is, when reading, the message is not removed from the queue, but becomes invisible to other recipients. If the message was processed correctly, it must be explicitly removed from the queue. If there are read errors or another exception - in general, if the timeout expires, the message will not be explicitly deleted - it will again appear in the head of the queue and become available for reading by other recipients.
Correct processing of the message looks like this:
Timeout for each message you can set your own (on the recipient side). Messages can only contain lines that are encoded in base64 for forwarding. In addition, there is an opportunity to "see" the contents of the message, without taking it from the queue, and you can read the message in "packets" - several at once.
REST api is provided for working with Azure Storage Queue, which allows working with queues regardless of platform. By the way, this is currently the only messaging technology in Azure, with which you can work directly from JS.
As you can see, the service provides not very wide functionality, or rather, strictly defined. Nevertheless, it provides an opportunity to raise the probability of message delivery to the recipient compared to forwarding messages using model A (without an intermediary). Such a queue is well suited for architectures when the recipient is not always online. Example - the sender performs some activity during the day, logs its activity in the queue, and at the end of the day the receiver appears in the network, processes this information, generates reports, for example, and again “falls asleep” until the next day.
Finally, Azure Storage Queue is the cheapest messaging service in Azure.
Azure Service Bus is a service that provides several messaging capabilities. In principle, they can be divided into two types:
Fig. 3
Azure Service Bus Topics & Subscriptions (Fig. 4) also implements model B, but in a completely different way.
Fig. four
In the service bus namespace, so-called topics are created. This can literally be understood as “themes”. Subscriptions to themes are also created there - subscriptions. A message from the sender passes a specific topic (the sender decides which topic to send the message to) and already in the topic it is automatically filtered and copied into each subscription that satisfies the filter. The message is not stored in the topic. If there is no suitable subscription for the message, it disappears. If there are several suitable subscriptions, it will be copied to each of them.
Subscriptions create message recipients. For each filter is specified (which may be empty, it will mean that all the messages that come to the corresponding topic will be copied to the subscription). The filter can be changed without changing the subscription, while the messages that are already there will not be lost, and the new ones will be received by the updated filter.
Reading messages by the recipient can be organized using two mechanisms: ReceiveAndDelete and PeekLock . The first involves the usual reading of a message from the queue - after a request is received for a message in a subscription, it is sent to the recipient and removed from the queue. The second mechanism, as in Azure Storage Queues, involves some kind of timeout to process the message. And if, upon its expiration, there is no confirmation that the message was successfully processed, it again becomes visible to other recipients.
Read more about Azure Storage Queues here .
About Azure Service Bus Queues here .
A comparison of Azure Storage Queues and Azure Service Bus Queues .
About Azure Service Bus Topics & Subscriptions here .
Which implementation of messaging to choose for the project, you decide. First of all, you need to build on how critical the delivery of messages is and how much you are willing to pay for it.
In conclusion, I would like to note that Microsoft Azure technologies are developing very rapidly, new features appear, some become outdated, so it's always better to check the latest information on the official website.
Author: Anastasia Belokurova, .NET Developer
Anton - a man enthusiastic and very active, knows how to inspire ideas. The practice ended a long time ago, after some time, Anton broke up with DataArt, but continues to be actively interested in and promote Microsoft Azure, often consults our colleagues on Azure technology and offers to participate in various thematic events. In particular, he holds various seminars, trainings, meetings dedicated to this cloud, and not long ago, the first Ukrainian Azure Community in our country was created in Kiev.
This is a group of like-minded people who are actively interested in technology, share their knowledge and communicate on the subject of Azure. Once a month or two meetings are held, each devoted to a separate topic. Often these meetings take place just at the DataArt office in Kiev.
')
The theme of the meeting, where, in particular, I spoke, chose to exchange messages using different Azure services. We heard about Azure Storage Queues, Azure Service Bus Queues, Azure Service Bus Topics & Subscriptions.
I will share the technical part. There are three types of messaging models.
Fig. one
In the case of model A, the message is sent directly from the sender to the recipient. This is the simplest model, but it has flaws:
- it is difficult to scale (yes, yes, these are clouds!);
- it is easy to lose messages (for example, the recipient goes offline);
- difficult to implement retry policy.
Because of these shortcomings, models B are mainly used in Azure (the message gets into a kind of broker, the recipient "picks up" it from there) and B (the message gets into the broker, and the broker sends it to the recipient).
Azure Storage Queues
This is the most common FIFO queue. It corresponds to model B.
Within one Azure account, you can create an arbitrary number of queues with unique names. The queue name must be url-friendly. Each queue can contain an arbitrary number of messages, but the size of a single message is limited to 64 KB. That is, the queue is not intended for exchanging data, but only for exchanging tasks.
(A question to think: what if the message for some reason does not fit in 64K? Think in the context of Azure.)
Fig. 2
Sending a message to the queue looks like this:
// Retrieve storage account from connection string CloudStorageAccount storageAccount = CloudStorageAccount.Parse(_cloudConnectionString); // Create the queue client CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient(); // Retrieve a reference to a queue CloudQueue queue = queueClient.GetQueueReference("somecorrectqueuename"); // Create the queue if it doesn't already exist queue.CreateIfNotExists(); string messageToSend = string.Format("message #{0}", i); // Create a message and add it to the queue. CloudQueueMessage message = new CloudQueueMessage(messageToSend); queue.AddMessage(message); The capacity of Azure to send to the queue - up to 20 thousand messages per second.
The peculiarity of this queue is a specific timeout (by default 30 seconds) for processing a single message. That is, when reading, the message is not removed from the queue, but becomes invisible to other recipients. If the message was processed correctly, it must be explicitly removed from the queue. If there are read errors or another exception - in general, if the timeout expires, the message will not be explicitly deleted - it will again appear in the head of the queue and become available for reading by other recipients.
Correct processing of the message looks like this:
// Retrieve storage account from connection string CloudStorageAccount storageAccount = CloudStorageAccount.Parse(_cloudConnectionString); // Create the queue client CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient(); // Retrieve a reference to a queue CloudQueue queue = queueClient.GetQueueReference("somecorrectqueuename "); while (true) { // Get the next message CloudQueueMessage retrievedMessage = queue.GetMessage(); if (retrievedMessage != null) { //Process the message in less than 30 seconds, and then delete the message queue.DeleteMessage(retrievedMessage); } else { Thread.Sleep(2000); } } Timeout for each message you can set your own (on the recipient side). Messages can only contain lines that are encoded in base64 for forwarding. In addition, there is an opportunity to "see" the contents of the message, without taking it from the queue, and you can read the message in "packets" - several at once.
REST api is provided for working with Azure Storage Queue, which allows working with queues regardless of platform. By the way, this is currently the only messaging technology in Azure, with which you can work directly from JS.
As you can see, the service provides not very wide functionality, or rather, strictly defined. Nevertheless, it provides an opportunity to raise the probability of message delivery to the recipient compared to forwarding messages using model A (without an intermediary). Such a queue is well suited for architectures when the recipient is not always online. Example - the sender performs some activity during the day, logs its activity in the queue, and at the end of the day the receiver appears in the network, processes this information, generates reports, for example, and again “falls asleep” until the next day.
Finally, Azure Storage Queue is the cheapest messaging service in Azure.
Azure Service Bus Queues, Azure Service Bus Topics & Subscriptions
Azure Service Bus is a service that provides several messaging capabilities. In principle, they can be divided into two types:
- Brokered bus - asynchronous messaging; saves messages, sends them when the recipient is ready to read them, uses two mechanisms: 'queues' and 'topics and subscription'
- Relayed bus - synchronous messaging; Messages are not saved anywhere, they are sent “on Wednesday” and, if the recipient is ready, he will read them, if not - messages are lost.
- Azure Service Bus Queues (Fig. 3) assumes model B (Fig. 1). It is fundamentally different from Azure Storage Queues in that messages from the broker are sent to the recipient when it reports readiness, and not vice versa. You can also send several messages at a time (using transactions). In addition, if Azure Storage Queues support the At-Least-Once delivery guarantee, that is, messages will be read at least once (or even under certain scenarios and more), Azure Service Bus Queues support At-Least-Once At “Most-Once Guaranteed” means that the message will be read no less and no more than once.
Fig. 3
Azure Service Bus Topics & Subscriptions (Fig. 4) also implements model B, but in a completely different way.
Fig. four
In the service bus namespace, so-called topics are created. This can literally be understood as “themes”. Subscriptions to themes are also created there - subscriptions. A message from the sender passes a specific topic (the sender decides which topic to send the message to) and already in the topic it is automatically filtered and copied into each subscription that satisfies the filter. The message is not stored in the topic. If there is no suitable subscription for the message, it disappears. If there are several suitable subscriptions, it will be copied to each of them.
Subscriptions create message recipients. For each filter is specified (which may be empty, it will mean that all the messages that come to the corresponding topic will be copied to the subscription). The filter can be changed without changing the subscription, while the messages that are already there will not be lost, and the new ones will be received by the updated filter.
Reading messages by the recipient can be organized using two mechanisms: ReceiveAndDelete and PeekLock . The first involves the usual reading of a message from the queue - after a request is received for a message in a subscription, it is sent to the recipient and removed from the queue. The second mechanism, as in Azure Storage Queues, involves some kind of timeout to process the message. And if, upon its expiration, there is no confirmation that the message was successfully processed, it again becomes visible to other recipients.
Read more about Azure Storage Queues here .
About Azure Service Bus Queues here .
A comparison of Azure Storage Queues and Azure Service Bus Queues .
About Azure Service Bus Topics & Subscriptions here .
Which implementation of messaging to choose for the project, you decide. First of all, you need to build on how critical the delivery of messages is and how much you are willing to pay for it.
In conclusion, I would like to note that Microsoft Azure technologies are developing very rapidly, new features appear, some become outdated, so it's always better to check the latest information on the official website.
Author: Anastasia Belokurova, .NET Developer
Source: https://habr.com/ru/post/257975/
All Articles