Design and naming of queues
A few rules on how to organize points, queues and how to correctly name them so that it is convenient.
Each queue must represent only one job type. Do not mix different types of messages in the same queue. And when this rule is observed, we can clearly name the queue with the task presented to them.
Avoid re-sending messages to the queue. If you find that your subscriber is trying to re-send any messages to other queues without actually processing, something is likely not designed correctly. Routing is the responsibility of the exchange points, not the queues.
')
Providers do not need to know anything about queues. One of the main ideas of AMQP is to share the responsibility of points and queues, so that suppliers do not need to worry about the message getting to the subscriber.
Suppose you want to design points and queues for “user-related” write events. Record events will run in one or more applications, and these messages will be used by some other applications.
The first question that is usually asked is different events of the same object (the user object in this example), should one exchange point be used to publish all three events or use 3 separate points for each event? Or, in short, a single exchange point, or a lot?
Before answering this question, I want to ask another question: do we really need a separate point for this case? What if we abstract all 3 types of events as a “write” event, whose subtypes are “created”, “updated” and “deleted”?
The simplest solution is to create a “user.write” queue, and publish all the messages of a user’s write event to this queue through the global exchange point.
The simplest solution cannot work when there is a second application (with a different processing logic) that wants to subscribe to any messages posted in the queue. When several applications are signed, we at least need one point with a “fanout” type with bindings to multiple queues. Thus, messages are sent to a point, and it duplicates messages in each queue. Each queue represents a processing task for each application.
The second solution works fine if every subscriber really wants to handle all the “user.write” subtypes of events. For example, if the subscriber application is designed to simply store the transaction log.
On the other hand, it’s not very good when some subscribers are outside of your organization and you want to notify them only about certain specific events, for example, the app2 application should receive a message about the creation of a user and should not be aware of update and delete events.
To solve the problem above, we need to extract the “user.created” event from the “user.write” type. An exchange point with a type of “topic” can help us. When posting messages, we will use user.created / user.updated / user.deleted as routing keys at a point, so we can put the communication key “user. *” In the queue “user.write.app1” and the communication key “user.created” in the queue "user.created.app2".
The type of “topic” of the exchange point is more flexible in case there are potentially more types of events. But if you clearly know the exact number of events, you can also use the “direct” type to improve performance.
We return to the question “one point, or many?”. While all solutions use only one point, it works fine, nothing bad. In what situations can we need a few points?
Let's look at an example when, in addition to the created, updated, and deleted events described above, we have another group of events: input and output - a group of events that describe "user behavior" rather than "data recording". For different groups of events you may need completely different routing strategies and agreements on keys and queue names, for this you need separate exchange points.
Free translation of the article RabbitMQ Exchange and Queue Design Trade-off .
| exchange | type | binding_queue | binding_key | |-------------------------------------------------------| | user.write | topic | user.created.app2 | user.created | Concepts
AMQP (Advanced Message Queuing Protocol) is an open protocol for transferring messages between system components.
Supplier (Publishers / Producer) - a program that sends messages.
Subscriber (Consumer) - a program that receives messages. Typically, the subscriber is in a message waiting state.
Queue - a message queue.
Exchange point is the initial exchange point of the queue that deals with routing.
Supplier (Publishers / Producer) - a program that sends messages.
Subscriber (Consumer) - a program that receives messages. Typically, the subscriber is in a message waiting state.
Queue - a message queue.
Exchange point is the initial exchange point of the queue that deals with routing.
Rule 1
Each queue must represent only one job type. Do not mix different types of messages in the same queue. And when this rule is observed, we can clearly name the queue with the task presented to them.
Rule 2
Avoid re-sending messages to the queue. If you find that your subscriber is trying to re-send any messages to other queues without actually processing, something is likely not designed correctly. Routing is the responsibility of the exchange points, not the queues.
')
Rule 3
Providers do not need to know anything about queues. One of the main ideas of AMQP is to share the responsibility of points and queues, so that suppliers do not need to worry about the message getting to the subscriber.
Examples and Solutions
Suppose you want to design points and queues for “user-related” write events. Record events will run in one or more applications, and these messages will be used by some other applications.
| object | event | |------------------| | user | created | | user | updated | | user | deleted | The first question that is usually asked is different events of the same object (the user object in this example), should one exchange point be used to publish all three events or use 3 separate points for each event? Or, in short, a single exchange point, or a lot?
Before answering this question, I want to ask another question: do we really need a separate point for this case? What if we abstract all 3 types of events as a “write” event, whose subtypes are “created”, “updated” and “deleted”?
| object | event | sub-type | |-----------------------------| | user | write | created | | user | write | updated | | user | write | deleted | Solution 1
The simplest solution is to create a “user.write” queue, and publish all the messages of a user’s write event to this queue through the global exchange point.
Solution 2
The simplest solution cannot work when there is a second application (with a different processing logic) that wants to subscribe to any messages posted in the queue. When several applications are signed, we at least need one point with a “fanout” type with bindings to multiple queues. Thus, messages are sent to a point, and it duplicates messages in each queue. Each queue represents a processing task for each application.
| queue | subscriber | |-------------------------------| | user.write.app1 | app1 | | user.write.app2 | app2 | | exchange | type | binding_queue | |---------------------------------------| | user.write | fanout | user.write.app1 | | user.write | fanout | user.write.app2 | The second solution works fine if every subscriber really wants to handle all the “user.write” subtypes of events. For example, if the subscriber application is designed to simply store the transaction log.
On the other hand, it’s not very good when some subscribers are outside of your organization and you want to notify them only about certain specific events, for example, the app2 application should receive a message about the creation of a user and should not be aware of update and delete events.
Solution 3
To solve the problem above, we need to extract the “user.created” event from the “user.write” type. An exchange point with a type of “topic” can help us. When posting messages, we will use user.created / user.updated / user.deleted as routing keys at a point, so we can put the communication key “user. *” In the queue “user.write.app1” and the communication key “user.created” in the queue "user.created.app2".
| queue | subscriber | |---------------------------------| | user.write.app1 | app1 | | user.created.app2 | app2 | | exchange | type | binding_queue | binding_key | |-------------------------------------------------------| | user.write | topic | user.write.app1 | user.* | | user.write | topic | user.created.app2 | user.created | Solution 4
The type of “topic” of the exchange point is more flexible in case there are potentially more types of events. But if you clearly know the exact number of events, you can also use the “direct” type to improve performance.
| queue | subscriber | |---------------------------------| | user.write.app1 | app1 | | user.created.app2 | app2 | | exchange | type | binding_queue | binding_key | |--------------------------------------------------------| | user.write | direct | user.write.app1 | user.created | | user.write | direct | user.write.app1 | user.updated | | user.write | direct | user.write.app1 | user.deleted | | user.write | direct | user.created.app2 | user.created | We return to the question “one point, or many?”. While all solutions use only one point, it works fine, nothing bad. In what situations can we need a few points?
Solution 5
Let's look at an example when, in addition to the created, updated, and deleted events described above, we have another group of events: input and output - a group of events that describe "user behavior" rather than "data recording". For different groups of events you may need completely different routing strategies and agreements on keys and queue names, for this you need separate exchange points.
| queue | subscriber | |----------------------------------| | user.write.app1 | app1 | | user.created.app2 | app2 | | user.behavior.app3 | app3 | | exchange | type | binding_queue | binding_key | |--------------------------------------------------------------| | user.write | topic | user.write.app1 | user.* | | user.write | topic | user.created.app2 | user.created | | user.behavior | topic | user.behavior.app3 | user.* | Free translation of the article RabbitMQ Exchange and Queue Design Trade-off .
Source: https://habr.com/ru/post/443722/
All Articles