Message Queue

Message Queue

This post talks about message queues - why you need to know about them, think when planning an architecture, and use them in your application.

Why message queues?

Messages, along with the blocks of calculation and storage, make up the three main blocks in almost every block diagram of the system. Message queues are essentially the link between the different processes in your applications and provide a reliable and scalable interface to other connected systems and devices.

The turn - the data structure with the discipline of access to the elements of "first come - first come". Adding an item is possible only to the end of the queue, sampling is only from the beginning of the queue, and the selected item is removed from the queue.

Using Message Queuing

Ten reasons why message queues are a vital component for any architecture or application:

• Weak binding - message queues create implicit data exchange interfaces that allow processes to be independent of each other; that is, you simply define the format of messages sent from one process to another.
• Redundancy - Queues allow you to avoid cases of uneconomical use of process resources (for example, memory) as a result of storing raw (extra) information.
• Scalability - Message Queuing allows you to distribute information processing. Thus, they allow you to easily increase the speed at which messages are added to the queue and processed.
• Elasticity and the ability to withstand peak loads - message queues can act as a kind of buffer for storing data in case of peak load, thereby easing the load on the information processing system and preventing its failure.
• Fault tolerance - message queues allow you to separate processes from each other, so if the process that processes messages from the queue drops, then messages can be added to the processing queue later when the system recovers.
• Guaranteed delivery - the use of a message queue ensures that the message will be delivered and processed in any case (as long as there is at least one handler).
• Guaranteed delivery order - most message queuing systems are able to provide assurance that data will be processed in a specific order (most often in the order in which they arrived).
• Buffering — a message queue allows you to send and receive messages while working with maximum efficiency, offering a buffer layer — the process of writing to the queue can occur as quickly as it is able to execute the message queue, and not the message handler.
• Understanding data streams - message queues allow you to identify bottlenecks in application data streams, you can easily determine which queue is clogged, which is idle, and what needs to be done - add new message handlers or optimize the current architecture.
• Asynchronous communication — Message queues provide the ability to asynchronously process data, which allows you to place a message on a queue without processing, allowing the system to process the message later when it becomes available.

I can tell you more about practical recommendations - how and why people use message queues, but perhaps next time.

In general, the applications for message queues include:

')

• Data processing
• Data stream buffering
• Process management
• Integration and interaction of systems

The above areas should give you ideas on where message queues can be used. If they are not a standard part of your toolkit, you probably missed something that can reduce the complexity of your system and solve a bunch of problems.

Why SaaS?

Adding message queuing for cloud applications only makes sense if there is a net gain in terms of installation and operation. Adding an additional architectural layer responsible for queuing messages is not an easy task, especially if you decide to use your own solution or install third-party servers, as this will add additional costs for monitoring, tuning, managing and affecting the overall reliability and security of the system.



When message queues are easy to install, easy to use, highly available, and extremely reliable, everything becomes much easier.

Here the analogy of obtaining energy is appropriate. Progress ranged from windmills and coal-fired furnaces to industrial power plants and power lines. This last step, the industrialization of energy, changed the face of industry in the world. This reduced construction and manufacturing costs, changed cities, factories, and homes, and allowed new inventions, services, and businesses to be created.



Similarly, by connecting Message Queuing services, developers no longer have to support a huge set of services running on multiple servers and not be afraid of downtime due to system failure. In today's world, service providers take responsibility for managing servers, APIs, and other resources, and a developer abstracting from most physical constraints can concentrate on implementing his idea.

Benefits of migrating to cloud-based message queues include:

• Increased speed to market: applications and systems can be built much faster.
• Reduced complexity: reduced risk and overhead in strategic capacity. For example, it now seems to you that your own server with a raised and configured RabbitMQ seems to be the best solution, then in the long term with increasing load, HA requirements (high availability), early integration of third-party services can play a positive role.
• Increased scalability: the ability to easily scale performance and functionality

Where to begin?

There are not so many different services that provide message queue services:

• Amazon SQS
• Ironmq
• Stormmq
• Windows Azure Queues

In this case, SQS provides 100,000 free messages (I understand that for the entire period of use), IronMQ - 10,000,000 free messages per month (provided that you enter credit card information), and StormMQ is currently in a closed beta. Vplane availability of examples of use and libraries in the leaders of SQS, although IronMQ also supports the most popular languages ​​(Ruby, Python, C #, Java ...) and examples of use are already enough.



PS I hope I managed to put a drop of doubt in the choice of “put my MQ server or use a third-party service” and interest in existing SaaS solutions in the field of message queues.



upd : added Windows Azure Queues