Load Balancing: Firebase + RabbitMQ

Modern hosting systems (Heroku, Amazon, etc.) provide a wide range of devices and settings for designing server load balancing architecture. You can configure both a simpler Round Robin algorithm for sequential server unloading, and a more complex system that takes into account the number of instans, current load, environment, and other factors.

Today we will talk about the manual method of load control (one of). I must say that this method was not tested in harsh conditions, but it showed itself well enough in pre-production.

It should also be added that the way I’m going to talk about is not some kind of medicine or a substitute for balancer hosting platforms already provided, but in the situation of a reduced budget it can be a kind of plantain.

So let's get started. What we need:
- Firebase (Cloud NoSQL Database)
- Message Broker. In this case, RabbitMQ
')

Theory

Firebase

The Firebase theme has already been hit several times:
Creating an AngularJS application using Firebase
Authorization of users with AngularJS and Firebase
In one harness Polymers, Dart and Firebase

But within the framework of this article I will tell those things that you need to know to understand it.

Listeners
At the moment, Firebase provides several methods for “listening” to events occurring in the database. You can easily find out when a particular field has been changed, deleted, created or moved. For this, the on () and once () methods can be used. They differ only in that on () will be triggered every time, as a result, carried out with the base of actions, while once () is triggered once.

Examples:

var Firebase = require('firebase'); var dbConnectionUrl = 'https://foo-database.firebaseio.com/posts/java_script' var dbReference = new Firebase(dbConnectionUrl); dbReference.on('child_added', function(addedChildSnap){ var addedChildValue = addedChildSnap.val(); //addedChildSnap    .. "",     doDomethingWithAddedChild(childValue); }); dbReference.once('child_removed', function(removedChildSnap){ var removedChildValue = removedChildSnap.val(); doSomethingWithRemovedChild(removedChildValue); }); 

The "child_added" and "child_removed" events are part of the Firebase API (I did not invent them from my head). For more information on them, you can follow the links:
www.firebase.com/docs/web/api/query
www.firebase.com/docs/web/api/query/on.html

“Child_added” will be triggered every time a new post is added to “java_script” (note dbConnectionUrl) because we used the on () listener. In the case of child_removed (“shoots” when deleting an entity), we use once (), so the event will run once and, if we remove more than one element, no action will be taken.

RabbitMQ

Again, this technology has already been considered in Habré.
RabbitMQ tutorial 1 - Hello World
...
RabbitMQ tutorial 6 - Remote procedure call

Those. This is such a postman, whose goal is to receive and give messages. These messages can be accumulated and stored in it until they are completely sent. In the context of this article, it is worth knowing that this postman can give a message one by one, regardless of whether the listener successfully accepts them or give a new message only when the receiving party “tapped” him and said that the message was received, they say , thank. So, in our case, we need a second way to work properly.

General scheme

Front-End does not work with the Back-End at all
The specifics of Firebase are such that you can work with this database without using the server part, which provides some speed advantages.

Front-End makes changes to the base - the Back-End "listens" to these changes
Using standard Firebase tools, we can easily “listen” on actions performed in the database.

Back-end sends messages to the queue
Receiving a response from Firebase as a result of the action taken over the base, the back-end forms the object and sends it to the queue

RabitMQ Server sends messages
Data service is responsible for storing and "issuing" messages as they appear in the queue

Practice

Conditions
Database Url: https://foo-database.firebaseio.com/
Database Entities: users_approved, users_requested
( https://foo-database.firebaseio.com/users_approved , https://foo-database.firebaseio.com/users_requested )

Task
The front-end will create entities in the database (users_requested), in turn, the back-end will check objects for validity and perform subsequent magic.

I will not create a fully-functional Front-End, so as not to clutter an article with an extra code, so I’ll just show only the necessary functional parts. By the way, this also applies to the back-end part.

Front end

 var dbRootReference = new Firebase('https://foo-database.firebaseio.com/'); var newRequestedUser = { firstName: 'John', lastName: 'Connor', email: connor.john@domain.com }; dbRootReference.child('users_requested').push(newRequestedUser); 

This is enough to add a new entry to the “requested_users” from the client side.
Of course I missed the step with authentication to the database. Do not think that everything is very bad with security. Read more about the auth method here.

Back-End: RabbitMQ

 //     AQMP  - ,    RabbitMQ //     (amqp, node-amqp, bramqp  ..),       "amqplib" var amqp = require('amqplib'); var amqpConnectionUrl = 'amqp://localhost'; //  var amqpConnection = amqp.connect(amqpConnectionUrl); //     var QUEUE_USERS_REQUESTED = 'requestedUsers'; var QUEUE_USERS_APPROVED = 'approvedUsers'; //  (    ) amqpConnection.then(function (successAmqpConnection) { successAmqpConnection .createChannel() .then(function (amqpChannel) { //          //         ,        //  channel    requested  amqpChannel.assertQueue(QUEUE_USERS_REQUESTED, {durable: false, noAck: false}); //  channel    approved  amqpChannel.assertQueue(QUEUE_USERS_APPROVED, {durable: false, noAck: false}); //noAck      ,      TRUE (ack: true  noAck: false), //  ,      ,     //   QUEUE_USERS_REQUESTED amqpChannel.consume(QUEUE_USERS_REQUESTED, function (msg) { if (msg !== null) { var requestedUserDetails = JSON.parse(msg.content.toString()); //  JSON-  return removeUserFromRequests(requestedUserDetails) .then(validateUser) .then(function (isValid) { if (isValid) { return; } else { throw Error('user not valid'); } }) //   'users_approved' //dbRootReference.child(USERS_APPROVED).push(requestedUserDetails); .then(addUserAsApproved) .then(function (result) { //   ,         ack(); amqpChannel.ack(msg); // ,   ,    }) .catch(function (err) { //        amqpChannel.ack(msg); }); } else { //    //  ,   ,    -     amqpChannel.ack(msg); } }); //   QUEUE_USERS_APPROVED amqpChannel.consume(QUEUE_USERS_APPROVED, function (msg) { if (msg !== null) { var approvedUserDetails = JSON.parse(msg.content.toString()); //  JSON-  sendWelcomeMessage(approvedUserDetails) //   .then(function (result) { amqpChannel.ack(msg); // AQMP-,        }); } else { amqpChannel.ack(msg); // ,   ,    } }); }); }); // publisher'      function pushToUserRequestedQueue(requestedUser) { return amqpChannel.sendToQueue(QUEUE_USERS_REQUESTED, JSON.stringify(requestedUser)); } function pushToUserApprovedQueue(approvedUser) { return amqpChannel.sendToQueue(QUEUE_USERS_APPROVED, JSON.stringify(approvedUser)); } 

Back-End: Firebase

 //  var Firebase = require('Firebase'); var dbUrl = 'https://foo-database.firebaseio.com/'; var dbRootReference = new Firebase(dbUrl); var USERS_REQUESTED = 'users_requested'; var USERS_APPROVED = 'users_approved'; // 'users_requested'     dbRootReference.child(USERS_REQUESTED).on('child_added', function (requestedUserSnap) { var requestedUser = requestedUserSnap.val(); //      sendMessageToUserRequestedQueue(requestedUser); }); // 'users_requested'     dbRootReference.child(USERS_APPROVED).on('child_added', function (newApprovedUserSnap) { var approvedUser = newApprovedUserSnap.val(); //      sendMessageToUserApprovedQueue(approvedUser); }); 

Conclusion

That's all. When detecting data in the database on the 'users_requested' path, our server will send a message to the queue, which will be further processed based on the business logic embedded in the project. A new message will be processed only when it is given permission, thereby reducing the load on the server. Sending a message to a queue requires much less resources than, for example, performing a series of operations for validating and saving a user.

Pros & Cons

- using Firebase in this case, we reduce the response time of the database, as well as the processing speed
- we get the budget option load-balancer at the stage of creating MVP
- the absence of Map Reduce in Firebase makes it questionable in dealing with large amounts of data, on the other hand, Real-Time principle shows itself very well, so with the right approach everything can turn out quite well
- due to the fact that Firebase does not yet have a convenient ORM (as I think), the base is not very convenient to use
- not every application can be compatible with such a scheme, say, waiting for when the user's personal data will be changed upon his request is not very good, but in the case of processing orders of an online store, it is quite suitable
- Affordable API and dense non-blurry information brings pleasant moments to the study and development