Node.js and cote: simple and convenient development of microservices

Many consider that microservices are very difficult. In fact, with the right approach, this is not at all the case.

Microservices today are very popular, and the real adherents of this architecture almost bow to everything on which it is written "microservice." However, if we reject fanaticism, such an approach to software development is a worthy step forward, microservices can forever change how server-side software applications are created. Around microservices there is a lot of information noise, so it is worth highlighting the truly important properties of this architecture and work on simplifying its implementation and use where it is really needed.



If you treat microservices with caution, or feel that you are confused in this thread, know that you are not alone. From an architectural point of view, microservices are not so simple. The matter is aggravated by the ecosystem around them. Instead of directly solving the complex tasks associated with working with microservices, those who are engaged in it try to get away from direct solutions, simplifying one thing and complicating the other. As a result, this leads to the fact that we have to maintain a large set of technologies and deal with errors that cannot be debugged because of the complexity of the whole structure. It is not easy even to check the performance of the application. It's time to fix this situation.
')
Here we will step by step analyze the development of a web application based on microservices. This example may well be your first such project, and if you are good at developing for Node.js, it will take a few minutes to get everything working.

Before we get down to business, I would like to clarify that I am the author of the library for Node.js cote , which simplifies and accelerates the development of microservice-based applications.

An example implementation of an application based on microservices

There are many ways to demonstrate the implementation of microservice architecture, and, in fact, a well-developed example is all that an experienced developer needs. However, this material is designed for programmers with different levels of training, including beginners, so a simple but full-fledged example was chosen for the demonstration: an application for currency conversion. We will create a system of four services, the joint work of which will allow to organize the conversion in terms of currency fluctuations.


Overview of services. Each of them can scale independently. Arrows indicate the direction of data flows.

We will create a system consisting of the following parts.

1. The client of the currency converter (conversion client) - he can request the conversion of a certain amount of money in one currency into another currency.
   
   
2. Service of the currency converter (conversion service) - currency rates are known to him, he can respond to customer requests. In addition, this microservice can accept updates regarding currency conversion rates.
   
   
3. Arbitration service (arbitration service) - it maintains current exchange rate information, and, when changed, publishes updates.
   
   
4. The arbitration service administration tool (arbitration admin) is a service that you can implement yourself as a home task. It is needed to manage the arbitration service, allowing you to inform it of the need to update any currency conversion rate.

Before we get down to business, let's talk a little about the history of microservices.

Subjective view of microservice history

Let's take a look at the past. Ten years ago, when microservices still did not appear, service-oriented architecture (SOA, service-oriented architecture) was in high esteem. There were hundreds of solutions for building service-oriented applications, and around this, a serious consulting business was built.

We are going to be seven years closer to our time, here we will meet microservices. Although at a fundamental level, they are very different from SOA, those who had previously built service-oriented applications switched to a new technology. Everything is good, but behind these developers was luggage from the past, in which a lot of excess.

Overnight, existing development approaches were transferred to the new environment. Although the basic assumptions underlying microservices differed from SOA, these first solutions based on old technologies were presented to the public as “the only true ones”. As a result, among other things, we had AMQP, HTTP, or even, God forbid, SOAP. These were nginx, zookeeper, etcd, consul and several other solutions that were tried to give the appearance of the tools vital for creating microservices.

The problem with all this was that these technologies were originally created to solve very different problems. They were, so to speak, “soldered” to microservices. However, after the developers used some of these tools to build their own services, they, at best, left a sense of temporary improvised means. It looked like nailing with pliers. When there is nothing more at hand, you can hammer a nail, but is it not better to look for a normal hammer?

It was always obvious that the barrier to entry into the field of microservices is too high, although the transition to them and promised economic benefits. However, outside the window of 2017, we all deserve better than the old technologies, with the help of which they solve new problems. It's time to declare that in order to study, create and use microservices, in order to achieve their scaling when solving real problems, you need only Node.js.

Now let's talk about the architecture of modern microservices.

Five requirements for microservices

Here are the requirements that modern systems based on microservices must meet.

1. Automatic system configuration . Any system based on microservices is likely to include hundreds of services. Manual configuration of IP addresses, ports and API capabilities is almost impossible task.
   
   
2. High system redundancy . In the above-described system, microservices failures are quite possible, therefore, it is very important to have a mechanism for emergency switching to reserve capacity, the creation of which should not cause any special troubles and costs.
   
   
3. Fault tolerance of the system . The system must withstand and adequately handle abnormal events, such as network failures, errors in processing messages, timeouts, and so on. Even if some services have ceased to function, all the rest, not connected with them, should work.
   
   
4. Self-healing system . Failures and all sorts of errors should be treated as completely expected phenomena. In the implementation of the system should be incorporated automatic recovery of any lost due to a functional failure or service.
   
   
5. Automatic discovery of services . Existing services should automatically identify new services entered into the system, start interacting with them without manual intervention or downtime.
   

If your architecture meets these requirements and if you split the execution of most of your API requests into several independent services, then what you did is microservices.

These are not microservices.

I would like to emphasize once again one important thing: the architecture of microservices is not tied to any particular technology. Say, good old working queues and consumer flows are not microservices. Mail demons, notification mechanisms, any auxiliary services that only consume events and do nothing to process user requests are not microservices.

Is some administrative application and client program different? This separation also does not mean the use of microservice architecture. Server daemons that can receive and process HTTP requests ... And these are not microservices. And what if we have a cluster of servers, the computers in which are named after the moons of Jupiter? Does an administrator need to interfere with their work? If yes, then there are no microservices.

It is clear that in the wake of the popularity of the new technology it is tempting to call everything “microservice architectures” everything. However, this only creates information noise and does not allow other people to understand what microservices are and, therefore, use them in their developments.

Microservices have always strived for minimalism. “Heavy” technologies, about which they say that only they make it possible to create microservices, is the direct opposite of them.

Cote library and microservices

Microservices built on the basis of the cote library are configured automatically. The library uses broadcast or multicast IP-distribution, as a result, the demons on the same network find each other and automatically exchange the information necessary for setting up the connection. In this regard, cote meets the requirements of №1 and №5: "Automatic system configuration" and "Automatic discovery of services."

With the help of cote, it is possible to create multiple copies of services very efficiently, with low resources, while processing requests is scaled automatically. This means that the library also corresponds to requirement No. 2: “High level of system redundancy.”

If there are no services in the system to satisfy a certain request, cote caches it and waits until the required service is available. Since services are usually independent, such an organization of a system means its fault tolerance, and this corresponds to requirement No. 3: “Fault tolerance of the system”.

The remaining requirement # 4: “System self-recovery” is fulfilled due to the use of Docker, giving the opportunity to restart services in case of failures. Since the system runs automatic discovery of services, even if Docker decides to deploy a dropped service on a new machine, all remaining services will find it and interact with it.

Thus, cote gives the developer complete freedom in terms of infrastructure, taking care of fulfilling the basic requirements for systems built on the basis of microservices.

With cote, you can focus on the most important aspects of application development, leaving the library to perform supporting tasks.

I suppose I have told in an accessible and understandable manner what real microservices are. Now let's talk about how to implement all this.

▍Installing cote

We will create a system based on microservices in the Node.js environment, using the cote library, the main features of which we discussed above. It is available as a npm package for Node.js.

Install the cote:

npm install cote 

▍First acquaintance with cote

Whether you want to integrate cote with an existing web application, for example, based on express.js, as shown here , whether you are going to rewrite some of the monolithic application, or decide to transfer several existing microservices to cote, the job will be to create several instances cote components and use them according to your needs. Among these components, for example - Responder , Requester , Publisher , Subscriber , you will learn more about them below. These components are designed to interact with each other, allowing you to implement the most common application development scenarios.

A simple application, or a very small microservice, may well be built so that one Node.js process will have to implement one cote component. However, more complex projects require closer communication and joint work of many components and microservices. The cote library also supports such scripts.

Let's start with the client, whose file we call conversion-client.js . Its role is to request a currency conversion operation and, upon receiving a response, perform some actions.

Implementation of the request and response mechanism

The most common scenario of interaction between components, implemented in many applications, is a request-response cycle. Usually, a certain microservice requests the execution of a task from another microservice, or executes a request to it and receives a response from it. We implement a similar interaction scheme using cote.

To begin with, in the file conversion-client.js , we will connect cote.

 const cote = require('cote'); 

Nothing special happens here - the usual library call.

▍Creating Requester Component

Let's start with the Requester component, which will request the execution of a currency conversion operation. Both Requester and other components are constructor functions in the cote module, so they are created using the new keyword.

 const requester = new cote.Requester({ name: 'currency conversion requester' }); 

As the first argument, the constructors of the cote components take an object in which, at a minimum, the name property must be contained, which is the name of the component that serves to identify it. The name is mainly used as an identifier for the purpose of monitoring components; it turns out to be very useful when reading logs, since, by default, each component logs the names of other components it finds.

The components created by the Requester constructor are designed to send requests, it is assumed that they will be used together with the objects that the Responder constructor creates that respond to requests. If no such components are found, the Requester component will accumulate requests in the queue until the Responder appears, to which you can send these requests. If there are several Responder components, Requester will access them using the round-robin algorithm, balancing the load on them.

Let's create a query of type convert , which is designed to convert a certain amount, expressed in US dollars, into euros.

 const request = { type: 'convert', from: 'usd', to: 'eur', amount: 100 }; requester.send(request, (res) => {   console.log(res); }); 

Recall that so far we have worked in the file conversion-client.js . Here, just in case, its full code.

 const cote = require('cote'); const requester = new cote.Requester({ name: 'currency conversion requester' }); const request = { type: 'convert', from: 'usd', to: 'eur', amount: 100 }; requester.send(request, (res) => {   console.log(res); }); 

To execute this file, use the following command:

 node conversion-client.js 

Now, the query does not lead to anything useful, there is not even a log in the console, because in our system there is still no component capable of responding to the request and returning something in response.

Let this node.js process be executed, but for now let's deal with the Responder component, which can respond to requests.

▍ Creating the Responder component

First, like last time, we connect the cote and create an instance of the Responder object using the new keyword.

 const cote = require('cote'); const responder = new cote.Responder({ name: 'currency conversion responder' }); 

This part of our system will be represented by the file conversion-service.js . Each Responder , among other things, is an instance of an EventEmitter2 object. The answer to a certain request, say, of the type convert , is the same as listening for the convert event and processing it with a function that takes two parameters — the request and the callback function. The request parameter contains information about a single request; in general, it is the same request object that was sent by the Requester component discussed above. The second parameter is a callback function that is called with the transfer of what should be sent in response.

Here is how a simple implementation of the above mechanism looks like.

 const rates = { usd_eur: 0.91, eur_usd: 1.10 }; responder.on('convert', (req, cb) => { //  ,           cb(req.amount * rates[`${req.from}_${req.to}`]); }); 

Here is the complete code for the file conversion-service.js .

 const cote = require('cote'); const responder = new cote.Responder({ name: 'currency conversion responder' }); const rates = { usd_eur: 0.91, eur_usd: 1.10 }; responder.on('convert', (req, cb) => {   cb(req.amount * rates[`${req.from}_${req.to}`]); }); 

Save the file, and, in the new terminal, execute it with the node:

 node conversion-service.js 

Immediately after starting the service, you will see how the first request received from conversion-client.js will be executed. Information about this will go to the log. As a matter of fact, knowing everything that we just talked about, you can already start creating your own microservices.

Note that we did not configure IP addresses, ports, host names, or anything else. And also - congratulations! You have just created your first set of microservices.

Now, in different terminals, you can run multiple copies of each service and make sure that all this works fine. Stop several conversion services, restart them, and you will see that the system we have created meets the requirements for modern microservices. If you want to scale a project, deploy it on several servers or in several data centers, you can either use your own solution or use Docker features that allow you to solve many problems of infrastructure management.

Let us further develop our currency converter.

Tracking system changes using the publisher-subscriber mechanism

One of the advantages of systems built on the basis of microservices is that it is very easy to implement mechanisms in them that used to require serious investments in infrastructure. Among the tasks solved by such mechanisms are the management of updates and tracking changes in the system. Previously, this required, at a minimum, a branching queue infrastructure. Such things are not easy to scale, they are difficult to manage, it was one of the limiting factors for the development of such systems.

The cote library solves such problems in a completely natural and intuitive way.

Suppose we need an arbitration service that sets currency rates, and, if there are changes, notifies all instances of the converter services that they need to use the new rate values.

The most important thing here is the notification of all instances of the converter services. In a highly available application based on microservices, we can expect the presence of several identical services between which the load is distributed. When currency rates are updated, all these services should be informed of the changes. If there was only one converter service in the application, it would be easy to implement this with the help of the request-response mechanism. But since we are striving for a scalable architecture, we don’t want to limit ourselves to the number of simultaneously running services, we need a mechanism that notifies each of these services, moreover, it is necessary that they all receive such notifications at the same time. In cote, this is achievable through the mechanism of publishers and subscribers.

Of course, the arbitration service itself is not independent, it must be managed through some kind of API, so that it can receive information about new courses through special requests. As a result, for example, the administrator will be able to enter information about new courses into the system, which, as a result, will reach the converter services. In order to achieve this, the arbitration service must include two components. One of them is the Responder component, which is responsible for implementing the course update API from an external source, and the second is the Publisher component, which publishes updates notifying the converter services. In addition to this, the converter services themselves must include the Subscriber component, which will allow them to sign up for course updates. Let's see how to do it all.

▍Creation of arbitration service

Develop arbitration service. First, let's connect the cote and create a Responder object for the API. And now - a small, but very important detail regarding the creation of microservices using cote. Since Requester objects are configured automatically, each of them connects to each Responder object that it can detect, regardless of the types of requests that this Responder can respond to. This means that each Responder must respond to exactly the same set of requests, since Requester objects will distribute the load among all Responder objects to which they are connected, regardless of their capabilities, that is, without paying attention to whether they can handle requests of a particular type.

In this case, we are going to create a Responder component for a set of requests that is different from the one that microservices existing in the system can handle. This means that we need the new Responder different from the usual services that serve requests of type convert . In cote, you can do this by specifying the component key. Keys - the easiest way to control the interaction of services. Here's how to create a Responder component for arbitration service.

 const cote = require('cote'); const responder = new cote.Responder({ name: 'arbitration API', key: 'arbitration' }); 

Now we need a mechanism for tracking currency rates in the system. Let's say they are stored in a local variable in the scope of the module. This may well be the database to which the service is accessed, but in order not to complicate the example, let it be a local variable.

 const rates = {}; 

Now the Responder component will have to respond to requests such as update rate , allowing the administrator to update currency rates using the service application. At the moment, the integration of our system with an auxiliary application, something like an administrator's office, is not important, however, here is an example of how such an application can interact with Responder components implemented using cote running on the server. The arbitration service must have a Responder component that can receive information about new exchange rates.

 responder.on('update rate', (req, cb) => {   rates[req.currencies] = req.rate; // { currencies: 'usd_eur', rate: 0.91 }   cb('OK!'); }); 

As an exercise, you can create the Requester component so that, for example, it periodically makes requests like update rate , changing courses. Name the file with this mechanism arbitration-admin.js and inject something like a timer based on setInterval , which gives out different currency exchange rates for the arbitration service each time.

▍Create a Publisher component

Now we have a mechanism for updating currency rates, but the rest of the system knows nothing about it. In particular, we are talking about currency conversion services. In order to inform them of course changes, it is necessary, in the arbitration service, to use the Publisher component.

 const publisher = new cote.Publisher({ name: 'arbitration publisher' }); 

Now, when updating courses, you need to take advantage of the features of this component. As a result, an update rate request handler will need to be edited as shown below.

 responder.on('update rate', (req, cb) => {   rates[req.currencies] = req.rate;   cb('OK!');   publisher.publish('update rate', req); }); 

Now the work on the arbitration service is completed, below, as usual, its full code is shown, which should be in the arbitration-service.js file.

 const cote = require('cote'); const responder = new cote.Responder({ name: 'arbitration API', key: 'arbitration' }); const publisher = new cote.Publisher({ name: 'arbitration publisher' }); const rates = {}; responder.on('update rate', (req, cb) => {   rates[req.currencies] = req.rate;   cb('OK!');   publisher.publish('update rate', req); }); 

As long as there are no subscribers to the course update events in the system, the converter services do not recognize that the currency rates have changed. In order for the publisher-subscriber mechanism to work, you need to return to the conversion-service.js file that you already know and add the Subscriber component to it.

▍Create a Subscriber component

The order of working with components of the Subscriber type is no different from other components. Create, in the file conversion-service.js , a new instance of the corresponding object.

 const subscriber = new cote.Subscriber({ name: 'arbitration subscriber' }); 

Subscriber objects extend the functionality of EventEmitter2 objects, and although these services can run on computers located on different continents, any new data coming from the publisher will ultimately be perceived by the subscriber as an event that needs to be processed.

Let's add the following code to conversion-service.js , which will allow listening to updates that the arbitration service publishes.

 subscriber.on('update rate', (update) => {   rates[update.currencies] = update.rate; }); 

That's all. After all our microservices are launched, the conversion services will be synchronized with the arbitration service, receiving messages published by it. Conversion requests coming after updating courses will be executed with fresh data in mind.

We have just created three services, the joint work of which allowed us to implement a currency conversion system based on a microservice architecture. Here is a repository on GitHub, in which you can find all three microservices we were involved in here, and a service for automatic updating of currency rates. If you want, clone the repository and experiment with its contents.

What's next?

If the topic of developing microservices using cote is interesting for you, take a look at the library repository on GitHub, join our community on Slack. The project needs active participants, so if you want to make a contribution to turning microservices into a widespread, accessible tool for everyone, let us know.

Here is another repository in which you can find an advanced example representing the implementation of a simple cote based e-commerce application. In this example, in particular, there is the following.

• An administrative application with real-time updates to manage your product catalog and display sales information using the RESTful API (express.js).
  
  
• The page for customer users, through which they can buy goods. It is also updated in real time, using WebSockets (socket.io) technology.
  
  
• Microservice to serve the needs of users performing CRUD operations.
  
• Microservice for working with products, again, implementing CRUD operations.
  
• Microservice, allowing customers to make purchases.
  
• Payment microservice that deals with the processing of financial transactions arising as a result of purchases of goods.
  
• Docker Compose configuration for local system startup.
  
• Docker cloud configuration to run the system in Docker Cloud.
  

If you want to experiment with a cote at Docker or Docker Cloud, here ’s a webinar recording . Here you will find a step-by-step guide, which, from scratch, demonstrates the creation of a working system based on microservices that supports scaling and continuous integration.

Results

In this article we, very briefly, discussed the topic of developing applications based on microservices. This is an introduction to microservice architecture, a general overview of the development methodology. However, although the application we created here consists of only a few lines of code, the same approach can be used on much larger projects. It should be noted that we considered the cote library only in general terms.

I suppose we are on the verge of very interesting times when you can observe a change in approaches to software development. I warmly support the simplification path that the web development industry is following. The cote library is one of the steps along this path, and, in fact, this is just the beginning. I hope this material helped those who thought about microservices, but did not dare to introduce them into their projects.

Dear readers! Do you use microservices in your development?