As practice shows, a huge part of the problems arises not because of the decisions themselves, but because of how communication takes place between the components of the system. If there is a mess in the communication between the components of the system, then how not to try to write separate components well, the system as a whole will fail.

Caution! Inside the bike.

Problematics or problem statement

Some time ago it happened to work on a project for a company that brings to the masses such charms as CRM, ERM systems and derivatives. Moreover, the company produced a rather comprehensive product from software for cash registers to a call-center with the possibility of leasing operators in the amount of up to 200 souls.

I myself worked on the front-end application for the call-center.

It is easy to imagine that it is in the operator’s application that information from all system components flows. And if we take into account the fact that the operator and the administrator are not the only operator, but also the manager and the administrator, then you can imagine how many communications and information the application should “digest” and connect with each other.

When the project was already launched and even worked quite stably, the problem of system transparency arose to its full height.

Here is the point. There are many components and they all work with their data sources. But almost all of these components in their time were written as independent products. That is, not as an element of the overall system, but as separate solutions for sale. As a result, there is no single (system) API and no common standards of communication between them.

I will explain. Some component sends JSON, “someone” sends lines with key: value inside, “someone” sends binary at all and do what you want with it. But, and the final application for the call-center should have been this all to receive and somehow handle. And most importantly, there was no link in the system that could recognize that the format / structure of the data had changed. If a component sent JSON yesterday, and today decided to send a binary, no one will see it. Only the final application will start to fail as expected.

It soon became clear (for others, not for me, since I spoke about the problem at the design stage) that the lack of a “single language of communication” between the components leads to serious problems.

The easiest case is when a client has asked to change some dataset. The task is written off by a young man that “holds” a component for working with databases of goods / services for example. He does his work, the new dataset introduces and he, an asshole, everything works. But, the day after the update ... oh ... the application in the call-center suddenly starts to work not as expected of it.

You probably already guessed. Our hero has changed not only the dataset, but also the data structure that its component sends to the system. As a result, the application for the call-center is simply not able to work more with this component, and there other dependencies are flying along the chain.

We began to think about what we actually want to get at the exit. As a result, they formulated the following requirements for a potential solution:

First and foremost: any change to the data structure should immediately be displayed in the system. If someone, somewhere has made changes and these changes are incompatible with what the system expects - the error should occur at the stage of the component tests that was changed.

The second . Data types should be checked not only at compile time, but also run-time.

Third . Since a large number of people work on components with a completely different skill level, the description “language” should be simpler.

Fourth . Whatever the solution, it should be as comfortable as possible to work with it. If possible, the IDE should highlight as much as possible.

The first thought was to implement protobuf. Simple, readable and easy. Strong data typing. It seems to be what the doctor ordered. But, alas, not all protobuf syntax seemed simple. In addition, even the compiled protocol required an additional library, but Javascript was not supported by the authors of protobuf and was the result of the work of the community. In general, they refused.

Then there was an idea to describe the protocol in JSON. How much easier?

Well, then I quit. And on this this post could have been completed, since after my departure, no one further dealt with the problem of particularly close engagement.

However, taking into account a couple of personal projects, where the question of communication between the components again rose to its full height, I decided to start implementing the idea myself. What are we talking about and below.

So, I present to your attention the ceres project, which includes:

• protocol generator
• provider
• customer
• implementation of transports

Protocol

The task was to do so:

• it was easy to set the message structure in the system.
• it was easy to determine the data type of all message fields.
• it was possible to define auxiliary entities and refer to them.
• and of course, that all this is highlighted IDE

I think that in a completely natural way, not pure Javascript was chosen as the language into which the protocol was converted, but Typescript. So all that the protocol generator does is turn JSON into Typescript.

To describe the messages available in the system, you just need to know what JSON is. With which, I am sure, no one has any problems.

Instead of Hello World, I offer an equally hackneyed example - chat.

{ "Events": { "NewMessage": { "message": "ChatMessage" }, "UsersListUpdated": { "users": "Array<User>" } }, "Requests": { "GetUsers": {}, "AddUser": { "user": "User" } }, "Responses": { "UsersList": { "users": "Array<User>" }, "AddUserResult": { "error?": "asciiString" } }, "ChatMessage": { "nickname": "asciiString", "message": "utf8String", "created": "datetime" }, "User": { "nickname": "asciiString" }, "version": "0.0.1" } 

Everything is so ugly. We have a couple of NewMessage and UsersListUpdated events; as well as a couple of UsersList and AddUserResult requests. There are two other entities: ChatMessage and User.

As you can see the description is quite transparent and understandable. A little about the rules.

• The object in JSON will become a class in the generated protocol.
• The value of the property is the definition of a data type or a reference to a class (entity).
• Nested objects from the point of view of the generated protocol will become “nested” classes, that is, nested objects will inherit all the properties of their parents.

Now you just need to generate a protocol to start using it.

 npm install ceres.protocol -g ceres.protocol -s chat.protocol.json -o chat.protocol.ts -r 

As a result, we get the protocol generated on Typescript. We connect and use:

So, the protocol already gives something to the developer:

• IDE highlights what we have in the protocol. Also IDE highlights all expected properties.
• Typescript, which will certainly tell us if something is wrong with the data types. Of course, this is done at the development stage, but the protocol itself at run-time will check the data types and throw an exception if a violation is detected
• In general, you can forget about validation. The protocol will do all the necessary checks.
• The generated protocol does not require any additional libraries. Everything he needs for work, he already contains. And it is very convenient.

Yes, the size of the generated protocol can surprise you, to put it mildly. But, do not forget about the minification, to which the generated protocol file lends itself well.

Now we can "pack" the message and send

 import * as Protocol from '../../protocol/protocol.chat'; const message: Protocol.ChatMessage = new Protocol.ChatMessage({ nickname: 'noname', message: 'Hello World!', created: new Date() }); const packet: Uint8Array = message.stringify(); // Send packet somewhere 

It is important to make a reservation, the packet will be an array of bytes, which is very good and correct in terms of traffic load, since sending the same JSON is "worth", of course, more expensive. However, the protocol has one chip - in debug mode, it will generate readable JSON, so that the developer can “look” at the traffic and see what happens.

This is done directly in run-time.

 import * as Protocol from '../../protocol/protocol.chat'; const message: Protocol.ChatMessage = new Protocol.ChatMessage({ nickname: 'noname', message: 'Hello World!', created: new Date() }); // Switch to debug mode Protocol.Protocol.state.debug(true); // Now packet will be present as JSON string const packet: string = message.stringify(); // Send packet somewhere 

On the server (or any other recipient), we can easily unpack the message:

 import * as Protocol from '../../protocol/protocol.chat'; const smth = Protocol.parse(packet); if (smth instanceof Error) { // Oops. Something wrong with this packet. } if (Protocol.ChatMessage.instanceOf(smth) === true) { // This is chat message } 

The protocol supports all major data types:

Within the protocol, these data types are called primitive. However, another feature of the protocol is that it allows you to add your own data types (which are called "additional data types").

For example, you probably already noticed that ChatMessage has a created field with a datetime data type. At the application level, this type corresponds to Date , and inside the protocol is stored (and forwarded) as uint32 .

Adding your type to the protocol is quite simple. For example, if we want to have an email data type, say for the following message in the protocol:

 { "User": { "nickname": "asciiString", "address": "email" }, "version": "0.0.1" } 

All you need is to write a definition for the email type.

 export const AdvancedTypes: { [key:string]: any} = { email: { // Binary type or primitive type binaryType : 'asciiString', // Initialization value. This value is used as default value init : '""', // Parse value. We should not do any extra decode operations with it parse : (value: string) => { return value; }, // Also we should not do any encoding operations with it serialize : (value: string) => { return value; }, // Typescript type tsType : 'string', // Validation function to valid value validate : (value: string) => { if (typeof value !== 'string'){ return false; } if (value.trim() === '') { // Initialization value is "''", so we allow use empty string. return true; } const validationRegExp = /^(([^<>()\[\]\\.,;:\s@"]+(\.[^<>()\[\]\\.,;:\s@"]+)*)|(".+"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/gi; return validationRegExp.test(value); }, } }; 

That's all. By generating a protocol, we will get support for a new email data type. If we try to create an entity with the wrong address, we will get an error

 const user: Protocol.User = new Protocol.User({ nickname: 'Brad', email: 'not_valid_email' }); console.log(user); 

Oh...

 Error: Cannot create class of "User" due error(s): - Property "email" has wrong value; validation was failed with value "not_valid_email". 

So, the protocol simply does not allow "bad" data into the system.

Note that when defining a new data type, we specified a couple of key properties:

• binaryType is a reference to the primitive data type to be used for storing, encoding / decoding data. In this case, we indicate that the address is an ascii string.
• tsType is a reference to the Javascript type, that is, how the data type should be represented in the Javascript environment. In this case, we are talking about string.
• It is also worth noting that we only need to define the new data type at the time of generating the protocol. At the output, we get a generated protocol that already contains a new data type.

Detailed information on all protocol capabilities you can see here ceres.protocol .

Provider and client

By and large, the protocol itself can be used to organize communication. However, if we are talking about the browser and nodejs, then the provider and the client are available.

Customer

Creature

To create a client, you need the client itself and the transport.

Installation

 # Install consumer (client) npm install ceres.consumer --save # Install transport npm install ceres.consumer.browser.ws --save 

Creature

 import Transport, { ConnectionParameters } from 'ceres.consumer.browser.ws'; import Consumer from 'ceres.consumer'; // Create transport const transport:Transport = new Transport(new ConnectionParameters({ host: 'http://localhost', port: 3005, wsHost: 'ws://localhost', wsPort: 3005, })); // Create consumer const consumer: Consumer = new Consumer(transport); 

The client, as well as the provider, are designed specifically for the protocol. That is, they will work only with the protocol (ceres.protocol).

Developments

After the client is created, the developer can subscribe to events.

 import * as Protocol from '../../protocol/protocol.chat'; import Transport, { ConnectionParameters } from 'ceres.consumer.browser.ws'; import Consumer from 'ceres.consumer'; // Create transport const transport:Transport = new Transport(new ConnectionParameters({ host: 'http://localhost', port: 3005, wsHost: 'ws://localhost', wsPort: 3005, })); // Create consumer const consumer: Consumer = new Consumer(transport); // Subscribe to event consumer.subscribe(Protocol.Events.NewMessage, (message: Protocol.Events.NewMessage) => { console.log(`New message came: ${message.message}`); }).then(() => { console.log('Subscription to "NewMessage" is done'); }).catch((error: Error) => { console.log(`Fail to subscribe to "NewMessage" due error: ${error.message}`); }); 

Note that the client will call an event handler, only if these messages are completely correct. In other words, our application is insured against incorrect data and the NewMessage event handler will always be called with an instance of Protocol.Events.NewMessage as an argument.

Naturally, the client can generate events.

 consumer.emit(new Protocol.Events.NewMessage({ message: 'This is new message' })).then(() => { console.log(`New message was sent`); }).catch((error: Error) => { console.log(`Fail to send message due error: ${error.message}`); }); 

Notice, we do not specify event names anywhere, we simply use either a reference to a class from the protocol, or we transmit its instance.

We can also send a message to a limited group of recipients, specifying a simple object of the type { [key: string]: string } as the second argument. Within ceres, this object is called query .

 consumer.emit( new Protocol.Events.NewMessage({ message: 'This is new message' }), { location: "UK" } ).then(() => { console.log(`New message was sent`); }).catch((error: Error) => { console.log(`Fail to send message due error: ${error.message}`); }); 

Thus, by additionally specifying { location: "UK" } , we can be sure that only those customers who have defined their position as UK will receive this message.

To associate the client with a specific query , you just need to call the ref method:

 consumer.ref({ id: '12345678', location: 'UK' }).then(() => { console.log(`Client successfully bound with query`); }); 

After we have connected the client with the query , he has the opportunity to receive "personal" or "group" messages.

Requests

We can also make requests

 consumer.request( new Protocol.Requests.GetUsers(), // Request Protocol.Responses.UsersList // Expected response ).then((response: Protocol.Responses.UsersList) => { console.log(`Available users: ${response.users}`); }).catch((error: Error) => { console.log(`Fail to get users list due error: ${error.message}`); }); 

Here it is worth noting that as the second argument we specify the expected result ( Protocol.Responses.UsersList ), which means our request will be successfully completed only if the answer is an instance of UsersList , in all other cases we will “fall” into catch Again, this insures us against the processing of incorrect data.

The client himself can act and those who can process requests. To do this, you only need to "identify" yourself as the "responsible" for the request.

 function processRequestGetUsers(request: Protocol.Requests.GetUsers, callback: (error: Error | null, results : any ) => any) { // Get user list somehow const users: Protocol.User[] = []; // Prepare response const response = new Protocol.Responses.UsersList({ users: users }); // Send response callback(null, response); // Or send error // callback(new Error(`Something is wrong`)) }; consumer.listenRequest(Protocol.Requests.GetUsers, processRequestGetUsers, { location: "UK" }).then(() => { console.log(`Consumer starts listen request "GetUsers"`); }); 

Note, optionally, as the third argument, we can specify a query object that can be used to identify the client. Thus, if someone sends a query with a query , say, { location: "RU" } , then our client will not receive such a query, since its query { location: "UK" } .

The query can include an unlimited number of properties. For example, you can specify the following

 { location: "UK", type: "managers" } 

Then, in addition to the complete match query, we also successfully process the following queries:

 { location: "UK" } 

or

 { type: "managers" } 

Provider

Creature

To create a provider (as well as create a client), the provider and the transport are needed.

Installation

 # Install provider npm install ceres.provider --save # Install transport npm install ceres.provider.node.ws --save 

Creature

 import Transport, { ConnectionParameters } from 'ceres.provider.node.ws'; import Provider from 'ceres.provider'; // Create transport const transport:Transport = new Transport(new ConnectionParameters({ port: 3005 })); // Create provider const provider: Provider = new Provider(transport); 

From the moment the provider is created, it can accept connections from clients.

Developments

As well as the client, the provider can "listen" to messages and generate them.

Listen

 // Subscribe to event provider.subscribe(Protocol.Events.NewMessage, (message: Protocol.Events.NewMessage) => { console.log(`New message came: ${message.message}`); }); 

We generate

 provider.emit(new Protocol.Events.NewMessage({ message: 'This message from provider' })); 

Requests

Naturally, the provider can (and should) "listen" requests

 function processRequestGetUsers(request: Protocol.Requests.GetUsers, clientID: string, callback: (error: Error | null, results : any ) => any) { console.log(`Request from client ${clientId} was gotten.`); // Get user list somehow const users: Protocol.User[] = []; // Prepare response const response = new Protocol.Responses.UsersList({ users: users }); // Send response callback(null, response); // Or send error // callback(new Error(`Something is wrong`)) }; provider.listenRequest(Protocol.Requests.GetUsers, processRequestGetUsers).then(() => { console.log(`Consumer starts listen request "GetUsers"`); }); 

Here there is only one difference from the client, the provider in addition to the request body will receive a unique clientId , which is automatically assigned to all connected clients.

Example

In fact, I really do not want to bore you with excerpts from the documentation, I am sure it will be easier and more interesting for you to just see a short piece of code.

Sample chat you can easily install by downloading the source and making a couple of simple steps

Install and run the client

 cd chat/client npm install npm start 

The client will be available at http: // localhost: 3000 . Open at once a couple of tabs with the client to see the "communication".

Installation and start of the provider (server)

 cd chat/server npm install ts-node ./server.ts 

The ts-node package is surely familiar to you, but if not, then it allows you to run TS files. If you don’t want to install, just compile the server and then run the JS file.

 cd chat/server npm run build node ./build/server/server.js 

Shaw? Again?!

Anticipating questions about why the hell to reinvent the next bike, because there are so many solutions already worked out, starting from protobuf and ending with hardcore joynr from BMW, I can only say that it was interesting to me. The whole project was done solely on personal initiative, without any support, in his spare time.

That is why your feedback is of particular value to me. In an attempt to somehow motivate you, I can promise that for every star on the github, I will stroke the hamster (which I don’t like to say the least). For fork, ufff, I'll scratch his tum to him ... brrrr.

Hamster is not mine, son of a hamster .

In addition, in a couple of weeks the project will go to testing to my former colleagues (which I mentioned at the beginning of the post and who were interested in what the alfa version turned out). The goal is to debug and run on several components. I really hope that it will work.

Links and Packages

The project lodges on two repositories.

• ceres sources: ceres.provider, ceres.consumer and all transports available today.
• ceres.protocol protocol generator source

NPM the following packages are available

• ceres.protocol protocol generator
• ceres.provider provider
• ceres.consumer client
• ceres.provider.node.longpoll transport for long polling based provider
• ceres.provider.node.ws transport for a provider based on a web socket
• ceres.consumer.browser.longpoll transport for a client based on long polling
• ceres.consumer.browser.ws transport for Web Socket-based client

Good and light.