We create microservice architecture together with Apache Kafka and .NET Core 2.0

Good day! Apache Kafka is a very fast distributed message broker, and today I will tell you how to “prepare” it and implement a simple microservice architecture from console applications using it. So, anyone who wants to get acquainted with Apache Kafka and try it out in business, welcome under the cat.

Overview

Introduction

This material does not in any way pretend to a thorough description of Apache Kafka, nor to the subtle questions of building a microservice architecture. The only thing you need to know is how to build applications on the .NET platform. We will use .Net Core 2.0

So what do we eventually create? An application that tells you what to call your child. For simplicity, it will give random male and female names from a pre-compiled list. The system will consist of two console applications and one library.
')
The idea is to build not a “monolithic”, but a distributed application. Thereby, we will provide ourselves with the groundwork for future scaling and many other advantages described, for example, here.

Here is the structure of our system:

The 3 blue “rectangles” on the sides are console applications. In fact, those two below are microservices, and MainApp is a user application, through which we will request names. NameService will be a universal service that can generate either male or female names.
The orange “rectangle” in the middle is the message broker Apache Kafka. A message broker is what binds all parts of our system together. In our case, we will use Apache Kafka, but with the same success could use RabbitMQ, ActiveMQ, or some other.

And this is how MainApp interacts with Apache Kafka:

It works according to the following scheme:

The user requests some data (in our case, a male or female name).
MainApp sends a message (in the diagram this is “Commands”) to Apache Kafka, which is automatically received by all the services we need.
These services respond by sending another message (data in the diagram) to Apache Kafka. MainApp receives this message from Apache Kafka (this is “Data” in the diagram), which contains the information we need, and provides it to the user.

Each service interacts with Apache Kafka in a similar “two-way” scheme.

Notice that MainApp knows nothing about NameService, and vice versa. All interaction takes place through Apache Kafka. But both MainApp and NameService must use the same “communication channels”. In practice, this means that, for example, the name of the topic, to which MainApp sends messages, must completely coincide with the name of the topic, from which it "listens" to NameService messages.

As you can see, the work of Apache Kafka in our example is to transfer messages between different elements of the system. It is this, exceptionally fast and reliable, that she does. Of course, she has other opportunities, you can read about them on the official website here.

What is Apache Kafka

Apache Kafka is a distributed message broker. In essence, this is a system that can transmit your messages very quickly and efficiently. Any type of data can be used as messages, since for Kafka it is just a sequence of bytes. Apache Kafka can work both on one machine and on several machines, which together form a cluster and increase the efficiency of the entire system. In our case, we will run Apache Kafka locally, and use the Confluent library to interact with it.

It is important to understand how Apache Kafka works. We can write messages to it, and we can read from it. All messages in Kafka belong to one topic or another (topic). A topic is like a title, and it must be defined for each message that we want to convey in Apache Kafka. Similarly, if we are going to read messages from Kafka, we must indicate with which topic these messages will be.

The topic is divided into sections, and we indicate their number, as a rule, on our own. The number of sections in the topic is of great importance for performance, you can read about it here

Practical part

Download and run Apache Kafka 0.11

At the moment, the latest version is version 0.11. Download the archive from the official site (https://kafka.apache.org/downloads) and unpack it into any folder. Next from the console you need to run 2 files (zookeeper-server.start and kafka-server-start) as follows.

We open the first console (if we unpacked to the C drive, we open it on behalf of the Administrator, just in case), go to where we unpacked our archive from Kafka, and enter the command:
bin \ windows \ zookeeper-server-start.bat config \ zookeeper.properties

After that, if everything is good and this process did not stop shortly after the start, open the second console in the same way, and start the Apache Kafka itself
bin \ windows \ kafka-server-start.bat config \ server.properties

We have just launched Zookeeper and Apache Kafka with the default settings specified in zookeeper.properties and server.properties respectively. Zookeeper is a necessary element, without it Apache Kafka does not work.

Complete information on the launch and configuration of Kafka can be viewed on the official website.

Begin to code

So, Kafka is running, now let's create our “distributed“ application. It will consist of 2 console applications and one library. As a result, we will get a solution from 3 projects, which will look like this:

Our library is a “wrapper” around the Confluent.Kafka library, we need it to interact with Apache Kafka. In addition, it will be used by each of our console applications.

The library is designed for the target .NET Core 2.0 platform (Although, it could have been created with the same success for the .NET Standard platform). Its code is presented below. Please note that it is necessary to download the nuget package Confluent.Kafka.

MessageBus.cs

using System; using System.Collections.Generic; using System.Text; using System.Threading; using Confluent.Kafka; using Confluent.Kafka.Serialization; namespace MessageBroker.Kafka.Lib { public sealed class MessageBus : IDisposable { private readonly Producer<Null, string> _producer; private Consumer<Null, string> _consumer; private readonly IDictionary<string, object> _producerConfig; private readonly IDictionary<string, object> _consumerConfig; public MessageBus() : this("localhost") { } public MessageBus(string host) { _producerConfig = new Dictionary<string, object> { { "bootstrap.servers", host } }; _consumerConfig = new Dictionary<string, object> { { "group.id", "custom-group"}, { "bootstrap.servers", host } }; _producer = new Producer<Null, string>(_producerConfig, null, new StringSerializer(Encoding.UTF8)); } public void SendMessage(string topic, string message) { _producer.ProduceAsync(topic, null, message); } public void SubscribeOnTopic<T>(string topic, Action<T> action, CancellationToken cancellationToken) where T: class { var msgBus = new MessageBus(); using (msgBus._consumer = new Consumer<Null, string>(_consumerConfig, null, new StringDeserializer(Encoding.UTF8))) { msgBus._consumer.Assign(new List<TopicPartitionOffset> { new TopicPartitionOffset(topic, 0, -1) }); while (true) { if (cancellationToken.IsCancellationRequested) break; Message<Null, string> msg; if (msgBus._consumer.Consume(out msg, TimeSpan.FromMilliseconds(10))) { action(msg.Value as T); } } } } public void Dispose() { _producer?.Dispose(); _consumer?.Dispose(); } } }

Some explanations to the library code

The wrapper is created in order to simplify the entire interaction with Apache Kafka and concentrate on the moments in the interaction of the system elements with each other. There are two methods in the library: SendMessage () and SubscribeOnTopic, as part of the tutorial, there is no need anymore. Back in SubscribeOnTopic, we subscribe to a topic and continuously “listen” to messages, so in order to subscribe to several topics, it is better to run them in separate threads, which we will do next when using this library using Task.Run () constructs.

Next, we will build the “main” application MainApp, and then our “microservice” NameService, which we will launch later in duplicate. Each of them will be responsible for the generation of either male or female names.

The code for a simple console application, MainApp, for the target .NET Core 2.0 platform is shown below. Please note that it is necessary to add a link to the library that we have just built and which is in the MessageBroker.Kafka.Lib namespace.

MainApp.cs

 using System; using System.Threading; using MessageBroker.Kafka.Lib; using System.Threading.Tasks; namespace MainApp { class Program { private static readonly string bTopicNameCmd= "b_name_command"; private static readonly string gTopicNameCmd = "g_name_command"; private static readonly string bMessageReq = "get_boy_name"; private static readonly string gMessageReq= "get_girl_name"; private static readonly string bTopicNameResp = "b_name_response"; private static readonly string gTopicNameResp= "g_name_response"; private static readonly string userHelpMsg = "MainApp: Enter 'b' for a boy or 'g' for a girl, 'q' to exit"; static void Main(string[] args) { using (var msgBus = new MessageBus()) { Task.Run(() => msgBus.SubscribeOnTopic<string>(bTopicNameResp, msg => GetBoyNameHandler(msg), CancellationToken.None)); Task.Run(() => msgBus.SubscribeOnTopic<string>(gTopicNameResp, msg => GetGirlNameHandler(msg), CancellationToken.None)); string userInput; do { Console.WriteLine(userHelpMsg); userInput = Console.ReadLine(); switch (userInput) { case "b": msgBus.SendMessage(topic: bTopicNameCmd, message: bMessageReq); break; case "g": msgBus.SendMessage(topic: gTopicNameCmd, message: gMessageReq); break; case "q": break; default: Console.WriteLine($"Unknown command. {userHelpMsg}"); break; } } while (userInput != "q"); } } public static void GetBoyNameHandler(string msg) { Console.WriteLine($"Boy name {msg} is recommended"); } public static void GetGirlNameHandler(string msg) { Console.WriteLine($"Girl name {msg} is recommended"); } } }

A bit of explanation to the MainApp code

Have you seen many string readonly variables at the beginning? These are the names of all the topics and messages that we will send to them. In other words, the title and text messages. All the services with which our MainApp will interact should know about them, because the names of the topics must match. For example, bTopicNameCmd is the name of the topic for the service team that we need to get a male name (gTopicNameCmd is the same). The service must be subscribed to the eponymous topic in order to receive messages from it and then do something.

Likewise, our MainApp is subscribed to topics to which our NameService services transmit useful information. For example, the variable bTopicNameResp is the name of the topic, which is provided for ready-made male names that NameService generated. The service sends the name to this topic, and MainApp receives them from there.

The following is the code of the microservice NameService. Please note, here, too, you need to add a link to the library we have already created in the namespace MessageBroker.Kafka.Lib

NameService.cs

 using System; using System.Threading; using System.Threading.Tasks; using MessageBroker.Kafka.Lib; namespace NameService { class Program { private static MessageBus msgBus; private static readonly string userHelpMsg = "NameService.\nEnter 'b' or 'g' to process boy or girl names respectively"; private static readonly string bTopicNameCmd = "b_name_command"; private static readonly string gTopicNameCmd = "g_name_command"; private static readonly string bTopicNameResp = "b_name_response"; private static readonly string gTopicNameResp = "g_name_response"; private static readonly string[] _boyNames = { "Arsenii", "Igor", "Kostya", "Ivan", "Dmitrii", }; private static readonly string[] _girlNames = { "Nastya", "Lena", "Ksusha", "Katya", "Olga" }; static void Main(string[] args) { bool canceled = false; Console.CancelKeyPress += (_, e) => { e.Cancel = true; canceled = true; }; using (msgBus = new MessageBus()) { Console.WriteLine(userHelpMsg); HandleUserInput(Console.ReadLine()); while (!canceled) { } } } private static void HandleUserInput(string userInput) { switch (userInput) { case "b": Task.Run(() => msgBus.SubscribeOnTopic<string>(bTopicNameCmd, (msg) => BoyNameCommandListener(msg), CancellationToken.None)); Console.WriteLine("Processing boy names"); break; case "g": Task gTask = Task.Run(() => msgBus.SubscribeOnTopic<string>(gTopicNameCmd, (msg) => GirlNameCommandListener(msg), CancellationToken.None)); Console.WriteLine("Processing girl names"); break; default: Console.WriteLine($"Unknown command. {userHelpMsg}"); HandleUserInput(Console.ReadLine()); break; } } private static void BoyNameCommandListener(string msg) { var r = new Random().Next(0, 5); var randName = _boyNames[r]; msgBus.SendMessage(bTopicNameResp, randName); Console.WriteLine($"Sending {randName}"); } private static void GirlNameCommandListener(string msg) { var r = new Random().Next(0, 5); var randName = _girlNames[r]; msgBus.SendMessage(gTopicNameResp, randName); Console.WriteLine($"Sending {randName}"); } } }

A few explanations to the code NameService

The service works as follows:

First we decide whether the service will generate the male or female names (i.e., just choose a random name from the prepared list, in our case)
Subscribe to the appropriate topic

In the body of the event handler method, we send a message with a prepared name to the topic to which MainApp is already subscribed. And this event occurs every time MainApp sends a message stating that you need to get some name.

Run

At this stage, you, in theory, should already have a ready-made solution with all the necessary code. Then you can proceed as follows: configure the solution so that 2 applications start at once (MainApp and NameService), and start them ( Just check that you already have Apache Kafka running ). In NameService, enter 'b', or 'g', to configure the service to generate male or female names, after which, in the same way, enter 'b' or 'g' into MainApp, but to get these names. Then in MainApp you should get some name.

At this stage, we get the names of only one gender. Suppose only male. Now we want to get the names of the female. Go to the folder where our NameService project is going, and start another service in the console using the " dotnet NameService.dll " command .
We enter the 'g' command in it, and now, when we query the female name in MainApp, we get it.

By the way, this way you can run as many NameService entities as you like, and this is one of the advantages of the microservice architecture. For example, if one of the services “falls”, the whole system will not collapse, because we have other services that do the exact same job.

One thing: if we, for example, run 5 NameService items, then 5 names will come to MainApp, not just one. This is due to the Apache Kafka settings that are specified in the server.properties file. As part of the tutorial, I deliberately do not touch this, so as not to complicate the material.

Conclusion

In this article, I wanted to describe the principle of building a microservice architecture as simply and easily as possible and introduce the reader to the distributed message broker Apache Kafka using a live example. I hope it worked out, and thanks for your attention :)

Links to materials used in the article

Source: https://habr.com/ru/post/336058/

All Articles