Geekly Articles each Day
📜 ⬆️ ⬇️

Experience in building an integration platform based on ServiceMix (Camel) and RabbitMQ

As soon as the company has at least two information systems that need to exchange data, the question arises how to organize their interaction. The options are many: file exchange, links between databases, web or rest services, various messaging systems, outdated RPC and CORBA, new-fashioned gRPC, etc. The choice depends on the preferences of the project participants and on the capabilities of the systems (system architecture, platform used, availability of a ready-made API, etc.). Suppose you have chosen some kind of exchange, the systems began to interact, everything is fine. But then a third system emerges, with which you also need to integrate, then a fourth, etc. It is necessary to sit down again and choose the mode of exchange, and it’s not a fact that we can limit ourselves to the technologies already used (somewhere this is dictated by the limitations of the new systems, somewhere the developer insisted on another technology or wanted to try something new). As the number of systems grows, the number and complexity of the interactions between them grow, the number of technologies used grows. As a result, the entire integration architecture of the company begins to resemble the tangled tangle of multi-colored threads, somehow linking the company's systems, which are more and more difficult to unravel when parsing errors and completions. Sooner or later, thoughts of creating a unified integration environment begin to come in, which will transparently and expandably connect all systems together.

In this article I will talk about the experience of using Apache ServiceMix (Camel) and RabbitMQ to build such an integration environment.

Stack technology


RabbitMQ


RabbitMQ is an AMQP based messaging system. I will not describe the system in detail, on Habré there are already a few articles that describe in detail the functionality of the system, I’ll tell you about where and how RabbitMQ is used here.


')
Through RabbitMQ we exchange data with the company's information systems. A set of queues is created for each system. For example, you need to organize a search for customers by name and date of birth in the accounting system. Create a couple of queues. One incoming with respect to the accounting system, we will send requests to it with the name and date of birth of the client. The accounting system listens to the incoming queue and processes incoming requests. The second is outgoing, to which the accounting system will send answers with a list of found customers that match the conditions of the request.

Why it is convenient:


Teaching the system to interact with RabbitMQ is not difficult. RabbitMQ has ready-made clients for various platforms (Java, .Net, Python, etc.). The client is simple and straightforward. Code that reads messages from a queue and / or sends messages to a queue takes up several lines. It is clear that not every system can be made friends with RabbitMQ, for example, in the case of Legacy and box systems, this will be difficult. In this case, the exchange is built using technologies that support these systems, somewhere we call stored procedures in the database, somewhere we use web and rest services, somewhere else. In order to organize such interaction for many other integration tasks, Apache ServiceMix is ​​used.

Apache ServiceMix

ServiceMix - Karaf container with a predefined set of bundles that will be useful for solving various integration tasks.



A little more detail about what is included in the product:

  1. Actually Karaf is the container in which the bundles work and which allows them to be managed: set / delete / stop / start, view logs, see dependencies of components, etc.
  2. A wide range of bundles that perform classic integration functions: validation, various transformations (for example, from JSON to XML, transformations using XSLT, etc.), enrichment, routing, split and join, monitoring, execution of integration processes, etc. .
  3. A wide range of different adapters: file adapters, adapters for web and rest services, JMS, RabbitMQ, Kafka, etc. A complete list of adapters can be found on the Camel website.

Restricting the use of only pre-installed bundles is unlikely to succeed, but for the beginning of the set should be enough. Since we work with the Karaf container, then you can install any necessary bundles, you can do this either by installing features (bundle bundles) or simply by installing individual bundles. Of course, you can write your own bundles, or wrap in third-party java libraries.

Apache camel


A key component of ServiceMix is ​​the Apache Camel framework, which allows you to build integration processes that are called routes in Camel terminology.
Let me show you with an example of what a route is:



This is an example of a simple route that converts incoming multi-format messages to a common output format. Depending on the message format, the route routes it to the appropriate transformation, which converts the message to a general format, and then sends the result to the output.

Camel supports various notations for describing routes, the most common Java DSL and Spring XML. We use Spring XML. This is how the route in the picture would look like in Spring XML notation:

<route id="Normalizer"> <from uri=" endpoint" /> <!--           --> <choice> <when> <xpath>/*[local-name()='Format_1']</xpath> <to uri="xslt:classpath:xslt/Format_1_Transformation.xslt" /> </when> <when> <xpath>/*[local-name()='Format_2']</xpath> <to uri="xslt:classpath:xslt/Format_2_Transformation.xslt" /> </when> <when> <xpath>/*[local-name()='Format_3']</xpath> <to uri="xslt:classpath:xslt/Format_3_Transformation.xslt" /> </when> </choice> <to uri=" endpoint" /> </route>

A very nice addition is that Camel is perfectly integrated with Spring. You can use the familiar Spring XML, in which the bins are defined, and in the same Spring XML you can define Camel routes. In this case, bins can be called from routs, and bins can be accessed from routs. Calling bins from routs is implemented with Camel's intrinsic flexibility; you can pass a message body to a bin method, you can pass headers + body, or you can pass only the value of a specific XML message tag specified via an XPATH expression, and use the result of the method in the choice to determine route. And all this is almost one line.

Here is an example of Camel style elegance:

 <camel:camelContext> <route id="Bean method invocation"> <from uri=" endpoint" /> <when> <simple>${bean:authManager?method=checkToken(${body})}</simple> <to uri=" " /> </when> </route> </camel:camelContext> <bean id="authManager" class="pachage.AuthManager" /> 

 public class AuthManager { public boolean checkToken(@Body Document xml, @XPath("/Root/Token/@Value") String token) { return checkSessionToken(token); } }

Another important concept in Camel is the endpoint (hereinafter referred to as endpoint). Routes can read messages from endpoints, can send messages to endpoints. Endpoint may be, for example, a RabbitMQ queue, a file in a directory, a published rest service, etc. If the route reads endpoint, then when a message arrives at this endpoint, the route starts processing it. This allows you to publish the route, i.e. give the opportunity to access it by external systems. If you have a router that performs some task, for example, checks the correctness of the data filled in the questionnaire, then you can publish it as a web service, or give the opportunity to access it through JMS, or you can do both to external systems could use the capabilities of your route, someone through calls to the web service, and someone through the exchange of messages through the JMS queue.

Endpoints are also used to enable routes to interact with each other. One route can send a message to an endpoint that reads another route, thus passing the message to it for processing. This allows you to create your own palette of routes, implementing various functions that are in demand in different places of your application. For example, logging messages. Instead of performing the same set of logging actions each time, you can simply transfer the processing of messages to a route specially designed for this purpose.

Integration architecture


Our company uses a number of information systems. In order to organize an integration environment through which systems will exchange data, you first need to connect our systems to the integration platform. For this, an adapter is developed for each system on ServiceMix, which will be responsible for exchanging data with the system and converting data formats.


For each system, one data exchange technology is selected between the system and ServiceMix. You can choose a few, but this complicates the implementation, both on the system side and on the ServiceMix side. In general, the use of several different technologies is not justified, but technically it can be fully implemented. We mainly use RabbitMQ to communicate with systems (we create sets of queues through which ServiceMix exchanges messages with integrable systems). But there are other cases in which the set of ready-made adapters, which is part of ServiceMix, helps us a lot. For example, in the case of our accounting system, we use stored procedures in the database. To call stored procedures, use the MyBatis component, which allows you to map messages that run on ServiceMix to the parameters of stored procedures. Here is an example of such a mapping for the user login procedure by its ID:

 <select id="setLogin" statementType="CALLABLE" parameterType="java.util.Map"> {call esb.SetLogin ( @UserId=#{UserId, mode=IN}, @LoginId=#{LoginId, mode=IN} )} </select>

Adapters are also responsible for converting the formats in which the systems interact with the integration platform to the internal format. Someone prefers to exchange in JSON format, someone uses their own XML format, but inside the integration platform the components exchange data in the internal canonical XML format. Now XML is losing popularity due to its heaviness and the emerging alternatives in the form of JSON, Protobuf, etc. But, in my opinion, XML is still convenient for solving integration problems. There are many useful technologies, such as XSLT and XPATH, that greatly simplify life, plus it is quite human-readable.

Routing of messages between components (adapters) is carried out on the basis of routing rules. Routes within a single component of the integration platform interact via internal Camel endpoints (direct, seda). Between themselves, the components are exchanged via RabbitMQ queues. This makes the components of the integration platform independent. The fall of one component does not affect the performance of others. With a temporary increase in load, messages will accumulate in the component queue, without affecting the rest of the exchange.



How is such an architecture better than direct interaction between systems on a point-to-point basis:


fault tolerance




Fault tolerance at the RabbitMQ level is achieved by creating a cluster and synchronizing queues. Each message that enters the queue on one of the RabbitMQ nodes is replicated to other nodes according to the synchronization policy (you can replicate to all the nodes of the cluster, you can replicate to a certain number of nodes, you can not replicate at all). We use a configuration of three nodes with replication of messages to all nodes. This allows the cluster to remain fully functional in the event that two of the three nodes fall. But you need to understand that this is the most costly option in terms of the processing time of each message and the disk space occupied. On the client, all three nodes are registered to RabbitMQ, while the connection is opened to one of the nodes, but if it drops, the client transparently switches to the other nodes and continues to work.

ServiceMix fault tolerance is achieved by using a cluster of two ServiceMix nodes. In this case, part of the bundles work in parallel, if this is permissible. For example, adapters that read one RabbitMQ queue can easily do this in parallel, one message will always receive only one of them, but the messages will be evenly distributed between the two nodes. In the event of a fall of one of the nodes, the second node takes over the entire load. Some bundles are active only on one node, but when it drops, bundles are activated on the second node. For example, if the adapter reads a shared network directory, then simultaneously reading the same file can lead to duplication and collisions. This mode of operation is achieved through the use of shared locks based on Hazelcast. The bundle that first captures the lock goes into active mode and performs its function, the second bundle hangs waiting for the lock to be released.

Hello world


At the end I want to give a simple example of Hello world application on Camel, which we will run on ServiceMix. Our application will simply write the phrase “Hello world” to the ServiceMix log once when launching the bundle (for example, when you deploy the bundle or restart ServiceMix).

  1. We download the ServiceMix distribution, unpack and run servicemix.sh (bat).
  2. Create and configure a new maven project.
    In src / main / resources, you need to create the META-INF directory in which to create the spring subdirectory.
    Since we need to build a bundle - edit pom.xml (add the packaging and build instructions):

     <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>HelloWorld</groupId> <artifactId>HelloWorld</artifactId> <version>1.0.0-SNAPSHOT</version> <name>HelloWorld</name> <description>HelloWorld</description> <packaging>bundle</packaging> <build> <plugins> <plugin> <groupId>org.apache.felix</groupId> <artifactId>maven-bundle-plugin</artifactId> <version>3.0.1</version> <extensions>true</extensions> <configuration> <instructions> <Bundle-SymbolicName>${project.artifactId}</Bundle-SymbolicName> <Import-Package>*</Import-Package> <Export-Package>${project.groupId};version=${project.version}</Export-Package> </instructions> </configuration> </plugin> </plugins> </build> </project>

    There is no need to add any maven dependencies.
  3. Configure Camel context and route.
    In the spring directory, you need to create a file camel-context.xml (the loader automatically searches for a file with a description of the Camel context in META-INF / spring and starts routes). In the file camel-context.xml we place the following contents (comments are given in the text):

     <?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:osgi="http://www.springframework.org/schema/osgi" xmlns:camel="http://camel.apache.org/schema/spring" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd http://www.springframework.org/schema/osgi http://www.springframework.org/schema/osgi/spring-osgi.xsd http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd"> <!--  Camel  --> <camel:camelContext xmlns="http://camel.apache.org/schema/spring" id="HelloWorldContext"> <!--   --> <route id="HelloWorldRoute"> <!--          --> <from uri="timer://startTimer?repeatCount=1" /> <!--      Hello world --> <setBody> <constant>Hello world</constant> </setBody> <!--      --> <log message="${body}"/> </route> </camel:camelContext> </beans>

    In order for our route to perform its task (recorded the text “Hello world” in the log), you need to send a message to it. In our case, this task is solved by the timer <from uri = "timer: // startTimer? RepeatCount = 1" /> , which, following the instructions repeatCount = 1 , sends a message to the route entry once. Since the timer sends an empty message to the input of the route, we need to fill it with something - put the text “Hello world” <setBody> in the message body. At the end of the route, we output the contents of the message body to the log <log message = "$ {body}"> .
  4. Putting our project: mvn package
  5. Deploy the assembled bundle.
    One way to secure a bundle in ServiceMix is ​​to copy the jar file into the deploy directory. Copy the built jar to the ServiceMix / deploy directory.

Now look at the ServiceMix log: ServiceMix / data / log / servicemix.log . In addition to the information that we installed and launched the new bundle, the inscription “Hello world” should appear:

HelloWorldRoute | 43 - org.apache.camel.camel-core - 2.16.3 | Hello world

Try to enter the ssh console and view the Camel context-list contexts and the route-list routes list. In the output of the commands, you will find our HelloWorldContext and HelloWorldRoute.

findings


In conclusion, I want to say that ServiceMix and Camel are excellent products for building integration solutions. The functionality is amazing, for almost any task you can find an elegant and simple solution. It can be seen that the products are very well thought out, the developers really did their best. In our company, ServiceMix has been used for 2 years already, at the moment we have integrated about 16 of our information systems and have developed more than 150 integration services. Some components still had problems, but there are always similar components from which to choose or, as a last resort, develop your component. In general, the products work stably and reliably, personally my impression is the most positive. The undoubted advantage is the openness of the source code and the absence of the need to buy a license. At the same time, the products are absolutely not inferior to commercial counterparts. I also want to note that ServiceMix and Camel can be used not only for solving integration tasks. For example, Karaf is perfect for deploying web applications into it. We use this feature to provide web interfaces to administrators for setting up an integration platform. Camel integrates seamlessly with various business components. For example, we use Drools (business rules engine) to assign incoming transactions to the corresponding portfolio according to certain rules (enrichment of the transaction with a portfolio), which are configured by users from the middle office. A very interesting product is Activiti (BPM engine), which can work in a Karaf container and integrate with Camel. Together, this allows you to create very interesting solutions that combine integration and business functions.

Source: https://habr.com/ru/post/358724/

More articles:


All Articles