Tugging a Camel or integration with Camel. Part 1

The story of one project.

Have you ever dreamed of camels? So I don't either. But when you work with Camel for the third year, not only camels begin to dream.
In general, I will share my experience, write about camels and teach you how to cook them. This is a series of articles in three parts: the first part will be for those who are interested in stories and torments of creativity; the second is more technical, about integration patterns, their application, and the third part is about errors and debugging.
If you need to combine your services, here you will find out why Camel is good. If you want to learn how to use something new, here we start with the basics. If you like the stories and the original chips that are in each team, then read on.

Integration task

To begin with, there was a need for a service bus. We are developing a large system that affectionately “behind the eyes” was called a monster. The monster turned out to be big and scary, but in reality it was one of the BPM systems (business process managment system). It all started a few years ago. Once at a meeting, the project manager started talking about future plans:
- Colleagues, in the near future we plan to integrate with a large number of internal and external systems. Now we need to work out a systematic approach so that our analysts can start preparing tasks.

Then for some time he talked about the systems with which we have to integrate. There were both external well-known systems, for example, elibrary.ru, and internal ones that were yet to be created. At the end of the story, characteristic features of integration with different types of systems were identified. For external systems, this is a big uncertainty in the description of the processes and objects of the subject area (business objects) that were to be exchanged; but it was clear the direction of data transmission (loading or unloading); Additional requirements for the protection of data transmission channels, ensuring guaranteed delivery and prerequisites for creating tools for correcting erroneous data were reviewed. For the second - the internal systems, there was more clarity. Clearly described the functional purpose of each of the systems. Due to this, it was possible to work out the range of objects to be transferred. That's just for these systems, the uncertainty manifested itself at the level we will - we will not, we will have time - we will not have time to implement. Without going into the details of the application area, I will limit myself to typical tasks that we have focused on.

1. It is necessary to download information to the main system using a standard protocol. At the time of discussion for the external systems two were mentioned: HTTP, ftp; for internal: JMS, HTTP, NFS, SMB and / or using the RMI interface. What architecture could we use to solve this problem?
2. Suppose you need to upload information from the main system and transfer it to an external system using one of the standard protocols. What could be the ways to solve this problem?
3. Or it is required to transfer one, two, n business objects. How could we prepare the data and what format to use?
4. Suppose you need to send an email through an email server that is temporarily unavailable. To do this, the user will need to either wait for the availability of the latter, or receive a message about the inability to finish his work now. How could we build a system that guarantees delivery and does not block user work?
5. We assume that we need to be able to take information from the external system at 1 or 3 am, but our system at that time interrupted its work for maintenance. How to make it so that during the non-availability of the main system the processes of unloading and loading of data do not stop?
6. And the last task: you must be able to quickly change the login and password to access the external system or the communication protocol from ftp to sftp. How could we make it so that the settings can be changed flexibly without disrupting the operation of the main system?

After discussing the problem, we came to the decision to create a standalone application that will connect the main system with others. The idea of ​​using an intermediate link is not new and is known in the world of integration as the service bus of an enterprise or an ESB. Let's outline the range of functionality that our application should have. The basis for the formulation of tasks. It should be able to:

• interact with other systems using standard protocols: HTTP, FTP, as well as using the JMS protocol or using the RMI interface.
• read, write and create files in the local file system to work with NFS
• send emails via SMTP server.
• parse one of the transport formats XML, JSON
• have a flexible setting
• to be able to describe the rules for the harmonization of formats in one of the Java programming languages, Groovy. We worked with these languages, so we could count on the absence of additional overhead to study.
• the ability to work with XSLT and xPath would be an added plus.

From reasoning to practice

After some time, when the search process only gained momentum, analysts, at the request of the client, wrote the first statement:
- Guys, we wrote the play here, we need to do it quickly. The customer was waiting for her yesterday, so try.
')
It happens so often with us, but we always have time to make new functionality in time. This time is no exception, we were waiting for a large amount of work and short deadlines for implementation. Although we could not use a service bus for this production, we used the formulation as the first practical task for creating a prototype.

In the meantime, we had to choose the technologies on which our prototype was to be built. To invent your bicycle and start from scratch is great, but too expensive. Proprietary solutions also did not use because the result was needed quickly, but it takes time to agree and resolve financial issues. Therefore, turned to opensource projects. At that time, the choice was small, so they could appreciate the merits of the “camel” at a glance.

As can be seen from the diagram, Apache Camel is a modular, easily extensible framework for integrating applications. The main structural elements: component model, routing mechanism, message processing mechanism. A component model is a collection of factories that create routing endpoints. For example, an endpoint may be an abstraction sending messages to a broker JMS queue. A routing mechanism associates endpoints with message processing abstractions. The latter, a message processing mechanism, allows you to manipulate message data. For example, convert to a different format, validate, add new content, log and more. All three architectural components are modular, and thanks to this, Camel’s capabilities are constantly expanding. You can get acquainted with the architecture in detail in Wikipedia and on the official site . Only having appeared, Apache Camel got a solid collection of components. There were components to satisfy all our requirements: JMS, HTTP, ftp, file, SMTP, xPath, xslt, XStream, Groovy, Java. Like the author of Which Integration Framework Should You Use - Spring Integration, Mule ESB or Apache Camel? , we chose Camel. This article compares three frameworks: Spring Integration, Mule ESB and Apache Camel. The key advantage the author describes as follows:

... Apache Camel due to its awesome Java, Groovy and Scala DSLs, combined with many supported technologies.

The ability to use fluent Java DSL instead of “clumsy” XML has become a big advantage for us. The question arises: what is clumsiness? XML is a great markup language, but JSON or YAML have long preferred it. They are preferred because of simplicity, better readability, fewer supporting information and simpler parsing algorithms. Programming languages ​​such as Java, Groovy, Scala have full support for modern IDEs, which means that unlike XML, debugging and refactoring are possible. Camel had no doubts, and he formed the basis of our service tire. The fact that this project was used by other companies added confidence in the right choice.
The main question that remained was how to integrate Camel into our monsters.

Flour choice: JMS vs. RMI.

The integration of the service bus and our monster could be implemented using one of the many components supported by Camel. Based on the tasks outlined above, we have formed the requirements: the connection must be stable and guarantee the delivery of messages. We stopped at three options: the first two were synchronous RMI and HTTP, and one asynchronous JMS. Of the three, we focused on the two simplest variants suitable for our Java projects: JMS, RMI. JMS (Java message service) is a standard for sending messages, it regulates the rules for sending, receiving, creating and viewing messages. Second, RMI (Remote Method Invocation) is a remote procedure call programming interface; it allows calling methods of one JVM in the context of another. The standard remote call procedure involves wrapping Java objects and passing. In fairness, it is worth noting that opposing JMS and RMI is not correct because JMS can be a transport and an integral part of RMI. We contrast the standard RMI implementation with the JMS implementation ActiveMQ. Previously, we used RMI to integrate two applications. Why was he chosen then? When both applications are in Java, there is nothing simpler than RMI. In order to use it, it is enough to describe the interface and register an object that implements this interface. But we had the opportunity to solve problems that arise with RMI when transferring a large amount of data between applications: memory was clogged up, and applications “added”. We looked for ways to solve this problem and discussed it with JavaOne JVM developers. It turned out that the garbage collectors in the virtual machine and the distributed collectors are different things. Everything rested on the fact that for the standard garbage collector it was possible to choose its type and adjust the optimal parameters, but for the distributed such possibility was not. If we talk about other differences, RMI limited integration to applications running on the JVM, but JMS did not. In addition to the difficulties described, there was a desire to learn something new: to abandon RMI and use an alternative solution.

Camel's first prototype

Let's go back to prototype creation. The first practical task for the service bus was this: the user initiates the process of uploading data, the system prepares them and sends them to the service bus. All the work of data delivery falls on her. The figure is an example of a problem statement in the symbols of enterprise application integration patterns (EIP).

The service bus combines the reception of messages from the JMS channel, their conversion and sending via HTTP. The JMS channel and the message channel in HTML format marked in the figure were supposed to be implemented using Camel’s JMS and Jetty components. The data transformation process could be implemented in Java and / or use template engines, such as, for example, VM (Apache Velocity). The proposed data transfer scheme is implemented in Java DSL in one line. Example:

from("jms:queue:se.export") .setHeader(Exchange.HTTP_METHOD,constant(org.apache.camel.component.http.HttpMethods.POST)) .process( new JmsToHttpMessageConvertor() ) .inOnly("jetty:{{externalsystem.uri}}"); 

The example above shows the Java DSL route. A route is a description of the message route. In Camel, descriptions can be of two types: Java DSL and XML DSL. Characteristics of such a route are the starting point and one or more endpoints designated by the from and to tokens, respectively. The route describes the message path from the start to the end point. If the end points are specified sequentially, the message will be transmitted to the first one, the service bus will wait for a response, which will then be sent to the next point. There may be routes that select the desired endpoint ( Dynamic Router ) or send a message to several points at once ( Recipient List ). The from and to tokens parameter is a string with a URI. A URI is a triple of parameters, consisting of the name of the Camel component, a resource identifier, and connection parameters. Let's take an example:

 from(“jms:queue:se.export?timeToLife=10000”) 

This is the entry point description that uses the JMS component. The JMS component provides the ability to retrieve data from the queue: se.export resource. Where queue is the type of message channel, it can be either a queue, or a topic. Next comes the channel name “se.export”. A queue with the same name will be created by the message broker. The last part of the URI, the parameters of the end point: “timeToLife = 10,000”, means that the lifetime of the packet is 10 seconds.
From the example it is clear how we planned to organize the transfer of data, in the next article there will be more real code and examples.
So, we solved the data transfer problem, created a prototype integration bus that consisted of Camel and was almost ready for implementation. It remained to solve the problem of its proper and convenient settings.

Prototype setup

I am very impressed with this topic, as it is difficult to find practical advice and examples of implementation.
Our stand structure is as follows:
Each developer has his own copy of the software, and he fully sets it up before launching it. There are two test benches for manual testing, and of course, there is a working system. The task statement is obtained as follows:

• fine tuning on each of the stands should have been done simply by editing the minimum number of parameters in the system configuration files
• a large setting (adding new functions) was performed imperceptibly for all stands

In addition, if there was an opportunity to use the project builder (Maven) on the developers' booths, then there was no such possibility on the test stands and the working server.
There are a lot of difficulties in this task: Camel is associated with a JMS broker, which forces you to use different channels for different stands or different message brokers. We went the easiest way by launching the ActiveMQ broker built into the bus. In this case, it remains to provide connection settings for different servers.

Let's move on to examples of using parameters in Camel settings:

1. Example from camel-config.xml
   

    <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> <property name="locations"> <list> <value>classpath:system.properties</value> <value>classpath:smtp.properties</value> </list> </property> </bean> 

   
   In this example, two files are specified with settings for Spring.
2. An example that uses the settings specified by the two files from the previous example.
   

    <bean id="someServiceStartPolicy" class="org.apache.camel.routepolicy.quartz.CronScheduledRoutePolicy"> <property name="routeStartTime" value="${config.someService.routeStartTime}"/> <property name="routeStopTime" value="${config.someService.routeStopTime}"/> </bean> 

   
   The properties themselves are set in the string "$ {config.someService.routeStartTime}"
   
3. An example in which several files with settings are transferred to the Camel context
   

    <camelContext id="rootCamelRoute" xmlns="http://camel.apache.org/schema/spring"> <propertyPlaceholder id="properties" location="smtp.properties, system.properties"/> … </propertyPlaceholder> </camelContext> 

   
   
4. An example of using parameters in Camel routs in Java DSL
   

    from("jms:queue:email.send") .setHeader("to", simple("${headers.email}")) .setHeader("from", simple("${properties:config.smtp.from}")) .to("{{config.smtps.server}}?username={{config.smtps.user}}&password={{config.smtps.password}}&contentType=text/html&consumer.delay=60000") 

   
   It uses several ways to address parameters. Here they are:
   - in a simple dialect with the string “$ {properties: config.smtp.to}”
   - in the endpoint URI with the string “{{config.smtps.server}}”
   Parameter names can be any, the lines are taken from the example above.
   

Let's present the real problem:
there is a service that sends letters via the service bus to the SMTP server; such a service should send messages to users for the working system, and for the test system, send all letters to one mailbox.
Then this task is transformed into the addition of different logic for some routes performed on the test and production systems.

Here is an example of how such a problem can be solved using parameters in Camel routes.

 from("jms:queue:event.recoverypass") .setHeader("to", isDebug() ? simple("${properties:config.smtp.to}") : simple("${headers.email}")) .setHeader("from", simple("${properties:config.smtp.from}")) … // some other headers .choice() .when( header("password").isNotNull() ) .setHeader("subject", simple("${properties:config.passwordNotify}")) .to("velocity:vm/email/newPasswordNotify.vm") .otherwise() .setHeader("subject", simple("${properties:config.recoverypass}")) .to("velocity:vm/email/recoveryPassword.vm") .end() .to("{{config.smtps.server}}?username={{config.smtps.user}}&password={{config.smtps.password}}&contentType=text/html&consumer.delay=60000") 

The sample code above is needed for the password recovery service. The following logic is implemented: the user clicks on the button to recover the password, an email is sent to him with a temporary link to generate a new password, the user follows the link, and the same route sends the user a new password by mail. This is probably the reason to complete the first part, it remains only to take stock.

Results

The prototype of the bus was completed: there were several routes that perform the transfer of messages, configuration files appeared to facilitate the configuration and deployment of the bus. Already by the results of the first steps in mastering Camel, it was possible to talk about the great potential of such an approach. The simplicity and conciseness of writing routes is fascinating. It seems that one line can do everything. But you should pay attention to the fact that working with routs requires a change of thinking. For programmers who have not dealt with them, it is better to immediately turn to the book “Patterns of integration of corporate applications”. The effect of understanding integration patterns is comparable to the effect of knowing design patterns in a classic OOP .

On this bye, bye. See you in the next part. Let me remind you, it will be devoted to the use cases of Camel.