BizTalk Server 2009

Hello dear habrozovateli. In this post I want to tell you about the product for automating and managing business processes BizTalk Server 2009.

The narration will be full of examples, because it is easier to understand the possibilities of the product when solving real problems.

Introduction

For a long time, management systems for medium and large companies include many software packages that are introduced as the company grows and develops. Also, these systems require communication with programs and services outside the enterprise. It looks like this:
')


To understand the problem area, I will give an example of the simplest business process:

1. The user creates a document "Order goods" through the web portal
2. The order must be transferred to all other enterprise systems (for accountants to pay the invoice, to the logistic system for accounting and delivery of goods, to Datawarehouse for compiling historical data and counting kpi, etc.)

We have a problem: how to make all components work with each other?

If the logic of data transfer is implemented directly at the system level where this data is created, then we proceed to the necessity of implementing one-to-many communication for each of the systems. This will lead to a large number of links between existing systems and the need to add new links when we want to introduce another system.



To solve such problems and was created BizTalk Server (BTS). It introduces a new level with which the systems interact, and takes on the responsibility of delivering messages to subscribers. Now, in our example, it is not necessary to send a document from the web portal to all other systems. It is enough to send it to BizTalk, and then be sure that the message is delivered to each of the systems requiring this. Obviously, the number of links with this approach will significantly decrease. This will reduce the cost of managing systems as a whole (after all, we have removed the duties of communicating with other systems), and will also make it easy to add new ones.

BTS architecture

Here is the simplest service sufficient to describe the architecture of the BTS:

1. At the incoming port, we receive a message initiating a business process;
2. We process the message;
3. Send a message to one or more outgoing ports;

But this process takes place at the server level:


First, a few definitions from what is seen in the picture:

Receive Adapter - input port. Required to receive messages from the outside world. It supports various protocols and ways to access applications.
Orchestration is the performing part of the business process. Here you can write C # code, transform messages, throw away and catch exceptions, cause compensation for successfully completed segments, etc. But the main functionality that the orchestration provides is the message processing patterns. I will give one of them in the section with an example.
Send Adapter is the same as Receive Adapter only for sending messages.

Well, now the process of the service in more detail:

1. The Receive Adapter receives a message on one of the many available protocols from the outside world;
2. The message is passed through Receive Pipeline (with the capabilities of transformation and validation);
3. The message gets into the Message Box (MB) - message storage inside the database based on MS SQL Server;
4. A constantly working agent records the presence of an unprocessed message and the recipient in the form of orchestration;
5. The agent calls the orchestration with the input message to him;
6. The orchestration performs all the necessary work and generates a new message that goes back to the Message Box (most often the structure of the new message is different from the original one);
7. The agent again records the appearance of the message in the MB and searches for the recipient. This time it is the Send Adapter;
8. The message is transmitted to the Send Adapter which, in conjunction with Send Pipeline, by necessity, validates, transforms and sends the message to the outside world using any of the available protocols;

Now that the base example is clear, I will try to describe on its basis some useful and even unique features that BTS provides:

1. A large number of protocols with which incoming (outgoing) ports can work (File, SMTP, HTTP, WCF, MSMQ, SOAP and others). In addition to protocols, there is also support for various applications: Oracle, SQL Server, WebSphere MQ, SharePoint, Tibco, SAP, JMS, JD Edwards, Dynamics, and others. If your system is not listed, then there is an SDK with which you can develop an adapter yourself;
2. Received messages can be converted inside the Pipeline to the most convenient format before they get into the orchestration. Out of the box there is support for working with Flat files and rich EDIFACT processing capabilities. All these formats BizTalk itself converts to XML with a predetermined XSD scheme.
   And again, if you want to receive input, for example, Excel files or Binary files, you can write the conversion procedure to XML yourself, and you can directly drop it into the Message Box, and then parse the message inside the orchestra.
   I would like to especially note the convenience of processing EDIFACT messages. All available formats are supported, but if one of your partners deviates from the standard, you can easily correct the format for it. In addition to the message processing framework, BizTalk also provides an infrastructure for exchanging documents. It can keep a record of the uniqueness of messages, enter special settings for each of the partners and automatically send confirmations of acceptance of an EDI document. Also, all this can work through the AS2 protocol;
   
3. In the described example, asynchronous processing of the message takes place, but in general it can be synchronous or mixed. For example, if we receive an HTTP request that requires an answer, then we can respond to it only after we are convinced of the successful sending of the message to the outgoing port;
4. Design patterns involve the reuse of components alone without the need for duplication. For example, if you need to receive a message not via FileSystem but via FTP, this will in no way affect the implementation - you just need to change the type of the incoming port and set it to the correct address;
5. Contextual message routing. Very convenient functionality! If you have several components processing a single message type, then you can sign them to receive using the XSD schema description. Thus, as soon as the Message Box drops XML that satisfies a given XSD structure, all interested subscribers will immediately receive a copy of it. Naturally adding a new subscriber does not affect recipients other than the sender. If you draw an analogy with JMS, it's as if you had one topic with one publisher and many subscriber. But the difference is that in the case of contextual routing, you do not need a topic, but only need a contract to receive the message in the form of an XSD scheme;
6. A little about scaling. As you noticed, Okestreyshen receives a message directly from the database. He does not know where it came from and to whom it was intended; its task is to take the message, process it and put it back. Therefore, you can install as many servers as you like, each of which will be subscribed to receive messages. And at that moment, when suddenly there will be 1000 of them in the database - Load Balancer will automatically distribute them equally for each server, depending on its current load;
7. Storing messages in the database gives another positive point. If an error occurs when sending a message to a port (for example, the channel is closed by the firewall), the message will be rejected and stored in the database with a description of the error. When the administrator detects a problem, he can correct the problem with the network if necessary and resend the message to the port. You can also implement automatic reprocessing strategies. All this is done without any participation of the developer through a convenient and informative BTS administration console;
8. A few words about the console itself is a powerful tool for managing and monitoring the status of one or more BizTalk servers. Here you can observe which services are active, control their launch, configure scheduling, check how routing messages occur between components, have there been errors recently and much more;
9. BRE (Business Rule Engine) - a mechanism for introducing functionality into a service on the fly (Dependency injection). To use BRE, the developer inserts the Bussines Rule element into the orchestration and implements the processing in the context of this rule. Then the service is deployed to the server and starts working. Over time, the business model may change, then the Analyst will have to create a new rule in a special user-friendly editor and add it to the BRE repository by associating with the required service. At the same time, the previously created process will function on the basis of the changed rule, without the need to involve the developer and restart the service;
10. BAM (Business Activity Monitoring) - all elements within the orchestration that are used in the operation of the service contain a set of parameters that are available to the BAM server. For example, the object of remapping xml messages provides access to all fields specified by XSD scheme. Having this data, business users, without the participation of the developer, can create data samples, charts and diagrams based on the data passing in the XML message through the service. All this work occurs regardless of the process of the service itself and can change without the need to stop the service. Results can be displayed in office documents or on a special BAM portal.

Service example

Here is an example of a more sophisticated service, showing the capabilities of the orchestration in managing the flow of the business process (the numbers in the list correspond to the items in the designer of the orchestration in the picture):

1. The user in his system creates an order of goods. A positive or negative answer should come to his request, depending on various factors determined by external, in relation to the user systems. The order is input to the BizTalk adapter, then to the MessageBox, and then the execution is passed to the orchestra;
2. The service sends the received order to the manager to check the validity (after all, the user could order goods not related to the work of the company);
3. The same order is immediately sent to the warehouse to check whether it can be carried out (the Goods may simply not be available);
4. Next comes the ParallelActions Shape, which implements the Parallel Convoy Pattern. This element will complete execution only when all its constituent elements have completed. And 2 facts should work: the service should receive answers from the manager and from the warehouse.
   In general, waiting can last long enough (for example, if the Manager is on vacation). In this case, BizTalk has the ability to set the type of Orchestration: Long Running. Then, while waiting for a response from both systems, the state of the orchestration is serialized and stored in the database (in terms of BTS, this process is called Application Dehydration). And as soon as both answers are received, the service will reload into memory and complete the ParallelActions Shape.
   There is one subtle point. In a real system, there are always many users and the manager application and the warehouse can receive several messages at the same time, respectively, and they may not respond in the order in which the requests came. Hence the question: how does BizTalk know to which of the working service instances the answer from the external system is intended?
   The answer is simple, logical and is realized in just a couple of clicks in the orchestration designer. Each outgoing message on the Send Port must have a Correlation_ID. In this case, the OrderNo field is ideal for its role. It must be specified directly from the context of the XML message. The same Correlation_ID is also set for the response to the request. Thus, we have many services waiting for an answer and many answers. The Call Management Agent will use the OrderNo field to communicate with each other. That is, if instance “A” sent the order number “35”, and instance “B” ordered “47”, then the answer to order “47” will be returned to instance “B”, and to order “35” - “A”;
   
5. After the replies are received, the results will remain from the manager and from the warehouse. If everything is OK, the user will receive a notification about the successful completion of his order, and the order itself will go to the system of the supplier of goods. Otherwise, the user will receive a message describing the reason why his order cannot be satisfied. At this service will complete;

In this example, the service worked with 4 ports. Each of them has its own system (for example, it can be Oracle Retail, Dynamics, JMS ports and Web Sphere or something else), but the service itself should not worry about which one. Details of message delivery, authorization, message conversion to the desired format for each of the systems occur at the level of the adapter and the associated Pipeline. Also, the problems of transport and possible delivery errors are also solved using standard tools, and does not require implementation at the service level (although, if necessary, this can be done inside the orchestration).

As you can see, this approach to creating business processes implies solely the implementation of business logic and allows you to completely abstract away from the systems with which you have to exchange messages.

Thank you for your attention, I hope you were interested.

Literature:

• WROX: Professional BizTalk Server 2006
• APRESS: Pro BizTalk 2009
• PACKT: SOA Patterns with BizTalk Server 2009