How we work with directories on the integration bus

Principles of solution

When integrating corporate systems, the task of managing reference data arises. To solve this problem, Master Data Managment (MDM) is often used. MDM is a repository that contains “reference” reference data, the so-called “golden records”. The directories in MDM contain cleared complete and consistent data.

Often, MDM is used as a platform for centralized reference directories. Reference data are entered and validated in MDM, and from there they are replicated to IT systems. This approach has several problems.

• Creating a reference data model that fits all systems is not easy.
• Reference data becomes detached from applications.
• Replication of data from MDM often requires major system modifications. For out-of-box systems, such refinement can be very expensive.

Another approach is that each business system stores directories locally and organizes data entry. When exchanging messages between systems, the integration bus performs the transformation from the format of one system to the format of another. When this occurs, and the transformation of reference data.
')

Transformation on the integration bus.

We use the second approach. All business system interactions occur via the integration bus. The bus (in our case, the Oracle Service Bus) transforms the message that the supplier’s system sends into a message that the system can understand. Such a transformation involves mapping the values ​​of reference books.

Data on how directories are mapped between systems is stored in a relational database, in our case, Oracle. In the tables, it will be recorded as from a dictionary value in one system to get the value in another system. That is some kind of structure:

(source_system, source_value, valid_from, valid_to, target_system, target_value)

Data in reference books change very rarely, and is used very often. In order not to access the database each time, directories are cached on the bus, and in a format that the bus can immediately use.

For caching we use Oracle Coherence . This is a very, very paid product. However, in this case, all its mega-functions are not used, so it is quite possible to replace it with a free solution (for example, hazelcast ). Read more about coherence here . Also, the coherence license is included in various Oracle Suite.

Using a cache has obvious advantages:

• data is stored in memory
• data is stored in serialized form
• data can be indexed
• database synchronization can be done asynchronously

The cache is distributed and synchronization between nodes is done by Coherence itself. When adding or removing a server, the cluster rebalances data between nodes.


Distributed Cache Map is used for reference data. During the start of the Oracle Service Bus, a cache is created inside the JVM that holds the data in memory. Each physical server has a coherence server that stores directories (in memory and on disk) and synchronizes with the database.

When transforming, osb workflow calls coherence via Java callout. You can also access via an Enterprise Java Bean call.