Visualization of integration applications

Since I began to perform the duties of a system architect, I often have to draw rectangles and arrows than to write program code. This could be fought, for example, during sleepless nights, to participate in open source projects, to create proof of the feasibility of the concept and the demonstration code, but there, too, you need to draw rectangles to demonstrate the architecture. This article is devoted to the visualization of messaging in distributed systems, service-oriented architecture (SOA) and microservice applications using agile development methodology (this term has lost its meaning, but there is no more appropriate one in this case).

What I like about software in recent years is that in most of the organizations in which I have worked, the principles of a lean and flexible software development methodology are valued. Everyone wants to create ready-made software (and not documentation), quickly produce a result (and not engage in long planning), eliminate losses and respond to changes. To implement these principles, management methods such as Scrum, Kanban and technical methods adopted from the methodology of extreme programming (for example, unit testing and pair programming), as well as CI / CD and DevOps methods are used. In general, I decided to put together in one place the schemes and design tools that I use in my daily work with distributed systems.

Problems of the “4 + 1” and “UML Death” View Model

Each project is accompanied at the start by big ambitions, but every time there is not enough time to bring everything to perfection, and in the end you have to give out something that more or less works. And this is good - in this way, the world around us helps us avoid embellishments and abide by such principles as YAGNI and KISS. As a result, we fulfill the necessary minimum and adapt to changes.
')
Most of the schemes that I met were based on Philip Cracten’s “4 + 1” representation model , which has a development, process, and also a logical and physical representation.


Representation model "4 + 1". Image taken from A Practical Guide to Enterprise Architecture, by James McGavern, Scott W. Ambler, Michael E. Stevens, James Lynn, Vikas Sharan, Elias C. Joe, 2003.

I really like the ideas and goals underlying this model: using separate ideas and points of view to consider certain constraints and positioning for different participants. This model is great for describing complex software architectures, but when I use it for integration applications, I encounter two problems.

Chart Applicability

Typically, these views are described using a unified modeling language (UML), and for each view you need to create one or more UML diagrams. If I need to create 15 types of UML diagrams to describe and explain the system architecture in an accessible manner, this is contrary to the purpose of the UML.


Death by UML (Death by UML). Fragments of the chart of Paulo Merson from Wikipedia .

Most likely, in the organization there are only one or two people who have the tools to create such a complex set of diagrams and are able to understand and update them. Inconvenient for perception and outdated charts are no more useful than outdated documentation. Complicated diagrams that do not carry much value quickly become a burden, rather than valuable documents describing the current state of an ever-changing system.

Another drawback of existing UML diagrams is that they primarily describe object-oriented architectures, rather than channel-based and filter-based architectures. In messaging applications, the main role is not played by the structure, but by the methods of interaction between the elements, routing and data flows. Diagrams of object classes, components, packages, etc. - this is far from an effective form of describing the processing of processes based on channels and filters. UML behavior diagrams (for example, activity and sequence diagrams) are more meaningful, but they also do not make it easy to describe the basic concepts of integration applications, such as filtering and routing based on content.

Applicability of representations

Representations reflecting various aspects of the system are an excellent form for describing its concept, however, the existing representations of the 4 + 1 model do not correspond to the practice of developing and deploying modern software. The idea of ​​a directional flow is when a logical representation is first created, then the ideas of development and processes arising from it, and at the end the corresponding physical representation is not always consistent with reality. The life cycle of system development is different from the traditional (waterfall) sequence of gathering requirements, design, implementation and maintenance.


Software development life cycle. Fragment of Web Serv .

In practice, alternative development approaches are used, including flexible methodology, prototyping, “synchronization and stabilization” and “overshoot and stabilization”. Over time, not only the development process, but also its participants are changing. When using techniques such as DevOps, developers need to know the final model of physical deployment, and operating specialists must know application processing routes.

Modern architectures (for example, microservices) also affect the presentation. On the one hand, the number of microservices is so large that the knowledge of one of them does not bring much benefit; on the other hand, general knowledge of microservices is difficult to put into practice. It is extremely important to choose the appropriate level of abstraction, which combines general ideas about the system with sufficiently detailed information about it.

Integration Application Visualization Tools

The most useful model, from my point of view, is described by Simon Brown and is called the C4 model (I advise you to download a free copy of Simon’s excellent book The Art of Visualizing Software Architecture ). Describing his model, Simon notes the importance of having a single set of abstractions, rather than general notation (for example, UML), and using a small set of diagrams to create abstractions at various levels: system context, containers, components, and classes. I like this approach because it implies a movement from the general to the particular, starting with the most general presentation with its subsequent detailing at each new level.

The C4 model is not always ideal for building middleware and application integration, but is more efficient than previous ones. If we used it, the system context diagram would be a single rectangle with the words "enterprise service bus", "middleware", "communication middleware" or "microservice" with dozens of arrows diverging in all directions. Not too helpful, is it? The container diagram is more convenient, but the term “container” is so overloaded with various meanings (“virtual machine container”, “application container”, “docker container”) that the value of its use for communicative purposes is small. Component and class diagrams are also not a good choice, since channels and filters are based on enterprise integration patterns, not classes and packages.

The question arises: what kind of charts to use? I use three types of diagrams, which are abbreviated as SSD (although it doesn’t sound as good as C4): system context, service design, and deployment.

System context diagram

This model displays all services (both SOA and micro services) with inputs and outputs. As a rule, external systems are located in the upper part of the model, services - in the middle, and internal services - in the lower part. Sometimes internal and external services are located partly above the intermediate level, and partly below it, as shown in the figure below. Next to the arrows, it is also useful (although not necessary) to specify protocols (for example, HTTP, JMS, file) and data formats (XML, JSON, CSV). If the number of services is large, then the protocols and data formats can be indicated on the service level diagrams. Using the arrows, I show the services that initiate requests, rather than the direction of data flows.


System context diagram

This diagram gives a good general idea of ​​the structure of a distributed system: its services, internal and external communications, types of interactions (with protocols and data formats) and their initiators.

Service structure diagram

This diagram shows what happens in each of the rectangles corresponding to the connecting services in the system context diagram. To create this diagram, it is recommended to use EIP icons and connect them to each other by message flows. A service can have multiple data streams, support various protocols, and also implement real-time mechanisms and batch processing.


Service structure diagram

Deployment Chart

The two previous diagrams are logical representations of the system as a whole and each of its services separately. Deployment diagrams show the deployment locations of each service. Instances of the same service can run on different nodes; Services may be active on some nodes and passive on other nodes. Sometimes services are managed by a load balancer.



Deployment Chart

The deployment diagram shows how individual services and the system as a whole relate to nodes (physical and virtual).

Tool Tips

System context and deployment diagrams consist only of rectangles and arrows; to create them do not need special tools. To create a diagram of the structure of the service, you will need a tool with the function of integrating EIP icons. At the moment, I know the following tools with support for EIP icons:

• Mac OS: OmniGraffle with graffletopia icons. With this tool, I created all the diagrams for the book Camel Design Patterns ( Camel Design Patterns );
• Windows: Enterprise Architect by Sparx Systems with Harald Westfal EIP badges. In addition, there are MS Visio templates;
• • Internet: LucidCharts , which by default includes EIP icons. This tool is the easiest to use and is available anywhere.

In addition, there are a number of other development tools that can be used to create EIP diagrams:

• Red Hat JBoss Developer Studio ;
• Sirius EIP Designer ;
• Talend Open Studio for ESB ;
• Hawtio , a plug-in for Camel, which can display Camel routes performed as EIP diagrams.

The system context diagram visually displays the overall structure of the system and the coverage of its services; The service structure diagram describes the service well, and the deployment diagram is useful in designing the physical implementation of the contents of the two previous diagrams.

IT professionals can find tasks that take up all their working time. I am sure that if we had more time, we would have come up with a dozen more useful ideas, but without these three we could not describe the application for integration. As Antoine de Saint-Exupery once said: "Perfection is achieved not when there is nothing to add, but when there is nothing to remove."