Integration - bikes

You Khan, if you do not got into the business process of the customer, and you dig only in the IT process.

This general rule almost knows no exceptions. Because the IT structure often does not own the data and does not know why they are needed further. And when it comes to integration (without which not a single implementation of a digital tool can do, be it a trendy chat bot or old school CRM) - this needs to be understood very precisely and in detail.
')
I saw a lot of different stories: and how one tiny service for working with faxes after implementation began to behave like a virus and fight the mail server and other services of the company (by the way, won), and how the “small and medium business” segment did not include the average business (to the sincere surprise of the IT department), and how people just didn’t know what the meaning of the processed data was and why they are needed - this often concerns finances.

So let's tell a couple of tales about what happens when you just need to take and connect several IT systems. Delov something!

What is integration in the Russian sense?

The most frequent examples are:

• You need to get something from the available systems. For example, there is a ticket booking system on the site, there is a check-in system at the airport. If you connect them, you can make online registration. Naturally, not everything is so simple - in the course of execution there will be a lot of questions.
• Need to connect something new. For example, there is an Internet bank and there is a good old ABS for bank employees. It is necessary to separate the execution layer and the interface layer: what the employee in the department does and what the client himself does in the Internet bank are interfaces. ABS do not care who requested a transfer from one account to another: a client is online or he is in the office or by phone. The main thing is that the operation is sufficiently authorized.
• Need to build a chain. That is, that several systems work as a single unit, so that operations begin, for example, in one of the systems, continue in the second, and end in the third. This is how transfers work in banks: if you transfer from card to deposit, then several systems are usually used at once - ABS for deposits, card processing, mobile or Internet banking, SMS gateway, etc.

The general approach is that the systems need to somehow communicate. You can do it the old-fashioned way, through the exchange of uploads and throwing files over long distances, it is possible a little more modern, it is possible through web services and so on. What is important is not a technical implementation, but a general principle. By integration, we understand the situation when IT systems of an enterprise operate at costs below a given level. Yes, you can do it on your knee with your grandmother and floppy disks, but remember that grandma has grandchildren, a lunar calendar of planting seedlings and all that, and the IT infrastructure should always work like a clock, quickly rebuild and not be a brake on business processes this infrastructure work.
That's for all this and you need the right integration.

Approaches are different. Sometimes we use direct data exchange, but more often we separate interfaces (service consumers - web applications, mobile applications, partner systems, etc.) and service providers (ABS, ERP, and so on). And then we build, for example, a bus for data exchange. If we talk about the "grandmother file transfer" as the most ancient integration technology, the bus approach is already an industrial age. Interfaces send data to the bus, the bus carries it to the executable modules and “tells” them what to do. Along the way, data formats or interaction protocols can be converted, data can be added from third systems, operations can be coordinated in different systems, etc. Then the execution results go back to the interfaces. This approach allows you to hang any number of subsystems, is more difficult to implement in the early stages, but with a more or less large information infrastructure, it makes sense because it allows you to save time and money due to reuse, reliability, transparency and a whole bunch of other buns.

Although I saw once that the customer had a surprisingly stable formation of five systems that ran directly into each other’s bases and exchanged data normally. Year four it worked like a clock. Then they learned that it was possible to make a tire, and there were six buggy systems - five available and another tire. So, too, is not always ideal. The mistake was that they took the bus and, without thinking, transferred their direct data exchanges as is to it. We received an additional layer of complexity and a point of failure, but did not receive any benefits from it - it was necessary to think out the architecture and processes more correctly.

In the real world in its pure form, the ideal architecture is extremely rare. Because to build everything correctly and working like a clock, you first need to:

• Understand what is happening (and each department will have its own vision)
• Understand what a business needs in the end (that is, you need someone who sees the big picture, and this is often not even a CIO)
• Understand why people resist change, do not want to let in their systems and share their data (most often, someone will not steal or someone will lose control)

This is where the fun begins.

How does the project begin

Usually the customer does not exist in a vacuum, and he has some kind of integration. That is, something is already working, and try not to touch it. There is, for example, a call center, he can pull up customer data from a common base. It is organized through the ass, but organized, and everything is already well. Five years pass, support builds crutches around it. Then the system begins to acquire new functionality, which is actually not for her. For example, on the basis of a corporate messenger, an exchange of large files suddenly appears; from the whole company, just to draw graphs, and so on. From my point of view, it looks like this:

- Guys, you're from IT, you're smart. Do us just such a small thing quickly and cheaply.
- For this we need such and such a system, it is not easy.
- Come on, blind in a quick way, this is a temporary building.

And there is nothing more permanent than such makeups.

As a result, approximately once every 5–7 years, customers understand that they need to introduce a new large system or replace the already suffocating old one - and here comes the sea of ​​everything that has been tolerated over the years. Begins the raking of errors.

It never happens that "since Monday everything is new." Many people think that the new system is when everyone will sleep well. Not. At first, this is most often new failure points, not a panacea.

Here, take the same introduction of sexy Internet banks. This is the front to the ABS, that is, the core of the bank. It is necessary to connect the ABS, credit office and all subsystems, we need a bus and interfaces. At first, “on crutches” are molded with unloadings, and the bank is not very online. Then comes the understanding that you can break down each operation of the front office into abstractions and make an API to the system used by bank employees, and give this API to the Internet bank. And only then comes the understanding why the tire.

We teach people to negotiate

As I said above, if you do not get into the business processes, the project is doomed.

Here is an example: at first everything is just technically. Two departments, in one, data are processed in 1C (accounting), in the second - in HRMS (personnel management system). As long as there were two hundred employees, everything was fine: they unloaded the CSV, imported it into another system, gave out the salary and took everything into account. After 6 years, the staff was under a thousand. Six people in the personnel department were engaged in manually performing the script cycles. We came to write this script. Sounds easy, right?

It is impossible to transplant personnel officers to 1C and vice versa (more precisely, economically unreasonable) - their butt does what the application software of another department cannot. Therefore, it is possible either to make a master data warehouse on the basis of one of the systems, or to make it outside.

And here in this place a political squabble already begins, whose system is more important. Then - who and how will upload data, in what formats, in what encoding. The epic holivar between 1251 and UTF-8 in one company lasted for a month of meetings (this is in another project, usually the tire solves this issue with ease).

Then you have to agree on which data format is universal. Here both departments tell what they need in the database and what they will use. Only then, when they understand exactly what and how, you can decide. In this story we used index storage - on the integration server there were links to different pieces of the database (and converters) - as a card in the library, there was no separate data lake.

The key point is for people to agree among themselves what data will be used, how and where, who puts them there, from where and why. If it is - everything else is already a technique. That is what we are helping to do, and from the fact that business needs, technical solutions are born.

If the example of accounting and HR managers is too abstract and simple, imagine a situation where one bank absorbs another. Just think that you need to connect and finish, so that different IT systems work together and allow in the branches to do everything that customers used to do. I would really like to read the chronicles of such a CIO, for example. But this is usually under the hard NDA.

Stories

The first story is not a bike. Just look at how SMEV developed - an integration bus of the scale of the whole country. This is data sharing between departments. For example, so that when you receive your passport, you do not have to collect information and bring a military officer. It is logical, right? The state has all the documents on you, and you just need to request them within the system. But no. Who knows this story - now it is quite difficult to connect to SMEV. Because now the third reincarnation of the tire is already in operation. The first version was beta level - it was very difficult to teach departments to simply determine what data they have and how to unload them. In the second they connected many, and as a result a wild amount of functions appeared without a proper level of abstraction, which were not always optimally designed. In the third version we updated the documentation, introduced more checks and appraisals. Now the data comes to the system in a more or less uniform format.

In general, parts of any integration are data, interfaces, people, applications. Applications - of course, are data handlers. For example, if you need to give a salary to Ivanov, then the processor can do this. Data is an abstraction in its pure form, that is, information about how to distinguish Ivanov from other people, what his salary is, and so on. Interfaces are where teams come from, for example, enterprise HR software or XLS. People are people who want and do something. They are usually overlooked.

Here is another example. They helped one organization build customer service and sell more. At the same time, the organization is not young, it has a lot of information systems, and even more data. To begin with, it was necessary to understand, and who are the clients of this organization? The same person could be Vasya Ivanov (or, in general, “Fluffy Chepushila”, if the data are from social networks), Ivanov Vasily, Ivanov V. A. or “passport number 1234 5678901”. First you need to understand how many people are here: is this all the same Vasya or are these typing errors? That organization had several million customer records. The rating was five records per person. They made cleaning, deduplication - reduced to 1.4 records per person. There was a lot of ambiguous data from the paper, including from the old ledgers - only the name and initials could be there.

Often the customer incorrectly puts the TZ, without thinking what and how. For example, we had a requirement for 100 transactions per second. But at the same time, the existing system was able to process only 5 records per minute. They said they are expanding. I had to change the hardware, licenses, rebuild a lot in the infrastructure. It turned out that they needed about 2 records per second (120 per minute), and this could be done on the current hardware with minimal changes. And this is the requirement for the next 5 years. Not considered - spent a lot of extra money. We ask: they say, why a hundred transactions? They explain to us: this is a special analyst considered. When we later celebrated the commissioning, this analyst admitted: they say, the number is beautiful.

In itself, implementation is rarely done for a long time. More - design. At this stage, the owners of the systems need to agree on what data will open to whom, who is responsible for the data, and what control procedures. The anthill begins to evolve - it is wildly difficult organizationally and psychologically. Because people in Russia are kings: “I will not give my systems, my data outside, but they will suddenly say that we are not working well. I do not want this. Big risk, but with proper perseverance is solved.

Most efforts at this stage may require a meeting to survive. There was, for example, a project to merge several subsystems into one company with state participation. And for a part of the subsystems were responsible other companies that lived there for a very long time before that. And they used to work at their own pace. The worst thing happened in the middle of the design - one of the companies really wanted to increase its importance, and we came to the meetings with them every week. For half a year they worked out a decision that the difference between the numerical indicators of the systems being introduced is not very important, but it is important that all this work.

Or another example: in one bank, everyone was gathered for meetings, while my participation was clearly 5 minutes. In the end, it was minus 3-4 hours. In fact - a fully spent day, which is very disappointing. But if the customer is important, nothing can be done.

There is an opinion that any approach can be template, that is, we have a specific task that can be decomposed before doing there action 1, action 2, action 3, shift the file. And then under it choose a ready-made tool or write your own, if you understand what you are doing. But you must first understand the task, why we do it. When faced with a specific integration task, you solve a specific integration task.

It happens, after building, they knock on us: they say, “your tire does not work”. That's right, a new entity has appeared - blame everything on it! Therefore, specialized monitoring systems are being introduced in the right projects in order not to find out for long and painfully what has fallen and where. Such tools immediately show which subsystem does not respond and where. In one of the projects, those who were responsible for the tire, that is, we received the first alert.
- Your tire is not working!
- No, it doesn't work for you right here.
- Aah, well. Do not relax there.

Or here's the delivery-acceptance. There are 8 services, you need to pass them in one test day. Two of our modules, the rest were written by other companies. Business on Saturday "plays" in its activities in the new system, taking turns testing modules. All perfectly. Saturday is closed, in the evening everyone is going home. Switching. This is the right process.

Another bank took over the experience, built the process in the same way. Only tested not on the full infrastructure, but in parts. Without reducing everything to the full process, without checking the connectives between the modules (more precisely, checking the basic, at the level of simple tests). And on Monday after the switch-over, it turns out that no one at the customer had the idea to check the interaction of the systems under certain conditions. That is, they tested all the individual services, but not how they interact with each other, Karl! Begin to request sausage, send something, receive, something else. In the evening, one of the subsystems begins to create a stream of data that simply overloads the other, and a chain reaction begins. Everything falls.

The reason - in order to put complex tests or implement a comprehensive project, ideally you need to have a single general contractor, a customer, from which you can ask him to organize everything, or organize it yourself. You will not believe, but this is not always the case. How many problems were there when it hurts because a lot of people are responsible for the integration from the mold to the mold.

Broadband data highway

We, in fact, work as translators. There is a customer from a business that says: "We need this one." And no one understands it further. Therefore, the problems, goals and aspirations of the business must be translated into technical language in order to communicate to the developers. And then the problems of developers to convey to the business so that they understand what will fall if you do not take into account their needs. And first of all - how to build this all into architecture, because most often a kind of quilt is formed at the enterprise: here is one, here is another, here is the third. These are completely different languages, different reference books, different commands. Until then, they somehow get along with each other, but the task of the whole integration is to make such a broadband highway with data from which they come from all sides, and everyone understands: where, where, on what cars and why.

The worst thing is that the IT department and business live in different realities. IT does not always know what is really important for a business (which operations are most valuable, for example, where it is necessary to optimize right to the core, and where it is absolutely not important), and the business does not know what problems IT has. “Servers should be bought, but you will not survive the new year” - and suddenly everything becomes clear. And before that, they simply did not know that “the maximum calculated load” is what is related to the queues for the holidays. In such cases, you need someone who can translate the IT language into a business and help them negotiate with each other.

And I also adore stories when someone silently changes something in ready-made systems. We had one hero who “brought order to the directory” before the New Year, deleting the records that seemed unnecessary to him. As a result, several trucks of products rotted at the customs, because the data did not go. We knew what had broken, but the person who had access to the router was on vacation. According to the regulations, only he could do it. According to the safety rules, you need to come into the room, open one lock, then the second lock, enter the server room, open the cage, open the console there, record the information in the console and do everything in the reverse order.

That's about it. So if you decide to change something in IT, you must first agree with the whole company among themselves, and only then implement it. And in order to negotiate - we need people who understand what the task is, what and how to do, they will translate it into a language understandable to all participants and help build everything. If you do only part of the work, for example, to design an ideal system, but not convince people that you need to work with it, there will be failure.

Well, just in case my email is: AnIvanov@croc.ru.