Tens of thousands of purchases that are transparent as glass: unraveling a tangle
To restore order in the procurement of a large bank is not easy. Especially when they are divorced on two independent complex systems of ERP and EDS. When VTB and VTB24 merged, we also had an integration of information systems, and now a single procurement process passes through them. What to do? Process Mining came to the rescue - one of the most interesting technologies for research, analysis and monitoring of business processes. But at the same time very difficult to use.
Process Mining is an approach to analyzing business processes using advanced technologies in the field of data collection and processing. We saw a lot of expensive, large projects, where process analyzes were taken using Process Mining. Despite the fact that these projects were completed, in 80% of cases, the beautiful schemes obtained did not work. But the sad statistics did not frighten us, and we also decided to unravel our tangle of processes through Process Mining. Details under the cut.
As we have already said, the implementation complexity was primarily due to the fact that after the merger of VTB and VTB24, the procurement process in the bank goes through several information systems that are responsible for different stages of the process. In addition, we had to take into account historical information from the decommissioned system. As a result, we got a heterogeneous set of IT data sources - IBM Lotus database, MS SQL database, Oracle database, SAP (integration through RFC). To complete the picture, the data sources are located in different network segments - this also had to be taken into account in the solution architecture and integration methods. By the way, we have a separate post about merging network banking segments. But back to the business processes of procurement.
In fact, the good desire to restore order in business processes has decomposed into two tasks :
')
Technological implementation of the solution in the bank includes the following components. The Process Mining platform is implemented on the basis of Celonis software, the data collection component is Pentaho DI + PostreSQL, the data storage and data mart is a Vertica column database. The Pentaho DI + PostreSQL bundle allows us to centrally collect and process data from sources (IBM Lotus, Oracle, MS SQL, SAP DFC). Vertica is a productive column-type database that allows us to store data in compressed form and process large massive requests faster. That is why Vertica serves as a data source for Celonis, which takes the data model for further automated mapping of the business process and subsequent analysis.
Our key tool is Celonis, used for Process Mining. It has a rich internal visualization and analytics, which can be extended with the help of the built-in Python API, which opens access to all modern data analysis approaches.
In general, each of the components we selected perfectly performs its own task. At the same time, they all combine well as a single solution. The new platform allows you to provide Process Mining as a service, with a customized level of detail and frequency of data updates. For some tasks of the bank, we provide data for Process Mining every 15 minutes. But in the context of this task there is no need to update the data more often than once a day.
In Celonis, it is very convenient to create excel reports based on analytical views, which always makes the calculation of the tool transparent. We came to the conclusion that, together with the implemented KPI, it is convenient to have on the same sheet a report with a complete list of transactions (events) on the basis of which the KPI was calculated. As a result, we can solve analytical tasks and tasks for internal reporting in parallel - this is an important advantage.
A digital model of a business process, assembled in a similar way, can detect: multiple rounds of agreement; time delays in status; inefficient or most busy artists; best and worst divisions in a KPI context and more. Analyzing information in terms of processes, it is easy to make the transition from the analysis of numbers to optimization. Information on each purchase, we can view in Celonis, and with the entire history of changes - earlier for this, we would have to contact almost dozens of systems.
Using Process Mining, we can analyze both a specific purchase and a sample of interest by type, division or other parameters over time. So you can easily identify inefficient steps of the process or, for example, find the reasons for the deviation of the process from a given model. For example, this is how we learned that contract negotiation is usually one of the longest steps in the process. And they counted the percentage of purchases that do not fall into the final status, and identified the reasons for this.
If we go further, then Process Mining allows us not only to identify problems based on multifaceted statistics, but also to discover the best ways to go through the procurement, to understand why everyone does not use it.
Okay, Process Mining is great, but what about the specific objectives of the project? We successfully coped with the first one within the stated time frame. Initially, it was necessary to restore business processes only for the purchases of the Information Technology Department, but after receiving the first results and demonstrating them, the internal customer asked to scale the solution for all purchases of the bank. And we managed to do this without moving the agreed time frame.
With the second task, calculating the KPI, everything was not so simple. Strict requirements for errors in the calculations of KPI demanded an improved quality of the collected data - 96-98% compared to sources. This quality was not achieved immediately, it took time for the financial department to devote us in particular to the business process. The process mining center of the bank and the financial department jointly identified poor-quality data and features of technical implementations, which sometimes distorted the process models.
At the end of the project, we were among those 20% of the lucky ones that Process Mining really helped. And this is not luck. Building a process model based on real-world data that is updated daily, calculating process indicators and bringing it all to beautiful and convenient analytical concepts is only part of the story. In many projects, they miss something without which no Process Mining will work - data quality . We have done a lot of work with the in-house customer to improve the quality of the data so that our system can not only carry out the analysis, but also prepare regular reports for making important management decisions.
As a result of the project, our understanding of Process Mining has changed somewhat. This is an approach to collecting disparate information about the process and its subsequent in-depth analysis using modern tools. Moreover, the approach involves a continuous and consistent collection, recording and analysis of events from information systems about the target object of research, and its evolution in the process of movement.
Our solution based on Process Mining technology has proven to be useful for a large number of different users involved in the procurement process. Now, within the framework of a unified system, they can deeply analyze these processes, monitor the status of specific purchases, KPIs and, finally, automate reporting. If we talk about numbers, the introduction of Process Mining and the implementation of a set of measures by the financial department made it possible to reduce the time of the procurement process by 25%, while the total number of purchases increased by 3 times.
Celonis has a rich marketplace with paid addons. But we came to the conclusion that it is better to develop your own customized fine-tune tools using the Celonis API in Python. We will share this experience in the following articles.
Process Mining is an approach to analyzing business processes using advanced technologies in the field of data collection and processing. We saw a lot of expensive, large projects, where process analyzes were taken using Process Mining. Despite the fact that these projects were completed, in 80% of cases, the beautiful schemes obtained did not work. But the sad statistics did not frighten us, and we also decided to unravel our tangle of processes through Process Mining. Details under the cut.
As we have already said, the implementation complexity was primarily due to the fact that after the merger of VTB and VTB24, the procurement process in the bank goes through several information systems that are responsible for different stages of the process. In addition, we had to take into account historical information from the decommissioned system. As a result, we got a heterogeneous set of IT data sources - IBM Lotus database, MS SQL database, Oracle database, SAP (integration through RFC). To complete the picture, the data sources are located in different network segments - this also had to be taken into account in the solution architecture and integration methods. By the way, we have a separate post about merging network banking segments. But back to the business processes of procurement.
In fact, the good desire to restore order in business processes has decomposed into two tasks :
')
- restore business process based on data from all sources - for subsequent data-driven optimization.
- calculate key performance indicators (KPI) of work processes - for management reporting
Technological implementation of the solution in the bank includes the following components. The Process Mining platform is implemented on the basis of Celonis software, the data collection component is Pentaho DI + PostreSQL, the data storage and data mart is a Vertica column database. The Pentaho DI + PostreSQL bundle allows us to centrally collect and process data from sources (IBM Lotus, Oracle, MS SQL, SAP DFC). Vertica is a productive column-type database that allows us to store data in compressed form and process large massive requests faster. That is why Vertica serves as a data source for Celonis, which takes the data model for further automated mapping of the business process and subsequent analysis.
Our key tool is Celonis, used for Process Mining. It has a rich internal visualization and analytics, which can be extended with the help of the built-in Python API, which opens access to all modern data analysis approaches.
In general, each of the components we selected perfectly performs its own task. At the same time, they all combine well as a single solution. The new platform allows you to provide Process Mining as a service, with a customized level of detail and frequency of data updates. For some tasks of the bank, we provide data for Process Mining every 15 minutes. But in the context of this task there is no need to update the data more often than once a day.
In Celonis, it is very convenient to create excel reports based on analytical views, which always makes the calculation of the tool transparent. We came to the conclusion that, together with the implemented KPI, it is convenient to have on the same sheet a report with a complete list of transactions (events) on the basis of which the KPI was calculated. As a result, we can solve analytical tasks and tasks for internal reporting in parallel - this is an important advantage.
A digital model of a business process, assembled in a similar way, can detect: multiple rounds of agreement; time delays in status; inefficient or most busy artists; best and worst divisions in a KPI context and more. Analyzing information in terms of processes, it is easy to make the transition from the analysis of numbers to optimization. Information on each purchase, we can view in Celonis, and with the entire history of changes - earlier for this, we would have to contact almost dozens of systems.
Using Process Mining, we can analyze both a specific purchase and a sample of interest by type, division or other parameters over time. So you can easily identify inefficient steps of the process or, for example, find the reasons for the deviation of the process from a given model. For example, this is how we learned that contract negotiation is usually one of the longest steps in the process. And they counted the percentage of purchases that do not fall into the final status, and identified the reasons for this.
If we go further, then Process Mining allows us not only to identify problems based on multifaceted statistics, but also to discover the best ways to go through the procurement, to understand why everyone does not use it.
Okay, Process Mining is great, but what about the specific objectives of the project? We successfully coped with the first one within the stated time frame. Initially, it was necessary to restore business processes only for the purchases of the Information Technology Department, but after receiving the first results and demonstrating them, the internal customer asked to scale the solution for all purchases of the bank. And we managed to do this without moving the agreed time frame.
With the second task, calculating the KPI, everything was not so simple. Strict requirements for errors in the calculations of KPI demanded an improved quality of the collected data - 96-98% compared to sources. This quality was not achieved immediately, it took time for the financial department to devote us in particular to the business process. The process mining center of the bank and the financial department jointly identified poor-quality data and features of technical implementations, which sometimes distorted the process models.
At the end of the project, we were among those 20% of the lucky ones that Process Mining really helped. And this is not luck. Building a process model based on real-world data that is updated daily, calculating process indicators and bringing it all to beautiful and convenient analytical concepts is only part of the story. In many projects, they miss something without which no Process Mining will work - data quality . We have done a lot of work with the in-house customer to improve the quality of the data so that our system can not only carry out the analysis, but also prepare regular reports for making important management decisions.
As a result of the project, our understanding of Process Mining has changed somewhat. This is an approach to collecting disparate information about the process and its subsequent in-depth analysis using modern tools. Moreover, the approach involves a continuous and consistent collection, recording and analysis of events from information systems about the target object of research, and its evolution in the process of movement.
Our solution based on Process Mining technology has proven to be useful for a large number of different users involved in the procurement process. Now, within the framework of a unified system, they can deeply analyze these processes, monitor the status of specific purchases, KPIs and, finally, automate reporting. If we talk about numbers, the introduction of Process Mining and the implementation of a set of measures by the financial department made it possible to reduce the time of the procurement process by 25%, while the total number of purchases increased by 3 times.
Celonis has a rich marketplace with paid addons. But we came to the conclusion that it is better to develop your own customized fine-tune tools using the Celonis API in Python. We will share this experience in the following articles.
About the merger of large banks at different levels we can read something else:
- How did VTB come to common knowledge
- How to safely combine the network segments of the three major banks: share tricks
- How to pack three large banks in one site
- Three data centers without seams, or how VTB protects business systems
- Secrets of the resource forest: how we gathered together the IT structure of VTB
Source: https://habr.com/ru/post/452824/
All Articles