Everything you wanted to know about BPM, but were afraid to ask
There are many publications on the net about why BPM (Business Process Management) should be implemented in your company. As usual, the benefits that BPM implementation gives to business are formulated:
This is not all that can be mentioned, but a rather typical set of platform advantages.
Is it really so cloudless? Isn't it time for everyone to throw away the old tools and completely switch to the new platform?
I would like to talk about what problems you can expect during the implementation process, or rather, when developing real-world applications. My impressions are based on experience with the IBM Websphere BPM platform, for definiteness, from versions 7.5 to 8.5 inclusive.
')
Consider the above items in order.
What are we trying to achieve? So that analysts, or even business users, can draw a process diagram that will then be executed in exactly the same way by the server. Or at least they could understand the essence of the process by looking at a diagram drawn by someone else.
Do we get what we wanted to achieve? More likely no than yes.
What is the root of the problem? I would say that there are two of them:
The fact is that the process that is carried out within the framework of the BPM system almost never boils down only to the diagram that a business user or analyst sees. To it is still attached a lot of details lying at different levels of detail, which somehow determine the behavior of the process. This, for example, is the data accompanying the process instance, or the integration of the process with other existing systems. All these details are usually reduced to code in another programming language.
Practice shows that without these components the process is practically useless, and at the same time, they are not visible on a typical BPMN diagram.
On the other hand, business processes are software, albeit somewhat unusual. Processes have a rather long development and maintenance cycle, and the convenience of maintenance rather quickly becomes important. Also, it is very important to manage changes in the application in general, and in close connection with the so-called issue tracking systems (for example, JIRA). That is, we need to know who, when, what exactly and why made changes to the process, and within what tasks.
Thus, on the one hand, business users have rather a false impression that they understand the logic of the process, part of which is hidden from them in the code. From the developers' point of view, the presentation in the form of a diagram only complicates development, primarily because the diagrams do not agree well with the development life cycle, including change management.
Now let's talk about the methods and tools for the developer.
If for traditional development tools more or less tools are invented (for example, refactoring), metrics and measurement methods that allow to estimate the size of a project (in code lines, usually), and such factors as test coverage, number of potential errors per code unit, complexity code, and much more, then you can forget about a lot of this when moving to BPM. In any case, I am not aware of any work in this direction. In a sense, this rejects development many years ago. You need to re-learn to refactor, collect project statistics, and learn to evaluate their complexity and size in order to plan development correctly.
So, tools for code analysis and automatic refactoring do not exist in practice. Worst of all, the very creation of such tools is a big question. Imagine a fairly typical development scenario on a more traditional platform. You have a versioning system, say SVN or Git, a development environment (IDE), and other tools. So, if your favorite IDE is not able to do something, you can, as a rule, either develop an extension for it or use third-party tools to work directly with the code. For example, a copy & paste detector, or any other tool, many examples of which have been known to us since the advent of UNIX. This is possible because your program code is traditionally just text. You can use any tool, and transfer the modified code back to Git, or just collect the necessary statistics, analyze the code, etc.
In the case of BPM, this approach is not available to you, or is very limited. It is impossible to automate many routine operations, and there is not always access to develop third-party automation tools. Access to the process code is possible, but rather theoretically - you can export the process as a set of BPMN schemes, and use various tools for analyzing the code - provided that you develop them yourself.
As a result, such a trivial task on traditional platforms, like a uniform change in the properties of many elements of a process diagram instead of something like Find & Replace, results in a dull routine work that can be done only with your hands and which makes you hate this tool in five minutes.
And last but not least. If for more traditional projects you are able to estimate the size of the application being created (in lines of code, for example) informally, or according to some model of the CoCoMo type, then in the case of BPM you find yourself in an incomprehensible situation. You have at least two types of code - diagrams, and a “normal” programming language, maybe not even one. How long does it take to draw a process diagram? What does it depend on? What is the size and complexity of diagrams? And if for ordinary code you can apply known models to estimate labor intensity, then such models are not constructed for diagrams and the questions have no answers.
Some necessary components for building processes: conditions, parallel branches, expectations, events of course exist. Only here their level is rather low, and the possibilities of combining, and as a result, the reuse of ready-made components, leave much to be desired.
In the process of evolution of development tools and programming languages, ways of creating reusable components have long been developed and more or less settled. Basically, they are reduced to data types and functions on these types, as well as to the means of composition of these basic elements into a complete application.
Try, for example, to organize a cycle in the process, using only the standard set of tools. It’s as if you’re going back to a time when you were still drawing flowcharts in programming books — that is, 40 years ago. Another very typical, but far from trivial, case is when you need to organize asynchronous interaction with an external service for you, that is, send him a request, and then wait for a response. Even if you can implement the logic you need, you will hardly be able to reuse it in another similar process, because the implementation will be scattered throughout the application in such a way that it will be impossible to allocate a reusable component. Moreover, the root of evil in this case is trivial and lies on the surface: BPMN does not contain what is called “higher order functions”, if we speak in terms of the OP. Or generics, if you remember OOP and Java. You cannot write a generic component, for example, to sort a list, abstracting from the type of the list item. You cannot pass functions (active components) as parameters. There is no way to describe a metacomponent if you can afford to call it that.
At present, two paradigms in the form of functional programming and OOP can be considered as predominant. In the case of BPM, there is no such development, and the current state of the theory is, frankly, deplorable. In fact, there is not even the paradigm itself - there are only some practices that can be recommended only with limitations. Moreover, if we say functional programming is based on mathematics, and in some cases the correctness of the program can be proved formally, and in the case of OOP and procedural programming, it is sometimes possible to prove the correctness at least partially, in the case of BPM I have not heard of such work.
Not defined and approaches to refactoring and testing processes. There is not even the concept of refactoring. Everything looks as if the developers of the tool believe that the process is built once and for all, and never develops. And there are no mistakes, technical duty, smells and other things in it.
Moreover, since the process is often a set of parallel activities, and cannot be performed outside the BPM system, writing tests is also an unsolved task today. The means available for this cannot be considered adequate, despite the fact that only services can be tested today - i.e. synchronous components of the process, for testing the processes themselves as BPMN diagrams (especially with parallelism), the author does not know any means (although they can be developed, there are enough APIs for this, but this is not an easy task).
All programs are developing. If the program does not develop - it is most likely that no one needs the program. There are exceptions - but they are very rare. This means that the BPM process can and should evolve, and go through several versions. These changes need to be monitored. Sometimes we develop two application branches — say, a bugfix for version 1, and a new version 2. Sometimes you need to merge changes into them into one version.
Why is it all bad in the case of BPM?
First, the changes are poorly foreseeable. The initial view of the diagram is not the text, but the picture. Squares, arrows, diamonds. And in comparison with the usual code in the form of text, which is usually one-dimensional, and for which all operations like diff / merge / patch, etc., have long been defined, several new dimensions are added here. For example, the color of coloring squares, or their relative position on the sheet. You get a significant level of information noise, simply because part of the changes in the diagram absolutely does not affect its work.
As a result, we do not have such a simple and obvious possibility, which since the time of unix is ​​called the word diff. You cannot simply and automatically compare two different versions of a process to each other, at different levels of detail (ie, an application, a process diagram, a component, a part of a diagram, a line of code in a language).
At best, you know that such and such a number such a user has changed a certain diagram. What exactly, why, and what the change actually was - you cannot find out, or it is difficult. All that is available is the ability to open (for reading), and see how a particular version looked. However, this process is not fast. And the most significant - you see not the difference, you see either one or another version of the code. To compare and understand where there are differences is your task, and it turns out to be completely non-trivial.
And the second usual possibility is merge or at least patch. It is also absent de facto. You cannot transfer changes from one branch to another, from one version to another, including selectively, saving changes made in both branches. In practice, it deprives the branches of any meaning.
Accordingly, when we talk about business process flexibility, we should always keep in mind the problem just discussed. We can change the process as flexibly as we like — but very rarely will we know for sure exactly what has changed. And when something breaks down - the search for the cause can result in lengthy excavations in the code.
So, briefly summarize again some of the problems:
In other words, business processes in the form of BPMN diagrams are fairly flexible and easy to understand for, say, analysts, but the diagrams are very limited as a development tool.
It so happened that in the current project I had to deal with a very similar product. This is MS Business Intelligence Development Studio, and what they are developing in it is SQL Server Integration Services. And what’s very characteristic is that this completely different product, made by another company for other tasks, tries to achieve the same goals in some way, and faces exactly the same problems.
What conclusion I made for myself - I think it is quite understandable. I will stay away from such tools, whenever possible. If you have other impressions about practical work with BPM, or say SSIS, I suggest to talk about them in the comments, and I hope that the discussion will be useful for us.
- Visual modeling and execution of business processes.
- A set of ready-made components for building flexible business processes.
- User interaction to perform manual actions.
- Flexibility to configure business processes.
- Support for versioned business processes.
This is not all that can be mentioned, but a rather typical set of platform advantages.
Is it really so cloudless? Isn't it time for everyone to throw away the old tools and completely switch to the new platform?
I would like to talk about what problems you can expect during the implementation process, or rather, when developing real-world applications. My impressions are based on experience with the IBM Websphere BPM platform, for definiteness, from versions 7.5 to 8.5 inclusive.
')
Consider the above items in order.
Visual modeling
What are we trying to achieve? So that analysts, or even business users, can draw a process diagram that will then be executed in exactly the same way by the server. Or at least they could understand the essence of the process by looking at a diagram drawn by someone else.
Do we get what we wanted to achieve? More likely no than yes.
What is the root of the problem? I would say that there are two of them:
- Proposed process modeling languages ​​are too primitive for real-world tasks, and other languages ​​are required to create a real workflow.
- A language in the form of diagrams, like BPMN, doesn’t fit well into a typical program development cycle.
The fact is that the process that is carried out within the framework of the BPM system almost never boils down only to the diagram that a business user or analyst sees. To it is still attached a lot of details lying at different levels of detail, which somehow determine the behavior of the process. This, for example, is the data accompanying the process instance, or the integration of the process with other existing systems. All these details are usually reduced to code in another programming language.
Practice shows that without these components the process is practically useless, and at the same time, they are not visible on a typical BPMN diagram.
On the other hand, business processes are software, albeit somewhat unusual. Processes have a rather long development and maintenance cycle, and the convenience of maintenance rather quickly becomes important. Also, it is very important to manage changes in the application in general, and in close connection with the so-called issue tracking systems (for example, JIRA). That is, we need to know who, when, what exactly and why made changes to the process, and within what tasks.
Thus, on the one hand, business users have rather a false impression that they understand the logic of the process, part of which is hidden from them in the code. From the developers' point of view, the presentation in the form of a diagram only complicates development, primarily because the diagrams do not agree well with the development life cycle, including change management.
Tools
Now let's talk about the methods and tools for the developer.
If for traditional development tools more or less tools are invented (for example, refactoring), metrics and measurement methods that allow to estimate the size of a project (in code lines, usually), and such factors as test coverage, number of potential errors per code unit, complexity code, and much more, then you can forget about a lot of this when moving to BPM. In any case, I am not aware of any work in this direction. In a sense, this rejects development many years ago. You need to re-learn to refactor, collect project statistics, and learn to evaluate their complexity and size in order to plan development correctly.
So, tools for code analysis and automatic refactoring do not exist in practice. Worst of all, the very creation of such tools is a big question. Imagine a fairly typical development scenario on a more traditional platform. You have a versioning system, say SVN or Git, a development environment (IDE), and other tools. So, if your favorite IDE is not able to do something, you can, as a rule, either develop an extension for it or use third-party tools to work directly with the code. For example, a copy & paste detector, or any other tool, many examples of which have been known to us since the advent of UNIX. This is possible because your program code is traditionally just text. You can use any tool, and transfer the modified code back to Git, or just collect the necessary statistics, analyze the code, etc.
In the case of BPM, this approach is not available to you, or is very limited. It is impossible to automate many routine operations, and there is not always access to develop third-party automation tools. Access to the process code is possible, but rather theoretically - you can export the process as a set of BPMN schemes, and use various tools for analyzing the code - provided that you develop them yourself.
As a result, such a trivial task on traditional platforms, like a uniform change in the properties of many elements of a process diagram instead of something like Find & Replace, results in a dull routine work that can be done only with your hands and which makes you hate this tool in five minutes.
And last but not least. If for more traditional projects you are able to estimate the size of the application being created (in lines of code, for example) informally, or according to some model of the CoCoMo type, then in the case of BPM you find yourself in an incomprehensible situation. You have at least two types of code - diagrams, and a “normal” programming language, maybe not even one. How long does it take to draw a process diagram? What does it depend on? What is the size and complexity of diagrams? And if for ordinary code you can apply known models to estimate labor intensity, then such models are not constructed for diagrams and the questions have no answers.
A set of ready-made components and reuse
Some necessary components for building processes: conditions, parallel branches, expectations, events of course exist. Only here their level is rather low, and the possibilities of combining, and as a result, the reuse of ready-made components, leave much to be desired.
In the process of evolution of development tools and programming languages, ways of creating reusable components have long been developed and more or less settled. Basically, they are reduced to data types and functions on these types, as well as to the means of composition of these basic elements into a complete application.
Try, for example, to organize a cycle in the process, using only the standard set of tools. It’s as if you’re going back to a time when you were still drawing flowcharts in programming books — that is, 40 years ago. Another very typical, but far from trivial, case is when you need to organize asynchronous interaction with an external service for you, that is, send him a request, and then wait for a response. Even if you can implement the logic you need, you will hardly be able to reuse it in another similar process, because the implementation will be scattered throughout the application in such a way that it will be impossible to allocate a reusable component. Moreover, the root of evil in this case is trivial and lies on the surface: BPMN does not contain what is called “higher order functions”, if we speak in terms of the OP. Or generics, if you remember OOP and Java. You cannot write a generic component, for example, to sort a list, abstracting from the type of the list item. You cannot pass functions (active components) as parameters. There is no way to describe a metacomponent if you can afford to call it that.
At present, two paradigms in the form of functional programming and OOP can be considered as predominant. In the case of BPM, there is no such development, and the current state of the theory is, frankly, deplorable. In fact, there is not even the paradigm itself - there are only some practices that can be recommended only with limitations. Moreover, if we say functional programming is based on mathematics, and in some cases the correctness of the program can be proved formally, and in the case of OOP and procedural programming, it is sometimes possible to prove the correctness at least partially, in the case of BPM I have not heard of such work.
Not defined and approaches to refactoring and testing processes. There is not even the concept of refactoring. Everything looks as if the developers of the tool believe that the process is built once and for all, and never develops. And there are no mistakes, technical duty, smells and other things in it.
Moreover, since the process is often a set of parallel activities, and cannot be performed outside the BPM system, writing tests is also an unsolved task today. The means available for this cannot be considered adequate, despite the fact that only services can be tested today - i.e. synchronous components of the process, for testing the processes themselves as BPMN diagrams (especially with parallelism), the author does not know any means (although they can be developed, there are enough APIs for this, but this is not an easy task).
Process version support
All programs are developing. If the program does not develop - it is most likely that no one needs the program. There are exceptions - but they are very rare. This means that the BPM process can and should evolve, and go through several versions. These changes need to be monitored. Sometimes we develop two application branches — say, a bugfix for version 1, and a new version 2. Sometimes you need to merge changes into them into one version.
Why is it all bad in the case of BPM?
First, the changes are poorly foreseeable. The initial view of the diagram is not the text, but the picture. Squares, arrows, diamonds. And in comparison with the usual code in the form of text, which is usually one-dimensional, and for which all operations like diff / merge / patch, etc., have long been defined, several new dimensions are added here. For example, the color of coloring squares, or their relative position on the sheet. You get a significant level of information noise, simply because part of the changes in the diagram absolutely does not affect its work.
As a result, we do not have such a simple and obvious possibility, which since the time of unix is ​​called the word diff. You cannot simply and automatically compare two different versions of a process to each other, at different levels of detail (ie, an application, a process diagram, a component, a part of a diagram, a line of code in a language).
At best, you know that such and such a number such a user has changed a certain diagram. What exactly, why, and what the change actually was - you cannot find out, or it is difficult. All that is available is the ability to open (for reading), and see how a particular version looked. However, this process is not fast. And the most significant - you see not the difference, you see either one or another version of the code. To compare and understand where there are differences is your task, and it turns out to be completely non-trivial.
And the second usual possibility is merge or at least patch. It is also absent de facto. You cannot transfer changes from one branch to another, from one version to another, including selectively, saving changes made in both branches. In practice, it deprives the branches of any meaning.
Accordingly, when we talk about business process flexibility, we should always keep in mind the problem just discussed. We can change the process as flexibly as we like — but very rarely will we know for sure exactly what has changed. And when something breaks down - the search for the cause can result in lengthy excavations in the code.
So, briefly summarize again some of the problems:
- Visual modeling and execution of business processes - in part. The model does not reflect the reality for the analyst, and is inconvenient for the developer.
- A set of ready-made components for building flexible business processes is not. Component creation tools are inadequate.
- Support for versioned business processes - in part. Means are inadequate.
In other words, business processes in the form of BPMN diagrams are fairly flexible and easy to understand for, say, analysts, but the diagrams are very limited as a development tool.
What about others?
It so happened that in the current project I had to deal with a very similar product. This is MS Business Intelligence Development Studio, and what they are developing in it is SQL Server Integration Services. And what’s very characteristic is that this completely different product, made by another company for other tasks, tries to achieve the same goals in some way, and faces exactly the same problems.
- Visual modeling of ETL processes works relatively well as long as you write them from scratch. As soon as you try to move from version to version, familiar problems are the same as for BPMN.
- It is also bad with search and navigation within the project.
- Create reusable components in the same rudimentary state. Reuse is close to zero.
- Everything is also bad with refactoring, using third-party tools for working on the project.
What conclusion I made for myself - I think it is quite understandable. I will stay away from such tools, whenever possible. If you have other impressions about practical work with BPM, or say SSIS, I suggest to talk about them in the comments, and I hope that the discussion will be useful for us.
Source: https://habr.com/ru/post/304628/
All Articles