Automate it
How many programmers do you need to make a form for a user? No one! For this, it is enough to create a tool with the help of which even housewives will be able to create forms for web solutions.
Imagine a portal with a huge number of different user forms. For example, such as gosuslugi.ru, where there are about 10,000 electronic forms of obtaining public services. Submitted? Now imagine that there are about a thousand people who generate the requirements for creating these forms, that is, customers, about 10 times less than the forms themselves. And finally, imagine how many developers are needed to make about 300 unique, with their cockroaches, forms in a month? About 100! And they must be highly qualified so that they can do everything on their own. Interns here will not cope. And also each form must be analyzed before development, after - tested, given to the customer for acceptance, corrected comments and transferred to the installation. This means that we need more testers and analysts.
Total is about 200 highly skilled people for a month. If we assume that such a team is able to be assembled, which in itself is almost impossible, the cost of such development will be off scale and many times higher than the adequate cost of such work on the market. Obviously, to solve such a problem, you need to automate something. But what?
')
To understand the scale of the disaster, let's see what needs to be done in order to develop and submit to the installation a working form for receiving the service.
In any case, you need to automate something, you say. And you will be right. The first candidate is point 4. We need to make a service that will accept XML as input and send it to the SMEV. OK, done, done. Still? Damn it, point 3! The process of signing is always the same. OK, done another service. At the input XML, at the output signed XML. Cool! More automation, Karl! There are two points left - the creation of a form and the generation of an XML document according to the format of the department. Hmm, here's the problem ...
How can you automate actions that are different each time? After all, the departments of work for the service (form and XML) are written by the departments, and it is not necessary to expect uniformity of requirements in a situation when customers compete with each other.
It was decided that we need a tool that allows:
- as simple as possible to describe the appearance of the form, format-logical control for the fields, logical links between the fields.
- to give the opportunity as simply as possible on the basis of the completed form to generate an XML-document of the desired format.
What will an experienced developer say about such a task? It is necessary to write a framework. We formed the basic requirements for it and ran to code:
- Standardize the appearance of forms, create a set of widgets
- Easy customization, customization of widgets
- Ability to connect to the development of junior developers and interns
- Inheritance by the previously developed forms of all subsequent changes in the core
- Automatic generation of output XML by template
As technology was chosen:
- Database layer: Oracle Database
- Application layer: Java 1.6
- Backend template engine: Freemarker
- Frontend framework: jQuery gaining popularity at that time
The results were not long in coming. Literally a month after the beginning of the development of the framework, a rather simple scheme of work lined up:
What are the main advantages we got? Backend and front-end developers were no longer required to develop the forms themselves. The main thing that was required of a specialist was to know the syntax and capabilities of Freemarker and be able to use the ready-made library. For most forms, these skills were enough. For complex cases, more experienced developers were involved, where it might have been necessary to dig with html, javascript or vice versa with java. The most experienced were engaged in the development of the framework framework, where deep knowledge of front and back technologies was already required.
The core development team consisted of 4-8 people at different times - 2 fronts, 2 back-ups, a tester and a team leader. The minimum service development team was 2 people — half the analyst, the developer, and half the tester. The implementation of the framework allowed the formation of several independent service development teams, and subsequently several dozens of such teams, distributed geographically. Each team could create 1-3 services per month, depending on their complexity, and the time-to-market of each form averaged 35-50 days.
Thus, the productivity of twenty teams (~ 50 people) was about 40 services per month. Not bad. But not enough. The task was 300 per month. It was already not economically profitable to build up the team, it was necessary to have a more optimal technical solution.
Countless developers and analysts spent days and nights in front of their monitors, coding, analyzing, calling customers, then coding again, then analyzing again, then correcting errors, and finally getting a finished service. The number of services that had to be done increased exponentially and the problem became obvious. Some decision was required. The worst enemies of the existing approach were known: complexity, speed (or rather, its absence), the need for technical knowledge in the field of development, as well as the human factor - the more a person participated in the development of a service, the more mistakes could be made, both due to inattention, and because of misunderstanding and “broken phone” when traveling setting a task from the customer to the developer.
So, the task was as follows:
- reduce service development time
- reduce the number of people involved in the design process
- further reduce the requirements for the necessary qualifications of developers
There is an expression - “if you want it to be good, do it yourself”. It is completely wrong from the point of view of project management, but it is precisely this that formed the basis for the next round of development of the form development system.
“Why are developers here at all?” We asked ourselves. And really, what was the developer doing in our case? Collected the page with calls of widgets and specified settings for them. Why not make a user interface for this? As in Delphi or Visual Studio.
Looking ahead, I will say that the results were fantastic, but for this I had to spend some time and effort. We have announced to all development teams that we have stopped the development of the old framework, leaving it on support and started developing a new one.
The difference in terms of technology was the strict separation of the data model and presentation. The backend templating engine was no longer required, the framework application server was a set of REST services, with minimal business logic inside. The main part of the logic was on the client - in the browser.
The framework consisted of two main components: the form designer and the form player. In the form designer, they are created and edited in a visual mode, and in the player are executed - the form itself is drawn, validation rules, relationships between fields work.
The main functions of the constructor:
The main field types (widgets):
The main functions of the player:
The main idea of ​​the new framework is the abstraction from everything except the screen form itself. If you represent him as a person, he will be one of those who say: “Do you have any complaints about buttons?” The architecture principle is very simple: “My task is to display the form, perform all the actions described in the constructor, send data. I don’t care about the rest, especially what happens to this data there. ” Due to this approach, we have achieved amazing flexibility, and were able to use any system counterparties to which intermediate and final requests are sent, could use any authentication system to provide access to the forms, embed the player on any web page of any application in any environment.
Creating a form in the constructor looks like this:
- Create a form
- Choose from the dropdown menu the items we need, place on the form
- In the attribute inspector, we customize the fields as we need with the help of check marks, buttons and lists
- Customize the validation of fields by selecting the desired verification method or a set of methods from the list of available
- Customize the relationship between fields
- We enclose XSLT template for forming the resulting XML document
- Specify details of the recipient system
- The form is ready!
Another huge advantage is the independence of the form description from the execution platform. Due to the fact that the form is stored as a meta description in JSON format, which only describes the composition of the form components and their interrelationship, but does not describe the presentation, it is enough to develop the form once in the constructor and it can be executed on any device for which the form player exists: for example, on iOS or Android, SmartTV and others.
What are the main advantages of the second development of the framework? Developing the forms themselves no longer required developers at all. The form creator was required only to be able to use the graphical interface of the form designer and to be able to write a simple XSLT transform. That was enough for most forms. There were, of course, complex cases, then they turned to the kernel development team, which, upon request, developed a necessary function or widget for a week or two.
Thus, the composition of the service development team was reduced to one person, since no development was essentially required. Analytics with no development skills were able to create forms. In a month, one person was able to make 5-7 forms, and the time-to-market was reduced to ten days. As a result, the performance of the same fifty people amounted to about 300 forms per month. Exactly what is needed!
Here are the testimonials of several representatives of the service development teams after the pilot period of using the new framework:
And this is the review of the framework development team leader:
What is the main insight of this story? I singled out for myself 3 main points.
And what have you noted for yourself? It will be cool to hear in the comments. If the topic is interesting, then I am ready to continue the cycle of articles about the Form Designer.
What else can we talk about:
- On the technological component - the architecture, the tools used, the relationships between the components, the points of expansion and customization.
- On the organization of the work of a large number of people to solve the problem of “developing 10 thousand services”, on creating a community of form development, on training, certificates, interns, students, and so on.
- On the development of functions of the framework and how from a highly specialized tool for developing services it has become a universal internal product of the company for solving various tasks.
10 thousand electronic services
Imagine a portal with a huge number of different user forms. For example, such as gosuslugi.ru, where there are about 10,000 electronic forms of obtaining public services. Submitted? Now imagine that there are about a thousand people who generate the requirements for creating these forms, that is, customers, about 10 times less than the forms themselves. And finally, imagine how many developers are needed to make about 300 unique, with their cockroaches, forms in a month? About 100! And they must be highly qualified so that they can do everything on their own. Interns here will not cope. And also each form must be analyzed before development, after - tested, given to the customer for acceptance, corrected comments and transferred to the installation. This means that we need more testers and analysts.
Total is about 200 highly skilled people for a month. If we assume that such a team is able to be assembled, which in itself is almost impossible, the cost of such development will be off scale and many times higher than the adequate cost of such work on the market. Obviously, to solve such a problem, you need to automate something. But what?
')
What is a service?
To understand the scale of the disaster, let's see what needs to be done in order to develop and submit to the installation a working form for receiving the service.
- Develop a custom form
Portal page (s) where the user fills in the necessary information to apply for a service - Prepare an XML document request to the departmental information system
Before sending to the external information system, the data that the user entered on the form must be converted into a special format XML document. Which - determines the agency, which is the customer development services. - Wrap and digitally sign an XML document
The generated XML-document is not sent directly to the IP department, but to a special bus called SMEV - the system of inter-agency electronic interaction. This system also requires that incoming XML documents have a specific format. Therefore, before sending, we must “wrap up” the XML document created at stage 2 into another XML format, according to the requirements of the SIEV, and then also sign the digital signature of the public services portal. - Send prepared XML document
Only after all these magical actions is the document being sent to the department’s information system through the SMEV.
Automation, Karl!
In any case, you need to automate something, you say. And you will be right. The first candidate is point 4. We need to make a service that will accept XML as input and send it to the SMEV. OK, done, done. Still? Damn it, point 3! The process of signing is always the same. OK, done another service. At the input XML, at the output signed XML. Cool! More automation, Karl! There are two points left - the creation of a form and the generation of an XML document according to the format of the department. Hmm, here's the problem ...
Classics of the genre
How can you automate actions that are different each time? After all, the departments of work for the service (form and XML) are written by the departments, and it is not necessary to expect uniformity of requirements in a situation when customers compete with each other.
It was decided that we need a tool that allows:
- as simple as possible to describe the appearance of the form, format-logical control for the fields, logical links between the fields.
- to give the opportunity as simply as possible on the basis of the completed form to generate an XML-document of the desired format.
What will an experienced developer say about such a task? It is necessary to write a framework. We formed the basic requirements for it and ran to code:
- Standardize the appearance of forms, create a set of widgets
- Easy customization, customization of widgets
- Ability to connect to the development of junior developers and interns
- Inheritance by the previously developed forms of all subsequent changes in the core
- Automatic generation of output XML by template
As technology was chosen:
- Database layer: Oracle Database
- Application layer: Java 1.6
- Backend template engine: Freemarker
- Frontend framework: jQuery gaining popularity at that time
The results were not long in coming. Literally a month after the beginning of the development of the framework, a rather simple scheme of work lined up:
- The analyst receives the TK for the development of the service, validates it and, if necessary, works with the customer, after which it transfers it to the developer.
- A developer using Freemarker sprinkles the necessary widgets onto a page simply by placing in the code a call to the necessary macro from the core library of widgets. When called, it indicates the necessary attributes of the widget: for example, the field code, field binding, validators, etc. After that, the developer, based on the requirements for the output XML format, using the same Freemarker, generates a document template, indicating the field code in the right place.
- The form is sent to testing, debugged and then transferred to the customer for testing. After receiving the OK from the customer, a release is formed with the forms that are currently ready and is transmitted to the installation.
What are the main advantages we got? Backend and front-end developers were no longer required to develop the forms themselves. The main thing that was required of a specialist was to know the syntax and capabilities of Freemarker and be able to use the ready-made library. For most forms, these skills were enough. For complex cases, more experienced developers were involved, where it might have been necessary to dig with html, javascript or vice versa with java. The most experienced were engaged in the development of the framework framework, where deep knowledge of front and back technologies was already required.
The core development team consisted of 4-8 people at different times - 2 fronts, 2 back-ups, a tester and a team leader. The minimum service development team was 2 people — half the analyst, the developer, and half the tester. The implementation of the framework allowed the formation of several independent service development teams, and subsequently several dozens of such teams, distributed geographically. Each team could create 1-3 services per month, depending on their complexity, and the time-to-market of each form averaged 35-50 days.
Thus, the productivity of twenty teams (~ 50 people) was about 40 services per month. Not bad. But not enough. The task was 300 per month. It was already not economically profitable to build up the team, it was necessary to have a more optimal technical solution.
Innovate the way!
Countless developers and analysts spent days and nights in front of their monitors, coding, analyzing, calling customers, then coding again, then analyzing again, then correcting errors, and finally getting a finished service. The number of services that had to be done increased exponentially and the problem became obvious. Some decision was required. The worst enemies of the existing approach were known: complexity, speed (or rather, its absence), the need for technical knowledge in the field of development, as well as the human factor - the more a person participated in the development of a service, the more mistakes could be made, both due to inattention, and because of misunderstanding and “broken phone” when traveling setting a task from the customer to the developer.
So, the task was as follows:
- reduce service development time
- reduce the number of people involved in the design process
- further reduce the requirements for the necessary qualifications of developers
There is an expression - “if you want it to be good, do it yourself”. It is completely wrong from the point of view of project management, but it is precisely this that formed the basis for the next round of development of the form development system.
“Why are developers here at all?” We asked ourselves. And really, what was the developer doing in our case? Collected the page with calls of widgets and specified settings for them. Why not make a user interface for this? As in Delphi or Visual Studio.
Looking ahead, I will say that the results were fantastic, but for this I had to spend some time and effort. We have announced to all development teams that we have stopped the development of the old framework, leaving it on support and started developing a new one.
The difference in terms of technology was the strict separation of the data model and presentation. The backend templating engine was no longer required, the framework application server was a set of REST services, with minimal business logic inside. The main part of the logic was on the client - in the browser.
The framework consisted of two main components: the form designer and the form player. In the form designer, they are created and edited in a visual mode, and in the player are executed - the form itself is drawn, validation rules, relationships between fields work.
The main functions of the constructor:
- Visual form design.
- Setting the format-logical control (validation) of fields.
- Configuring relationships between fields of a form (for example, hide a block or field depending on the value of another field).
- Create and edit directories for use in drop-down lists.
- Save and then use your own widgets.
- Mode preview form.
The main field types (widgets):
- text field
- date entry widget
- radio button, checkbox
- drop-down list
- file upload field
- KLADR or FIAS address input widget
- button
- table
The main functions of the player:
- execution of the form
- fulfillment of validation rules, interrelations between fields, intermediate and final requests to external systems, described by the designer during form development
- ability to embed anywhere on any web page
- custom visual themes
The main idea of ​​the new framework is the abstraction from everything except the screen form itself. If you represent him as a person, he will be one of those who say: “Do you have any complaints about buttons?” The architecture principle is very simple: “My task is to display the form, perform all the actions described in the constructor, send data. I don’t care about the rest, especially what happens to this data there. ” Due to this approach, we have achieved amazing flexibility, and were able to use any system counterparties to which intermediate and final requests are sent, could use any authentication system to provide access to the forms, embed the player on any web page of any application in any environment.
Creating a form in the constructor looks like this:
- Create a form
- Choose from the dropdown menu the items we need, place on the form
- In the attribute inspector, we customize the fields as we need with the help of check marks, buttons and lists
- Customize the validation of fields by selecting the desired verification method or a set of methods from the list of available
- Customize the relationship between fields
- We enclose XSLT template for forming the resulting XML document
- Specify details of the recipient system
- The form is ready!
Another huge advantage is the independence of the form description from the execution platform. Due to the fact that the form is stored as a meta description in JSON format, which only describes the composition of the form components and their interrelationship, but does not describe the presentation, it is enough to develop the form once in the constructor and it can be executed on any device for which the form player exists: for example, on iOS or Android, SmartTV and others.
What are the main advantages of the second development of the framework? Developing the forms themselves no longer required developers at all. The form creator was required only to be able to use the graphical interface of the form designer and to be able to write a simple XSLT transform. That was enough for most forms. There were, of course, complex cases, then they turned to the kernel development team, which, upon request, developed a necessary function or widget for a week or two.
Thus, the composition of the service development team was reduced to one person, since no development was essentially required. Analytics with no development skills were able to create forms. In a month, one person was able to make 5-7 forms, and the time-to-market was reduced to ten days. As a result, the performance of the same fifty people amounted to about 300 forms per month. Exactly what is needed!
Here are the testimonials of several representatives of the service development teams after the pilot period of using the new framework:
Thank you very much for your help with the constructor, we managed to close a big contract with it. In fact, more than 120 forms were implemented in a little over a week, and about 110 forms were put into production.
With the use of the new platform, the development goes much faster, the analyst can completely control the whole process and make edits very quickly. The analyst can do it himself, almost 90-100%
I want to say thanks to the developers - how much speeds up the process! I have 4 forms per day. With our deadlines for the Khanty-Mansi Autonomous Okrug, it would be impossible to do it manually with one developer
And this is the review of the framework development team leader:
The project was remembered by magic, or rather the feeling that you are a magician. At the very least, the reaction of many people to whom we demonstrated the product was similar to the reaction to the focus. It is very nice when you do it with the help of the tool you made to surprise the skeptic. Immediately you realize that you and your team are doing the right things.
Performance indicators
| Indicator | Old framework | New framework |
| Kernel development team size, man | 4-8 | 4-8 |
| Size of service development team, person | 2-3 | one |
| Team Forms Performance / Month | 2-3 | 5-7 |
| Average development time of one form, days | 25 | four |
| Time-to-market, days | 35-40 | ten |
| Average number of forms / month with 50 people | 40 | 300 |
Insight
What is the main insight of this story? I singled out for myself 3 main points.
- In a commercial company, any innovation, technological, organizational or some other should be economically in demand and justified.
- Making the task “in the forehead” in a rapidly developing IT world means instantly losing competitiveness. Most often because of the overestimated cost and / or timing.
- It is necessary to “sharpen a saw” (S.Kovi “7 Skills of highly effective people”). If you are developing tools, you have a competitive and technological advantage in the market.
And what have you noted for yourself? It will be cool to hear in the comments. If the topic is interesting, then I am ready to continue the cycle of articles about the Form Designer.
What else can we talk about:
- On the technological component - the architecture, the tools used, the relationships between the components, the points of expansion and customization.
- On the organization of the work of a large number of people to solve the problem of “developing 10 thousand services”, on creating a community of form development, on training, certificates, interns, students, and so on.
- On the development of functions of the framework and how from a highly specialized tool for developing services it has become a universal internal product of the company for solving various tasks.
Source: https://habr.com/ru/post/263391/
All Articles