The history of a heavy project: a little about the bureaucracy, infrastructure and software development process

The history of a heavy project: a little about the bureaucracy, infrastructure and software development process

Any small project wants to grow into Facebook in something significant. This does not always happen, but if it does happen, a whole host of problems and tasks arise that would not seem to exist when there were five developers in the project, but who require considerable attention after the project has become fully functional. Below is a summary of the history of a relatively large project, thoughts about bureaucracy, as well as a description of the infrastructure and the development process - I think this experience will be useful to someone.



The customer is a fairly large investment bank. The number of end users: more than 10 thousand.



The project team

• more than 200 developers (31 of them are leaders, 7 are master timlids)
• 3 architects, one of them is in charge
• 19 testers (1 lead, 2 on load tests, the rest on functional testing)
• 5 system administrators whose responsibility is DBMS management and higher. Administering the OS and hardware in the area of ​​responsibility of the special HW team
• a variable number (from 2 to 16) of analysts working on part time
• 12 technical writers and translators
• 3 project managers, constantly managing one - second and third substitute during vacations, illness, or acting as an assistant project manager when active and accessible.

All employees work in geographically separated offices. Analysts in London, New York and Frankfurt am Main. Development centers are in Eastern Europe, but a significant part of the developers are de facto working remotely, so that in the project, in addition to the Hungarians, there are Bulgarians, Russians, as well as Ukrainians and Belarusians.

')

What is the developed system and a brief history

In essence, this is a kind of mix of CRM, OSS for financial organizations and specialized software for investment banking. The project began more than 15 years ago and was originally developed in Perl, the rest of the code on which is still (as of 2013) present in the system. When it became clear that you won’t get far away on Perl, a team was formed inside the IT block, which slowly began to rewrite the existing functionality to a new platform (Java). The quiet work did not last long - as a result of a smaller financial customer's acquisition of a financial institution, a system for PHP was “organically” connected, which was, simply, the web interface of the end users, while the back-end was a broker that manually processed submitted applications. It was originally planned to completely rewrite the PHP part, since it was relatively small.



Unfortunately, according to such a crucial factor as the time to bring the functionality to production (go to market), the Java team was losing to the PHP team, despite the almost two-fold advantage of the first in number, and almost 3-fold advantage in the payroll. As a result, all new products first appeared on PHP, and only after a certain time did the Java command reproduce, sometimes only partially, this functionality on J2EE. Let us leave aside the reasons for this phenomenon, but taking this opportunity to say that in my subjective opinion, the Java team was completely satisfied with this situation, which was deprived of the “happiness” of communicating with the end customer, the requirements were reached in a fairly clean and refined form, and Damocles the sword of time over it practically did not hang.

About attempts to avoid internal development

Attempts were made periodically to get away from in-house development, transferring the project to an external integrator. Tenders were held, teams were looked at, even a talk was started about the development of a competence center specifically for the bank, but each time something did not grow together. The history of the failure of the last attempt was particularly epic - it was decided to choose a query, which was a hodgepodge of small improvements, independent of each other, as a litmus test. One of these improvements was to add the ability for authorized users to add only visible comments to the news. The architect and the seller from the eminent integrator cheerfully painted the plan and rolled out the cost of refinement. On the contrary, improvements with commenting on the news were labor intensive - 8 man-hours. Now it is not known what fate this estimate came to the author of the request to Madrid, but literally the next day a uniform dressing was arranged at the video communication conference, where the son of some Spanish employee live made the functionality in literally 10 minutes. It is clear that the boy used ready-made pieces of SQL and Java code written by him, but a task for which you need one table in the database and one page of code didn’t pull for 8 hours. Management feels like a maximum of half an hour.

It is quite natural that neither the vendors nor the architect of the integrator could substantiate the almost 16-fold difference in the assessment of labor costs, neither the charisma of the main vendor, nor the reference to Brooks with his evolution of the system software helped. The idea of ​​internal development, continued to live and win.

Product or not product?

Attempts to lead the project correctly, to get a product manager, to make a clear plan for releases also failed. To IT as a whole and to the project in particular, the business had a purely utilitarian attitude - IT services the business. There is security, there are services for office cleaning and delivery. And there is an IT service that asks for money not as an example anymore, but which should not dictate conditions to the business - you cannot imagine a cleaner who invades the meeting room and asks everyone to go out for half an hour under the pretext of the necessary cleaning? So how is IT better? So all regional offices were allocated budgets for the implementation of new features of the system, which was an easy shock for the product manager: so, there is a line - ordering stationery, there is a line - delivering coffee \ tea \ water to the office and there is a budget line, requests for changes to his product? Naturally, practically no centralization of requirements was meant by such a model - if a unit has money left on the budget line, IT must implement this functionality. As for the relevance of the requirements, the local business knows better what reports or opportunities they need.

Process flow

Over several years, intrigues, optimizations, and the inclusion of common sense have resulted in the process of organizing work in a scheme, a simplified version that I bring to your attention.



Members

End User - the end user of the system

Manager - Project Manager

Tester - Tester

Architect - Architect

Analyst - analyst

Application Developer - developer

Code reviwer - lead developer responsible for code review







1. The end user applies for new functionality through the BPM system. Now this is a self-written solution, the option of migration to Oracle BPM is being considered.

2. The project manager accepts the application, filtering unrealizable / duplicating.

3. The project manager sends the application to the analyst / architect, again through the BPM system.

4a. The analyst / architect formulates the task in terms of first the business / functional requirements, then in the form of a private technical assignment. The text of the task goes to Source Control (Git), the task in Jira and BPM, the requirements are traced (Rational ClearCase).

4b. The architect predicts the deadline for completing the task and adds the information to the project management system (Microsoft Project Server + improvements).

4c. The project management system informs the end user about the projected timing and cost of the task.

5. The developer takes the latest version of the source codes, implements the implementation of the functional.

6. The developer places the code on code review (via Gerrit).

7. CI system (Hudson) picks up the code, conducts a test assembly.

8. The test build report goes to the reporting system (based on Microsoft SQL Server Analysis Services). The person responsible for the code review is sent a notice of the need to conduct a code analysis.

9. Code reviwer gives the go-ahead for the changes made.

10. The code is transferred to the main branch of development.

11. The CI system picks up the code from the main branch, does the assembly.

12. Report on which is laid out in the reporting system.

13. CI system produces deploy code on the test environment.

14. The CI system calls functional and load testing systems.

15. Test systems in the automatic mode are carried out on the test environment of the testing procedure (JMeter, Selenium, RationalRobot).

16a. Report on which is laid out in the reporting system. The tester is notified of the need to conduct tests.

16b. Testing systems change task status in a project management system.

17. The tester performs testing.

18. If successful, the status changes in the BPM system, as

19. The project manager receives a notice.

20a. The BPM system initiates the transfer of the assembly to the pre prod environment.

20b. Mark about this event is placed in the project management system.

20c. About what the end user is notified, who can see the functionality on the pre prod environment.

21. The project manager and the end user check the functionality on the pre prod environment.

22. In the event that a decision is made to transfer the functionality to the production unit, the project manager through the BPM system initiates the task to deploy to a productive environment.

In the background

1. The automatic environment setting system (Puppet) aligns the landscape in terms of the basic software settings.

2. The monitoring and status tracking system (Zabbix) monitors the infrastructure, including not only dev, test, pre-prod and prod, but also other infrastructure, including gerrit, Git, Hudson, etc.

3. The backup system, in addition to backup project repositories and infrastructure software, backs up images of pre-prod landscape machines in case of force majeure or the need to demonstrate previous system releases.

4. Once a day, a static code analysis is performed, primarily to search for vulnerabilities.



The above scheme carries a certain touch of gigantomania and idealism - RP often neglects its responsibilities to view the new functionality on pre prod, Code reviwer confirms the code from familiar trusted developers, etc. without looking. but in general, this scheme is quite digestible for really big projects. And, the most remarkable thing is that you can get a scheme for another project out of it simply by cutting off unnecessary, clearly redundant parts.

Dry conclusions, denounced in the advice that the author made after several years of participation in the project

1. Everyone who is in IT. Relations IT business in Russia and Western Europe are different, sometimes dramatically. "They have" IT, it is a servant, in the good sense of the word. Allow and privileged and highly paid, but does not pretend to dominate the role and do not swing the right to the board of directors. We are seriously at the CIO forums discussing why CIO is Career Is Over, how to deal with it, and why CEOs often get from the position of financial director, but almost never from the position of IT director. The position of IT in the corporate hierarchy, somewhere between AXO and accounting. Humble, it will be so in Russia.
2. Advice to novice architects. "For large projects, there is no properly selected architecture." It is impossible to design without crutches something that will develop over a couple of years. All you can think of is loose coupling and a reasonable level of abstraction.
3. Advice to developers. "Then there will not be." No one for serious systems will not give the opportunity to "rewrite correctly." Business does not understand this, understand you. In the end, you do not baptize children with this code.
4. Advice to project managers. In the classical triangle, quality-time-budget, the budget is usually fixed, the quality category is quite abstract for top management (they do not give functionality, so quality is quality), so instead of a triangle, get ready to fight only on one front - time front . Just consider this when planning.
5. There are no rules, forget what is written above.