Tales about the harsh Russian IT and digitalization victims

Russia is irrational. There are the right practices, there are a hundred times the rake felt, but still, something epic happens with enviable constancy. Sometimes for the reason: “Well, it’ll certainly blow me over,” sometimes: “Always did, and worked,” sometimes just because of mistakes. Perhaps in the genes.

The first vivid example of incredible game (details are slightly changed at the request of the security guards). The customer is engaged in capital construction. I ordered a system several years ago from a contractor that manages all this (in particular, estimated work). The system was installed on a dozen rather large facilities, and was introduced. Suddenly, the customer decided to ask him to give him the source code. As it turned out, their existing contractor had plans for them to develop the software and then sell the result as SaaS in the market. The contract says nothing about the code. We had a fight.
')
When they called us to understand, there were about 10 different software versions (releases from 0.9 to 2.4). There are 1.5 sources, this version was collected once from them. No documentation. And the system needs to be developed and developed. They counted “rewrite everything again” and “modify 1.5” and settled on the second - TtM three to four months against the year. They taught us how to collect support specialists, corrected the sources, reduced the codebases, made the infrastructure, organized one "casting", where the source is received, collected and distributed there. It cost us and the customer a lot of hemorrhoids.

Come in, I'll show you something more about how you can make a mistake with the development process, and what interesting consequences this leads to.

Another example

The customer - also a rather big company - rolls ERP releases. And the joke is that the system is so healthy that there is nowhere to test it on a full base or something similar. There is simply no infrastructure. More precisely, there is, but it is impossible to do a load test, only small local things are checked. The release rises - and no one knows how he will behave in practice. Once, everything has notably fallen so much when the release did not behave as it wanted. In the end, they invited us to see what can be done. We discussed that they want to see the HP Performance Center, made a pilot, then integrated, trained, delivered. Now releases through it. These normalized operations are tested, a summary of operations SLA.

Or, the state customer has had import substitution. Business comes to us and says: our IT specialists told us that it is very difficult to replace the Oracle base with the Post-Congress. We do not believe them, consult. Two weeks of proceedings - and the result: “Well, yes, changing the base is easy. Just then you have to rewrite the entire application level. Little. Approximately 90%. You have huge packages, you need to transfer the layer of business logic - and come. The developers who wrote the kernel can no longer be found, because the system is eight years old. " They believed the IT team. It turned out that they simply argued insufficiently clearly.

We look somewhere safety, here is an epic example . Fortunately, this one is still simple, there simply no one has done anything for five years, and the business is not of a federal scale.

The following example. We put the same story with the acceptance of releases. Ideal situation: a third-party contractor writes the code, passes it for testing, passes the tests, puts it all into the repository, and from there it is assembled and distributed. Handsomely? Handsomely. We have been living in this company for three years now without exception (before that, there were not all departments and teams). But the customer has a “fund of algorithms and programs”. There, each artist ships a blank or a listing of the source code. They smolder there. As it turned out during the audit, anime was recorded on one disc in general. And even if there is a de facto code in the fund, then there is no sense.

Another similar customer. They have a typical pain - many contractors. They made an intra-industry integrator, they check that the contractor brings the correct software. There were problems in that there are rights of third parties (open source libraries with viral open licenses, for example) - unscrupulous contractors can deliver all this without licensing in order. In the case of open source, you can still cope with the problem somehow, but sometimes commercial libraries come across. Guilty of who will be - guess three times. Then one of their contractors also went bankrupt, and the customer lived with this code. Such situations must be caught at an early stage. We helped set up the process correctly. We have a solution such as automation of algorithms and programs. Technical documentation, versions, source codes, contracts and all links to tenders. With an average CIO life of two to three years, this really helps the next one to figure it out right away.

We are also introducing agile in Russia. I don’t laugh at the circus now. Almost every time it begins as a fashionable story on business digitalization. The main thing is that everyone is trying to understand what these words are. But no one understands. Concepts order, hire strange people. They say the words “it seems”, “hypothesis”, startups are invited, acceleration is muddied, smoothies are drunk - in general, it all has the outward signs of some kind of valley. Then Agile begins to apply, but it does not take off. If you make a serious system, you need to check it for a long time. If you don’t put in processes, then sprints will be long (a month or two), if you set up processes, then you need to start with the testing infrastructure, devoops, make a delivery pipeline, work processes between teams and within teams. And all this is usually forgotten. If the processes of the Old Believers are 98%, that the project will not be done. In the end, then we rake and run. In general, we do not complain: bread too. But sometimes I just want to somehow explain, or what, that IT is first an infrastructure, and then fast TtM. And not vice versa.

What usually goes wrong? Set of examples

My colleagues and I have gathered here such a set of vivid problems that are not very objective, but very well describing the situation. Of course, we only go to places where it’s bad, and it’s not necessary to extrapolate it to the state of the industry. But still, I am sure you will learn something from your company. A small part. In general, everything is described by the words: "process inefficiency, unmotivated employees, poor tools or poorly used tools." Well, now - examples.

1. Penalty for errors in software contributed by developers.

I don’t even know how to rationally describe it. Just a fine for identified bugs. Live now with this knowledge. Naturally, any release (even small) is released ten times slower than it could.

2. Meetings from morning to evening.

The developer must attend them. He is silent and nods his head into the phone. These are the very meetings when the whole project team is needed at the meeting, plus the department head to control. It is impossible not to come. But there is almost no sense in participating. This is a traditional mistake of the project manager, called "excessive control."

3. Cargo cult. We take flexible methodologies and implement them rigidly.

The best I've seen: implemented aggile, but worked as before. They just made developers come to the daily stand-up. They are built and tell: I did nothing. It happens every day.

4. Tools: we implement to report that implemented.

“We have a Continious Integration server, and only an administrator can add a task.” “We have implemented the storage of binary assemblies, and there is a very small disk, you need to delete the old ones, and there are only the last three versions.” Or here: a task management system in an ex-file on a shared disk. So backlog often lead even in large companies, despite the fact that there is Jira. In this case, until the task falls into this file, no one will do it. He is official.

Another company has an internal knowledge base, but everything is stored directly in the version control system: it’s more convenient for the manager. Operating system distributions even add there. When there is no person responsible for the version control system, managers can put gigabyte files to transfer them to the counterparty, because in Dropbox the place has run out.

5. Coding standards: written without understanding or not updated for 10 years.

Just a couple of examples: we require 100% code coverage with tests and documentation. In particular, all third-party libraries. The downside is the lack of standard tests. Recently I saw how an engineer deployed the system, and the user cannot log in, because the key from the test to the prod was not changed.

Once again I saw how the leader wrote the code in Notepad.exe, then threw it into the compiler without errors. He was still studying with punch cards. Such a skill certainly deserves respect. But only until this professional deformation begins to influence the standards for the rest of the development department.

6. Errors of regulations.

For example, a fixed lunch time, which is fully walked. This is a symptom. I ask why. They explain: if you are sitting in the workplace at lunch, then you are the target. Must respond to letters in five minutes and so on.

Excessive bureaucratization: often following documented procedures that leads to a pile of paper. The same test plans for every sneeze. And this is a description of each filter, including data types in each interface field, field length, and so on. This is usually solved by examples, and not by full detail.

In one company there was no person with responsibility for the current release, sometimes the deadlines for releasing products for two weeks were broken.

Communications: something comes from the customer in the mail. Moreover, he writes to the one who last spoke with him. Even if you want to do this, comments on the task can be in different places, including instant messengers. Then someone left, someone came, someone deleted the mail - that's all.

7. Fighting the security guards.

This includes manual updates of all systems (and you need to automatically spill it, this saves a lot of time), physical workarounds with a flash drive over network nodes, port allocation for three weeks, and so on.

8. Communication between developers and analysts is not established.

This is such a jamb that is repeated on every fifth project. It’s just that the analyst wrote what is needed, the developer developed it and a month later showed it ready. The analyst runs in horror because he did not mean this. For three weeks this month, the developer worked in vain, although he could show part of the project, and the analyst could ask how things are going. There are methodologies that simply close this, but the problem is that in this situation both parties do not understand why the project is needed and how it goes.

9. Conference Driven Development.

The leader saw something cool at the conference, and they introduced it without a trace. Three months later repeated. As a result, the report is beautiful, but the work is worth it.

Due to internal conferences, it is still possible to bring to the result according to the scheme “Always 80% ready”. On public reports, “Almost done,” and it's endless. Bringing to 100% never happens. Why? Well, for example, see point 1.

Separately, I note the under-examination of third-party systems. You read the article - wow, cool, let's use it, and then just learn how. And you come across a lot of restrictions, because the vendor poured honey into his ears only at the first two meetings. We ourselves stepped on a rake. There was an internal implementation of a system for online stores of documentation on the design of infrastructures, including Data centers for very interesting objects. A case of limiting the cost of goods of 100 million was discovered. The trick is that, in the subject area, unit prices are higher. And there, the system had to shovel the index for the search, and we have investments of 1 gigabyte in the documents. And the expected indexing time is one month. The vendor did not warn about this.

10. Scary to make changes.

There is such a state of the project: you need refactoring, come back in a month. But there are no tests, no documents, nothing. And the developers are sitting: "We are afraid to make changes, we are afraid to break everything."

I also saw how one company developed integration with a system that cannot be deployed on the developer's side, because it is very difficult. Testers test the data transfer by the service (as far as possible). Exit to the customer’s stands before the delivery - and another two weeks of improvements, because the transaction model was wrong. It’s good practice to make a stub for this system that returns some answers. As a result, they did so, it became easier to test and accept.

Another customer can write economic descriptions in his usual language. On that project, we gave a certain pseudo-language constructor (a set of standards), which is then converted into analyst data. In the spirit of "If Vanya is on sick leave, then he will not be able to go to production."

And the final chord. Someone else in one company is coding right on the prod. "There is one little fiddle, I know what I'm doing." Thank you, dear man, but if I find you, I don’t even know what to do with you.

In general, spoke out. If you find out your problems, write to oeremeev@croc.ru in the mail. For large businesses, we have almost free pilots with express diagnostics (searches for bottlenecks in 3-5 days). If someone from your business needs to be explained that it is impossible to put up with technical debt, - write too, we can count this normally.