This happens at least once in the life of every programmer, project manager, or team leader. You get a whole bunch of paired dung. If you're lucky, just a few million lines. The original authors have long since flown to warm countries, and the documentation, if any, is hopelessly outdated.

Your task: to get out of this mess.

After you let go of the first instinctive reaction (to run away), you start working on the project, knowing full well that company executives are watching your progress. Failure is not an option. But so far, judging by the alignment, it is precisely the failure that seems the most likely outcome. So what to do?
')
I (not) was lucky to be in this situation several times. And we with a small group of friends found out that with proper skills this is a very profitable business - to take on such heaps of steaming misery and turn them into healthy, supported projects. Here are some tricks we use:

Backup copy

Before starting any action, back up everything that may be relevant to the project. This is to ensure that no information is lost that may be useful in the future. There can always be some stupid question that you cannot answer in a day or two after the changes have been made. Such problems especially arise with configuration data, they are usually not included in the versioning scheme, so it will be good luck to at least restore something from a periodic backup. So it is better to survive than sorry. Copy everything to the safest place and never in your life touch it if it is not in read-only mode.

An important prerequisite: make sure you have an assembly process that actually produces what works in production.

I completely missed this step, suggesting its obviousness and the presence of the assembly process in almost everyone, but many commentators on HN pointed it out and were absolutely right. The first thing to do is make sure that it is in production at the moment. This means that you should be able to build a version of the software that — if your platform works in this way — will byte by byte match the current build in production. If you cannot achieve this, then get ready for some unpleasant surprises when you try to send a commit to production. Make every effort to test, and when everything you need is in place, and you will be confident in the performance, roll out in production. Be prepared to immediately return to the previous version and make sure that total journaling of everything that can come in handy during the - inevitable - debriefing.

Do not touch the DB

If possible, freeze the database schema until you have completed the first stage of the improvements. You will be ready to change it when a complete understanding of the code base appears, and the legacy code is completely left behind. Change the scheme before this point - and there may be real problems, because you will lose the possibility of launching the old and new code base side by side with a reliable foundation from the database on which everything is built. Keeping the database in complete immunity, you can compare the effect of the new business logic code with the old one. If everything works as stated, there should be no difference.

Write your tests

Before any changes, write as much as possible end-to-end and integration tests. Make sure these tests are issued correctly and test as many scenarios as you can think of how the old code works (get ready for surprises here). These tests will have two important functions: they will help eliminate any misconceptions at a very early stage and they will serve as a protective fence if you start writing new code to replace the old one.

Automate all of your testing, if you already have a CI experience, use it, and make sure your tests run fast enough to run a full set of tests after each commit.

Collecting metrics and logs

If the old platform is still available for development and equipment. Do this in a completely new database table, add a simple counter to each event that you just come up with, and a simple function that will increment these counters based on the name of the event. In this way, you can implement an event log with timestamps with just a few extra lines of code and get a good idea of ​​how many events of one type lead to events of another type. One example: the user opens the application, the user closes the application. If two events lead to some kind of backend calls, then between these two counters in the long term there should be a constant difference, this difference represents the number of applications that are currently open. If you see a lot more openings than closing an application, you know that there must be a way the application quits (at least a crash). For each event that you find, there is some connection to other events. Usually you will make an effort to achieve permanent relationships, except for the obvious error somewhere in the system. You will strive to reduce those counters that correspond to errors, and to maximize the counters along the chain to the indicator set at the beginning. (For example, users who are trying to pay for a purchase should result in payments received).

This very simple trick turns any backend application into an accounting system, and just like with a real accounting system, the numbers must match so that you have no problems anywhere.

Over time, this approach will become invaluable in improving the health of the program and will be complemented perfectly with the change log of the source control system, in which you can determine the point in time when the bug appeared, and what effect it had on various counters.

I usually set up these counters with a five-minute resolution (so that it goes 12 bucks per hour), but if your application generates fewer or more events, then you can change the interval at which the buckets are created. All counters share one DB table, so each counter is just a column in this table.

Change only one thing at a time

Do not fall into the trap of improving the operational reliability of the code or platform on which it runs, and at the same time adding new features or fixing bugs. It will give you a huge headache, because now you have to ask at every step what the desired outcome of the action is, and this cancels some of the tests that were done earlier.

Platform changes

If you decide to move the application to another platform, then do it, but leave everything else unchanged . If you want, you can add documentation or tests, but no more than that, all business logic and interdependencies should be kept the same.

Architecture changes

The next step is changes to the application architecture (if required). At this point, you are free to change the high-level structure of the code, usually reducing the number of horizontal links between modules and thereby reducing the amount of code that is active during each interaction with the end user. If the old code was monolithic in nature, now is the time to make it more modular, break large functions into smaller ones, but keep the names of variables and data structures as they were.

User HN mannykannot paid attention - rightly - that there is not always such an opportunity, and if you are not particularly lucky, you will have to dig deeper to be able to make any architectural changes. I agree with this and should have mentioned it, so therefore here is a small addition. I would also like to add that if you make high-level and low-level changes at the same time, at least try to limit them to one file or, in the worst case, to one subsystem to limit the amount of changes as much as possible. Otherwise it will be very difficult when you have to debug the changes you just made.

Low level refactoring

At this point, you should have a good understanding of what each module is doing, and you are ready for a real job: refactoring the code to improve operational reliability and prepare the code for new functionality. This happens to be the part of the work that takes the most time, write documentation on the go, do not change the module until you have thoroughly documented it and have not received a feeling of a complete understanding of its functionality. Feel free to rename variables and functions, as well as data structures, to increase clarity and consistency, add tests (also unit tests, if the situation dictates this).

Fix bugs

You are now ready to make changes visible to real end users. The first order of battle will be a long list of bugs accumulated over the years in the queue of tickets. As usual, first make sure the problem still exists, write a test for this purpose, and then fix the bug, your CI and written end-to-end tests should save you from errors that you can make due to lack of understanding or some extraneous question.

Database update

If everything you need is done and you have a solid and supported code base again, then there is an option to change the database schema or replace the database altogether with another model if you planned to do so. All that you have done by this time will help you make this change in a responsible manner without any surprises, you can thoroughly test the new database with the new code, and all the tests are available and guarantee that your migration will take place without a hitch.

Running on a roadmap

Congratulations, you are safe and ready for the introduction of new functionality.

Never even think about large-scale rework

Large-scale rework - a type of project that is almost guaranteed to fail. First, you start in a completely uncharted territory, so it’s not known where to start, and second, you put all the problems aside on the very last day - the day before you put your new system into life. And then you fail terribly. Assumptions of business logic will be wrong, suddenly you will understand why that old system did certain things the way it did, and in general you will come to the conclusion that the guys who assembled the old system may not be such idiots, in the end . If you really want to break up the company (and your own reputation to boot) in every sense, start a massive alteration, but if you are smart, then this possibility is not even considered.

So, as an alternative, work step by step.

To unravel one of these balls in the fastest way, you need to take any element from the code that you understand (it may be a minor piece of code, but it may be a key module), and try to improve it step by step, staying in the old context. If the old build tools are no longer available, you will have to use some tricks (see below), but at least try to keep as much code alive as possible, which is proven to work by starting your changes. Thus, as the code base improves, your understanding of what it actually does will grow. A typical commit should consist of a pair of lines.

Release!

Every change made is released in production, even if the change is not visible to the end user, it is important to take the smallest possible steps, since you do not have an understanding of the system, there is a high probability that only in the working environment you realize the existence of the problem. If this problem occurs right after your little change, you get several advantages:

• would probably be trivial to find out what went wrong
• You will be in an excellent position to improve the process.
• and you will immediately update the documentation to display the facts just revealed

Use proxies to your advantage

If you're doing web development, praise the gods and put a proxy between the end users and the old system. Now you have the control of what requests go to the old system, and which requests you redirect to the new system, which makes it much easier and more accurate to control what to run and who will see it. If you have a smart enough proxy, you could probably use it to send some traffic to a new system for individual URLs, until you are satisfied that everything works as expected. If your integration tests are connected to this interface, then it is even better.

Yes, but it all takes too much time!

Well, how to look. Indeed, if you follow these steps, you need some re-work. But it does work, and any optimization of this process assumes that you know more about the system than you probably know. I have a reputation and I really do not want negative surprises while working like this. If you are unlucky, this may lead the company to failure or there may be a real threat to cause confusion in the work of clients. In such a situation, I prefer complete control and a reinforced concrete process, rather than an attempt to save a couple of days or weeks, which puts at risk the successful outcome. If you are more cowboy in character - and your boss agrees - then it may be acceptable to take more risks, but most companies will choose a slightly slower, but much more faithful way to victory.