Rules for the development of complex systems. The story of one project
Hi, Habr. My name is Alexander. And I want to share my story of work on one large and complex project.
In this article there will be no code and diagrams, it will only contain the creation history “from and to” of the project itself. I think many will be interested in this article. So let's go!
It all started in the summer of 2011. At that time I was 3 years old full-time freelancer. So my job is freelancing. He has worked and is still working only with Western customers. The main specialization is the development of projects related to the recognition of images, text, etc.
')
It all started with the fact that I, as always, checked the mail in the morning, cleaned the spam, was engaged in routine work. I usually don’t see what is in my spam, but then I saw a letter from a very real address. I opened a letter in which one company was looking for a programmer to finish a large western project. Moreover, this company demanded a programmer from my city and necessarily with experience in the field of recognition. I answered this letter for the sake of curiosity. Literally an hour later I received an answer. And after two we phoned the project manager. At first it seemed to me that there was nothing difficult in the finalization, the usual set of functionality. After a brief conversation with the manager, I announced my price, that is, the rate per hour. And on this we said goodbye. The next day I was told that they agreed to my price tag and were given a test task. I successfully completed it within an hour, and we moved on. And here the fun begins. First, I was invited to the office in order to sign a non-disclosure agreement. Secondly, and this is logical, the source code of the project I was promised to give only after the signing of the contract. To be honest, it confused me, I don’t even know why. And intuition did not let me down. I demanded at least part of the source code in order to assess the complexity of the work and asked to tell more about the project. As it turned out, the project at that time was underway for three years and I was a 4 (!) Performer. Before me, the American company worked, then the Indians, then the company that hired me, tried to implement the project with the help of one girl programmer, and then it was all offered to rake me. I was not just surprised, but very alert. Then I learned a lot of amazing things, for example, that the customer hadn’t seen the program for 2 years, but had seen only screenshots, and the Indians fed the promises of this customer. I did not fit in my head how this can be realized. Hindu manager should be given the medal "For resourcefulness."
After I listened to the amazing story, we agreed with the manager that he would give me the source code and I would appreciate the scale of the tragedy. To make it clearer, I will tell you more about the project. This project is a tool for engineers, architects, electricians and other people who are building houses, skyscrapers, in a word, buildings. It is used to calculate the various elements on building plans, calculate areas, measure lengths and estimate. Roughly speaking there is a construction plan and there are sockets on it. We need to recognize and count how many of these outlets. For recognition used a library written by another programmer. The project itself is written in C #. My task was to put everything together and refine the additional functionality, as well as bring the program to a less stable state. Everything seems simple and elementary. I thought so too. But it was not there.
After I got the source, I tried to compile the project. I did not succeed. After a brief analysis, I corrected the errors and still launched the project. But, unfortunately, it did not work as it should. After several hours of code analysis, I came to the conclusion that the whole problem is in the recognition library. At that time I had a 64-bit “seven”, and the manager had a 32-bit one. Everything worked for him, I haven't. I asked to compile the library for a 64-bit platform. But the developer of the library, foaming at the mouth, argued that it was not a matter of bitness. I couldn’t prove anything to him, since he gave very little information about his library and generally kept it as the apple of his eye. Time passed and I had to at least complete the search process. I spat on everything and set myself a 32-bit version of the OS. And lo and behold! It all worked. Distracting, I want to say about the library, in the future it still turned out to be in its bit depth.
I began to analyze the code. The first impression was just disgusting. I was shocked. I will give a list of what angered me in the project.
The project is very much connected with graphics, but for its output and processing the usual PictureBox was used. The smallest plan size is 5400x3600 pixels. Knowledgeable people will understand that for PictureBox this is a rather problematic topic with the output of large images and their processing. Do not forget that in addition to the plans themselves, a lot of information is also displayed (areas, text, symbols found, etc.). When you run a project with 5 small plans, the program will certainly fall with the error "Out of memory". What was a very big problem, since the main idea was that the tool had to work on the principle of “launched and forgotten”.
Rather, its absence. I spent 6 months to understand this architecture, but I didn’t manage it. Classes are connected not only incomprehensibly, but simply chaotic. In one class, methods can be written that are responsible for different logically unrelated operations. Especially "pleased" class called "RubberBand", if I understood correctly, then from English it translates as "gum", although it may be some kind of term. I dont know. So, this class was something so comprehensive. It was possible to find methods for working with the database, graphics, interface, as well as the calculation part. From the very beginning, I thought it was a kind of class that was used as a draft. That is, it is not used, and only some practices are stored in it. Because, at first glance, there is no logic in it, but only individual pieces of code. But I was wrong - this is the most important governing class.
Everything is also ambiguous. Due to the fact that several companies were involved in the project, the code was of different stylistics, writing. I got the impression that the naming of the variables was “from the bald.” Often, variables were named, for example: int rjik, double opgdd. But this is not the main thing. After several days of refactoring, the variable problem was resolved.
Next are the methods. It's a little simpler here. Although no - not easier. The difficulty is that many methods are also named either meaninglessly, or without carrying any semantic load at all. Distracting, I want to note that when I parsed the code, I got the impression that the naming happens according to the principle that I see, I write. Well, for example, I see a table - a name variable on the Indian "table".
I especially remember one method called “bindingDB (bool Draw)”. It contained more than 800 (!) Lines of code. If you thought that it was somehow connected to the database, then you, like me, are mistaken. I spent a few days to understand why he and break it. This method was unique. He was called from almost all other classes. He was responsible for the calculation of statistics, graphics output, interaction with the GUI and much more. And many lines of code were copied from the previously mentioned class RubberBand.
We turn to the comments. There were few comments. Most of them were irrelevant. Previous developers left comments of this type: "Here we assign the variable i 0". And in some confused places there were no comments.
I also want to note that there were a few pieces of code that were impudently stolen from the codeproject.
Here I can not say anything bad about architecture. Since it was designed by the Americans. But you can write a whole book about stored procedures, about how NOT to write queries.
Immediately, I note that this architecture, as conceived by the program owner, should have ensured continuous operation in the organization’s local network. The server part had to be installed on the server, and handle all the most difficult operations, that is, recognition procedures. And the client parts had to stand on the machines of the workers, who would simply form the projects and launch them for analysis, which was supposed to occur on the server. After analysis, all data should be recorded in the database, after which all client parts should be synchronized with the database.
In reality, everything looked different. After forming the project on the client, he was sent to the queue on the server. After entering the queue, the client was blocked until it was analyzed on the server. Imagine there is an organization with 5 client machines. On each of the machines one project will be formed (the average project is 10 pages and 10 desired pictures). Such an average project is calculated in the region of 1 hour. That is, if 5 projects are formed, the last project will begin only after 4 hours. And all these 4 hours the client will be frozen. Hindus, in my opinion, do not know what multithreading is.
I was given 2 months to solve all the problems. In fact, this resulted in 7 months of not only reworking, but also updating the new functionality. In the end, even after my witchcraft, we got an unusable product. No, the point here is not that I could not patch it, but the fact is that the project was originally crookedly designed, which subsequently led to a very deplorable result. A lot of things like flickering, memory overflow, slow work and so on. It was simply impossible to remove. And we with the customer decided that we need to rewrite the project from scratch.
For a month I had to provide a demo version. Which will cover 30% of all functionality. In February, I started designing and developing the project.
Next, I will provide a list of what I have noted for myself in the design.
This is a prerequisite for any more or less large project. It must be respected so that further development does not have to rewrite a significant portion of the code. Any chip or ryushechka should join in the minimum time. And the code should be clear and flexible.
An important point of any designer, as the success and quality of development often depends on the technologies used. As a programming language, I chose C # and the .NET 4.0 framework. It was originally incorporated in the project itself, so I did not change anything.
I chose MSSQL 2008 r2 from the DBMS, I am more used to working with MS, so I trust them.
To work with graphics, I chose XNA 4.0. A good powerful thing, although it has its drawbacks, but they are insignificant over the advantages.
Here I was extremely attentive and cautious when designing. Since I wanted to keep the code clean, easy to understand and avoid the “Indian code”, first I described all classes on A4 sheets, each sheet was named, for example, “ImageProcessing”, “Rendering”, “Reporting”, etc. Further, these sheets were connected according to logic, for example, sheets with graphic classes were connected separately from classes for working with the database. After all these manipulations, I got a class architecture. I also entered the database architecture. Many may say that it is wrong, but it is convenient and comfortable for me to design.
When naming methods and variables, I tried to call them so that in the future there was no confusion and wasting time understanding the value of a method or variable. In my code there are no such variables as “a”, “b”, “c”, since such a designation does not carry a semantic load. The exceptions are the variables i, j - I am used to using them in cycles. And also the variables x, y, z are coordinates.
I leave comments before each method to understand what this method is. In the methods themselves, I try to leave more comments. Since this greatly simplifies the understanding of the code in the future.
I know that many sin by not using exception handling constructs. But they must be used without fail.
In high-load systems, clearing the memory is a necessary condition for the normal operation of the product. Therefore, be sure to clean the memory.
I used to actively use it. I wrote a method or class. Then, if necessary, I took and copied the code. But with this project, I basically refused such a method. Since I'd rather spend 15 minutes writing code, I will spend an hour debugging and bugs. Just often when we copy code, we subconsciously can not change some things in it, because of which errors may appear.
Initially try to write some basic things, and only then fasten chips to them. Since if you start writing all at once, you get a “spray”. That is, you spray your attention and time. And often it turns out that both in the main points of the problem and in the additional problems too. It turns out "Neither myself nor the people."
The result of my work was a product that is already announced in the United States. And a couple of days will begin selling it. For four months I was able to develop a product that three previous companies could not develop in 3 years. I'm not saying that I am a mega cool programmer. I say that the quality of the product, the speed of development, the correction of errors, depends on those points that I described above. And now if we compare the two products - new and old, then this is heaven and earth. And in future projects, I will stick to one simple rule: “It is better to spend more time on planning and design, so that you do not spend much more time on debugging and fixing”, which I wish for you.
In this article there will be no code and diagrams, it will only contain the creation history “from and to” of the project itself. I think many will be interested in this article. So let's go!
Start
It all started in the summer of 2011. At that time I was 3 years old full-time freelancer. So my job is freelancing. He has worked and is still working only with Western customers. The main specialization is the development of projects related to the recognition of images, text, etc.
')
It all started with the fact that I, as always, checked the mail in the morning, cleaned the spam, was engaged in routine work. I usually don’t see what is in my spam, but then I saw a letter from a very real address. I opened a letter in which one company was looking for a programmer to finish a large western project. Moreover, this company demanded a programmer from my city and necessarily with experience in the field of recognition. I answered this letter for the sake of curiosity. Literally an hour later I received an answer. And after two we phoned the project manager. At first it seemed to me that there was nothing difficult in the finalization, the usual set of functionality. After a brief conversation with the manager, I announced my price, that is, the rate per hour. And on this we said goodbye. The next day I was told that they agreed to my price tag and were given a test task. I successfully completed it within an hour, and we moved on. And here the fun begins. First, I was invited to the office in order to sign a non-disclosure agreement. Secondly, and this is logical, the source code of the project I was promised to give only after the signing of the contract. To be honest, it confused me, I don’t even know why. And intuition did not let me down. I demanded at least part of the source code in order to assess the complexity of the work and asked to tell more about the project. As it turned out, the project at that time was underway for three years and I was a 4 (!) Performer. Before me, the American company worked, then the Indians, then the company that hired me, tried to implement the project with the help of one girl programmer, and then it was all offered to rake me. I was not just surprised, but very alert. Then I learned a lot of amazing things, for example, that the customer hadn’t seen the program for 2 years, but had seen only screenshots, and the Indians fed the promises of this customer. I did not fit in my head how this can be realized. Hindu manager should be given the medal "For resourcefulness."
After I listened to the amazing story, we agreed with the manager that he would give me the source code and I would appreciate the scale of the tragedy. To make it clearer, I will tell you more about the project. This project is a tool for engineers, architects, electricians and other people who are building houses, skyscrapers, in a word, buildings. It is used to calculate the various elements on building plans, calculate areas, measure lengths and estimate. Roughly speaking there is a construction plan and there are sockets on it. We need to recognize and count how many of these outlets. For recognition used a library written by another programmer. The project itself is written in C #. My task was to put everything together and refine the additional functionality, as well as bring the program to a less stable state. Everything seems simple and elementary. I thought so too. But it was not there.
After I got the source, I tried to compile the project. I did not succeed. After a brief analysis, I corrected the errors and still launched the project. But, unfortunately, it did not work as it should. After several hours of code analysis, I came to the conclusion that the whole problem is in the recognition library. At that time I had a 64-bit “seven”, and the manager had a 32-bit one. Everything worked for him, I haven't. I asked to compile the library for a 64-bit platform. But the developer of the library, foaming at the mouth, argued that it was not a matter of bitness. I couldn’t prove anything to him, since he gave very little information about his library and generally kept it as the apple of his eye. Time passed and I had to at least complete the search process. I spat on everything and set myself a 32-bit version of the OS. And lo and behold! It all worked. Distracting, I want to say about the library, in the future it still turned out to be in its bit depth.
I began to analyze the code. The first impression was just disgusting. I was shocked. I will give a list of what angered me in the project.
1. Components and controls.
The project is very much connected with graphics, but for its output and processing the usual PictureBox was used. The smallest plan size is 5400x3600 pixels. Knowledgeable people will understand that for PictureBox this is a rather problematic topic with the output of large images and their processing. Do not forget that in addition to the plans themselves, a lot of information is also displayed (areas, text, symbols found, etc.). When you run a project with 5 small plans, the program will certainly fall with the error "Out of memory". What was a very big problem, since the main idea was that the tool had to work on the principle of “launched and forgotten”.
2. The logic of the architecture of the source code of the program.
Rather, its absence. I spent 6 months to understand this architecture, but I didn’t manage it. Classes are connected not only incomprehensibly, but simply chaotic. In one class, methods can be written that are responsible for different logically unrelated operations. Especially "pleased" class called "RubberBand", if I understood correctly, then from English it translates as "gum", although it may be some kind of term. I dont know. So, this class was something so comprehensive. It was possible to find methods for working with the database, graphics, interface, as well as the calculation part. From the very beginning, I thought it was a kind of class that was used as a draft. That is, it is not used, and only some practices are stored in it. Because, at first glance, there is no logic in it, but only individual pieces of code. But I was wrong - this is the most important governing class.
3. Variables, methods and comments.
Everything is also ambiguous. Due to the fact that several companies were involved in the project, the code was of different stylistics, writing. I got the impression that the naming of the variables was “from the bald.” Often, variables were named, for example: int rjik, double opgdd. But this is not the main thing. After several days of refactoring, the variable problem was resolved.
Next are the methods. It's a little simpler here. Although no - not easier. The difficulty is that many methods are also named either meaninglessly, or without carrying any semantic load at all. Distracting, I want to note that when I parsed the code, I got the impression that the naming happens according to the principle that I see, I write. Well, for example, I see a table - a name variable on the Indian "table".
I especially remember one method called “bindingDB (bool Draw)”. It contained more than 800 (!) Lines of code. If you thought that it was somehow connected to the database, then you, like me, are mistaken. I spent a few days to understand why he and break it. This method was unique. He was called from almost all other classes. He was responsible for the calculation of statistics, graphics output, interaction with the GUI and much more. And many lines of code were copied from the previously mentioned class RubberBand.
We turn to the comments. There were few comments. Most of them were irrelevant. Previous developers left comments of this type: "Here we assign the variable i 0". And in some confused places there were no comments.
I also want to note that there were a few pieces of code that were impudently stolen from the codeproject.
4. Database
Here I can not say anything bad about architecture. Since it was designed by the Americans. But you can write a whole book about stored procedures, about how NOT to write queries.
5. Client server
Immediately, I note that this architecture, as conceived by the program owner, should have ensured continuous operation in the organization’s local network. The server part had to be installed on the server, and handle all the most difficult operations, that is, recognition procedures. And the client parts had to stand on the machines of the workers, who would simply form the projects and launch them for analysis, which was supposed to occur on the server. After analysis, all data should be recorded in the database, after which all client parts should be synchronized with the database.
In reality, everything looked different. After forming the project on the client, he was sent to the queue on the server. After entering the queue, the client was blocked until it was analyzed on the server. Imagine there is an organization with 5 client machines. On each of the machines one project will be formed (the average project is 10 pages and 10 desired pictures). Such an average project is calculated in the region of 1 hour. That is, if 5 projects are formed, the last project will begin only after 4 hours. And all these 4 hours the client will be frozen. Hindus, in my opinion, do not know what multithreading is.
Problem solving
I was given 2 months to solve all the problems. In fact, this resulted in 7 months of not only reworking, but also updating the new functionality. In the end, even after my witchcraft, we got an unusable product. No, the point here is not that I could not patch it, but the fact is that the project was originally crookedly designed, which subsequently led to a very deplorable result. A lot of things like flickering, memory overflow, slow work and so on. It was simply impossible to remove. And we with the customer decided that we need to rewrite the project from scratch.
New project
For a month I had to provide a demo version. Which will cover 30% of all functionality. In February, I started designing and developing the project.
Next, I will provide a list of what I have noted for myself in the design.
1. Scalability and flexibility
This is a prerequisite for any more or less large project. It must be respected so that further development does not have to rewrite a significant portion of the code. Any chip or ryushechka should join in the minimum time. And the code should be clear and flexible.
2. Technology
An important point of any designer, as the success and quality of development often depends on the technologies used. As a programming language, I chose C # and the .NET 4.0 framework. It was originally incorporated in the project itself, so I did not change anything.
I chose MSSQL 2008 r2 from the DBMS, I am more used to working with MS, so I trust them.
To work with graphics, I chose XNA 4.0. A good powerful thing, although it has its drawbacks, but they are insignificant over the advantages.
3. Database architecture and classes
Here I was extremely attentive and cautious when designing. Since I wanted to keep the code clean, easy to understand and avoid the “Indian code”, first I described all classes on A4 sheets, each sheet was named, for example, “ImageProcessing”, “Rendering”, “Reporting”, etc. Further, these sheets were connected according to logic, for example, sheets with graphic classes were connected separately from classes for working with the database. After all these manipulations, I got a class architecture. I also entered the database architecture. Many may say that it is wrong, but it is convenient and comfortable for me to design.
4. Variables, methods and comments
When naming methods and variables, I tried to call them so that in the future there was no confusion and wasting time understanding the value of a method or variable. In my code there are no such variables as “a”, “b”, “c”, since such a designation does not carry a semantic load. The exceptions are the variables i, j - I am used to using them in cycles. And also the variables x, y, z are coordinates.
I leave comments before each method to understand what this method is. In the methods themselves, I try to leave more comments. Since this greatly simplifies the understanding of the code in the future.
I know that many sin by not using exception handling constructs. But they must be used without fail.
5. Memory
In high-load systems, clearing the memory is a necessary condition for the normal operation of the product. Therefore, be sure to clean the memory.
6. Copypast
I used to actively use it. I wrote a method or class. Then, if necessary, I took and copied the code. But with this project, I basically refused such a method. Since I'd rather spend 15 minutes writing code, I will spend an hour debugging and bugs. Just often when we copy code, we subconsciously can not change some things in it, because of which errors may appear.
7. From larger to smaller
Initially try to write some basic things, and only then fasten chips to them. Since if you start writing all at once, you get a “spray”. That is, you spray your attention and time. And often it turns out that both in the main points of the problem and in the additional problems too. It turns out "Neither myself nor the people."
Result
The result of my work was a product that is already announced in the United States. And a couple of days will begin selling it. For four months I was able to develop a product that three previous companies could not develop in 3 years. I'm not saying that I am a mega cool programmer. I say that the quality of the product, the speed of development, the correction of errors, depends on those points that I described above. And now if we compare the two products - new and old, then this is heaven and earth. And in future projects, I will stick to one simple rule: “It is better to spend more time on planning and design, so that you do not spend much more time on debugging and fixing”, which I wish for you.
Source: https://habr.com/ru/post/143804/
All Articles