Excessive complexity as a reason for additional hidden costs

We often tell someone phrases like "I only need a few hours to do it / implement / implement it." But after some time after the end of the work, we suddenly realize that we have to regularly fix bugs and errors, explain the device and the algorithm of work to other engineers and developers, or help answer customer questions. And it turns out that the amount of time spent on supporting your offspring many times exceeds the few hours that were spent on its creation.

In software development, one of the most difficult to assimilate rules is that with increasing complexity of a product, hidden costs also grow. Sometimes complexity is only a consequence of some other problems. It is very difficult to find a balance between the “selling” and “loading” properties of a product, especially in the light of determining its cost and regulating supply and demand. It is equally difficult to engage in maintaining the quality of the product and expanding the range of consumers. Or, say, develop a rich text editor that works equally quickly and consistently on all devices and supports real-time document collaboration. In other words, it is often necessary to complicate a product so that it becomes successful in the market.

On the other hand, we often have to contend with the complexity of the product, which we ourselves have allowed. For example, they wrote an application in a language that is owned by very few employees, and in the future we are faced with difficulties of support. Or they introduced a new, fresh technology, which they really wanted to try after reading reviews, but it turned out that it had a number of flaws that no one suspected. Or they have developed a new function that is needed by a very limited number of users, but disproportionately a lot of time and effort is being spent on fixing bugs.
')
The increasing complexity of the product leads to an increase in hidden costs. Decisions made by us in the course of software development, affect not only the speed of product release. They are reflected in how much time and effort it takes to promote and support. Moreover, the excessive complexity to take place not only in the code or functionality of the product, this also affects the organization of work in the development team and the structure of the team itself.

Code complexity

The complexity of the code does not show a linear dependence on the number of lines. It grows combinatorially: each line of code can interact with many other lines. And the degree of complication in this case is very difficult to realize, so many developers tend to downplay the time required to create a large and complex project. In particular, this is one of the reasons that rewriting a project often breaks all deadlines.

When the code becomes very complicated, it becomes more difficult to expand it, think about it as a whole, and detect bugs. Difficult to understand the relationships and data flows when you need to find the cause of an error. At the same time, developers periodically try not to touch at all some of the most complex parts of the code, even if making changes there would be the most logical step. Or avoid working together on the most difficult places, although this may be much more effective.

Systemic complexity

Engineers and software developers like to tinker with new technologies, applications, devices, devices and components. This is a manifestation of innate curiosity, usually peculiar to the people of these professions. Most of them believe that the next new technology can magically solve some current problems. For example, when Pinterest started expanding their resource in 2011, they used six different storage technologies and databases: MySQL, Cassandra, Membase, Memcache, Redis, and MongoDB. And all this diversity was served by a team of back-engineers of three people. Each of the solutions with which they experimented, on paper promised to solve some limitations of their system. But in practice it turned out that all these products could not cope, and each in its own way. In this case, management and support took a lot of time and effort. In the end, Pinterest came to the decision that it would be easier to scale up your service by adding new servers, rather than introducing new software technologies. The team abandoned some of the products and concentrated on the development of the remaining ones.

Fragmentation of infrastructure into too many systems leads to increased costs. All of them will require a lot of attention, it will become more difficult to pool resources to create reusable libraries for each system, it is more difficult to maintain day-to-day services, to understand the causes of specific failures and to quickly recover performance. Yes, and the return on each system is unlikely to be the maximum, because they will be given less time.

Product complexity

The product may be overly complicated due to someone's mistaken vision or unlimited ambitions, which leads to a blurring of the objectives and target audience of the product. Sometimes the fact that developers want to become the best everywhere, instead of concentrating on something key, can already speak of an inability to explain the purpose of a product to users. Its complication leads to an increase in the system complexity and complexity of the code. The team begins to add new features, implement new modules to support them, inflate the code base. If the product already has broad functionality, then adding a new feature or changing an existing one leads to the fact that more efforts are needed to understand and master all this abundance.

Complex products usually suffer from a large number of bugs and security holes. Engineers and data processing and analysis specialists have to consider much more variables and write more reports, rather than focusing on how users use the product. Team members are constantly switching their attention between different projects. And the time spent on maintaining the functionality of the product could be spent on optimizing the code, paying for “technical debt” and other useful actions.

Organizational complexity

All three of the above types of complexity lead to the complication of the organizational structure of the development team. We have to hire new employees, because of the proliferation of the product there is a lack of hands. And large teams require more communications, more coordination of actions, which does not always improve efficiency. Yes, and the process of finding employees, with all these interviews, can take a lot of time from team members. But beginners also have to train, to bring up to date.

An alternative to recruitment can be the division of a team into smaller groups. Perhaps in some of them there will even be one person each. These teams can assign different areas of responsibility. This allows you to prevent an excess of communication links, but the “teams of one person” also have a number of disadvantages: reduced motivation, limited duration of the working process, deterioration in the ability to learn, etc. Such a specialist is himself-self-the team cannot consult or get help from colleagues, because he is alone, and therefore the solution of some tasks may take a lot of time from him. In teams, you can quickly get an opinion from colleagues about your work, who will point out mistakes, cheer you. And loners will not get it either. As a result, this often leads to a decrease in productivity, a drop in the quality of work and ... an unintended complication of the code base or infrastructure.

How to deal with complexity

In 1980, Anthony Hoar said: “There are two ways to create software architecture. The first is to make it so simple that there are obviously no flaws in it. The second is to make it so difficult that it does not have obvious flaws. ” How do we protect ourselves from the costs associated with the unobvious shortcomings generated by complexity?

You can use different strategies.

Optimize for the sake of simplification. As far as possible, avoid any complications. Estimate the cost of support. Ask yourself if the solution is worth 20% of the problem so that the project is complicated, or is it enough to close the problem by 80%?

Be clear on the team for the mission or vision of the product. The authors of the book Team Geek told how they instructed the Google Web Toolkit development team, inspiring them to an accurate description of the mission. The ensuing discussion of the content and the submission form helped to find out that the leading developers really disagree with the general vision of how the final product should look! The team had to sit down and discuss all the views expressed, and as a result, the goal was developed: “The mission of GWT is to dramatically improve the experience of using the web, allowing developers to use existing Java tools to build uncompromising AJAX in all modern browsers.” How much time and effort would they have spent in vain if they had not discussed all their differences at an early stage?

Build large systems with simpler components. Google is a good example of an organization that focuses on building powerful, basic structures that are then widely used in all kinds of applications. They have such basic components as Protocol Buffers , Google File System , Stubby servers. On top of these components, they add other structures, like MapReduce and BigTable . And based on this, many applications are built, including a large-scale search engine, Google Analytics service, Google News cluster, Google Earth data processor, Google Zeitgeist data analysis service and much more.

Be clear about the interfaces between modules and services. Reducing links between service modules allows reducing the combinatorial complexity of even a small code. In 2002, Jeff Bezos decided that Amazon would move in the direction of a service-oriented architecture, and that all development teams would communicate with each other only through service-level interfaces. While this transformation required enormous development costs, the separation of code and service logic made it easy to create Amazon Web Services, which is very successful today.

Periodically "pay technical debt." We always create applications in the conditions of lack of necessary data. In response to changing conditions, the code base grows, which leads to an increase in entropy. Complication of the project becomes a kind of tax on future developments. And to reduce this "tax" can be the allocation of time for relevant work. Many teams and developers do this in between projects. But even one-time events, kind of subbotnik, sometimes is enough. For example, in Quora, a Code Cleanup Day was held, during which developers were engaged in removing unused code from the code base.

Use statistics to get rid of unclaimed functions. At Yammer, when someone discovers that extending the functionality or keeping it at the same level requires too much effort from the team, they begin to analyze product usage statistics, finding out if the functionality is in demand by users. If it is not claimed, or it is claimed weakly, then this functionality is discarded or limited. Such a strategy reduces costs in the same way that simplifying a code reduces “technical debt”.

Group existing projects by subject. This allows team members to share suitable solutions, makes it easy to discuss design, parse code, or create reusable libraries.

When students learn programming, they can afford to experiment with complex decisions in any way, because after graduation, their projects will be lost. But the erroneous decisions of a professional developer can significantly affect his career. Do not complicate things unnecessarily.