Change in perception of complexity

I want to share very subjective thoughts about changing attitudes towards complexity over the past 50 years. Perhaps my observations concern all of engineering, but I’ll be careful not to write about software development.

In recent years, I began to feel that I was missing something in software development methods. The whole world happily and with the song accelerates, creates software faster and faster, and I slow down. To keep up, you have to overcome the inner barrier of an obscure nature, to act abroad which is beneficial but unpleasant (to me). It is not clear why it is profitable. And it is not clear why it is unpleasant.

Today I understood. In the old days, they struggled with complexity, now they ignore it. The transition between these views was long and smooth, but one can already see striking differences.
')
I studied mainly based on materials created in the last century, “ancient manuscripts,” and I also read classical science fiction, so I unwittingly became an adherent of the “old school”. What is the difference?

When the computers were large and the software was small, we were only confronted with complexity and were frightened of it. Not having understood, they began to struggle with the source of fear. Hence the desire to formalize everything, standardize, classify.


At first we won the fight, because we only entered a new area for us and took on the most “simple” tasks. Success has spawned even more formalisms and standards: sequential software development models (waterfalls), OOP (in the form in which it became popular), “hard” protocols and formats (XML), relational databases, corresponding idioms (RAII), etc .

Over time, the tasks became more complex and it became harder to resist the difficulty. This problem had two effects:

1. spawned over-complicated "freaks" - short-lived technologies trying to recapture another islet from complexity;
2. changed the perception of the world by the developers themselves.

As a result, a new attitude to complexity was formed. She was simply accepted as one of the properties of our universe. Like gravity. Fencing off gravity is pointless and expensive. So it is with complexity.

As this view spread, relevant technologies began to evolve: flexible methodologies, NoSQL databases, dynamic typing languages, simple protocols and formats (JSON), flexible idioms like Fail Fast.


A new view on complexity turned out to be significantly more profitable in many cases (but not all). Instead of spending a huge amount of resources on creating a completely isolated formalism from the external world of the system, we can adapt our solutions to the already existing landscape, skillfully using its irregularities to our advantage, promptly correcting the problems that we directly encounter.

But in order to enjoy these benefits, it is necessary to stop being afraid of complexity, which is not so easy.