Translation of the book "Social Architecture": Chapter 5. Design, development, innovation

"The size and diversity of the community is key."

Let's look at innovations that Wikipedia defines as “the development of new values ​​through solutions that meet new requirements, no obvious needs or needs of old customers or markets for new ways of adding value”. In fact, this means solving problems in a cheaper way. It sounds simple, but the stories of the collapsed technology giants say the opposite. I will try to explain why teams often do not understand this correctly, and suggest a way to create something innovative.

The story of two bridges

Two old engineers talked about life and showed off their outstanding projects. One of the engineers described how he developed one of the greatest bridges ever created.



“We built it above the river gorge,” he told his friend. “It was wide and deep. We spent two years studying the soil and selecting projects and materials. We hired the best engineers and designed the bridge, which took another five years. We hired the largest engineering companies to create structures, towers, control posts and roads that would connect the bridge with the main highways. Dozens of people died during construction. Below the level of the road we started the trains and laid a special path for cyclists. The years of my life are connected with this bridge. ”



The second person thought for a while, and then spoke. “One evening my friend and I, having thrown on vodka, decided to transfer the rope across the gorge,” he said. “Just a rope tied to trees. There were two villages, one on each side. At first, people pulled bales along this rope using belts and pulleys. Then someone threw the second rope and made a path along which you can walk. She was dangerous, but her children adored. Then a group of people remade it, making it stronger, and women with their products began to walk on it every day. On one side of the bridge, a market grew, which gradually turned into a large city, since there was plenty of room for houses. The rope bridge was replaced with a wooden one so that horses and carts could cross it. Then the city built a real stone bridge, with metal crossbars. Later, they replaced the stones with steel and today the cable-stayed bridge is in the same place. ”

')

The first engineer listened in silence. “Funny,” he said, “my bridge was dismantled ten years after we built it. It turned out that it was built on the wrong place, and no one wanted to use it. Some guys threw the rope over the gorge, a few miles further along the stream, and there they all walked. ”

How ZeroMQ lost its roadmap

When I represented ZeroMQ at the Mix-IT conference in Lyon (France) at the beginning of 2012, I was asked several times about the roadmap. My answer was: no more roadmap. We had them, and we eliminated them. Instead of several experts trying to determine the next steps, we allowed events to evolve naturally. The audience did not really like my answer. Too not french.



However, the history of ZeroMQ clearly shows why there are problems with road maps. In the beginning, we had a small library development team, several participants and no approved roadmap. Therefore, we put together our plans and tried to organize them in the form of releases. “Here,” we wrote, “what will be in the next release.”



As the releases were published, we ran into a problem: it’s easy to promise something and it’s much harder to do it according to plan. First, most of the work was voluntary, and it’s not quite clear how to get volunteers to follow the roadmap. Secondly, the priority of tasks changes significantly over time. Therefore, we made promises that we could not keep, and the outgoing releases did not fit the roadmap.



Another problem was that when defining a roadmap, we seemed to have stalled the territory, which made it difficult for others to take part. People prefer to participate in the realization of what they consider their ideas. A compiled list of what needs to be done looks more like a chore rather than an interesting opportunity to participate.



As a result, we were faced with traumatic changes in ZeroMQ, from which the road maps did not save us, despite a bunch of strengths and words spent on “doing everything right”. The result is incompatible changes in the API and protocols. It was clear that we needed a new approach, defining the process of making changes. Software developers don’t like the idea that powerful, effective solutions can appear without smart developers checking and thinking about them. And yet, then in Lyon, no one would question evolution. This is strange and ironic, so I will further reflect on this phenomenon, because on its basis the ZeroMQ community has been developing since the beginning of 2012.



In the dominant theory of innovation it is said that ingenious individuals, having considered a problem situation, give an accurate and accurate solution. Sometimes they have inspirations in the style of “eureka!” - and the solution is ready. The inventor and the process of invention itself are rare, precious, and sole. History knows many such lonely heroes. We owe them our modern world.



However, looking closer, you will see that something here does not converge. The story does not tell about lonely inventors. She tells us about people who are lucky, who have stolen or appropriated authorship over ideas that emerged from the work of many other people. She tells us about brilliant people who made the coup and then spent decades in fruitless and meaningless searches. Such large and well-known inventors like Thomas Edison were actually good at systematizing the various studies carried out by large groups of scientists. This is similar to the statement that Steve Jobs invented every device created by Apple. A good myth, good for marketing, but absolutely meaningless from a practical point of view for science.



The history of the last decades, which is better fixed and more difficult to manipulate, clearly demonstrates this. The Internet is definitely one of the most innovative and rapidly developing technologies, the formation of which has a large amount of reliable information. This technology has no inventor. Instead, there is a huge mass of people who carefully and successfully solved a long series of current problems, recorded their answers and made them available to all. The innovative nature of the Internet is provided not by a small select group of Einsteins. It is provided with RFC documents that can be used and improved by anyone, hundreds and thousands of smart, though not uniquely smart people, open source software that anyone can use and improve. It comes from sharing, mixing and scaling a community. It comes from the constant increase in the number of good decisions and getting rid of bad ones.



Therefore, here is an alternative theory of innovation:

1. There is an infinite area of ​​problems / solutions.
2. The area changes over time depending on external circumstances.
3. We can accurately perceive only those problems that are close to us.
4. We can estimate the profitability / cost of the problem using the solution market.
5. There is an optimal solution for any problem to be solved.
6. We can achieve this optimal solution in a heuristic and mechanical way.
7. Our intellect can speed up this process, but not replace it.

Therefore:

• Individual creativity is less important than the process. Smarter people can work faster, but move in the wrong direction. A collective vision of reality helps us to follow the current path and not to deceive ourselves or others.
• We do not need road maps if we have a well-established process. Over time, as solutions compete for market share, functionality will grow.
• We are not so much inventing solutions as we find them. Condolences to creative people: creativity is only a golem processing information, polishing its own ego and preoccupied with raising karma.
• Intellect is a social effect, although it feels like something personal. A person cut off from others stops thinking. We can neither identify problems nor evaluate their solutions without other people.
• Community size and diversity is key. Larger and more diverse communities cover more relevant tasks, solve them more accurately and do it faster than a small group of experts.

So when we trust the lone experts, they make classic mistakes. They focus on ideas, not problems. They focus on the wrong problems. They make wrong conclusions about the value of the problems to be solved. And they do not use what they are working on.



Can we apply the above theory in practice? At the end of 2011, I began to document C4 and similar contracts and use them in ZeroMQ and in closed source projects. I call the underlying process “Simplicity Oriented Development,” or abbreviated ODA (Simplicity Oriented Design, SOD). This is a reproducible way to develop simple and elegant products. It organizes people in flexible chains of solution providers who can quickly and cheaply navigate the problem area. They do this by creating, testing, and maintaining minimal acceptable solutions, called “patches,” or discarding them. Viable products consist of a long succession of patches applied one on top of the other.



Firstly, ODA is essential because this is how we develop ZeroMQ. It is also the basis for the development process, which we use when creating large ZeroMQ applications. Of course, you can use any software architectural methodology with ZeroMQ.

To better understand how we arrived at ODA, let's look at alternatives.

Trash-Oriented Design

The most popular type of development in large organizations is “Trash-Oriented Design”. TOD is based on the belief that in order to make money we need cool ideas. This stubborn pop-up crap is a powerful crutch for those who lack imagination. The theory goes like this: ideas are rare, so the trick is to grab them. Like people far from music, they admire a guitar player, not realizing that great talents are so cheap that they literally play on the streets for a penny.



The main exhaust of TOD is costly “thinking”: concepts, engineering documentation and products that go straight to the trash. It turns out this way: Creative People come with a long list of "we can do X and Y". I saw endlessly detailed lists of all the amazing things a product could do. We were all guilty of this. As soon as the creative work on the generation of ideas was accomplished, it was just a matter of their execution.



Then the managers and their consultants pass on their brilliant ideas to the designers who create tons of impeccably formulated documents. Those, in turn, take the ten best ideas of managers and turn them into hundreds of projects, stunning foundations of life.

These developments are passed on to developers who scratch their heads and wonder who came up with this nonsense. They begin to argue, but the designers have descended from Olympus, and, in the end, it’s not for mortal developers to argue with creative people and expensive consultants.



Then the developers wander back to their dens, humiliated, compelled by the whip to build a gigantic and “very elegant” junk. And this work is hysterical, because the designers do not take into account the real costs. Even small moods can turn into weeks of work. As the project slows down, managers force developers to work overtime on evenings and weekends.



As a result, something like a work product sees the light. This is something creaky, fragile, complex and ugly. Designers brand developers for their incompetence and pay consultants more so that they do makeup to the pig, and little by little the product starts to look better.



By this time, managers have already begun to try to sell the product and found, unexpectedly, that nobody needs it. Without a shadow of a doubt, they boldly throw millions of dollars at an advertising campaign explaining to the public why this product is urgently needed. They make deals with other organizations to push it into a lazy, stupid and ungrateful market.



After twelve months of tense advertising campaign, the product still does not bring profit. Worse, he dramatically fails and is branded by the press as a complete failure. The company slowly takes it to the warehouse, dismisses consultants, buys a competing product of a small startup and calls it version two of its own product. Hundreds of millions of dollars are thrown to the wind.



Meanwhile, another vision manager somewhere in the organization pours himself an extra glass of tequila and tells the marketing department about his brilliant idea.



TOD could be a caricature if it were not so common. About nineteen of the twenty products that are ready to be released to the market by large companies will fail (yes, 87% of the statistics are done on the spot). Of the twenty, only one will succeed, and even then thanks to aggressive advertising and the weakness of competitors.



The basic moral of TOD is clear, but difficult to digest: ideas are cheap. With no exceptions. There are no brilliant ideas. Anyone who starts a conversation with the words “Oh! We can also do this! ”Must be beaten with the zeal of wandering evangelicals. It's the same as sitting in a cafe at the foot of the mountain, drinking hot chocolate and saying to others: “Hey, I have a great idea, we can climb this mountain! And build there on top of the chalet! With two saunas! And the garden! Hey, and we can provide it with electricity using solar panels! Dude, that's cool! What color will we paint it in? In the green! No, in blue! Ok, go and do it, but for now I’ll stay here and get some charts and graphs! ”



For a good start to a successful development process, collect the real problems that people face. The second step is to evaluate these problems using the basic question “How much will a solution to this problem cost?”. After that, you can make a list of problems that are worth solving. Good solutions to real-world problems will be a successful product. Their success will depend on how good and cheap the solutions are, and how important the problem is (and, unfortunately, how big the marketing costs can be). But their success will also depend on how much effort they require, in other words how simple they will be.



Now, having struck the dragon of absolute uselessness, we will attack the demon of complexity.

Complexity-Oriented Design

Really good development teams and small companies can usually create decent products. But most of the products still get too complicated and less successful than they could be. This is all because teams of specialists, even the best of them, often stubbornly practice complexity-oriented development, (Complexity-Oriented Design, COD), as I call it. And it works like this:

• Management correctly identifies some interesting and complex problems that are attractive from an economic point of view. This is where they fall on the TOD gauge.
• The team enthusiastically begins to create prototypes and work on the core. All this works as it was intended, and the team, having caught fire even more, delves into the intense development and discussion of architecture, the creation of elegant schemes, beautiful and slim.
• Management returns and inspires the team to solve even more complex problems. We tend to equate costs with cost, so the more complex and expensive the solution is, the more it will be possible to help out for it - as it seems to them.
• The team consists of engineers who love to create things, and they come into action. They create and create and create, and all this ends with a massively beautifully designed complexity.
• The market when meeting with the product, scratches behind the ear and asks: "What, seriously, and this is the best solution that you find?". Yes, people use products, if at the same time they do not have to spend their own money to climb the mountain of manuals.
• Management receives positive feedback from its large customers, who share the view that the higher the cost (of learning and use), the higher the value, and continues to push the process.
• At this time, somewhere in the world, a small team is engaged in solving the same problem, using the best approach, and a year later spreads the current market situation into small pieces.

CODs are characterized by teams that are obsessed with solving the wrong problems and are subject to collective mania.



COD products are usually large, ambitious, complex and unpopular. Much of open source software comes from COD. For developers it is incredibly difficult to stop and stop expanding the project in order to cover more and more potential problems. They argue: "What if someone wants to do X?", But they never ask themselves: "How much does X really cost?"



A good example of COD in practice turned out to be Bluetooth, a complex, with an overly complicated construction, a set of protocols that users hate. It continues to exist only because in the patented industry there are no real alternatives. Bluetooth is perfectly protected, which is almost useless for a contactless protocol. At the same time, it lacks a standard API for developers, which means it is really expensive to use it in applications. On a group discussion channel #zeromq, a Wintre member once wrote how he was furious to find that XMMS 2 had a working plugin system, but he could not play music.



COD is a rabbit hole for developers and engineers, in which they continue and continue to search for technical solutions. They add more and more features, closing their eyes to the economic side of their work.



The main lessons of COD are simple, but bitter in taste:

• To do something in which there is no need now is meaningless. It does not matter how talented or brilliant you are - if you are doing what you do to nobody you need, you lose time.
• Problems are often unequal. Some are easy to solve, others are difficult. Ironically, but solving simple problems often benefits people than solving complex problems. Therefore, if you allow your developers to work on random things, most likely they will focus on the most interesting, but not actual tasks.
• Engineers and developers like to do different things and decorate them, and this will inevitably lead to complexity. It is extremely important to have a “stop tap”, a way to set short, strict deadlines that will force people to look for less significant, simple answers to the most important tasks.

Simplicity Oriented Design

Finally, we come to a rare and valuable Simplicity Oriented Design (SOD). This process begins with implementation: we do not know what we should do until we start doing something. Putting forward ideas or large-scale projects is not just useless, but it hinders the development of truly accurate solutions. Truly dainty tasks are hidden as cherished oases, and any activity other than seeking them only envelops them with fog. You need to be mobile, move fast and light.



SOD works as follows:

• We compile a list of interesting problems (by observing how people use technology or other products) and place them from simple to complex, examining and defining ways of using.
• We take the simplest, most dramatic problem and look for the minimum number of acceptable solutions or “patches” for it. Each patch solves the original and approved problem in the most optimal way.
• When evaluating patches, we are guided by the following question: “Can we find a simpler solution to the problem?” We can measure the complexity by the number of concepts and models with which the user will have to read or select at random to use the patch. Less is better. The perfect patch solves the problem without requiring anything from the user.
• The development of our product consists in creating a patch that solves the problem of “proof of concept” and which is then integrated into a single line of more mature products consisting of hundreds of thousands of patches one on top of another.
• We are not doing anything that would not be a patch. We enforce this by formal rules that require that every action or duty be tied to the main and approved by everyone task, clearly articulated and documented.
• We build our projects as a chain of solution providers, where each project can provide tasks to its “suppliers” and receive patches in response. The supply chain is a stop-stop, because when people are eagerly waiting for an answer, we have to work in narrow time frames.
• Individuals can work on any project and make patches for problems that they think are important. None of them "owns" projects; they can only force them to follow formal rules. A single project can have many variations, each can overgrow with different patches competing with each other.
• Projects export formal and documented interfaces, so source projects (client) are in the dark about the work being done. At the same time, they can compete for the attention of client projects, creating a free and competitive market.
• We tie our supply chain to real users and external customers, and we conduct the whole process in fast cycles so that the problem received from outside users can be analyzed, evaluated and solved by the patch in a few hours.
• At every moment, starting with the first patch, our product is ready for release. This is important, because most of the patches will be wrong (10-30%), and only by giving the product to users, we can find out which patches are problematic and need to be improved.

SOD is a bottom-up algorithm, a reliable way to find optimal solutions to the most important problems in an uncharted area. You do not need to be a genius to use SOD, you just need to be able to see the difference between fogging activity and progress in solving real problems.



People say that such algorithms have limitations. You can focus on solving local problems. But this is how life works: we collect small gradual improvements for a long time. There are no brilliant developers. We reduce the risk associated with the locality of problems, covering the whole area, and in general this is a controversial issue. Restrictions do not go away, they are like the laws of physics. The theory says that this is how innovations work, so it’s better to accept it and work with it, rather than being guided by faith in magic.



Realizing the upward nature of innovation, you will understand why some teams, companies or products get stuck in a fictional country of diminishing prospects. They simply lack the diversity and collective wisdom to find the best peaks to strive for. When Nokia closed their open-source projects, they cut off their oxygen.



A truly good developer with a good team can use SOD to create world-class products, quickly and accurately. To get the most out of SOD, a developer must use the product for a long time, starting from the first day, and develop his ability to sense problems such as inconsistency, unusual activity, and other types of problems. We tend not to notice many annoying phenomena, but a good developer pays attention to them and finds a way to patch them. The essence of the development process is to fix product problems.



In open source projects we do this work in public. There is no such moment “let's open the code”. When they do this, in my opinion, this suggests that people do not understand the meaning of open source projects - to involve users in your research and build a community around the main architecture.



Translation of the book "Social Architecture":

• Preface. Crowd wisdom
• Chapter 1. Toolkit
• Chapter 2
  
  - Emotional burnout of volunteers
  
  - How to capture / protect an open-source project
  
  - The myth of individual intelligence
• Chapter 3
  
  - ZeroMQ Community
  
  - Psychology of software architecture
  
  - Importance of contracts
  
  - 4 steps to a self-governing community
• Chapter 5
  
  - Design, development, innovation
  
  - Stratagem for success open source projects
• Chapter 6. Living Systems

---

Accepting applications to the accelerator for filtech startups continues

If you have projects / startups with a high share of technologies that match the values ​​of Filtech , feel free to submit an application .

Until February 25, there is still time!



Chat in Telegram

A community of people developing PhilTech projects or simply interested in the topic of technologies for the social sector.



#philtech news

News about projects in ideology #philtech and links to useful materials.



Facebook Page

News of Philtech startups, Russian and international events from the world of technology for philanthropy.



Subscribe to the newsletter

Weekly newsletter from the world of technology and philanthropy.