Richard Hamming: Chapter 26. Experts

“What led you to success may become ineffective in the future.”

Hi, Habr. Remember the awesome article "You and your work" (+219, 2244 bookmarks, 351k readings)?

So Hamming (yes, yes, self-checking and self-correcting Hamming codes ) has a whole book based on his lectures. Let's translate it, because the man is talking.

This book is not just about IT, it is a book about the thinking style of incredibly cool people. “This is not just a charge of positive thinking; it describes the conditions that increase the chances of doing a great job. ”
')
We have already translated 8 (out of 30) chapters.

Chapter 26. Experts

(For the translation, thanks to Mitroshin Evgeny, who responded to my call in the “previous chapter.”) Who wants to help with the translation - write in a personal or mail magisterludi2016@yandex.ru

As noted in the fifth chapter, the number of problems appearing before us is growing exponentially, and as a solution, we cope with this with the help of specialization. This statement over the years only becomes more truthful:

The expert is the one who knows everything in something small, the erudite is the one who knows little, but about everything.

In a dispute between an expert and an erudite, an expert usually wins, simply because he uses incomprehensible terms and refers to his specific experience, which often has nothing to do with the subject of the dispute. However, the opinion of experts must be considered. Since experts at the same time play a very important role and, sometimes, hinder the development of the industry, their opinions should be evaluated from all sides. Often the expert underestimates the problem posed, and the erudite cannot cope with the task due to the lack of deep knowledge. Anyone who thinks that he understands the task, but in fact does not, is a real scourge, compared to those who know that he does not understand anything in the question posed.

Kuhn (note: Thomas Samuel Kuhn, American physicist, historian and philosopher of science) in his book Structures of Scientific Revolutions, considers the structure of scientific progress and introduces the concept of paradigm as a description of the normal state of science. He notes that in any single scientific field, there is almost always a set of generally accepted assumptions that are often not discussed, which are presented to students, and they, in turn, easily take them on faith. There is also a similar set of opposing hypotheses. Workers of science, working in this vein, expand and complicate the sphere to impossibility, often ignoring the contradictions that may arise in their way.

From time to time, usually because of these forgotten or ignored contradictions,
an unexpected paradigm shift arises, as a result of which new judgments become dominant and new questions appear along with answers to old problems. Changes of this kind in the generally accepted paradigm of science usually foreshadow significant progress in the development of the sphere. As an example, one can cite the special theory of relativity and quantum mechanics in physics.

At first, changes are accepted with bayonets by key individuals who put a lot of effort into working on the study of the previous picture, but usually, as Kuhn and other researchers notice, new ideas prevail over old ones. Sometimes the adoption of a new paradigm takes more time than was devoted to the life of the pioneer of new ideas! For example, I mentioned earlier the continental drift theory that Thomas Dick touched on in 1838 (note: here the author refers to the work of Celestial Scenery, or the Wonders of Heavens Displayed), and then Alfred Wegener in his book dated 1900. When my wife and I were children, we (independently of each other, because we were not familiar yet) read Alfred Wegener's book and noticed that, indeed, the coastlines of Africa and South America are very similar to each other, a powerful argument was also Wegener's observation - he notes, not only the shape of the shores is similar, but even some rock formations on two continents match each other like two parts of a mosaic! Despite the fact that even in the eyes of the child, this theory seemed obvious, experts of that time did not actively consider it, geologists did not take the theory seriously.

In favor of the theory of continental drift, there is another argument - the distribution of various forms of life among the ages. Similar forms were found in physically separated places, such that they inspired thoughts about "earthen bridges" that were once on the surface, and then sank. Their estimated number and location seemed incredible to me. Biologists studying the ancient life forms tried to explain their observations using the hypotheses about Panghei and Gondwana, the forerunners of modern continents, and did not particularly think about the mythical "earth bridges", however, geologists did not want to listen to anyone. The theory of continental drift was approved by oceanographers only after the Second World War, during the study of the ocean floor and magnetic anomalies, during which the theory of spreading was confirmed.

Of course, geologists can now argue that they always somehow believed in the drift and that it was necessary only to shed some light on the details in order for them to accept the theory that is now accepted as the most plausible. This is a typical example of a paradigm shift. At first, when it’s a long time, when it’s not very good, they don’t perceive the hypothesis (I don’t know how many theories in general have survived such a struggle, and I don’t know), however, when it still enters the minds, those who opposed it , they begin to assert that, in general, they never took it off completely. You have probably heard many similar examples, the Wright brothers heard before their historic flight that the device could not take off heavier than air, they also spoke of the impossibility of flying at supersonic speed, etc. Statements about the impossibility of doing something right before it comes to life sound amazing.

I really like one example, based on the assumption that it is impossible to pump water to a height of more than ten meters (note: 33 feet). When an employee of the patent bureau rejected a patent application claiming that this was possible, the inventor showed how he pumps water into the tank on the roof of his house, which was clearly located more than ten meters from the ground. How did he do it? He used the construction shown in fig. 26.1, resorted to the help of standing waves, which were not known in the patent office (note: here I am not an expert, perhaps we mean patent US3743446). The bureau considered this impossible, because it was written in the books on which those specialists studied, no one thought about what principles lay in the basics.

All evidence of impossibility is based on judgments that may or may not be applied in a particular situation.

When an expert encounters something new, his expertise and habit of looking at things remains with him. If something does not fit into his picture of the world, then it either goes unnoticed or is fitted to the picture. Therefore, truly innovative ideas are rarely born within the sphere, and, in general, experts would not be very correct to blame for this, because trying to explain something using old and proven methods is more appropriate before seeking new solutions.

Everything that is recognized as impossible is obviously based on some statements and if one or several statements turn out to be wrong, then the “impossibility” is questioned, however, experts rarely meticulously address the verification of basic assumptions before declaring impossibility. There is a rather old expression on this score:

“If an expert says that something can be done, then he is most likely right. But if you are told that it is impossible to do this, then you better ask someone else. ”

Kuhn and other historians of science were focused on studying significant changes; it seems to me that the principle is common to all, including the smaller ones.



For example, while working at Bell Telephone Laboratories, I naturally had to follow the frequency approaches to numerical analysis, and therefore apply it to the numerical methods I used for the problems that I was asked to solve.

If you use the functions that customers are familiar with, the conclusions from the solutions obtained may show that there are other things that they did not initially consider. I found the frequency approach very useful, but some of my close friends, not from Bell Telephone Laboratories, regularly made fun of me at every meeting over the years. They simply followed a polynomial approach, although when questioning they could not give any real reason for this - it was just how everything should be done, therefore, it was the right way to do something.

A blowdown in the direction of the experts is not the purpose of this chapter, the reasons why I am writing this below:

• First, you will repeatedly meet experts on your way and it will be nice if you are familiar with their type.
• Secondly, after a while, many of you will become experts and I hope that you will listen to everything said and will not hinder progress, as your predecessors sometimes did
• Thirdly, it seems that the current pace of development is much faster than they were before, which means that paradigms will change more often and you will have to endure more changes than I found
• Fourth, I would like to think that I have found the right words in order to save many of you from becoming useless, after the established paradigm in your area has changed.

Let me also touch on one point that we’ve hardly discussed before. It seems that most of the great, innovative ideas originate outside the sphere of research, rather than in the circle of its researchers. You remember the example of the theory of continental drift. Consider archeology - one of the central tasks that archeologists face is to determine the age of the finds.

Previously, it was solved only with the help of complex and not always reliable methods of stratigraphic dating. Now, in most cases, radiocarbon analysis is used. Where did he come from? From physics! Hardly any expert in archeology could come up with something like that. Another colorful example - the step finder (note: the switch's predecessor, an element of an automatic telephone station, patent US447918) was invented by the funeral home Elmon Strowger, annoyed by the telephone operator's behavior, whose husband was the owner of the rival bureau and calls to Elmon were often redirected to competitors.

Such cases take place in most scientific fields, but, unfortunately, they are not often mentioned in textbooks. Einstein at the beginning of his career worked in the patent office - he could not find official work in the university environment. A little later, he was recognized and got the opportunity to occupy prestigious positions, being in Berlin, and then in Princeton.

Thus, the expert is faced with a dilemma. Hundreds of crazy ideas are swarming outside his research, but among them there may be those who will later occupy a leading position. What strategy of behavior will be rational?

Many people prefer to ignore all new trends coming from outside, thus depriving themselves of the opportunity to adapt if the paradigm suddenly changes. Those who consider the proposed innovative ideas, can spend a lot of time in pursuit of a ghostly correct concept, which in the end will bring something new. Obviously, the strategy you choose depends entirely on your attitude to these ideas. I want you to be prudent when you come to this crossroads. You do not need to simply go with the flow, you have to see your goal and have a plan for how to get there. No need to immediately reject any, even perhaps insane, ideas as soon as they appear before you, especially if it was expressed by a person who is not an expert in this field of research - after all, it could be the very thing that will change everything! On the other hand, this does not mean that you can and should take into account everything that is in the air. Up to this point I have talked about changes in scientific paradigms, but, to me, this approach applies to any area of ​​research. It seems that the reason is the same everywhere - sometimes specialists are overly self-confident, they put a lot of effort into studying the current picture and are quite inert. I think you yourself can remember the many similar cases that have taken place in the history of science.

I touched on two main problems associated with expert opinion. The first problem is that they are extremely self-righteous. Secondly, they are not always able to assess where the scope of judgments ends, on which their knowledge and their applicability to new issues is based. Recall my story with fast Fourier transforms and the Cooley-Tukey algorithm, which could be called the Tukey-Hamming algorithm (note Chapter 16). This is not the only example where I made such a mistake, forgetting that progress does not stand still and can drastically change the view of things that seemed previously impossible. No matter how ashamed I was to mention this, I told you about this case in order to convey to you the importance of the question in all its glory. How do you plan to avoid such a scrape when your turn comes? At one time no one told me about such things, I hope you will not make mistakes like mine.

With more active development of technologies in the future, this kind of errors will occur more often, at least it seems to me. Experts live in their closed theoretical world, of course, they are often right, but they are deaf to the opinions of others. Despite all their merits, the experts are a real scourge of our society, with their infinite confidence in their unlimited knowledge and the lack of doubt in their infallibility. When you meet face to face with a weighty problem, remember my words from chapter 8 - “what will you accept as evidence of your wrong?” . Ask yourself this question more often, along with “why am I so sure that it works that way?”, Especially when it comes to the area where you are considered an expert.

Experts do not always so openly resist change, as I describe above. Let me review my experience at Bell Labs at the dawn of the computer age. All my leaders succeeded in the field of mathematics and in their golden years working with computers was not considered something prestigious, rather as a suitable occupation for yesterday's schoolchildren. The chiefs knew how to deal with mathematics, computers in their eyes were a rank far lower than a living mathematician, it was pointless to discuss this with them, I would say that as the machines developed, they began to hate them a little. I had to work with computers despite the rejection (usually implicit) of my leadership - they always said that with the help of a machine I could hardly solve a particular problem, although, usually, by that time it had already been solved by me. But I tried not to get involved in these disputes, it was more interesting for me to do something that they could not do, I was fascinated by the thought that a union of man and machine could create. I do not undertake to calculate now how many times I have cited this argument against the sometimes very implicit attacks on computers in those times. And this is in such a sublime place as Bell Labs.

I hope to influence your perception of the world, so that you avoid these ills and know-alls when you become experts. All I ask is that, please, remember more often all the features I mentioned and try to look at yourself from the side - are they characteristic of you? I made a promise to myself that when I climbed higher, I would behave more cautiously, in the end I refrained from participating in making decisions on current issues in the field of computer computing.



Of course, I will express my opinion if asked, but I don’t want to be a burden for the new generation, as my predecessors have sometimes been. For clarity, I tried to illustrate the pace of progress with the help of Figure 26.II. At the beginning, in the region of 1935 and earlier, the derivative is positive, those who knew their business at that time eventually gained success and these were my future leaders. When computers came into the picture, the picture changed slightly, now the derivative is negative. This suggests that the previous methods of work that previously led to success are no longer so widely applicable. There may be a small amount of truth in this chart. In my opinion, progress has not quite stopped, but rather it is gradually accelerating, which means that the advice below is useful to you:

What has led you to success may become ineffective in the future.

Please remember this when you are on the edge of progress and give the new generation a little more chance of success, more than your mentors gave you. Earlier, I told you (note in Chapter 4), as one of my friends argued that Hamming does not fully understand the error correction codes and I have to admit - it looks like he was right! I truly believe that the venerable expert is often a burden for progress. Again, Einstein, who took such an important step for the development of quantum mechanics, in his work related to the photoelectric effect (note here probably refers to the article of 1905 “On one heuristic point of view concerning the origin and transformation of light”), and later when quantum mechanics began to actively develop, opposed it. Physicists do not like to recollect this case, since he represents Einstein not in the best light, they forgive him for this, however they can recognize that the person who opened the new field for research did not understand the whole breadth of his discovery and, it would be better if later listened with a much more skeptical attitude.

Now the last and most powerful reason why I am telling you all this. I have seen and see how many experts again and again find themselves overboard after the subject area of ​​their expertise changes due to a paradigm shift. Take a look at the history of computer computing through my eyes.In Chapter 4, I told you about three things that were met with resistance among programmers: symbolic programming languages, high-level software and FORTRAN, at the time of its appearance. What happened to those who were against these innovations? In the end, they were not needed by anyone!

I have a close friend, a big fan of analog technology, who taught me a lot in the field of analog computers when I was a manager at Bell Labs. When digital technology began to evolve, he was always not averse to paying attention to the advantages of analog technology (which she had at that time). Little by little, he began to lose grip and in the end he had to do several other things. When I retired and went to teach (I always wanted to do this, it seemed to me that the old guard of researchers only inserts sticks in the wheels of the young), he also retired.

However, I left the company with good memories of the time spent in it, which can not be said about it, judging by our conversations after. If you do not develop with your professional environment, then you will be expected. You need to keep your nose in the wind. When I lived in California, I had the opportunity to communicate with former naval captains in stories which often expressed a dislike for the career path they had made. Could it be otherwise?

If you ignore an important (for you) promotion, it will probably not be the best way to reflect on the memories of your career later.

In these lectures I have repeatedly resorted to this kind of everyday stories. They describe many situations very well, I know many other examples. I began to notice these “theories” for quite some time, time and experience showed that some were true and others were not. This is not the ultimate truth, but simply conclusions from a multitude of observations, with which I try to prove the correctness of my convictions. Of course, you will say that I am only looking for confirmation of my guesses; however, as a scientist, I used to look for and refute, so some of the judgments were eventually rejected. If you consider what I have said, you will notice that the truth of most of these stories is very intuitive and relies more on human traits than anything else. Of course, we are all human, but this is not what gives us the right not to try to step over our instincts.We threw a thin veil over our primitive instincts and this veil is what we call civilization. To be civilized, in particular, means “to think first, and then do.” So I'm just trying to add a bit of self-criticism to you, so that you become a little more “civilized” and, perhaps, this will lead you to success.

At the beginning of the chapter, I told you how to handle experts, but at the end I gave you a couple of tips that will be useful to you when you yourself become an expert. Do not make the mistakes that I made!

To be continued...

Who wants to help with the translation - write in a personal or mail magisterludi2016@yandex.ru