Survivor bias

Do you read business blogs in which the author repeatedly describes his failures?
No, because you want to learn from a successful experience, and not to learn about the experience gained from a guy who has not yet succeeded.

However, the fact that you study only the experience of successful people can be a greater problem than it seems .

I will give a few stories that will help to understand the seriousness of this problem.
')

Bullet marks

During the Second World War, British bombers were sent daily to bomb the Germans. Most of the aircraft did not return home, and those that returned were covered with bullet marks from German anti-aircraft guns and fighters.

Wanting to increase the likelihood of crews returning home, British engineers studied the location of bullet trails. Engineers believed that in those places where the most hits, you need to additionally book the body of the aircraft. Of course, a pattern was found: many tracks were located on the wings, tail and in the area of ​​the tail gun. There were few tracks on the cockpit and fuel tanks.

It is logical to conclude that you need to add armor in places where there are most tracks. But this is not true.


Aircraft with bullets in the cockpit and fuel tanks could not return home , and on the returned aircraft traces of bullets were found just in fairly fortified places. Important information was from downed aircraft, and not from those who returned.

This is a literal example of “survival bias” - to draw conclusions based on data that is available or convenient, thereby distorting the results obtained methodically.

And do not suffer most of the tips on business the same? You read about success, but what about those businesses that “could not return home”? Perhaps , just like with downed planes, failures contain more useful information than successes?

Hiding other data

Scientific journals love to publish outstanding results. Therefore, the results of studies that did not lead to statistically important results are not published. Such results are discarded or quietly surrendered to the archive.

This practice is called the “card index effect” and this is a particularly insidious form of survivor bias, because everything happens unnoticed. Peter Norvig said this well :

When the published work says “statistically, this is possible in one case out of twenty,” then it is likely that similar experiments were conducted twenty times, but the results of other studies were simply not published.

Pharmaceutical companies use the effect of filing cabinets in order to deliberately distort results in the right direction. The problem has become so acute that the journals are calling for the creation of a publicly accessible database of research to prevent fraud:

For example, more than 66% percent of studies of the effects of antidepressants on depressed children showed that drugs are no better than sugar syrup, but companies published only research results with positive results.

If all studies were recorded, then doctors would know that positive results are only a fraction of all the results.

Washington Post

And do not suffer most of the tips on business the same? Harvard Business School publishes research only successful businesses. To paraphrase Peter, what if twenty other coffee houses had the same ideas, the same assortment and the same perseverance as Starbucks, but they did not succeed? How can such information affect what we know about the success of Starbucks?

Experimental evidence of extrasensory perception

Dr. Joseph Rhine applied the rigor of experimental psychology to the study of paranormal phenomena in general and extrasensory perception in particular. He created a sensation in the 1930s with his experiments on testing people for their ability to predict the order of cards in a Zener shuffled deck (cards with images of a circle, a square, a star, and wavy lines).

In a typical experiment, 500 people were tested for "strong telepathic abilities." It was believed that a person has such abilities, if he guessed the order of cards in a 25-card deck much more often than usual. Selected at the first stage were tested again and the part was again screened out. After the third test, one person was supposed to remain. It was believed that the actions of this person statistically confirmed the presence of extrasensory perception.

In order to show that this conclusion is just another type of survivor bias, consider the following experiment. I believe that some people are “eagles.” When tossing a coin, they drop an “eagle” much more often than it should according to probability theory. So, I put a thousand people in a room and ask everyone to flip a coin 10 times. Most likely, some Margaret will throw ten "eagles" in a row! The probability that she will throw ten “eagles” in a row is 1 out of 1024, so I’ll decide that Margaret has unusual abilities.

In fact, the last statement is all true, but it is confusing. The probability that Margaret will throw ten “eagles” in a row is 1 out of 1024, but I’ve done another experiment. I asked a thousand people to throw coins and “found” Margaret among them.

The likelihood that someone in a crowd of a thousand people will throw ten eagles in a row is as much as 62%! Due to the fact that so many people try to accomplish the feat, some usually unbelievable events happen. Margaret's abilities have nothing to do with it!

And do not suffer most of the tips on business the same? Take me for example. I founded three successful startups, here’s proof that I know what I’m talking about, and you should do it, as I advise you, right? Or, maybe, I just guessed the order of the cards three times, and there is no reason to believe that I would guess the fourth one.

Specific Examples of Survivor Bias at Business Advisors

Up to this point, I rhetorically asked if it was possible that business advice could be seriously susceptible to survivor bias. Steven Levitt , co-author of the Phreconomics, investigated this question specifically.

He read Good to Great by Jim Collins (Good to Great by Jim Collins), a book that explored 11 companies. These companies, from mediocrity, lagging behind, turned into a stock market sensation. It was concluded that all these companies have a “culture of discipline”. Many millions of copies of this book have been sold, and therefore it serves as a good example of books on the subject of business advice.

One of the eleven "excellent" companies was Fannie Mae (Fannie Mae) and Stephen Levitt was reading the book at the very moment when the company was dying under the influence of the financial crisis. Interesting, he thought, and how are things in the rest of the "excellent" companies?

It turned out to be not very good. Fanny May was not really the only example of total collapse. The book praised the now bankrupt company Circuit City.

Why didn't these companies continue to do well? It turns out that Jim began preparing for writing a book from a list of 1,435 companies and then chose eleven from this list. This is nothing more than the described experiment to prove the presence of extrasensory perception!

In addition, Jim did not bother to find out whether the remaining 1,424 companies practiced “discipline culture”. Maybe this is something that all companies do, regardless of the success achieved.

Maybe this book is an exception to the rule? No, Stephen researched another book from the 1980s - In Search of Excellence - and found the same thing.

Stephen came to the conclusion, to which I am leading you:

These business books are mostly retrospective: what did companies do to become successful? The future is always difficult to predict, and understanding the past is valuable, but on the other hand, the implicit message in these books is: the principles that these companies used not only brought them success in the past, but also allow them to succeed in the future.

Not only is the conclusion, as it turns out, not true, but it also calls into question the original message of these books, isn't it?

So how to make sure that some business advice does not suffer from the bias described? Of course, nothing. But, nevertheless, the very fact that you know about the existence of a survivor's bias means that you are less likely to fall under the influence of this bias. So reading this article will be of some benefit.

In addition, give preference to advice that will make you think and answer yourself to uncomfortable questions, rather than advice to march in the indicated direction. Use the tips as an amplifier of your own thoughts, do not take them on faith, as a sermon.