Nature asks if modern scientific experiments are reproducible?

Accidentally in the stream of news and information came across an article in Nature Scientific Reports . It presents data from a survey of 1500 scientists on the reproducibility of research results. If earlier this problem was raised for biological and medical research , where it is explainable on the one hand (false correlations, the overall complexity of the systems under study, sometimes even scientific software is blamed ), on the other hand it has a phenomenological character (for example, mice tend to behave differently with scientists different sexes ( 1 and 2 )).

However, all is not smooth and with more natural science disciplines, such as physics and engineering, chemistry, ecology. It would seem that these disciplines are based on “absolutely” reproducible experiments conducted in the most controlled conditions, alas, amazing - in all senses of the word - the result of the survey: up to 70% of researchers encountered non-reproducible experiments and results obtained not only by other groups of scientists , BUT and by the authors / co-authors of published scientific works!

Every sandpiper praises its swamp?

Although 52% of respondents indicate a crisis of reproducibility in science, less than 31% consider published data to be fundamentally incorrect and most indicated that they still trust published works.
')

Question: Is there a crisis of reproducible results?

Of course, you should not chop off the shoulder and lynch all science as such only on the basis of this survey: half of the respondents were still scientists who were connected, in one way or another, with biological disciplines. As the authors note, in physics and chemistry the level of reproducibility and confidence in the results obtained are much higher (see the diagram below), but still not 100%. But in medicine, things are very bad compared to the rest.


Marcus Munafo (Marcus Munafo), a biological psychologist from the University of Bristol (England) has a long-standing interest in the problem of reproducibility of scientific data. Recalling the times of student youth, he says:

Once I tried to reproduce the experiment from literary sources, which seemed simple to me, but I just could not do it. I had a crisis of trust, but then I realized that my experience was not such a rarity.

Question: How many already published works in your industry are replicable?

Latitude and longitude depth problems

Imagine that you are a scientist. You come across an interesting article, but the results / experiments can not be reproduced in the laboratory. It is logical to write about this to the authors of the original article, ask for advice and ask clarifying questions. According to the survey, less than 20% have ever done it in their scientific career!

The authors of the study note that perhaps such contacts and conversations are too complicated for the scientists themselves, because they reveal their incompetence and inconsistency in certain issues or reveal too many details of the current project.

Moreover, an absolute minority of scientists attempted to publish a refutation of irreproducible results, while confronted with opposition from editors and reviewers, who demanded to minimize the comparison with the original research. Is it any wonder that the chance to report the irreproducibility of scientific results is about 50%?


First question: Did you try to reproduce the results of the experiment?
Second question: Did you try to publish your attempt to reproduce the results?

It may be worth then, at least within the laboratory, to carry out a check for reproducibility? The saddest thing is that a third of the respondents didn’t even EVER think about creating reproducible data verification methods. Only 40% indicated that they regularly use such techniques.


Question: Have you ever developed special techniques / technical processes to improve the reproducibility of results?

Another example, a biochemist from the United Kingdom who did not want to disclose her name, says that trying to repeat, reproduce work for her laboratory project simply doubles the time and material costs, giving nothing and not bringing new things to work. Additional checks are carried out only for innovative projects and unusual results.

And of course, the eternal Russian questions that began to torture foreign colleagues: who is to blame and what to do?

Who is guilty?

The authors identified three main problems of reproducible results:

• The pressure from the authorities that the work was published in time
• Selective reporting (apparently, it means the silence of some data that "spoil" the whole picture)
• Insufficient data analysis (including statistical)

Question: What factors are guilty of irreproducible scientific results?
Answers (top to bottom): –– Sample reporting –– Head pressure –– Poor analysis / statistics –– Insufficient experiment repeatability in the laboratory –– Insufficient supervision –– Lack of methodology or code –– Poor experiment planning –– Lack of raw data from the primary laboratory –– Fraud –– Insufficient testing by experts / reviewers –Problems with replay attempts –Required technical expertise to reproduce –Variance of standard reagents - “Nyudachka and pichalka”

What to do?

Of the 1,500 respondents, more than 1,000 experts spoke in favor of improving statistics in collecting and processing data, improving the quality of supervision by the bosses, as well as more rigorous planning of experiments.


Question: What factors will help improve reproducibility?
Answers (top to bottom): - Better understanding of statistics - More rigorous oversight - Improved planning of experiments - Training - Inside laboratory testing - Improving practical skills - Stimulating to formal data re-checking - Inter-laboratory testing - Allocating more time to manage projects - Improving scientific standards journals –– Allocating more time to work with lab records

Conclusion and some personal experience

First , even for me, as a scientist, the results are staggering, although I have already got used to some degree of irreproducibility of results. This is especially pronounced in the works performed by the Chinese and Hindus without external “audit” in the form of American / European professors. It’s good that they recognized the problem and thought about its solution (s). Tactfully keep silent about Russian science in connection with the recent scandal , although many honestly do their work.

Secondly , the article ignores (or rather, does not consider) the role of scientific metrics and peer-reviewed scientific journals in the emergence and development of the problem of non-reproducibility of research results. In pursuit of the speed and frequency of publications (read, the increase in citation indices) the quality drops sharply and there is no time left for additional verification of the results.

As they say, all the characters are fictional, but based on real events. It was somehow possible for one student to review the article, because not every professor has the time and energy to read articles thoughtfully, so the opinion of 2-3-4 students and doctors, which makes up a review, is collected. A review was written, it pointed out that the results were not reproducible by the method described in the article. This was clearly demonstrated to the professor. But in order not to spoil relations with the “colleagues” - after all, everything turns out for them - the review was “adjusted”. And these articles published 2 or 3 pieces.

It turns out a vicious circle. The scientist sends the article to the editor of the journal, where he points out the “ desired ” and, most importantly, “ undesirable ” reviewers, that is, in fact leaving only positive attitudes towards the authors. They review the work, but they cannot “foul in comments” in the black way and try to choose the lesser of two evils - here is a list of questions that need to be answered, and then we will publish the article.

Another example of which Nature’s editor was talking about just a month ago was the Grazel solar panels . Due to the huge interest in this topic in the scientific community (after all, they still want an article in Nature!), The editors had to create a special questionnaire in which they needed to specify a lot of parameters, provide equipment calibrations, certificates, etc., to confirm that the efficiency measurement method The panels conforms to some general principles and standards.

And, thirdly , when once again you hear about the miraculous vaccine, winning everything and everyone, a new story about Jobs in a skirt , new batteries or the harm / benefits of GMOs or radiation of smartphones , especially if it was promoted by yellowing writers from journalism, Then treat with understanding and do not make hasty conclusions. Wait to confirm the results of other groups of scientists, the accumulation of the array and data samples.

And what do you, dear Habra / GT users, think about the reproducibility of scientific data? Share your opinion in the comments!

PS: The article was translated and written in haste, about all the errors and inaccuracies noticed, please write in the LAN.

---

Sometimes briefly, and sometimes not so much about the news of science and technology, you can read on my Telegram channel - welcome;)