5th article that shaped my career: “Why Most Published Research Findings Are False” Ioannidis, 2005

“Published research findings are sometimes refuted by subsequent evidence, with ensuing confusion and disappointment. (…) There is increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims.”

Ioannidis, 2005

That is how starts the scary Ioannidis paper, which is still nowadays the most downloaded paper from PLoS Medicine (with 2,904,779 page views as I write). He has analyzed some of the most highly regarded published research findings in medicine over almost 15 years and found that out of 45 that claimed to have uncovered effective interventions, 34 claims had been retested and 14 of those (or 41%) had been proven wrong or significantly exaggerate.

It was my great friend Milton Moraes, that has always been a scientific reference for me, who told me to read it.

The high rate of non-replication (lack of confirmation) of research discoveries is a consequence of the convenient, yet ill-founded strategy of claiming conclusive research findings solely on the basis of a single study assessed by formal statistical significance, typically for a p-value less than 0.05.

Ioannidis, 2005

“The probability that a research finding is indeed true depends on the prior probability of it being true (before doing the study), the statistical power of the study, and the level of statistical significance”

Ioannidis, 2005

The model of null hypothesis for research studies is very powerful, but increases the probability of false positive findings. In my opinion, there is a general lack of statistical knowledge the drives articles to probably false positive conclusions from the beginning, when prior to the research, researchers did not create a proper experimental design. The ‘expectation’ of a given result (see again the importance of human factor?!), can also strongly influence the outcome of a study.

Finally, there is the lack of independent replicate testing. This is, I believe, not caused by ignorance, but rather by the same reasons presented in this paragraph.

The author explains that even though research efforts are globalized and several research teams are working on the same or similar questions, single teams focus on isolated discoveries and interpret research experiments in isolation. The result is that at least one study claiming a research finding which is unlikely to be true.

Based on his own observations, the author deduced several corollaries about the probability that a research finding is indeed true. And these… are so true!

• The smaller the studies conducted and the effect size in a scientific field, the less likely the research findings are to be true.
• The greater the number and the lesser the selection of tested relationships, the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true.
• The greater the financial and other interests and prejudices in a scientific field, the hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.

He goes on showing calculations to prove that most research findings are false for most research designs and for most fields in science. Even worst, according to him claimed research findings are, very often, simply accurate measures of the prevailing bias in a given field.

Ioannidis is a monster! He had authored more than 400 peer-reviewed papers, 40 books and book chapters or so, and much more. An article about him on The Atlantic Monthly (2010) tells the history of this physician who devotes his scientific career to educate medical doctors about the evidence they use in their clinic.

Large-scale evidence should be targeted for research questions where the pre-study probability is already considerably high, so that a significant research finding will lead to a post-test probability that would be considered quite definitive. Large-scale evidence is also particularly indicated when it can test major concepts rather than narrow, specific questions. (…) In some research designs, efforts may also be more successful with upfront registration of studies. (…) Finally, instead of chasing statistical significance, we should improve our understanding of the range of R values—the pre-study odds—where research efforts operate.

Ioannidis, 2005

This article is a must read for anyone worried about the pursue of the truth and the development of products and services based on scientific evidence.