It’s not everyday that you are told – nay, proven! – that you cannot trust more than half of the published peer-reviewed work in your field. For nitpickers, I am using the word “proven” in its scientific sense, and not the philosophical “well, nothing can be technically really proven, etc…”
In an astonishing feat of collaboration, 270 psychologists from all over the world replicated 100 of the most prominent studies in their field, as published in 2008 in 3 leading journals: Psychological Science (leading journal in all psychology), Journal of Personality and Social Psychology (leading journal in social psychology), and Journal of Experimental Psychology: Learning, Memory, and Cognition (leading journal in cognitive psychology). All this without any formal funding! That’s right, no pay, no money, no grant (there was some philanthropy involved, after all, things cost). Moreover, they invited the original authors to take part in the replication process. Replication is possibly the most important step in any scientific endeavor; without it, you may have an interesting observation, but not a scientific fact. (Yes, I know, the investigation of some weird things that happen only once is still science. But a psychology study does not a Comet Shoemaker–Levy 9 make)
Results: 64% of the studies failed the replication test. Namely, 74% social psychology studies and 50% cognitive psychology studies failed to show significant results as originally published.
What does it mean? That the researchers intentionally faked their results? Not at all. Most likely the effects were very subtle and they were inflated by reporting biases fueled by the academic pressure and the journals’ policy to publish only positive results. Is this a plague that affects only psychology? Again, not at all; be on the lookout for a similar endeavor in cancer research and rumor has it that the preliminary results are equally scary.
There would be more to say, but I will leave you in the eloquent words of the authors themselves (p. aac4716-7):
“Humans desire certainty, and science infrequently provides it. […]. Accumulating evidence is the scientific community’s method of self-correction and is the best available option for achieving that ultimate goal: truth.”
Reference: Open Science Collaboration (28 August 2015). PSYCHOLOGY. Estimating the reproducibility of psychological science. Science, 349(6251):aac4716. doi: 10.1126/science.aac4716. Article | PDF | Science Cover | The Guardian cover | IFLS cover | Decision Science cover
By Neuronicus, 13 October 2015
The Science Portal 