64% of psychology studies from 2008 could not be replicated

Free clipart from www.cliparthut.com
Free clipart from http://www.cliparthut.com

It’s not everyday that you are told – nay, proven! – that you cannot trust more than half of the published peer-reviewed work in your field. For nitpickers, I am using the word “proven” in its scientific sense, and not the philosophical “well, nothing can be technically really proven, etc…”

In an astonishing feat of collaboration, 270 psychologists from all over the world replicated 100 of the most prominent studies in their field, as published in 2008 in 3 leading journals: Psychological Science (leading journal in all psychology), Journal of Personality and Social Psychology (leading journal in social psychology), and Journal of Experimental Psychology: Learning, Memory, and Cognition (leading journal in cognitive psychology). All this without any formal funding! That’s right, no pay, no money, no grant (there was some philanthropy involved, after all, things cost). Moreover, they invited the original authors to take part in the replication process. Replication is possibly the most important step in any scientific endeavor; without it, you may have an interesting observation, but not a scientific fact. (Yes, I know, the investigation of some weird things that happen only once is still science. But a psychology study does not a Comet Shoemaker–Levy 9 make)

Results: 64% of the studies failed the replication test. Namely, 74% social psychology studies and 50% cognitive psychology studies failed to show significant results as originally published.

What does it mean? That the researchers intentionally faked their results? Not at all. Most likely the effects were very subtle and they were inflated by reporting biases fueled by the academic pressure and the journals’ policy to publish only positive results. Is this a plague that affects only psychology? Again, not at all; be on the lookout for a similar endeavor in cancer research and rumor has it that the preliminary results are equally scary.

There would be more to say, but I will leave you in the eloquent words of the authors themselves (p. aac4716-7):

“Humans desire certainty, and science infrequently provides it. […]. Accumulating evidence is the scientific community’s method of self-correction and is the best available option for achieving that ultimate goal: truth.”

Reference: Open Science Collaboration (28 August 2015). PSYCHOLOGY. Estimating the reproducibility of psychological science. Science, 349(6251):aac4716. doi: 10.1126/science.aac4716. Article | PDF | Science Cover | The Guardian cover | IFLS cover | Decision Science cover

By Neuronicus, 13 October 2015

Advertisements

Choose: God or reason

Photo Credit: Anton Darcy
Photo Credit: Anton Darcy

There are two different ways to problem-solving and decision-making: the intuitive style (fast, requires less cognitive resources and effort, relies heavily on implicit assumptions) and the analytic style (involves effortful reasoning, is more time-consuming, and tends to assess more aspects of a problem).

Pennycook et al. (2012) wanted to find out if the propensity for a particular type of reasoning can be used to predict one’s religiosity. They tested 223 subjects on their cognitive style and religiosity (religious engagement, religious belief, and theistic belief). The tests were in the form of questionnaires.

They found that the more people were willing to do analytic reasoning, the less likely they were to believe in God and other supernatural phenomena (witchcraft, ghosts, etc.). And that is because, the authors argue, the people that are engaging in analytic reasoning do not accept as easily ideas without putting effort into scrutinizing them; if the notions submitted to analyses are found to violate natural laws, then they are rejected. On the other hand, intuitive reasoning is based, partly, on stereotypical assumptions that hinder the application of logical thinking and therefore the intuitive mind is more likely to accept supernatural explanations of the natural world. For example, here is one of the problems used to asses analytical thinking versus stereotypical thinking:

In a study 1000 people were tested. Among the participants there were 995 nurses and 5 doctors.
Jake is a randomly chosen participant of this study. Jake is 34 years old. He lives in a beautiful home in a posh suburb. He is well spoken and very interested in politics. He invests a lot of time in his career. What is most likely?
(a) Jake is a nurse.
(b) Jake is a doctor.

Fig. 1 from Pennycook et al. (2012) depicting the relationship between the analytical thinking score (horizontal) and percentage of people that express a type of theistic belief (vertical). E.g. 55% of people that believe in a personal God scored 0 out of 3 at the analytical thinking test (first bar), whereas atheists were significantly more likely to answer all 3 questions correctly (last bar)
Fig. 1 from Pennycook et al. (2012) depicting the relationship between the analytical thinking score (horizontal) and percentage of people that express a type of theistic belief (vertical). E.g. 55% of people that believe in a personal God scored 0 out of 3 at the analytical thinking test (first bar), whereas atheists were significantly more likely to answer all 3 questions correctly (last bar)

First thing that comes to mind, based on stereotypical beliefs about these professions, is that Jake is a doctor, but a simple calculation tells you that there is 99.5% chance for Jake to be a nurse. Answer a) denotes analytical thinking, answer b) denotes stereotypical thinking.

And yet that is not the most striking thing about the results, but that the perception of God changes with the score on analytical thinking (see Fig. 1): the better you scored at analytical thinking the less conformist and more abstract view you’d have about God. The authors replicated their results on 267 additional more people. The findings were still robust and independent of demographic data.

Reference: Pennycook, G., Cheyne, J. A., Seli, P., Koehler, D. J., & Fugelsang, J. A. (June 2012, Epub 4 Apr 2012.). Analytic cognitive style predicts religious and paranormal belief. Cognition, 123(3): 335-46. doi: 10.1016/j.cognition.2012.03.003.  Article | PPT | full text PDF via Research Gate

by Neuronicus, 1 October 2015