New article format aims to prevent 'sloppiness' in science
WIth Cortex journal's Registered Reports — featured in The Guardian — papers are reviewed before data collection to avoid publication bias
By Chris Chambers, PhD, and Sergio Della Sala, PhD Posted on 13 June 2013
Professor Sergio Della Sala of the University of Edinburgh is Editor-in-Chief of Cortex, an international journal devoted to the study of cognition and of the relationship between the nervous system and mental processes.
Here, the editors write about their Registered Reports program, which seeks to eliminate publication bias. The initiative was featured in a June 5 open letter to The Guardian: "Trust in science would be improved by study pre-registration," signed by Dr. Chambers, Dr. Marcus Munafo and more than 80 others.
A version of this article also appeared in Editors' Update.
[divider] On May 1st, Cortex launched an innovation in scientific publishing called a Registered Report. Unlike conventional publishing models, Registered Reports split the review process into two stages.
Initially, experimental methods and proposed analyses are pre-registered and reviewed before data are collected. Then, if peer reviews are favorable, we offer authors “in-principle acceptance” of their paper. This guarantees publication of their future results providing that they adhere precisely to their registered protocol. Once their experiment is complete, authors then resubmit their full manuscript for final consideration.
Why should we want to review papers before data collection? The reason is simple: because the editorial process is too easily biased by the appearance of data. Rather than valuing innovative hypotheses or careful procedures, too often we find ourselves applauding impressive results or being bored by non-significant effects. For most journals, issues such as statistical power and technical rigor are outshone by novelty and originality of findings.
By venerating findings that are eye-catching, we incentivize the outcome of science over the process itself, forcing aside other vital issues. One of these sacrificial lambs is statistical power – the likelihood of detecting a genuine effect in a sample of data. Several studies in neuroscience suffer from insufficient statistical power, so – driven by the need to publish – scientists inevitably mine their underpowered datasets for statistically significant results. Many will p-hack, cherry pick, and even reinvent study hypotheses to ‘predict’ unexpected findings.
Such practices cause predictable phenomena in the literature, such as poor repeatability of results, a prevalence of studies that support stated hypotheses, and a preponderance of articles in which obtained p values fall just below the significance threshold. Furthermore, an anonymous survey recently showed that these behaviors are not the actions of a naughty minority – in psychology and neuroscience they are the norm. We ourselves are guilty.
[note color="#f1f9fc" position="right" width=400 align="alignright" margin=10]
To submit a Registered Report
To find out more on how to prepare and submit a Registered Report, see the Cortex Guide for Authors.[/note]
Registered Reports will help minimize these practices by making the outcome of experiments almost irrelevant in reaching editorial decisions. Cortex is the first journal to adopt this approach, but our underlying philosophy is as old as the scientific method itself: If our aim is to advance knowledge then editorial decisions must be based on the strength of the experimental design and the likelihood of a study revealing definitive results – and never on how the results themselves appeared.
[note color="#f1f9fc" position="right" width=400 margin=10 align="alignright"]
Data freely available
The experimental data submitted with Registered Reports will be made publicly available so that other scientists can re-use it and explore the results in unanticipated ways, explained Dr. Zudilova-Seinstra, Content Innovation Manager for Journal and Content Technology at Elsevier STM Journals. She said this approach is part of Elsevier’s priority to be a data-friendly publisher.[/note]
We know that other journals are watching Cortex to gauge the success of Registered Reports. Will the format be popular with authors? Will peer reviewers be engaged and motivated? Will the published articles be influential? We have good reasons to be optimistic.
In the lead-up to Registered Reports, many scientists have told us that they look forward to letting go of the toxic incentives that drive questionable research practices. And our strict peer review will ensure that our published findings are among the most definitive in cognitive neuroscience.
"Confronting the 'sloppiness' that pervades science" —The Guardian, December 7, 2012
"Trust in science would be improved by study pre-registration" —The Guardian, June 5, 2013
"Scientific publishing as it was meant to be" — NeuroChambers blog, April 10, 2013
"Registered Reports: A new publishing initiative at Cortex" — Editorial in Cortex, March 2013
"Should research data be publicly available?" — Elsevier Connect, May 30, 2013