Spliting the review process into two stages
Journal Cortex launches Registered Reports
By Dr Chris Chambers and Professor Sergio Della Sala | Associate Editor and Editor-in-Chief of the Elsevier journal Cortex Posted on 8 May 2013
On May 1st, Cortex launched a new innovation in scientific publishing called a Registered Report. Unlike conventional publishing models, Registered Reports split the review process into two stages. Initially, experimental methods and proposed analyses are pre-registered and reviewed before data are collected. Then, if peer reviews are favourable, we offer authors “in-principle acceptance” of their paper. This guarantees publication of their future results providing that they adhere precisely to their registered protocol. Once their experiment is complete, authors then resubmit their full manuscript for final consideration.
Cortex is an international journal devoted to the study of cognition and of the relationship between the nervous system and mental processes.
Why should we want to review papers before data collection? The reason is simple: because the editorial process is too easily biased by the appearance of data. Rather than valuing innovative hypotheses or careful procedures, too often we find ourselves applauding impressive results or being bored by non-significant effects. For most journals, issues such as statistical power and technical rigor are outshone by novelty and originality of findings.
By venerating findings that are eye-catching, we incentivize the outcome of science over the process itself, forcing aside other vital issues. One of these sacrificial lambs is statistical power – the likelihood of detecting a genuine effect in a sample of data. Several studies in neuroscience suffer from insufficient statistical power, so – driven by the need to publish – scientists inevitably mine their underpowered datasets for statistically significant results. Many will p-hack, cherry pick, and even reinvent study hypotheses to ‘predict’ unexpected findings.
Such practices cause predictable phenomena in the literature, such as poor repeatability of results, a prevalence of studies that support stated hypotheses, and a preponderance of articles in which obtained p values fall just below the significance threshold. Furthermore, an anonymous survey recently showed that these behaviours are not the actions of a naughty minority – in psychology and neuroscience they are the norm. We ourselves are guilty.
Registered Reports will help minimise these practices by making the outcome of experiments almost irrelevant in reaching editorial decisions. Cortex is the first journal to adopt this approach, but our underlying philosophy is as old as the scientific method itself: If our aim is to advance knowledge then editorial decisions must be based on the strength of the experimental design and the likelihood of a study revealing definitive results – and never on how the results themselves appeared.
We know that other journals are watching Cortex to gauge the success of Registered Reports. Will the format be popular with authors? Will peer reviewers be engaged and motivated? Will the published articles be influential? We have good reasons to be optimistic. In the lead-up to Registered Reports, many scientists have told us that they look forward to letting go of the toxic incentives that drive questionable research practices. And our strict peer review will ensure that our published findings are among the most definitive in cognitive neuroscience.
George A. Lozano says: June 15, 2013 at 10:00 am
Finally! Something to force people to adhere to the scientific method.
J. Bruno Debruille says: August 5, 2013 at 1:58 pm
This is a very good initiative. However, how will you make sure that authors will submit before obtaining their data?
Chris Chambers says: August 5, 2013 at 3:45 pm
Thanks for your query. Authors must do the following when submitting a Stage 2 manuscript:
(a) collectively certify that no data collection (other than approved pilot data) was collected prior to the date of in-principle acceptance
(b) provide a laboratory log indicating the range of dates during which data collection took place
(c) upload time-stamped raw data files for free public access
This issue is address further in the detailed guidelines for Registered Reports:
And quick answers to FAQs may be found in slides 25-29 of this recent talk I gave at the University of Sussex:
J. Bruno Debruille says: August 5, 2013 at 4:34 pm
This is great.
You have a fantastic idea. I hope many journals will follow you.
J. Bruno Debruille, MD, PhD
Chair of the Research and of the Ethics board of the Douglas Mental Health University Institute, McGill
J. Bruno Debruille says: August 5, 2013 at 4:49 pm
As an REB chair, I would very much appreciate research projects accepted by reviewers internationally known for their expertise in the field. I hope your initiative eventually creates a standard and, we, REBs, will have to examine only research projects accepted by journals.
J. Bruno Debruille says: August 6, 2013 at 1:30 am
Another way of preventing submissions after data collection would be to work with research ethics boards (REBs). Research projects should be submitted to REBs only after being accepted by your journal. Then, when receiving the data, you could ask for the REBs letter of acceptance. You would have the date….