Registered Reports: A step change in scientific publishing
Professor Chris Chambers, Registered Reports Editor of the Elsevier journal Cortex and one of the concept’s founders, on how the initiative combats publication bias
By Professor Chris Chambers Posted on 13 November 2014
Last year, Cortex launched an exciting initiative called Registered Reports – a format of empirical article that places study pre-registration at the center of peer review.
Our aim with Registered Reports is to enhance the transparency and reproducibility of science by reviewing study protocols before experiments are conducted. If we think the protocol has merit we will commit, in advance, to publishing the outcomes. Armed with this provisional acceptance of their work, authors can perform the research safe in the knowledge that the results themselves will not determine the article's publication. At the same time, readers of the final paper can feel more confident that the work is reproducible because the initial study predictions and analysis plans were independently reviewed.
Gauging the community's reaction
Registered Reports represents a major departure from standard peer review, and at the time of the Cortex launch there was much uncertainty about how the shift would be regarded. Would the scientific community take notice? Would the format be popular? What kinds of submissions would we receive? To raise the profile of the initiative, we wrote an open letter to The Guardian, signed by more than 80 scientists and members of journal editorial boards. Together, we called for Registered Reports to be offered across the Life Sciences as a way to liberate academia from the grip of managerial incentives that favor the production of publications over genuine discovery.
Since then, the response from the scientific community suggests that Registered Reports are poised to transform the publishing landscape. In addition to Cortex, where the first completed articles will soon be published, we've seen the format taken up by more than a dozen journals across Neuroscience, Psychology, Psychiatry, Biology, Nutrition, and Medicine, with many more in the pipeline. I'd like to highlight just a few examples:
- Earlier this year, the journal Social Psychology published a high-profile special issue of Registered Reports that tested the reproducibility of classic psychological phenomena.
- A new journal, Comprehensive Results in Social Psychology, is dedicated entirely to the Registered Reports.
- eLife is hosting a special issue of Registered Reports to assess the reproducibility of cancer biology research.
- Another major journal, to be named in the coming weeks, is poised to unveil them across the full spectrum of sciences ranging from Physics to Psychology.
This is just a snapshot of the progress we've seen since last summer (see here for a complete and regularly updated list of journals, frequently asked questions, and more).
A range of problems threaten the integrity of the scientific method. Such practices have been documented most thoroughly in Psychology, which is characterized by a paucity of replication studies, insufficient statistical power, a high prevalence of cherry picking (also known as p-hacking), post hoc hypothesising, lack of data sharing, and a journal culture marked by publication bias. Registered Reports are designed to counteract all of these problems.
Understanding the appeal of Registered Reports
So why is this proving such an attractive reform? First and foremost, the idea of accepting papers before results are known moves us beyond the assumption that the visibility of a scientific study should depend on its outcome. A number of Social Sciences and Life Sciences journals are locked in the grip of this powerful bias, prioritizing the publication of positive, novel findings while rejecting those that are negative. Even in medical research, where trial protocols have been pre-registered for decades, null or negative findings are far less likely to be published. By selectively reporting positive results we distort the literature, needlessly populating journals with false conclusions and sabotaging the ability of science to self-correct.
The reason for this publication bias is simple human nature: in judging whether a manuscript is worthy of publication, editors and reviewers are guided not only by the robustness of the method but by their impressions of what the results contribute to knowledge. Do the outcomes constitute a major advance, worthy of space within a journal that rejects the majority of submissions? Results that are novel and eye-catching are naturally seen as more attractive and competitive than those that are null or ambiguous, even when the methodologies that produce them are the same. This bias, in turn, creates perverse incentives for individuals. When we reward scientists for getting "publishable results", we encourage a host of questionable practices to produce them (see Figure 1).
The unique selling point of Registered Reports is that they eliminate the need for scientists to strive for "publishable results". Registered Reports enshrine the ethos that science earns its stripes from the value of the research question and the rigor of the method, and never from whether the data sing a good tune. This idea is as old as the scientific method itself; in fact, it almost feels wrong to call Registered Reports an innovation in publishing when it is closer to being a restoration – a reinvention of publishing and the peer-review process as it was meant to be.
Some scientists have expressed fears that Registered Reports could restrict creativity by requiring authors to adhere to a fixed research methodology. In fact – and this is important to emphasize – the Registered Reports initiative places no restrictions whatsoever on creativity, flexibility or the reporting of serendipitous findings. While it is true that the pre-specified methods in a Registered Report must be followed, there are no bounds on the reporting of additional unregistered analyses. The only requirement is that such additional material is labelled transparently so that readers know which analyses were pre-registered and which were exploratory.
Where next for Registered Reports?
Ultimately, it is up to all of us to determine the future of any reform, and if the community continues to support Registered Reports then that future looks promising. Each field that adopts this initiative will be helping to create a scientific literature that is free from publication bias, that celebrates transparency, that welcomes replication as well as novelty, and in which the reported science will be more reproducible. Registered Reports isn't a one-shot cure for scientific publishing, but with every new journal that offers the format, and with every new article published, we strengthen the scientific record and offer scientists a positive incentive to embrace best practice.
Figure 2 illustrates the editorial pipeline for Registered Reports at Cortex, which has also been adopted by several other journals, including Drug and Alcohol Dependence. Details on review criteria and answers to frequently asked questions are available online.
Chris Chambers (@chrisdc77) is a Professor of Cognitive Neuroscience at Cardiff University, Section Editor for Registered Reports at Cortex and AIMS Neuroscience, and Chair of the Registered Reports Committee at the Center for Open Science. His main research interests include the psychology and neuroscience of human impulse control, the interaction between science and the media, and evidence-based public policy.