Are scientific editors reliable gatekeepers of the publication process?

Are scientific editors reliable gatekeepers of the publication process?

Gatekeepers
The Biological Conservation editorial team often acts as an editorial gatekeeper, as shown in this 2018 photo in front of an actual gate. From left to right: Danielle Descoteaux (USA, publisher) Richard Primack (USA), Robin Pakeman (UK), Tracey Regan (Australia), Vincent Devictor (France), Richard Corlett (China), Liba Pejchar (USA), Bea Maas (Austria) and David Johns (USA).

The scientific community assumes the publication process is reliable and fair, with the best papers being published only after rigorous review. Scientific editors act as “gatekeepers” in this publishing process, deciding whether a paper is even sent out for peer review, or alternatively “desk rejected”, that is returned to the author without peer review. While the review process has been extensively investigated - for example to determine reviewer consistency, and whether reviewers exhibit gender bias - as far as we know, the topic of editor consistency has never been experimentally examined. Do editors make arbitrary decisions on which papers to send out for review and which ones to desk reject? Or is there consistency in what editors decide to do?

We addressed these questions in respect of manuscripts submitted to the journal Biological Conservation, and the results of our study were recently published in the journal., We are reasonably confident, however, that the results are likely to apply to other scientific journals.

Study design

The handling process of Biological Conservation is structured so that all new submissions are sent directly to the Editor in Chief, who either desk rejects them or assigns them to an editors whose expertise is aligned with the subject, for further evaluation and/or initiating the review process.  For our study, conducted in 2018, we asked ten editors of the journal to each evaluate the same forty manuscripts, each of which had been submitted to the journal and previously evaluated in 2017 by an editor. All forty manuscripts had been filtered by the Editor-in-Chief in 2017 to be within the scope of the journal and of acceptable quality. They had then been delegated to handling editors that year. The manuscripts selected for the study represented a variety of topics and article types, a mixture of male and female corresponding authors, and involved authors from many countries. Of these 40 papers, the editors in 2017 had sent out 20 of these papers for review and desk rejected the other 20 papers.

The ten editors involved in the 2018 study did not know which manuscripts had actually been sent out for review (20 in total) and which had been desk rejected (20 in total) in 2017. Editors are generally assigned papers only in their area of expertise, so editors in this experiment were sometimes evaluating papers outside their normal topic areas.

Consistency among editors

We were pleasantly surprised to find that there was demonstrable consistency in the evaluations. While complete agreement among editors (that is, when all editors come to the same decision) was rare, for 73% (29) of manuscripts, seven or more of the 10 editors independently agreed on the decision of whether to review or desk reject the submission in question.

Agreement with past decisions

In 2018, editors tended to make decisions that echoed the actual decisions made in 2017. Of those manuscripts desk rejected in 2017, on average 70% of editors in 2018 agreed they should be rejected “again”. Similarly, when considering manuscripts sent out for review in 2017, on average 67% of editors agreed with the decision in 2018.

Importance of expertise

Editors were also asked if a manuscript was in their area of expertise or not. Surprisingly, editors did not manifest a higher rate of desk rejection for manuscripts outside their area of expertise. We regard this as a positive sign that editors can assess each paper on its own merits regardless of their individual background.

Overall, we found that editors are reasonably consistent in their decisions to send a paper out for review, or to desk reject it, and that they agreed with past decisions. However, disparities in agreement with decisions reveal the unsurprising subjectivity editors bring to the process. We are pleased that this study has demonstrated a significant degree of consistency in the peer review system in the area of editorial decision-making, but we see some room for improvement. One way in which we believe the process could be reinforced is by encouraging editors to seek the opinion of one or two additional editors before making a decision on papers that are not obvious candidates for either review or rejection. Whilst this would slow down the decision process for these papers, it seems reasonable that doing so would add an additional layer of rigor to the decision-making.

Transparency and fairness in publication

The editors of Biological Conservation are committed to improving transparency and fairness in the review and publication process. We have previously examined whether or not gender, nationality, or academic age affect acceptance rates for papers, how editors use reviews to make decisions, and the characteristics of our reviewers. We hope that the open communication of these results and of our practices will also inspire other editors to critically examine their own gatekeeping activities, and in the process promote research that is both robust and just. We welcome comments and questions about the study and how to implement its findings to improve the rigor of peer review; please comment below.

Tags


Contributors


Written by

Richard B. Primack

Written by

Richard B. Primack

Richard B. Primack is a Professor of Biology at Boston University (USA) specializing in how climate change affects plants and animals. He is the author of three widely used conservation biology textbooks; local co-authors have helped produce 36 translations with local examples. He is an Editor of the journal Biological Conservationand for nine years was Editor-in-Chief.  Richard frequently gives talks and writes for the public on issues of climate change, including the popular book Walden Warming: Climate Change Comes to Thoreau’s Woods.

Written by

Danielle Descoteaux

Written by

Danielle Descoteaux

Danielle Descoteaux is publisher of journals in the fields of ecology and biodiversity at Elsevier. She oversees seventeen journals, including the newly launched Environmental and Sustainability Indicators, an open access journal promoting research on indicators to drive environmental management, policy formulation and interdisciplinary research assessing complex environmental interactions. Danielle is committed to climate justice and volunteers with Mothers Out Front, an organization committed to a swift and just transition away from fossil fuels.

Written by

Vincent Devictor

Written by

Vincent Devictor

Vincent Devictor is a research director at CNRS (France) and the Editor-in-Chief of Biological Conservation. He specializes in community ecology and conservation biogeography issues. Vincent has developed concepts and techniques to transform large scale biodiversity monitoring data into meaningful indices for ecology and conservation. He recently completed a second PhD in the philosophy of science to better understand the science-policy interface involving biodiversity knowledge.

Written by

Laurent Godet

Written by

Laurent Godet

Laurent Godet is a research director at CNRS (France) working in biogeography. He is studying the impacts of landscape anthropization on biodiversity, at global (macroecology), regional (historical geography & natural history) and local scales (landscape ecology), mainly focusing on birds and coastal areas.

Written by

Lucy Zipf

Written by

Lucy Zipf

Lucy Zipf is a PhD candidate at Boston University in Richard Primack’s Lab. She studies the effects of climate change and land management on insectivorous birds and their prey. She is also involved in outreach and community education efforts through Mass Audubon.

Comments


comments powered by Disqus