Is open peer review the way forward?
As our publishing peer review reports trial nears its second birthday, we hear from the participants
By Bahar Mehmani, PhD Posted on 22 September 2016
Would you reveal your name in a published review report? That’s the question we asked reviewers last year in an article about our publishing peer review reports pilot. The goal of the pilot is simple: make the peer-review process more transparent and improve the recognition reviewers receive for their work.
Now, almost two years on, we can see that reviewers are accepting review invitations knowing their comments to authors will be published if the manuscript gets accepted and many are publishing their names and even listing the review on their ORCID profile. Researchers are reading the reports and reviewers appear to be learning from them,
Piloting a radical idea
Open review has cropped up in a number of journals in recent years, reflecting a growing trend towards more transparent science. But it was something very new for Elsevier when we started the pilot 18 months ago.
Five journals signed up and their editors chose to have review reports typeset and published, in the format of an article, alongside the reviewed research paper on ScienceDirect. When invited to review for these journals, reviewers are informed of the pilot. Like with the normal peer-review process, they can then choose to accept or decline the invitation. If they accept to review, their comments will be published once the manuscript is accepted; however, the reviewer will remain anonymous unless they choose to be publicly named.
Five journals are currently involved in the pilot:
The review reports are made freely accessible, interlinked to the original articles and are given a separate Digital Object Identifier (DOI) – a unique character string used to identify electronic documents such as research papers. This means the reviewers who choose to publish the report with their name can claim the report as a publication and include it on their ORCID profile.
To find out how the pilot is going and determine whether to scale up, we asked reviewers who have published reports or declined to review what they thought about open review. We also interviewed editors and authors. What did we find?
Publishing peer review reports – impact on reviewers
Of the reviewers who accepted the invitation:
- 95 percent said publishing review reports didn’t influence their recommendation
- 76 percent said the fact their reports will be publicly available didn’t change their wording
- 45 percent gave us consent to reveal their names
- 36 percent of those who preferred to stay anonymous said they will reveal their names next time they review for the journal
- 98 percent said they will accept further review invites for the journal
Of the reviewers who declined, 91 percent said their decision was not influenced by the open review; most (68 percent) stated a lack of time as their reason for declining.
On average, 45 percent of reviewers who accepted the invitation revealed their identity. Dr. Mario Musella, Associate Professor of Surgery at University of Naples Federico II in Italy and one of the reviewers who revealed their identity, explained: “I definitely had no relationship with the authors, so I considered it fair to express my identity, with my points of view, in the review.”
Mr. Tim Williams, retired surgeon and Editor for the Travelling Surgical Society of Great Britain and Ireland, believes such transparency could also have benefits for the articles being reviewed.
“I am obsessional about my writing, and I would hope that others are about theirs too,” he said. “I think publication of reviewers' names may make them more conscious of the need for clarity and in some instances for restraint and encouragement. I would like to influence and improve the quality of the papers submitted, both in their science and in the way they are written.”
Publishing peer review reports – impact on editors and authors
Although many of the reviewers did not perceive a change in the way they conducted the review or wrote the report, 33 percent of editors did identify an improvement in the overall quality of review reports.
70 percent of those editors said the pilot resulted in reports that are “more in depth and constructive for authors to improve the quality of their manuscript.” It seems the pilot is helping researchers by providing them with examples of good peer-review reports as learning resources.
The pilot did not significantly impact authors’ decisions to submit; depending on the journal, 25-50 percent of authors said they would be more inclined to submit to a journal with open review. Only a small fraction of authors said they would prefer not to publish in such a journal.
Readers are certainly engaged: in many cases, one out of three clicks through to a pilot article on ScienceDirect led to someone reading the review report. This shows that usage is quite high and suggests open review is valuable to readers as well as reviewers, editors and authors.
Scaling up open review
With these positive results we now have the signal from reviewers, editors and authors to move forward with scaling up the pilot. We are developing the processes and systems in a way that minimizes the manual work required by journal managers and suppliers.
The pilot will continue until August 2017, at which point we hope to be ready to offer an open review option to more journal editors. By the end of 2017, we will be able to make it possible for reviewers to choose to publish their review reports in any Elsevier journal.
In the meantime, we are continuing to collect feedback from reviewers, editors and authors. What do you think? As a reviewer, would you like it if your report was published alongside the article? Would you reveal your name? Or, as an author, would you like to know who reviewed your article? We welcome your thoughts in the comments section below.
The success of the publishing peer review pilot to date is the result of a team effort. Thank you to the pilot team: Masako Takeda, Helen Heriz-Smith, Dominika Kaluza, Jan Hanraads, Gilles Jonker and Marco Casola.