Pilot designed to help reviewers win recognition leads to better quality reviews, say editors

As Elsevier’s publishing peer review reports trial approaches its second anniversary, we hear what the research community has to say

By Bahar Mehmani, PhD     Posted on 26 September 2016

When five Elsevier journals agreed to join a new pilot project back in 2014, we shared a common goal; we wanted to see if we could make the peer-review process more transparent and improve the recognition reviewers receive for their work.

The method we chose was simple. We decided to publish reviewers' reports alongside the reviewed article on ScienceDirect and to give each review report its own Digital Object Identifier (DOI) – a unique character string used to identify electronic documents such as research papers.

Now, almost two years on, we can see that reviewers are accepting review invitations knowing their comments to authors will be published if the manuscript gets accepted and many are publishing their names and even listing the review on their ORCID profile. Researchers are reading the reports and reviewers are learning from them, and some of the editors involved have noticed an improvement in the quality of the reports they receive.

How the pilot works - the nuts and bolts

When invited to review for these journals, reviewers are informed of the pilot. Just as with the normal peer-review process, they can choose to accept or decline the invitation. If they agree to review, their comments will be published once the manuscript is accepted; however, the reviewer will remain anonymous unless they choose to be publicly named.

The review report is made freely accessible, interlinked to the original article, and the DOI the report receives means that if reviewer has chosen to reveal their name, they can then claim the report as a publication and include it on their ORCID profile.

Five journals are currently involved in the pilot:

• Agricultural and Forest Meteorology
• Annals of Medicine and Surgery
• Engineering Fracture Mechanics
• Journal of Hydrology: Regional Studies
• International Journal of Surgery

To determine the pilot's success and to aid our decision about upscaling, we asked the reviewers for those journals how they found the open review process. We also interviewed editors and authors.

Publishing peer review reports – impact on editors and authors

Although many of the reviewers did not perceive a change in the way they conducted the review or wrote the report, 33 percent of editors did identify an improvement in the overall quality of review reports.

70 percent of those editors said the pilot resulted in reports that are “more in depth and constructive for authors to improve the quality of their manuscript.” It seems the pilot is helping researchers by providing them with examples of good peer-review reports as learning resources.

Dr Riaz Agha BSc(Hons), MBBS, MSc Oxf, MRCSEng, FHEA, FRSA, FRSPH, is Managing and Executive Editor of International Journal of Surgery and Editor-in-Chief of Annals of Medicine and Surgery, two of the journals in the trial. He commented: "We felt publishing reviewers' comments was an important part of our research transparency program at the journals and we are glad Elsevier could facilitate it.

"The results of the pilot show that in both journals, 3-4 times as many people were more likely to publish in the journal than less likely to, and the majority of authors liked very much having reviewer reports next to their article once published. This is an important endorsement of greater research transparency and we are glad our authors support it.”

The pilot did not significantly impact authors’ decisions to submit; depending on the journal, 25-50 percent of authors said they would be more inclined to submit to a journal with open review. Only a small fraction of authors said they would prefer not to publish in such a journal.

Readers are certainly engaged: in many cases, one out of three clicks through to a pilot article on ScienceDirect led to someone reading the review report. This shows that usage is quite high and suggests open review is valuable to readers as well as reviewers, editors and authors.

Publishing peer review reports – impact on reviewers

Of the reviewers who accepted the invitation:

• 95 percent said publishing review reports didn’t influence their recommendation
• 76 percent said the fact their reports will be publicly available didn’t change their wording
• 45 percent gave us consent to reveal their names
• 36 percent of those who preferred to stay anonymous said they will reveal their names next time they review for the journal
• 98 percent said they will accept further review invites for the journal

Of the reviewers who declined, 91 percent said their decision was not influenced by the open review; most (68 percent) stated a lack of time as their reason for declining.

On average, 45 percent of reviewers who accepted the invitation revealed their identity. Dr. Mario Musella, Associate Professor of Surgery at University of Naples Federico II in Italy and one of the reviewers who revealed their identity, explained: “I definitely had no relationship with the authors, so I considered it fair to express my identity, with my points of view, in the review.”

Mr. Tim Williams, retired surgeon and Editor for the Travelling Surgical Society of Great Britain and Ireland, believes such transparency could also have benefits for the articles being reviewed.

“I am obsessional about my writing, and I would hope that others are about theirs too,” he said. “I think publication of reviewers' names may make them more conscious of the need for clarity and in some instances for restraint and encouragement. I would like to influence and improve the quality of the papers submitted, both in their science and in the way they are written.”

Scaling up open review

With these positive results we now have the signal from reviewers, editors and authors to move forward with scaling up the pilot. We are developing the processes and systems in a way that minimizes the manual work required by journal managers and suppliers.

The pilot will continue until August 2017, at which point we hope to be ready to offer an open review option to more journal editors. By the end of 2017, we will be able to make it possible for reviewers to choose to publish their review reports in any Elsevier journal.

In the meantime, we are continuing to collect feedback from reviewers, editors and authors. What do you think? As a reviewer, would you like it if your report was published alongside the article? Would you reveal your name? Or, as an author, would you like to know who reviewed your article? We welcome your thoughts in the comments section below.