Exploring Improvements to the Peer-Review System
New initiatives on easing the pressure on peer review
By John Lardee and Adrian Mulligan Posted on 4 June 2012
Peer review has a long history; it has been a part of scientific communication since the appearance of the first journals in the 1660s. The Royal Philosophical Transactions is accredited as being the first journal to introduce peer review.
Each year more than 1.3 million learned articles are published in peer-reviewed journals. Such is its importance that according to Ziman (1968)1 it is ‘the lynchpin about which the whole business of science is pivoted.’
However, the expansion of the global research community and the year on year increase in the number of papers published means the pressure on the peer-review system has grown. Moreover, as the pressure has increased, so too has the volume of those questioning peer review’s effectiveness. Some are worried by bias and are concerned it is not objective, others are anxious about the length of time it takes for an article to go through the peer-review process and some worry about its efficiency. Richard Smith2, former Editor of the BMJ, said the following about peer review in 2006:
“.. it is slow, expensive, profligate of academic time, highly subjective, something of a lottery, prone to bias, and easily abused.”
In response to the perceived challenges, peer review has evolved and continues to do so. Working with you, the Editor, we hope to be able to improve and streamline the peer-review process, ultimately easing the burden on both reviewers and Editors. In this article, we take a closer look at initiatives in Elsevier that tackle some of the challenges in peer review and evaluate the progress of some of these pilots.
Peer Review Grand Challenge
Running from March to May 2012, this web-based Challenge invited submissions on any aspect that could significantly add to the current peer-review system. Entries could range from designing a completely new system, to working within an existing peer-review method (like the single blind system).
The Challenge also welcomed entries that explored how publishers and Editors can help early career researchers become reviewers, or how reviewers can be recognized by either their institutes or publishers.
The entry phase of the Challenge closed on 7th May and the judges are now going through the submissions to pick out up to 10 finalists, whose ideas will be posted on the Challenge website. We will be inviting comments from the community on these ideas before the judges make their final decisions, taking into account any relevant community comments. Please do check the Peer Review Challenge website from 12th June onwards for details of the finalists!
For more information on this initiative, please contact Clare Lehane, Executive Publisher, STM Publishing, email@example.com
Cascading of manuscripts
As an Editor, you may frequently be confronted with manuscripts that are out of scope or are simply not suitable for the journal; however, they still contain sound research. With this in mind, we have developed the complementary Article Transfer Service (ATS) which allows the paper to be moved to a more appropriate journal. Currently, Editors within the fields of Pharma Sciences, Physics and Immunology, are able to offer authors this option and, if the author agrees, we can promptly transfer the manuscript on their behalf.
Key advantages of the Article Transfer Service:
- Editors can make faster, more informed decisions on manuscripts;
- authors receive faster decisions without the need to reformat or resubmit;
- reviewers benefit from a lighter burden due to a reviewer sharing policy where reviews have already taken place; and
- authors can publish in a journal that maximizes the impact of their research.
Results so far:
- Editors have offered to transfer up to 35% of rejected manuscripts and up to 35% of offered transfers have been taken up by authors.
- Up to 20% of those transfers have been accepted by the receiver journals.
We also surveyed a number of participants in the ATS scheme and discovered the following:
- 67% of Editors think that the ATS benefits the authors, while 75% agree that having reviewer reports is beneficial.
- 55% of authors are active promoters of the scheme.
- 86% of the reviewers are willing to recommend an alternative journal to the Editor.
For more information on this pilot, please contact John Lardee, Senior Project Manager, Publishing Services, firstname.lastname@example.org.
The Reviewer Guidance Program
From feedback we know that reviewers, especially those new to the task, would value more guidance on how to peer review. This program, which is still in the developmental stages, has been created to answer that need and will consist of both theory and hands-on practice.
Theory: By attending a Reviewer Workshop, participants will be introduced to the concept and background of peer review as well as peer-review fundamentals, publication ethics and the role of a reviewer. They will also examine a specific case study. Reviewer Workshops have been taking place for a while now and participants have told us that they feel more confident after attending one. Since it is not always possible to physically attend a workshop, we are now looking into the possibility of offering a distance learning (online) alternative.
“The Reviewer Guidance Program is not only an experience that helps early career researchers become better reviewers, but also to be more critical in analyzing their own papers before submitting. In addition, this is a great opportunity for junior scientists to network with their more senior peers.” Irene Kanter-Schlifke
Hands-on practice under mentorship: This part of the Reviewer Guidance Program aims to provide participants with the experience of independently reviewing at least two manuscripts inside a specially-created EES (Elsevier Editorial System). Each trainee is supported by a mentor who discusses the reviews with them and gives feedback and guidance. The mentor finally decides when a trainee has gained enough experience to start reviewing live manuscripts. After the program, each trainee receives a certificate of participation from Elsevier. We began piloting this module at the end of last year and the first feedback is promising. One trainee commented: “I’m now more familiar with rating papers and I’m more critical when I read papers.” The mentors involved in this module, often journal Editors, also see the benefits of this initiative; one remarked: “This module is a nice opportunity to learn how to efficiently review manuscripts. Often junior scientists have no idea how it works. As well, they can better understand how their manuscripts will be reviewed.”
During the Reviewer Guidance Program we will guide participants in how to write review reports in such a way that they answer the needs of both the Editor and the author. Another expected benefit is that the program should contribute to increasing the number of trusted – and usually enthusiastic – reviewers available for Editors to call on. Irene Kanter-Schlifke is a Publisher for Pharmacology and Pharmaceutical Sciences and is closely involved in the pilot. She adds: “The Reviewer Guidance Program is not only an experience that helps early career researchers become better reviewers, but also to be more critical in analyzing their own papers before submitting. In addition, this is a great opportunity for junior scientists to network with their more senior peers.”
If you are interested in organizing a Reviewer Workshop at your institute, please contact your publisher.
Results so far: We are currently evaluating feedback and expect to do a further pilot in due course.
For more information on this program, please contact Irene Kanter-Schlifke, Publisher Pharmacology & Pharmaceutical Sciences, STM Publishing, email@example.com, or Angelique Janssen, Project Manager, Publishing Services, firstname.lastname@example.org.
Published reviewer reports
Reviewers play such a vital role in the peer-review process yet their contribution often remains hidden. In addition, open reviewer reports increase peer-review transparency and assist good articles to gain authority. With that in mind, we thought why not publish reviewer reports alongside the final article on SciVerse ScienceDirect?
At the beginning of this year, we began doing just that on the journal Agricultural and Forest Meteorology.
We know from the feedback we have received that Editors welcome such a public acknowledgement of reviewers’ contributions, and we hope this step will enhance the quality of the review reports and help to capture / attract good reviewers for the journal.
How does it work?
Both authors and reviewers for the journal are informed about the new process and reviewers can indicate whether they want their name disclosed on ScienceDirect. Editors then decide if the reviewer reports are appropriate to publish alongside the article as supplementary material.
Results so far: The pilot launch attracted positive international media attention. It was also suggested that open reviewer reports could play a useful role in training early career researchers as reviewers. So far, reviewer reports have been published alongside around 13 manuscripts.
For more information on this pilot, please contact Gilles Jonker, Executive Publisher, Physical Sciences, email@example.com.
Open peer commentary format
In this pilot, we have asked experienced researchers to submit a one page comment on a (review) article for the journal Physics of Life Reviews. These comments are published in the same issue as the article. On average, five comments are published with the article and the author can write a rebuttal article.
Figure 2. An example of an article with open peer commentary in SciVerse ScienceDirect.
Results so far: Since the pilot was launched in January 2010, the journal has seen an increase in papers (2011 - 85 and 2010 - 74; previously the journal received around 12 papers per year). There has also been a sharp increase in usage – roughly 3,000 downloads per month compared to 2,000 per month in 2009.
For more information please contact Charon Duermeijer, Publishing Director Physics, firstname.lastname@example.org.
Traditionally in peer review, Editors have chosen to approach reviewers they consider are suitably qualified to comment on a manuscript, or who would find the subject matter interesting.
But what if the reviewer could select the manuscript themselves? For a year now, we have been experimenting with this additional peer-review system on the journal Chemical Physics Letters. Each week, a selected pool of reviewers receives an overview of the new submissions. If they like a paper because it matches their expertise and interest, they can decide to review it. Because they make the decision themselves, we ask them to review the manuscript within a week.
Martin Tanke, Managing Director of Elsevier’s STM Journals, explains: “The 2009 Peer Review Survey, which we conducted with our partner Sense About Science, showed that a significant number of reviewers were sometimes hesitant to review an article because of a lack of expertise in that particular field. In addition, researchers made clear they want to improve peer review by improving article relevancy and speeding up turnaround time. PeerChoice can contribute to solving both issues.”
Figure 3. An example of the email overview a reviewer receives.
Results so far: The time taken to review the manuscript has been slightly reduced, while the time taken to accept an invitation has been halved.
For more information on this pilot, please contact Egbert van Wezenbeek, Director Publication Process Development, Publishing Services,email@example.com.
All these pilots have been launched with one aim in mind; to support and improve the peer-review process to the benefit of Editors, authors and reviewers.
We would love to hear your thoughts on these new approaches and your suggestions for improvements.
1Ziman, J.M. (1968), Public Knowledge: an essay concerning the social development of science. London: Cambridge University Press.
2Smith, R. Peer Review: A Flawed Process at the Heart of Science and Journals. Journal of the Royal Society of Medicine April 2006 99.4: 178–182.
- A Helping Hand for Early Career Reviewers – Editors’ Update, Issue 32, June 2011
- ARTICLE TRANSFER SERVICE: Helping Authors Publish in the Right Journal – Reviewers’ Update, Issue 9, December 2011
SENIOR PROJECT MANAGER, PUBLISHING SERVICES
For the last 15 years, John has been involved in managing projects to improve author, Editor and reviewer experiences with Elsevier’s products and services. Recent projects include the Article Transfer Service and the Find Reviewer Tool. John’s approach to project management is an agile one: “To develop services and products iteratively together with our Editors, authors and reviewers”. John has a Master of Science in Informatics from the Technical University of Delft.
DEPUTY DIRECTOR, RESEARCH & ACADEMIC RELATIONS
Adrian has 14 years of experience in STM publishing. The last 10 years he has spent in research have given him the unique opportunity to study the scholarly community. Recently, in partnership with Sense About Science, Adrian worked on a large scale study that examined attitudes of researchers towards peer review. He has presented on peer review at various conferences, including STM, ESOF, AAP and APE. Adrian’s background is in archaeology with a BA Honours degree and a Master of Science from Leicester University. He also has a diploma in Market Research from the Market Research Society.
Andreas Demetriou says: June 6, 2012 at 8:26 pm
I applaud the whole initiative. I am particularly positive to the idea of having reviewers selecting manuscripts for review.
Roberto Torrent says: July 6, 2012 at 9:08 am
I like the idea of having reviewers selecting manuscripts for review. How can one proceed if too many reviewers opt for the same paper: by chronological order?. What explanation is given to those not selected to review the manuscript?.
Egbert van Wezenbeek, Director Publication Process Development says: July 6, 2012 at 1:32 pm
Thank you for your question. We do have mechanisms in place to prevent too many reviewers selecting the same manuscript. These include: 1) once four reviewers have selected the manuscript the selection function is disabled; 2) it is also disabled if two reviews have already been submitted; and 3) one week after the manuscript first appears on the system we remove it (these time parameters can be changed as desired). These measures are also explained to reviewers in the guidelines we send them.