How we can better support and recognize reviewers
New platform will provide reviewer rewards
By Dr. Joris van Rossum Posted on 3 March 2014
We know that finding, retaining and rewarding reviewers are long-term pain points for editors. Scientists are increasingly busy and often find it difficult to free up time to do reviews. At the same time, new approaches to peer review are being developed, for example, working in a more open and collaborative manner or making use of the latest technology. That makes these times challenging, as well as exciting, and this is reflected in the enthusiasm and energy with which new experiments are being launched within our organization.
My team is behind a number of these peer-review pilots and our decision to carry them out in an experimental setting, i.e. test the concepts with a limited number of journals, is deliberate. It means we can learn quickly and be flexible. If a pilot proves unsuccessful, we can swiftly shift our attention to other areas. However, if the results are encouraging, we can upscale and roll it out to more journal titles. Below I outline a few of the pilots currently taking place.
New platform will provide reviewer rewards
This experiment looks at addressing the need of reviewers to be better recognized for their work. Reviewers indicate that they like to review manuscripts; they feel it is an important service to their communities and it keeps them abreast of the latest developments. At the same time, we know they often feel that they are not fully recognized for their work.
With this in mind, Elsevier set up a Peer Review Challenge in 2012. We asked entrants to submit an original idea that would significantly improve or add to the current peer-review process. The winner was Simon Gosling, a Lecturer in Climate Change and Hydrology at The University of Nottingham. He proposed the creation of a ‘reviewer badges and rewards scheme' as an incentive for reviewers. Elsevier has since been working with him to implement his vision and, in early February, we began piloting a digital badge system with a selection of journals in our Energy portfolio. Via Mozilla OpenBadges, reviewers are issued with badges that they can display on their Twitter, Facebook and Google+ pages.
A second phase of the pilot is due to be launched this month - a ‘reviewer recognition’ platform for approximately 40 journals. Upon completion of a review for one of these titles, reviewers are provided with a link to a personal page on the platform that displays their reviewer activity. Based on their contributions to the journal, they are appointed statuses – for example, ‘recognized reviewer’ for those completing one review within two years, and ‘outstanding reviewer’ for those that have completed the most reviews. They are also able to download certificates based on their achievements and discount vouchers. We hope the platform will make the important work of reviewers more visible and encourage them to engage with Elsevier journals. Following the pilot, our aim is to make the platform available to all Elsevier titles.
We are continuously looking at how we can increase the visibility of the contribution made by reviewers; in another pilot, the journal Agricultural and Forest Meteorology has been making its review reports accessible on ScienceDirect. We now want to extend the experiment to more journals and see if we can provide the reports with DOIs (Digital Object Identifiers). In this way, the reports will be better acknowledged as an essential part of the scientific literature.
Article Transfer Service for soil science journals
As an editor, you may frequently be confronted with manuscripts that are out of scope or are simply not suitable for the journal; however, they still contain sound research. For some time now, we have been offering the complementary Article Transfer Service (ATS), which is currently active for more than 300 of our journals. ATS allows editors to recommend that authors transfer their submitted papers – and any accompanying reviews – to another Elsevier journal in the field, without the need to reformat them.
A new experiment with six Elsevier soil science journals aims to improve on this service. If participating editors decide not to accept a paper, they can now choose from two important options in Elsevier’s Editorial System (EES):
- They can choose a new option, ‘decline’, which means the paper is not suitable for their title. If this option is chosen, the author will always have the option to transfer the article, with the review reports, to another journal.
- They can decide to ‘reject’ the paper. If they choose this option, the author will not be invited to submit to any other journal in the pilot.
Gilles Jonker, Executive Publisher for soil science, explained: “The editors of these journals were confronted with a strong growth in submitted articles and found it increasingly difficult to find reviewers. To help address these issues, an agreement was reached to harmonize the editorial policies of the six journals, honor another editor’s decision to reject a paper, as well as give authors more autonomy in finding an alternative journal.”
Early pilot results show a good uptake by editors of the ‘decline’ decision option. Authors are also embracing the concept and are accepting transfers to journals within the cluster that better fit the scope of their articles. “Later this year we should be able to see whether this pilot study has indeed addressed reviewer fatigue and improved the quality of submitted articles,” said Jonker.
Experimenting with the peer-review process via Mendeley
Last, but not least, Elsevier is exploring ways in which Mendeley can be used to improve the peer-review process. Mendeley, a London-based company that operates a global research management and collaboration platform, was acquired by Elsevier in April 2013. Researchers worldwide use Mendeley’s desktop and cloud-based tools to manage and annotate documents, create citations and bibliographies, collaborate on research projects and network with fellow academics. These advanced collaborative features could benefit the peer-review process. Manuscripts can be annotated online, and these annotations can be shared in private groups. Moreover, editors and reviewers can discuss manuscripts in discussion forums. We are curious to see whether peer review within this environment will streamline the peer-review process, increase its efficiency and, in the end, lead to a better manuscript review. As part of this experiment, papers will be brought within the Mendeley environment - naturally only with the consent of the reviewers and editors. This pilot began with a few titles earlier this year. If it proves successful we will look to make it more widely available.
If you have any comments or suggestions for new peer-review pilots, I would really like to hear from you. You can contact me at firstname.lastname@example.org
- Learn about other initiatives designed to support reviewers in Recognizing your top reviewers, Editors' Update, Issue 39, June 2013.
Dr. Joris van Rossum
DIRECTOR PUBLISHING INNOVATION
For the past 12 years, van Rossum has been involved in the launch and development of products and initiatives within Elsevier. From its inception he worked as a Product Manager on Scopus, the largest abstract and citation database of peer-reviewed literature, and he worked on Elsevier’s search engine for scientific information as Head of Scirus. Later, he developed the Elsevier WebShop, which offers support and services for authors at many stages of the publication workflow. In his current role, van Rossum is focused on testing and introducing important innovations with a focus on peer review. He holds a master’s of science in biology from the University of Amsterdam, and a PhD in philosophy from VU University Amsterdam.
Robin Shorthose says: March 4, 2014 at 10:16 am
Experimentation is encouraging> Do you have mandated review periods to determine how successful your experiments have been? Please be gentle with older reviewers, if you need to keep them. We are often told what we CAN do, when it is often what we MAY do. Keep up the experiments..
Joris van Rossum says: March 4, 2014 at 2:39 pm
Dear Dr. Shorthose, thank you for your response and your encouragement. Indeed, for every experiment we determine how and when we will review results. This is a crucial element in executing experiments. In light of their experience, more senior reviewers are a great asset for Elsevier so we definitely want to ensure that our processes and systems are optimally designed for them as well.
Abdel-Zaher M Abouzeid says: March 4, 2014 at 8:52 pm
I suggested once to provide the active reviewers with a permanent access to ScienceDirect. The was, when one reviews an article you get a month's free access to ScienceDirect. This means there is no difference between an active and a non-active reviewer, which is discouraging the active ones.
Jacques Maurissen says: March 6, 2014 at 3:20 pm
As a reviewer, I spend an inordinate amount of time on problems that I keep seeing in submitted manuscripts. In my field, it is not unusual, as a reviewer, to have to ask the authors to reconcile the total number of animals with the sum of all animal groups, to ask for what happened to missing animals, to ask for how the multiplicity problem is addressed and what are its possible consequences, to ask the authors to use the appropriate unit for statistical analysis, to ask for missing data analyses for a number of otherwise tested variables, to ask the authors to address all the data they have collected, to ask the authors to address the assumptions underlying the statistical tests they use, to ask the authors to reconcile their statistical analysis with their study design, to ask the authors to also cite references that are contrary to their hypothesis, to ask them to read the papers they actually cite (some authors totally misrepresent what they cite), etc.
You write above the following: "We are curious to see whether peer review within this environment will streamline the peer-review process, increase its efficiency and, in the end, lead to a better manuscript review."
If the potential authors were to be made aware of the fact that these problems have to be addressed BEFORE submission of their manuscript, peer-reviewers will not have to spend as much time on these recurring problems, would possibly reduce the number of necessary revisions and, at least, would be able to increase the efficiency of the peer-reviewing process. This would make the peer-review process less burdensome, would potentially increase the quality of the submitted and published papers, and, consequently, would increase the credibility of the journal and the publisher.
I realize that now already the Editor screens out a number of totally inadequate manuscripts. If for the other manuscripts, the authors had to address the recurrently occurring problems encountered in many manuscripts before submitting, I would find it, as a peer-reviewer, much more rewarding to examine papers that have already reached a minimum standard of REPORTING, so that the reviewer can focus on the QUALITY of the manuscript, instead of having to ask for the same things over and over again. Spending less time on asking for missing information and more time on evaluating the substance of the paper would definitely result in a more rewarding experience for the reviewer, in the publication of potentially more credible and replicable findings, and, consequently, in an increased score for the Journal and for the Publisher. The only losers would be those who cannot provide the rigorous scientific information necessary to support their findings. At a time when many journals evaluate the lack of replicability of scientific findings (editorials, symposia, individual articles, …) and hoax papers have been able to get through the screen of peer-review, this would be welcome news.
D Seamon says: March 6, 2014 at 7:33 pm
Just so you know: I review a good number of manuscripts for one of your journals. Because of skyrocketing copying costs, our administration at my institution is telling us we may shortly have to pay for all paper copies we make. If that happens, I will no longer be able to review for your journals–I am not going to pay for copies that offer me no personal benefit. And don't suggest I use the digital copy for review. I need to work from paper.
I would suggest you start paying reviewers a set amount for each review. At least say you will pay for copying costs.
Joris van Rossum says: March 11, 2014 at 4:13 pm
Thank you all for your comments. Concerning financially compensating for peer review, we listen to views in both directions and, on balance, the editor and reviewer community at the moment seem to believe that the negative aspects of paying reviewers outweigh the benefits. The majority of editors seem to prefer that we don't pay reviewers. Besides the potentially negative effects (e.g. will a financial compensation not interfere with a genuine interest to review a scientific work?), there is also the issue of determining what the right level of compensation is.
Robin Mockett says: March 7, 2014 at 1:22 am
With publication fees being so high for open access journals, maybe reviewers could be given the equivalent of "frequent flier" miles in the form of decreased fees if their articles were accepted by publishers for whose journals they had reviewed. Such a system would allow greater rewards for greater numbers of reviews. For quality control, editors could assess the value of each review when it was received and assign a score that affected the size of the discount.
For frequent reviewers who no longer have active laboratories or write research manuscripts, an alternative recognition upon accumulating sufficient review service would be an invitation to write a guest editorial/commentary/opinion or review article, etc. (also subject to peer review, of course).
John Nagle says: March 9, 2014 at 5:58 pm
Not infrequently, a reviewer will make comments or suggestions, well beyond the usual oversights and clarifications, that make a substantially better paper, and the authors agree. The editor should ask both the authors and the reviewer if they would like the paper to acknowledge the reviewer. I would find this a better reward than a few dollars or a plaque.
Another reward: reviewers who have a lot say about a paper/field should be invited to write a perspective/comment for the issue where the paper is published. Of course, the perspective would also be reviewed. Some journals are doing this.
Joris van Rossum says: March 11, 2014 at 3:25 pm
Thanks very much to all of you who have made some great suggestions. I will look further into them. One aspect we'll have to take into account in this particular case, however, is the fact that not all reviewers want to reveal their names. But in instances where they are happy to do so, your suggestions make a lot of sense.
Joan-Albert Sanchez-Cabeza says: March 11, 2014 at 2:41 am
If paying reviewers was not an option (why not?), what about waiving costs of color figure reproduction or Open Access?
Dr John Philips says: April 30, 2014 at 8:35 am
For anyone who's not familiar with Open Access:
There are alternatives, but Open Access seems like the best option.