Interviews with Peer Review Challenge winners
In our interviews with the three challenge winners we ask them questions about their winning ideas - 'Elsevier Reviewer Badges and Rewards,' Top Reviewer Incentives' and 'Peer Review Annotation Application'
By Ursula Culligan Posted on 1 December 2012
Elsevier Reviewer Badges and Rewards
Simon Gosling, Lecturer in Physical Geography, Nottingham University, UK
Q. Your idea which you have termed Elsevier Reviewer Badges and
Rewards scheme is about officially acknowledging the time and effort
spent by reviewers. What motivated you to come up with this idea?
A. As an early career researcher I have reviewed many papers, over 30 so far in my career, and whilst reviewing is something I enjoy and have learned a lot from, it does take up a lot of time. I felt that reviewers should get some recognition for the time and effort they invest and in coming up with my idea I wanted a system that would be fair and also sustainable.
Q. How important do you think it is for reviewers to have their work acknowledged in an official way?
A. I think it's very important. If a reviewer's contribution to the peer review process is more visible it should help in their career progression. In most institutions, when considering promotions a researcher's contribution to the academic community is one of the factors looked at but what isn't often considered is the role you have played in peer reviewing. I think it would be nice to have a standardized measure or index outlining reviewers' contributions to the peer review process.
Q. You suggest that badges would appear against a reviewers' online profile. Do you envision this online profile as being public?
A. I would see it as something that would ideally be public and hopefully it would then encourage researchers to write better reviews. In my experience as a Guest Editor of a Special Issue you don't always receive quality reviews. If there is public acknowledgment of reviewers' contributions which takes into account the quality of reviews they have carried out, this could be an incentive for some reviewers to put more effort into their reviews.
Q. The perception of someone being a good reviewer encompasses an
array of factors, the number of reviews they carry out, the quality of
the review and their timeliness, for example. Would the Reviewer Badges
scheme, which is based on the number of reviews carried out, be an
accurate measure of who is a good reviewer?
A. In my initial idea I mentioned the scheme being based on the number of reviews as an illustrated example, due to the word limit placed on challenge submissions. Ideally, in practice I would see a review being measured around a variety of factors, including things such as quality and timeliness. To give you an idea of how it could work, after reading a review editors would be required to answer 3-4 quick questions about the review. Answers would be selected from a scale, which could be a range from 1 to 10. Each question evaluating the review would be given a different weighting factor, depending on the importance of the factor being assessed. For example, quality could be given more weight than quantity. The combination of factors being assessed would be a pretty accurate measure of what constitutes a good reviewer. With these scores being recorded against reviewers, as well as it benefiting reviewers by recognizing their work there will also be a benefit for editors, as when selecting reviewers they will have a recognizable quality measure at their finger tips.
Q. Do you think institutes will take into account such a scheme when considering the career progression of researchers?
A. I think they would if it were introduced by a major publisher. For the scheme to be really effective and capture a researchers' total contribution to the peer review process the scheme would have to be adopted across the publishing industry. That would be my vision.
Q. One train of thought is that it is the responsibility of
researchers to review if they expect others to review their articles,
and that adding a reward element to the peer review process may not
motivate the best reviewers. What do you think about such views?
A. In assigning 'contributor points' to reviewers we would be looking at the overall quality of a review not just the number carried out and so through the scheme we would hopefully encourage researchers to write better quality reviews and to become better reviewers. There is a reward element, with the suggestion of discounts, but the main motivation behind my idea is to find a way of more publicly giving recognition to the work of a reviewer and to help with their career progression, not just within a reviewers' institute but internationally.
Visit the Peer Review Challenge website for further details of Simon's winning entry.
Top Reviewer Incentives
Michael Muthukrishna, Graduate student, Department of Psychology, University of British Columbia, Canada
Q. Your suggestion is to have a public reviewer points system. What format do you envision this taking?
A. I see it taking the form of the type of peer review you currently see being used by popular online websites and forums, such as Reddit, StackOverflow, Slashdot and Ebay. All vibrant peer review communities where reviewer "points" in the form of badges or "karma" are the only incentives. These incentives rely on nothing more than reputation, which we know to be a powerful motivator of human behavior. The way I would see it working would be something along these lines. A reviewer would submit their review to the editor who would give the review initial ratings, such as speed of review. The review, along with others submitted, would then be sent to the author in the normal way. After the final decision has been made the author would be sent a few brief questions about the review. The combination of the editor and author ratings would be combined into an overall score for the review based on an algorithm, which differentially weights scores based on factors, such as acceptance or rejection. This score would be added to the reviewers public points system. The issue of anonymity is very important and can be addressed by using pseudonyms for the display of reviewer points obfuscating the link between feedback and points, only providing a leader board, emailing reviewer points privately, or any combination of these.
Q. That reputation is a powerful motivator of human
behavior is the cornerstone of your idea. Do you think though that there
will be some reviewers who would be wary of contributions to the peer
review process being more public?
A. The reviewer points system though public would keep the identities of reviewers anonymous as reviewers would have a pseudonym and who was behind a pseudonym would never be revealed by those in control of the board. There could be reviewers though who are at the top of the board and who would wish to have their profile more public. They might choose a pseudonym which reveals their real name.
Q. How important is it for a young researcher to have the level of
their contribution to the peer review process officially acknowledged?
A. Acknowledgment is incredibly important. At present, reputations in the review process are missing, and the rewards for being a reviewer are small and not visible, therefore there are no real incentives to taking on a review. By having a reviewer points system where reviewers could use their reviewer identity on their CV for example, would give acknowledgment within their institute and the wider community as to the contribution they have made to the peer review process.
Q. You talk about a 'public review points system, rewarding
points for number of reviews, speed of reviewing papers and other
aspects of the review process that need to be incentivized.' What other
aspects do you think are important when evaluating contribution to the
peer review process?
A. I think the main aspect which is important, aside from quantity and speed which are mentioned, is the quality of a review. How useful the reviewers comments are for the editor and author is important to consider when evaluating contribution to the peer review process.
Q. Do you feel public acknowledgment would incentivize reviewers to take on more reviews?
A. I don't think reviewers would be incentivized to carry out more reviews but I do think that a public reviewer points system could act as an encouragement for reviewers to improve the quality of their reviews.
Visit the Peer Review Challenge website for further details of Michael's entry.
Peer Review Annotation Application
Koen Hufkens, Postdoctoral research associate, Faculty of Bioscience Engineering, Ghent University, Belgium
Q. Your idea is to create an integrated review application. Can you explain how you think this will help reviewers and authors.
A. One of my main concerns as a reviewer is that doing editorial work in different systems slows you down. If the whole process can be integrated then it would speed things up considerably. With an integrated review application reviewers would be able to make comments directly on the manuscript and there would then be an option to print off a summary, which would bring together all comments into a review template. The application should ideally be browser based, allowing the reviewer the possibility to go back and forth when working on a paper. From the author's perspective instead of receiving comments separately from their original manuscript the comments would be integrated, providing more clarity and ease in processing the requested changes.
Q. Developing online systems requires investment. What would be the benefits for publishers in making this investment?
A. There are already tools out there that offer similar services so I don't think an integrated review application would require a lot of investment but would rather give online systems submission systems a new purpose. One of the benefits to publishers, as well as for authors and reviewers, would be a quicker publication process, and by integrating the review process it would make it easier from the publisher stand point to keep track of the review process.
Q. It can be tough for editors to find willing and appropriate
reviewers who are motivated to review time and time again. Would you see
an improved online platform as a motivation to carry out more reviews?
A. I think so. What I find annoying is the time you have to put into reviewing a manuscript. Researchers are busy people and often have to review in their free time. If the amount of time they have to invest is reduced by an easier, more integrated system they may be willing to accept more review requests. For me personally, it's not the task itself which is off-putting but the time it takes to complete a review.
Q. When accepting to review for a journal, would an
easier and more integrated online system be as much of an incentive as
the quality and reputation of the journal itself?
A. The quality of a journal is an important factor but you can also be asked to review bad manuscripts for good journals. So it's not a given that a paper from a good journal will be easier to review. For myself the journal isn't as important as that it is an interesting topic and in my area of expertise. If in addition, the journal the request came from had an integrated review application that would be an added incentive.
Q. You suggest having a standalone app for mobile devices. Do you think this would be widely used by researchers?
A. Most scientists are up to speed with new technology and an app could provide researchers with an added opportunity to do their work more quickly, without having to learn much new technology. If the integrated review application tracks changes in a similar fashion to software which is already in use an app should be quick and easy to use.
Visit the Peer Review Challenge website for further details of Koen's entry.
Ursula Culligan has over 12 years experience in STM Journals publishing and has worked in both communications and publishing roles. Ursula is Customer Communications Manager based in Amsterdam and responsible for Editorial Communications and Elsevier's Reviewers' Update newsletter.