[caption id="attachment_17305" align="alignleft" width="102"] Ursula Culligan[/caption] [caption id="attachment_17313" align="alignright" width="150"] Simon Gosling, PhD[/caption] When Reviewers’ Update invited readers to submit ideas to improve the peer-review system last year, more than 800 took up the challenge. Dr. Simon Gosling (@Simon_Gosling), Lecturer in Physical Geography at the University of Nottingham, was the overall winner of Elsevier's Peer Review Challenge. Elsevier’s journal publishing colleagues are looking into the possibility of a pilot this year to try out this idea, which he calls the Elsevier Reviewer Badges and Rewards scheme.
Here, Dr. Gosling talks with Ursula Culligan about his winning entry.
This Q&A is from the December 2012 issue of Reviewers' Update, which also featured interviews with runners-up Michael Muthukrishna and Dr. Koen Hufkens. Those interviews will be featured here in subsequent articles.
Ms. Culligan is Customer Communications Manager at Elsevier. She has over 12 years experience in STM journals publishing and has worked in both communications and publishing roles. Based in Amsterdam, she is responsible for editorial communications and Elsevier’s Reviewers’ Update newsletter.
[divider]Your idea, which you call the Elsevier Reviewer Badges and Rewards scheme, is about officially acknowledging the time and effort spent by reviewers. What motivated you to come up with this idea?
As an early career researcher, I have reviewed many papers — over 30 so far in my career — and whilst reviewing is something I enjoy and have learned a lot from, it does take up a lot of time. I felt that reviewers should get some recognition for the time and effort they invest, and in coming up with my idea, I wanted a system that would be fair and also sustainable.
How important do you think it is for reviewers to have their work acknowledged in an official way?
I think it’s very important. If a reviewer’s contribution to the peer review process is more visible, it should help in their career progression. In most institutions, when considering promotions, a researcher’s contribution to the academic community is one of the factors looked at, but what isn’t often considered is the role you have played in peer reviewing. I think it would be nice to have a standardized measure or index outlining reviewers’ contributions to the peer review process.
You suggest that badges would appear against a reviewers' online profile. Do you envision this online profile as being public?
I would see it as something that would ideally be public and hopefully it would then encourage researchers to write better reviews. In my experience as a Guest Editor of a Special Issue you don’t always receive quality reviews. If there is public acknowledgment of reviewers’ contributions which takes into account the quality of reviews they have carried out, this could be an incentive for some reviewers to put more effort into their reviews.
The perception of someone being a good reviewer encompasses an array of factors — the number of reviews they carry out, the quality of the review and their timeliness, for example. Would the Reviewer Badges scheme, which is based on the number of reviews carried out, be an accurate measure of who is a good reviewer?
In my initial idea, I mentioned the scheme being based on the number of reviews as an illustrated example, due to the word limit placed on challenge submissions. Ideally, in practice I would see a review being measured around a variety of factors, including things such as quality and timeliness. To give you an idea of how it could work, after reading a review editors would be required to answer three to four quick questions about the review. Answers would be selected from a scale, which could be a range from 1 to 10. Each question evaluating the review would be given a different weighting factor, depending on the importance of the factor being assessed. For example, quality could be given more weight than quantity. The combination of factors being assessed would be a pretty accurate measure of what constitutes a good reviewer. With these scores being recorded against reviewers, as well as it benefiting reviewers by recognizing their work there will also be a benefit for editors, as when selecting reviewers they will have a recognizable quality measure at their finger tips.
Do you think institutes will take into account such a scheme when considering the career progression of researchers?
I think they would if it were introduced by a major publisher. For the scheme to be really effective and capture a researchers’ total contribution to the peer review process the scheme would have to be adopted across the publishing industry. That would be my vision.
One train of thought is that it is the responsibility of researchers to review if they expect others to review their articles, and that adding a reward element to the peer review process may not motivate the best reviewers. What do you think about such views?
In assigning "contributor points" to reviewers, we would be looking at the overall quality of a review, not just the number carried out, and so through the scheme we would hopefully encourage researchers to write better quality reviews and to become better reviewers. There is a reward element, with the suggestion of discounts, but the main motivation behind my idea is to find a way of more publicly giving recognition to the work of a reviewer and to help with their career progression, not just within a reviewers’ institute but internationally. Visit the Peer Review Challenge website for further details of Simon’s winning entry.