[caption id="attachment_17305" align="alignleft" width="102"] Ursula Culligan[/caption] [caption id="attachment_13520" align="alignright" width="150"] Michael Muthukrishna[/caption]
When Reviewers’ Update invited readers to submit ideas to improve the peer-review system last year, more than 800 took up the challenge. Michael Muthukrishna, a PhD student in the Department of Psychology of the University of British Columbia, Canada, was one of two runners up in Elsevier's Peer Review Challenge.
Here, Muthukrishna talks with Ursula Culligan about his entry: Top Reviewer Incentives
This Q&A is from the December 2012 issue of Reviewers' Update, which also featured interviews with the winner, Dr. Simon Gosling (@Simon_Gosling), and Dr. Koen Hufkens. Elsevier Connect featured the interview with Dr. Gosling last month and will publish the interview with Dr. Hufkens in a subsequent issue.
Culligan is Customer Communications Manager at Elsevier. She has over 12 years experience in STM journals publishing and has worked in both communications and publishing roles. Based in Amsterdam, she is responsible for editorial communications and Elsevier’s Reviewers’ Update newsletter.
[divider]Q: Your suggestion is to have a public reviewer points system. What format do you envision this taking?
A: I see it taking the form of the type of peer review you currently see being used by popular online websites and forums, such as Reddit, StackOverflow, Slashdot and eBay — as vibrant peer review communities where reviewer “points” in the form of badges or “karma” are the only incentives. These incentives rely on nothing more than reputation, which we know to be a powerful motivator of human behavior. The way I would see it working would be something along these lines. A reviewer would submit their review to the editor who would give the review initial ratings, such as speed of review. The review, along with others submitted, would then be sent to the author in the normal way. After the final decision has been made the author would be sent a few brief questions about the review. The combination of the editor and author ratings would be combined into an overall score for the review based on an algorithm, which differentially weights scores based on factors, such as acceptance or rejection. This score would be added to the reviewers public points system. The issue of anonymity is very important and can be addressed by using pseudonyms for the display of reviewer points obfuscating the link between feedback and points, only providing a leader board, emailing reviewer points privately, or any combination of these.
Q: That reputation is a powerful motivator of human behavior is the cornerstone of your idea. Do you think though that there will be some reviewers who would be wary of contributions to the peer review process being more public?
A: The reviewer points system, though public, would keep the identities of reviewers anonymous as reviewers would have a pseudonym and who was behind a pseudonym would never be revealed by those in control of the board. There could be reviewers though who are at the top of the board and who would wish to have their profile more public. They might choose a pseudonym which reveals their real name.
A: How important is it for a young researcher to have the level of their contribution to the peer review process officially acknowledged?
A: Acknowledgment is incredibly important. At present, reputations in the review process are missing, and the rewards for being a reviewer are small and not visible, therefore there are no real incentives to taking on a review. By having a reviewer points system where reviewers could use their reviewer identity on their CV, for example, would give acknowledgment within their institute and the wider community as to the contribution they have made to the peer review process.
Q: You talk about a 'public review points system, rewarding points for number of reviews, speed of reviewing papers and other aspects of the review process that need to be incentivized.' What other aspects do you think are important when evaluating contribution to the peer review process?
A: I think the main aspect which is important, aside from quantity and speed which are mentioned, is the quality of a review. How useful the reviewers comments are for the editor and author is important to consider when evaluating contribution to the peer review process.
Q: Do you feel public acknowledgment would incentivize reviewers to take on more reviews?
A: I don’t think reviewers would be incentivized to carry out more reviews, but I do think that a public reviewer points system could act as an encouragement for reviewers to improve the quality of their reviews. Visit the Peer Review Challenge website for further details of Michael’s entry.