Refereeing Behavior and the Determinants of Altruism

Collaborating with Elsevier on a study into how we can motivate pro-social behavior

Study Overview

Raj Chetty

We chose to collaborate with Elsevier on a study into how we can motivate pro-social behavior. Timely and useful refereeing work is usually understood to be a favor to colleagues, an unpaid but important task for the profession. As such, it is a good candidate for comparing financial incentives with other more inexpensive alternatives, in situations where the work benefits the public more than it rewards the worker.

The study has highlighted some key factors:

  • Deadlines are extremely effective.
  • Moderate financial incentives invoke a large response.
  • Public comparison to peers has a moderate effect.
  • There are different effects when looking at younger referees in comparison to established tenured ones.
  • There is no backlash when financial rewards are missing or phased out.

Conducting the Experiment

In 2010, we randomly assigned approximately 1,000 referees to one of four groups:

  • A control group with the usual 6-week due date.
  • A short deadline group with a 4-week due date.
  • A cash incentive group where a $100 payment was made for reports submitted by the shorter deadline.
  • A social incentive group where referees’ turnaround times were publicly posted on the journal’s website.

Emmanuel Saez

Always following the same process the majority of referees received multiple invitations throughout the study, which ended in November 2011. Using data retrieved from the Elsevier Editorial System (EES), the team studied refereeing times both before and after the experiment, as well as comparing times to other Elsevier journals. The differing approaches had small but significant impacts on whether referees agree to review a paper, but do not appear to have generated differential selection. The shorter deadline, of a 4-week due date, reduced turnaround times by an average of 10 days. The cash offer doubled this effect, further reducing referee times by an additional 10 days and the social incentive treatment reduced turnaround times by approximately 5 days. Less surprising in hindsight is that the study showed that tenured professors are most responsive to social pressure, while untenured referees are most responsive to deadlines and cash offers.

The study also showed that a common misgiving about financial incentives -- that they may crowd out intrinsic (or altruistic) motivation -- does not appear to apply here. Referees are no slower at concurrent or later unpaid jobs. Finally, all the approaches show modest impacts on the quality (length) of reports, the recommendations of the referees and/or the final decision of the editors.

Presenting the study

László Sándor

This study was presented at the National Tax Association Annual Conference in November 2011 and will be part of a session on journals and academic publishing at the Summer Institute of the National Bureau of Economic Research in July 2012. The session will be attended by editors of leading journals, who may decide to re-evaluate their own journal's policies based on the evidence from this innovative experiment conducted by the research team, along with Elsevier.

This article first appeared in Reviewers' Update, Issue 11, July 2012.

comments powered by Disqus