On several occasions in the past year, I have spoken with journal editors about the process of peer review. For some, it is the cornerstone of their work, a pleasurable opportunity to interact within their professional network. But for others, it is a chore, and a dispiriting chore at that. When they speak of being ignored or turned down by reviewers, you hear distress and even puzzlement in their voices: ‘I’m doing my part, the author is doing hers, so why won’t you play yours?’

To be fair, there are probably as many reasons to turn down a review invitation, as there are to accept one. Everyone is busy, so we can take that as given. But there are some explanations that are harder to understand. I’ve noticed, to take one example, that some academics have an astonishingly small comfort zone and refuse entreaties to pass judgment on anything a millimeter outside their realm of expertise. That automatically involves turning down requests from another discipline, another country, and even a journal with which they are not familiar. In short, they like to feel that they are working on a warm, familiar Kuhnian puzzle, and they become unhappy, even disoriented, if asked to venture outside their own zone of reference.[1] For this type of reviewer, the biggest kick is trying to guess who wrote the manuscript in question, and by definition that demands that they stay in a small pasture. [In my experience, academics in this group loved graduate school, and have yet to relinquish their vinyl record collection.]

Other academics are more flexible but are more concerned for appearances. They have very refined radar for the academic caste system, know the impact factor of every journal in their field and absolutely won’t slide too far down the Web of Science rankings; some will review the paper as requested, but you sometimes get the feeling that it is hard for them to type and hold their nose at the same time.

To stay with line of thought for a moment—we might ask ourselves just what are the incentives to do a conscientious review? Done well, it can be a major time sink. Editors do occasionally sight the legendary Loch Ness monster of reviews—the three-to-four page response in which the referee goes back to the original data [perhaps in other fields, she returns to the archive, or builds a new laboratory], and essentially tests the paper in a Popperian manner until it drops to bits; alternatively, like a medieval torture test, if the paper floats, the author is innocent and it can be published.[2] With a review like this in hand, you feel as an editor that you can ask the author to fix the split infinitives, respect the journal’s word limit and generally re-write it to your specifications.

But consider the human cost. The anonymous reader spends a vast amount of time doing the work that the author should have done himself [or his advisor should have done with him]. She gets no tangible reward—and increasingly, there isn’t even the satisfaction of seeing better science in print. Many authors [especially those with publication pressures upon them] simply withdraw the manuscript, and cycle down the list of journals in the field until they reach one that will publish without all those pesky changes. Graduate students test out their abilities at referees’ expense but are often reluctant to work on revisions. And even with incontrovertible evidence that their work is flawed, plenty of authors are willing to argue out what they will and won’t do to pander to editors [as they see it]. So, in some ways, it is almost rational to provide the other kind of legendary assessment, the one that simply states ‘reject—this is unacceptable’. Of course, that’s succinct, but not actually very helpful, unless you plan simply to inform the author that ‘we must reject the paper as Professor Bauhaus insists upon it’.

I call this situation, in which authors manipulate the system at the expense of reviewers and editors, “free-ridership”, and it’s increasingly rational precisely because the academic system encourages it. In the UK, the departmental review process demands that personnel produce papers but offers no reward whatsoever for the chores that make it possible—such as reviewing manuscripts of books and papers. Much the same is true in the US, where an endless cycle of annual, three-, six- and twelve-year reviews, plus post-tenure assessments, insists upon production but does little to recognize the infrastructure that makes it all happen. Hence, all those unhappy editors.

So, how does one remain an editor and keep one’s sanity? From my perspective, there are a number of strategies that are helpful, even liberating. Here are some suggestions that can make one’s life easier without doing any damage to the integrity of the peer review process. Not all are relevant to every journal, but most have general applicability, in my experience.

• First, don’t review every paper. If you have any doubts about a manuscript’s ability to compete for scarce journal space, then save a great deal of time—and precious reviewer labor—and return the paper as ‘unsuitable for review’. In my experience—including that as an author—most of us would rather receive a rejection in a week rather than in three to six months. And if you are wrong, then the paper will surface elsewhere, with no damage done.
• Don’t expect to get three to five readers for every manuscript. Quality is more important than quantity, and if reviewers are drained less, then they are more likely to provide quality reviews in turn. And if you finish up with two reviews that disagree, use the Editorial Board to adjudicate.
• Don’t return to the same referees [or Board members] every time—use them sparingly, and ask yourself why you like Reviewer B: is it the quality of her insights, or just that she always replies within three weeks? Don’t sacrifice objectivity for productivity.
• See reviewers as part of the community served by the journal. If your journal supports a blog, make sure that you acknowledge their contribution. In some fields, it is customary to list all the reviewers used in the previous year. Invite the conscientious ones on to your Board—most institutions like to see their names and their affiliates in print.
• Work to build up a community of scholars who have an investment in the quality of the journal. It is certainly appropriate to use past authors, who should have positive feelings for the journal, and who have a vested interest in ensuring that the journal in which they publish has high standards.
• Be creative about finding new recruits and do not be apprehensive about their quality; some may not do a good job, but then you don’t have to keep using them. Go outside the comfort zone; if you are finding it hard to get responses from those on the faculty, then use advanced graduate students—after all, they should be knowledgeable in the field and should have more disposable time to devote to reviewing. And they should be keen to be seen to do a conscientious job as jobs get harder to find.
• Alternatively, look further afield; in many emerging economies, reviewing for foreign journals is often viewed as prestigious and is even rewarded; their use of the English language may be idiosyncratic but be realistic—it’s probably much better than your Turkish or Spanish.
• And finally stay with your database; grey is not the same as dead, and emeritus faculties are often pleased to be asked for their still-relevant opinions.

Most of you reading this will recognize these as tactical responses to the ever-increasing demand for journal space as the market place becomes truly global. My editorial workload has probably trebled in the past five years, and so these have been necessary patches and fixes. This still leaves bigger strategic issues untouched. There need to be structural rewards for all the aspects of publication—notably review—and not simply the production of material. Graduate students—those not driven from the academy by a shortage of jobs—must learn that being in a community involves obligations and not simply rewards. And of course we are all trying to deal with the opportunities, and costs, of new media and an avalanche of new technologies. These are, of course, bigger issues than we can deal with here, but we can make small changes to our own practice, with some helpful results.

Andrew Kirby has been a journal editor in one guise or another for nearly thirty years. He has sat on nine editorial boards and recently reviewed a manuscript for his 33rd journal. He welcomes comments.

To cite this article, please use: Andrew Kirby, “Editors’ View: Finding & retaining reviewers”, Elsevier Editors’ Update, Issue 27, August 2009

[1]. Referring to Thomas Kuhn’s ideas of normal science, in which an academic community is committed to a specific set of research puzzles.
[2]. Karl Popper told us that we should attempt to rebut an argument rather than confirm it.