Peer Review Researched

The first peer-review of scientific journal articles as we know it was conducted in the middle of the seventeenth century. More recently, research into peer review by Adrian Mulligan outlines practices for editors to effectively attract and retain the best-suited referees for their journals.

A Senior Research Analyst with six years' experience at Elsevier, Mulligan is responsible for the Author Feedback Programme (AFP), which helps in the understanding of author reactions. In 2002, he began to notice a sequence of responses in the AFP that focussed his attention on the referee process and prompted closer attention via a series of focus groups and interviews. "We monitor standards in refereeing and we began to see mixed responses," says Mulligan. The completed research has been presented at three recent editor's conferences and an article containing the findings has been published in   Perspectives in Publishing, no. 2, .

Author choice
"Prestige is obviously attached to an article published in a well-respected journal," says Mulligan. Notwithstanding this separate issue of the journal's reputation, Mulligan's research finds that the refereeing standard of a journal is second only to time-to-publication when an author selects a journal for publication.

In fact, refereeing is perceived to be a key element for authors when selecting a journal. Since contented, engaged referees who receive papers relevant to their field review papers more promptly than others, the issue of time-to-publication is essentially linked to refereeing.

Referee position
The single most unloved aspect of editor-referee relations from the standpoint of the referee is the issue of advance notice. Before committing to a review, referees overwhelmingly want to receive an abstract of the paper so they can gauge whether the paper is pertinent to their field. If a timeline for review completion is included with the abstract, they can gauge also whether they can commit time to the review.

What's more, a speedily-completed review process is seen by many referees as automatically inviting more requests. Because of this, many referees ignore the paper until they have been reminded to get on with the process by the journal editor. See graphic 1 for a summary of the juxtaposition of motivations identified for refereeing.

While reviewing is seen as an academic duty, a whole range of criteria influence a referee's decision to review a paper. For example, with more submissions from emerging and non-native-English-speaking authors, the basic issue of poor language can be a factor. As well, issues of authors' and referees' professional agendas and issues of anonymity continue to play a role. "A reviewer may be in competition with the author of a paper," says Mulligan. Of course, if the editor is aware of such a case he or she should move on to the next referee.

See graphic 2 for a view of the criteria found to be involved in selecting a paper to referee.

Editorial factors
When scouting for referees, editors often match possible referees to the subject of the paper under consideration. Mulligan's research supports the importance of the matching process. Mulligan suggests that "ScienceDirect® (the Elsevier electronic platform) makes electronic searching of field and citation a rapid process for editors." Scopus®, set for launch in October 2004, will make that process even easier.

Ideally, editors should adhere to a minimum standard in quality for papers, in terms of content and in terms of language. Mulligan's research shows that if referees are aware that there is a standard below which they will not be expected to review, referee engagement and satisfaction are more likely to improve.

Communication key
The study findings reaffirm that editor-referee communications are crucial. These communications span the full range of the process from personalising contact with each referee to making sure that referees know they are appreciated once the job is done. Probably the most important aspect is being in touch on an ongoing basis without disconcerting gaps.

Referees, particularly the best referees, feel overburdened with requests to review. Editors creating a maximum limit of reviews to be done per referee per year will remove the sense of overburdening. This can stimulate a speedier review process from relieved referees.

Follow-up communication is very important. Editors can circulate referee responses among the other referees (often anonymously) so referees can gauge their interpretation of the paper in light of the whole body of response. "Referee comments can be seen by other referees," says Mulligan. "This solution stops referees with an agenda." It is crucial, however, to maintain whatever anonymity referees find most comfortable.

Acknowledgement is widely seen by referees as an essential element of the process. A list of a journal's referees published annually is a simple gesture that carries a lot of goodwill. See graphic 3 for a list of further possible incentives to help retain referees.

Cycle improvement
Skilled, experienced authors want to submit papers to journals with high-quality, timely referees. With best quality referees and valuable papers, the quality of the journal improves. The research supports the perception that every aspect of the journal seems to improve when suitable referees can be attracted and retained. See graphic 4 for a summary of best practices identified by the study.

Expert relations
As simple as they all may sound, these issues are identified by the study as significant. A friendly and communicative relationship with the editor coupled with some recognition of the efforts made by the referee improves retention. Advance notice of papers and a system by which referees fulfil no more than a known and reasonable number of reviews per year add to referee satisfaction and efficiency. "We thought it would be useful to research and document the various approaches so that editors can make informed decisions," says Mulligan. See graphic 5 for a summary of conclusions.

Quality of practice
Of course, the single most important goal of every editor must be to produce the best quality journal possible. Not surprisingly, there are many journal editors already carrying out practices described in Mulligan's research. The research into peer review compiled by Mulligan is not a set of rules but it can provide a set of possible guidelines. An editor can choose to enhance their editorial routine from the body of practice described in the research. This can lead to the improvement of their journal's quality through the attraction and retention of suitable referees and, ultimately, the attraction of the best papers.

Please send responses to this article to EditorsUpdate@elsevier.com