Reviewers may not always agree, but that's not necessarily a bad thing
Unanimous agreement among reviewers may make for a fairly easy decision for editors and give authors an unambiguous indication as to the quality and relevance of their work, but what happens when reviewers disagree?
A recent article published in Angewandte Chemie by Bornmann & Daniel (2008) examined the peer review process in the same journal and reported some interesting and powerful conclusions supporting the Peer Review Process.
The research measures the level of agreement between reviewers on some 1899 papers and found that reviewers show a low level of agreement on whether to accept or reject a submitted article to the journal – this agreed with a similar study on the same journal from the 1980s (Daniel, 1993). But rather than being a blow to peer review, authors of the papers say it shows that the editors have selected appropriate, active and competent reviewers who take this important role seriously. Indeed Bornmann & Daniel quote Bailar (1991):
“Too much agreement is in fact a sign that the review process is not working well, that reviewers are not properly selected for diversity, and that some are redundant”
Bornmanm & Daniel also used Scopus to analyse rejected articles from the sample (54%) and found that the majority were later published in journals with lower impact factors in all cases and on average were cited less than articles accepted by Angewandte Chemie. Both these findings affirm that the peer review process of the journal serves as a level of predictive validity – reviewers can often spot those papers that will go on to be important within the field.
Several other studies have also tried to assess the ultimate fate of rejected articles during the peer review process. Opthof et al. (2000) studied articles rejected by Cardiovascular Research in the mid 1990s and found that 50% were subsequently published in lower impact factor journals.
In a similar study, Liesegang et al. (2007) assessed over 2000 manuscripts submitted to the American Journal of Ophthalmology and found exactly the same result. Nemery (2007) reported that most rejected articles from Occupational and Environmental Medicine were subsequently published within two years of rejection and once again were in lower impact factor journals that the study journal. Similar results were also reported by Ray et al. (2000) with Annals of Internal Medicine and McDonald et al. (2007) with American Journal of Neuroradiology.
The fact that all of these studies have reported similar findings is an interesting perspective on the peer review process, suggesting that
- Reviewers make a judgement call dependent on the journal inviting them to review, asking the fundamental question: “is this paper right for this journal”.
- If a paper is rejected authors will automatically select a lower impact factor journal to re-submit their paper. These studies agree with similar findings from Elsevier author research.
- Reviewers of these lower impact factor journals would also change their review approach in accordance with the perceived level of the journal.
- A percentage of rejected papers (normally ranging from 7–25%) are never published: either the authors fail to have their papers accepted in another journal or decide to re-write their article and try again.
1. Bornmann, L., Daniel, H.D., Angewandte Chemie (2008) v.47, pp. 7173-7178
2. Bailar, J.C., Behavioural Brain Science (1991) v.14, pp. 137
3. Langfeldt, L., Social Studies Science (2001) v.31, pp. 820
4. Daniel, H.D., Angewandte Chemie (1993) v.105, pp. 247
5. Opthof, T., Furstner, F., Van Greer, M., Coronel, R., Cardiovascular Research (2000) v. 45, pp. 255-258
6. Leisegang, T.J., Shaikh, M., Crook, J.E., American Journal of Ophthalmology (2007) v. 143, pp.551-560
7. Nemery, B., Occupational Environmental Medicine (2001) v. 58, pp. 604-607
8. Ray, J., Berkwits, M., Davidoff, F., Annals of Internal Medicine (2000) v. 109, pp. 131-135
9. McDonald, R.J., Cloft, H.J., Kallmes, D.F., American Journal of Neuroradiology (2007) v. 28, pp. 1430-1434