Is peer review just a crapshoot?
How do reviewer recommendations influence editor decisions? And are Chinese authors treated fairly?
By Richard B. Primack, PhD, Ahimsa Campos Arceiz, PhD, and Lian Pin Koh, PhD Posted on 31 March 2015
An important concern in the scientific publication process is how well reviewers evaluate the quality of papers and how their recommendations influence editors’ decisions to accept or reject papers. Are there some comprehensible patterns in the review process or is it just a crapshoot — a random process with reviewers and editors making arbitrary decisions?
Additionally, a particular concern for researchers in China and other countries with rapidly developing scientific communities is whether there are potential biases affecting their manuscripts in the review process. Do editors treat authors from China fairly, or are their papers given an especially harsh treatment?
In our study, which was recently published in the journal Biological Conservation, we examined 4,575 manuscripts submitted to Biological Conservation over the past 7 years, and a selected subset of 2,093 papers that were sent out for review.
Right from the beginning, we could see that Chinese authors face special difficulties in getting their papers published: 71% of the papers from China were rejected without review in contrast to a far lower 50% of papers from English-speaking countries. Although identifying the causes of the higher rejection rate was beyond the scope of our paper, the reasons are probably due to a combination of the topics being less relevant to the journal, lower quality of research work, and less effective presentation of the material.
How do reviewer recommendations affect editorial decisions?
Of the articles that were sent out for review, reviewer recommendations strongly influenced the outcome of the review process and the decisions by the editors. Papers having two reviewers’ recommendations to accept and minor revision have much greater chance of acceptance (more than 98%) than papers receiving two reviewers’ recommendations that include a major revision and a reject or two rejections (which results in an acceptance rate of less than 5%): (Figure 1).
When articles from China and English-speaking countries were compared, similar reviewer recommendations resulted in comparable outcomes. However, papers from China that were sent out for review had more negative reviewer recommendations of reject and major revision than did papers from English-speaking countries (Figure 2).
Of papers that were sent out for review, the acceptance rate is 51% for papers from English-speaking countries and 42% for papers from China. If we consider the fate of all the submissions, including the papers that were rejected without review, papers from English-speaking countries have about a 25% acceptance rate and papers from China have a 12% acceptance rate.
Reviewer rejection as the “kiss of death”
In considering the effects of reviewer recommendations on editors’ decisions to accept or reject a paper, it is surprising that even though there are four different categories of reviewer recommendations (accept, minor revision, major revision and reject), editors clearly do not use these categories on a simple linear scale. Reviewer recommendations of accept and minor revision are virtually identical in their positive effects on editor decisions; two such recommendations of Accept and Minor result in a greater than 98% of acceptance (Figure 1).
In contrast, papers receiving at least one recommendation of reject (“the kiss of death”) were almost always rejected; in fact receiving even one reviewer recommendation of reject lowers the probability of a paper being accepted by more than 50% when combined with a reviewer recommendation of accept or minor revision. Recommendations of major revision resulted in a greater than 16% decrease in the probability of the paper being accepted in contrast with a recommendation of a minor revision.
Variability of reviews has a negative outcome
Papers with more consistent reviews (e.g., both reviewers recommending a major revision) had a greater chance of acceptance than did papers with more variation (e.g. minor revision and reject). Similarly, papers with a minor and major recommendation had a great probability of acceptance than papers with reviewer recommendations of accept and a reject. These results are likely due to editors weighting negative recommendations more heavily than positive ones, especially recommendations of reject.
We found no evidence of editor bias against papers from China when receiving the same reviewer recommendation; however, reviewer recommendation for papers from China had a greater degree of agreement than did reviewers of papers from English-speaking countries (e.g., intra-class correlation of 0.25 vs. 0.55), due to reviewers of papers from China often agreeing that papers should be rejected or require major revision. The significant agreement between pairs of reviewers who are reviewing the same paper (as reflected by significant intra-class correlations) for papers from both China and from English-speaking countries indicates that reviewers are able to determine the quality of the paper using objective and repeatable criteria. However, the fact that these correlations are much lower than 1.00 also indicates that most of the time the two reviewers of the paper are coming up with different opinions on the quality of the paper.
Harsh reviewers from China
In their cover letter, authors from China sometimes request reviewers from China, presumably hoping that Chinese reviewers will provide more favorable or sympathetic reviews than will reviewers from outside of China. In fact, we found the opposite, with reviewers from China judging papers from China more harshly (43% recommending rejection) than did reviewers from other countries (with only 15 % of the reviewer from the UK recommending rejection).
Reviewers provide useful information, and editors make reasonable decisions
Our results demonstrate that the review process is not a crapshoot; reviewers are providing useful information and editors are using this information to make reasonable decisions. Authors from a non English-speaking country such as China have a tougher time getting their work published, but this is not due to a negative bias by reviewers from English-speaking countries or editors. The next step in evaluating the review process might be to examine the usefulness of the extensive comments of the reviewers in the decision-making progress. And also, more attention needs to be devoted to supporting scientists from China and other developing countries to help them reach their potential in carrying out and publishing their research.
Read the study
Elsevier has made this article freely available for six months, until September 24, 2015:
Campos-Arceiz, A, Primack, RB, and Koh, LP: “Reviewer recommendations and editors’ decisions for a conservation journal: Is it just a crapshoot? And do Chinese authors get a fair shot?” Biological Conservation (June, 2015)
- Baethge, C, Franklin, J, Mertens, S: “Substantial agreement of referee recommendations at a general medical journal – A peer review Evaluation at Deutsches Ärzteblatt International,” PLoS ONE (2013)
- Bornmann, L., Mutz, R., Daniel, H-D: “A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants,” PLoS ONE (2010)
- Lee, CJ, Sugimoto, C.R., Zhang, G., Cronin, B: “Bias in peer review,” Journal of the American Society for Information Science (2012)
- Primack, RB: “Editorial: Why did we reject your article?” Biological Conservation (2009)
- Primack, RB, Ellwood, E, Miller-Rushing, AJ, Marrs, R, Mulligan, A: “Do gender, nationality, or academic age affect review decisions? An analysis of submissions to the journal Biological Conservation,” Biological Conservation (2009)
- Primack, RBMarrs, R: “Editorial: Bias in the review process,” Biological Conservation (2008)
- Xie, Y, Zhang, C, Lai, Q, “China’s rise as a major contributor to science and technology,” Proceedings of the National Academy of Sciences (2014)
Elsevier Connect Contributors
Dr. Richard B. Primack is Professor of Biology at Boston University and Editor-in-Chief of Biological Conservation, an Elsevier journal focusing on the protection of biodiversity. He is also a Humboldt Research Fellow in Germany and Distinguished Overseas Professor at the Northeast Forestry University in China. Dr. Primack studies the effects of climate change on the plants and animals of Massachusetts using the observations of Henry David Thoreau from Concord in the 1850s as a starting point. This research is the focus of his new popular book titled Walden Warming: Climate Change Comes to Thoreau's Woods.
Dr. Ahimsa Campos-Arceiz (firstname.lastname@example.org) is Associate Professor in the School of Geography at the University of Nottingham Malaysia Campus. He is a conservation ecologist, and his main research interest lies in the behavior, ecology, and conservation of Asian megafauna, especially elephants. He is based in Peninsular Malaysia, where he leads the Management & Ecology of Malaysian Elephants (MEME) research project and works closely with local authorities. Dr Campos-Arceiz’s work is increasingly leaning towards the social sciences because he believes that understanding (and changing) human behavior is the main challenge for tropical conservation science in the 21st century. In his free time, Dr. Campos-Arceiz also enjoys playing with data in side-projects like the one described in this article.
Dr. Lian Pin Koh is Associate Professor and Chair of Applied Ecology & Conservation at the University of Adelaide in Australia. He was recently awarded an Australian Research Council Future Fellowship (Level II). Lian Pin’s research focuses on addressing emerging environmental and socioeconomic challenges facing tropical developing nations, including the threats of intensifying land-use conflicts to natural ecosystems and wildlife.
Lian Pin is Founding Director of the nonprofit ConservationDrones.org, Regional Technical Advisor for Conservation International, a TED speaker, Editor-in-Chief of Cogent Environmental Science, and an Editor of Biological Conservation.
He obtained his bachelor's and master’s degrees from the National University of Singapore in 2003. He received his PhD in Ecology and Evolutionary Biology from Princeton University in 2008. He then moved to Zurich for his post-doctoral training at the Swiss Federal Institute of Technology (ETH Zurich). In 2011, he was appointed a Swiss National Science Foundation Professor.