When peer reviewers go rogue: collaborating with Wageningen University to detect and prevent citation manipulation

Wageningen and Elsevier are developing innovative analytical methods to detect citation manipulation; view their analysis of 500,000 reviewers

Philippe Terheggen quote

Occasionally, adding citations to scientific articles during the peer-review process can contribute to improving the quality and integrity of research. It can also help authors avoid accusations of plagiarism or give their paper more context.

However, sometimes editors, reviewers and authors add irrelevant citations with the goal of increasing citations to certain researchers (“citation pushing”) or journals (“citation stacking”). This “citation manipulation” search results list from Retraction Watch shows numerous examples.

Wageningen University & Research and Elsevier have together developed innovative analytical methods for detecting and preventing this form of scientific misconduct. The collaboration plays an important role in taking a quality-over-quantity approach to research output evaluation. After Wageningen University’s call for action on this issue, their experts – Prof. Jan Willem van Groenigen (also Editor-in-Chief of Geoderma) and Prof. Coen Ritsema of the Department of Environmental Sciences and information specialists Dr. Theo Jetten and Dr. Ellen Fest – joined forces with Elsevier colleagues to investigate the scale of the issue among peer reviewers.

Prof. Jan Willem van Groenigen, Prof. Coen Ritsema, Dr. Ellen Fest and Dr Theo Jetten of Wageningen University are working with Elsevier colleagues to investigate citation manipulation by peer reviewers.

As Prof. Arthur Mol, Rector Magnificus of Wageningen University, explained:

Of course, scientific integrity in relation to publishing is bigger than citation stacking or pushing. Since the imperative ‘publish or perish’ has become such an important factor in our scientific community, it is important to safeguard all aspects of quality control of the publishing process. That is a joint responsibility for both the scientific and the publishing community. And that is why we called for action and there is still much work to do. However, this is a promising first step.

We are happy to work with Elsevier and any other interested party in making the publication process more fair. We look forward to forthcoming steps in combatting citation pushing and stacking, by Elsevier as well as other journal publishers.

Reviewer manipulation of citations is rare

According to the results of their analysis, just 0.8 percent of 69,000 reviewers were associated with suspicious citation patterns. Nonetheless, there is still cause for concern, said Dr. Philippe Terheggen, Managing Director of STM Journals at Elsevier:

Although rare, even one case of citation manipulation can have a ripple effect on the scientific community. Detection is an important step in making sure that journal citations can continue to be trusted. Stemming from our efforts to support research in every way, I am just happy that we could together answer Wageningen University & Research’s call for action on this form of misconduct, with our analytical capabilities. We plan to further expand the existing suite of tools we offer researchers to support research integrity and trusted information.

By analyzing 500,000 reviewers and their citations in Scopus, a distribution of normal citation patterns was determined. More than 270,000 reviewers were never cited in any of the papers they reviewed for Elsevier. In a subsequent more detailed screening of the 69,000 most prolific researchers that reviewed at least 5 publications with an Elsevier journal and had at least 5 publications, 0.8 percent of reviewers were associated with suspicious citation patterns based on ≥50 percent citations to their papers having been added to submissions that they reviewed.

It must be emphasized that these numbers do not always equal unethical behavior, in some cases there are good reasons for these added citations. For all reviewers with very suspicious citation activity, Elsevier is sharing the information with editors who have the expertise needed to assess the reviewer reports in detail. If editors find that reviewer citations are superfluous in several cases, the evidence is then shared with both the reviewer and their institute. This includes the number of citation additions suggested per article by a reviewer. Unless there is a very good explanation, they are no longer sent papers to review or welcome to sit on Elsevier journal editorial boards.

Co-author Catriona FennellJeroen Baas, Director of Product Management for Elsevier’s Research Products division, presented this first large-scale analysis of citation manipulation in journals at the 17th International Conference on Scientometrics & Informetrics (ISSI) in Rome. The paper, co-authored by Jeroen and Catriona Fennell, Director, Journal Services, is hosted as a pre-print here on SSRN.

Netherlands Code of Conduct for Research IntegrityElsevier and Wageningen are sharing the analysis and outcomes to raise awareness in the wider scientific community and support the needed ethical discussions around what can and should be done to prevent misconduct. In addition, thanks to a recommendation from Wageningen University, the Netherlands Code of Conduct for Research Integrity now recognizes citation pushing as a form of misconduct.

What’s next?

Now that Elsevier can detect citation manipulation in published papers, the next step is to prevent it earlier and before publication. Before reviewing submissions, reviewers of scientific research are now reminded that all citations that they request of authors must be genuinely relevant and manipulation is unacceptable.

To help prevent manipulation further, in 2018, Elsevier also added the text below to reviewer instruction emails in the editorial systems and to reviewer guidelines, in line with new COPE recommendations:

Please note any suggestion that the author includes citations to reviewers' (or their associates') work must be for genuine scientific reasons and not with the intention of increasing reviewers' citation counts or enhancing the visibility of reviewers' work (or that of their associates).

Quick question for you

Which terms do you most associate with Elsevier? (check all that apply)

Data and analytics
Research platforms
Technology
Decision support tools
Publishing
Books and journals
Scientific articles
Healthcare content

Tags


Contributors


Comments


comments powered by Disqus