Pressures in system cause researcher concerns over quality, per new peer review survey

An international survey by Sense About Science and Elsevier finds that 82% researchers want more information to accompany research papers

Peer review report main image

Download the reportResearchers have confidence in peer review, but strains on the system are causing them to worry about quality, according to a new survey of more than 3,000 researchers conducted by Sense About Science and Elsevier. The survey indicates that researcher satisfaction with peer review has increased over the past decade and is high overall, and that it is a valued and necessary part of the scientific process. However, there are concerns about the quality of science as the rate of article publication accelerates and pressures in the system push researchers to publish ever more articles.

The findings, drawn from respondents in a range of disciplines, career stages and locations, build on the conclusions of a 2009 survey also conducted by Sense About Science and Elsevier.

Tracey Brown, OBETracey Brown, Director of Sense about Science, explained:

What this survey shows is that our inroads into getting wider understanding of peer review are now not enough for research publishing in the 2020s or for people to understand what quality checks have been done. We urgently need to see a common language and greater transparency about what has been reviewed. Does it include data? What criteria were used? Is it even a published study? Remember, many research users are arriving at information via search engines, not academic portals.

Since 2009, there have been multiple technological advances in the ways research is shared and disseminated, as well as the ways peer review is conducted. For researchers and reviewers, these advances can speed up the process, ensure that research is reviewed by the right experts, and get articles from submission to publication faster. At the same time, these changes can bring challenges; it needs to be clear whether something reported by third party – for example, a preprint, a piece of code – has been peer reviewed. There are also opportunities for misrepresentation or exaggeration of findings, which has sparked discussions around research quality and trust in science.

Key findings

Here are some highlights from Trust and Peer Review: Current researchers’ perspectives 10 years on:

  • Respondents want more information available alongside research papers, including an indicator to show whether someone else had tried to reproduce the research (82%).

Researchers’ views on additional information sources required to assess research outputs. (Source: Figure 10, <em>Quality, trust & peer review</em>, Sense About Science and Elsevier, 2019)

  • Most researchers (76%) believe that data and supporting material that accompany research articles should be reviewed.

Researchers’ views on whether data and other supporting material should be reviewed (by specialty). (Source: Figure 9, <em>Quality, trust & peer review</em>)

  • Two thirds of participants feel that reviewers would benefit from clearer guidance on reviewing criteria, while many felt that recognition of reviewing work was important for maintenance of a healthy peer review system, most commonly from employers.

The steps researchers consider important for maintaining a healthy peer review system. (Source: Figure 9, <em>Quality, trust & peer review</em>)

  • Relatively few researchers (38%) agree that the public understand the concept of peer review. There is strong support (77%) for a clear indication that material has been reviewed, and most (70%) believe that explaining research context and the implications of research in lay terms would be most helpful.

  • Despite the trend towards using technology to evaluate evidence, few respondents (8%) think an evaluation that relied on artificial intelligence would qualify as peer review.

Overall, respondents to the 2019 survey were happy with the current evaluation system, and satisfaction has increased since the 2009 survey from 69 percent to 75 percent. And almost all respondents each time (91 percent in 2009 and 90 percent in 2019) indicated that they believed that peer review improves the quality of research.

Respondents also strongly agreed each time (85 percent in 2009 and 84 percent in 2019) that without peer review, there would be no control in scientific communication.

However, confidence in the reliability and trustworthiness of research outputs has yet to hit a similar level: 62 percent of respondents found most research trustworthy, but over a third (37 percent) admitted they had doubts over the quality of at least some research outputs they had encountered in the week prior to the survey – including those they thought had been peer reviewed.

Adrian MulliganAdrian Mulligan, Research Director for Customer Insights at Elsevier, explained:

Researchers provided a range of reasons as to why they might find some outputs untrustworthy, ranging from biases in peer review, doubt over sources and unsupported claims. Perhaps unsurprisingly, many respondents noted that questioning findings was a key part of their role; however, we also saw respondents commenting that pressures in the system were leading to poor quality research.

Sarah Pritchard, Dean of Libraries at Northwestern University. makes a similar observation in Elsevier’s Research Futures report: “The notion that a person needs to have lots and lots of ‘objects’ credited to them to get tenure is not going to change. It’s going to continue to be what counts when making the judgment call, ‘Is this person ready to get tenure?’”

That sentiment is reflected in comments from the survey respondents – for example, this comment by a researcher in elecrrical/electronic engineering in Italy, aged 36 to 55:

The university/research system forces researcher to publish a ridiculous amount of articles… researchers do not have time to properly test and validate the results. Trying to reproduce the results highlighted in the paper is often impossible, because there are missing details or errors.

The peer review report also identifies that the increasing pressure to publish has given rise to predatory journals, described by one interviewee for the report as “the dark side of open access.” These journals accept money for publication without checking articles for legitimacy and without providing the editorial and publishing services normally associated with publishers.

Adrian concluded:

This study highlights the growing pressure being placed on research communication and importantly the value of peer review. Maintaining the integrity of the peer review system is paramount, so it's important various stakeholders work together to ensure reviewers receive recognition and have clear guidance and quality tools to support them in their roles.

Quick question for you

Which terms do you most associate with Elsevier? (check all that apply)

Data and analytics
Research platforms
Technology
Decision support tools
Publishing
Books and journals
Scientific articles
Healthcare content

Tags


Contributors


Comments


comments powered by Disqus