So does the public finally ‘get’ peer review?
How a 10-year collaboration is raising awareness of the role of evidence in policy, news and society
By Julia Wilson Posted on 29 September 2015
This is Peer Review Week, a virtual celebration of the important role peer review plays in maintaining quality in science. Sense About Science and Elsevier have been working together for almost 10 years to convey the value of peer review to the public, policymakers and journalists. We encourage people to ask “is it peer reviewed?” when assessing claims about science – on issues ranging from the dangers of mobile phone radiation to the health benefits of detox diets.
The early days
Peer Review Week
Peer Review Week 2015 runs from Monday to Friday (September 28 to October 2) and features blog posts and interviews by participating organizations, a social media campaign, webinars and more. Peer Review Week grew out of informal conversations between ORCID, ScienceOpen and Wiley. Sense About Science joined to ensure that the wider benefits of peer review – such as a quality mark and tool for making sense of science claims – are shared with the public.
To participate on Twitter, use the hashtag #peerrevwk15
We’ve come a long way in 10 years. When Sense About Science was set up in 2002 as a charity to help people make sense of science and evidence, the practices of scientific publishing seemed a strangely kept secret. It was a time when the front pages of many newspapers raged with headlines and horror stories about genetic modification, the MMR vaccine and nuclear waste disposal. Often, little attention was being paid to the quality or status of research findings, and in particular to whether they had been peer reviewed.
Meanwhile, when asked about peer review, rather than considering the public interest, scientists’ minds went either to the paper they had in the system or to the papers sitting threateningly on their desks awaiting review.
The early work of Sense About Science was to open up that process. We convened a working group of scientists in 2004 to look at public questions about science. They concluded in a report that greater effort was needed by research institutions, journals, publishers and others to share the workings of peer review in scholarly publishing. In response to their recommendations, Sense About Science produced a public guide to peer review – I don’t know what to believe – published in 2005 with help from Elsevier, Wiley, the Medical Research Council, the Royal Society of Biology, and the Royal Pharmaceutical Society. It explains the peer review process to the public as a tool for helping people make sense of science claims.
Many researchers and publishers were skeptical. For a start, they didn’t think the public would find the dry academic process of peer review interesting. And they also worried that promoting the value of peer review would promote false certainties. What about times when the system doesn’t work and cases of fraud, bias and misconduct get through? What if the public thinks that peer reviewed means true?
We argued that people can accept that mistakes exist while finding the general rule of thumb – that peer reviewed means scrutinized – useful. They can understand that one paper is not the end of the story and that conflicting views coexist and compete. They see this in the criminal justice system all the time, after all. Like peer review, the criminal justice system is not flawless; bias, delays and mistakes happen, and sometimes the wrong decisions are made. Most people understand those flaws and still see the value in the system.
Reaction to the peer review guide
In publishing the guide, I am afraid we took the skeptical comments too much on board. We printed just 10,000 copies. We were surprised then that the guide immediately generated public discussion, with coverage in national radio and newspapers as well as specialist publications. Peer review was newsworthy. Within a few months, 58,000 copies had been requested, and a decade later, over a quarter million have been sent.
The requests revealed a broad swathe of society involved in processing and mediating scientific information – doctors, midwives, librarians, public health officials, MPs, safety bodies, teachers, helpline operators, horticultural societies, parenting groups, medical research charities and patient advocacy groups. Many of these are the people who, when the newspapers move on from a passing scare story, are still answering calls weeks later from anxious people wondering whether they should be worried. Contrary to the fears of some researchers, the public was quite able to understand that peer review is an indicator of scrutiny rather than the final word.
“A crisis in peer review”
Having increased public awareness and understanding of peer review, a few years later we became concerned about growing talk of “a crisis in peer review.” We were hearing concerns about the global expansion of scholarly research and a shortage or reviewers, hostile reaction in the press to the University of East Anglia email exchange about trying to stop publication of some climate research, and increased discussion about the mounting pressures on researchers to get grants and publish papers, leaving little time to review papers. Was the global peer review system about to collapse under this weight?
To get a better picture of the situation, in 2009, with Elsevier, we conducted a survey of authors and reviewers about these issues. There were questions on the role of the peer review system, how well it was understood, the time spent reviewing papers and the motivations for reviewing. The Peer Review Survey 2009 was one of the largest ever global surveys of authors and reviewers, and the findings are published in the journal Serials.
The biggest surprise was that overall satisfaction with peer review was very high. Only 9 percent of authors and reviewers said they were dissatisfied with the system. Motivations to review were altruistic. Reasons such as “playing my part as a member of the academic community” and being able to improve a paper ranked much higher (90 percent and 85 percent) than gaining personal recognition or enhancing one’s career (34 percent and 46 percent).
Ylann Schemm, Head of Corporate Responsibility at Elsevier who has worked closely with us over the years on our peer review outreach, said, “What I remember keenly was the impressive number of responses to the study, with over 4,000 researchers globally. It told us just how engaged the scientific community is with peer review and how to continuously evolve it.”
While improvements to peer review were strongly supported, talk of crisis was clearly much exaggerated. However, the survey did draw our attention to a certain lack of understanding of peer review amongst researchers and the need to provide better training and outreach. In response, we have been running peer review workshops for early career researchers where editors share insights about how they manage the process and experienced reviewers give tips and advice for reviewing papers. Several early career researchers from our Voice of Young Science network decided they wanted to share the insights from the workshops more widely. They interviewed editors and researchers and shared their tips in a guide for their peers Peer review: the nuts and bolts, published in 2012. The guide has been hugely popular in the UK and with researchers around the world including the US, China, Romania, Turkey and France. There is clearly a demand for this kind of support.
Three years later, peer review training is much higher on the professional development agenda, and the guide has just been incorporated into an online course for researchers on the Elsevier Publishing Campus.
Here we are today, looking at a much improved situation. Our peer review work is now backed by library services, publishers and editors. It is enshrined in BBC editorial policy, in civil service training and in high school education. Millions of people have read the Chinese translation of the guide, and the US version has been popular across the board from media to policy to patient groups. Information about whether findings have been peer reviewed is routinely sought by journalists, and details of the scientific publication are regularly included in institutions’ press releases and in news reports. Public debates about peer review are now a common feature at global interdisciplinary science conferences, such as the American Association for the Advancement of Science (AAAS) or the biannual EuroScience Open Forum.
In recent years, we have also been bringing a wider public audience into discussions about the quality of research findings through Ask for Evidence – our public campaign to encourage people to ask advertisers, policymakers, companies and journalists for evidence for their claims. It brings people into a discussion about evidence on the issues that matter to them – whether it’s claims about pesticides and bees or policies for improved road safety. Thousands of people have used our website, which makes asking for evidence — and getting help understanding that evidence, from statistics to peer review — as easy as possible for the public. Understanding the status of research findings gives people the confidence to hold public figures and companies accountable. We’ve seen articles corrected, posters taken down, and companies change their policies in response to people asking for evidence.
We have brought this campaign and others to the US with the newly formed Sense About Science USA, and we’re working together with Elsevier to improve public understanding of science.
Awareness of peer review has improved greatly over the past decade. I think we’re at an exciting point where the public is more informed and empowered than ever to take part in discussions about science and evidence and assess claims that matter to them, and the scholarly publishing world is responsive to researchers’ needs. We are seeing exciting new models and platforms for improving reviewing and supporting early career researchers. We have come a long way from the days when researchers didn’t see the point in talking about peer review.
What’s new in peer review?
By Bahar Mehmani, PhD (@mehmanibahar), Publishing Innovation Manager, Elsevier
Reviewers are important to us. They are experts in their research area who spend hours on a manuscript they are invited to review on a voluntary basis to ensure that it is sound and worthy of being published. Their work is essential to the advancement of science, medicine and technology – but it’s also anonymous. In most cases, they are unknown to authors and their community, and their work remains unrecognized. Elsevier’s Reviewer Recognition Program aims to engage reviewers and reward them for the work they do. The program features several projects and experiments:
The Reviewer Recognition Platform
The Reviewer Recognition Platform offers reviewers a personalized profile page, documenting their reviewing history. If a reviewer has completed at least one review in two years, they become a “Recognized Reviewer.” ‘Outstanding Reviewer” status is awarded to those in the top 10th percentile in terms of the number of completed reviews for a specific journal in two years. Editors can also hand pick reviewers and award them with a “Certificate of Excellence” and other perks. Reviewers can download review certificates, end of year review reports and electronic badges via the platform, create their public profile page and claim their reviews for non-Elsevier journals. Reviewers can also volunteer to review for their favorite Elsevier journals.
The Platform offers discounts for several Elsevier services, including Elsevier’s WebShop, which offers professional English language editing, translation and illustration services for researchers preparing their articles, and the Elsevier Book Store.
Peer review reports as articles
The Publishing Peer Review Reports pilot publicly recognizes reviewers’ intellectual contribution to accepted articles through the official publication of their reports. Review reports are attributed a separate DOI and are published next to the accepted paper on ScienceDirect. Participating journals include:
- Agriculture and Forest Meteorology
- Annals of Medicine and Surgery
- Engineering Fracture Mechanics
- Journal of Hydrology: Regional Studies
- International Journal of Surgery
Cross-Reviewing (CR) is a new innovation that allows reviewers to see each other’s reports once all reports have been submitted. It provides reviewers with a short window of time in which to discuss the reports. During this time, they are given the opportunity to provide additional information or make further recommendations to the editor based on this discussion, before they make a final decision.
Elsevier Connect Contributor
As Development Director at Sense About Science, Julia Wilson is responsible for fundraising activities and planning as well as the international program of work. Julia launched the Ask for Evidence campaign in Boston in 2013 and oversaw the launch of the US version of the public guide to peer review. She previously coordinated the Voice of Young Science program, where she headed a campaign that pressured the World Health Organization to respond to the promotion of homeopathy for serious diseases in Africa. She has led many Standing up for Science media workshops in the UK and internationally, including South Africa.
Julia has a degree in Biology from the University of Manchester and joined the Sense About Science team in 2009.
Sense About Science
Sense About Science works with thousands of scientists, scientific bodies, research publishers, policymakers, the public, community groups and media, to create an environment that supports clear and open public discussion about research and its findings. They aim to ensure that the public’s questions about evidence are heard and answered. Through award-winning public campaigns, they share the importance of evidence and respond when it is misrepresented.
By Bahar Mehmani, PhD, Joris van Rossum, PhD, and Ursula van Dijk | Posted on 03 Sep 2015
Reviewers can now list their entire review history on Elsevier’s Reviewer Recognition platformBy Hannah Foreman | Posted on 28 Aug 2015
Quick Guides on the Elsevier Publishing Campus are available in five languagesBy Paige Shaklee, PhD, and Helena Cousijn, PhD | Posted on 06 Jul 2015
Two journals at Elsevier take a stab at giving data a seal of approvalBy Chris Peters, PhD | Posted on 03 Jul 2014
Authors urge people to "ask for evidence" when confronting questionable policies “in the interests of safety”By Julia WIlson | Posted on 10 May 2013
Adaptation of ‘I Don’t Know What to Believe’ based on input from research community in ChinaBy Julia Wilson | Posted on 08 Feb 2013
New US version aims to help people make sense of science debates in the newsBy Joseph Alpert, MD | Posted on 05 Dec 2012
A seasoned journal editor and professor gives advice to peer reviewers