Uncovering clues to the societal impact of research

Elsevier and Wageningen University & Research studied the links between media coverage, policy and impact

Linda Veldhuizen screenshot video
Linda Veldhuizen, PhD, Sustainable Development Solutions Network Coordinator for SDG2 at Wageningen University & Research. Watch the video at the end of this story.

An estimated 821 million people worldwide were undernourished in 2017, representing 11 percent of the global population, according to the Food and Agriculture Organization of the United Nations (FAO). UN Sustainable Development Goal (SDG) 2 aims to end hunger, but despite years of decline in undernourishment, it’s on the rise again.

Research is one way to address this problem. About 6,500 people at Wageningen University & Research (WUR) in the Netherlands are working on a range of sustainability issues, including tackling hunger. From developing innovative agriculture technologies to working out ways to reduce food waste and increase security, they aim to contribute to achieving the UN’s goal of zero hunger by 2030.

Influencing society and policymakers is an important part of this. But how can researchers measure the contribution their work is making to achieve a goal like zero hunger? In other words, how can we measure the societal impact of science? And what can an institution do to increase its impact on society and policy?

Theo Jetten, PhDTo find out, Elsevier has been working with WUR since April 2017. Dr. Theo Jetten, Executive Secretary of the Graduate School for Production Ecology and Resource Conservation at WUR, described their collaboration project:

What we have done … is to visualize an aspect of relevance to society, and that's called outreach. So how does your work end up in society? For instance, by means of Twitter, which is a proxy of outreach, or news mentions, which is a proxy of outreach? We expect that this will become more and more important in the assessment of scientific research.

WUR’s research output is disseminated to scientists, stakeholders and the wider public, through partnerships, projects, publishing in academic journals and through the media, social media and policy – a path often seen as reflecting societal impact. WUR colleagues were curious to know whether this societal impact was in line with the academic impact of their research output – and whether the researchers and their work are having the impact they desire.

Wageningen partnered with Elsevier to develop a project that would reveal their societal impact and its link with research output. Dr. Mark Siebert, Director of Engagement Programs and Strategy at Elsevier and one of the project leaders, explained:

The topic of societal impact with focus on media resulted as a practical area of mutual interest, and it seemed to be a good path to take given its strategic importance in light of the Sustainable Development Goals. On that basis, the team explored specific research questions.

Media and social media: markers of impact?

In collaboration with Elsevier, WUR studied its societal impact by monitoring mentions of its research output in the mainstream news media, social media and policy documents. The aim of the project was to investigate the key drivers influencing research use by policymakers and the media. WUR wanted to understand the societal impact of its research, and Elsevier wanted to explore the use of its tools in doing this, and to identify potential areas for improvement. The findings were presented at the SDG conference Towards Zero Hunger: Partnerships for Impact, held at WUR in August. They are already helping Elsevier make its news monitoring tools more helpful for analyzing societal impact, Dr. Siebert said.

Wageningen University & Research is among the top institutions in its field, with a mission to “explore the potential of nature to improve the quality of life.” The institution focuses on three main areas: Food and food production; the living environment; and health, lifestyle and livelihood. Wageningen has produced more research output in terms of peer-reviewed publications related to SDG2 in the past five years than its peers, but this research output seemed to be getting less media attention than other institutions – particularly those in native English-speaking countries.

Wageningen is out-performing its peers in terms of scientific quality but lagging behind on media coverage. (Publications in Scopus 2014-18 analyzed in SciVal; Social media from Plum Analytics 2014-18, news data from NewsFlo 2014-18)

Peter Darroch, PhD Dr. Peter Darroch, Senior Manager of Engagement Strategy and Planning at Elsevier, was the project’s data wrangler for Scopus, SciVal and Plum Analytics, and expert consultant for the interpretation and use of the data. He explained how answering questions could lead to improvements in the tools:

Is social media attention good? Should they be targeting local, national or international media? What does it mean for getting invited to policy conversations? Our products could link these things together and both sides would benefit: Elsevier’s products could deliver information, data and insights, and we could use the findings to provide insights for our product development.

The team developed a three-part project – using media, social media and policy mentions as a proxy for societal impact – to answer the following questions:

  1. How does Wageningen perform compared to its peers regarding SDG2?
  2. Are media and social media mentions related to author, article or journal metrics?
  3. Do scientific quality and/or media or social media attention increase the likelihood of being picked up by policy documents?

To get an overview of Wageningen’s media and social media coverage, the team looked at publications and key researchers related to SDG2. They analyzed the media attention in context and compared it with that of other institutions.

By analyzing NewsFlo data on media mentions against SciVal data on publications and Scopus data on researchers, they found there was no significant relationship between the volume or quality of the scientific output and news media mentions in Wageningen’s case. However, they did find that in general, the more the output and the higher the quality, the more likely the research is to be picked up by the media.

Media coverage does not reflect quality

Using data from Plum Analytics, they found that social media mentions are not related to news media mentions for WUR researchers and competitors, although being active on Twitter may have a positive (but not significant) effect on the relationship between Twitter mentions and media mentions. Since there is no clear relationship between media mentions and an article, the author and the journal’s metrics, there is still a challenge for institutions to get their best research into the news media and social media.

The media and social media investigations were simple using NewsFlo, Plum and SciVal, but the process highlighted some overarching challenges with analyzing this kind of information and using it as a proxy for impact.

For example, the team focused on the official name of the institution – Wageningen University & Research – but quickly discovered that journalists weren’t using the full name so a lot of data was missing. They also searched for the names of key researchers, which weren’t always included in articles. The same goes for Plum Analytics – they would search using DOIs, but they weren’t always mentioned on social media.

In addition to the challenge of collating data from Scival, Newsflo and Plum, another challenge arose: language. The tools are global so they capture English language media – but it’s virtually impossible to cover all languages, so coverage in local languages like Dutch may be missed. Wageningen is a significant player in the field of sustainability science, and scientific publications demonstrate that, but they are appearing in the media less frequently than their peers. Could that be because they’re communicating in Dutch, or getting the attention of local and national Dutch media rather than international English media?

Does impact come from press or policy?

The project also aimed to understand the relationship between the quality and publicity of research and the likelihood of it being used in a policy document. Dr. Darroch said:

Policymakers have no time to read articles, but institutions want to influence policy with their research, so this is the kind of thing to investigate: Can you use social media or mainstream media to get your research into policy documents? Or is getting people into networks involved in choosing consultation panels to make these documents more effective?

The team analyzed a sample document from the FAO to uncover the links. Two of their main findings provided actionable information for Wageningen:

  • Review articles were more likely to be included in a policy document.
  • Studies cited in policy documents get more media attention than similar studies.

Elaine van Ommen Kloeke, PhDDr. Elaine van Ommen Kloeke, project lead and Senior Product Manager of Elsevier’s media monitoring tool NewsFlo, commented:

An interesting finding to add, with caution, is that there is still no clear indication of how research is being picked up by policymakers. There is a signal that research mentioned in policy documents receives more media attention, but in the case of the policy document we investigated, we don’t know whether the research got media attention because of the policy or was used in the policy because it had had media attention.

Linda Veldhuizen, PhDAnother explanation could be that these were the better articles and got more media attention because of that, noted Dr. Linda Veldhuizen, Sustainable Development Solutions Network Coordinator for SDG2 at Wageningen. Although a definite link between the two factors was not established in the research, she believes the link may exist in some instances. The findings will help inform Wageningen’s outreach strategy, enabling them to maximize the impact of their research connected to SDG2. Dr. Veldhuizen said:

I would recommend for researchers to get to know their relevant policymakers. When you have findings, think about the people you should reach with those findings, and then reach out to your communications department or other people to help you devise a strategy of how to reach these partners. There are people who can help you with that to make sure that your research is being used.

Using the results to improve the tools

The team looked at the results to see if any changes could be made to improve the support Elsevier’s tools provide in analyzing societal impact. For NewsFlo, looking into how journalists pick up news –and the variations on the institutions’ names they use when mentioning institutions like Wageningen – could be helpful, Dr. van Ommen Kloeke said:

We were using SciVal together with Scopus, NewsFlo and Plum, and even with all of those together, we still noticed we can't actually capture everything. For instance, we couldn't capture all the media attention around the references within a policy document very easily. That's why we were limited to one policy document. We’re now already adapting our technology to capture more relevant news for Wageningen so they can again showcase what their societal impact is through media attention.

Scopus, SciVal and Plum will benefit from the information too, Dr. Darroch said:

Societal and policy impact is very relevant for every field in every country. More visibility means a better institution; the more the public sees you, the better your reputation, researchers and students. What we have got through this project is an insight into what would be good for us to start trying to track.

We’re already in discussions about how to take our tools to the next level. The findings of this collaboration with WUR have confirmed use cases we knew about from other interviews – we’ve confirmed that our data and tools can provide significant insights, and we’ve confirmed the potential future direction for product development. This kind of information is important as it means we can demonstrate through concrete use cases how our tools and services benefit universities. Now we’re developing the information into roadmaps, bringing together use cases from around the world.

Watch the video



Written by

Lucy Goodchild van Hilten

Written by

Lucy Goodchild van Hilten

After a few accidents, Lucy Goodchild van Hilten discovered that she’s a much better writer than a scientist. Following an MSc in the History of Science, Medicine and Technology at Imperial College London, she became Assistant Editor of Microbiology Today. A stint in the press office at Imperial saw her stories on the front pages, and she moved to Amsterdam to work at Elsevier as Senior Marketing Communications Manager for Life Sciences. She’s now a freelance writer at Tell Lucy. Tweet her @LucyGoodchild.


comments powered by Disqus