Measuring the impact of science: a multi-stakeholder challenge

Researchers say their work has impact; a collaboration network and the right tools can help them measure it

Impact-of-Science-conference-group-photo.jpg

Science is not performed in a metaphorical vacuum; its results have impacts on society and the economy, and increasingly funders and institutions are looking to understand the broader impact of research. Despite this trend, a recent Elsevier survey revealed that while 66 percent of researchers track their scholarly influence, only 46 percent monitor the societal impact of their work.

The survey asked 3,511 researchers around the world questions about the impact of their research. What impact – if any – do they think it has? How do they measure its impact? And do they have the support they need to demonstrate the impact of their work?

The results of the survey were presented at the 2016 Impact of Science conference, held in Amsterdam in June. Now in its fourth year, the conference is an outcome of an ongoing collaboration between Elsevier and the Dutch nonprofit organization ScienceWorks, facilitating an important conversation among all relevant stakeholders around the measurement of the impact of research on society.

Prof. Louise Gunning, Chair of the Dutch National Research Agenda (Nationale Wetenschapsagenda) and former Dean and President of the University of Amsterdam, presents at the 2016 Impact of Science conference. (Photos for AESIS by Mo P&I Studio, the Netherlands)

The 2015 conference saw the establishment of the Network for Advancing & Evaluating the Societal Impact of Science (AESIS), which brings together university strategists, policy makers, research funding organizations and science assessment experts. The network was established by ScienceWorks on the basis of discussions with us over the past few years. John Walker, Elsevier’s Director of Strategy, Research Management, represents Elsevier as an expert on the board. Since 2012, we have partnered with ScienceWorks to convene a variety of stakeholders to address the issue of measuring the societal impact of research, beyond the impact it has on academia. Frank Zwetsloot, Managing Director of ScienceWorks, commented on the partnership:

It is clear that science has a huge impact on society, the economy and our culture, and it’s vital to measure this impact effectively. This is a big challenge, as meaningful evaluation activities need to speak to all relevant stakeholders. Like Elsevier, we saw that there was a real need for those stakeholders to come together. We both wanted to facilitate this dialogue, and that’s when the idea of the Impact of Science conference was born. We worked closely together to set up this new conference. By working with Elsevier, we not only had a reliable partner but also gained access to the relevant data and research intelligence expertise.

The conference is one key element of the partnership, facilitating these discussions. This year’s theme was “Governmental and institutional methods to advance the societal impact of science,” getting right to the fundamental question of how we measure impact.

David Sweeney, Director of Research, Education and Knowledge Exchange at the Higher Education FundingCouncil for England (HEFCE) in the center; Prof. Daniel Zajfman, President of the Weizmann Institute of Science, Israel, at right; and David Goldston, Director of Government Affairs for the Natural Resources Defense Council and former Chief of staff on the House Science, Space and Technology Committee, USA.

Funding bodies demand information on impact of research projects

Research grants are competitive, and today many funding bodies require applicants to provide information on the impact they expect their work to have on society. This information may include data on the impact of previous research, collected by the institution, publisher or individual researcher. Larger national programs also collect this information, such as the Research Excellence Framework (REF) in the UK and the Standard Evaluation Protocol (SEP) in the Netherlands, showing the performance of institutions throughout the country.

EU funding in particular requires researchers and institutions to provide this information in relation to the political and societal goals of Horizon 2020, the EU Framework Programme for Research and Innovation. The program includes the section Science with and for Society, which “allows all societal actors (researchers, citizens, policy makers, business, third sector organizations etc.) to work together during the whole research and innovation process in order to better align both the process and its outcomes with the values, needs and expectations of European society.”

Data on the impact of research plays an important role in supporting the work done by scientists and other stakeholders together, helping researchers and their institutions benefit from this and many other sources of funding. But as Dr. Rob Hamer, VP of Research & Development at Unilever, is quoted on the AESIS website: “Society should define WHAT is needed; science should be solely responsible for HOW the problem is solved.”

To make the information that provides the foundation for discussions and collaboration with stakeholders valuable, it must address the needs of those stakeholders – starting with the researchers themselves.

The researcher’s view of impact

John WalkerOur recent survey aimed to understand how researchers see the impact of their work and find out what can be done to improve its measurement. It was part of a larger survey examining the drivers and influences on the communication of scholarly research, run in May 2016. Walker presented the results of the survey at the Impact of Science conference, including researchers’ need for access to better impact tracking and evaluation tools. He said:

The survey was enlightening in many ways, and some of the results were unexpected. A significant number of researchers believe their research has an impact beyond academia, and many say it should. But less than half actually track the wider impact of their research. There is an opportunity here – by helping researchers with the right tools, we can enable them to evaluate and demonstrate the impact of their research, giving them access to funding and public engagement and ultimately boosting their impact on society.

Two-thirds of survey respondents track their scholarly influence, but only 46 percent monitor the societal impact of their work. Similarly, less than half of respondents (45 percent) agreed that science should always have an impact. Those who agreed mentioned the source of their funding and that research should benefit society. As a biological sciences researcher from Spain said: “We work mostly with public money, so society should know what we do and ideally it should impact society.”

Not all respondents felt their work had impact, but many would value support:

  • 1 percent said their research had no impact at all.
  • Impact reported included improved quality of life (31 percent), commercial application (25 percent) and appropriate government policy (17 percent)
  • 43 percent said they feel well supported by their institutions in communicating their societal impact, for example by communications departments
  • 36 percent feel they would benefit from improved systems for tracking impact
  • 33 percent said they would appreciate training

So we need to provide the right tools for researchers to track their impact. This in itself is a challenge, Walker explained:

There is something of a catch 22 in this area in the sense that there will be limited attention unless there are technologies to support the tracking and measurement of impact. But unless there is evidence of attention and commercial demand, there will be limited investment in these technologies. Nevertheless, at Elsevier we are investing and making progress in this area. For example, our SciVal tool can now analyze and benchmark a university’s level of academic-industry collaboration, as well citations from patents to academic articles. We will soon be adding metrics on the level of media attention paid to a university’s output, and we plan to add other impact-related metrics.


Sample tweets

Impact of Science conference

The 2016 Impact of Science conference is the fourth in the series. Hosted by AESIS, which is convened by ScienceWorks, the conference aims to:

  • Engage with key stakeholders in the research strategy, policy and funding arena.
  • Enhance relationships with existing customers and stakeholders.
  • Target new customers and new contacts with existing customers.

The conference was held in Amsterdam on June 9 and 10. It brought together science policymakers, science assessment experts, university strategists, scientometricians and university data providers to explore the topic “Governmental and institutional methods to advance the societal impact of science.”

To find out more and view the program, visit the conference website.

Research Intelligence at Elsevier

Elsevier’s work in this area has resulted in a number of solutions that researchers and institutions can use to evaluate their impact, including through citations, patents, collaboration, press coverage and article usage. In particular, Elsevier’s Research Intelligence solutions address the most pressing challenges researchers and research managers face.

These tools and services could provide the support so many researchers say they are lacking. This would empower them with the information they need to evaluate and demonstrate the impact their research has on society. And as we continue to partner with organizations like ScienceWorks and take part in networks such as AESIS, these solutions will continue to grow stronger and address the needs of the many stakeholders involved.


Photos for AESIS by Mo Barends, P&I Studio, the Netherlands.

Contributors


Related Stories


Comments


comments powered by Disqus