Forget miracles – how can we gauge the real impact of science?

It’s a problem research funders everywhere grapple with: evaluating what science can do for society

Prof. Daniel Sarewitz referred to Sidney Harris's famous cartoon when talking about pitfalls in determining the public value of research. See the full image below. (© ScienceCartoonsPlus.com)
Prof. Daniel Sarewitz referred to Sidney Harris's famous cartoon when talking about pitfalls in determining the public value of research. See the full image below. (© ScienceCartoonsPlus.com)

It’s a familiar scenario to anyone who keeps up with the news: conflicting research findings spark heated public debate on topics from climate change to whether women in their 40s should get mammograms and men in their 50s should get PSA tests. Of course, experts on all sides are willing to step in and testify before lawmakers and write op-eds.

It’s just one symptom of which Dr. Daniel Sarewitz calls a “system in crisis,” referring to a “crisis of politicization and legitimacy” while questioning the quality of research and the system that supports it.

“Certainly one does not want to pour money into a system that’s leading to unreliable research, that’s increasingly unproductive and that turns science into a weapon for politics,” he said.

Daniel Sarewitz, PhD, Professor and Co-director of the Consortium for Science, Policy, and Outcomes at Arizona State University, gives gives a keynote presentation on research evaluation at the Research Funders Summit.

Dr. Sarewitz, Professor and Co-Director of the Consortium for Science, Policy & Outcomes at Arizona State University, made these comments as a keynote speaker at Elsevier’s Research Funders Summit, focusing on the challenges of demonstrating the impact of science – and how our understanding of impact is evolving.

He started by asking two questions that are key to researchers and research leaders as well as those who fund research: How can we know whether our investments in science are leading to the outcomes we hope for? And how can we measure that impact?

“The intellectual and practical efforts to understand, assess and improve impact are pretty much in their infancy,” he said. When gauging the value and social impact of science, he explained,  we need to go beyond simplistic categories like “basic versus applied” and familiar quantitative measures like number of publications and patents or monetary ROI.

Dr. Sarewitz considers this a matter of urgency: “I think the system is in deep trouble in a number of ways,” he said, speaking of crises in three areas:

  • He referred to a “crisis in quality” in some of the research itself, exacerbated by inappropriate incentives for scientists, and manifested in part by the difficulty of replicating scientific results – a challenge Elsevier has been tackling with various reproducibility intiatives.
  • He spoke of a “crisis in public value,” questioning whether the promises used to justify support for science could be met without significant institutional reform, and pointing toward the need for new approaches to assessment and accountability.
  • And he talked about the “crisis of politicization and legitimacy.” Conflicting research findings are leading to ongoing public debate about important topics, resulting in the weaponization of science for politics.

Because of the complexity of scientific research and the system that supports it, these issues need more sophisticated tools and policies, he said.

“Impact is achieved in a complex systems context,” he explained. “Yet science policy lacks theories, methods and tools for understanding and assessing impact in such a context. …

“How do you know if you’re asking good questions – whether you’re asking questions that will lead to the outcome you want to achieve? How do you know if you’re getting that right?”

Prof. Sarewitz said the complexity of scientific research demands different ways of understanding its impact. He outlined some possible new approaches, such as Public Value Mapping to understand the plausibility of research having its desired impact and Reconciling Supply and Demand (RSD) to make sure knowledge gained from research is useful and usable.

© <a href="http://sciencecartoonspplus.com/" target="_blank">ScienceCartoonsPlus.com</a>Public value mapping is a framework he developed with his colleague Prof. Barry Bozeman to assess the capacity of research programs to achieve social goals:

The key point about public value mapping research is that it takes the ‘then a miracle occurs’ part of science policy – that is, we put the money into science, we create the great science and that leads to social impact – and replaces that with very granular case studies that look at the types of questions we’ve been asking, not with an eye toward prediction but with an eye toward understanding plausibility and logic.

Here are some questions to ask as part of the Public Value Mapping process. (Source: Bozeman, 2002)

For RSD, Dr. Sarewitz applies the economics concept of “supply and demand” to the results of research. It’s framework that can be used to determine whether the knowledge gained from research is “useful and usable.”

This chart is from a 2007 paper Prof. Daniel Sarewitz co-authored in Elsevier’s journal <em>Environmental Science and Policy</em>. It visualizes a method to gauge whether research is relevant to those who need it.

Prof. Sarewitz summed up the need for such systems by saying:

Science isn’t simply scientists in the lab; It’s a complex production process that also includes processes of learning and engagement and … organizational and institutional dynamics. If you don’t have a way of understanding all of those in the context of a spectrum – from the value that scientists care about to the values users care about – it’s going to be really hard for you to say with any confidence that the knowledge you are working on as a researcher is actually going to lead to the impact … that society would like.

Quick question for you

Which terms do you most associate with Elsevier? (check all that apply)

Data and analytics
Research platforms
Technology
Decision support tools
Publishing
Books and journals
Scientific articles
Healthcare content

Tags


Contributors


Comments


comments powered by Disqus