Advancing responsible research assessment: NICHD’s story

How a leading biomedical research funder turned to the world of evaluation science for inspiration

By Linda Willems and Christopher Belter - May 17, 2021  6 mins
Responsible Metrics Article

Disclaimer: Chris Belters' participation in this article and webinar does not constitute an endorsement of any vendor, product, or service. All opinions are his own and do not necessarily reflect those of NICHD, the NIH, or the US Government.


The idea that the assessment of researchers and their outcomes should be “responsible” is nothing new. But factors such as the increasingly competitive environment that researchers operate in, and the growth of open science and accountability, have given the topic a new resonance.

In this article, Christopher Belter, Program Analyst for US biomedical research funder Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), explains how his agency is responding to these shifts.

In the second article of this two-part series, we will look at the steps that Elsevier is taking in this area, not only within its business and solutions, but via the work of its International Center for the Study of Research (ICSR).


Chris BelterSince Chris Belter joined NICHD four months ago, the idea of responsible research assessment has kept him busy. “I’ve been thinking about it almost non-stop,” he admits.

The NICHD falls under the wider remit of the US National Institutes of Health (NIH). As Program Analyst for NICHD’s Division of Extramural Research, Chris’ role involves taking the lead in a range of activities, from designing and conducting analyses, to providing guidance to colleagues.

He adds: “Wrapped up in all of that is a larger, less tangible function and that’s to ensure that analyses are used responsibly. It’s my job to provide context for how the analyses should be approached, used, and incorporated into the decision-making process.”

Chris devoted his first weeks in the role to thinking about how NICHD can improve assessment. The result? A new conceptual research assessment framework is now out for review with NICHD’s program officers.

And, as Chris notes, he didn’t have to look far for inspiration: “I realized that to get closer to this idea of responsible metrics, we needed to borrow from evaluation science. The evaluation framework that will help us map what we do and how we measure it.”

Resources versus results

Chris knew that drawing up a framework would involve striking the right balance between “responsibility” and “feasibility”.

He explains: “Irresponsible use of data, metrics, and indicators puts an enormous pressure on researchers to do the right kinds of research, produce research outputs, publish them in the right places, and do all kinds of things that aren’t really appropriate for the healthy conduct of science. So, it was always clear that whatever assessment method we developed at NICHD, we would have to ensure it didn’t make the situation worse.”

Accuracy has been another big focus. “It’s not only about making sure that data sources and indicators are as accurate as possible, but that we are using the right indicators for the things we are trying to measure. We have a very broad field of research, so we can’t take a one-size-fits-all approach.”

Chris also had to consider the feasibility of any assessment measures proposed: “We currently have 3,700 active projects associated with 400 different research funding opportunity announcements that produce close to 10,000 publications every year. Many of our staff focus on administering our grants. Our assessment and accountability team is small. And, as a funding agency, we don’t have the kind of access to researchers that folks at the university- or laboratory-level enjoy.”

NICHDTo develop his framework, Chris followed a five-step approach. “One of the core principles of assessment is to evaluate a program based on the good that it is trying to do in the world. So, we needed to work out what we ultimately wanted to achieve and then work our way backward.”

STEP 1:  Establish the long-term outcomes

NICHD has two – develop new research and improve health.

STEP 2: Set intermediate-term outcomes

Chris and the team brainstormed a series of outcomes to help them reach those long-term outcomes

STEP 3: Develop short-term outcomes

Chris notes: “We wanted to capture the kind of levers we can start pressing now.” He adds: “What’s nice about the short-term outcomes we’ve chosen is that they are relevant to both our long-term outcomes.”

STEP 4: Map these against your organization’s activities

For Chris, this involved dividing NICHD’s various programs into key categories, which the framework will help them assess the impact of.

STEP 5: Design relevant indicators

This step is still under development. Chris explains: “Now we have our draft framework, we need indicators for each individual goal and indicators that track our progress from one goal to the next. Then we will have a framework that not only captures what we need to measure but puts those measures in context and ensures we are interpreting them in the best way we can.”

He adds: “Research is a complicated process, and while I don’t think we can ever really prove direct causality between our framework fields, I don’t think that should stop us from trying to make that link. That’s what prompts us to ask the right kinds of questions.”

How the framework works in practice

To illustrate how the framework can support NICHD’s various programs and activities, Chris has developed examples for two classic NICHD funding initiatives - Impact for a clinical research project and Impact for a sociological research center, both illustrated below.

For Chris, developing the draft framework is only the start of the process. “Now we need to finish it, actually use it, and make sure that we reassess it on an ongoing basis.”

He adds: “I think it’s going to help us be a bit more responsible in what we are trying to do. Ultimately, it’s about making sure it can measure the things that we care about – not just the things that we can actually measure.”


Chris shared NICHD's plans for a research evaluation framework during the Library Connect webinar Advancing responsible research assessment: implications for librarians and their users, held in May 2021 (timestamp 03:50 - 27:05). View the webinar.

About NICHD

The mission of NICHD is to lead research and training to understand human development, improve reproductive health, enhance the lives of children and adolescents, and optimize abilities for all. According to Chris “In practice, that means we fund a lot of different types of research, from physical and mental health to fertility and socio-economics – all kinds of things that impact the healthy development of kids into adults.”

NICHD

Contributors


Linda Willems
Written by

Linda Willems

Written by

Linda Willems

After starting her working life as a newspaper journalist (covering everything from amateur dramatics to murder trials), Linda Willems held a variety of communications roles before joining Elsevier. During her six years with the company, she focused on researcher communications and edited several of Elsevier’s researcher-focused publications. She's now a freelance writer and owner of Blue Lime Communications.

Christopher Belter
Written by

Christopher Belter

Written by

Christopher Belter

Christopher Belter
Lead Analyst, Division of Extramural Research
Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD)

Chris Belter is the lead analyst for the Division of Extramural Research at the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), a division of the US National Institutes of Health. At NICHD he designs and conducts analyses to inform the Institute’s funding decisions, provides strategic guidance on the use of data and analyses in those decisions, and coordinates analytical activities across the Division. Prior to joining NICHD, created and led bibliometric service programs at the NIH Library and the NOAA Central Library. Chris has a master’s degree in Library Science from the University of Maryland and a BA in Religion from Shenandoah University.

Comments


comments powered by Disqus