Measuring what matters: library-led insight at a National Laboratory

Understanding research impact has become a top priority at research institutions in recent years. This is especially true for institutions with government mandates to explore solutions on behalf of national interests.

At the Pacific Northwest National Laboratory (PNNL), a U.S. Department of Energy (DOE) national laboratory focused on chemistry, earth science, and innovations that protect critical infrastructure, understanding research performance can inform major decisions about resource allocation and future scientific direction.

Redesigning how publication metrics are reported

Working in collaboration with her manager, Miriam Blake, Mary Frances led the redesign of PNNL’s publication metrics report. Instead of a dense PDF, the new version is a compact, two-slide visual summary that’s presented quarterly to the oversight board.

Mary Frances’ long-standing interest in data visualization techniques was a great asset in the redesign. “Visualizations help make connections without having to use so many words.”

The simplified format also stirs more discussion. By sharing the report in a presentation, Miriam can offer background on the data that directly addresses the questions in the room. “It’s easier to talk about two slides rather than this long document,” Mary Frances said. “When you’re at this presentation, you can quickly be up to speed.”

By helping stakeholders develop a critical understanding of publication data, the library is actively shaping institutional insight and improving how research performance is discussed.

Going deeper with topic-based analysis

A second, more experimental initiative piloted a new approach to analyzing impact in relation to PNNL’s scientific objectives. They wanted to explore how to gain valuable insight into specific research topics that were within the larger directorate areas of the laboratory but also crossed those boundaries. They selected one key research area, chemical catalysis, to develop the approach.

The project presented technical and semantic challenges. PNNL’s internal objectives in the topic area didn’t map neatly to existing bibliometric topic structures. “Our lab objectives are really detailed and even AI was having trouble identifying topics, so we changed the approach and went with a known factor.” Rather than searching predefined topics, the team searched authors from their laboratory they knew were publishing on chemical catalysis. Using SciVal, they developed topic clusters based on the work of these researchers, then expanded the dataset to incorporate global research output in those clusters across all institutions. “Then we could use that to benchmark against other laboratories working on those topics,” explains Mary Frances. It also allowed them to capture output from collaborations across directorates in their laboratory that otherwise would not have been included.

While still in a pilot phase, this deeper analysis highlights how the laboratory is contributing to this specialized research space within the boundaries of the field. Feedback from stakeholders is forthcoming, but for Mary Frances, the value lies in being able to test more granular methods to see if topic-level analysis reveals more actionable insights.

She’s also seeking ways to include research outputs that are frequently cited, but often invisible in citation databases. “Besides journal articles, we publish conference papers, books, data sets, and technical reports. Those show up in the Office of Scientific and Technical Information database (OSTI) but they don’t show up in other bibliographic databases, so getting citation metrics on those is very difficult” she explains.

Eventually Mary Frances hopes to expand the topic-based approach piloted in this analysis to inform other key laboratory objectives. “We know how the labs are doing as a whole, and to some extent different directorates, but regarding these specialized topic areas, can we better understand how they are moving the science forward?"

As PNNL seeks to understand impact better, the library is building internal capacity to support it. Mary Frances is training other librarians to work with bibliometric tools and contribute to the reporting process. “I’m helping our team of research librarians who have not worked as much in this space become familiar with the tools, so I’m not the only one doing it.”

She also offers practical advice for other libraries that want to strengthen their role in institutional reporting. “Start with your customer’sask, what’s important to them,” she said, and then broaden their horizons a bit – “introduce what they may not have realized they need. They may say, ‘yeah, we hadn’t thought of that.’”

At PNNL, the library’s commitment to starting with what matters to stakeholders and making it clear and useful is helping research value to be better understood across the organization. And it’s proving that librarians with the right tools and skills can become trusted collaborators and key players in research strategy.