Beyond citations: new ways to understand research impact

Universities have become highly practised at counting the outputs of research. Publications, citations, grants and patents all offer useful signals, but they say less about whether evidence reached the people who could use it.

Basil Mahfouz’s doctoral research at UCL, supported by Elsevier, offers one example of a different approach. His analysis compared research cited by governments with closely related work that had received no policy attention. Conventional bibliometric indicators had relatively little power to explain the difference. The authors involved, and their positions within research networks, were more influential.

This kind of work sits within metascience, a growing field that applies research methods to the research system itself. Sir Geoff Mulgan, Professor at UCL and Mahfouz’s doctoral supervisor, describes it as “the science of science”: a way to examine how knowledge is created, selected and put to use.

A research system that can be studied at scale

Scholarly literature is unusually well-structured. Papers have abstracts, authors, references and institutional affiliations. Collaboration patterns can be mapped, while citations show how ideas move between fields.

Mahfouz argues that advances in AI and data science now make it possible to analyse this information at a scale that was previously out of reach. The same methods could be applied across research generation, peer review, evaluation and use.

That opens up questions that standard indicators struggle to answer. Which areas of research are reaching public decision-makers? Where does relevant work remain concentrated inside disciplinary networks? Which researchers appear to have useful evidence but limited access to the channels through which policy finds it?

These questions are closer to diagnosis than measurement. They help explain how the research system behaves, including the points at which knowledge stops travelling.

Moving beyond a retrospective view

Most research evaluation looks backwards. Citation counts show what has already attracted attention. Policy citations reveal which papers have already entered government documents.

Mahfouz also used predictive analytics to look for emerging research with characteristics similar to work previously used in policy. The model could be applied across a larger body of literature to identify papers or topics with potential relevance before they had established a visible record of influence.

For universities, this could support a more detailed understanding of their own research portfolios. Mulgan notes that institutions often want their work to be useful but have limited methods for identifying where their output may be poorly aligned with the contexts in which it could be applied.

A university might use these methods to find strong work that needs better policy engagement, or to see where important public questions are receiving little attention across its research base. The analysis could also reveal how language, geography and professional networks shape the visibility of different researchers.

The limits belong inside the evaluation

Predictive tools learn from historical patterns. If previous policy use favoured particular institutions, countries or networks, a model may absorb those tendencies.

Mahfouz acknowledges this directly: using the past to identify future relevance provides a useful starting point, but it carries clear limitations.

This places more responsibility on the people interpreting the results. They need to ask what the data omits, which forms of influence are being rewarded and whether apparently neutral patterns reflect older inequalities.

Mulgan argues that this reflective layer will become more important as researchers and institutions work with increasingly powerful AI tools. His concern is that better access to information does not automatically produce better judgement; users still need to examine relevance, accuracy and what may be missing.

Used carefully, metascience could help institutions see how their research moves beyond publication and where that movement breaks down. That would give research leaders a firmer basis for deciding which work needs support, connection or closer examination.

Watch the full interview