E-mail Updates Sign Up for E-mail Updates

The dawn of predictive analytics to measure research performance: SciVal’s Topic Prominence

How drilling down into the granularity of 97,000 topics can help institutions and researchers plan, secure funding – and demonstrate their impact

SciVal-topics-wheel-excerpt.jpg
The "Wheel of Science" provides a clear overview of the Prominent Topics your institution is active in. (Source: SciVal's Topic Prominence in Science; Scopus data)

Editor’s note: This month we are exploring the theme “using technology to extract knowledge from data.” Here, a Senior Product Manager for SciVal writes about a new functionality in SciVal designed to help researchers see into the future.


Until now, research performance analytics tools have focused mainly on the past, showing what has been done and comparing that to the performance of peers. Increasingly, however, researchers and research managers at institutions and governments need to know where to focus their attention for the future. Those who do this are better positioned to secure more funding, find the right collaborators and perform the most competitive analyses.

So at Elsevier, we looked to turn traditional research analysis around to face the future and build on the solid citation connections that are the hallmark of international and multidisciplinary science. Now, our research performance benchmarking platform SciVal is delivering a game changer in research planning: Topic Prominence in Science.

Dr. Richard Klavans, chairman of SciTech Strategies, the company that Elsevier partnered with to develop the new capability, explained why looking forward is so important:

Researchers have limited time and resources; they want to spend their time on the science rather than reworking unsuccessful grant applications. Knowing where to focus for the best chances of success helps make the process more efficient. Knowing which topics have the highest momentum can help researchers decide what to work on next, and with whom. And they can showcase their work in those high-momentum areas, highlighting their leadership.

How it works

Topic Prominence in Science divides the full spectrum of published research – 95 percent of the Scopus corpus of data – into 97,000 distinct topics based on citation paths. These topics are ranked using a new indicator called “Prominence,” which is calculated by weighing together the current impact of papers clustered in a topic (citation count), how much attention these papers receive (Scopus views count) and the prestige of the journals in which they are published (CiteScore). The prominence rank is indicative of the current momentum of a Topic.

Where the predictive power of Prominence comes in is its correlation with future funding.

To prove this, SciTech Strategies assigned 314,000 grants worth $203 billion from the STAR METRICS database, a large project-level funding database that accounts for 24 percent of US federal funding, to all 97,000 Topics through textual similarity. The grant data were split into two time periods for each Topic and the correlation analyzed. The model showed that Prominence accounted for 37 percent of the variance of future funding.

“If we split the grants into two 3-year periods, we find that Prominence combined with early funding explains 71 percent of the natural variability in funding in the final three years, which is extremely good,” Dr. Klavans explained. “From the perspective of SciVal, this adds a layer of usability that is a gold mine for researchers and research managers: looking at what has high Prominence can help them discover what is likely to get funded.”

More about the Prominence indicator

Prominence is a new indicator that shows the current momentum of a Topic. It is calculated by weighing 3 metrics for papers clustered in a Topic:

  • Citation Count in year n to papers published in n and n-1
  • Scopus Views Count in year n to papers published in n and n-1
  • Average CiteScore for year n

Read more about the formula behind the indicator Prominence and its correlating with future funding: “Research Portfolio Analysis and Topic Prominence” by Richard Klavans and Kevin W.

A future-focused planning capability

Topic Prominence responds to the evolving information needs of customers, Dr. Klavans explained:

Every researcher needs to justify that what they’ve done has impact. But they also need to look at where they’re going in the future.

For example, if you are an academic, there might be five Topics related to the work you do. The Topic where you publish the most is very familiar to you – you know the leaders, their biases and where they apply for funding. That’s your home community; although it might not get much funding, you care about it, and it’s the source of your ideas. But look down the list and you might come across a different, related Topic that is rising in Prominence and therefore likely to be a way to generate funding for your work. Intelligence gathering is an important activity, and scientists are doing it all the time. As Dr. Klavans said, “the problem isn’t knowing your own area, it’s knowing your neighbor’s neighbor.”

Research managers will look at different things. How do you justify assigning funding to one research group over another? What evidence can you gather and use to advocate for the establishment of a new department? Topic Prominence becomes a source of evidence and a talking point, both internally and externally. There’s also a showcasing role, as Dr. Klavans explained:

If you want to demonstrate you’re at the forefront, leading the way in a field, Prominence offers an alternative way of doing that, adding to the collection of indicators already available that demonstrate research impact. It’s a good way to decide which areas of research to put in the spotlight.

A sample of an institution's dashboard with the "Wheel of Science," which gives an overview of the Prominent Topics the institution's researchers are active in. (Source: SciVal's Topic Prominence in Science; Scopus data)

This powerful predictive analysis is already providing valuable insights for institutions around the world. Simoné “Sisi” Nutt, User Experience Lead for SciVal, has tested Topic Prominence with SciVal users, and the feedback has been overwhelmingly positive:

We haven’t seen anything like this before in terms of meeting users’ needs. Researchers relate to the Topics, they strongly resonate. The Topics really reflect the research areas they are working in, who they’re working with, the topics they’re covering and what their peers are doing. Researchers are excited by the possibilities Prominence gives them, and they are already getting a better understanding of what they can do to make their grant applications more efficient and their work more impactful.

Also, having the ability to cut across the traditional journal subject areas, and zoom into the granularity of analysis the citation clustering brings, more powerfully reflects what is actually happening in science, who is leading and who is emerging, what competitors are doing – and on which Topic exactly.

Revealing the technology

At SciTech Strategies, Dr. Klavans works with company president Dr. Kevin Boyack to develop cutting-edge technologies to understand and model the structure and dynamics of science and technology at a highly detailed level. Their goal has been to analyze topics across, rather than within, a range of subjects and industries to help research institutions see which topics have momentum today and predict which will likely have momentum tomorrow. This approach aligns with Elsevier’s role as an information analytics business. As Dr. Boyack explained:

To analyze research strengths, opportunities and risks, old disciplines are deadly. They’re used for establishing institutions, for control purposes, not for the sort of investigation a scientist does. We wanted to create something that resonates with researchers on a personal level, and it happened to be part of what Elsevier has rededicated itself to – helping scientists do their work, quickly and with joy.

In the table view, you can see all the Topics an Institution is active in, which serves as a starting point for further analysis. You can sort by the Institution’s Scholarly output or by the Global Prominence of the Topic. (Source: SciVal’s Topic Prominence in Science; Scopus data)

SciTech Strategies needed to design a new way of breaking down research areas into topics. Building an accurate, usable network of topics that researchers can feel connected to and identify with requires analysis – not just of papers but also of collaborations over time.

They clustered all Scopus publications from 1996 onwards – about 35 million – into 97,000 different “Topics” using citation paths – the first time this level of analytics has been done across the entire Scopus corpus. This provides a much better reflection of reality than citations alone, showing who is working with whom on papers, for example. They defined a Topic as a collection of documents with a common intellectual interest. Topics can be large or small, old or new, growing or declining. They are dynamic and can evolve but can also be dormant. Researchers are not limited to one Topic – they can contribute to many, shaping the connections and the research map as they publish their work.

At first glance, having so many Topics seems to complicate the picture, where traditionally 27 subject areas with 334 subcategories would have been used. But repeatedly over the last 25 years, the team has shown that researchers identify much more closely with fine-grained Topics that define the structure of science at the problem level. The granularity also makes analysis easier, opening the door to this new indicator.

“We will learn more over time to what extent research managers would need a more aggregated level of Topics,” said Dr. Klavans. “If that is the case, it’s something we can implement. From what we’ve seen, the fine-grained Topics are welcomed by researchers. Overall we’re excited to offer our audience this window onto their future planning.”

About SciVal

SciVal offers quick, easy access to the research performance of 8,500 research institutions and 220 countries worldwide. It enables customers to visualize research performance, benchmark relative to peers, develop collaborative partnerships and analyze research trends. As SciVal uses advanced data analytics, users can instantly configure and process enormous amounts of data, and generate on-demand data visualizations relevant to specific challenges. SciVal runs on Scopus data, which covers more than 69 million records from 22,000 journals and 5,000 publishers worldwide.

Using technology to extract knowledge from data

Empowering Knowledge page

To secure more funding, find the right collaborators and perform the most competitive analyses, researchers need to know where to focus their attention. Elsevier worked with SciTech Strategies to develop a new functionality in SciVal designed to help researchers see which topics are prominent now, and which are likely to be in the future.

Tags


Contributors


Related stories


Comments


comments powered by Disqus