Japan’s bid to define institutional research metrics

A collaboration in Japan seeks to bolster universities’ competitiveness by finding more adaptive measures of research performance

© istock.com/DrAfter123

Japan has a long-standing heritage of ground-breaking research and academic freedom, producing no fewer than 25 Nobel laureates. And on a lighter note, it has clinched an Ig Nobel Prize for Improbable Research for the past 10 years. Universities (the country has 86 national ones) have been a cornerstone of this proud tradition, yet in a rapidly changing world, institutions must also adapt to a new research landscape which is increasingly competitive and globalized.

Throughout Asia, many universities have been successful in doing this, yet Japan, despite its research strengths, has experienced a declining trend in international rankings, causing widespread concern amongst its academic community.

When the Japanese government set out its goal to have 10 Japanese universities included in the top 100 of the Times Higher Education (THE) World University Rankings within 10 years, there were questions raised about whether applying the same metrics to all institutions – regardless of their location, size or scientific focus – represented the best way forward. While at first glance a standard “one-size-fits-all” approach might seem fair, it can also be argued that the ranking criteria – which is geared primarily towards the recruitment of international students and measures impact primarily against English-language citations – does not accurately reflect research performance, particularly as it does not account for non-English publications. These problems were exacerbated last year when the rankings criteria were abruptly changed, causing many institutions to drop several places and jeopardizing their research funding, which was benchmarked against their rankings.

Yet Japanese scholars nevertheless recognized that their country’s steady decline in global university rankings represented a problem that needed to be addressed. The question then became: How can they do this in a way that preserves both research strengths and academic freedoms?

Preserving research strengths and academic freedom

Amane Koizumi, MD, PhDA new initiative seeks to answer to this question. It aims to identify a set of metrics to better understand and monitor Japan’s unique research landscape so universities can take concrete steps towards improving their research competitiveness on the global stage.

“Japanese universities are focusing on improving their global research competitiveness, and an evidence-based approach is essential in order to achieve this,” said project lead Dr. Amane Koizumi, a professor at the National Institutes of Natural Sciences. “University rankings are not sufficient in themselves, as they do not provide guidance to universities on how they may actually improve.”

In May, an academic subcommittee meeting was held at Japan’s Ministry of Education Culture, Sports, Science and Technology (MEXT) to discuss whether alternative metrics might better reflect research performance, taking into account different functions and subject areas to analyze each institution’s global research positioning and competitiveness.

This led to the launch of a 2-year project bringing together scholars in areas such as scientometrics and institutional research to explore a range of viable alternatives. They were joined by a board of international advisors including Dr. John Green, Honorary Research Associate at Imperial College; Dr. Lisa Colledge, Director of Research Metrics for RELX Group (Elsevier’s parent company); and M’hamed Aisati , Director of Content and Analysis at Elsevier from Elsevier and experts in Snowball Metrics, an independent initiative pioneered by a group of UK Universities which uses a broad array of indexes with transparent and consequently reproducible calculations.

Dr. Koizumi explained:

We do not even have a comprehensive Japanese national database for non-English scientific literature, for example. That is a key part of visualizing our own research performance. Elsevier is supporting the project primarily by providing us with access to its databases such as Scopus and SciVal — which are the most extensive collections of English-language research literature in the world. Yet they are also collaborating with researchers to devise better ways of calibrating the many existing indexes to account more accurately for the differences between various scientific fields, and account for the ways that institutions of different sizes operate within different countries. The long-term goal here is to ultimately arrive at an agreed model of multi-dimensional metrics for evaluating research performance.

Upcoming symposium in Tokyo

Around 200 participants are expected to attend a kick-off symposium in Tokyo on October 25 to gather feedback and share the project goals with the research community. Following the symposium, the project will start in earnest, informed by those initial discussions, with results of expected to be published by the end of June 2017.

Dr. Koizumi, principal investigator of this project, emphasized the importance of taking individual characteristics of each institution into account when determining their research impact:

We need an adaptive and intelligent system to take these differences into account. For example, Japan has a strong tradition in the social sciences, and a wealth of knowledge in areas such as literature, history and traditional medicine. In the project, we focus on how we can analyze our research performance by institutional location, size or scientific subject area, and crucially including non-English literature. In the first instance, the team will list all possible indexes to measure research performance and subsequently select core-indexes by scientific subject areas. The end goal will be to propose concrete and comprehensive ways to analyse research performance by institutions and by scientific subject areas.

The symposium will attempt to address some of these issues by investigating how Snowball Metrics can be applied to the Japanese institutional landscape. This new adapted version of the model will then be shared broadly with the research community, both in Japanese and in English. The longer-term goal is to pave the way for enabling Japan and other countries to build their own adaptive metrics models, accounting for variations in culture, geography, subject area and language, thus contextualizing each university’s true impact.

Also contributing to this report: Yoshiko Kakita, Regional Solution Sales Manager, and Ludivine Allagnat, Senior Academic Relations Manager, both from Elsevier’s Tokyo office.



comments powered by Disqus