Q&A: How to use bibliometrics to evaluate interdisciplinary research

Eleonora Palmaro, a bioengineer at the Italian Institute of Technology (and an Elsevier trainee) shares her insights

Sophia Katrenko, PhD, is a data scientist who leads Elsevier’s Data Science team.In October, a conference organized by Elsevier and hosted at Ca' Foscari University in Venice brought leading professors and researchers together to discuss “Advanced Research Management Tools” at the Italian Research Management Workshop. Delegates from universities in Venice, Bologna and Belfast met with Elsevier colleagues to discuss global rankings, research assessment, smart data and algorithms, and the use of software such as Pure.

At the conference, Eleonora Palmaro, a biomedical engineer in charge of publications analysis at the Italian Institute of Technology (IIT), gave insight into IIT’s best practices for research performance assessment in interdisciplinary areas. She also shared her experience as an intern at Elsevier’s Amsterdam headquarters, where she works with our Data Science team to learn about the data and methodologies we use to support the research assessment process.

Recently I interviewed her for Elsevier Connect.

Eleonora, what is bibliometrics and how does it contribute to advancing science?

Eleanora Palmero analyzes the research performance at the Italian Institute of Technology (IIT) in comparison with universities worldwide using bibliometric indicators. As an intern with Elsevier’s Data Science team, she is learning about the data and methodologies we use to support the research assessment process.Bibliometrics is the quantitative analysis of science through its products, such as publications and bibliographies. In the last few years, bibliometric indicators have been officially brought into national assessment exercises (including the Italian one), researchers to decide where to publish, to assess the impact of their research, and to find experts in their research area. It also provides reliable instruments to direct funding because it helps to define where it is most profitable to invest. However, social science and arts also need, for instance, a qualitative parameter like the peer review of experts, on which the ranking of journals also depends.

As we know, research would not exist without productive and performative research institutions. What kind of data do we need to assess the productivity of a research institution?

To assess the productivity of a research institution, we consider not only scientific publications but the institute’s ability to attract funding through international projects as well as researchers’ impact on industry – for example, their patents or start-ups.  

You work for an Italian research center whose scientific activity started in 2006. How is the IIT doing in terms of research performance? And what are they doing to further improve the impact of their scientific production?

The institute has obtained excellent results, confirming that we are going in the right direction. Due to its international nature, the IIT puts qualitative and quantitative evaluation at the center of scientific planning and efforts to attract successful foreign researchers. An external evaluation committee verifies that scientific goals have been achieved and makes sure that the institute adopts international research management standards. This activity goes hand in hand with a yearly internal annual assessment of employees’ performance, according to an MBO (Management by Objective) model and with the evaluation of research program. Furthermore, the institute voluntarily underwent the 2012 national research assessment, emerging as one of the best research institutes of its size. To further improve evaluation criteria, we are also developing customized ad-hoc assessment practices.

IIT promotes interaction among research areas. What are the major difficulties in assessing interdisciplinary research sectors? 

One of the major difficulties we face during the assessment of interdisciplinary products, including our own, is to identify core research areas. It is particularly challenging for disciplines like robotics -- a branch of engineering that overlaps with electronics, computer science, artificial intelligence, mechatronics, nanotechnology and bioengineering – to set boundaries and determine which articles belong to it. If a disciplinary area is not defined by current classification systems, our approach is to look at the content of scientific products and search for some key concepts that we have previously identified with a dictionary.

Despite very recent progress, Italian research institutions seem unable to retain their researchers and to attract foreign “brains.” According to the Italian National Statistics Agency (Istat), only 6 percent of PhDs in Italy are non-Italian. This share is slightly higher than a decade ago (2.2 percent). How does the IIT tackle the “brain drain” and favor “brain circulation”?

Brain drain is a real problem only if it is not balanced by “brain gain.” Balancing does not always occur, especially because Italian research infrastructure and salaries are usually not up to international standards. Besides that, red tape contributes to discouraging international researchers. Thanks to its international character, our institute manages to attract a significant number of international researchers. Currently, 30 percent of the PhDs doing their research at the IIT have foreign nationality. We cannot add much more to this question, however, because PhD titles are not released by the IIT.

What did you like the most about you experience in the Netherlands? Is there anything you are taking away from working for Elsevier? What are the skills and experiences that you will bring back to IIT?

Of the Netherlands and in particular Amsterdam, I like the international environment, the capability of different cultures to live peacefully together, the possibilities to experience in just one city different kinds of food, music, sports, exhibitions, events. All these experiences are combined with a unique scenario characterized by the canals, the different profiles of the houses, the countless bikes, the windmills.

From the job’s point of view, I gained more awareness of the importance of bibliometric analysis as tool to help to make strategic decisions. I understood the state-of-the-art tools and practices of this discipline and the challenges we face. I appreciated for the first time the power of data mining. I will go back with a broader knowledge of the tools, with more ideas and with the desire to study more to become a data scientist. I am very grateful for this internship. With the communication and exchange of ideas and people between companies and research institutes and universities can be born tomorrow’s solutions.

Elsevier Connect Contributor

Dr. Sophia Katrenkois Team Lead for Data Science at Elsevier, focusing on information extraction from various data sources, data modeling and analysis. Sophia has extensive knowledge of predictive analytics, having worked for over 10 years in the fields of natural language processing and, more recently, risk modeling.

Prior to joining Elsevier, she had been building default prediction models for small and medium enterprises at the Capital Tool Company, G-20 SME Finance Challenge Winner 2010. Sophia had been affiliated with the University of Utrecht, University of Amsterdam (the Netherlands), Tuebingen University (Germany) and Lviv Polytechnic University (Ukraine). She has been involved in organizing events on machine learning, served on the program committee of multiple international conferences and published over 30 papers.

Sophia holds a PhD in Computer Science from the University of Amsterdam on the topic of information extraction, and an MSc in Computer Science (with distinction) from Lviv Polytechnic National University, Ukraine.

comments powered by Disqus

Related Stories