In recent years, research has been increasingly measured based on bibliometric indicators, such as the Impact Factor. Now, in the midst of global recession, those holding the purse strings are looking for strategic investments, and funding is increasingly being awarded based on similar performance indicators.

The economic crisis is affecting academia, just as it is in most other industries. A lot of funding comes directly from governments, which are already diverting budgets to help free up the money markets. Meanwhile, publishing, an essential component of scientific development, is mainly carried out by private companies or associations, many of which are facing higher costs and lower returns for the foreseeable future. The natural reaction in any recession is to cut investment and contain costs to ride out the storm. But for science, funding is all about investing in innovation for the future.

For those organizations dispensing funds, the aim is to identify the best candidates for funding as efficiently as possible, and to do this they are utilizing many different indicators to help them find those institutes and groups that can offer the highest returns on investment based on their performance.

Indicating potential
According to Dr. Henk Moed, of the Centre for Science and Technology Studies (CWTS) in the Department of Social Sciences at Leiden University, the Netherlands: “Performance-based funding is gaining ground, and indicators are becoming an essential aspect of this process.”

Once upon a time, none of this really mattered to research institutes, but times have changed. Moed has witnessed this trend growing over the past decade. “A lot of funding comes from government sources. Politicians are demanding greater value for money in all areas, and science is no exception. This means that applicants for funding will have to be able to demonstrate research impact or risk losing funding over the long term.

“At CWTS, we measure research performance on behalf of other organizations, such as research institutes and research councils. What we do is quite relevant during times of economic recession. The results of our research help institutes identify areas of productivity, enabling them to see where they can make budget cuts and where they should protect budgets – for instance, for highly productive groups.”

While experts like Moed often carry out performance-impact analyses based on indicators, research institute managers are increasingly using specially developed tools themselves to analyze performance within their own institute. These tools can also generate analytical breakdowns of research performance across an institute, and provide data for benchmarking exercises both within the institute and with its peers.

Moed stresses that such indicators do not aim to replace qualitative analyses made through peer assessments of impact. “These indicators have been developed to support qualitative evaluations by supplying background and historical data on past performance.”

Showing your best face
Indicators may make life easier for those assessing performance, but it is far more important to ensure that we are serving science first. Some researchers can feel pressured to meet performance targets rather than doing the very best research.

Moed confirms that is a concern for those using them as well. “Both the producers and users of indicators are aware of the importance of looking at the impact of such indicators on researcher and editor behavior. This is quite a new idea. The UK’s Research Assessment Exercise (RAE), for instance, is heavily based on indicators, and there the discussion is focused on the effects this could have on research.”

Little empirical research has been done in this area, but anecdotal evidence abounds. For instance, some editors have learned that particular types of behavior can improve their journal’s Impact Factor, and it is natural that they would act accordingly. In most cases, this can improve a journal’s relevance to the community it serves.

Moed says: “In a sense, all indicators suffer this phenomenon: where there are rules, some people will learn to play them very well. But it’s worth keeping in mind that even when indicators aren’t used, game playing takes place. If you know you are being evaluated and have a good idea of what you are being evaluated on, you tend to show your best side. It’s human nature. If an author knows that one criterion against which his overall performance will be evaluated will be how many papers he has in certain journals, he will naturally work towards getting himself published in them. This happens regardless of whether it is the best place to publish that research.”

These effects on editor and researcher behavior will happen as a result of performance analysis, regardless of how such analyses are conducted, and the effects will often be positive as people work towards achieving greater impact in their research efforts. So, while there may be less funding available, the outlook is that this will result in more efficient research, rather than halting the progress of scientific innovation altogether.

Criteria for all
Another concern is that standardized assessment criteria could lead to standardized research, as people aim to meet the targets set for their performance.

Moed believes that this will depend on how institutions use these indicators. “As I said before, research performance analysis only gives you the data, the trends. To really understand research performance, the qualitative aspect is essential, and of course, how this is applied depends very much on the institution. While analysis might show that a particular research group is underperforming, the institute directors might know that this group is on the verge of a major breakthrough, and might therefore decide to protect their budget.”

He also believes that each country should have a plurality of funding sources, each with their own, different assessment criteria. For instance, research councils assess proposals and are essentially looking towards the future; national research funds tend to base their awards on past performance, while other funds are allocated directly to a university to re-distribute on its own criteria.

“A good mix of funding sources and reviews based on a wide range of criteria, both quantitative and qualitative, should serve to protect all types of good research, whichever criteria it happens to do well in,” says Moed.

To cite this article, please use: Michelle Pirotta, “Rewarding performance”, Elsevier Editors’ Update, Issue 27, August 2009