Journals are the most important means of representing results in the field of science communication, particularly in the natural science and medical fields. With the use of bibliometrics, it is possible to analyze in detail the role of a journal in a specific field of science. Elsevier and the Centre of Science and Technology Studies (CWTS) at Leiden University in the Netherlands have been working together for the last 21 years to analyze journal performance.

Ton van Raan is the Director of CWTS. We talked to him about the relationship between bibliometrics and journal performance and asked him what tools are available to measure this performance.

“Bibliometrics is about how scientists cite each other and how many times a paper is cited, taking into account the citation characteristics of the field,” explains van Raan. “Citations of one specific publication might not be statistically significant in themselves but they will be as part of a larger entity or a network. Bibliometrics is a method by which citation analysis can provide information about the role of a journal such as scope, impact and prominent authors in a particular field.”

Different tools for different uses
There are a number of tools that can be used for the purpose of measuring journal performance, depending on which information is required. A publisher of several journals might be interested in knowing what their share of the market is. “They might, for example, have five journals in the field of immunology and would like to know how many of these journals are increasing their market share,” adds van Raan. This measure of journal performance is market-related and involves just figures.

The next step in citation analysis is to look at the impact of the papers in a journal, for instance how often the journal is cited by other journals. A publisher can own several journals but they might attract little interest and are therefore not being cited frequently. This means, as a publisher, that you have a strong share of the market in terms of the number of papers, but your journals may have very little impact. It is therefore necessary to carry out a citation analysis in order to establish the impact of the papers in your journal.

Next, you need to analyze how the impact is developing over time, whether it is increasing or decreasing, and what the impact is per document type, such as letters, reviews, etc. Different document types have different citation characteristics. It is important to know what the contribution of these different documents to the total impact of the journal is. “If you look at the share of a journal within all the journals of a publisher as far as its impact is concerned, it is very possible that the share is not large but yet the journal is frequently cited, comparatively speaking,” according to van Raan. In certain fields, like mathematics, the citation level, or amount of references made in a paper to other papers, can be quite low, which can be rather different in other fields, such molecular biology, where it is quite high. Therefore, it is of crucial importance to normalize the impact factor according to the standards of the field. man, the media and making babies’, Professor Robert Winston, one of the world’s leading authorities on the human reproductive system, talked about his life and work; while Nick Arnold, best-selling author of the Horrible Science books for children covered gruesome, curious and revolting in a family event.

The next important issue to look at is the distribution of impact in terms of citations of papers within a journal. This distribution is very skew. Only a few papers receive many citations, while most papers receive very few or none. It is important for a publisher to know who the most impact-attracting authors or institutions in a journal are. “For Elsevier, we analyze which papers contribute most to the impact of the journal. We also establish for these highest-impact papers whether they come from only one or a restricted group of authors or institutions, or whether they are more evenly spread” explains van Raan.

It is also important to look more precisely at the relationship of a journal with other journals. Impact is generated by papers in other journals which may come from a different field. If you analyze the citations received by your journal, you can build up a good impression of the influence of the journal in its own field and possibly also in other fields. You can make a profile of the journal, in which it becomes visible how it is cited by other journals and how it affects other fields.

Network analysis
A further step in journal performance involves analyzing all the citation relationships between journals in a specific field of science. A network analysis of a journal is carried out, making it possible to establish how the citations from other journals give structure to this network of journals. Thus, on the basis of the ‘citation traffic’ between journals, it is possible to create a picture of all of these links between journals and to make a landscape of related journals, some close and others less so. This is what is known as journal mapping, which is a very important tool for editors of journals to establish what the positions of all journals in the network are and, especially, the position of their competitors. “Network analysis of related but different journals based on citation relations gives information regarding all these connections. This makes it possible for us to then build networks of journals in which often a number of smaller clusters can be identified which indicate separate subfields and, perhaps, even emerging research themes,” concludes van Raan.

To cite this article, please use: Gloria Kenny, "The role of bibliometrics in journal performance", Elsevier Editors' Update, Issue 21, February 2008

Useful links:
  Centre of Science and Technology Studies (CWTS) at Leiden University



 Please send responses to: EditorsUpdate@Elsevier.com