The role of reviewers in improving journal quality
Although scientific journals vary greatly in scope and subject matter, and even in their processes, there is one goal all journal editors share: to maintain and improve the quality of their journals. Reviewers’ Update spoke to two prominent journal editors about the crucial role reviewers play in improving journal quality.Reviewers bring success to Cities
Veteran editor Andrew Kirby has been a book review and journal editor since the 1970s, and has been Editor-in-Chief of Cities for almost 15 years. The journal, which offers a comprehensive range of articles on all aspects of urban policy, provides an international and interdisciplinary platform for the exchange of ideas and information between urban planners and policy makers from national and local government, non-government organizations, academia and consultancy.
“Because our journal is international in scope, the quality and availability of reviewers is a particularly important issue to us,” Kirby explains. “After all, even though we may touch upon the same topic more than once, we may very well cover the issue from the perspective of two different countries, with two different governments and different circumstances. In that way, it’s rare for us to use the same reviewer more than even once a year.”
Kirby says that while Impact Factors are a great way to measure the results of improvement efforts, they should never become the primary focus. “Impact Factors are always relative — to other journals in the same field for instance — and are always open to deliberate manipulation by editorial policy. That’s why it’s important to continue to focus on the journal itself, and not solely on the numbers.”
The rewards of reviewing
Given the broad scope and international perspective of Cities, Kirby says that the search for reviewers is an ongoing struggle that is renewed with each issue. But Kirby suspects that there’s more to it than a vast subject matter.
“Even though I send out a large number of requests for reviewers, I can expect that at least 30% of the invitations will go completely unanswered. Either reviewers don’t receive the requests (because they are mistakenly identified as SPAM), or they simply don’t take the time to respond. Of those that do respond, the answer is frequently no.”
“Reviewers are often too busy with other work or their own research to invest in a quality review. And, I suspect, they don’t perceive much of an immediate incentive or longer-term reward for getting involved in peer review. Offering any kind of external incentive would be likely to have a negative impact on the quality of the reviewers who accept. The rewards have to come from the academy in terms of recognition for this kind of professional service.”
Interestingly, Kirby says that reviewers from countries such as Turkey and China, whose academies are strongly influenced by English-language journals, tend to be more enthusiastic about reviewing than their American and British counterparts. “Often, they are excited by the opportunity to participate in the process, and their reviews are therefore of a high quality. It’s even more impressive when you consider that these colleagues are reading manuscripts in English, which is not their first language, and this creates an added level of difficulty for them.”
One approach to successful reviewing
When asked what can be done to encourage more reviewers to get involved, Kirby laughs, “I suppose if I knew the answer to that, I would be in high demand in the publishing world!”
But Kirby’s approach with Cities is deceptively simple, and yet highly effective. “I treat each author and each reviewer exactly the way I’d like to be treated when I submit a manuscript. We make every effort to respond quickly and professionally, to make the review process as simple and user-friendly as possible and to treat each scientist with whom we come into contact with respect and appreciation.”
Often, this approach leads to a longer-term relationship with authors than initially anticipated. “Sometimes, an author submits a manuscript that is interesting and noteworthy, but yet somehow falls short of the quality that we need for our journal. Instead of just rejecting it outright, we work with the authors – relying heavily on the opinions and input of the reviewers – to reshape and revise the article until it is ready for publication.” Kirby says that even if this means a paper goes through multiple rounds of review, in the end, the result is a high-quality paper that both the author and the journal can be proud of.
Kirby has embraced a fully electronic process – from submission to proof correction through to publication – which most of his authors and reviewers find helpful and time-efficient. Through tools like Scopus and EES, Kirby is able to reduce the time needed to seek and find appropriate reviewers, thereby making the entire process smoother. It has allowed him and his editorial colleagues to handle over 150 submissions annually, a significant increase from even five years ago.
“When authors submit a paper, they expect it to be reviewed by their peers in the field, and to be handled efficiently. If only those expectations could always be in place when those same scientists are called upon to assess another author’s manuscript. I don’t think we’d have as much difficulty finding quality reviewers if all scientists could see the issue from the other perspective, and give back to the community that serves them well.”
Focusing on journal improvement
Although he was just appointed Editor-in-Chief at the beginning of 2008, Roger Narayan has already been making great strides with his journal, Materials Science and Engineering-C: Materials for Biological Applications (MSE-C). In fact, significant improvements in submissions, citations and recognition were already noted within the first six months after Narayan took the lead.
“When it comes to journal quality, it’s important to remember that no single factor is as influential as the sum total of all improvement efforts collectively,” Narayan says. “An editor must always be willing to look at the process from all sides, to determine the best course of action.”
In the case of MSE-C, that meant getting back to basics. Narayan broadened the subheading and scope of MSE-C, to fill noticeable gaps in subject areas, thereby improving the journal on a broad scale.
This also meant increased submissions – papers that previously fell outside of the journal’s scope could now be accepted under the new format. But other improvement efforts, such as enlisting the help of experts in the field to not only contribute to the journal, but also to sit on the Editorial Board, helped boost MSE-C’s visibility and reputation. Special issues and the journal’s involvement in society meetings and events all played a role, as well.
Reviewers make the difference
Despite all of these fundamental and structural changes, Narayan still believes that the journal’s reviewers have a tremendous impact on overall journal quality.
“For MSE-C, reviewers play two distinct and highly significant roles. First, reviewers have the difficult task of evaluating manuscripts and ensuring that they are technically, mathematically, scientifically, and even grammatically sound,” he says.
But equally important to Narayan is the role that reviewers play as sparring partners. “Reviewers are themselves experts in the field, actively researching and writing on similar topics, so I take their opinions very seriously. MSE-C reviewers are one source I use to determine the ‘hot topics’ of the moment. They let me know when it’s time to delve deeper into a new subject area, and also when to back off from a topic that is considered ‘over-done’ or out of date. In that way, our reviewers help us ensure we remain at the cutting edge of our field.”
From the reviewer’s perspective
Jun-ichi Anzai has been reviewing for MSE-C for a number of years. He was involved in the review process before Narayan was appointed Editor, and has recognized the significant improvements that have occurred in the past eight months. So far in 2008, Anzai has reviewed no less than 13 papers for MSE-C, and says he’s noticed a significant broadening of the subject matter of the papers he reviews.
“To me, the decision to review is mostly determined by whether the subject matter of the paper fits into my own research,” Anzai says. “I often review for MSE-C for that very reason.” In fact, Anzai says that the manuscripts he reads often stimulate his own interests and perspectives, and he’d be happy to encourage his colleagues and associates to review for the journal as well. “The review process helps improve the overall quality of scientific publications, and can sometimes even provide new information and insights that the reviewer will find helpful in his or her own work,” he says.
The role of the Impact Factor
In 2007, MSE-C had a total of 293 submissions. By June 2008, 296 manuscripts had been submitted. The journal’s Impact Factor has also been on the rise this year. But Narayan emphasizes that these scores are merely a measurement of his success – they are not the success itself.
“Although some editors use their Impact Factor score as a goal in itself, I see it as more of an indication of the success of our other efforts,” Narayan explains. By simply focusing on improving the quality of the journal, we can expect to see improvements in the scores. And by continuing to provide quality articles reviewed by experts in our field, we look forward to even further progress.”
Reviewers are crucial
Although discussion continues about the best way to manage the peer review process and all its components, editors are in agreement about the crucial role peer review plays in overall journal quality. Peer review has been, and will continue to be, an invaluable process in ensuring sound scientific publication, and the continuing growth of scientific knowledge and understanding.
To that end, both Kirby and Narayan encourage scientists and researchers to do their part to contribute to this process. They suggest that reviewers keep an open mind when approached to review a paper, to remember that they too will be dependent upon reviewers when submitting their own manuscripts, and to contribute where they can to the publication of high-quality science – both as authors and as reviewers.