(Re-)reviewing peer review with psychologists and cognitive scientists

An Elsevier publisher discusses peer review and its future with researchers

A scientist himself, cartoonist <a href="http://lab-initio.com/">Nick D. Kim, PhD</a>, takes a playful jab at the peer review process. (Used with permission)When I get feedback about peer review from publishers and scientists, I sometimes call to mind the well-worn Churchill quote that — to paraphrase slightly — traditional peer review is the worst form of review except all those other forms that have been tried from time to time. Nevertheless, we should continue to challenge and reform the system we have become perhaps too comfortable with, and continue to try other variations as well.

The Psychonomic Society meeting in Long Beach, California, last November was an excellent opportunity for us to take a step back from the day-to-day reality of research and reflect on the current state and possible future of peer review. The “us” in this case was a group of researchers in cognitive science and psychology, and myself as Elsevier publisher of journals in these fields. So here are some thoughts that surfaced during this discussion.

Peer reviewing as an opportunity to develop yourself

One of the issues raised — particularly pertinent for younger researchers reviewing for the first time — was the evaluation of one’s peer review. Although the editor’s decision implies whether or not an editor had agreed with reviewer’s comments, many journals don’t have a mechanism in place to provide direct feedback to reviewers on how they actually did — which leads me to think that constructive criticism or praise from editors to reviewers would likely be welcome.

In cooperation with conference organizers and institutions around the world, Elsevier has been addressing this issue in reviewer workshops, where publishers, early career researchers and experienced editors work together to highlight the importance of the reviewer in the publishing process and outline the fundamentals of reviewing a scientific paper.

If we encourage and develop better peer reviewers, who then have a closer relationship to the journal that is playing a role in their development, there’s no doubt that the benefits would be shared amongst all involved. This should drive us to explore new ways in which publishers can support the collegiate networks of researchers that are so essential to the development of science. Indeed, many researchers are already exploring new ways of building networks via social media with ResearchGate, Academia.edu, and Mendeley: tools that cut across all publishers and give researchers the free, easy and intuitive means for communicating with scientists around the world, helping to advance their career and aid personal development.

As good as it gets?

Journals predominantly apply the traditional form of (single-blinded) peer review, where the authors don't know who the reviewers are but the reviewers do know the authors, and the editor coordinates the process and makes a decision before a paper is being published. In the spirit of science itself, which only advances through experimentation, publishers are challenging the system by trialing different variations of peer review.

Reactions to such innovations have been mixed. Post-publication peer review did not seem to be as popular an idea amongst the cognitive scientists and psychologists in the room. How would one ensure that all papers receive attention? Would there be a growing inequality in reviewing where the more prominent research from top institutions would receive a lot of attention and review, whereas smaller, lesser known research would fail to find any peer review? One of the aims of pre-publication peer review is to provide an equal process for all, and even if it doesn’t always achieve this aim, it is not unreasonable to assert that it has an equalising pressure on an unequal system.

But would we expect such a reaction to post-publication peer review if we were to get a group of mathematicians together? Clearly there is significant variation that would depend on the specific circumstances of particular fields, meaning that there may be no one-size-fits-all approach — just like there isn’t at the moment. Any innovations and developments in peer review will need to be closely in line with the needs and cultures of each academic community rather than playing to a perceived idea of “The Academic Community” as one homogeneous body.

If anyone has any other ideas, or would like to share their thoughts or feedback, then please join the conversation in the comments below, or contact me. I look forward to hearing more about how we can continue to challenge, improve and experiment in the future.


Elsevier Connect Contributor

Adam FraserAdam Fraser (@AdamFraserPsych) is Associate Publisher for Cognitive Science Journals at Elsevier, working in the area of cognitive science and psychology. His portfolio includes Brain and Cognition, Cognitive Psychology, Computers in Human Behavior and other journals. He received his undergraduate and master’s degrees from the University of Warwick and is based in London.

comments powered by Disqus

Related Stories