Experimenting with collaborative peer review
We look at the experiences of three Cell Press journals which carried out trials on Mendeley
By Dr. Karen Carniol, Dr. John Pham and Meredith LeMasurier Posted on 13 November 2014
At Elsevier, we are focused on working with our journal editors and reviewers to improve and streamline the peer-review process. So when we acquired the global research management and collaboration platform Mendeley in 2013, we were keen to explore how peer review might benefit from its advanced collaborative features.
Researchers worldwide use Mendeley's desktop and cloud-based tools to manage and annotate documents, create citations and bibliographies, collaborate on research projects and network with fellow academics.
As reported earlier this year in Reviewers' Update, we launched a series of small pilots which saw research papers brought within the Mendeley environment with the consent of the relevant reviewers and editors. The journals involved in the trials were Molecular Cell, Neuron, and Cell, which are part of the Cell Press family. While there were variations in each pilot, the common factor in all cases was that the reviewers remained anonymous to one another. In this article, we highlight the results of each trial and our planned next steps.
The Molecular Cell pilot
Reviewers were invited to discuss the article only after they had completed their reviews the conventional way. Ten manuscripts were included in the pilot, and all six editors on the team participated.
Responses were positive. In a post-pilot survey, 17 of 24 reviewers responded, and ~94 percent said they liked the discussion and would be willing to participate in a similar discussion in the future. Five of the 10 authors we approached responded to the survey; they unanimously agreed that the interactive review process made it clearer to them how to revise their paper and the editor's summary of the discussion was helpful. Like reviewers and authors, editors also found the forum useful, particularly for reaching consensus on the experiments needed for reconsideration, and when reviewers disagreed. The process also prompted interesting meta-discussion about peer review and the role of the reviewer. While there were clear positives, the process took longer than conventional peer review (since the discussion was tacked onto the regular process). Some reviewers expressed concern about the extra work, and editors felt they needed to spend additional time preparing to lead the discussions.
The Neuron pilot
This experiment was more complex than the Molecular Cell trial. Here, reviewers were encouraged to perform the review process end-to-end in Mendeley. Ten manuscripts were included in the pilot and seven editors contributed. Reviewers were encouraged to annotate the PDF directly, but were also asked to provide summary comments that the editors used to initiate discussion.
Authors and reviewers were responsive to our requests to include them in the pilot, with only one author choosing not to participate. We found the reviewers tended not to use Mendeley's PDF annotation function, with the majority preferring to provide only summary comments. The quality and extent of the interactive discussions varied: in cases where the reviewers were largely in agreement that the paper did not have potential for Neuron, the discussion was more limited, whereas papers that were viewed as being more promising generally led to a more extensive discussion. The editors felt that the process was most valuable in these cases, and this sentiment was reinforced by reviewer feedback.
The Cell pilot
Reviewers were asked if they would like to directly interact only after the traditional review process was complete and they had been offered the opportunity to see each other's responses.
The responses were mixed and depended on the circumstances: the greater the divergence of reviewers' views on the paper, the greater their interest in interacting. It may therefore be wise for editors to make the option for interactive peer review available but not insist on it for every paper.
Enthusiasm for collaborative review appears to be generally high among researchers and we now plan to expand our ability to implement it. Improved efficiency; reducing logistical hurdles and the maintenance of fast turnaround times for authors will remain key priorities. One major challenge with using Mendeley for interactive peer review is coupled with a clear silver lining: so many of our customers already have Mendeley accounts that in order to ensure they could log in anonymously as a peer reviewer, they had to provide us with an alternate email address. With that in mind, we are now exploring how reviewers can use their EM/EES/Evise account information to seamlessly engage in collaborative review.
Thank you to Matthew Green from Mendeley, Elsevier Strategy Analyst Sweitse van Leeuwen, and Cell Press Editorial Coordinator Patrick Hannon for their support in implementing these Mendeley peer-review trials.