Peer review using today’s technology

© istockphoto.com/Urupong It is fair to say that without the PDF and email, scholarly publishing would not be where it is today. Such technologies underpin the entire system and they are all the more vital given the incredible numbers involved.  A 2014 study based on Scopus data demonstrated that the number of published articles continuously increased from 1.3 million in 2003 to 2.4 million in 2013 and there is no reason to believe that this trend has changed since ^[1].

This level of growth would not be possible if researchers still needed to send hard copy manuscripts via the postal system and editors/reviewers had to confine their communications to offline systems. To speed the process up and respond to the modern researcher’s need to see their research published as swiftly as possible, most journals employ submission systems, which are in essence an electronic version of the old log books journals used for recording and tracking manuscripts.

The pitfalls of peer review

Despite the various advances and enhancements, researchers still have good cause to believe that peer review is slow and inefficient. One reason is that the typical submission and review processes they encounter still don’t seem to belong to the internet age. Referees still generally receive a file, download it and provide their comments with much to-ing and fro-ing, uploading and downloading and often frustration along the way. Sometimes the devil is in the details for example with line numbers, which may appear differently on the reviewer’s copy of a paper vs. that which the author(s) prepare.

Editors in their turn have to spend time in finding out which reviewer has focused most on which part of the manuscript in order to prepare a balanced decision letter and reviewers themselves often feel distanced from the outcome of the operation (and even if they do have the opportunity to view comments made by their peers, it is often unclear as to who has done what).

Deus ex machina?

When we consider the world of academic publishing, we note that more and more online reading and annotation services are emerging each year and evolving to become more user friendly. These services support a wide range of academic use cases, moreover. Canvas(opens in new tab/window) for example enables supervisors to provide feedback on students’ work using online annotations. Hypothes.is(opens in new tab/window) allows academics to keep track of their annotations across manuscripts. Reference managers like Papers(opens in new tab/window) and Mendeley(opens in new tab/window) are offering annotation features to remind users of the important passages in the papers they read. Thanks to these new possibilities, services that allow online reviewing of manuscripts have started to emerge. An eye-catching example is https://www.liquidtext.net/(opens in new tab/window). Not only does this tool enable a user to select text and turn it in to a note, it also facilitates comprehension of a group of notes by creating relations among them in an intuitive and engaging manner. This “compare and contrast” feature among various passages in a text makes it easy to think about the research paper, for example, and in the case of reviewing, can help to provide a balanced critique.

This entire peer review process could be made considerably easier, swifter and more efficient if we could leverage such annotation tools and discussion forum technologies. In fact, here at Elsevier we have been experimenting with such tools during 2018 and have noted a good deal of positive feedback from users when doing so.

Peerful: leveraging technology to improve peer review

Back in 2017 we at Elsevier started to test the impact of online peer review on reviewer performance. It was clear to us that success would depend on the user-friendliness of the tool as well as its integration with editor and author workflows. Keeping this firmly in mind and working closely with our colleagues in Mendeley we developed Peerful, a powerful online peer review tool which enables referees to annotate, tag and discuss a manuscript as well as answering journal-specific manuscript quality questions and providing an overall decision recommendation. We launched the experiment in 2018 and tested it with two journals. For fifty randomly selected reviewers, the access link to the manuscript led them to Peerful instead of the normal submission system.

Surveying reviewers afterwards revealed the benefits of such an online manuscript annotation tool.

Most reviewers where happy that they didn’t need to download the manuscript and start their comments by indicating sections and line numbers. As one user put it: “This tool is AMAZING! … It is quicker, very efficient allowing me to highlight and comment instead of typing out page / line numbers every time; having it in a single window persuades me to finish the review in one sitting instead of putting it down to pick up later; I love how it pulls everything together into a single report at the end. Overall, excellent.”

Peering towards a brighter future for online peer review

The apparent satisfaction with the tool has encouraged us to continue iterating and looking for ways to further expand the experiment with a view to one day integrating it directly into our editorial systems. For example, we would like to develop Peerful to enable a referee not only to annotate but also to initiate discussions with the editor(s) and/or each other once they have completed a review. We are also hoping to further expand this element to enable editors to include the authors themselves in the discussion forum and thus generate a true discussion around the evolution of the paper.

These are plans for future, which keep us working towards bringing peer review online and making it a more rewarding experience. By introducing the tool to more journals in the future and expanding the initiative beyond reviewers to authors and editors, we are hoping to improve both the reviewer experience and the speed of peer review itself. That in turn will hopefully drive a better author experience and help streamline a system better able to handle the millions of submissions being reviewed each year.

References

1. https://www.researchtrends.com/researchtrends/vol1/iss38/5/(opens in new tab/window)