New article type verifies experimental reproducibility
Information Systems journal creates “Invited Reproducibility Paper”
By Hylke Koers, PhD, and Rebecca Capone Posted on 11 April 2016
Reproducibility — the hallmark of scientific research — is still more of an ideal than a practical reality for many research communities in computer and computational sciences. As it now stands, much of the research published in computer science journals and presented at conferences cannot be readily and fully reproduced, raising fundamental questions about the quality and validity of the research and the experimental methods used.
While the situation is not unique to computational sciences (see “Estimating the reproducibility of psychological science” and references therein), there are challenges specific to the field. The complexity of in silico systems, with their many interdependencies and variables; the requirement of precise systems configurations; the great task of documenting and preserving every aspect of the experimental environment; and the effort to validate these conditions present formidable roadblocks to reproducibility.
Invited Reproducibility Papers
For more on Invited Reproducibility Papers, we invite you to read this editorial in Information Systems: Fernando Chirigati et al: “A collaborative approach to computational reproducibility” (March 2016)
Information systems are the software and hardware systems that support data-intensive applications. The journal Information Systems publishes articles concerning the design and implementation of languages, data models, process models, algorithms, software and hardware for information systems. Read more.
Information Systems, a data science journal published by Elsevier, has devised a solution to the question of reproducibility by establishing a new article type: the Invited Reproducibility Paper.
Authors of selected published articles are invited to co-author, with the journal’s reproducibility reviewers, a report by which the experiment described in their published article is reproduced and verified. Authors capture their experimental environment (ReproZip and Docker are recommended to fully package and visualize the experiment), and together with their code and data make it available to the reviewers and readers via Mendeley Data. Invited Reproducibility Papers not only verify the experiment but incentivize both authors and reviewers to participate in reproducibility by creating a publishable and citable article.
This initiative is one of various ways Elsevier is helping to improve reproducibility in research through innovative content and technology. Other initiatives include the Minimal Data Standards and Resource Identification Initiative, SoftwareX, and the database linking program together with other provisions to share research data. In the computer science domain, this initiative is a logical continuation of the Executable Paper Grand Challenge and subsequent pilots.
Learn about Executable Papers
This video illustrates a pilot project that resulted from the Executable Paper Grand Challenge. The pilot has since concluded, but it has led the way for initiatives like Invited Reproducibility Papers.
Read about Executable Papers on Elsevier Connect.
Elsevier Connect Contributors
Dr. Hylke Koers is the Head of Content Innovation at Elsevier, leading a team that is responsible for enhancing the online article format to better capture and present modern-day research. Part of Elsevier's Article of the Future program, this includes improved online presentation, better support and visualization of digital content, and contextualization of the article by linking with data repositories and other sources of trusted scientific content on the web.
Before joining Elsevier in 2010, Hylke received a PhD in theoretical astrophysics from the University of Amsterdam and served as a postdoctoral research associate at the Université Libre de Bruxelles. He is based in Amsterdam.
Rebecca Capone is a Publisher for Theoretical Computer Science journals at Elsevier, joining the organization in 2008 after managing Life Science journals for Wiley-Blackwell. She has led various reproducibility initiatives at Elsevier, including co-organizing the 2011 Executable Paper Grand Challenge.