Why it's time to publish research “failures”

Publishing bias favors positive results; now there's a movement to change that

Source: <a href="https://xkcd.com/892" target="_blank">xkcd.com</a>After hours of testing, retesting, checking and confirming, Dr. Jones closes her lab book; her experiments have not disproved the null hypothesis and they don’t support her new theory. She leaves her negative results behind, moving on to work on something new in the hope that she’ll have a positive outcome next time.

Meanwhile, what happened in her lab will stay in her lab. No point in publishing negative results, after all.

Or is there?

Publication bias affects the body of scientific knowledge in different ways, including skewing it towards statistically significant or “positive” results. This means that the results of thousands of experiments that fail to confirm the efficacy of a treatment or vaccine – including the outcomes of clinical trials – fail to see the light of day.

Now, a movement is gaining momentum to counter this bias.

Recently, the World Health Organisation (WHO) announced its position on publishing the results of clinical trials, calling for the main findings to be submitted for publication in a peer reviewed journal within 12 months of study completion, and for all previously unreported results – including negative findings – to be published.

Meanwhile, new journals like New Negatives in Plant Science, approaches like Registered Reports and initiatives like the All Trials campaign are trying to balance the literature by encouraging researchers to publish their negative results.

To understand the challenge they face, it helps to understand why research and publication are so focused on positive results. The answer is not as simple as you might think.

In his 1963 collection Conjectures and Refutations: The Growth of Scientific Knowledge, science philosopher Karl Popper wrote: “Refutations have often been regarded as establishing the failure of a scientist, or at least of his theory. It should be stressed that this is an inductivist error. Every refutation should be regarded as a great success. … Even if a new theory … should meet an early death, it should not be forgotten; rather its beauty should be remembered, and history should record our gratitude to it.”

Perhaps the best-known example of a scientific refutation occurred in 1897 when Albert A. Michelson and Edward W. Morley published the results of their experiment to detect the relative motion of matter through the “luminiferous aether.” Their paper “On the relative motion of the Earth and the luminiferous ether,” in the American Journal of Science, described what has come to be known as the most famous failed experiment ever. The authors summarized their disappointing findings:

This new explanation… failed to account for the fact proved by experiment that the aberration was unchanged when observations were made with a telescope filled with water.

A few years later, Einstein developed his theory of special relativity, motivated in part by Michelson and Morley’s lack of evidence for the luminiferous ether. This is widely recognized as the starting point for the “second Scientific Revolution,” and it contributed to Michelson being awarded the 1907 Nobel Prize in Physics – a rare accolade for a “negative” result. But why is it such an unusual example?

The power of negative results

In their editorial in Disease Models & Mechanisms, Dr. Natalie Matosin and her colleagues from the University of Wollongong and the Schizophrenia Research Institute in Sydney, Australia argue that “Science is, by its nature, a collaborative discipline, and one of the principal reasons why we should report negative results is so our colleagues do not waste their time and resources repeating our findings.”

The majority of negative results serve this purpose – to add to our knowledge and function as a collaboration tool. But some, like the Michelson-Morley experiment, turn out to have huge significance.

When he looked back over his work to chronicle his career, Prof. Dr. Anthony Cerami, translational medicine pioneer and the Hermann Boerhaave Visiting Professor of Medicine at Leiden University, said he was surprised to see how significant his negative results had been:

Many of the biggest discoveries of my career were the results of failure of another research project. ... Failure strikes a negative tone, but it appeared in my personal history that it was an essential experience on the path to important discoveries.

Prof. Cerami describes one of these failures. He and his colleagues had come up with a number of compounds with the potential to fight trypanosome infection, a parasite that affects cattle. They travelled to Kenya to test one particular compound in cattle, which promptly died within minutes of administration.

The “treatment” soon became the joke of the town, but Prof. Cerami saw the value in the results, noticing that there must be something other than the parasite causing wasting in the cattle. He was right, and the protein they eventually isolated was tumor necrosis factor (TNF). He continued to work on the protein, eventually revealing its important role in inflammatory diseases including rheumatoid arthritis.

Publication bias: a publishing problem?

Despite their potential, negative results are repeatedly relegated to the lab books, the drawers and the trash bins. This is not a new phenomenon – research published in Controlled Clinical Trials in 1987 showed that statistically significant clinical trial results were three times more likely to be published than those supporting the null hypothesis.

Perhaps surprisingly, the researchers concluded that rather than being a result of editorial decisions, this was because scientists were failing to write up and submit papers describing their negative results.

When Matt Shipman, public information officer at North Carolina University, wrote about publishing negative results, he saw a flurry of social media activity – many researchers thought it was important, but some weren’t so keen on the idea. Peter Dudek was one of the people who responded on Twitter: “If I chronicled all my negative results during my studies, the thesis would have been 20,000 pages instead of 200.”

The academic community has developed a culture that overwhelmingly supports statistically significant, “positive” results. Researchers themselves strive for these results and rush to publish them, leaving the “failed” attempts in the dust. How can this culture be shifted towards valuing negative results?

The role of publishing in changing perceptions

Publishing is a major factor in academic success, so the publishing industry is in a position to change this academic culture and publication bias. In recent years, publishers and journals have been encouraging more submissions of negative results, particularly for clinical trials.

This is an area in which publication bias is being tackled from different angles. Launched by Sense About Science in 2013, the All Trials campaign calls for “all past and present clinical trials to be registered and their full methods and summary results reported.” According to the campaign website, “Millions of volunteers have participated in clinical trials to help find out more about the effects of treatments on disease, yet that important ethical principle about reporting has been widely ignored.”

The WHO’s announcement is a big win for the All Trials campaign, whose response explains the importance of publishing all results:

This is a strong statement from the WHO. It has been welcomed by patients, doctors and researchers worldwide. The best available evidence shows that around half of all the clinical trials that have ever been carried out have never reported results. … It means that information about the medicines we use every day is at risk of being lost forever.

To promote and enable the ongoing publication of new clinical trials data, many journals are encouraging submissions describing negative results. One journal published by Elsevier is taking it a step further: Cortex has launched a section called Registered Reports, which commits to publishing the outcome of a trial before it has been completed.

“Our aim with Registered Reports is to enhance the transparency and reproducibility of science by reviewing study protocols before experiments are conducted,” said Prof. Chris Chambers, Registered Reports Editor of Cortex and one of the concept’s founders, in the launch announcement. “If we think the protocol has merit we will commit, in advance, to publishing the outcomes. Armed with this provisional acceptance of their work, authors can perform the research safe in the knowledge that the results themselves will not determine the article's publication.”

A plant science journal dedicated to negative results

It’s not just the clinical trials literature that is missing negative results – any experimental discipline that works on the basis of a hypothesis runs the risk of this bias.

Emma Granqvist, PhD“All these results exist, but they are not published,” said Dr. Emma Granqvist, Publisher of Elsevier’s New Negatives in Plant Science, who was a researcher in molecular plant science. “Researchers are saying that if they submit these results, journal editors will reject their manuscripts because they’re not interesting or high impact.”

New Negatives in Plant Science aims to address this by publishing only negative results. “Some people disagree with having a journal purely for negative results, saying it’s the wrong approach – that all journals should publish these results,” Dr. Granqvist said. “Ideally, this would be the case, but it’s not realistic. A shift like this takes time – it’s about changing attitudes. This is the first step, a spotlight on the fact that negative results are largely missing from the literature.”

The language around results has an effect, too, Dr, Granqvist added. “By calling these results ‘negative,’ they are immediately less appealing – especially to the researchers themselves, who usually neglect to publish them.

“This isn’t an issue in every discipline; in mathematics, for example, counter-proofs and proofs by contradiction are all considered logical and are not as readily labeled as a failure,” she explained. “Perhaps we can take some inspiration from fields like this and start to value all results more equally.”

Related stories

Positively negative journals

An increasing number of journals proactively publish negative results. Here are a few of them:

  • New Negatives in Plant Science. Publishes hypothesis-driven, scientifically sound studies that describe unexpected, controversial, dissenting, and/or null (negative) results in basic plant sciences.
  • Journal of Negative Results. Provides an online medium to publish peer-reviewed, sound scientific work in ecology and evolutionary biology that is scientifically rigorous but does not rely upon arbitrary significance thresholds to support conclusions.
  • Journal of Negative Results in BioMedicine. Provides a platform for the publication and discussion of unexpected, controversial, provocative and/or negative results in the context of current tenets.
  • Journal of Pharmaceutical Negative Results. Publishes original, innovative and novel research articles resulting in negative results; publishes theoretical and empirical papers that report the negative findings and research failures in pharmaceutical field.
  • The All Results journals. The main objective of the four journals – Chem, Nano, Biol and Phys – is to recover and publish negative results, valuable pieces of information in science.


Elsevier Connect Contributor

Lucy Goodchild van HiltenAfter a few accidents, Lucy Goodchild van Hilten discovered that she’s a much better writer than a scientist. Following an MSc in the History of Science, Medicine and Technology at Imperial College London, she became Assistant Editor of Microbiology Today. A stint in the press office at Imperial saw her stories on the front pages, and she moved to Amsterdam to work at Elsevier as Senior Marketing Communications Manager for Life Sciences.

She’s now a freelance writer at Tell Lucy. Tweet her @LucyGoodchild.

comments powered by Disqus

Related Stories