Scientific controversy? Systematic review can help

The field of chemical risk assessment is taking a cue from medicine


Science is full of controversy. Is red wine good for you? Are there parallel universes? Is the antimicrobial triclosan toxic to humans? The aim of scientific research is to answer questions like these, but what happens when two or more studies produce conflicting results?

With standardized methodologies for aggregating and analyzing studies, you can make the best use of the best evidence available. Ultimately, this approach – called systematic review – saves time and resources, avoids unnecessary research and, in the case of medicine, saves lives.

Paul Whaley

Paul Whaley, an independent scientific consultant and researcher at Lancaster University, is one of the guest editors of “Systematic Review Methods for Advancing Chemical Risk Assessment,” a special issue of Environment International.

Despite accounting for the most highly cited papers in medicine, systematic review is not yet widely used in chemical risk assessment. We have put together the first ever dedicated special issue on systematic review methods in chemical risk assessment. Published in Environment International, the issue features different methodologies we believe will be fundamental to resolving some of the biggest controversies in chemical risk assessment.

One such controversy is whether the chemical triclosan, which has been banned in soaps in the EU but is still commonly used in cosmetics, is toxic to humans. One of the articles in the special issue is the first systematic review in environmental health to show that triclosan is “possibly toxic” based on its adverse impacts on the thyroid hormone thyroxine. Thyroid hormone disruption indicates developmental toxicity.

Medical meta-analyses

Before systematic review, medics relied on experts to settle disputes. If there was more than one opinion about a treatment or diagnosis, for example, rather than reviewing all the evidence, an expert would be asked for their opinion. This meant practice could be based on opinion – often biased – rather than direct evidence.

In the late 1960s, doctors thought that if you gave a pregnant woman expecting to give birth prematurely a dose of steroids, you could reduce respiratory failure and respiratory illness in the infant. Research involving more than 1,000 pregnant women revealed a clear clinical benefit – if the mother was given steroids, the infant would be less likely to have problems. This could have had a significant positive impact on infant mortality rates, but because the results weren’t communicated quickly, it didn’t filter into general practice.

Doctors called for more studies, so a lot of follow-up trials were done. However, they were all very small and individually inconclusive, adding to the confusion about whether steroids should be given. This rumbled on for 17 years until a large meta-analysis was done, aggregating the results of all the 18 clinical trials that had been carried out. The meta-analysis – a systematic review – revealed clear clinical benefits, and it’s now standard practice to prescribe steroids.

This example shows how effective systematic review can be in helping settle a matter of scientific dispute. It also shows how much unnecessary and low-quality research can be done at the expense of an intervention being put in place.

With systematic review, once you know you’ve found and aggregated all the evidence, you can build a clearer picture and everyone can see what the evidence is really saying. It’s now the most cited type of research in the medical literature, and when a researcher proposes a new clinical trial, they first have to do a systematic review of the literature to demonstrate that the trial is necessary and that the answer is not already known.

In medicine, systematic review methodologies have transformed the way we use evidence to inform practice. We think it could have the same impact in chemical risk assessment.

Toxic triclosan

There’s a lot of controversy in chemical risk assessment. Recently, there has been disagreement over whether glyphosate, the world’s most widely used herbicide, causes cancer. The International Agency for Research on Cancer (IARC) and the European Food Safety Authority (EFSA) both conducted reviews of the evidence but came up with the opposite results: IARC says glyphosate is a carcinogen, EFSA says it isn’t.

This is exactly the sort of dispute a systematic review could help settle. If the methods EFSA used had been standardized and open to scrutiny, it would be much easier to see whether the difference in results was down to interpretation or limitations in how EFSA or IARC reviewed the evidence.

One of the reviews in the special issue goes some way to settling another dispute, over triclosan. An antimicrobial, triclosan is used in many consumer products, such as liquid hand soap, toothpaste, plastics and children’s toys, and it is found in humans and the environment. There is little evidence of the benefit of using triclosan in soaps, for example, and plenty of evidence suggesting it could contribute to antimicrobial resistance and disrupt hormones. However, studies have produced differing results.

Tracey J. Woodruff, PhD, MPHA team of scientists at University of California, San Francisco (UCSF), led by Dr. Tracey Woodruff, Director of UCSF’s Program on Reproductive Health and the Environment, developed a systematic review methodology for environmental health called the Navigation Guide. They applied it to answer the question: “Does exposure to Triclosan have adverse effects on human development or reproduction?”

The team first searched the literature, finding several studies on rats. They narrowed their focus to thyroid hormone disruption, which is an indicator of developmental harm, and pooled the data. They found that there is “sufficient” non-human evidence and “inadequate” human evidence of an association between exposure to triclosan and the concentration of the hormone thyroxine, concluding that triclosan could have a detrimental effect on human development or reproduction.

Paula Johnson, PhD, MPHDr. Paula Johnson, the lead author of the paper and leader of the California Safe Cosmetics Program at the California Department of Public Health, commented:

Consumers generally seem to believe that products available for purchase are proven to be safe and that the government would prevent unsafe products from being sold to us. This is not true. For example, the FDA is charged with regulating cosmetics, but it does not have the legal authority to approve products before they go on the market. Private companies have the responsibility to ensure the safety of their products, but no law requires specific tests to demonstrate the safety of individual products or ingredients. Products may contain ingredients with very limited safety testing or, for example, no data on reproductive effects from prenatal exposure.

Agencies should adopt systematic review methods to evaluate the toxicity of chemicals. The public could greatly benefit from this in terms of health and simply from a consumer right-to-know perspective. We could benefit even further if data gaps were acknowledged and chemicals had to be shown safe before widely used. The efforts underway to use systematic review methods at the National Toxicology Program and the US Environmental Protection Agency should bring greater scientific rigor and transparency to the process, and it is exciting to be a part of this effort through proof of concept studies applying the Navigation Guide.

Making risk assessment systematic

Using systematic review methodologies means all the evidence is considered, bringing greater transparency and reducing risk of bias to aggregating existing knowledge than we get when simply asking an expert for their opinion. It involves:

  • Specifying a study question
  • Writing a protocol for how the review will be conducted
  • Gathering all the evidence comprehensively
  • Judging the quality of every study included using the same criteria
  • Defining the type of evidence needed for a decision before conducting the review

The approach has worked well in medicine and has been adopted in social science, forestry and ecosystem research. But it hasn’t really taken off in chemical risk assessment.

We’re looking at developing systematic review techniques for aggregating information and evidence about the risks to health posed by environmental chemicals, in order to make hazard and risk assessment more scientifically robust. But this presents some challenges.

We have a much more varied and heterogeneous base to work with than medics do; they have a lot of clinical trials, which are reasonably manageable in terms of methodological uniformity and recognition for need for blinding, and tend to be structurally similar. But in toxicology and environmental science, we have a variety of epidemiological and animal studies to deal with. This means you have to aggregate studies that are very different.

It’s always a challenge to change the way things are done, and changing this at the structural level is quite difficult – you need a lot of evidence to show people something is amiss. Changing the way people approach this is one of the aims of this special issue. It’s about explaining why we’re optimistic about systematic review methods, and exploring what some of the methods we need to develop might be.

Dr. Woodruff shares this optimism:

We have missed too many early warning signals about the toxicity of certain chemicals and products. By applying systematic review methods to environmental health, we hope to reveal toxic warning signals earlier and improve decision-making regarding the use of toxic chemicals.

We need to show that chemicals are safe before putting them into our communities and our bodies. Our results from this and other case studies of applying the Navigation Guide systematic review method demonstrate proof-of-concept that systematic review methods are ready and capable of transforming science-based decision making about environmental chemicals.

Ultimately, we want to see systematic review methods implemented in chemical risk assessment. It’s easy to be skeptical about changes, but we hope the special issue will enthuse people; it should go some way to demonstrating that it’s feasible to make risk assessment much more systematic.

Read the special issue

Elsevier’s journal Environment International has published a special issue: Systematic Review Methods for Advancing Chemical Risk Assessment. The articles are freely available until May 18, 2017.

The featured review is available open access:

Johnson, Paula et al: “Application of the Navigation Guide systematic review methodology to the evidence for developmental and reproductive toxicity of triclosan,” Environment International (May 2016).

Authors of the featured review

Dr. Paula Johnson is with the California Department of Public Health, where she leads the California Safe Cosmetics Program. The program investigates and informs the public about hazardous ingredients in cosmetic products sold in California. She helped develop the Navigation Guide and complete the first case studies applying the methodology while a postdoctoral fellow with UCSF’s Program on Reproductive Health and the Environment. She holds a PhD in environmental health sciences from the University of Michigan and a master’s degree in public health from San Diego State University. Her background includes research on endocrine disrupting chemicals in consumer goods and product development in private industry.

Dr. Tracey J. Woodruff is a professor at the University of California, San Francisco (UCSF) Department of Obstetrics, Gynecology, and Reproductive Sciences, and the Philip R. Lee Institute for Health Policy Studies. She is director of the UCSF Program on Reproductive Health and the Environment. Dr. Woodruff has done extensive research and policy development on understanding and characterizing exposures to environmental chemicals and related adverse health effects, with a focus on early-life development. She also has extensive expertise in methods for translating scientific findings, with an emphasis on risk assessment and systematic reviews. Dr. Woodruff has authored numerous scientific articles and book chapters. She is the PI of a US Environmental Protection Agency (US EPA) and National Institute of Environmental Health Sciences (NIEHS) Children’s Environmental Health Center at UCSF. Previously, Dr. Woodruff was a senior scientist and policy advisor in the US EPA Office of Policy.

The journal

Environment International is an international, multidisciplinary journal with a specific focus on the impacts of human activities and environment on human health. It serves as a platform for academics, practitioners, regulators and policy makers to share and discuss insights into how our human activities and the environment shape and impact human health now and in the future. In doing so, it also brings forward opportunities to minimize health risk and maximize resilience. Topics covered include the impact of air quality on human health, the health implications and impacts of climate change and the use of systematic review methods for environmental and human health risk assessment.

Main image by


Written by

Paul Whaley

Written by

Paul Whaley

Paul Whaley is an independent scientific consultant and researcher at Lancaster University, investigating how systematic review methods can be adapted for use in toxicology and chemical risk research, to help ensure we are making best use of the best evidence in the course of developing environmental policy.
Written by

Lucy Goodchild van Hilten

Written by

Lucy Goodchild van Hilten

After a few accidents, Lucy Goodchild van Hilten discovered that she’s a much better writer than a scientist. Following an MSc in the History of Science, Medicine and Technology at Imperial College London, she became Assistant Editor of Microbiology Today. A stint in the press office at Imperial saw her stories on the front pages, and she moved to Amsterdam to work at Elsevier as Senior Marketing Communications Manager for Life Sciences. She’s now a freelance writer at Tell Lucy. Tweet her @LucyGoodchild.


comments powered by Disqus