Why systematic reviews matter

This article was written as part of the Cochrane International Mobility Programme.

Introduction

The vast scale of scholarly literature occasions various problems. One is how to comprehensively record and assess the state of knowledge on a particular topic. A potent means of doing so is that of systematic reviews. The continuous growth of research, coupled with the demand to systematically summarize the available evidence to inform decisions from consumers and stakeholders, led to the formal development of systematic reviews (SRs) in the late 20th century^1-3.

Systematic reviews search, appraise and collate all relevant empirical evidence in order to provide a complete interpretation of research results. Although conventional SRs are typically used in clinical research and social sciences, they have found application in various other subject areas for example in advertising, education, international development, public policy, ecology, environmental sciences, engineering and basic science research^4-7

A brief history of systematic reviews

The first example of a systematic review was conducted in 1753 by James Lind, who published a paper that aimed to provide a concise and unbiased summary of evidence on scurvy^{8 9}. However, it was not until the 1970’s and 1980’s when more attention was paid to the growing need to improve the state of evidence synthesis.

In 1972, Archie Cochrane published a textbook titled “Effectiveness and Efficiency: Random Reflections on Health Service”¹⁰. Cochrane drew attention to the vital importance of randomized control trials in determining the effectiveness of health treatments. This led to a greater international emphasis on the need to improve research synthesis by policy makers, academics, and clinicians³. Gradually, topic areas outside of healthcare also adopted SRs as a way of comprehensively and systemically summarizing existing research.

How to conduct a systematic review

If you are considering embarking on a systematic review, there are several issues you need to contemplate if you wish to conduct one. In healthcare, for example, the first step would be to define an explicit research question by using the PICOTS (Population, Intervention, Comparator, Outcome, Timing, Setting) framework¹¹, and also register the protocol for the review on PROSPERO (https://www.crd.york.ac.uk/prospero/(opens in new tab/window)), the international database of prospectively registered systematic reviews. Protocols provide a complete detailed description of the process by which the review will be conducted. Registering the protocol reduces research bias, duplication of effort, resource waste, and provides greater transparency¹². Outside of medical sciences, protocols can be uploaded to Open Science Framework (https://osf.io(opens in new tab/window)).

You must adopt a comprehensive, objective and reproducible search strategy to capture all relevant sources of evidence. In doing so, you can be confident of having incorporated all the appropriate material for the topic at hand. A thorough search strategy should involve multiple databases, registries, sources of grey literature (https://onlinelibrary.london.ac.uk/resources/databases/opengrey(opens in new tab/window))¹³, conference proceedings and abstracts. Following the predefined eligibility criteria, you then need to analyze the screened search results to extract data from those publications that meet the inclusion criteria.

Don’t forget to assess the risk of bias when applicable (i.e., in clinical research). Ideally, these methodological steps should preferably be performed by two authors independently, one of which is a methodologist and the other a content area expert. Summarizing the results of the included studies and interpreting their findings in the light of certainty of evidence and their applicability are the final steps of completing a systematic review. You can also include a meta-analysis if applicable.

In order to assess the methodological quality of systematic reviews in biomedical sciences, checklists like the AMSTAR – “A MeaSurement Tool to Assess systematic Reviews"¹⁴ (https://amstar.ca) can be utilized. Lastly, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)¹⁶ (https://prisma-statement.org/) checklist, is a minimum set of items for reporting in systematic reviews: include one with your full article. The flow diagram can also be adopted to use for non-medical research (https://prisma-statement.org/PRISMAStatement/FlowDiagram).

Why are systematic reviews important?

Systematic reviews offer a number of benefits. For starters, they deliver a clear and comprehensive overview of available evidence on a given topic. Moreover, SRs also help identify research gaps in our current understanding of a field. They can highlight methodological concerns in research studies that can be used to improve future work in the topic area¹⁷. Lastly, they can be used to identify questions for which the available evidence provide clear answers and thus for which further research is not necessary¹⁸.

The process of conducting systematic reviews, especially for new authors, will prove to be a worthwhile endeavour. Authors refine their knowledge on the subject area of interest, develop new research ideas, and gain critical skills in synthesising existing literature.

We hope that you have found this introduction to systematic reviews helpful. Additional information about SRs can be found on the Cochrane website(opens in new tab/window). If you have any questions or observations, please feel free to comment below.

References

1. Meerpohl JJ, Herrle F, Reinders S, et al. Scientific value of systematic reviews: survey of editors of core clinical journals. PLoS One 2012;7(5):e35732. doi: 10.1371/journal.pone.0035732 [published Online First: 2012/05/09]

2. Higgins JPT, Green S, Cochrane Collaboration. Cochrane handbook for systematic reviews of interventions Version 5.1.0 [updated March 2011]. London: The Cochrane Collaboration,, 2011:1 online resource. doi: 10.1002/9780470712184

3. Chalmers I, Hedges LV, Cooper H. A brief history of research synthesis. Eval Health Prof 2002;25(1):12-37. doi: 10.1177/0163278702025001003 [published Online First: 2002/03/01]

4. O’Hagan EC, Matalon S, Riesenberg LA. Systematic reviews of the literature: a better way of addressing basic science controversies: American Physiological Society Bethesda, MD, 2018. doi: 10.1152/ajplung.00544.2017

5. Gilbody S, Wilson P, Watt I. Benefits and harms of direct to consumer advertising: a systematic review. BMJ Quality & Safety 2005;14(4):246-50. doi: 10.1136/qshc.2004.012781

6. Pullin AS, Stewart GB. Guidelines for systematic review in conservation and environmental management. Conservation biology 2006;20(6):1647-56. doi: 10.1111/j.1523-1739.2006.00485.x

7. Petticrew M. Systematic reviews from astronomy to zoology: myths and misconceptions. Bmj 2001;322(7278):98-101. doi: 10.1136/bmj.322.7278.98

8. Lind J. A treatise on the scurvy. In three parts. Containing an inquiry into the nature, causes, and cure, of that disease. London,: A. Millar 1753. doi:10.1136/bmj.330.7482.92-a

9. Clarke M, Chalmers I. Reflections on the history of systematic reviews. BMJ Evid Based Med 2018;23(4):121-22. doi: 10.1136/bmjebm-2018-110968 [published Online First: 2018/06/21]

10. Cochrane AL. Effectiveness and efficiency: random reflections on health services. London: Nuffield Provincial Hospitals Trust 1972. doi: 10.1017/cbo9781107256644

11. Santos CMdC, Pimenta CAdM, Nobre MRC. The PICO strategy for the research question construction and evidence search. Revista latino-americana de enfermagem 2007;15(3):508-11. doi: 10.1590/s0104-11692007000300023

12. Stewart L, Moher D, Shekelle P. Why prospective registration of systematic reviews makes sense. Syst Rev 2012;1:7. doi: 10.1186/2046-4053-1-7 [published Online First: 2012/05/17]

13. Mahood Q, Van Eerd D, Irvin E. Searching for grey literature for systematic reviews: challenges and benefits. Research synthesis methods 2014;5(3):221-34. doi: 10.1002/jrsm.1106

14. Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Bmj 2017;358:j4008. doi: 10.1136/bmj.j4008

15. Chandler J, Churchill R, Higgins J, et al. Methodological standards for the conduct of new Cochrane Intervention Reviews. The Cochrane Library 2013 doi: 10.4073/cpg.2016.3

16. Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Annals of internal medicine 2009;151(4):264-69. doi: 10.7326/0003-4819-151-4-200908180-00135

17. Eagly AH, Wood W. Using research syntheses to plan future research. The Handbook of Research Synthesis: Russell Sage Foundation 1994:485-500. doi: 10.1002/(SICI)1097-0258(19970330)16:6<713::AID-SIM430>3.0.CO;2-4

18. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Obstet Gynecol 2009;114(6):1341-5. doi: 10.1097/AOG.0b013e3181c3020d [published Online First: 2009/11/26]