Exploring Factors that Limit Data Quality on Vaccines

The goal of the World Health Organization’s (WHO) Expanded Programme on Immunization (EPI)(opens in new tab/window) is to ensure universal access to life-saving vaccines. Vaccines already save the lives of millions of children around the world. But progress in reaching even more people has stalled. One important factor "according to" researchers reporting in Elsevier’s journal Vaccine is poor quality or incomplete data available to health workers doing the vaccinations and to decision makers making essential calls about how to proceed.

“To run a proper expanded program on immunization, it's vital to have the data,” said Nargis Rahimi of the Shifo Foundation, Sweden. “Data is a catalyst for us to do the right things and to do them right.”

Data is also essential to ensuring accountability, transparency, and ownership, she says. Prior to their new study, only one literature review had taken a deep dive into data quality on immunization. It also hadn't explored in depth what the characteristics of data quality problems were.

To address these challenges, Rahimi and colleagues including Katherine Harrison, Karolinska Institutet, and M. Carolina Danavaro-Holliday, WHO Expanded Programme on Immunization, conducted a scoping review to explore the state of data quality on immunizations in low and middle income countries (LMIC). The researchers asked three questions:

1. What is the state of data quality on the Expanded Program on Immunization?

2. What are the factors related to low or high data quality?

3. What can be done to improve data quality?

For answers, the researchers searched four electronic databases to find all of the literature relevant to these issues through February 2018. Their search initially turned up hundreds of articles. But the researchers narrowed the list to just 21 that focused on vaccination, vaccination policy, and related topics. Some of the articles focused on a single country, including Zimbabwe, Nepal, and Nigeria. Others captured information on a longer list of countries.

The studies detailed widespread over-reporting of immunization coverages, with estimates ranging from 119% up to more than 200%. In addition to problems in accuracy, they also noted problems with data completeness, timeliness and consistency. They uncovered problems at many levels, including tools, governance, and people. The top recommendations for improving the quality of data focused on better mechanisms to monitor data quality and improved training and supervision of health workers.

To improve data quality, Rahimi and colleagues say it will be important to improve data collection tools, prioritize capacity building among health workers and decision makers, and strengthen governance. Reaching these goals will take a financial investment and commitment at all levels of the health system. At the same time, they note that any proposed solutions must be sustainable even without outside funding.

The new findings to identify the problems and barriers to high quality data on immunizations in low and middle income countries are just the beginning. But Rahimi’s team is confident that the motivations to improve data quality and get vaccines to more people are there."If one accepts that ‘what gets measured gets done’, the findings of this review should raise significant concerns amongst EPI programmes in LMIC,” the researchers write. “Many systemic factors contribute to poor quality immunization data, especially the challenges faced by health workers. Having identified the existence and nature of data quality problems, action is required. The valuable work of small-scale studies to address data quality, with mechanisms such as performance feedback, must be built upon. EPI’s well-documented successes, as well as the significant government and donor investment it attracts, should provide ample motivation to seek the high quality data required.”

A Conversation with Nargis Rahimi

I talked with Nargis Rahimi, corresponding author of the Atlas-winning paper, about the factors limiting data quality in the expanded programme on immunization and how those barriers might be overcome in low and middle-income countries.  Listen now.(opens in new tab/window)

What are the goals of the expanded program on immunizations? Many people know expanded program on immunization. This program basically makes sure that every person is protected from vaccine-preventable diseases.

Immunizations are obviously very important to public health, what got you thinking about issues of data quality in this context? At Shifo Foundation, we strive to make sure everyone gets life-saving vaccines and other essential health services. What we've realized is the data that can help organizations and health workers reach every person with vaccines is suboptimal. If I give you concrete examples: let’s say you are a health worker and the goal is to immunize every child in the community. You need to know how many children are in the community, how old they are and who has received vaccine, who’s coming for follow up and who's missing. So, this is vital information that nurses should possess to vaccinate children. What we've seen in many countries and especially low income countries is this information isn't easily available or the quality of data is poor, which means if a health worker doesn't know how many kids are in the community, she can't properly plan to deliver vaccine to all of them.

The same happens in decision making at all levels. If you are responsible for a big community on the district level, you need to know how many facilities deliver vaccines, which health facilities have enough vaccine and supplies, which have enough health workers. Are they doing outreach or not? You see, to run a proper expanded program on immunization, it's vital to have the reliable data. So, data is a catalyst for us to do the right things and to do them right. Also, these are fundamental aspects to ensure there is accountability, there is transparency, and proper ownership of the program. That's why we wanted to make sure we do this assessment and review the literature using a scoping review method and get answers to our questions.

How did you address this issue in your Atlas-winning paper? We have three main objectives in mind. We wanted to look at the state of data quality in low- and middle-income countries. We wanted to assess what factors cause low or high data quality and thirdly what could be done to improve data quality.

What does data quality mean? How is it measured? That's a great question because what we have realized after reviewing many papers was that there wasn’t really a consensus of what we mean by data quality. One of our recommendations is actually to define data quality in a way that could be accepted and used by everyone. For me personally, data quality comprises various aspects. Good quality data should be complete, consistent, reliable, timely. These are some of the ingredients which we can use to asses if data is of high quality or not. When looking at the literature we were using published articles and referring to how authors defined data quality.

Why is data quality on immunization poor? What are key barriers to better quality data? There were a number of factors behind poor data quality and when I refer to data quality we were looking at both the nominator and denominator. You can call it the vaccine coverage indicator. Some factors that are causing poor data quality we can group into specific chunks or structures and one of them is the tools. The tools used by health workers to collect and report the information. We refer to this as structural weaknesses in health information systems. If I can give you some examples from the field. There are tools to collect health information that can be challenging for health workers to fill in—for instance, if they are using multiple, bulky journals to collect data. The second factor was governance. The availability and implementation of policies to make sure data reporting, analysis, and interpretation -- all the processes for data management are in place. We've seen a lack of standardization in data collection tools within countries and sometimes among facilities. They use different types of processes. This is another factor. The third is connected with the people. A lot of articles highlight the detrimental effect of inadequate health worker training and capacity building in health workers in areas of data management. You may imagine when people do not understand why data is important and why it should be properly collected, reported, analyzed and used, you don't value data and give it the priority or importance [it needs]. This is another factor we've identified.

Where do you go from here? How do you hope this info can be put to work to improve things? What we are describing in the article are some of the ways the literature is suggesting next steps. One is based on improving the processes and strengthening the capacity of health workers to manage data properly. From what I’ve learned when working on the paper and my own experience is that there are four structures: 1) People, 2) Processes, 3) Tools, and 4) Governance. These should be strengthened simultaneously as they all have a big influence on data quality. When you strengthen the capacity of people to collect and use data, you also need to make sure health workers are given user-friendly, proper tools that reduce their administration. You need tools that don't add a lot of burden to the few health workers we have especially in low- and middle-income countries. We need health workers to use the best tools available. One also has to make sure health workers have the capacity and standard operating procedures to help them navigate in their daily jobs. The data collection happens as you provide the services and of course the governance, the structure, the policies available in every country, district and health system also need to be enabling for health workers. These are our findings and recommendations.

Is there anything else you want to say about the work? I would like to finish by saying that data is the greatest tool for decision making. So, I hope we will wake up one day to find out that data is really used. People use data in all spheres of their lives including to deliver lifesaving vaccines to everyone. So, once data use and data quality is strengthened I believe we will see a different world. And I hope that with being consciously aware and competent in health and other sectors, it will really help us to have transparent, accountable health and other systems to make sure especially that marginalized and underprivileged communities and societies that are being neglected or that are not having access to vital health services are not left behind. That’s why we need to use data. We need to work on strengthening data quality and its use to achieve all the goals including the SDG goals that we dream to achieve.

What’s next for you? At Shifo Foundation, we are working with Data for Action interventions to strengthen data use at health facilities at both the sub-national and national level. More personally, I started a company called Laprica.com to empower parents to make the right decisions using evidence-based information. This work would not have been possible without my experience at Shifo and as a parent. Parents have a great responsibility in their journey of raising children and usually are not equipped with the knowledge, tools, and strategies to support their child's development. Laprica offers evidence-based programs to parents across the globe.

Related resources

• ●

  Shifo Foundation(opens in new tab/window)

• ●

  The Expanded Programme on Immunization (WHO)(opens in new tab/window)

• ●

  Vaccines and Diseases (WHO)(opens in new tab/window)

About Vaccine

Vaccine is unique in publishing the highest quality science across all disciplines relevant to the field of vaccinology - all original article submissions across basic and clinical research, vaccine manufacturing, history, public policy, behavioral science and ethics, social sciences, safety, and many other related areas are welcomed.