IRD Global (Electronic Immunization Registry and Mobile-Based Conditional Cash Transfers to Increase Vaccination)

Published: August 2022

What is the program?

IRD Global (IRD) is a global-health focused organization that runs Zindagi Mehfooz (ZM), an electronic immunization registry, in Pakistan. The ZM platform can be used to deliver several programs, including mobile-based conditional cash transfers (mCCTs) to incentivize immunization, the focus of this page.1

Does it work?

The primary benefit of IRD’s mCCT and ZM program is reduction in mortality for vaccinated children up to age five. (more)

Other benefits of IRD’s program that we incorporate into our overall estimate of the program's impact are:

• Reduction in mortality beyond the age of five. Reductions in mortality from vaccine-preventable disease also occur for vaccinated people after the age of five.
• Development benefits. Early-life health improvements from vaccines might lead to increases in income when vaccinated children become adults.
• Consumption benefits. Mobile-based incentives received by caregivers may improve well-being through increased consumption. (more)

We have also attempted to account for:

• Additional benefits (e.g., morbidity effects, benefits of herd immunity) and negative or offsetting impacts (e.g., crowding out of IRD by other vaccination promotion programs, inflation). Rather than explicitly modeling these, we have applied percentage adjustments based on our best guesses. (more)
• Downside adjustments. We adjust for risk of wastage, the quality of monitoring and evaluation, and our level of confidence in funds being used for their intended purpose. (more)

What is the effect on child mortality?

We view reducing child mortality by improving the uptake of vaccines that prevent diseases that cause mortality as the primary benefit of IRD’s program.

IRD plans to run a mobile phone-based conditional cash transfer (mCCT) program in the seven highest-risk (characterized by low Penta-3 and Measles-1 vaccination rates) districts in Sindh Province, Pakistan.2 We model this by splitting the districts of Sindh into two groups: one with higher baseline vaccination rates and one with lower baseline vaccination rates. Our model incorporates the benefits of cash incentives only for the low-coverage group.

We estimate that IRD’s program will avert 0.7 child deaths per 1,000 infants eligible for the program in Sindh,3 based on the following factors:

• The effect of IRD’s mCCTs and ZM on vaccination rates. Our estimate is based primarily on an unpublished RCT of the effect of SMS reminders and mCCTs from IRD’s ZM platform in Korangi town in Sindh. Because the program that was implemented for the trial and the program IRD plans to implement at scale are not identical, we make several adjustments to this estimate to reflect our best guess of the effect of ZM and the mCCTs program IRD plans to implement at scale. (more)
• The prevalence of vaccine-preventable diseases in Sindh. We estimate that the probability of death from vaccine-preventable disease for unvaccinated children under 5 in Sindh is 1.4% in high-coverage districts and 1.8% in low-coverage districts. This estimate is based primarily on data from the Institute for Health Metrics and Evaluation (IHME) on causes of death for children under 5. We estimate the probability of death among the unvaccinated using vaccination coverage at the time of the IHME data and estimates of vaccine efficacy. We adjust the probability of death for high-coverage and low-coverage districts, because we expect areas with lower baseline vaccination rates to have worse health outcomes for reasons beyond vaccination coverage. (more)
• The effect of increased vaccination on prevalence of vaccine-preventable diseases. We estimate that vaccines reduce recipients’ risk of vaccine-preventable disease by 83%. This is based on meta-analyses of vaccines' effects on vaccine-preventable diseases and adjustments for "non-specific" effects on mortality from diseases other than those directly targeted by vaccines. We assume this is constant across districts. (more)

Is the program effective at increasing vaccination rates?

The majority of the benefits of IRD’s program come from increasing the number of children vaccinated, which leads to reductions in child deaths, as well as reductions in deaths at older ages and development benefits. We model IRD as increasing vaccination rates through both the ZM platform, which includes SMS reminders for immunization, and the addition of mCCTs to that platform.

To estimate IRD’s program's effect on vaccination rates, we rely heavily on an unpublished randomized controlled trial of the effect of SMS reminders and mCCTs, provided through IRD’s ZM platform, in Korangi town in Sindh. (more)

We extrapolate from this trial to generate a best guess for the effect of IRD’s program during the implementation period in Sindh. We account for differences in baseline coverage, incentive size, and several additional factors.

• We estimate that ZM, which provides SMS reminders and other components, increases coverage by 2-3 percentage points. We view this as broadly consistent with other literature on the effect of SMS reminders and on immunization registries, though we have not done a thorough comparison of effect sizes from the trial to this broader literature. (more)
• We estimate that mCCTs increase coverage by an additional 9 percentage points in low-coverage districts (mCCTs are not provided in high-coverage districts.) We also benchmark against a randomized controlled trial of conditional cash transfers in North West Nigeria by New Incentives, which finds a similar effect. We view these findings as broadly consistent with other literature on the effect of mCCTs on vaccination rates, though we have not done a thorough comparison of effect sizes from the trial to this broader literature. (more)

Overall, we view the strength of the evidence on the effect of ZM and mCCTs on vaccination rates as moderate. While we view the evidence for the effect of incentives and SMS reminders for immunization as strong, we’re highly uncertain about the appropriate effect size to use for the effect of IRD’s program at scale in Sindh, and we make several speculative adjustments to account for differences between the evidence we use and implementation in Sindh.

Korangi trial

IRD recently completed a three-year randomized controlled trial, with partial funding from a grant recommended by GiveWell, on the effects of mobile-phone based incentives in Korangi town in Sindh province. We have reviewed an initial draft of this study (the Korangi trial).4 The results are unpublished at this time. We use the trial to inform our best guess of the effect of both ZM and mCCTs on vaccination rates.

The trial was individually randomized and included seven arms: five arms that received both incentives and SMS reminders, one arm that only received SMS reminders, and a control arm that did not receive SMS reminders or incentives. Incentives were distributed via mobile top-ups and, in one arm, Easypaisa (an electronic cash transfer service). Beyond method of payment, the five arms receiving incentives were further separated by incentive amount (low versus high) and incentive schedule (flat rate payments versus increasing payments), and all five arms were subdivided by incentive design (lottery-based versus guaranteed payments).5 We focus on the incentive arms providing low incentives via mobile top-up (roughly $0.80 per visit), high incentives via mobile top-up (roughly $2.50 per visit), SMS-only, and control.6

Infants were enrolled at either the BCG, Penta-1 or Penta-2 visits.7 The trial reported effects on Penta-3, Measles-1, and Measles-2 vaccines. We have also reviewed results on Penta-1 and Penta-2.

Overall, we view this evidence as moderate quality.

Effect of ZM on vaccination

IRD enrolls caregivers and infants and tracks whether they have received their necessary vaccinations. Once enrolled, ZM provides caregivers with SMS reminders and other programs.8

We looked at the results from the unpublished Korangi trial and made several adjustments to the estimate of effect size. Note that since the results of the Korangi trial are unpublished, we are not able to link to our full analysis of the effect size of ZM on vaccination, including the adjustments discussed below.

• Baseline coverage. The baseline coverage rates we use in our model are lower than those observed in the Korangi trial. We extrapolate effects from the trial using the percentage reduction in children not receiving each vaccine. Because baseline vaccination coverage is lower than in the trial, we guess that the effect of ZM in terms of percentage points will be higher during the implementation period than during the trial.
• Effects on those enrolled vs. unenrolled in ZM. We assume the effect of SMS reminders and other ZM features only accrues to caregivers enrolled in ZM. As a result, we benchmark the effect based on percent unvaccinated among children enrolled in ZM.9
• BCG. We think it's highly unlikely SMS reminders would have a substantial effect on BCG coverage, since BCG is delivered at the first enrollment opportunity and few caregivers are likely to receive reminders for BCG as a result.
• Additional components beyond SMS reminders. We adjust the effect upward to account for additional components that ZM provides beyond SMS reminders that we guess also increase vaccination coverage. These include attendance and geographic information system (GIS) tracking of vaccinators and predictive analytics. We’re highly uncertain about the appropriate size of this adjustment.
• Implementation at scale. We make a small downward adjustment for weaker implementation at scale. While we guess that implementation quality was higher in the trial, we are not very concerned about this, since the ZM platform is operating at scale in Sindh already.
• Effect of COVID-19 on responsiveness to SMS reminders. We make a downward adjustment for concerns that effects of SMS reminders might be smaller as a result of coronavirus disruptions, which may present additional barriers to vaccination.
• Study quality. While the trial was pre-registered and there are no obvious flaws (no obvious baseline imbalance, low attrition), the study is unpublished and has not been vetted externally. As a result, we make a downward adjustment.

Our best guess after applying these adjustments is that ZM increases vaccine coverage by 2 percentage points in high-coverage districts and 3 percentage points in low-coverage districts. (This percentage point increase is a weighted average of increases in vaccination rates across each visit, weighted by the percentage of vaccine-preventable deaths that can be averted through vaccines delivered at each visit.)

We also undertook a superficial review of external evidence on the effects of immunization registries, which generally include SMS reminders and other services, as well as evidence on effects of SMS reminders.

• Our shallow review of the effect of immunization registries finds effect sizes ranging from 0-22 percentage points in increased coverage across the vaccine sequence. This corresponds to a weighted effect of 9 percentage points, which exceeds our best guess for effect size in both high- and low-coverage districts.10
• Our page on SMS reminders for vaccinations provides a shallow review of the literature and concludes: “We are aware of six randomized controlled trials (RCTs) of SMS reminders for vaccination, five in sub-Saharan Africa and one in Guatemala. […] There is some limited evidence that SMS messages can notably increase vaccination rates in some sub-Saharan populations that already access vaccination services or clinic delivery and have moderate baseline vaccination rates. Of two studies with high baseline vaccination rates, both in Kenya, one found a notable effect on vaccination rates and the other found no effect. The four RCTs that found an effect of SMS reminders on vaccination rates (8.7 - 20 percentage point increase in third visit at age 14-16 weeks) measured effects only on early infant vaccination, and not on measles vaccination at nine months.”
• A recent meta-analysis, which we have not vetted, finds SMS reminders increase vaccination coverage by 16% across countries, 39% in low-income countries, and 19% in lower middle-income countries.11

We view these studies as supporting the effect of SMS reminders and other services provided by immunization registries on vaccination rates. However, we have not reviewed this work in depth and have not undertaken a thorough comparison, based on baseline coverage and other factors, in order to translate them directly into the effect sizes we estimate.

Overall, we are fairly confident immunization registries that include SMS reminders increase vaccination rates, but we are highly uncertain about the magnitude of the effect.

Effect of mCCTs on vaccination

The ZM platform also permits incorporating mobile-based cash incentives on top of SMS reminders and other services.

Our best guess of the effect of mCCTs takes the following factors into account. Note that since the results of the Korangi trial are unpublished, we are not able to link to our full analysis of the effect size of mCCTs on vaccination, including the adjustments discussed below.

• Baseline coverage. The baseline coverage rates we use in our model are lower than those observed in the Korangi trial. We extrapolate effects from the trial using the percentage reduction in children not receiving each vaccine. Because baseline vaccination coverage is lower than in the trial, we guess that the effect of mCCTs in terms of percentage points will be higher during the implementation period than during the trial.
• Effects on those enrolled vs. unenrolled in ZM. We assume the effect of SMS reminders and ZM features only accrues to caregivers enrolled in ZM. However, we assume effects of mCCTs are similar for those enrolled vs. unenrolled. While those who are not enrolled initially may face stronger barriers to vaccination (e.g., they are more isolated from health clinics), there are reasons they may be more responsive (e.g., they have lower income and are more affected by a monetary incentive).12 As a result, we benchmark the effect based on percent unvaccinated among all children in Sindh, not just those enrolled in ZM.
• BCG. We extrapolate effects on BCG using the effect on Penta-1. The trial was not designed to detect an effect of mCCTs on BCG, since participants were enrolled in the program and learned about the availability of mCCTs at or after the BCG visit. We anticipate that when the program is implemented, individuals would be aware of the mCCT program prior to receiving the BCG vaccine. As a result, the availability of the mCCT program would increase BCG coverage.13
• Incentive sizes. We use an incentive size of $1.26 (approximately 200 PKR) per visit in low-coverage districts. The Korangi trial uses similar incentive sizes (roughly $0.80 for low incentive groups and roughly $2.50 for higher incentive groups), but to extrapolate, we calculate percentage reduction in unvaccinated population per dollar and multiply that by dollars of incentive.

We make (or consider) the following adjustments to our best guess of the effect of mCCTs.

• Spending power in Pakistan today vs. during trials. A U.S. dollar today can purchase more goods in Pakistan than it could at the time of the Korangi trial, driven in large part by a US dollar purchasing more Pakistani rupees today than in recent years. We make an upward adjustment for this.
• Better advertising of incentive at scale. In the Korangi trial, IRD did not communicate broadly about the incentives. During the implementation period, IRD plans to include broader communication about the incentive program.14 We guess that doing so would increase the effect of the program and include an upward adjustment for this.15
• Implementation at scale. We guess implementation quality would be lower outside of the trial context and when implemented at a larger scale and include a downward adjustment for this.16
• Effect of COVID-19 in responsiveness to mCCTs. COVID-19 disruptions could weaken the effect of mCCTs by causing disruptions to vaccine supply (e.g., vaccinators not coming into work, vaccine supply chain disruptions) or vaccine demand (e.g., discouraging individuals from coming to clinics out of fear of COVID-19 infection). However, it is also possible that COVID-19 increases responsiveness to mCCTs (e.g., because individuals have lower income and therefore have more to gain from mCCTs). We guess that these considerations balance each other out, so we do not include an adjustment for this.17
• Spillovers. The trial was individually randomized, and we would guess there would be some spillovers in treatment and control (i.e., caregivers who are in the cash transfer group share the information with others, who bring infants in for vaccinations thinking they too will get transfers). This would bias the effect downward. We include an upward adjustment for this.
• Study quality. While the trial was pre-registered and there are no obvious flaws (no obvious baseline imbalance, low attrition), the study is unpublished and has not been vetted externally. As a result, we make a downward adjustment.

Our best guess with these adjustments and only putting weight on the Korangi trial is that mCCTs increase coverage by 8 percentage points in low-coverage districts. (This percentage point increase is a weighted average of increases in vaccination rates across each visit, weighted by percent of deaths that can be averted through vaccines delivered at each visit.)

We also benchmark effects of mCCTs to effects observed in a randomized controlled trial of New Incentives, a conditional cash transfer program in Nigeria.

• We compare to this trial because (1) it is a program we are familiar with, so we are able to more easily draw an apples-to-apples comparison, taking into account differences in programs and setting, (2) based on a shallow review, we concluded the New Incentives results were broadly consistent with the larger literature on the effect of conditional cash transfers on vaccination, and (3) New Incentives includes the type of broad communication about the incentives that we expect for mCCTs when implemented across Sindh.
• The New Incentives trial finds cash transfers of $1.40 per visit for BCG, Penta-1, Penta-2, and Penta-3 visits and $5.50 for the Measles-1 visit increase coverage by 14-27 percentage points across the sequence, which corresponds to a reduction of 39%-50% in children not receiving vaccines across the sequence.
• To compare these to the mCCTs results, we make a downward adjustment for the assumption that mobile transfers have a smaller effect than direct cash transfers, an upward adjustment for higher spending power per USD in Pakistan today relative to the New Incentives trial, and a downward adjustment due to COVID-19 disruptions. We also account for differences in baseline vaccination coverage and dollar amounts used for incentives by estimating reduction in percentage unvaccinated per dollar for each vaccine.

We combine these effects by putting 20% weight on the New Incentives results and 80% weight on the Korangi results. Our best guess is a 9 percentage point increase in coverage in low-coverage districts.

We have also conducted a very shallow review of the literature on mobile conditional cash transfers specifically. We view it as broadly supporting the results of the Korangi trial, but we have not rigorously compared effect sizes (taking into account purchasing power-adjusted incentive size, baseline vaccination rates, and additional specifics of the intervention and context).18

Key uncertainties and judgment calls in our analysis:

• What will vaccination coverage be in the near term, given disruptions due to COVID-19? Cost-effectiveness depends heavily on both average coverage rates and variation in rates across Sindh (due to targeting). We are using 2020-2021 coverage as our best guess of coverage in the next 1-3 years, but we’re highly uncertain about this assumption, given potential disruptions due to COVID-19. IRD said it expects coverage rates to be temporarily higher in the near term due to expanded outreach activities (EOAs) and that vaccinations that occur as a result of these EOAs are less responsive to incentives.19 If coverage rates do decline in the coming months, this would increase cost-effectiveness because (a) lower baseline coverage means providing incentives to fewer "always-takers," leading to lower costs, (b) a higher share of unvaccinated infants could be affected by incentives, leading to a higher effect on coverage, and (c) a lower share of infants receiving vaccinations through EOAs, which may be less responsive to incentives, leading to a higher effect on coverage.20 We’re uncertain how much weight to put on these claims.
• What is the appropriate adjustment to use for differences in vaccination coverage between those with and without a cell phone? We adjust coverage upwards slightly because we guess the subset of those individuals in Sindh who have access to a cell phone and are therefore eligible to receive mCCTs have slightly higher baseline coverage, which lowers the effect size. Because we estimate close to all caregivers have a cell phone or are able to access one (see below), this adjustment is small.
• How reliable are ZM estimates? We have not done a thorough review of immunization data from ZM to understand what IRD does to ensure accuracy. For example, there may be an incentive for vaccinators to over-report coverage in order to hit targets set by authorities.
• Are there vaccinations missing from ZM data? Data from IRD are based on infants vaccinated through ZM divided by estimated live births. We assume few vaccinations are missed by ZM (e.g., due to vaccinations being captured on paper records, infants vaccinated at clinics where ZM is not active, and vaccinations occurring through campaigns), based on conversations with IRD.21 If these data missed a significant share of routine immunizations, then we would be underestimating baseline coverage.
• What are the effects of mCCTs on vaccines earlier in the sequence (BCG, Penta-1 and Penta-2)? These drive cost-effectiveness because they account for the largest share of vaccine-preventable disease. However, the trial was not designed to detect an effect on BCG, Penta-1 and Penta-2 were not primary outcomes in the analysis, and measured effects on those vaccines have less statistical power than the primary outcomes (Penta-3, Measles-1 and Measles-2).
• We model the program's effect on vaccination rates as its effect on the proportion of people who are unvaccinated, rather than as its effect on the proportion of people who are vaccinated or as a percentage point reduction. We use this approach because it seems reasonable to assume that the program's effects would be among those who would be unvaccinated in the absence of the program, because these would be the individuals that could respond to the intervention. There is also some evidence in favor of this approach.22 However, if we based our estimates on percentage point reductions observed in the Korangi trial or percentage increases in the proportion vaccinated, our estimated effect sizes would be much smaller.23
• Are areas or individuals with low baseline vaccination rates less responsive to mCCTs? It is possible these individuals face the highest constraints on vaccination (e.g., strong aversion to vaccination, limited access to clinics) so they would be less responsive. On the other hand, if these are the poorest individuals, they may be more responsive to incentives. We assume these effects cancel each other out, but if one is larger than the other, our best guess for effect size could be an over- or under-estimate.
• Should we expect the same effect on indirectly as directly incentivized vaccines? In our model, we assume effects on Penta-1, for example, are the same for other vaccines provided at the same time (e.g., PCV-1 and Rota-1). If effects are weaker for these vaccines, cost-effectiveness would be lower.24
• How would our best guess of the effect size change if we undertook a more thorough review of the literature on electronic immunization registries, SMS reminders, and mobile-based conditional cash transfers? We have done a superficial review of this literature, and it is possible that a more thorough review could change our best guess of the effect size.
• What are the appropriate adjustments to make to the evidence from the Korangi trial? We make several speculative adjustments, and different values for these parameters could change our estimate substantially.

What is the prevalence of vaccine-preventable disease and mortality in areas where IRD will operate?

Our best guess is that the probability of death for an unvaccinated child under five from a disease that can be prevented by one of the vaccines incentivized by IRD’s program is 1.4% in high-coverage districts and 1.8% in low-coverage districts.

We rely primarily on IHME data on child mortality from vaccine-preventable diseases in Pakistan. We adjust for the following:

• Proportion of deaths from vaccine-preventable diseases caused by specific pathogens (i.e., disease etiology). We follow a similar approach as we did for New Incentives, which is described here. IRD-specific calculations are here.
• Deaths that occur before vaccines are administered. We follow a similar approach as we did for New Incentives, which is described here. IRD-specific calculations are here.
• Vaccination coverage. Because the IHME data include children who are vaccinated and unvaccinated, we estimate the probability of death for unvaccinated children specifically. We follow a similar approach as we did for New Incentives, which is described here. IRD-specific calculations are here.
• Subnational adjustments. We guess that the mortality rate is higher in low-coverage districts for unvaccinated children due to worse health overall. We have not thoroughly investigated this parameter but apply a rough adjustment that is broadly in line with what we assumed for New Incentives. Our approach for New Incentives is described here. IRD-specific estimates are here.

Key uncertainties

• How reliable are cause of death estimates from IHME and other sources we use for etiology adjustments? Our cost-effectiveness analysis relies heavily on the proportion of deaths that are due to vaccine-preventable diseases. We have high uncertainty about these estimates.25
• What are the appropriate adjustments for probability of death from different diseases in Sindh and between high- and low-coverage areas? Our current assumptions are speculative, and it’s possible that further work could change our best guess.26
• Should we adopt a different moral weight for diseases prevented by vaccination in IRD's case? For example, if IRD is averting more neonatal deaths via BCG vaccine against tuberculosis than New Incentives' program does, we may want to apply a lower moral weight, since our moral weights place less value on neonatal deaths.27
• Should we include estimates of how probability of death for vaccine-preventable diseases and vaccination coverage will change in the next 5 years? We include a rough downward adjustment for vaccination coverage increasing and probability of death from vaccine-preventable disease decreasing in our exclusion tab,28 but it’s possible that using forecasts of vaccine coverage and vaccine-preventable disease could update our estimates.

Does increased vaccination lead to reductions in disease?

The extent to which IRD’s program leads to reductions in mortality depends on the extent to which the vaccines lead to reductions in the diseases those vaccines target. We estimate that the vaccines promoted, either through ZM or mCCTs, by IRD reduce recipients' likelihood of contracting the diseases targeted by 83%.

This estimate is based primarily on meta-analyses of the effects of vaccines provided as part of routine immunization in Pakistan. Our best guess is that these vaccines are 71% effective against the diseases included in our probability of death estimates above. A description of these meta-analyses are in our report on New Incentives here.29

We apply adjustments to this initial estimate:

• All-cause mortality effects. We apply an upward adjustment to account for observed "all-cause" mortality effects — i.e. it appears that vaccines have mortality benefits beyond their impacts on the targeted diseases. We follow a similar approach as we did for New Incentives, which is described here. IRD-specific calculations are here.
• Concerns about vaccine efficacy in low-income countries. We make downward adjustments due to concerns that vaccine efficacy in low-income countries is lower than observed in randomized controlled trials estimating vaccine efficacy. We had substantial concerns about vaccine efficacy for New Incentives, due to concerns about biomarkers results suggesting very low measles vaccine efficacy. We do not have evidence of lower efficacy in Pakistan but still make a slight downward adjustment because we guess some of the same concerns we described in the New Incentives case apply for Pakistan as well. IRD-specific estimates are here.
• Imperfect coverage. We include a 95% coverage adjustment to account for less than 100% coverage of all doses of vaccines in trials. This is a rough guess.

We have a number of uncertainties about these estimates. Because vaccine efficacy figures are used to calculate not just effect of increased vaccination but also to derive probability of death among unvaccinated (see above), the model is particularly sensitive to these parameters.

• We have not thoroughly vetted the meta-analyses we use to quantify vaccine efficacy, and it's possible that further review would lead us to make adjustments to these values.
• We assume vaccine efficacy is maintained through childhood (up to the age of five) and then begins to decline. We have not thoroughly investigated this assumption for specific vaccines.
• We are highly uncertain whether the sizes of our adjustments for concerns about vaccine coverage, vaccine efficacy, and non-specific effects are appropriate.
• We've chosen to use a relatively simple cost-effectiveness model of the benefits of increased vaccination compared to some other models used in the literature. It is possible our model leaves out key features or additional positive or negative effects that would substantially affect our cost-effectiveness estimate (e.g., disease transmission dynamics, herd immunity). We incorporate some such factors in the "Adjustments for effects excluded from our core model" section of our cost-effectiveness analysis, but only as very rough guesses.

What are the benefits beyond child mortality?

In addition to effects on child mortality, we also model benefits of IRD’s program occurring through:

• Reductions in mortality from vaccine-preventable disease in vaccinated people after the age of five. Risk of vaccine-preventable diseases also occurs at ages beyond five. Vaccines may confer immunity beyond the age of five, reducing risk of mortality as vaccinated infants age. We follow a similar approach as we did for New Incentives, which is described here. IRD-specific calculations are here.
• "Development benefits," i.e., increases in income later in life due to early-life health improvements. We follow a similar approach as we did for New Incentives, which is described here. We downweight development benefits to account for lower probability of death in Pakistan, relative to the New Incentives context. IRD-specific calculations are here.
• Consumption benefits from the mobile transfers themselves. We follow a similar approach as we did for New Incentives, which is described here. We downweight consumption benefits to account for mobile top-ups having a lower effect on consumption than cash. IRD-specific calculations are here.

Additional benefits and negative or offsetting impacts

There are several benefits, as well as negative or offsetting impacts, of IRD’s program that we have not modeled explicitly in the main CEA but that we do incorporate into our final cost-effectiveness estimate as rough guesses. Rather than explicitly modeling these, we have applied percentage adjustments based on our best guesses. We use similar values as we did for New Incentives, which are described here. IRD-specific calculations are here.

Downside adjustments

We also adjust for risk of wastage, the quality of monitoring and evaluation, and our level of confidence in funds being used for their intended purpose. We use similar values as we did for New Incentives, which are described here. IRD-specific calculations are here.

What do you get for your dollar?

Cost per child immunized

We estimate that the cost per additional child vaccinated is $73 in high-coverage districts and $112 in low-coverage districts.30 This includes costs to IRD, as well as costs to the government and Gavi. We also account for the probability that IRD’s funding may cause other actors to shift funds from a less cost-effective use to a more cost-effective use or from a more cost-effective use to a less cost-effective use.

Our approach

Our estimate of the total costs of the program includes:

• All costs paid by IRD to run ZM and mCCTs.
• Costs paid by other donors that would not have been incurred in the absence of IRD’s program. We would guess those are mostly shouldered by the government and Gavi. These actors incur costs for each additional child vaccinated as a result of ZM and mCCTs that would not have been vaccinated if these programs did not exist.

To estimate costs paid by IRD to run ZM and mCCTs, we start with total costs to run ZM and the mCCTs program, excluding the costs of incentives. We divide this by the number of infants who are eligible to receive incentives (i.e., with caregivers who are able to access a cell phone). This number is calculated by multiplying the birth cohort in Sindh province by percent that are eligible.31

We then add costs of incentives. This includes both the incentive amount and transaction costs. We make an upward adjustment to costs to account for fraud (e.g., individuals receiving transfers who are not bringing in infants to be vaccinated).32

To estimate costs to the government and Gavi, we rely on estimates of immunization costs from the World Health Organization (WHO) and divide by number of infants vaccinated in order to estimate cost per child vaccinated.33

We also account for cases where we believe the charity's funds have caused these other actors to shift funds from a less cost-effective use to a more cost-effective use ("leverage") or from a more cost-effective use to a less cost-effective use ("funging"). We consider three sources of leverage and funging:34

• By increasing the number of infants receiving vaccines, IRD’s program causes the government and Gavi to spend more on vaccination. We assume that this additional spending is more cost-effectively used to increase vaccination rates than what it would have been spent on in the counterfactual. This increases cost-effectiveness.
• There is some chance another donor or the government would fund ZM and mCCTs in the absence of GiveWell’s recommendation. We assume that spending on IRD’s program would be more cost-effective than the counterfactual opportunity that a donor or the government would fund. This decreases cost-effectiveness.
• There is also some chance another donor or the government would fund ZM alone in the absence of GiveWell’s recommendation. We assume that spending on ZM alone would be more cost-effective than the counterfactual opportunity that the donor or the government would fund. This decreases cost-effectiveness.

Key uncertainties:

• How likely is it that the government or another donor would fund ZM or both ZM and mCCTs? We estimate there is a 10% chance the government would fund the equivalent of three years of ZM.35 We think it’s even more unlikely the government would fund mCCTs in the next three years. We estimate there is a 5% chance another funder would support ZM and mCCTs over the three-year period.36 However, in both cases, we’re uncertain about the appropriate probabilities to use.37
• What are the appropriate adjustments to use for fraud?
• How much does it cost the government and Gavi to provide vaccines? We have used data from WHO and made adjustments to account for our understanding that Gavi is planning to enter an “accelerating transitioning phase” that would require the government to bear a larger share of costs and eventually phase out Gavi support, but have not vetted this parameter in depth.38
• How confident should we be in budgeted costs? We traditionally rely on actuals from previous years. However, since this is a new program, we need to rely on budgeted costs provided by IRD.
• Based on conversations with IRD, we think it’s more likely (50%) that the Sindh government would be willing to take on the costs of the ZM program at the end of the 3-year period and that this probability would be increased by our recommending funding of mCCTs.39 We have not incorporated the effect of increasing the likelihood of the government funding ZM beyond the grant period in our leverage and funging adjustments.

Cost effectiveness

Our best guess is the program is roughly eight times as cost-effective as unconditional cash transfers from GiveDirectly, which we use as a benchmark for comparing the cost-effectiveness of different programs.

To estimate cost-effectiveness, we use the benefits and costs to IRD and other actors described above. For full details of our cost-effectiveness analysis, see this spreadsheet.

Note that there are limitations to this kind of cost-effectiveness analysis, and we believe that cost-effectiveness estimates such as these should not be taken literally, due to the significant uncertainty around them. We provide these estimates (a) for comparative purposes and (b) because working on them helps us ensure that we are thinking through as many of the relevant issues as possible.

Sources

Document	Source
Chernozhukov et al. 2018 (working paper)	Source (archive)
Eze et al. 2021	Source (archive)
GiveWell, Electronic immunization registry review, 2021	Source
GiveWell, Electronic immunization registry studies and effect sizes, 2021	Source
GiveWell, Intervention report for New Incentives (Conditional Cash Transfers to Increase Infant Vaccination), 2021	Source
GiveWell, Korangi trial preliminary results calculations, 2021 (unpublished)	Unpublished
GiveWell, Main IRD CEA, 2021	Source
GiveWell, Northern Nigeria mortality adjustment, 2021	Source
GiveWell, Probability of death sense checks, 2021	Source
GiveWell, Shallow review of mCCTs for vaccination, 2021	Source
GiveWell's non-verbatim summary of a conversation with IRD, August 13, 2021	Source
GiveWell's non-verbatim summary of a conversation with IRD, August 17, 2021	Source
GiveWell's non-verbatim summary of a conversation with IRD, August 31, 2021	Source
GiveWell's non-verbatim summary of a conversation with IRD, July 13, 2021	Source
GiveWell's non-verbatim summary of a conversation with IRD, June 29, 2021	Source
GiveWell's non-verbatim summary of a conversation with IRD, September 14, 2021	Source
IRD, Coverage estimates by EPI targets, 2021 (unpublished)	Unpublished
IRD, Korangi trial methods, 2021 (unpublished)	Unpublished
IRD, Korangi trial results for Penta-1 and Penta-2, 2021 (unpublished)	Unpublished
IRD, Korangi trial results, 2021 (unpublished)	Unpublished
IRD, Proposal for scale-up of mCCT in Sindh (working draft), September 22, 2021	Source
IRD, Zindagi Mehfooz overview, 2021	Source
Lamberti et al. 2016	Source

• 1

  See the "What is the program?" section of our write-up on the grant to IRD Global here.

• 2

  “The mCCTs Sindh project will provide conditional cash transfers to parents/caregivers of children between the ages of 0-23 months residing in the 7 high-risk districts of Sindh province, for completing each of the six EPI recommended immunization visits [...] The high-risk districts are categorized based on the 20th percentile of crude penta-3 and crude measles-1 vaccines district coverage rates (Figure 2). There are 6 districts (20.0%) below the 20th percentile for penta-3 vaccine (coverage of 55.86%) which include Hyderabad, Karachi East, Karachi West, Karachi Central, Jacobabad, and Sujawal. Additionally, there are 6 districts (20.0%) below the 20th percentile for measles-1 vaccine (coverage of 40.87%) including Karachi Central, Karachi East, Karachi West, Jacobabad, Kambar, and Sujawal. The 7 high-risk districts below the 20th percentile for crude penta-3 and measles-1 coverage are, therefore: Karachi Central, Karachi East, Karachi West, Hyderabad, Sujawal, Jacobabad, and Kambar.” IRD, Proposal for scale-up of mCCT in Sindh (working draft), September 22, 2021, Pgs. 5-7.

• 3

  Take the sum of these cells in our cost-effectiveness analysis.

• 4

  Study methodology and results are unpublished at this time.

• 5

  IRD, Korangi trial methods, 2021 (unpublished)
  Because the five incentive arms were subdivided based on lottery status, there were technically 12 arms in the trial:

  1. Easypaisa Flat Rate With Lottery
  2. Easypaisa Flat Rate Without Lottery
  3. HighIncentive Flat Rate With Lottery
  4. HighIncentive Flat Rate Without Lottery
  5. HighIncentive Sharp Rate With Lottery
  6. High Incentive Sharp Rate Without Lottery
  7. Low Incentive Flat Rate With Lottery
  8. Low Incentive Flat Rate Without Lottery
  9. Low Incentive Sharp Rate With Lottery
  10. Low Incentive Sharp Rate Without Lottery
  11. SMS
  12. Control

• 6

  We pool across “flat rate” arms (which provided the same incentive amount per visit) and “sharp rate” arms (which provided lower incentives for BCG, Penta-1, Penta-2, and Penta-3 visits and higher incentives for Measles-1 and Measles-2 visits). We also pool across lottery vs. non-lottery arms. Since the program will include non-lottery flat rate incentives, these arms are the most relevant. Focusing on only those arms makes only a small difference in effect sizes, and we view it as discarding extra supporting evidence for the effect of incentives.

• 7

  IRD, Korangi trial methods, 2021 (unpublished)

• 8

  See IRD, Zindagi Mehfooz overview, 2021, slide 6.

• 9

  Based on data we received from IRD, we expect approximately 92% of the annual birth cohort in Sindh is enrolled in ZM. IRD, Coverage estimates by EPI targets, 2021 (unpublished)

  • 1,510,179 (number of children enrolled) / 1,638,388 (estimated birth cohort) = 92% (% of Sindh birth cohort enrolled in ZM)
  • % of birth cohort that received a Penta-1 vaccination: 84.6%
  • % of children enrolled in ZM that received a Penta-1 vaccination: 91.7%

  Because coverage rates are higher conditional on enrollment in ZM, this leads to lower effect size in percentage points than assuming these effects occur for all individuals in Sindh, including those not enrolled in ZM.

• 10

  The findings are summarized in this write-up. Best guesses on effect sizes come from this spreadsheet.

• 11

  Eze et al. 2021, Table 2. Pooled RR 1.16 (95% CI 1.10-1.21) for all studies (19). Pooled RR 1.39 (95% CI 1.19-1.62) for low-income countries (1 study) and pooled RR 1.19 (95% CI 1.13-1.26) for lower middle-income countries (14 studies).

• 12

  We do exclude from the effect those without cell phone access.

  “Since zero-dose children are of low socioeconomic status, mCCTs could be particularly effective at incentivizing immunizations among this group, and also may have secondary impacts across other health indicators—by bringing otherwise isolated and vulnerable children into contact with the health system. However, there could also be challenges in reaching these children (e.g., lack of access to phones).” GiveWell's non-verbatim summary of a conversation with IRD, July 13, 2021.

• 13

  Caregivers receive BCG close to birth, but even if they deliver at a facility, they may need to return to the health facility to receive it. “Challenges with coverage of Bacillus Calmette–Guérin (BCG) vaccine. Although BCG is meant to be administered when a child is born, health facilities in Sindh typically administer the vaccine on fixed days to avoid wasting doses, multiple of which are contained in one BCG vial. If a child is born on a day when BCG is not being administered, caregivers are told to return for the vaccination, although only some return. IRD expects mCCTs to be effective in incentivizing caregivers to return for BCG vaccination.” GiveWell's non-verbatim summary of a conversation with IRD, August 13, 2021

• 14

  See IRD, Proposal for scale-up of mCCT in Sindh (working draft), September 22, 2021, section 3: Communications and Community Engagement Strategy.

• 15

  “The importance of communications to the effectiveness of mCCTs. IRD's experience with tuberculosis treatment programs suggests that simple messaging through billboards and television advertising can substantially increase program reach and ultimately the number of individuals that seek care.” GiveWell's non-verbatim summary of a conversation with IRD, August 31, 2021

  “IRD's expectation that the effect size of its mCCT program at scale would be higher than what was observed in the randomized controlled trial (RCT) of the program, due to a broader communications strategy including widespread advertisements. In the RCT, only individuals visiting health centers were made aware of the program, although there was also likely some small-scale spread through word of mouth.” GiveWell's non-verbatim summary of a conversation with IRD, August 13, 2021

  “The large potential impact of a communications strategy on the cost-effectiveness of mCCTs. A study in Haryana, India, for example, found that identifying and targeting messaging to community leaders significantly increased the impact of mCCTs. Initially, the communications strategy for IRD's mCCT program will only include infrequent text messages, although different strategies (e.g., more frequent communications, television advertising) could be rolled out and tested at a later point in time.” GiveWell's non-verbatim summary of a conversation with IRD, September 14, 2021

• 16

  For example, IRD reports fewer management personnel for IRD’s mCCT program at scale, compared to the RCT. “Fewer management personnel for IRD's mCCT program at scale, compared to in the RCT of the program. Due to the smaller scale of the RCT, IRD was able to have one field worker at each health center included in the study to coordinate activities, engage with partners and government officials, and conduct monitoring and evaluation. At scale, it will only be feasible to have one district immunization officer (DIO) per district to fulfill this role, although DIOs are senior-level personnel.” GiveWell's non-verbatim summary of a conversation with IRD, August 31, 2021

• 17

  “Reasons that the COVID-19 pandemic could increase the effectiveness of mCCTs. The majority of the population in Pakistan is of middle and low socioeconomic status, and has had less opportunity to earn income and more free time available during COVID-19 lockdowns. As a result, mCCTs could be particularly effective.” GiveWell's non-verbatim summary of a conversation with IRD, August 13, 2021

• 18

  More in this write-up.

• 19
  • "Increased vaccination coverage observed during the COVID-19 pandemic, due to enhanced outreach activities (EOAs). From March 2020 to April 2021, vaccination coverage appears lower than the rates observed prior to the COVID-19 pandemic (i.e., March 2019 to March 2020). However, multiple rounds of EOAs were conducted over the past three months, significantly increasing vaccination coverage. Over the next year, coverage rates are likely to remain similar to their current level. However, EOAs will not be sustained in the long term, and coverage rates will likely return to or fall below the levels observed prior to the COVID-19 pandemic over the next three years." GiveWell's non-verbatim summary of a conversation with IRD, September 14, 2021
  • "A decline in the percentage of immunizations occurring at fixed centers in Sindh. Prior to COVID-19, approximately 60% of child vaccinations occurred through fixed centers, and 40% through outreach. However, throughout the COVID-19 pandemic, the percentage of children visiting fixed centers for vaccination has remained significantly lower due to increased availability of vaccines through outreach. Many caregivers have likely become accustomed to the increased outreach, which could result in lower vaccination coverage rates once outreach activities become less frequent.
    • “This may change the overall impact of mCCTs, which IRD has found to be more effective for fixed center vaccinations than for those that occur through outreach. Additionally, children are typically not able to receive the full schedule of immunizations through outreach.” GiveWell's non-verbatim summary of a conversation with IRD, September 14, 2021.

• 20

  “A decline in the percentage of immunizations occurring at fixed centers in Sindh. Prior to COVID-19, approximately 60% of child vaccinations occurred through fixed centers, and 40% through outreach. However, throughout the COVID-19 pandemic, the percentage of children visiting fixed centers for vaccination has remained significantly lower due to increased availability of vaccines through outreach. Many caregivers have likely become accustomed to the increased outreach, which could result in lower vaccination coverage rates once outreach activities become less frequent.

  • “This may change the overall impact of mCCTs, which IRD has found to be more effective for fixed center vaccinations than for those that occur through outreach. Additionally, children are typically not able to receive the full schedule of immunizations through outreach.” GiveWell's non-verbatim summary of a conversation with IRD, September 14, 2021.

• 21

  “The close to 100% coverage of ZM among routine vaccination workers in Sindh. All government vaccination workers in Sindh, who provide the large majority of vaccinations, use ZM. Vaccinators at private clinic networks also typically use ZM, although there are likely some individual private clinics that do not. A very small proportion of campaign vaccination workers, a temporary staff made up of teachers and healthcare workers, use ZM.” GiveWell's non-verbatim summary of a conversation with IRD, June 29, 2021.

• 22

  We have not done a thorough review of this literature. Two data points give us some confidence in this assumption:

  • Using data from a large-scale RCT in India, Chernozhukov et al. 2018 (working paper) tested what factors were associated with high responsiveness to incentives for immunization and other nudges. They find that villages with low baseline vaccination rates have the highest responsiveness to incentives.
    • "This paper develops a generic approach to use any of the available ML tools to predict and make inference on heterogeneous treatment or policy effects. [...]" p. 3
    • "We apply our method to a large-scale RCT of nudges to encourage immunization in the state of Haryana, Northern India. [...] The experiment was a cross randomized design of three main nudges: providing incentives,sending SMS reminders, and seeding ambassadors." p. 4
    • "These results suggest that the villages with low levels of pretreatment immunization are the most affected by the incentives. These are in fact the only variables that consistently pop up from the CLAN [classification analysis]." p. 27

  From a meta-analysis of SMS reminders “We found relatively lower effectiveness of SMS reminders in upper middle-income countries although these countries have better childhood immunisation infrastructure than lower middle-income and low-income countries. Upper middle-income countries are likely to have high vaccination rates and thus limited potential for SMS reminders to improve the childhood immunisation rates further (ceiling effect).” Eze et al. 2021, p. 10.

• 23

  For example, if the Korangi trial had found that the program had increased coverage for the Penta-1 vaccine from 80% at baseline to 90% after the program, we would model that as a 50% decrease in the unvaccinated population, because the proportion of unvaccinated people decreased by half (20% to 10%). Two alternative approaches would be to model this effect as a 10 percentage point increase in vaccination rates (90% - 80% = 10%) or a 12.5% increase in the vaccinated population ((90% - 80%) / 80% = 12.5%), either of which would lead us to much lower estimates of the program's effect.

• 24

  PCV and the rotavirus vaccine, which were more recently introduced, may be more likely to be subject to stockouts, which could limit the program's effect on coverage for these vaccines:

  • “More recently introduced vaccines, such as pneumococcal conjugate vaccine (PCV) and rotavirus vaccine, are more likely to encounter stockouts.”
  • “Cash incentives for immunization have the potential to increase stockout rates due to increased demand. The RCT of IRD's mCCT program found slight differences in coverage for vaccines meant to be provided at the same visit, due to stockouts. However, as part of implementing its program, IRD plans to use the data it generates on vaccine demand to ensure sufficient supply.” GiveWell's non-verbatim summary of a conversation with IRD, June 29, 2021.

• 25

  One rough sense check is to compare under-5 mortality between Pakistan and Nigeria. This provides a benchmark, which doesn’t rely on attributing deaths to specific factors or assumptions on vaccination coverage and vaccine efficacy, for how much we should expect the share of deaths addressed by an incentives for vaccination program to vary between settings.

  In 2019, under-5 mortality was 60% as high in Pakistan as in Nigeria (10% for Nigeria, 6.3% for Pakistan). This closely matches the probability of death among unvaccinated we’re estimating between countries (1.6% for Pakistan, before subnational adjustments, and 2.6% for Nigeria, before subnational adjustments, which is also a roughly 60% difference). Calculations are here.

• 26

  We have not replicated the estimates from North West Nigeria here for Sindh or for specific districts with high vs. low coverage.

• 27
  • The moral weights we apply to various benefits of IRD's mCCT program are listed here. See our approach to moral weights in this document, including how we value neonatal deaths relative to all under-5 deaths.
  • IHME doesn’t provide data on neonatal tuberculosis deaths, but it may be possible to get it from a different source.

• 28

  See this cell.

• 29

  We use slightly different vaccine efficacies for rotavirus vaccine and measles.

  • For rotavirus, we use the Southern Asia-specific vaccine efficacy of rotavirus vaccination in the meta-analysis by Lamberti et al. 2016. They find vaccine efficacy of 50.0 (95% CI 34.4-61.9) against severe rotavirus diarrhea in Southern Asia (see Table 1). For New Incentives, we use the Sub-Saharan Africa-specific estimate.
  • For measles, we use vaccine efficacy for two doses of measles. We assume efficacy is higher for two doses, relative to one dose. For New Incentives, we use a value of 0.85 for one dose, so we guess a value of 0.90 for two doses of measles. We have not thoroughly vetted this estimate, but it’s unlikely to change our bottom line.

• 30

  See calculations here.

• 31

  We assume eligibility is lower for low-coverage districts due to lower cell phone access. Calculations are here.

• 32

  The fraud rate is based on a subjective guess and is similar to what we have assumed for New Incentives. See this cell of our cost-effectiveness analysis.

• 33

  See this spreadsheet for details.

• 34

  See this spreadsheet for details.

• 35

  This is based on conversations with IRD, suggesting it’s fairly unlikely the government would fund ZM over the next 2-3 years:

  • “IRD's upcoming nine-month contract with Gavi to develop a plan for transitioning ownership of ZM to the government of Sindh. The grant will only fund planning activities. Actual government ownership of ZM (e.g., conducting data analysis without assistance from IRD) will take at least 2-3 years, and would initially only include around 10% of the estimated $1.8 million annual cost of operating ZM. The government of Sindh, however, is strongly committed to ZM and would have already begun taking on partial ownership if not for the disruptions caused by the COVID-19 pandemic.
    • “In particular, the government has expressed strong interest in taking on the costs of district field coordinators, who have been trained by IRD and are key components of immunization infrastructure at the district level.
  • “The strong probability that ZM operations will remain unfunded beyond September 2021, absent additional funding from GiveWell or another donor.” GiveWell's non-verbatim summary of a conversation with IRD, July 13, 2021.

  We discuss this here.

• 36

  We discuss this here.

• 37

  We also have not incorporated any adjustments to account for drop-off in quality if this was transitioned to the government. This would increase cost-effectiveness.

  We also assume the government is the counterfactual funder, rather than Gavi. If Gavi were the funder of either ZM or mCCTs, this would increase cost-effectiveness because we assume the counterfactual value of Gavi's spending is higher than that of the government, so funging Gavi is more cost-effective than funging the government.

• 38

  See this spreadsheet for details.

• 39

  “The relatively high likelihood that the Sindh government will begin funding and operating ZM independently of IRD within three years. If the government is unable to take ownership of ZM within three years, it is less likely it will be able to do so in the longer-term.” GiveWell's non-verbatim summary of a conversation with IRD, August 17, 2021.