IRD Global — Mobile Conditional Cash Transfers for Immunizations (October 2021)

Note: This page summarizes the rationale behind a GiveWell Incubation Grant to IRD Global. IRD staff reviewed this page prior to publication.

Summary

In October 2021, GiveWell recommended a three-year grant of up to $25 million (our best guess is $16 million)1 to IRD Global (IRD) to implement the following:

  • A mobile phone-based conditional cash transfer (mCCT) program in seven high-risk/low immunization coverage districts in Sindh Province, Pakistan (more).
  • Ongoing technical assistance and operational support for the Sindh Electronic Immunization Registry (SEIR aka Zindagi Mehfooz (ZM)) throughout Sindh Province, Pakistan (more).

We are recommending this grant because:

  • We think it's cost-effective. IRD's program averts child deaths by increasing vaccination rates, which in turn leads to fewer deaths from vaccine-preventable diseases (VPDs).
  • It has learning value. We directed another grant to support a rigorous evaluation of this program, the preliminary results of which could inform our funding decisions as soon as late 2022.2 We also expect to learn whether ZM could be leveraged as a platform for other cost-effective programs in the future.
  • If the work funded by this grant is successful, there may be cost-effective opportunities to support the expansion of IRD's program within and beyond Pakistan.

Our main reservation is related to uncertainty about the cost-effectiveness of this grant. Specifically, the studies we rely on to estimate the impact of the mCCT program differ from IRD's program in important ways. We also have very limited evidence to draw on to estimate the impact of ZM alone. Finally, we are uncertain about vaccination coverage rates, which are an important input into our cost-effectiveness estimate for the program. We expect to learn more and reduce our overall level of uncertainty over the course of this grant.

The grant was funded by Open Philanthropy. Note that on our recommendation, Open Philanthropy made a bridge grant to IRD in June 2021 for $336,402 to support ZM while we conducted our investigation.

We have had the following conversations about updates on this grant:

Published: August 2022

Table of Contents

The organization: IRD Global

IRD is a global health-focused organization with country offices in South Africa, the United Arab Emirates (UAE), Pakistan, Bangladesh, Vietnam, Indonesia and Singapore, and programs in several other countries.3 IRD has expertise in various program areas, including:

  • Maternal and Child Health (MCH)
  • Infectious diseases
  • Neglected tropical diseases
  • Non-communicable diseases
  • Youth engagement
  • Mental health4

IRD’s MCH program team has spent over a decade in developing and scaling-up the ZM-EIR (more), and has shown an appetite for testing innovative approaches to using that platform, including AI-powered chatbots, predictive analytics, digitized and interactive gamified training modules for Frontline Health Workers, and vaccine supply chain support.5

The program

This grant covers two activities: (1) continued technical assistance and implementation of a mobile-based electronic immunization registry (EIR) called Zindagi Mehfooz, and (2) roll-out of a mobile conditional cash transfer program for immunizations that leverages that immunization registry platform.

Zindagi Mehfooz: Electronic Immunization Registry in Sindh Province

Since 2011, IRD has developed and implemented an electronic immunization registry (EIR) called Zindagi Mehfooz (ZM). After a series of pilots and limited scale deployments, it was scaled-up in coordination with the Sindh provincial government in 2017.6 It was later expanded to other regions of Pakistan (Islamabad and Gilgit districts).7

ZM captures data on demographics and child vaccination status.8 Beyond recording immunization data, the ZM platform is also capable of additional support and functionality, including immunization decision support for vaccinators in determining the appropriate vaccines to offer at any given visit, and a pregnant women and birth registry to capture child births.9 ZM is interoperable with other tech platforms including DHIS2 and the government’s Expanded Programme on Immunization (EPI) Management Information System.10 In 2020, the most recent year with full data available at the time of analysis, ZM enrolled around 1.8 million children in Sindh Province, representing roughly all of the estimated birth cohort.11

Data validation

IRD has said that it currently runs checks to confirm the accuracy of ZM data (e.g., calling phone numbers and making in-person clinic visits to confirm infants registered as receiving a particular vaccine actually got it). However, it said it does not have systematic data collected on these checks (e.g., percentage of recorded immunizations that passed audits) that it can share with us.12 As a result, we have not independently evaluated data audits generated by the ZM platform.

IRD is planning on running more consistent data validation and compliance/fraud checks on ZM data in the context of this program.13

Mobile conditional cash transfers for immunizations

IRD plans to run a mobile phone-based conditional cash transfer (mCCT) program in the seven highest-risk (characterized by low Penta-3 and Measles-1 vaccination rates) districts in Sindh Province, Pakistan.14

In short, caregivers who bring their children in for EPI routine vaccines, which are provided by the Sindh government free of charge, will receive incentives over the course of six visits.15 IRD tested a similar mCCT program in a three-year RCT in Korangi town in Sindh Province, working with J-PAL and with partial funding recommended by GiveWell (more in the intervention report).

IRD also plans to engage caregivers via SMS, mass media, and community engagement campaigns about the importance of vaccinating children, and to work with its government partners to improve vaccine supply by identifying and addressing challenges in the vaccine supply chain.16

Incentives

IRD plans to offer mCCTs with a net worth (i.e., after tax deduction) of 200 PKR (approximately $1.20) for each of the six immunization visits.17 The following table shows the schedule and list for directly and indirectly18 incentivized vaccines.

Pakistan’s EPI Schedule & Vaccines Incentivized19
Visit # Age Directly incentivized Indirectly incentivized
1st Birth BCG OPV-0, Hep B
2nd 6 weeks Penta-1 OPV-1, PCV-1, Rota-1
3rd 10 weeks Penta-2 OPV-2, PCV-2, Rota-2
4th 14 weeks Penta-3 OPV-3, PCV-3, IPV-1
5th 9 months Measles-1 Typhoid, IPV-2
6th 15 months Measles-2 --

Caregiver enrollment in program

Caregivers will be eligible to participate in the program if they have a child 0-23 months of age enrolled in ZM and have cell phone accessibility (our understanding is the phone numbers provided by caregivers do not have to be their own). Caregivers will be able to enroll in the program during routine well-child visits, at “outreach locations,” and in government-designated birthing centers.20

Based on our review of the Korangi RCT, our current best guess is that only 5%-10% of caregivers will be unable to participate in IRD's mCCT program due to a lack of mobile phone accessibility.21

Cash transfer distribution

Cash transfers will be distributed via mobile airtime top-ups (credits that can be used to pay for mobile services) to be disbursed directly for immediate use to caregivers’ registered cell phone numbers.22 Mobile airtime top-ups were chosen above direct cash distribution primarily because mobile top-ups are more widely accessible compared to electronic cash transfers, which require a government identification card.23

Caregiver engagement

IRD plans to communicate with caregivers about routine immunizations and mCCTs via a mass media campaign, sign boards at child wellness centers, mobile immunization vans, a helpline, community engagement, and various other channels.24

Budget

IRD’s fixed three-year budget for the program is $9,913,126.25 Additionally, IRD has estimated a cost of $6,168,374 over three years to cover incentives and variable costs associated with administering incentives.26

Because the total cost for incentives is unknown (it will depend on enrollments in the program), we recommended that Open Philanthropy approve up to $15,066,861 overall across the three years for incentives, totaling roughly $25 million when program costs are included.27 However, our best guess, based on how much we estimate the program is likely to increase vaccination, is that the total cost of the program will be close to the $16 million estimate from IRD.

IRD’s $9,913,126 fixed program budget breaks down as follows:

  • Personnel – approximately $6.3 million, including salaries for staff from IRD Pakistan and global IRD staff. This includes salaries for around 100 people over three years, totaling roughly $5.6 million, with the remaining $0.7 million covering the costs of travel and fieldwork.28
  • Procurement – approximately $1.2 million, including costs associated with sending SMS messages to participants and cloud-based server hosting.29
  • Laptops and mobile phones for vaccinators – approximately $820,000. Our current best guess is that these costs will be incurred once every three years as equipment wears out.30
  • Other costs – approximately $700,000, including costs associated with communications and engagement.31
  • Indirect costs – approximately $950,000, or 10% of the total program budget.32

Variable costs associated with incentives include:33

  • PKR 200 (approximately $1.26) exclusive of tax for each vaccination visit directly incentivized.34
  • Charges associated with distributing mCCTs, including 10% transaction costs and 3% tax.
  • 5% indirect costs additional to the programmatic indirect costs referenced above.

Does it work?

We model the core benefit of IRD's ZM platform and mCCT program as an increase in child vaccination coverage, leading to a reduction in child mortality. See our intervention report for more details.

Cost-effectiveness

Based on our cost-effectiveness analysis, our best guess is that this program is approximately eight times as cost-effective as unconditional cash transfers, which is in the range of cost-effectiveness of programs we expect to direct funding to as of 2021.35 See our intervention report for more details.

Note that there are limitations to this kind of cost-effectiveness analysis, and we believe that cost-effectiveness estimates such as these should not be taken literally, due to the significant uncertainty around them. We provide these estimates (a) for comparative purposes and (b) because working on them helps us ensure that we are thinking through as many of the relevant issues as possible.

Is there room for more funding?

In Sindh

We don’t think that it’s likely (<5% probability) that another donor would support IRD’s proposed mCCT program in high-risk districts in Sindh in the next three years.36

Based on a light review, we were unable to identify any other funders providing mCCTs for immunizations in Pakistan or elsewhere at substantial scale.37 Additionally, our impression is that it is unlikely the Pakistani government would fund a large-scale mCCT program for vaccinations.

Our best guess is that demonstrated success in this program will lead to further opportunities to scale the program across all of Sindh, which we think would yield roughly $11-12 million a year in room for more funding.38

Elsewhere in Pakistan

We think it’s plausible that demonstration of the success of the mCCT program in Sindh would encourage other provinces in Pakistan to adopt a similar program, if funding were available.39 On the other hand, our impression is that political considerations can also weigh into decisions on whether a given program is adopted elsewhere, and we have not explored these in depth in other areas of Pakistan.

Beyond Pakistan

We do not have a strong sense of the likelihood that an intervention like this would be feasible to be scaled up by IRD beyond Pakistan, but it’s our broad impression that there is global interest in adopting electronic immunization registries, particularly in the context of COVID-19 vaccine delivery.40

Risks and reservations

Major differences between the mCCT program that will be implemented and evidence used to inform cost-effectiveness. We rely heavily on an RCT of the effect of SMS reminders and mCCTs conducted in Korangi town in Sindh. The program evaluated in this trial has key differences from the program that IRD will implement in Sindh:

  • The trial enrolled and notified caregivers about the mCCT program at early vaccine visits.41 As a result, the trial has limited ability to detect effects on uptake of vaccines early in the sequence. These early vaccines drive cost-effectiveness, so high uncertainty about effects on these vaccines means high uncertainty about cost-effectiveness.42
  • The program implemented during the grant period will include communications activities to inform caregivers about mCCTs before early vaccine visits. We anticipate this will increase cost-effectiveness, but we have high uncertainty about the magnitude.43

In light of these concerns, we also put some weight on the effects observed from an RCT of New Incentives' conditional cash transfer for immunization program in Nigeria. That program also included communications activities and the RCT permitted measuring effects on vaccines early in the sequence, so we view it as providing one benchmark for the effect of IRD’s program, though we do make adjustments to account for differences between New Incentives and IRD’s program and their respective settings.44

High uncertainty about the impact of the ZM-EIR. We extrapolate the effects of ZM based on the Korangi trial’s estimates of the effect of SMS reminders, which are a component of ZM, on vaccination coverage. We view these effects as broadly consistent with other evidence on the effect of SMS reminders on vaccination. However, we are uncertain about how these will generalize beyond the trial and about the magnitude of benefits of additional services provided by ZM.45 While the evaluation of the IRD program will provide some additional information about the effect of mCCTs, it will not be able to provide estimates of the effect of ZM on vaccination.46

High uncertainty about vaccination coverage in the near term, especially due to COVID-19. Cost-effectiveness depends heavily on coverage rates, which affect both costs (high counterfactual coverage means paying incentives to more caregivers who would have brought their children in for vaccinations regardless) and effect size (higher counterfactual coverage means less room to improve coverage). Similar to our cost-effectiveness analysis for New Incentives, we use recent data on vaccination coverage from IRD as our best guess of near-term vaccination coverage and make a downward adjustment to cost-effectiveness to account for coverage increases in the future.47 However, if coverage is higher or lower than we’ve estimated, this could have large impacts on cost-effectiveness.48

The Korangi RCT is not published yet. The RCT findings have not yet been published. There is some possibility that findings may change after undergoing peer review.

We hired a third-party researcher with whom we have an established relationship to review the analysis strategy and reproduce the analyses to confirm the results. The review did not identify any issues that we thought would meaningfully change the main findings we used to make our decision.

Political support is key to program success. Due to the program's integration with the provincial government systems, there is a risk that changes in the provincial government leadership or priorities could put it at risk.

Our best guess is that support from the health department will not be an existential risk to the program’s success during the course of this three-year grant. We are less certain about the likelihood of major political or contextual changes in the longer term.

"Crowding out" other motivations for vaccinating children. It is possible that, by creating a financial motivation to vaccinate infants, IRD’s program could "crowd out" intrinsic motivations for caregivers to bring children in for vaccinations. This might potentially lead to lower vaccination rates after the program is discontinued in an area than there would have been if the program had not been implemented.49

In our investigation of New Incentives (another program offering conditional cash transfers for immunizations), it told us of one case in Northern Nigeria in which some caregivers reportedly refused to vaccinate their infants after in-kind incentives for a polio vaccination campaign were suspended.50 Based on a light search, we were unable to identify other examples of this.

It also seems plausible that as a result of, for example, IRD’s efforts to communicate about the benefits of vaccination or improved social norms toward vaccination as a result of higher rates caused by the program, IRD’s program could lead to increases in vaccination rates that persist even if incentives are later removed. However, this is speculative, and we have not explored evidence for or against this.

Overall, we judge it unlikely that this consideration would significantly offset the program's benefits, though we have not investigated this question in detail.

Plans for follow up

Impact evaluation. We have recommended funding an evaluation of the program alongside implementation. We will be writing more about that grant decision on a separate page.

Routine updates. IRD will report annual progress on indicators listed in its log frame. GiveWell staff will also have access to dashboards outputting key indicators on a real-time basis.51 During the initial months of the scale up, we will hold monthly calls with IRD to track progress. After the program has been fully operationalized (expected in mid-2022), we may reduce the frequency of our calls with IRD.

Broadly, we expect that this scale up will have gone successfully if the mCCT program has been rolled out in all seven districts by August 2022,52 and if 80% or more of the estimated annual birth cohort in these districts is enrolled by the end of 2023.

Forecasts

By October 2022, we think there is an 85% likelihood that IRD will successfully roll out the mCCTs program in all targeted districts. By October 2023, we think there is a 50% likelihood that GiveWell will assess this program in these seven districts as being at least eight times more cost-effective than cash transfers ("8x cash"), using 2021 methodology.

Finally, we think there is a 70% likelihood that by October 2023 GiveWell will have recommended further funding to expand this program beyond the initially targeted districts, either in Sindh province or in another province where ZM is operating.

Our process

  • In 2015, GiveWell recommended funding to IRD's Korangi RCT.
  • During follow-up conversations to get an update on the grant, we learned more about IRD's overall portfolio, and identified its mCCT program as an area of further interest based on preliminary RCT findings and our general impression that mCCTs for child vaccinations was a promising area.
  • We had a number of conversations with IRD to learn more about its mCCT program and ZM platform.
  • IRD shared ZM enrollment and baseline vaccination coverage rates data.
  • We spoke with funders of IRD.
  • IRD submitted a program scale-up proposal, budget, and various other planning documents, which we discussed and revised over several conversations.
  • We created a cost-effectiveness analysis and wrote an intervention report for IRD's mCCT program, which were reviewed and vetted internally.

Sources

Document Source
GiveWell, IRD CEA (upper bound costs), 2021 (unpublished) Unpublished
GiveWell, Main IRD CEA, 2021 Source
GiveWell's non-verbatim summary of a conversation with IRD, August 12, 2021 Source
GiveWell's non-verbatim summary of a conversation with IRD, August 13, 2021 Source
GiveWell's non-verbatim summary of a conversation with IRD, August 17, 2021 Source
GiveWell's non-verbatim summary of a conversation with IRD, August 31, 2021 Source
GiveWell's non-verbatim summary of a conversation with IRD, July 13, 2021 Source
GiveWell's non-verbatim summary of a conversation with IRD, July 27, 2021 Source
GiveWell's non-verbatim summary of a conversation with IRD, June 29, 2021 Source
GiveWell's non-verbatim summary of a conversation with IRD, September 14, 2021 Source
Government of Pakistan, Benazir Income Support Programme , "Overview" Source (archive)
IRD, "About Us" Source (archive)
IRD, "Global Footprint" Source (archive)
IRD, "Zindagi Mehfooz (Safe Life)" Source (archive)
IRD, Budget for mCCT scale-up (Sindh province-wide), 2021 (redacted) Source
IRD, Budget for mCCT scale-up (Sindh low-coverage districts), 2021 (redacted) Source
IRD, EPI vaccination schedule, government of Sindh (English translation) Source
IRD, Korangi trial methods, 2021 (unpublished) Unpublished
IRD, Log frame for program indicators, 2021 Source
IRD, Monitoring plan for scale-up of mCCTs in Sindh (working draft), 2021 Source
IRD, Proposal for scale-up of mCCT in Sindh (working draft), September 22, 2021 Source
IRD, Responses to GiveWell's questions, September 28, 2021 Source
IRD, Slide deck on maternal and child health program Source
IRD, Spreadsheet of children enrolled in ZM and estimates of annual birth cohort in Sindh, 2021 Source
IRD, Spreadsheet of vaccines administered by mode of delivery, 2021 (unpublished) Unpublished
IRD, Timeline for mCCT scale-up, 2021 Source
IRD, Zindagi Mehfooz overview, 2021 Source
IRD, ZM application codebook Source
PATH, Digital Square: Electronic Immunization Registries in Low- and Middle-Income Countries, 2021 Source
The World Bank, Press release: "World Bank Supports Expansion of the Ehsass Social Protection Program in Pakistan to Increase Household Resilience to Economic Shocks," 2021 Source (archive)
  • 1The $16 million budget is an estimate due to uncertainty in the amount that will be needed for cash incentives, the component of IRD's mCCT program meant to induce caregivers to have children vaccinated. Aside from $9,913,126 in fixed program costs, IRD has estimated $6,168,374 variable costs for incentives and associated fees and overhead. We recommended that Open Philanthropy approve up to $15,306,018 overall across the three years for incentives, which represents the absolute upper bound in terms of what might be needed to cover incentives for all children in each annual birth cohort in Sindh over the three-year grant period (more).
  • 2See our write-up on the grant for this evaluation here.
  • 3
    • "IRD is a global health delivery & research organization that works in over 15 countries. Based in Singapore, IRD has country affiliate offices in Bangladesh, Indonesia, Pakistan, South Africa, UAE and Vietnam. IRD is committed to improving the lives of vulnerable communities by building a network that shares ideas and innovations to create global impact." IRD, "About Us"
    • For a map of IRD's country offices and programs, see IRD, "Global Footprint"

  • 4IRD, Slide deck on maternal and child health program, slide 4.
  • 5See IRD, Zindagi Mehfooz overview, 2021, slides 21, 34, 35, and 37.
  • 6
    • "ZM started in 2011 as a phone-based electronic immunization registry to improve immunization coverage and timeliness." IRD, Zindagi Mehfooz overview, 2021, slide 3.
    • "In the Sindh province, the Provincial Health Department has collaborated with Interactive Research and Development (IRD) and the Indus Hospital and Health Network (IHHN) to provide technical support for the implementation of a phone-based digital immunization registry program ‘Zindagi Mehfooz’ across all 30 districts." IRD, "Zindagi Mehfooz (Safe Life)"
    • IRD, Comments on a draft of this page, April 28, 2022 (unpublished)

  • 7IRD, Comments on a draft of this page, April 28, 2022 (unpublished)
  • 8See IRD, ZM application codebook.
  • 9IRD, Zindagi Mehfooz overview, 2021, slides 4-5 and 23-24.
  • 10IRD, Comments on a draft of this page, April 28, 2022 (unpublished)

  • 11IRD, Spreadsheet of children enrolled in ZM and estimates of annual birth cohort in Sindh, 2021, sheet "Sheet 1," row "2020".
    • Enrollments (1,763,421) / Estimated birth cohort (1,725,479) = ~102%.
    • Note that birth cohort figures are rough. We have not independently verified the estimated EIR enrollments or birth cohort data IRD provided.

  • 12“Checks to verify the accuracy of ZM data. IRD makes random phone calls and in-person center visits to confirm caregivers' contact information and to ensure that immunizations recorded in ZM actually occurred. Government health officials conduct similar checks. Additionally, vaccinations recorded in ZM are tagged with timestamps and geolocation data, and this activity is also checked during the random visits IRD conducts. Currently, IRD does not systematically conduct data checks and is unable to share aggregated metrics (e.g. % of recorded immunizations that passed an audit).” GiveWell's non-verbatim summary of a conversation with IRD, August 12, 2021, Pg. 1.
  • 13IRD is planning on running regular fraud checks for the mCCT program, including caregiver surveys (phone and in-person), creation of a complaints mechanism, spot checks by an internal audit team, and automated monitoring of immunization data to spot unusual and suspicious patterns as flags for focused monitoring.
    "A central concern with the implementation of incentive programs on a large scale is the potential for leakage and corruption, which are rampant in many government programs [...] We will address these risks by surveying (phone and in-person) caregivers, setting up a complaints mechanism, and systematic process reviews and spot checks by internal audit team and district coordinators. We will also monitor immunization data to spot unusual and suspicious patterns as flags for focused monitoring." IRD, Monitoring plan for scale-up of mCCTs in Sindh (working draft), 2021, Pgs. 1-2. See additional details in the document.

  • 14“The mCCTs Sindh project will provide conditional cash transfers to parents/caregivers of children between the ages of 0-23 months residing in the 7 high-risk districts of Sindh province, for completing each of the six EPI recommended immunization visits [...] The high-risk districts are categorized based on the 20th percentile of crude penta-3 and crude measles-1 vaccines district coverage rates (Figure 2). There are 6 districts (20.0%) below the 20th percentile for penta-3 vaccine (coverage of 55.86%) which include Hyderabad, Karachi East, Karachi West, Karachi Central, Jacobabad, and Sujawal. Additionally, there are 6 districts (20.0%) below the 20th percentile for measles-1 vaccine (coverage of 40.87%) including Karachi Central, Karachi East, Karachi West, Jacobabad, Kambar, and Sujawal. The 7 high-risk districts below the 20th percentile for crude penta-3 and measles-1 coverage are, therefore: Karachi Central, Karachi East, Karachi West, Hyderabad, Sujawal, Jacobabad, and Kambar.” IRD, Proposal for scale-up of mCCT in Sindh (working draft), September 22, 2021, Pgs. 5 and 10.
  • 15IRD, Proposal for scale-up of mCCT in Sindh (working draft), September 22, 2021
    • “The mCCTs Sindh project will provide conditional cash transfers to parents/caregivers of children between the ages of 0-23 months residing in the 7 high-risk districts of Sindh province, for completing each of the six EPI recommended immunization visits [...] IRD will provide comprehensive support to facilitate Government’s EPI-Sindh, which provides free-of-cost vaccines via fixed centers, outreach, and mobile vaccination activities, through the following…" Pg.
    • Supplemental vaccinations occurring via campaigns will not be incentivized: "The national and sub-national campaigns for additional doses [...] are not eligible for the incentive." Pg. 22. IRD has shared data with us showing that roughly 0.1% of vaccines delivered to children enrolled in ZM in Sindh are administered via campaigns. IRD, Spreadsheet of vaccines administered by mode of delivery, 2021 (unpublished)

  • 16IRD, Proposal for scale-up of mCCT in Sindh (working draft), September 22, 2021
    • "During the Korangi trial, we subcontracted the SMS reminder service to Telenor (one of the four main Mobile Network Operators (MNOs) in the country). However, for the ZM scale-up by the Government, Intellexal Solutions was subcontracted, which is currently handling the SMS reminder service for all children (>5 million) enrolled in the ZM EIR throughout the province." Pg. 8.
    • "We will deploy a hybrid mass media communications and community engagement model to engage, motivate and inform caregivers regarding routine immunizations and mCCTs in all the mCCT districts." Pg. 17.
    • "A vaccinator is a front-line health worker employed by the immunization centers to administer vaccinations through fixed sites and outreach. Through GSM and GIS-based tracking of vaccinators, ZM-EIR will monitor vaccinator attendance and compliance. Moreover, through capturing the client flow and number of vaccinators, the ZM-EIR will assist in determining if the centers have adequate levels of staff. Facility-level reports will be generated to assist supervisors in their management [...] Maintaining vaccine stock is essential for the effective planning of vaccination services. ZM-EIR assists in the management of vaccination supply through recording each vaccination administered and accurate consumption data [...] To ensure vaccine potency, it is crucial to store the vaccinations at the required temperatures. ZM-EIR field reports (generated through site visits on a daily basis) would identify a lack of functional equipment at the centers [...] The project will share attendance, stock report, and cold chain equipment status with EPI-Sindh to ensure the availability of critical supply-side components for vaccine delivery." Pgs. 29-30.

  • 17
    • "In the mCCTs scale-up project, unlike the trial, we will have only one fixed amount of payment (USD 1.26/PKR 200) for each immunization visit across all the selected 7 high risk districts." IRD, Proposal for scale-up of mCCT in Sindh (working draft), September 22, 2021, Pg. 6.
    • “The minimum incentive size of 200 Pakistani rupees (around $1.20) needed to ensure mCCTs are effective. A lower incentive size is likely to have a relatively small effect. It is possible that lower incentive amounts could be effective in the short-term, however, due to the negative economic impact that the ongoing COVID-19 pandemic is having on many individuals in Pakistan.” GiveWell's non-verbatim summary of a conversation with IRD, August 17, 2021, Pg. 2.
    • IRD, Comments on a draft of this page, April 29, 2022 (unpublished)

  • 18“Multiple vaccines are provided at each visit in Pakistan's routine immunization schedule, although IRD's mCCT program will only explicitly incentivize a subset of these vaccines that are at low risk of stockouts. In the RCT of the program, stockouts only caused very small differences in coverage between directly and indirectly incentivized vaccines provided at the same visit (i.e., less than 0.5 percentage points).” GiveWell's non-verbatim summary of a conversation with IRD, August 31, 2021, Pgs. 1-2.
  • 19

  • 20"Enrollment Sites & Eligibility
    1. EPI-registered public immunization centers (1,546) and private centers offering EPI services (230):
      a. Eligibility criteria: 0-23 month old child visiting for any of the 6 immunization visits.
    2. Routine Outreach services (in all districts, excluding sub-national campaigns) are offered through public immunization centers.
    3. Government designated birthing centers participating in the birth registry:
      a. Eligibility criteria: newborns and neonates receiving BCG/birth OPV.

    Eligibility for mCCT disbursement post-vaccination
    For Mobile airtime transfer

    1. The caregiver has a valid pre-paid mobile number." IRD, Proposal for scale-up of mCCT in Sindh (working draft), September 22, 2021, Pg. 23.

  • 21See this cell for references.

  • 22

  • 23

  • 24IRD, Proposal for scale-up of mCCT in Sindh (working draft), September 22, 2021
    • "We will deploy a hybrid mass media communications and community engagement model to engage, motivate and inform caregivers regarding routine immunizations and mCCTs in all the mCCT districts [...] Our mass media campaign will involve [...] A social media campaign (YouTube, Facebook, Instagram, and Tik Tok) focusing on user-generated content, advertisements, and collaboration with key influencers for motivating parents to get involved in the project [...] Transport branding of 150 rickshaws coupled with rickshaw rallies in all 7 high-risk districts to provide additional visibility and reach to on-ground communities. We will also brand five mobile immunization vans to support this effort additionally [...] Rebranding of 398 EPI centerboards to reflect mCCTs intervention branding to link centers to communications campaigns and increase the visibility of the project [...]" Pgs. 17-18.
    • "A toll-free dedicated helpline/contact (0800 number) will be provided (through advertisement, posters at EPI centers, SMS, EPI cards) to users/caregivers for confidentially registering complaints through:
    1. SMS
    2. Phone call
    3. Email" Pgs. 26-27.

  • 25See IRD, Budget for mCCT scale-up (Sindh low-coverage districts), 2021 (redacted), sheet "Budget Summary," rows "TOTAL INSTITUTION BUDGET in US Dollars" and "Total" for "PHONES AND LAPTOP COSTS."
    • 9,093,449 + 819,677 = $9,913,126

  • 26See IRD, Budget for mCCT scale-up (Sindh low-coverage districts), 2021 (redacted), sheet "Budget Summary," row "Total" for "INCENTIVE COSTS - SCENARIO 1 (post-Covid19)."
  • 27The $15,066,861 estimate for total incentive costs assumes 10% “wastage” of incentives due to fraud and 100% vaccination coverage in districts receiving mCCTs across all three years, the latter of which we believe is highly unlikely to occur. We view this as an upper bound on the amount of incentives that would be disbursed. GiveWell, IRD CEA (upper bound costs), 2021 (unpublished)
    • Total upper bound cost: $9,913,126 + $15,066,861 = $24,979,987

  • 28

  • 29See budget total in IRD, Budget for mCCT scale-up (Sindh low-coverage districts), 2021 (redacted), sheet "Budget Summary," row "Procurement."
  • 30

  • 31See IRD, Budget for mCCT scale-up (Sindh low-coverage districts), 2021 (redacted), sheet "Budget Summary," rows "Mobile Van Operational Cost," "Communications Budget," and "Contingency Costs"
    • 229,480 + 364,073 + 80,388 = $673,941

  • 32See IRD, Budget for mCCT scale-up (Sindh low-coverage districts), 2021 (redacted), sheet "Budget Summary," row "Overhead / Indirect costs."
    • 974,298 / 9,913,126 = ~10%

  • 33See IRD, Budget for mCCT scale-up (Sindh low-coverage districts), 2021 (redacted), sheet "Budget Summary," section "INCENTIVE COSTS - SCENARIO 1 (post-Covid19)."
  • 34 "In the mCCTs scale-up project, unlike the trial, we will have only one fixed amount of payment (USD 1.26/PKR 200) for each immunization visit across all the selected 7 high risk districts." IRD, Proposal for scale-up of mCCT in Sindh (working draft), September 22, 2021, Pg. 6.
  • 35As of October 2021, we were primarily looking to recommend grants that we estimate are 8 or more times as cost-effective as GiveDirectly's unconditional cash transfer program, and were willing to consider recommending a limited amount of funding to grants that are between 5 and 8 times as cost-effective as GiveDirectly. For an example of the cost-effectiveness of our recommendations, see this page.
  • 36 This is our qualitative impression based on multiple conversations with IRD and its partners.

  • 37It should be noted that the World Bank funds conditional cash transfers in Pakistan via Pakistan’s domestic poverty alleviation program called Ehsaas. It could be possible that the World Bank would eventually prioritize immunizations in the context of its CCT programming.

    Pakistan also runs a domestic unconditional cash transfer program focused on poverty alleviation called the Benazir Income Support Programme.

    • "The Government of Pakistan launched the Benazir Income Support Programme (BISP) in July 2008 to serve as the premier national safety net institution with the primary objective of consumption smoothing and alleviating adverse effects of slow economic growth. The country at that time was experiencing rapid food price inflation, with prices of basic necessities reaching a 30-year high, and economic fallout from the global financial crisis. The Programme targets cash transfers to vulnerable and deserving women and their families from the poorest households across the country irrespective of political affiliations, racial identity, geographic location, and religious beliefs." Government of Pakistan, Benazir Income Support Programme , "Overview"

  • 38
    • "The potential for IRD's mCCT program in Sindh to motivate other provincial governments in Pakistan to implement similar programs. 2023 will also be an election year in Pakistan, which could create political incentives for establishing successful social programs." GiveWell's non-verbatim summary of a conversation with IRD, September 14, 2021, Pg. 2.
    • IRD previously shared a budget for scaling up the program across all of Sindh. That budget was $11,776,572 in Year 3 (once the program was at scale) for all districts in Sindh. IRD, Budget for mCCT scale-up (Sindh province-wide), 2021 (redacted)
    • We also guess there is additional room for more funding outside of Sindh, though we have not modeled this room for more funding explicitly.

  • 39
    • "The potential for IRD's mCCT program in Sindh to motivate other provincial governments in Pakistan to implement similar programs. 2023 will also be an election year in Pakistan, which could create political incentives for establishing successful social programs." GiveWell's non-verbatim summary of a conversation with IRD, September 14, 2021, Pg. 2.
    • mCCTs and ZM cost roughly $9.50 per high-risk child (see GiveWell, Main IRD CEA, 2021). Balochistan, a neighboring province that has expressed interest in ZM, has an annual birth cohort of around 450,000 children, and we’d expect that all of the districts in Balochistan would be considered high-risk. Conversations with IRD on May 19 and June 2, 2021 (unpublished)

  • 40
    • "IRD has submitted ZM to a request for proposals (RFP), issued by World Health Organization Pakistan, for the establishment of an electronic immunization registry in provinces other than Sindh. It has also submitted ZM to a Gavi RFP for Pakistan, Bangladesh, and Indonesia and is engaging with various partners to generate interest in expanding ZM to Nigeria." GiveWell's non-verbatim summary of a conversation with IRD, August 17, 2021, Pg. 1.
    • "With increasing digitalization of health systems in low- and middle-income country (LMIC) contexts, there is increasing interest from governments, implementing partners, and investors to introduce and scale EIRs [...] with the introduction of COVID-19 vaccines, some countries are introducing new EIRs to track COVID-19 vaccine deployment [...] EIRs can be used to capture data about COVID-19 vaccine delivery, and the data that EIRs capture on routine vaccination can help with understanding and curbing the devastating impacts of global pandemics." PATH, Digital Square: Electronic Immunization Registries in Low- and Middle-Income Countries, 2021, Pgs. 1, 17, and 35.

  • 41IRD, Korangi trial methods, 2021 (unpublished)

  • 42See our cost-effectiveness analysis here.
  • 43
    • “The importance of communications to the effectiveness of mCCTs. IRD's experience with tuberculosis treatment programs suggests that simple messaging through billboards and television advertising can substantially increase program reach and ultimately the number of individuals that seek care.” GiveWell's non-verbatim summary of a conversation with IRD, August 31, 2021
    • “IRD's expectation that the effect size of its mCCT program at scale would be higher than what was observed in the randomized controlled trial (RCT) of the program, due to a broader communications strategy including widespread advertisements. In the RCT, only individuals visiting health centers were made aware of the program, although there was also likely some small-scale spread through word of mouth.” GiveWell's non-verbatim summary of a conversation with IRD, August 13, 2021
    • “The large potential impact of a communications strategy on the cost-effectiveness of mCCTs. A study in Haryana, India, for example, found that identifying and targeting messaging to community leaders significantly increased the impact of mCCTs. Initially, the communications strategy for IRD's mCCT program will only include infrequent text messages, although different strategies (e.g., more frequent communications, television advertising) could be rolled out and tested at a later point in time.” GiveWell's non-verbatim summary of a conversation with IRD, September 14, 2021

  • 44See our intervention report for details.

  • 45

    See our intervention report for details.

  • 46This is because ZM has already been scaled across Sindh, and there is therefore no control group to compare against:
    • “All government vaccination workers in Sindh, who provide the large majority of vaccinations, use ZM. Vaccinators at private clinic networks also typically use ZM, although there are likely some individual private clinics that do not. A very small proportion of campaign vaccination workers (mostly temporary staff made up of teachers and healthcare workers besides vaccinators) use ZM.” GiveWell's non-verbatim summary of a conversation with IRD, June 29, 2021, Pg. 2.

  • 47See this cell in our cost-effectiveness analysis.
  • 48See our intervention report for details.

  • 49We have a similar concern for New Incentives' CCT for immunization program in Nigeria (more details here).
  • 50“A concern . . . stakeholders (including UNICEF representatives) have shared with us is about incentives replacing intrinsic motivations to immunize infants in the absence of incentives. This was experienced by WHO when some women in Northern Nigeria started refusing vaccinations if they didn't get the in-kind donations that were previously being offered during Polio campaigns.” New Incentives, Responses to questions from GiveWell, September 23, 2020 (unpublished).
  • 51"Our Monitoring methodology and framework are guided by a theory of change with a consolidated list of indicators outlined in the Draft mCCT Log Frame_v2 [...] Annual Progress Reports will be shared with the donor. The reports will cover a narrative of activities, milestones achieved, plan deviations, learnings and adaptations, key indicators, and interim analysis on the effect of mCCTs on improving coverage rates. [...] Moreover, view access to the Government's EIR (ZM) dashboard will be provided to designated persons of the GiveWell team. This will provide real-time coverage data including summary statistics and visualizations, and insights at the provincial, district, town, UC, and facility levels." IRD, Proposal for scale-up of mCCT in Sindh (working draft), September 22, 2021, Pgs. 25, 32-33.
  • 52When we originally recommended the grant, we believed scale-up would be completed by July 2022. However, the grant agreement was signed a month later, updating our projection to August 2022.