University of Chicago — Evaluation of Mobile Conditional Cash Transfers (November 2021)

Note: This page summarizes the rationale behind a GiveWell Incubation Grant to the University of Chicago. In 2024, we recommended additional funding to support the evaluation, as we explain in the addendum. University of Chicago staff reviewed this page prior to publication.

Addendum added: May 2024

Published: August 2022

Background

The grant to the University of Chicago is to support the evaluation of a program implemented by another recent grantee, IRD.

In October 2021, IRD received a GiveWell Incubation Grant to run a mobile phone-based conditional cash transfer (mCCT) program in the seven highest-risk (characterized by low Penta-3 and Measles-1 vaccination rates) districts in Sindh.3 The program will provide caregivers who bring their infants in for routine vaccinations with cash incentives via mobile top-ups.4 The mCCT program will be rolled out over the course of one calendar year in the seven high-risk districts according to a randomly generated sequence.5 For more details on the October 2021 grant to IRD, see this page.

Planned activities

The evaluation team will undertake the following activities during the grant period:

• Assess the change in vaccination coverage caused by IRD's mCCT program, using data from the ZM platform.6 The evaluation will exploit the randomized, staggered roll-out of the mCCT program in seven districts to assess the impact of the program.7 The main outcome of interest for GiveWell is the number of children who are registered in ZM and receiving each vaccine in the routine immunization sequence,8 since this is a key parameter in our cost-effectiveness analysis of IRD's program.
• Conduct checks on the accuracy of ZM data and the implied coverage rates using phone surveys and in-person household surveys. It is possible that ZM data and government estimates of children over- or underestimates true coverage, a key input in our cost-effectiveness analysis. The University of Chicago evaluation team will check the accuracy of the ZM estimate of the number of children immunized and coverage rates via surveys of caregivers who are and are not enrolled in ZM.9
• Assess the expected mechanisms for increased uptake via a household survey and additional empirical analysis. The household survey will collect information on child age, immunizations received, knowledge of the incentives program, knowledge of immunizations, education, language, and other characteristics.10 The evaluation team will then test coverage rates of households or union councils (UCs) with different characteristics after the introduction of incentives.11
• Evaluate ways to improve the effectiveness of mCCTs through alternative communication approaches. The evaluation team will work with IRD to compare the effect on vaccination of four different approaches to sharing information about immunizations and the incentives program:
  • Identifying community members to serve as immunization "ambassadors" who inform others about the incentive program,
  • Varying content of SMS messages,
  • Using a WhatsApp bot to answer common questions and concerns about vaccination,
  • Sharing television ads that answer key questions about immunization.12

  These approaches will be randomized at the individual and community level, which should provide sufficient statistical power (though we have not requested power calculations).13 The effect on vaccination rates will be measured through vaccination coverage from ZM.14

• Test for negative or positive spillovers from the mCCT program. By increasing the number of children receiving vaccinations, the mCCT program may lead to negative spillovers (e.g., reducing health care workers' time spent on other activities) or positive spillovers (e.g., causing more children to visit clinics and receive other services beyond vaccinations).15 To estimate the extent of these spillovers, the evaluation team will measure the effect of the mCCT program on the activity of health workers and the distribution of preventative goods (such as family planning supplies).16 They will also attempt to measure effects on uptake of non-immunization health activities (e.g., caregivers bringing children to clinics when they suspect they have an illness or for regular health check-ups).17

The evaluation team will share both preliminary and final results with GiveWell and also plans to submit the resulting research for publication.18

Budget

The evaluation team has estimated a budget of $1,602,868.19 Costs are split across the University of Chicago and IRD. A breakdown is in this spreadsheet.

Case for the grant

We are recommending this grant because:

• The evaluation may provide an update to our best guess of the cost-effectiveness of IRD’s mCCT program and, in turn, change how much funding we recommend to it. We think IRD's mCCT program is promising and that we could recommend large amounts of funding to the program in the future. This evaluation provides a relatively inexpensive chance to potentially update our best guess on the cost-effectiveness of the program. This is driven by the following factors:
  • Large potential funding opportunities for similar programs in the future. Our best guess is that IRD's mCCT program is cost-effective, and we think it's possible we could recommend a substantial amount of funding to mCCT programs in the future. We have recommended a grant for IRD's program in seven districts in Sindh province. We estimate that these districts could absorb a total of $6 million annually. While we have not thoroughly reviewed additional funding opportunities, we guess that there is an additional $19 million annually in other districts in Sindh, Balochistan province in Pakistan, and other areas in Pakistan or beyond where IRD could potentially implement its program in the near term.20 As a result, if the evaluation updates our estimate of IRD's program, it could result in large changes in the amount of funding we recommend to these programs.
  • Low evaluation costs due to the use of ZM data. Because we will have real-time, individual-level data from ZM, we think that this evaluation will be relatively low cost compared to the information value, leading to higher cost-effectiveness.
  • Moderate uncertainty about the effect of mCCTs on vaccination rates in our current cost-effectiveness analysis. We estimate that IRD's mCCT and ZM program has a similar cost-effectiveness to opportunities to which we expect to direct marginal donations, based in part on our best guess that mCCTs increase coverage of vaccinations among infants in Sindh. However, this is based on two randomized controlled trials of programs that have several differences from the program being implemented by IRD during the grant period and the programs we may consider funding in the future.21 As a result, we have some uncertainty about the effect of mCCTs on vaccination rates in Sindh and other potential implementation settings. We think it's possible that seeing results from the evaluation could update our best guess of the cost-effectiveness of IRD's program.
  • Additional uncertainty about the reliability of ZM data, mechanism for effect on vaccination rates, and spillovers. The household survey will provide information on the mechanisms through which incentives increase vaccination rates, how reliable the ZM data are, and whether there are positive or negative spillovers from the program. This could influence how much confidence we have in the observed effect of mCCTs on vaccination rates and further update our assessment of cost-effectiveness.
  • Moderate confidence in the methodological approach. We view the evaluation methodology as moderately strong. We believe that exploiting the staggered roll-out of the program and ZM data pre- and post-mCCT roll-out should produce an estimate of the effect of mCCTs that we might put substantial weight on, though we do have some concerns (including limited statistical power and questions about the quality of the ZM data), which we discuss below.
  • External validity of evaluation findings to other areas for which we might provide funding. We also guess the evaluation will have sufficient external validity to these areas that we will see the findings as a meaningful update on our best guess for the effect of mCCTs in those areas.
• We also think the grant itself could uncover approaches to improve the cost-effectiveness of mCCTs. The evaluation will pilot several approaches for increasing the impact of mCCTs, such as varying the content of SMS messages and airing television ads to promote immunization alongside the program. If these are found to increase impact, they could be integrated into the program going forward, improving its cost-effectiveness.
• We have high confidence in the evaluation team. We have high confidence in the evaluation team, and Dr. Glennerster in particular, and believe they are likely to execute the planned activities. This is based on Dr. Glennerster's expertise in using randomized controlled trials to assess the effect of incentives for immunization22 and her reputation in development economics.23 We think this increases the likelihood the evaluation is successful at improving our estimate of cost-effectiveness and identifying ways to improve cost-effectiveness.

Based on this, we have built a back of the envelope model to assess the promisingness of this grant. This model incorporates our best guesses on how much we expect the evaluation to change our future funding decisions and cost-effectiveness of grants we recommend. The rough model indicates that this grant likely compares favorably to other giving opportunities this year. We view this as a rough check on where the value of this grant comes from and a way to make sure we've considered key drivers of value. We do not put a lot of stock in the specific cost-effectiveness estimate provided.

Risks and reservations

• There is a reasonable chance the evaluation will not provide an update on the effect of mCCTs, due to some limitations in the evaluation. Our specific concerns are about:
  Power. The small number of districts (seven) and quick roll-out (one district per month) limit statistical power. The evaluation team has provided power calculations showing that there will be "an 80% chance of distinguishing the impact of mCCT from zero if the true effect of the program was to reduce the number of children not receiving Penta-3 by 23 percent (equivalent to a 10.9 percentage points effect) and a 95% confidence interval was used."24 Our current CEA assumes a 9 percentage point effect on Penta-3. Our best guess of the effect of mCCTs (9 percentage points) is less than the minimum detectable effect for the trial (11 percentage points). As a result, we think it's more likely than not the results will be underpowered, even if there is a relatively large effect on vaccination rates.
  • More uncertain results in Year 1. We guess that the effect of mCCTs during the first few months of implementation will be smaller as the program "ramps up." We plan to incorporate this when we update our estimate of cost-effectiveness (i.e., assuming the effects of mCCTs are 25% lower during the ramp-up period than they will be after the ramp-up period). We also anticipate Year 1 results will have less power, since less data will have been collected by the end of Year 1 than by the end of Year 2. Both of these will give us additional uncertainty about Year 1 results and may cause us to update less based on first year results.
  • COVID-19 disruptions and other aspects of external validity. It's possible that changes in behavior by households (e.g., caregivers being less likely to leave their houses due to risk of infection) or clinics (e.g., additional outreach efforts beyond what is typical) as a result of COVID-19 mean that results have limited external validity post-COVID-19.
  • Other factors more meaningfully driving cost-effectiveness. The effect of mCCTs is one factor driving our cost-effectiveness analysis, but there are several others as well. It is possible other updates to cost-effectiveness due to factors beyond the effect of mCCTs may make any updates to the effect on mCCTs less meaningful.

  Even if there is a large chance this evaluation does not update our cost-effectiveness substantially, we think there is enough of a chance of a positive or negative update that this grant is cost-effective, though we have high uncertainty about this. We also think that some of the other aspects of the evaluation (e.g., testing different communication approaches, validating ZM data through household surveys) are less subject to these limitations.

• We may make some funding recommendations to IRD before we receive all results. While we expect to have initial results on the effect of mCCTs based on ZM data in late 2022, we will not receive full results until 2023, and we don't anticipate having results from the other evaluation components until 2024-2025. As a result, it's possible we'll make decisions on providing additional funding to IRD's program before we receive all results from the evaluation. We still think it's possible we will meaningfully update our understanding of the cost-effectiveness of supporting IRD, even based on initial results in 2022. In addition, because we think there are large potential funding gaps for IRD's mCCT programs over several years, there will still be an opportunity to update our view and, in turn, substantially change how much funding we recommend to IRD's mCCT programs, even if all results are not available for a few years.
• There may be limited funding gaps for similar programs by IRD. Our best guess is that there are enough funding gaps for IRD's mCCT program to make learning more about its cost-effectiveness valuable, but we're uncertain about funding opportunities for IRD's program beyond high-risk districts in Sindh. If opportunities beyond Sindh are limited, then the value of the grant will be limited as well.
• Publishing effects on vaccination rates that are positive but not statistically significant due to limited statistical power may be misinterpreted as evidence mCCTs don't work and reduce support for IRD's mCCT program and similar programs. We plan to mitigate this problem by being clear about any caveats, particularly lack of statistical power, when we communicate about the results.

Plans for follow-up

We plan to have approximately quarterly check-in calls with the evaluation team to discuss progress. We expect to receive results on the timeline listed above.

Forecasts

We think there is a 50% chance that we will deem the Year 1 results sufficiently conclusive that we put substantial weight (at least 50%) on the results in developing our best guess of the effect of mCCTs on vaccination rates by October 2022.

We think there is a 70% chance that we will deem the Year 2 results sufficiently conclusive that we put substantial weight (at least 50%) on the results in developing our best guess of the effect of mCCTs on vaccination rates by October 2024.

Our process

• In 2021, GiveWell investigated a grant to IRD to run an mCCT program in Sindh and recommended a grant in October 2021.
• As we concluded that investigation, we had several conversations with both IRD and Rachel Glennerster about pairing the program roll-out with an evaluation.
• We reviewed a proposal from Dr. Glennerster and asked for additional information and clarification on that proposal.

Sources

• 1

  This report from the evaluation team includes an assessment of 1) the impact of the randomized scale-up of IRD's program on immunization rates, and 2) any evidence of systematic fraud or inaccuracies in the electronic immunization registry data that IRD uses.

• 2

  More information on these studies can be found in these slides: Tailoring Deadlines to Improve Childhood Immunization Timeliness, and Information ambassadors with mCCTs program to increase immunization rates in Sindh, Pakistan.

• 3

  "The high-risk districts are categorized based on the 20th percentile of crude penta-3 and crude measles-1 vaccines district coverage rates (Figure 2). There are 6 districts (20.0%) below the 20th percentile for penta-3 vaccine (coverage of 55.86%) which include Hyderabad, Karachi East, Karachi West, Karachi Central, Jacobabad, and Sujawal. Additionally, there are 6 districts (20.0%) below the 20th percentile for measles-1 vaccine (coverage of 40.87%) including Karachi Central, Karachi East, Karachi West, Jacobabad, Kambar, and Sujawal. The 7 high-risk districts below the 20th percentile for crude penta-3 and measles-1 coverage are, therefore: Karachi Central, Karachi East, Karachi West, Hyderabad, Sujawal, Jacobabad, and Kambar." IRD Global, Proposal for scale-up of mCCT in Sindh (working draft), September 22, 2021.

• 4

  "Mode of disbursement: Incentives will be disbursed as mobile airtime that can be transferred within 0-48 hours of the vaccination visit." IRD Global, Proposal for scale-up of mCCT in Sindh (working draft), September 22, 2021.

• 5

  The seven districts and timeline for roll-out is in this spreadsheet, provided by IRD. According to this plan, mCCTs will be rolled out in all seven districts by July 2022.
  "The seven districts which have been chosen to receive the mCCT program (based on their baseline levels of immunization) will be phased into the program based on a randomized schedule with one district per month entering the program for a period of seven months." @University of Chicago team, Evaluation plan for Immunization Incentive Roll Out in Sindh Pakistan, 2021@.

• 6

  "Data generated by the ZM electronic database will be the primary source of data to evaluate the impact of the program." @University of Chicago team, Evaluation plan for Immunization Incentive Roll Out in Sindh Pakistan, 2021@.

• 7

  "To evaluate the causal impact of the ZM program on coverage and timeliness, we will use a randomized stepped wedge design. The seven districts which have been chosen to receive the mCCT program (based on their baseline levels of immunization) will be phased into the program based on a randomized schedule with one district per month entering into the program for a period of seven months. The evaluation team will conduct the randomization. Data generated by the ZM electronic database will be the primary source of data to evaluate the impact of the program. A panel estimating equation will be used to identify the impact utilizing ZM data from before the introduction of the program, during the phase in period and post phase in." @University of Chicago team, Evaluation plan for Immunization Incentive Roll Out in Sindh Pakistan, 2021@.

• 8

  "The three main outcomes of interest are: number of additional children receiving at least one vaccination as a result of the program; persistence through the vaccination schedule; and timeliness by which children receive their vaccines. The first will be measured by estimating the increase in the number of children registered in the ZM system (children are registered when they are brought in for their first shot). Coverage rates by vaccine or the average number of vaccines received by child speak to the second objective. Note however, that an increase in more marginalized children coming into the system can bring down the average number of vaccines received per child even while the total number of vaccines increases. One way to address this is to estimate the total vaccines delivered by district for each vaccine. The same challenge arises with respect to measuring timeliness: timeliness of those who would have been in the system anyway could increase while new entrants could bring down the average receiving their vaccines in a timely way. Again, estimating the increase in the number of children receiving their vaccinations in a timely way solves this potential bias." @University of Chicago team, Evaluation plan for Immunization Incentive Roll Out in Sindh Pakistan, 2021@.

• 9

  "1.1A Phone-based verification surveys: As part of the active monitoring, these surveys will be conducted by the helpline operators and will include questions for caregivers on whether their child received a vaccine, whether they received an incentive, what was the incentive transfer time, whether they were asked for a tip and whether there were supply issues e.g. long waiting time, stock-outs etc. at the center. These calls will be conducted by 8 helpline operators. Each operator will be conducting around 550 calls per month (25 calls/day) which would cover around 3.2% (4,400/137,316 visits) of the total expected immunization visits in a month. Any areas of concern highlighted during these surveys, including discrepancies reported between the incentive status and the transfer time as informed by the caregiver, caregivers reporting to have paid 'tips' to vaccinators (overtly or covertly), vaccinators offering to collude to increase incentive amount, vaccinators threatening to withhold incentive for any reason, or supply issues at the immunization center, will be logged on the ticketing system*." @IRD Global, Monitoring plan for scale-up of mCCTs in Sindh (working draft), 2021@.
  "We would use inperson interviews to check whether what ZM reports is accurate. This would include checking whether those ZM reports as having been vaccinated were actually vaccinated, as well as checking for people who may be vaccinated but are not in the ZM database (e.g. from a door to door campaign). Note that we can do the first of these using phone interviews for those with a phone but not the second." Rachel Glennerster, Email to GiveWell, November 3, 2021 (unpublished).

• 10

  "They would be asked about: child age, immunization received, incentives received, whether older children in the family had been immunized, knowledge of the incentive program and of immunizations. Basic socioeconomic indicators would be collected including income proxies (not a full income survey), education, language, and settlement status (i.e., are they refugees or migrants)." @University of Chicago team, Evaluation plan for Immunization Incentive Roll Out in Sindh Pakistan, 2021@.

• 11

  "By collecting socioeconomic data on locations and comparing those to areas that saw a big increase in vaccination with the introduction of mCCT, and by analyzing the characteristics of those who vaccinated their youngest child but did not vaccinate older children it will be possible to give some information about the type of children who got vaccinated as a result of the mCCT program." @University of Chicago team, Evaluation plan for Immunization Incentive Roll Out in Sindh Pakistan, 2021@.

  "One of the key benefits of the household survey is to get a better handle on why people are not getting vaccinated. How much is it misinformation, how much just time/travel cost, how much procrastination. This is critical for understanding whether the program will work elsewhere. Our hypothesis is that a small nudge helps people who are not strongly opposed to vaccination but only slightly unsure or face travel and time costs and procrastinate. If we see a very big difference in the types of people who are not getting vaccinated at all (even after the incentive) and those who are coming in for at least some vaccination then we need to look at whether those types of people are more common in other parts of Pakistan. We can also get a handle on how this might replicate to other parts of Pakistan by looking at how much the issue in those areas is people dropping off over the schedule and how much its ppl getting no vaccines, and whether the mCCT program helps on the no vaccine group (something we have not tested yet). The latter does not require a household survey." Rachel Glennerster, Email to GiveWell, October 2, 2021 (unpublished).

• 12

  "Four different add-ons to the mCCT will be piloted and tested:

  1. Community influencer: Caregivers would be asked for the names of people in the community who would be good at getting the message out about the CCT program. LHW would be asked to help track these people to see if they would be willing to act as immunization "Ambassadors". These people would be sent a text explaining the program and asking them to inform caregivers in the community about the program. [...]
  2. SMS messages: Varying the content of SMS messages drawing on lessons from behavioral economics studies. The SMS messages will be sent [by] IRD (or another legitimate/trusted source) encouraging parents to get their children vaccinated -- using reminders, nudges, social proof, and other short texts. [...]
  3. WhatsApp bot: The WhatsApp bot is a basic program that answers common questions/ concerns about vaccination (e.g., where is the closet clinic, is it free, how old my child should be, are vaccines safe, etc.). [...]
  4. Adverts in the media: More content filled pieces answering key questions about immunization would be delivered through cable/ local TV channels and radio stations randomized at the lowest possible geographic level."

  @University of Chicago team, Evaluation plan for Immunization Incentive Roll Out in Sindh Pakistan, 2021@.

• 13

  "Yes, we will randomize at lower levels than district level so will have sufficient power. The plan is to do some community level randomization and some with individual-level randomization." Rachel Glennerster, Email to GiveWell, November 3, 2021 (unpublished).

• 14

  "Outcomes for this part of the evaluation would be vaccination coverage (from ZM), knowledge about the program (from the household survey)." @University of Chicago team, Evaluation plan for Immunization Incentive Roll Out in Sindh Pakistan, 2021@.

• 15

  "One concern with any program that seeks to promote one activity by individuals and health workers is that it reduces the effort spent on other health activities. . . . By introducing children into the health system there may be positive spillovers of the mCCT program." @University of Chicago team, Evaluation plan for Immunization Incentive Roll Out in Sindh Pakistan, 2021@.

• 16

  "We will seek access to data on the activity of LHWs and distribution of preventative goods (including family planning supplies) in an effort to track possible positive spillovers by releasing LHW time to spend on these important activities." @University of Chicago team, Evaluation plan for Immunization Incentive Roll Out in Sindh Pakistan, 2021@.

• 17

  "Yes - we will look at whether they are more likely to sign-up for other non-immunization health activities (e.g. bring their kids to clinic for regular health checks, or when they have diarrhea). We will need double check exactly what is the best measure to do this." Rachel Glennerster, Email to GiveWell, November 16, 2021 (unpublished).

• 18

  "Do you plan to publish the findings from this research? "Yes." Rachel Glennerster, Email to GiveWell, September 30, 2021 (unpublished).

• 19

  The full budget is in this spreadsheet.

• 20

  Our guesses on funding gaps are in this spreadsheet.

• 21

  The program evaluated in this evaluation has key differences from that program that IRD will implement in Sindh:

  • The trial enrolled and notified caregivers about the mCCT program at early vaccine visits. IRD Global, Korangi trial methods, 2021 (unpublished), p. 4. As a result, the trial has limited ability to detect effects on uptake of vaccines early in the sequence. These early vaccines drive cost-effectiveness, so high uncertainty about effects on these vaccines means high uncertainty about cost-effectiveness.
  • Because infants were notified at enrollment, the effect from the trial is based on those who would have been enrolled in ZM. We have high uncertainty about how the effect will generalize to those who are unenrolled. Our best guess is that they will have a similar effect, but we are unsure.
  • The program implemented during the grant period will include communications activities to inform caregivers about mCCTs before early vaccine visits. We anticipate this will increase cost-effectiveness, but we have high uncertainty about the magnitude.

  More detail is in our intervention report on IRD's program.

• 22

  Dr. Glennerster's research on incentives for immunization includes Banerjee, Duflo, and Glennerster 2010 and the Korangi trial.

• 23

  Dr. Glennerster's works have been cited thousands of times by other scholars. See Dr. Glennerster's Google Scholar profile here.

• 24

  @University of Chicago team, Evaluation plan for Immunization Incentive Roll Out in Sindh Pakistan, 2021@.