IDinsight — Endline Evaluation of New Incentives RCT

Published: January 2020

Note: This page summarizes the rationale behind a GiveWell Incubation Grant to IDinsight. Staff from IDinsight and New Incentives reviewed this page prior to publication.

The case for the grant

New Incentives has been a GiveWell Incubation Grant recipient since 2014. Its current program, which began in late 2016, provides conditional cash transfers (CCTs) to parents of infants who receive routine vaccines1 at clinics in Northwest Nigeria, where infant vaccination rates are low.2

Over several years of working with New Incentives, we have been impressed by its implementation competency, monitoring data, willingness to pivot based on expected cost-effectiveness,3 commitment to ensuring that this RCT is well-designed,4 and the breadth and depth of its engagement with our questions. New Incentives has reported strong operational success to us, including higher than expected volumes of caregivers participating in its program.5

It appears plausible to us that IDinsight's RCT of the program will show strong enough impact for us to recommend New Incentives as a top charity. The amount of funding necessary to complete the RCT at this stage is fairly small relative to the project's total costs, and we believe the results will likely allow us to understand whether or not New Incentives' program is effective at increasing immunization coverage.

As part of this project, IDinsight has conducted a baseline survey and midline evaluation. Findings from both were consistent with the possibility that the RCT will end up finding the program to be effective. Specifically:

• IDinsight's baseline survey for the RCT found low rates of immunization and low rates of caregivers reporting strongly-held anti-vaccination views (more).
• IDinsight conducted a midline evaluation to provide some limited information that would update us about the risk of the program having no impact; we generally consider the results of the midline evaluation to be a positive indication about the program's effectiveness, though with low confidence (more).

The fact that this project has now reached its final stage after several years of co-development with our partners also factored into our decision to make this grant in two important ways. First, though we have made significant past investments in this project, our current CEA estimate for the value of this grant only projects remaining future costs, and this contributes to the grant looking relatively cost-effective. Second, we put weight on the qualitative consideration that declining to make a grant at this late stage in the project would have immediate major negative consequences for our partners and would likely harm GiveWell's ability to form partnerships in the future.

We also believe this RCT might be somewhat valuable for learning about the effectiveness of immunization incentives in general. We conducted a literature search and found very limited existing evidence about the impact of incentives for immunization in sub-Saharan Africa. However, we believe the value of the information produced by the RCT will depend largely on how widely the results are read and applied outside of GiveWell, and we base the case for this grant primarily on the internal value of the RCT results for GiveWell's evaluation process.

Due to being the only funder of this project and its direct relevance to our future funding decisions, we have had a substantially deeper level of engagement in the design and process of this RCT than we ever have before. We expect a secondary benefit of this grant to be its learning value for us about funding projects of this kind more broadly.

Planned activities and budget

The total projected cost of the endline RCT is $1,955,436, including all fieldwork and IDinsight staff costs.6 IDinsight requested a smaller initial disbursement from this grant of $1,666,112 because it has some residual funds available.7

As part of the structure of this grant, IDinsight may also request contingency funding of up to $439,850 as unexpected costs arise, for a total project cost of up to $2,395,286. IDinsight has agreed to return any unused funds at the end of the project.8

We expect IDinsight to complete the following activities by roughly June 30, 2020:

• Conduct the endline pilot and make final recommendations for implementing the endline survey.
• Conduct the endline survey.
• Produce a written endline report on the impact of New Incentives' program on vaccination rates as measured by the RCT.

We expect this grant to be the final funding recommendation to IDinsight for the New Incentives RCT before we determine whether New Incentives should be a top charity.

Results from previous stages of the RCT

Baseline results

Before New Incentives began its program, IDinsight conducted a baseline survey of immunization rates in the study context, as well as of attitudes toward and barriers to immunization. The survey found that baseline immunization rates were low,9 there was little evidence of strong caregiver objections to immunization,10 and lack of knowledge was the primary reason cited for not immunizing.11 All of these results updated us positively on New Incentives' scope for impact.

Midline results

IDinsight recently shared with us preliminary results from a midline survey of clinics enrolled in the RCT. This survey was designed to provide some limited information to update us about the potential risk of the program having no impact. If the midline data had shown a small effect or no effect, we would have considered this a warning sign that the program was likely ineffective, particularly because we expect the midline data to overestimate New Incentives' impact (see footnote for more).12

The midline survey found that the number of recorded vaccinations more than doubled in New Incentives' clinics while remaining generally unchanged in control clinics.13 The occurrence of peaks and valleys in the administrative data on immunization volumes also matches fairly well to the timing of New Incentives' program rollout.14 While we don't take these large observed effects at face value (for the reasons described in the footnote above), this change is large enough to increase our confidence that New Incentives is having at least some effect on immunization rates. Overall, we consider the midline survey results to be a positive update on the program's effectiveness.

However, these midline results could also be consistent with relatively small program impacts. In addition, the midline results are only an indicator for the first half of the study treatment period, and program impact could change between midline and endline. We used data from the baseline and midline surveys to create a range of estimates for possible program impact up to the midline. Our post-midline best guess is that New Incentives has increased measles vaccination rates by about 9 percentage points,15 with low-end to high-end estimates ranging from 3 to 13 percentage points,16 and other vaccination rates by about 13 percentage points,17 with low-end to high-end estimates ranging from 4 to 21 percentage points.18

The midline results make use of randomized treatment and control groups but are based on clinics' administrative records of immunizations.19

These data are not comparable to the higher-quality community data we intend to collect in the endline survey, and we expect the endline results to supersede the midline results in our final analysis.

Cost-effectiveness

We produced two different kinds of cost-effectiveness analyses for this grant: an estimate of the cost-effectiveness of New Incentives' incentives for immunization program (more), and an estimate of the expected cost-effectiveness of making this grant (more).

Value of the program

Our current best guess is that New Incentives' program is roughly 5x as cost-effective as cash transfers (based on reasoning described below). However, this estimate is highly uncertain, and a wide range of potential cost-effectiveness levels for this program seem plausible to us, including higher levels of cost-effectiveness.

There are currently two major sources of uncertainty in our cost-effectiveness estimate:

1. Lack of endline results from the RCT. As described above, the confidence intervals around the estimates in our current best-guess cost-effectiveness analysis (CEA) are wide, and we expect endline results to supersede midline results in our final analysis.
2. Lack of confidence in our cost-effectiveness model. Our current best guess of New Incentives' cost-effectiveness is derived from two semi-independent CEA modeling exercises that generated substantially different cost-effectiveness estimates for this intervention:
   • At the beginning of this project, we created our main CEA. This model is much longer than the typical GiveWell CEA, complex, and somewhat outdated (the latest version is from early 2018).
   • In preparation for making this grant, we created a new CEA. While this new model is quite rough and deliberately simplified, we put some weight on it because the relative simplicity reduces the likelihood of conceptual and calculation errors relative to the older, very complex model.

   The old CEA suggests that New Incentives' program is roughly 4x as cost-effective as cash,20 while the new CEA suggests that it's roughly 8x as cost-effective as cash.21 We put 70% weight on the old CEA model and 30% weight on the new CEA model, yielding a current cost-effectiveness estimate of 5x cash.22

Due to our lack of confidence in how comparable these cost-effectiveness estimates are to our cost-effectiveness estimates for our top charities, the case for this grant relies to a large degree on qualitative arguments about finishing a project that we believed to be valuable when we began funding it and which has continued to show promising signs (as mentioned above). We expect to make substantial changes to our cost-effectiveness model of New Incentives' program as part of our future investigation into potentially recommending New Incentives as a top charity.

While we are quite unsure how changes to our model will affect our cost-effectiveness estimate, we think it's unlikely that our cost-effectiveness estimate will fall enough upon review to rule out the chance that New Incentives' program could be competitively cost-effective within a reasonable range of RCT results. In preparing to make this grant, we did a limited review of our old CEA to try to identify and conduct sanity checks on key inputs that have a large impact on our cost-effectiveness estimate.

Value of this grant

Our value-of-the-grant model for this project suggests that additional funding for the New Incentives RCT is roughly 10x as cost-effective as cash at this stage.23 This model is highly dependent on assumptions about New Incentives' future room for more funding should it become a GiveWell top charity.24

This value-of-the-grant model is somewhat conservative, since it includes a fairly generous estimate of the potential costs of an additional grant to New Incentives to extend its program until we're able to make a decision about recommending it as a top charity.25 The model also doesn't account for any direct benefits of funding New Incentives' program beyond facilitating the RCT, though (as discussed above) we currently estimate that the program itself is about 5x as cost-effective as cash.

This model only counts endline costs; it doesn't reflect the all-in costs of the New Incentives project since its inception. While we believe that this is the most useful way of framing the value of our current decision to proceed with endline data collection, our value-of-the-grant estimate is not directly comparable to value estimates for early-stage projects for this reason.

Risks and reservations

• The most problematic outcome of this project from a learning perspective would be inconclusive RCT results (i.e. results that leave us with significant remaining uncertainty about whether to make New Incentives a top charity). While we predict that the RCT results will likely be conclusive, inconclusive results are possible if the program's impacts end up being lower than we expect.26
• We currently plan to collect saliva biomarkers during the endline pilot to corroborate caregiver reports of immunizations received. We expect this to update us on the reliability of biomarker tests for measles immunity in a Nigerian context, and we expect to use this information to determine if collecting biomarkers during the endline survey will be worth its cost.

See also the risks and reservations described in our write-up of the November 2017 grant to New Incentives, most of which still hold true.

Plans for follow-up

GiveWell expects to complete a top charity review and make a decision about whether to recommend New Incentives as a top charity by November 2020.

Internal forecasts

For this grant, we are recording the following forecasts:

Confidence	Prediction	By time
65%	New Incentives is a top charity and is ≥ 3x as cost-effective as cash	November 2020
50%	New Incentives is a top charity and is ≥ 5x as cost-effective as cash	November 2020
22.5%	New Incentives is a top charity and is ≥ 7.5x as cost-effective as cash	November 2020
5%	New Incentives is a top charity and is ≥ 10x as cost-effective as cash	November 2020
15%	The RCT results are inconclusive, such that after seeing them we have significant uncertainty about whether to make New Incentives a top charity	November 2020
25%	We cite our learning experience from the New Incentives RCT as part of our reasoning for funding a future RCT (including any RCTs related to current GiveWell Incubation Grant recipients, such as Evidence Action's Beta Incubator)	December 2024
5%	The New Incentives RCT results are cited by another funder or agency when making a recommendation for or against pursuing CCTs for immunization, or as a reference in future research	December 2024

Our process

We have made a number of previous grants27 to both New Incentives and IDinsight in relation to this project and have been working with both organizations for several years. We have been preparing to make this endline grant since early 2019. Since that time, we have engaged in regular phone calls with IDinsight, New Incentives, and an outside expert that works with New Incentives to discuss major decisions for endline.

We've also spent a limited amount of time investigating the results from the first draft of the midline survey that IDinsight recently shared with us, evaluating the progress of the project so far, updating our best guesses of the impact and cost-effectiveness of this program (though we plan to make much more substantial changes to our cost-effectiveness model in the future), and reevaluating our priors on how likely New Incentives is to become a top charity.

Sources

Document	Source
GiveWell, Best guess effect size for New Incentives, June 2019	Source
GiveWell, Endline Grant Updated Version of New Incentives CEA, early 2018	Source
GiveWell, New Incentives CEA, June 2019	Source
GiveWell, Studies of incentives for immunization	Source
GiveWell, Value of the grant model for New Incentives Endline RCT	Source
IDinsight, New Incentives Evaluation Baseline Report	Source
IDinsight, New Incentives Evaluation Budget Comparisons	Unpublished
IDinsight, New Incentives Midline Report	Source
New Incentives, CCTs for Immunizations Proposal	Source

• 1

  These vaccines include:

  • Measles
  • BCG (bacille Calmette-Guerin) against tuberculosis
  • Pentavalent vaccine, doses 1, 2, and 3
  • Pneumococcal conjugate vaccine (PCV), doses 1, 2, and 3

  See New Incentives, CCTs for Immunizations Proposal, p. 1: "While New Incentives incentivizes all vaccination visits it… [aims to] focus transfer conditions on a few highly cost-effective vaccinations (BCG, Penta, PCV, and Measles)."

• 2

  "Our survey found that routine immunization coverage across Katsina and Zamfara is low. A third of 12 to 16-month olds (33.6%, 95% confidence interval (CI): 32.2%, 35.0%) have received at least one injectable vaccine… and only 10.2% (95% CI: 9.1%, 10.9%) of 12 to 24-month olds are fully immunized..." IDinsight, New Incentives Evaluation Baseline Report, p. 30

• 3

  Most significantly, in 2016, New Incentives pivoted from a program that used cash transfers to incentivize women in Nigeria to deliver in a health facility to its current program of incentives for immunization. See the write-up of our 2016 grant to New Incentives for more details.

• 4

  For instance, New Incentives has spearheaded both a) engaging an outside expert to vet our decisions and b) coordinating a shared project timeline to ensure that all decisions are made on time and reviewed.

• 5

  This is our impression based off of a) monitoring a dashboard maintained by New Incentives and b) projected increases in New Incentives' monthly program costs due in part to an increase in expected volume of transfers.

• 6

  IDinsight, New Incentives Evaluation Budget Comparisons

• 7

  The residual funds include the amount of this grant, as well as unused funds that remain from this previous grant.

• 8

  IDinsight, New Incentives Evaluation Budget Comparisons

• 9

  "Our survey found that routine immunization coverage across Katsina and Zamfara is low. A third of 12 to 16-month olds (33.6%, 95% confidence interval (CI): 32.2%, 35.0%) have received at least one injectable vaccine… and only 10.2% (95% CI: 9.1%, 10.9%) of 12 to 24-month olds are fully immunized..." IDinsight, New Incentives Evaluation Baseline Report, p. 30

• 10

  "Mistrust or fear of vaccination... was the most infrequently cited reason for not vaccinating children – only 5.5% [of caregivers surveyed] cited medical reasons such as fears of side effects, bad reactions to previous vaccinations, or a fear of needles. This low percentage indicates that allaying fears around side effects may not be a crucial element to include in awareness campaigns." IDinsight, New Incentives Evaluation Baseline Report, p. 50

• 11

  "Lack of knowledge was the primary reason for caregivers (53.2%) not vaccinating their children. The lack of knowledge category includes reasons such as not knowing where and when to get vaccines, as well as misconceptions like not needing to vaccinate healthy children." IDinsight, New Incentives Evaluation Baseline Report, p. 49.

• 12

  The main reasons we expect the midline results to overstate the actual effects of New Incentives' program are:

  • New Incentives' program may cause clinics to improve their recordkeeping around immunizations. We think it's possible that some of the observed increase in immunizations may reflect an increase in immunizations recorded rather than in the actual volume of immunizations taking place.
  • Immunization incentives might attract an unusually large number of patients from outside of treatment clinic catchment areas, leading to increases in out-of-catchment vaccinations that could distort the data. Because the endline survey will only reach people who live within treatment clinic catchments, we don't expect this effect to exist at endline (the experiment was designed to leave a buffer area around most treatment clinic catchments to reduce the likelihood that control households attend treatment clinics).

• 13

  "Our analysis indicated that recorded volumes of incentivized vaccinations in both Tally Sheets and CIRs in period 3 were roughly double what we would have expected in the absence of treatment... a doubling in the recorded volume of vaccines exceeds the maximum increase that could result from improved record keeping alone." IDinsight, New Incentives Midline Report, p. 15

• 14
  • See IDinsight, New Incentives Midline Report, p. 13, Figure 1.
  • "Recorded volume in control clinics... is effectively constant over time. By contrast, recorded volume in treatment clinics… spikes during the eight months of program ramp-up (November 2017–June 2018; middle sections of graphs). This change stabilizes at a lower – but still high – level in the operational period (July 2018–January 2019; right-most sections of graphs)." IDinsight, New Incentives Midline Report, p. 13

• 15

  GiveWell, Best guess effect size for New Incentives, June 2019, cell B6

• 16

  GiveWell, Best guess effect size for New Incentives, June 2019, cells B44 and B37

• 17

  GiveWell, Best guess effect size for New Incentives, June 2019, cell C6

• 18

  GiveWell, Best guess effect size for New Incentives, June 2019, cells B45 and B38.

• 19

  "The primary goal of midline was to get an early and rough indication of New Incentives’ impact using vaccination volumes recorded in clinic administrative records. We found that the number of recorded vaccinations was statistically significantly and meaningfully higher in treatment than control facilities…" IDinsight, New Incentives Midline Report, p. 5.

• 20

  GiveWell, Endline Grant Updated Version of New Incentives CEA, early 2018, "New Incentives" sheet, cell B15

• 21

  GiveWell, New Incentives CEA, June 2019, "Results" sheet, cell B4

• 22
  • The old model estimates New Incentives' program to be 3.65x as cost-effective as cash transfers. GiveWell, Endline Grant Updated Version of New Incentives CEA, early 2018, "New Incentives" sheet, cell B15
  • The new model estimates New Incentives' program to be 8.2x as cost-effective as cash transfers. GiveWell, New Incentives CEA, June 2019, "Results" sheet, cell B4
  • (.7 x 3.65) + (.3 x 8.2) ≈ 5

• 23

  GiveWell, Value of the grant model for New Incentives Endline RCT, cell B38

• 24

  Our current model assumes New Incentives will have $20 million in annual room for more funding. See GiveWell, Value of the grant model for New Incentives Endline RCT, row 12.

• 25

  See GiveWell, Value of the grant model for New Incentives Endline RCT, row 4.

• 26

  The New Incentives RCT will have 80% power to detect effects as small as 9-10 percentage point increases in immunization rates, consistent with our expectations based on early design decisions. In order to achieve power to detect smaller effect sizes reliably, we would have had to enroll more clinics (our treatment clusters) in the study, which would have stretched the logistics of New Incentives' program scale-up and threatened our ability to leave buffer areas between treatment and control clinics. As a result, we decided that the current scale of the program represents a good balance between these costs and our ability to obtain information about the range of outcomes for which we believe the program is likely to be cost-effective.

• 27
  • To New Incentives: November 2017, April 2017, and November 2016
  • To IDinsight: October 2016, May 2017, April 2018, and February 2019