New Incentives — Nigeria Expansion and Extension (May 2023)

Note: This page summarizes the rationale behind a GiveWell-recommended grant to New Incentives. New Incentives staff reviewed this page prior to publication.

Published: July 2023

Summary

What we think this grant will do

With this grant, New Incentives will provide cash transfers (of up to approximately $11)1 to caregivers who bring their infants to clinics for routine vaccinations. New Incentives also conducts outreach to caregivers about the importance of vaccinating children, and works with its government partners to improve vaccine supply by identifying and addressing bottlenecks in the vaccine supply chain.

This grant, along with approximately $19.6 million applied from prior grants, will support the program's expansion to up to 47 additional local government areas (LGAs) in Nigeria and extend support by another year (through the end of 2025) for existing operations in 104 LGAs. By the end of 2023, New Incentives expects to have expanded to all LGAs in 11 northern Nigerian states that meet GiveWell's cost-effectiveness bar and are feasible to operate in.

We think this grant will lead to an increase in the number of infants receiving routine vaccinations, which will, in turn, lead to reduced mortality, increased income in adulthood, and other benefits.

For more about what we think this grant will do, see the "Planned Activities" section below.

Why we made this grant

• We believe that New Incentives' program is highly cost-effective.2 Intuitively, this is because we think:
  • New Incentives' program increases the number of children who receive routine vaccinations. We believe low vaccination rates in northern Nigeria are partly a result of demand side barriers to vaccination (e.g., opportunity costs of taking children to clinics, low perceived benefit of vaccination, etc.). We think New Incentives' program will help address these barriers by providing cash transfers and raising awareness about the benefits of vaccination. Based on the results of a randomized controlled trial (RCT) of New Incentives' program, we estimate the program increases vaccination rates by approximately 22 percentage points.3
  • Routine vaccinations reduce the likelihood of contracting, and dying from, vaccine-preventable diseases. We estimate that the vaccines incentivized by New Incentives' program reduce the risk of contracting the diseases targeted by those vaccines by 46% to 85%, depending on the vaccine.4 We assume that this reduction in disease incidence results in a similar reduction in associated mortality.5
  • Without vaccinations, many children in Nigeria could die from vaccine-preventable diseases. We estimate that unvaccinated children under the age of five have a 1.6% to 3.9% chance of dying from vaccine-preventable diseases in the states covered by this grant.6
  • While the per-child cost of this program is higher than many other programs we fund, the benefit per child enrolled is also very high. This program provides cash transfers to caregivers, a cost not incurred by many other programs we fund. It also relies on a large network of in-country staff to disburse the cash transfers. Our most recently published cost-effectiveness analysis estimated a cost per child of $24.85 for New Incentives' program,7 compared to a cost per child of approximately $6 to $7 for Malaria Consortium's seasonal malaria chemoprevention program. Yet we estimate that New Incentives' program is above our cost-effectiveness bar in many areas because we believe the cash transfers result in a significant increase in vaccination rates and lives saved, and thus result in a comparable amount of benefits per dollar spent.

  A sketch of our most recently published cost-effectiveness analysis (as of June 2023) is below. We use Bauchi as an example of cost-effectiveness in one of the five states where New Incentives has fully expanded its program (i.e., New Incentives is operating in all areas where security allows).

  	Bauchi
  Grant size (arbitrary)	$100,000
  Child mortality benefits
  Cost per infant vaccinated in areas served by New Incentives	$24.85
  Number of infants vaccinated in areas served by New Incentives	4,236
  Percentage of infants in areas served by New Incentives who would be vaccinated without the program	39%
  Increase in vaccination rates from New Incentives' program (percentage points)	22%
  Percentage of vaccinated children who would have been unvaccinated without New Incentives' program	36%
  Probability that unvaccinated children die from vaccine-preventable diseases before their 5th birthday	3.7%
  Adjusted effect of vaccines on vaccine-preventable diseases	76%
  Deaths averted in children under 5 as a result of the program	43
  Cost per under 5 death averted	$2,333
  Moral weight for each under 5 death averted	116
  Cost-effectiveness from under-5 mortality reduction only (x cash)	15
  Primary benefit streams (as % of overall effect)
  Under-5 mortality effects	70%
  Over-5 mortality effects	20%
  Development benefits	7%
  Consumption benefits	2%
  Additional upside and downside adjustments	105%
  Charity-level adjustments	93%
  Intervention-level adjustments	133%
  Downstream costs incurred as a result of GiveWell funding (leveraging)	93%
  Chance that the government would fund this or a similar program in our absence (funging)	91%
  Final cost-effectiveness, after all adjustments (x cash)	22

  This is calculated as follows: ($100,000 / $24.85) / 95%.

  We expect that the majority of vaccinated infants (95%) in areas served by New Incentives will be enrolled in New Incentives' program. We factor this 95% into our calculation to account for the small proportion of infants that might be vaccinated from other sources.

  We estimate 39% of children would have been vaccinated without New Incentives, and we estimate New Incentives' program increases vaccination rates by 22 percentage points, so we estimate 61% of children will be vaccinated. This implies that 36% of those vaccinated would have been unvaccinated without the program.

  22%/61% = 36%

  Calculated as: $100,000 / 43

  We use moral weights to make cost-effectiveness comparisons between interventions achieving different types of outcomes (e.g., averting the death of a child from malaria vs. doubling consumption for a year for a very poor household). See this page for a discussion of how GiveWell uses moral weights.

  Reducing deaths of children under the age of five is the largest source of benefits we model for New Incentives' program, but we believe the program has other benefits as well. This section breaks down how much different benefit streams contribute to the overall effect of the program. Note that this section only includes benefit streams we directly model. We adjust for additional benefits and downsides to the program in the following section.

  We believe vaccines can confer immunity from vaccine-preventable diseases beyond the age of five, though we expect fewer deaths are averted in older age groups than in children under the age of five.

  We would guess that vaccination in infancy may have "development benefits" — i.e., early-life health improvements from vaccines might lead to improvements in income when vaccinated children become adults. While we have not reviewed direct evidence for development benefits from vaccination in infancy, we have included these benefits to be consistent with our cost-effectiveness evaluations of other interventions that improve health early in life.

  New Incentives provides cash transfers to caregivers who bring their children to clinics for routine vaccinations. We expect those cash transfers can modestly improve a household's quality of life, through increased consumption.

  We believe there are likely additional benefits and downsides of New Incentives' program beyond those modeled above. This section summarizes our best guess of how those additional considerations contribute to the overall impact of New Incentives' program.

  Adjustments of 100% indicate we believe those additional factors reduce the overall impact of New Incentives' program, while adjustments >100% indicate additional factors that we believe increase the overall impact of the program.

  It's often difficult to explicitly model different kinds of wastage or other charity-level issues directly in our cost-effectiveness analysis. These adjustments take into account our best guess of how different charity-level factors (e.g., false monitoring results, possible changes in charity priorities, etc.) affect our best guess of true cost-effectiveness.

  We only directly model some of the possible effects of the intervention. Many effects, such as averted treatment costs, are excluded to keep the model tractable.

  This row sums the speculative guesses we use to estimate the impact of outcomes we haven't explicitly modeled. The "Supplemental intervention-level adjustments" tab of our cost-effectiveness analysis includes more details on how these adjustments are calculated and some brief justifications.

  "Leverage" refers to GiveWell funding causing other entities to spend more on the program than they otherwise would have. We assume this funding would have been spent on programs that create value. We account for this by subtracting the counterfactual value of spending caused by GiveWell funding.

  In this case, we think a grant to New Incentives will cause Gavi and the government of Nigeria to devote resources to support routine childhood vaccinations. We think that without our funding, Gavi and the government of Nigeria would devote their resources to other health programs. Our downward adjustment reflects the funding that Gavi and the government of Nigeria spend as a result of GiveWell's funding and the value that would have been created by programs they otherwise would have supported with that funding.

  "Funging" refers to GiveWell funding causing other entities to spend less on the program than they otherwise would have. See this blog post for a description of our approach to leverage and funging.

  Calculated as: (15 / 70%) * 105%

  Since our initial cost-effectiveness estimate of 15x cash only incorporates mortality reduction in children under the age of five, we divide that estimate by 70% to account for the additional benefits from other benefit streams we model (mortality reduction in older individuals, development benefits, and consumption benefits).

  We then multiply by our total estimate for additional upside and downside adjustments, to account for the impact of other factors not directly modeled.

  You can see our full cost-effectiveness analysis and simple version here. [August 2023 Update: The updated cost-effectiveness analysis mentioned elsewhere in this page has now been published here]

• We believe there are additional areas in northern Nigeria where New Incentives' program would be cost-effective. We have already funded expansion to a subset of LGAs in the states covered by this grant. We believe it is likely that additional LGAs in those states will be cost-effective, and we want New Incentives to have enough funding available to expand to other eligible LGAs in a timely manner. We plan to review the results of baseline vaccine coverage assessments for new LGAs as we receive them, and New Incentives will only initiate operations in those LGAs if cost-effectiveness meets GiveWell's bar.
• We believe there is a low likelihood this program would be funded without GiveWell's support. New Incentives receives very limited additional funding from sources other than GiveWell and donors influenced by GiveWell's recommendations, and we assume a low probability that the government of Nigeria would fund this program in GiveWell's absence.
• We have a very strong qualitative impression of New Incentives. We have worked with New Incentives on its cash transfer program for over five years, and we believe New Incentives stands out for its focus on maximizing cost-effectiveness and its dedication to identifying, responding to, and being transparent about issues it faces.

For more about why we made this grant, see "The case for the grant" section below.

Main reservations

• How effective are vaccines in northern Nigeria compared to vaccine study contexts? We estimate the benefits of New Incentives' program based on our best guesses of (1) how likely an unvaccinated child is to die from vaccine-preventable diseases in areas covered by the program and (2) how much vaccination reduces that risk of death. For the former, we rely on estimates of child mortality due to vaccine-preventable diseases from the Institute for Health Metrics and Evaluation's Global Burden of Disease. However, record-keeping on the causes of child mortality can be unreliable. If mortality from vaccine-preventable diseases in unvaccinated children is lower than estimated, New Incentives' program would avert fewer deaths than we currently estimate. To estimate how much vaccination reduces the risk of death, we use the results of meta-analyses from the Lives Saved Tool, Mangtani et al. 2014, and Thumburu et al. 2015. As described in our review of New Incentives' program, we continue to be uncertain about vaccine efficacy in northern Nigeria versus the contexts in which studies of the efficacy of childhood vaccines were carried out. If vaccine efficacy is less than we have estimated (approximately 80% of the efficacy in trial contexts), this would lower the cost-effectiveness of New Incentives' program.
• How will the effect of New Incentives' program at scale compare to the effect seen in the randomized controlled trial (RCT) of the program? Our understanding of the effect of New Incentives' program is largely based on the results of an RCT of the program in Jigawa, Katsina, and Zamfara. We are just starting to receive follow-up vaccination coverage survey results for areas where New Incentives has been operating for 6 to 12 months. As we receive and analyze these results, we will have a clearer signal as to whether the effect size observed in the RCT has been maintained as New Incentives scales and expands to new states not included in the RCT. However, unlike the RCT results, the data on trends in vaccination coverage in areas New Incentives is operating will not distinguish between changes caused by New Incentives and those due to other factors occurring in the same period.

For more information, see the "Risks and reservations" section below.

Funding for the grant

This grant was funded by about $11.1 million in donations to GiveWell's Top Charities Fund in the first quarter of 2023, about $1.1 million from Giving What We Can's Top Charities Fund, about $0.3 million from Effective Altruism Australia, and about $0.06 million from Effective Altruism New Zealand.

Planned activities

New Incentives provides cash transfers to incentivize caregivers to bring infants to clinics for routine childhood vaccinations. This program operates in northern Nigeria and seeks to increase uptake of routine vaccinations, raise public awareness of the benefits of vaccination, and reduce the frequency of vaccine stockouts.8

In November 2022, we made a $30 million grant to New Incentives to support expansion of its program to 67 local government areas (LGAs) in the states of Adamawa, Gombe, Kaduna, Kano, Kebbi, and Taraba. New Incentives had not been operating in those states prior to the November grant.

This grant, along with approximately $19.6 million from prior grants,9 will provide funding for New Incentives to:

• Expand its program to up to 47 additional LGAs in Adamawa, Kaduna, Kano, Kebbi, and Taraba by the end of 2023 and maintain operations in those LGAs through December 2025.10
• Continue operations in another 104 LGAs for an additional year (through December 2025). Previous grants funded operations in these LGAs through December 2024.11

See our review of New Incentives for a breakdown of spending categories in its program. This grant will support the same program at a greater scale, so we expect the cost structure to be similar. With this grant, New Incentives will have sufficient funding to operate through December 2025 in all feasible, cost-effective LGAs in the 11 states highlighted on the map below.12

LGA-specific funding decisions

For the expansion portion of this grant, New Incentives will only initiate operations in new LGAs after GiveWell has determined that they are likely to meet our cost-effectiveness bar.13 This means that while this grant provides funding for operations in up to 47 additional LGAs, New Incentives will be left with excess funding if some of those LGAs do not meet GiveWell's bar. We have discussed this possibility with New Incentives and plan to account for any excess funding in future grants we make to New Incentives.

Historically, expansion decisions have been made on the basis of (1) whether state-level estimates of cost-effectiveness were above GiveWell's bar and (2) whether baseline vaccination coverage was below a threshold set by New Incentives.14 New Incentives is now expanding from an initial set of states that were selected for very low baseline vaccination coverage to states that may have higher baseline vaccination coverage.15 We are now moving to a process that incorporates GiveWell's input on cost-effectiveness for a specific subset of LGAs (rather than the state as a whole) before New Incentives starts operations in a given area. This change is driven by the expectation that a greater proportion of the areas assessed at baseline will have vaccination coverage that leads GiveWell to conclude that they are below our cost-effectiveness bar.

Moving forward, New Incentives will continue conducting vaccine coverage assessments to estimate baseline coverage in a given area prior to working there. These assessments are done at the cohort level (typically including about 6 LGAs),16 and New Incentives plans to conduct such assessments covering all 47 LGAs in 2023.17 We will assess the cost-effectiveness of each cohort on the basis of the coverage assessment results as we receive them, and New Incentives will only initiate operations in the associated LGAs with funding from this grant if cost-effectiveness meets GiveWell's bar.

Budget and grant amount

This grant assumes a lower cost per child reached than prior grants. As a result of this change in our assumptions, we now believe that New Incentives will be able to conduct the grant activities covered in prior grants with less funding than originally estimated and thus we expect that New Incentives will be able to roll forward funding received in prior grants for 2023-2025 program activities. Since New Incentives is expected to have funds remaining from prior grants, a smaller grant amount is required to support the broad set of activities described above than would have been required otherwise. More detail on the updates:

• Cost per child: Our last grant to New Incentives assumed a cost per child of $25.42.18 However, New Incentives' cost per child has decreased over time, so after analysis of the most recent spending data and discussion with New Incentives, we revised this assumption to $20 per child.19 This reduces the grant amount needed to cover the same number of children by about 20%.20
• Rollover from prior grants: Since we typically provide grants to cover multiple years of a program, previous grants to New Incentives have included funding for 2023 and beyond.21 Those grants were made based on our best guesses of New Incentives' cost per child at the time of the grant decisions, with our best guesses ranging from $25 to $31.22 However, New Incentives' cost per child has steadily decreased in recent years. This means we effectively granted more funding than New Incentives will ultimately need to cover the intended number of children. To account for this expected excess, we calculate the difference in funding needed using the current cost per child assumption ($20 per child) compared to the previous cost per child assumptions. We then subtract the total from the budget required for the activities included in this grant. This reduces the amount of new funding needed by approximately $19.6 million.

The case for the grant

The case for this grant differs from our prior grants to New Incentives as we are not basing the grant decision on updated cost-effectiveness estimates.23 Instead, we will be assessing the cost-effectiveness of each cohort within a state as baseline coverage results from each cohort's vaccine coverage assessment are received (as explained above). We are choosing to provide funding now, before determining the cost-effectiveness of the cohorts covered in this grant, for the following reasons:

• We have already funded expansion to a subset of LGAs in the states covered by this grant (based on previous cost-effectiveness estimates).
• We want New Incentives to have enough funding available to expand to eligible LGAs in a timely manner.

In addition to these timeliness considerations, we considered the following:

• We believe New Incentives' program is cost-effective in existing geographies and believe that there is additional room to expand that will be greater than 10x as cost-effective as unconditional cash transfers, which is our current cost-effectiveness bar.24
• We believe there is a low likelihood this program would be funded without GiveWell's support.
• We have a very strong qualitative impression of New Incentives.

Assessing all LGAs in a given state

We have already provided funding for New Incentives to expand to a subset of LGAs in the states covered by this grant, based on previous cost-effectiveness estimates.25 New Incentives does not sequence expansion to LGAs within a given state based on expected need, so we have no reason to believe that the 47 expansion LGAs covered in this grant would be more or less cost-effective on average than those covered in the prior grant. Since vaccine coverage assessment results might indicate that some of these LGAs are cost-effective and these assessments can be conducted at relatively low cost,26 we believe it is worthwhile to have New Incentives conduct the assessments for these additional LGAs.

Timely expansion

We want to ensure New Incentives will have sufficient funding on hand to expand to each cohort that GiveWell assesses as above our cost-effectiveness bar shortly after it completes its baseline assessment. Minimizing the time between completing baseline assessments and initiating operations is preferred (though not essential) as (1) it preserves the integrity of the coverage assessment as a baseline, with follow-up surveys then happening 12 months later (after New Incentives has been operating in an area for 6 to 12 months),27 and (2) it maintains momentum within New Incentives' operations and with local stakeholders. We considered disbursing individual tranches of cohort-specific funding after reviewing each set of baseline assessment results, but ultimately decided against this model as (1) it would add time and complexity to our process that did not seem proportional to the added benefit, and (2) we have high confidence that we will make future grants to New Incentives and believe any excess funding provided in this grant can be put toward those future grants (see the Cost-effectiveness section below).

Cost-effectiveness

At the time of our November 2022 grant decision, our cost-effectiveness analysis for New Incentives estimated the program to be above 10x cash in all 11 states covered under this grant, and above 18x cash in five of those states.28 We are currently working on updates to our cost-effectiveness analysis, which may cause some of the 11 states to fall below our 10x bar.29 However, the process outlined above will ensure that only cohorts with expected cost-effectiveness of greater than 10x will be funded. We have relatively high confidence that the upcoming changes to our cost-effectiveness analysis will not cause the five states with previous estimates of greater than 18x cash to fall below our current 10x bar. Consequently, we did not delay this grant decision until the newest version of our cost-effectiveness analysis is finalized, as we believe any excess funding from this grant can be used to fund further extensions of the program (beyond 2025) in the five states with very high cost-effectiveness.

Low likelihood of funding from other sources

New Incentives receives very limited additional funding from sources other than GiveWell and donors influenced by GiveWell's recommendations, and we assume a low probability that the government of Nigeria would fund New Incentives or a similar program in the absence of GiveWell.30 It is possible that the risk of crowding out other philanthropic funding could increase in the future if interest in demand-side incentives for vaccination grows. For example, in January 2023, Gavi launched an RFP for "demand-side incentives to address immunization challenges," which could include programs such as New Incentives'. However, we have not yet incorporated any associated risk into our cost-effectiveness analysis for New Incentives, as we have not yet seen:

• Funding being spent on vaccine demand generation in Nigeria on the current scale of New Incentives' program
• Large initiatives from major funders that focus on conditional cash transfers for routine vaccinations
• Interest from major funders in funding New Incentives

Strong qualitative impression of New Incentives

We have a strong qualitative assessment of New Incentives after more than five years of working together on its conditional cash transfer program.31 We believe New Incentives stands out for its focus on maximizing cost-effectiveness and its dedication to identifying, responding to, and being transparent about issues it faces.

Risks and reservations

• Risk that some cohorts will be less than 10x cash: We decided to make this grant prior to having results from New Incentives' baseline vaccine coverage assessments.32 While this grant provides sufficient funding for New Incentives to operate in all cohorts, it is likely that not all cohorts will meet our cost-effectiveness bar and that New Incentives will have excess funding as a result of this grant. We see this as fairly low risk as we plan to roll any excess funding into a future grant to extend funding through 2026 in areas we assess to be above our cost-effectiveness bar.
• Impact of exchange rate on cost per child: The favorable dollar-to-naira exchange rate that New Incentives has received in recent years has resulted in a decreasing cost per child and positive impact on cost-effectiveness. We have based this grant amount on New Incentives receiving a slightly less favorable exchange rate than it is currently receiving, given the risk that exchange rate dynamics may shift in Nigeria. There is a risk that meaningful changes to the exchange rate could result in us having over- or under-funded this grant. There is also a risk that a substantial decrease in the exchange rate received could meaningfully increase the cost per child and negatively impact cost-effectiveness.33
• Uncertainty regarding the effect of vaccines on child mortality in northern Nigeria: We estimate the benefits of New Incentives' program based on our best guesses of (1) how likely an unvaccinated child is to die from vaccine-preventable diseases in areas covered by the program and (2) how much vaccination reduces that risk of death. For the former, we rely on estimates of child mortality due to vaccine-preventable diseases from the Institute for Health Metrics and Evaluation's Global Burden of Disease.34 However, record-keeping on the causes of child mortality can be unreliable.35 If mortality from vaccine-preventable diseases in unvaccinated children is lower than estimated, New Incentives' program would avert fewer deaths than we currently estimate. To estimate how much vaccination reduces the risk of death, we base our estimates of vaccine efficacy on the results of meta-analyses from the Lives Saved Tool, Mangtani et al. 2014, and Thumburu et al. 2015.36 As described in our review of New Incentives' program, we continue to be uncertain about vaccine efficacy in northern Nigeria versus the contexts in which randomized controlled trials (RCTs) of the efficacy of childhood vaccines were carried out. If vaccine efficacy is less than we have estimated (approximately 80% of the efficacy in trial contexts),37 this would lower the cost-effectiveness of New Incentives' program. We plan to consider whether to fund additional data collection on this question.
• Ability of New Incentives to continue to scale and spend increasingly larger annual budgets while maintaining program quality: New Incentives spent around $9.7 million in 2022.38 Based on this grant and prior grants, we are expecting New Incentives to be able to absorb annual budgets of up to $30.3 million in 2023, and $42.5 million in 2024. Based on its track record, we are optimistic that it will be able to absorb this funding. We reviewed New Incentives' latest monitoring data as part of this grant decision and did not identify any concerns in regards to program quality that would cause us to reconsider additional support for the program.39 However, there remains a risk that New Incentives will not be able to deliver the same program quality at this increasingly larger scale.
• Very limited data on program effect outside of the randomized controlled trial (RCT) context: Our understanding of the effect of New Incentives' program is based largely on the results of an RCT of the program in Jigawa, Katsina, and Zamfara.40 We are just starting to receive follow-up vaccination coverage survey results for cohorts where New Incentives has been operating for 6 to 12 months. As we receive and analyze these results, we will have a clearer signal as to whether the effect size observed in the RCT has been maintained as New Incentives scales and expands to new states not included in the RCT.41 However, unlike the RCT results, the data on trends in vaccination coverage in areas New Incentives is operating will not distinguish between changes caused by New Incentives and those due to other factors occurring in the same period.

Plans for follow up

• We will finalize the latest version of our cost-effectiveness analysis [August 2023 Update: This cost-effectiveness analysis has now been published here], and New Incentives will continue to share baseline coverage results from each cohort's baseline vaccine coverage assessment for GiveWell to make cohort-specific funding decisions.
• We will continue to have monthly calls with New Incentives and get written monthly program updates. Key questions we plan to discuss include:
  • Expansion experience:
    • Is scale up happening at the pace New Incentives projected? What are the implications for New Incentives' annual targets and expected room for more funding for future grants?
    • What proportion of LGAs included in this grant and in the November 2022 grant has New Incentives started working in? What proportion have been excluded on the basis of cost-effectiveness less than 10x cash?
    • How have conversations with state and local governments gone regarding not operating in areas where the vaccine coverage assessment results have indicated cost-effectiveness is less than 10x cash?
    • Has the switch to making cohort-level funding decisions caused any delays or other complications for New Incentives' operations?
    • How has the security situation evolved since the most recent elections, and how is it impacting New Incentives' expansion and operations, if at all?
  • Follow up on specific trends from monitoring review:
    • Has the percentage of enrolled infants who are more than six months old at the point of enrollment decreased following the new measures New Incentives adopted?42
    • How do New Incentives' retention rates (i.e., the frequency with which enrolled infants are brought back for subsequent vaccinations) compare to rates during the RCT? Has Penta3 retention recovered since the end of 2022?43
    • How does the real value of the incentives now compare to what it was during the RCT? What has been the impact on caregivers and staff of increasing transportation costs? How is New Incentives responding?
  • Fraud:
    • Has New Incentives seen any new or increased evidence of fraud and, if so, what forms of fraud?
    • What is New Incentives' latest perspective on the rate of double enrollments (one form of fraud), and what monitoring metric reflects its best-guess estimate of double enrollments?
  • Supply-side dynamics:
    • Have there been any changes in the rates of vaccine stockouts, and how is New Incentives responding to stockouts?
    • What new information has New Incentives learned about the supply landscape, particularly in regard to other stakeholders' ability to finance additional vaccines to meet future increases in demand?
  • Room for more funding:
    • What is the latest exchange rate that New Incentives is receiving, and does New Incentives have any expectations for how it will change in the future?
    • Has New Incentives raised funding from non-GiveWell sources?
    • How are conversations with other stakeholders and potential funders progressing in regard to their interest in conditional cash transfers for generating vaccine demand?
• We will continue to review follow-up survey results in addition to the baseline assessment results, and consider implications for cost-effectiveness.
• We will periodically track several key metrics (see our most recent update here), based on the data New Incentives routinely collects.
• We will periodically review New Incentives' cost per infant enrolled and incorporate updates into our cost-effectiveness analysis.

Internal forecasts

Our process

Our process for this grant relied heavily on (a) our prior work modeling the cost-effectiveness of New Incentives, and (b) our monthly discussions with New Incentives and the monthly written progress updates it shares with us. For this particular grant, we:

• Had additional calls with New Incentives to discuss the proposed grant approach and align on the grant amount
• Conducted a room for more funding analysis to determine the size of this grant
• Incorporated newly available data into our cost per child analysis and updated our cost per child assumption accordingly
• Reviewed New Incentives' latest monitoring data and updated our spreadsheet tracking key monitoring metrics
• Reviewed New Incentives' latest perspectives on the vaccine supply landscape and confirmed no red flags
• Reviewed IDInsight's preliminary report on the 12-month follow-up survey for New Incentives' first four cohorts and confirmed no red flags
• Discussed implications of the Nigerian elections with New Incentives

For internal review, a Senior Researcher and a Senior Research Associate who were not otherwise involved in the grant investigation gave feedback on the plan for investigating the grant. They also reviewed the case for making the grant and gave feedback prior to final grant approval by a Senior Program Officer.

Sources

• 1

  See the amount offered for each vaccination visit on New Incentives' website. These incentives total 5,000 naira, which as of June 12, 2023, was equal to $10.81 USD (calculated via Google).

• 2

  While we do not yet have cost-effectiveness estimates for each of the LGAs covered under this grant (more in this section), our prior modeling work suggests the program is likely cost-effective.

• 3

  As New Incentives continues to expand to states which are further from the three states included in the RCT, we expect to vary the increase in vaccination rate based on state-specific estimates of baseline vaccine coverage (i.e., we think states with higher baseline coverage are likely to see smaller improvements and vice versa).

• 4
  • See our estimates of vaccine efficacy here.
  • Note that our estimates of vaccine efficacy do not include a few vaccines that are indirectly incentivized by the program (i.e., the polio, yellow fever, and meningitis A vaccines). These vaccines are considered "indirectly incentivized" because New Incentives does not provide cash transfers to caregivers whose infants receive those vaccinations, but these vaccines are offered at the same time as other directly incentivized vaccines. See the routine vaccination schedule on New Incentives' website.
  • We do not directly model the benefits of these indirectly incentivized vaccines (except for the rotavirus vaccine) because we do not expect them to have as large of an impact on child mortality as the vaccines we directly model. Instead we include supplemental adjustments to account for the benefits from these vaccines.

• 5

  See this cell note in our cost-effectiveness analysis.

• 6

  See our calculations of the probability of death from vaccine-preventable diseases here. This probability of death refers to the probability that, after the recommended age of vaccination, an unvaccinated child will die before the age of five.

• 7

  New Incentives' cost per child has decreased over time, so we expect to include a lower cost per child estimate in the next version of our cost-effectiveness analysis. We expect to publish an updated cost-effectiveness analysis later this year. [August 2023 Update: This cost-effectiveness analysis has now been published here]

• 8

  "New Incentives runs a conditional cash transfer program in North West Nigeria. Caregivers who bring their infants to clinics for routine vaccines, which are provided through government clinics free of charge, can receive a total of $11 over the course of five clinic visits. New Incentives also does outreach to caregivers about the importance of vaccinating children, and works with its government partners to improve vaccine supply by identifying and addressing bottlenecks in the vaccine supply chain." See our review of New Incentives for more details.

• 9

  See discussion in the Budget and grant amount section below. Our calculation of rollover funding from previous grants is here.

• 10
  • See the breakdown of these LGAs by state here.
  • Note that expansion to these LGAs will be conditional on their estimated cost-effectiveness. See the LGA-specific funding decisions section for more information on how these expansion decisions will be made.

• 11

  See the breakdown of these LGAs by state here.

• 12

  As of November 2022, New Incentives was operating in all areas where security allowed in Katsina, Zamfara, Jigawa, Sokoto, and Bauchi. Expansion to Gombe state was delayed from November to December 2022. New Incentives, November 2022 Program Update (unpublished)

• 13

  Note that (a) our cost-effectiveness analyses are simplified models that are highly uncertain, and (b) our cost-effectiveness threshold for directing funding to particular programs changes periodically. As of May 2023, our bar for directing funding is about 10x as cost-effective as GiveDirectly's unconditional cash transfers. See GiveWell’s Cost-Effectiveness Analyses webpage for more information about how we use cost-effectiveness estimates in our grantmaking.

• 14
  • Before initiating operations in a new area, New Incentives conducts assessments to measure baseline vaccination coverage. Previously, New Incentives typically moved forward with expansion when baseline coverage of the BCG vaccine (against tuberculosis) was less than 80%. New Incentives, conversation with GiveWell, March 11, 2022 (unpublished)
  • "New Incentives will group local government areas (LGAs) it expands to within a given state at a given point in time into ‘expansion groups’. New Incentives will then collect coverage data in these expansion groups once before the start of operations to establish baseline coverage rates." IDinsight, Coverage monitoring – analysis plan, 2021, p. 1. (Note that New Incentives and GiveWell refer to these expansion groups as "cohorts.")

• 15

  See, for example, the estimated effective coverage rates here.

• 16

  "New Incentives will group local government areas (LGAs) it expands to within a given state at a given point in time into ‘expansion groups’. New Incentives will then collect coverage data in these expansion groups once before the start of operations to establish baseline coverage rates." IDinsight, Coverage monitoring – analysis plan, 2021, p. 1. (Note that New Incentives and GiveWell refer to these expansion groups as "cohorts.")

• 17

  New Incentives, April 2023 Program Update (unpublished).

• 18

  See here.

• 19

  See here for further rationale on the $20 cost per child estimate.

• 20

  $20.00 / $25.42 = 0.7868; 1 - 0.7868 = 0.2132, or about 20%

• 21

  For example, we made a grant to New Incentives in May 2022 that covers:

  • "Expansion to 25 new LGAs by the end of 2022 and funding for these LGAs in 2023
  • All 92 LGAs in 2024 (combined with existing funds)." GiveWell, "New Incentives – Nigeria Expansion (May 2022)"

• 22

  See the cost per child assumptions from previous grants here.

• 23

  By contrast, our November 2022 grant to New Incentives was based on an average estimated cost-effectiveness of 14.7x cash in the states covered by that grant. See this section of our November 2022 grant page.

• 24

  As of May 2023, our bar for directing funding is about 10x as cost-effective as GiveDirectly's unconditional cash transfers. See GiveWell’s Cost-Effectiveness Analyses webpage for more information about how we use cost-effectiveness estimates in our grantmaking.

• 25
  • "The current grant would fund expansion to 67 LGAs by mid-2023, with funding for 12 LGAs through December 2024 and funding for 55 LGAs through December 2025. As of the time of this grant, New Incentives expected that the 67 additional LGAs would be located in 6 states it has not previously worked in [Adamawa, Kaduna, Kano, Kebbi, Taraba, and Gombe], however, these plans are subject to further discussions with government partners and findings from baseline vaccination coverage assessments." GiveWell, "New Incentives – Nigeria Expansion (November 2022)"
  • Note that the expansion funded by our November 2022 grant is still in progress (we are still awaiting the baseline vaccination coverage assessment results for some cohorts) and thus we do not yet know if all LGAs funded by that grant will be cost-effective.

• 26

  Based on conversations with New Incentives, we estimate the cost of conducting vaccination coverage assessments to be approximately $7,400 per LGA (excluding staffing and indirect costs), and the annual operational budget per LGA to be around $210,000. Thus, we estimate these assessments account for only about 3.5% of the annual operational budget for an LGA. New Incentives, conversation with GiveWell, April 11, 2023 (unpublished)

  $7,400 / $210,000 = 0.0352

• 27

  "New Incentives plans to group the Local Government Areas (LGAs) to which it is expanding into cohorts. For each cohort, it plans to:

  1. Conduct a baseline assessment of vaccination coverage. New Incentives will use the results of the baseline assessment when considering which LGAs to work in.
  2. Conduct follow-up assessments of vaccination coverage in the LGAs it works in, one year after the baseline assessment and every six months after that." GiveWell, "New Incentives' Coverage Assessments: Plans as of October 2021," 2023

  "Once operating in all LGAs in an expansion group (usually within six months of completion of baseline), New Incentives will obtain follow-up RI coverage estimates in these LGAs in six months intervals." IDinsight, Coverage monitoring – analysis plan, 2021, p. 1.

• 28

  We estimated cost-effectiveness of greater than 18x cash in Bauchi, Gombe, Jigawa, Kano, Katsina, Kebbi, Sokoto, and Zamfara. We estimated cost-effectiveness of greater than 10x cash in the remaining three states covered by the grant: Adawama, Kaduna, and Taraba. See our estimates here.

• 29

  We expect to publish an updated cost-effectiveness analysis later this year. [August 2023 Update: This cost-effectiveness analysis has now been published here]

• 30
  • For example, in the funding cycle preceding this one, New Incentives filled less than 7% of its funding gap with funding from non-GiveWell sources. New Incentives raised $2,220,000 from other funders, had $1,485,800 in budget surplus from a prior funding cycle, and GiveWell made a grant of $30,186,650 to fill the remaining funding gap: $2,220,000/ ($2,220,000 + $1,485,800 + $30,186,650) = 0.06585. New Incentives, Financial forecasts 2021-2026 (unpublished)
  • See our rationale for the low likelihood of government funding in this cell note.

• 31
  • "New Incentives started this program in May 2017." GiveWell, "New Incentives," 2022.
  • All GiveWell-directed grants to New Incentives can be found on this page.

• 32
  • "New Incentives will group local government areas (LGAs) it expands to within a given state at a given point in time into ‘expansion groups’. New Incentives will then collect coverage data in these expansion groups once before the start of operations to establish baseline coverage rates." IDinsight, Coverage monitoring – analysis plan, 2021, p. 1. (Note that New Incentives and GiveWell refer to these expansion groups as "cohorts.")
  • We expect to publish an updated cost-effectiveness model, using results from the baseline vaccine coverage assessments received to date, as an addendum to this page once complete. [August 2023 Update: This cost-effectiveness analysis has now been published here]

• 33

  See our cost per child projections based on varying exchange rates here.

• 34

  See these estimates on the "Probability of death" tab of this spreadsheet.

• 35

  For example, the Institute for Health Metrics and Evaluation explains that mortality data can be flawed: "This last piece – the underlying cause of death – is what GBD researchers use to produce mortality statistics. Understanding the underlying cause of death in a population allows public health officials to develop interventions that target the root cause. But picking out the underlying cause of death can be difficult. Many countries in the world do not have functioning vital registration systems, and even for those countries who do, there are often high levels of misclassification and vague reporting of causes of death on death certificates." Alexander, "Determining causes of death: How we reclassify miscoded deaths," 2018.

• 36

  For more information, see this section of our review of New Incentives' program.

• 37

  See this row in our cost-effectiveness analysis for New Incentives.

• 38

  See here.

• 39

  See this monitoring data here.

• 40

  For more information on the RCT, see IDinsight, Impact Evaluation of New Incentives, Final Report, 2020.

• 41

  More on our plan for analyzing these results here.

• 42
  • New Incentives has observed a relatively high rate of enrolled infants who are more than six months old at the point of enrollment. See here. If this rate is sustained, there will be a negative impact on cost-effectiveness, as a meaningful proportion of the mortality averted through vaccination occurs in the first six months.
  • "New Incentives introduced new measures in cash disbursement review that assess whether an infant is older than 1 year old at the point of enrollment, as well as whether infants are significantly older than their date of birth suggests to have additional data to triangulate estimated age of infants given inconsistencies in date of birth." New Incentives, comments during review of this page, June 29, 2023 (unpublished)

• 43

  See the latest trend in Penta3 retention rates here.