Note: This page summarizes the rationale behind a GiveWell-recommended grant to New Incentives. New Incentives staff reviewed this page prior to publication.
Summary
In May 2022, GiveWell recommended a $35 million grant1 to New Incentives. The grant was funded in part by donations to the Top Charities Fund between January and March 2022 ($21.5 million) and in part through a grant from Open Philanthropy ($13.7 million). New Incentives runs a conditional cash transfer (CCT) program in North West Nigeria which seeks to increase uptake of routine immunizations through cash transfers, raise public awareness of the benefits of vaccination, and reduce the frequency of vaccine stockouts. New Incentives expects to use this funding to continue to expand in northern Nigeria and to extend its work in its current areas of operation through 2024. New Incentives is one of GiveWell's top charities.
We recommended this grant because we believe that the work that the grant will support will be cost-effective. New Incentives' program is among the most cost-effective programs we know of. We will have the opportunity to learn about the impact of the grant through New Incentives' surveys of vaccine coverage before and after it begins working in an area and through its routine monitoring. New Incentives has a track record of fast growth and stands out for its dedication to identifying, responding to, and being transparent about issues it faces.
Published: August 2022
Table of Contents
Planned activities and budget
New Incentives provides cash transfers to incentivize caregivers to bring babies to clinics for routine childhood vaccinations. This CCT program operates in the North West region of Nigeria and seeks to increase uptake of routine immunizations, raise public awareness of the benefits of vaccination, and reduce the frequency of vaccine stockouts.2 This grant will expand New Incentives' current program to new geographies within Nigeria as well as extend the timeline of funding in currently operating areas. We have previously funded New Incentives to expand its program to 67 local government areas (LGAs) in northern Nigeria and maintain that scale through the end of 2023.3 The current grant would fund:4
- Expansion to 25 new LGAs by the end of 2022 and funding for these LGAs in 2023
- All 92 LGAs in 2024 (combined with existing funds)
New Incentives projected that this funding would enable it to reach 1,559,000 additional infants.5 The total funding needed to support a program of this size is $40,052,900,6 and New Incentives projected that it would have $4,908,300 available from sources other than this grant and budget surplus,7 leaving room for more funding in the amount of $35,144,600.8 A breakdown of the budget by year is available here. See our review of New Incentives for a breakdown of spending categories in its program. This grant will support the same program at a greater scale, so we expect the cost structure to be similar.
The case for the grant
- Cost-effectiveness – We estimate that this grant will meet our bar for cost-effectiveness. More below.
- Learning opportunities – We will have the opportunity to learn about the impact of the grant through New Incentives' surveys of vaccine coverage before and after it begins working in an area and through its routine monitoring. More below.
- Track record – New Incentives has a track record of fast growth, and we have a strong qualitative assessment of New Incentives. More below.
Cost-effectiveness
Based on our cost-effectiveness analysis of New Incentives, we believe the program is in the range of cost-effectiveness of programs we expect to direct funding to, as of mid-2022. At the time we recommended this grant, we were primarily looking to recommend grants that we estimated were more than 10x as cost-effective as GiveDirectly and were more tentatively considering grants that were between 6x and 10x as cost-effective as GiveDirectly.
Note that our cost-effectiveness analyses are simplified models that do not take into account a number of factors. There are limitations to this kind of cost-effectiveness analysis, and we believe that cost-effectiveness estimates such as these should not be taken literally due to the significant uncertainty around them. We provide these estimates (a) for comparative purposes to other grants we have made or considered making, and (b) because working on them helps us ensure that we are thinking through as many of the relevant issues as possible.
Our estimate is that this program is 13.6 times as cost-effective as GiveDirectly's program.9
We discuss our model of New Incentive's cost-effectiveness on this page. In considering this grant, we made one very minor update to our cost-effectiveness model. We received updated program information from New Incentives that led us to increase our estimate of the cost per infant enrolled in New Incentives' program from $28.93 to $29.52.10
See this spreadsheet for the impact of this change on our cost-effectiveness estimates.11
Learning opportunities
We will have the opportunity to learn about the impact of the grant through New Incentives' surveys of vaccine coverage before and after it begins working in an area and through its routine monitoring. New Incentives plans to reassess vaccination coverage after it has worked in each of the new areas for a year, and then every six months.12 See this page for how we expect to use findings from these assessments to update our cost-effectiveness analysis of New Incentives' program.
New Incentives' track record
New Incentives has a track record of fast growth,13 and this grant is given under the expectation that it can continue a quick pace of growth over the next year.
We have a strong qualitative assessment of New Incentives. We believe New Incentives stands out for its dedication to identifying, responding to, and being transparent about issues it faces. For examples of how New Incentives approaches issues that arise, see our "Risks and reservations" section below.
Risks and reservations
- Rapid scale up – Supporting a rapidly growing organization means we are increasing funding while not yet being able to learn much about the impact of the funds we've contributed before. An alternative to making this grant would be to wait and get more results before funding New Incentives to expand further. In particular, we do not yet have updates on:
- When rotavirus vaccines will be introduced in the areas where New Incentives works. In 2021, New Incentives told us that it expected rotavirus vaccines to be introduced in 2022.14
- Our expectation remains that many of the infants enrolled with the funding from this grant will be eligible for a rotavirus vaccine. Note that even if no rotavirus vaccines are given, this grant would still meet our criteria. Our CEA models New Incentives' cost-effectiveness at 13.6x. Benefits from the rotavirus vaccine account for 21% of the overall benefits of the intervention, so, even if we assume no benefits from the rotavirus vaccine, we'd still estimate that the grant is more than 10 times as cost-effective as cash transfers.15
- How vaccination rates change after New Incentives begins operating in an area. We won't have an update on this for another 6-18 months, when results from the first vaccination coverage reassessments become available.16
- Managing telecommunications shutdowns – There have been telecommunications shutdowns in response to security challenges in New Incentives' operating areas that slowed down its work recently and may undermine trust in the program in those areas going forward. The telecommunications shutdowns have ended for now. New Incentives is seeking to adapt by enrolling infants who received their first vaccines during the shutdowns (who generally would only be eligible if they enrolled in the program when receiving their first vaccine), preparing for enrollments during future shutdowns, and focusing expansion on areas with lower levels of security threat.
- Vaccine shortages – In our most recent grant page for New Incentives, we mentioned that there had been an increase in vaccine stockouts.17 The number of disbursement days with a vaccine out of stock was high (~15%) for several months in 2021. This then fell to around 5% (the rate seen during the RCT) for three months at the end of 2021. Since our last grant, days with stockouts ticked up again in February and March 2022 (16 and 18%, respectively).18 New Incentives has sought to address this issue by improving coordination among those responsible for stocking vaccines at zonal, state, and local levels of government.19 We will continue to follow up with New Incentives to see if stockouts remain an issue.
- Managing incorrectly given incentives – In our most recent New Incentives grant page, we mentioned that New Incentives had identified some cases where its staff gave cash incentives to caregivers, even though the infants had not received the incentived vaccine. The identified cases were ones where the vaccine in question was out of stock that day. It's not clear to us whether this issue is limited to when vaccines are out of stock or whether it indicates a wider problem. Since then, we have discussed this more with New Incentives. We have learned that 15-20% of disbursement days include a visit by a New Incentives supervisor or auditor.20 These visits include assessments of whether vaccinations are given. New Incentives tries to make these auditors unidentified, so that clinic staff and New Incentives field officers won't change their behavior because they know they are being observed.21 When possible, New Incentives field officers sit next to the vaccinator to observe if infants receive the vaccinations before the disbursement. When that is not possible, field officers are expected to check every two hours that clinic staff are marking on the child's health card that they received a vaccine (with a gold pen) only after the vaccine has been given.22 Auditing measures increase our confidence that incidences of fraud are being reported; however, this is an issue we will continue to discuss with New Incentives.
- Government opposition – Government opposition is a potential risk to New Incentives' ability to operate its program, as we have written before. New Incentives has the support of several, but not all, government agencies in Nigeria. It has the support of the state agencies whose permission it requires to work in clinics in each state and several national health and social protection agencies. New Incentives has kept us updated on its relationships with and support to state-level officials and other national agencies. We've had a number of conversations for past grants on the question of governmental support and did not have more conversations for this grant investigation.
Plans for follow up
- We will continue to have monthly calls with New Incentives and get written monthly program updates. Key questions we plan to discuss include:
- Is scale up happening at the pace New Incentives projected?
- What proportion of disbursement days have been missed for any reason? What are the implications for New Incentives' room for more funding?
- Have rotavirus vaccines been introduced in the areas where New Incentives operates?
- What forms of fraud is New Incentives seeing evidence of?
- What actions has New Incentives taken to support its government partners to reduce vaccine stockout rates?
- Has New Incentives raised funding from non-GiveWell sources?
- Has New Incentives been tracking inflation and considering updates to incentive amounts? Has the real value of the incentive decreased meaningfully?
- How is government support progressing?
- For all clinics that New Incentives is starting to partner with (since late 2021), New Incentives will do baseline and follow-up surveys to measure changes in vaccination rates. We have published a page about how we plan to use those results.
- We will periodically track several key metrics (see our most recent update here), based on data the New Incentives routinely collects, including on the proportion of infants that return for later vaccines and stockout rates.
- We will update the cost per infant estimate in our cost-effectiveness analysis with later data.
Internal forecasts
| Confidence | Prediction | By time |
|---|---|---|
| 35% | New Incentives will be unable to operate for >20% of the disbursement days in the clinics it has begun operations in during the second six months of 2022. This may be due to security concerns, telecommunication shutdowns, or other unanticipated factors. | End of 2022 |
| 30% | New Incentives will have an average measles 1 retention rate >85% in the second six months of 2022. (Measles 1 retention during the RCT was 91%. It has been 81% since the end of the RCT.) | End of 2022 |
| 25% | Number of stockouts per 100 disbursement days in clinics that New Incentives works in will be >15 in the second six months of 2022, according to New Incentives' data. (This figure was 5 out of 100 days during the RCT and 13 since the RCT ended.) | End of 2022 |
| 40% | Rotavirus vaccines will be available and incentivized by New Incentives in >75% of the clinics it operates in. | End of 2022 |
Our process
Our process for this grant relied heavily on (a) our prior work on modeling the cost-effectiveness of New Incentives, and (b) our monthly discussions with New Incentives and the monthly written progress updates it shares with us. For this particular grant, all of the information we used to make our decision came from those conversations and updates, in addition to documents that New Incentives shared with us. We made minor updates to our cost-effectiveness model to increase our cost-per-infant-enrolled estimates. See our most recent grant page prior to this one for more discussion of updates to our cost-effectiveness analysis for this program and stakeholders we have spoken to.
For internal review, two Program Officers who were not otherwise involved in the grant investigation gave feedback on the plan for investigating the grant.
Sources
- 1
Exact grant amount: $35,144,600. GiveWell, Budget for mid-2022 grant to New Incentives
- 2
See our report on New Incentives for more details.
- 3
See our grant page about this expansion here.
- 4
Funding Cycle: Q2 2022:
- $40,052,900 for 1,559,000 infants
- Support expansion to 25 new LGAs by the end of 2022
- "The additional funds combined with existing funds would support operations for all 92 LGAs through Dec 2024.…" New Incentives, 2021-2025 Financial Forecasts (unpublished). - 5
"$40,052,900 for 1,559,000 infants" New Incentives, 2021-2025 Financial Forecasts (unpublished).
- 6
"Funding Cycle: Q2 2022:
- $40,052,900 for 1,559,000 infants" New Incentives, 2021-2025 Financial Forecasts (unpublished). - 7
Some of the projected $4,908,300 is other funds raised (including donors giving based on GiveWell's recommendation of New Incentives as a top charity) and some is budget surplus from 2021:
- "Estimated Funds from 2022 Q1-Q2 from GiveWell Top Charity Recommendation: $2,120,000 (assumes potential commitments of $522,000, $1,000,000, $98,000; $500,000 in addition from other donors)
- Estimated Budget Surplus and Additional Funds Raised as of February 2022: 2021 Budget Surplus of $2,016,200 and additional funds raised of $772,100 to scale-up the program (per contributions received as of early February 2022) are applied to this funding gap." New Incentives, 2021-2025 Financial Forecasts (unpublished).$2,120,000+$2,016,200+$772,100= $4,908,300
- 8
See here.
- 9
See our 2022 GiveWell cost-effectiveness analysis — version 5, sheet "New Incentives," row "Cost-effectiveness in multiples of cash transfers, after all adjustments"
- 10
See the updated version of our cost per infant immunized analysis for New Incentives here. See the previous version here.
- 11
Changes to our models of the cost-effectiveness of other interventions can also affect our estimates for New Incentives, because our models are interconnected. For full details on what changed between versions of our cost-effectiveness analysis, see our changelog here and our spreadsheet on the effects of those changes here and here.
- 12
- "The first follow-up will occur approximately 12 months after baseline." IDinsight, Coverage Monitoring Analysis Plan, 2021, Pg. 1.
- "Once operating in all LGAs in an expansion group (usually within six months of completion of baseline), New Incentives will obtain follow-up RI coverage estimates in these LGAs in six months intervals." IDinsight, Coverage Monitoring Analysis Plan, 2021, Pg. 1.
- 13
New Incentives operated in 98 clinics at the start of 2021. As of October 2021, it was operating in 511. New Incentives, Cost Model with Assumptions (unpublished). New Incentives notes (email to GiveWell, June 24, 2022) that this "reflects the clinics we were operating in during October 2021 and excludes those we expanded to but were not operating in during October 2021 due to the telecommunications shutdown."
- 14
“However, as a result of delays in the vaccine rollout, the rotavirus vaccine has not yet been added to the routine immunization schedule in Nigeria, and New Incentives now guesses that it will not be added to the schedule in the states in which its program operates until 2022.” GiveWell, "Allocation of Funds Donated to the Top Charities Fund in Q2 2021", 2021
- 15
13.6 x (1 - 0.21) = 10.7x as cost-effective as cash transfers.
- 16
See here.
- 17
- An average of 15% of disbursement days had stockouts in February to August. Percent of disbursement days with stockouts. February 2021: 20.64%, March 2021: 23.94%, April 2021: 11.50%, May 2021: 4.97%, June 2021: 13.78%, July 2021: 9.33%, August 2021: 18.46%. Average = 14.66%. See here.
- During the prior period in which New Incentives' impact was rigorously measured (and on which our cost-effectiveness model is based), the average percent of disbursement days with stockouts was 5%. See here.
- 18
See GiveWell, New Incentives monitoring results [April 2022], sheet "Source: Indicators [thru Mar 2022]" row "% disbursement days with stockouts"
- 19
New Incentives, Program Updates, April 1, 2022 (unpublished).
- 20
Call with New Incentives, May 9, 2022 (unpublished).
- 21
Call with New Incentives, May 9, 2022 (unpublished).
- 22
Call with New Incentives, May 9, 2022 (unpublished).