New Incentives — Nigeria (January 2022)

Note: This page summarizes the rationale behind a GiveWell-recommended grant to New Incentives. New Incentives staff reviewed this page prior to publication.

Published: March 2022

Planned activities and budget

New Incentives has budgeted $10,018,000 to expand from 35 LGAs at the start of 2022 to 67 LGAs by the end of 2022. This budget is for the expansion in 2022 and to maintain operations in those new LGAs in 2023. It projects that this funding will enable it to reach an additional 532,800 infants.1

New Incentives projected that it would raise $607,000 from other donors, leaving a funding gap of $9,411,000.2

See our review of New Incentives for a breakdown of spending categories in its program. This grant will support the same program at a greater scale, so we expect the cost structure to be similar.

The case for the grant

• We estimate that this grant will meet our bar for cost-effectiveness. More below.
• We expect to get meaningful updates on the impact achieved by this grant, through data collected by New Incentives. More below.
• We believe New Incentives stands out for its dedication to identifying, responding, and being transparent about issues it faces. See "risks and reservations" section below.

Cost-effectiveness

Based on our cost-effectiveness analysis of New Incentives, we believe the program is in the range of cost-effectiveness of programs we expect to direct funding to, as of 2022. Our estimate is that New Incentives is 14x as cost-effective as GiveDirectly's program, which provides unconditional cash transfers to poor households in low-income countries.3 At the time we recommended this grant, we were primarily looking to recommend grants that we estimated were more than 8x as cost-effective as GiveDirectly, and were willing to consider recommending a limited amount of funding to grants that were between 5x and 8x as cost-effective as GiveDirectly.

Note that our cost-effectiveness analyses are simplified models that do not take into account a number of factors. There are limitations to this kind of cost-effectiveness analysis, and we believe that cost-effectiveness estimates such as these should not be taken literally due to the significant uncertainty around them. We provide these estimates (a) for comparative purposes to other grants we have made or considered making, and (b) because working on them helps us ensure that we are thinking through as many of the relevant issues as possible.

We discuss our model of New Incentives' cost-effectiveness on this page. In considering this grant, we made the following updates:

• Cost to New Incentives per enrolled infant. Our model previously used data from a time when New Incentives operated at a smaller scale than it does now. We have now updated the model to use data on costs and enrollments through August 2021.4 This includes costs of cash transfers that have not yet been disbursed to enrolled infants because they have not yet reached the age at which some vaccinations are given or because they haven't returned yet for scheduled vaccines.5 We estimate that New Incentives is spending $29 per enrolled infant, down from $38 in our previous estimate.6
• Costs to the government of Nigeria and Gavi. New Incentives pays for the cash incentives but not for the costs to buy and administer vaccines to the additional infants that New Incentives induces to get vaccinated. We guess that those costs are paid for by the government and by Gavi, the Vaccine Alliance.7 We updated the data source that we use for these costs, which significantly increased our estimate of the cost per vaccine that the government and Gavi incur.8 Together, this and the change in our estimate of New Incentives' costs largely canceled out.9
• Disease burden in the states that New Incentives operates in. Previously, we used estimates from the Institute of Health Metrics and Evaluation (IHME) of vaccine-preventable mortality across Nigeria, with a rough adjustment for evidence that rates of vaccine-preventable mortality are higher in the states that New Incentives operates in than average for Nigeria.10 Recently, we have contracted with IHME to access state-level disease burden estimates for Nigeria. We replaced the national estimate and adjustment with the state-level estimates. This increased our estimate of cost-effectiveness by 30%.11

Monitoring of program impact

New Incentives is now conducting baseline vaccination coverage assessments before starting to work in new areas.12 It plans to reassess vaccination coverage after it has worked in each of the new areas for a year, and then every six months.13 See this page for how we expect to use findings from these assessments to update our cost-effectiveness analysis of New Incentives' program.

New Incentives also will track other metrics, such as the proportion of enrolled infants that return for each subsequent vaccination and vaccine stockouts at clinics.14 Details here.

Risks and reservations

• New Incentives has been scaling up rapidly.15 An alternative to making this grant would be to wait and get more results before funding New Incentives to expand further. In particular, we do not yet have updates on:
  • When rotavirus vaccines will be introduced in the areas New Incentives works. This accounts for 21% of the value of the intervention in our CEA.16 Earlier this year, New Incentives told us that it expected rotavirus vaccines to be introduced in 2022.17
  • How vaccination rates change after New Incentives begins operating in an area. We won't have an update on this for another 12-24 months, when results from the first vaccination coverage reassessments (described above) become available.18
• For several months in the second half of 2021, in parts of two states that New Incentives operates in, Zamfara and Katsina, the government ordered complete shutdowns of telecommunications as a response to security threats.19 New Incentives suspended its operations in these areas for the length of the shutdowns. Caregivers could bring their infants to clinics for vaccines but they didn't receive a cash incentive from New Incentives during this time.20 New Incentives is considering ways to make up for missed incentives and/or put in place better systems for maintaining incentives during any future shutdowns.21 New Incentives revised its projection for how quickly it could expand down modestly as a result of the shutdowns.22 We expect that the 2021 shutdown will have limited impact on New Incentives' long-term cost-effectiveness and ability to grow, but it is possible that these events indicate an increased risk level in general for the program. It is possible that the shutdowns could reduce trust in the program and thus effectiveness of the transfers; we will get some data on this question from the vaccination coverage assessments that New Incentives is conducting (discussed in previous section).
• Since February 2021, rates of clinics being stocked out of vaccines has risen. An average of 15% of disbursement days had stockouts in February to August;23 during the prior period in which New Incentives' impact was rigorously measured (and on which our cost-effectiveness model is based), it was 5%.24 New Incentives has told us about some steps it is taking in response: it held a meeting with state government officials in September at which the state ministries of health agreed "to visit national stakeholders to advocate for additional vaccines";25 and it is working to identify the clinics with the biggest stockout issues and do community outreach to encourage caregivers whose infants weren't served on previous visits to try again.26
• Recently, New Incentives identified some cases where its staff gave cash incentives in cases where the infant had not received the incentived vaccine. The identified cases were ones where the vaccine in question was out of stock that day. After reviewing its data, New Incentives determined that 0.2% of transfers were given for vaccines that were recorded as out of stock on the day the transfer was given (and therefore must have been given against policy).27 It's not clear to us whether this issue is limited to when vaccines are out of stock or whether it indicates a wider problem. New Incentives believes that the main reasons this would occur would be due to stockouts or to vaccine hesitancy.28 Earlier research indicated little if any vaccine hesitancy.29 New Incentives plans to do some additional data analysis to investigate the issue30 and we will follow up with New Incentives about this.
• As we've written before, "Government opposition is a potential risk to New Incentives' ability to operate its program. New Incentives has the support of several government agencies in Nigeria, including the state agencies whose permission it requires to work in clinics in each state and several national health and social protection agencies. Support for CCTs for immunization within the government of Nigeria is not universal, however, and its critics have raised concerns about whether CCTs are the most effective strategy for increasing immunization rates and whether CCTs, once ended, could lead to backlash against clinic staff or to caregivers declining to vaccinate infants without incentives. We believe these are important questions and have decided to recommend the program after considering them."

Plans for follow up

• We will continue to have monthly calls with New Incentives and get written monthly program updates from New Incentives. Key questions we plan to discuss include:
  • Is scale up happening at the pace New Incentives projected?
  • Have rotavirus vaccines been introduced in the areas where New Incentives operates?
  • What is New Incentives' plan for improving safeguards against and/or quickly identifying cases of disbursements being made without vaccines being received?
  • What is the status of telecommunications shutdowns in the areas New Incentives operates? What is New Incentives' plan for program continuity during future shutdowns?
  • If stockouts have remained elevated, how is New Incentives responding?
  • Has New Incentives raised funding from non-GiveWell sources?
• For all clinics that New Incentives is starting to partner with as of late 2021, New Incentives will do baseline and follow-up surveys to measure changes in vaccination rates. We have published a page about how we plan to use those results.
• We will periodically track several key metrics (see our most recent update here), based on data the New Incentives routinely collects, including on the proportion of infants that return for later vaccines and stockout rates.
• We will update the cost per infant estimate in our cost-effectiveness analysis with later data.

Internal forecasts

Confidence	Prediction	By time
50%	New Incentives will be unable to operate for >20% of the disbursement days in the clinics it has begun operations in during the first six months of 2022. This may be due to security concerns, telecommunication shutdowns, or other unanticipated factors.	July 2022
35%	Number of stockouts per 100 disbursement days in clinics that New Incentives works in will be >15 in the first six months of 2022, according to New Incentives' data.	July 2022
65%	Rotavirus vaccines will be available and incentivized by New Incentives in >75% of the clinics it operates in.	End of 2022

Our process

Our process for this grant relied heavily on (a) our prior work on modeling the cost-effectiveness of New Incentives, and (b) our monthly discussions with New Incentives and the monthly written progress updates it shares with us. For this particular grant, we:

• Used newly available data to update the cost-effectiveness analysis (discussed above).
• Spot checked New Incentives financial forecasting model.
• Spoke with another funder who recently declined to fund New Incentives.
• Spoke with three Nigeria-based New Incentives staff who manage security and stakeholder relations.

For internal review, a Senior Researcher who was not otherwise involved in the grant investigation gave feedback on the plan for investigating the grant.

Sources

Document	Source
GiveWell, "All Content on New Incentives," 2021	Source
GiveWell, "Allocation of Funds Donated to the Top Charities Fund in Q2 2021"	Source
GiveWell, "New Incentives (Conditional Cash Transfers to Increase Infant Vaccination)," 2021	Source
GiveWell, "New Incentives," 2020	Source
GiveWell, "New Incentives' Coverage Assessments: Plans as of October 2021"	Source
GiveWell, "Our Top Charities"	Source
GiveWell, 2022 V1 to 2022 V2 CEA change tracker	Source
GiveWell, Changelog for cost-effectiveness version 2, 2022	Source
GiveWell, Cost-effectiveness analysis version 2, 2022	Source
GiveWell, New Incentives CEA supplemental information [December 2021] (public)	Source
GiveWell, New Incentives cost per infant immunized, 2021 (public)	Source
GiveWell, New Incentives monitoring results, 2021 (public)	Source
GiveWell, Northern Nigeria mortality adjustment, June 2021, (public)	Source
IDinsight, Coverage Monitoring Analysis Plan, 2021	Source
IDinsight, New Incentives Evaluation Baseline Report, 2019	Source
New Incentives, 2021-2024 Financial Forecasts	Unpublished
New Incentives, Cost Model with Assumptions, 2021	Unpublished
New Incentives, Email to GiveWell, December 16, 2021	Unpublished
New Incentives, Program Updates	Unpublished

• 1

  “Funding Cycle: Q4 2021:

  • $10,018,000 for 532,800 infants
  • Support expansion to 32 new LGAs by the end of 2022
  • The additional funds combined with existing funds would support operations for all 67 LGAs through Dec 2023.” New Incentives, 2021- 2024 Financial Forecasts (unpublished).

• 2

  New Incentives, 2021- 2024 Financial Forecasts (unpublished).

• 3

  See here, GiveWell, Cost-effectiveness analysis version 2, 2022, Sheet: New Incentives
  

• 4 See the previous version of our cost per infant immunized analysis here, and the updated version of our cost per infant immunized analysis here. For more information on this update, see change 2 of version 2 of our 2022 cost-effectiveness analysis: GiveWell, Changelog for cost-effectiveness version 2, 2022
  
• 5
  • See here for the conditional cash transfer commitments by month.
  • “In order to provide a more complete and conservative accounting of NI-ABAE’s finances, we also include “CCT Commitments” in non-GAAP financial reports such as the one you viewed in row 88. These are (a) accruals of the remaining CCT payment schedule for the newly enrolled infants in the month in question, net of (b) the amounts in row 10 above, and also net of (c) any amounts that we previously accrued (in the “CCT Commitments” line) but subsequently determined would not become “8000 Grants (CCTs)” GAAP expenses.” New Incentives, Email to GiveWell, December 16, 2021 (unpublished).

• 6

  For our current estimate, see here. For our previous estimate, see here.
  

• 7

  See here for our estimate of costs paid for by the government and Gavi.
  

• 8

  In our previous estimate, we estimated the costs incurred by the government as $20.04 per additional (counterfactual) full immunization and $6.45 for Gavi per additional (counterfactual) full immunization. In our current estimate, costs incurred by the government are estimated at $48.98 per additional (counterfactual) full immunization and $40.91 for Gavi per additional (counterfactual) full immunization.
  

• 9Overall, these combined updates to our cost per infant immunized analysis led to an increase of 1.5 in our cost-effectiveness estimate for New Incentives. See here.
  
• 10
  See these adjustments here.
  
• 11See change 1 of version 2 of our 2022 cost-effectiveness analysis: GiveWell, Changelog for cost-effectiveness version 2, 2022. Our cost-effectiveness estimate increased from 9.6 to 12.4, a change of ~30%. See here.

• 12

  For the first cohort for which it will conduct these assessments, New Incentives plans to do the baseline assessment in September to November 2021. New Incentives, call with GiveWell, September 7, 2021 (unpublished).

• 13
  • "The first follow-up will occur approximately 12 months after baseline." IDinsight, Coverage Monitoring Analysis Plan, 2021, Pg. 1.
  • "Once operating in all LGAs in an expansion group (usually within six months of completion of baseline), New Incentives will obtain follow-up RI coverage estimates in these LGAs in six months intervals." IDinsight, Coverage Monitoring Analysis Plan, 2021, Pg. 1.

• 14

  See “Retention through vaccination schedule” and “Supply-side issues” sections of our page on New Incentives.

• 15

  New Incentives operated in 98 clinics at the start of 2021. As of October 2021, it was operating in 511. New Incentives, Cost Model with Assumptions (unpublished).

• 16

  See here in GiveWell, New Incentives CEA supplemental information, December 2021.

• 17

  “However, as a result of delays in the vaccine rollout, the rotavirus vaccine has not yet been added to the routine immunization schedule in Nigeria, and New Incentives now guesses that it will not be added to the schedule in the states in which its program operates until 2022.” GiveWell, “Allocation of Funds Donated to the Top Charities Fund in Q2 2021,” 2021.

• 18

  See here.

• 19
  • "Enrollments in September reduced drastically due to the telecom shutdown in Zamfara and parts of Katsina (representing ~40% of all clinics), stockouts (particularly BCG), and delays in expansion due to Rapid Assessments." New Incentives, September 24, 2021 update (unpublished)
  • "There is no change in status of the Telecommunications Shutdown in Katsina and Zamfara." New Incentives, November 19, 2021 update (unpublished).
  • "All states are beginning to restore telecommunications access. Telecom access has been restored in Sokoto and Kaduna while we are seeing gradual restoration in Zamfara and Katsina. We anticipate that most of the LGAs will have restored access by the end of this year." New Incentives, December 17, 2021 update (unpublished).

• 20
  • New Incentives told us that clinics are still open and vaccines are being distributed during the shutdown. Call with New Incentives on October 26, 2021 (unpublished).
  • "We enrolled 14,792 new infants into the program in October 2021, as against 6,885 new infants in September 2021. We had projected to enroll 178,761 infants by the end of October, actual enrollments are 146,401. We have enrolled 152,498 infants into the program in 2021 so far, which is 57.6% of the 264,859 infants goal for 2021. There is no change in status of the Telecommunications Shutdown in Katsina and Zamfara. 3 LGAs in Sokoto have partially rescinded the Shutdown, but these are some of the most security compromised LGAs in the state so the reason for the selection is unclear. With the staff who have temporarily relocated from Zamfara to Sokoto, we have screened 41 clinics and have been training staff for conducting Rapid Assessments. We have continued to ask staff in areas affected by the Telecommunications Shutdown to limit non-essential travel and have hired 2 new Security Focal Points to strengthen our security processes. As part of our initiatives to inform staff and managers about security, monthly reports such as this are disseminated internally." New Incentives, November 19, 2021 update (unpublished)

• 21
  • New Incentives told us that it is piloting methods to enroll children who got BCG during the shutdown and considering strategies for the future. Call with New Incentives on October 26, 2021 (unpublished).
  • "Field teams expect that temporary shutdowns like the telecommunications shutdown may potentially cause greater delayed or reduced vaccination uptake among always-vaccinators than if the program were to cease operations permanently in a given area. This is because caregivers know that the intention of the program is to return and may wait to receive the incentives. We are continuing to experiment long-term solutions for temporary shutdowns to reduce potential negative effects. We are also exploring solutions to get administrative data from clinic staff when they travel outside of the shutdown areas. This will help us understand whether pausing the program temporarily caused vaccination uptake to go below levels prior to when the incentives started. We have some preliminary data on this, but it is very incomplete and we are unable to make any determination using it." New Incentives, November 19, 2021 update (unpublished).

• 22

  "On October 26, 2021, we shared that we estimate needing $811,200 less in funding for 2022 as a result of the telecommunications shutdown. This estimate was based on the number of enrollments we estimate missing because of the telecommunications shutdown (31,200 missed enrollments). We analyzed the number of disbursements we expect previously enrolled infants in areas affected by the telecommunications shutdown to miss due to the telecommunications shutdown, assuming the shutdown lasts 6 months. This figure is $164,600. If we assume that 35% of infants will still return for these vaccination visits and incentives (either because they could not receive the vaccinations due to security issues or delayed vaccination during the shutdown), this figure becomes $106,990. Based on this, we estimate needing $918,190 less in funding for 2022." New Incentives, November 19, 2021 update (unpublished).

• 23

  Percent of disbursement days with stockouts. February 2021: 20.64%, March 2021: 23.94%, April 2021: 11.50%, May 2021: 4.97%, June 2021: 13.78%, July 2021: 9.33%, August 2021: 18.46%. Average = 14.66%. See here.

• 24

  See here.

• 25

  “A meeting of stakeholders in Katsina, Jigawa, Zamfara, and Sokoto States was held on September 16, 2021, in Kano State.… Part of the resolution by the 4 state SMOHs is to visit national stakeholders to advocate for additional vaccines.” New Incentives, September 24, 2021 update (unpublished).

• 26

  Conversation with New Incentives, November 23, 2021 (unpublished).

• 27

  "Auditors identified that there were disbursement days in which disbursements were being given without administration of required vaccines. This was identified in audit visits where the CCT for PENTA/PCV stages was being given on a day when PCV was out of stock. Part of the contributor in this case was incorrect information communicated by the manager. As a result of this, we identified the frequency with which this might have occurred, a summary of the cases from the last 3 years can be found below. The distribution of the vaccine that was out of stock is as follows: PCV(62.5%), PENTA (22.8%), and BCG (14.7%). In total, this represents less than 0.2% of disbursements. This will be closely monitored moving forward. We are looking into improving the identification of such cases by correlating other stockout data and the number of injections reported by caregivers to get a better best guess estimate of the maximum frequency." New Incentives, December 17, 2021 update (unpublished).

• 28

  Call with New Incentives, December 21, 2021 (unpublished).

• 29

  "Most caregivers cited lack of knowledge or ambivalence and relatively few caregivers cited socio-cultural reasons or mistrust and fear, as reasons for not vaccinating. It seems likely that an incentive, coupled with awareness raising activities, can overcome these stated reasons for not vaccinating." IDinsight, New Incentives Evaluation Baseline Report, 2019, p. 71.

• 30

  “We are looking into improving the identification of such cases by correlating other stockout data and the number of injections reported by caregivers to get a better best guess estimate of the maximum frequency." New Incentives, December 17, 2021 update (unpublished).