New Incentives — Nigeria Expansion (November 2022)

Note: This page summarizes the rationale behind a GiveWell-recommended grant to New Incentives. New Incentives staff reviewed this page prior to publication.

Published: January 2023

Planned activities and budget

New Incentives provides cash transfers to incentivize caregivers to bring infants to clinics for routine childhood vaccinations. This CCT program operates in northern Nigeria and seeks to increase uptake of routine immunizations, raise public awareness of the benefits of vaccination, and reduce the frequency of vaccine stockouts.2 This grant will fund expansion of New Incentives' current program to new geographies within Nigeria. Prior to this grant, New Incentives had raised funding to work in 92 local government areas (LGAs) through the end of 2024, primarily through GiveWell and our funding partners.3 The current grant would fund expansion to 67 LGAs by mid-2023, with funding for 12 LGAs through December 2024 and funding for 55 LGAs through December 2025.4 As of the time of this grant, New Incentives expected that the 67 additional LGAs would be located in 6 states5 it has not previously worked in, however, these plans are subject to further discussions with government partners and findings from baseline vaccination coverage assessments.

New Incentives projected that this funding would enable it to reach 1,355,698 additional infants.6 The total funding needed to support a program of this size is $33,892,450, and New Incentives projected that it would have $3,705,800 available from sources other than this grant and from budget surplus, leaving room for more funding in the amount of $30,186,650.7 See our review of New Incentives for a breakdown of spending categories in its program. This grant will support the same program at a greater scale, so we expect the cost structure to be similar.

The case for the grant

• Cost-effectiveness - We estimate that this grant will meet our current bar for cost-effectiveness. More below.
• Learning opportunities - We will have the opportunity to learn about the impact of the grant through New Incentives’ surveys of vaccine coverage before and after it begins working in an area and through its routine monitoring. More below.
• Track record - New Incentives has a track record of fast growth, and we have a strong qualitative assessment of New Incentives. More below.

Cost-effectiveness

Based on our cost-effectiveness analysis of New Incentives, we believe the program is in the range of cost-effectiveness of programs we expect to direct funding to, as of late-2022.8 Our estimate is that this program, in the 6 expansion states, is 14.7 times9 as cost-effective as unconditional cash transfers. We discuss our model of New Incentives’ cost-effectiveness on this page. In considering this grant, we made the following updates to that model:

• Cost to New Incentives per enrolled infant - Our previous estimate was based on New Incentives’ spending through October 2021, when New Incentives was operating at a smaller scale than in 2022.10 We have now updated the model to use data on costs and enrollments through July 2022. We estimate that New Incentives is spending $25 per enrolled infant, down from $29 in our previous estimate.11
• Cost to the government of Nigeria and Gavi - New Incentives pays for the cash incentives but not for the costs to buy and administer vaccines to the additional infants that New Incentives induces to get vaccinated. Our understanding is that these costs are paid for by the government and by Gavi, the Vaccine Alliance. We estimate that the government is spending $34 per enrolled infant (down from our previous estimate of $49) and that Gavi is spending $29 per enrolled infant (down from $41).12 This change is driven by assuming 30% of the total costs of vaccination in Nigeria from 2014-2018 were fixed costs and excluding these from the final estimates.13 We believe it is important to incorporate an estimate of fixed costs into our model because fixed costs, such as the costs of maintaining a national immunization platform, will be incurred by the Nigerian government or Gavi whether New Incentives' program incentivizes additional vaccinations or not. While our confidence in the specific assumption we make about fixed costs is low, we believe incorporating our best guess about fixed costs brings our cost estimates closer to reality than not accounting for them at all. Together, this and the decrease in our estimate of New Incentives’ costs increased cost-effectiveness by an average of ~3.5x cash across the 10 states which were modeled in the prior version of the cost-effectiveness model.14
• Cost-effectiveness by state - Our previous cost-effectiveness model did not consider cost-effectiveness on an individual state-by-state basis. The model now assesses cost-effectiveness for each state, using state-specific disease burden estimates from the Institute for Health Metrics and Evaluation (IHME).15
• Counterfactual vaccination coverage rate - The counterfactual vaccination coverage rate is the vaccination coverage we assume in the absence of New Incentives’ program. Previously, we were using the vaccination coverage rate found in the control group at the end of the randomized controlled trial of New Incentives’ program.16 However, we now have additional data on baseline coverage rates from the surveys New Incentives has been conducting in local government areas (LGAs) that it is expanding to. So far, it has largely been finding that coverage rates at baseline are lower than they were in the control group.17 We have updated the counterfactual coverage rate used in our cost-effectiveness analysis by putting some weight on our prior assumption (from the RCT) and some weight on the more recent surveys. More on our methodology here. This has reduced our estimate of the counterfactual vaccination coverage rate.18 As a result, our estimate of New Incentives’ cost-effectiveness has increased by an average of ~1.6x cash across the 10 states which were modeled in the prior version of the cost-effectiveness model.19 Cost-effectiveness is increased because New Incentives must pay out incentives to everyone bringing their infant to be immunized, regardless of whether or not they would have done so in the absence of the program. Reducing the proportion of individuals who would have brought their infants to be vaccinated in the absence of incentives while maintaining the same absolute increase in proportion of individuals who bring their infants to be vaccinated in response to the incentives reduces the total number of incentives New Incentives pays, thus reducing costs while achieving the same impact.
• Counterfactual value of Gavi spending - We have previously assumed that the counterfactual value of Gavi spending (i.e., the value of Gavi putting a dollar to a use other than funding the additional vaccinations induced by New Incentives' program) is similar to that of the Global Fund’s.20 We have revised this estimate based on our understanding that Gavi is typically successful in fundraising for its full desired portfolio. This implies that we should think of the counterfactual value of a Gavi dollar as reflecting other opportunities Gavi donors would fund, rather than Gavi programs. The methodology behind the new estimate is further described here. This new estimate is lower, meaning that even more value is derived from New Incentives leveraging Gavi funds. This has increased our estimate of the cost-effectiveness of New Incentives’ program by about 1.1x across states.21

Learning-opportunities

We will have the opportunity to learn about the impact of the grant through New Incentives' surveys of vaccine coverage before and after it begins working in an area and through its routine monitoring.22 New Incentives plans to reassess vaccination coverage after it has worked in each of the new areas for a year, and then every six months.23 See this page for how we expect to use findings from these assessments to update our cost-effectiveness analysis of New Incentives' program.

New Incentives’ track record

New Incentives has a track record of fast growth,24 and this grant is given under the expectation that it can continue a quick pace of growth over the next year.

We have a strong qualitative assessment of New Incentives after 5+ years of working together on its conditional cash transfer program.25 We believe New Incentives stands out for its focus on maximizing cost-effectiveness and its dedication to identifying, responding to, and being transparent about issues it faces. For examples of how New Incentives approaches issues that arise, see our "Risks and reservations" section below.

Risks and reservations

• Rapid scale up – Supporting a rapidly growing organization means we are increasing funding while not yet being able to learn much about the impact of the funds we've contributed before. An alternative to making this grant would be to wait and get more results before funding New Incentives to expand further. In particular, we do not yet have updates on how vaccination rates change after New Incentives begins operating in an area. We expect to have an update on this in the next year, when results from the first vaccination coverage reassessments become available.26
• Impact of organizational scale on program quality - Since the RCT ended in February 2020, New Incentives has grown to work in ~29 times as many clinics as it was working in during the RCT.27 Given the complex nature of the program (e.g., relative to mass distribution of health commodities) and the fact that it distributes cash (incentivizing various people involved to attempt to defraud the program), it seems intuitive that it would be very difficult to deliver the same quality of results as during the RCT. New Incentives has already uncovered a number of problems and instituted changes in how it monitors and incentivizes staff performance.28 As a result of these concerns, we considered whether an adjustment to the improvement in vaccination coverage observed in the RCT was warranted. However, we decided not to make an adjustment at this time as the data we have does not yet clearly support the need for an adjustment, and we were not able to formulate a strong theory of change for why the improvement in vaccine coverage would be substantially lower than in the RCT. Key evidence included:
  • Retention metrics: We have not yet seen any meaningful impact on retention rates (i.e., the frequency with which enrolled infants are brought back for subsequent vaccinations) as New Incentives scales.29 We view retention rates as an intermediate metric - short of coverage rates - for whether the program is delivering results, and would expect retention to be reduced if program quality was substantially compromised.
  • Fraud metrics: New Incentives collects data on fraud metrics like the number of children that appear to have more than one scar from BCG vaccination (for tuberculosis) –an indicator of children being enrolled in the program twice– and fraud observed by auditors and supervisors who visit clinics and attempt to anonymously observe operations. There's no indication in the fraud metrics that we've been tracking that fraud has been increasing.30 For example, metrics on the frequency of repeat enrollments suggest low rates of repeat enrollments, consistent with the adjustment we use for this risk in our cost-effectiveness analysis.31
  • Impact of vaccine stockouts: The rates of vaccine stock outs have increased relative to those experienced during the RCT. During the RCT there were ~8 stock outs of any vaccine per 100 disbursement days. In the period since the RCT, this has increased to ~19 stock outs of any vaccine per 100 disbursement days. However, the increase in stock outs has primarily been driven by an increase in stock outs of indirectly incentivized vaccines (~1 stock out per 100 disbursement days during the RCT and ~9 in the period since), whereas stock outs of directly incentivized vaccines (which make up the majority of the program's benefits in our cost-effectiveness analysis) have increased only modestly (~7 stock outs per 100 disbursement days during the RCT and ~10 in the period since).32 We roughly modeled the maximum impact of the increase in stockouts of directly incentivized vaccines on cost-effectiveness using conservative assumptions33 and found the impact on coverage to be only ~0.3 percentage points.34 Based on this finding and the fact that we have not seen a notable decrease in retention rates since the RCT (see above), we do not believe that increases in rates of stock outs are meaningfully impacting the program's cost-effectiveness.
  • Conversations with New Incentives staff: We spoke with two Nigeria-based New Incentives operations staff, in addition to our monthly calls with New Incentives’ CEO and COO. These conversations bolstered our confidence that New Incentives is being very proactive in mitigating the risks of scaling, including efforts to reduce fraud, maintain infant retention, and address organizational and managerial challenges.35 This is supported by the available data, per the above. Some select examples from our discussions:
    • The amount of time that New Incentives staff spends with each caregiver when disbursing the incentive has increased substantially. This allows staff to 1) ask probing questions which can help identify any fraud and 2) educate on the importance of vaccination and the date of the next immunization which can help support retention.36
    • New Incentives has implemented a new staff performance management system as of June 2022 which increases oversight of staff activities with the goal of ensuring continued quality with scale.37
• Expansion to areas with lower potential for impact - It is possible that, as New Incentives expands to states beyond the original three states included in the RCT,38 , the impact of the program will decrease. This could be due to baseline vaccination rates already being higher in these areas, or because of more strongly-held reasons for not vaccinating. New Incentives expects to see similarly low vaccination rates in the expansion states. We will learn more about baseline coverage rates in the expansion areas prior to New Incentives beginning operations there and New Incentives will only move forward with expansion if the baseline coverage suggests cost-effectiveness above 10x cash (more on this in “Plans for follow up”). However, we will only learn more about whether other factors are affecting the impact of the program in new areas after New Incentives has conducted follow up surveys. New Incentives’ security manager told us that the security risks in the expansion states are unlikely to provide challenges for the program beyond those encountered in existing states.39
• Impact of New Incentives' program on vaccine supply - As New Incentives expands to more states, vaccine demand will continue to grow and increasing pressure may be put on vaccine supply at all levels, including nationally. National vaccine supply dynamics is an area of limited visibility for both GiveWell and New Incentives. This is an area that we will continue to monitor and hope to learn more about. For the time being, our understanding is that the government has been responsive to increased demand.40 At the state and LGA level (where New Incentives supply side officers play a more active role – see our description of New Incentives' supply side work here) there is a risk that limited supply may be disproportionately funneled to LGAs where New Incentives is operating.41 This would decrease rates of stockouts in New Incentives LGAs but cause harm by increasing rates of stockouts elsewhere. New Incentives notes that its supply side work supports all LGAs in a state—not just those where the program is operating—and that it is unlikely that supply would be reallocated from one LGA to another.42
• Impact of inflation on incentive value - Judging from national estimates of the cost of goods in Nigeria, the value of the incentive to the average Nigerian went down by about 25% between 2019 (during the RCT) and 2021 (most recently available World Bank data).43 We are concerned that a large enough reduction in the value of the incentive due to inflation could reduce the effect size of the program (i.e., the % improvement in vaccination coverage). New Incentives believes that cost increases for the largely rural populations that it serves are likely below the national average, and notes that infant retention rates have remained high.44

Plans for follow up

• We will work with New Incentives to determine what value of baseline vaccination coverage to use as a maximum for determining whether or not New Incentives will expand in a given set of LGAs. As needed, funding will be reallocated to areas with lower baseline coverage and thus with cost-effectiveness above GiveWell’s bar. The expected updates that may change our perspective on cost-effectiveness are:
  • Higher quality data on vaccine coverage rates in the expansion areas than we've seen to date, which will be collected by New Incentives prior to starting work in each area
  • Planned changes to our cost-effectiveness analysis which will assume a smaller effect size (i.e., the % improvement in vaccination coverage) in areas with higher baseline coverage rates relative to the control group during the RCT45
• We will continue to have monthly calls with New Incentives and get written monthly program updates. Key questions we plan to discuss include:
  • Is scale-up happening at the pace New Incentives projected?
  • What proportion of disbursement days have been missed for any reason? What are the implications for New Incentives' room for more funding?
  • Which sets of LGAs has New Incentives decided to start working in? How do the baseline vaccination rates in these cohorts compare to coverage rates in the control group during the RCT?
  • Are rotavirus vaccines being given at most disbursement days?
  • What forms of fraud has New Incentives seen any evidence of?
  • If stockouts have remained elevated, how is New Incentives responding?
  • What have New Incentives’ retention rates been in relation to what they were during the RCT?
  • What are the results of the assessments of vaccine coverage before and after New Incentives starts operating in an LGA? Are we able to fully understand and have confidence in the results?
  • How does the real value of the incentives now compare to what it was during the RCT? How is New Incentives responding?
  • Has New Incentives raised funding from non-GiveWell sources?
  • How has the nature of government opposition to New Incentives’ program changed (more)? How, if at all, is it affecting New Incentives’ operations?
• We will periodically track several key metrics (see our most recent update here), based on the data that New Incentives routinely collects, including on the proportion of infants that return for later vaccines and stockout rates.
• We will continue to update the cost per infant estimate in our cost-effectiveness analysis on an at least annual basis to monitor for changes.

Internal forecasts

Confidence	Prediction	By time​​
55%	New Incentives will have an average measles retention rate >85% in 2023. (Measles retention during the RCT was 91%. It has been 87% since the end of the RCT and was 86% in the first 7 months of 2022.)	End of 2023
50%	Number of stockouts of directly incentivized vaccines per 100 disbursement days in clinics that New Incentives works in will be >10, according to New Incentives' data. (This figure was 6.8 per 100 days during the RCT. Since the RCT ended, it's been 9.5 per 100 days. In the first 7 months of 2022, it was 9.2 per 100 days.)	End of 2023
50%	Rotavirus vaccines46 will be in stock at >80% of the disbursement days that New Incentives attends in 2023.	End of 2023
80%	Rotavirus vaccines will be in stock at >60% of the disbursement days that New Incentives attends in 2023.	End of 2023
55%	New Incentives, with input from GiveWell, will implement the program in all six of the new expansion states (Adamawa, Kaduna, Kano, Kebbi, Taraba, and Gombe) after assessing baseline vaccination coverage.	End of 2023
70%	New Incentives, with input from GiveWell, will implement the program in at least four of the new expansion states (Adamawa, Kaduna, Kano, Kebbi, Taraba, and Gombe) after assessing baseline vaccination coverage.	End of 2023

Our process

Our process for this grant relied heavily on (a) our prior work on modeling the cost-effectiveness of New Incentives, and (b) our monthly discussions with New Incentives and the monthly written progress updates it shares with us. For this particular grant, we:

• Updated several parameters in our cost-effectiveness model (discussed above).
• Reviewed New Incentives’ recent results on a number of metrics (stockouts, retention, fraud, etc.) and compared them to levels during the RCT. Metrics that raised concern are discussed above.
• Spot checked New Incentives' financial forecasting model.
• Spoke with Nigeria-based New Incentives staff: program operations staff, New Incentives’ security manager, and staff involved in supply side work.
• Spoke with an immunization expert who works in Kaduna (one of the expansion states).

For internal review, two Senior Researchers and a Senior Research Associate who were not otherwise involved in the grant investigation gave feedback on the plan for investigating the grant.

Sources

Document	Source
Gavi, "Dealing with diarrhoea: Nigeria introduces rotavirus vaccine into its immunisation plan," August 2022	Source (archive)
GiveWell, "2023 Cost-Effectiveness Analysis changelog"	Source
GiveWell, "All content on New Incentives"	Source
GiveWell, "New Incentives — Nigeria expansion (May 2022)"	Source
GiveWell, "New Incentives (conditional cash transfers to increase infant vaccination)"	Source
GiveWell, "New Incentives' Coverage Assessments: Plans as of October 2021"	Source
GiveWell, "New Incentives"	Source
GiveWell, 2022 V5 to 2023 V1 CEA change tracker	Source
GiveWell, Cost-effectiveness analysis – version 1, 2023	Source
GiveWell, Cost-effectiveness analysis – version 5, 2022	Source
GiveWell, Cost-effectiveness of November 2022 New Incentives grant	Source
GiveWell, Effect of stockouts on vaccination rates [October 2022]	Source
GiveWell, Gavi counterfactual value, 2022	Source
GiveWell, New Incentives CEA - Baseline coverage rates update, 2022	Source
GiveWell, New Incentives CEA supplemental information [August 2022]	Source
GiveWell, New Incentives cost per infant immunized [2021]	Source
GiveWell, New Incentives cost per infant immunized [August 2022]	Source
GiveWell, New Incentives monitoring results [July 2022]	Source
IDinsight, Coverage Monitoring Analysis Plan, 2021	Source
IDinsight, Impact evaluation of New Incentives, Final Report, 2020	Source

• 1

  Exact grant amount: $30,186,650.

• 2

  "New Incentives runs a conditional cash transfer program in North West Nigeria. Caregivers who bring their infants to clinics for routine vaccines, which are provided through government clinics free of charge, can receive a total of $11 over the course of five clinic visits. New Incentives also does outreach to caregivers about the importance of vaccinating children, and works with its government partners to improve vaccine supply by identifying and addressing bottlenecks in the vaccine supply chain." See our review of New Incentives for more details.

• 3

  Givewell previously recommended a grant in May 2022 that would fund "expansion to 25 new LGAs by the end of 2022 and funding for these LGAs in 2023; all 92 LGAs in 2024 (combined with existing funds)." See our grant page about this grant here.

• 4

  "Funding Cycle: Q3-Q4 2022: Description: Expand from 92 LGAs to a total of 159 LGAs (67 new LGAs) by mid 2023. The additional funds combined with existing funds would support operations for all 159 LGAs through December 2024 and 55 LGAs (Jan-Jun 2023 expansion) through December 2025 based on the results of coverage surveys and other critical operational factors." New Incentives, 2021-2025 Financial forecasts (unpublished). We confirmed the latest plans on a call with New Incentives on November 22, 2022. New Incentives, conversation with GiveWell, November 22, 2022 (unpublished).

• 5

  The six states are Adamawa, Kaduna, Kano, Kebbi, Taraba, and Gombe. New Incentives, email to GiveWell, November 11, 2022 (unpublished).

• 6

  “Funding Cycle: Q3-Q4 2022: Description: . . . Enroll a minimum of 1,355,698 additional infants by the end of December 2025." New Incentives, 2021-2025 Financial forecasts (unpublished).
  

• 7

  New Incentives, 2021-2025 Financial forecasts (unpublished).
  

• 8Note that a) our cost-effectiveness analyses are simplified models that are highly uncertain, and b) our cost-effectiveness threshold for directing funding to particular programs changes periodically. As of late 2022, our bar for directing funding is about 10x as cost-effective as GiveDirectly's unconditional cash transfers. See GiveWell’s Cost-Effectiveness Analyses webpage for more information about how we use cost-effectiveness estimates in our grantmaking.
  
• 9See calculation here.
  
• 10See this grant page for more information about the previous expansion of New Incentives' program.
  
• 11See the previous version of our cost per infant immunized analysis here, and the updated version of our cost per infant immunized analysis here. For more information on this update, see change 3 of version 1 of our 2023 cost-effectiveness analysis here.
  
• 12See the previous version of our cost per infant immunized analysis here, and the updated version of our cost per infant immunized analysis here. For more information on this update, see change 3 of version 1 of our 2023 cost-effectiveness analysis here.
  
• 13See here.
  
• 14See here. 3.5x cash represents the unweighted average increase in cost-effectiveness across the 10 states with prior cost-effectiveness estimates (Adamawa, Bauchi, Gombe, Jigawa, Kano, Katsina, Kwara, Sokoto, Taraba, and Zamfara). This excludes Kaduna and Kebbi which had not previously been modeled, and includes states which have been supported by prior grants but are not explicitly supported by this grant.

  Average: (2.9 + 3.9 + 3.5 + 4.2 + 3.5 + 3.7 + 1.5 + 4.1 + 3.0 + 4.2)/10 = 3.45
  

• 15For more information on this update, see change 7 of version 1 of our 2023 cost-effectiveness analysis here. See our new cost-effectiveness model here. See the state-specific disease burden estimates that inform that model here.
  
• 16See the previous version of our cost-effectiveness analysis here.
  
• 17For a comparison of the new data on baseline coverage rates to the coverage rates from the RCT, see the "Counterfactual vaccination coverage - NI" sheet of this spreadsheet.
  
• 18Our estimate of the counterfactual vaccination coverage rate was reduced from 48% to 39%. More information here. See our calculations here.
  
• 19See here. 1.6x cash represents the unweighted average increase in cost-effectiveness across the 10 states with prior cost-effectiveness estimates (Adamawa, Bauchi, Gombe, Jigawa, Kano, Katsina, Kwara, Sokoto, Taraba, and Zamfara). This excludes Kaduna and Kebbi which had not previously been modeled, and includes states which have been supported by prior grants but are not explicitly supported by this grant.

  Average: (1.2 + 1.9 + 1.6 + 2.1 + 1.6 + 1.7 + 0.2 + 2.0 + 1.3 + 2.1)/10 = 1.57
  

• 20See the previous version of our cost-effectiveness analysis here.
  
• 21See here.
  
• 22See the latest New Incentives monitoring results through July 2022 here. An overview of its monitoring practices is on our charity review page here.
  
• 23

  "The first follow-up will occur approximately 12 months after baseline." IDinsight, Coverage Monitoring Analysis Plan, 2021, p. 1.

  "Once operating in all LGAs in an expansion group (usually within six months of completion of baseline), New Incentives will obtain follow-up RI coverage estimates in these LGAs in six months intervals." IDinsight, Coverage Monitoring Analysis Plan, 2021, p. 1.
  

• 24New Incentives operated in 98 clinics at the start of 2021. As of October 2021, it was operating in 511. New Incentives, Cost Model with Assumptions (unpublished). As of October 2022, it was operating in 2,796. New Incentives, Program updates (unpublished).
  
• 25"New Incentives started this program in May 2017." GiveWell, "New Incentives"
  All GiveWell-directed grants to New Incentives can be found on this page.
  
• 26"New Incentives plans to group the Local Government Areas (LGAs) to which it is expanding into cohorts. For each cohort, it plans to:
  1. Conduct a baseline assessment of vaccination coverage. New Incentives will use the results of the baseline assessment when considering which LGAs to work in.
  2. Conduct follow-up assessments of vaccination coverage in the LGAs it works in, one year after the baseline assessment and every six months after that.

  For the first cohort for which it will conduct these assessments, New Incentives plans to do the baseline assessment in September to November 2021, and to begin enrolling infants in the program in these LGAs in the three months following the baseline assessments." GiveWell, "New Incentives' Coverage Assessments: Plans as of October 2021"
  

• 27
  • "The RCT window ran from July 2018 to October 2019. The endline survey took place from November 2019 to February 2020." IDinsight, Impact evaluation of New Incentives, Final Report, 2020
  • New Incentives worked in “98 clinics during the RCT period.” As of October 24, 2022, “The program is now operating in 83 LGAs in Nigeria, covering 2,796 clinics.” New Incentives, Program Updates (unpublished).

• 28

  "We carried out a detailed review of managerial field oversight and introduced major changes in June after several months of insufficient improvement. We developed a framework to assess managers as we were increasingly finding significant gaps in managerial oversight, including Field Officers and other staff not carrying out expected activities and managers not being able to ensure that their team members carry out expected activities. It is very important for managers to meet minimum performance standards in order to scale program quality. The framework provides the organization objective metrics to take into account when assessing manager field oversight performance. Managers who cannot meet the minimum passing score of 70% for their teams will face employment consequences by July 15. We have tested and assessed related strategies in depth. This is a critical change for New Incentives to make at this stage before additional LGAs are expanded to." New Incentives, Program updates (unpublished)

  "We are detecting increased reports of clinic staff forcing caregivers to purchase certain items (ex:paracetamol) as a requirement to provide immunization services. This forces caregivers to share part of the incentive [...] When we detect these types of issues on a repeated basis, based on a process reviewed by State stakeholders, we engage LGA officials and clinic staff to discuss the issues (type of issue and occurrences) and ask them to sign a letter acknowledging the issue and committing to stop the behavior. We have encountered cases where the issue either persists or clinic staff refuse to sign the letter and threaten to start charging more. In such cases, we temporarily pause cash disbursements at the clinic in order to ensure that the issue gets resolved and the behavior does not spread to other clinics. We have paused cash disbursements at 8 clinics so far." New Incentives, Program updates (unpublished)

• 29See the latest New Incentives monitoring results through July 2022 here.

• 30

  See the latest New Incentives monitoring results through July 2022 here. Note that we may consider tracking more or different metrics in the future, as we are not confident that the metrics currently tracked are holistic.

• 31

  See the latest metrics on repeat enrollments here and see the adjustment we make for repeat enrollments in our cost-effectiveness analysis here.

• 32

  See the latest New Incentives monitoring results for stock outs through July 2022 here.

• 33

  We assumed the following: 1) Stockouts of directly incentivized vaccines on 8.6% of disbursement days (see here), 2) only 50% of infants who are not served due to a stockout return to be immunized another day (we believe this is conservative because preliminary, unpublished data shared with us by New Incentives suggests this is likely higher e.g., ~80%). We also did not account for the possibility that stockouts may sometimes be of only one or two vaccines rather than all vaccines we are modeling the benefit of – accounting for this likelihood would decrease the impact of stockouts on cost-effectiveness because instead of assuming that children encountering stockouts are not vaccinated at all and receive none of the associated benefits, we would be assuming that a subset of children still do receive incentivized vaccines that are not stocked out, and thus receive part of the modeled benefits.

• 34See calculations here.

• 35

  New Incentives operations staff (Nigeria-based), conversations with GiveWell, October 2022 (unpublished)

• 36

  The individual we spoke with cited an increase from 1.5 to 7 minutes, though we have not validated that this is standard practice through additional conversations. New Incentives operations staff (Nigeria-based), conversation with GiveWell, October 2022 (unpublished)

• 37"We carried out a detailed review of managerial field oversight and introduced major changes in June after several months of insufficient improvement. We developed a framework to assess managers as we were increasingly finding significant gaps in managerial oversight, including Field Officers and other staff not carrying out expected activities and managers not being able to ensure that their team members carry out expected activities. It is very important for managers to meet minimum performance standards in order to scale program quality. The framework provides the organization objective metrics to take into account when assessing manager field oversight performance. Managers who cannot meet the minimum passing score of 70% for their teams will face employment consequences by July 15. We have tested and assessed related strategies in depth. This is a critical change for New Incentives to make at this stage before additional LGAs are expanded to." New Incentives, Program updates: July 5, 2022 update (unpublished)
  
• 38The three states included in the RCT were Katsina, Zamfara, and Jigawa. "This study is an impact evaluation of the NI-ABAE CCTs for RI Program in Katsina, Zamfara, and Jigawa States in North West Nigeria[.]" IDinsight, Impact evaluation of New Incentives, Final Report, 2020

• 39

  New Incentives’ security manager, conversation with GiveWell, September 29, 2022 (unpublished)

• 40

  This is our understanding based on communications with New Incentives. New Incentives, email to GiveWell, October 24, 2022 (unpublished)

• 41

  This may be a risk if New Incentives' supply side efforts to improve vaccine availability are disproportionately focused on clinics or local government areas (LGAs) where it operates (e.g., requesting and receiving additional vaccine supply at the cost of other clinics or LGAs which are not as proactive in requesting the necessary supply).

• 42

  “Vaccines are allocated through a calculation made based on factors such as estimated population and consumption. Each LGA is allocated a consignment per quarter in which maximum and minimum stock levels are indicated. SCCOs can authorize LGAs to conduct vaccine ‘mopping’ in which LGAs can request stock from neighboring LGAs with an excess stock, but we do not expect the SCCO to authorize such a request if it would compromise those minimum levels. In general, LGAs are hesitant to participate in vaccine mopping to preserve adequate stock for their clinics and because they have no incentive to share their stock. NI promotes vaccine supply in all LGAs through transportation support to pull supplies (typically from the State’s Cold Chain storage) in both NI- and non-NI LGAs when needed. We are cognizant of the possibility that LGAs where we operate might get prioritized and try to take measures to avoid this.” New Incentives, email to GiveWell, October 24, 2022 (unpublished)

• 43See monitoring on inflation here.

• 44New Incentives, conversation with GiveWell, September 27, 2022 (unpublished)

• 45To date, we have assumed a flat percentage point increase in vaccination coverage based on the RCT results. However, as New Incentives moves into areas with varying levels of baseline coverage, we expect that we will want to reduce the effect size for areas with higher baseline coverage.
  
• 46
  • In August 2022, rotavirus vaccine rollout was officially launched and the government announced that it would reach all 36 states in the next year, with 19 states in the northern part of the country being prioritized. "Dr Shuaib says: 'The vaccine rollout is supported by development partners including Gavi, the United Nations Children's Fund, the World Health Organization, and the Bill and Melinda Gates Foundation. The vaccine is expected to be given to about seven million children across the 36 states in Nigeria for free over the next year.' He goes on to say that the vaccine distribution will take place in phases, starting immediately with the 19 northern states with low immunisation coverage as well as the Federal Capital Territory." Gavi, "Dealing with diarrhoea: Nigeria introduces rotavirus vaccine into its immunisation plan," August 2022
  • As of November 2022, New Incentives wrote “We expect that the rotavirus vaccine has started getting administered at approximately 89% of operational clinics[.]” New Incentives, Program updates: November 21, 2022 update (unpublished)