New Incentives

Donate

What is on this page: New Incentives is one of GiveWell’s top charities. The main factor driving our grantmaking to New Incentives is an analysis of its program’s cost-effectiveness. We summarize our cost-effectiveness analysis in detail in our separate report. This page provides additional information on New Incentives’ program, our qualitative assessment of New Incentives, and the monitoring and evaluation data it shares. This information feeds into our overall recommendation (alongside cost-effectiveness) and provides additional context so that we can understand and appropriately model the impact of its program.

Table of Contents

Published: April 2024

Summary

What do they do? New Incentives runs a conditional cash transfer program in northern Nigeria. The program aims to increase uptake of routine vaccinations (which is low in this part of Nigeria) through small cash incentives (worth approximately $7 in total). In areas where it works, New Incentives also conducts awareness-raising activities to encourage vaccination, and various activities to strengthen the vaccine supply chain. (More)

We recommend New Incentives’ program as a top charity because of its:

  • Sole focus on a program that we think is very cost-effective (more).
  • Standout transparency and commitment to making its program cost-effective (more).
  • Processes for gathering and sharing detailed monitoring data on all aspects of its program (more).

Our main reservations about New Incentives are:

  • New Incentives is at an earlier stage of its development compared to GiveWell’s other top charities. While it has grown successfully in recent years, it might be more susceptible to unexpected problems or a decline in the quality of program delivery as it grows further. (More)
  • New Incentives’ program (which handles a large volume of cash directly) poses a higher risk of fraud than other programs that GiveWell funds. (More)
  • There is some opposition to New Incentives’ program within the Nigerian federal government, which poses a potential threat to the sustainability of the program long-term. (More)

These assessments are based on the following components:

What do you get for your dollar? GiveWell believes New Incentives is one of the most cost-effective programs donors can support. We estimate that it costs roughly between $1,000 and $5,000 (varying by state) to avert a death in areas where New Incentives works. We think New Incentives’ program is cost-effective because it leads to a substantial increase in vaccination rates (11 to 22 percentage points depending on the location) at moderate cost (approximately $20 per child enrolled, and ~$40 to $140 per additional child vaccinated), and vaccines are effective at protecting against child mortality.

We also have a number of uncertainties about our analysis of the cost-effectiveness of New Incentives’ program. These include:

  • Whether vaccine efficacy is lower in Nigeria than suggested by evidence from other locations.
  • Whether our analysis is adequately accounting for increases in vaccination rates that would have taken place anyway, regardless of the program.

See our separate report on the cost-effectiveness of its program for more details.

What information has New Incentives shared about its program? We ask organizations that we fund to share monitoring data and other detailed information on their programs. We use the data as inputs in our cost-effectiveness analysis, and its quality and reliability also inform our overall assessment of the program. Our analysis of New Incentives’ cost-effectiveness is based on a randomized controlled trial (RCT). New Incentives has collected several monitoring indicators that allow us to compare performance of the program today to performance in the RCT. Overall, our assessment of the data is that New Incentives has maintained a high level of program quality at scale, similar to the level achieved in the RCT, although we have seen a rise in supply problems for some vaccines. We will continue to monitor this issue as New Incentives grows further, since we think programs often face unexpected challenges as they scale. (More)

New Incentives’ monitoring data does not show evidence of increasing fraud since the RCT. But we believe this is inherently a high risk for its program, and we’re still uncertain whether there are types of fraud that wouldn’t be captured by this data. (More)

New Incentives also conducts follow-up surveys to assess vaccination coverage after it has worked in each of the new areas for a year, and then every six months. We have only just begun analyzing these surveys in detail, but expect them to provide useful information about the program’s effect in the future. (More)

What is GiveWell’s qualitative assessment of New Incentives? We make qualitative assessments of our top charities alongside our cost-effectiveness analyses to inform our grantmaking. Overall, our qualitative assessment of New Incentives is very strong, even compared to our other top charities. Factors we take into account for this include (more):

  • We think New Incentives stands out for its dedication to identifying, responding to, and being transparent about issues it faces. For example, it has proactively brought challenges it is facing to our attention even when they might negatively affect our view of its program.
  • We think that New Incentives shares GiveWell’s focus on evidence-based decision making and maximizing the cost-effectiveness of its program to an unusual level among organizations that GiveWell funds.

1. What do they do?

1.1 In a nutshell

New Incentives runs a conditional cash transfer (CCT)1 program in northern Nigeria. The program aims to increase uptake of routine childhood vaccinations (which is low in this part of Nigeria)2 through small cash incentives. Caregivers who bring their children for routine vaccines, which are provided through government clinics free of charge, can receive a total of 6,000 naira3 (about $7, as of December 2023) over six visits.4

In addition to disbursing incentives, New Incentives also:

  1. Conducts awareness-raising activities to inform caregivers about the program and the importance of vaccinating children. (More)
  2. Works with the government and other partners to ensure sufficient vaccine supply by identifying and addressing bottlenecks in the vaccine supply chain. (More)

New Incentives started this program in May 2017.5 The program was evaluated by a randomized controlled trial (RCT) from August 2017 until February 2020, which found that it increased vaccination rates of key childhood vaccines by 14-21 percentage points.6 As of January 2024, New Incentives’ program is operating in nine states in Nigeria.7 Between January and November 2023, New Incentives enrolled 1.39 million infants in its program.8

In Nigeria, New Incentives is known as the All Babies Are Equal (ABAE) Initiative.9

States where New Incentives operates its CCT program, as of January 2024, are highlighted in blue.10

1.2 Vaccines incentivized

New Incentives directly incentivizes (i.e., provides cash transfers conditional on children receiving) the following vaccines:

  • Bacille Calmette-Guérin vaccine (BCG) against tuberculosis
  • Pentavalent vaccine (Penta) against diphtheria, tetanus, pertussis (whooping cough), hepatitis B, and Haemophilus influenzae type b
  • Pneumococcal conjugate vaccine (PCV) against pneumococcal disease
  • Measles (MCV) vaccine

Some of these vaccines are delivered in multiple doses. Other vaccines (Hepatitis B, OPV and IPV (the vaccines against polio), meningitis A, yellow fever, and rotavirus11 ) are delivered at the same visits as the directly incentivized vaccines. We consider these vaccines to be "indirectly incentivized" because we’d expect the cash incentives to increase the number of children who receive these vaccines too. See the vaccination schedule below.12

Age of visit Directly incentivized Indirectly incentivized
Birth BCG Hep B, OPV0
6 weeks Penta1, PCV1 OPV1, Rotavirus
10 weeks Penta2, PCV2 OPV2, Rotavirus
14 weeks Penta3, PCV3 OPV3, IPV, Rotavirus
9 months Measles1 Yellow Fever, Meningitis A
15 months Measles213 n/a

Caregivers receive a portion of the overall cash incentive at each visit. As of December 2023, New Incentives offers caregivers 1,000 naira14 (about $1.10) per visit. This incentive schedule has been updated over time to account for inflation and increase caregivers’ motivation to receive all recommended vaccines (details in footnote).15

1.3 How does the program work?

The main goals of New Incentives' program are:

  1. Increasing vaccine demand by (a) distributing cash incentives and (b) conducting awareness-raising activities.
  2. Ensuring sufficient vaccine supply by supporting the government and other partners to identify and address bottlenecks in the vaccine supply chain.

Distributing cash incentives

Vaccines incentivized by New Incentives are administered by the staff of government clinics that partner with New Incentives (more). Vaccinations and cash disbursements take place on days during which clinics provide routine immunizations ("immunization days"). Some children are also reached through outreach efforts targeted at communities with particularly low vaccination rates. (More)

After a child is vaccinated, New Incentives checks whether the child meets the eligibility criteria to receive a cash incentive (more) and, if so, disburses the incentive to the child's caregiver (more). New Incentives also conducts audits and remote vetting of cash disbursements to prevent fraud and ensure disbursements reach eligible caregivers (more).

Partnership with clinics

New Incentives pursues partnerships with clinics that16 :

  • Offer routine immunization services
  • Are located in an area where the clinic is safe to visit during the day
  • Are public clinics
Eligibility for enrollment

Children are usually enrolled in New Incentives' program during their first vaccination visit (for BCG, for example, see the schedule above17 ), with one exception (more below). The eligibility criteria for an infant to enroll in the program are:

  1. The infant's caregiver lives in a local government area (LGA) where the program is operating. This is checked by looking at the residence (settlement) reported on the Child Health Card (CHC)18 by the clinic staff and by asking questions of caregivers.19
  2. The infant received a BCG vaccine on that immunization day (and has not previously received a BCG vaccine). Receipt of the BCG vaccine is used as a criterion for enrollment because it is the first vaccine in Nigeria's routine immunization schedule (scheduled to be given shortly after birth).20 Infants brought in later (up to 12 months of age) will still be administered the BCG vaccine at their first clinic visit and can be enrolled in the program at that point.21 New Incentives uses a fresh BCG vaccination mark as evidence that a child has been vaccinated with BCG that day, and it uses the scar left by the BCG vaccine as a way of checking that the infant has not previously received BCG.22
  3. The caregiver gives consent to collect their and their infant’s personal data.23

An exception applies to these eligibility criteria:24

  • In the first 6 months of operations at a newly expanded clinic, infants who received the BCG vaccine before New Incentives' expansion can be enrolled in the program if the infant’s name was previously recorded in the Child Immunization Register (CIR), a ledger that tracks which infants have received vaccinations at the clinic.

Our understanding is that this exception, introduced in 2022, is designed to reach additional children who could benefit from the program, but who would otherwise not be eligible (e.g., because they received BCG before New Incentives began operating in a particular area). The proportion of children enrolled under this exception (i.e., those who received BCG before they were enrolled) was 20% in 202225 and stayed high throughout 2023 (18% - 29% depending on the month in 2023).26 We see this as potentially concerning (more below), but we did not have strong prior expectations for how high this proportion should be. As of the time of writing we have not yet investigated the implications of this in depth, and are considering whether to make an adjustment to our cost-effectiveness analysis to account for this.

Eligibility to receive cash transfers

To be eligible to receive a CCT, an infant must meet the following criteria27 :

  1. Be enrolled in New Incentives' program. New Incentives checks that infants are enrolled by asking caregivers to provide their All Babies Are Equal (ABAE) ID, a unique ID provided by New Incentives at the point of enrollment (more below).28 This ensures New Incentives can identify each caregiver and avoid disbursing incentives to caregivers whose infants are not eligible for the program.
  2. Have received a directly incentivized vaccination during the same immunization day. New Incentives verifies this by checking that there is a vaccination marked with the current date and a golden dot on the child's Child Health Card (a document provided to caregivers that reports the infant's residence, date of birth, and vaccinations the infant has received on which dates).29 The golden dot comes from a golden pen provided by New Incentives’ staff. Clinic staff mark each child’s Child Health Card with the pen to indicate that a child has received a vaccine.30
  3. Have had both arms checked for a BCG scar, and have had the injections reported on their Child Health Card confirmed with their caregiver.31 Our understanding is that New Incentives checks for BCG scars to assess whether infants are being enrolled in the program more than once, and confirms the injections reported on the Child Health Card with caregivers to ensure that the reported vaccinations were actually administered.
  4. Have had a New Incentives Field Officer directly observe their vaccination.32
  5. Be age eligible (e.g., under 1 year old for new enrollments, older than 9 months if receiving the first dose of the measles vaccine).33 New Incentives verifies this by comparing the current date with the date of birth reported on the Child Health Card.34
Immunization days

Cash incentives are disbursed during immunization days.35 The process is:

  1. New Incentives assigns one or more Field Officers (FO) per clinic.36 On immunization day (or the day before), the FO collects cash from an ATM before traveling to the clinic.37
  2. At the start of the immunization day, clinic staff hold a health talk for caregivers, during which the FO also introduces the New Incentives program.38
  3. Clinic staff provide new children with a Child Health Card (CHC) and add them to the Child Immunization Register. A CHC is a document provided to caregivers that reports the infant's residence, date of birth, vaccinations the infant has received on which dates, and other information.39
  4. Infants are vaccinated according to the routine immunization schedule and their child health card. After administering a vaccine, the clinic staff write down the vaccination date on the CHC and mark it with a golden dot near the record of the vaccination.40
  5. For new infants, the FO then confirms their eligibility for the program according to the criteria discussed above. Caregivers of eligible children are provided with an ABAE Card (a card that includes basic information about the New Incentives program and immunization schedule) and an ABAE ID (applied to both the ABAE Card and the Child Health Card).41 As part of the enrollment process, the FO takes a picture of the infant's BCG injection mark.42
  6. For all infants, the FO disburses cash to their caregivers after confirming that they are eligible to receive a cash disbursement that day (according to the criteria discussed above). The FO then takes a picture showing the caregiver holding the cash, the child, and the Child Health Card (including the ABAE ID).43 Later, the pictures are reviewed by remote staff to check for possible fraud (see below).44

New Incentives also provides cash incentives during targeted outreach events (vaccination sessions targeted at settlements with particularly low vaccination rates).45 New Incentives identifies settlements for targeted outreach based on data and feedback from New Incentives staff, clinic staff, and community members. At these events, clinic staff administer vaccines and New Incentives staff enroll new infants and disburse incentives according to the same criteria outlined above. Clinic staff are reimbursed by New Incentives for transportation costs for targeted outreaches.46 We have not investigated these events in detail, and our impression is that they account for a low proportion of the caregivers reached by the program.

Process compliance and anti-fraud checks

New Incentives’ Field Officers collect evidence over the course of each disbursement day and submit it using a custom New Incentives app. Remote New Incentives staff (called “console agents”) are then assigned to review the data to check for potential fraud, as part of the process for approving staff expenses. The data console agents check includes:

  • Disbursement photos: For each disbursement, Field Officers are instructed to take a photo of the caregiver, their child, Child Health Card and ABAE ID, and the cash disbursed (more above). Console agents review these photos to check that incentives appear to have been given to eligible recipients (e.g., caregivers of young infants who have child health cards with an ABAE ID attached) and that caregivers received the correct amount of cash.47 In order to provide a deterrent against fraud, New Incentives deducts money from Field Officers’ submitted expenses if the money shown in the disbursement photo does not match the amount reported by the Field Officer.48 Our understanding is that these checks are conducted for all disbursements.
  • Location reviews: On every disbursement day, Field Officers are required to submit an "attendance photo" of themselves in the clinic or at the outreach location. They also submit background photos of the clinic every 2 hours over the course of the day.49 Console agents check this data along with photo metadata (i.e., data showing the time and location the photos were taken). This is to check that disbursements took place at an approved location and that the Field Officer was the person conducting the disbursements.50
  • Before and After Disbursement Session Reports: Console agents review photos taken of the cash on hand at the beginning and end of each disbursement day, to ensure the sums match what would be expected based on the number of disbursements.51 They also check a photo of the clinic tally sheet and compare the number of vaccinations recorded to the number of disbursements given, to validate that incentivized vaccinations were recorded.52

Other anti-fraud processes used by New Incentives:

  • Biometric reviews: New Incentives uses the Amazon Rekognition service to review the photos of caregivers taken during immunization days. The service is employed to assess potential duplicate enrollments and duplicate disbursements. A sample of flagged matches are manually reviewed by select New Incentives staff.53 Our understanding is that these checks happen after each disbursement day, meaning that the biometric review is not able to detect fraud in real time. Instead, it is used to deter fraud and identify possible fraud after the fact.54 Note that GiveWell has not independently vetted how this service is used or how effective it is.
  • Audits: New Incentives employs staff in the role of auditors who visit each clinic at least every eight weeks to assess compliance with protocols and check for evidence of fraud by (a) observing activities; (b) interviewing caregivers, clinic staff, and New Incentives staff; and (c) conducting an office money spot check (i.e., verifying that the FO is in possession of the amount of cash expected based on New Incentives' records).55 New Incentives reports that ~98% of these audits are unannounced.56 Separately, New Incentives Field Managers audit at least 5% of disbursement days. New Incentives reports that ~92% of these visits are unannounced.57 We do not have a strong understanding of how these visits differ from the main audit procedure.
  • Caregiver interviews: During the enrollment and disbursement process, Field Officers ask each caregiver whether they gave out "dashes," or tips, to anyone at the clinic.58 Auditors and Field Managers also ask caregivers whether they gave tips to Field Officers during their audits. This data is then submitted through a New Incentives app and reviewed to identify possible fraud.
  • Vendor spot checks: Auditors conduct periodic calls to randomly selected vendors from which New Incentives staff submitted expense claims. These calls are designed to verify that the expense claims were legitimate.59
  • Fraud investigations: Cases of suspected fraud are collected through the processes above, as well as a hotline and anonymous staff reporting channels.60 Those are then assessed by auditors, who review the evidence to establish whether there is reason to believe fraud occurred.61

We discuss how we interpret the risk of fraud to New Incentives’ program, and how effectively these procedures account for it, below.

Awareness-raising activities

New Incentives' awareness-raising activities aim to increase the demand for routine vaccinations by increasing awareness of the program and sharing information with caregivers about immunization. These activities fall into two main groups: activities during immunization days and activities outside immunization days.

Activities during immunization days

  • During the health talk (before children are vaccinated), a New Incentives Field Officer (FO) introduces the program to caregivers, discusses the benefits and potential adverse effects of immunization, and explains eligibility requirements.62
  • After cash incentives are disbursed, FOs communicate and/or indicate on caregivers' ABAE Cards the date of the next vaccination.63
  • New Incentives advertises the program by putting up posters outside the entrance of partner clinics.64

Activities outside of immunization days

  • Awareness meetings. Meetings in targeted communities aimed at increasing knowledge of the program and addressing concerns about vaccinations.65 The meetings are typically run by New Incentives staff and clinic staff, who distribute awareness cards (including basic information about the program).66
  • Community member outreach. New Incentives recruits community members to increase program awareness and to track infants who are behind schedule for receiving vaccinations and encourage their caregivers to complete the immunization schedule. Community members receive a stipend of 1,000 naira.67 Community members can include town announcers or town criers, village leaders, youth members, traditional birth attendants (TBAs), community health workers (CHWs), and others. For example, "town criers" may be engaged to make announcements informing caregivers about immunization days and outreach sessions.68
  • Engagement with traditional and religious leaders. New Incentives engages with traditional and religious leaders to understand the sources of vaccine hesitancy and methods to help address these. These efforts can result in community discussions on the benefits of vaccination and alignment with religious beliefs, reasons for providing incentives, as well as announcements in places of worship.
  • Ambassadors. Field Officers sometimes ask supportive caregivers to promote the program in their areas of residence or help bring back enrolled infants who might be delayed in getting their next vaccinations. Ambassadors are unpaid volunteers.69
  • Radio messages. New Incentives introduced paid radio advertisements in November 2023 in the states of Bauchi and Gombe. These advertisements provide information on New Incentives' program and the benefits of vaccination. New Incentives may expand the radio messages to additional states after assessing their impact in Bauchi and Gombe.70

During review of this page, New Incentives shared the following estimates of the frequency and reach of awareness activities:71

Activity Engagements per clinic, future expected frequency Caregivers reached per engagement (expected)
Immunization day activities 1.25 per week 20
Awareness meetings 1 every 2 months 40
Community member engagements72 2-3 per month -
Targeted outreach 1 per month 12

New Incentives also shared data on how new program participants say they became aware of the program:73

Where participants heard about the program 2018-202074 2021-202375
Neighbor 48% 58%
Friends 39% 52%
Family 31% 43%
Health worker at this clinic 15%
New Incentives staff 11%
Traditional birth attendant 11% 9%
Village leader 11% 11%
Town crier 7% 8%

As of the time of writing (December 2023), we have not independently reviewed this data.

Supporting vaccine supply

New Incentives’ supply-side work aims to ensure consistent vaccine supply at the clinics where it operates. Its work can be divided into responsive and preventive activities.

Responsive
Specialist New Incentives staff (called “supply side officers”) identify incidents of low supply. These include stockouts (where a clinic does not have a vaccine throughout an immunization day); runouts (where a clinic runs out of a vaccine during an immunization day); shortages of Child Health Cards, syringes, or other supplies; and incidents where children were not served.76 Supply side officers identify these cases through a mix of methods including77 :

  • Field Officers’ reports from each immunization day.
  • Phone calls to government staff responsible for ensuring vaccine supply (“cold chain officers”) at the state, local and zonal levels and at “apex clinics” (larger clinics that store vaccines for clinics with more limited storage capacity). These calls aim to identify whether vaccines are in low supply and if any vaccines are due to expire in the following months.
  • Information from other New Incentives teams (e.g., audit findings).
  • Cases where New Incentives Field Officers have identified expired vaccines being used at clinics (more below).

Supply side officers then attempt to resolve supply problems where they’re identified.78 This can include79 :

  • Conducting follow-ups and reminding cold chain officers to divert vaccine stocks to clinics where they are needed (e.g., by retrieving stocked out vaccines from apex facilities, LGA cold stores, or state cold stores).
  • Encouraging cold chain officers to submit vaccine utilization and vaccine requests (the form needed to request vaccines from national supplies) on time, and checking they have the information they need to do so.
  • Occasionally providing financial support to relieve bottlenecks (for instance, by paying for transport costs to deliver vaccines).

We think it is possible that New Incentives' program increases the likelihood of vaccine supply shortages in areas where New Incentives does not operate, though we currently judge this risk to be low. We discuss this potential risk further in our report on the cost-effectiveness of New Incentives' program.

Separately, Field Officers conduct vaccine vial monitor (VVM)80 checks at partner clinics to ensure that vaccines are in a usable condition and have not expired. Field Officers randomly select one vial of each directly incentivized vaccine during immunization days and check it is in a usable condition. Where any vaccines are not usable, Field Officers request that clinic staff check all the vaccines at the clinic. They also record data on these checks in a log so that supply side officers can verify the VVM assessments and escalate issues to respective government agencies when warranted.81

Preventive
New Incentives also aims to prevent supply issues from occurring. It uses strategies including:

  • Requesting same-day “top-up” disbursements of vaccines from Apex Clinics or LGA Cold Stores, when New Incentives staff have notified their managers or supply side officers that there is a pending stockout at their clinic.82
  • Notifying local cold chain officers about clinics that use an unusually high volume of vaccines (“high-consumption facilities”), and encouraging them to allocate more vaccines to these clinics.83
  • Reviewing stock availability at state and zonal levels against estimated consumption and prompting requests for vaccines ahead of widespread vaccine stockouts.
  • Engaging with vaccine supply stakeholders to share findings and coordinate to close supply gaps where possible, including through regular monthly round table meetings with state and zonal cold chain officers.

New Incentives collects extensive data on supply-related issues - see the monitoring section below for details on how we interpret this data.

Relationships with government

New Incentives works in government clinics and requires the support of agencies within each Nigerian state responsible for those clinics to operate. When it expands to a new state, New Incentives negotiates and signs a Memorandum of Understanding (MoU) with the state authorities, setting out each party’s responsibilities.84 New Incentives reports that it also has support from and has previously signed MoUs with several national health and social protection agencies.85

Overall, our impression is that New Incentives’ relationships with authorities at the state and local level in states where it works are strong, and it conducts a number of activities to strengthen those relationships (e.g., Joint State Meetings (roundtables) with health officials in the states where it works).86 However, we haven’t investigated this aspect of its work in detail. Our understanding of these relationships is also based largely on discussions with New Incentives itself. In 2021, we spoke to two government officials and a UNICEF representative to understand their perspectives on New Incentives' work,87 but we have not prioritized further conversations with New Incentives’ partners since then.

New Incentives also faces some opposition to its program among some members of the Nigerian federal government. Critics have raised concerns about the program, particularly that there could be a backlash if New Incentives discontinues its program in a given area, leading to falling vaccination rates.88 We discuss how these concerns affect our view of the program in detail in our separate report on the cost-effectiveness of New Incentives’ program. Overall, our best guess is that this consideration does not significantly offset New Incentives’ program benefits. But we see it as an important priority to engage more with critics of the program in the future.

Our understanding is that the decision-making authority to support New Incentives’ program rests primarily with the states where it works, and therefore that the opposition in other parts of the government is not a formal obstacle to New Incentives delivering its program.89 However, we see this opposition as a concern for other reasons:

  • We think that it’s possible that this could impede New Incentives’ ability to deliver the program in other ways.
  • In general, we are wary about the risk of undermining government authorities in locations where we fund programs. This is for a mix of reasons, including governments possibly having access to better local information than we do, and being potentially more responsive to the wishes of their citizens.

We do not currently account for the second of these concerns. We currently account for the first concern with a -5% adjustment in our cost-effectiveness analysis, discussed here in our separate report on the cost-effectiveness of New Incentives’ program.90 This adjustment is small because New Incentives has been running its program since 2017 and we have not yet seen examples of opposition from the authorities impeding its ability to deliver the program. We plan to continue monitoring this issue to understand whether it appropriately captures the risk to New Incentives’ program.

1.4 New Incentives as an organization

Overview

New Incentives was founded in 2011. It initially began piloting a program using conditional cash transfers (CCTs) for the prevention of mother-to-child transmission of HIV in Nigeria, and pivoted towards its current program (focused on routine childhood vaccinations) in 2016.91 As of 2023, it is solely focused on delivering its childhood vaccination program in northern Nigeria.

Staff structure

As of the end of 2023, New Incentives employs almost 3,000 staff, with most staff based in the parts of northern Nigeria where the program operates. Some of the most common New Incentives staff include:

  • Field Officers, who are responsible for disbursing CCTs, running awareness activities, and collecting information about vaccine supply during immunization days.92
  • Field Managers, who are responsible for supervising Field Officers and identifying and addressing problems with enrollment, retention, and vaccine supply at the clinic level.93
  • Console agents, who conduct the anti-fraud checks discussed above.

Spending breakdown

Around one third of New Incentives’ costs are the cash incentives themselves, with the rest split between staff compensation, supply costs, and other activities. See below for a breakdown between November 2017 (around when the program began) and December 2023 (the most recent data available).94

Expense category Total ($m) %
Conditional cash transfers $10.78 23%
Staff compensation $8.90 19%
Transport and supply costs for in-clinic and awareness activities $8.73 19%
Contractors and Consultants $6.12 13%
Stakeholder Relations (meetings and vaccine transport) $0.70 2%
Other $6.32 14%
Cash transfer commitments95 $4.66 10%
Total $46.21 100%

2. Monitoring and information sharing

2.1 Overview

GiveWell asks our top charities to share detailed monitoring information on their programs. The aim of reviewing this information is to assess whether the program is being conducted to a high quality and whether it is reaching recipients as intended. We use the outputs from these reviews as part of our cost-effectiveness analyses to make funding decisions.

Because GiveWell’s analysis of New Incentives’ program is based on a 2017-20 RCT, we have focused most of our attention on analyzing data that could help us understand how the program’s impact at scale might differ from the RCT. We have reviewed (or plan to review) three main sources of information:

  • Program performance indicators (retention through the vaccination schedule, repeat enrollments, supply-side issues, and reduction in the value of the incentive because of inflation).
  • Data on vaccination rates from coverage surveys conducted by New Incentives in areas where it works. We can compare these to the expected impact of the program based on the RCT.
  • Fraud monitoring data that we can compare to the RCT.

Our findings:

  • Program performance indicators: These have largely remained stable since the RCT, although we have seen a worsening in the rate of supply-side problems at New Incentives-supported clinics for some vaccines. Overall, we interpret this as evidence that New Incentives’ delivery of the program has remained high-quality at scale. (More)
  • Data on vaccination rates: As of December 2023, we have only just begun analyzing these surveys. We plan to incorporate them in our analysis of the program in the future. (More)
  • Fraud monitoring data: We have not seen evidence of fraud risk increasing since the RCT. But we remain uncertain about this because we’re not sure whether the monitoring data we’ve seen would capture all kinds of possible fraud. (More)

Our main open questions and uncertainties are:

  • We’re concerned that fraud is inherently a large risk in a program like New Incentives’, and it’s possible there are types of fraud that the monitoring indicators we have reviewed to date do not account for. (More)
  • Although we’ve updated our analysis of New Incentives’ monitoring data for successive grant decisions, our last in-depth review of the methodology used to collect the data was in 2020 (shortly after the RCT). This is a less intensive level of review than for GiveWell’s other top charities, and it’s possible we have missed things or that aspects of our analysis are out of date.
  • We do not have a strong understanding of how some of the monitoring indicators we use are collected and aggregated.96 It’s possible that this means that some aspects of our interpretation are incorrect.

2.2 Assessing the program’s impact over time

New Incentives maintains a database of over 100 monitoring indicators that it tracks to assess program performance.97 A summary of the main indicators we’ve reviewed is in this spreadsheet. In this report, we focus specifically on the indicators that we think are most important for understanding how the impact of the program might have changed since the RCT.

Overall, we think that the monitoring information we’ve seen from New Incentives is relatively high-quality and comprehensive. Our understanding is that many monitoring indicators are based on disbursement data gathered by New Incentives Field Officers during all disbursements, reducing our concerns about comprehensiveness and sampling bias. In the last systematic review we conducted of its monitoring methods (in 2020, available here), we assessed the risk of bias for most of the data collection methods it used as medium or low.

Retention through the vaccination schedule

Why is this important? Retention rates refer to the proportion of children enrolled in New Incentives’ program who receive subsequent vaccine doses at any point in time.98 If retention rates decrease over time, this might indicate that New Incentives' program is becoming less effective at incentivizing immunizations.

How we interpret the evidence
Our bottom line: Retention rates have been stable since the RCT, suggesting New Incentives’ program is still motivating for caregivers.

In more detail:

  • New Incentives collects and shares data on retention rates for each of the 3 doses of the Penta vaccine, and both doses of the measles vaccine.99 Our understanding is that this data is based on entries submitted by Field Officers to a New Incentives app during the disbursement process. Field Officers record the vaccines that a child received during their visit, according to their Child Health Card.100 Our understanding is that this data should be recorded for every child.
  • Overall, retention rates have stayed high, at levels consistent with those observed in the RCT. For example, retention for the first dose of the Penta vaccine was 95% in 2022 (compared to 93% in the RCT), and 88% (in 2022) vs 91% (RCT) for the first dose of the measles vaccine.101
  • The retention rate for the second dose of the measles vaccine has been considerably lower than other vaccines since it was introduced in 2021 (70% overall in 2022).102 We see this as unsurprising, since this dose comes last in the vaccination schedule (at 15 months).103 In January 2023, New Incentives decided to increase the incentive for the measles 2 visit from 500 to 1,000 naira in an attempt to increase vaccination rates.104 We have not yet analyzed how effective this change has been.
  • A limitation in this evidence is that it cannot distinguish between the impact of New Incentives’ program and other factors on retention rates.

Repeat enrollments

Why is this important? Caregivers have an incentive to enroll their infants in the program multiple times in order to receive additional incentives. It is possible that some infants are enrolled in the program more than once, either at the same or different clinics.105 We think that infants are unlikely to benefit from receiving the same vaccine more often than it is scheduled to be received, so if we count children who are enrolled more than once as new children, then we will overestimate the impact of the program.

How we interpret the evidence
Our bottom line: We estimate that ~10% of all New Incentives enrollments are repeat enrollments, based on data New Incentives collects on BCG scarring. This has been stable since the RCT, suggesting some (stable) level of caregiver fraud over time.

In more detail:

  • New Incentives aims to prevent the same child being enrolled multiple times by limiting enrollment to the visit in which the child receives the BCG vaccine (with an exception described above), and (b) checking new enrollees for BCG scars, which indicate that the child received the BCG vaccine previously.106 This system is imperfect because most but not all children develop BCG scars and because scars take a couple of weeks to form, leaving a window for re-enrollment.107
  • Field Officers also look at children’s arms when they return for subsequent vaccinations and record the number of BCG scars on each arm using a New Incentives app.108 By comparing the proportion of children with scars to the proportion we’d expect to have scars (from independent data sources), we can estimate the proportion of children who have received BCG vaccination more than once. We use this as a proxy for children being enrolled in the program multiple times, since we’d guess that the main reason children receive more than one BCG vaccination in areas where New Incentives operates is to get additional cash incentives.109
  • Our best guess, based on a quick literature review, is that 90% of children who are vaccinated against BCG develop a scar that will be detected and correctly identified when checked for (details in footnote).110 By contrast, 99% of children checked by Field Officers at return vaccination visits in 2022 had 1 or more BCG scars.111 Based on this data, we estimate that roughly 10% of total enrollments in the program in 2022 were repeat enrollments (calculation in footnote).112 Scar formation is a biological process which we’d guess has substantial variation, and so our estimates of re-enrollment based on scar formation are moderately uncertain.
  • While we see this proportion as moderately high, overall we are less concerned about repeat enrollments than other possible fraud risks. That is because (i) our estimates of repeat enrollments have stayed relatively steady over time113 , (ii) we think New Incentives’ processes for deterring repeat enrollments (such as the BCG scar check) are reasonable and (iii) we are already accounting for repeat enrolled children in our analysis of New Incentives’ costs, meaning we’re less concerned that this factor could lead us to overestimate the impact of the program.

Supply-side issues

Why is this important? If supply-side problems mean that children are not vaccinated when they arrive at New Incentives-supported clinics, the program might become less effective at boosting vaccination rates over time (e.g., if caregivers learn to expect that vaccines will not be available).

How we interpret the evidence
Our bottom line: The rate of stockouts has significantly risen since the RCT. This largely reflects supply problems with indirectly incentivized vaccines, especially the rotavirus vaccine, which was introduced in the routine immunization schedule in 2022.114 Because indirectly incentivized vaccines are only a small portion (~20%) of the program benefits we model, we haven’t yet included an adjustment to account for this. We plan to keep monitoring this issue.

In more detail:

  • New Incentives collects and shares a number of different supply-side monitoring indicators (we list 13 indicators in our latest summary of New Incentives’ monitoring data).115 We have paid most attention to data on the number and rate of vaccine stockouts during disbursement days. This data is recorded by Field Officers based on their observations, discussions with clinic health staff, and the clinic’s immunization tally sheet, and submitted through a New Incentives app through a daily survey called “Clinic Daily”.116
  • We would expect this data to be reasonably reliable. We would guess that clinic health staff might be incentivized to underreport stockouts, but that this probably does not apply to the same extent to New Incentives Field Officers (since they are not directly responsible for vaccine stock in the clinics where they work). However, we have not investigated this in detail and it's possible that the data might be biased in ways we haven’t considered. Supply-side data is available for a very high proportion of disbursement days (98% since the end of the RCT), increasing our confidence that this data provides a reasonably comprehensive picture of the supply dynamics in New Incentives-supported clinics.117
  • Overall, the data we have suggests that there has been an increase in supply-side issues over time. The rate of stockouts of any vaccine rose from ~8 per 100 disbursement days during the RCT to ~47 per 100 disbursement days in 2022.118
  • We see this as somewhat concerning. But as of December 2023, we have not included an adjustment in our cost-effectiveness analysis to account for the risk that this increase could weaken the impact of the program. This is largely because the increase in stockouts is concentrated heavily in indirectly incentivized vaccines, particularly rotavirus (which was only introduced in the Nigerian immunization schedule in August 2022).119 The stockout rate for directly incentivized vaccines rose only modestly, from ~7 per 100 disbursement days during the RCT to ~10 per 100 disbursement days in 2022.120 A high proportion (~80% as of December 2023) of the benefits of New Incentives’ program in GiveWell’s cost-effectiveness model come from directly incentivized vaccines, reducing our concern that this could significantly weaken the impact of the program.121
  • We plan to continue to monitor this issue over time and will consider an adjustment in our analysis to account for increasing supply-side issues, especially if we do not see a decrease in rates of rotavirus stockout over time (we would expect to see this decrease as the length of time since it was introduced in the immunization schedule increases).

Reduction in the value of the incentive because of inflation

Why is this important? Inflation might cause the real value of New Incentives' cash transfers to decrease, which may weaken their effectiveness as an incentive.

How we interpret the evidence
Our bottom line: We think the real value of the incentives has fallen somewhat since the RCT. We’re less concerned about this than other potential risks because retention rates have stayed high, suggesting caregivers are still motivated by the program.

In more detail:

  • We have monitored this issue by (i) analyzing the real value of the cash incentives over time relative to their value in the RCT (based on data from the Consumer price index for Nigeria)122 and (ii) analyzing data shared by New Incentives on caregivers' transportation costs for health clinic visits. Our understanding is that the latter metric is based on Field Officers asking caregivers to report the one-way cost of getting to the clinic (not the roundtrip cost), and is asked for every disbursement.123
  • Our analysis suggests there was a substantial fall in the real value of the incentive from the RCT through 2021 (76% of its 2019 value in 2021).124 Since that time New Incentives has increased the nominal value of the incentives by 50% (from 4,000 naira in the RCT to 6,000 naira as of August 2023, more above).125 We would expect this to have mitigated the erosion of the incentives’ value to some extent, although we have only analyzed the real value of the incentive up to the end of 2021. New Incentives has informed us that as of November 2023 it believes the current incentive is less valuable than it was during the RCT in spite of this change, because of rapid inflation in Nigeria over this period.126
  • There has also been a moderate increase in the average cost reported by caregivers for transportation to the clinic (79 naira in March - December 2020 to 99 naira in 2022),127 and a slight decrease in the proportion of disbursements reporting 0 transportation cost (75% to 68% in the same period).128 The change in transportation costs is roughly in line with inflation over the same period, and so doesn’t affect our overall view of the real value of the incentive.
  • To account for any residual risk that inflation is reducing caregivers’ willingness to come to vaccine appointments, we use a rough -5% adjustment in our cost-effectiveness analysis.129 Our concern about this is also reduced because New Incentives has maintained high retention rates over time (more above), suggesting its program is still motivating to caregivers. This analysis could be wrong because of lags in our analysis of retention rates (which only goes up to the end of 2022), or if a fall in the real value of the incentive meant that caregivers were less likely to enroll their children in the first place. This would not be picked up by the retention rate.

2.3 Data on vaccination rates

Why is this important? Our analysis of New Incentives’ program is based on extrapolating data from the 2017-2020 RCT to new locations, and accounting for changes to program delivery based on program monitoring data. We do not (as of December 2023) use direct data on vaccination rates in areas where New Incentives works to assess the impact of the program. Cross checking against this kind of data would be another tool to assess the direct impact of the program over time, and we think it’s likely we’ll start including this data in our analysis in the near future.

How we interpret the evidence
Our bottom line: New Incentives has begun to conduct vaccine coverage surveys in areas where it works. We plan to incorporate these into our analysis of the program in the future, but haven’t done so yet.

In more detail:

  • In September 2021, New Incentives began conducting household surveys to assess baseline vaccination coverage before starting to work in new areas.130 New Incentives then conducts follow up surveys to reassess vaccination coverage after it has worked in each of the new areas for a year, and then every six months.131
  • We see the follow up surveys as having significant strengths and weaknesses. Their main strength is that they directly measure vaccination rates in areas where New Incentives works, providing additional evidence about the program’s impact on vaccine coverage outside experimental conditions. Their main weakness is that factors apart from New Incentives' program may influence the vaccination rates observed, and we do not observe changes that might have occurred in the absence of New Incentives' program. This means they are less useful for understanding the causal impact of New Incentives’ program. More information on how we expect to use this data can be found in our pre-analysis plan here.
  • As of September 2023, we have only reviewed 12-month follow up survey results for four "cohorts" (groups of neighboring local government areas that are grouped together for surveying purposes), representing ~13% of New Incentives' scale at the time.132 We do not expect the results at 12 months to be very informative for the reasons described here, so we do not currently put any weight on these results in our analysis of the program.
  • However, we expect to continue to receive additional results and continue to analyze the data over time. Ultimately, we hope to use these results as part of our analysis of New Incentives’ program in line with our pre-analysis plans.

2.4 Fraud monitoring data

Why is this important?
We think fraud is likely to be a major risk in any program that handles large volumes of cash. We think there are multiple ways in which fraud could take place and multiple actors who could potentially commit fraud (e.g., caregivers, New Incentives staff, and clinic staff).
GiveWell believes that some types of fraud are more worrying than others, depending on their implications for our analysis of New Incentives’ program. Two ends of this continuum are:

  1. Fraud that results in higher costs, without reducing the number of additional vaccinations: While this is a concern, we think that our analysis would account for this type of fraud (since the higher costs would be incorporated into our estimates of New Incentives’ costs per child enrolled). One example in this category is expense fraud (i.e., New Incentives staff could submit falsified or inflated expenses).
  2. Fraud that reduces the number of additional vaccinations, or suggests that our estimates of additional vaccination are inflated: An example in this category is a New Incentives staff member recording fake disbursements and keeping the cash for themselves. We see this type of fraud as particularly worrying because it would mean our analysis models the benefit of some vaccinations that didn’t in fact occur.

The most difficult fraud to detect might be fraud that involves collusion between multiple actors (such as clinic staff, New Incentives Field Officers, and other New Incentives staff). Fraud that includes collusion between multiple parties may be less likely, but could have a higher impact.

How we interpret the evidence
Our bottom line: The monitoring data we’ve seen does not show any increase in fraud since the RCT. We remain uncertain about this because we think fraud is inherently a high risk for New Incentives’ program, and we don’t know if the monitoring data we’ve seen would capture all kinds of potential fraud.

In more detail:
Our summary of New Incentives’ monitoring includes 10 fraud prevention indicators (listed here).133 These include the proportion of disbursements accompanied by a picture of the caregiver holding the cash,134 the proportion of caregivers who report giving dashes (tips) to clinic or New Incentives staff when asked by Field Officers during the disbursement process,135 and the proportion of audits reporting fraud by clinic or New Incentives staff.136

Overall, the monitoring data we have reviewed suggests that fraud has remained at the (relatively low) levels observed in the original RCT. For example:

  • The proportion of audits reporting fraud by either New Incentives or clinic staff was approximately 4% in the RCT and 4% in 2022.137
  • The proportion of caregivers reporting that they gave tips was 0.1% in the RCT and 0.4% in 2022.138
  • Throughout the program, 100% of disbursements logged on New Incentives’ platform have been accompanied with a photo of the caregiver holding the incentive.139
  • As discussed above, our estimate of the repeat enrollment rate (an indicator of fraud) has remained steady since the RCT, at approximately 10%.

In 2020, we conducted a review of possible fraud scenarios and determined at the time that the risk of fraud not accounted for by New Incentives’ existing procedures was low. We account for the possibility that fraud is occurring that isn’t otherwise accounted for in our cost-effectiveness analysis with a rough adjustment for “false monitoring results” (-2% as of April 2024).140 This accounts for the risk that the data we have reviewed is biased in some way, although it does not distinguish between fraud and other concerns about the data collection method. This adjustment aligns with our overall impression (based on many conversations with New Incentives over time) that it has strong procedures for identifying and responding to fraud.

However, we still have a number of open questions and uncertainties about this issue. We’re concerned that fraud is inherently a large risk in a program like New Incentives’, and it’s possible there are types of fraud that the monitoring indicators we have reviewed to date do not account for. For example:

  • Field Officers might record fake disbursements to friends and family members and pocket the cash. We don’t know if there are ways this could avoid detection, and think it’s possible that this could be missed by New Incentives’ remote console agents so long as these disbursements took place at a clinic and all the required elements of the caregiver photo were submitted appropriately.
  • Different New Incentives staff members or teams could collude to defraud the program (e.g., auditors colluding with Field Officers). We’re uncertain how effective New Incentives’ current procedures would be at catching this type of fraud.141
  • We would guess that the data on dashes (tips) is underestimated. The data is collected based on caregiver reports to Field Officers during the disbursement process, and we would expect caregivers to feel some pressure to report inaccurately (e.g., since caregivers may not think that this report would be confidential and may get back to clinic staff). Field Officers are also unlikely to report tips that they receive themselves. We do not currently see this as a significant threat to the program because we’d expect this kind of fraud to be taking place at a relatively low level, and for any worsening trends to show up in our monitoring of retention rates (which have stayed relatively consistent over time).

We’re also concerned that our analysis may cover risks that we and New Incentives have anticipated. One continuing concern is that there may be additional risks that we have not anticipated.

We’re planning to investigate this question in more detail in the future. We expect to update our 2020 review of the program’s fraud risks142 and think through scenarios that might be missed by New Incentives’ current procedures. We may also update our adjustment for “false monitoring results” in light of this analysis.

3. Qualitative assessment

In theory, our funding recommendations are maximizing for one thing: total improvement in well-being per dollar spent. This is what our cost-effectiveness estimates intend to capture.

In practice, there are costs and benefits that we do not observe and are not estimated in our models. We make qualitative assessments to account for these unmodeled costs and benefits. We use these assessments alongside our cost-effectiveness estimates to inform our funding recommendations.

As one tool for thinking through and communicating about impressions we have that aren't captured in our cost-effectiveness estimates, we assess each organization on eight dimensions on a four-point scale (“Stands out”; “Relatively strong”; “Average”, “Relatively weak”). We believe our top charities are exceptional relative to the majority of organizations and so these assessments are intended to capture differences among GiveWell top charities, rather than absolute rankings among all charitable organizations.

Overall, our qualitative assessment of New Incentives is very strong, even compared to our other top charities. We believe it stands out for its focus on maximizing cost-effectiveness and its dedication to identifying, responding to, and being transparent about issues it faces. Our latest assessment of New Incentives (for 2023) is in the table below.

Dimension What does this capture? Assessment
Responses to our questions When we ask the organization a question, do its answers generally either indicate that it has thought through the question before or show us why getting an answer is not important to understanding its work? Stands out
Prioritization discussions Do the organization's explanations about how it allocates funding among different locations and program participants seem to be aimed at maximizing its impact per dollar? Stands out
Self-improvement and attitude toward mistakes Does the organization proactively share information with us and publicly about mistakes it has made? Stands out
Role in field Is the organization producing research aimed at informing policymakers or other implementers? Does it participate in global conversations about its field of work? Average
Responsiveness Does the organization send us information by mutually agreed-upon deadlines? Is it responsive to our emails? Stands out
Giving us feedback Does the organization catch our mistakes and let us know, thus improving our research? Does the organization make useful suggestions for how we could improve our research process and cost-effectiveness models? Relatively strong
Quality of information shared Have the documents that the organization has shared with us contained significant errors? Has the organization told us things that were inaccurate? Has the information provided been easy to interpret and use? Have the organization's projections of when it would achieve its goals generally been accurate? Relatively strong
Incorporating feedback from participants and last mile providers How does the program collect feedback from program participants and from program implementers, i.e. those directly delivering the program? How does the program incorporate feedback to improve service delivery? Relatively strong

Some of the main factors informing our assessment are:

Self-improvement and attitude towards mistakes (stands out)

  • We value working with organizations that proactively bring negative updates to our attention. We think that New Incentives is highly transparent in informing us about challenges it is facing, even when it knows these may affect our evaluation of the program.
  • For example, in 2023, New Incentives raised an unprompted concern about an increasing proportion of children older than 6 months at enrollment (this is permitted under the program’s eligibility criteria, but is intended to be an exception).143 New Incentives had reviewed the underlying data and found that some of these children were meaningfully older (e.g., 1-2 years) and that their date of birth was misreported. If a high portion of children were getting vaccinated later than recommended, we believe they would be unlikely to be receiving the full benefits of vaccination, because mortality for several of the diseases is particularly high during the first 6-12 months of life. As a result, we would potentially consider down-adjusting our cost-effectiveness estimates. In this case, New Incentives immediately started working to understand the drivers of this trend and implementing protocols to address it.144
  • In September 2023, New Incentives informed us that it had found an error in its monitoring data on duplicate enrollments. New Incentives runs caregiver photos through a face-matching service to identify potential duplicates, and reports on the rate of photos flagged as duplicates at the same clinic, at clinics within a 10km vicinity, and at clinics within a 20km vicinity. New Incentives discovered the rates of duplication within 10km or 20km were underreported (due to a coding error).145 Although underreported rates of duplicate enrollment would imply the program was reaching fewer children, New Incentives brought this error to our attention and kept us updated as it began rerunning the data for the affected time period.146

Prioritization discussions (stands out)
We think that New Incentives shares GiveWell’s focus on evidence-based decision making and maximizing the cost-effectiveness of its program.

  • An important question when we have considered funding expansion grants for New Incentives is whether the program will be cost-effective in new locations. In discussions about this, we have found New Incentives to be a very thoughtful partner, and we have always been able to come to full agreement efficiently on where New Incentives expands to. This has sometimes involved us demonstrating to New Incentives that the funding could be used more cost-effectively elsewhere (either within New Incentives program or, through making a smaller grant, through another program). At other times it involved New Incentives convincing us that we were missing something important in our assessment.
  • Our impression is also that New Incentives is focused on decreasing the cost of the program per child enrolled to an unusual degree among GiveWell top charities. This is reflected in a consistent decrease in our estimates of New Incentives’ cost per child enrolled since the RCT, from ~$35 between 2019-20 to ~$20 in 2022.147 We think that a substantial part of this decline is due to external factors, e.g., the fall in the value of the Naira relative to the dollar148 but we still think New Incentives’ focus on cost-effectiveness has played a role in keeping costs low.

Incorporating feedback from participants and last mile providers (relatively strong)
In general, we would expect that organizations will deliver more effective programs if they have mechanisms for gathering and acting on feedback from program participants.

New Incentives has a large staff footprint in the areas where it works in northern Nigeria, providing frequent opportunities for touchpoints between New Incentives staff and program participants. It also has systems for gathering feedback from program participants and stakeholders on its program (e.g., during immunization sessions and awareness-raising meetings in communities), and acting on the results.149
For example:

  • Before making the decision to change the incentive structure for its program in 2023, New Incentives gathered feedback from caregivers and other partners (e.g., state health officials).150 The positive feedback it received on this change informed its decision to go ahead.151
  • New Incentives’ enrollment criteria include an exception, whereby children who received the BCG vaccine before New Incentives began operating in a given area can still be enrolled in the program in some circumstances (more above). New Incentives reports that this decision was based on feedback from both caregivers and clinic staff that it would be an unfair penalty on caregivers whose children were vaccinated before New Incentives expanded to that location.152

We would guess that these processes allow it to gather better data on its program to improve delivery, although we have not yet seen detailed information on the methodology used to collect this feedback.

Other factors

Track record scaling the program: New Incentives has rapidly scaled this program since it began in 2017. The number of children enrolled has risen from ~90,000 children in 2019153 to ~1.39m between January and November 2023154 and its annual spending has risen from ~$2.6m (June 2019 - May 2020) to ~$11.3m (Jan - Dec 2022).155 We were concerned that program quality could be negatively impacted by rapid scaling, or that unexpected problems could arise. While this continues to be a concern as New Incentives expands further,156 the evidence we have seen so far suggests that New Incentives is still delivering the program at a similar level of quality as in the randomized controlled trial (with the exception of an increase in the rate of vaccine stockouts, see our monitoring review for further details). This increases our confidence that the program is still being delivered well, and is still motivating caregivers to receive additional vaccines.

 

As it has scaled up, New Incentives has also adapted its program over time. For example, our understanding is that it has introduced an increasing number of anti-fraud checks (e.g., a face recognition process and a requirement that staff submit attendance photos to verify that they are present at the clinic as intended).157 While we have not individually assessed each of these changes for its impact on the program, they contribute to our overall impression that New Incentives is dedicated to trying to tackle problems as they emerge and maximize the impact of the program.

4. What do you get for your dollar?

GiveWell recommends interventions and organizations that we believe are cost-effective in the sense of saving or improving lives as much as possible for as little money as possible. We summarize the full reasoning behind our cost-effectiveness analysis for New Incentives in our separate report. In summary, as of April 2024, we think:

  • It costs roughly between $1000 and $5,000 to avert a death through New Incentives’ program.158 This equates to being 11 - 51x as cost-effective as direct cash transfers (GiveWell’s benchmark for comparing different programs). This is because:
    • Based on an RCT of New Incentives’ program, we believe the program leads to a large increase in the number of children receiving routine vaccinations (11 to 22 percentage points depending on the location).159
    • There is strong evidence that vaccination leads to a substantial reduction in child mortality (~50% reduction in vaccine-preventable disease).160
    • We believe that vaccination probably leads to other benefits like reduced mortality at older age groups and increases in income later in life.161

See our separate report for more details.

5. Previous New Incentives grants

6. Sources
Document Source
Amazon, "What is Amazon Rekognition?" Source (archive)
Atimati and Osarogiagbon 2014 Source
Central Bank of Nigeria, Exchange Rates Source (archive)
Gambo et al. 2014 Source (archive)
Gavi, "Dealing with diarrhoea: Nigeria introduces rotavirus vaccine into its immunisation plan," Source (archive)
GiveWell, Allocation of funds donated to the top charities fund in Q2 2021 Source
GiveWell, Analysis of enrollment discrepancy Source
GiveWell, GiveWell's Cost-effectiveness analyses Source
GiveWell, How we produce impact estimates Source
GiveWell, New Incentives — Nigeria expansion and extension (May 2023) Source
GiveWell, New Incentives (Conditional Cash Transfers to Increase Infant Vaccination) Source
GiveWell, New Incentives monitoring and evaluation 2017-2020 and plans for future evaluation (September 2020) Source
GiveWell, New Incentives monitoring results [December 2022] Source
GiveWell, New Incentives program data, 2020 Source
GiveWell, New Incentives Spending 2017 - 2023 Source
GiveWell, New Incentives vaccine coverage and treatment effects estimates by state [May 2023] Source
GiveWell, New Incentives' coverage assessments: plans as of October 2021 Source
GiveWell, Questions for New Incentives on program activities, May 18, 2020 Source
GiveWell, Update: New Incentives cost per infant immunized [May 2023] Source
GiveWell's CEA of New Incentives' conditional cash transfers to increase infant vaccination Source
IDinsight, Coverage Monitoring Analysis Plan, 2021 Source
IDinsight, Impact Evaluation of New Incentives, Final Report Source
IDinsight, Impact of Conditional Cash Transfers on Routine Childhood Immunizations: Evidence from North West Nigeria Source
IDinsight, New Incentives Impact Evaluation, Preliminary Results Brief, 2020 Source
Lagarde et. al. 2009 Source
New Incentives home page Source (archive)
New Incentives, “History” Source (archive)
New Incentives, "How it works" Source (archive)
New Incentives, "Our work" Source (archive)
New Incentives, Audits and Investigations Guidelines (February 2024) Source
New Incentives, Overview of Fraud Prevention Procedures (February 2024) Source
New Incentives, Program Protocols (January 2024) Source (archive)
New Incentives, Responses to 13-Feb-2020 Questions from GiveWell Source
New Incentives, Supply-Side Action Protocol (January 2024) Source
Odujinrin and Ogunmekan 1992 Source
Wall Street Journal, Nigerian Naira/U.S. Dollar historical prices Source (archive)
World Bank, Consumer price index, Nigeria Source (archive)
World Health Organization, "Vaccination schedule for Nigeria" Source (archive)
World Health Organization, “Vaccine vial monitor (VVM)” Source
World Health Organization, BCG vaccines position paper, February 2018 Source
  • 1

    “Conditional cash transfers (CCT) provide monetary transfers to households on the condition that they comply with some pre‐defined requirements.” Lagarde et. al. 2009, abstract. In this case, the requirement is that caregivers bring their children into clinics for vaccinations.

  • 2

    We currently (as of April 2024) estimate that the proportion of children vaccinated at baseline in the 9 states where New Incentives works varies from 25% to 69%, depending on the state. We estimate that the proportion of children vaccinated in all other Nigerian states (where New Incentives does not currently operate) is 79% on average (unweighted). See this row in our cost-effectiveness analysis.

    “Vaccinated” in this context is an aggregate measure. We weight the proportion of children receiving each vaccine dose by the share of mortality that could be averted by that dose. See this sheet for our calculations.

  • 3

    See New Incentives’ ‘How it Works’ page, accessed October 23, 2023.
    Note: New Incentives has made a number of updates to the incentive schedule (originally 4,000 naira for all incentivized vaccinations) over time:

    • New Incentives originally added a 500 naira incentive for measles 2 (which had not yet been introduced into Nigeria's routine vaccination schedule at the time of the RCT). This brought the total incentive across all visits to 4,500 naira. In early 2023, New Incentives increased the incentive for measles 2 to 1,000 naira. This brought the total incentive across all visits to 5,000 naira.
    • In August 2023, New Incentives decided to change its schedule to offer 1,000 naira per visit (6,000 in total).

    New Incentives, January 14th 2022, December 16th 2022, July 7th 2023 Program Updates (unpublished).

  • 4

    This conversion uses the naira:USD market exchange rate, which the Nigerian Central Bank reports as ~887 naira per $1 as of December 7th, 2023 (6,000 naira / 887 naira per $1 = $6.76). See this page. While we would expect Purchasing Power Parity (PPP) adjusted exchange rates to be a better indicator of the goods and services the incentive can purchase than market exchange rates, we use market exchange rates here for consistency with other aspects of this page (e.g., New Incentives’ cost per child enrolled, which uses market exchange rates).

    Note the incentive tested in the RCT was 4,000 naira. We discuss how the value of the incentive has changed over time in this section of the report.

  • 5

    "New Incentives was not operating in the study clinics in months prior to Oct 2017 and was fully operational in all study clinics by July 2018, the beginning of the RCT window." IDinsight, New Incentives Impact Evaluation, Preliminary Results Brief, 2020, p. 14.
    Note from New Incentives, provided when reviewing a draft of an earlier version of this page: "The program started in May 2017 when the 'learning sites' commenced enrollment."

  • 6

  • 7

    “We closely collaborate with the state governments of Bauchi, Gombe, Jigawa, Kaduna, Kano, Katsina, Kebbi, Sokoto, and Zamfara in northern Nigeria.” New Incentives, “Our work” page (as of January 2024).

    New Incentives will also consider expanding to Adamawa and Taraba, but as of October 2023 New Incentives had not yet identified cost-effective areas to operate in in either of those states. See our May 2023 grant page for further details on New Incentives’ most recent expansion.

  • 8

    “We have enrolled 1,388,431 infants in 2023”. New Incentives, Program updates, December 2023 (unpublished).

  • 9

    Note from New Incentives, provided when reviewing a draft of this page: "The name New Incentives was not accepted during corporate registration in Nigeria so the available name, All Babies Are Equal (ABAE) Initiative was used instead."

  • 10

    Created using https://www.mapchart.net/.

  • 11

    The rotavirus vaccine was introduced into the Nigerian childhood vaccination schedule in August 2022. Gavi, "Dealing with diarrhoea: Nigeria introduces rotavirus vaccine into its immunisation plan," August 30, 2022.

  • 12

    Adapted from New Incentives, ‘How it works’ page, accessed July 29th 2023.

    We include all the directly incentivized vaccines and rotavirus in our main analysis. We exclude the remaining indirectly incentivized vaccines, because we expect that they will provide a small proportion of the overall benefits and the additional time required to add them to our analysis would not be worth the added complexity. See this section of our separate report on the cost-effectiveness of New Incentives’ program for further details.

  • 13

    New Incentives began disbursing incentives for the second dose of the measles vaccine in January 2022. "We have started testing the sixth cash transfer of N500 for MCV 2. We are planning community mobilization, town crier announcements, and awareness efforts across our clinics to inform caregivers about the MCV 2 incentive and eligibility criteria." New Incentives, Program Updates, January 14, 2022 (unpublished).
    Note that as of December 2023, New Incentives offers 1000 naira for the measles 2 visit.

  • 14

    New Incentives, ‘How it works’ page, accessed July 29, 2023. The conversions here use the naira:USD market exchange rate, ~887 naira per USD as of December 7th 2023 (1,000 naira / 887 naira per $1 = $1.13). See this page. While we would expect Purchasing Power Parity (PPP) adjusted exchange rates to be a better indicator of the goods and services the incentive can purchase than market exchange rates, we use market exchange rates here for consistency with other aspects of this page (e.g., New Incentives’ cost per child enrolled, which uses market exchange rates).

  • 15

    In the RCT, New Incentives used a 4,000 naira schedule (500 for BCG, 500 for each dose of Penta and PCV, and 2,000 for the measles vaccine—500 + (3 * 500) + 2,000 = 4,000 naira). See IDinsight, Impact Evaluation of New Incentives, Final Report, p. 11, table 1. This schedule has been updated over time. As of December 2023, New Incentives used a 6,000 naira schedule (1,000 naira per visit, 6 visits * 1,000 naira = 6,000 naira). New Incentives, ‘How it works’ page, accessed December 14th 2023.

  • 16

    Routine immunization services: “Clinics that are not operational or not offering RI [routine immunization] (as confirmed at the LGA and state level) are sorted out and not visited for a Clinic Screening." New Incentives, Clinic screenings analysis, "Statistics: All Clinics Screened" sheet, cell A35.
    Safe to visit during the day: “Clinics that are in no go security areas (as confirmed at the LGA and state level) are sorted out and not visited for a Clinic Screening. No Go Zones cannot even be accessed during daytime." New Incentives, Clinic screenings analysis, "Statistics: All Clinics Screened" sheet, cell A36.

    Public clinics: New Incentives informed us during a review of this page that it typically does not work in private clinics, as the clientele at those clinics tend to be comparatively higher income.

  • 17

    While BCG is recommended to be given at birth, our understanding from speaking with New Incentives is that the majority of births in the areas where the program is operating do not occur in health facilities and thus infants are typically brought to clinics sometime after birth to receive the BCG vaccine. Call with New Incentives, August 8, 2023 (unpublished).

  • 18

    A CHC is a document provided to caregivers that reports the infant's residence, date of birth, and vaccinations the infant has received on which dates.

  • 19

    New Incentives staff verify that caregivers live within an operating LGA by:

    • Encouraging clinic staff to write the caregiver’s settlement on the Child Health Card.
    • Checking the settlement on the Child Health Card themselves, to ensure it’s within the clinic LGA.
    • Asking the caregiver to confirm the settlement, and that it matches the settlement on the Child Health Card.

    "How do we find out if a caregiver is from the LGA?

    • The clinic staff probes to write the correct address/settlement on the Child Health Card. (You have to remind clinic staff if they don’t do this!)
    • You as the field officer check the address/settlement on the Child Health Card to verify that the address is in the LGA. In addition, you ask the caregiver again about which settlement she is from. If the written settlement is in the catchment and the beneficiary gives the same answer again, then the caregiver is eligible."

    New Incentives, Program Protocols (January 2024), p. 12.

  • 20

    See the Nigeria childhood immunization schedule here.

  • 21

    “Infants must be age eligible (e.g. under 1 year old if new enrollment, older than 9 months if receiving the MCV 1 vaccine).” New Incentives, Program Protocols (January 2024), p. 12.

  • 22

    BCG vaccination typically leaves a scar, providing a more objective measure of vaccination rates. "BCG vaccination usually causes a scar at the site of injection due to local inflammatory processes. However scar formation is not a marker for protection and approximately 10% of vaccine recipients do not develop a scar." WHO, BCG vaccines position paper, February 2018, p. 84.

    “Eligibility for Program Enrollment - only caregivers qualify that:...
    …2. Have an infant with a fresh BCG vaccination mark (given on the same day)."
    New Incentives, Program Protocols (January 2024), p. 11.

  • 23

    “13. Consent
    ASK: Do you consent to allowing us to process the following types of data: Personal Information (Name, Date of Birth, contact number), Sensitive Information (Medical Data, Biometric Data), Data about Child / Children?
    Beneficiary or caregiver said Yes
    Beneficiary or caregiver said No -&gt SKIP to close without saving
    The response of the beneficiary or caregiver means unsure or refused to consent to some -&gt SKIP to close without Saving”
    New Incentives, Program Protocols (January 2024), p. 21.

  • 24

    “Exceptions to the enrollment eligibility criteria are as follows:

    • In the first 6 months of operations at a newly expanded clinic, infants can be enrolled if the infant’s name was previously recorded in the Child Immunization Register (CIR)
    • Infants who received BCG on immunization days when FO was not present (e.g. outreaches, campaigns, and clinic paused days) can be considered, if verified through the CIR
    • After enrolling an infant on the basis of the CIR, the Date of Disbursement should be added in front of the infant’s name so that the row cannot be reused”

    New Incentives, Program Protocols (January 2024), p. 12.

    Note: New Incentives also introduced an exception for children who received BCG during a “Network Shutdown” (telecommunications being down in some locations for several months in late 2021). Our understanding is that, at the time of writing (December 2023), this exception is no longer likely to apply to a substantial group of children because the children who received BCG during this period are likely to be older than 2, and therefore not eligible to enroll in the program.
    New Incentives, Program Updates, September 2021 and January 2022, unpublished.

  • 25

    See here in our supplementary analysis of New Incentives’ monitoring.

  • 26

    New Incentives, Program Monitoring Indicators 2023 (unpublished).

  • 27

    “Eligibility for CCT Disbursements - to be eligible to receive a CCT, the following criteria must be met:

    1. Infant is enrolled in the All Babies program
    2. Infant is administered the required directly incentivized vaccination(s) during the same immunization day
    3. Data in Child Immunization Register (CIR), Child Health Card (CHC), and Tally Sheet is properly entered and matches
    4. Both arms of infant checked for BCG Scar and the number of injections match the CHC
    5. FO observed vaccine administration and can provide attestation that each required vaccine has been duly administered to the infant in full, and FO ensured that Gold Dot is being added only after vaccine administration
    6. Infants must be age eligible (e.g. under 1 year old if new enrollment, older than 9 months if receiving the MCV 1 vaccine)”

    New Incentives, Program Protocols (January 2024), p. 11.

  • 28

    “Eligibility for CCT Disbursements - to be eligible to receive a CCT, the following criteria must be met:

    • Infant is enrolled in the All Babies program”

    New Incentives, Program Protocols (January 2024), p. 11.

  • 29

    “CHC Check: 1. Settlement 2. Gold Dots 3. Next visit [Field tag: Child_Health_Card_Check]
    ACTION: Confirm that the Child Health Card has 1) SettlementName, 2) Gold Dots next to vaccinations, and 3) Next Visit Date. If any are missing, send the caregiver back to the nurse. Did you confirm all three items and if not correct, send the caregiver back to the clinic staff?” New Incentives, Program Protocols (January 2024), p. 21.
    “Child Health Card: tracks vaccinations of infant, handed out to caregiver" New Incentives, Program Protocols (January 2024), p. 6.

  • 30

    “The Gold Pen [protocol] is used by vaccinators to indicate that a vaccine recorded in a Child Health Card was actually administered (because sometimes the data might be recorded in the CHC by another clinic staff or independent of administering the vaccine). This not being followed could reflect improper recording or vaccination." New Incentives, email to GiveWell, March 14, 2023 (unpublished).

  • 31

    “BCG Scar on Arm…
    IF REPEAT VISIT: ANSWER: How many BCG scars does the infant have?
    Personally verify on the upper left and right arm of the child.” p. 24.
    "Nr Injections: Ask…
    ASK: How many injections did your infant get today? Enter the number. Example: if the infant received 2 injections, enter2. If you are not clear or don’t know, enter 99.
    Nr Injections: Child Health Card…
    ANSWER: How many injectable vaccinations on the infant’s Child Health Card with today’s date also have a gold dot? Enter the number. Example: if two injectable vaccinations have today’s date but only one has a gold dot, enter 1. If you are not clear or don’t know, enter 99.
    Nr Injections: No Match…
    IF NUMBERS DON’T ADD UP: Does the number of injections match the number of gold dots for injectable vaccines (Note: Use this only for injectable vaccines, not oral vaccines such as Rota)?
    Yes, continue
    No, stop and inquire more!" p. 31.
    New Incentives, Program Protocols (January 2024).

  • 32

    “FO observed vaccine administration and can provide an attestation that each required vaccine has been duly administered to the infant in full, and FO ensured that Gold Dot is being added only after vaccine administration”
    New Incentives, Program Protocols (January 2024), p. 12.

    Note that New Incentives reports it has found it challenging to enforce this requirement in the past. In November 2023, it reported that it has made changes to more strictly enforce this requirement, including additional training for Field Officers, changing seating arrangements so that Field Officers sit near clinic staff, and tracking the proportion of audits that find Field Officers do not directly observe vaccinations. New Incentives, Program Update, November 2023 (unpublished).

  • 33

    “Infants must be age-eligible (e.g. under 1 year old if new enrollment, older than 9 months if receiving the MCV 1 vaccine).” New Incentives, Program Protocols (January 2024), p. 12.

    The Nigerian government recommends that children receive measles vaccinations after 9 months of age. Our understanding is that this is because the measles vaccine is less effective before this, but we have not closely reviewed the evidence for this claim. Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).

  • 34

    “Measles Age Alert…Check to see that today is really 9 months or more after the infant’s birth. The vaccine is not as effective if an infant gets the Measles vaccine before 9 months after birth!” New Incentives, Program Protocols (January 2024), p. 31.

    New Incentives informed us during a review of this page that it allows for a little leeway when checking that an infant is at least 9 months old, as the date of birth reported on the Child Health Card may not be exact.

  • 35

    "Field staff go to clinics on immunization days to distribute cash transfers after the condition (vaccination given) has been verified." New Incentives, Program Protocols (January 2024), p 2.

  • 36

    “Two staff working at the same clinic
    If two staff work at the same clinic they manage all aspects of the program individually (each gets cash from his account, each enrolls with his personal doForms account, each fills out his personal myDay app). The only difference is that only one person, the lead staff at this clinic, fills out the Clinic Daily form.”
    New Incentives, Program Protocols (January 2024), p. 4.

  • 37

    The New Incentives FO collects money from the ATM before heading to the clinic; the money is stored in a waist bag. They can carry a maximum of 200,000 naira (about $550—200,000 naira / 887 naira per $1), but usually carry less than 100,000 (about $280—100,000 naira / 887 naira per $1). If an FO runs out of money during the day, they can go and get more from the ATM, ask their manager for help, or close up the disbursement. This is then recorded in the supply dashboard and addressed by the field manager. Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).

    Note that the above figures are from 2020, and may not be reflective of the cash amounts carried by Field Officers now.

  • 38

    “Beginning of Each Enrollment Day at the Clinic…Participate in the health talk to underline main messages about our program and clearly communicate the program’s eligibility criteria. The six different visits and the related cash transfers should also be clearly addressed (hold up one of the All Babies cards when speaking).” New Incentives, Program Protocols (January 2024), p. 15.

    New Incentives informed us during review of this page that Field Officers also share information about the benefits of vaccination when disbursing the cash incentives.

  • 39

    “CHC Check: 1. Settlement 2. Gold Dots 3. Next visit [Field tag: Child_Health_Card_Check]
    ACTION: Confirm that the Child Health Card has 1) SettlementName, 2) Gold Dots next to vaccinations, and 3) Next Visit Date. If any are missing, send the caregiver back to the nurse. Did you confirm all three items and if not correct, send the caregiver back to the clinic staff?” New Incentives, Program Protocols (January 2024), p. 21.

    “Child Health Card: tracks vaccinations of infant, handed out to caregiver" New Incentives, Program Protocols (January 2024), p. 6.

  • 40

    This mark is made with a pen provided by New Incentives. In the middle of the day, the FO also checks whether the golden pen is being used by the vaccinating nurse and is being used correctly.

    “Use of Gold Pen…Observe each infant getting vaccinated. How did the nurses use the gold pen? Remind them to always use the gold pen but only after the vaccination took place (not during registration!).”
    New Incentives, Program Protocols (January 2024), p. 68.

  • 41

    “IF NEW BENEFICIARY: ACTION: Put a matching pair of All Babies ID stickers on the Child Health Card and All Babies card. Stamp the stickers so that if the sticker is removed and placed on another Child Health Card, we will easily know. Did you do this?
    a. Yes, I assigned both All Babies ID stickers and stamped the sticker on the Child Health Card
    b. No, I assigned both stickers but was not able to stamp the Child Health Card”
    New Incentives, Program Protocols (January 2024), p. 33.

  • 42

    “IF NEW BENEFICIARY: ACTION: Take a photo of the fresh BCG injection mark on the upper left arm of the child.” New Incentives, Program Protocols (January 2024), p. 25.

  • 43

    Specifically, the FO takes a picture of (a) the fresh BCG injection mark (for enrollment), (b) the side of the CHC, including the ABAE ID (for disbursement), and (c) the other side of the CHC, including the vaccination list (for disbursements).
    “IF NEW BENEFICIARY: ACTION: Take a photo of the fresh BCG injection mark on the left upper arm of the child. Ensure that you take the photo close to the upper left arm and ensure it is sharp by tapping the screen." p. 25.
    “Photo field: ACTION: Take a photo of the left side of the Child Health Card and SAY: “I am registering you with this photo.” (Ensure that the All Babies ID and all caretaker data is visible.)" p. 39.
    Photo field: ACTION: Take a photo of the right side of the Child Health Card and SAY: “I am capturing your vaccinations.” Ensure that all vaccinations are visible." p. 41.
    ACTION: Take a photo of caregiver, infant and cash. SAY: “I am documenting that you actually got the money.” All bills must be visible, the caregiver’s face must be visible, the infant but NOT necessarily its face must be visible. Will you do this?
    a. Yes, I will take the Incentive Photo and narrate this” p. 41.
    New Incentives, Program Protocols (January 2024).

  • 44

    New Incentives noted in response to reviewing a draft of this page that this is done “[i]n a secure manner using an application that maintains data security and access controls.”

  • 45

    "Description: Conducted by holding an immunization day at a settlement, with participating clinic staff to address concerns related to distance or sometimes non-compliance. . . .
    Costs (Time and Monetary): Costs incurred are the same as those for a regular immunization day (transportation cost, one full FO day). Small fuel stipend of around N1,000-N2,000 is given to clinic staff, based on distance to outreach location and number of clinic staff." GiveWell, Questions for New Incentives on program activities, May 18, 2020, p. 4.

  • 46

    Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).

  • 47

    New Incentives informed us during a review of this page that console agents also check if the same photos are being reused and submitted for multiple disbursements, and review ABAE IDs flagged by the system as duplicate disbursements or as a terminated ABAE ID.
    We do not have a good understanding of how these checks work in practice.

  • 48

    Conversation with New Incentives, September 2023 (unpublished).

  • 49

    “Bi-hourly Monitoring
    - Employee takes photos at least once every 2 hours, these are reviewed by an independent unit to assess whether the location of disbursement submissions align with the monitoring photos
    - Clinics not meeting this requirement are marked as Fail and multiple instances of this can lead to disbursements at clinics being paused and/or investigation visits”
    New Incentives, comments on a draft of this page, January 2024 (unpublished).

  • 50

    Conversation with New Incentives, September 2023 (unpublished).
    New Incentives informed us during review of this page that Field Officers' travel logs are also reviewed to confirm if activities took place at the intended location.

  • 51

    Reviewers look at transactional data and associated photos to count the amount of cash displayed and the sum of cash disbursed from individual photos is used to calculate the total amount disbursed by an employee for each Disbursement Day.
    “Before and After Disbursement Session Reports
    - All disbursement staff report their Cash on Hand before and after Disbursement Day (daily differences between Cash on Hand before and after the Disbursement Day are reconciled against the total cash disbursed to caregivers by employee)
    - Each report contains an After Clinic Visit photo so that independent reviewers can count the employee’s Cash on Hand to cross-check the figures reported by the employee
    - Transactional data, including the amount of cash disbursed and photo evidence of recipient holding the cash disbursement, are compared to expense reports
    - Independent reviewers review transactional data and associated photos to count the amount of cash displayed; the sum of cash disbursed from individual photos is used to calculate the total amount disbursed by an employee per Disbursement Day”.
    New Incentives, comments on a draft of this page, January 2024 (unpublished).

  • 52

    Clinic staff are responsible for recording vaccinations in the Clinic Tally Sheet. If tally sheets are not being filled out consistently, New Incentives will flag this to the clinic in question and may pause disbursements at that clinic until proper protocols are followed. Conversation with New Incentives, September 28, 2023 (unpublished).

    For new enrollments, console agents also review a photo of the Child Immunization Register (CIR) to ensure that newly enrolled infants were added. New Incentives, comments on a draft of this page, January 2024

  • 53

    “- Face Search and Face Comparison features of the Amazon Rekognition service are used to compare the photos of the caregiver accompanying the infant of each disbursement session to assess potential duplicate enrollments and duplicate disbursements; a sample of flagged matches are manually reviewed.
    - The caregiver accompanying the infant at the point of enrollment is compared to caregivers within a 20km radius to assess whether the same caregiver has been enrolled more than once with the same infant.” New Incentives, comments on a draft of this page, January 2024 (unpublished).

  • 54

    For example, this check could identify cases where Field Officers are repeatedly disbursing incentives to the same friends or family, or provide feedback to Field Officers that they need to be more careful to identify caregiver fraud.

  • 55

    "Observations by Auditor of the following risk areas at a minimum:

    • Clinic staff protocol non-compliance
    • NI-ABAE staff protocol non-compliance…
    • Beneficiary fraud
    • Clinic staff fraud
    • NI-ABAE staff fraud

    Inputs from beneficiaries regarding:...

    • Prospective fraud
    • Behavior of clinic staff

    Inputs from clinic staff regarding:...

    • Security issues…
    • Prospective fraud

    Inputs from NI-ABAE Staff…

    • Security issues…
    • Prospective fraud
    • Behavior of clinic staff

    Office Money Spot-Check…

    • Last week submitted balance (check Google Form submission)
    • Expenses in current week (as per myDay App)
    • Current bank account balance (Carry out balance SMS check, if network available. Check last balance SMS if network not available)...
    • Total Cash on Hand (including disbursement & transport expenses since morning):
    • Difference in actual vs. expected balance"

    New Incentives, Audits and Investigations Guidelines (February 2024), pp. 5-6.

  • 56

    “- At least 5% of Disbursement Days receive visits by an independent unit of Field Auditors who submit an audit report, ~98% are unannounced.” New Incentives, Overview of Fraud Prevention Procedures (February 2024), p. 6.

  • 57

    “- At least 5% of Disbursement Days receive visits by Field Managers who submit a supervision report, ~92% are unannounced”. New Incentives, Overview of Fraud Prevention Procedures (February 2024), p. 6.

  • 58

    “Dash test [Field tag: Dash_Test]
    ASK: Did you or will you dash anyone at the clinic today” New Incentives, Program Protocols (January 2024), p. 46.

  • 59

    “- Vendors are randomly selected based on a defined sampling methodology and called by a member of the Audits and Investigations Unit to confirm the veracity of reported expenditure.” New Incentives, Overview of Fraud Prevention Procedures (February 2024), p. 5.

    “Auditors are to review select expense receipts submitted by the staff by making calls to the vendors and verifying if the full expenses were actually paid by the staff…Each Auditor has a target call of minimum 60 calls per week.” New Incentives, Audits and Investigations Guidelines (February 2024), p.4.

  • 60

    Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).

  • 61

    "This includes (a) reviewing data on the case (b) if warranted, on-ground investigation (c) issuance of Disciplinary Actions. . . . Cases where further investigation is needed. . . HR review and confirmation" New Incentives, Protocol: Inquiries and Investigations (unpublished).

  • 62

    “Beginning of Each Enrollment Day at the Clinic…Participate in the health talk to underline main messages about our program and clearly communicate the program’s eligibility criteria. The six different visits and the related cash transfers should also be clearly addressed (hold up one of the All Babies cards when speaking).” p. 15.

    “Explain how immunizations play a role in protecting against many common and deadly diseases.” p. 13.
    New Incentives, Program Protocols (January 2024).

  • 63

    "All Babies Card: Next Visit Date
    108. Add Next Visit to All Babies Card…
    ACTION: On the All Babies card, make a check mark next to today's transfer. Add the next visit date in the following format. Example: "I I I I" for four weeks. Did you do this?
    a. Yes, I made a check mark next to today's transfer and added the next visit date in the number of weeks format
    b. No, I was not able to make a check mark or was not able to add the next visit date in the number of weeks format
    c. Not applicable (infant fully vaccinated)
    109. Explain Next Visit…
    All Babies card: ACTION: Tell the beneficiary 'Today you got . . . Naira for . . . vaccines. In . . . weeks, on . . . date, you will get another . . . Naira.' Clearly show the card to the caretaker as you say this. (Explain the date with numbers of weeks. Example: 'You come back Wednesday four weeks from today. That is week 1, week 2, week 3 and in week 4 you come back on Friday.' Use your fingers to count the weeks and refer to the symbols on the card that indicate four weeks.) Did you do this?
    a. Yes, I explained this to the beneficiary while pointing to the All Babies card
    b. No, I was not able to explain this or was not able to point to the All Babies card
    c. Not applicable (infant fully vaccinated)"
    New Incentives, Program Protocols (January 2024), p. 37.

  • 64

    "Program Posters outside the entrance of treatment clinics and blue (distinctive) plastic bags handed to caregivers to increase visibility" New Incentives, Responses to 13-Feb-2020 Questions from GiveWell.

    Note that New Incentives informed us during review of this page that it has paused distribution of the promotional bags due to difficulty verifying their distribution. New Incentives may restart distribution in the future.

  • 65

    "Conducted by visiting medium and large settlements with participating clinic staff and influential community leaders to address concerns related to non-compliance, vaccine suspicion, and to increase program awareness. Sometimes this is paired with a Targeted Outreach." GiveWell, Questions for New Incentives on program activities, May 18, 2020.

    New Incentives informed us during review of this page that, "depending on the need and feasibility, influential stakeholders such as village leaders, ward focal persons, traditional leaders, district heads, health educators, LGA representatives, partners, and others are invited to support these sessions."

  • 66

    Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).

  • 67

    "Community Members Engagements Description: Small stipends are paid for activities including Defaulter Tracking, filtering out-of-catchment infants, identifying households and settlements with low uptake of immunizations, resolving identified uptake issues, attracting unimmunized infants, referrals, increasing program awareness, maintaining community support, and recently, for helping maintain COVID-19 adherence at clinics (like social distancing). The type of community member is selected based on the perception of the influence, recommendation by local leaders, and the value we get from their engagements. While most are selected for short periods of time to achieve specific goals (e.g. to increase defaulters in a settlement with reduced number of enrollments), some have longer standing engagements (e.g. identify out-of-catchment infants, helping maintain COVID-19 measures adherence at clinics). . . .
    Costs (Time and Monetary): Stipend of N500 is typically given for each community member engaged, with N1,000 given when multiple objectives need to be achieved or travel to additional settlements is needed. These are given during immunization days at the clinic. Around half of the recent contribution is at clinics for the filtering of out-of-catchment infants and for adherence of COVID-19 Measures." GiveWell, Questions for New Incentives on program activities, May 18, 2020, p. 3.

    During review of this page, New Incentives confirmed the community member stipend is 1,000 naira as of December 2023.

  • 68

    "Town Criers Conducted by making announcements using loudspeakers at settlements, informing caregivers about immunization days and outreaches. This serves as a reminder and helps maintain program awareness. Sometimes, town criers can also be engaged for Defaulter Tracking, in addition to their typical responsibilities.
    Target Group: Caregivers, community members, village and traditional leaders.
    Reach: Estimate of 20 caregivers reached per engagement.
    Costs (Time and Monetary): N500 per week of engagement, given during immunization days at the clinic." GiveWell, Questions for New Incentives on program activities, May 18, 2020, p. 4.

  • 69

    Ambassador Model
    Objective: Using beneficiaries from settlements with many defaulters to help track them down. The same can also be done with Volunteer Community Mobilizers (VCM) and Traditional Birth Attendants (TBA) that are motivated and frequent affected settlements. The end of the Health Talk is the time to recruit these Ambassadors. Ambassadors do a small community service and are unpaid…
    Ambassadors are only asked to go back to their home settlement and help track beneficiaries that they know already."
    New Incentives, Program Protocols (January 2024), p. 54.

  • 70

    New Incentives only plans to consider using radio messages in states where the program has fully expanded.

  • 71

    New Incentives informed us that these estimates are based on program data from 2021-2023. We have not independently reviewed that data to confirm these estimates.

    In 2020, New Incentives outlined its plans for awareness activities at scale here: GiveWell, Questions for New Incentives on program activities, May 18, 2020. See GiveWell, New Incentives program data, 2020, for details of the changes New Incentives expected to make to the frequency of activities and the total number of caregivers reached by past and future activities.

  • 72

    New Incentives did not provide an estimate of the number of caregivers reached via community member engagements. We expect that these numbers would be hard to track.

  • 73

    This data is based on information collected by FOs at the end of each disbursement. Choices are not mutually exclusive.
    "IF NEW BENEFICIARY: ASK: Where did you first hear about the All Babies program? Select all that apply.
    Had not heard about the program before today
    Neighbor
    Friends
    Family
    Traditional birth attendant
    Town crier
    Village leader
    Religious leader
    Radio
    Awareness card (small card from All Babies)
    UNICEF Voluntary Community Mobilizer (VCM)
    Community Health Worker
    Health worker at this clinic
    All Babies employee
    Other
    Unclear"
    New Incentives, Program Protocols (January 2024), pp. 33-34.

  • 74

    See GiveWell, New Incentives program data, 2020, "Awareness activities" sheet.

  • 75

    Note that we have not independently reviewed this data.

  • 76

    “The Supply-Side team’s primary mandate is to identify cases of stockout at any NI-ABAE affiliated facility. The primary cases of concern are any incident of stockout or runout of vaccines of any type, stockout or runout of CHCs, or an incident in which eligible children were not served by the facility.” undefinedNew Incentives, Supply-Side Action Protocol (January 2024), p. 4.

  • 77

    “Sources of case identification include:

    • Review of the Supply-Side Dashboard
    • Phone calls to ZCCO, SCCO, LCCO or officers in Apex facility
    • Significant cases identified during review of the VVM and Expiry Review Sheet
    • Information from other units like audit findings
    • Informal or formal conversation during other discussions or meetings (This is adhoc and would be carried out mainly by the SROs, SRM and SRD)”
      New Incentives, Supply-Side Action Protocol (January 2024), pp. 4-5.

    “As team members conduct routine calls to SCCOs, LCCOs, Apex clinics and other data sources, the content of those calls is recorded in the corresponding logs. Undesirable answers from the Fortnightly call with SCCO and LCCO Log or Fortnightly Call to Apex Clinics Log are transferred to the LGA or apex clinic level case log (3rd tab in the Stakeholder Relations & Supply-side Case Log).

    When filling the call logs, an undesirable answer is when a YES answer is obtained for questions about vaccine expiration or a NO answer to questions of enough vaccines or collection of VM1 forms in either the Fortnightly call with SCCO and LCCO Log or Fortnightly Call to Apex Clinics Log.” New Incentives, Supply-Side Action Protocol (January 2024), p. 5.

  • 78

    Field Officers and their managers are also trained in identifying and resolving stockouts at the facility level.

    "If a potential stockout is identified, Field Officers will coordinate with the routine immunization providers at their facilities to access top-up supplies that can meet that day’s demand. In many cases, these supplies are available at other nearby facilities or at the Apex facility, and stockouts are resolved promptly. However, the ability to access such supplies can vary based on the facility’s distance from another facility or cold store, provider motivation to resolve such a stockout, and vaccine availability at other stores." New Incentives, review of a draft of this page, January 2024

  • 79

    "Common steps toward resolution are:

    • Conducting follow-ups, reminders, or placing subtle pressure on stakeholders across all levels to carry out their responsibilities. This can include utilization and request forms, signing off on distributions, or pulling vaccines down the supply chains.
    • Advising and reminding LCCOs to pick up vaccines from the State store
    • Providing transportation support to the LCCO to pick up vaccines from the State or Zonal stores
    • Request to the LCCO to allocate reserved vaccines to our clinics
    • Advising FO or LCCO to retrieve stocked out vaccines from neighboring apex facility, LGA cold store, or State cold store
    • FOs cross check the clinic daily record with the clinic daily summary and the vaccine monitoring form (VM1 and VM2) at the health facility to improve supply-side data quality which feeds into the monthly report that the clinic sends to the LGA. This information is then used to fill the vaccine utilization and vaccine request required to receive vaccines from the national level.
    • Conduct weekly phone calls to the Apex facility, SCCOs, and LCCOs to identify stockouts at these levels as well as incidents of vaccines that are expiring soon. Also check to see if they have received the monthly report from the health facility to use in filling the vaccine utilization and vaccine request required to receive vaccines from the national level.
    • Reminder to the LCCO and SCCO to start this process of developing vaccine utilization and vaccine request on time and then follow up to ensure early submission."
      New Incentives, Supply-Side Action Protocol (January 2024), p. 6.

  • 80

    “VVMs are small indicators that adhere to vaccine vials and change colour as the vaccine is exposed to cumulative heat, letting health workers know whether the vaccine has exceeded a pre-set limit beyond which the vaccine should not be used.” World Health Organization, “Vaccine vial monitor (VVM)”, accessed October 23rd, 2023.

  • 81

    These activities are paraphrased based on New Incentives, Program Protocols (January 2024), pp. 60-66.

  • 82

    “Day-Of Stockout Incidents: In the event that a clinic has noted impending stockout on distribution day, FOs are encouraged to reach out to SSOs for support. In such a circumstance, SSOs can communicate to Apex clinics for a same-day ‘top-up’ disbursements that RI providers may access.” New Incentives, Supply-Side Action Protocol (January 2024), p. 9.

  • 83

    “Ongoing Stockout Incidents in High Consumption Facilities: SSOs are uniquely suited to support LGAs to identify and prevent stockout in high-consumption facilities due to their knowledge of stockout patterns. SSOs can proactively notify and coach LGAs on vaccine allocation strategies to prevent stockout through under-disbursement.” New Incentives, Supply-Side Action Protocol (January 2024), p. 9.

  • 84

    This understanding is based on many conversations with New Incentives over time.

  • 85

    New Incentives, responses to GiveWell questions, September 2020, unpublished.

    New Incentives also has a signed MoU with the Sultan Foundation for Peace and Development, which formalizes support from some traditional and religious leaders. New Incentives, Program Updates, November 2023 (unpublished)

  • 86

    “We plan to continue investing in state relationships, carrying out Joint State Meetings, and positioning the state to speak to program activities and results. At the same time, we are hearing stakeholders communicate that the incentives help increase coverage among hard-to-reach populations and security-compromised areas.” New Incentives, Program Updates, 7th July 2023 (unpublished).

  • 87

    Two of the people we spoke to expressed explicit support for the continued scale-up that this grant would support. The other individual, while not opposed to continued scale-up, expressed some concerns about program sustainability if New Incentives were to cease its support to a state. See this section of our 2021 grant page.

  • 88

    This understanding is based on many conversations with New Incentives over time.

  • 89

    New Incentives, responses to GiveWell questions, September 2020, unpublished.

  • 90

    This adjustment is intended to capture all kinds of potential non-funding bottlenecks, but we expect government opposition to be the most significant. This adjustment is relatively small because we have not yet seen evidence that it is harming New Incentives’ ability to deliver its program.

    See this section of our cost-effectiveness report on New Incentives’ program.

  • 91

    “2011: Inspired by evidence on the effectiveness of conditional cash transfers while doing research with the Center for Effective Global Action, Svetha Janumpalli raises seed funding from Unorthodox Philanthropy to create New Incentives…
    2012-13: New Incentives begins encouraging existing nonprofits to start offering conditional cash transfers and then pilots a program with a local partner in Nigeria for the prevention of mother-to-child transmission of HIV. New Incentives presents the pilot program at the 2012 Clinton Global Initiative Annual Meeting…

    2016…By the end of the year, New Incentives reassesses the evidence and the program’s theory of change for reducing neonatal mortality. The organization decides to pivot based on the evidence and starts collecting data from health clinics about childhood vaccination rates and challenges.” New Incentives, “History” page, accessed October 23rd, 2023.

  • 92

    New Incentives, Program units and responsibilities (unpublished):

    • "Logistics and Field Officers (120)." Slide 4.
    • "Enrolling beneficiaries into the program based on strict eligibility criteria
    • Detailed reporting on enrollments and diligent cash management." Slide 10.
    • "Reporting of supply-side issues through FM Check-in; same-day submission of Clinic Daily
    • Maintaining cordial relationship with clinic staff, community leaders, Imams, TBAs, LLOs, and other stakeholders that are part of the clinic's catchment area
    • Communicating expected supply-side requirements to clinic staff." Slide 23.

  • 93

    New Incentives, Program units and responsibilities (unpublished):

    • "Field Managers (10)." Slide 4.
    • "Managing the performance of each clinic under supervision." Slide 10.
    • "Field Manager (FM)
    • Ensuring that FOs are meeting their responsibilities . . .
    • Identifying and prioritizing core issues at clinics and settlements . . .
    • assessing the quality of completed activities and budget expenditure." Slide 17.

  • 94

    See this spreadsheet for further information.

    • Conditional cash transfers: 23% of spending
    • Cash transfer commitments: 10% of spending
    • 23% + 10% = 33% of spending on cash incentives

  • 95

    Cash transfer commitments are expected spending on cash transfers for future disbursements as enrolled infants complete routine immunization visits.

  • 96

    Our monitoring summary contains the definition used by New Incentives for each one of 47 indicators we have analyzed (see this sheet). However, this dashboard does not contain information on the data generating process for these indicators and we remain uncertain about how some indicators are defined (see the rest of this section for examples).

  • 97

    The 100 figure is based on New Incentives, Program Monitoring Indicators, unpublished. Our summary of New Incentives’ monitoring indicators contains a subset (47 indicators, as of January 2024) from this larger set.

  • 98

    See this section of our monitoring analysis for the definition for each indicator. In a conversation with New Incentives in September 2023 (unpublished), we confirmed our understanding that, if considering a specific month, the indicator refers to the % of infants who received the vaccine who were due to receive it that month. Numbers for prior months will get updated over time as infants may get vaccinated later than they were originally due.

  • 99

    See this section of our monitoring analysis.

  • 100

    “ANSWER: Which incentivized immunization(s) did the infant receive today based on the Child
    Health Card? SELECT ALL THAT APPLY.” New Incentives, Program Protocols (January 2024), p. 28.

  • 101

    See these rows in our monitoring analysis.

  • 102

    See this row in our monitoring analysis.

  • 103

    See the Nigeria childhood immunization schedule here.

  • 104

    "We are continuing to take actions to improve Measles 2 immunization rates and believe that an increase to 1,000 Naira (from 500 Naira) for the Measles 2 CCT (CCT #6 given at the 15-month RI visit) could lead to positive results. This would increase the total amount of cash transfers from 4,500 Naira to 5,000 Naira. It would represent a 25% increase from the program measured during the RCT, in-line with Nigeria’s inflation rates." New Incentives, Program Update, December 2022 (unpublished). New Incentives confirmed that it is moving forward with this change in an (unpublished) call with GiveWell in January 2023.

  • 105

    This hypothesis is supported by the fact that, after reviewing data collected during the RCT, we could not explain roughly 17% of enrollments reported by New Incentives on the basis of estimates of the number of infants living in and around clinic catchment areas. See GiveWell, Analysis of enrollment discrepancy, 2020, "% BCG disbursements not explained by OOC_per catch" sheet, for calculations.

  • 106

    “Old BCG Scar on Arms…IF NEW BENEFICIARY: ANSWER: Does the infant have an old BCG scar? Personally verify on the upper left and right arm of the child.” New Incentives, Program Protocols (January 2024), p. 24.

  • 107

    New Incentives has told us that Field Officers also usually mark the infants' fingers with a marker following BCG vaccination, meant to serve as a temporary vaccination marker during the period before scar formation. New Incentives believes this procedure helps deter duplicate enrollments of the same infant. New Incentives, conversation with GiveWell, September 28, 2023 (unpublished).

    We are not aware of how often this procedure is followed or how durable these marks are.

  • 108

    “BCG Scar on Arm…
    IF REPEAT VISIT: ANSWER: How many BCG scars does the infant have?
    Personally verify on the upper left and right arm of the child.” New Incentives, Program Protocols (January 2024), pp. 23-24.

  • 109

    Note: New Incentives has also shared monitoring data on other indicators that could provide evidence on this question:

    We do not currently use these metrics to estimate the rate of repeat enrollments, because we’re unsure how accurate New Incentives’ face recognition process is, and our understanding is that interviews are only conducted for a (possibly non-representative) sample of caregivers, whereas the BCG scar checks conducted by Field Officers are intended to take place at every disbursement. However, we’ve not investigated these indicators in as much detail as we have the data from the main BCG scar check data, and we may consider these in more detail in the future.

  • 110

    We found three studies reporting scarring rates of 82-96%. These average to 87% (unweighted) which is roughly in line with the 90% cited by WHO as the BCG scar rate in its 2018 position paper. We therefore use 90% as our rough best guess for the BCG scar rate.

    "BCG vaccination usually causes a scar at the site of injection due to local inflammatory processes. However scar formation is not a marker for protection and approximately 10% of vaccine recipients do not develop a scar." WHO, BCG vaccines position paper, February 2018, pg. 84 (downloadable here).

    "Two hundred and six subjects (96.3%) had a postvaccination BCG scar." Atimati and Osarogiagbon 2014.

    "The prevalence of BCG scar was 79.5% among the male infants and 84.7% among the female infants, while the overall prevalence of BCG scar was 81.5%." Gambo et al. 2014.

    "Although 84.2% had physical evidence of BCG inoculation only 69.8% had developed detectable sensitization to the tubercle bacilli as shown by the Mantoux test." Odujinrin and Ogunmekan 1992.

    • (96.3% + 81.5% + 84.2%) / 3 = 87.33% unweighted average BCG scar rate

  • 111

    In particular, ~99% had only one BCG scar in 2022 and &lt0.1% had two or more BCG scars. See these rows in our New Incentives monitoring analysis.

  • 112

    We estimate that the gap between the expected scar rate (90%) and the actual scar rate (99%) in 2022 was 9 percentage points (99% - 90% = 9%). As a proportion of the expected scar rate, this is ~10% of the total (9% / 90%).

    This calculation uses the following assumptions:

    • Caregivers of children who have a BCG scar after their first shot will not re-enroll for a second BCG shot. This is because New Incentives checks for BCG scars at the point of enrollment, and the presence of a BCG scar makes the caregiver ineligible for a BCG cash disbursement.
    • Some caregivers of children who do not have a BCG scar after their first shot will re-enroll in the program again to receive a second BCG cash disbursement. Since scarring occurs around 90% of the time, most of these children will have BCG scars after their second shot.

    See this row in our analysis of New Incentives’ cost per child enrolled for further details.

  • 113

    Our estimate of repeat enrollments was 8% in the first 2 years of the program, and rose to 10% in years 3 and 4. See this row of our analysis of New Incentives’ cost per child enrolled for more details.

  • 114

    The rotavirus vaccine was introduced into the Nigerian childhood vaccination schedule in August 2022. Gavi, "Dealing with diarrhoea: Nigeria introduces rotavirus vaccine into its immunisation plan," August 30, 2022.

  • 115

    As we note above, this is only a partial list of New Incentives’ monitoring indicators. As of January 2024, New Incentives has shared 24 supply monitoring indicators with GiveWell. Source: New Incentives, Program Monitoring Indicators, unpublished.

  • 116

    “Which vaccines were out of stock today according to the Facility Daily Immunization Tally Sheet or your observations? Confirm a potential stockout with a nurse as no vaccines tracked might in rare cases also mean that no child needed this particular vaccine.” p. 78.

    Field Officers also record whether any children were not served that day due to supply issues and the number of caregivers who have left by midday because the waiting times seemed too long (based on their observations).

    "All Children Served…
    Were all caregivers/infants served today or did the immunization day end prematurely (e.g. no time; ran out of vaccines)?
    All served
    Not all served
    Nr Children Not Served… how many caregivers were not served, i.e. sent home without receiving incentives? ____________" p. 72.
    New Incentives, Program Protocols (January 2024) .

  • 117

    See this row in our analysis of New Incentives’ monitoring. This metric is defined as the number of Clinic Daily surveys submitted as a proportion of the number of disbursement days (definition here). While we’d expect this to be a reasonably reliable measure of comprehensiveness, we haven’t investigated the methodology used to collect this indicator in detail and there may be reasons that this is less useful (e.g., if any supply-side questions are not required fields to complete the survey).

    “Specification…Submitting a Clinic Daily Record implies, Supply-side data is Submitted for the Session.
    This is the ratio between number of Clinic Daily records received to the Number of Disbursement Sessions.” New Incentives monitoring results, December 2022.

  • 118

    See this row in our summary of New Incentives’ monitoring data.

  • 119

    The rotavirus vaccine was introduced into the Nigerian childhood vaccination schedule in August 2022. Gavi, "Dealing with diarrhoea: Nigeria introduces rotavirus vaccine into its immunisation plan," August 30, 2022.

  • 120

    See this row in our summary of New Incentives’ monitoring data.

  • 121

    As of April 2024, the rotavirus vaccine accounts for 19% - 25% of the modeled benefits in our cost-effectiveness analysis, varying by state. See this row in our cost-effectiveness analysis. We do not explicitly model the benefits of the other indirectly incentivized vaccines and instead use a rough +4% adjustment to account for them, since we expect their share of the overall program benefits to be small. See this section of our report on the cost-effectiveness of New Incentives’ program for further details.

    Note: The 19% - 25% range includes the following states in Nigeria: Bauchi, Gombe, Jigawa, Kano, Katsina, Kebbi, Sokoto, and Zamfara. We focus on these states for the reasons discussed in this section of our report on the cost-effectiveness of New Incentives’ program.

  • 122

    See this row in our monitoring analysis. The indicator we have calculated looks at the ratio between the naira value of the incentives and the Nigeria consumer price index, as a proportion of the same ratio in the RCT. We use consumer price index data for Nigeria from the World Bank, available here.

  • 123

    “How much did you spend on one-way transportation to reach the clinic today?” Probe and enter the cost of one-way travel only, NOT roundtrip. Enter 0 if they did not pay anything for transport (example: if they walked to the clinic. Enter 99 if they do not know or refused to answer.” New Incentives, Program Protocols (January 2024), p. 23.

    See these rows in our monitoring analysis for these indicators over time.

  • 124

    See this row in our monitoring analysis.

  • 125

    New Incentives changed to a 6,000 naira schedule (1,000 naira per visit) in August 2023. Our understanding is that New Incentives made this change to make the initial enrollment visit more attractive, and to mitigate the impacts of inflation. New Incentives, Program Updates, July 7, 2023 (unpublished).

    • (6,000 - 4,000) / 4,000 = 50% increase in nominal value of incentives

  • 126

    GiveWell, conversation with New Incentives, November 14th, 2023 (unpublished).

  • 127

    Note that this is the reported one-way cost of transportation among caregivers reporting a non-zero transportation cost.

  • 128

    See these rows in our monitoring analysis.

  • 129

    See this row in our cost-effectiveness analysis.

  • 130

    "New Incentives will group local government areas (LGAs) it expands to within a given state at a given point in time into ‘expansion groups’. New Incentives will then collect coverage data in these expansion groups once before the start of operations to establish baseline coverage rates." IDinsight, Coverage Monitoring Analysis Plan, 2021, Pg. 1.

  • 131

    "The first follow-up will occur approximately 12 months after baseline." IDinsight, Coverage Monitoring Analysis Plan, 2021, Pg. 1.

    "Once operating in all LGAs in an expansion group (usually within six months of completion of baseline), New Incentives will obtain follow-up RI coverage estimates in these LGAs in six months intervals." IDinsight, Coverage Monitoring Analysis Plan, 2021, Pg. 1.

  • 132

    The first 4 cohorts include 18 LGAs as shown here. As of September 2023, New Incentives was working in 140 total LGAs (New Incentives, September 2023 Program Updates, unpublished).

    18 / 140 = 12.9% of New Incentives' program (not weighted by population targeted in each LGA).

  • 133

    Note that our summary is not inclusive of all of the fraud-related indicators that New Incentives monitors.

  • 134

    See this row in our monitoring analysis. Our understanding is that this indicator is based on data from console agents’ reviews of the photos submitted by New Incentives Field Officers using a New Incentives app after each disbursement, although we haven’t investigated the methodology used to produce this indicator in detail.

  • 135

    See this row in our monitoring analysis. Our understanding is that this metric is based on the number of caregivers answering the following question, as a proportion of the total number of disbursements reported by New Incentives Field Officers using a New Incentives app.

    “ASK: Did you or will you dash anyone at the clinic today…
    Yes
    No
    Dash Test Who…
    ASK: Did you or will you dash anyone at the clinic today? This is particularly for dashing to somebody at the clinic who might be in some way or another taking advantage of their position. If they give some cash to their family or neighbor, that is okay as that’s a choice that they are making…
    Clinic Staff
    All Babies Staff
    Security Guard
    UNICEF VCM
    Community Health Worker
    Government Staff
    Other"
    New Incentives, Program Protocols (January 2024), p. 47.

    Note: we think it is likely that this data is underestimated, since caregivers may not feel comfortable reporting this to New Incentives staff (e.g., since caregivers may not think that this report would be confidential and may get back to clinic staff). Field Officers are also unlikely to report tips that they receive themselves.

  • 136

    See this row in our monitoring analysis. We last reviewed the methodology used to collect this data in our 2020 review of New Incentives’ monitoring (available here), but have not updated our analysis over time to understand if the methods used are the same.

  • 137

    See this row in our monitoring analysis.

  • 138

    See this row in our monitoring analysis.

  • 139

    See this row in our monitoring analysis.

  • 140

    See this row in our cost-effectiveness analysis.

  • 141

    New Incentives has informed us that its data system assigns reviews of data submitted by Field Officers to console agents automatically and in real-time, so collusion between console agents and Field Officers would be difficult as there is no way to know in advance which console agent will be reviewing a specific Field Officer's data. However, we have not investigated this question in depth and we are unsure how well New Incentives' procedures would catch other forms of collusion between staff. If this type of fraud was taking place, we believe that it would be a significant risk to the program given the role of the console agent review in verifying that disbursements were legitimate. GiveWell, conversation with New Incentives in September 2023 (unpublished).

    We have also not investigated the possibility of fraud that includes collusion between field staff and other staff at New Incentives, including (for example) those with control of audit functions or information technology.

  • 142

    Available here. See this sheet for our analysis of the methodology used for New Incentives’ audit process, this sheet for our analysis of the methodology used for its other fraud prevention processes, and this sheet for our analysis of the overall fraud risk to the program.

  • 143

    This figure was 14% in 2022. See this cell in New Incentives’ monitoring spreadsheet.

  • 144

    Unpublished conversation with New Incentives, February 2023.

  • 145

    New Incentives, September 2023 Program Updates, unpublished; New Incentives, conversation with GiveWell, September 12, 2023, unpublished.

  • 146

    We look at the rate of duplicate enrollments within a 20km vicinity in our monitoring review. We have not yet updated these figures to reflect the corrected figures provided by New Incentives, but we plan to do so when we conduct our next monitoring review. We have preliminarily reviewed the corrected figures and do not believe they will substantially impact our assessment of the program. This is in part because we use a different method to estimate the number of repeat enrollments in New Incentives’ program (discussed above), which has generally found a higher % of repeat enrollments than New Incentives’ face-matching process (e.g., ~10% vs 4.5% in 2022), and we have not independently assessed how accurate New Incentives’ process is.

  • 147

    We estimate that the cost per child fell from $35.40 between June 2019 and May 2020 to $20.73 between June and December 2022. See this row in our analysis.

  • 148

    The value of the naira fell from 0.002778 per US dollar on June 1st, 2019 to 0.002171 per US dollar on January 1st, 2023 (roughly the same time period covered by this cost comparison). This suggests a fall of approximately 22% in the relative value of the naira, compared to a ~41% decrease in New Incentives’ cost per child enrolled (from ~$35 to ~$20). While this is only a rough comparison, it suggests that the falling value of the naira cannot account for all of New Incentives’ falling costs. Historical exchange rate data is from Wall Street Journal, Nigerian Naira/U.S. Dollar historical prices.

    • 0.002778 - 0.002171 / .002778 = 21.9% fall in value of the naira
    • $35.40 - $20.73 / $35.40 = 41.4% fall in New Incentives' cost per child enrolled

  • 149

    New Incentives, Program Updates, 7th July 2023 (unpublished)
    “Feedback Input Mechanisms from Program Participants and Implementers:

    • Data collected from cash disbursement sessions and clinic visits that collect data on clinic service reliability, clinic staff availability or absenteeism, vaccine supply issues, and other problems like charging for health services that are supposed to be free
    • Interviews during disbursement sessions
    • Observations during immunization days and feedback from clinic staff
    • Feedback from families and community members/leaders during awareness meetings, defaulter tracking, and other activities
    • Caregiver interviews by auditors during immunization sessions
    • Caregiver interviews by managers during immunization sessions
    • Caregiver interviews during audit visits in communities
    • A phone hotline (with associated audits)

    Feedback from stakeholders, LGA partners, other partners, and NGOs (often though SERICC meetings)

    • Feedback from monitoring and learning officers and site visits (informal interviews with small groups of caregivers similar to focus groups)
    • 1:1 meetings between team members
    • Anonymous feedback forms, whistleblower policy, and reporting channels
    • Open door policies and communication channels for complaints to be reported across levels
    • All-hands meetings and question forms (staff members are frequently encouraged to share questions and concerns)

    Feedback Review, Validation, Reporting and Actioning:

    • Feedback data is cleaned, validated, and in some cases, combined with other information to present a holistic picture
    • Dashboards and reports containing feedback from different data collection mechanisms is shared with teams to take actions and make improvements, as well as flag discrepancies
    • 1:1 meetings are reviewed and follow up items from previous meetings are checked to confirm actions taken
    • Reports on service delivery gaps are shared with partners at local, state and zonal levels to increase transparency and visibility of service delivery issues (including stockouts)
    • An issue tracking system called myDay Issues is in place to track the status and resolution of issues reported by staff”

  • 150

    The incentive structure was previously 500 naira for the first four visits, 2,000 naira for the fifth and 1,000 naira for the sixth. New Incentives changed this to a 1,000 naira per visit schedule. This increased the overall incentive from 5,000 to 6,000 naira, but reduced the incentive for the first measles dose (visit 5) to 1,000 naira.
    New Incentives, Program Updates, 7th July 2023 (unpublished).

    • (4 * 500) + 2,000 + 1,000 = 5,000 naira overall (old incentive structure)
    • 6 * 1,000 = 6,000 naira overall (new incentive structure)

  • 151

    New Incentives, Program Updates, 7th July 2023 (unpublished):
    “Feedback from Caregivers
    Process: Monitoring and Learning Officers took a participatory approach to this question, guiding the caregivers through the process by asking a series of questions, which the caregivers responded well to. The caregivers were very responsive to calculating the change in the total amount of incentives as it helped them better understand the impact on the total amount disbursed.

    • 1. How many RI sessions do you attend with your child?
    • 2. How much do you receive for each session? Can you please calculate the total amount?
    • 3. Assuming each session were to be incentivized at N1,000 per session, including a reduction for measles 1 from N2,000 to N1,000, how much will that be?
    • 4. NI is assessing whether it can update its incentive structure. How do you feel if each session is incentivized at N1,000 per session instead of the current structure?

    Feedback: Very Positive

    • There was an overwhelming positive response to the proposed update to the incentive structure due to the increased total as well as increase per disbursement. Caregivers understood that the Measles 1 amount would be N1,000 instead of N2,000 but that the overall amount will be greater, and appreciated that the organization is considering such a change.

    Feedback from Stakeholders
    Process: The change in disbursement structure was discussed with stakeholders from all currently operating states – Katsina, Bauchi, Jigawa, Kano, Gombe, Zamfara, and Sokoto. These were done primarily with senior health officials such as representatives of State Primary Health Care Development Agencies, PM SERICCs, and Director of PHCs.
    Feedback: Very Positive

    • The feedback was overwhelmingly positive and supportive. They felt that this was a good response to the increases in transportation costs (especially due to the removal of the fuel subsidy) and agreed that this would help with more unified messaging of N1,000/visit.”

  • 152

    New Incentives, conversation with GiveWell, September 12, 2023, unpublished.

  • 153

    See this section of GiveWell, New Incentives program data, 2020.

  • 154

    “We have enrolled 1,388,431 infants in 2023”. New Incentives, Program updates, December 2023 (unpublished).

  • 155

    See this row of our accompanying spreadsheet for the annual breakdowns.

  • 156

    In GiveWell’s May 2023 expansion grant, we expected New Incentives to be able to absorb annual budgets of up to $30.3 million in 2023, and $42.5 million in 2024. See this page for more details.

  • 157

    These checks are discussed above.

  • 158

    See this row in our cost-effectiveness analysis. This range includes the following states in Nigeria: Bauchi, Gombe, Jigawa, Kano, Katsina, Kebbi, Sokoto, and Zamfara for the reasons discussed in this section of our separate report.

    We also estimate that the program costs roughly $40 to $140 per additional child vaccinated. We calculate this by dividing an arbitrary grant size (in our CEA we use $1,000,000) by our estimate of the number of additional children reached by the program with that grant. For example:

    • Jigawa cost per additional child vaccinated: $1,000,000 / 9,052 children reached = $105
    • Bauchi cost per additional child vaccinated: $1,000,000 / 12,088 children reached = $83

  • 159

    See this row of our cost-effectiveness analysis. This range includes the following states in Nigeria: Bauchi, Gombe, Jigawa, Kano, Katsina, Kebbi, Sokoto, and Zamfara for the reasons discussed in this section of our separate report.

  • 160

    See this section of our separate report for more details.

  • 161

    See these sections of our cost-effectiveness analysis.