New Incentives - November 2020 Version

We have published a more recent review of this organization. See our most recent report on New Incentives.

Donate

Published: November 2020; Last Updated: October 2022

Summary

What do they do? New Incentives (newincentives.org) runs a conditional cash transfer (CCT) program in North West Nigeria which seeks to increase uptake of routine immunizations through cash transfers, raising public awareness of the benefits of vaccination and reducing the frequency of vaccine stockouts. (More)

Does it work? A randomized controlled trial (RCT) of New Incentives' program found strong evidence that the program increases vaccination rates. (This RCT was funded by Open Philanthropy on GiveWell's recommendation.) There is also strong evidence that vaccines effectively prevent the diseases they target. New Incentives collects data on an ongoing basis to monitor whether CCTs reach intended recipients. (More)

What do you get for your dollar? We estimate that the average cost to New Incentives for each child who is fully immunized through its program and who wouldn't otherwise have been fully vaccinated is $89. This estimate excludes costs paid by the Nigerian government, Gavi, and other donors who fund vaccinations in Nigeria. We estimate that, on average, the average total cost to all actors to fully immunize one child through New Incentives' program who wouldn't have been vaccinated otherwise is $179. The numbers of deaths averted and other benefits of the program are a function of a number of difficult-to-estimate factors, which we discuss in detail below. (More)

Is there room for more funding? We conduct "room for more funding" analysis to understand what portion of New Incentives' ideal future budget it will be unable to support with the funding it has or should expect to have available. We may then choose to either make or recommend grants to support those unfunded activities. (More)

New Incentives is recommended because of its:

• Focus on a program with very strong evidence of effectiveness and cost-effectiveness.
• Processes for monitoring that CCTs reach the intended recipients and that other aspects of the program operate effectively.
• Room for more funding – we believe New Incentives will be able to use additional funds to enroll additional infants.
• Transparency – New Incentives shares significant information about its work with us, and we are able to closely follow and understand its work.

Major open questions:

• Compared to our other top charities, New Incentives is at an earlier stage of its organizational development. In the next year, we expect it to roughly double the number of infants it reaches. This could lead to changes in cost per enrolled infant, effectiveness of its incentives, or unexpected problems. It may scale up less quickly than we expect, reducing its room for more funding.
• New Incentives' impact depends on what proportion of caregivers would not have vaccinated their infants in New Incentives' absence and what portion of those caregivers change their behavior in response to cash incentives and information. The RCT of the program was designed to measure these proportions by employing a control group and randomly selecting which clinics received the program. As the program scales up, it will become more difficult to measure New Incentives' impact. This situation differs to some degree from that of many of our other top charities, which deliver health commodities that have reasonably well-understood biological mechanisms.
• Government opposition is a potential risk to New Incentives' ability to operate its program. New Incentives has the support of several government agencies in Nigeria, including the state agencies whose permission it requires to work in clinics in each state and several national health and social protection agencies. Support for CCTs for immunization within the government of Nigeria is not universal, however, and its critics have raised concerns about whether CCTs are the most effective strategy for increasing immunization rates and whether CCTs, once ended, could lead to backlash against clinic staff or to caregivers declining to vaccinate infants without incentives. We believe these are important questions and have decided to recommend the program after considering them.

Our review process

We first recommended a grant to New Incentives in 2014. Our review process has consisted of:

• Making seven Incubation Grants to support New Incentives' development (details here).
• Funding IDinsight to lead, in partnership with Hanovia Limited, a randomized-controlled trial of the impact of New Incentives' program on infant vaccination rates in 2018-2020.
• Extensive communication with New Incentives' US-based leadership, founders Svetha Janumpalli and Pratyush Agarwal.
• Reviewing documents New Incentives shared with us about its operations, monitoring, costs, plans for the future, and risks it faces.
• Conversations with academics who have worked on evaluating programs to increase demand for vaccinations and a funder that has worked on immunizations in Nigeria.

All content on New Incentives, including details of past grants, blog posts and conversation notes, is available here.

What do they do?

New Incentives runs a conditional cash transfer (CCT) program in North West Nigeria. Caregivers who bring their infants to clinics for routine vaccines, which are provided through government clinics free of charge, can receive a total of $11 over the course of five clinic visits. New Incentives also does outreach to caregivers about the importance of vaccinating children, and works with its government partners to improve vaccine supply by identifying and addressing bottlenecks in the vaccine supply chain.

New Incentives started this program in May 2017. The program was evaluated by a randomized control trial (RCT) from July 2018 until February 2020.1 In 2019, New Incentives enrolled approximately 90,000 children, made 380,000 disbursements, and disbursed about $820,000.2 It currently works in three states: Katsina, Zamfara and Jigawa.

In Nigeria, New Incentives is known as All Babies Are Equal (ABAE) Initiative.3

Vaccines incentivized

New Incentives directly incentivizes (i.e., makes cash transfers conditional on infants receiving) the following vaccines:

• BCG (against tuberculosis)4
• PENTA (against diphtheria, tetanus, whooping cough, hepatitis B, and Haemophilus influenzae type b)5
• PCV (against pneumococcal disease)6
• MCV (against measles)7

These vaccines are part of Nigeria's routine immunization schedule and are provided by the government in public clinics (more on this below). Some of these vaccines are provided in more than one dose. New Incentives also indirectly incentivizes the Hepatitis B vaccine, OPV and IPV (the vaccines against polio), and the yellow fever vaccine (these vaccines are on the same schedule and are administered at the same visits as the directly incentivized vaccines; see schedule below).8 The meningitis A vaccine (MenA) was introduced into the Nigerian routine immunization schedule in 2019 as a single dose administered at 9 months (the same time as the measles and yellow fever vaccines) and is also one of the indirectly incentivized vaccines.9

Immunization schedule

Age of visit	Directly incentivized	Indirectly incentivized
At birth	BCG	Hep B, OPV0
6 weeks	PENTA1, PCV1	OPV1
10 weeks	PENTA2, PCV2	OPV2
14 weeks	PENTA3, PCV3	OPV3, IPV
9 months	Measles	Yellow Fever, MenA

Between 2020 and 2021, Nigeria is expected to add two doses of rotavirus and a second dose of measles (MCV2) to the schedule.10 New Incentives plans to indirectly incentivize the rotavirus vaccine;11 it has not yet decided how it will incentivize MCV2 and plans to pilot different options.12

Size of conditional cash transfers

New Incentives provides CCTs worth 500 naira (approximately $1.40) for each of the first four vaccine visits (from birth to 14 weeks) and one CCT worth 2000 naira (approximately $5.50) for a fifth visit for the measles vaccine (at 9 months).13 New Incentives decided on the transfer amount for the first four visits with the goals of covering the cost of transportation and providing a small additional incentive. The larger incentive amount for the measles vaccine was decided upon taking into account lower baseline vaccination rates for that vaccine, a relatively long time gap between the fourth and fifth vaccination visits, and the measles vaccine's high impact on mortality.14

Activities

The main goals of New Incentives' program are:

1. increasing vaccine demand by (a) distributing CCTs and (b) raising awareness about the importance of vaccinating children and the availability of CCTs for vaccination
2. improving vaccine supply by identifying and addressing bottlenecks in the vaccine supply chain

We cover these components in turn below.

Distributing CCTs

Vaccines incentivized by New Incentives are provided by the staff of government clinics that partner with New Incentives. Vaccinations and disbursements take place on days during which clinics provide routine immunizations ("immunization days").

When a child is vaccinated, New Incentives checks whether the child meets the eligibility criteria to receive a CCT and, if so, disburses a CCT to the child's caregiver. New Incentives also conducts audits and remote vetting of cash disbursements to ensure disbursements reach eligible caregivers.

Partnership with clinics

New Incentives pursues partnerships with clinics that:

• offer routine immunization services15
• are located in an area where the clinic is safe to visit during the day16
• are no more than three hours away from an ATM, bank, or point-of-sale terminal17

During the RCT, each of New Incentives' partner clinics aimed to serve infants in its own catchment area (that is, the geographic area whose population it is intended to serve), and New Incentives aimed to disburse incentives during all immunization days at partner clinics. At scale, New Incentives plans to determine the location and frequency of disbursements on the basis of the number and location of clinics and the estimated population that can access these clinics rather than aiming to disburse incentives during all immunization days at all clinics. New Incentives believes that a large proportion of infants who received disbursements during the RCT came from outside the catchment areas of the partner clinics. Since, at scale, partner clinics in a local government area (LGA) will be located closer to each other than during the RCT, New Incentives expects they will at times decrease the frequency of disbursements to sustain the same level of cost per child at scale.18

Eligibility for enrollment

Children are enrolled in New Incentives' program during their first vaccination visit (for BCG, for example, see the schedule above). The eligibility criteria for an infant to enroll in the program are:

1. The infant's caregiver is a resident of the clinic's catchment area (going forward the criterion will be residency in the LGA in which the clinic is located). This is checked by looking at the residence reported on the Child Health Card (see below) by the clinic staff and by asking questions of caregivers.19 New Incentives employs this criterion to avoid excessive vaccination volumes at its partner clinics.20
2. The child has received a BCG vaccine during that immunization day (and has not previously received a BCG vaccine). Because infants and caregivers do not consistently have documents that would allow New Incentives to identify them, New Incentives uses a fresh injection mark as evidence that a child has been injected with BCG during the day, and it uses the scar left by the BCG vaccine as a way of checking that children are not enrolled more than once.21

Eligibility for CCT disbursement

To be eligible to receive a CCT, an infant must:

1. Be enrolled in New Incentives' program. New Incentives checks that infants are enrolled by asking caregivers to provide the infant's ABAE ID (see below).22 This ensures New Incentives can identify each child and avoids repeatedly disbursing incentives to the same child for the same vaccination. (See the eligibility criteria for enrollment above.)
2. Have received a directly incentivized vaccination during the same immunization day. New Incentives verifies this by checking that there is a vaccination marked with the current date and a golden dot on the child's Child Health Card (see below).23
3. Be older than nine months, if receiving the measles vaccine. New Incentives verifies this by comparing the current date with the date of birth reported on the Child Health Card.24 The Nigerian government recommends that children receive measles vaccinations after 9 months of age.25 (We have not closely reviewed the evidence for this claim.)

Immunization days

Disbursements are provided during immunization days.26 The process is as follows:27

1. New Incentives typically assigns one field officer (FO) to a clinic.28 On immunization day (or the day before), the FO collects cash from an ATM, bank, or point-of-sale terminal before traveling to the clinic.29
2. At the start of the immunization day, clinic staff hold a health talk for caregivers, during which the FO introduces the New Incentives program.30
3. Clinic staff provide new children with a Child Health Card (CHC). A CHC is a document provided to caregivers that reports the infant's residence, date of birth, and vaccinations the infant has received on which dates.31
4. Infants are vaccinated according to the routine immunization schedule. When administering the vaccine, the clinic staff also write down the vaccination date on the CHC and mark it with a golden dot near the record of the vaccination.32
5. After vaccinations, the FO enrolls new infants and disburses cash to the caregivers of eligible infants. (See the eligibility criteria for enrollment and disbursement above.) Newly enrolled children are provided with an ABAE Card (a card that includes basic information about the New Incentives program and immunization schedule) and an ABAE ID (an identification number printed on stickers that are applied to the CHC and the ABAE Card).33
6. If the infant is being enrolled, the FO takes a picture of the infant's BCG injection mark. For all disbursements, the FO takes a picture showing the caregiver holding the cash, the child, and the ABAE Card.34 Later, the pictures are checked by remote staff (see below).35

During the RCT, New incentives disbursed approximately 80% of incentives on immunization days at clinics.36 At scale, New Incentives does not expect this to change drastically.37

Process compliance and anti-fraud checks

New Incentives monitors immunization days through:

• Console checks: Remote staff (called "console agents") check (a) that cash was disbursed to caregivers by cross-checking different sources that track disbursements and reviewing evidence submitted by FOs,38 and (b) that caregivers who received cash were eligible, by reviewing the pictures taken by FOs during immunization days documenting caregivers' eligibility.39
• Audits: New Incentives employs staff in the role of auditors who visit each catchment roughly every two months to assess compliance with protocols and check for evidence of fraud by (a) observing activities; (b) interviewing caregivers, clinic staff, and New Incentives staff; and (c) conducting a money spot check (i.e., verifying that the FO is in possession of the amount of cash expected based on New Incentives' records).40
• Fraud investigations: Cases of suspected fraud are collected through the console checks, audits, and staff complaints.41 Those are then assessed by auditors, who review the evidence to establish whether there is reason to believe fraud occurred.42

Awareness-raising activities

New Incentives' awareness-raising activities aim to increase the demand for routine immunization by increasing awareness of the program and sharing information with caregivers about immunization.43

During immunization days, New Incentives conducts the following awareness-raising activities:

• During the health talk, New Incentives' FO introduces the program and explains eligibility requirements.44
• After enrollment, FOs distribute promotional plastic bags to increase program visibility.45
• After CCTs are disbursed, FOs communicate and/or write down on caregivers' ABAE Cards the date of the next vaccination.46
• New Incentives advertises the program by putting up posters outside the entrance of partner clinics.47

New Incentives runs "awareness meetings" in targeted communities aimed at increasing knowledge of the program and addressing concerns about vaccinations. The meetings are run by New Incentives staff and clinic staff.48 Awareness cards (including basic information about the program) are distributed during these meetings.49 New Incentives also organizes targeted outreach sessions, which are vaccination sessions that occur at settlements, aimed at addressing low vaccination rates in certain areas.50 Vaccines for these outreach sessions are delivered by clinic staff, who are reimbursed by New Incentives for transportation costs.51

New Incentives also recruits community members to increase program awareness and to track infants who are behind schedule for receiving vaccinations and encourage their caregivers to complete the immunization schedule. Community members receive a stipend of $1.40 to $2.80.52 New Incentives also hires "town criers" to make announcements informing caregivers about immunization days and outreach sessions. New Incentives provides stipends for town criers of $1.40 per week of engagement.53

New Incentives shared the following estimates of frequency and reach of awareness activities:54

Activity	Engagements per clinic, future expected frequency	Estimated number of caregivers reached per engagement
Immunization day activities	2 per week	30
Awareness meetings	1 every 3 months	20-30
Community member engagements	1-2 per month	9
Targeted outreach sessions	1 per month	11
Town criers	1 every 1-2 months	20

New Incentives also shared data on how new program participants say they became aware of the program:55

Program Awareness Source	%
Neighbor	48%
Friends	39%
Family	31%
Traditional birth attendant	11%
Village leader	11%
Town crier	7%

Supporting vaccine supply

New Incentives' supply-side work targets vaccine supply at the local, zonal, and state levels for BCG, PENTA, PCV, OPV, measles, and yellow fever vaccines.56 This work consists of collecting data on issues with the supply of vaccines, and investigating and addressing the problems tracked. Supply chain issues are identified via:57

• Clinic-level data: The day before a scheduled immunization day, New Incentives' FO calls partner clinic staff to check whether the clinic has vaccinations in stock for the following day.58 During each immunization day, FOs fill out a form about the vaccine stock at the clinic.59
• Data at the apex clinic, zonal, and state levels: Every two weeks, New Incentives staff call cold chain60 officers working at "apex clinics" (larger clinics that store vaccines for clinics with more limited storage capacity), and at the LGA, zonal and state levels to collect information about vaccine stocks.61

Based on the data collected, New Incentives staff identify problems with the vaccine supply, such as stockouts, low vaccine stocks, or local authorities not having received the required documentation from clinics (in order to receive vaccine supply, clinics need to submit reports detailing their utilization and supply needs).62 New Incentives then informs relevant decision-makers about problems identified63 and, occasionally, provides financial support to relieve bottlenecks (for instance, by paying for transport costs to deliver vaccines).64

Staff structure

New Incentives' Operations Unit is in charge of distributing CCTs and running awareness-raising activities, as well as providing support for supply-side activities. Employees in this unit include:

• Field Officers, who are responsible for disbursing CCTs, running awareness activities, and collecting information about vaccine supply during immunization days.65
• Field Managers, who are responsible for supervising field officers, reviewing expenses, and identifying and addressing problems with enrollment, retainment, and vaccine supply at the clinic level.66
• State Field Managers, who are responsible for supervising field managers, tracking targets, and managing the state budget.67

Other New Incentives staff members include:68

• Supply-Side Officers, who are in charge of identifying issues based on data collected by Field Officers and maintaining regular contact with cold chain officers at the area, local, and zonal levels.
• Stakeholder relations team which is responsible for managing stakeholder relations and resolving supply-side issues at the local and state levels.
• Security management team, who are responsible for staff security and safety procedures.
• National Coordinator, who manages stakeholder relations and resolves supply-side issues at the national level, in addition to the general coordination role between operations, security, stakeholder relations, and supply-side activities.
• Software engineer.

New Incentives staff who review the data collected by FOs include:69

• Console Agents
• Auditors

Spending breakdown

Below we break down New Incentives' spending between November 2017 (when the program began) and February 2020 (the most recent data available as of this report).

Expenses, November 2017 to February 202070

Expense category	Total (m)	%
Conditional Cash Transfers (CCTs)	$1.7	29%
Staff compensation	$1.5	26%
Transport and supply costs for in-clinic and awareness activities	$1.2	21%
Contractors and consultants, including console agents71	$0.5	9%
Stakeholder relations (meetings and vaccine transport)	$0.2	3%
Other	$0.7	12%
Total expenses	$5.7	100%

Does it work?

On a separate page, we discuss the evidence on the effectiveness of New Incentives' program. We conclude that there is strong evidence that the intervention increases vaccination rates. Below, we discuss factors we plan to evaluate to assess the program's ongoing impact over time and possible negative and offsetting effects of the program.

Is the intervention effective?

A randomized controlled trial (RCT) of New Incentives' program found strong evidence that the program increases vaccinations. (This RCT was funded by Open Philanthropy on GiveWell's recommendation.) Combined with evidence that (a) vaccine-preventable diseases are a significant cause of child mortality in North West Nigeria, and (b) vaccines effectively prevent the diseases they target, we believe New Incentives is likely to reduce child mortality in North West Nigeria. More details are available in our intervention report.

Assessing the program's impact over time

New Incentives collects data on various aspects of its program's performance. In this section, we highlight certain types of data that New Incentives collects that we plan to use to help us assess how the program's impact over time compares to that measured during the RCT.

Retention throughout the vaccination schedule

Why this matters: If retention rates decrease over time, this may indicate that New Incentives' program is becoming less effective at incentivizing immunizations.

Available evidence: New Incentives estimates retention rates by calculating the percentage of enrolled infants who receive later vaccines.72 At enrollment, New Incentives provides infants with an ABAE ID (see above). When providing disbursements, New Incentives collects the infant's ABAE ID data, allowing it to tag the disbursements to a specific infant.73 We plan to compare New Incentives' retention rates over time to its retention rates during the period of the RCT.74

Repeat enrollments

Why this matters: It is possible that some infants are enrolled in the program more than once, either at the same or different clinic locations.75 We would guess that infants are unlikely to benefit from receiving the same vaccination more times than it is scheduled to be received, so if we count infants who are enrolled more than once as unique infants, then we will overestimate the cost-effectiveness of the program.

Available evidence: Caregivers have an incentive to enroll their infants in the program multiple times in order to receive additional CCTs. New Incentives aims to prevent the same infant being enrolled multiple times by (a) only allowing enrollment during a visit in which the infant receives the BCG vaccine, and (b) checking new enrollees for BCG scars, which indicate that the infant received the BCG vaccine previously. This system is imperfect because most but not all infants develop BCG scars and because scars take a couple of weeks to form, leaving a window for re-enrollment. To check the frequency of the same infant being enrolled multiple times, Field Officers look at infants' arms when they return for subsequent vaccinations and record the number of BCG scars on each arm.76

The World Health Organization (WHO) reports that about 90% of infants vaccinated with BCG develop a scar.77 Data that New Incentives has collected indicates that 97% of infants had a BCG scar when they came for subsequent vaccinations, while nearly no infants had two or more scars,78 which suggests that a small proportion of infants received the BCG vaccination more than once. Based on this data, we estimate that 8% of disbursements were for repeated vaccines.79
New Incentives has told us that it plans to conduct periodic household surveys to estimate vaccination coverage in the areas where it works, which could potentially detect large differences in the number of children enrolled in the program and the number of children found to be vaccinated in the community. This may shed further light on the frequency of infants receiving the same vaccination more than once. We do not yet know the details of how these surveys will be conducted.80 New Incentives is also considering employing biometric identification of caregivers as an additional method of identifying infants and preventing infants from being enrolled more than once.81

Supply-side issues

Why this matters: An increase in the number of vaccine stockouts and/or infants not served per immunization day may indicate that the program is becoming less effective at causing additional infants to be vaccinated.82

Available evidence: New Incentives' FOs collect data on stockouts (i.e., vaccines missing at the beginning of an immunization day), runouts (i.e., vaccines running out during an immunization day), and infants not served at the end of each immunization day. New Incentives estimates that there were five stockouts, one runout, and 20 infants not served per 100 immunization days during the period of the RCT.83 This data:

• covers 97% of immunization days84
• includes stockout and runout data for all vaccines directly incentivized
• primarily relies on FOs' direct observations, supplemented with clinic records and conversations with nurses as needed85

Reduction in the value of the transfer due to inflation

Why this matters: Inflation might cause the real value of New Incentives' CCTs to decrease, which may weaken their effectiveness as an incentive.

Available evidence: In addition to reviewing independent information on inflation rates in Nigeria, we plan to review data that New Incentives collects on caregivers’ transport costs as evidence of changes over time in overall costs in the areas where New Incentives works. Caregivers’ reported transport costs are collected during immunization days by FOs who ask each caregiver how expensive it is for them to reach the clinic. There may be an incentive for caregivers who traveled to the clinic from non-eligible areas to underreport their true transportation costs (since those costs could imply that they traveled from further away than they claim).86

Fraud at the expense of program participants

Why this matters: If fraud reduces the value of the CCTs that caregivers receive, New Incentives' ability to incentivize caregivers may decrease. Fraud of this type could include FOs not giving caregivers the full transfer amount or clinic staff, FOs, or others getting kickbacks after transfers.

Available evidence: Sources of evidence on this type of fraud include:

• Data on CCT disbursements: After disbursing a CCT, FOs take pictures of the caregiver holding the cash, the infant, and the ABAE ID. The photos are then reviewed by console agents.87 New Incentives reports that console agents have comprehensively reviewed the pictures and have observed caregivers holding the expected cash amount in 99% of cases.88
• Data on "dashes": New Incentives FOs ask caregivers, after each disbursement, whether they gave out "dashes," or tips, to clinic or New Incentives staff. Of the respondents, 0.1% reported giving dashes.89
• Data from audits: During audits, auditors collect data on fraud through (a) direct observation and (b) interviews of clinic staff, New Incentives staff, and program participants. This includes asking program participants whether they have noticed fraud by New Incentives or clinic staff. Auditors are instructed to conduct caregiver interviews out of earshot of FOs and clinic staff. We have seen reports for audits covering 5% of immunization days; 4.8% of those audits reported cases of suspected New Incentives or clinic staff fraud.90
• Data on New Incentives' fraud mitigation measures: See below for details.

Potential negative or offsetting effects

In our report of the program’s effectiveness, we discuss potential negative and offsetting effects included in our cost-effectiveness estimate. Below, we discuss potential negative and offsetting effects of the program that we believe to be too small or unlikely to have been included in our quantitative model.

Risk of HIV-infected children developing disseminated BCG disease

Children who are HIV-infected when vaccinated with BCG at birth are at increased risk of developing disseminated BCG disease (a disease with symptoms resembling tuberculosis, the disease against which the BCG vaccine is used). According to the World Health Organization, up to 4 in 1000 HIV-positive infants vaccinated with BCG develop disseminated BCG disease, and the disease has a case-fatality rate greater than 70%.91 Our understanding is that HIV prevalence is relatively low in the areas where New Incentives works.92

Side effects from repeated immunizations

As discussed above, we believe there is some risk of caregivers bringing infants to receive the same vaccine multiple times within a short interval. This might negatively affect infants' health beyond the ordinary side effects of vaccinations. However, we expect that this is unlikely to have significant negative effects on the treated population, since (a) we estimate that a relatively small percentage (roughly 10%) of children enrolled in the program receive repeated immunizations (more below) and (b) a brief review of the evidence did not indicate that repeated immunizations are likely to cause significant negative health effects.93

Security threats to staff

New Incentives works in areas at moderate to high risk of security threats.94 New Incentives reports that, over roughly two and a half years, it has recorded 23 incidents that were connected to the program in some way and that resulted in theft, injury, or death.95 The list includes incidents that involved program staff outside their work capacity as well as those that did not involve program staff but may relate to the program. Sixteen of the 23 incidents involved theft, including minor theft such as phones and cash. Five of the 23 incidents involved injury, and two involved deaths, including an incident in which two people died and three people were kidnapped. This incident did not involve New Incentives staff directly, but the assailants mentioned a vaccination cash transfer program as a reason for the attack. It is unclear from the report whether they said this because they intended to steal the CCT money or for some other reason.96

New Incentives procedures to decrease risks to its staff include:

• Collecting information about potential security threats and communicating with staff about threats. Information about security incidents is sourced on an ongoing basis by FOs,97 as well as by New Incentives' Security Manager and Security Focal Point.98 New Incentives has shared the list of these incidents with us.99 The list includes an average of 40 reports per month from June 2019 (when this logging system was introduced) to October 2020 (the date of our review); it was up to date at the time of our review. We spot-checked records for 50 incidents: for all of them, it was indicated whether further steps should be taken to mitigate the risk and, if so, what action had been taken. When an ongoing and urgent threat is identified, the Security Manager directly contacts relevant FOs.100 To communicate less-urgent security information, the Security Manager compiles weekly summaries, which are circulated to managers, who share them with FOs.101 At times, New Incentives designates high-risk areas as "no go" and prevents FOs from creating work plans or submitting expenses for these areas in the app used for this purpose.102 We have seen the Security Manager's weekly reports for five weeks in 2020; these reports describe the process used to identify threats and designate areas as "no go."103
• Training staff to avoid security threats where possible and address them where necessary. The onboarding process for New Incentives staff members includes a UN safety training course and a training course on New Incentives' internal security procedures.104 Ongoing monthly training is also provided, covering topics including road safety and abduction and kidnapping.105 We have seen the Security Manager's weekly reports on onboarding training for five weeks in 2020.106

Discontent of people who are not served on a particular day

It is possible that New Incentives' program causes discontent among caregivers who have to wait a long time at clinics or who are not served during a particular immunization day. We would guess the negative effects from discontent of this kind are likely small, because:

• There appear to be only a limited number of children who leave without being served. FOs collect data on infants not served at the end of each immunization day. During the RCT, New Incentives estimates that, on average, 20 infants were not served for every 100 immunizations days during which New Incentives disbursed cash.107
• When interviewed by New Incentives auditors, only 4% of caregivers complained about long wait times or not being served during a certain immunization day.108

"Crowding out" other motivations for vaccinating children

It is possible that, by creating a financial motivation to vaccinate infants, New Incentives' program "crowds out" intrinsic motivations to vaccinate infants. This might potentially lead to lower vaccination rates after the program is discontinued in an area than there would have been if the program had not been implemented. New Incentives told us of one case indicating this might be a concern in the area where it works: some caregivers reportedly refused to vaccinate their infants after in-kind incentives for a polio vaccination campaign were suspended.109 We would guess this potential effect would be partly offset by New Incentives' work to educate caregivers about immunization.110 Overall, we judge it unlikely that this consideration would significantly offset the program's benefits, though we have not investigated this question in detail.

Increased fraud and theft

New Incentives' program might lead to increased fraud or theft, including:

1. Fraud by caregivers, clinic staff, and/or New Incentives staff at the expense of New Incentives (e.g., non-eligible caregivers tampering with Child Health Cards to receive transfers)
2. Fraud by clinic staff and New Incentives staff at the expense of caregivers (e.g., clinic staff and New Incentives staff taking a portion of the disbursement)
3. Theft by third parties (e.g., stealing the cash transfer from caregivers or New Incentives’ staff)

New Incentives' procedures to prevent the first type of fraud are described here, and procedures to prevent the second type are described here. New Incentives reports nine cases of theft by third parties affecting New Incentives staff to date.111 New Incentives does not collect information about theft by third parties affecting program participants; it believes the risk is low, in part because caregivers only receive relatively small sums.112

Increased vaccine supply shortages in areas where New Incentives does not work

It is possible that New Incentives' program increases the likelihood of vaccine supply shortages in areas that New Incentives does not work in, via a combination of (a) CCTs and awareness-raising activities increasing demand for vaccines in areas New Incentives works in, and (b) vaccine supply support provided by New Incentives increasing vaccine supply flow to the areas it works in by diverting supply from other areas.113

As part of their supply support activities, New Incentives works to improve the communication between states, local government areas, and clinics in the states where it works. We would guess this benefits a wider set of areas than the ones where New Incentives works.114 As a result, we would guess it is unlikely that New Incentives has a significant negative effect overall in areas in which it doesn't work. However, we have not seen data addressing this directly.

Crowding out World Bank CCT funding

The World Bank has provided funding for a CCT program targeting very low-income households in Nigeria, under which each state can choose the condition for the transfers. Our understanding is that Katsina chose an education-focused condition for this program over a health-focused condition in part because it judged the health sector to already be better served than the education sector; knowledge of New Incentives' program may have affected that assessment. (Jigawa has chosen health-focused conditions for the transfers, while Zamfara had not selected a condition as of June 2020.)115 It is possible that New Incentives could have a similar effect on other government decisions in the future, in ways that are hard to predict. However, we believe this consideration is unlikely to significantly offset the benefits of New Incentives' program, since (a) our understanding is that New Incentives was only one of several organizations whose activities affected this decision, and (b) the World Bank program is expected to reach less than 1% of the population.116

What do you get for your dollar?

What is the cost per child immunized?

We estimate that the average cost to New Incentives for each child who is enrolled in its program is $30.117 This estimate includes all children enrolled in the program, regardless of whether or not they receive their full course of immunizations, and regardless of whether or not we believe that they did so as a result of New Incentives' program.

We estimate that the average cost to New Incentives for each child who is fully immunized through its program and who wouldn't otherwise have been fully vaccinated is $89.118 This estimate is higher because not all infants enrolled in the program receive their full course of immunizations and because we believe that a portion of enrolled infants would have been immunized without New Incentives' program. This estimate excludes costs paid by the Nigerian government, Gavi, and other donors who fund vaccinations in Nigeria. We estimate that the average total cost to all actors to fully immunize one child through New Incentives' program who wouldn't have been vaccinated otherwise is $179.119 These estimates rely on a number of uncertain assumptions.

Our approach

Our estimate of the total costs of the program includes:

• All costs paid by New Incentives.
• Other costs of additional vaccinations that occur because of New Incentives' program.
  We would guess those are mostly shouldered by the Nigerian government and Gavi.120

We start with this total cost figure and apply adjustments in our cost-effectiveness analysis to account for cases where we believe the charity's funds have caused these other actors to shift funds from a less cost-effective use to a more cost-effective use ("leverage") or from a more cost-effective use to a less cost-effective use ("funging"). Since the costs borne by the Nigerian government and Gavi account for a substantial percentage of total program costs, we believe spending by New Incentives causes these actors to spend a considerable amount of funding on immunizations that would likely have otherwise been spent less cost-effectively. This increases our cost-effectiveness estimate for New Incentives' program.

Some of the key assumptions we've made in estimating the cost-effectiveness of New Incentives' program over the next few years include the following:

• We expect New Incentives' future costs to be similar to its costs in the recent past. We divide New Incentives' total costs from June 2019 through August 2021 by the number of infants enrolled in the program during that time, as measured by the number of cash disbursements for the BCG vaccination.121 This is a key uncertainty in our model, since New Incentives is planning to scale up its program over the next few years, and operating at a different scale may lead to significant increases or decreases in charity costs.
• We incorporate an 8% adjustment to account for our estimate of the proportion of children who have been enrolled in the program multiple times (see above for more detail).122
• We use data published by WHO and by the government of Nigeria to estimate the government's costs and Gavi's costs.123

What is the cost per death averted?

See our cost-effectiveness model for estimates of the cost per death averted through New Incentives' incentives for immunization program.

Note that our cost-effectiveness analyses are simplified models that do not take into account a number of factors. For a list of factors excluded from our model, see this section of our report on New Incentives' incentives for immunization program.

There are limitations to this kind of cost-effectiveness analysis, and we believe that cost-effectiveness estimates such as these should not be taken literally, due to the significant uncertainty around them. We provide these estimates (a) for comparative purposes and (b) because working on them helps us ensure that we are thinking through as many of the relevant issues as possible.

Is there room for more funding?

We conduct "room for more funding" analysis to understand what portion of New Incentives' ideal future budget it will be unable to support with the funding it has or should expect to have available. We may then choose to either make or recommend grants to support those unfunded activities.

Room for more funding analysis

In general, we assess top charities' funding needs over a three-year period.124 We ask top charities to report their ideal budgets over the next three years, along with information about their current available funding and funding pipeline. The difference between a charity's three-year budget and the funding we project that it will have available to support that budget is the charity's "room for more funding."

The main components of our room for more funding analyses are:

• Available funding. We ask top charities to report how much funding they currently hold in the bank, including in reserves,125 and how much of this funding is committed or expected to be spent on specific future activities. The difference between these figures is the amount available to allocate to the charity's unfunded spending opportunities.
• Expected funding. We project the amount of additional funding that top charities will receive to support their work over the next three years. These projections represent our best guesses based on top charities' past revenue and our understanding of their funding pipelines. They typically include funding currently held by GiveWell to be granted to the top charity, projected funding due to being a GiveWell top charity,126 and, if the top charity is part of a larger organization, projected unrestricted funding from that parent organization. They exclude any funding we may specifically recommend to the top charity subsequent to the analysis. We add this projected funding to the amount available to allocate to the charity's unfunded spending opportunities.
• Spending opportunities. We ask top charities to report their ideal budgets in each of the next three years and to provide details on the specific spending opportunities included in these budgets. These opportunities are typically presented as one program year in a specific implementation geography, and they can represent either an extension of the top charity's previous support to a geography or an expansion of support to a new geography. We ask top charities to report the order in which they would prioritize funding these opportunities, which helps us to understand how available and expected funding will be allocated and what the marginal impact of additional funding beyond that amount would be.

  A charity's room for more funding represents the total budget for the charity's spending opportunities, less its available and expected funding. For example, if a charity proposes spending $50 million over the next three years and holds $10 million in uncommitted funding, and we project that it will receive an additional $15 million in revenue over the next three years, that charity's room for more funding is $25 million. (Note that a charity's total room for more funding figure includes funding gaps at all levels of cost-effectiveness—see below.)

  Grant investigation process

  Room for more funding analysis is a key part of our grant investigation process. We periodically request the information described above from top charities and update our room for more funding analyses. Our default is to update each top charity's room for more funding analysis annually, though we may choose to do so more or less frequently. The cadence on which we conduct updates depends largely on how often we grant funding to a top charity127 and how much we expect that charity's funding and budgets to have changed since our most recent funding decision.128 We have typically updated our analysis of New Incentives' room for more funding prior to large grant decisions.

  After completing such an update, we may then choose to investigate potential grants to support the spending opportunities that we do not expect to be funded with the charity's available and expected funding, which we refer to as "funding gaps." The principles we follow in deciding whether or not to fill a funding gap are described on this page.

  The first of those principles is to put significant weight on our cost-effectiveness estimates. We use GiveDirectly's unconditional cash transfers as a benchmark for comparing the cost-effectiveness of different funding gaps, which we describe in multiples of "cash." Thus, if we estimate that a funding gap is "10x cash," this means we estimate it to be ten times as cost-effective as unconditional cash transfers. As of November 2020, we have typically funded opportunities that meet or exceed a relatively high bar: 8x cash, or eight (or more) times as cost-effective as GiveDirectly's unconditional cash transfers. (Note that a charity's total room for more funding figure includes funding gaps at all levels of cost-effectiveness.)

  If we decide to fill a funding gap, we either make a grant from our Top Charities Fund129 or recommend that another funder—typically Open Philanthropy130 —makes a grant. This page lists all grants made or recommended by GiveWell. Typically, when GiveWell donors make a donation to a top charity,131 we don't expect that donation to be directed to a specific funding gap, but rather to contribute to supporting the overall portfolio of opportunities included within a charity's room for more funding.

  New Incentives' room for more funding

  We will consider making or recommending grants to fill funding gaps identified by New Incentives. As of November 2020, we expect to have sufficient funding at our discretion to fill all funding gaps we identify among our top charities that meet our current cost-effectiveness bar of 8x cash or better; we also expect to fill some funding gaps in the 5-8x cash range (for more details, see this blog post). For GiveWell donors who want to support the highest-priority funding needs among our top charities, we recommend donating to the Top Charities Fund. (We updated the name of this fund in September 2022; more information here.)

  New Incentives as an organization

  We use qualitative assessments of our top charities to inform our funding recommendations. See this page for more information about this process and for our qualitative assessment of New Incentives as an organization.

  Sources

  Document	Source
  Centers for Disease Control, "Ask CDC - Vaccines & Immunizations"	Source (archive)
  Centers for Disease Control, "Tuberculosis Fact Sheets: BCG Vaccine"	Source (archive)
  Dr. Obinna Ebirim, National Coordinator, New Incentives, email to GiveWell, June 12, 2019 (unpublished)	Unpublished
  Dr. Obinna Ebirim, National Coordinator, New Incentives, email to GiveWell, May 27, 2019 (unpublished)	Unpublished
  European Centre for Disease Prevention and Control, "Prevention and control measures for measles"	Source (archive)
  Gavi, "Demand promotion and community engagement"	Source (archive)
  Gavi, "Pentavalent vaccine support"	Source (archive)
  Gavi, Report to the Board, 2018	Source (archive)
  GiveWell, Cost-effectiveness analysis – version 4, 2022	Source
  GiveWell, New Incentives cost per infant immunized [February 2022]	Source
  GiveWell, New Incentives costs, 2021	Source
  GiveWell, New Incentives monitoring and evaluation 2017-2020 and plans for future evaluation	Source
  GiveWell, New Incentives program data, 2020	Source
  GiveWell, Questions for New Incentives about monitoring and evaluation	Source
  GiveWell, Questions for New Incentives about potential negative and offsetting effects	Source
  GiveWell, Questions for New Incentives about scale up (unpublished)	Unpublished
  GiveWell, Questions for New Incentives on program activities, May 18, 2020	Source
  Hesseling et al. 2009	Source (archive)
  IDinsight, Impact Evaluation of New Incentives in North West States of Zamfara and Katsina: Report on June Field Activities, 2017	Source
  IDinsight, New Incentives enrollment data discrepancies	Source
  IDinsight, New Incentives Evaluation Baseline Report, 2019	Source
  IDinsight, New Incentives Impact Evaluation, Preliminary Results Brief, 2020	Source
  Moro et al. 2019	Source
  National Agency for the Control of AIDS, "Nigeria Prevalence Rate"	Source (archive)
  New Incentives, ABAE Work Plan (unpublished)	Unpublished
  New Incentives, Answers to GiveWell's questions about scale up (unpublished)	Unpublished
  New Incentives, Audit Protocol	Source
  New Incentives, BCG Scar Update	Source
  New Incentives, Budget for scaling up in 2021 and maintaining the scale in 2022 and 2023, June 29, 2020 (unpublished)	Unpublished
  New Incentives, Cash Management Dashboard (unpublished)	Unpublished
  New Incentives, Cash Management Errors, Error Breakdowns	Source
  New Incentives, CCTs for immunizations pilot, November 2016 to February 2017	Source
  New Incentives, Clinic and settlement security assessments (unpublished)	Unpublished
  New Incentives, Clinic Daily form	Source
  New Incentives, Clinic screenings analysis	Source
  New Incentives, Console disbursement review fields	Source
  New Incentives, Disbursement Reviews	Source
  New Incentives, Disbursement Reviews errors	Source
  New Incentives, Disbursement Volume & Program Reach, Beneficiary Transport Cost	Source
  New Incentives, Disbursement Volume & Program Reach, Beneficiary Transport Costs and tips	Source
  New Incentives, Disbursement Volume & Program Reach, Disbursement Volume	Source
  New Incentives, Disbursement Volume & Program Reach, Program awareness	Source
  New Incentives, Financial Summary RCT period	Source
  New Incentives, Incidents involving staff (unpublished)	Unpublished
  New Incentives, Inquiries dashboard, BCG Scar verification	Source
  New Incentives, Log of Enquiries (unpublished)	Unpublished
  New Incentives, Overview of NI-ABAE Anti-Bribery and Security Policies	Source
  New Incentives, Program Protocols 2018-2020	Source
  New Incentives, Program units and responsibilities (unpublished)	Unpublished
  New Incentives, Progress Dashboard, Clinic Supply Outages and disbursement days, costing analysis period	Source
  New Incentives, Progress Dashboard, Clinic Supply Outages and disbursement days, misc periods	Source
  New Incentives, Progress Dashboard, Clinic Supply Outages and disbursement days, RCT period	Source
  New Incentives, Progress Dashboard, Second Visit & Immunisation Rate	Source
  New Incentives, Protocol for Console Agents (unpublished)	Unpublished
  New Incentives, Protocol: Inquiries and Investigations (unpublished)	Unpublished
  New Incentives, Responses to 13-Feb-2020 Questions from GiveWell	Source
  New Incentives, Responses to questions from GiveWell, September 23, 2020 (unpublished)	Unpublished
  New Incentives, RI Hotline Responses (unpublished)	Unpublished
  New Incentives, Security incidents dashboard (unpublished)	Unpublished
  New Incentives, Security Procedures and Status (unpublished)	Unpublished
  New Incentives, Stakeholder Dashboard	Source
  New Incentives, Stakeholder Relations & Supply-side Case Log (unpublished)	Unpublished
  New Incentives, Supply-Side Action Protocol	Source
  New Incentives, Supply-Side Dashboard, Vaccine Stockouts & Ranouts	Source
  New Incentives, Supply-Side Dashboard, Vials VVM & Expiry Assessment	Source
  New Incentives, Supply-Side Workflow and Plan	Source
  New Incentives, Updated budget for expanding at full speed in the next three years (unpublished)	Unpublished
  New Incentives, Vaccinations Dashboard, Comparison Between Periods, BCG scar data	Source
  New Incentives, Vaccinations Dashboard, Comparison Between Periods, costing analysis period	Source
  New Incentives, Vaccinations Dashboard, Comparison Between Periods, misc	Source
  New Incentives, Vaccinations Dashboard, Comparison Between Periods, RCT period	Source
  New Incentives, Vaccinations Dashboard, Vaccination Volume and Median Age	Source
  Nigeria Strategy for Immunisation and PHC System Strengthening (NSIPSS), 2018	Source (archive)
  Pratyush Agarwal, Co-founder and COO, New Incentives, email to GiveWell, August 10, 2020 (unpublished)	Unpublished
  Smith 2019	Source (archive)
  Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversation with GiveWell, June 16, 2020 (unpublished)	Unpublished
  Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversation with GiveWell, March 10, 2020 (unpublished)	Unpublished
  Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished)	Unpublished
  Svetha Janumpalli, Founder and CEO, New Incentives, email to GiveWell, June 24, 2020 (unpublished)	Unpublished
  Svetha Janumpalli, Founder and CEO, New Incentives, emails to GiveWell, June 19 to December 9, 2019 (unpublished)	Unpublished
  US Department of State, Bureau of Consular Affairs, "Nigeria Travel Advisory," 2020	Source (archive)
  World Health Organization, "Immunization, Vaccines and Biologicals: Pneumococcal disease"	Source (archive)
  World Health Organization, "WHO African Region: Ethiopia, EPI logistics"	Source (archive)
  World Health Organization, BCG vaccines: WHO position paper – February 2018	Source (archive)
  World Health Organization, GHO data, HIV/AIDS	Source
  World Health Organization, Immunization Financing Indicators, Indicator 3: Total expenditure (from all sources of financing) on routine immunization	Source (archive)
  World Health Organization, Information Sheet: Observed Rate of Vaccine Reactions Bacille Calmette-Guérin (BCG) Vaccine, 2012	Source (archive)
  World Health Organization, Nigeria: WHO and UNICEF estimates of immunization coverage: 2019 revision	Source (archive)
  World Health Organization, Vaccine-preventable diseases monitoring system, 2020 global summary, Nigeria	Source (archive)

  • 1
    • "New Incentives was not operating in the study clinics in months prior to Oct 2017 and was fully operational in all study clinics by July 2018, the beginning of the RCT window." IDinsight, New Incentives Impact Evaluation, Preliminary Results Brief, 2020, p. 14.
    • Note from New Incentives, provided when reviewing a draft of this page: "The program started in May 2017 when the 'learning sites' commenced enrollment."

  • 2

    See GiveWell, New Incentives program data, 2020, "Program basics" sheet, for details.

  • 3

    Note from New Incentives, provided when reviewing a draft of this page: "The name New Incentives was not accepted during corporate registration in Nigeria so the available name, All Babies Are Equal (ABAE) Initiative was used instead."

  • 4

    "BCG, or bacille Calmette-Guerin, is a vaccine for tuberculosis (TB) disease." Centers for Disease Control, "Tuberculosis Fact Sheets: BCG Vaccine"

  • 5

    "Pentavalent vaccine protects against five major diseases: diphtheria, tetanus, pertussis (whooping cough), hepatitis B and Haemophilus influenzae type b (DTP-hepB-Hib)." Gavi, "Pentavalent vaccine support".

  • 6

    "There are 2 available pneumococcal conjugate vaccines (PCV) that target either 10 or 13 of the most prevalent serotypes." World Health Organization, "Immunization, Vaccines and Biologicals: Pneumococcal disease".

  • 7

    "[I]n countries with endemic measles, the first dose of measles containing vaccine (MCV) is given as early as nine months, often complemented by another dose during the second year of life." European Centre for Disease Prevention and Control, "Prevention and control measures for measles".

  • 8

    TABLE 1: New Incentives and the Routine Immunization Schedule, IDinsight, New Incentives Evaluation Baseline Report, 2019, p.15.

  • 9

    Note from New Incentives, provided when reviewing a draft of this page in October 2020.

  • 10
    • "MCV2 will be rolled out in northern states in 2020. Nationwide rollout of the Rotavirus vaccine is being planned for 2020." New Incentives, Responses to 13-Feb-2020 Questions from GiveWell.
    • Updated timeline based on @Pratyush Agarwal, Co-founder and COO, New Incentives, email to GiveWell, August 12, 2020 (unpublished)@.

  • 11

    Table 1: New Incentives and the Routine Immunization Schedule, IDinsight, New Incentives Evaluation Baseline Report, 2019, p.15.

  • 12

    Some options it is considering include (a) splitting the current measles CCT or (b) adding a second CCT of the same size for the second dose. Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).

  • 13

    "Briefly mention the stages and the amount (500 naira at Birth, 6 weeks, 10 weeks, and 14 weeks, and 2,000 naira at 9 months for Measles)." New Incentives, Program Protocols 2018-2020, p. 47 (questions 58-59). See GiveWell, New Incentives program data, 2020, "Program basics" sheet, for conversion calculations.

  • 14

    "As vaccination visits are largely free (or a maximum of N100), we hypothesize that the cash amounts only need to cover transportation costs and provide a small additional incentive. Earlier vaccination visits in the Nigerian Child Immunization Schedule (see table) are incentivized with lower amounts given the higher baseline rates. The fifth and final visit in the schedule at 9 months (Measles and Yellow Fever) will be incentivized with a higher amount. This takes into consideration the low baseline rates around 30-50%, the lag to the 14-week vaccination visit, and the high impact of the Measles vaccine on mortality." New Incentives, CCTs for immunizations pilot, November 2016 to February 2017, p. 1.

  • 15

    "Clinics that are not operational or not offering RI [routine immunization] (as confirmed at the LGA and state level) are sorted out and not visited for a Clinic Screening." New Incentives, Clinic screenings analysis, "Statistics: All Clinics Screened" sheet, cell A35.

  • 16"Clinics that are in no go security areas (as confirmed at the LGA and state level) are sorted out and not visited for a Clinic Screening. No Go Zones cannot even be accessed during daytime." New Incentives, Clinic screenings analysis, "Statistics: All Clinics Screened" sheet, cell A36.

  • 17

    Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished) and note from New Incentives, provided when reviewing a draft of this page. Note, however, that no clinic was disqualified on this basis during the screening preceding the RCT.

  • 18

    Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversation with GiveWell, June 16, 2020 (unpublished).

  • 19
    • New Incentives, Program Protocols 2018-2020, p. 32, provides two responses to the question, "How do we find out if a mother is from the catchment area?"
      • "The nurse probes hard to write the correct address/settlement on the Child Health Card.
        (You have to remind nurses if they don't do this!)"
      • "You as the field staff check the address/settlement on the Child Health Card to verify
        that the address is in the catchment area. In addition, you ask the caretaker again about
        which settlement she is from. If the written settlement is in the catchment and the
        beneficiary gives the same answer again, then the mother is eligible."
    • In addition, New Incentives also collects data on transport costs. It uses this data as a check of eligibility during disbursement, since high transport costs are evidence of the caregiver coming from a settlement outside of the catchment area. Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).

  • 20

    New Incentives does not plan to make residence in the program catchment an eligibility criteria because it wants to increase the chances the caregiver will report the true residence settlement. New Incentives plans to conduct immunization sessions outside the partner clinic, in areas of low coverage, and it aims to use residence data to identify low-coverage areas. In addition, New Incentives does not expect there to be a high risk that clinics will be overwhelmed because partner clinics will be reasonably close to one another at scale. Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversation with GiveWell, June 16, 2020 (unpublished).

  • 21
    • "Fresh BCG vaccination mark: New infants should visit our desk only if/after they
      received their BCG immunization that day. We check to see that their mark is fresh to
      confirm this." New Incentives, Program Protocols 2018-2020, p. 32.
    • "23. IF NEW BENEFICIARY: ANSWER: Does the infant have an old BCG scar? Personally verify on the upper left and right arm of the child. Does the infant have an old BCG scar and which arms did you check?
      a. Yes, there is an old BCG scar and I checked both of the infant's arms for the old BCG scar
      → SKIP to close without saving
      b. Yes, there is an old BCG scar but the infant is new and has required evidence
      c. No, there is no old BCG scar and I checked ONE of the infant's arms for the old BCG scar
      d. No, there is no old BCG scar and I checked BOTH of the infant's arms for the old BCG
      scar
      24. IF NEW BENEFICIARY: ANSWER: Does the infant have a fresh BCG injection mark?
      a. No, infant is NOT eligible to enroll → SKIP to close without saving
      b. No, but infant is new and there is required documentation
      c. Yes, infant is eligible to enroll
      d. Unclear, I can't tell if the infant has a fresh BCG injection mark (uncommon response!)" New Incentives, Program Protocols 2018-2020, p. 43.
    • This requirement is waived if the child is too old to receive BCG, or can prove they received BCG at a different health facility. New Incentives, Program Protocols 2018-2020:
      • "However, for cases where they may have gotten enrolled elsewhere (e.g. infant got BCG at the point of delivery at another hospital), an exception can be made with the right evidence and process. Ensure that the caregiver presents the CHC from the clinic (non-ABAE Clinic) where the BCG was given. Do not renew cards otherwise we will be exposing the criteria to fraud by the clinic staff (highest risk) and the caregivers.
        • Confirm with Line-Listing register from LGA to confirm if that infant lives in the particular settlement or not. You can get the updated Line Register copy from the LGA once a month.
        • If you make the correct selections, you will be asked for one more photo after the BCG photo: circle the line with the name of the infant and caregiver in the Line Register copy, then click the photo.
        • Mark the BCG immunization with blue dot as always (note: this implies that the clinic staff will not make a golden dot since the infant did not receive BCG at the clinic).
      • OLD BCG on Arms questions in doforms
        Select the first option: "Yes, there is an old BCG scar and I checked both of the
        infant's arms for the old BCG scar." P. 69.
      • "Exception in rare cases: infants that did not get a BCG vaccination (because they are too old when they get the first vaccination, e.g. 1.5 years old) are eligible too." P. 64.

  • 22
    • Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).
    • "ACTION: Take a photo of the left side of the Child Health Card and SAY: 'I am registering you with this photo.' (Ensure that the All Babies ID and all caretaker data is visible.) Will you do this with the next field?" New Incentives, Program Protocols 2018-2020, p. 48.

  • 23
    • "ACTION: Confirm that the Child Health Card has 1) Settlement Name, 2) Gold Dots next to
      vaccinations, and 3) Next Visit Date. If any are missing, send the caretaker back to the nurse. Did you confirm all three items and if not correct, send the caretaker back to the clinic staff?" New Incentives, Program Protocols 2018-2020, p. 41.
    • "If the caregiver has misplaced the ABAE Card, then a new one can be issued by New Incentives staff, based on information on the CHC. If the CHC is lost then a new one can be issued by the clinic staff, after checking information on the Child Immunisation Registry. If both are lost, then the FO collects identifying information from the caregiver, and submits a 'history retrieval' request to remote staff. If remote staff is able to match the information to an existing profile, a new ABAE card is issued. This usually requires time, so caregivers are asked to come back during a different immunisation day." New Incentives, Program Protocols 2018-2020, p. 64-66.

  • 24

    "HIDDEN: ACTION: Measles Age ALERT - The infant got a Measles vaccine today. Check to see that today is really 9 months or more after the infant's birth. The vaccine is not effective if an infant gets the Measles vaccine before 9 months after birth! Example: an infant born in November 2017 cannot get Measles in June 2018 (only 8 instead of 9 months). -- Do NOT pay the Measles incentive if the infant is less than 9 months old. Ask the caregiver to come back for a second Measles shot at 9 months and only pay the N2000 at that point.
    a. I checked and paid the N2000 as the infant is more than 9 months old
    b. I checked and DID NOT PAY the N2000 as the infant is less than 9 months old (do not
    send this form)
    c. NA"
    New Incentives, Program Protocols 2018-2020, p. 44.

  • 25

    Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).

  • 26

    "Field staff go to clinics on immunization days to distribute cash transfers after the condition (vaccination given) has been verified." New Incentives, Program Protocols 2018-2020, p. 3.

  • 27

    In addition to the steps described below, Field Managers use phone calls to screen each Field Officer for COVID-19 symptoms, conduct a pre-travel security review, and ask about potential vaccine supply issues. Before leaving for immunization day, Field Officers fill out a brief form covering various questions about potential COVID-19 symptoms, known or suspected cases in their locality, in addition to other questions. See more about New Incentives' response to COVID-19 here.

  • 28

    Unless it expects more than 75 disbursements per day. Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).

  • 29
    • The New Incentives FO collects money from the ATM before heading to the clinic; the money is stored in a waist bag. They can carry a maximum of 200,000 naira (about $550), but usually carry less than 100,000 (about $280). If an FO runs out of money during the day, they can go and get more from the ATM, ask their manager for help, or close up the disbursement. This is then recorded in the supply dashboard and addressed by the field manager. Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).
    • "Collect cash at ATM (according to cash management regulations of New Incentives)
      - Ensure that you have Cash for cash transfers in waist bag. . . .
      - Ensure that you have a proper table and chair at the clinic. This is important for orderly review and payout." New Incentives, Program Protocols 2018-2020, p. 35.

  • 30

    "Participate in the health talk to underline main messages about our program and clearly communicate the program's eligibility criteria. The five different visits and the related cash transfers should also be clearly addressed (hold up one of the All Babies cards when speaking)." New Incentives, Program Protocols 2018-2020, p. 36.

  • 31
    • "Child Health Card: tracks vaccinations of infant, handed out to mother" (and picture) New Incentives, Program Protocols 2018-2020, p. 10.
    • After delivering and checking CHCs, clinic staff compiles the Child Immunisation Registry, a document held at the clinic that tracks information about infants served. "Child Immunization Register: tracks infants at the clinic; sorted by date of birth or by settlement" (and picture) New Incentives, Program Protocols 2018-2020, p. 11.

  • 32
    • This mark is made with a pen provided by New Incentives. In the middle of the day, the FO also checks whether the golden pen is being used by the vaccinating nurse and is being used correctly.
    • "How did the nurses use the gold pen? Remind them to always use the gold pen but only after the vaccination took place (not during registration!).
      • Gold pen is used during registration (wrong)
      • Gold pen is used after vaccination (correct)
      • Gold pen is not used (wrong)"

      New Incentives, Clinic Daily form, p. 7.

    • "Hand out golden pen to the nurse(s) who will be administering the vaccines." New Incentives, Program Protocols 2018-2020, p. 36.
    • "ANSWER: How many injectable vaccinations on the infant’s Child Health Card with today’s date also have a gold dot?" New Incentives, Program Protocols 2018-2020, p. 44.

  • 33

    The sticker is also stamped so that it becomes easier to tell if it has been moved from a different CHC/ABAE card: "42. IF NEW BENEFICIARY: ACTION: Put a matching pair of All Babies ID stickers on the Child Health Card and All Babies card. Stamp the stickers so that if the sticker is removed and placed on another Child Health Card, we will easily know." New Incentives, Program Protocols 2018-2020, p.45.

  • 34

    Specifically, the FO takes a picture of (a) the fresh BCG injection mark (for enrollment), (b) the side of the CHC, including the ABAE ID (for disbursement), and (c) the other side of the CHC, including the vaccination list (for disbursements). See New Incentives, Program Protocols 2018-2020:

    • "IF NEW BENEFICIARY: ACTION: Take a photo of the fresh BCG injection mark on the left upper arm of the child. Ensure that you take the photo close to the upper left arm and ensure it is sharp by tapping the screen." P. 41.
    • "Take a photo of the left side of the Child Health Card and SAY: 'I am registering you with this photo.' (Ensure that the All Babies ID and all caretaker data is visible.)" P. 48.
    • "ACTION: Take a photo of the right side of the Child Health Card and SAY: 'I am capturing your vaccinations.' Ensure that all vaccinations are visible." P. 49.

  • 35

    New Incentives noted in response to reviewing a draft of this page that this is done “[i]n a secure manner using an application that maintains data security and access controls.”

  • 36

    "Clinic session: (82%) and outreach sessions (17%); outreach session are routine sessions conducted in settlement; campaigns (~1%); campaigns are non-routine sessions conducted in settlements." New Incentives, Responses to 13-Feb-2020 Questions from GiveWell, table on p. 3.

  • 37

    Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversation with GiveWell, June 16, 2020 (unpublished).

  • 38

    The console agents cross-check (a) "Field expenses" (i.e., expense records submitted by FOs), through an app called "myDay"; they are used for all expenses (including disbursements and awareness-raising activities); (b) "Cash report," reports of the amounts of cash held by the FO before and after visiting the clinic, and a picture of cash after visiting the clinic; (c) disbursement records, as logged by FOs on doForm, the form used to guide FOs during disbursements and to collect infants' information; and (d) console agents' review of the pictures taken by FOs to document each disbursement. Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversation with GiveWell, March 10, 2020 (unpublished).

  • 39
    • Among other things, console agents check whether the BCG picture shows an old BCG scar, whether the BCG picture shows a fresh injection, whether the CHC picture includes the ABAE ID, and whether the ABAE ID sticker is torn.
    • "BCG Photo Quality: present, missing, blurry (Present/Missing indicates BCG Scar is present
      or missing)
      New BCG Scar: Y/N
      Identity Photo Quality: clear, blurry, incomplete, crossed, wrong
      ABAE ID: ABAE ID
      ABAE sticker torn : Y/N" New Incentives, Console disbursement review fields, p. 3.

  • 40

    "Minimum Inputs Expected from Disbursement and Activity Audits
    Observations by Auditor on the following points: Clinic Staff Protocol Non-Compliance . . . NI-ABAE Staff Protocol Non-Compliance. . . .
    Inputs from Beneficiaries . . . Inputs from Clinic Staff . . . Inputs from the NI-ABAE Staff . . . Office Money Spot Check . . . Auditor conclusion: Do you think the FO is managing the Office Money correctly?" New Incentives, Audit Protocol, pp. 3-4.

  • 41

    Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).

  • 42

    "This includes (a) reviewing data on the case (b) if warranted, on-ground investigation (c) presenting the findings to the Disciplinary Committee. . . . Cases where further investigation is needed. . . Management Review and confirmation" New Incentives, Protocol: Inquiries and Investigations (unpublished).

  • 43

    In particular, focusing on the educational aspect, New Incentives reports using awareness activities to educate caregivers about misconceptions about vaccinations, concerns about side effects, and the need for immunization. Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).

  • 44

    "Participate in the health talk to underline main messages about our program and clearly communicate the program's eligibility criteria. The five different visits and the related cash transfers should also be clearly addressed (hold up one of the All Babies cards when speaking)." New Incentives, Program Protocols 2018-2020, p. 36.

  • 45

    "Program Posters outside the entrance of treatment clinics and blue (distinctive) plastic bags handed to caregivers to increase visibility." New Incentives, Responses to 13-Feb-2020 Questions from GiveWell. (The bags are also used to help caregivers keep the CHCs and ABAE cards safe.)

  • 46

    "All Babies Card: Next Visit Date
    58. ACTION: On the All Babies card, make a check mark next to today's transfer. Add the next visit date in the following format. Example: "I I I I" for four weeks. Did you do this?
    a. Yes, I made a check mark next to today's transfer and added the next visit date in the number of weeks format
    b. No, I was not able to make a check mark or was not able to add the next visit date in the number of weeks format
    c. Not applicable (infant fully vaccinated)
    59. All Babies card: ACTION: Tell the beneficiary 'Today you got . . . Naira for . . . vaccines. In . . . weeks, on . . . date, you will get another . . . Naira.' Clearly show the card to the caretaker as you say this. (Explain the date with numbers of weeks. Example: 'You come back Wednesday four weeks from today. That is week 1, week 2, week 3 and in week 4 you come back on Friday.' Use your fingers to count the weeks and refer to the symbols on the card that indicate four weeks.) Did you do this?
    a. Yes, I explained this to the beneficiary while pointing to the All Babies card
    b. No, I was not able to explain this or was not able to point to the All Babies card
    c. Not applicable (infant fully vaccinated)" New Incentives, Program Protocols 2018-2020, p. 47 (questions 58-59).

  • 47

    "Program Posters outside the entrance of treatment clinics and blue (distinctive) plastic bags handed to caregivers to increase visibility" New Incentives, Responses to 13-Feb-2020 Questions from GiveWell.

  • 48

    "Conducted by visiting medium and large settlements with participating clinic staff and influential community leaders to address concerns related to non-compliance, vaccine suspicion, and to increase program awareness. Sometimes this is paired with a Targeted Outreach." GiveWell, Questions for New Incentives on program activities, May 18, 2020.

  • 49

    Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).

  • 50

    "Description: Conducted by holding an immunization day at a settlement, with participating clinic staff to address concerns related to distance or sometimes non-compliance. . . .
    Costs (Time and Monetary): Costs incurred are the same as those for a regular immunization day (transportation cost, one full FO day). Small fuel stipend of around N1,000-N2,000 is given to clinic staff, based on distance to outreach location and number of clinic staff." GiveWell, Questions for New Incentives on program activities, May 18, 2020, p. 4.

  • 51

    Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).

  • 52

    "Community Members Engagements Description: Small stipends are paid for activities including Defaulter Tracking, filtering out-of-catchment infants, identifying households and settlements with low uptake of immunizations, resolving identified uptake issues, attracting unimmunized infants, referrals, increasing program awareness, maintaining community support, and recently, for helping maintain COVID-19 adherence at clinics (like social distancing). The type of community member is selected based on the perception of the influence, recommendation by local leaders, and the value we get from their engagements. While most are selected for short periods of time to achieve specific goals (e.g. to increase defaulters in a settlement with reduced number of enrollments), some have longer standing engagements (e.g. identify out-of-catchment infants, helping maintain COVID-19 measures adherence at clinics). . . .
    Costs (Time and Monetary): Stipend of N500 is typically given for each community member engaged, with N1,000 given when multiple objectives need to be achieved or travel to additional settlements is needed. These are given during immunization days at the clinic. Around half of the recent contribution is at clinics for the filtering of out-of-catchment infants and for adherence of COVID-19 Measures." GiveWell, Questions for New Incentives on program activities, May 18, 2020, p. 3. A stipend of N500 is equivalent to $1.38; a stipend of N1,000 is equivalent to $2.76 ($1.38 x 2 = $2.76). See GiveWell, New Incentives program data, 2020, "Program basics" sheet, for conversion calculations.

  • 53

    "Town Criers Conducted by making announcements using loudspeakers at settlements, informing caregivers about immunization days and outreaches. This serves as a reminder and helps maintain program awareness. Sometimes, town criers can also be engaged for Defaulter Tracking, in addition to their typical responsibilities.
    Target Group: Caregivers, community members, village and traditional leaders.
    Reach: Estimate of 20 caregivers reached per engagement.
    Costs (Time and Monetary): N500 per week of engagement, given during immunization days at the clinic." GiveWell, Questions for New Incentives on program activities, May 18, 2020, p. 4. A stipend of N500 is equivalent to $1.38; see GiveWell, New Incentives program data, 2020, "Program basics" sheet, for conversion calculations.

  • 54
    • NI outlined their plans for awareness activities at scale. GiveWell, Questions for New Incentives on program activities, May 18, 2020.
    • Our understanding is that this is expected to increase the total number of caregivers reached through awareness activities. See GiveWell, New Incentives program data, 2020, "Awareness activities" sheet, for details of the changes New Incentives expects to make to the frequency of activities and the total number of caregivers reached by past and future activities.

  • 55
    • See GiveWell, New Incentives program data, 2020, "Awareness activities" sheet.
    • This data is based on information collected by FOs at the end of each disbursement. Choices are not mutually exclusive.
    • "IF NEW BENEFICIARY: ASK: Where did you first hear about the All Babies program? Select all
      that apply.
      a. Neighbor
      b. Friends
      c. Family
      d. Traditional birth attendant
      e. Town crier
      f. Village leader
      g. Religious leader
      h. Awareness card (small card from All Babies)
      i. UNICEF Voluntary Community Mobilizer (VCM)
      j. Community Health Worker
      k. Health worker at this clinic
      l. All Babies employee
      m. Other
      n. Unclear" New Incentives, Program Protocols 2018-2020, p. 45.

  • 56

    "Our focus is to ensure uninterrupted supply of these key vaccines: BCG, PENTA, PCV, OPV, Measles, and Yellow Fever (YF) to our 98 clinics in the 3 states." New Incentives, Supply-Side Workflow and Plan, p. 1.

  • 57

    "The various means of identification and those responsible are:

    • Review of the supply side dashboard (The SSOs are responsible for reviewing pages 1-2 while the SROs and SRM are responsible for reviewing pages 3-10)
    • Calls to ZCCO, SCCO, LCCO or officers in Apex facility over the course of the appropriate week since this alternated biweekly between calls to Apex clinics and calls to LCCO (The SSO is responsible for this role and will be held accountable for any calls not being made).
    • Review of the Field Check-in Dashboard. [The SRM (for reports from Katsina) and SROs (for cases from Jigawa and Zamfara) are responsible for this role and will be held accountable for unidentified or untransferred cases from these check-ins into the case logs).
    • Any significant case identified during review of the VVM and Expiry Review Sheet.
    • Information from other units like audit findings.
    • Informal or formal conversation during other discussion or at a meeting (This is adhoc and would be carried out mainly by the SRO, GRM and FD)."

    New Incentives, Supply-Side Action Protocol, p. 2.

  • 58

    Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).

  • 59

    This includes vaccine stockouts (i.e., vaccines were not present at the beginning of the immunization day) and runouts (i.e., vaccines were exhausted during immunization day); expiring and expired vaccines; and vaccine vial monitors (VVMs, sensors which measure whether a vial has been exposed to excessive heat). Data is collected on the "ClinicDaily" form that is filled in during each immunization day. See New Incentives, Clinic Daily form, pp. 2-8.

  • 60

    "The 'cold chain' is a system of storing and transporting vaccines at recommended temperatures from the point of manufacture to the point of use. That is, the role of the cold chain is to maintain the potency of vaccines." World Health Organization, "WHO African Region: Ethiopia, EPI logistics".

  • 61

    "Calls to ZCCO, SCCO, LCCO or officers in Apex facility over the course of the appropriate week since this alternated biweekly between calls to Apex clinics and calls to LCCO (The SSO is responsible for this role and will be held accountable for any calls not being made)." New Incentives, Supply-Side Action Protocol, p. 2.

  • 62

    This is based on our review of the New Incentives, Stakeholder Relations & Supply-side Case Log (unpublished), "LGA or Apex Clinic level Case Log" sheet, Columns C and E. Most cases logged are "Stockouts," "Runouts," "VM1A or 1B for Previous Month Uncollected from clinics," or "Antigen not enough for the next 1 or 2 weeks."

  • 63

    Examples of this include (from New Incentives, Stakeholder Relations & Supply-side Case Log (unpublished), "LGA or Apex Clinic level Case Log" sheet): "SSO communicate with Clinic staff/SCCO/LCCO on supply-side"; "SCO urged the LCCO to follow up with the SCCO so that the vaccines are made available"; and "SRO discussed stockout issues with SCCO, LCCO and FM. The SCCO has promised to increase the allocation for Kwanda PHC to avoid consistent stockouts meanwhile the FM asked the LCCO to go to the LGA and get the antigens tomorrow."

  • 64

    Examples of actions logged include:

    • "Supporting requests from LGAs for transportation to pick vaccine from the State Cold Store; Reason or Issue: LCCO serving our clinics occasionally do not have the funds to go and pick up vaccines causing widespread stockout; SMART Objective: To avert supply-side issues due to inability of LCCO to make the trip to State to topup vaccine stock." New Incentives, ABAE Work Plan (unpublished), "Stakeholder Relations" sheet, cell E10:G10.
    • "Support for Special Request from the States - Logistic Support for 34 LGA CCOs to Collect Vaccines from State Store. Reason or Issue: States make request to partners and to ensure a predictable expenditure, a limit was set and a quarterly frequency was agreed. SMART Objective: To sustain the cordial relationship existing with the States while maintaining our obligation as a partner organization." New Incentives, ABAE Work Plan (unpublished), "Stakeholder Relations" sheet, cell E193:G193.
    • "What happens, if clinic staff reports vaccine shortages the day before immunisation days? Does NI take any steps to address the shortage before the following day?
      Every morning Stakeholder Relations Officers (SROs) check prior-to-immunization-day submissions made by FMs through FM Check-in. If it indicates a potential vaccine Stockout or Runout, SROs initiate follow-ups in the following sequence:
      • SRO calls FO to understand the issue, followed by a call to the Clinic In-Charge or RI Focal Person to identify the source of the issue (usually either apex clinic or LGA cold store, sometimes State, Zonal, or National). For clinics without cold storage (majority), SROs call the apex clinic officer, while for clinics with cold storage (apex clinics) the SROs call the LCCO (Local Government Cold Chain Officer).
      • Once the source of the issue is diagnosed, action is taken to avoid the vaccine Stockout or Runout. An example of action taken is to advise LCCO to pick up vaccines from the State store, occasionally providing transportation support for the LCCO to restock from the State store. In many cases, the prior-to-immunization-day conversation between the clinic staff and FO and between the FO and FM prompts the FO to resolve these cases by calling the LCCO, which is part of why the SRO calls the FO first. Action taken by FM or FO is covered in the column for 'What has the FM or FO done?' in Clinic level Case Log, while action by the SRSS team is primarily recorded in 'What did the SRD, GRM, SRO or / and SCO do?' column. SCO is an acronym for Supply Chain Officers (LCCOs and SCCOs)."
        GiveWell, Questions for New Incentives on program activities, May 18, 2020, p. 1.

  • 65

    New Incentives, Program units and responsibilities (unpublished):

    • "Logistics and Field Officers (120)." Slide 4.
    • "Enrolling beneficiaries into the program based on strict eligibility criteria
      Detailed reporting on enrollments and diligent cash management." Slide 10.
    • "Reporting of supply-side issues through FM Check-in; same-day submission of Clinic Daily
      Maintaining cordial relationship with clinic staff, community leaders, Imams, TBAs, LLOs, and other stakeholders that are part of the clinic's catchment area
      Communicating expected supply-side requirements to clinic staff." Slide 23.

  • 66

    New Incentives, Program units and responsibilities (unpublished):

    • "Field Managers (10)." Slide 4.
    • "Managing the performance of each clinic under supervision." Slide 10.
    • "Field Manager (FM)
      Ensuring that FOs are meeting their responsibilities . . .
      Identifying and prioritizing core issues at clinics and settlements . . .
      assessing the quality of completed activities and budget expenditure." Slide 17.

  • 67

    New Incentives, Program units and responsibilities (unpublished):

    • "State Field Managers (3)." Slide 4.
    • "Senior Field Manager Ensuring that FMs and FOs are meeting responsibilities by conducting in-person supervision visits, reviewing submissions and providing coaching
      Assessing status of Units Objectives and KPIs
      Developing and managing State Budget Requests based on KPIs . . .
      Ensuring Clinic Schedule and Settlement List are maintained and up-to-date." Slide 11.

  • 68
    • "Could you share the number of people who work in the Stakeholder Relations and Supply-Side unit, disaggregated by role?
      There are 7 people in the SRSS Unit.
      National Coordinator (NC -1): Manages Operations, Security, and SRSS Units, and manages stakeholder relations and resolves supply-side issues at the National level.
      Stakeholder Relations Director (SRD -1): Manages SRSS unit, manages stakeholder relations and resolves supply-side issues at the Zonal and State levels, and intervenes when issues are beyond the resolution of the SRM or the SROs.
      Stakeholder Relations Manager (SRM - 1): Supervises Supply-Side Officers (SSOs), manages stakeholder relations and resolves supply-side issues at the LGA and State levels in Katsina State.
      Stakeholder Relations Officer (SRO - 2): Manage stakeholder relations and resolve supply-side issues at the LGA and State levels in Zamfara and Jigawa States.
      Supply-Side Officers (SSO - 2): Review dashboards, transfer cases to the SRSS Case Log, call the apex and LGA stakeholders intermittently, coordinate communication to resolve each case and escalate to SROs and SRM for resolution when needed.
      With this structure, the SRSS Unit is able to have physical presence necessary for relationships in each important geography (Abuja, the capital, and each of the program states), while covering Supply-Side efforts in a majority of the LGAs in each of the states." GiveWell, Questions for New Incentives on program activities, May 18, 2020, pp. 7-8.
    • "Security Focal Point (SFP) . . . Following-up and assessing validity of information gathered with other sources." New Incentives, Program units and responsibilities (unpublished), slide 35.
    • "Security Manager . . . Overseeing Security Unit objectives and responsibilities." New Incentives, Program units and responsibilities (unpublished), slide 35.
    • "Software engineer." New Incentives, Program units and responsibilities (unpublished), slide 4.

  • 69

    "Console Manager, Console Supervisors (4), Console Agents (21), Auditors (3)." New Incentives, Program units and responsibilities (unpublished), slide 4.

  • 70

    See GiveWell, New Incentives program data, 2020, "GW elaboration of NI operating expenses" sheet, for details.

  • 71

    See GiveWell, Questions for New Incentives on program activities, May 18, 2020, pp. 9-11.

  • 72

    "[GiveWell:] My understanding is that immunization Rates in the Progress Dashboard refer to the number of children who received BCG who went on to receive the other vaccines, and are calculated on the basis of disbursements data, and ABAE ID information. Is this correct?
    [New Incentives:] The Immunization Rates in the Progress Dashboard are based on the number of children enrolled who went on to receive the other vaccines. The rest of the understanding is correct." GiveWell, Questions for New Incentives about monitoring and evaluation, p. 6.

  • 73

    "ACTION: Take a photo of the left side of the Child Health Card and SAY: 'I am registering you with this photo.' (Ensure that the All Babies ID and all caretaker data is visible.) Will you do this with the next field?" New Incentives, Program Protocols 2018-2020, p. 48.
    

  • 74

    Results are available here: GiveWell, New Incentives monitoring and evaluation 2017-2020 and plans for future evaluation, "Results" sheet. More details about the strength of this evidence here.

  • 75

    This hypothesis is supported by the fact that, after reviewing data collected during the RCT, we could not explain roughly 17% of enrollments reported by New Incentives on the basis of estimates of the number of infants living in and around clinic catchment areas. See GiveWell, Analysis of enrollment discrepancy, 2020, "% BCG disbursements not explained by OOC_per catch" sheet, for calculations.

  • 76
    • "23. IF NEW BENEFICIARY: ANSWER: Does the infant have an old BCG scar? Personally verify on the upper left and right arm of the child. Does the infant have an old BCG scar and which arms did you check?
      a. Yes, there is an old BCG scar and I checked both of the infant's arms for the old BCG scar
      → SKIP to close without saving
      b. Yes, there is an old BCG scar but the infant is new and has required evidence
      c. No, there is no old BCG scar and I checked ONE of the infant's arms for the old BCG scar
      d. No, there is no old BCG scar and I checked BOTH of the infant's arms for the old BCG
      scar
      24. IF NEW BENEFICIARY: ANSWER: Does the infant have a fresh BCG injection mark?
      a. No, infant is NOT eligible to enroll → SKIP to close without saving
      b. No, but infant is new and there is required documentation
      c. Yes, infant is eligible to enroll
      d. Unclear, I can't tell if the infant has a fresh BCG injection mark (uncommon response!)" New Incentives, Program Protocols 2018-2020, p. 43.
    • See GiveWell, New Incentives monitoring and evaluation 2017-2020 and plans for future evaluation, "BCG scarring: methods" sheet, for more details on our assessment of the evidence.

  • 77

    "However scar formation is not a marker for protection and approximately 10% of vaccine recipients do not develop a scar." World Health Organization, BCG vaccines: WHO position paper – February 2018, p. 84.

  • 78

    In June 2020, New Incentives started collecting data on the BCG scars of infants returning after the BCG vaccine. We have seen data covering roughly four months (June to October 2020). This data indicated that roughly 97% of infants had one BCG scar, roughly 3% had no BCG scar, and a small percentage (0.04%) had two or more scars. See GiveWell, New Incentives monitoring and evaluation 2017-2020 and plans for future evaluation, "Results" sheet, for details.

  • 79

    See GiveWell, New Incentives cost per infant immunized [February 2022], "Per-year analysis" sheet, for calculations.

  • 80

    "We anticipate having a good understanding of our program coverage when expanding to LGAs by using VTS population estimates, structuring immunization days and protocols to effectively limit infants from outside the LGA, and running rapid surveys to calibrate internal coverage estimates. . . . Added quarterly LGA review meetings which will be used to review coverage performance, vaccination training and best practices, supply-side issues, and controls for fraud and out-of-catchment infants." Svetha Janumpalli, Founder and CEO, New Incentives, email to GiveWell, June 24, 2020 (unpublished)

  • 81

    New Incentives reports that biometric identification methods for infants are not yet developed enough to be used in the program. New Incentives expects caregiver biometric identification to be an improvement over the current system, but it will still be an imperfect identification process, since New Incentives ultimately aims to identify the infant (rather than the caregiver), the same caregiver might accompany multiple infants, and the same infant might be accompanied by different caregivers. Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversation with GiveWell, June 16, 2020 (unpublished).

  • 82

    Note that neither of these might be the case if, for instance, an increase in the number of stockouts, runouts, and/or infants not served were the result of increased demand. We are unsure if or how we will be able to confidently disentangle these effects.

  • 83

    See GiveWell, New Incentives monitoring and evaluation 2017-2020 and plans for future evaluation, "Results" sheet, for details.

  • 84

    See GiveWell, New Incentives monitoring and evaluation 2017-2020 and plans for future evaluation, "Supply side: methods" sheet.

  • 85

    More details about how this data is collected can be found here: GiveWell, New Incentives monitoring and evaluation 2017-2020 and plans for future evaluation, "Supply side: methods" sheet.

  • 86
    • See GiveWell, New Incentives monitoring and evaluation 2017-2020 and plans for future evaluationd, "Transport costs: methods" sheet, for more details.
    • Note that New Incentives' new model (in which clinics will be located closer to each other ) might change transportation costs for program participants. We therefore plan to track transportation cost data under the new model over time rather than directly comparing transportation costs under the new model with transportation costs reported during the RCT.

  • 87

    We have spot-checked about 20 pictures from New Incentives, Cash Management Dashboard (unpublished). See GiveWell, New Incentives monitoring and evaluation 2017-2020 and plans for future evaluation, "Fraud deterrence and mitigation" sheet, for details.

  • 88

    See GiveWell, New Incentives monitoring and evaluation 2017-2020 and plans for future evaluation, "Disbursement fraud_CCT survey: methods" sheet, for details.
    

  • 89

    Caregivers who report dashing someone are asked to categorize the recipient as one of the following: "Clinic Staff," "All Babies Staff," "Security Guard," "UNICEF VCM," "Community Health Worker," "Government Staff," or "Other." Of those, 6.6% report giving tips to clinic staff and 0.6% to New Incentives staff. See GiveWell, New Incentives monitoring and evaluation 2017-2020 and plans for future evaluation, "Results" sheet, for more information.

    We see this data as weak evidence on whether caregivers are giving some of the CCT to clinic staff or others; it is likely weakest evidence for whether New Incentives FO are capturing part of the CCT because FOs are the ones asking the question.

    For more details about the strength of this evidence, see GiveWell, New Incentives monitoring and evaluation 2017-2020 and plans for future evaluation, "Disbursement fraud_CCT survey: methods" sheet.

  • 90

    See GiveWell, New Incentives monitoring and evaluation 2017-2020 and plans for future evaluation, "Results" sheet, for the survey results and GiveWell, New Incentives monitoring and evaluation 2017-2020 and plans for future evaluation, "Disbursement fraud_audit: methods" sheet, for more information on how the data was collected.

  • 91
    • "Children who are HIV-infected when vaccinated with BCG at birth are at increased risk of developing disseminated BCG disease. However, if HIV-infected individuals, including children, are receiving ART, are clinically well and immunologically stable (CD4% >25% for children aged <5 years or CD4 count ≥200 if aged >5 years) they should be vaccinated with BCG. In general, populations with high prevalence of HIV infection also have the greatest burden of TB; in such populations the benefits of potentially preventing severe TB through vaccination at birth are outweighed by the risks associated with the use of BCG vaccine. Therefore, it is recommended that in such populations:
      • Neonates born to women of unknown HIV status should be vaccinated as the benefits of BCG vaccination outweigh the risks.
      • Neonates of unknown HIV status born to HIV infected women should be vaccinated if they have no clinical evidence suggestive of HIV infection, regardless of whether the mother is receiving ART.
      • Although evidence is limited, for neonates with HIV infection confirmed by early virological testing, BCG vaccination should be delayed until ART has been started and the infant confirmed to be immunologically stable (CD4 >25%)."

      World Health Organization, BCG vaccines: WHO position paper – February 2018, p. 95.

    • Adverse events linked to BCG vaccination range from mild, localized complications to more serious, systemic or disseminated BCG disease in which M. bovis BCG is confirmed in one or more anatomical sites far from both the site of injection and regional lymph nodes. Disseminated BCG disease is associated with a case-fatality rate of >70% in infants. By comparison, the background mortality rate among South African HIV-infected infants is 12.2 per 100 person-years (95% CI: 8.2–17.4).
      Systemic or disseminated BCG disease may be clinically indistinguishable from tuberculosis and can only be confirmed through positive mycobacterial culture species identification, preferably by polymerase chain reaction (PCR) for the RD1 genetic region that is lost during attenuation of BCG" Hesseling et al. 2009.
    • "Disseminated BCG disease is associated with a case-fatality rate of >70% in infants. . . . [The risk of disseminated BCG disease] has been shown to be 1100 to 4170 per 1 million in HIV-infected infants routinely vaccinated at birth." Hesseling et al. 2009.

  • 92

    "NIGERIA PREVALENCE RATE: Zamfara 0.5%, Jigawa 0.3%, Katsina 0.3%" National Agency for the Control of AIDS, "Nigeria Prevalence Rate".

  • 93
    • "The risk of a reaction at the injection site following certain injected vaccines, such as DTaP (diphtheria, tetanus, and pertussis) or pneumococcal vaccine, increases if the doses are not separated by the recommended amounts of time. In these cases, it is the spacing of the doses, not the number of doses, that creates the risk. These reactions can be unpleasant, but they are not life-threatening." Centers for Disease Control, "Ask CDC - Vaccines & Immunizations".
    • "We searched VAERS for US reports where an excess dose of vaccine was administered to a person received from 1/1/2007 through 1/26/2018. . . .
      More than three-fourths of reports of an excess dose of vaccine did not describe an AHE [adverse health effect]. Among reports where an AHE event was reported, we did not observe any unexpected conditions or clustering of AEs [adverse events]." Moro et al. 2019, abstract.

  • 94
    • The US Department of State reports that violent crime is common throughout Nigeria. Within the country, the states in which New Incentives currently works (Katsina, Zamfara and Jigawa) are not among those flagged as "higher-risk." "Violent crime—such as armed robbery, assault, carjacking, kidnapping, and rape—is common throughout the country. Borno and Yobe states and Northern Adamawa State – Do Not Travel . . . . Adamawa, Bauchi, Borno, Gombe, Kaduna, Kano, and Yobe states – Do Not Travel . . . . Coastal areas of Akwa Ibom, Bayelsa, Cross Rivers, Delta, and Rivers states (with the exception of Port Harcourt) – Do Not Travel." US Department of State, Bureau of Consular Affairs, "Nigeria Travel Advisory," 2020.
    • As of October 2020, New Incentives considered 20% of their partner clinics’ catchment areas to be "high risk." This means one or more cases in which armed violence resulting in death has been recorded near the clinic, but New Incentives judged it to be safe to travel in the area during the day. The assessment was made by New Incentives Security Manager on the basis of information collected from a variety of sources, including FO reports. See the main text below for more information about this process. New Incentives, Clinic and settlement security assessments (unpublished).

  • 95
  • "As of 17-June-2020, there have been 23 incidents in total. The cases include those related to minor theft (e.g. phone and sometimes cash), minor injuries (e.g. bruises and scorpion bites), and encounters with bandits." GiveWell, Questions for New Incentives about potential negative and offsetting effects, p. 1.

  • 96

    New Incentives, Security Procedures and Status (unpublished), "Incidents Involving Staff" sheet:

    • Incidents resulting in death:
      • "Year: 2017; Week: 47; A field staff in [redacted] . . . was involved in a road traffic accident (RTA) as he was traveling on a non-work day from [redacted] after visiting his relative in a hospital there. On his way back from the hospital, his vehicle got into an accident with a lorry, which put him in a critical condition. He was rushed to the hospital but unfortunately he passed away on 22-Nov."
      • "Year: 2018; Week: 36. On Tuesday 4/9/18 at about 1:00am armed bandit attacked [redacted], killed two people and kidnapped three people including two children of the district head. Reportedly, the bandit was specifically asking for [routine immunization (RI)] incharge of the clinic, saying that they were informed that the RI incharge and our staff (FV) [Field Volunteer] goes around some of their settlement and sometime in the clinic disbursing cash to some women."
    • Incidents resulting in injuries:
      • "Year: 2018; Week: 29. One of our FVs . . . was involved in a motor accident on his way back from the outreach session. The tyre of the vehicle he boarded bursted and the driver lost control consequently hitting a roadside tree. [The FV] sustained an injury to the shoulder. He is stable presently and will seek further medical attention."
      • "Year: 2019; Week: 35. At [redacted], about 4 men armed with dangerous weapons attempted to burgle and rob an NI/ABAE staff residence in [redacted]. The armed youths took about 30 minutes attempting to break the barriers (doors and windows) and attacked neighbours who had responded to the distress calls of the residents of the apartment inflicting various degrees of injuries on the first responders. Upon arrival of security forces, the attackers withdrew from the residence and proceeded to raid at least two nearby houses where they robbed residents of cash and personal effects. Subsequently, members of the community mobilized in their numbers, confronted the robbers causing them to withdraw from the location."
      • "Year: 2019; Week: 46. Two of our FOs including the clinic staff assigned (VCHEW) had an accident today on their way back from the outreach session and sustained slight bruises, and in the process [name redacted] broke his phone and right now he is unable to send his data and do the end process for today."
      • "Year: 2020; Week: 4. Staff reported; 'On their way back from [redacted], they were involved in an accident. He said on their way back they were involved in an accident together with the RI staff, falling on a stone and sustaining injuries and a crack to his phone screen; though phone still remains functional."
      • "Year: 2020; Week: 25. FO was involved in a RTA. Someone with a car hit FO on a commercial bike and as a result FO's office phone got lost. FO sustained a minor injury on his right leg and was treated in a hospital. The person responsible for the accident has gone ahead to replace FOs phone."

  • 97

    FOs collect information on a rolling basis through drivers (while traveling to the clinic), town criers, and community members (during outreach sessions). When a clinic is marked as "high risk," FOs reach out to clinic staff and community members (for outreach sessions) the day before an immunization day to collect information about security threats. If the FO determines there is reason for concern, they have the option of not joining the outreach session. Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).

  • 98
    • The Security Manager collects information from security industry networks and media on a rolling basis. The Security Focal Point makes calls to local government authorities every two weeks to collect information about security threats.
      Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).
    • "SM and SFP: Verifies all security reports received through myDay to ensure that these are true incident.
      Verifications entails
      - Reach out to our key sources of information (community stakeholders, security forces, security networks, staff, sister NGO security units)
      - Triangulation of information from more than one source
      - Consideration of the authenticity of each source."
      New Incentives, Security Procedures and Status (unpublished), "Security Procedures and Status" sheet, cell C26

  • 99

    New Incentives, Security incidents dashboard (unpublished).

  • 100

    Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).

  • 101
    • "The security incidents are compiled into reports sent internally every week" New Incentives, Overview of NI-ABAE Anti-Bribery and Security Policies, p. 2.
    • "SFP: To collate and send out all security notifications to NC, SRD, SFMs, SM, COO, and external members" New Incentives, Security Procedures and Status (unpublished), "Security Procedures and Status" sheet, cell C14.
    • "SM: Reviews the Security Incident Review sheet to see if himself, FMs, SFMs or NC has taken action for cases where an action is required. This is a follow up of his responsibility to call the attention of the Manager concerned to a recommended action." New Incentives, Security Procedures and Status (unpublished), "Security Procedures and Status" sheet, cell C11.
    • "FM: Will discuss each FOs travel to the clinics or settlements based on highlighted Risk Rating for Disbursement Day or Field Activity, and submit this as part of the FM Check-in. FOs should be able to report if they are uncomfortable to travel based on the Security Assessment and Recommendations and recommend for activities to be cancelled." New Incentives, Security Procedures and Status (unpublished), "Security Procedures and Status" sheet, cell C13.

  • 102
    • Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).
    • "All staff: anybody who uses the app, must acknowledge and adhere to these measures
      System: Displays Travel Safety Measures from Clinic and Settlement Security Assessments. Will not allow expenses or in-person activities to get logged against No Go clinics and settlements." New Incentives, Security Procedures and Status (unpublished), "Security Procedures and Status" sheet, cell C18.

  • 103

    New Incentives, Security Procedures and Status (unpublished), Week 18-22, 2020.

  • 104

    "All Staff:
    1. Completion of UN B-SAFE and certificates in Zoho
    2. Signed CSP + Quiz + Statement
    3. Companion Card Received."
    New Incentives, Security Procedures and Status (unpublished), "Security Procedures and Status" sheet, cell C19.

  • 105

    New Incentives training includes (among other topics):

    • UN Be Safe training: how to prepare for travel, emergency communications, and how to respond to violence
    • Training on the Country Security Plan, including standard operating procedures on road safety/travel, cash management, abduction, political, ethnic and religious instability, bad governance, and health risks.
    • Road safety training, including control measures to prevent life-threatening accidents, such as using seatbelts and enforcing speed limits.
    • Abduction and kidnapping, including procedures to follow in cases of suspected abduction.

    Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).

  • 106

    New Incentives, Security Procedures and Status (unpublished), Week 18-22, 2020.

  • 107

    See GiveWell, New Incentives monitoring and evaluation 2017-2020 and plans for future evaluation, "Results" sheet, and GiveWell, New Incentives monitoring and evaluation 2017-2020 and plans for future evaluation, "Supply side: methods" sheet.

  • 108

    Our main concern about this estimate is that it is based on a small sample size (it includes 0.3% of caregivers served to date). (See GiveWell, New Incentives monitoring and evaluation 2017-2020 and plans for future evaluation, "Negative and offsetting effects" tab, for more details).

    New Incentives also incentivizes caregivers who have not been served at the end of the day to return by providing them with numbered tags that allow them to be served first during the following immunization day: "Advantages of Numbered Tags . . . Beneficiaries who came in on a previous immunization day already get prioritized so that they get to return home sooner, it also increases their confidence that they will get the disbursement, if eligible." New Incentives, Program Protocols 2018-2020, p. 39.

  • 109

    “A concern . . . stakeholders (including UNICEF representatives) have shared with us is about incentives replacing intrinsic motivations to immunize infants in the absence of incentives. This was experienced by WHO when some women in Northern Nigeria started refusing vaccinations if they didn't get the in-kind donations that were previously being offered during Polio campaigns.” New Incentives, Responses to questions from GiveWell, September 23, 2020 (unpublished).

  • 110

    New Incentives reports using awareness-raising activities to educate caregivers about misconceptions about vaccinations, concerns about side-effects, and the need for immunization. Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).

    New Incentives also told us it believes incentives play a different role in its program than they did in the polio vaccination campaign, since campaigns provide vaccinations door-to-door (and thus create limited costs for caregivers), while New Incentives mostly incentivizes vaccinations in clinics (which have higher time and transport costs for caregivers).

    “Note: we have received conflicting reports and opinions regarding in-kind incentives during Polio campaigns. CCTs for Routine immunization are different from offering in-kind incentives or ‘pluses’ during Polio campaigns where caretakers do not bear any cost to vaccinate their children with an oral vaccine at their home. We take concerns around intrinsic motivation and perverse incentives very seriously. We do not think these are unique to the application of CCTs to immunizations as similar concerns exist for application of CCTs to other health and non-health behaviors. We have incorporated messaging in various components of the program to help caretakers understand that vaccinations are important to protect their children against deadly diseases.” New Incentives, Responses to questions from GiveWell, September 23, 2020 (unpublished).

  • 111

    New Incentives, Incidents involving staff (unpublished):

    • "Year: 2017, Week: 49. On 10-Oct, staff reported the following: According to him . . . he carried motorcyclist to convey him to [redacted], about 3 km from colony then the motorcyclist said he want to ease himself, [he] agreed and the man stopped, immediately he stopped 2 men came out from the farm one with cutlass and the other with pistol, they asked him to take a small foot path which he . . . complied, as they went in they met two other guys each with cutlass and axe, he . . . saw the motorcyclist parked his own bike and joined them, they asked for money and handset which he . . . gave them, they searched all compartments of his back bag they could not find any thing because all the money is in his waist bag except transport money."
    • "Year: 2018, Week: 28. On 10-July, staff reported that 'Thieves broke into his home and Stole cash amounting to N71000 and his big Npower tab phone with 2 chargers and power bank. Staff also reported incident to the police."
    • "Year: 2019; Week: 6. This evening at about 8 pm I received a call from my FV . . . reporting that his room was buggled at [redacted] and all his valuable was stolen including the organization's phone."
    • "Year: 2019; Week: 20. On Sunday 12/05/19, at about 9:30 pm, FM got a call from [FO] who called and narrated how his phone was stolen as he went for evening prayers after breaking his fast. He mentioned that he had dropped his work phone inside his room to charge it, as the battery of the phone was very low. But when he returned from the mosque, he noticed that the phone has been stolen."
    • "Year: 2019; Week: 23. It was reported that today 06/06/19 at around 1:00 pm, armed bandit attacked 2 of our staff on their way back home from the . . . clinic. According to the staffs the attack took place at [redacted] immediately after crossing the river they met 2 bandits with 2 AK 47 rifles waiting for their arrival. After several questions from the bandit they said they are suspecting the bike men are informants, so they are going to hurt them, but finally they collected 1 office smartphone, 1 personal phone, N62,000 for Disbursement, and N13,000 for Transport from [redacted], and they collected N69,000 for Disbursement N13,000 only for the Transport, from [redacted], the incident took place about 4km to . . . clinic."
    • "Year: 2020; Week: 10. On 10-Mar along [redacted] road, it was reported that someone stole FOs phone on his way from [redacted] to [redacted] in public car. FO reported that he had the phone conveniently placed in his pocket during the trip only to discover when he alighted from the vehicle that the phone was no longer there. Despite making several efforts in retrieving the phone by going to the park to lodge same complain, it proved abortive."
    • "Year: 2020; Week: 16. Staff reported that: At [redacted], at night before sleeping, he plugged his phone to charge around 12:30 am as he always do, only to wake up in the morning around 6:43 am and could not find the phone. It is evident the phone must have been stolen because the door was not properly locked. Amongst the items stolen are; 1 NI/ABAE Phone and 2 other phones , belonging to his mother and brother."
    • "Year: 2020; Week: 20. At [redacted], staff reported that her home was burgled around 0300hrs on 14-May and valuables including her personal phone, generator set and twenty five thousand naira (25,000) belonging to NI/ABAE was stolen. She further reported that the thieves attempted to make away with her car when they were awoken by the car alarm. At the instance of the alarm, the thieves allegedly withdrew from the scene."
    • "Year: 2020; Week: 20. At [redacted], Staff went to the Bank to request for a Bank statement as requested by HR during one of his non-clinic operational days. Upon coming out from the bank, he realized his bike has been stolen. In a secret compartment of the bike is a pass issued to the staff to allow access during COVID-19 lockdown."

  • 112

    "[New Incentives] We believe that this is infrequent since the bandits rely on the community for support. Also, it is unlikely that bandits would target female caregivers traveling to the clinic who are receiving small amounts like N500 and N2,000. While we ask questions regarding caregivers losing part of their cash transfers due to bribes or 'dashes' during each disbursement, we do not specifically ask whether caregivers have experienced theft by third parties." GiveWell, Questions for New Incentives about potential negative and offsetting effects, p. 1.

  • 113

    New Incentives reports that there are cases in which this might have occurred: "While we don't have data, below are the cases where we anticipate that supply-side efforts could have led to shortages at other clinics in the state:
    Sometimes if there is a stockout at the LGA store, our clinic might request vaccines from a nearby clinic. These requests can be refused if the neighboring clinic does not think they have adequate additional stock to spare, we have seen many cases of refusals from nearby clinics. There are at least 87 cases recorded where we attempted to borrow vaccines from a nearby clinic (these cases can be found by searching for 'borrow' in the Clinic level Case Log).
    The consumption increase due to our clinics could cause a short-term shortage in the LGA (we have records of this happening in one of the LGAs in Zamfara with 5 of our partner clinics)." GiveWell, Questions for New Incentives about potential negative and offsetting effects, p. 1.

  • 114
    • See above for details on New Incentives supply support activities.
    • New Incentives reports positive feedback on its work on supply activities at the LGA level: "We have received acknowledgements from State stakeholders for the utilization reports at New Incentives partner clinics as well as improvements in documentation at the LGA through our Fortnightly calls." GiveWell, Questions for New Incentives about potential negative and offsetting effects, p. 2.

  • 115
    • "Question 1: Do you know whether Katsina's decision not to include immunization in its conditions was influenced by the existence of the New Incentives program?
      Response to Question 1: We are unsure to what extent our program has influenced the choice of Katsina State. Our last discussion with the Social Safety Net Office in Katsina State on this matter revealed that Katsina State prioritizes various sector including health and education but feels that health, unlike education, has been receiving lots of interventions by both Government and NGOs. The government official we spoke to gave examples of the government's Saving One Million Lives Program for Result (SOML-PforR) which cuts across Maternal, Newborn and Child Health (MNCH), the New Incentives CCTs for Routine Immunizations program, and Save the Children's nutrition program. An example of Katsina State government's investments in health is the State Emergency Routine Immunization Coordination Centre (SERICC); Katsina State is one of the 18 States in Nigeria that have the equivalent state setup of the National Emergency Routine Immunization Coordination Centre (NERICC). The purpose of these coordination centres is to utilize a war-room like approach of allocating more resources to closely monitor trends, identify issues and solutions, and implement them to improve routine immunization coverage." Dr. Obinna Ebirim, National Coordinator, New Incentives, email to GiveWell, June 12, 2019 (unpublished).
    • "Katsina State has been implementing the base unconditional cash transfer and has recently selected girl-child education as their conditionality. . . . Zamfara State is yet to choose a conditionality." Dr. Obinna Ebirim, National Coordinator, New Incentives, email to GiveWell, May 27, 2019 (unpublished).
    • "Based on our latest understanding, Zamfara is still at the base phase and yet to get to the top-up phase where the conditionality is chosen by the state." GiveWell, Questions for New Incentives about potential negative and offsetting effects, p. 2.
    • “Jigawa State chose health as their conditionality for the top up conditional cash transfer and got above average in the supply-side evaluation and re-evaluation.” Dr. Obinna Ebirim, National Coordinator, New Incentives, email to GiveWell, May 27, 2019 (unpublished).

  • 116

    Dr. Obinna Ebirim, National Coordinator, New Incentives, email to GiveWell, June 12, 2019 (unpublished):

    • "Question 1: Do you know whether Katsina's decision not to include immunization in its conditions was influenced by the existence of the New Incentives program?
      Response to Question 1: We are unsure to what extent our program has influenced the choice of Katsina State. Our last discussion with the Social Safety Net Office in Katsina State on this matter revealed that Katsina State prioritizes various sector including health and education but feels that health, unlike education, has been receiving lots of interventions by both Government and NGOs. The government official we spoke to gave examples of the government's Saving One Million Lives Program for Result (SOML-PforR) which cuts across Maternal, Newborn and Child Health (MNCH), the New Incentives CCTs for Routine Immunizations program, and Save the Children's nutrition program. An example of Katsina State government's investments in health is the State Emergency Routine Immunization Coordination Centre (SERICC); Katsina State is one of the 18 States in Nigeria that have the equivalent state setup of the National Emergency Routine Immunization Coordination Centre (NERICC). The purpose of these coordination centres is to utilize a war-room like approach of allocating more resources to closely monitor trends, identify issues and solutions, and implement them to improve routine immunization coverage."
    • "Comment 1: In 2019-2023, the World Bank CCT program is being operated as an experiment and is only expected to reach a small portion of the population (<1% of the population for the conditional transfer; plus Katsina State is not including immunization in its conditions and we don't yet know whether Zamfara State will)
      Response to Comment 1: The World Bank CCT program is being operated as a program. The reason why the proportion of the entire population of the State that is benefiting is small (<1%) is because the target of the program is not the entire population but the poorest of the poor that are in the National Social Register."

  • 117

    See GiveWell, New Incentives cost per infant immunized [February 2022] (public), "Summary" sheet.

  • 118

    We estimate that a donation of $100,000 to New Incentives would fully vaccinate the equivalent of 1,123 infants who wouldn't have been vaccinated otherwise. $100,000 / 1,123 = ~$89. See our calculations in the "Number of vaccinations supported with a hypothetical donation" section of our cost-effectiveness analysis for New Incentives.

  • 119

    We estimate that a donation of $100,000 to New Incentives would lead to a total of $200,991 spent by all actors (New Incentives, Gavi, and the Nigerian government) as a result of the program and would fully vaccinate the equivalent of 1,123 infants who wouldn't have been vaccinated otherwise. $200,991 / 1,123 = ~$179. See our calculations in the "Number of vaccinations supported with a hypothetical donation" section of our cost-effectiveness analysis for New Incentives.

  • 120

    We estimate the cost per child fully immunized (excluding costs covered by New Incentives) in Nigeria on the basis of data reported by WHO on vaccine coverage rates and total spending on routine vaccination by year. See GiveWell, New Incentives cost per infant immunized [February 2022], "Government and Gavi costs" tab, "Average of annual coverage rates by vaccine in Nigeria, 2014-2018," "Expenditure on vaccines and size of targeted populations in Nigeria, 2014-2018" and "Average cost per child fully immunized, 2014-2018" sections. We rely on estimates from 2014 to 2018 because those are the only recent years for which we have seen routine vaccine expenditure data from WHO. See "Source: WHO gov expenditure routine immunization" sheet and "Source: WHO total expenditure routine immunization" tabs.

    We are aware of a few limitations of this estimate:

    • We have not carefully vetted the sources WHO relies on for its estimates of vaccine coverage or spending on routine immunizations for Nigeria.
    • The spending totals WHO reports appear to be for routine immunization only, but the coverage estimates may include immunization through non-routine programs (e.g., immunization campaigns).
      • See "Source: WHO total expenditure routine immunization" and "Source: WHO gov expenditure routine immunization" tabs.
      • Our understanding is that the coverage and spending estimates reported by WHO refer to average costs per child fully immunized, but we would ideally like to estimate the costs per additional child fully immunized due to New Incentives' program, which may differ from average costs.

    After estimating the cost per child fully immunized (excluding costs covered by New Incentives), we estimate the proportion of that cost covered by the Nigerian government and the proportion covered by Gavi. We rely on two sources of information to estimate those proportions:

    • Data reported by WHO on Nigerian government spending on routine immunization programs between 2014 and 2018.
    • The Nigeria Strategy for Immunisation and PHC System Strengthening (NSIPSS), 2018 plan, which includes a proposed split for covering vaccination costs between the Nigerian government and Gavi between 2018 and 2028, with costs covered by the Nigerian government rising and costs covered by Gavi phasing out over time. (See p. 24, Figure 8.)

    We are uncertain about the extent to which the plan to phase out Gavi contributions over time in Nigeria Strategy for Immunisation and PHC System Strengthening (NSIPSS), 2018 has been implemented. As a rough guess for the proportion of immunization costs covered by Gavi and the Nigerian government in 2022 and 2024, we average the proportion of routine immunization costs covered by the Nigerian government from the historical data we have seen from WHO for 2014 to 2018 with the planned proportion of immunization costs covered by the government in 2022 to 2024 from Nigeria Strategy for Immunisation and PHC System Strengthening (NSIPSS), 2018. See GiveWell, New Incentives cost per infant immunized [February 2022], "Government and Gavi costs" tab, "Costs covered by Gavi and government" section.

    Limitations of our estimate of the proportion of immunization costs covered by the Nigerian government and Gavi include:

    • As noted above, we are uncertain about the extent to which the plan described in Nigeria Strategy for Immunisation and PHC System Strengthening (NSIPSS), 2018 (p. 24, Figure 8) to phase out Gavi contributions over time has been implemented.
    • We have not carefully vetted the sources WHO relies on for the estimates it reports for total government spending on routine immunization programs in Nigeria.
    • We assume that all immunization costs are covered by the Nigerian government or Gavi (excluding costs covered by New Incentives). The NSIPSS mentions that other donors might contribute to the immunization costs, but the size of the contributions is not specified and we would guess it is unlikely to be a large portion of the total. We would guess Gavi and the government shoulder the majority of the costs because the immunization plan outlined in NSIPSS is based on Gavi and government spending, and the document refers to other donors' contribution as "supplementing'' these funds.
      • "Of this amount $2.7 billion is earmarked for vaccines for routine immunization and supplementary immunization activities (SIAs), with $1.95 billion to be provided by the Nigerian Government, and $773.2 million requested from Gavi." Nigeria Strategy for Immunisation and PHC System Strengthening (NSIPSS), 2018, p. 11.
      • "Total vaccine support by the Alliance to the Government of Nigeria would amount up to US$ 773 million." Gavi, Report to the Board, 2018, p. 12.
      • "Government is also engaging with the Bill and Melinda Gates Foundation and other donors, for additional grant financing to be tied to specific performance indicators. This would supplement Gavi support and domestic resources towards immunization financing from 2018 to 2028." Nigeria Strategy for Immunisation and PHC System Strengthening (NSIPSS), 2018, p. 71.

  • 121

    New Incentives' total costs during this period were approximately $6.6 million, and about 250,000 disbursements were made for the BCG vaccination. See GiveWell, New Incentives cost per infant immunized [February 2022], "Per-year analysis" sheet, for details.

    The BCG disbursement is the first in the series of disbursements made by New Incentives' program and is a key requirement for receiving the remaining disbursements. New Incentives also tracks the number of enrollments as separate from cash disbursements for BCG, but we have chosen to use BCG disbursements as the measure for number of enrollments. In a previous version of our cost per infant immunized analysis, we checked the sensitivity of the estimate to using enrollment data rather than BCG disbursement. The estimate of cost per child decreases by about 1% when using data on the number of enrollments. See GiveWell, New Incentives costs, "Costing_NI" sheet, for details.

  • 122

    See "https://docs.google.com/spreadsheets/d/1gAy4_Y4SvvE-R2sTEE7jio3pf4Ak1JP…">GiveWell, New Incentives cost per infant immunized [February 2022], "Per-year analysis" sheet, for details.

  • 123

    See GiveWell, New Incentives cost per infant immunized [February 2022], "Source: WHO gov expenditure routine immunization" sheet and Nigeria Strategy for Immunisation and PHC System Strengthening (NSIPSS), 2018, figure 8, p. 24.

  • 124

    For a discussion of why we consider funding a charity's work up to three years in the future, see this blog post.

  • 125

    Some of our top charities have a policy of holding funding reserves. In our room for more funding analyses, we typically include reserved funding as funding available to support program activities. We do this both to ensure consistency across top charities (as not all top charities hold reserves) and to understand the true effect of granting additional funding (i.e. whether additional funding would support undertaking additional program activities versus building or maintaining reserves).

  • 126

    GiveWell maintains a list of all charities that meet our criteria, along with a recommendation to donate to our Top Charities Fund (we updated the name of this fund in September 2022; more information here). Some donors give based on our top charity list but do not follow our donation recommendation. In our projections of future funding, we typically count only one year of funding that an organization receives as a result of being on our list of top charities in order to retain the flexibility to change our recommendations in future years.

  • 127

    We update a top charity's room for more funding analysis more frequently if we grant funding to it more frequently.

  • 128

    We update a top charity's room for more funding analysis more frequently if we have reason to believe that its funding and budgets have changed substantially.

  • 129

    For a list of grants we have made from our Top Charities Fund, see this page, section "Past recipients of the Top Charities Fund." (We updated the name of this fund in September 2022; more information here.)

  • 130

    Open Philanthropy, a philanthropic organization with which we work closely, is the largest single funder of our top charities. The vast majority of Open Philanthropy's current giving comes from Good Ventures. We make recommendations to Open Philanthropy each year for how much funding to provide to our top charities and how to allocate that funding among them. An example of these recommendations from November 2020 can be found on this page.

  • 131

    This includes donations made to charities on our checkout page, donations made directly to the organizations, and donations through other third-party organizations that share GiveWell’s recommendations (e.g., One for the World).