New Incentives (Conditional Cash Transfers to Increase Infant Vaccination)

Published: September 2024

Previous versions of this page:

• April 2024 version
• December 2021 version
• May 2021 version
• November 2020 version

Summary

Basics

New Incentives runs a conditional cash transfer (CCT) program in northern Nigeria. The program primarily aims to increase vaccination rates through distributing cash incentives. Caregivers who bring their children for routine vaccinations, which are provided through government clinics free of charge, can receive a total of 6,000 naira (about $5) over six visits. As of September 2024, New Incentives is planning to offer an additional 5,000 naira incentive at the measles 2 vaccination visit for caregivers whose infants completed the entire routine immunization schedule. This would bring the total incentive amount to 11,000 naira (or about $9.50).

Clinic staff employed by the government conduct the vaccinations themselves, while New Incentives' field officers check children’s eligibility, enroll children into the program, and distribute cash incentives. (More)

New Incentives also conducts awareness-raising activities and aims to ensure sufficient vaccine supply by working with partners to address bottlenecks in the vaccine supply chain (see our separate report on New Incentives as an organization for more detail). Around a third of GiveWell’s funding for New Incentives is used for cash incentives, with the rest going to staff costs, distribution and transport costs, awareness-raising activities, and other costs (see here in our separate report on New Incentives).

How cost-effective is it?

We estimate that it costs approximately $1,500 to $6,000 (depending on the specific state)1 to avert a death through the program in the areas where New Incentives works. We also estimate that the program equates to being 10x - 42x as effective as spending on unconditional cash transfers (GiveWell’s benchmark for comparing different programs).

In simple terms, we think New Incentives’ program is cost-effective because:

• New Incentives' program increases the number of children who receive routine vaccines. Vaccination rates in northern Nigeria are low (~40% to ~70% in locations where New Incentives works), and we think New Incentives’ program addresses some of the barriers to vaccination (e.g., opportunity costs of taking children to clinics and perceived low value of vaccinations). (More) Extrapolating from the results of a randomized controlled trial (RCT) of New Incentives' program and triangulating against other studies of vaccine incentive programs, we estimate the program increases vaccination rates by approximately 9 - 18 percentage points in areas where it now works. (More)

• Child mortality is very high in northern Nigeria, and we’d expect many children to die from vaccine-preventable diseases without vaccination. We estimate that unvaccinated children under the age of five have an approximately 3% to 5% chance of dying from vaccine-preventable diseases before their fifth birthday in the locations where New Incentives works. Our analysis is mainly based on estimates of mortality from vaccine-preventable disease from the Global Burden of Disease (GBD) model. (More) We also estimate that vaccines indirectly avert about 0.75 deaths for every death they directly avert from the disease they target. This is based on evidence that vaccination programs sometimes have larger impacts on mortality than would be expected from their impact on the diseases they target alone (more).

• The vaccines that New Incentives incentivizes are effective at reducing mortality. Based largely on evidence from vaccine RCTs, we estimate that being vaccinated reduces a child’s mortality risk from vaccine-preventable diseases by approximately 50% - 55% from the point of vaccination up to age five (more). Our impression is also that routine childhood vaccination is widely viewed in the global health community as an effective way to reduce child mortality, strengthening our confidence in these estimates. (More)

• While the per-child cost of this program is higher than many other programs we fund, the benefit per child enrolled is also very high. This program provides cash transfers to caregivers, a cost not incurred by many other programs we fund. It also relies on a large network of in-country staff to disburse the cash transfers. We currently think that each child enrolled in the program costs New Incentives approximately $18 (more), compared to a cost per child of approximately $5 to $6 per year for Malaria Consortium's seasonal malaria chemoprevention (SMC) program. Although the per-child cost is high, we estimate that New Incentives' program is above our cost-effectiveness bar in many areas because vaccines are highly effective at averting mortality.

• We think New Incentives’ program leverages a large amount of external funding and is unlikely to be funded by other actors if we did not fund it. We attempt to adjust our cost-effectiveness analysis to account for the impact of the program on other actors’ spending. In this case, we think New Incentives’ program leverages a large amount of funding from Gavi and the Nigerian government (who pay for purchasing the vaccines and the staff to administer them). This increases our cost-effectiveness estimate because we think that the leveraged funding would have otherwise gone to less cost-effective programs (more). We also think that the program, or a similar vaccine incentive program, would be unlikely to be funded by other actors if we did not fund it (because we have seen relatively little interest from other actors in funding comparable programs). We account for the chance that another actor would fund the program or a similar one in GiveWell’s absence with a downward adjustment of approximately 10%, which is small compared to some other programs we fund (more).

• In addition to averting child mortality, we think that the program probably results in various additional benefits. These include:
  • Reduced mortality for children in later life, because we think vaccines provide some long-term protection against disease. This accounts for 10% to 13% of the benefits we model, varying by state (more).
  • Increased income for children in later life, by averting disease in a sensitive developmental period of childhood. This accounts for approximately 20% of the benefits we model (more).
  • Increased consumption for children and their families in the short term because of the cash incentives themselves. This accounts for around 2% to 9% of the benefits we model, varying by state (more).
  • Various other benefits that we don’t model and apply as percentage best guesses. These include reduced morbidity from disease and averted costs that would have been spent on treatment of disease. These increase our cost-effectiveness estimate by 50% (more).

We use a cost-effectiveness analysis to quantify our reasoning. Here is a summary of our analysis, using one state, Bauchi, as an example.

We’ve also considered other perspectives that might not be captured explicitly in these cost-effectiveness estimates (e.g., whether experts see New Incentives as a good program). Overall, none of the work we’ve done on these questions has substantially undermined our view that the case for New Incentives is strong.

However, we have spent considerably less time on these questions than we have on our main cost-effectiveness model. We hope to engage more with them in the future. (More)

How could we be wrong?

Overall, we are reasonably confident in the case for New Incentives’ program, and our level of uncertainty is in a similar range to GiveWell’s other top charities. New Incentives’ program is more complex than these other programs (implying additional uncertainty) and may have a higher risk of unintended negative consequences (e.g., risk of fraud, potential for backlash if the program ends). However, we think it is less likely that New Incentives’ program or similar programs would be funded in GiveWell’s absence than for other top charities, giving us confidence that our funding is leading to additional positive impact.

Our key open questions:

• Are we adequately accounting for increases in vaccination rates that would have taken place anyway? The RCT found a substantial increase in control group vaccination rates over the course of the study. Other sources of data also show vaccination rates were increasing in northern Nigeria before New Incentives began scaling up. Because we’d expect the program to be less effective in areas with higher baseline coverage, this raises a concern that the impact in the future will be smaller than the impact observed in the RCT. We currently account for this by assuming vaccination coverage will continue to improve by the same rate seen in the years before New Incentives' scale up (2013-2021). However, our estimates could be meaningfully off in either direction. We also haven’t deeply investigated what other programs are being delivered to increase vaccination rates in northern Nigeria, which increases our uncertainty on this issue. (More)

• How reliable are the mortality estimates we rely on? Our cost-effectiveness analysis uses estimates of mortality from IHME's Global Burden of Disease (GBD) Project. These estimates are based on a number of modeling assumptions that we have not reviewed in detail, and we have substantial uncertainty about them. While we use GBD estimates across a number of our programs, we are particularly uncertain in this case because our analysis requires us to take a stance on the proportion of each cause of death attributable to different pathogens (e.g., the share of diarrhea deaths attributable to rotavirus), which adds additional uncertainty. We also assume that 0.75 deaths are indirectly attributable to vaccine-preventable disease for every death directly attributed to these diseases. This is a rough best guess and could be improved with further research. (More) Our 25th - 75th percentile confidence interval for a child’s probability of death before age five in Bauchi is 3.7% to 8.0%. This implies a program cost-effectiveness of 13 to 28 times as cost-effective as direct cash transfers (“13x-28x”).

• How effective are vaccines at reducing mortality in Nigeria? Some evidence we have seen suggests that the measles vaccine might be less effective in Nigeria than would be expected based on published literature. We adjust for this risk in our analysis (reducing our estimate of vaccine efficacy by around 20%). But our adjustment is very speculative and further research could lead us to update this upward or downward. (More) Our 25th - 75th percentile confidence interval for vaccine efficacy against vaccine-preventable disease (which incorporates this adjustment) is 37% to 69%, implying a cost-effectiveness of 15x to 28x in Bauchi.

• How will the effect of New Incentives' program in states with higher vaccine coverage compare to the effect in the RCT of the program? Our understanding of the effect of New Incentives' program on vaccination rates is largely based on the results of an RCT of the program conducted in Jigawa, Katsina, and Zamfara in 2017-2020. We extrapolate the RCT results to other states based on their vaccine coverage rates, but we are unsure how well these results will translate to new states. (More) New Incentives has begun to conduct vaccine coverage surveys in areas where it works, and we plan to use these as another source to check that the program is having the expected effect at scale. But we’ve only just started to analyze these surveys (as of February 2024), and don’t incorporate them into our analysis yet. (More)

• How high is the risk of fraud? We’d expect fraud to be a significant risk for a program like New Incentives, which handles large volumes of cash. We currently account for this with an approximately -10% adjustment to the number of children enrolled (reflecting that some children may be enrolled multiple times) (more), and -2% adjustment to account for other ways New Incentives’ monitoring data could be wrong (including fraud) (more). New Incentives has a number of systems to monitor and prevent fraud, but it’s possible that there are types of fraud these are unable to detect. We discuss this issue in detail on a separate page.

• Are we properly accounting for the community-level benefits of vaccines? Our analysis is based on the impact of vaccination for individual recipients reported in RCTs. We use rough supplementary adjustments of approximately 25% to account for the impact of increased vaccination on disease transmission, including during outbreaks, and the potential for development of herd immunity. (More) Incorporating these as percentage guesses allows us to use a relatively simple main cost-effectiveness model. However, our understanding is that community-level benefits are considered one of the major benefits of vaccination, and it's possible that explicitly modeling them could change our estimates up or down. We plan to get more input from epidemiologists and disease modelers on our analysis in the future. (More)

• What happens to vaccination rates if the program ends? By introducing a financial incentive for vaccination, New Incentives' program may lead to lower vaccination rates after the program is discontinued than there would have been if the program had never been implemented. We also think it’s possible that the program could lead to longer term increases in vaccination by changing social norms and increasing knowledge about vaccination, or strengthening supply chains. .

  As of September 2024, we have investigated this issue at a moderate level of depth. Our best guess is that discontinuing New Incentives' program may lead to a roughly 1-3% drop in vaccination rates for subsequent children born to caregivers who previously had a child enrolled during the program. In our current cost-effectiveness analysis, we apply a -11% to -32% adjustment to states in Nigeria where New Incentives’ program does not currently operate. This adjustment aims to capture the potential impact from a decrease in vaccination rates if the program was introduced in these locations and then withdrawn. We view this adjustment as highly speculative because there is limited high-quality evidence on the effect of discontinuing an incentives program targeted at childhood immunizations in low- and middle-income countries. We see the long-term impact of program discontinuation as an important priority area for future research and plan to seek opportunities to generate additional evidence. (More)

• Will New Incentives maintain program quality at scale? Since the RCT, New Incentives has expanded quickly and is operating in nine states (as of September 2024). We think it’s plausible that this could lead the program to be delivered less effectively (e.g., if managers have less effective oversight, or vaccine supply is not able to keep up with demand). The monitoring data we’ve reviewed has remained broadly stable since the RCT, although we have seen an increase in supply problems for some vaccines. We interpret this to mean that New Incentives has been able to deliver the program to a similar high quality as it has grown, although we plan to keep monitoring this. (More)

• Are there errors in our analysis we’ve missed? Our analysis of New Incentives’ program is more complicated than most of the other programs we support, and the program is comparatively newer (meaning our analysis has been reviewed less rigorously). It’s possible that this increases the likelihood of errors that could affect our bottom line.

• How accurate was our analysis of New Incentives in hindsight? GiveWell’s cost-effectiveness analyses are “forward-looking” and aim to predict the impact a program will have at the time we make a grant decision. We have paid less attention to backward checks to understand how accurate the predictions in our New Incentives grants were. This is a weakness in our approach and something we aim to improve in the future. (More)

• 1

  See this row in our cost-effectiveness analysis.

• 2

  “Vaccines train your immune system to create antibodies, just as it does when it’s exposed to a disease. However, because vaccines contain only killed or weakened forms of germs like viruses or bacteria, they do not cause the disease or put you at risk of its complications.” World Health Organization, “Vaccines and immunization.”

• 3

  See World Health Organization, “Immunization dashboard”, “Vaccination coverage globally” section.

• 4

  See the following maps showing each country’s vaccination rates for the main vaccines we model in our analysis of New Incentives’ program:

  • DTP: see Nigeria (56% vaccinated) in the following map, which uses the WHO/UNICEF, “DTP3 (% of one-year-olds immunized)" dataset.. Our World in Data, "Share of one-year-olds vaccinated against diphtheria, pertussis, and tetanus," 2021
  • BCG: see Nigeria (75% vaccinated) in the following map, which uses the WHO/UNICEF, “BCG (% of one-year-olds immunized)" dataset. Our World in Data, "Share of one-year-olds vaccinated against tuberculosis, 2021"
  • Measles: see Nigeria (59% vaccinated) in the following map, which uses the WHO/UNICEF, “MCV1 (% of one-year-olds immunized)" dataset. Our World in Data, "Share of one-year-olds vaccinated against measles, 2021"
  • HiB: see Nigeria (56% vaccinated) in the following map, which uses the WHO/UNICEF, “Hib3 (% of one-year-olds immunized)" dataset. Our World in Data, "Share of one-year-olds vaccinated against Haemophilus influenzae type B, 2021"
  • PCV: see Nigeria (52% vaccinated) in the following map, which uses the WHO/UNICEF, “3.b.1 - Proportion of the target population with access to pneumococcal conjugate 3rd dose (PCV3) (%) - SH_ACS_PCV3" dataset. Our World in Data, "Share of children vaccinated with pneumococcal conjugate, 2021"
  • Rotavirus: Nigeria has no data for rotavirus, based on the WHO/UNICEF, “RotaC (% of one-year-olds immunized)" dataset. See this map: Our World in Data, "Share of one-year-olds vaccinated against rotavirus, 2021"

• 5
  • “Northern Nigerian communities were less likely to be vaccinated compared to Southern Nigerian communities. Studies suggested this was driven by higher levels of education and wealth in the South, higher proportions of Muslim households in the North, and greater insecurity in the North.” Mahachi et. al, 2022
  • See also National Bureau of Statistics and United Nations Children's Fund, Multiple Indicator Cluster Survey 2021, Statistical Snapshot Report, 2022, p. 30, figure labeled “Immunisation Coverage across Nigeria.”

• 6
  • “In partnership with WHO, Gavi, the Vaccine Alliance, UNICEF, and other partners, the Government of Nigeria, through the National Primary Health Care Development Agency (NPHCDA), administers 13 vaccines on its Expanded Programme on Immunization (EPI) schedule.” World Health Organization, “Beyond the numbers—Nigeria steps up measures to reach eligible children with potent vaccines,” May 19, 2023.
  • “The Expanded Programme on Immunization (EPI) was launched in 1974 to ensure that all children, in all countries, benefited from life-saving vaccines… There are now 13 vaccines (antigens) recommended by WHO for the EPI programme.” World Health Organization, “Essential Programme on Immunization”.

• 7
  • "In Nigeria, routine childhood vaccinations are provided at government clinics free of charge, but caregivers often find it difficult to afford transportation and face other challenges to taking their baby to a clinic – often half a day’s trip away or longer. These round-trip journeys to the clinic must be made six times to complete a childhood routine vaccination schedule, all within the first year of a baby’s life." New Incentives, "How it works"
  • See the infographic at New Incentives, "How it works".

• 8

  This conversion uses the naira:USD market exchange rate, which the Nigerian Central Bank reported as 1,165 naira per $1 as of April 25, 2024. See this page(archived version available here). 11,000 / 1,165 = 9.442.

  New Incentives has made a number of updates to the incentive schedule (originally 4,000 naira for all incentivized vaccinations) over time:

  • New Incentives originally added a 500 naira incentive for measles 2 (which had not yet been introduced into Nigeria's routine vaccination schedule at the time of the RCT). This brought the total incentive across all visits to 4,500 naira. In early 2023, New Incentives increased the incentive for measles 2 to 1,000 naira. This brought the total incentive across all visits to 5,000 naira.
  • In August 2023, New Incentives decided to change its schedule to offer 1,000 naira per visit (6,000 in total).
  • In April 2024, New Incentives decided to pilot offering an additional 5,000 naira incentive at the measles 2 vaccination visit for caregivers whose infants completed the entire routine immunization schedule. As of May 2024, New Incentives had begun piloting this additional incentive in a few clinics, and expected to roll it out across the program later in 2024, conditional on the pilot proceeding well.

  New Incentives, January 14th 2022, December 16th 2022, September 8th 2023, May 13th 2024 Program Updates (unpublished).

• 9

  “CCTs are now present in 64 countries, a dramatic increase from 2 countries in 1997 and 27 in 2008.”
  “Conditional cash transfers (CCTs) are periodic monetary benefits to poor households that require beneficiaries to comply with specific behavioral requirements to encourage investments in human capital (such as school attendance, immunizations, and health checkups).” World Bank Group, The state of social safety nets 2015, p. 1 and 8.

• 10
  • Adapted from New Incentives, "How it works"’.
  • We directly model the benefits of increasing coverage of the directly incentivized vaccines and the rotavirus vaccine in our cost-effectiveness analysis. We do not directly model the remaining indirectly incentivized vaccines. We expect that they will provide only a small proportion of the overall benefits and so the additional time required to directly model their benefits would not be worth the added complexity (more below). Instead, we account for these vaccines with a rough percentage adjustment (+4% as of February 2024) elsewhere in our analysis.

• 11

  Oral polio vaccine.

• 12

  Inactivated polio vaccine.

• 13
  • “We closely collaborate with the state governments of Bauchi, Gombe, Jigawa, Kaduna, Kano, Katsina, Kebbi, Sokoto, and Zamfara in northern Nigeria.” New Incentives, “Our work” page (as of February 2024).
  • Note: Our cost-effectiveness analysis includes columns representing all 36 states in Nigeria and the Federal Capital Territory, but New Incentives does not currently operate in most of those states.

• 14

  We used the following process:

  • Two GiveWell staff members familiar with our New Incentives research gave confidence intervals for the parameters in the Simple CEA sheet of our cost-effectiveness analysis. The intervals were for New Incentives’ program in Bauchi state (the example program we discuss in this report). These intervals address the following question: “For each parameter, provide an upper and lower bound for the range of values that you think has a 50% probability of containing the true value (i.e., a 25% probability the true value is lower than the range, and a 25% probability the true value is higher).”
    • Where we felt that it was difficult to form an intuitive judgment about a parameter (e.g., because it comprised several separate parameters), we gave intervals for each individual parameter.
    • For example, our estimates of the probability that a child will die from vaccine-preventable disease before age 5 are aggregated from (i) the probability that an unvaccinated child will die directly of vaccine-preventable causes before age 5 (more) and (ii) our estimate of the number of deaths indirectly caused by vaccine-preventable diseases per direct death (more).
  • A third GiveWell staff member reviewed the intervals given by the first two staff members and decided upon a final interval for each parameter, using their subjective judgment.
  • We applied the intervals used for Bauchi to other states as follows:
    • We first calculated both the relative differences and absolute differences between our best guesses and our 25th and 75th percentile values for each parameter. For example, if our best guess value for a parameter was 50% with 25th and 75th percentile values of 25% and 75%, then the relative differences would be -50% and +50%, and the absolute differences would be -25 percentage points and +25 percentage points.
    • We assigned confidence intervals to each country that matched the widths of our confidence intervals for Bauchi, putting some weight on the relative difference between the confidence interval bounds and the best guess and some weight on the absolute difference. The higher our best guess value for a parameter in a country was compared to our best guess for Bauchi, the more weight we put on the absolute difference between the confidence interval bounds and our best guess. The purpose of this method was to avoid assigning extremely wide confidence intervals to parameters in countries where our best guess was much higher than our best guess for Bauchi. In some limited cases, we manually adjusted estimates where the intervals seemed implausible, compared to our best guess for Bauchi.
  • Finally, we used monte carlo simulations to transform the confidence intervals for individual parameters into an aggregated confidence interval for each parameter. These are the intervals that appear in the summary of this page, and the Bauchi columns in the Sensitivity Analysis sheet of our cost-effectiveness analysis.

• 15

  We use the term “counterfactual” to refer to the state of the world that would exist if we did not provide funding to a grantee for a program. In this case, we refer to children who are “counterfactually vaccinated” to mean children who are vaccinated as a result of New Incentives’ program who would not otherwise have been vaccinated. We exclude children whose caregivers receive the cash incentive, but who we think would have been vaccinated anyway, even without the program.

• 16

  $1 million is an arbitrary amount that we use to quantify the benefits of the program in the rest of our analysis.

• 17

  See this row of our cost-effectiveness analysis.

• 18

  We calculate this by dividing $1 million by our estimate of the number of additional children reached by the program in each state, e.g., ~9,900 in Bauchi.

• 19

  See this row of our cost-effectiveness analysis. Note that this is a forward-looking estimate of what New Incentives' cost per child enrolled will be in 2024. We chose to use a forward-looking projection here rather than relying on cost data from previous years because New Incentives is making some changes to its program in 2024 that we think should be factored into our cost per child estimate. The biggest change from a costs standpoint is the introduction of an additional 5,000 naira incentive for caregivers whose infants complete the full routine immunization schedule.
  For more details, see this section of our November Top Charity Fund allocation to New Incentives.
  This additional incentive brings the total incentives available to caregivers from 6,000 naira to 11,000 naira, which we think is a substantial enough change to warrant an adjustment to our cost per child calculations. Our cost per child estimate relies on New Incentives' assumptions about how program changes will affect its cost per child and about the foreign exchange rate it will receive in 2024. While relying on New Incentives' assumptions could cause us to over or underestimate cost per child, we were uncertain whether attempting to model the impact of the program changes ourselves would produce a more accurate estimate, and ultimately did not think the capacity tradeoff was worthwhile. See our cost per child estimate here.

• 20

  See this sheet in our analysis for a breakdown by category.

• 21

  We estimate that the cost per child fell from $38.91 between March 2020 and December 2020 to $17.26 between January and December 2023. We think this is because a decreasing share of New Incentives’ overall costs were start-up costs during that period. See this row in our analysis.

• 22

  $23,399,097 / 1,518,511 = ~$15.41. See calculation here.

• 23
  For more details, see this section of our November 2023 Top Charity Fund allocation to New Incentives.
  
• 24

  See New Incentives' estimate here.

• 25

  Note that this foreign exchange rate assumption represents New Incentives' best guess as of February 2024, but the real foreign exchange rate is likely to fluctuate. Our understanding is that Nigeria operated two exchange rates, an official rate and a parallel rate, until June 2023, but after that time has allowed the exchange rate to be set by the market.

  “Nigeria has for years operated multiple exchange rates for the naira—with the official exchange rate dictated by the central bank, while a far higher unofficial rate determined the price of imported commodities like wheat, which are priced in dollars. The exchange rate now will be determined by market forces and no longer the central bank, a move that analysts on Thursday said would boost inflows of money and help stabilize an economy battered by surging inflation and a record unemployment rate.” Associated Press, "Nigeria lets market set currency exchange rate to stabilize economy, woo investors, 2023.

• 26

  Our calculation is based on:

  • Expected BCG scar rate: 90%. This is based on a quick literature review. We found three studies reporting scarring rates of 82-96%. These average to 87% (unweighted) which is roughly in line with the 90% cited by WHO as the BCG scar rate in its 2018 position paper ("BCG vaccination usually causes a scar at the site of injection due to local inflammatory processes. However scar formation is not a marker for protection and approximately 10% of vaccine recipients do not develop a scar." p. 84). We therefore use 90% as our rough best guess for the BCG scar rate. (More details above).
  • Scar rate in New Incentives’ monitoring data: New Incentives tracks the percentage of enrolled children with BCG scars. In 2023, it found that 99.64% of children had one BCG scar and 0.01% had two scars (New Incentives monitoring results as of December 2023). This is higher than the 90% rate we’d expect. Our best guess is that the explanation for this is that some children received the BCG vaccine more than once, and developed a scar on a subsequent occasion even if they didn’t scar at first.
  • We calculate our ~10% repeat enrollment estimate as the difference between the expected and actual scar rate / the expected scar rate + the percentage of children with two or more scars in 2023. This is 10% / (90% + 0.01%) = 10.72% See this section of our analysis for our calculations.

• 27

  See here for this calculation.

• 28

  See here for this calculation.

• 29

  Our calculation is based on publicly available data on vaccine coverage, costs, and population size between 2014 and 2018:

  • Vaccine coverage: We estimate that overall vaccine coverage was 51% in Nigeria between 2014 and 2018, using WHO and UNICEF estimates of coverage (available here).
  • Vaccine costs: We use publicly-available WHO estimates of expenditure on routine vaccinations between 2014 and 2018, compiled in this sheet. The WHO estimates that total spending in this period in Nigeria was ~$1.6 billion ($314 million per year), and the government covered around 29% of this. See this section of our analysis.
  • Population size: We use estimates from the Global Burden of Disease project (compiled here) that the average population under age 1 in Nigeria was ~6.9 million between 2014 and 2018. Assuming 51% of these children were vaccinated, this implies 3.5 million children per year vaccinated.
  • % of fixed costs: Our estimates are for the cost of each additional child vaccinated to the Nigerian government and Gavi. We would guess that this is lower than the average cost of each child vaccinated, because some proportion of total costs are fixed costs. We estimate that 30% of total costs to these actors for routine vaccinations are fixed costs. This is a very rough guess.

  Using these inputs, we estimate that each additional child vaccinated costs $62.92. ($314m x (100% - 30%) / ~3.5m children = $62.92. See this section of our analysis for our calculations.
  Finally, we roughly allocate costs between Gavi and the Nigerian government (as indicated in this section):

  • Based on the WHO data cited above, we estimate that the Nigerian government covered 29% of vaccine costs between 2014 and 2018, and Gavi covered 71%.
  • Based on Nigeria’s 2022 - 2024 strategy for immunization, which sets out a roadmap for transitioning vaccination spending to the government over time, we estimate that future costs will be covered 80% by the government and 20% by Gavi.
  • A straight average of these gives an estimate of 54% costs covered by the government and 46% covered by Gavi. This is a rough guess.
  • Multiplying these percentages by the total cost per additional child vaccinated ($62.92) gives our final estimates of $34.29 per additional child vaccinated to the government, and $28.64 to Gavi.

• 30

  See this row of our cost-effectiveness analysis. .

• 31

  We estimate that the cost per child fell from $38.91 between March 2020 and December 2020 to $17.26 between January and December 2023. We think this is because a decreasing share of New Incentives’ overall costs were start-up costs during that period. See this row in our analysis.

• 32

  See this row of our cost-effectiveness analysis.

• 33

  See this row of our cost-effectiveness analysis.

• 34

  We use the term “coverage” to refer to the proportion of children vaccinated. We define “baseline vaccination coverage” as the proportion of children who would be vaccinated in the absence of New Incentives’ program.

• 35

  See this row of our cost-effectiveness analysis.

• 36
  • "New Incentives will group local government areas (LGAs) it expands to within a given state at a given point in time into ‘expansion groups’. New Incentives will then collect coverage data in these expansion groups once before the start of operations to establish baseline coverage rates." IDinsight, Coverage monitoring analysis plan, 2021, p. 1.
  • These expansion groups are called "cohorts."

• 37

  See the number of LGAs included in each cohort for which we have received coverage survey results (as of March 2024) here.

• 38

  "The Multiple Indicator Cluster Survey (MICS) was carried out in 2021 by the National Bureau of Statistics (NBS) as part of the Global MICS Programme. . . . The Global MICS Programme was developed by UNICEF in the 1990s as an international multi-purpose household survey programme to support countries in collecting internationally comparable data on a wide range of indicators on the situation of children and women." National Bureau of Statistics (NBS) and United Nations Children's Fund (UNICEF), Multiple Indicator Cluster Survey 2021, Statistical Snapshot Report, 2022, p. ii.

• 39

  See the MICS data we use in this sheet. Note that we do not use the MICS 2017 data in this sheet. This feeds into our calculations in a different part of our analysis.

• 40

  See this section of our analysis for the rules we apply to the data.

• 41

  For example in Zamfara, where New Incentives started operations before instituting coverage surveys in 2021. "New Incentives will group local government areas (LGAs) it expands to within a given state at a given point in time into ‘expansion groups’. New Incentives will then collect coverage data in these expansion groups once before the start of operations to establish baseline coverage rates." IDinsight, Coverage monitoring analysis plan, 2021, p. 1.

• 42

  Reasoning for each:

  • Measles vaccine: We think the coverage surveys may systematically underestimate coverage of the measles vaccine because they only survey children through 12 months, whereas MICS surveys 12-23 month olds. The first dose of the measles vaccine is recommended to be administered at 9 months, though some infants may receive it later. Thus, the MICS data captures infants getting the measles vaccine after 12 months (see the cell notes in this section for more detail). ("The unit of analysis is the individual 6-12-month-old infant." IDinsight, Coverage monitoring analysis plan, 2021, p. 4)
  • PCV vaccine: Because the coverage surveys do not include PCV, we use the coverage estimates for the Penta vaccine according to the rules above, and apply an adjustment based on the ratio of PCV coverage to Penta coverage observed in the MICS data (see this section for our calculations). (The coverage surveys include coverage of BCG, Penta, and measles vaccines, as indicated by this research question the assessments are intended to answer: "What is the self-reported coverage of the following routine childhood immunizations among 6-12 month olds at a given point in time in the state-LGA cohort: BCG, any Penta, Measles 1, full vaccination (loose)?" IDinsight, Coverage monitoring analysis plan, 2021, p. 2)
  • Rotavirus vaccine: We do not have data from either source on the rotavirus vaccine because it was not introduced in Nigeria’s immunization schedule until August 2022 (see Gavi, "Dealing with diarrhoea: Nigeria introduces rotavirus vaccine into its immunisation plan," August 30, 2022). We assume rotavirus coverage is the same average of Penta and PCV coverage, as both vaccines are administered at the same time (see this section for more detail).

• 43

  See this section of our analysis. For example, in Bauchi we estimate that the second dose of PCV contributes 19% to the overall under-five mortality benefit of the program, and the first dose of the measles vaccine contributes 6%.

• 44

  Our calculations are in this sheet. Each of the factors we include are discussed below.

  • Mortality associated with diseases targeted by each vaccine: We use state-level estimates of mortality from vaccine-preventable diseases from the Institute for Health Metrics and Evaluation's 2021 Global Burden of Disease model to calculate the proportion of vaccine-preventable deaths attributed to the diseases targeted by each vaccine. These are the same as the estimates that we use to model risk of death from vaccine preventable-disease, and we apply the same adjustments (discussed in this section).
  • Vaccine efficacy: We then adjust those proportions to account for the estimated efficacy of each vaccine in preventing disease. Adjusting for vaccine efficacy increases the overall contribution of vaccines with high estimated efficacy (such as measles) and decreases the contribution of vaccines with lower estimates of efficacy (such as the rotavirus vaccine). We discuss our vaccine efficacy estimates in this section.
  • Efficacy by dose: Our vaccine efficacy estimates are for a full course of that vaccine. To translate dose-specific coverage to overall coverage for vaccines with multiple doses (PCV, Penta, and rotavirus), we came up with rough estimates of the marginal efficacy of each dose (i.e., how much protection is provided by a first dose compared to a second dose, etc.), based on a quick literature review. Our calculations are in this section of our analysis. The literature broadly matched our understanding that the marginal efficacy of the final dose of each vaccine tends to be lower than previous doses.

• 45

  See this row of our analysis for the coverage survey data and this row for the MICS data.

  Our method

  We estimate the level of overreporting by comparing caregiver-reported data on BCG vaccination coverage to data from the coverage surveys on BCG scarring (BCG vaccination typically leaves a scar, providing a more objective measure of vaccination rates). We roughly assume, based on a quick literature review, that 90% of children who are vaccinated against BCG develop a scar that will be detected and correctly identified when checked for (details discussed below).

  • The proportion of children with BCG scars in coverage surveys to date has been 56%, compared to 62% of children reported to be vaccinated by caregivers.
  • These values imply that the true rate of BCG vaccination was ~1% higher than caregivers reported (56% / 90%) / 56% - 100% = ~1%). See calculations here.
  • We make this same adjustment to coverage of all vaccines (not just BCG), assuming that the level of bias was probably around the same regardless of the specific vaccine.
  • We use a smaller (~0%) adjustment for the MICS data because we estimate that only 22% of the MICS data comes from self-reported questions (the rest comes from reviewing children’s vaccination cards). See this section of our analysis.

  Data on BCG scarring rates

  We found three studies reporting scarring rates of 82-96%. These average to 87% (unweighted) which is roughly in line with the 90% cited by WHO as the BCG scar rate in its 2018 position paper. We therefore use 90% as our rough best guess for the BCG scar rate. Because the estimates from the literature rely on an individual detecting and correctly identifying BCG scars (rather than being a theoretical percentage of all infants who receive BCG who scar, we no longer make additional adjustments for the percentage of scars which are detected or the percentage of detected scars which are in fact BCG scars. We had included such adjustments in earlier versions of our CEA. See here.

  • "BCG vaccination usually causes a scar at the site of injection due to local inflammatory processes. However scar formation is not a marker for protection and approximately 10% of vaccine recipients do not develop a scar." World Health Oragnization, BCG vaccines: WHO position paper – February 2018, p. 84.
  • "Two hundred and six subjects (96.3%) had a postvaccination BCG scar." Atimati and Osarogiagbon 2014.
  • "The prevalence of BCG scar was 79.5% among the male infants and 84.7% among the female infants, while the overall prevalence of BCG scar was 81.5%." Gambo et al. 2014.
  • "Although 84.2% had physical evidence of BCG inoculation only 69.8% had developed detectable sensitization to the tubercle bacilli as shown by the Mantoux test." Odujinrin and Ogunmekan 1992.

• 46

  See our analysis of self-report bias in the RCT here.

• 47

  We looked at vaccination coverage estimates from the DHS 2013, MICS 2016-17, DHS 2018, and MICS 2021 surveys when making this estimate.

• 48

  For example, our most recent investigation considered the reallocation of New Incentives' surplus funding to support program additions from 2024-2026. We therefore used 2025 as the middle year of the grant period.

• 49

  See these calculations in the "Baseline vaccination coverage for the target population, aggregated across vaccines" section on the 'Aggregation across vaccines' sheet of our cost-effectiveness analysis.
  

• 50

  See this row of our cost-effectiveness analysis.

• 51

  We divide the impact of the program by the share of unvaccinated children at baseline to estimate the reduction in unvaccinated children caused by the program: 22pp / (100% - 36%) = ~33%. More information in this section.

• 52

  "This study is an impact evaluation of the NI-ABAE CCTs for RI Program in Katsina, Zamfara, and Jigawa
  States in North West Nigeria funded by Open Philanthropy at the recommendation of GiveWell."
  "The evaluation consisted of a two-arm clustered randomized controlled trial." IDinsight, Impact evaluation of New Incentives, final report, 2020, p.7.

• 53

  The report is available here: IDinsight, Impact evaluation of New Incentives, final report, 2020.

• 54

  “The evaluation consisted of a two-arm clustered randomized controlled trial. We worked with NIABAE to identify clinic catchments in the three evaluation states that met its operational criteria. We then randomly selected our sample of 167 clinics from among these clinics.” IDinsight, Impact evaluation of New Incentives, final report, 2020, pg. 7.

• 55

  "We measured most outcomes using caregiver reports from a household survey, during which enumerators also checked for BCG vaccine scars and recorded data from vaccination records kept in the home." IDinsight, Impact evaluation of New Incentives, final report, 2020, pg. 7.

• 56

  Adapted from IDinsight, Impact evaluation of New Incentives, final report, 2020, Table 3, pg. 23.

• 57

  This is referred to as “Penta 1” throughout the IDInsight final RCT report. However, our understanding based on viewing the survey questions (not published) is that "Penta 1" refers to receiving any dose of the Penta vaccine.

• 58

  In addition to the primary outcomes, this includes all three doses of PCV, and the three doses of Penta individually.

• 59

  See this section of our analysis for the impact on each vaccine dose.

• 60

  “Fully vaccinated” here refers to children receiving BCG, all 3 doses of the Penta vaccine, and the measles vaccine. “When using the strict definition of full immunization (BCG vaccine, Penta vaccine 1-3, Measles 1 vaccine), the likelihood of being fully immunized was 27 percentage points (95% CI: 23, 31, p-value 0.001) higher among children in the treatment areas.” IDinsight, Impact evaluation of New Incentives, final report, 2020, pg. 28.

• 61

  Note that this aggregation method implies that the program increased take-up of each vaccine by a different amount. This is different to our previous method for estimating the program’s effect size, which viewed the effects on the three main primary study outcomes as noisy estimates of the same underlying effect (more in a previous version of our research report here, footnote 7). We think it is plausible that New Incentives’ program would have different effects depending on where a vaccine falls in the schedule (e.g., less effect on early vaccines and more effect on later vaccines), although we’re unsure about this.

• 62

  We use the term “coverage” to refer to the proportion of children vaccinated.

• 63

  See this row in our supplementary analysis.

• 64

  See this section in our analysis.

• 65

  See this section in our analysis.

• 66

  "Across the study area, coverage in the control group was substantially higher at endline than at baseline. This was true for each of the primary study vaccines, with the largest difference for the Measles vaccine: 17.8% of children at baseline versus 57.2% of children at endline had reportedly received Measles 1 vaccine (Table 16). We did not expect a change of this magnitude." See also Table 16, IDinsight, Impact evaluation of New Incentives, final report, 2020, pg. 50.
  Note that the baseline survey was conducted in Zamfara and Katsina states only, and so these estimates aren’t directly comparable to the overall coverage estimate (including Jigawa state) that we discuss above.

• 67

  "Changes in the questionnaire or enumerator technique could have led to increased recording of vaccinations via the survey even if true vaccination coverage had not changed. This could result from either improved recall or increased social desirability, or both. This hypothesis could explain why we do not see similar increases in clinic tally sheets, which record relatively stable control-group vaccination volumes between baseline and endline (see Appendix I)." IDinsight, Impact evaluation of New Incentives, final report, 2020, pg. 51.

• 68

  “I don’t understand the reason for the huge increases in vaccination rates in the control areas in the RCT. I agree with your conclusion that those effects are unlikely to change the determination that NI’s program significantly increased vaccinations but I have two reservations. First, it seems important to determine what it was that increased vaccination rates in the control areas that much because that change was bigger than the treatment effect from NI. My hunch is that it was survey effects or spillovers that weren’t totally captured. I have seen these big changes in control group outcomes before in some of my own studies.”

  “The RCT report shows a very big increase in clinic outreach days in the control areas (from 2 to 3.5 month, a 75% increase). Is that accounted for in your model?” Dr. Jessica Cohen, Bruce A. Beal, Robert L. Beal and Alexander S. Beal Associate Professor of Global Health, Comments on GiveWell’s draft New Incentives Report, November 2023, unpublished. Note that Dr. Cohen’s response was shared before we compared the control group results to independent vaccine coverage surveys.

• 69

  See this spreadsheet for our calculations.

  Our approach

  • We compared RCT data on control group vaccine coverage increases from baseline to endline (in Katsina and Zamfara states only, since the baseline survey was only conducted in these states) with 3 independent vaccine coverage surveys conducted before and after the RCT: (1) MICS 2017 (here), (2) DHS 2018 (here), and (3) MICS 2021 (here). We compared data for BCG and measles 1 because they are the first and last vaccines covered in the RCT (in terms of time of vaccination).
  • The New Incentives baseline survey took place in 2017, and the endline took place in 2020 (in between these surveys). We extrapolated results from the three surveys using a linear trend to estimate what we’d expect vaccine coverage to have been in the New Incentives RCT at baseline and endline in both states. In both states and for both vaccines there was a trend of increasing BCG and measles vaccine coverage between 2016-2017 and 2021.
  • We compared the implied increase in vaccine coverage with the observed increase in the New Incentives RCT. Overall, the implied increase was 13 percentage points for the BCG vaccine and 12 percentage points for the measles 1 vaccine based on extrapolating from independent surveys, compared to 28 percentage points (BCG) and 38 percentage points (measles) in the New Incentives RCT. This implies 48% of the control group increase (BCG) and 30% (measles) is explained by the regional trend, or roughly 40% overall.
  • There are two reasons why we think these could be underestimates:
    • Different age ranges: The RCT used a different age range (12-16 months) than DHS/MICS vaccine coverage (12-23 months). We’d expect that this means the RCT would show larger increases than DHS/MICS given the same underlying changes. For example, for BCG received at birth, if coverage at birth increases over a short period of time, the BCG coverage measured among children aged 12-16 months will increase more rapidly than coverage measured amongst children aged 12-23 months because the children receiving the increase make up a larger share of the measured population.
    • New Incentives clinics may not be representative: The RCT was conducted in a set of clinics that met NI’s operational criteria, which meant excluding clinics that had high security risk and that were barely functional, amongst other reasons. We think it’s plausible that coverage would have increased faster in these clinics than average, regardless of New Incentives’ program.

  Uncertainties

  • The DHS/MICS state level results are fairly noisy and there are substantial survey-to-survey fluctuations. For example, in Zamfara in the DHS/MICS, BCG coverage was 19% in 2016, 16% in 2018, and 51% in 2021. This means we’re uncertain whether it’s valid to assume a linear trend.
  • Our analysis is based on only two states and two vaccines. Given the noisy estimates mentioned in the previous bullet, this means this comparison is fairly uncertain.

• 70

  Note: For our estimate of the main effect size of the program (21 percentage points), we use IDinsight’s estimates of the impact on all vaccine doses, not just the pre-specified outcomes (BCG, measles, and any dose of Penta vaccine). We do not see this as a significant threat to the validity of our analysis because (i) there were also significant increases in the pre-specified outcomes, (ii) we also incorporate information in the pre-specified outcomes, and (ii) we think it’s plausible that the impact of the program might vary according to when a vaccine is delivered in the vaccine schedule.

• 71

  From the independent review by the International Initiative for Impact Evaluation (3ie): "Overall, I found that IDinsight was receptive to suggestions, made major adjustments between baseline and endline that substantially improved the design, was diligent in documenting the Stata code, and their work and assessment are a valid assessment of the New Incentives Conditional Cash Transfer program. While there are remaining questions about the change in coverage (greatly improved) in control areas, and the poor accuracy of the biomarkers and its implications for the efficacy of the measles vaccine in Nigeria, we do not believe that these questions change the interpretation and conclusions of the study. The New Incentives program substantially and significantly increased immunizations rates." International Initiative for Impact Evaluation, Quality assurance of IDinsight's evaluation of New Incentives, 2020, pg. 3.

• 72

  “Vaccination volumes recorded at clinics have experienced statistically significantly larger increases since baseline in treatment clinics than in control clinics. Using clinic tally sheets (which count vaccination doses given at each clinic by vaccine and by month), we found that the average increase in monthly BCG vaccine doses administered between baseline and endline was 17.6 (95% CI: 9.1, 26.2, p-value < 0.001) doses greater in treatment clinics than control clinics.” IDinsight, Impact evaluation of New Incentives, final report, 2020, pg. 31-32.

• 73

  IDinsight, Impact evaluation of New Incentives, final report, 2020, Figures 16-18, pg. 65-66.

• 74

  “Programmatically, New Incentives’ theory of change continues to look promising. First, the sources of vaccinations measured align with the program. Outside of the national measles campaigns, almost all infants receive vaccinations from the sources New Incentives’ program will cover: health facilities and health facility outreach. Second, most caregivers cited lack of knowledge or ambivalence and relatively few caregivers cited socio-cultural reasons or mistrust and fear, as reasons for not vaccinating. It seems likely that an incentive, coupled with awareness raising activities, can overcome these stated reasons for not vaccinating. Finally, New Incentives’ program appears to be unique. While small incentives are relatively common, incentives worth more than 500 Naira are rare and only two caregivers received cash in our sample.” IDinsight, New Incentives evaluation baseline report, 2019, p. 71.

• 75

  We use an internal validity adjustment of -12%. Applying this to the impact of New Incentives’ program on full vaccination coverage of 27 percentage points (the measure we use here for comparison since it’s available across almost all of the other studies we looked at) implies an updated impact of 24 percentage points. See this row of our supplementary analysis for this calculation. This estimate would still be higher than any of the other studies we included in our analysis. Applying our internal validity adjustment to the estimate of program impact we actually use in our main analysis (21 percentage points, discussed above) implies an impact of ~18% (21pp x (100% - 12%)). This is higher than all but one of the other studies we looked at (Banerjee et. al. 2010).

• 76

  This is based on the estimates that we calculate above from the RCT: that it increased vaccine coverage by 21 percentage points from a baseline of 36%. 21pp / (100% - 36%) = ~33%. See this row in our analysis.

• 77

  See this section in our cost-effectiveness analysis.

• 78

  We reviewed nine studies in a 2021 World Bank meta-analysis of conditional cash transfers for immunization. We also conducted a literature search for additional studies. We found two studies published since the World Bank analysis and one study (Bannerjee et. al. 2010) that tested the impact of non-cash incentives (raw lentils and plates) that was not included in the World Bank analysis. Although this is not strictly speaking a conditional cash transfer for immunization, we included this study because the monetary value of these was similar to the cash transfer of the New Incentives program. The final study was the New Incentives RCT. See this spreadsheet for a summary of all 13 studies. This column summarizes which studies were randomized controlled trials.

• 79

  The exception to this is Morris et al. 2004, which reports outcomes on DTP3 instead. Since DTP3 comes last in the vaccination schedule, we think this is a reasonable proxy for full vaccination. See this row in our accompanying spreadsheet.

• 80

  See this column in our analysis spreadsheet.

• 81

  See the graphs in our accompanying spreadsheet. There seems to be a negative relationship between baseline coverage and effect size, such that effect size reduces as baseline coverage increases. We think this makes intuitive sense, as higher baseline coverage implies a smaller pool of caregivers that can be influenced by incentives.

• 82

  We use the following calculation:

  • (RCT effect size x RCT weight) + (skeptical prior x skeptical prior weight) = (27pp x 70%) + (16pp x 30%) = 24pp.
  • This implies a downward adjustment of ~12% ((24 / 77) - 100%). See this section of our accompanying spreadsheet.

• 83

  See this chart in our accompanying spreadsheet.

• 84

  See this section of our cost-effectiveness analysis.

• 85

  See this sheet for our calculations. These calculations include deaths averted across all age groups, and they account for variation between vaccines in disease burden, vaccine efficacy, baseline coverage, and treatment effect of the program. They also account for GiveWell’s moral weights.

• 86

  See this row of our cost-effectiveness analysis.

• 87

  See this row of our cost-effectiveness analysis.

• 88

  We refer to the diseases targeted by routine childhood vaccinations that are included in our analysis as “vaccine-preventable disease.” This does not include diseases targeted by other vaccines not included in our model.

• 89

  See this row of our cost-effectiveness analysis.

• 90

  These estimates (available here) are from the 2021 Global Burden of Disease model.

• 91

  See this sheet for our calculations.

• 92

  See this row in our cost-effectiveness analysis. Because the GBD age bands (0 to 6 days, 7 to 27 days, under 1 year, and 1 to 4 years) don’t align exactly with the scheduled vaccination dates, this adjustment relies on some rough guesses.

  For example, in Bauchi State, the GBD model estimates that 9% of under-five meningitis deaths occur in the first 27 days of life (see this row). We roughly guess that the proportion of under-five meningitis deaths occurring in the first six weeks (before the first doses of PCV and HiB are scheduled) is 10% higher than this, or 10% in total (9% x 110% = 10%).

• 93

  See this row in our analysis.

• 94

  GiveWell conducted a literature review in 2020 to learn more about what proportion of pneumonia deaths are caused by different pathogens. As part of this review, we spoke to Dr. Maria Knoll, a researcher at the Bloomberg School of Public Health, Johns Hopkins University. Dr. Knoll told us:

  • The best available estimates of the contribution of SP and Hib to mortality are available at View-Hub, a data visualization platform for vaccine-treatable diseases (created at Johns Hopkins and supported by Gavi and BMGF).
  • The WHO and member states use View-Hub data rather than IHME data to quantify their SP and Hib burdens.
  • She expressed skepticism about the IHME data, in part because it allegedly does not reflect recent vaccination trends, in contrast to View-Hub data.
  • The model used to generate the SP and Hib data in View-Hub was created specifically for SP and Hib, whereas IHME methods are more general and therefore potentially less accurate for specific conditions.

  The GiveWell literature review concludes: “I recommend using View-Hub estimates. I found Dr. Knoll’s arguments persuasive and they conform to my priors about IHME, although I would feel more confident if I had independent confirmation that the WHO and member states use View-Hub data and that they are typically viewed as superior by other researchers.” (Pg. 3 here).

  We also decided to use View-Hub rather than other studies of pneumonia disease etiology (PERCH and GABRIEL), in part because these studies provide estimates of the etiology of severe pneumonia hospitalization (not mortality).

• 95

  See this write-up for further discussion of how we ought to weight the findings of the PERCH study in our estimates of the proportion of pneumonia deaths that are caused by S. pneumoniae and HiB.

• 96

  We exclude rotavirus from this adjustment because rotavirus was only introduced in Nigeria’s childhood vaccination schedule in 2022 (see Gavi, "Dealing with diarrhoea: Nigeria introduces rotavirus vaccine into its immunisation plan," August 30, 2022), and so we would not expect it to have any impact on the GBD mortality estimates. See this row in our supplementary analysis.

• 97

  This calculation uses the following formula:

  • Mortality among unvaccinated children = Mortality overall / (% of unvaccinated children) + (% vaccinated x % relative risk as a result of vaccination).

  As an example, in Bauchi, we estimate that the vaccine-preventable mortality rate among children who have not been vaccinated is 3.0% / ((100% -30%) + (30% x (100% - 53%)) = ~3.5% (see this row).

  This calculation relies on estimates of vaccine coverage for each state in our analysis. For this, we use vaccine coverage data from the Multiple Indicator Cluster Survey (MICS), a household survey which provides data on vaccine coverage at the state level. This is the same survey that we use in our estimates of New Incentives’ impact on vaccination rates (discussed above). Our estimates are calculated as follows in this section:

  • Since the under-5 mortality data we use comes from GBD 2021, we want our estimate of vaccination coverage for that cohort of children to represent the average vaccination rate across the 5 years preceding the GBD 2021 estimates. We assume that the under 5 population represented by GBD 2021 consists of children born between 2017 and 2021, and we estimate vaccination coverage was increasing by roughly 5% each year over that period (see our calculations here). To roughly approximate average vaccination coverage across that cohort, we estimate vaccination coverage for the midpoint of the cohort (2019). Data collection for MICS 2021 ran from September to December 2021 (See "Fieldwork" dates on p.iii of the MICS 2021 Report). Vaccination coverage was measured for children aged 12-23 months, so we assume the children for whom vaccination coverage was measured were born mostly in 2020. To estimate vaccination coverage for children born in 2019, we back out the 2021 MICS estimates by 5% (since we expect vaccination coverage was increasing by roughly 5% each year from 2017 to 2021). See our calculation here.
  • We adjust the MICS 2021 coverage estimates by -1% for self-report bias, using the same method discussed above.
  • MICS provides data on how many children have received each dose of a given vaccine, whereas our vaccine efficacy estimates are for a full course of that vaccine. To translate dose-specific coverage to overall coverage for vaccines with multiple doses (PCV, Penta, and rotavirus vaccines), we came up with rough estimates of the marginal efficacy of each dose (i.e., how much protection is provided by a first dose compared to a second dose, etc.), based on a quick literature review. Our calculations are in this section.

  As an example, this produces the following estimates for vaccine coverage in Bauchi State:

  • PCV: 37%
  • DTP 37%
  • HiB: 37%
  • Measles: 29%
  • BCG vaccine: 47%
  • Rotavirus: 0%

  A weighted average of these vaccines, using each vaccine’s contribution to the primary benefit of New Incentives’ program (reducing under-five mortality), is 30% in Bauchi. See this row in our cost-effectiveness analysis.

• 98

  See this row in our cost-effectiveness analysis.

• 99

  Our estimate of 0.75 for malaria is a rough best guess, based primarily on evidence that malaria control interventions often have bigger impacts on mortality than their impacts on malaria-related mortality alone would imply. See this section of our report on seasonal malaria chemoprevention for more details.

  • We would guess that non-specific effects would be similar for vaccines because the age profile of the beneficiaries is the same (children under 5) and the mechanism seems similar (i.e., beneficial health effects reduce the likelihood of death from other causes).
  • We'd guess these effects might be somewhat lower for vaccines, since all of these vaccines together are preventing a slightly larger share of overall deaths than SMC (and so there are fewer deaths "available" to prevent indirectly). However, vaccines also prevent a wider range of diseases. As a result, an adjustment of 0.75 indirect deaths seemed appropriate for both programs.

• 100

  See our original cost-effectiveness analysis for New Incentives here.

• 101

  Higgins et al. 2016 finds:

  • The BCG vaccine decreases all-cause mortality by 30% in clinical trials: "BCG vaccine was associated with a reduction in all cause mortality: the average relative risks were 0.70 (95% confidence interval 0.49 to 1.01) from five clinical trials." pg. 1.
  • The measles vaccine decreases all-cause mortality 26% in clinical trials: "Receipt of standard titre MCV was associated with a reduction in all cause mortality (relative risks 0.74, [CI] 0.51 to 1.07) from four clinical trials." pg. 1.
  • The DTP vaccine increases all-cause mortality by 38%, though this is from observational studies that the authors rate as having a high risk of bias: "Receipt of DTP (almost always with oral polio vaccine) was associated with a possible increase in all cause mortality on average (relative risk 1.38, [CI] 0.92 to 2.08) from 10 studies at high risk of bias." pg. 1.

• 102

  Lucero et al. 2009 finds that PCV decreases all-cause mortality by 11%: "Pooled vaccine efficacy (VE) for VT‐IPD was 80% (95% confidence interval (CI) 58% to 90%, P < 0.0001); all serotypes‐IPD, 58% (95% CI 29% to 75%, P = 0.001); World Health Organization X‐ray defined pneumonia was 27% (95% CI 15% to 36%, P < 0.0001); clinical pneumonia, 6% (95% CI 2% to 9%, P = 0.0006); and all‐cause mortality, 11% (95% CI ‐1% to 21%, P = 0.08). Analysis involving HIV‐1 positive children had similar findings." pg. 2.

• 103

  0.74 (for measles) x 0.70 (for BCG) x 0.89 (for PCV) = 0.46.

• 104

  This rough calculation is based on our earlier cost-effectiveness analysis implying a 12% reduction in all-cause mortality, and our rough estimate based on Higgins and Lucero that these vaccines might lead to a 60% reduction in all-cause mortality. (60% - 12%) / 12% = ~4.

• 105

  Further reservations include:

  • Wide confidence intervals. The 95% confidence intervals in Higgins et al. 2016 and Lucero et al. 2009 include a risk reduction on all-cause mortality of 0%.
  • Characteristics of infants in trials may not be representative. Two of the BCG trials were limited to a sub-sample of low-birthweight infants, for whom we would guess this effect is larger than for the average infant in northern Nigeria today.
  • There is high uncertainty about the mechanisms for non-specific effects. Our impression is that the mechanisms driving non-specific effects are poorly understood, which leads us to discount them slightly.
  • There is evidence of a negative effect of DTP on mortality. The negative effect of DTP on mortality suggests that we ought to downweight non-specific effects, though this is based on observational studies with high risk of bias.

  Sources:

  • Low birthweight infants: “We considered four results from cohort studies to be at very high risk of bias and excluded them from meta-analyses. The clinical trial results, including two at low risk of bias in low birthweight infants and two in Native American children in the 1930s and 40s, suggested a beneficial effect of BCG on mortality (average relative risk 0.70, 95% confidence interval 0.49 to 1.01).” Higgins et al. 2016, pg. 4.
  • Evidence of a negative effect of DTP on mortality: “Receipt of DTP (almost always with oral polio vaccine) was associated with a possible increase in all cause mortality on average (relative risk 1.38, 0.92 to 2.08) from 10 studies at high risk of bias; this effect seemed stronger in girls than in boys.” Higgins et al. 2016, abstract.

• 106

  These are the specific diseases targeted by the vaccines that Higgins et al. 2016 and Lucero et al. 2009 find lead to large impacts on all-cause mortality (PCV, the measles vaccine, and the BCG vaccine).

• 107

  We estimated in an earlier version of our analysis that the vaccines incentivized by New Incentives’ program would reduce all-cause mortality by approximately 12%, if they only reduced mortality through the diseases they directly avert (see here). 12% x (1 + 0.75) = ~21%.

• 108

  For example, we estimate the probability of death before age 5 from vaccine-preventable diseases ranges from ~1.3% in Kaduna to ~3.9% in Zamfara. The range for other states in Nigeria is considerably wider. See this row in our cost-effectiveness analysis.

• 109

  For estimates from IGME, see this page.

• 110

  See this row of our cost-effectiveness analysis.

• 111

  Note: We exclude hepatitis B and yellow fever (indirectly incentivized) because IHME estimates show a very low probability of death from these diseases in Nigeria and so we do not think they would make a significant difference to our analysis. See IHME data here.
  We exclude the oral polio vaccine because we expect that there is a very low probability of death from polio in Nigeria today.

  • The last reported case of wild polio virus in Nigeria was in 2016. Global Polio Eradication Initiative, "Polio-free countries".
  • Cases of vaccine-derived polio have been reported in recent years in Nigeria, but we expect that these case numbers are small enough that the average probability of child mortality from polio remains very low.
  • "No cVDPV2 case was reported this week. There have been 16 cVDPV2 cases reported this year and 48 cases in 2022." Global Polio Eradication Initiative, "Where we work: Nigeria", accessed July 26, 2023.

  Instead of explicitly modeling them, we account for the benefits of these vaccines with a rough +5% adjustment elsewhere in our analysis (discussed here).

• 112

  These efficacy numbers correspond to 1 minus the relative risk.

• 113

  IDinsight, New Incentives evaluation, pre-analysis plan, 2019, pg. 2.

• 114

  "Pooled vaccine efficacy (VE) for VT‐IPD was 80% (95% confidence interval (CI) 58% to 90%, P < 0.0001); all serotypes‐IPD, 58% (95% CI 29% to 75%, P = 0.001); World Health Organization X‐ray defined pneumonia was 27% (95% CI 15% to 36%, P < 0.0001); clinical pneumonia, 6% (95% CI 2% to 9%, P = 0.0006); and all‐cause mortality, 11% (95% CI ‐1% to 21%, P = 0.08). Analysis involving HIV‐1 positive children had similar findings." Lucero et al. 2009, pg. 2.

• 115

  World Health Organization, Diphtheria, pertussis, tetanus vaccines information sheet, 2014.

• 116

  Fulton et al. 2016:

  • "Study outcomes were required to be based on the current WHO definition of (1) "typical" pertussis (>14 days of cough with at least one of the following: paroxysmal cough, inspiratory whoop, or posttussive vomiting, in addition to laboratory confirmation); or (2) "severe" pertussis (>21 days of paroxysmal cough with laboratory confirmation of Bordetella pertussis infection, or epidemiological linkage). Studies using less stringent clinical criteria were also included if their laboratory criteria provided a high level of confidence for pertussis infection (eg, positive culture or polymerase chain reaction assay for B. pertussis)." Pg. 1101.
  • "Regional Databases, with no date restrictions, for English-language studies using the following search terms: pertussis, whooping cough, DTwP, DTaP, vaccine, efficacy, morbidity, and mortality." Pg. 1101.
  • "Meta-analysis of the 2 aP vaccine efficacy studies generated a random-effects pooled vaccine efficacy of 84% (95% confidence interval [CI], 81%–87%; P<.00001; Figure 2)." Pg. 1107.
  • “Meta-analysis of 3 wP vaccine effectiveness studies (assessing the Behringwerke, Pasteur/Mérieux, and SmithKline Beecham formulations) yielded an overall wP vaccine effectiveness of 94% (95% CI, 88%–97%) (both I2 = 0%).”
  • We use the effect of the acellular pertussis vaccine in the cost-effectiveness analysis, since it is based on two RCTs and is the effect used in the Lives Saved Tool (LiST); see Table 3, Pg. 1103. We do not use the estimate for whole-cell vaccines (94% efficacy).

  The WHO does not seem to take a stance on using acellular or whole-cell vaccines, though we have not reviewed in depth; see Word Health Organization, Table 2: Summary of WHO Position Papers, Recommended Routine Immunizations for Children, 2023.
  Note: Fulton et al. also reports an estimate for whole cell vaccines of 94% efficacy. We opted to use the more conservative estimate in our analysis, and did not check whether the acellular or whole cell vaccine is used in the Nigerian immunization schedule. We have since learned that many countries use the whole cell vaccine. Because this only makes a small difference to our bottom line, we have not yet checked which type of vaccine is used in Nigeria.
  “Two forms of vaccine are in use, the whole-cell vaccine (wP), and the acellular vaccine (aP). Whole-cell pertussis vaccines were developed first and are suspensions of the entire B. pertussis organism that has been inactivated, usually with formalin…Given that the relative protective efficacy of the best wP and aP vaccines are comparable and the adverse events of both vaccines are relatively minor, wP vaccines remain the vaccine of choice in many developing countries.” World Health Organization, "Pertussis" page.

• 117

  See calculations here.

• 118

  See World Health Organization, Haemophilus influenzae type b (Hib) Vaccination Position Paper – July 2013, pg. 414. “Hib infection and disease start with colonization of the nasopharynx. Following colonization the organism can cause disease either (i) through invasion of the bloodstream with secondary spread to other sites leading to meningitis, pneumonia, and other serious diseases including septic arthritis, osteomyelitis, pericarditis, cellulitis and epiglottitis (referred to collectively as invasive Hib disease) or (ii) through contiguous spread to the paranasal sinuses or the middle ear leading to sinusitis and otitis media.”

• 119

  "Nine randomized studies were included in the analysis. Pooled vaccine efficacy using a fixed effects model against confirmed invasive Hib disease following the 3, 2 and 1 primary dose schedule were 82% [95% confidence interval (CI) 73-87], 79% (95% CI 54–90) and 65% (95% CI 23–84), respectively, and the overall efficacy was 80% (95% CI 72–85)." Thumburu et al. 2015, pg. 31.

• 120

  Note: we only incorporate the impact of the first dose of the measles vaccine in our analysis. This is because the second dose was only added to the Nigerian routine vaccination schedule in 2022. We do not expect the additional dose to have a significant impact on the overall mortality reduction from the program, and so we have not prioritized adding it to our estimates.
  See Figure 1, Sudfeld, Navar, and Halsey 2010, p. I50.

  The same meta-analysis estimates efficacy for two doses of ~98%. This is based on serological and observational studies rather than randomized controlled trials.

  “The effect of a second dose of measles vaccine on measles disease or mortality compared with no vaccination has not been evaluated on individual children in prospective randomized studies as this type of trial would be unethical. Therefore, the best estimate of the effect of a two-dose measles vaccine schedule on measles mortality must be extrapolated from serology data, studies looking at effectiveness of two dose vs one-dose measles vaccination, and observational studies. Caution should be taken when using serology data to estimate the impact on mortality. A recent WHO review of serology studies determined that a median 97% [inter-quartile range (IQR) 87–100%] of children that failed to seroconvert to first dose measles vaccine developed immunity after a second dose.64 If 85% efficacy is assumed for single dose measles vaccine, these serology results would correlate to an efficacy of 99.6% for two dose measles vaccine with a range of 98.1–100% based on the IQR of the review. In addition, the effectiveness of two doses of measles vaccine will vary by setting based on the age of vaccination. Epidemiologic studies comparing the effectiveness of early two dose vaccination vs single dose have found varying results in developing country settings; a study in Niger found two doses (first dose at 6–8 months and second at 9 months) was 23% less effective than single dose whereas studies in India (first dose at 9–12 months and second at 15–18 months) and Guinea Bissau (first dose at 6–8 months and second at 9–12 months) determined two doses of vaccine were respectively 83 and 90% more effective than one dose of measles vaccine.66–69 In order to produce a conservative estimate of the efficacy of two dose measles vaccine per LiST rules, we felt an input of 98% based on the lower quartile of the WHO two dose measles vaccine serology review was reasonable.” Sudfeld, Navar, and Halsey 2010, p. I52.

• 121

  "Diseases prevented: Disseminated disease and meningitis caused by M. tuberculosis." Feikin et al. 2016, Annex 10A, p. 4

• 122

  Mangtani et al. 2014, Figure 5, pg. 478.

• 123

  "Two forms of TB are life threatening: disseminated or miliar disease, and meningitis." Plotkin et al. 2018, pg. 1098.

  Note: Our understanding is that BCG vaccine efficacy wanes significantly over time and offers lower protection to older children and adults. Our understanding is that it still provides significant protection against the most severe forms of TB when given in infancy, but we have not dug into this literature in detail.

  “BCG is usually administered only in infants, immediately after birth, in countries that have a high incidence of TB. The vaccine then produces an early immune response that has been demonstrated to protect children against severe forms of TB. In particular, BCG protects very well against the development of disseminated forms of TB. Usually TB occurs in the lungs, but the bacteria can also be found in other parts of the body – this is called dissemination. In children, the bacteria can be found in the brain – this is called TB meningitis. The BCG vaccine is very effective at protecting against TB meningitis and is a great example of how vaccines can be of huge benefit.
  However, this immunity usually wanes in adolescence and thereafter. Protection by BCG in adults is highly variable – ranging from 0% to 80% depending on the country and environment. The reasons for this remain a mystery and much effort has been placed recently in developing biomarkers that will identify which new vaccines will eventually yield long-lasting immunity. Biomarkers are signals that one can pick up in blood or other clinical specimens that give a predictive sense of whether a vaccine is going to work. If a certain set of signals in blood predicts good protection, we can check if a new vaccine also induces the same set of signals.” Gavi, “TB prevention has relied on the same vaccine for 100 years. It’s time for innovation," 2021.

• 124

  "Diarrhea remains the second leading cause of death around the world for children under 5 years of age. Because the majority of diarrhea deaths occur in the low and middle-income countries, the etiologic agents responsible for diarrhea deaths among young children are unknown. Using hospitalization data as a best estimate of severe diarrheal disease and a proxy for diarrhea mortality, it has been estimated that rotavirus may be responsible for up to 39% of child deaths, the majority of which occur in low and middle income countries." Walker and Black 2011, pg. 1.

• 125

  See Lamberti et al. 2016, Table 1.

• 126

  “The Lives Saved Tool is a mathematical modeling tool which allows users to estimate the impact of coverage change on mortality in low and middle income countries.” LiST, Home page, accessed July 26, 2023. The tool provides meta-analyses as sources for its estimates.
  To obtain the vaccine efficacy estimates above, we accessed LiST’s ‘Explore Data’ tool (available here), and looked at its sources for estimates of vaccine efficacy for children under-five. We used the efficacy estimates cited in its sources for each disease available (see the footnotes for the table above where we cite the meta-analyses available via the LiST tool).

• 127

  An "intention-to-treat" (ITT) analysis includes all RCT subjects originally allocated to receive the intervention, whereas a "per-protocol" analysis is a comparison of treatment groups that includes only subjects who in fact completed the intervention.
  The meta-analyses referenced in LiST include a combination of ITT and per-protocol effects. Where possible, we've used ITT effects.

• 128

  See this section of our analysis.

• 129

  The effect size for measles is based on a combination of RCTs and observational studies, but there is other evidence consistent with the effect size reported from serological studies of measles (see sources below). Our impression (which we have not investigated in detail) is also that measles has been reduced in many countries that have had good vaccination coverage.

  A Cochrane review (Di Pietrantonj et al. 2020) finds an effect of 95% for a single dose at age 9 months to 15 years, though this is also based on cohort studies and includes a much broader age range. We therefore do not put any weight on it in our analysis.

  "The proportion of children who develop protective antibody levels following measles vaccination depends on the presence of inhibitory maternal antibodies and the immunological maturity of the vaccine recipient, as well as on the dose and strain of the vaccine virus as described in detail below. In general, some 85−90% of children develop protective antibody levels when given one dose of MCV at 9 months of age, and 90−95% respond when first vaccinated at 12 months of age. Median MCV effectiveness (i.e. protection from disease) following a single dose of MCV administered at 9−11 months of age was 84% (interquartile range [IQR] 72–95%) across several studies, and increased to 92.5% (IQR 84.8–97%) among children first vaccinated at 12 months or older. Thus most, but not all, children are protected following a single dose of MCV." World Health Organization, Immunological Basis for Immunization Series, Measles, 2020, pg. 12.

• 130

  The study we use for HiB efficacy (Thumburu et al. 2015) uses ITT effects. However, other studies use a combination of ITT and per-protocol effects. The appropriate adjustment would also take into account efficacy achieved through partial vaccination.

• 131

  "Vaccination with the protein-polysaccharide conjugate vaccine, PCV7, has significantly reduced the burden of pneumococcal disease in populations where it is in widespread use and has had an important public health benefit. This vaccine targets only 7 of the more than 92 pneumococcal serotypes, and there have been concerns that the non-vaccine serotypes (NVTs) could increase in prevalence and reduce the benefits of vaccination." Weinberger, Malley, and Lipsitch 2011, pg. 1.

• 132

  For example, we estimate that the diseases targeted by PCV are responsible for 36% of vaccine-preventable deaths among children under age five in Bauchi, compared to 5% for the measles vaccine. See this section of our analysis.

• 133

  Note that this figure does not appear directly in our cost-effectiveness analysis. In our cost-effectiveness analysis, we adjust each vaccine’s efficacy to reflect lower vaccine efficacy in Nigeria (more) and imperfect coverage during the underlying trials (more) in this sheet. We then aggregate them here in our analysis.

  We present the aggregated estimate of unadjusted vaccine efficacy here for simplicity. This change to the order of operations does not make a quantitative difference.

  Note: The variation between states is because different diseases are responsible for varying shares of mortality in each state.

• 134

  See this sheet in our cost-effectiveness analysis for these estimates.

• 135

  The adjustments we apply are:

  • Removing deaths that we think are likely to occur before each vaccine is administered (more).
  • An adjustment for disease etiology (cause of disease) (more).
  • An adjustment for unvaccinated children having higher mortality (more).

  We do not apply the adjustment for all-cause mortality to these estimates because we are unsure how vaccine nonspecific effects vary across vaccines. This is equivalent to assuming nonspecific deaths are the same across vaccines.

• 136

  See this section of our cost-effectiveness analysis.

• 137

  For example, Thumburu et al. 2015 includes studies from Finland, Chile, Gambia, the US, and Indonesia, among others.

• 138

  One piece of evidence for this is that the meta-analysis we rely on to estimate rotavirus efficacy finds efficacy varies widely by region. Efficacy in sub-Saharan Africa (the estimate we use in our analysis) is significantly lower than efficacy in wealthy countries. “Efficacy against severe rotavirus diarrhea ranged from 90.6% [95% confidence interval (CI): 82.3–95.0] in the developed region to 88.4% (95% CI: 67.1–95.9) in Eastern/Southeastern Asia, 79.6% (95% CI: 71.3–85.5) in Latin America and the Caribbean, 50.0% (95% CI: 34.4–61.9) in Southern Asia and 46.1% (95% CI: 29.1–59.1) in sub-Saharan Africa”, Lamberti et al 2016, pg 1.

• 139

  See this row in our cost-effectiveness analysis.

• 140

  This interval was selected by GiveWell staff members familiar with our New Incentives analysis based on their subjective judgment. It does not appear in our cost-effectiveness analysis.

• 141

  We assign the following weights based on our subjective impression of how informative each source is about the likely efficacy of the measles vaccine in Nigeria today:

  • 10% weight to the main LiST measles vaccine efficacy estimate.
  • 20% weight to the results from the IDinsight biomarker pilot.
  • 35% weight to each of Fowotade et al. 2015 and Uzicanin and Zimmerman 2011.

  The weights are in this section of our cost-effectiveness analysis.

• 142

  This figure is a weighted average of the -26% (measles) and -19% (non-measles) adjustments, weighted by each vaccine’s contribution to reducing deaths among children under five. The adjustment varies slightly between states because we think that each vaccine contributes a different amount to the program benefit in each state.

• 143

  See this row in our cost-effectiveness analysis.

• 144

  We have considered two reasons why the biomarker results could be measles-specific, although we haven’t deeply investigated either:

  1. Children in Nigeria could receive the measles vaccine too young. The first dose of measles is given at 9 months in the Nigeria vaccination schedule, compared to 12 - 15 months (for the MMR vaccine) in many high-income countries. This could be too early for them to develop antibodies in response to vaccination.
  2. No measles 2 vaccine: At the time the biomarker pilots were conducted, Nigeria only offered one dose of the measles vaccine (compared to two doses in many high-income countries). We would guess that this is insufficient to induce full immunity for some children.

• 145

  See here.

• 146

  We estimate vaccine efficacy as a function of (i) the test sensitivity (the proportion of children with antibodies correctly identified by the test), (ii) the proportion of children falsely reported to be vaccinated, (iii) the proportion of children who were immune before the test, because of a previous measles infection, and (iv) the proportion of children testing positive.

  Our calculations are in this supplementary spreadsheet. We use the following assumptions:

  • Test sensitivity: we use a reported figure that the test sensitivity is 86%. We do not have permission to share the source for this estimate.
  • False report: we roughly guess that 15% of children are falsely reported to be vaccinated when they were not.
  • Prior immunity: we roughly estimate that 5% of children had a prior measles infection. This is a rough estimate, based on a number of sources, discussed here.
  • Rate testing positive: we use the pilot’s finding that 22% of children tested positive.

  These assumptions imply a vaccine efficacy of 26% (see calculation in this row).
  Note: Our analysis assumes that vaccine efficacy is linearly correlated with the proportion of children developing antibodies. We have not deeply investigated this assumption.

• 147

  "This study was designed to assess the seroconversion rate of measles vaccine among infants receiving measles immunization in Ilorin, Nigeria. . . . Only 286 (71.5%) of the vaccines returned to give post-vaccination samples. All the infants screened had low pre-vaccination measles antibody titers. Thirty one (8.0%) of the infants had measles prior to vaccination. The seroconversion pattern showed that 196 (68.6%) of the infants developed protective antibody titers." Fowotade et al. 2015, Abstract

• 148

  "This study was designed to assess the seroconversion rate of measles vaccine among infants receiving measles immunization in Ilorin, Nigeria. . . . Only 286 (71.5%) of the vaccines returned to give post-vaccination samples. All the infants screened had low pre-vaccination measles antibody titers. Thirty one (8.0%) of the infants had measles prior to vaccination. The seroconversion pattern showed that 196 (68.6%) of the infants developed protective antibody titers." Fowotade et al. 2015, Abstract

• 149

  "The loss in vaccine virus titers in our centre may be attributed partly to repeated thawing and freezing due to erratic power supply and the absence of standby power generating set serving the immunization clinic where the vaccines are stored. Another contributory factor is the absence of a refrigerator thermometer to ensure that the correct storage temperature is maintained. The low potency found has been the usual trend in potency studies carried out by various researchers in Nigeria. This finding is however different from vaccine studies carried out in other parts of the world. Techatharyat et al. in Thailand and Saha et al. in India reported measles vaccine potency test results of 100.0% and 95.0% respectively. Other adverse factors such as poor handling by vaccinators, existence of chains of salesmen, lack of good storage system for vaccines and difficulty in maintaining a cold chain system have also been suggested to be responsible for the loss in potency of vaccines used in Nigeria and other African countries." Fowotade et al. 2015, Discussion.

• 150

  "These vaccines were obtained from United Nations International Children’s Emergency Fund (UNICEF), through the State EPI Unit and were stored in the freezer until required for immunization. These lyophilized Ruvax vaccines were maintained in cold chains and reconstituted each time per vial according to manufacturer’s instruction." Fowotade et al. 2015, Materials and Methods.

• 151

  See this row in our cost-effectiveness analysis.

• 152
  • “One hundred and twenty-two out of 130 children (93.9%) who had received DPT had protective levels of anti-tetanus IgG compared to 278 out of 288 children (96.5%) who had received the pentavalent vaccine… DPT and pentavalent vaccines are equally effective in inducing protective levels of anti-tetanus IgG in children”, Uket et al. 2018, abstract.
  • “Four hundred and eighteen children participated in the study. The mean IgG antibody level was 1.021 ± 0.9 IU/ml. Four hundred children (95.7%) had protective levels of antibodies”, Ekanem, Uket, and Okpara 2018, abstract.

• 153
  • “The prevalence of BCG scar was 79.5% among the male infants and 84.7% among the female infants, while the overall prevalence of BCG scar was 81.5%”, Gambo et al. 2014, results.
  • “Two hundred and six subjects (96.3%) had a post-vaccination BCG scar”, Atimati and Osarogiagbon 2014, abstract.
  • “84.2% had physical evidence of BCG inoculation”, Odujinrin and Ogunmekan 1992, abstract.

• 154

  “Only one‑third (49/138) of those vaccinated had identifiable BCG scars”, Orogade et al. 2013, abstract.

• 155
  • “Of the vaccinated children, 157 (57.5%) developed protective measles virus HI antibody, which is not enough to maintain protective herd immunity”, Onoja and Adeniji 2013, abstract
  • "A total of 150 (52.8%) of the 284 children who received measles vaccine returned for post-vaccination screening, and of these, 82 (54.7%) seroconverted (titres, 10-320) following vaccination; 68 (45.3%) did not seroconvert (Table 1)." Adu et al. 1992, pg. 458.
  • “The seroconversion pattern showed that 51(60%) had potent antibody titres ranging from 1:40 to 1:1280, while the remaining 34 (40%) had a low antibody titres between < 1:20 and 1:20”, Omilabu et al. 1999, abstract.

• 156

  Many have small sample sizes and many are published in less well-recognized journals.

• 157
  • Uzicanin and Zimmerman 2011 find a median vaccine efficacy for individuals who were 9-11 months old at age of first dose of 77%, with vaccine efficacy ranging from 73% in the WHO Africa region to 96% in the WHO European region (Table 2, pg. 145).
  • "Results of this literature review suggest that the VE of MCV1 administered at 9–11 months of age is generally lower than 85%, which is the usual expected rate of immune response after vaccination at that age." Uzicanin and Zimmerman 2011, pg. 135.
  • "Generally, the reasons related to low VE estimates can be grouped into 3 broad categories, including (1) issues related to study methods; (2) program-related factors, such as appropriate vaccine storage, handling, and administration; and (3) host-related factors, most notably, age at vaccination." Uzicanin and Zimmerman 2011, pg. 144.

• 158

  BCG vaccine: we reviewed Pimpin et al. 2013, which notes that the BCG vaccine is less effective closer to the equator. It is not clear to us whether this result applies to BCG vaccine administered at birth and to the BCG vaccine's effects on fatal forms of tuberculosis.

  “In general, the protective effect of BCG vaccination was either absent or low in studies conducted close to the equator, whereas there was reasonably consistent evidence of good protection observed in studies conducted at latitudes exceeding 40°. Relatively high protection was observed in studies (Saskatchewan Infants and MRC) conducted above 50° latitude: rate ratio 0.22 (95% CI 0.16 to 0.30), corresponding to a VE of 78% (95% CI 70% to 84%). Latitude explained a substantial amount of the between-study variation in the protective effect of BCG vaccination." Pimpin et al. 2013, p. 24.

  "Because of the evidence that BCG protects against miliary and meningeal tuberculosis, in developing countries BCG vaccination is recommended at birth (or first contact with health services), taking into account HIV status. Our systematic review suggests that BCG also confers protection against pulmonary disease, the greatest burden from tuberculosis, when administered both in infancy and at school age, providing that children are not already infected with M tuberculosis or sensitised to other mycobacterial infections. Protection against pulmonary disease was seen in the Bombay Infants trial suggesting that, even close to the equator, if BCG is administered prior to exposure to tuberculosis and environmental mycobacteria it can provide significant protection. Further evidence of protection in populations close to the equator from BCG given before infection would strengthen these findings." Mangtani et al. 2014, p. 479.

  PCV: The two African trials in a meta-analysis of the effects of PCV we reviewed, Lucero et. al. 2009, have smaller effects than trials in other settings, though the difference does not appear to be statistically significant, and it is not clear what is causing this difference.

  • Cutts et al. 2005 in Gambia: risk ratio of 0.57 [0.38, 0.85].
  • Klugman et al. 2003 in South Africa: risk ratio of 0.58 [0.26, 1.31].

  The overall risk ratio is 0.42 [95% CI 0.25, 0.71]. Effects are lower in the American Indian trial (O'Brien 2003) (0.48), Gambia (Cutts 2005), South Africa (Klugman 2003), and the Philippines (Lucero 2009) (1.00), and highest in Finland (Eskola 2001 and Klipi 2003) and the US (Black 2000) (0.11-0.33). (See Figure 4, p. 15). It is not clear to us why these effects vary. The commentary does not explicitly mention changes varying by distance from equator or lower impact in low- and middle-income countries.

• 159

  The 95% guess appears here in our cost-effectiveness analysis.

• 160

  This appears as a 100% ratio between vaccine-preventable disease and vaccine-preventable mortality in our cost-effectiveness analysis. See this row.

• 161

  See this section in our cost-effectiveness analysis.

• 162

  This excludes our supplemental adjustments discussed in section 4.6. See this row in our cost-effectiveness analysis.

• 163

  For ease of modeling, we quantify our uncertainty ranges for benefits other than mortalities averted among people under age 5 as percentage adjustments applied to other modeled benefits of the program. For Bauchi:

  • Our 25th to 75th percentile range for mortalities averted after age 5 is an adjustment ranging from +6% to +28% (with the percentage adjustment applied to the modeled benefits of averted mortality for children under five). See this row in the “Sensitivity analysis” section of our cost-effectiveness analysis.
  • This roughly translates to between 4% and 18% of the total modeled benefits of the program, as shown in the table above.
  • This conversion is imperfect because varying the proportion of the program's total impact that is contributed by one type of benefit will also affect the proportion of impact contributed by other types of benefits, but we think the values presented in the table are a reasonably good approximation of our uncertainty level for this parameter.

• 164

  Note: We use a similar method to the one described above to estimate vaccine-preventable mortality among unvaccinated children (as opposed to all children). This method incorporates estimates of vaccine coverage among this age group during the time period in which this age group would have been scheduled to receive child vaccinations. To roughly estimate this, we assume that people aged 5 - 14 were 50% less likely to have been vaccinated than children under age 5 today, 15 - 49 year olds were 80% less likely to have been vaccinated, and 50 - 74 year olds were 90% less likely to have been vaccinated. See this section in our cost-effectiveness analysis.

• 165

  A discount rate of 0.5% implies that we discount the value of averting a death by 0.5% if it happens one year from now, compared to if it happens now. The purpose of this discount rate is to account for uncertainty about future events, for example, possible fundamental changes which would render the program ineffective. The discount rate compounds, so deaths averted far in the future are valued less than deaths averted in the near future. See this document for more details on discount rates.

• 166

  Our method for 15 - 49 year olds and 50 - 74 year olds is the same, but we apply larger discounts for reduced mortality and vaccine efficacy in the future.

• 167

  To reach this figure, we also apply our discount rate. This slightly reduces the effective number of deaths we think will be averted in the future, because each death counts for less than if it was averted now. See this row of our cost-effectiveness analysis for our calculation.

• 168

  This calculation takes into account both the numbers of deaths averted in each age group and GiveWell’s moral weights for averting a death in each age group (we assign more weight to averting the deaths of children than adults).

• 169

  See this row in our cost-effectiveness analysis. Note that this estimate doesn’t account for the benefits we incorporate as rough supplemental adjustments, and so can’t be interpreted as an estimate of the total benefits of the program.

• 170

  For ease of modeling, we quantify our uncertainty ranges for benefits other than mortalities averted among people under age 5 as percentage adjustments applied to other modeled benefits of the program. For Bauchi:

  • Our 25th to 75th percentile range for income effects is an adjustment ranging from +6% to +64% (with the percentage adjustment applied to the total modeled benefits of averted mortality). See the "Adjustment to account for developmental benefits (long-term income increases)"row on the 'Sensitivity analysis' sheet of our cost-effectiveness analysis.
  • This roughly translates to between 4% and 28% of the total modeled benefits of the program, as shown in the table above.
  • This conversion is imperfect because varying the proportion of the program's total impact that is contributed by one type of benefit will also affect the proportion of impact contributed by other types of benefits, but we think the values presented in the table are a reasonably good approximation of our uncertainty level for this parameter.

• 171

  Our estimate of the development effects of deworming programs is based on long-run follow-ups to the experiment described in Miguel and Kremer 2004. See our report on mass deworming programs for our full assessment of the study. For malaria programs, we rely on the findings of two natural experiments: Bleakley 2010 and Cutler 2010. See this section of our report on mass distribution of insecticide-treated nets (ITNs) for a detailed discussion of these studies.

• 172

  “This study uses the malaria-eradication campaigns in the United States (circa 1920), and in Brazil, Colombia and Mexico (circa 1955) to measure how much childhood exposure to malaria depresses labor productivity. The campaigns began because of advances in health technology, which mitigates concerns about reverse causality. Malarious areas saw large drops in the disease thereafter. Relative to non-malarious areas, cohorts born after eradication had higher income as adults than the preceding generation. These cross-cohort changes coincided with childhood exposure to the campaigns rather than to pre-existing trends. Estimates suggest a substantial, though not predominant, role for malaria in explaining cross-region differences in income.” Bleakley 2010, abstract.

  “We examine the effects of exposure to malaria in early childhood on educational attainment and economic status in adulthood by exploiting geographic variation in malaria prevalence in India prior to a nationwide eradication program in the 1950s. We find that the program led to modest increases in household per capita consumption for prime age men, and the effects for men are larger than those for women in most specifications. We find no evidence of increased educational attainment for men and mixed evidence for women.” Cutler 2010, abstract.

• 173

  See this section of our report on insecticide-treated nets for more detail on these estimates.

• 174

  See this row in our cost-effectiveness analysis.

• 175

  See this row in our cost-effectiveness analysis.

• 176

  For the income increase per household member, see here in our cost-effectiveness analysis. For the proportion of benefits this makes up, see this row.

• 177

  For ease of modeling, we quantify our uncertainty ranges for benefits other than mortalities averted among people under age 5 as percentage adjustments applied to other modeled benefits of the program. For Bauchi:

  • Our 25th to 75th percentile range for income effects is an adjustment ranging from +2% to +7% (with the percentage adjustment applied to all the other modeled benefits of the program, including averted mortality and long-term income effects). See this row in the “Sensitivity analysis” section of our cost-effectiveness analysis.
  • This roughly translates to between 2% and 6% of the total modeled benefits of the program, as shown in the table above.
  • This conversion is imperfect because varying the proportion of the program's total impact that is contributed by one type of benefit will also affect the proportion of impact contributed by other types of benefits, but we think the values presented in the table are a reasonably good approximation of our uncertainty level for this parameter.

• 178

  To compare cost-effectiveness across different programs, we use "moral weights" to quantify the benefits of different program impacts (e.g., increased income versus reduced deaths). We benchmark the value of each benefit to a unit value of 1, which we define as the value of doubling someone’s consumption for one year. For more on our approach to moral weights, see this document.

  We model units of value for income increases in terms of the natural log of consumption. The logarithmic model captures the idea that money has diminishing value as you get more and more of it. For example, our model considers a 50% increase in income as a little better than 50% as good as an 100% increase in income. Using this model, a $1.34 increase in income from a $286 baseline equates to 0.01 units of value. See this row in our cost-effectiveness model.

• 179

  New Incentives has made a number of changes to its incentive schedule over time, from the original 4,000 naira incentive offered during the RCT:

  • New Incentives originally added a 500 naira incentive for measles 2 (which had not yet been introduced into Nigeria's routine vaccination schedule at the time of the RCT). This brought the total incentive across all visits to 4,500 naira. In early 2023, New Incentives increased the incentive for measles 2 to 1,000 naira. This brought the total incentive across all visits to 5,000 naira.
  • In August 2023, New Incentives decided to change its schedule to offer 1,000 naira per visit. This brought the total incentive across all visits to 6,000 naira.
  • In April 2024, New Incentives decided to pilot offering an additional 5,000 naira incentive at the measles 2 vaccination visit for caregivers whose infants completed the entire routine immunization schedule. As of May 2024, New Incentives had begun piloting this additional incentive in a few clinics, and expected to roll it out across the program later in 2024, conditional on the pilot proceeding well. The calculations in our cost-effectiveness analysis assume that this incentive will be offered in all states where New Incentives operates.

  New Incentives, January 14, 2022; December 16, 2022; September 8, 2023; May 13, 2024 Program Updates (unpublished).

• 180

  We calculate this as follows:

  Average per visit transportation cost among caregivers who reported paying for transportation in 2023: 149 naira (one-way cost) x 2 = 298 naira round trip cost
  Percentage of disbursements for which caregivers reported no transportation cost in 2023: 68%

  Average total transportation cost across the full routine immunization schedule: (298 naira x 6 visits) x (1 - 68%) = 577 naira

  See the 2023 transportation cost data here.
  

• 181

  See this spreadsheet for our calculations (underlying data from StatCompiler available here). The states we include in this average are: Zamfara, Katsina, Jigawa, Sokoto, Bauchi, Kaduna, Gombe, Kano, and Kebbi. For simplicity and because these estimates make such a small difference to our bottom line, we use a cross-state average rather than a state-specific estimate.

• 182

  The calculation behind this value can be found in the "Cash" section of the "Parameters" tab at this link. Our analysis is based on Haushofer and Shapiro 2013, an RCT of GiveDirectly’s program. In Haushofer and Shapiro 2013, non-durable expenditure in the control group is measured as USD 157.40 PPP per month (p. 49). We multiply this value by 12 to put it in annual terms, converted to nominal USD, and divided by the average household size.

• 183

  New Incentives’ monitoring data is available here. In 2023, New Incentives estimated that retention rates varied between 75% and 96%, depending on the specific vaccine. These rates were similar to those found in the RCT.

• 184

  The World Bank reports estimates in 2022 that Nigeria’s GDP per capita was $2,184 and Kenya’s was $2,099 (estimates available here).

• 185

  Our understanding is that there is some opposition to New Incentives’ program among some individuals in the Nigerian federal government and that concerns about sustainability and potential backlash effects are parts of this concern. See this section of our separate page on New Incentives’ program for more details. New Incentives also reports hearing this concern from other partners.
  “A concern . . . stakeholders (including UNICEF representatives) have shared with us is about incentives replacing intrinsic motivations to immunize infants in the absence of incentives. This was experienced by WHO when some women in Northern Nigeria started refusing vaccinations if they didn't get the in-kind donations that were previously being offered during Polio campaigns.” New Incentives, Responses to questions from GiveWell, September 23, 2020 (unpublished)

• 186

  New Incentives field officers identify incidents of low supply (e.g. due to stockouts or runouts of vaccines during immunization days), and report them to the relevant stakeholders responsible for ensuring vaccine supply. New Incentives also proactively requests additional vaccines and regularly engages with stakeholders to share information on and close supply gaps. In addition, New Incentives conducts awareness-raising activities, such as advertising the program at partner clinics, holding community meetings to talk about the benefits of vaccines, and running radio advertisements for the program. More on these aspects of its work here and here on our separate page about New Incentives' program.

• 187
  • “A concern . . . stakeholders (including UNICEF representatives) have shared with us is about incentives replacing intrinsic motivations to immunize infants in the absence of incentives. This was experienced by WHO when some women in Northern Nigeria started refusing vaccinations if they didn't get the in-kind donations that were previously being offered during Polio campaigns.” New Incentives, Responses to questions from GiveWell, September 23, 2020 (unpublished).
  • While we view these reports as concerning, we have not identified evidence in support of systematic negative effects from removal of the polio vaccine incentives. New Incentives also told us it believes incentives play a different role in its program than they did in the polio vaccination campaign, since campaigns provide vaccinations door-to-door (and thus create limited costs for caregivers), while New Incentives incentivizes vaccinations at clinics (which have higher time and transport costs for caregivers).
    • “Note: we have received conflicting reports and opinions regarding in-kind incentives during Polio campaigns. CCTs for Routine immunization are different from offering in-kind incentives or ‘pluses’ during Polio campaigns where caretakers do not bear any cost to vaccinate their children with an oral vaccine at their home. We take concerns around intrinsic motivation and perverse incentives very seriously. We do not think these are unique to the application of CCTs to immunizations as similar concerns exist for application of CCTs to other health and non-health behaviors. We have incorporated messaging in various components of the program to help caretakers understand that vaccinations are important to protect their children against deadly diseases.” New Incentives, Responses to questions from GiveWell, September 23, 2020 (unpublished).
  • We would guess that the incentives in the polio vaccination campaign were more likely to crowd out intrinsic motivation or generate distrust in the healthcare system than the incentives offered by New Incentives' program.
    • We think it’s plausible that incentives viewed more as external rewards would be more likely to crowd out intrinsic motivation. In New Incentives’ context, caregivers bring their children in to clinics to receive their vaccinations. We think the incentives offset some of the time and transportation costs, and thus may support caregivers’ desires to bring their children in for vaccinations. In contrast, because the polio vaccine was provided in a more convenient way (door-to-door), the incentives may have been seen as an external reward that focused attention away from caregivers’ intrinsic motivation to vaccinate their child.

• 188

  “Demand-Side Incentives (DSIs) have primarily been used to address inter-generational poverty (including through incentivizing well-child and education programs) but have also shown promising results in increasing demand and coverage of specific health services, including vaccination. Conditional Cash Transfers (CCTs) and Conditional Non-Cash Transfers (CTs) are DSI approaches which provide beneficiaries with money or valuable goods for accomplishing a task or meeting specified requirements. In the context of immunization programs, CCT’s and CT’s can be applied by granting caregivers financial or non- financial incentives for immunizing their children.” Gavi, "Request for proposal (RFP) Demand-Side Incentives to Address Immunization Challenges", 2023, p. 5.

• 189

  Onwujekwe et al. 2020 and Ezenwaka et al. 2021 were non-randomized studies that found mixed effects post-program (a persistent increase in public health facility use roughly 2 years after withdrawal with some evidence of decreased trust in the healthcare system).

  • Onwujekwe et al. found that in the group which received CCTs, public health facility use remained higher 2 years after the withdrawal of the program, indicating a durable effect of cash incentives. There did not appear to be a persistent effect in the group which received the SURE-P program but not CCTs.
    • “A second issue concerns the extent to which the large up-front cost can be considered not only as a way of encouraging current use of skilled delivery services but as an investment that generates a longer term impact on health seeking behavior going well-beyond the end of the programme. The timing of the study allows us to examine what happened after the ending of funding for the intervention. Information on use of services 2 years beyond the end of the programme suggest that the effect of the intervention may have continued well-beyond the period of funding. The investment in the supply side into equipment and the facility was not immediately lost when the programme closed so potentially explaining the continued effect. It is notable, however, that the persistent effect is largely evident in the CCT areas and not the SP areas which also received the supply side investment.” Onwujekwe et al 2020.
    • “This finding is supported by a community survey undertaken as part of the wider realistic evaluation of the SURE-P programme: 70% of women living in CCT areas said that their trust in public services increased as a result of the introduction of CCTs compared to 49% in SURE-P only areas (15). Following withdrawal of CCTs, the majority of those women in those areas (62%) reported continued high levels of trust. This is reinforced by patterns of delivery which suggested that there was no statistical difference in the proportion of women reporting using a public facility during and after the CCT programme (69 ± 6.4% before compared to 64 ± 7.8% after).” Onwujekwe et al 2020.
  • Ezenwaka et al. also reported qualitative evidence of reduced trust in the healthcare system.
    • “Results: The CCTs contributed to increasing facility attendance and utilization of MCH services by reducing the financial barrier to accessing healthcare among pregnant women. However, there were unintended consequences of CCT which included a reduction in birth spacing intervals, and a reduction of trust in the health system when the CCT was suddenly withdrawn by the government.” Ezenwaka et al. 2021, abstract.

• 190

  Studies of cash incentives for non-vaccination health behaviors in low- and middle-income countries found no evidence of negative post-program effects, or evidence of positive health behaviors persisting after incentives were taken away. Studies we’ve reviewed:

  • McCoy et al. 2017, an RCT of ~800 participants estimating the effect of CCTs and food transfers incentivizing antiretroviral therapy adherence in Tanzania. The primary outcome was the percentage of participants who possessed antiretrovirals for over 95% of the time they were tested (medication possession ratio (MPR) ≥95%). It found that the program had a ~22 percentage point effect at the end of the intervention and roughly 20 percentage points six months after that.
    • "At three clinics, 805 participants were randomized to three groups in a 3:3:1 ratio, stratified by site: nutrition assessment and counseling (NAC) plus cash transfers (~$11/month, n=347), NAC plus food baskets (n=345), and NAC-only (comparison group, n=113, clinicaltrials.gov NCT01957917)" (abstract)
    • "Cash or food was provided for 6 or less consecutive months, conditional on visit attendance. The primary outcome was medication possession ratio (MPR ≥ 95%) at 6 months." (abstract)
    • "The primary intent-to-treat analysis included 800 participants. Achievement of MPR≥95% at 6 months was higher in the NAC+cash group compared to NAC-only (85.0% vs. 63.4%), a 21.6 percentage point difference (95% confidence interval (CI): 9.8, 33.4, p<0.01). MPR≥95% was also significantly higher in the NAC+food group versus NAC-only (difference=15.8, 95% CI: 3.8, 27.9, p<0.01). When directly compared, MPR≥5% was similar in the NAC+cash and NAC+food groups (difference=5.7, 95% CI: −1.2, 12.7, p=0.15). Compared to NAC-only, appointment attendance and LTFU were significantly higher in both the NAC+cash and NAC+food groups at 6 months. At 12 months, the effect of NAC+cash, but not NAC+food, on MPR≥95% and retention was sustained.” (abstract)
    • "At 12 months, MPR≥95% was significantly greater among those in the NAC+cash group than the NAC-only group (74.9% vs. 55.4%, adjusted difference=20.3, 95% CI: 8.4, 32.2, p<0.01) in both ITT and adjusted analyses." (p. 7)
      • 74.9% - 55.4% = roughly a 20 percentage point increase in MPR≥95% for the NAC+cash group compared to the NAC-only group.
  • de Walque, Dow, and Nathan 2014, an RCT including ~2400 participants, estimating the effect of CCTs incentivizing safe sexual practices in Tanzania. It found a 25% decrease in sexually transmitted infections for a $20 transfer after 12 months. A follow-up at 24 months found a decrease of approximately 20%, although this was not statistically significant.
    • “The Rewarding Sexually Transmitted Infection Prevention and Control in Tanzania (RESPECT) study is a randomized controlled trial testing the hypothesis that a system of rapid feedback and positive reinforcement that uses cash as the primary incentive can be used to reduce risky sexual activity among young people, male and female, who are at high risk of HIV infection. The study enrolled 2,399 participants in 10 villages in rural southwest Tanzania.” (abstract)
    • "The intervention arm was further divided into two subgroups, one receiving a high value payment of up to $60 over the course of the study ($20 payments every four months)" (abstract)
    • "at month 12 (column 3) the high value CCT arm corresponds to a 25% risk reduction compared to the control group (relative risk of 0.749 statistically lower than 1)" (p. 13)
    • See Table 3, p.28: “Month 12 combined prevalence of 4 STIs tested at every round: 0.749** and Table 3, p.28: “Month 24 combined prevalence of 4 STIs tested at every round:0.798” (pg. 28).
      • "*** p<0.01, ** p<0.05, * p<0.1 denote a Relative Risk significantly different from 1." Table 3, pg. 28.
  • Fahey et al. 2021, a follow-up to a trial of cash or food transfers for monthly healthcare appointments for adults starting HIV treatment in Tanzania. The study found no statistically significant differences in retention in care between groups 24 and 36 months after the program ended, and a non-significant trend in favor of the intervention group after 36 months, indicating no evidence of crowding out.
    • “We traced former participants in a 2013–2016 trial, which randomised 800 food-insecure adults starting HIV treatment at three clinics to receive either usual care (control) or up to 6 months of cash or food transfers (~US$11/month) contingent on timely attendance at monthly clinic appointments. The primary intention-to-treat analysis estimated 24-month and 36-month marginal risk differences (RD) between incentive and control groups for retention in care and all-cause mortality, using multiple imputation for a minority of missing outcomes. We also estimated mortality HRs from time-stratified Cox regression." (abstract)
    • "From 3 March 2018 to 19 September 2019, we determined 36-month retention and mortality statuses for 737 (92%) and 700 (88%) participants, respectively. Overall, approximately 660 (83%) participants were in care at 36 months while 43 (5%) had died. There were no differences between groups in retention at 24 months (86.5% intervention vs 84.4% control, RD 2.1, 95% CI −5.2 to 9.3) or 36 months (83.3% vs 77.8%, RD 5.6, –2.7 to 13.8), nor in mortality at either time point. The intervention group had a lower rate of death during the first 18 months (HR 0.27, 95% CI 0.10 to 0.74); mortality was similar thereafter (HR 1.13, 95% CI 0.33 to 3.79).” (abstract).
  • Czaicki et al. 2018, a trial comparing food and cash incentives for healthcare appointment visits for HIV-positive adults in Tanzania against standard of care. The study found no difference in intrinsic motivation between groups (as measured by the Treatment Self-Regulation Questionnaire (TSRQ)) at baseline, endline, and approximately six months after the program ended.
    • “We analyzed data from 469 individuals randomized to one of three study arms: standard of care, short-term cash transfers, or short-term food assistance. Eligible participants were: 1) ≥18 years old; 2) HIV-infected; 3) food insecure; and 4) initiated antiretroviral therapy (ART) ≤90 days before the study. Food or cash transfers, valued at ~$11 per month and conditional on attending clinic visits, were provided for ≤6 months. Intrinsic motivation was measured at baseline, 6, and 12 months using the autonomous motivation section of the Treatment Self-Regulation Questionnaire (TSRQ). We compared the change in TSRQ score from baseline to 6 and 12 months and the change within study arms." (abstract)
    • The mean intrinsic motivation score was 2.79 at baseline (range: 1–3), 2.91 at 6 months (range: 1–3), and 2.95 at 12 months (range: 2–3), which was 6 months after the incentives had ended. Among all patients, the intrinsic motivation score increased by 0.13 points at 6 months (95% CI (0.09, 0.17), Cohen’s d = 0.29) and 0.19 points at 12 months (95% CI (0.14, 0.24), Cohen’s d = 0.49). Intrinsic motivation also increased within each study group at 6 months: 0.15 points in the food arm (95% CI (0.09, 0.21), Cohen’s d = 0.37), 0.11 points in the cash arm (95% CI (0.05, 0.18), Cohen’s d = 0.25), and 0.08 points in the comparison arm (95% CI (-0.03, 0.19), Cohen’s d = 0.21); findings were similar at 12 months. Increases in motivation were statistically similar between arms at 6 and 12 months.” (abstract)

• 191

  “The M-SIMU study was a cluster-randomized controlled trial conducted in Gem and Rarieda sub-counties, Siaya County, Kenya in 2013-2015. The M-SIMU study evaluated the impact SMS reminders alone or coupled with KES 75 (~US$0.88 in August 2015) or KES 200 (~US $2.35 in August 2015) mobile phone-delivered incentives on vaccination coverage and timeliness. The M-SIMU study enrolled caregivers and infants aged 0-4 weeks from 152 within the Kenya Medical Research Institute (KEMRI) Health and Demographic Surveillance System (HDSS).27 Kagucia 2018 (p. 296)

• 192

  “For the three intervention groups, SMS reminders were sent three days and the day before scheduled immunisation visits at ages 6 weeks, 10 weeks, and 14 weeks for the three doses of pentavalent vaccine and age 9 months for measles vaccine using the free and open-source. RapidSMS platform.” Gibson et al. 2017, p. e430

• 193

  “In addition to receiving SMS reminders, caregivers were sent either 75 KES (group 3) or 200 KES (group 4) to
  their mobile phone for each timely dose of pentavalent and measles vaccine received, defined as vaccination
  within 2 weeks of the Expanded Programme on Immunisations (EPI) scheduled date (ie, pentavalent1 at 6 weeks, pentavalent2 and pentavalent3 4 weeks after the previous pentavalent dose, and measles at 9 months).26
  Incentives were sent to participant’s mobile phones using a mobile-money programme22 and through their
  preferred mobile network (eg, Safaricom, Airtel). Gibson et al. 2017, p. e430

• 194

  Gibson et al. 2017, Table 3.
  SMS plus 75 KES results for timely measles completion: RR: 1.37 (1.19–1.59)
  SMS plus 200 KES results for timely measles completion: RR: 1.42 (1.23–1.63)

• 195

  "Post-trial follow-up visits were conducted between August 4, 2018 and November 30, 2018. CIs visited 1,467 of 1,599 M-SIMU households (91.7%)." Kagucia 2018 (p. 318)

• 196

  “132 M-SIMU households were not visited as funding for the study ended before the households could be visited. Of the households visited, 218 (14.9%) follow-up visits were completed. Follow-up visits were not conducted for the remaining households either because they did not meet MSBC eligibility criteria (n= 1,000; 68.2%) or were lost to follow-up (n= 249; 17.0%). Of 1,000 caregivers not meeting MSBC eligibility criteria, 711 (71.1%) did not have a child born after M-SIMU, 203 (20.3%) had a SC younger than age 10 months, 62 (6.2%) had a SC but were enrolled in the M-SIMI study and 24 (2.4%) did not have an MCH booklet for the SC. " Kagucia 2018 (p. 318)

  In total 914 (711 + 203) out of the original 1,599 households were excluded due to lack of an eligible sibling at the time of follow-up. 57% = 914/1,599

• 197

  “The primary objective was to assess whether first dose measles containing vaccine (MCV1) timely coverage, i.e., the proportion of children vaccinated at or before age 9 months and 2 weeks (9m+2w) was significantly different among SC of SMS+200KES caregivers compared to SC of Control caregivers.” Kagucia 2018 (p. 289)

• 198

  “This translated to 18.1% lower MCV1 timely coverage among SMS+200KES SC compared to Control SC that was not statistically significant (95% CI: -39.3%, 3.1%; p= 0.095)” Kagucia 2018 (p.322)

• 199

  “However, in the period after the M-SIMU study, MCV1 timely coverage was significantly lower by 21.7% among children whose caregivers were previously enrolled in the M-SIMU SMS+75KES arm compared to those whose caregivers were in the M-SIMU Control arm (95% CI: -41.9%, -1.6%; p= 0.035).” Kagucia 2018 (p.322)

• 200

  With a sample size of 1,278 households in the follow-up, the study estimates that it would have been powered to detect an absolute difference of ≥18% in MCV1 timely completion rates between the SMS+200KES and control group included. The follow-up ended up only including 218 households, so we expect that it was underpowered for even fairly large effect sizes above 18%. We have not vetted the original power calculations or calculated power calculations with the smaller sample size.
  “We also estimated the effect size able to be detected with a sample size of 1,278. In the effect size estimation we assumed: a type 1 error (alpha) of 5%; power (1-beta) of 80%; 50.8% measles coverage among SC of M-SIMU Control arm caregivers at age 9m+2w (data from M-SIMU); at least 157 SC enrolled per M-SIMU caregiver study arm; 38 clusters (villages) per M-SIMU caregiver study arm; an average cluster size of four SC; and a coefficient of variation (CV) equal to 0.089 which was the CV observed in the M-SIMU study.6 Based on these assumptions and accounting for clustering, the study was powered to detect an absolute difference of ≥18% in the proportion of SC born to M-SIMU SMS+200KES arm caregivers receiving MCV1 by age 9m+2w (the primary objective’s time cutoff) compared to those of M-SIMU Control arm caregivers.” Kagucia 2018, p. 306.

• 201

  “Compared to Control SC, MCV1 overall coverage was 20.8% lower and 6.2% lower among SMS+200KES SC and SMS only SC, respectively, however these differences were not significant (SMS+200KES aRD 95% CI: -42.1%, 0.6%; p= 0.057 and SMS only aRD 95% CI: -22.2%, 9.8%: p= 0.451). MCV1 overall coverage was significantly lower by 18.4% among SMS+75KES SC compared to Control SC (aRD 95% CI: -36.2%, -0.5%; p= 0.044)” Kagucia 2018 (p.291)

• 202

  "The total M-SIMU sub-sample included in the MSBC study represented 13.6% (218/1599) of the entire M-SIMU sample and 12.2% (44/360), 15.5% (60/388) 12.6% (56/445) and 14.3% (58/406) of households enrolled in the M-SIMU Control arm, SMS only arm, SMS+75KES arm and SMS+200KES arm, respectively" Kagucia 2018, p. 318.

• 203

  During the original trial, the subset of control group households later included in the follow-up reported higher MCV1 coverage compared to the overall control group. "In addition, we observed that MCV1 timely coverage in the sub sample of M-SIMU Control children included in the MSBC study was substantially greater than coverage among all Control M-SIMU children” (Kagucia 2018, pp. 334-335)

• 204

  "we observed that [...] MCV1 timely and overall coverage in the sub-sample of SMS+75KES M-SIMU children included in the MSBC study were substantially lower than among all SMS+75KES M-SIMU children.” (Kagucia 2018, pp. 334-335)

• 205

  As far as we could tell, Kagucia 2018 does not run significance tests on the differences in vaccine-seeking behavior in the follow-up groups compared to the broader cohort in the original study.

• 206

  “In comparing M-SIMU baseline characteristics across M-SIMU study arm of the sub-sample included in the MSBC study, there were no statistically significant differences in the distribution of M-SIMU child’s sex, location of delivery for the M-SIMU child, maternal education, maternal age, travel time to a health facility, region of residence or socioeconomic quintile. However, phone ownership at the start of the M-SIMU study was not equally distributed in this sub-sample; 52.3% (n= 23) of Control arm caregivers owned a phone at the beginning of the M-SIMU study compared to 45.0% (n= 27) of SMS only arm caregivers, 26.8% (n= 15) of SMS+75KES arm caregivers and 33.9% (n= 20) of SMS+200KES arm caregivers (p= 0.040; Table 5.5). Because of this imbalance, regression analyses were adjusted for mobile phone ownership status at the start of M-SIMU.” (Kagucia 2018, pp. 320-321)

• 207

  “MCV1 timely coverage was 11.2% lower among children of SMS only caregivers after the MSIMU study compared to during the M-SIMU study, although this difference did not achieve statistical significance (adjusted DID -11.2%; 95% CI: -36.1%, 13.7%; p= 0.376). This decrease translated to a statistically insignificant difference of 11.5% in MCV1 timely coverage among SMS only SC compared to Control SC (95% CI: -32.8%, 9.8%; p= 0.290). Crude and adjusted RR and RD estimates did not differ markedly (Figure 5.4).” (Kagucia 2018, p. 323)

• 208

  More on these aspects of New Incentives' work here and here on our separate page about New Incentives' program.

• 209

  For example,

  • New Incentives’ other program activities could mitigate negative effects by improving caregivers' knowledge of vaccinations, thus leading them to complete more vaccinations after the program ends.
  • On the other hand, removing other services alongside CCTs could lead to larger negative effects by reducing caregivers’ ability to access vaccinations or acting as a signal that vaccination is less valuable.

• 210

  We assigned a 15% overall weight to Kagucia 2018. (See these cells of our analysis.) Within this study, we put somewhat more weight on the 200KES arm (10% out of the 15%) because we think the higher incentive amount is more representative of New Incentives' program. Details below:
  In February 2024, New Incentives estimated that the average naira:USD exchange rate it would receive in 2024 would be 1,250 naira per 1 USD. Source: New Incentives, Summary of Funding Gap Estimates (unpublished). This implies an incentive of $1.47 USD on average per caregiver visit (although note that in New Incentives' program, almost half of the total value of the incentives is provided for completion of all six vaccination visits).

  • 11,000 naira total incentive / 1,250 naira per USD = $8.80 USD.
  • $8.80 USD / 6 visits = $1.47 USD incentive per visit, on average.

  As of August 1st 2024, the KES:USD exchange rate is 157 KES per 1 USD. Source: this page. This implies an incentive of $1.27 per timely vaccine completed in the SMS+200KES arm and an incentive of $0.48 per timely vaccine completed in the SMS+75KES arm.

  • 200 KES incentive per timely vaccine completed / 157 KES per USD = $1.27 USD.
  • 75 KES incentive per timely vaccine completed / 157 KES per USD = $0.48 USD.

• 211

  “Fourth, the incentive implementation mimicked that of a real program at scale: government health workers distributed the bracelets as part of routine immunization days at clinics, and the incentive was implemented at a large scale, reaching more than 35,000 children and for a prolonged time of two years.” Karing, Finetti, and Kuloszewski 2024 (p. 8)

• 212

  “Karing (2024) tested the effects of social signaling incentives–in the form of color-coded bracelets–given to children when they came timely for immunization. A total of 120 government clinics were randomized into four groups. [...] The bracelets were implemented for two years from July 2016 to August 2018. Health workers were trained to give bracelets to children when they came to the clinic according to the design rules of each treatment.” Karing, Finetti, and Kuloszewski 2024 (pp. 7-8)

• 213

  "All three treatment groups included an initiation bracelet given to children upon the first vaccine visit. Since 99.7% of parents in this context initiate vaccination, the first bracelet does not carry any signaling value and primarily functions as a small material reward.”

  • ‘The first treatment consisted solely of the initiation bracelet: children received a bracelet of their chosen color between yellow and green at their first vaccination visit, replaced by a same identical bracelet at vaccines four and five. We refer to this treatment as the “Initiation Reward”.’
  • ‘The second treatment included an additional incentive: the initiation bracelet set for all as the yellow bracelet was exchanged for a green one at the fourth vaccine if completed on time. While informative about one’s vaccine status, parents assign little importance to the fourth vaccine and do not gain social benefit from signaling its timely completion. Instead, its value comes from clinic staff potentially praising the parent and the novelty of receiving a different bracelet color. We refer to it as the “Double Reward”.’
  • ‘The third treatment consisted in exchanging the yellow initiation bracelet for a green one at the last vaccine if all vaccines are completed on time. The green bracelet received at the fifth visit, perceived by parents as one of the most important vaccine, was a strong signal of parents’ intrinsic motivation to care for their child’s health. We refer to it as the “Signaling Reward”. If the child came late in either the Double or Signaling Reward treatments, the bracelet was replaced by the same identical yellow bracelet.”’

  Karing, Finetti, and Kuloszewski 2024 (pp. 7-8)

• 214

  "The third treatment consisted in exchanging the yellow initiation bracelet for a green one at the last vaccine if all vaccines are completed on time. The green bracelet received at the fifth visit, perceived by parents as one of the most important vaccine, was a strong signal of parents’ intrinsic motivation to care for their child’s health. We refer to it as the “Signaling Reward”. If the child came late in either the Double or Signaling Reward treatments, the bracelet was replaced by the same identical yellow bracelet.”
  Karing, Finetti, and Kuloszewski 2024 (pp. 7-8)

• 215

  “Compared to the Control Group, signaling treatments combined lead to an increase in timely vaccination rates for four vaccines, from 74 to 80% (p=0.069), and for five vaccines, from 57 to 63% (p=0.082). The average effects mask substantial heterogeneity by treatment: Signal at 4 has a small, insignificant effect on the share of children receiving timely vaccinations for four vaccines (2 percentage points, p=0.58), and has no impact on the timely completion of five vaccines. In contrast, Signal at 5 has a large effect on the share of children receiving timely vaccinations for five vaccines (13.3 percentage points, p=0.001). This effect remains large and significant (10.5 percentage points, p=0.004) when comparing Signal at 5 to the Uninformative Bracelet, suggesting social signaling is the primary mechanism.“ Karing 2024, p. 4

  We estimate the change in the number of timely vaccines received by calculating the average total number of additional timely vaccines completed in the Signal at 5 group / the average total number of timely vaccines completed in the control group. 0.0868 = 0.351 / 4.042
  Karing 2024, Table IV, p. 43

• 216

  “We conducted a follow-up survey in the 597 communities of the original study between December 2020 and August 2021. In this survey, we list all children born and living in these communities since the experiment ended… A total of 19,318 children were identified as eligible for inclusion in our study sample. Eligibility was defined by being born after December 1, 2017 and attending one of the 119 study clinics for immunizations.” Karing, Finetti, and Kuloszewski 2024 (p. 9)

• 217

  “To assess the extent to which the incentives influence parents’ intrinsic motivation, we examine whether direct exposure to the incentives during the experiment affects parents’ vaccination decisions post-experiment, compared to parents who were only indirectly exposed to the incentives. As incentives were tied to timely vaccination, our primary outcome is the total number of vaccines completed on time.” Karing, Finetti, and Kuloszewski 2024 (p. 3)
  “Timely vaccination is defined in the same way as in the paper Karing (2024). The timeliness cut-offs are informed by WHO guidelines stating that the DTP series should be completed by six months of age (WHO 2018). There is no strict guidelines for the measles vaccine, but we use the same definition allowing for a 2.5 months buffer window for each vaccine.” Karing, Finetti, and Kuloszewski 2024 (footnote 5 on p. 12)

• 218

  "parents who had a child during the experiment and were thus directly exposed to the bracelet incentives in the Initiation and Double Reward treatments complete 5% (p=0.05) and 11% (p<0.001) fewer vaccines on time, respectively, compared to parents in the Control group who had a child during the experiment. The Signaling Reward shows less pronounced and insignificant negative effects (-3%, p=0.29)." Karing, Finetti, and Kuloszewski 2024 (p. 12).

• 219

  “Our analysis shows that direct exposure to extrinsic incentives diminishes parents’ motivation to vaccinate subsequent children on time. These effects are common across all three incentives treatment, but vary in magnitude and significance. Specifically, parents who had a child during the experiment and were therefore directly exposed to bracelet incentives in the Initiation and Double Reward treatments complete 5% (p=0.05) and 11% (p<0.001) fewer vaccines on time, respectively, compared to parents in the Control group who had a child during the experiment.” Karing, Finetti, and Kuloszewski 2024 (p. 12)

• 220

  The authors hypothesize that these bracelets did not have any signaling value because parents do not get any social benefits from signaling completion of the incentivized visits. “All treatments included an initiation bracelet given to children at the first vaccination visit. Since 99.7% of parents in this context initiate vaccination, the first bracelet does not carry any signaling value and mainly functions as a small material reward. The first treatment consisted solely of the initiation bracelet and we refer to it as Initiation Reward. The second treatment included an additional incentive: the initiation bracelet was exchanged for a new color at the fourth vaccine if completed on time. While informative about one’s vaccine status, parents assign little importance to the fourth vaccine and do not gain social benefit from signaling its timely completion. Instead, its value comes from clinic staff potentially praising the parent and the novelty of receiving a different bracelet color. We refer to it as Double Reward.” Karing, Finetti, and Kuloszewski 2024 (p. 4).

• 221

  “The Initiation and Double Reward treatments did not show positive effects during their implementation.” Karing, Finetti, and Kuloszewski 2024 (p. 4).

• 222

  “The Signaling Reward shows less pronounced and insignificant negative effects (-3%, p=0.29).” Karing, Finetti, and Kuloszewski 2024 (p. 12)

• 223

  “The third treatment consisted of the exchange of the initiation bracelet at the last vaccine if all vaccines were completed on time. Owning the second bracelet is considered a significant indicator of a parent’s care for their child’s health because parents regard the last visit as one of the most important ones. We refer to it as a Signaling Reward.” Karing, Finetti, and Kuloszewski 2024 (p. 3)

• 224

  The authors hypothesize that the signaling bracelets focused caregiver attention on the intrinsic reasons for vaccinating their children because caregivers perceived the last visit as one of the most important. Thus, receiving an incentive tied to completion of the last visit may have emphasized their care for their child’s health. By contrast, the bracelets provided in the other arms may have focused attention on external reasons for vaccination and crowded out intrinsic motivation.

  • “We hypothesize that in the Signaling Reward, parents’ attention was focused on showing their intrinsic motivation to others via the signal, which mitigated the crowd-out effect. This suggests that incentives that allow parents to focus on the reason they take the action may counteract the crowding-out of intrinsic motivation.” Karing, Finetti, and Kuloszewski 2024 (p. 13)

• 225

  “Conversely, parents who were only indirectly exposed to the incentives do not show this decline, suggesting that the adverse effects are driven by the direct experience of receiving the incentive, rather than community-level factors such as changes in clinic staff’s behavior or shifts in community vaccination norms. Table IV (Panel B, column 1) displays the results for indirectly exposed parents, indicating minimal effects of incentive treatments on the total number of vaccines received on time: coefficients hover around zero (-0.14 to 0.002) and are insignificant (p=0.14-0.98).” Karing, Finetti, and Kuloszewski 2024 (pp. 12-13)

• 226

  The study authors confirmed this in an email to GiveWell, September 6, 2024 (unpublished).

• 227

  “We conducted a follow-up survey in the 597 communities of the original study between December 2020 and August 2021. In this survey, we list all children born and living in these communities since the experiment ended. Our methodology mirrored that of the original study. The survey included questions about each child’s vaccination history, ownership of a bracelet, awareness of the bracelet program, and exposure to other incen-tives for immunization. To confirm whether parents had previously been exposed to the program, we asked how long they had lived in the community and if they had a newborn between 2016 and 2018. A total of 19,318 children were identified as eligible for inclusion in our study sample. Eligibility was defined by being born after December 1, 2017 and attending one of the 119 study clinics for immunizations.3 Of these, 11,235 children were alive at the time of the survey, old enough to have completed all vaccinations, and their primary caregiver resided in one of the study communities, making it possible for us to collect their immunization data.” Karing, Finetti, and Kuloszewski 2024 (p. 9)

• 228

  "Tables II and III detail socio-demographic characteristics of parents who were directly exposed to incentives and those who were not directly exposed, respectively. The former sample is well balanced across all indicators with the exception of a significant but small imbalance on the birth order of children. Double Reward parents have 5% (p0.08) and 2% (p=0.09) fewer children on average than those in the Control group and Signaling Reward, respectively." Karing, Finetti, and Kuloszewski 2024 (pp. 9-10)

• 229

  “Tables I, II and III displays characteristics and attrition for our main analysis sample. Table I shows consistent survey success rates across control and treatment groups, ranging from 66.3 to 72.9% (F= 1.09, p=0.36). The primary reasons for attrition include that parents have moved (14%), the child is deceased (8%), and that parents were traveling at the time of the survey (7.9%). Notably, the attrition rates and their cause do not differ significantly between the control and incentive groups.” Karing, Finetti, and Kuloszewski 2024 (p. 9)

• 230

  We think it’s likely that caregivers may misremember or forget their children’s vaccination status over time. In particular, caregivers who participated in an intervention aimed at improving vaccination coverage may be more likely to overreport completed vaccinations compared to control groups.
  In unpublished slides shared with GiveWell, the researchers who conducted Karing, Finnetti, Kuloszewski 2024 reported that they verified vaccination status using “in-person survey using information from the child’s growth card when available (95%), and from the parent’s recall otherwise.” We think this largely eliminates concerns about recall bias because the majority of vaccine verification did not rely on participants' memory.

• 231

  We think the incentives provided by New Incentives increase vaccination uptake by offsetting the time and transportation costs required for caregivers to bring children in for immunizations. Caregivers may perceive the incentives as stipends which support their decision to vaccinate their children.

• 232

  We put 25% weight on Karing, Finetti, and Kuloszewski 2024 overall. Within this study, we allocate slightly less weight (5% out of the 25%) to the arm where the bracelets appeared to have a social signaling value ("Signaling Reward"). We think the bracelets offered in the other two treatment arms are more similar to receiving a financial incentive, so we put 10% weight on each.

• 233
  • Kagucia 2018 is a follow-up study of Gibson et al 2017, which incentivized timely completion of four vaccines:
    • "In addition to receiving SMS reminders, caregivers were sent either 75 KES (group 3) or 200 KES (group 4) to their mobile phone for each timely dose of pentavalent and measles vaccine received, defined as vaccination within 2 weeks of the Expanded Programme on Immunisations (EPI) scheduled date"
  • Karing, Finetti, and Kuloszewski 2024 followed up on Karing 2024, which incentivized timely completion of the fourth or fifth vaccine:
    • "In each of the first two bracelet treatments—hereafter Signal at 4 and Signal at 5 - children receive a yellow bracelet upon their first vaccination. In the Signal at 4 treatment, the yellow bracelet is exchanged for a green bracelet once a child completes all of the first four vaccines in a timely manner. In clinics assigned to the Signal at 5 treatment, the nurse exchanges the yellow bracelet for a green bracelet once a child completes all five vaccinations in a timely manner." (pp. 3-4).

• 234

  In order to be eligible for enrollment into New Incentives’ program, infants need to be under 1 year old; so this means very delayed vaccinations would not be incentivized in practice. For more details on the eligibility criteria New Incentives uses for disbursing CCTs, see this section of our report on New Incentives' program.

• 235

  “Compared to Control SC, MCV1 overall coverage was 20.8% lower and 6.2% lower among SMS+200KES SC and SMS only SC, respectively, however these differences were not significant (SMS+200KES aRD 95% CI: -42.1%, 0.6%; p= 0.057 and SMS only aRD 95% CI: -22.2%, 9.8%: p= 0.451). MCV1 overall coverage was significantly lower by 18.4% among SMS+75KES SC compared to Control SC (aRD 95% CI: -36.2%, -0.5%; p= 0.044)” Kagucia 2018 (p.291)

• 236

  Kagucia 2018 found large reductions in MCV1 coverage at 12 months similar in direction and magnitude to the primary results on timely completion.

  • “The absolute decrease in MCV1 overall coverage among SMS+200KES arm SC compared to Control SC was 20.8% and was not statistically significant (aRD 95% CI: -42.1%, 0.6%; p= 0.057).”
  • “But SMS+75KES SC had significantly lower MCV1 overall coverage than Control SC in the period after the M-SIMU study (aRD 18.4%, 95% CI: -36.2%, -0.5%, p= 0.044).”

  (Kagucia 2018, pp. 324-325)

  Karing, Finetti, and Kuloszewski 2024 found no statistically significant reductions in completion at 15 months in any treatment arm, in contrast to the significant reduction they found in the Initiation and Double Rewards groups. “We then examine the completion of all vaccines by 15 months, which is the due date for the second measles dose and the next milestone in the immunization schedule. Panel A of Table V reveals that direct exposure to incentives does not have a pronounced effect on the completion of vaccinations by 15 months. The coefficients on the total number of vaccines received in the Initiation and Double Reward treatments are negative but insignificant at -0.12 (p=0.22) and -0.075 (p=0.55).” Karing, Finetti, and Kuloszewski 2024 (pp. 13-14)

• 237

  “Compared to Control SC, MCV1 overall coverage was 20.8% lower and 6.2% lower among SMS+200KES SC and SMS only SC, respectively, however these differences were not significant (SMS+200KES aRD 95% CI: -42.1%, 0.6%; p= 0.057 and SMS only aRD 95% CI: -22.2%, 9.8%: p= 0.451). MCV1 overall coverage was significantly lower by 18.4% among SMS+75KES SC compared to Control SC (aRD 95% CI: -36.2%, -0.5%; p= 0.044)” Kagucia 2018 (p.291)

• 238

  The study authors confirmed this in an email to GiveWell, September 6, 2024 (unpublished).

• 239

  In the original M-SIMU trial, control group coverage for all vaccines included in the study (BCG, 3 doses of the polio vaccine, 3 doses of the pentavalent vaccine) except measles was at least 97% at 12 months. “With the exception of MCV, vaccination coverage in the control arm for most vaccines at age 12 months was high, ranging from 97% (Polio3) to 98% (Penta3). MCV1 coverage at age 12 months was 84%.” (Kagucia 2018, p. 58)

  In the original bracelets study (Karing 2024), control group coverage for all vaccines except measles was at least 90% at 12 months. “Almost all children in the Control Group receive the first three vaccines by 12 months, with completion rates at 99.3, 98.4 and 95.9%. Despite this, a substantial drop-off remains for the fourth and fifth vaccines, with only 91.7% and 68.6% of children, respectively, receiving these vaccines by one year of age.” (Karing 2024, p. 23)

  In states where New Incentives operates at the time of writing, we estimate that vaccination coverage excluding the measles vaccine ranges between 14% to 72%, with an average Penta3 completion rate of 33% at baseline. See these cells in our CEA.

• 240

  “The context of childhood immunization in Sierra Leone offers a relevant policy application to the question of motivation crowd-out. With vaccination rates nearing the levels necessary to reach herd immunity, policy makers face the challenge of increasing take-up among the “last mile” parents without undermining the majority who are already vaccinating. Further, in this setting, routine vaccination is a well-known health behavior for which individuals have established beliefs about its costs and benefits.” Karing, Finetti, and Kuloszewski 2024 (pp. 2-3).

• 241

  Email from New Incentives, August 16, 2024 (unpublished).
  Our understanding is that Joint State Stakeholder Meetings are roundtables with health officials and other stakeholders in the states where New Incentives operates. See this section of our separate report on New Incentives' program.

• 242

  To calculate the hypothetical exit year, we estimate the year that the marginal program cost-effectiveness will fall below 8x (our current bar for funding top charities). For our calculations, see here.
  Our approach:

  • We start with baseline coverage during the midpoint of the current grant period and project forward counterfactual coverage using our assumption that non-vaccination rates fall by 5% each year. For more on this, see this section of the report.
  • We estimate that New Incentives’ program causes a 29% reduction in the unvaccinated rate each year. For more on this, see this section of the report.
  • We assume the program receives a hypothetical $1 million grant each year and that cost per child enrolled remains the same over time (roughly $18 per child). We discount the value of the program per counterfactually vaccinated child by .5% in each subsequent year. These are simplifying assumptions.
    • Over time, we expect cost per child will decrease somewhat based on historical declines (see this row of our analysis of New Incentives' cost per infant enrolled), but we are unsure whether and how much it will continue to decline.
    • We apply a 0.5% discount rate for the future value of the program, consistent with the approach used elsewhere in this report to estimate the value of vaccination on future mortality risk.This potentially understates the actual decline in benefits over time for children who are counterfactually vaccinated by the program, e.g., because of other general health improvements which could reduce the spread and deadliness of vaccine-preventable diseases, and thus the benefits of vaccination.
    • We assume the overestimated cost per child and underestimated decline in program benefits over time roughly cancel each other out, but this could mean we are over or underestimating the future cost-effectiveness of the program.
  • We estimate cost-effectiveness each year until the value falls below 8x.

• 243

  We assume that caregivers will have 5 children in total on average based on the total fertility rate in Nigeria estimated by the World Bank for 2022 (5.1 total births per woman).

  • We expect the total fertility rate will decline somewhat over time, and that fertility rates are slightly higher in northern Nigeria. We think these two effects roughly cancel each other out, so we have not made any adjustments to the fertility rate for future periods in northern Nigeria.

• 244

  We estimate that only caregivers of children enrolled in the last 5 years of the program will have at least one additional child who needs to be vaccinated after the program ends. In simple terms, this is because we think children enrolled earlier during the program implementation period are less likely to have younger siblings who need to be vaccinated after the program ends.

  • We assume that caregivers will have roughly 5 children in total based on the World Bank's estimated fertility rate of (5.1) for Nigeria in 2022.
  • As a simplification, we assume caregivers are reached by New Incentives halfway through their childbearing years and will have 2 additional children after the first child enrolled.
  • Furthermore, we assume that children will be born ~2.5 years apart based on median birth intervals reported in DHS18 data, and do not adjust for potential changes in birth spacing over time. In expectation, this means that:
    • The fourth (second to last) child will be born 2.5 years after the child enrolled in New Incentives' program.
    • The fifth (last) child will be born 5 years after the child enrolled in New Incentives' program.
  • Therefore, only children enrolled in the last 2.5 years of the program are likely to have at least 2 additional siblings who need to be vaccinated post-program; and only children enrolled in the last 5 years of the program are likely to have at least 1 additional sibling who needs to be vaccinated post-program.

  We also assume that some of the children in New Incentives’ program are younger siblings of previously enrolled children. The post-program siblings of these children would be double-counted, so we make rough adjustments to estimate the number of caregivers represented.

  • For children enrolled in the last 5 years of the program, we roughly guess that 20% of them are siblings of other children enrolled in the same period. We use a slightly smaller value for children enrolled in the last 3 years of the program (10%) because we expect the share of siblings to decrease as program duration shortens.
  • For children enrolled in the last 2.5 years of the program, we roughly guess that 10% of them are siblings of other children enrolled in the same period.

  In Niger, for example, where we estimate that the program would be implemented for three years with roughly 165,000children enrolled over that period of time, we expect:

  • (3/3) * (100-10)% = ~90% of children enrolled during the program implementation period to have at least one sibling who needs to be vaccinated post-program. This is equal to roughly 148,000 siblings .
  • (2.5/3) * (100-10)% = ~75% of children enrolled during the program implementation period to have at least two siblings who need to be vaccinated post-program. This is equal to roughly 124,000 siblings.

• 245

  “Conversely, parents who were only indirectly exposed to the incentives do not show this decline, suggesting that the adverse effects are driven by the direct experience of receiving the incentive, rather than community-level factors such as changes in clinic staff’s behavior or shifts in community vaccination norms. Table IV (Panel B, column 1) displays the results for indirectly exposed parents, indicating minimal effects of incentive treatments on the total number of vaccines received on time: coefficients hover around zero (-0.14 to 0.002) and are insignificant (p=0.14-0.98).” Karing, Finetti, and Kuloszewski 2024 (pp. 12-13)

• 246

  We estimate the counterfactual vaccination rate for siblings using the counterfactual vaccination rate estimated in the hypothetical exit year (see this row of our analysis). This may underestimate the vaccination rate for siblings who are born further out from the program withdrawal year because we expect counterfactual vaccination rates to increase over time. For example, if we expect the program to exit in 2027, we would expect siblings who are born in 2029 to have somewhat higher counterfactual vaccination rates.

• 247

  We estimate the number of children harmed by multiplying the number of siblings * their counterfactual vaccination rate * the expected decrease in vaccination coverage.

• 248
  • Kagucia 2018 specified M-SIMU caregivers were eligible to be included in the follow-up only if they had an immediate younger sibling aged at least 10 months: “M-SIMU caregivers were eligible to enroll in the MSBC study if: the vaccination status of the child enrolled in the M-SIMU (M-SIMU child) study was verified using the maternal and child health (MCH) booklet at age 12 months; the M-SIMU child had an immediate younger sibling (i.e., subsequent born child) aged at least 10 months old; and the vaccination status of the subsequent child (SC) could be verified via MCH booklet.” Kagucia 2018 (p. 297)
  • We are less sure in the case of Karing, Finetti, and Kuloszewski 2024, but we think it’s likely that the follow-up predominantly included only the next subsequent sibling because the follow-up was conducted relatively soon after the program ended (1-3 years), and subsequent children in the follow-up study were only included if they were at least 12 months of age. We think it’s very unlikely that many caregivers will have had more than one additional child in this time frame.
    • “The bracelets were implemented for two years from July 2016 to August 2018.” Karing, Finetti, and Kuloszewski 2024 (p. 8)
    • In the follow-up: “We include children born after May 1, 2019, who were at least 12 months old by the time last observed.” Table IV (p. 26) Karing, Finetti, and Kuloszewski 2024

• 249

  See this row of our analysis.

• 250

  We estimate a -32% adjustment in Taraba state. Prior to applying an adjustment, our estimate of cost-effectiveness in Taraba was around 8.5x.

  • To see the program’s cost-effectiveness prior to post-program effects, set the adjustment in this cell to zero.

  We think the adjustment is especially high for two reasons:

  • Baseline vaccination rates in Taraba (we estimate 74%) are high relative to the other locations we modeled (63% in Yobe, and 61% in Niger), so we expect fewer children to benefit each year the program is running.
  • We expect the program would only run for two years prior to falling below our cost-effectiveness bar (8x), so fewer total children benefit over the duration of the program,

• 251

  I.e. (-15% + -32% + -11%) / 3. We don’t include Niger - cohort 32 in our average because Niger state is already captured in the average.

• 252

  In general, we expect the adjustment would be somewhat larger in areas where vaccination rates are higher at the start of the program, and thus, where the program would likely operate for a shorter period of time.

  • If an area has higher vaccination rates at the start of a program, we expect to fund the program for fewer years. This is because cost-effectiveness is generally lower in areas where the vaccination rates are high at the start of the program, and we expect cost-effectiveness to decline over time as counterfactual vaccination rates generally improve.#
  • A shorter program duration means fewer children are vaccinated during the program relative to the number who don't get vaccinated after the program (assuming that a longer program duration does not meaningfully increase the size of the negative post-program effects). This would result in a larger negative adjustment.

  However, in locations where vaccination rates are higher, it is possible that post-program effects would differ for other reasons. For example, there may be stronger social norms around vaccinations, which could mitigate negative effects.

• 253

  As of August 2024, this is Bauchi, Gombe, Jigawa, Kaduna - cohort 17, Kano, Katsina, Kebbi, Sokoto, and Zamfara.

• 254

  We estimate the total hypothetical implementation duration using a similar approach to what is outlined above for Niger. See this worksheet for our calculations.

• 255

  We estimate the impact of post-program effects over the total hypothetical implementation duration using a similar approach to what is outlined above for Niger (i.e. we estimate when the program would fall below 8x, and calculate the relative number of children vaccinated because of the program during and after). We estimate that the post-program effects adjustment would be roughly -3% to -14% in these areas if the program operated for the full hypothetical program duration.

  • As a simplification, we use 2024 as the starting year of the program. We expect this to overestimate the adjustment in locations where New Incentives started operating prior to 2024 because the adjustment does not capture the children counterfactually vaccinated by the program in the earlier years.

  If the program were to withdraw sooner, we would expect the post-program effects adjustment to be larger. For example, in Bauchi, we estimate that for every 100 children counterfactually vaccinated by the program over 16 years, 6 children would not be vaccinated after the program. If, however, the program only operated from 3 years (assuming the program started in 2024), we expect 15 children to be counterfactually not vaccinated after the program (set this value to 3).

• 256

  To estimate the adjustment with 10 additional operating years, set the hypothetical program duration in this row to 10.

• 257

  This is because we think they would be challenging to model effectively or their likely effect is small enough that we do not expect explicit modeling to be worth the time investment required.

• 258

  See this row of our cost-effectiveness analysis.

• 259

  In our cost-effectiveness analysis, this appears as (mortality and morbidity effects). We have grouped them here for simplicity. Because of rounding, these sum to 4% (3.6% + 0.7%).

• 260

  We add each percentage adjustment together to obtain our total adjustment.

• 261

  Step one: Estimate the likely size of each effect (a rough subjective best guess).

  • Step two: Evaluate each effect using three criteria, each assigned a score of up to 3:
    • Can it be objectively justified (i.e., is there direct evidence for the effect)? (Where 0 is little evidence and 3 is strong direct evidence).
    • How easy would it be to model? (Where 0 is impossible and 3 is simple).
    • Consistency - is this effect included in our cost-effectiveness analysis for other programs? (Where 1 is no, 2 is partially, and 3 is yes).
  • Step three: Convert the total three criteria score out of 9 into a percentage (e.g., a total score of 5 converts to a percentage of 50%).
  • Step four: Weight our estimate of each effect by this percentage to produce our overall adjustment (e.g., an effect size guess of 15% multiplied by a three criteria score of 60% = an adjustment of 9% (15% * 60% = 9%)).

  We use the method described for all supplemental intervention-level adjustments in our cost-effectiveness analysis for this program, with the exception of our estimate of costs saved from averted malaria. We account for these savings using an explicit model, discussed in more detail here.

• 262

  See this spreadsheet for more details on our reasoning for each adjustment.

• 263

  "Vaccination confers both direct and indirect effects. The direct effect implies protection against disease in vaccinated individuals. Indirect protection is when susceptible individuals avoid infection because the people who surround them are immunized. The magnitude of indirect effects is a function of transmissibility of the infectious agent, population mixing patterns, distribution of vaccine, and distribution of immunity in the population. 'Herd immunity' refers to population-scale immunity." Nymark et al. 2017.

• 264

  “Herd immunity is usually achieved by interrupting the transmission of the organism by preventing infections in immunized individuals. Thus, there is less opportunity for the unimmunized individual to be exposed to the organism.” Pollard et al. 2015.

• 265

  We believe the excluded morbidity effects are likely to be small (no more than 5% of the total DALY burden). This is based on the finding that only a small percent of DALYs is due to years lost to disability (YLDs) for children under 5 (less than 2% of DALYs across all diseases; see here). Excluding these morbidity effects seems consistent with our decision to exclude morbidity from our cost-effectiveness analyses of other interventions primarily aimed at reducing mortality. See here for our DALY analysis.
  Definitions:

  • DALY stands for “disability-adjusted life year.” The Global Burden of Disease Project defines it as: “a universal metric that allows researchers and policymakers to compare very different populations and health conditions across time. DALYs equal the sum of years of life lost (YLLs) and years lived with disability (YLDs). One DALY equals one lost year of healthy life.”
  • YLDs (years lived with disability) are years lived in less than ideal health. YLLs are measured by multiplying the prevalence of a health condition by the “disability weight” (how bad it is) of that condition. We use YLDs as a measure of morbidity.
  • YLLs (years of life lost) are years lost due to premature mortality. For example, if the longest life expectancy for men in a given country is 75, but a man dies of cancer at 65, this would be 10 years of life lost due to cancer. We use YLLs as a measure of mortality.

  Definitions are from the Institute for Health Metrics and Evaluation, "Global Burden of Disease (GBD)".

• 266

  Note: in our cost-effectiveness analysis this appears as two rows: (1) mortality (+4%), (2) morbidity (+1%). Because of rounding, together these sum to 4% (3.6% + 0.7%). See here. We have grouped them here for simplicity.

• 267

  Yellow fever and acute hepatitis B have a much lower DALY burden for children under five (53 and 69 per 100,000, respectively), relative to other diseases prevented by vaccines (see this column of our supplementary spreadsheet).
  For polio, meningitis A, mumps, rubella, and varicella, our impression is that the number of cases are low, but we haven’t attempted to quantify the burden. In our cost-effectiveness analysis, we include a rough guess for additional benefits from effects on these diseases.

• 268

  As of September 2024, these are the Against Malaria Foundation (AMF), Malaria Consortium, Helen Keller International, and New Incentives. See our "Top charities" page here.

• 269

  Our model estimates a 14% upward adjustment for long-lasting insecticidal nets (here), a 20% upward adjustment for seasonal malaria chemoprevention (here) and a 21% upward adjustment for vitamin A supplementation (here). We did not model these benefits for New Incentives, and so the New Incentives figure is benchmarked to the other three programs.

• 270

  The RCT found that New Incentives’ program increased the timeliness of the measles vaccine, also increased the timeliness of the Penta 1 vaccine (on some but not all measures), and led to a decrease in timeliness for the BCG vaccine, although this was small and not statistically significant. See IDinsight, Impact evaluation of New Incentives, final report, 2020, Table 8, pg. 30.

  “The incentive had the largest impact on the timeliness of the Measles 1 vaccine. Children in the treatment group who received the Measles 1 vaccine were 33 percentage points (95% CI: 28, 38; p-value < 0.001) more likely to have received the vaccination within one month of the recommended age (9 months) compared to those who received the Measles1 vaccine in the control group. Measles 1 vaccinations were also more likely to be timely in the treatment group than in the control group when using a cutoff of within 2 weeks of the recommended age." IDinsight, Impact evaluation of New Incentives, final report, 2020, pg. 30.

• 271

  The measles vaccine is recommended at 9 months in high-transmission settings to balance higher efficacy with age (which suggests vaccinating later) with high risk of measles for those unvaccinated (which suggests vaccinating sooner).

  “The age at vaccination is one of the most important determinants of the immune response to MCV, with older infants usually showing better responses than younger infants (Figure 4). The optimal age for measles vaccination is determined by consideration of the age-dependent increase in seroconversion rates following measles vaccination and the average age of infection. In regions of intense MV transmission, the average age of infection is low and the optimal strategy is to vaccinate against measles at as young an age as possible (usually 9 months of age, Figure 4). By contrast, in settings where MV transmission has been reduced, the age of administration of the first dose of MCV can be increased to 12 months or older. Antibody responses to MCV increase with age up to around 15 months because of the declining levels of inhibitory maternal antibodies and decreasing immaturity of the immune system. This immaturity of the immune system in neonates and very young infants includes a limited B-cell repertoire and inefficient mechanisms of antigen presentation and T-lymphocyte help. The recommended age at vaccination must balance the risk of primary vaccine failure, which decreases with age, against the risk of MV infection prior to vaccination, which increases with age.” World Health Organization, Immunological basis for immunization series, measles, 2020, pg.13.

  As a result, vaccinating too soon may risk lowering vaccine efficacy while vaccinating too late may risk measles infection. We have not sought to vet this claim or quantify this effect, and we include a small adjustment for improved timeliness in our cost-effectiveness analysis.

• 272

  See this spreadsheet, "Investment of income increases" rows for each intervention, for details on the specific values used for each program. Note that we have not recently updated these adjustments, and so the variation in values between programs may no longer be accurate. We have not prioritized updating this because it makes a very small difference to our cost-effectiveness estimates.

• 273
  • “Most studies indicate that the degree of mucosal immunity in the intestine is significantly less than that provided by OPV, although this difference may be less pronounced in the pharyngeal mucosal lining.” World Health Organization, Standards and specifications, "Poliomyelitis".
  • “While IPV elicits a much weaker mucosal immune response than OPV,5 and is thus less effective at averting transmission, it is very protective against disease.” Alfaro-Murillo et al. 2020. Note that we have not reviewed the evidence for this in detail.

• 274
  • “In very rare cases, the administration of OPV results in vaccine-associated paralysis associated with a reversion of the vaccine strains to the more neurovirulent profile of wild poliovirus.” World Health Organization, Standards and specifications, "Poliomyelitis".
  • A write up on this issue is available here.
  • New Incentives may offset this risk through its effect on inactivated polio vaccine (IPV), which may lower risk of outbreak from oral polio vaccines: "Circulating VDPVs occur when routine or supplementary immunization activities (SIAs) are poorly conducted and a population is left susceptible to poliovirus, whether from vaccine-derived or wild poliovirus. Hence, the problem is not with the vaccine itself, but low vaccination coverage. If a population is fully immunized, they will be protected against both vaccine-derived and wild polioviruses." WHO, Poliomyelitis: Vaccine derived polio, 2017.

• 275

  “Vaccine-associated paralytic poliomyelitis (VAPP) is a rare adverse event associated with oral poliovirus vaccine (OPV). This review summarizes the epidemiology and provides a global burden estimate… Using all risk estimates, VAPP risk was 4.7 cases per million births (range, 2.4–9.7), leading to a global annual burden estimate of 498 cases (range, 255–1018). If the analysis is limited to estimates from countries that currently use OPV, the VAPP risk is 3.8 cases per million births (range, 2.9–4.7) and a burden of 399 cases (range, 306–490).” Platt et al. 2014. Note that we have not reviewed this evidence in detail.

• 276

  "Vaccination with the protein-polysaccharide conjugate vaccine, PCV7, has significantly reduced the burden of pneumococcal disease in populations where it is in widespread use and has had an important public health benefit. This vaccine targets only 7 of the more than 92 pneumococcal serotypes, and there have been concerns that the non-vaccine serotypes (NVTs) could increase in prevalence and reduce the benefits of vaccination." Weinberger, Malley, and Lipsitch 2011, pg. 1.

• 277
  • "We currently employ over 2,000 FOs." New Incentives, responses to GiveWell questions, July 5, 2024 (unpublished)
  • We discussed qualifications for the Field Officer position on a recent call with New Incentives. New Incentives, conversation with GiveWell, July 9, 2024 (unpublished)

• 278
  • New Incentives informed us that as of June 2024, base pay for the Field Officer position was 40% higher than the minimum wage in Nigeria, and that most Field Officers earned more than the base salary through bonuses, employer contributions, and expense buffers. New Incentives, responses to GiveWell questions, July 5, 2024 (unpublished)
  • New Incentives does not seem to have difficulty finding applicants for the Field Officer position. "From June 2023 to June 2024, we received approximately 20,000 external applications for the Field Officer position and recruited about 1,300 FOs in all states of operations during that same time period." New Incentives, responses to GiveWell questions, July 5, 2024 (unpublished)
  • Note that we have not reviewed New Incentives' compensation policy in detail and do not feel well-placed to judge whether the Field Officer salary is reasonable for the role. Our assumption that the Field Officer role is a decent job opportunity is based largely on the high number of applicants to the role and our assumption that unemployment among this population is fairly high.

• 279

  See IDinsight, Impact evaluation of New Incentives, final report, 2020, Table 13, pg. 36: "Outcome: Ever visited clinic… Adjusted OLS results: 0.05 ([95% CI] 0.03, 0.08)"

• 280

  We also received feedback from Dr. Jessica Cohen in her review of our report that this could cause us to underestimate impact: “Not including potential benefits of clinic visits for young children (where health concerns can be addressed in addition to the receipt of vaccinations) could cause you to underestimate impact.” Dr. Jessica Cohen, Bruce A. Beal, Robert L. Beal and Alexander S. Beal Associate Professor of Global Health, Comments on GiveWell’s draft New Incentives Report, November 2023 (unpublished).

• 281

  "Children who are HIV-infected when vaccinated with BCG at birth are at increased risk of developing disseminated BCG disease. However, if HIV-infected individuals, including children, are receiving ART, are clinically well and immunologically stable (CD4% >25% for children aged <5 years or CD4 count ≥200 if aged >5 years) they should be vaccinated with BCG. In general, populations with high prevalence of HIV infection also have the greatest burden of TB; in such populations the benefits of potentially preventing severe TB through vaccination at birth are outweighed by the risks associated with the use of BCG vaccine. Therefore, it is recommended that in such populations:

  • Neonates born to women of unknown HIV status should be vaccinated as the benefits of BCG vaccination outweigh the risks.
  • Neonates of unknown HIV status born to HIV infected women should be vaccinated if they have no clinical evidence suggestive of HIV infection, regardless of whether the mother is receiving ART.
  • Although evidence is limited, for neonates with HIV infection confirmed by early virological testing, BCG vaccination should be delayed until ART has been started and the infant confirmed to be immunologically stable (CD4 >25%)."

  World Health Organization, BCG vaccines: WHO position paper – February 2018, p. 95.
  Adverse events linked to BCG vaccination range from mild, localized complications to more serious, systemic or disseminated BCG disease in which M. bovis BCG is confirmed in one or more anatomical sites far from both the site of injection and regional lymph nodes. Disseminated BCG disease is associated with a case-fatality rate of >70% in infants. By comparison, the background mortality rate among South African HIV-infected infants is 12.2 per 100 person-years (95% CI: 8.2–17.4).
  Systemic or disseminated BCG disease may be clinically indistinguishable from tuberculosis and can only be confirmed through positive mycobacterial culture species identification, preferably by polymerase chain reaction (PCR) for the RD1 genetic region that is lost during attenuation of BCG" Hesseling et al. 2009.
  "Disseminated BCG disease is associated with a case-fatality rate of >70% in infants. . . . [The risk of disseminated BCG disease] has been shown to be 1100 to 4170 per 1 million in HIV-infected infants routinely vaccinated at birth." Hesseling et al. 2009.

• 282

  "Children who are HIV-infected when vaccinated with BCG at birth are at increased risk of developing disseminated BCG disease. However, if HIV-infected individuals, including children, are receiving ART, are clinically well and immunologically stable (CD4% >25% for children aged <5 years or CD4 count ≥200 if aged >5 years) they should be vaccinated with BCG. In general, populations with high prevalence of HIV infection also have the greatest burden of TB; in such populations the benefits of potentially preventing severe TB through vaccination at birth are outweighed by the risks associated with the use of BCG vaccine. Therefore, it is recommended that in such populations:

  • Neonates born to women of unknown HIV status should be vaccinated as the benefits of BCG vaccination outweigh the risks.
  • Neonates of unknown HIV status born to HIV infected women should be vaccinated if they have no clinical evidence suggestive of HIV infection, regardless of whether the mother is receiving ART.
  • Although evidence is limited, for neonates with HIV infection confirmed by early virological testing, BCG vaccination should be delayed until ART has been started and the infant confirmed to be immunologically stable (CD4 >25%)."

  World Health Organization, BCG vaccines: WHO position paper – February 2018, p. 95.

• 283

  "HIV PREVALENCE AMONG PERSONS AGE 15-64 YEARS: Zamfara 0.4%, Jigawa 0.3%, Katsina 0.3%" National Agency for the Control of AIDS, "Nigeria Prevalence Rate," 2019.

• 284

  "The risk of a reaction at the injection site following certain injected vaccines, such as DTaP (diphtheria, tetanus, and pertussis) or pneumococcal vaccine, increases if the doses are not separated by the recommended amounts of time. In these cases, it is the spacing of the doses, not the number of doses, that creates the risk. These reactions can be unpleasant, but they are not life-threatening." Centers for Disease Control, "Ask CDC - Vaccines & Immunizations".

  "We searched VAERS for US reports where an excess dose of vaccine was administered to a person received from 1/1/2007 through 1/26/2018. . . .
  More than three-fourths of reports of an excess dose of vaccine did not describe an AHE [adverse health effect]. Among reports where an AHE event was reported, we did not observe any unexpected conditions or clustering of AEs [adverse events]." Moro et al. 2019, abstract.

• 285

  As of October 2020, New Incentives considered 20% of their partner clinics’ catchment areas to be "high risk." This means one or more cases in which armed violence resulting in death has been recorded near the clinic, but New Incentives judged it to be safe to travel in the area during the day. The assessment was made by New Incentives Security Manager on the basis of information collected from a variety of sources, including field officer (FO) reports. New Incentives, Clinic and settlement security assessments (unpublished)

• 286

  "As of 17-June-2020, there have been 23 incidents in total. The cases include those related to minor theft (e.g. phone and sometimes cash), minor injuries (e.g. bruises and scorpion bites), and encounters with bandits." GiveWell, Questions for New Incentives about potential negative and offsetting effects, 2020, p. 1.

• 287

  New Incentives, Security Procedures and Status (unpublished), "Incidents Involving Staff" sheet:
  Incidents resulting in death:

  • "Year: 2017; Week: 47; A field staff in [redacted] . . . was involved in a road traffic accident (RTA) as he was traveling on a non-work day from [redacted] after visiting his relative in a hospital there. On his way back from the hospital, his vehicle got into an accident with a lorry, which put him in a critical condition. He was rushed to the hospital but unfortunately he passed away on 22-Nov."
  • "Year: 2018; Week: 36. On Tuesday 4/9/18 at about 1:00am armed bandit attacked [redacted], killed two people and kidnapped three people including two children of the district head. Reportedly, the bandit was specifically asking for [routine immunization (RI)] incharge of the clinic, saying that they were informed that the RI incharge and our staff (FV) [Field Volunteer] goes around some of their settlement and sometime in the clinic disbursing cash to some women."

  Incidents resulting in injuries:

  • "Year: 2018; Week: 29. One of our FVs . . . was involved in a motor accident on his way back from the outreach session. The tyre of the vehicle he boarded bursted and the driver lost control consequently hitting a roadside tree. [The FV] sustained an injury to the shoulder. He is stable presently and will seek further medical attention."
  • "Year: 2019; Week: 35. At [redacted], about 4 men armed with dangerous weapons attempted to burgle and rob an NI/ABAE staff residence in [redacted]. The armed youths took about 30 minutes attempting to break the barriers (doors and windows) and attacked neighbours who had responded to the distress calls of the residents of the apartment inflicting various degrees of injuries on the first responders. Upon arrival of security forces, the attackers withdrew from the residence and proceeded to raid at least two nearby houses where they robbed residents of cash and personal effects. Subsequently, members of the community mobilized in their numbers, confronted the robbers causing them to withdraw from the location."
  • "Year: 2019; Week: 46. Two of our FOs including the clinic staff assigned (VCHEW) had an accident today on their way back from the outreach session and sustained slight bruises, and in the process [name redacted] broke his phone and right now he is unable to send his data and do the end process for today."
  • "Year: 2020; Week: 4. Staff reported; 'On their way back from [redacted], they were involved in an accident. He said on their way back they were involved in an accident together with the RI staff, falling on a stone and sustaining injuries and a crack to his phone screen; though phone still remains functional."
  • "Year: 2020; Week: 25. FO was involved in a RTA. Someone with a car hit FO on a commercial bike and as a result FO's office phone got lost. FO sustained a minor injury on his right leg and was treated in a hospital. The person responsible for the accident has gone ahead to replace FOs phone."

• 288

  We last asked New Incentives for detailed information on its procedures in 2020. New Incentives has also shared a log of security monitoring activities, communications of security findings to staff, and security training for staff. GiveWelly lightly reviewed weeks 17-21 of 2021 and found that security processes were reported to have taken place as expected. We have not conducted a more detailed review since that time. See this section of our New Incentives monitoring analysis.

• 289
  • Collecting information about potential security threats and communicating with staff about threats. Information about security incidents is sourced on an ongoing basis by field officers (FOs), as well as by New Incentives' Security Manager and Security Focal Point. New Incentives has shared a list of these incidents with us. The list included an average of 40 reports per month from June 2019 (when this logging system was introduced) to October 2020 (the date of our review); it was up to date at the time of our review. We spot-checked records for 50 incidents: for all of them, it was indicated whether further steps should be taken to mitigate the risk and, if so, what action had been taken. When an ongoing and urgent threat is identified, the Security Manager directly contacts relevant FOs. To communicate less-urgent security information, the Security Manager compiles weekly summaries, which are circulated to managers, who share them with FOs. At times, New Incentives designates high-risk areas as "no go" and prevents FOs from creating work plans or submitting expenses for these areas in the app used for this purpose. We have seen the Security Manager's weekly reports for five weeks in 2020; these reports describe the process used to identify threats and designate areas as "no go."
  • Training staff to avoid security threats where possible and address them where necessary. The onboarding process for New Incentives staff members includes a UN safety training course and a training course on New Incentives' internal security procedures. Ongoing monthly training is also provided, covering topics including road safety and abduction and kidnapping. We have seen the Security Manager's weekly reports on onboarding training for five weeks in 2020.

  Sources:

  • FOs collect information on a rolling basis through drivers (while traveling to the clinic), town criers, and community members (during outreach sessions). When a clinic is marked as "high risk," FOs reach out to clinic staff and community members (for outreach sessions) the day before an immunization day to collect information about security threats. If the FO determines there is reason for concern, they have the option of not joining the outreach session. Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).
  • The Security Manager collects information from security industry networks and media on a rolling basis. The Security Focal Point makes calls to local government authorities every two weeks to collect information about security threats. Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).
  • "SM and SFP: Verifies all security reports received through myDay to ensure that these are true incident.Verifications entails - Reach out to our key sources of information (community stakeholders, security forces, security networks, staff, sister NGO security units) - Triangulation of information from more than one source - Consideration of the authenticity of each source." New Incentives, Security Procedures and Status (unpublished), "Security Procedures and Status" sheet, cell C26
  • "The security incidents are compiled into reports sent internally every week" New Incentives, Overview of NI-ABAE Anti-Bribery and Security Policies, p. 2.
  • "SFP: To collate and send out all security notifications to NC, SRD, SFMs, SM, COO, and external members" New Incentives, Security Procedures and Status (unpublished), "Security Procedures and Status" sheet, cell C14.
  • "SM: Reviews the Security Incident Review sheet to see if himself, FMs, SFMs or NC has taken action for cases where an action is required. This is a follow up of his responsibility to call the attention of the Manager concerned to a recommended action." New Incentives, Security Procedures and Status (unpublished), "Security Procedures and Status" sheet, cell C11.
  • "FM: Will discuss each FOs travel to the clinics or settlements based on highlighted Risk Rating for Disbursement Day or Field Activity, and submit this as part of the FM Check-in. FOs should be able to report if they are uncomfortable to travel based on the Security Assessment and Recommendations and recommend for activities to be cancelled." New Incentives, Security Procedures and Status (unpublished), "Security Procedures and Status" sheet, cell C13.
  • System: Displays Travel Safety Measures from Clinic and Settlement Security Assessments. Will not allow expenses or in-person activities to get logged against No Go clinics and settlements." New Incentives, Security Procedures and Status (unpublished), "Security Procedures and Status" sheet, cell C18.
  • "All Staff:1. Completion of UN B-SAFE and certificates in Zoho 2. Signed CSP + Quiz + Statement 3. Companion Card Received." New Incentives, Security Procedures and Status (unpublished), "Security Procedures and Status" sheet, cell C19.
  • New Incentives training includes (among other topics):
    • UN BSafe training: how to prepare for travel, emergency communications, and how to respond to violence
    • Training on the Country Security Plan, including standard operating procedures on road safety/travel, cash management, abduction, political, ethnic and religious instability, bad governance, and health risks.
    • Road safety training, including control measures to prevent life-threatening accidents, such as using seatbelts and enforcing speed limits.
    • Abduction and kidnapping, including procedures to follow in cases of suspected abduction.

  Svetha Janumpalli, Founder and CEO, and Pratyush Agarwal, Co-founder and COO, New Incentives, conversations with GiveWell, March 25-27, 2020 (unpublished).

• 290

  The number of security incidents reported involving New Incentives staff was 0.4 per 100 disbursement days in March - December 2020, 0.2 in 2021, 0.1 in 2022, and 0.1 in 2023. See this row of our monitoring analysis.

• 291

  New Incentives also operates in an environment where corruption is a general concern. In 2023 Nigeria scored 25/100 on Transparency International's Corruption Perceptions Index.
  

• 292

  Those controls include, but are not limited to, requiring photographic documentation of worksites throughout the day, checking that GPS data matches the expected work location, extensive data submission requirements, minimum activity durations, and setting caps on allowable expenses levels. If those requirements are not met, the expenses for the associated workday may be revised or rejected. For example, if a Field Officer were to give out cash to friends or family but record it as incentives disbursed during vaccination visits, those expenses would in theory be rejected for lack of corroborating data.
  

• 293

  "We made over 240 terminations in 2023 due to poor performance, negligence, and/or expense falsification. Though difficult, these terminations have been extremely helpful. We have learned that other approaches such as issuing other forms of Disciplinary Action or asking for falsified expenses to be reimbursed are insufficient. We have seen a shift in diligence and increased honesty since making these terminations. We have learned that what staff value most are long-term jobs with reliable pay and health benefits in a mission-oriented organization; terminations serve as an important fraud deterrent." New Incentives, Program Updates, January 5, 2024 (unpublished)

• 294
  • Further details on expense approvals: In February 2023, New Incentives introduced a policy of providing each Field Officer 10,000 naira per month to cover accidental errors in expense submissions that may lead to some submitted expenses not being reimbursed. In May 2024, 30% of Field Officers had expense revisions or rejections that exceeded the 10,000 naira buffer. For the majority of Field Officers (>60%) those extra costs were less than 10,000 naira. New Incentives reports that, from January to June 2024, 4% of total expense submissions were revised or rejected." New Incentives, responses to GiveWell questions, July 8, 2024 (unpublished)
  • On staff complaints: New Incentives has informed us that many current and former staff have complained about expense rejections and requested wage increases. New Incentives often shares this type of staff feedback in its monthly written program updates, most recently in its March 2024 program update. New Incentives, Program Updates, March 8, 2024 (unpublished)
  • New Incentives has told us that it has a ticketing system in place for staff to raise complaints and dispute unreimbursed expenses, and that it makes corrections for valid disputes.

• 295
  • On recruitment: "From June 2023 to June 2024, we received approximately 20,000 external applications for the Field Officer position and recruited about 1,300 FOs in all states of operations during that same time period." New Incentives, responses to GiveWell questions, July 5, 2024 (unpublished)
  • On staff retention:
    • "About 220 FOs resigned in 2023 and 96 in 2024 YTD (this includes staff members under active investigation for misrepresenting funds). We currently employ over 2,000 FOs." New Incentives, responses to GiveWell questions, July 5, 2024 (unpublished)
    • 220 / (2,000 + 220) = 9.9%. Note that this is intended to be a rough estimation of staff turnover, as we did not ask for the exact number of terminations/resignations or the exact number of employed Field Officers.

• 296

  Days on which clinics provide routine immunizations are called “immunization days”. See this section of our report on New Incentives’ program for more details.

• 297

  See this row in our summary of New Incentives’ monitoring.

• 298

  New Incentives, Incidents involving staff (unpublished):

  • "Year: 2017, Week: 49. On 10-Oct, staff reported the following: According to him . . . he carried motorcyclist to convey him to [redacted], about 3 km from colony then the motorcyclist said he want to ease himself, [he] agreed and the man stopped, immediately he stopped 2 men came out from the farm one with cutlass and the other with pistol, they asked him to take a small foot path which he . . . complied, as they went in they met two other guys each with cutlass and axe, he . . . saw the motorcyclist parked his own bike and joined them, they asked for money and handset which he . . . gave them, they searched all compartments of his back bag they could not find any thing because all the money is in his waist bag except transport money."
  • "Year: 2018, Week: 28. On 10-July, staff reported that 'Thieves broke into his home and Stole cash amounting to N71000 and his big Npower tab phone with 2 chargers and power bank. Staff also reported incident to the police."
  • "Year: 2019; Week: 6. This evening at about 8 pm I received a call from my FV . . . reporting that his room was buggled at [redacted] and all his valuable was stolen including the organization's phone."
  • "Year: 2019; Week: 20. On Sunday 12/05/19, at about 9:30 pm, FM got a call from [FO] who called and narrated how his phone was stolen as he went for evening prayers after breaking his fast. He mentioned that he had dropped his work phone inside his room to charge it, as the battery of the phone was very low. But when he returned from the mosque, he noticed that the phone has been stolen."
  • "Year: 2019; Week: 23. It was reported that today 06/06/19 at around 1:00 pm, armed bandit attacked 2 of our staff on their way back home from the . . . clinic. According to the staffs the attack took place at [redacted] immediately after crossing the river they met 2 bandits with 2 AK 47 rifles waiting for their arrival. After several questions from the bandit they said they are suspecting the bike men are informants, so they are going to hurt them, but finally they collected 1 office smartphone, 1 personal phone, N62,000 for Disbursement, and N13,000 for Transport from [redacted], and they collected N69,000 for Disbursement N13,000 only for the Transport, from [redacted], the incident took place about 4km to . . . clinic."
  • "Year: 2020; Week: 10. On 10-Mar along [redacted] road, it was reported that someone stole FOs phone on his way from [redacted] to [redacted] in public car. FO reported that he had the phone conveniently placed in his pocket during the trip only to discover when he alighted from the vehicle that the phone was no longer there. Despite making several efforts in retrieving the phone by going to the park to lodge same complain, it proved abortive."
  • "Year: 2020; Week: 16. Staff reported that: At [redacted], at night before sleeping, he plugged his phone to charge around 12:30 am as he always do, only to wake up in the morning around 6:43 am and could not find the phone. It is evident the phone must have been stolen because the door was not properly locked. Amongst the items stolen are; 1 NI/ABAE Phone and 2 other phones , belonging to his mother and brother."
  • "Year: 2020; Week: 20. At [redacted], staff reported that her home was burgled around 0300hrs on 14-May and valuables including her personal phone, generator set and twenty five thousand naira (25,000) belonging to NI/ABAE was stolen. She further reported that the thieves attempted to make away with her car when they were awoken by the car alarm. At the instance of the alarm, the thieves allegedly withdrew from the scene."
  • "Year: 2020; Week: 20. At [redacted], Staff went to the Bank to request for a Bank statement as requested by HR during one of his non-clinic operational days. Upon coming out from the bank, he realized his bike has been stolen. In a secret compartment of the bike is a pass issued to the staff to allow access during COVID-19 lockdown."

• 299

  New Incentives told us: "We believe that this is infrequent since the bandits rely on the community for support. Also, it is unlikely that bandits would target female caregivers traveling to the clinic who are receiving small amounts like N500 and N2,000. While we ask questions regarding caregivers losing part of their cash transfers due to bribes or 'dashes' during each disbursement, we do not specifically ask whether caregivers have experienced theft by third parties." GiveWell, Questions for New Incentives about potential negative and offsetting effects, 2020, pg. 1.

• 300

  As of 2020, New Incentives reports that there are cases in which this might have occurred: "While we don't have data, below are the cases where we anticipate that supply-side efforts could have led to shortages at other clinics in the state:
  Sometimes if there is a stockout at the LGA store, our clinic might request vaccines from a nearby clinic. These requests can be refused if the neighboring clinic does not think they have adequate additional stock to spare, we have seen many cases of refusals from nearby clinics. There are at least 87 cases recorded where we attempted to borrow vaccines from a nearby clinic (these cases can be found by searching for 'borrow' in the Clinic level Case Log).
  The consumption increase due to our clinics could cause a short-term shortage in the LGA (we have records of this happening in one of the LGAs in Zamfara with 5 of our partner clinics)." GiveWell, Questions for New Incentives about potential negative and offsetting effects, 2020, pg. 1.

• 301

  “Vaccines are allocated through a calculation made based on factors such as estimated population and consumption. Each LGA is allocated a consignment per quarter in which maximum and minimum stock levels are indicated. SCCOs can authorize LGAs to conduct vaccine ‘mopping’ in which LGAs can request stock from neighboring LGAs with an excess stock, but we do not expect the SCCO to authorize such a request if it would compromise those minimum levels. In general, LGAs are hesitant to participate in vaccine mopping to preserve adequate stock for their clinics and because they have no incentive to share their stock. NI promotes vaccine supply in all LGAs through transportation support to pull supplies (typically from the State’s Cold Chain storage) in both NI- and non-NI LGAs when needed. We are cognizant of the possibility that LGAs where we operate might get prioritized and try to take measures to avoid this.” New Incentives, email to GiveWell, October 24, 2022 (unpublished).

  “NI works with LCCOs and SCCOs to resolve stockout concerns in both operating and non-operating locations. In 2023, our team launched the Monthly RI Vaccine Round Table, a virtual meeting attended by ZCCOs and SCCOs from all states in NW and NE Zones. The goal of this meeting is to foster relationships amongst key partners, gain insight into the vaccine supply space, and proffer targeted solutions to supply challenges. NI also doesn’t limit its financial support for vaccine distribution to NI-operating states and LGAs. For example, in Q1 2023 we provided support to all the LGAs in Katsina for Measles antigen to be distributed to all the LGAs in the state, not only those where we operate.” New Incentives, Vaccine Supply-Side Dynamics: Follow-up, March 2023 (unpublished).

• 302

  See this row in our cost-effectiveness analysis. Note, we also consider other grantee-level factors in our analysis that we do not discuss in this report:

  • Risk of wastage:
    • Double treatment (children not benefitting from the program because they have already received it elsewhere). We account for children receiving more than one of the same vaccination separately in our analysis of program costs (here).
    • Ineffective goods (whether the program might use goods that are out of date, poorly made, or otherwise ineffective). We account for this separately in our adjustment for lower vaccine efficacy in Nigeria (here).
    • Goods purchased and left in storage until they expire. We exclude this factor because New Incentives does not itself distribute vaccines. More detail in this cell note.
  • Factors affecting our confidence in funds being used for their intended purposes:
    • Change of priorities (i.e., New Incentives uses funds designated for the core program for something else). We do not believe that this is a meaningful risk with New Incentives, which runs a single program.
    • Within-org fungibility (i.e., our funding frees up funding for New Incentives to use on another program). We do not believe that this is a meaningful risk with New Incentives, which runs a single program.

  We do not discuss these factors in this report because we assign a 0% downward adjustment in each case. See this section of our cost-effectiveness analysis for more details on our reasoning.

• 303

  See this section in our cost-effectiveness analysis.

• 304

  For information about the various audits for compliance and fraud that New Incentives conducts, see this section of our New Incentives report.

• 305

  "The Central Bank of Nigeria (CBN) recently announced that they will be replacing ₦200, ₦500, and ₦1,000 bills. They plan on starting this on December 15, 2022 while allowing time until January 31, 2023 to replace all the old bills. We expect that this could result in a temporary shortage of bills that can be used to disburse CCTs to caregivers and could result in a temporary increase in the number of infants not served." New Incentives, Program Updates, November 2022 (unpublished)

• 306

  See this row of our cost-effectiveness analysis.

• 307

  See this row in our cost-effectiveness analysis.

• 308

  See this section of our cost-effectiveness analysis.

• 309

  For details of how we estimate this, see this section.

• 310
  • We estimate that each dollar spent by the Nigerian government generates 0.005 units of value if used for other activities, and each dollar spent by Gavi generates 0.007 units of value.
  • In total, we think these actors spend ~$620,000 on vaccines per $1 million spent by New Incentives (~$340,000 for the Nigerian government, ~$284,000 for Gavi).
  • This implies that shifting these resources away from other activities results in 3,704 units of value being lost. (~340,000 x 0.005) + (~284,000 x 0.007) = 3,704.

  See this row in our cost-effectiveness analysis.

• 311

  See these rows in our cost-effectiveness analysis.

• 312

  3,334/82,884 = ~4%.

• 313

  See this row in our cost-effectiveness analysis.

• 314

  We estimate that the Nigerian government’s spending on other activities generates 0.005 of value per $, compared to 0.083 for New Incentives’ program in Bauchi. 0.005 / 0.083 = ~6%. See this section for how we generate these estimates.

• 315

  See this row in our cost-effectiveness analysis.

• 316

  7,783 / 82,884 = ~9%.

• 317

  See this row of our cost-effectiveness analysis.

• 318

  See these rows of our cost-effectiveness analysis.

• 319

  See this section of our cost-effectiveness analysis.

• 320

  See this section of our cost-effectiveness analysis.

• 321

  See this section of our separate report on New Incentives’ program for more details.

• 322

  Nigeria Strategy for Immunisation and PHC System Strengthening (NSIPSS), 2018.

• 323

  See Nigeria Strategy for Immunisation and PHC System Strengthening (NSIPSS), 2018, section 6.3, pgs 34 - 40.

• 324

  We learned about this program in 2019, but have not deeply investigated what has happened since then. As of 2020, our understanding was that in the three states where New Incentives worked at the time:

  • Katsina chose an education-focused condition for this program over a health-focused condition in part because it judged the health sector to already be better served than the education sector; knowledge of New Incentives' program may have affected that assessment.
  • Jigawa has chosen health-focused conditions for the transfers.
  • Zamfara had not selected a condition as of June 2020.

  Dr. Obinna Ebirim, National Coordinator, New Incentives, email to GiveWell, June 12, 2019 (unpublished):
  "Comment 1: In 2019-2023, the World Bank CCT program is being operated as an experiment and is only expected to reach a small portion of the population (<1% of the population for the conditional transfer; plus Katsina State is not including immunization in its conditions and we don't yet know whether Zamfara State will)
  Response to Comment 1: The World Bank CCT program is being operated as a program. The reason why the proportion of the entire population of the State that is benefiting is small (<1%) is because the target of the program is not the entire population but the poorest of the poor that are in the National Social Register."
  “Jigawa State chose health as their conditionality for the top up conditional cash transfer and got above average in the supply-side evaluation and re-evaluation.”
  “We are unsure to what extent our program has influenced the choice of Katsina State. Our last discussion with the Social Safety Net Office in Katsina State on this matter revealed that Katsina State prioritizes various sector including health and education but feels that health, unlike education, has been receiving lots of interventions by both Government and NGOs. The government official we spoke to gave examples of the government's Saving One Million Lives Program for Result (SOML-PforR) which cuts across Maternal, Newborn and Child Health (MNCH), the New Incentives CCTs for Routine Immunizations program, and Save the Children's nutrition program. An example of Katsina State government's investments in health is the State Emergency Routine Immunization Coordination Centre (SERICC); Katsina State is one of the 18 States in Nigeria that have the equivalent state setup of the National Emergency Routine Immunization Coordination Centre (NERICC). The purpose of these coordination centres is to utilize a war-room like approach of allocating more resources to closely monitor trends, identify issues and solutions, and implement them to improve routine immunization coverage."
  "Katsina State has been implementing the base unconditional cash transfer and has recently selected girl-child education as their conditionality. . . . Zamfara State is yet to choose a conditionality."
  "Based on our latest understanding, Zamfara is still at the base phase and yet to get to the top-up phase where the conditionality is chosen by the state." GiveWell, Questions for New Incentives about potential negative and offsetting effects, p. 2.

• 325

  See this row of our analysis.

• 326

  See this row of our cost-effectiveness analysis.

• 327

  0.005 / 0.083 = ~6%.

• 328

  See this cell in our supplementary analysis.

• 329

  Health

  • We use data from the Uganda National Health Expenditure Accounts in 2013-2014 (available here) as a proxy for how domestic governments in low-income countries allocate their health spending. In that year, approximately 30% of the health spending was allocated to HIV/AIDS, 20% to malaria, and the rest to other programs (see this column in our accompanying spreadsheet).
  • We roughly guess the dollar value required to save a life for eight of the largest categories of spending (e.g., we guess $30,000 of spending on HIV/AIDS programs is required to save one life (here), compared to $10,000 for malaria (here). As a simplifying assumption, we divide each category of spending into spending intended to save adult lives (e.g., HIV/AIDS), or spending intended to save under-five lives (e.g., malaria) (see this column in our accompanying spreadsheet). This allows us to calculate the weighted average cost required to save one child’s life ($12,254) and one adult life ($30,000).
  • Finally, we translate these figures into standardized units of value using GiveWell’s moral weights for the value of averting a child death (116 units) and adult death (73 units). We also use our estimates of the proportion of spending allocated to programs intended to save adult lives (55%) and under-five lives (45%). This results in an overall weighted average of 0.0056 units of value per US dollar spent on health programs by domestic governments. See this section of our supplementary analysis for our calculations.

  Education

  • As with health, we use data from Uganda as a proxy for how domestic governments in low-income countries allocate their education spending. Specifically, we rely on World Bank 2014 data that 7% of Uganda’s GDP in 2014 went towards primary education, 3% went towards secondary education and 2% went towards tertiary education (here). We then rescale these figures, resulting in estimates that 59% of education spending in Uganda goes towards primary education, 25% towards secondary education and 16% towards tertiary education (here).
  • As a simplifying assumption, we assume that the primary benefit of education spending is that it increases long-run income and consumption. We benchmark our estimates to our estimates of the impact of GiveDirectly’s unconditional cash transfer program (which we think also increases income and consumption). Specifically, we roughly guess that primary education spending is 100% as valuable as GiveDirectly’s program, secondary education is 70% as valuable and tertiary education is 50% as valuable (here).
  • We take a weighted average of these values, using the % of government spending on each category of education as weights. This results in an overall estimate of 0.0028 units of value per $ spent on education programs by domestic governments. See this section of our supplementary analysis for our calculations.

  Social security

  We use a similar approach to estimating the value of social security spending as education spending. Specifically:

  • We use data from Uganda as a proxy for how domestic governments in low-income countries allocate their social security spending. We use World Bank 2014 estimates that social security spending accounted for 0.62% of Uganda’s GDP (here). Within this, 0.29% of GDP is in-kind spending, 0.1% is social pension spending, 0.1% is conditional cash transfers, and 0.05% is other cash transfers (here).
  • We roughly guess the value generated by each of these four largest categories of social security spending, excluding the smaller categories. As with education spending, we use the simplifying assumption that the primary benefit of social security spending is that it increases income and consumption. Benchmarking to GiveDirectly’s cash transfer program, we guess that cash transfers generate 100% of the value of GiveDirectly’s program and in-kind and social pension spending each generate 70% of the value (here).
  • We take a weighted average of these values, using the % of government spending on each category of social security spending as the weights (re-scaled to ignore smaller categories). This results in an overall estimate of 0.0026 units of value per $ spent on social security programs by domestic governments. See this section of our supplementary analysis for our calculations.

• 330

  See here in our cost-effectiveness analysis.

• 331

  See this row of our cost-effectiveness analysis.

• 332

  0.007 / 0.083 = ~8%.

• 333

  For example, pledges to Gavi in 2020 (for the 2021-2025 period) exceeded its replenishment ask: “US$8.8 billion worth of new commitments were pledged by public and private sector donors towards Gavi’s replenishment, well exceeding the target set in support of Gavi’s investment opportunity for the next 5 years. These pledges will add to the US$1.7 billion previously secured by Gavi, bringing its resources for 2021-2025 to more than US$10.5 billion. Gavi is now well positioned to accelerate the roll out of current vaccines, reach 300 million more children in developing countries by 2025, and in so doing contribute to saving 7 to 8 million lives.” Global Vaccine Summit London 2020.

• 334

  The rate of stockouts of any vaccine rose from ~8 per 100 disbursement days during the RCT to ~45 per 100 disbursement days in 2023. While we see this as concerning, we do not currently see it as a significant threat to the program because the increase in stockouts is concentrated heavily in indirectly incentivized vaccines, particularly the rotavirus vaccine (which was only introduced in the Nigerian immunization schedule in August 2022, see Gavi, "Dealing with diarrhoea: Nigeria introduces rotavirus vaccine into its immunisation plan," August 30, 2022). These form a relatively small share of the overall benefits of the program. More here and in this row of our summary of New Incentives' monitoring data.

• 335

  "New Incentives will group local government areas (LGAs) it expands to within a given state at a given point in time into ‘expansion groups’. New Incentives will then collect coverage data in these expansion groups once before the start of operations to establish baseline coverage rates." IDinsight, Coverage monitoring analysis plan, 2021, pg. 1.
  For the dates of the surveys we have received from New Incentives, see here.

• 336

  Some of the estimates we use in our main analysis of vaccine effects rely in part on meta-analyses that include some observational studies. See the footnotes in this section for more details.

• 337

  King et al. 2020: Each percentage point increase in PCV coverage was associated with a 1-2% decrease in non-traumatic infant mortality:

  • Results from Study 1: "Adjusted negative binomial regression of monthly mortality against three-dose coverage found every 1 percentage point increase in coverage was associated with a 2.0% (95% CI: 1.5 to 2.4; p value<0.001) decrease in mortality"
  • Results from Study 2: “Across 354 geographical clusters, three-dose PCV13 coverage ranged from 58% to 100% and mortality ranged from 0 to 87/1000 live births. Cluster-level impact analysis, using negative binomial regression adjusted for RV1 coverage, found that every 1% absolute increase in three-dose coverage by geographical cluster was associated with a 1.3% reduction in non-traumatic infant mortality (95% CI: 0.3% to 2.4%; p value=0.02).”

  Bar-Zeev et al. 2018: Each percentage point increase in vaccine coverage was associated with a 1.1% decrease in diarrhea-related mortality. “Two-dose vaccine coverage ranged from 63·6% to 100% across clusters; each percentage point increase in vaccine coverage was associated with a 1·6% (95% CI 0·8–2·5) lower diarrhoea-associated mortality rate (appendix). Adjusting for sociodemographic covariates, the reduction was 1·1% (95% CI 0·9–1·3).”
  Sifuna et al. 2023: A coverage increase from 0-75% was associated with a 44% reduction in all-cause child mortality. “Following its introduction in late 2014, the span of rotavirus vaccine coverage for children increased to 75% by 2017. Receiving the rotavirus vaccine was associated with a 44% reduction in all-cause child mortality (95% confidence interval = 28-68%, p < 0.0001), but not diarrhea-specific mortality (p = 0.401).”

• 338

  See GiveWell’s grant page to IRD Global for more details.

• 339

  “US$8.8 billion worth of new commitments were pledged by public and private sector donors towards Gavi’s replenishment, well exceeding the target set in support of Gavi’s investment opportunity for the next 5 years. These pledges will add to the US$1.7 billion previously secured by Gavi, bringing its resources for 2021-2025 to more than US$10.5 billion. Gavi is now well positioned to accelerate the roll out of current vaccines, reach 300 million more children in developing countries by 2025, and in so doing contribute to saving 7 to 8 million lives.” Global Vaccine Summit London 2020

  “Over US$7.5 billion was pledged by 31 public and private sector donors, which means that Gavi, the Vaccine Alliance is well positioned to achieve these milestones, expand vaccine coverage, strengthen health systems and accelerate the introduction of new vaccines. The pledges announced today will be combined with US$2 billion in resources already committed for the 2016-2020 period to enable Gavi to meet the US$9.5 billion cost of funding vaccine programmes in developing countries over the next five year period.” Gavi, Reach Every Child: Gavi Pledging Conference, 2015

• 340

  “Respondents who had either never vaccinated their children or had missed one or more vaccinations (89.8% of the sample), were asked the question 'Why have they not received these vaccinations?'. Responses were categorized by enumerators into one of 18 possible categories”. IDinsight, New Incentives evaluation baseline report, 2019, p. 48.

• 341

  IDinsight, New Incentives evaluation baseline report, 2019, figure 14, p. 49.

• 342

  We received feedback in 2022, in a submission to GiveWell’s “Change our Mind” contest, that we should use an epidemiological model rather than a static model: “Our first critique is the core modelling approach taken for this cost-effectiveness model. We understand the model developed by GiveWell to be a static model; the gold standard of models for vaccination cost effectiveness are dynamic models (1). Models such as Susceptible-Infected-Recovered (SIR) models may be especially suited. However, we understand from the literature that it is not straightforward to estimate the extent to which these types of models provide differing estimates to static models.” Naik and Field 2022, p. 1.

• 343

  "The World Health Organization (WHO) has recommended a new vaccine, R21/Matrix-M, for the prevention of malaria in children. . . . The R21 vaccine is the second malaria vaccine recommended by WHO, following the RTS,S/AS01 vaccine, which received a WHO recommendation in 2021." World Health Organizatgion, "WHO recommends R21/Matrix-M vaccine for malaria prevention in updated advice on immunization," 2023.

• 344

  We estimate that each dollar donated to GiveDirectly’s direct cash transfer program generates 0.003 units of value. 0.005 / 0.00335 = ~1.5.

• 345

  We estimate that each dollar donated to GiveDirectly’s direct cash transfer program generates 0.003 units of value. 0.007 / 0.00335 = ~2.