Published: August 2018
Note: This page summarizes the rationale behind a GiveWell Incubation Grant to New Incentives. New Incentives staff reviewed this page prior to publication.
Summary
As part of GiveWell’s work to support the creation of future top charities, in November of 2017, New Incentives received a GiveWell Incubation Grant of $5,944,203 to support its program offering conditional cash transfers for infant vaccination in North West Nigeria. This grant is the first (and major) tranche of a grant intended to fully support New Incentives' operations through May 2020, for the duration of a randomized controlled trial (RCT) of the program.1 Separately, as part of our ongoing partnership, IDinsight will receive GiveWell Incubation Grant funding to conduct the RCT and analyze its results.
Table of Contents
The organization
History
GiveWell has recommended several grants to New Incentives since it began operating conditional cash transfer (CCT) programs in Nigeria in 2014.2 New Incentives currently operates a program that offers CCTs to incentivize infant vaccination in Nigeria.3 This is the third iteration of its program.
Initially, New Incentives distributed CCTs for maternal delivery in health facilities and treatment-seeking for the prevention of mother-to-child transmission of HIV. When it became clear that New Incentives was unlikely to reach enough HIV-positive expectant mothers for this program to be highly cost-effective, it expanded to incentivize facility-based delivery more broadly for women with high-risk pregnancies. GiveWell conducted a review of the independent evidence for the effect of facility-based delivery on infant mortality and determined that facility delivery would not be a GiveWell priority program. Following conversations with GiveWell, New Incentives decided to pivot to CCTs incentivizing infant vaccination, which is a GiveWell priority program. New Incentives first started exploring CCTs to incentivize infant vaccination in the fall of 2014. In November of 2016, New Incentives received a GiveWell Incubation Grant to support this programmatic transition. In 2017, New Incentives began to operate its infant vaccination CCT program in states in North West Nigeria.
Leadership
New Incentives is run by Svetha Janumpalli (Founder and CEO), Patrick Stadler (Co-Founder and Chief Strategy Officer), and Pratyush Agarwal (Co-Founder and Chief Operating Officer).4 Ms. Janumpalli has been our primary contact at New Incentives, though we have also spoken regularly with Mr. Agarwal and Mr. Stadler. GiveWell has communicated extensively with New Incentives over the past few years, and overall we have a strong positive impression of the organization:
- We have communicated well with its leadership.
- New Incentives has provided thoughtful, detailed answers to our critical questions.
- New Incentives has proactively shared a great deal of detailed programmatic data with us. We have never seen New Incentives hesitate to share information on its program with us (unless it had what we considered to be a good reason).
- New Incentives has been very transparent in allowing us to write about its programs and publish documents.
The intervention
New Incentives provides conditional cash transfers (CCTs) to parents of infants who receive vaccines at clinics in North West Nigeria, where infant vaccination rates are low.5 Its program model involves hiring and training field staff to attend immunization days at health clinics and disburse cash transfers after verifying that a vaccination was given, monitor clinics to ensure that vaccines are in stock and not expired, and provide photographic monitoring that cash transfers are given in the correct amount to parents whose infants have received a vaccine.6 The cash transfers are in the amount of 500 naira (about $1.40 at December 2017 exchange rates) for each of the first four vaccine visits (at birth, 6 weeks, 10 weeks, and 14 weeks) and 2,000 naira (about $5.60 at December 2017 exchange rates) for the 9-month measles vaccine.7 We do not have good data on the income of participants in New Incentives' program, but we believe that these transfers are small enough to be non-coercive.8
New Incentives previously ran a pilot of this program in central and southern Nigeria, in addition to testing the program at ten "learning sites" in North West Nigeria.9 The pilot found that the program increased retention of infants who received at least one vaccination in a clinic.10 New Incentives staff believe that this increase in retention is likely to be replicated at the learning sites and in the RCT in the North West.11
It is our impression that vaccination is a cost-effective means of preventing illness and death in young children.12 We think it is plausible that small cash incentives are effective at significantly increasing the vaccination rate of young children in areas that have low vaccination uptake but suitable supply of vaccines, though we are highly uncertain about what effect size to expect from New Incentives' program. We have conducted a preliminary investigation into CCTs for infant vaccination but have not completed a full report. Our preliminary cost-effectiveness analysis of New Incentives13 indicates that this is plausibly a fairly cost-effective intervention, though we have substantial uncertainty in our model as it currently stands. In our review of studies of incentives for vaccination, we found little evidence specifically relevant to New Incentives' context (a low-vaccination context in sub-Saharan Africa), but we think the literature on small incentives for behavior change suggests that an effect is plausible.
Goals for the grant
The purpose of this grant is to support New Incentives' program of CCTs for infant vaccination in North West Nigeria from November 2017 to May 2020, comprising the full expected duration of a randomized controlled trial (RCT) of the program. IDinsight has separately received funding to conduct the RCT as part of its partnership with GiveWell. Our goal for this grant, and for the RCT of New Incentives' program, is to inform an evaluation of New Incentives as a potential GiveWell top charity by November 2020. Regardless of its outcomes, we expect to write publicly about the RCT and about our view of incentives for vaccination as an intervention. Details on the proposed study design of the RCT as of June 2017 (the most recent version that was available at the time we decided to recommend the grant) are available here.
Budget
In November 2017, New Incentives received a total of $5,944,203 toward its program through May 2020, which it plans to allocate as follows:14
- $1,370,216 (23%) to U.S. and Nigerian management costs. The total management cost is $1,599,461 but the difference is covered by a previous grant.15
- $1,697,855 (29%) to cash transfers.
- $778,183 (13%) to field costs for clinics, including clinic screenings.
- $1,463,289 (25%) to other field costs related to CCTs.
- $634,660 (11%) to phase out the program in May-September 2020 in the event that the RCT finds a clearly negative result and/or GiveWell decides not to pursue the program further. If the program does not phase out, this will be put toward New Incentives' ongoing operating costs.
This disbursement of $5,944,203 in November of 2017 is the first of two expected tranches of a grant supporting New Incentives' program for the duration of the RCT. GiveWell recommended a total grant amount of up to $6,689,277, intending for the second tranche of the grant to allow flexibility in the number of clinics to be randomized to receive the program. As of this writing, we expect that the second tranche of this grant will additionally fund an increase in the duration of the preparatory period prior to the start of the RCT (when infants are born and begin receiving vaccines).
RCT timeline
The rough projected timeline for the RCT is described below.16 This timeline is approximate and we would not be surprised if some portions are delayed.
- August-October 2017: Baseline data collection.17
- November 2017-January 2018: Rollout period. Staff are trained and the program is activated at clinics.
- February-June 2018: Ramp-up period. Incentives are disbursed, but mother-infant pairs enrolled during this period are not considered part of the RCT.
- July-October 2018: RCT infants are born.
- April-July 2019: RCT infants become eligible for measles vaccination.
- August-November 2019: Grace period for RCT infants to receive measles vaccination.
- November 2019: Endline survey. This date is approximate and has been deliberately left somewhat flexible. The endline survey will evaluate vaccination outcomes in infants 12-16 months old living in the catchment areas of clinics participating in the RCT.
- December 2019-March 2020: Analysis. These dates may be subject to change.
- April-June 2020: Results expected. These dates may be subject to change.
Monitoring & evaluation
We have reviewed New Incentives' monitoring procedures at a shallow level. Our impression is that New Incentives conducts high-quality program monitoring (i.e. checking for beneficiary fraud and ensuring that beneficiaries receive both the incentive and the vaccines). New Incentives' monitoring processes include:18
- Field staff gathering data (such as evaluations of vaccine supply and beneficiary records) in online forms.
- Photographing beneficiaries holding bills as proof that they are receiving the correct amount, which enables back-checks by remote contractors.
- Field staff keeping records of vaccination, which can be checked against clinic records (though we expect clinic recordkeeping to be low quality).
- Stamping child health cards in such a way as to prevent beneficiary fraud (detail in footnote).19
The upcoming RCT constitutes a strong impact evaluation of the program and represents the first rigorous evaluation of New Incentives' impact on vaccination rates. We do not expect that it is feasibly cost-effective to experimentally evaluate New Incentives' impact on health outcomes.
Details on the proposed study design of the RCT as of June 2017 (the most recent version that was available at the time we decided to recommend the grant) are available here.
Cost-effectiveness of New Incentives' program
At the time we recommended the grant, our best guess was that New Incentives' program is approximately five times as cost-effective as unconditional cash transfers (UCTs), which is competitive with the cost-effectiveness of our current top charities.20 We have since made some revisions to our model that have marginally increased our estimate of the program's cost-effectiveness.21 However, our cost-effectiveness analysis is in its early stages, and we believe it may change significantly as we receive more information and revise our model. We are highly uncertain about a number of important parameters, including the expected effect of the program on vaccination rates and parameters related to disease incidence and mortality.
Our rough cost-effectiveness model at the time we decided to recommend the grant suggested that:22
- If New Incentives increased vaccination rates by 6.1 percentage points, it would be about 3 times as cost-effective as UCTs, which would make it competitive with our current top charities.
- If New Incentives increased vaccination rates by 17 percentage points, it would be about 12 times as cost-effective as UCTs and 2 times as cost-effective as the Against Malaria Foundation, which would make it a strong candidate for receiving marginal GiveWell-directed funds.
While we are very unsure what effect size to expect from New Incentives' program, a 17 percentage point increase seems intuitively plausible:
- A baseline survey that IDinsight conducted in 2017 in two of the three states where the RCT will take place found very low vaccination rates for children 12-16 months old (WHO/UNICEF estimates of national vaccination rates for an unspecified age group in parentheses for comparison): 24.4% (53%) had received Bacille Calmette-Guerin (BCG) vaccination, 21.4% (49% DTP or pentavalent vaccine) had received at least one dose of pentavalent vaccination, 5.7% (42% DTP or pentavalent vaccine) had received the full three doses of pentavalent vaccination, and 15.4% (42%) had received a measles vaccination.
- IDinsight, New Incentives Evaluation Baseline Report, Pg 22, Table 2a: Immunization Coverage for 12 to 16-month olds Across Katsina and Zamfara. Publication of the full report from this baseline survey is forthcoming.
- National comparison values are from the WHO/UNICEF Estimates of National Immunization Coverage under tabs BCG, DTP1, DTP3, (representing vaccination with DTP or with pentavalent vaccine in countries where pentavalent vaccine is available) and MCV1 (first routine dose of measles vaccine). We do not know the age range of children represented in these statistics.
- New Incentives staff have told us that preliminary observations at its 10 learning sites suggest that vaccination rates in the target population may be even lower than expected, which leaves substantial room for an increase in vaccination rates.
- New Incentives has told us that, qualitatively, it has observed a large increase in demand for vaccination (crowding in clinics) at the pilot sites and the 10 learning sites due to the program.23
Because we expect our cost-effectiveness estimate to change as we gain more information, and because in general we expect such changes to be negative rather than positive, our current expectation is that New Incentives will need to demonstrate a somewhat more robust effect than a 6.1 percentage point increase in order to be a strong top charity candidate.
Cost-effectiveness of the grant
Based on our rough calculations,24 we think that this grant is slightly more cost-effective, in expectation, than the counterfactual use of the grant funding. Our rough estimate is that the grant is 3.2 times as cost-effective as UCTs, whereas the counterfactual (donating the money to the current top charities that we model as having the most cost-effective funding gaps) is roughly 2.4 times as cost-effective as UCTs, in expectation. These estimates incorporate our best guesses about several factors related to top charities in the future, such as the probability that we will find new highly cost-effective giving opportunities, our top charities' room for more funding, and the amount of money moved by GiveWell's top charity recommendations.
They also incorporate the probability that New Incentives will become a top charity and the implications thereof. It is reasonably likely that this grant will inform our decision of whether or not to consider New Incentives as a potential top charity. Given our relatively high credence that New Incentives has at least enough of an effect on vaccination rates to make it a top charity contender and our estimate that it has a large amount of room for more funding to scale up in North West Nigeria,25 there is a good chance that this grant will lead to the positive identification of a new GiveWell top charity. This encompasses two possible outcomes:
- Based on this RCT and further evaluation, we may recommend New Incentives as a top charity, but not as our top recommendation for marginal donations. In this scenario, we would recommend that New Incentives receive some funding (including an incentive grant and possibly capacity-building funds) that otherwise would have gone to our top recommendation for marginal donations, and we would have added a new type of intervention to those implemented by our top charities.
- Based on this RCT and further evaluation, we may recommend New Incentives as a top charity, and as our top recommendation for marginal donations. In this scenario, the RCT will have caused funding to move from the counterfactual (the charity or charities that would otherwise have been our top recommendation for marginal donations) to New Incentives. We roughly think about the value of this outcome as the amount of GiveWell-directed money moved to New Incentives over the coming years multiplied by the difference between the value of New Incentives and the value of the counterfactual, divided by the cost of the grant.
Fungibility
In the absence of this grant, there is a very low chance that the RCT would take place, since it is a collaborative project designed for the purpose of informing GiveWell's decision making. In the absence of this grant, New Incentives might either seek operational funding from other sources to support its incentives for vaccination program or switch to a different program activity.
Speculative long-term benefits
Apart from the direct impacts of the program, we believe that there may be long-term, difficult-to-quantify positive returns to creating new, outstanding organizations that are more cost-effective than UCTs. These potential returns include:
- Demonstrating to the nonprofit sector and academics that such organizations can be created, which may increase their future involvement in starting or supporting such organizations.
- Increasing the pool of non-GiveWell funders for cost-effective organizations or programs by providing a proof of concept for the program in question. This could include increasing large funders' interest in cost-effective organizations and/or increasing governments' interest in adopting such programs. We expect that increasing the base of non-GiveWell funders for these organizations would make these organizations more sustainable over the longer term, since GiveWell is unlikely to continue to be a major source of funding for a given program indefinitely.
We also think it is possible that New Incentives' model could eventually serve as a platform for small incentives for other health behaviors and could lead to improvements in clinic service delivery (e.g. vaccine stockage, staff capacity, and recordkeeping).
We did not include any of these potential benefits in our decision to make this grant because they are very speculative and difficult to quantify.
Risks and reservations
Concerns about study design
The full details of the proposed study design for this RCT as of June 2017 (the most recent version that was available at the time we decided to recommend the grant) are available here. There are several elements of the study design that we think are not ideal (though we believe that the study is likely to be informative despite these issues):
- Sample size: Due to a number of practical constraints, the RCT's sample size is smaller than would be ideal for clearly informing our decision about whether New Incentives has sufficient impact to be a GiveWell top charity.
- Practical constraints include:
- In order to manage spillover effects (i.e. women in the catchment areas of control clinics traveling to treatment clinics in order to receive the incentive), there must be some buffer distance between treatment and control clinics. Hence, if clinics were too close together, only one would be selected to participate in the RCT (as either a control clinic or a treatment clinic), or else the clinics would be randomized as a pair and contribute only one additional combined cluster catchment area to the RCT sample size.
- More clinics than expected were excluded due to either being located in insecure areas or being too remote for New Incentives staff to access as frequently as would be necessary during the RCT or in the program at scale (about once per week).
- More clinics than expected were found to have only one staff member available for vaccination activities. Since New Incentives is uncertain whether such clinics would be able to handle the increases in clinic volume that it expects to come as a result of the program (and therefore is uncertain whether it will include these clinics in its program at scale), it decided to include only some of them in the RCT.
- IDinsight expressed concern that if the sample size were too large, it would become increasingly challenging to hire and train enough survey enumerators to complete surveys in a timely manner, leading to possible data quality concerns.
- Study power: According to IDinsight's power curve calculations, including more clinics in the study would not substantially increase the study power. These calculations rely on some assumptions (such as effect size and inter-cluster correlation) about which we have little information, and if the assumptions are inaccurate, it is possible that the study power will be lower than expected.
- Impact on informativeness of the study: If the program's impact on vaccination rates is just large enough for us to consider New Incentives as a potential top charity (a roughly 6 percentage point increase), there is a decent chance that the study would fail to detect this effect. However, if its impact is large enough to make a robust case for it to become a top charity (a roughly 13 percentage point increase) or large enough to make New Incentives a candidate for marginal GiveWell-directed funds (i.e. robustly more cost-effective than AMF; a roughly 17 percentage point increase), the study is unlikely to fail to detect these effects.
- Practical constraints include:
- Representativeness of the program at scale:
- Since New Incentives is uncertain whether the program at scale will operate in clinics that have only one staff member or are somewhat security-compromised, we were uncertain whether to include such clinics in the RCT and ultimately decided to include only some such clinics.
- Due to the need for a geographical buffer between treatment and control clinics, the study includes proportionally fewer urban clinics than will be included in the program at scale (since urban clinics are generally closer together and therefore more susceptible to spillover effects). This could cause the RCT to overestimate the impact of the program if it has a higher impact in rural populations (for example, due to a lower baseline vaccination rate). It is also possible that this could cause the RCT to underestimate the impact of the program if it has a higher impact in urban populations (for example, due to greater receptiveness in urban areas), though this seems intuitively less likely to us. We expect that subgroup analysis will reveal whether the program has different effects on rural clinics compared to urban ones.
- New Incentives' program is fairly new, and it is deliberately limiting publicity during the RCT to minimize spillover from treatment to control clinics. By the time the program is operating at scale, it will have had more publicity, and trust in the program will likely increase after it has been operating for a longer period of time (assuming the program has built a positive reputation in this time).
- New Incentives' learning clinics have initially experienced high patient volumes. This has led to longer wait times as the clinics adjust to the higher volumes of patients on immunization days and as they work through the backlog of older infants who are eligible for vaccinations. We expect this to be the case at the RCT clinics as well, which could possibly reduce the measured effect of the program by driving away some potential participants. We expect that the multi-month ramp-up period before the RCT will partially address this, but that the problem will be further reduced in the program at scale.
- Compact segment sampling: Due to the size of this RCT, IDinsight expressed that it would be infeasible to hire and train enough high-quality enumerators to survey every household at endline while maintaining data quality. Furthermore, surveying at every household in a clinic’s catchment area would not be cost-effective, as the marginal costs of surveying in each additional household provide diminishing returns to the study’s power and precision. For these reasons, the RCT used compact segment sampling during the baseline survey and plans to do so for the endline survey as well. Compact segment sampling involves dividing settlements within clinic catchment areas into segments geographically (for example, by using Google Maps) and surveying every household within the randomly selected segments. IDinsight believes this technique is challenging due to uncertain population data and settlement boundaries, and because not all structures identified in satellite images are residential. IDinsight will verify the location of survey areas prior to the endline survey. GiveWell does not have sufficient expertise in study design to independently determine the costs and benefits of using compact segment sampling or using a combination of approaches to surveying within the same study. The forthcoming pre-analysis plan will describe the compact segment sampling method in detail.
- Reduction of spillover effects: Spillover effects of the program could occur if people in non-treatment areas hear about the program and travel to the nearest treatment clinic rather than to a nearer control clinic. In order to mitigate this, the study will include buffer areas between control and treatment sites and randomize some neighboring clinics as pairs. The RCT was initially planned to take place in two states (Katsina and Zamfara), but all stakeholders agreed to add a third state (Jigawa) in order to include more clinics while maintaining buffer areas.
There are early indications that caregivers are not traveling long distances to reach treatment clinics; data from learning sites in June 2017 suggested that 97% of caregivers are coming from settlements less than 5 kilometers away from a clinic;26 however, we find it plausible that caregivers might travel longer distances, especially for a 2000 naira incentive for measles vaccination. The program has not yet been widely marketed, and New Incentives intends to expand marketing of the program after the RCT. We think it is plausible that increases in social marketing might make caregivers more likely to travel longer distances to a treatment clinic.
- Data quality: The study will measure vaccination status by checking a combination of a) child health cards retained by caregivers, which should have a record of any vaccinations the child has received, b) caregivers' reports of vaccinations received by a child, and c) clinic vaccination registers. We do not expect any one of these sources to yield particularly high-quality data. New Incentives roughly estimates that about 50% of caregivers can produce their child's health card; we are unsure how reliable caregivers' reports are; and we have a general impression that clinic records are not particularly strong. We believe that combining these methods and cross-checking the results is the most feasible means of determining infants' vaccination status. The forthcoming pre-analysis plan will include additional details on how vaccination status will be measured.
All of these methods are common in studies of vaccination coverage; IDinsight considered using other methods and concluded that they would be infeasible. In particular, biomarker testing would produce more reliable results, but is logistically difficult.
Clinics in the RCT may receive an intervention aimed at strengthening data quality, such as assistance with the supply of child health cards. It is our hope that any support or technical assistance offered during the RCT would also be part of the program at scale, to avoid overestimating the impact of the program at scale. It is possible that the program itself may improve the quality of administrative records.
- Informativeness of the primary outcome of the RCT: Since the RCT is designed to measure an impact on vaccination rates, rather than on disease incidence or mortality, it is possible that the measured impact of the program could be higher than the true impact.
- While the vaccines used are generally considered efficacious, they may lose potency due to interruptions in the cold chain or supply chain problems. New Incentives has told us that all clinics use vaccine biomarkers that indicate whether a vaccine has been exposed to heat.27 Additionally, New Incentives monitors vaccine expiry, but it is more complicated to monitor vaccine potency, and to our knowledge, there are few existing studies of vaccine field efficacy. IDinsight considered using biomarkers (i.e. blood or saliva tests) to determine immune status, but concluded that this would be prohibitively challenging. Challenges include the logistics of obtaining and transporting biosamples (including maintaining cold chain), cultural acceptability, and accuracy of biomarker tests for various vaccines.
- Because the RCT measures the program's impact on vaccination rates, not disease incidence or mortality, we must make assumptions about disease incidence and mortality when evaluating the impact of the program on child mortality. There are not good sources of data on the incidence and fatality of the relevant diseases in North West Nigeria.
Nigeria's government conditional cash transfer program
As part of a broader National Social Safety Nets Project, the Nigerian federal government is preparing to implement a five-year conditional cash transfer (CCT) program targeting poor or vulnerable populations. We do not know the implementation timeline of this program, and because of its large scale, we expect that it may be subject to delays. Each participating state will select one area of focus from a list of possible conditionalities for the cash transfers, one of which includes child vaccination status.28 New Incentives and IDinsight have told us that all three of the states involved in the RCT (Katsina, Zamfara, and Jigawa) are participating in the government CCT program, and that the government CCT program is geographically targeting approximately one third of each state. We do not yet know which condition each state will select. Our understanding is that the condition selected may change over time.29
Households targeted by the program will receive a guaranteed monthly transfer of 5,000 naira (roughly $14 USD) and an additional 5,000 naira upon fulfillment of the selected condition.30 The government aims to reach a minimum of one million households in at least 24 states, which the World Bank estimates is approximately 10% of the poor.31 We believe that this leaves substantial room for New Incentives to have an impact, even if states choose the vaccination condition. However, it seems likely that, if successfully implemented, the presence of the government program would affect the magnitude of New Incentives' impact.
According to the World Bank, the expected closing date of the program is June 30, 2022.32 GiveWell will likely be evaluating New Incentives as a potential top charity in 2020, so it is likely that the government program will be in place when we are making this decision. We are concerned about several possible outcomes of the potential coexistence of New Incentives' and the government's CCT programs, in terms of both the effectiveness of New Incentives' program and on the quality of the data we get from the RCT:
- If any of these states select vaccination status as the condition for the CCTs, this could potentially reduce New Incentives' impact on vaccination rates by increasing the baseline vaccination rate in the absence of New Incentives' program.
- It is possible that a government CCT for vaccination program could begin in the near future but end earlier than planned, such that it is no longer operating by the time we are evaluating New Incentives as a potential top charity (in 2-3 years). In this scenario, if a government CCT for vaccination program begins rolling out partway through the RCT, the RCT may underestimate the impact that New Incentives would have in the absence of the government CCT program.
- From GiveWell's perspective, the worst-case scenario (in terms of the impact of the CCT program on the usefulness of the RCT data) would be one in which one or more of the RCT states choose vaccination as the condition for the transfer and the government CCT program is eventually implemented, but is initially ineffective or delayed. This would notably decrease our sample size for measuring the impact of New Incentives' program in the context in which it will ultimately operate (i.e. alongside the government CCT program). We are highly uncertain about the likelihood of this scenario.
If none of the states in the RCT select vaccination status as the condition, we think it is quite unlikely that a government CCT for vaccination program will exist in these states in 2-3 years.
We considered delaying the RCT until we had more information about the government CCT program to increase our confidence that we were evaluating the program in the context in which it will be operating in 2-3 years, when we may evaluate New Incentives as a potential top charity; we decided to move forward with the RCT because we estimate that the government CCT program will only affect a small proportion of the RCT population and because we do not expect to learn significantly more about the timeline of the government program in the near term.
Implementation challenges
- The program is likely to stress clinic staff capacity and lead to long wait times on immunization days. New Incentives is monitoring this challenge closely and working with clinics to mitigate it.
- Clinics sometimes run out of vaccines.33
- Some potential beneficiaries may be suspicious of the program, which could reduce uptake.
- In North West Nigeria, it is traditional for postpartum women and neonates to stay indoors for a few weeks to recover from birth and to limit the infant's exposure to a foreign environment. This tradition poses a challenge to vaccinating infants on schedule (the BCG vaccine is recommended as soon as possible after birth, and the next three vaccination visits are recommended at 6, 10, and 14 weeks), though it is not necessarily a challenge to starting and following the vaccination schedule slightly later.34
Lower than expected cost-effectiveness due to unavailability of vaccines
For our estimation of the cost-effectiveness of New Incentives' program as a potential 2020 top charity, we assume not only the availability of all vaccinations currently available as part of routine vaccination in Nigeria, but also the availability of vaccination for rotavirus. Rotavirus vaccination is not currently available through the routine health system in Nigeria. As of this writing, the World Health Organization estimates that rotavirus vaccination will be available in October 2019.35 However, we would not be surprised if roll-out of the vaccine is delayed or takes longer than anticipated. We are moderately comfortable with an assumption that rotavirus vaccination will be available by 2020. However, it is possible that rotavirus vaccination, which is an important component of the estimated cost-effectiveness of the program, will not be available to program participants by the time we are evaluating New Incentives as a potential top-recommended charity.
Possible negative impacts of the grant
While we think it is unlikely that this grant will do harm, the main ways in which we can imagine it having a negative impact are:
- Reducing the likelihood that caregivers return with their infants for future vaccinations: Despite New Incentives' warnings, its learning clinics were unprepared for the large increases in patient volumes caused by the program on immunization days. This led to long wait times, and in some cases, caregivers had to return with their infants on a later immunization day. We think it is possible that resulting frustration and wasted time may make caregivers less likely to return to the clinic for future vaccinations. New Incentives' program now includes working with clinics and stakeholders to better manage clinic volumes, so the RCT will demonstrate whether the program as a whole (including some potential negative effects on caregivers' wait times) ultimately increases vaccination rates. We expect that caregivers will generally consider it worthwhile to vaccinate their infants and receive the cash incentive despite possibly longer wait times. New Incentives notes that caregivers who are not served the same day are given a small tally card to encourage them to return the next week. This was requested by caregivers to increase the likelihood that their husbands would give them permission to return to the clinic.
- Drawing resources away from nearby clinics: It is possible that the program will have a negative effect on clinic operations generally, on clinic or health system priorities, or on nearby clinics that are not part of the program. In particular, we have discussed concerns with New Incentives that the health system at the local government area (LGA) or state level may redirect clinic staff to clinics where New Incentives works to help manage higher clinic volumes on immunization days, typically one day per week. This may have a negative effect on nearby clinics from which staff may be redirected.
Internal forecasts
We are experimenting with recording explicit numerical forecasts of the probability of events related to our decision-making (especially grant-making). The purpose of this exercise is to record the implicit predictions that inform our decisions and to make it possible for us to look back on how well-calibrated and accurate those predictions were. For this grant, we are recording the forecasts below, all of which we consider to be fairly rough. Except where otherwise noted, the end date for all predictions is the end of 2020.
- New Incentives increases vaccination rates by >17 percentage points and this is detected by the RCT: 15%
- New Incentives increases vaccination rates by >17 percentage points and this is not detected by the RCT: small probability, close to 0%
- New Incentives increases vaccination rates by between 6 and 17 percentage points and this is detected by the RCT: 55%
- New Incentives increases vaccination rates by between 6 and 17 percentage points and this is either not detected by the RCT or is unclear: 15%
- New Incentives increases vaccination rates by <6 percentage points and we either conclude as much or are uncertain enough that we choose not to pursue New Incentives further: 15%
- New Incentives increases vaccination rates by <6 percentage points and we falsely believe it is higher and do pursue New Incentives further: 5%
- After seeing the RCT results, we are significantly uncertain about whether or not to recommend New Incentives as a top charity: 20%
- GiveWell estimates that New Incentives is >3x as cost-effective as GiveDirectly: 50%
- GiveWell estimates that New Incentives is >2x as cost-effective as AMF: <10%
- New Incentives becomes a top charity by November 2020: 50%
Plans for follow-up
Our plans for following up on this grant include:
- Continuing to be somewhat involved in ongoing discussions regarding the RCT and New Incentives' progress in scaling up to RCT size.
- Reviewing midline results of the RCT in 2019. At this time, we expect that the study will include a midline survey using administrative data sources; however, these plans may change.
- Revising our cost-effectiveness model of New Incentives.
- Reviewing preliminary endline results of the RCT.
We expect to have final results from the RCT in 2020, in time to review New Incentives as a potential 2020 top charity. If the results are clearly negative, New Incentives plans to close down its program. This grant provides enough funding to do so.36 In this scenario, it is possible we would consider recommending a grant to support New Incentives' leadership on the next step in their careers. We estimate this would be a maximum of a year's salary for each of three staff members, or roughly in the range of $250,000. We are not sure whether we would move forward with this, since we expect that New Incentives leadership would have about six months to think about their next steps while closing down the program.
If the results are clearly positive,37 we expect to evaluate New Incentives as a potential top charity in 2020 and consider whether to recommend that donors donate to New Incentives. If we expect to direct additional funding to New Incentives, and New Incentives has a need for funding to bridge the period between the end of the RCT and expected end-of-year donations, we will also consider this funding gap.
If the RCT returns an inconclusive result,38 we expect to spend a few months determining whether it would be plausible for us to conduct additional research at a relatively low cost. In this scenario, we expect to provide additional funding, either to enable New Incentives to close down at a responsible rate or to support additional evaluation.
Key questions for follow-up
- Operations
- Does New Incentives encounter any major challenges to operating in RCT clinics?
- Over time, how do clinics adapt to higher volumes?
- Are there indications that the program has an effect (positive or negative) on the surrounding health infrastructure?
- Midline
- How (if at all) does the midline suggest that we should update our expectations regarding the program's effect on vaccination rates?
- Are there any early indicators of differential effects by various characteristics (e.g. rural vs. urban settings)?
- Endline
- What is the program's effect on vaccination rates?
- Are there any differential effects by various characteristics?
- Modeling
- How can we improve our model of the effects of vaccination on child mortality?
Our process
As part of our collaboration with IDinsight to aid in the creation or identification of additional GiveWell top charities, together with New Incentives we developed a plan to evaluate New Incentives' impact with an RCT. This is the first time GiveWell has been so closely involved in planning an RCT. We have had extensive discussions with IDinsight and New Incentives throughout this process on topics including:
- Power calculations and the number of clinics to include in the study
- Inclusion or exclusion of clinics based on clinic criteria such as staff capacity, weekly travel distance for New Incentives staff, and security
- Compact segment sampling
- Data collection methodology
- Size of transfers for the measles vaccine
- Nigeria's government CCT program
Sources
- 1
The grant also includes funding for the program to phase out in 2020, which New Incentives may choose to do in the event that the RCT does not find a promising result or GiveWell decides not to pursue further evaluation of the program. If the program does not phase out, we expect that this funding will be used to support New Incentives' ongoing operational costs.
Because of uncertainty in the full amount needed for New Incentives' operations through May 2020, in November of 2017, New Incentives received a GiveWell Incubation Grant covering the minimum expected budget need for this time period and is approved for up to $745,074 in additional funding based on the final size of the RCT.
- 2
For more detail, see our page about our first grant to New Incentives.
- 3
"Conditional Cash Transfers (CCTs) are small sums of money that individuals in need can earn after meeting various education and health benchmarks such as school attendance or clinic visits. In the case of New Incentives, your donation gives poor mothers in rural West Africa small stipends on the condition that they vaccinate their children against deadly diseases. The stipends allow the women to afford transport to the clinic where the vaccinations are provided and food for their families." New Incentives webpage: How It Works
- 4
- 5
- "New Incentives’ program offers incentives for certain vaccines (chosen based on cost-effectiveness and expected mortality reduction), while other vaccines are monitored for increased uptake but not directly incentivized. New Incentives has now disbursed over 27,000 conditional cash transfers (CCTs) to mothers after verifying that their child has been immunized." GiveWell's non-verbatim summary of a conversation with Svetha Janumpalli, Pratyush Agarwal, and Patrick Stadler on August 2, 2017, Pg 1.
- IDinsight's 2017 baseline survey (in two out of three states where the RCT will take place) found very low vaccination rates for children 12-16 months old (WHO/UNICEF estimates of national vaccination rates for an unspecified age group in parentheses for comparison): 24.4% (53%) had received Bacille Calmette-Guerin (BCG) vaccination, 21.4% (49% DTP or pentavalent vaccine) had received at least one dose of pentavalent vaccination, 5.7% (42% DTP or pentavalent vaccine) had received the full three doses of pentavalent vaccination, and 15.4% (42%) had received a measles vaccination.
- IDinsight, New Incentives Evaluation Baseline Report, Pg 22, Table 2a: Immunization Coverage for 12 to 16-month olds Across Katsina and Zamfara. Publication of the full report from this baseline survey is forthcoming.
- National comparison values are from the WHO/UNICEF Estimates of National Immunization Coverage under tabs BCG, DTP1, DTP3 (representing vaccination with DTP or with pentavalent vaccine in countries where pentavalent vaccine is available), and MCV1 (first routine dose of measles vaccine). We do not know the age range of children represented in these statistics.
- 6
- "On each vaccination day, New Incentives staff perform a brief supply-side check to verify the vaccine cold chain has not been broken. This involves checking the vaccine vial monitors and also verifying the vaccines in stock have not expired." IDinsight's 2017 report of a site visit to New Incentives, p. 5.
- [On vaccination days at a clinic, after an infant receives vaccination, they are seen by a New Incentives staff member.] "Once the NI field staff member confirms the baby’s eligibility, the staff member pays the mother the appropriate amount and records this figure three times: electronically using a smartphone, on a paper tally sheet, and by taking a photo of the mother with her cash. The field staff member also takes a photo of the mother’s child health card so that other NI staff can verify that the field staff is correctly determining eligibility. Before the mother leaves, the staff member applies a blue dot to the child health card to guard against double payment." IDinsight's 2017 report of a site visit to New Incentives, p. 6.
- "All clinics in Nigeria use vaccine bio-monitors that indicate when a vaccine has been exposed to heat. So far, there have been no issues with accessing the vials and no issues with vaccine quality or expiration." GiveWell's non-verbatim summary of a conversation with Svetha Janumpalli and Patrick Stadler on December 16, 2016, Pg 3.
- 7
- New Incentives and IDinsight conducted a small experiment testing the effect of different measles vaccine incentive sizes on measles vaccine uptake, and found that there was no clear trend that increasing the incentive amount resulted in an increase in vaccine uptake rates for groups initially receiving 2,000 and 3,000 naira. (Publication of the results from this experiment is forthcoming.)
- IDinsight, New Incentives Evaluation Baseline Report, Pg 11, Table 1: Schedule of Immunizations Incentivized by New Incentives. Publication of the full report from this baseline survey is forthcoming.
- For more details of the New Incentives program, see IDinsight's 2017 report of a site visit to New Incentives.
- 8
- In 2014-2016, as part of the enrollment process for its programs in southern and central Nigeria promoting the prevention of mother-to-child transmission of HIV and delivery in a health facility, New Incentives asked enrollees about their daily income and their husbands' daily income. New Incentives noted to us that income data quality was low because many women do not work during pregnancy and many do not know the income of their husbands. Out of 8,632 respondents, 842 (10%) reported on income earned themselves (average 9,530 naira per month, or $1.06 per day), and 416 (5%) reported on their husband's income (average 32,939 naira per month, or $3.66 per day). New Incentives, Income Statistics and Use/Spending of Cash Transfers Dashboard
- A manuscript on a 2013 maternal vaccination intervention in Adamawa state, North East Nigeria, notes that, "the average daily earnings per household is approximately 1,000 naira and that per person is 144 naira in our sample; the average transportation cost to and from the health clinic is about 250 naira among those who need to pay for the transportation, while 50 percent of women do not pay for the transportation," Sato and Takasaki 2016, footnote pp. 4-5.
- Sato, Ryoko and Yoshito Takasaki. "Peer Effects on Vaccination: Experimental Evidence from Rural Nigeria." CIRJE Discussion Paper F-1002 (2016).
- 9
- "New Incentives is running a pilot of its incentives for immunization program at sites in central and southern Nigeria." Pg 1.
- "New Incentives is also conducting learning activities in ten clinics in North West Nigeria, where it plans to conduct an RCT." Pg 2.
- 10
"New Incentives is running a pilot of its incentives for immunization program at sites in central and southern Nigeria. New Incentives’ program offers incentives for certain vaccines (chosen based on cost-effectiveness and expected mortality reduction), while other vaccines are monitored for increased uptake but not directly incentivized. New Incentives has now disbursed over 27,000 conditional cash transfers (CCTs) to mothers after verifying that their child has been immunized. The cash transfers have increased retention among infants who come to a clinic for at least one immunization:
- 89% of infants in the pilot who had turned 10 weeks old had completed their 10-week immunization visit, compared to a baseline of 66%.
- 87% of infants in the pilot who had turned 14 weeks old had completed their 14-week immunization visit, compared to a baseline of 54%.
- 90.9% of infants in the pilot who had turned 9 months old had completed their 9-month measles immunization visit, compared to a baseline of 31%.
"The baseline figures used here are weighted averages of the number of infants from each clinic whose due date for a given visit has passed. These figures include data only from infants who have been brought to a clinic at least once for an immunization; they do not reflect the percentage of the total population of infants who receive these immunizations.
…
"The baseline immunization rates above are based on administrative data that may be unreliable. New Incentives believes it has taken a conservative approach to interpreting the data such that, if anything, the impact of the program will be underestimated."
- 11
"New Incentives is also conducting learning activities in ten clinics in North West Nigeria, where it plans to conduct an RCT. New Incentives suggests that the high retention rates in the pilot program suggest that in the learning sites and RCT in the North West, once infants are enrolled in the program, retention (receipt of later vaccines) will be high." GiveWell's non-verbatim summary of a conversation with Svetha Janumpalli, Pratyush Agarwal, and Patrick Stadler on August 2, 2017, Pg 2.
- 12
- 13
For more detail on the cost-effectiveness model we used when deciding to recommend this grant, see GiveWell's 2017 preliminary cost-effectiveness analysis of New Incentives.
A more recent version of our cost-effectiveness model that includes baseline vaccination rates is available here: GiveWell's early 2018 preliminary cost-effectiveness analysis of New Incentives. Our rough estimate of the bottom line cost-effectiveness of New Incentives' program has not changed significantly from the previous version.
- 14
For more detail, see New Incentives RCT Budget November 2017 - May 2020.
- 15
For this calculation, see New Incentives RCT Budget November 2017 - May 2020, tab "RCT Budget", cell C9, which calculates "Total US and Nigerian management costs through May 2020" as $1,599,461 and subtracts ($407,546.67/16)*9, the portion of these costs which are covered by the previous "Learning Grant."
- 16
- As of August 2017, the projected RCT timeline was as follows:
- "August-October 2017: baseline survey
- October-December 2017: roll out, i.e. activating clinics and training staff
- January-March 2018: ramp up, i.e. the program will operate for three months before enrolling infants as part of the RCT. This allows the program time for community engagement and to ensure that staff are working to high standards before beginning the RCT.
- April-July 2018: infants to be enrolled in the RCT are born
- August 2018-July 2019: infants complete their immunization schedule"
- This timeline was shifted later by one month because GiveWell recommended this grant in November 2017 instead of October 2017 as anticipated.
- Since August 2017, we have gained more detailed timeline information from conversations with New Incentives and IDinsight staff.
- In March 2018, we discussed with New Incentives the possibility of a 2-3-month delay in the start of the RCT (from May 2018 to July or August 2018) to allow the program a longer "ramp-up" period, in which incentives are disbursed but enrolled mother-infant pairs are not considered part of the RCT. Any agreed-upon extension of the RCT timeline beyond May 2020 will be funded by an additional tranche of GiveWell Incubation Grant funding.
- As of August 2017, the projected RCT timeline was as follows:
- 17
A mass measles vaccination campaign was scheduled for the last week of October 2017. In order to avoid confusion among respondents that might lead to inaccurate survey responses, IDinsight, New Incentives, and Hanovia (a survey firm) worked quickly to complete baseline data collection in Katsina and Zamfara states prior to the campaign.
We have seen draft results from the baseline survey, and publication is forthcoming.
- 18
- "New Incentives has developed a console approach to increase quality and reduce costs. This approach includes the following elements:
- Demographic details and immunization history are collected on each woman enrolled in the program.
- The field worker marks a dot on each bill the mother receives and takes a photo of her with her baby's ID number and the cash transfer in her hand. The dots assist online workers who review the photos and verify how much cash recipients received.
- An independent online worker counts how much cash is in the photo. Any discrepancies between the cash in the picture and the field staff’s report are reported.
- A cash management console agent follows a detailed seven-page protocol on what to verify and triangulate and updates this information in a dashboard in real time." Pgs 3-4.
- "New Incentives is now focusing on maintaining feedback loops to staff, maintaining high quality in its verifications of immunizations and disbursement of cash, and continuing close monitoring of clinics’ vaccine supplies." Pg 4.
- "New Incentives has developed a console approach to increase quality and reduce costs. This approach includes the following elements:
- 19
"The cash management console agents identified one case of beneficiary fraud in which a staff member in charge of disbursement reported giving out 115,000 naira, and it was later discovered during a verification process that only 111,000 naira of this was given to people who enrolled in the program. The other 4,000 naira was paid to the mothers of two infants who were not enrolled in the program.
"When women enroll their infant in the program, they receive a sticker on the child's health card with a unique ID number on it, as well as an additional card for the New Incentives program. In the above instance of beneficiary fraud, mothers enrolled in the program gave their stickers to the mothers of two infants not enrolled in the program who had not received immunizations, who were then mistakenly given cash transfers. New Incentives staff believe that these mothers pretended to be in the program rather than actually enrolling because the enrollment period had already ended.
"New Incentives has since created a procedure to prevent this type of beneficiary fraud from recurring. Under the new system, when a woman is enrolled in the program, a sticker is placed on the child's health card and stamped such that the stamp marks both the sticker and the card, so that it will be obvious if the sticker is moved to a different card. This type of problem has historically taken a lot of management time to address, and the new system should enable New Incentives to prevent fraud with less active engagement as it scales up.
"It would have been relatively easy to implement a technological fix to this problem, such as having the staff member in charge of disbursement look up the beneficiary number in a database and check whether that person has received the immunizations and the cash transfer. However, loading the database may have slowed down the process and resulted in delays that would prevent New Incentives from disbursing cash transfers to women as soon as their infant is vaccinated."
- 20
Unconditional cash transfers are the program implemented by GiveDirectly, one of our top charities, against which we benchmark the cost-effectiveness of our other top charities. We roughly estimate that our other top charities range from 4 to 12 times as cost-effective as GiveDirectly.
In giving season 2017, we estimated that the top charities to which we recommended marginal funding were six times as cost-effective as GiveDirectly.
- 21
For more detail, see the 2017 and 2018 versions of our cost-effectiveness model:
- 22
GiveWell's 2017 preliminary cost-effectiveness analysis of New Incentives, "Test effect sizes for power calcs" tab, rows 29-30.
Note: Our model has been updated since we made the grant decision, so the model no longer reflects the numbers we list here. We expect our model to continue to evolve, and therefore do not place significant weight on its outputs. The numbers we list here are those we used in making the grant decision.
- 23
"New Incentives is running a pilot of its incentives for immunization program at sites in central and southern Nigeria. New Incentives’ program offers incentives for certain vaccines (chosen based on cost-effectiveness and expected mortality reduction), while other vaccines are monitored for increased uptake but not directly incentivized. New Incentives has now disbursed over 27,000 conditional cash transfers (CCTs) to mothers after verifying that their child has been immunized. The cash transfers have increased retention among infants who come to a clinic for at least one immunization:
- 89% of infants in the pilot who had turned 10 weeks old had completed their 10-week immunization visit, compared to a baseline of 66%.
- 87% of infants in the pilot who had turned 14 weeks old had completed their 14-week immunization visit, compared to a baseline of 54%.
- 90.9% of infants in the pilot who had turned 9 months old had completed their 9-month measles immunization visit, compared to a baseline of 31%.
"The baseline figures used here are weighted averages of the number of infants from each clinic whose due date for a given visit has passed. These figures include data only from infants who have been brought to a clinic at least once for an immunization; they do not reflect the percentage of the total population of infants who receive these immunizations.
…
"The baseline immunization rates above are based on administrative data that may be unreliable. New Incentives believes it has taken a conservative approach to interpreting the data such that, if anything, the impact of the program will be underestimated."
- 24
See GiveWell's 2017 BOTEC for New Incentives grant for more details.
- 25
We expect New Incentives' room for more funding to scale up its program in North West Nigeria to be in the tens of millions of dollars per year. We believe that the program would likely be less cost-effective in other regions of Nigeria where baseline vaccination rates are higher.
- 26
A journey under 5 km is measured as a journey costing less than 200 naira:
- "Currently, 97% of women are coming from settlements less than 5 kilometers away, as defined by being a journey that costs less than 200 Naira.
- Furthest journey in the clinic records was 1000 naira to go 15 km.
- The furthest journey in exit interviews was 300 Naira to go 8km."
IDinsight, Spillover Risk & Baseline Supervision, July 2017, Pg 3.
- 27
"All clinics in Nigeria use vaccine bio-monitors that indicate when a vaccine has been exposed to heat. So far, there have been no issues with accessing the vials and no issues with vaccine quality or expiration." GiveWell's non-verbatim summary of a conversation with Svetha Janumpalli and Patrick Stadler on December 16, 2016, Pg 3.
- 28
- For all World Bank documents related to this project, see World Bank National Social Safety Nets Project
- World Bank Project Appraisal Document, National Social Safety Nets Project:
- "Through this component, the Government aims to cover at least one million households, or approximately 10 percent of the poor, over a five-year period in at least 24 states. Each household will receive a base transfer of NGN 5000 (US$25, equal to about 15 percent of the national poverty line) per month. This base benefit is designed to ensure a minimum, reliable payment to promote consumption smoothing, address food insecurity and protect against economic shocks. Targeted households will be eligible for an additional monthly benefit of NGN 5000 upon the completion of agreed co-responsibilities." Pg 12.
- "Initially it is expected that most states will opt for “softer” co-responsibilities or accompanying measures such as awareness seminars on nutrition, family practices, health, sanitation, education, productive practices, and training sessions. Co-responsibilities are expected to evolve over the life of the project to include more activities related to use of health and education services (that is, use of health centers, primary school enrollment and attendance, girls’ senior secondary school attendance) as the participating states improve both supply side and project management capacity.
…
"The project has developed a menu of co-responsibilities around health and nutrition, education, environment and productivity (box 1). Each state will, depending on their conditions and priorities, choose their co-responsibility area.
Box 1. Menu of Co-responsibilities
Education:- Attend workshops on importance of child education
- Attend literacy classes
- Primary school enrollment
- Primary school attendance
- Girls’ attendance of senior secondary schools years 1–3
Health and nutrition:
- Attend health and nutrition education workshops, including family planning and maternal health
- Immunization of children
- Use of pre-natal and post-natal services
- Child growth monitoring
Environment:
- Attend workshops on hygiene and sanitation
- Attend workshops on environmental activities, such as tree planting
- Participate in local tree planting initiatives
Productivity:
- Open bank accounts and attend financial literacy training
- Attend group meetings and coaching sessions" Pg 13.
- World Bank Implementation Status & Results Report, National Social Safety Nets Project
- 29
"Initially it is expected that most states will opt for “softer” co-responsibilities or accompanying measures such as awareness seminars on nutrition, family practices, health, sanitation, education, productive practices, and training sessions. Co-responsibilities are expected to evolve over the life of the project to include more activities related to use of health and education services (that is, use of health centers, primary school enrollment and attendance, girls’ senior secondary school attendance) as the participating states improve both supply side and project management capacity." World Bank Project Appraisal Document, National Social Safety Nets Project, Pg 13.
- 30
- "Through this component, the Government aims to cover at least one million households, or approximately 10 percent of the poor, over a five-year period in at least 24 states. Each household will receive a base transfer of NGN 5000 (US$25, equal to about 15 percent of the national poverty line) per month. This base benefit is designed to ensure a minimum, reliable payment to promote consumption smoothing, address food insecurity and protect against economic shocks. Targeted households will be eligible for an additional monthly benefit of NGN 5000 upon the completion of agreed co-responsibilities." World Bank Project Appraisal Document, National Social Safety Nets Project, Pg 12.
- Based on an approximate 2018 conversation rate of 1 naira = 0.0028 USD, 5,000 naira is equivalent to approximately $14, rather than the $25 cited in the quote above. XE Currency Converter: NGN to USD
- 31
"Through this component, the Government aims to cover at least one million households, or approximately 10 percent of the poor, over a five-year period in at least 24 states." World Bank Project Appraisal Document, National Social Safety Nets Project, Pg 12.
We do not know what criteria the World Bank is using to ascertain poverty status.
- 32
World Bank Project Appraisal Document, National Social Safety Nets Project, Pg vii.
- 33
- "So far, stock-outs have not been a significant issue at most clinics, but vaccines have run out during immunization days in some facilities." GiveWell's non-verbatim summary of a conversation with Svetha Janumpalli and Patrick Stadler on December 16, 2016, Pg 4.
- "Supply-side monitoring, as well as clinics adjusting to changes in vaccine demand, seems to have reduced the frequency of vaccine stock-outs. Of the 109 "immunization days" that clinics hosted in July, there were six instances of a clinic running out of a vaccine in the middle of the day and two stock-outs; this is fewer stock-outs than there were during New Incentives' initial work at its learning sites." GiveWell's non-verbatim summary of a conversation with Svetha Janumpalli, Pratyush Agarwal, and Patrick Stadler on August 2, 2017, Pgs 4-5.
- 34
"Barriers to getting mothers to bring their infant to a clinic in the first 30 days include a) a cultural norm that encourages mothers and infants to stay indoors for the first few weeks after birth and b) for many beneficiaries, it takes a whole day to go to the clinic and back, so they often delay this for a few weeks after birth or after hearing about the program." GiveWell's non-verbatim summary of a conversation with Svetha Janumpalli, Pratyush Agarwal, and Patrick Stadler on August 2, 2017, Pg 2.
- 35
World Health Organization, Vaccine-Preventable Diseases Global Summary for Nigeria. See table "Immunization Schedule," listing "Rotavirus" as "from October 2019."
- 36
The grant includes about $600,000 to cover the costs of phasing out the program over the course of five months. If the program does not phase out, this funding will be used to support New Incentives' ongoing operations.
See New Incentives RCT Budget November 2017 - May 2020 for more details.
- 37
If the RCT were to find a >6 percentage point increase in vaccination rates with p<0.05, we would consider this a clearly positive result. However, the study is not powered to detect this effect. A more likely scenario in which we determine that the outcome is clearly positive would be roughly a 12 percentage point increase in vaccination rates with p<0.05.
- 38
If the RCT were to find a 4-6 percentage point increase in vaccination rates (which, according to our cost-effectiveness model at the time we made the grant decision, implies that it would be 2-3x as cost-effective as GiveDirectly), we may or may not decide to recommend New Incentives as a top charity depending on other factors about the organization and its program.
If the RCT were to find a >6 percentage point increase in vaccination rates but with a higher p-value than expected (for example, due to reduced study power as a result of an unforeseen event), we may consider providing additional funding to continue evaluating the program, or we may make another decision based on available information.