We’re planning to release updated top-charity recommendations in mid-November, and one of the questions our staff has been debating recently is whether to recommend New Incentives as a top charity.
We’ve decided that New Incentives doesn’t currently meet our criteria for a top charity because its program doesn’t have sufficient evidence supporting it. However, we have been extremely impressed with and think very highly of New Incentives’ staff and are considering how best to support them in the future and incentivize others to found an organization like they did.
In this post, we summarize the answers to the key questions we asked to determine whether New Incentives meets our criteria for a top charity recommendation and the options we’re considering for future support.
Background
New Incentives operates a conditional cash transfer (CCT) program in Nigeria to incentivize pregnant women to deliver in a health facility. New Incentives originally intended its CCT program to focus primarily on prevention of mother-to-child transmission (PMTCT) of HIV. However, under this model the program did not reach enough HIV-positive pregnant women to justify its operating costs, and in 2015, New Incentives expanded its program to target both HIV-positive women and HIV-negative women.
New Incentives was the first organization we supported as part of our experimental work to support the development of future top charities. It has been about two and a half years since New Incentives received its initial grant, and it now has a long enough track record implementing its program to be considered for a top charity designation.
Is New Incentives’ intervention evidence-backed?
New Incentives’ impact is made up of three components: (a) delivering cash to very poor people, (b) incentivizing HIV-positive pregnant women to deliver in clinics and get the medicines that prevent mother-to-child transmission of HIV, and (c) incentivizing pregnant women to deliver their babies in a health facility.
Because a relatively small portion of New Incentives’ beneficiaries are HIV-positive, because it costs New Incentives more than GiveDirectly to deliver each dollar, and because it is likely reaching individuals with higher incomes than GiveDirectly does, the impact that has the dominant effect on our view about whether or not New Incentives meets the standard we have for a top charity’s cost-effectiveness is the impact of facility delivery on neonatal mortality.
The evidence we have for the impact of facility delivery comes from (1) relevant randomized controlled trials (RCTs), (2) monitoring that New Incentives carries out, and (3) non-RCT evidence on the impact of facility delivery.
Overall, the evidence from the RCTs increases our confidence that an intervention that offers improved neonatal care could have a significant impact on neonatal mortality, but the evidence we have seen and New Incentives’ current monitoring of its program is insufficient to convince us that increasing the number of women who deliver at facilities has a similar impact.
Randomized controlled trial evidence
Two RCTs of low-intensity training programs for traditional birth attendants found significant (30-45%) reductions in neonatal mortality. These interventions are different than New Incentives’ intervention but may have a similar effect since they aim to increase the knowledge of traditional birth attendants so that they offer similar care to that which is offered in health facilities. We did not find any RCTs on facility delivery itself; these two RCTs are the most similar ones to New Incentives’ program that we identified. The interventions varied:
- In Gill et al. 2011, the intervention group received training and supplies related to common practices to reduce neonatal mortality immediately following birth. The study observed significant differences between the treatment and control group on practices such as drying the baby with a cloth and then wrapping it in a separate blanket (as opposed to using the same blanket), clearing the baby’s mouth and nose with a suction bulb (instead of a cloth), and using a pocket resuscitator (instead of mouth to mouth) (see Table 5, Pg. 8). We have not closely vetted this study but note some significant-seeming differences between the treatment and control birth attendants–in particular, the treatment group had significantly more education than the control group (see Table 1, Pg. 4).
- In Jokhio et al. 2005, the intervention group received supplies and 3 days of training focused on antepartum, intrapartum, and postpartum care, including activities such as: “how to conduct a clean delivery; use of the disposable delivery kit; when to refer women for emergency obstetrical care; and care of the newborn.” The intervention group was “asked to visit each woman at least three times during the pregnancy (at three, six, and nine months) to check for dangerous signs such as bleeding or eclampsia, and to encourage women with such signs to seek emergency obstetrical care.”
New Incentives’ monitoring
New Incentives’ staff interviews a nurse and conducts additional inspection at each health facility it considers working with. New Incentives reports the results of these interviews. Two questions are most relevant to our assessment of the similarity between the interventions studied in the RCTs discussed above and the care offered in facilities New Incentives works with.
New Incentives asks nurses at each health facility: 1) “What multiple steps do you take immediately after delivery?” and 2) “What are the essential steps immediately after birth in ensuring that the baby can breathe and is warm?”
For the first question, New Incentives counts how many of the following steps nurses say they take (without being prompted by the New Incentives staff member asking the question): a) Dry baby with cloth, b) Slightly rub baby, c) Clear airways, d) Use air mask if necessary, e) Regulate temperature (put on mother’s belly), f) Don’t know/refused to answer. For the second question, New Incentives captures a free form answer.
We have limited information about the differences in practices between the intervention and control groups in Jokhio et al. 2005, but we do have this information for Gill et al. 2011. (See Gill et al. 2011, Table 5, Pg. 8.) It does not appear that the way New Incentives evaluates answers to its first question can tell us whether nurses in the facilities with which it works follow the improved practices from Gill et al. 2011.
We aggregated the answers to the second question, and 17 of 54 answers explicitly mentioned using a bulb syringe or mucus extractor, which we would guess is equivalent to clearing the baby’s mouth and nose with a suction bulb in Gill et al. 2011 (another 11 mentioned ‘clear airways’ or ‘suck’ which might refer to the procedure used in Gill et al. 2011). We were not able to get additional relevant information from nurses’ answers to the second question.
New Incentives does not appear to ask questions that fully address the other major difference between the intervention and control groups in Gill: use of a resuscitation intervention.
The intervention offered by Jokhio et al. 2005 includes antenatal care in addition to intrapartum and postpartum care, and we don’t know what impacts each part of the intervention had.
Note that New Incentives does not systematically collect data on the type of care women who enroll in its program would have received had they not delivered in a facility, though it has done some limited surveys of traditional birth attendants in the areas it works in.
Non-randomized evaluations of the impact of facility delivery
We have not carefully reviewed these studies, and the studies we identified found mixed effects (including some studies finding higher neonatal mortality in facilities) but we have major questions about these studies’ ability to assess facilities’ causal impact.[1] In particular, women may be more likely to go to a facility for childbirth when they are experiencing complications, which could bias the results.
What is our best guess about New Incentives’ cost-effectiveness?
The most important questions in assessing New Incentives’ cost-effectiveness are (a) the impact its cash transfers have on rates of facility delivery and (b) the impact that increased facility delivery has on neonatal mortality.
New Incentives is conducting an RCT of its impact on (a) and preliminary results indicate that it had a significant impact on facility deliveries: 48% of women in the treatment group (i.e., all those who were offered the opportunity to enroll in the program even if they chose not to do so) delivered in a facility versus 27% in the control group. However, there are differences between the program studied by New Incentives’ RCT and its current program; the RCT only targeted HIV-positive women, so some portion of the impact may be attributable to educating women about the importance of PMTCT. The program studied in the RCT also provided larger cash transfers than New Incentives will provide in its ongoing program: the program originally gave 6,000 naira (approximately 19 US dollars) for enrollment, 20,000 naira for delivery, and 6,000 naira for an HIV test; the program currently gives 1,000 naira for enrollment and 10,000 naira for delivery.
As noted above, we have very limited information to rely on when forming an estimate of the impact of facility delivery on neonatal mortality, and we do not see the evidence from the RCTs described above as particularly relevant or informative.
However, in trying to arrive at our best guess of the impact of the program, we also considered the facts that:
- The interventions described in Gill et al. 2011 and Jokhio et al. 2005 are relatively low cost and of limited intensity, and they find significant decreases in neonatal mortality. This increases the plausibility that merely referring women to facilities for childbirth could have a similar, significant impact.
- Our intuition (supported by what appears to be conventional wisdom in the global health community) strongly implies that delivering in a facility (in general, without respect to the specific facilities New Incentives works with in Nigeria) is likely to lead to lower mortality than alternatives.
Philosophical value judgments
Based on the results from the RCTs, we would expect New Incentives’ program to primarily prevent deaths of very young children (largely those within the first days or week of life). In internal, staff discussions about New Incentives, we have asked ourselves how we value the lives of newborn children vs. the lives of those saved by malaria nets (the other life-saving intervention we currently recommend). We have not completed a thorough assessment of the ages at which people die from malaria, but our impression is that the median age of death is approximately 1.[2]
We believe there is no “right” answer to this question, but depending on one’s values, the answer could have a significant impact on the relative cost-effectiveness of New Incentives vs. the Against Malaria Foundation, and by extension our other top charities.
Key considerations include:
- One could simply sum the number of remaining years of life lost due to a death of a newborn vs a 1-year-old.
- One could focus solely on lives saved and treat all lives as equivalent.
- One might say that families and society have invested more in 1-year-olds and that 1-year-olds have more self-awareness and “personhood” than newborns, leading to valuing the 1-year-old more than the newborn.
Primarily for the last reason, the GiveWell staff who participated in these discussions tend to value 1-year-old lives over newborns, though our relative weights vary considerably.
Best guess cost-effectiveness estimate
Ultimately, we don’t have enough information to arrive at a reliable estimate of the impact of facility delivery on neonatal mortality. Our best guess is extremely rough, based primarily on intuitions formed based on limited data, and one that could easily shift significantly. We asked all staff who primarily work on GiveWell research to (a) guess the likely effect of New Incentives’ program on neonatal mortality and (b) enter the philosophical values discussed above. This yielded a median staff estimate that New Incentives was approximately as cost-effective as cash (in GiveDirectly’s program). Our cost-effectiveness model is here (.xlsx).
Is New Incentives transparent?
Yes – extremely. New Incentives has shared all of the information we have requested (and more) in a timely fashion. We feel that it is as good as any other organization we have ever engaged with on this criterion.
Options we’re considering for future support of New Incentives and/or its staff
We have discussed each of the following options with New Incentives and plan to let New Incentives’ preference drive our decision about which one to choose. In considering these options, we took into account (a) the likely direct impact funding would have and (b) the incentives that funding would create for others considering starting a new organization like New Incentives.
- Recommend that Good Ventures (a foundation with which we work closely that has provided past funding for our experimental work) provide an “exit grant” of approximately $1.2 million to New Incentives. New Incentives relied heavily on funding we recommended in its scale up, and abruptly stopping funding could cause it significant harm. Our impression is that funders often give grantees exit grants to offer them time to comfortably adjust their plans for fundraising and spending; this has been GiveWell’s experience with support from institutional funders. We would plan to benchmark our recommendation to the level of support New Incentives could have expected from us over the next two years (January 2017 – December 2018) as of the last time Good Ventures made a grant (March 2016). $1.2 million represents half what we would have projected New Incentives spending to be in 2017 and 2018 as of March 2016. (It grew faster than we expected since March 2016, so this is less than 50% of its projected operating expenses.)
- Recommend that Good Ventures agree to support some portion of New Incentives’ ongoing operations and a randomized controlled trial of New Incentives’ program’s impact on neonatal mortality. New Incentives’ program doesn’t seem cost-effective enough that we’d be willing to recommend that Good Ventures fully fund an RCT and New Incentives’ ongoing operations, but we’d consider recommending some, significant support (very roughly, we’d cap a recommendation at 50% of the total cost) if New Incentives could raise the rest of the funding elsewhere. This option would provide New Incentives with the opportunity to demonstrate that its program is more effective/cost-effective than we currently expect it to be as long as it is able to convince other funders to provide some support as well.
- Provide support to New Incentives/the New Incentives team to do something new. If New Incentives or its staff were interested in starting a new charity aiming to be a GiveWell top charity or significantly changing its program to focus on something more cost-effective, we would recommend that Good Ventures provide support.
We hope to decide soon about which option to pursue.
[Added December 19, 2016: GiveWell’s experimental work is now known as GiveWell Incubation Grants.]
Notes
[1] We identified two relevant meta-analyses. Chinkhumba et al. 2014, a meta-analysis of six prospective cohort studied of perinatal mortality in sub-Saharan Africa found 21% higher perinatal mortality in home deliveries compared to facility deliveries (OR 1.21 [1.02-1.46]) using a fixed-effects model, but this difference was not significant using a random effects model (OR 1.21 [0.79-1.84]).
We are also concerned that studies limited to the perinatal period may not capture longer-term neonatal effects. Tura et al. 2013, a meta-analysis of 19 studies (of various methodology) of the effect of facility delivery on neonatal mortality, found mixed results. Pooled results from low- and middle-income countries showed 29% reduction in risk of neonatal death associated with facility delivery. However, results of the studies were highly heterogeneous. Of the 8 studies in sub-Saharan Africa, 4 found effect near the pooled mean, and the other 4 did not find a statistically significant effect. (Of the four that did not find a significant effect, two studies found a nonsignificant effect close to the pooled mean of all studies, and two found no effect.)
A retrospective study based on the demographic and health surveys in Nigeria found that facility delivery is associated with increased neonatal mortality (adjusted odds ratio 1.28 [1.11-1.47], Fink et al. 2015, Figure 1, Pg. 5).
[2] Here is one paper we found. We have not vetted this paper. The simple average age of death in it is approximately 1.2 years (see Table 1).
The post New Incentives update appeared first on The GiveWell Blog.
