Note: This page summarizes the rationale behind a GiveWell grant to New Incentives. New Incentives staff reviewed this page prior to publication.
In a nutshell
In October 2024, GiveWell recommended a $4.8mil grant to New Incentives to distribute Oral Rehydration Solution (ORS) and Zinc (collectively, “ORSZ”) to caregivers bringing their children in for routine childhood immunizations.
We think this grant is a good use of funding because:
- We think the program is cost-effective. Intuitively this is because New Incentives reaches a large number of young children through its existing immunization program, ORSZ distribution is relatively inexpensive, ORSZ is effective at treating dehydration from diarrhea, and we think providing ORSZ to caregivers of young children will avert deaths during a particularly vulnerable period.
- We expect monitoring and evaluation (M&E) implemented alongside the grant to generate evidence about how the program impacts uptake.
Our main reservations about this grant are:
- We have limited evidence on how free ORSZ provision impacts uptake (especially this program model).
- The evaluation of the program will include surveys to ask about ORSZ use before and after the program began. There are no plans currently to randomize rollout and get a more rigorous measure of the program's impact on ORSZ usage to treat diarrhea.
- There are potential risks of free ORSZ distribution such as crowding out of private providers and increasing consumption of contaminated water for breastfeeding infants.
- New Incentives has not previously procured and delivered commodities at scale, and program implementation may be more difficult than expected.
Published: February 2025
Table of Contents
1. Summary
1.1 Background
Diarrhea is a significant cause of illness and death in Nigeria among children aged under five. Oral rehydration solution (ORS) is a type of fluid replacement, often administered alongside short-term zinc supplementation, to treat dehydration due to diarrhea.
New Incentives runs a program in northern Nigeria which provides conditional cash transfers (CCTs) to caregivers bringing their children in for routine immunizations. On immunization days, New Incentives field officers in government health clinics check children’s eligibility, enroll children into the program, and distribute cash incentives. For more on New Incentives’ immunization program, see our separate report here.
1.2 What we think this grant will do
A $4,759,596 grant will support New Incentives to distribute free ORS and zinc (ORSZ) co-packs to caregivers bringing their children in for immunizations in 8 states in Nigeria for 3 years. We expect New Incentives to provide 3 ORSZ co-packs in total to each caregiver and deliver education about using ORSZ to treat diarrhea alongside its existing immunization program. We think that free provision could increase use of ORSZ and, in turn, decrease diarrhea-related mortality in young children. (more)
1.3 Why we made this grant
We think funding ORSZ distribution with New Incentives’ program is highly cost-effective (we estimate between 20 - 45 times the value of cash transfers,1 across states where New Incentives operates). In simple terms, this is because:
- ORSZ is inexpensive; and New Incentives can leverage its existing program staff and presence in health clinics to deliver ORSZ to caregivers of young children relatively inexpensively. We estimate it would cost roughly $1.20 per child enrolled for New Incentives to deliver 3 ORSZ co-packs (in total for each child) to caregivers across routine immunization visits. (more)
- Infant diarrheal-related mortality rates are high where New Incentives works. We estimate a 1.1% - 2.6% probability an untreated child dies of diarrheal diseases before the age of 1. This is based on data from the Global Burden of Disease (GBD) model, and includes adjustments for vaccination coverage and indirect deaths that could be averted by averting diarrheal-related deaths. (more)
- Counterfactual usage of ORS to treat dehydration resulting from diarrhea is relatively low. Based on data from the 2021 MICS (Multiple Indicator Cluster Survey), we estimate 31 - 48% of children under 1 would receive ORS absent the program in the areas where New Incentives works. (more) In addition, we expect to update these figures with data from baseline surveys that New Incentives plans to conduct (more), and we will only disburse funding for locations where usage rates are sufficiently low.
- Distributing free ORS (and zinc) during routine immunizations could lead to increased uptake for treatment of dehydration. We estimate an 12-16 ppt increase in the number of children treated. This is based on a single, high-quality RCT of a one-time household ORSZ distribution in Uganda. (more)
- Treating dehydration with ORSZ is highly effective at averting deaths. We estimate a 60% reduction in mortality. (more)
- We also think treating young children with ORSZ is likely to result in other benefits like long-term income gains and savings from averted medical costs. (more)
- We think CHAI’s program costs more per child ($3.20 per under-5 child reached; or ~$1.90 per under-5 child in the target population)2 because CHAI’s program requires implementing campaigns to go out to households to deliver ORSZ.3 By contrast, New Incentives can leverage their existing program and clinic staff who are already at health clinics, so the marginal costs for ORSZ distribution are mostly for procurement and commodity costs (~$1.20 per child enrolled). If we were to model New Incentives’ program at a cost per child enrolled of $1.90, this would bring cost-effectiveness in Bauchi to ~23x.
- Deaths due to diarrheal diseases appear most concentrated in the youngest children, and we expect to provide enough ORSZ to target all untreated diarrheal incidents on average. We expect New Incentives’ ORSZ program to reach children during and just after infancy. By contrast, CHAI’s program is targeted at all children under the age of 5.4 Across states where New Incentives’ program operates, we estimate that roughly 50 - 70% of diarrheal deaths that occur in children under the age of 5 occur when children are under the age of 1. This means we think it’s likely that New Incentives’ ORSZ program reaches young children when their risk of dying due to diarrheal diseases is highest.
- On the other hand, CHAI’s program may reach more vulnerable populations than those who show up at clinics for immunizations. We think it’s plausible that a household distribution model would reach caregivers who are less likely to use ORSZ absent the intervention, or who would respond more strongly to receiving free ORSZ. This could mean that CHAI’s program is more effective at increasing uptake of ORSZ in settings where baseline usage is lowest.
- We think this grant has moderate learning value. New Incentives routinely surveys households in areas where it works, and we expect them to leverage these surveys to collect baseline and follow-up data on ORSZ usage and other related outcomes. By measuring pre-post changes in ORSZ usage, we expect to learn more about how the program impacts uptake. (more)
- We have a strong qualitative impression of New Incentives, and we think they are well-placed to implement and monitor the program. (more)
- We are uncertain about several key parameters in our CEA (more). In particular:
- We have very limited evidence on how free ORSZ provision impacts usage, particularly in infants. Our estimates are based on a single, high-quality RCT. The RCT studied a one-time household ORSZ distribution in Uganda targeted at children under 5, and we make a number of simplifying assumptions to apply the results to New Incentives’ context. We expect to learn more about program impact through monitoring and evaluation conducted alongside the grant. (more)
- Our estimates of diarrheal-related mortality rely on the Institute for Health Metrics and Evaluation (IHME)'s Global Burden of Disease (GBD) model, and we are unsure how reliably the model attributes deaths to specific causes in Nigeria. Our understanding is that attributing deaths to a particular cause is highly uncertain in low-income countries, and these estimates are based on a number of modeling assumptions that we have not reviewed in detail. (more)
- Costs may be underestimated because this is a new program, and New Incentives has not previously procured and delivered commodities at scale. We make a rough 20% adjustment for this, but this may not be adequate. We plan on learning more about actual costs through the grant. (more)
- We have some uncertainty about how ORSZ usage reduces mortality because we rely on quasi-experimental evidence on the effect of ORSZ, and we do not plan on learning more about this through the grant. (more)
- The evaluation design may not be robust enough to strongly update us on program impact. Changes in uptake will largely be assessed through pre-post measures of caregiver self-reported ORSZ usage, which will likely not provide robust estimates of the program’s causal impact. The evaluation plan is still being finalized, and we plan to review and continue to provide feedback to strengthen the design. (more)
- Providing free ORSZ distribution carries potential risks, such as increasing consumption of poor quality water in very young children and crowding out private providers. Based on discussions we’ve had with experts and other implementers, we think these risks are relatively small. (more)
- New Incentives has not previously procured and delivered commodities at scale, and the program may be more difficult to implement than anticipated. This could mean the program is less impactful than we expect, or that the program cannot be implemented. We think the risks here are very small because New Incentives intends to pilot and iterate prior to scaling; and if the program can’t be implemented, any unspent funding could be rolled over into their highly cost-effective immunization program. (more)
- We think the grant is highly cost-effective. We estimate that in the 8 locations where New Incentives plans to implement ORSZ distribution, cost-effectiveness ranges from 20-45x. Intuitively, we think the program is highly cost-effective because New Incentives can leverage its existing immunization program to deliver ORSZ inexpensively, we think diarrheal-related mortality is high in the populations they reach, and we expect that providing free ORSZ will increase uptake for treatment of diarrhea and reduce subsequent mortality in young children. (more)
- We expect to learn more about program impact through monitoring and evaluation conducted alongside the program activities. New Incentives plans to leverage its existing household surveys to collect baseline and follow-up data on ORSZ usage, and we expect to learn more about program impact by comparing pre-post measures of ORSZ usage. (more)
- We have a strong qualitative impression of New Incentives, and we think they’re well-suited to effectively implement and monitor this program. (more)
- New Incentives estimates it would cost ~$0.33 USD to distribute each co-pack, which equals ~$1 per infant for 3 total.
- Because this is a new program, we adjust the total costs upwards by 20% to account for unanticipated costs related to procurement. This is a rough guess.
- This is a new program and we think it is plausible there may be unexpected costs. We make a 20% adjustment to account for a possible underestimate, but this may not be sufficient. We expect to learn more about program costs through routine monitoring conducted alongside the grant.
- Costs might also be overestimated if the Naira devalues due to inflation.13
- To estimate ORSZ coverage in children under 5 years of age, we use state-level data from the 2021 MICS survey14 on ORS usage for children under 5 years of age who experienced diarrhea in the last two weeks.15 In Kaduna, this is 50%.
- To estimate the share of children under 1 year of age who would be treated by ORS, we calculate the [share of children under 1] / [the share of children under 5] treated in Nigeria overall,16 then apply that ratio (~91%) to the under-5 ORS usage rate in each state.
- We do not adjust for possible changes in counterfactual coverage over time because we are not aware of systematic changes to ORSZ usage over time.
- We also do not adjust coverage for differences between the population reached by New Incentives and the overall average because we think an adjustment could go either way (e.g., caregivers reached by New Incentives may be poorer and less able to access ORS, or they may have higher care-seeking behaviors than average).
- Effect size from Wagner et al. 2019. We start with the effect size from Wagner et al. 2019 (a brief summary of this study is included in the footnotes),17
a ~47% reduction in the number of children under 5 not treated for diarrhea after households were given ORSZ for free. For more on this trial, see our separate ORS and Zinc report.
- As a simplification, we assume this effect applies to children under 1 (prior to any adjustments), and implicitly that the same treatment effect applies for each diarrheal episode that occurs in children under 1.18
- Adjustment for share of caregivers reached with ORSZ. We apply a 25% upwards adjustment because we expect New Incentives' field officers will successfully deliver ORSZ to a higher proportion of enrolled children (75%)19 than the proportion of targeted households CHWs reached in the RCT (61%). The trial measured effects in all households, regardless of whether or not they received ORSZ,20 and we expect treatment effects to be larger for caregivers who successfully receive ORSZ.
- Internal validity adjustment. We apply a -20% internal validity (IV) adjustment to account for weaknesses in study quality. For more, see our separate ORS and Zinc report.
- External validity adjustment. We apply a -50% external validity (EV) adjustment to account for differences between the RCT and New Incentives’ program. In the trial, ORSZ was delivered to households and changes in ORSZ usage were measured after 4 weeks.21
We think there’s a higher chance that caregivers would not have ORS on hand to treat diarrhea in New Incentives’ context because:
- A longer time between immunization visits and more frequent diarrhea episodes means ORSZ is less likely to reach children preemptively in New Incentives’ program. To account for this, we applied a rough -40% EV adjustment. This is a rough guess that tries to capture the possibility that caregivers will not have sufficient ORSZ on hand because younger children will experience more frequent diarrhea episodes, and additional ORSZ uptake caused by the program may not persist after one month (more details in the footnotes).22
- Relative to a household distribution model, distributing ORSZ at clinics may increase the risk that packets are lost during transportation or resold. To account for this, we applied a rough -10% EV adjustment. We think the overall risk of ORSZ being resold is low based on a conversation with implementers./strong>23
- We have very limited evidence on the impact of free ORSZ distribution on uptake in children (and no evidence for this model of distribution). Our estimates are based on a single RCT, and the RCT studied a different model of ORSZ distribution (one-time mass household delivery) targeted at children under 5 in Uganda. Outcomes were only measured one month after distribution. We make a number of simplifying assumptions to apply the results to New Incentives’ context over a one year period. While we apply a moderately large adjustment for external validity (-50%) because we think there’s a higher risk caregivers will not have ORSZ on hand to treat diarrhea in infants under New Incentives’ model, we could be over or (under) estimating program impact.
- We do not adjust for several factors because we are unsure which way the adjustment is most likely to go: Populations reached at facilities and households may respond differently to receiving free ORSZ.24 Caregiver behavior may differ around ORS usage in infants and children under 5 more generally.25
- We may be overestimating the number of children New Incentives can successfully reach with ORSZ, especially in the first few years of the program. We expect New Incentives to reach a reasonably large share of children enrolled with ORSZ, but program implementation could be more challenging than anticipated. We expect to learn more about how many children are successfully reached through monitoring conducted alongside the program.
- We do not make any adjustments for spillover effects due to changes in caregiver behavior. Receiving ORS for free may make caregivers more or less likely to get ORS outside of New Incentives’ program for other children (either in present day or later born children) or household members. We do not adjust for these spillover effects because it’s not clear which direction the effects would go on average.26
- Reduced mortality for infants (more)
- Long-term income increases from averting disease in a sensitive developmental window of childhood (more)
- Medical costs averted (more)
- Sharing of ORSZ within the household, which increase our cost-effectiveness estimate by 10% overall
- Other intervention-level factors (relating to free ORSZ provision more generally), which increase our cost-effectiveness estimate by 0.5% overall
- Grantee-level factors (relating to the organization’s implementation of the program), which reduce our estimate by 7% overall
- We start with the state-level estimates from the IHME’s Global Burden of Disease (GBD) project that there is a 1.1% to 3.9% probability, across states, that a child will die of diarrheal disease before reaching age 5. In Kaduna this is 1.1%.
- Adjustment for the under-1 population. To estimate the probability of a child dying before age 1 (which is roughly the age we expect children to benefit from ORSZ distribution through New Incentives), we take the share of under-5 diarrheal deaths that occur when children are under-1 from GBD (48%-68% across states; 68% in Kaduna), and then apply the resulting ratio to the under-5 probability of death.
- Adjustment for late enrollments. We also make a small -5% adjustment (details in footnotes)27 to account for late enrollments into New Incentives’ program (in 2023, ~14% of children enrolled after the 6 month mark).
- We adjust diarrheal-related mortality risk for vaccination coverage. Some of the children reached by New Incentives’ program will benefit from both immunizations and ORSZ. We make two adjustments to account for diarrheal-related deaths averted through immunizations:
- Adjustments for rotavirus vaccination. Rotavirus vaccination had not been rolled out in Nigeria at the time of GBD2021 estimates. We adjust diarrheal burden for higher rotavirus coverage (90%)28 in the population enrolled by New Incentives because we expect rotavirus vaccination to avert some diarrheal deaths.29
- Adjustments for all other vaccines. We also include a more speculative -5% adjustment for diarrheal deaths indirectly averted by other vaccines. We think receiving childhood vaccinations indirectly reduces mortality from causes other than the diseases vaccines intended to target (for more, see our separate report here). In particular, we think it’s plausible that some childhood vaccines may indirectly reduce diarrheal-related mortality. We make a small, rough -5% adjustment to diarrheal mortality because we think there is likely to be minimal overlap in the diarrheal deaths averted by childhood immunizations and those we expect New Incentives’ ORSZ program to target (more details in the footnotes).30
- We adjust for all-cause mortality effects. We assume there are 0.33 indirect deaths averted for every direct death averted. This is based on our general ORSZ work that estimates 0.5 indirect deaths averted per direct death due to diarrheal diseases, and a subjective -35% adjustment to account for indirect deaths that may be targeted by both immunizations and ORSZ.
- We estimate that there are 0.5 indirect deaths averted for each direct diarrheal death averted. See the linked cell note for more detail.
- Some of these indirect deaths may be averted through immunizations that we account for in our model of New Incentives’ CCT program. As a rough guess, we reduce the adjustment by 35% to arrive at 0.33 indirect deaths.
- We think both immunizations and ORSZ reduce deaths from causes other than the diseases they are intended to target.31 For children benefiting from both interventions, if these secondary deaths are averted through similar mechanisms (e.g. a strengthened immune system), we might be double-counting the benefits in our model.
- We have not deeply investigated what other causes of death vaccinations and ORSZ are most likely to target, and the mechanisms through which either interventions averts deaths due to non-specific causes. As a rough guess, we reduce the indirect effect adjustment for ORSZ by 35%.32
- We adjust for higher mortality in the untreated population. Our understanding is that GBD estimates represent mortality among all children (treated and not treated with ORSZ). We expect untreated children to face a higher mortality risk. To account for this, we make an adjustment to estimate the mortality risk for untreated children only. The full explanation for a similar adjustment is included in our separate New Incentives intervention report.
Shortcoming and uncertainties:
- Focusing on the under-1 population is a simplification. We don’t expect ORSZ provided through the program, particularly at 9 months, to solely address diarrhea when children are under 1; but we expect this to reasonably approximate the addressable burden of the period when New Incentives would reach children with ORSZ (through infancy and a little after).
- Our calculations rely on estimates from GBD, and we have not conducted a detailed analysis of the modeling assumptions and process GBD uses to attribute deaths to specific causes in Nigeria. This could mean we are over (or under) estimating the diarrheal risk and associated mortality. For more on our general reservations around GBD estimates, see our separate report here. We expect to learn more about diarrheal incidence in the populations reached by New Incentives through program monitoring, and may update our estimate for diarrheal burden.
- The GBD estimates for relative diarrheal risk across age groups don't align with data from the 2021 MICS survey (see footnotes),33 or our qualitative understanding that the risk of diarrhea is likely lower in infants under 6 months because they are more likely to be exclusively or predominantly breastfeeding. We have not adjusted for this because we think it’s plausible that the most severe cases of diarrhea (i.e. those driving mortality) would be concentrated in the youngest children, and because we think the GBD estimates likely provide a reasonable approximation for mortality risk in the age range when we expect New Incentives’ ORSZ program to exert its effect (during and just after infancy).
- Diarrhea episodes and mortality may be concentrated in a subset of the population, so distributing 3 ORSZ co-packs on average may overestimate what share of diarrheal mortality is addressable through the program. We expect 3 ORSZ co-packs to target the average number of diarrheal episodes a child experiences under the age of 1.34 Diarrhea mortality may be concentrated in a subset of children who experience more episodes of diarrhea than the overall average (i.e. more than 2-3 episodes in the first year of life). If this is true, then we are likely overestimating diarrheal mortality addressable by New Incentives’ program because 3 co-packs would only cover a fraction of diarrheal episodes causing mortality.
- We don’t make any adjustments to account for differences in the population reached by New Incentives compared to the overall average because we think it’s plausible that an adjustment could go either way.35 We expect to learn more about diarrheal incidence in the populations reached by New Incentives through their surveys (although this won’t tell us about mortality). (more)
- We may not be adequately accounting for the impact of vaccinations on diarrheal mortality. We explicitly adjust for rotavirus coverage and make other adjustments to decrease diarrheal-related mortality in a vaccinated population. We could be underestimating the impact of vaccinations on mortality (and thus, overestimating the remaining deaths that ORSZ can target) if children counterfactually vaccinated through the program largely overlap with those who are counterfactually treated through ORSZ; or if the benefits of vaccinations are more concentrated in the youngest children (e.g., we’ve seen some evidence that rotavirus burden may be most concentrated in the youngest children, and that the vaccine may be most effective around the time of vaccination36 ).
What impact does ORS uptake have on mortality?
We estimate that being treated with ORS would reduce diarrhea mortality by 60%. This is based on quasi-experimental evidence on the effect of ORS on mortality, to which we apply adjustments for internal and external validity adjustments.For more see, our separate report on ORSZ.
- We use seasonal malaria chemoprevention (SMC) as a benchmark. Our estimate is that the value of income benefits from SMC are equivalent to 31% of the value from deaths averted for people under the age of 15 (for more, see our separate New Incentives intervention report).
- We apply a subjective adjustment that the impact of long-term increases from ORSZ usage would be 22% as large as those from SMC (more details in the footnotes).37
- Sharing within households We expect that some of the ORSZ distributed through the program will go to other household members.39 We do not currently have evidence that quantifies sharing within households, so as a rough guess, we assume that other household members will benefit 10% as much as infants.40 We expect to learn more about this through the grant and may update our calculations.
- Additional program benefits and downsides from ORSZ distribution. We make a ~0.5% adjustment for other excluded benefits from ORSZ based on our existing work.
- Grantee-level adjustments. We apply the same -7% adjustment as our existing New Incentives model, which includes a -2% adjustment for quality of monitoring and evaluation and a -5% adjustment for non-funding bottlenecks.41
- Program impact. New Incentives plans to conduct baseline and follow-up surveys roughly every 6 months to measure pre-post changes in ORSZ usage based on caregiver self-reports. The survey instrument and evaluation protocols are still being developed, and we expect to continue to review and provide feedback until they are finalized.
- New Incentives regularly conducts household surveys to collect data on vaccination coverage and other outcomes.43 They plan to add questions about ORSZ usage, diarrheal incidence, and other related outcomes to their routine surveys to measure outcomes related to the ORSZ program.44 Surveys will be conducted at baseline prior to ORSZ implementation in a given area, and then roughly every 6 months to measure changes in key outcomes.45
- Key outcomes that we expect to learn more about through the surveys include: diarrheal incidence, ORSZ usage in children under 5, changes in ORSZ uptake, and caregiver knowledge and awareness about ORSZ.46
- In general, we have fairly high confidence in the quality of New Incentives’ routine surveys (for more, see our separate report here). New Incentives has engaged evaluator (IDInsight) to collaborate on design of the survey instrument and evaluation protocols, and we expect them to provide independent analysis of the results. To improve comparability across programs, they are also drawing from the survey instruments and pre-analysis plans which will be used in the RCT we funded with CHAI’s ORSZ program in Bauchi.
- Impact on private providers. We expect the rapid assessments to include questions about where caregivers procure ORSZ, which would provide some evidence about the impact on private providers. Separate from the rapid assessments, New Incentives is exploring whether it is possible to collect data on ORSZ availability and price at private providers.47
- Program costs and feasibility. We also expect to learn more about program costs and feasibility through New Incentives’ general program monitoring. These would include outcomes such as how many children New Incentives’ field officers are reaching with ORSZ on immunization days.
- We are uncertain about several key parameters in our cost-effectiveness analysis. In particular, we have very limited evidence on the impact of free ORSZ distribution on uptake (and no evidence for a facility-based model like the one New Incentives plans to implement). We expect to learn more about some parameters (including program impact) through M&E conducted alongside the grant. (more)
- We may not learn enough about program impact through M&E. We expect to learn more about the program through baseline and follow-up surveys, but these surveys likely will not provide rigorous estimates of causal impact. (more)
- We think there are some potential risks to ORSZ distribution, including the possibility that children increase their consumption of poor quality water and that free distribution may interfere with the private market. Our impression from talking to experts is that these risks are relatively small. (more)
- This is a new program, and implementation could be more difficult than anticipated. We think the risks here are very small because New Incentives plans to stagger rollout and iterate on program design as the program scales. If the program could not be implemented effectively, unspent funding could be rolled into their highly cost-effective immunization program. (more)
- We have very limited evidence on the impact of free ORSZ distribution on increased treatment (in particular, for this model of distribution). A single RCT underlies our estimates, and the RCT studied a different model of ORSZ distribution (one-time mass household delivery) targeted at children under-5 in Uganda. We make a number of simplifying assumptions to extrapolate from this trial to New Incentives’ context. While we apply a moderately large -50% EV adjustment to the results, we may still be overestimating program impact.
- Our calculations rely heavily on estimates of diarrheal diseases from GBD, and we’re highly uncertain about how accurately GBD's model attributes deaths to particular diseases. Our best guess is that attributing deaths to a specific cause is highly uncertain in low-income countries, particularly for different age groups, and we have not investigated all the modeling assumptions used in the GBD estimates.
- Costs may be underestimated because this is a new program. We make a 20% upward adjustment for possible underestimates, but this may not be sufficient.
- We are somewhat uncertain how effective ORSZ is at reducing mortality because our estimates rely on quasi-experimental evidence. Our impression is that it is widely accepted that ORSZ is effective at reducing mortality due to diarrhea.
- The ORSZ program will be implemented in the same population we expect to benefit from New Incentives’ immunization program, and we may be double-counting benefits from both programs (or not accounting for positive synergistic effects). We make a number of adjustments to the disease burden to avoid double-counting deaths averted by both programs, but these may not be sufficient. We have not conducted in-depth research into possible interaction effects between the programs, including the possibility that the programs have positive synergistic effects.48
- The current evaluation plan is a pre-post analysis. Any observed changes in ORSZ usage or other outcomes may be caused by factors outside of New Incentives’ program, and we may not be able to attribute improvements to New Incentives’ program.49 Because this is a relatively small amount of funding and it’s highly plausible that free ORSZ distribution could have a moderate to large impact on uptake, we think a strong pre-post design is likely sufficient to update us on program impact.
- ORSZ usage will be measured through caregiver self-reports, and we think there's a strong incentive for caregivers to misreport usage if they think it will cause the program to continue, especially since ORSZ is often used for others in the household.50
We’ve explored some options to get objective measurements, but do not think there are strong alternatives.
- ORSZ usage appears to generally be monitored through self-reports,51 and researchers we spoke to did not have strong alternatives to self-reported measures. We expect NI’s survey to ask caregivers to show unused packets as one way to gauge bias in different locations,52 but we expect this to be a fairly noisy measure.
- ORSZ questions will be incorporated into New Incentives’ existing routine surveys, which limits the scope for learning. In general, we have a strong impression of the quality of New Incentives’ monitoring; however, only a subset of children surveyed for immunization will have recently experienced a diarrheal episode53 and be included in the ORSZ survey, so the sample size for ORSZ related measures will likely be smaller. This means that the surveys may not be adequately powered to detect changes in ORSZ usage and other outcomes.
- Providing free ORSZ could interfere with the private market. We think there’s some risk that providing large quantities of ORSZ for free could discourage caregivers from purchasing ORSZ on their own, or interfere with the private market. However, the overall quantity of ORSZ being provided is small and experts we spoke with do not think distributing the anticipated volumes is likely to negatively impact the market.54
- Providing ORSZ to caregivers of young infants may increase the uptake of poor quality water. We expect ORSZ distributed through New Incentives to target infants, and infants are more likely to be exclusively breastfeeding. By providing ORSZ (which needs to be mixed with water) to caregivers for treatment, there may be a risk that very young children increase their uptake of poor quality water. We expect these risks to be mitigated through education New Incentives provides emphasizing the importance of using clean water for ORSZ. We think the overall risk is low because our understanding is that ORSZ is still recommended for infants who are exclusively breastfeeding.55 For more, see our CHAI ORSZ grant page here.
- If the program is stopped, ORSZ usage or immunization rates may decrease below what they would have been if the program was never implemented.
- If we decline to renew funding after a scale-up, is there a risk that ORSZ usage drops below baseline because caregivers have gotten accustomed to receiving free ORSZ at a clinic? We think the risks are small because the overall quantity of ORSZ being provided is relatively low and only expected to cover the diarrheal episodes in infants. For caregivers who would have otherwise acquired ORSZ on their own, we expect them to continue procuring ORSZ for other household members while the program is ongoing. Therefore, we don’t expect the program to cause caregivers to get out of the habit of procuring ORSZ for their households, value ORSZ less, or otherwise decrease their motivation to use ORSZ.
- Could removing ORSZ cause caregivers to stop going to immunization visits? Because ORSZ will be distributed alongside cash incentives at immunization visits, withdrawing ORSZ may result in some caregivers to stop going to immunization visits. We think the risk is small because our understanding is that the value of ORSZ is low relative to cash incentives, so it seems reasonable that we would not expect negative effects on immunization uptake, assuming the cash incentives continued.
- New Incentives has not previously procured and distributed commodities at scale, and ORSZ distribution could be more difficult to implement than anticipated. This could mean the program can’t be implemented, or will have less impact than expected. Overall, we think these concerns are broadly mitigated because:
- To the extent that supply is a constraint, we expect these to be short-term issues rather than intractable ones. When we spoke with other implementers with more experience procuring commodities in Nigeria, they didn’t express concern about the volumes New Incentives planned on procuring.
- New Incentives plans to pilot in select LGAs or one state, iterate on the program design, and stagger rollout. This would allow New Incentives to identify and address some problems as the program scales.
- Any unspent funding could be rolled over into New Incentives’ immunization program. If New Incentives decides to stop implementation at any point, we expect they could use any unspent funding for their immunization program, which we think is highly cost-effective. More on New Incentives’ immunization program here.
- There’s a risk that operational capacity spent on ORSZ distribution could decrease the quality of the existing CCT program for immunization. Overall, we think this is a very low risk based on our strong, qualitative impression of New Incentives.
- We estimated cost-effectiveness of the ORS and zinc program using our existing work on ORS and zinc, and New Incentives’ program. Our model was reviewed by a Senior Research Associate and a Program Officer.
- We had multiple discussions (calls and over email) with New Incentives to learn more about their plans, provide feedback, ask questions, and discuss M&E.
- We also had multiple conversations with external experts and other groups working on ORS distribution in Nigeria to understand the landscape and barriers to ORS and zinc usage.
- 1
The estimate used here and elsewhere in the page of the value of cash transfers is out of date as of 2024. We are continuing to use this outdated estimate for now to preserve our ability to compare across programs, while we reevaluate the benchmark we want to use to measure and communicate cost-effectiveness.
- 2
$2.9 mil / 1.5 mil kids under 5 * 61% of households reached = $3.20 cost per child reached
- 3
"This is an ambitious program requiring training and supervision of many people in order to distribute ORS and zinc to a very large number of households." GiveWell, Clinton Health Access Initiative — Oral Rehydration Solution and Zinc Distribution in Bauchi, Nigeria, September 2023.
- 4
"This grant will support CHAI to design and implement a program in Bauchi, Nigeria, to distribute free ORS and zinc to all households with children under age 5." GiveWell, Clinton Health Access Initiative — Oral Rehydration Solution and Zinc Distribution in Bauchi, Nigeria, September 2023.
- 5
See a table of all grants we have made or recommended to New Incentives here.
- 6
At the time of writing, New Incentives’ CCT program operates in 9 different states: Bauchi, Gombe, Jigawa, Kaduna - cohort 17, Kano, Katsina, Kebbi, Sokoto, Zamfara + 10 High Risk LGAs in those states.
We are recommending funding in all these locations except Bauchi because of a preexisting ongoing ORSZ distribution program. - 7
"We recommend providing a co-pack or equivalent of 2 ORS and 10 tablets of Zinc every alternate visit:
BCG visit, Penta 2 visit, and MCV 1 visit." New Incentives, Costing Review of ORS, Zinc, and Paracetamol, 2024 (unpublished). - 8
See the timeline here: New Incentives, How We Encourage All Routine Childhood Vaccines, 2023.
- 9
New Incentives, conversation with GiveWell, January 9, 2024 (unpublished).
- 10
New Incentives, Costing Review of ORS, Zinc, and Paracetamol, 2024 (unpublished).
- 11
New Incentives, email to GiveWell, September 23, 2024 (unpublished).
- 12
Note that a) our cost-effectiveness analyses are simplified models that are highly uncertain, and b) our cost-effectiveness threshold for directing funding to particular programs changes periodically. As of late 2024, our bar for directing funding is about 10 times as cost-effective as unconditional cash transfers. See GiveWell’s Cost-Effectiveness Analyses webpage for more information about how we use cost-effectiveness estimates in our grantmaking.
- 13
We’ve observed rapid inflation in Nigeria since 2020. See our program monitoring indicators here.
- 14
"The Multiple Indicator Cluster Survey (MICS) was carried out in 2021 by the National Bureau of Statistics (NBS) as part of the Global MICS Programme. . . . The Global MICS Programme was developed by UNICEF in the 1990s as an international multi-purpose household survey programme to support countries in collecting internationally comparable data on a wide range of indicators on the situation of children and women." National Bureau of Statistics (NBS) and United Nations Children's Fund (UNICEF), Multiple Indicator Cluster Survey 2021, Statistical Snapshot Report, 2022, p. ii.
- 15
See here. The MICS survey data we use is based on whether children aged 0-59 months were treated for diarrhea, conditional on them having had diarrhea in the last two weeks. We make a slight leap from this to estimate the proportion of children aged 0-59 who are treated for diarrhea, i.e., it’s possible that any given child could be treated for some but not all diarrheal incidents.
- 16
We were unable to find more granular age breakdowns by state.
- 17
Wagner et al 2019 studied the effect of free distribution of ORS on usage in under 5 children in Uganda. The RCT evaluated the impact of community health workers providing ORS to households (either at households or via vouchers; the results were similar for those two arms). ORS usage was 74-77% in the treatment group compared with 56% in the control group, a 18-21 ppt increase.
- 18
A different way of framing this is that, for each diarrheal incident a child experiences before their 1st birthday, we expect that receiving ORSZ preemptively through New Incentives’ program would result in a 47% reduction in not being treated. Assuming that a child experiences 2-3 diarrheal incidents on average, and they receive 3 ORSZ packets from New Incentives, then their risk of not being treated decreases by 47% over the under-1 period.
- 19
- In 2023, New Incentives estimated that CCTs were offered on 88.9% of immunization days. We expect ORSZ distribution to happen alongside CCT distribution, so we would expect a similar number of days where ORSZ is distributed.
- To estimate the number of children who will receive ORSZ, we apply a rough -15% adjustment to account for potential ORSZ stockouts and other issues procuring the necessary amount of ORSZ during the implementation period.
- 20
“We assessed impact using an intention-to-treat (ITT) framework.” Wagner et al. 2019 (p. 1.)
- 21
“We used household surveys to measure ORS (primary outcome) and ORS + zinc use 4 weeks after the interventions began (between April and May 2017).” Wagner et al. 2019 (p. 1.)
- 22
There’s a sizable gap between the second and third expected distribution (at 10 weeks and 9 months). This schedule and potential delays in immunizations may mean that caregivers are less likely to have ORSZ on hand at the time of diarrheal episodes than in the RCT.
- For example, it’s possible that a child would experience 3 diarrheal episodes prior to their measles vaccination at 9 months. In this scenario, assuming New Incentives is the only source of ORSZ, even if the child was vaccinated on time and New Incentives delivered 3 co-packs at the scheduled visits, the caregivers would not have ORSZ on hand to treat the 3rd episode.
- Caregivers may also be less likely to use ORSZ to treat children for diarrheal diseases when more than a month has elapsed since they received ORSZ.
- The RCT only measured follow-up at one month, and we think it’s very likely that more than a month will elapse between diarrheal episodes and when ORSZ is distributed through New Incentives’ program. This is particularly true between the 10 week and 9 months visits, but also after the 9 month visit (i.e. we don’t expect a plurality of cases to occur at month 10).
- We expect program impact to wane the longer it’s been since distribution, because it’s more likely ORSZ will be used by others (e.g. adults or older children in the same household) or simply misplaced.
- 23
We spoke with implementers who work in Nigeria, and they believed the risk of reselling was low because a relatively small quantity of ORSZ was being distributed and the value was not high enough to incentivize significant reselling at the point of distribution.
- 24
The RCT studied a household delivery model whereas New Incentives will reach caregivers coming to health clinics for immunizations. Caregivers at clinics may have higher levels of baseline care-seeking behavior and respond more strongly to receiving free ORSZ; or caregivers reached in the community-based model might represent a population with lower awareness or knowledge of ORSZ, so free distribution could lead to larger increases.
- 25
The RCT had a target population of under-5s and caregiver behavior for under-1s may be different, due to breastfeeding and other factors. For example, caregivers who are predominantly or exclusively breastfeeding may be less likely to give their children ORSZ than caregivers of slightly older children.
- 26
- Free ORSZ distribution could have positive spillover effects by increasing awareness about ORSZ and getting caregivers into the habit of using ORSZ to treat diarrhea, thus making them more likely to procure ORSZ for treating others in the household.
- On the flipside, caregivers could get used to receiving free ORSZ, so they become less likely to procure more on their own. There’s also a small risk that free distribution of ORSZ disrupts supply and/or has negative impacts on private providers, thus making it more difficult for caregivers to get ORSZ on their own.
- 27
We apply a small -5% adjustment to account for the fact that 14% of children are reached after ~50% of the under-1 diarrheal episodes have occurred; but that ORSZ could be used to avert some diarrhea later in the 12-23 months period (or beyond). I.e., ~14% * ~50%, with a small downward adjustment.
- New Incentives enrolls ~14% of infants after 6 months (as of 2023). ORSZ delivered through the program would not address diarrheal episodes that happen during this period. IHME data shows that in Nigeria a little over half (.9 / 1.9) of diarrheal incidents in the under-1 range occur in the <6 month period.
- However, we expect New Incentives will still deliver the same amount of ORSZ co-packs to infants enrolled on a delayed schedule (i.e. caregivers would still receive 3 ORS co-packs total).
- IHME reports ~1.1-1.7 diarrheal incidents per year in children 12-23 months of age (compared to ~2-2.3 for children <1 year). So, while the diarrheal burden appears lower in the slightly older age group, we expect any ORSZ delivered by New Incentives roughly between the 0-23 months age range to still target a meaningful share of deaths.
- 28
- This is based on first dose (RV1) coverage for simplicity because we expect RV1 to confer roughly 75% of the total protection from the rotavirus vaccine.
- In 2023, New Incentives reported that Penta1 (when RV1 is scheduled) coverage was 96%. To arrive at 90% RV1 coverage, we make a small downward adjustment because rotavirus vaccines have a higher likelihood of being out of stock (13% vs 7% for all vaccines), so we expect coverage to be slightly lower. See New Incentives monitoring results [January 2024] (public)
- 29
For our calculations, see this range in our cost-effectiveness analysis.
- 30
At a high-level, our approach was to estimate how many indirect deaths averted by immunizations could be diarrheal deaths that overlap with the period when ORSZ would exert its effects. We assume that only indirect deaths averted roughly during the post neonatal period would overlap.
- We assume that all non-rotavirus vaccines (BCG, Pentavalent, PCV, measles) would avert roughly 0.75 deaths (from any cause) for each direct death averted. This means that roughly 43% of the deaths averted by vaccination when children are under the age of 5 are indirect deaths.
- 0.75 / (0.75 + 1) = 43%
- We expect that rotavirus vaccination would not indirectly reduce diarrheal deaths, so we deduct the share of indirect deaths attributable to rotavirus (~17-23%).
- We assume that deaths averted roughly during the post neonatal period would overlap with ORSZ. We estimate that roughly 23-33% of under 5 deaths would’ve occurred during the post neonatal period based on GBD estimates of all-cause mortality (unpublished).
- This is a rough approximation of the period of overlap when ORSZ and immunizations would exert their effects. In reality, we would expect some overlap in the 12-23 months period and some (much smaller) overlap in the neonatal period.
- This means that only about ~8-12% of indirect deaths averted through immunizations are likely to overlap with deaths targeted by ORSZ. Furthermore, we expect that only a portion of these would be diarrhea deaths.
- 43% * (100% - 23%) * 23% = 8%. 43% * (100% - 17%) * 33% = 12%.
- We assume that all non-rotavirus vaccines (BCG, Pentavalent, PCV, measles) would avert roughly 0.75 deaths (from any cause) for each direct death averted. This means that roughly 43% of the deaths averted by vaccination when children are under the age of 5 are indirect deaths.
- 31
In other words, we expect immunizations to reduce mortality from causes other than the specific pathogens the vaccine is intended to target; and ORSZ to reduce mortality from causes other than diarrhea.
- 32
This is a rough guess that is intended to capture:
- Lower overall disease burden in a population with high vaccination rates. Because populations reached by New Incentives are expected to have higher vaccination rates as a result of the program, we think there is likely a somewhat lower disease burden, and fewer indirect deaths “available” for ORSZ uptake to avert.
- Children benefiting from both programs. Because the children enrolled in the ORSZ program are also the same children enrolled in New Incentives’ CCT program, we expect some share of children to benefit from both programs (i.e. be counterfactually vaccinated and counterfactually treated with ORSZ). We think it’s plausible that some of the secondary deaths which would have been averted through additional ORSZ uptake are instead averted through immunizations (because children will likely be immunized before being treated for ORSZ).
- 33
- GBD suggests that there’s a higher incidence of diarrhea in children <1 year of age (1.9 cases per 100,000 children in Nigeria) compared to children between 12-23 months of age (1.1 per 100k in Nigeria).
- By contrast, MICS2021 (see Table C2.1 Reported disease episodes, p. 205) finds that 15.4% of children aged 0-11 months experienced a diarrhea episode two weeks prior to the survey compared to 18.9% in children aged 12-23 months. We don’t think these numbers are directly comparable to GBD’s, but this is suggestive evidence that diarrheal risk might be higher after infancy.
- 34
We calculate that the average number of diarrheal episodes a child under 1 in Nigeria will experience is approximately 1.9. The diarrheal disease incidence rate per 100,000 children under 1 is 191,339. 191,339/100,000 = 1.9.
- 35
For example, caregivers reached by New Incentives may be poorer and access lower quality drinking water than the overall average, or they may be a population with higher care-seeking behaviors.
- 36
Our adjustment for rotavirus vaccine coverage is fairly small because we estimate a relatively small share of diarrheal-related diseases are targeted by the rotavirus vaccine (around 46% of diarrheal deaths for children under the age of 1), and the vaccine is not particularly effective at averting these deaths (we estimate around 39%). We’ve seen some evidence which suggests a higher share of severe rotavirus gastroenteritis occur when children are younger; and that vaccine efficacy is highest around the time it is first administered, then rapidly wanes. From Clark et al 2019:
- “Efficacy is lower and wanes more rapidly in high-mortality settings, but in these settings, more than 60% of rotavirus gastroenteritis hospital admissions occur before age 1 year, and more than 90% occur before age 2 years.”
- “In settings with high mortality (24 observations), there were five vaccines with observation points for infant schedules. The pooled iVE was 66% (48–81) after 2 weeks of follow-up and 44% (27–59) after 12 months.”
- 37
In order to consistently compare the cost-effectiveness of ORSZ with other programs we model, we need to estimate the effect of ORSZ on long run income / consumption (“development effects”). However, we have not found any studies directly estimating the effect of ORSZinc on long-run income / consumption. To estimate the development effects of ORSZ, we compare its effect on factors we believe are correlated with long run income / consumption to the effect of seasonal malaria chemoprevention (SMC) on those same factors.
See this worksheet for our calculations. - 38
We include a rough adjustment intended to account for medical costs averted based on GiveWell’s estimates of the average benefits per DALY averted. We assume that morbidity will decrease by the same share as mortality, and we make a subjective downward adjustment due to our guess that a curative intervention (where the patient still becomes ill and may require health care access) will avert fewer medical costs than a preventive one (where the patient may never become ill).
- 39
We expect New Incentives to provide enough ORSZ to treat all infant diarrheal episodes (on average), but we estimate that free distribution of ORSZ will “only” result in a 23% reduction in children not being treated through the program. This is, in part, because we think some of the ORSZ distributed to infants will go to others in the households.
- 40
We expect most of the benefits would come from slightly older children (under the age of 5) being treated for diarrhea with ORSZ distributed by New Incentives’ program.
- 41
- We expect monitoring for the ORSZ program to be conducted through New Incentives’ existing monitoring, so we apply the same quality adjustment here.
- The “Non-funding bottlenecks” adjustment accounts for the possibility that New Incentives holds funding for a prolonged period without reaching beneficiaries. To the extent funding is not used for ORSZ, we expect it to be rolled into New Incentives’ immunization program, so we apply the same adjustment here.
- 42
Funging refers to GiveWell funding causing other actors to spend less on New Incentives’ program (or a similar ORSZ distribution program) than they otherwise would. We estimate:
- A 10% chance that the Nigerian government would fund this (or a similar) program if GiveWell did not. This is based on the assumption that the government would be at least as likely to fund an ORSZ program as New Incentives’ core program. For more on our reasoning, see our report here.
- A15% chance that other philanthropic actors would fund a comparable ORS distribution program. We think there’s a higher risk of other philanthropic funding replacing New Incentives’ program because we’re aware of several other organizations who have implemented ORSZ distribution programs in Nigeria.
- 43
These are household surveys conducted by trained enumerators which collect data on vaccination coverage and other related outcomes. For more, see here.
- 44
"New Incentives is also piloting the distribution of oral rehydration solution and zinc (ORSZ) for treating childhood diarrhea during routine immunization visits…. New Incentives plans to leverage this activity to collect additional data on diarrhea and usage of ORS and zinc. They intend to use this data to inform whether and how to integrate ORSZ distribution into its program as well as monitor changes in ORS coverage before and after its implementation." New Incentives, ORSZ Survey Protocol, October 2024 (unpublished).
- 45
"It will yield a series of cross-sectional estimates of self-reported coverage of diarrhea incidence and ORS and zinc usage. One of these estimates will be done prior to the start of distribution of ORSZ, while the others will be done at six month intervals once ORSZ distributions have begun." New Incentives, ORSZ Survey Protocol, October 2024 (unpublished).
- 46"The specific primary research questions that this activity will seek to answer are:
- What percentage of children surveyed have had diarrhea over 4 weeks prior to the
survey? - What is the average number of diarrhea episodes per child-year?
- What is the self-reported coverage of ORS at baseline among children with diarrhea within
the past 4 weeks? - Is there a change in ORS coverage at follow-up compared to baseline?
- What is the self-reported coverage of ORSZ at baseline among children with diarrhea
within the past 4 weeks? - Is there a change in ORSZ coverage at follow-up compared to baseline?
The secondary research questions that this activity will seek to answer are:
- What is the self-reported coverage of zinc at baseline among children with diarrhea within
the past 4 weeks? - Is there a change in zinc coverage at follow-up compared to baseline?
- What is self-reported knowledge of ORS among caregivers? Does this change between
baseline and endline?"
New Incentives, ORSZ Survey Protocol, October 2024 (unpublished).
- What percentage of children surveyed have had diarrhea over 4 weeks prior to the
- 47
Confirmed in New Incentives' review of a draft of this page, January 2025.
- 48
For example,
- Providing ORSZ at immunization visits may increase uptake of vaccinations.
- Treating children with ORSZ may also improve their general health and the immune system’s ability to mount an effective response to vaccination.
- 49
"Survey comparisons will be pre / post comparisons. These can be indicative of impact but will not
provide rigorous estimates of the programʼs attributable impact." IDinsight, Write-up on ORSZ survey options, 2024 (unpublished). - 50
In our conversation with the CHAI implementation team, they noted that usage of ORS for adults is quite pervasive, especially to rehydrate in the farming season. New Incentives has also flagged this as a concern.
- 51
From our separate report on ORS and zinc:
- “We apply an 80% adjustment to the estimate of the reduction of non-ORS usage to account for internal validity factors, including a lack of masking of data analysts and self-reporting by participants”
- “This study has several other limitations. First, our measure of ORS use relies on caretaker reports. Although caretaker reports are used to monitor ORS use globally, self-reported data rely on accurate memory and could be subject to social desirability bias.” Wagner et al. 2019, p. 15.
- 52
If a much lower share of caregivers in one location can successfully show a used packet relative to others, then that could indicate higher levels of self-report bias.
- 53
We currently expect the survey to include children who experienced a diarrheal episode in the last 4 weeks, but this may change in the final design.
- 54
- We spoke with experts and implementers with experience working in Nigeria, and they all believed that the risk of crowding out private providers was small because the quantities of ORSZ being distributed were low relative to demand.
- Experts also told us that they think ORSZ distribution could positively impact the local market by increasing demand, and incentivizing local manufacturers to increase supply.
- 55
The WHO’s recommendation for exclusive breastfeeding includes exceptions for medications including ORS: “Exclusive breastfeeding” is defined as giving no other food or drink – not even water – except breast milk. It does, however, allow the infant to receive oral rehydration salts (ORS), drops and syrups (vitamins, minerals and medicines).” WHO, “Breastfeeding”, 2015.
- 56
Confirmed by New Incentives' review of a draft of this page, January 2025.
- 57
We expect that we’ll estimate this the same way we do New Incentives’ conditional cash transfer program costs, by taking total actual costs over children enrolled for ORSZ.
Simple CEA (illustrative example)
Key uncertainties highlighted in yellow
| What we’re estimating | Kaduna, Nigeria - Cohort 17 (rounded) | Confidence Intervals (25-75th percentile) | Implied cost-effectiveness |
|---|---|---|---|
| Children enrolled for ORSZ (more) | |||
| Number of children enrolled over 3 years | ~255,000 | ||
| Cost of ORSZ (more) | |||
| Total cost of procuring and distributing 3 ORSZ co-packs per child enrolled | $1.20 | $1.00-$1.80 | 13x-23x |
| Total cost of 3 ORSZ co-packs for all children enrolled | $306,000 | ||
| Under-one mortality benefits | |||
| Share of children under 1 treated for diarrhea in counterfactual (more) | 46% | ||
| Given free ORS, reduction in number of children not treated for diarrhea in intervention, IV/EV adjusted (more) | 23% | 10%-40% | 8x-33x |
| Number of new children treated for diarrhea in intervention | 32,000 | ||
| Probability that a person will die of a diarrheal disease before reaching age 1, including mortalities indirectly attributable to diarrheal disease (more) | 1.1% | 0.5%-1.5% | 9x-28x |
| Effective reduction in mortality from ORSZ treatment, after IV/EV adjustments (more) | 60% | 37%-75% | 12x-25x |
| Number of under-1 diarrheal deaths averted due to program | ~200 | ||
| Initial cost-effectiveness estimate | |||
| Cost per under-one death averted | ~$1,500 | ||
| Moral value of averting the death of a person under age one | 100 | 50-150 | 10x-30x |
| Initial cost-effectiveness in terms of multiples of GiveDirectly's unconditional cash transfer program | 20x | ||
| Percentage of program impact coming from different benefits (as a % of modeled benefits) | |||
| Mortalities averted before age one | 83% | ||
| Development effects (more) | 6% | ||
| Cost savings (more) | 11% | ||
| Additional adjustments | |||
| Sharing within households (more) | 10% | ||
| Grantee-level adjustments (more) | -7% | ||
| Other excluded effects (more) | 0.5% | ||
| Adjustment for leverage | 0% | ||
| Adjustment for funging (more) | -22% | ||
| Final cost-effectiveness estimate | |||
| Final cost-effectiveness in terms of multiples of GiveDirectly's unconditional cash transfer program | ~20x | ||
For our full calculations see here, and for a simple version of our calculations here.
How does ORSZ distribution through New Incentives compare to other interventions GiveWell thinks are promising?
In September 2023, GiveWell funded CHAI to employ community health workers to deliver ORSZ to households with children under 5 in Bauchi, Nigeria. In Bauchi, we estimate that New Incentives’ program model would be ~36x compared to ~21x for CHAI’s program at scale. We think New Incentives’ ORSZ program is more cost-effective (in the populations it is targeting) because New Incentives’ program is significantly less expensive per child and the program is probably better targeted at the youngest children who are most vulnerable to diarrheal diseases.
Factors outside the model
1.4 Our main reservations
2. Planned activities and budget
2.1 The organization
New Incentives runs a conditional cash transfer (CCT) program in northern Nigeria that seeks to increase uptake of routine childhood vaccinations by providing cash transfers, raising public awareness of the benefits of vaccination, and partnering with the government to reduce the frequency of vaccine stockouts. New Incentives is a GiveWell Top Charity. In part, this means that we've directed significant funding to the organization and have seen it operate effectively,5 and we think further grants to New Incentives have a high likelihood of substantial impact (more about our criteria for Top Charities). For more information on New Incentives, including how it works and our qualitative assessment of the organization, see this page.
On immunization days, New Incentives has field officers in government health clinics who check children’s eligibility, enroll children into the program, and distribute cash incentives. For more on New Incentives’ CCT program, see our separate report here.
2.2 What we think this grant will do
We expect a $4,759,596 grant would support New Incentives to procure and deliver ORS and zinc (ORSZ) co-packs to caregivers bringing their children in for routine immunization visits for 3 years (roughly starting in mid to late 2025) across 8 locations where it works as of October 2024.6 New Incentives plans to deliver 3 ORSZ co-packs alongside CCTs at alternating immunization visits (BCG, Penta2, and Measles1).7 Because these immunization visits happen shortly after birth, at 10 weeks, and at nine months, we expect these ORSZ packs would primarily be used when the child is under 1 year old.8 We think providing free ORSZ to caregivers could lead to increased treatment for dehydration resulting from diarrheal events, preventing a significant number of young child deaths. For more on the free provision of ORSZ see our separate report.
In addition to providing ORSZ, New Incentives plans to conduct education and awareness raising activities at immunization visits to inform caregivers about the benefits and proper usage of ORSZ (including emphasizing the importance of using clean water and how to properly prepare ORSZ).9
New Incentives also plans to leverage its existing household surveys to collect data on ORSZ usage, diarrheal incidence rates, and other related outcomes that will allow us to learn more about whether free ORSZ distribution alongside CCTs is an effective model to increase uptake. (more)
2.3 Budget for grant activities
We expect a $4,759,596 budget to cover the costs of procurement and distribution in the following locations:
| State | Total cost |
|---|---|
| Gombe | $439,992 |
| Jigawa | $701,892 |
| Kaduna (Cohort 17) | $254,916 |
| Kano | $1,669,176 |
| Katsina | $810,144 |
| Kebbi | $426,024 |
| Sokoto | $247,932 |
| Zamfara | $209,520 |
| Total: | $4,759,596 |
This is based on estimates New Incentives shared that it would cost roughly $1 USD per infant to procure and deliver 3 ORSZ co-packs.10 New Incentives expects this budget to cover costs related to procuring, transporting, and storing ORSZ; and costs for training staff and any educational materials provided alongside ORSZ.11
Because this is a new program, we think it’s plausible that costs will be higher than anticipated. We have included a rough adjustment for additional costs in our cost-effectiveness analysis (more below), and would consider granting additional funding if needed. We also think it’s plausible that New Incentives may decide to increase the number of co-packs distributed, or alter the program design as a result of learning more through piloting and monitoring activities. We would evaluate any changes in program design and may grant additional funding if needed.
While we have approved funding for 3 years of implementation in these locations, we plan to only disburse funding for a given location if baseline surveys conducted by New Incentives indicate that ORSZ usage rates are low enough that the program will likely be cost-effective. (more)
3. The case for the grant
We are recommending this grant because:
3.1 Cost-effectiveness
We estimate that this program is roughly 20-45 times as cost-effective as unconditional cash transfers.12
Intuitively, we expect free provision of ORSZ to be highly cost-effective because ORSZ is inexpensive, and New Incentives is well placed to reach a large number of young children (before and just after infancy) with ORSZ through its existing immunization program. We think those young children are at a high risk of dying from diarrheal-related diseases; that lack of access to ORSZ is a major barrier to using ORSZ; and receiving free ORSZ through the program could reduce the number of children not treated for diarrhea, and avert deaths from diarrheal-related diseases.
For a summary of our cost-effectiveness analysis see above. We provide a more detailed explanation of our cost-effectiveness analysis below.
Note: We expect these exact numbers to change to incorporate information about ORSZ usage from New Incentives’ monitoring surveys. (more)
How much does the program cost?
We estimate it would cost roughly $1.20 for New Incentives to deliver 3 ORSZ co-packs (in total) to a caregiver at immunization visits.
| What we’re estimating | Value (rounded) |
|---|---|
| Estimated commodity and distribution costs for one ORSZ co-pack (USD) | $0.33 |
| Adjustment for additional costs | 20% |
| Co-packs delivered to each child enrolled | 3 |
| Cost per child enrolled in the program | $1.20 |
Our estimate of program cost per child enrolled is largely based on New Incentives’ estimates, with a modest (20%) upward adjustment for unanticipated costs.
Shortcomings and uncertainties:
How many people do we expect New Incentives to reach with ORSZ?
We estimate that 31% - 48% of children under 1 who experience diarrhea are treated with ORS at baseline where New Incentives works, varying by state. We estimate that distributing 3 ORSZ co-packs to caregivers bringing their children in for immunizations will reduce the number of children not treated with ORSZ after experiencing diarrhea by 23%. We use this to estimate the number of additional children treated as a result of the program (shown below). In Cohort 17 (a cluster of local government areas (LGAs) located in Kaduna state), this is roughly 32,000 additional children treated with ORSZ.
A summary of our calculations for Cohort 17 in Kaduna is below:
| What we’re estimating | Kaduna - Cohort 17 (value rounded) |
|---|---|
| Total children enrolled in the program (more) | 255,000 |
| Share of children under 1 treated for diarrhea with any ORS absent intervention (more) | 46% |
| Given free ORSZ, reduction in number of children not treated for diarrhea in intervention (more) | -23% |
| Children treated for diarrhea because of the intervention | ~32,000 |
How many children are reached through New Incentives’ program?
We expect New Incentives to enroll the same children for ORSZ and cash incentives. In Kaduna (Cohort 17), we estimate this would be roughly 255,000 infants enrolled over 3 years based on data provided by New Incentives.
How many children would receive ORSZ without New Incentives’ program?
We estimate between 31% - 48% of children under 1 would be treated for diarrhea with ORS based on Multiple Indicator Cluster Survey (MICS) 2021 survey data, depending on location. In Kaduna, we estimate baseline usage in infants is 46%.
| What we’re estimating | Kaduna - Cohort 17 (value rounded) |
|---|---|
| Share of children aged 0-59 months with diarrhea in the last two weeks who received any ORS (MICS2021 - Kaduna state) |
50% |
| Share of children aged 0-59 months with diarrhea in the last two weeks who received any ORS (MICS2021 - Nigeria overall average) |
46% |
| Share of children aged 0-11 months with diarrhea in the last two weeks who received any ORS (MICS2021 - Nigeria overall average) |
42% |
| Share of children under 1 treated for diarrhea absent intervention | 46% |
Our approach:
Shortcomings and uncertainties:
How much would the program increase ORSZ uptake?
We estimate that New Incentives’ program would lead to a 23% reduction in the number of children not treated for diarrhea.
A summary of our calculations is below:
| What we’re estimating | Overall (rounded) |
|---|---|
| Reduction in number of children not treated for diarrhea in Wagner et al. 2019 (“free and convenient group”) | 47% |
| Share of enrolled children given ORSZ by New Incentives | 75% |
| Share of children reached with preemptive ORSZ in Wagner et al. 2019 | 61% |
| Share of under-1 diarrhea cases covered by co-packs distributed, on average | 100% |
| Subtotal: Reduction in number of children not treated for diarrhea, adjusted for intervention coverage | 58% |
| Adjustment for internal validity (IV) | -20% |
| Adjustment for external validity (EV) | -50% |
| Given free ORSZ, reduction in number of children not treated for diarrhea in intervention, IV/EV adjusted | 23% |
Our approach:
Shortcomings and uncertainties:
What impact would increased ORSZ uptake have?
Our cost-effectiveness analysis models three main benefits resulting from New Incentives’ program:
A summary of the contributions of each type of benefit to our total modeled estimate of the value of the program is below, using Cohort 17 in Kaduna as an example:
| What we’re estimating | Kaduna - Cohort 17 |
|---|---|
| Reduced mortality for infants (more) | 83% |
| Long-term income increases (more) | 6% |
| Medical costs savings (more) | 11% |
In addition to these three benefits, we also include a number of supplemental adjustments to account for additional benefits and offsetting impacts. We divide these into:
Rather than explicitly modeling these, we have applied percentage adjustments based on our best guesses. (more)
After factoring in all these impacts (as well as our adjustments to account for the impact on other actors’ spending, discussed below), we estimate that it costs approximately $900 to $2,000 (varying by state) to avert a death through the program. In the sections below we discuss how we quantify each of these benefits.
Reduced mortality for infants
What is the probability of death among children who do not receive ORS?
We estimate a 1.1% - 2.6% probability that an untreated child dies of diarrheal diseases before the age of 1 across states. In Kaduna, this is ~1.1%.
| What we’re estimating | Kaduna - Cohort 17 |
|---|---|
| Probability that a person (treated and untreated) will die of diarrheal disease before reaching age 5, from the Global Burden of Disease (GBD) model |
1.1% |
| % of under 5 deaths due to diarrheal diseases that occur in children under 1, from the GBD model |
68% |
| Adjustment for under-1 diarrheal deaths missed due to late enrollments in New Incentives’ program (after 6 months of age) |
-5% |
| Subtotal: Probability that children (treated and untreated) reached by New Incentives will die of diarrheal disease before reaching age 1 (more) |
0.7% |
| Estimated proportion of children enrolled by New Incentives who receive at least 1 dose of the rotavirus vaccine |
90% |
| Share of diarrhea caused by rotavirus | 31% |
| Rotavirus vaccine efficacy rate with universal coverage | 39% |
| Share of under-1 diarrheal deaths indirectly averted by all other vaccines | 5% |
| Subtotal: Probability that a person will die of diarrheal disease before reaching age 1, adjusted for vaccination coverage (more) |
0.6% |
| Indirect deaths averted for each direct diarrhea death averted (more) | 0.33 |
| Subtotal: Probability that a person will die of a diarrheal disease before reaching age 1, including mortalities indirectly attributable to diarrheal disease |
0.8% |
| Share of children under 1 treated for diarrhea absent intervention (above) | 46% |
| Effective reduction in mortality from ORSZ coverage (below) | 60% |
| Probability that an untreated person will die of a diarrheal disease before reaching age 1, including mortalities indirectly attributable to diarrheal disease |
1.1% |
Our approach:
Long-term income increases
Based on other child health programs we’ve investigated (such as malaria and deworming), we think that treating young children for diarrhea during a sensitive development window may lead to long-term income increases in adulthood. We estimate that benefits from long-term income increases are 7% as large as the benefits from deaths averted.
Medical costs savings
By treating diarrheal diseases, ORSZ provision may also reduce treatment costs if children are more likely to recover and not need further treatment. We estimate that cost savings are between roughly 11-12% of total modeled benefits.
For more on our approach to estimating medical costs averted, see our separate report on ORS and Zinc (an overview of our approach is provided in the footnotes).38
Supplemental adjustments
How does the program affect other actors’ spending?
Part of our cost-effectiveness analysis involves asking what impact funding a program has on other actors’ spending. New Incentives’ program may lead other organizations or governments to spend more (we refer to this as "leveraging" funding, or “crowding in”) or less (we refer to this as "funging," from “fungibility,” or “crowding out”) on vaccines than they otherwise would.
We include a “leverage and funging” adjustment in our cost-effectiveness analysis to account for this. We do not expect the ORSZ program to leverage any funds, but we apply a funging adjustment of -22% to -23%, varying by state (more details in the footnotes).42
For a more detailed explanation of our approach, see our separate report on New Incentives’ conditional cash transfer program.
3.2 Moderate learning value
We expect monitoring and evaluation (M&E) implemented alongside the grant to address some of our key uncertainties around program uptake, program costs and feasibility, and the impact on private providers. Learning more about the program’s impact on uptake could improve our cost-effectiveness estimates, and cause us to direct more funding to this and similar programs in the future. We also expect New Incentives to use the learnings to inform program design and potentially increase impact.
Note: The monitoring and evaluation protocol is still being developed, and changes to the plans may impact what we expect to learn through this grant. We expect to publish the final monitoring protocol and pre-analysis plan after they are completed.
3.3 Strong qualitative impression of New Incentives
New Incentives is one of GiveWell’s Top Charities, and we think the organization has a strong track record as an implementer as well as strong processes for gathering and sharing detailed monitoring data for its programs. For more on our qualitative impressions of New Incentives, see our separate report here.
Throughout our investigation on the ORSZ program, New Incentives was responsive to our feedback, provided high-quality responses to our questions, and proactively highlighted potential risks and downsides of the program (including the possibility that the program could not be implemented at all).
4. Risks and reservations
Our main reservations about this grant are:
4.1 Uncertainty about key CEA parameters
We provide a more in-depth discussion of shortcomings and uncertainties in each section above. Here, we provide a brief overview of the key uncertainties:
4.2 We may not learn enough about program impact through M&E
We have very limited evidence on the impact of free ORSZ provision on usage, and plan to learn more through monitoring and evaluation conducted alongside this grant. The current evaluation strategy is largely based leveraging New Incentives’ ongoing surveys to estimate pre-post changes of caregiver self-reported usages. This evaluation approach likely will not provide robust estimates of causal impact.
Note: The monitoring and evaluation protocol is still being developed, and changes to the plans may address some of these limitations.
4.3 Potential risks of harm from ORSZ distribution
4.4 Program implementation could be more difficult than anticipated
There may be unexpected challenges to ORSZ distribution that mean New Incentives is unable to implement the program, or the program is much less cost-effective than expected. As of August 2024, New Incentives estimates a 20% chance that they will not be able to distribute ORSZ upon learning more.56 Overall, we think these concerns are broadly mitigated because we expect New Incentives to pilot and iterate on the program as it scales; and in the event that the program cannot be implemented cost-effectively, any unspent funding can be transferred to their immunizations program (which we think is highly cost-effective).
5. Plans for follow up
We plan to continue reviewing and providing feedback on the monitoring and evaluation plans as they are being developed. We expect to publish an update with the final protocol once details are finalized. After surveys are rolled out, we plan to review baseline results on an ongoing basis and update our cost-effectiveness analysis to incorporate estimates of baseline ORS usage.
After implementation starts, we plan to follow the progress in our regular check-ins with New Incentives. We have monthly calls with New Incentives and receive monthly written program updates. New Incentives also shares information on program performance across a variety of indicators. We have not yet aligned on indicators specific to the ORSZ program but expect them to include measures such as program costs and whether field officers can distribute the expected number of sachets.
6. Internal forecasts
For this grant, we are recording the following forecasts:
| Confidence | Prediction | By time | |
|---|---|---|---|
| Renewing funding | |||
| 62.5% | We will renew ORSZ funding in at least 8 states (i.e. the current states, excluding Bauchi) | End of December 2027 | |
| 72.5% | We will renew ORSZ funding in at least 6 states (i.e. the current states, excluding Bauchi) | End of December 2027 | |
| Baseline surveys | |||
| 55% | New Incentives will roll out their first baseline survey by March 2025. | End of March 2025 | |
| 82.5% | New Incentives will roll out their first baseline survey by June 2025. | End of June 2025 | |
| 47.5% | New Incentives will complete baseline ORSZ surveys in all cohorts (we may not receive the results then) by December 2025. | End of December 2025 | |
| 75% | New Incentives will complete baseline ORSZ surveys in all cohorts (we may not receive the results then) by March 2026. | End of March 2026 | |
| 80% | In all locations where baseline surveys are successfully completed, the ORSZ program will still look >10x after incorporating coverage information from baseline surveys into the CEA. | Following survey completion | |
| Program implementation | |||
| 10% | After learning more, New Incentives will decide not to go forward with program implementation prior to implementing any baseline surveys. | End of March 2025 | |
| 15% | After learning more, New Incentives will decide not to go forward with program implementation after baseline surveys have been rolled out in at least some locations. | End of 2025 | |
| 60% | New Incentives will propose increasing the number of co-packs to be delivered per child (i.e. up from 3) | End of December 2027 | |
| Program costs | |||
| 75% | Actual program costs per child enrolled57 (assuming no major changes to the program design; i.e. 3 co-packs delivered per child) will be at least 10% higher than New Incentives has budgeted for in this grant proposal. | End of 2026 | |
| 50% | Actual program costs per child enrolled (assuming no major changes to the program design; i.e. 3 co-packs delivered per child) will be at least 20% higher than New Incentives has budgeted for in this grant proposal. | End of 2026 | |
| 20% | Actual program costs per child enrolled (assuming no major changes to the program design; i.e. 3 co-packs delivered per child) will be at least 10% lower than New Incentives has budgeted for in this grant proposal. | End of 2026 | |
Note: We may add additional forecasts after monitoring and evaluations plans are finalized.