Note: This page summarizes the rationale behind a GiveWell grant to the Development Innovation Lab (DIL) at the University of Chicago. DIL staff reviewed this page prior to publication.
Summary
In January 2024, GiveWell recommended an $896,000 grant from our All Grants Fund to the Development Innovation Lab (DIL) at the University of Chicago to cover four months of additional research to complete its scoping activity, which GiveWell originally supported in January 2023, for a randomized controlled trial (RCT) powered to detect the effect of water chlorination on several outcomes, including mortality. The first of a two-phase grant, the January 2023 grant provided funding for DIL to begin "Phase 1" activities, including launching the RCT in Kenya, scoping the possibility of expanding the trial to include vouchers or in-line chlorination (ILC) in Nigeria or India, and, pending the outcome of scoping, launching a small pilot RCT in Nigeria or India. This would be followed by DIL’s proposal for "Phase 2," which would expand geographic reach and scope of the RCT with the goal of having adequate power to detect mortality effects. The bridge funding provided by this grant allows DIL to (a) continue scoping in both Nigeria and India, rather than focusing on only one, (b) assess the viability of using ILC in Nigeria as part of the Phase 2 study, and (c) cover its exit costs in the event GiveWell declines to fund the second phase of the grant.
We recommended this grant because:
- The additional research that this bridge grant supports should increase the quality of the Phase 2 RCT, making it more likely to generate reliable data that can inform both our and other funders’ decision-making in future.
- We think the Phase 2 RCT could be extremely valuable. Its results could substantially update our estimate of the effect of chlorination on mortality—a key uncertainty in our cost-effectiveness estimate—and influence a large amount of GiveWell’s future spending. Since the study will generate evidence on the delivery of water treatment programs at scale, the RCT may also lead other funders to direct more spending toward chlorination.
Our main reservations are:
- We are still uncertain whether the Phase 2 RCT can be powered to detect sufficiently small mortality effects, and therefore whether we will be inclined to recommend funding for it.
- We do not know whether this grant will be sufficient to cover DIL’s program costs for the full period between now and our ultimate decision on Phase 2, since we are uncertain about how long DIL’s additional research will take.
- We are concerned that this grant could reduce the incentive for DIL to adhere to our agreed-upon timelines in future.
Published: April 2024
Table of Contents
Background
In January 2023, GiveWell recommended a grant to support DIL in both launching and scoping a randomized controlled trial (RCT) on the impact of water chlorination. The study would be statistically powered to detect effects on under-two all-cause mortality, among other outcomes, and would help to refine program implementation on a larger scale.1 This was "Phase 1" of a two-stage grant; more information about the phases of the study is available here. DIL initially planned to have completed this work and submitted a proposal for the expanded RCT ("Phase 2") in fall 2023.2 However, in consultation with GiveWell, DIL has chosen to undertake additional work during Phase 1, in order to further refine its proposed Phase 2 trial. We now expect to make a funding decision on Phase 2 by the end of 2024. To cover the costs associated with this additional work, DIL approached GiveWell in November 2023 about the possibility of bridge funding, prior to our decision on Phase 2.
The grant
We are recommending an $896,000 grant to DIL to cover the cost of additional research activity in advance of its Phase 2 RCT proposal, and to cover exit costs in the event GiveWell chooses not to fund the Phase 2 study. Specifically, the grant will pay for:
- Additional scoping work and data collection, as well as program costs while this work is carried out ($596,000).3
DIL has continued with its Phase 1 intention of launching an RCT of chlorine vouchers in Kenya, and scoping the potential of adding a trial site in both Nigeria and India, rather than narrowing its focus to either Nigeria or India as originally planned. GiveWell has supported this decision, as we think that sustaining the full range of options will lead to a higher quality Phase 2 proposal. DIL has also chosen to investigate additional research questions in all three countries, as listed below. $596,000 of our grant will go toward these research costs, and toward ensuring program continuity between now and the time we make our Phase 2 funding decision in late 2024. Specific further research activities DIL has planned include:
- Nigeria: finding ways to sustain voucher redemption rates over time, such as by providing more vouchers at the time of collection, or by increasing the number of locations where vouchers can be redeemed;4
- Kenya: exploring ways to increase voucher take-up, which would increase the power of the study;5
- India: completing the in-line chlorination (ILC) pilot, which started late due to challenges obtaining permissions, and considering using administrative data during the RCT.6
- Scoping ILC in Nigeria ($150,000).7 DIL initially decided against scoping ILC in Nigeria, primarily due to the limited initial timeline of Phase 1 of the grant.8 However, we think ILC has the potential to increase the study’s power, because it is automated and doesn’t require behavior change, which generally leads to higher chlorination rates than chlorine vouchers.9 Because of this, we have asked DIL to explore ILC in Nigeria further. We do not expect DIL to complete this scoping work in time for our initial Phase 2 funding decision, but we do expect this work to allow us to decide, at a later date, whether and how to expand the trial to include ILC in Nigeria, having begun with a smaller initial design.
- Exit costs ($150,000). Should GiveWell decline to recommend funding for Phase 2, this portion of this grant would cover the cost of (a) winding down the existing research, and (b) DIL’s write-up of the results of Phase 1.10 DIL had not accounted for these exit costs in its original Phase 1 budget.11
The case for the grant
We are recommending this grant because:
- The activities covered by this grant will improve the quality of the Phase 2 RCT. The additional research that this grant supports will increase both the external and internal validity of the trial, making it more likely that the study generates reliable data that can inform both our and other funders’ decision-making in future. For example, the research activities in Kenya and Nigeria, discussed above, aim to increase the trial’s power, which in turn makes it more likely that the study will run in multiple countries.12 The results of a multi-site trial would be more generalizable than those of a single-site trial.
- We think the Phase 2 RCT could be extremely valuable. At present, our guess is that a Phase 2 RCT is likely to yield value of information that is significantly above our cost-effectiveness bar, although this remains highly speculative while we await a formal Phase 2 proposal and budget from DIL.13 We think this RCT is likely to be cost-effective because it could substantially update our estimate of the effect of chlorination on all-cause mortality—a key uncertainty in our cost-effectiveness analysis—and influence a large amount of GiveWell’s future spending. In addition, if the RCT shows that chlorination has a large effect on mortality or helps to refine the design of water chlorination programs at scale, we think the study could lead other funders to direct more support toward chlorination in future, which could increase the cost-effectiveness of their spending.
- We understand why these costs were not covered by the initial Phase 1 grant budget. Specifically, the additional costs covered by this grant can be attributed to (a) DIL’s decision to continue with Phase 1 studies in all three countries, rather than focus on two as originally planned, (b) our request that DIL explore ILC in Nigeria further, and (c) the fact DIL had not included in its original Phase 1 budget either the cost of its research while we make a Phase 2 funding decision, or the cost of winding down its activity if we decide against funding the second phase of the grant.
Risks and reservations
Our main reservations about this grant are:
- We do not yet know whether the RCT will be adequately powered to detect effect sizes that are meaningful to GiveWell. We are working with DIL to estimate the minimum detectable effect size they’d be powered to detect in the contexts they plan to work in. It is possible that the Phase 2 RCT will not be adequately powered, which would make us unlikely to fund the Phase 2 proposal. However, we expect that the additional Phase 1 activity supported by this grant will help us resolve this uncertainty, allowing us to make a more informed decision about Phase 2. It will also cover DIL’s exit costs in the event we decide not to fund the study.
- We do not know whether this grant will be sufficient to cover DIL’s time until we make a decision on Phase 2. The budget for this grant is based on DIL’s expectation that it will take four months to share additional research findings, and our expectation that it will take one month for us to make a funding decision.14 It is possible that this timeline is too optimistic, and that we should have made a larger grant.
- This grant could reduce the incentive for DIL to keep to its agreed timelines in future. Although we understand why the costs covered by this grant were not included in DIL’s original budget, DIL has fallen behind the timeline we agreed upon when we made our initial grant in January 2023. Providing this funding could lead DIL to believe that GiveWell will compensate for further overruns in future, which could encourage delays or under-budgeting for its Phase 2 work.
Plans for follow up
We plan to meet with DIL every two weeks to discuss its progress on launching the Kenya RCT and scoping in India and Nigeria, power calculations, and the timeline for its Phase 2 proposal. Once we receive that proposal, we plan to reach a funding decision on Phase 2 of the study within one month.
Internal forecasts
For this grant, we are recording the following forecasts:
| Confidence | Prediction | By time |
|---|---|---|
| 60% | We decide to fund Phase 2 of this grant. | September 2024 |
| 50% | Conditional on receiving funding, Phase 2 will include at least 2 sites. | September 2024 |
| 33% | GiveWell’s estimate of the effect chlorination has on U5 mortality will update positively as a result of the study. | End of 2028 |
| 33% | GiveWell’s estimate of the effect chlorination has on U5 mortality will update negatively as a result of the study. | End of 2028 |
| 33% | GiveWell’s estimate of the effect chlorination has on U5 mortality will not change as a result of the study. | End of 2028 |
Our process
This grant is based on our ongoing discussions about DIL’s progress on the Phase 1 grant that we recommended in January 2023. We are also in regular, ongoing email communication with DIL.
Sources
- 1
“While the research project described in this proposal has been designed primarily to provide information on the impact of water treatment on mortality, it will also produce:
- Shed light on how mortality impact varies with factors such as baseline child mortality and diarrhea rates, and with water treatment rates, thus allowing funders to better target water treatment interventions where they will have the most impact.
- Evidence on potential pathways of impact, obtained by (a) conducting verbal autopsies to determine the cause of deaths, (b) gathering data on age-specific mortality, and (c) collecting information on morbidity data. Such evidence may help resolve the discrepancy between experimental evidence from meta-analyses of water treatment RCTs on mortality and models focused on particular scientific pathways. For example, the proposed study would provide information on whether water treatment reduces mortality only following weaning of children from breastfeeding, consistent with diarrhea as the key pathway, or whether there are also effects on neonatal mortality. It will also provide information on the benefits and costs of making coupon redemption possible in shops as well as clinics. It may of course also demonstrate that certain approaches are not effective, which could also accelerate scale up by allowing resources to be focused on more promising approaches. For example, if we found that take up rates of water treatment through a coupon program in Nigeria were very low, this would point to focusing on other strategies, such as in-line chlorination (ILC), in Nigeria. We are and will continue to collaborate closely with partners, including Evidence Action and the governments of India and Kenya, both at operational and senior levels, so that this study will allow us to position programs for rapid scale-up, contingent on positive results.
- Detailed tracking data to enable long-run follow-ups, enabling future measurement of long-run health and mortality effects, with potential implications for both cost effectiveness and program design.”
Development Innovation Lab, Proposal for water treatment study, 2022, pp. 1-2.
- 2
“In fall 2023, DIL will submit a proposal for Phase 2.” GiveWell, Development Innovation Lab at the University of Chicago — RCT of Water Quality Interventions (January 2023).
- 3
Based on an initial cost estimate provided in Development Innovation Lab, email to GiveWell, January 17, 2024 (unpublished), and further discussions with DIL on the final grant amount.
- 4
Development Innovation Lab, email to GiveWell, January 11, 2024 (unpublished)
- 5
Development Innovation Lab, email to GiveWell, January 11, 2024 (unpublished)
- 6
Development Innovation Lab, email to GiveWell, January 11, 2024 (unpublished)
- 7
Based on a cost estimate provided in Development Innovation Lab, email to GiveWell, January 17, 2024 (unpublished).
- 8
Based on a conversation with DIL on December 12, 2023 (unpublished).
- 9
- “In-line chlorination is a technology for automatically disinfecting water at shared water collection points in low-income settings with unsafe water. We use the findings of water quality RCTs, adjusted for internal and external validity, to estimate reductions in all-cause mortality in children under five and people five and older. Pickering et al. 2019, the only published RCT of in-line chlorination with a diarrhea outcome, forms the basis of our adherence adjustment, a key input of our external validity adjustment. Clasen et al. 2015, the most recent Cochrane meta-analysis of water treatment trials, forms the basis for the morbidity reduction estimate and the plausibility limit derived from it. After adjustments, we estimate that in-line chlorination in Kenya reduces all-cause mortality by 11% in children under five and 2% in people five and over. We further estimate that development effects and medical costs averted account for 35% and 19% of the total benefit of the intervention, respectively.” GiveWell, Water quality interventions, 2022.
- “ILC devices can be used to cost effectively achieve high treatment rates in populations that use either piped water systems or communal storage tanks. They automatically dose water with chlorine without requiring users to change the way they collect and manage their drinking water, shifting the burden of water treatment from households to a service provider. ILC devices that do not require electricity for operation are commercially available.” Development Innovation Lab, Proposal for water treatment study, 2022, p. 3.
- "Although chlorine dispensers have higher take-up rates [than coupons], coupons can be further scaled where dispensers would not be appropriate. Development Innovation Lab, Proposal for water treatment study, 2022, p. 3.
- 10
Based on a cost estimate provided in Development Innovation Lab, email to GiveWell, January 17, 2024 (unpublished)
- 11
Development Innovation Lab, email to GiveWell, November 26, 2023 (unpublished)
- 12
This is because GiveWell would only consider expanding to a multi-site trial if it was sufficiently powered.
- 13
See our back-of-the-envelope calculation here: GiveWell, DIL Bridge Funding Water Vouchers Value of Information BOTEC [Jan 2024]. Note that a) our cost-effectiveness analyses are simplified models that are highly uncertain, and b) our cost-effectiveness threshold for directing funding to particular programs changes periodically. As of early 2024, our bar for directing funding is programs that are about 10 times as cost-effective as unconditional cash transfers. See GiveWell’s Cost-Effectiveness Analyses webpage for more information about how we use cost-effectiveness estimates in our grantmaking.
- 14
Development Innovation Lab, email to GiveWell, January 11, 2024 (unpublished).