Development Innovation Lab at the University of Chicago — Bridge Grant for RCT of Water Quality Interventions (January 2024)

Note: This page summarizes the rationale behind a GiveWell grant to the Development Innovation Lab (DIL) at the University of Chicago. DIL staff reviewed this page prior to publication.

Published: April 2024

Background

In January 2023, GiveWell recommended a grant to support DIL in both launching and scoping a randomized controlled trial (RCT) on the impact of water chlorination. The study would be statistically powered to detect effects on under-two all-cause mortality, among other outcomes, and would help to refine program implementation on a larger scale.1 This was "Phase 1" of a two-stage grant; more information about the phases of the study is available here. DIL initially planned to have completed this work and submitted a proposal for the expanded RCT ("Phase 2") in fall 2023.2 However, in consultation with GiveWell, DIL has chosen to undertake additional work during Phase 1, in order to further refine its proposed Phase 2 trial. We now expect to make a funding decision on Phase 2 by the end of 2024. To cover the costs associated with this additional work, DIL approached GiveWell in November 2023 about the possibility of bridge funding, prior to our decision on Phase 2.

The grant

We are recommending an $896,000 grant to DIL to cover the cost of additional research activity in advance of its Phase 2 RCT proposal, and to cover exit costs in the event GiveWell chooses not to fund the Phase 2 study. Specifically, the grant will pay for:

• Additional scoping work and data collection, as well as program costs while this work is carried out ($596,000).3 DIL has continued with its Phase 1 intention of launching an RCT of chlorine vouchers in Kenya, and scoping the potential of adding a trial site in both Nigeria and India, rather than narrowing its focus to either Nigeria or India as originally planned. GiveWell has supported this decision, as we think that sustaining the full range of options will lead to a higher quality Phase 2 proposal. DIL has also chosen to investigate additional research questions in all three countries, as listed below. $596,000 of our grant will go toward these research costs, and toward ensuring program continuity between now and the time we make our Phase 2 funding decision in late 2024. Specific further research activities DIL has planned include:
  • Nigeria: finding ways to sustain voucher redemption rates over time, such as by providing more vouchers at the time of collection, or by increasing the number of locations where vouchers can be redeemed;4
  • Kenya: exploring ways to increase voucher take-up, which would increase the power of the study;5
  • India: completing the in-line chlorination (ILC) pilot, which started late due to challenges obtaining permissions, and considering using administrative data during the RCT.6
• Scoping ILC in Nigeria ($150,000).7 DIL initially decided against scoping ILC in Nigeria, primarily due to the limited initial timeline of Phase 1 of the grant.8 However, we think ILC has the potential to increase the study’s power, because it is automated and doesn’t require behavior change, which generally leads to higher chlorination rates than chlorine vouchers.9 Because of this, we have asked DIL to explore ILC in Nigeria further. We do not expect DIL to complete this scoping work in time for our initial Phase 2 funding decision, but we do expect this work to allow us to decide, at a later date, whether and how to expand the trial to include ILC in Nigeria, having begun with a smaller initial design.
• Exit costs ($150,000). Should GiveWell decline to recommend funding for Phase 2, this portion of this grant would cover the cost of (a) winding down the existing research, and (b) DIL’s write-up of the results of Phase 1.10 DIL had not accounted for these exit costs in its original Phase 1 budget.11

The case for the grant

We are recommending this grant because:

• The activities covered by this grant will improve the quality of the Phase 2 RCT. The additional research that this grant supports will increase both the external and internal validity of the trial, making it more likely that the study generates reliable data that can inform both our and other funders’ decision-making in future. For example, the research activities in Kenya and Nigeria, discussed above, aim to increase the trial’s power, which in turn makes it more likely that the study will run in multiple countries.12 The results of a multi-site trial would be more generalizable than those of a single-site trial.
• We think the Phase 2 RCT could be extremely valuable. At present, our guess is that a Phase 2 RCT is likely to yield value of information that is significantly above our cost-effectiveness bar, although this remains highly speculative while we await a formal Phase 2 proposal and budget from DIL.13 We think this RCT is likely to be cost-effective because it could substantially update our estimate of the effect of chlorination on all-cause mortality—a key uncertainty in our cost-effectiveness analysis—and influence a large amount of GiveWell’s future spending. In addition, if the RCT shows that chlorination has a large effect on mortality or helps to refine the design of water chlorination programs at scale, we think the study could lead other funders to direct more support toward chlorination in future, which could increase the cost-effectiveness of their spending.
• We understand why these costs were not covered by the initial Phase 1 grant budget. Specifically, the additional costs covered by this grant can be attributed to (a) DIL’s decision to continue with Phase 1 studies in all three countries, rather than focus on two as originally planned, (b) our request that DIL explore ILC in Nigeria further, and (c) the fact DIL had not included in its original Phase 1 budget either the cost of its research while we make a Phase 2 funding decision, or the cost of winding down its activity if we decide against funding the second phase of the grant.

Risks and reservations

Our main reservations about this grant are:

• We do not yet know whether the RCT will be adequately powered to detect effect sizes that are meaningful to GiveWell. We are working with DIL to estimate the minimum detectable effect size they’d be powered to detect in the contexts they plan to work in. It is possible that the Phase 2 RCT will not be adequately powered, which would make us unlikely to fund the Phase 2 proposal. However, we expect that the additional Phase 1 activity supported by this grant will help us resolve this uncertainty, allowing us to make a more informed decision about Phase 2. It will also cover DIL’s exit costs in the event we decide not to fund the study.
• We do not know whether this grant will be sufficient to cover DIL’s time until we make a decision on Phase 2. The budget for this grant is based on DIL’s expectation that it will take four months to share additional research findings, and our expectation that it will take one month for us to make a funding decision.14 It is possible that this timeline is too optimistic, and that we should have made a larger grant.
• This grant could reduce the incentive for DIL to keep to its agreed timelines in future. Although we understand why the costs covered by this grant were not included in DIL’s original budget, DIL has fallen behind the timeline we agreed upon when we made our initial grant in January 2023. Providing this funding could lead DIL to believe that GiveWell will compensate for further overruns in future, which could encourage delays or under-budgeting for its Phase 2 work.

Plans for follow up

We plan to meet with DIL every two weeks to discuss its progress on launching the Kenya RCT and scoping in India and Nigeria, power calculations, and the timeline for its Phase 2 proposal. Once we receive that proposal, we plan to reach a funding decision on Phase 2 of the study within one month.

Internal forecasts

For this grant, we are recording the following forecasts:

Confidence	Prediction	By time
60%	We decide to fund Phase 2 of this grant.	September 2024
50%	Conditional on receiving funding, Phase 2 will include at least 2 sites.	September 2024
33%	GiveWell’s estimate of the effect chlorination has on U5 mortality will update positively as a result of the study.	End of 2028
33%	GiveWell’s estimate of the effect chlorination has on U5 mortality will update negatively as a result of the study.	End of 2028
33%	GiveWell’s estimate of the effect chlorination has on U5 mortality will not change as a result of the study.	End of 2028

Our process

This grant is based on our ongoing discussions about DIL’s progress on the Phase 1 grant that we recommended in January 2023. We are also in regular, ongoing email communication with DIL.

Sources

Document	Source
Development Innovation Lab, "About the Lab"	Source (archive)
Development Innovation Lab, email to GiveWell, January 11, 2024	Unpublished
Development Innovation Lab, email to GiveWell, January 17, 2024	Unpublished
Development Innovation Lab, email to GiveWell, November 26, 2023	Unpublished
Development Innovation Lab, Proposal for water treatment study, 2022	Source
GiveWell, conversation with DIL, December 12, 2023	Unpublished
GiveWell, Development Innovation Lab at the University of Chicago — RCT of Water Quality Interventions: Phase 1 (January 2023)	Source
GiveWell, DIL Bridge Funding Water Vouchers Value of Information BOTEC [Jan 2024]	Source
GiveWell, Water quality interventions, 2022	Source
GiveWell’s Cost-Effectiveness Analyses webpage	Source

• 1

  “While the research project described in this proposal has been designed primarily to provide information on the impact of water treatment on mortality, it will also produce:

  1. Shed light on how mortality impact varies with factors such as baseline child mortality and diarrhea rates, and with water treatment rates, thus allowing funders to better target water treatment interventions where they will have the most impact.
  2. Evidence on potential pathways of impact, obtained by (a) conducting verbal autopsies to determine the cause of deaths, (b) gathering data on age-specific mortality, and (c) collecting information on morbidity data. Such evidence may help resolve the discrepancy between experimental evidence from meta-analyses of water treatment RCTs on mortality and models focused on particular scientific pathways. For example, the proposed study would provide information on whether water treatment reduces mortality only following weaning of children from breastfeeding, consistent with diarrhea as the key pathway, or whether there are also effects on neonatal mortality. It will also provide information on the benefits and costs of making coupon redemption possible in shops as well as clinics. It may of course also demonstrate that certain approaches are not effective, which could also accelerate scale up by allowing resources to be focused on more promising approaches. For example, if we found that take up rates of water treatment through a coupon program in Nigeria were very low, this would point to focusing on other strategies, such as in-line chlorination (ILC), in Nigeria. We are and will continue to collaborate closely with partners, including Evidence Action and the governments of India and Kenya, both at operational and senior levels, so that this study will allow us to position programs for rapid scale-up, contingent on positive results.
  3. Detailed tracking data to enable long-run follow-ups, enabling future measurement of long-run health and mortality effects, with potential implications for both cost effectiveness and program design.”

  Development Innovation Lab, Proposal for water treatment study, 2022, pp. 1-2.

• 2

  “In fall 2023, DIL will submit a proposal for Phase 2.” GiveWell, Development Innovation Lab at the University of Chicago — RCT of Water Quality Interventions (January 2023).

• 3

  Based on an initial cost estimate provided in Development Innovation Lab, email to GiveWell, January 17, 2024 (unpublished), and further discussions with DIL on the final grant amount.

• 4

  Development Innovation Lab, email to GiveWell, January 11, 2024 (unpublished)

• 5

  Development Innovation Lab, email to GiveWell, January 11, 2024 (unpublished)

• 6

  Development Innovation Lab, email to GiveWell, January 11, 2024 (unpublished)

• 7

  Based on a cost estimate provided in Development Innovation Lab, email to GiveWell, January 17, 2024 (unpublished).

• 8

  Based on a conversation with DIL on December 12, 2023 (unpublished).

• 9
  • “In-line chlorination is a technology for automatically disinfecting water at shared water collection points in low-income settings with unsafe water. We use the findings of water quality RCTs, adjusted for internal and external validity, to estimate reductions in all-cause mortality in children under five and people five and older. Pickering et al. 2019, the only published RCT of in-line chlorination with a diarrhea outcome, forms the basis of our adherence adjustment, a key input of our external validity adjustment. Clasen et al. 2015, the most recent Cochrane meta-analysis of water treatment trials, forms the basis for the morbidity reduction estimate and the plausibility limit derived from it. After adjustments, we estimate that in-line chlorination in Kenya reduces all-cause mortality by 11% in children under five and 2% in people five and over. We further estimate that development effects and medical costs averted account for 35% and 19% of the total benefit of the intervention, respectively.” GiveWell, Water quality interventions, 2022.
  • “ILC devices can be used to cost effectively achieve high treatment rates in populations that use either piped water systems or communal storage tanks. They automatically dose water with chlorine without requiring users to change the way they collect and manage their drinking water, shifting the burden of water treatment from households to a service provider. ILC devices that do not require electricity for operation are commercially available.” Development Innovation Lab, Proposal for water treatment study, 2022, p. 3.
  • "Although chlorine dispensers have higher take-up rates [than coupons], coupons can be further scaled where dispensers would not be appropriate. Development Innovation Lab, Proposal for water treatment study, 2022, p. 3.

• 10

  Based on a cost estimate provided in Development Innovation Lab, email to GiveWell, January 17, 2024 (unpublished)

• 11

  Development Innovation Lab, email to GiveWell, November 26, 2023 (unpublished)

• 12

  This is because GiveWell would only consider expanding to a multi-site trial if it was sufficiently powered.

• 13

  See our back-of-the-envelope calculation here: GiveWell, DIL Bridge Funding Water Vouchers Value of Information BOTEC [Jan 2024]. Note that a) our cost-effectiveness analyses are simplified models that are highly uncertain, and b) our cost-effectiveness threshold for directing funding to particular programs changes periodically. As of early 2024, our bar for directing funding is programs that are about 10 times as cost-effective as unconditional cash transfers. See GiveWell’s Cost-Effectiveness Analyses webpage for more information about how we use cost-effectiveness estimates in our grantmaking.

• 14

  Development Innovation Lab, email to GiveWell, January 11, 2024 (unpublished).