Development Innovation Lab at the University of Chicago — RCT of Water Quality Interventions: Phase 1 (January 2023)

Note: This page summarizes GiveWell’s rationale for recommending a grant to the Development Innovation Lab (DIL) at the University of Chicago. DIL’s rationale is explained in its proposal here. DIL staff reviewed this page prior to publication.

Published: October 2023

Background

The Development Innovation Lab (DIL) at the University of Chicago is a development economics research group focused on creating an evidence base for international development projects.1 GiveWell has previously funded research work in which DIL-affiliated researchers were involved, including expansion of the Kenya Study of Water Treatment and Child Survival.

In our previous work on water quality, we estimated that water chlorination likely reduces all-cause mortality in children under five years old by about 12%.2 However, most randomized controlled trials (RCTs) of water quality interventions that we are aware of do not directly assess or report on the impact of the intervention on mortality, so we are uncertain about this estimate.3

In addition, there are open questions about how to best deliver water treatment in different contexts. Vouchers and in-line chlorination are both highly promising interventions, which could deliver water treatment at large scale and low cost. However, so far vouchers have only been tested at a modest scale.4 While in-line chlorination has been delivered more broadly, there are still open questions about how to deliver it at scale most efficiently.5

To fill this gap in evidence, GiveWell approached DIL in January 2022 to discuss its interest in conducting an RCT of water quality interventions that was statistically powered to detect the effects of chlorination on all-cause mortality and that would test vouchers at a large scale.

This grant provides funding for research on delivery of water quality interventions at scale and their effect on morbidity and all-cause mortality. At the time we recommended the grant, the Principal Investigator (PI) team for this research included:6

• Michael Kremer, DIL at University of Chicago
• Pascaline Dupas, Stanford University
• Amy Pickering, University of California, Berkeley
• Elisa Maffioli, DIL at University of Chicago and University of Michigan
• Akito Kamei, DIL at University of Chicago
• Sammy Khagayi, Kenya Medical Research Institute

Since recommending the grant, we have received a more complete list of PIs:

• Lead PI: Dr. Sanghamitra Pati, Scientist-G & Director ICMR-Regional Medical Research Centre (RMRC), Bhubaneswar
• Co-PI: Dr. Sidhartha Giri, Scientist-E ICMR-Regional Medical Research Centre (RMRC), Bhubaneswar
• Co-PI: Dr Krushna Chandra Sahoo, Public Health Specialist
• Co-PI: Dr. Jyotirmayee Turuk, Scientist C ICMR-Regional Medical Research Centre (RMRC), Bhubaneswar
• Co-PI: Dr. Anna Salomi Kerketta, Scientist F ICMR-Regional Medical Research Centre (RMRC), Bhubaneswar

Planned research activities

The water treatment interventions

DIL’s RCT in Kenya will use vouchers (also known as coupons) for free water treatment as the primary water quality intervention to be studied. These are printed or electronic vouchers that can be redeemed in a variety of locations, including one-stop shops and health clinics, to receive a free chlorine bottle.7

Depending on the outcome of the scoping phase (described below), research at sites in India or Nigeria may also utilize in-line chlorination as the intervention of interest. In-line chlorination employs devices that are compatible with existing water delivery infrastructure and automatically dispense chlorine to disinfect water at shared water collection points.8

The study phases

This is considered Phase 1 of a two-phase grant structure.

During this Phase 1 grant period, DIL will:

• Launch and fully execute an RCT of vouchers in Kenya with approximately 6,000 pregnant women;9
• Scope the potential of adding a trial site in India using in-line chlorination, or a trial site in Nigeria using either in-line chlorination or vouchers; and
• Launch and fully execute a small pilot RCT in preparation for a larger RCT in India or Nigeria, pending scoping outcomes.10

In fall 2023, DIL will submit a proposal for Phase 2.11 We expect that the Phase 1 scoping period will provide a better understanding of whether a multi-site trial that has enough statistical power to detect mortality effects is possible, and if so, in what countries and with which interventions.

If this is not possible, GiveWell and DIL will explore the possibility of substantially increasing the sample size in Kenya to gain enough statistical power to detect an approximately 5% minimum detectable effect (MDE) of the water quality intervention on under-two all-cause mortality.12

If GiveWell decides not to move forward with the Phase 2 grant, Phase 1 still fully funds a small RCT in Kenya, scoping in Nigeria for vouchers or in-line chlorination and in India for in-line chlorination, and, if scoping is promising, an additional small pilot RCT site in Nigeria and/or India.

Outcomes this RCT will measure

We expect that this RCT will provide evidence on:

• The effect of water chlorination interventions on under-two all-cause mortality;
• The factors that may affect mortality effects, such as intervention take-up and baseline water treatment, mortality, and diarrhea rates;
• The pathways of impact and relative effectiveness of various chlorine delivery mechanisms; and will
• Create the option for long-term follow-ups.13

Budget for grant activities

The budget for Phase 1 of this grant is $1,801,387.14 This includes staff salaries, travel to study sites, scoping in Nigeria, scoping in India, and a small pilot RCT in Nigeria or India.

The case for the grant

We are recommending Phase 1 of this grant because:

• In Phase 1, the small RCT of vouchers in Kenya and the small pilot RCT of vouchers or in-line chlorination in India or Nigeria will provide useful updates for our estimates of the cost-effectiveness of water chlorination interventions. We expect to get useful updates on chlorination rates, and though the study would be underpowered to detect mortality effects through Phase 1 alone, it would still be a useful input for updating our meta-analysis.15
• Phase 1 of this study creates the possibility of funding a larger, potentially multi-site RCT, which would have enough statistical power to detect under-two all-cause mortality.16 This would improve the precision of our pooled impact estimate of the effect of water chlorination on all-cause mortality.17
  • We estimate that the value of information gained from a $10 million trial is 12x. This includes information value only, and does not include the direct benefits of implementation to the treatment group in the RCT.18
  • Improving the mortality estimate could lead us to update our best guess for the cost-effectiveness of water quality interventions, which would in turn lead to a more informed and cost-effective allocation of resources.
• We expect the Phase 1 Kenya RCT will help us better understand vouchers as a standalone intervention, including its effects on chlorination take-up and mortality rates, as well as practical considerations for implementation. This RCT will improve the evidence base for vouchers specifically, since current evidence is limited.19 Our current best guess is that vouchers are 15 times as cost-effective as unconditional cash transfers (“15x”),20 ranging from 9x in Kenya to 41x in Nigeria.21 We view vouchers as a potentially promising water chlorination intervention to consider implementing at scale, similar to Dispensers for Safe Water and in-line chlorination.22
  • Given that the government of Kenya seems interested in scaling implementation of a vouchers program,23 there are likely lessons to learn about policy and scalability.
• A multi-site trial has the potential to be widely generalizable.
  • If scoping looks promising in Nigeria or India, and DIL is able to complete a multi-country, potentially multi-intervention trial, it would improve our understanding of how these water quality interventions generalize across different contexts with varying baseline chlorination rates, mortality rates, intervention take-up rates, and other factors.
• Results from this trial could influence potential wide-scale public sector adoption.
  • In addition to DIL’s impressions of the Kenyan government’s interest in vouchers,24 the Government of India has committed to providing piped water to every household in India through its Jal Jeevan Mission (JJM).25 Though JJM is focused on water access, we have heard from the DIL team and Evidence Action, which implements Dispensers for Safe Water and in-line chlorination in several other countries,26 that the Government of India is interested in scaling in-line chlorination or other water quality interventions.27
  • Though we currently estimate direct implementation of in-line chlorination to be only 6x in India,28 we think it’s highly plausible that technical assistance for in-line chlorination in India would be more promising. Our best guess is that there is also likely considerable variation in cost-effectiveness across the country, depending on baseline mortality rates, which we have not yet incorporated. Additionally, there is significant upside potential to better understanding intervention scale-up.29
• We view the study team as preeminent researchers in this field. The researchers undertaking this work have published extensively on water quality interventions.30 In discussions, they were transparent about study design, concerns, trade-offs, and learnings from preliminary scoping activities. We expect to check in with the team regularly on progress and are confident that the team is well-suited to execute the proposed RCT.

Risks and reservations

Our main reservations about this grant are:

• It’s possible that scoping will indicate that Nigeria and India are not promising for a full RCT, and we will need to substantially increase the sample size in Kenya to ensure that there is enough statistical power to detect the 5-7% mortality effect we anticipate. A single-site trial would have limited generalizability, given that under-five all-cause mortality rates are lower in Kenya than in other contexts we’d consider funding, such as Nigeria.31 We also think it’s possible that baseline chlorination rates are higher in Kenya, though this will be measured during scoping and piloting.32
  • Additionally, vouchers are likely not the best intervention for detecting the all-cause mortality effect of water chlorination interventions. In-line chlorination has higher take-up rates than vouchers, likely because it doesn’t require behavior change on behalf of the user, making it a better intervention to measure this effect size.33 There are ways to mitigate this risk, such as targeting vouchers to a population that is more likely to utilize them.34 It’s possible that a vouchers-only RCT would do little to update the precision of our pooled mortality impact estimate, and an uninformative null could present difficulties interpreting the findings.35
  • However, DIL has already completed a fair amount of due diligence on the feasibility of a trial in India and Nigeria, and seems optimistic that a multi-site trial is possible.36
• It’s possible that we will not fund Phase 2 of this grant at all. Potential reasons for this could include a lack of government buy-in in India or Nigeria that is discovered during the scoping phase, or significantly higher study costs than anticipated. This would mean that the Phase 1 RCT in Kenya would be under-powered to detect mortality effects on its own. However, it would still be useful for updating our meta-analysis and informing the evidence base for vouchers (see above).
• It seems possible that this trial could be at least partially funded by another funder if not by GiveWell, given DIL’s involvement and connections to other large funding organizations.37 However, we think it’s still worth moving forward, considering that this trial is highly informative for our research on water chlorination interventions.

Plans for follow up

We plan to meet regularly with the DIL team during the Phase 1 scoping portion of the grant. We expect to make a decision about funding for Phase 2 in fall 2023.

During Phase 1, we expect to learn about:

• Feasibility of a multi-site, multi-intervention RCT in Kenya, Nigeria, and India;
• Specifics for each site, including costs and statistical power;
• Site-specific needs, such as permissions and partnerships;
• How likely a program could be scaled in each country.38

GiveWell and DIL will discuss these factors to decide the most promising plan for Phase 2 activities.39

Internal forecasts

For this grant, we are recording the following forecasts:

Our process

• We published our water quality intervention report in April 2022, which relied heavily on the Kremer et al. 2022 meta-analysis and GiveWell’s subsequent alternate meta-analysis of chlorine interventions.
• We had multiple conversations with DIL between January 2021 and December 2022 surrounding its interest in:
  • Implementing an RCT of a water quality intervention, most likely vouchers, that is powered to detect an effect on mortality and could improve our meta-analysis impact estimate;
  • Expanding beyond a single-site trial of vouchers in Kenya to a multi-site trial of vouchers and/or in-line chlorination in some combination of Kenya, India, and Nigeria, pending initial scoping and pilot studies; and
  • Using a two-phased approach to structure the potential grants.
• We conducted in-depth research on vouchers for safe water and drafted an intervention report for this intervention in December 2022 (it was published here in June 2023).

Relationship disclosures

None

Sources

• 1

  “The Development Innovation Lab uses the tools of economics to develop innovations with the potential to benefit millions of people in low- and middle-income countries.” Development Innovation Lab, About the Lab.

• 2

  This estimate is an update to the 14% figure reported in our water quality intervention report, which was published in April 2022.

• 3

  “Most RCTs of water quality interventions were not designed to determine the impact of the intervention on mortality, and most publications do not report this outcome.” GiveWell, Water quality interventions, 2022.

• 4
  • Dupas et al. 2016 included a vouchers treatment group in Kenya of 382 individuals. Dupas et al. 2016, Table S1, “Baseline Characteristics and Balance Tests.”
  • Dupas et al. 2021 included a vouchers treatment group in Malawi of 872 households. Dupas et al. 2021, Table 7, “Cost-effectiveness.”

• 5
  • “In-line chlorination has not yet been implemented at scale. As it scales, Evidence Action will gather high-quality data that will improve our understanding of the program and the precision of our cost-effectiveness estimate.” GiveWell, Evidence Action's In-Line Chlorination Program — General Support (July 2022), “Summary.”
  • GiveWell recommended $5.6 million to Evidence Action's in-line chlorination program in Malawi in July 2022. This program is the first that we are aware of to implement in-line chlorination at scale. Thus, we believe there is still room for refinement of the technology and implementation techniques at scale.

• 6

  The DIL team plans to identify additional principal investigators in India and Nigeria. Amy Pickering, Assistant Professor, University of California, Berkeley, comment on a draft of this page, March 17, 2023.

• 7

  “One potential approach to increasing take-up of chlorination is to offer vouchers (also called coupons) for free chlorine treatment. Implementation details can vary, but in the trials we reviewed, this involved distributing to households yearly printed calendars where each month includes a voucher for a month’s supply of chlorine treatment (e.g., WaterGuard) for an average household size. Vouchers could then be redeemed in a variety of locations, including one-stop shops and health clinics (i.e., someone from the household would take the voucher to the one-stop shop every month to receive their free chlorine bottle).

  Researchers hypothesize that offering vouchers for free redemption of chlorine treatment could increase household chlorine use while reducing wastage by introducing a “hassle cost” for those who would otherwise not use chlorine distributed directly to their homes.

  Vouchers programs can target areas with relatively high all-cause mortality rates and low water chlorination rates to achieve higher counterfactual chlorination usage and mortality reductions. They may also target households with children under two or under five, or pregnant women, given the hypothesis that a large share of under-five deaths averted from chlorination are clustered in the neonatal to 12-month age range.” GiveWell, Chlorine vouchers, 2023.

• 8
  • “In-line chlorination is a technology for automatically disinfecting water at shared water collection points in low-income settings with unsafe water. We use the findings of water quality RCTs, adjusted for internal and external validity, to estimate reductions in all-cause mortality in children under five and people five and older. Pickering et al. 2019, the only published RCT of in-line chlorination with a diarrhea outcome, forms the basis of our adherence adjustment, a key input of our external validity adjustment. Clasen et al. 2015, the most recent Cochrane meta-analysis of water treatment trials, forms the basis for the morbidity reduction estimate and the plausibility limit derived from it. After adjustments, we estimate that in-line chlorination in Kenya reduces all-cause mortality by 11% in children under five and 2% in people five and over. We further estimate that development effects and medical costs averted account for 35% and 19% of the total benefit of the intervention, respectively.” GiveWell, Water quality interventions, 2022.
  • “ILC devices can be used to cost effectively achieve high treatment rates in populations that use either piped water systems or communal storage tanks. They automatically dose water with chlorine without requiring users to change the way they collect and manage their drinking water, shifting the burden of water treatment from households to a service provider. ILC devices that do not require electricity for operation are commercially available.” Development Innovation Lab, Proposal for water treatment study, 2022, p. 3.

• 9
  • This is a subset of the sample needed to detect a ~6% mortality effect. DIL estimates that this small sample would be able to detect a 20.5% mortality effect.
  • “For example, we estimate that a small initial HDSS study in Kenya with 6,000 pregnant women would have an MDE in a random effects model of 20.5%. (Of course this study is designed not primarily to shed light on mortality impacts but to inform the design of larger studies and to yield low cost information on morbidity, health care utilization and expenditure, etc.). A meta analysis which more heavily weighted studies in the same geography and with similar mortality rates would likely yield a lower MDE.” Development Innovation Lab, Proposal for water treatment study, 2022, p. 16.
  • “We estimate that spending the entire Phase II budget on a single coupon study in Kenya would lead to an MDE of 5.9%.” Development Innovation Lab, Proposal for water treatment study, 2022, p. 15.

• 10
  • Phase 1 of the study includes plans to launch and fully execute a small pilot RCT in Nigeria (using either vouchers or in-line chlorination) or India (using in-line chlorination). In Nigeria, a small pilot RCT of vouchers would target 12,600 pregnant mothers over three years, while a small pilot RCT of in-line chlorination would target around 6,600 pregnant mothers over two years. In India, a small pilot RCT of in-line chlorination would target 80 villages over two years. Development Innovation Lab, Phase I Budget, December 19, 2022 (unpublished).
  • The additional small pilot RCT site and water intervention type will be selected based on scoping results and assessing which option seems most promising for a multi-site RCT. While it’s likely that only one of these three additional RCT sites (vouchers in Nigeria, in-line chlorination in Nigeria, or in-line chlorination in India) will be selected and carried out, we would consider supporting multiple studies if scoping suggests more than one is very promising for further work.
  • “We’re budgeting for an initial study in India or Nigeria, because we expect that only one will be promising enough to launch a full RCT. In the event that it is possible to launch initial studies in all three countries, we could do so with smaller sample sizes or shorter follow-up periods.” Development Innovation Lab, Proposal for water treatment study, 2022, p. 4

• 11
  • Phase 2 will have a ceiling of approximately $8.2 million (Combined with the $1.8 million Phase 1 budget, this is equal to $10 million total across both phases): "We propose to produce each of these estimates, and with a total budget of $10m ($8.2m for Phase II) we expect fairly good precision." Development Innovation Lab, Proposal for water treatment study, 2022, p. 11.
  • It’s possible the ceiling could increase if we believe the value of information (VOI) learned from this grant could increase. Currently, our VOI model for this RCT estimates that the full Phase 1 and Phase 2 grant opportunity at $10 million is 12 times as cost-effective as unconditional cash transfers. See here for calculations.

• 12

  “Concentrating the Phase II budget in a single country would minimize the MDE for the context- specific effect in that country, albeit at the cost of lower precision to estimate the global effect or to characterize heterogeneity. We estimate that spending the entire Phase II budget on a single coupon study in Kenya would lead to an MDE of 5.9%.
  If we were to conduct a two-country study, we could also estimate an average treatment effect in the two contexts by pooling the data from these two contexts. For example, if we conducted coupon studies in Nigeria and Kenya (as in the above example), we would have an MDE of 6.2%. This is higher than the MDE for either single-country study because there is a fixed cost of working in each new country, so the sample for a two-country study is slightly smaller. However, a multi-site study would yield other benefits, such as providing more quantitative evidence to predict context-specific effects.
  We currently estimate that a single-country study of ILC in Nigeria would have an adjusted MDE of 3.6% (raw MDE 6.1%) and a single-country study of ILC in India would have an adjusted MDE of 2.4% (raw MDE 9%).” Development Innovation Lab, Proposal for water treatment study, 2022, p. 14-15.

• 13

  “While the research project described in this proposal has been designed primarily to provide information on the impact of water treatment on mortality, it will also produce:

  1. Shed light on how mortality impact varies with factors such as baseline child mortality and diarrhea rates, and with water treatment rates, thus allowing funders to better target water treatment interventions where they will have the most impact.
  2. Evidence on potential pathways of impact, obtained by (a) conducting verbal autopsies to determine the cause of deaths, (b) gathering data on age-specific mortality, and (c) collecting information on morbidity data. Such evidence may help resolve the discrepancy between experimental evidence from meta-analyses of water treatment RCTs on mortality and models focused on particular scientific pathways. For example, the proposed study would provide information on whether water treatment reduces mortality only following weaning of children from breastfeeding, consistent with diarrhea as the key pathway, or whether there are also effects on neonatal mortality. It will also provide information on the benefits and costs of making coupon redemption possible in shops as well as clinics. It may of course also demonstrate that certain approaches are not effective, which could also accelerate scale up by allowing resources to be focused on more promising approaches. For example, if we found that take up rates of water treatment through a coupon program in Nigeria were very low, this would point to focusing on other strategies, such as in-line chlorination (ILC), in Nigeria. We are and will continue to collaborate closely with partners, including Evidence Action and the governments of India and Kenya, both at operational and senior levels, so that this study will allow us to position programs for rapid scale-up, contingent on positive results.
  3. Detailed tracking data to enable long-run follow-ups, enabling future measurement of long-run health and mortality effects, with potential implications for both cost effectiveness and program design.” Development Innovation Lab, Proposal for water treatment study, 2022, p. 1-2.

• 14

  Development Innovation Lab, Phase I Budget, December 19, 2022 (unpublished).

• 15

  The Phase 1 Kenya vouchers RCT sample size (approximately 6,600 participants) is more than twice as large as the largest trial (Boisson 2013, 2,991 participants) included in our meta-analysis.

• 16
  • We believe this to be true because our impression is that the research team is high-quality and very experienced with executing well-conducted, large field trials of water interventions, including vouchers. Additionally, power calculations and budget projections for a few single- and multi-site RCT scenarios suggest that a well-powered trial within our budget constraint is feasible.
  • “We estimate that spending the entire Phase II budget on a single coupon study in Kenya would lead to an MDE of 5.9%.
    If we were to conduct a two-country study, we could also estimate an average treatment effect in the two contexts by pooling the data from these two contexts. For example, if we conducted coupon studies in Nigeria and Kenya (as in the above example), we would have an MDE of 6.2%. This is higher than the MDE for either single-country study because there is a fixed cost of working in each new country, so the sample for a two-country study is slightly smaller. However, a multi-site study would yield other benefits, such as providing more quantitative evidence to predict context-specific effects.
    We currently estimate that a single-country study of ILC in Nigeria would have an adjusted MDE of 3.6% (raw MDE 6.1%) and a single-country study of ILC in India would have an adjusted MDE of 2.4% (raw MDE 9%).” Development Innovation Lab, Proposal for water treatment study, 2022, p. 15.

• 17

  “The new study will have a much larger sample than the 15 studies in Kremer et al. 2022 combined (depending on how the study is designed it could have between 120,000 and 400,000 participants, compared to around 25,000 in the existing meta-analysis), and thus they would have the potential to substantially improve the precision with which the mean effect of water interventions can be estimated and to shed much more light on how effects vary with factors such as the death rate and the chlorination rate. The ability to do this would likely be maximized by allocating the budget across three or more different studies, although this would reduce sample size and thus the precision of the zero pooling estimate for each study. GiveWell had asked about a scenario with studies in Kenya and Nigeria. We estimate that combining a multi-site study in Kenya and Nigeria with the 15 studies in the Kremer et al. 2022 meta-analysis would yield an MDE of the global average mortality reduction between 0% (in which the overall global average estimated treatment effect through the meta-analysis would be significant with any non-negative effect in the new studies) and 2.5%. It is likely that we will be able to improve on this by optimizing the split of the budget across settings in order to maximize statistical power.” Development Innovation Lab, Proposal for water treatment study, 2022, p. 14.

• 18

  We view this as a conservative estimate. In addition to excluding the direct benefits of implementation to the treatment group during the trial, it also doesn’t include potential upside to influencing host-country governments or other funders.

• 19

  The two existing RCTs of chlorine vouchers (Dupas et al. 2021 and Dupas et al. 2016) both have limitations that this study could address. These limitations include: short-term follow-up periods, moderate attrition rates, lack of pure control group, lack of health outcome measurement, and self-reporting bias. This study could address each of these limitations by extending the follow-up period, measuring child and/or adult mortality (which is an objective outcome measure not subject to social desirability bias), including a control group, and improving follow-up rates. The current limitations of vouchers evidence likely wouldn’t prevent us from funding vouchers implementation, but this is an opportunity to improve the evidence and better understand this likely priority area. We discuss this in further detail in the “Evidence of vouchers on water chlorination” section of our intervention report.

• 20

  Note that (a) our cost-effectiveness analyses are simplified models that are highly uncertain, and (b) our cost-effectiveness threshold for directing funding to particular programs changes periodically. As of late 2022, our bar for directing funding is about 10 times as cost-effective as unconditional cash transfers. See GiveWell’s Cost-Effectiveness Analyses webpage for more information about how we use cost-effectiveness estimates in our grantmaking.

• 21

  See our calculations here.

• 22

  We directed approximately $70 million to Evidence Action’s Dispensers for Safe Water and in-line chlorination programs in 2022.

• 23

  DIL researchers conveyed their impressions from interactions they’ve had with Ministry of Health officials in Kenya. This interest could be overstated. DIL, conversation with GiveWell, November 2, 2023 (unpublished).

• 24

  DIL, conversation with GiveWell, November 2, 2023 (unpublished).

• 25

  “Jal Jeevan Mission is envisioned to provide safe and adequate drinking water through individual household tap connections by 2024 to all households in rural India. The programme will also implement source sustainability measures as mandatory elements, such as recharge and reuse through grey water management, water conservation, rain water harvesting. The Jal Jeevan Mission will be based on a community approach to water and will include extensive Information, education and communication as a key component of the mission. JJM looks to create a jan andolan for water, thereby making it everyone’s priority.” Government of India, Jal Jeevan Mission.

• 26

  We directed approximately $70 million to Evidence Action’s Dispensers for Safe Water and in-line chlorination programs in 2022.

• 27

  GiveWell’s non-verbatim summary of a conversation with Evidence Action, February 2, 2023 (unpublished)

• 28

  See here for calculations.

• 29

  For example, it seems plausible that scaling in-line chlorination in India would prove as a powerful proof of concept case for other countries or funders considering safe water investments.

• 30

  For example: Dupas et al. 2016, Dupas et al. 2021, Pickering et al. 2019, and Kremer et al. 2022.

• 31

  “According to DHS data, in Kenya the mortality rate for those attending no ANC visits is 2.7 times higher than for those who attend at least one ANC visit. According to the GBD in 2019, the U5 mortality rate (U5MR) was 41 deaths per 1,000 live births. We can therefore estimate an U5MR of 39 for those who attended at least one ANC visit, and 103 for those who attended no ANC visits in 2019.
  According to DHS data, in Nigeria the mortality rate for those attending no ANC visits is 1.5 times higher than for those who attend at least one ANC visit. In Nigeria, national ANC attendance rates are low, but this is driven by northern states. We therefore focus on southern states, which have both high ANC attendance and high child mortality. As table 1 shows, there are many states where ANC attendance rates and U5MR among ANC attendees are high.” Development Innovation Lab, Possible selection effects, 2022, pp. 1-2.

• 32

  Higher baseline chlorination rates would make it harder to detect the effect of a newly implemented chlorination intervention, since the difference between baseline and endline chlorination would be smaller and likely have a smaller effect on mortality.

• 33

  “However, the benefit of ILC is that it is likely to have near 100% take-up. This does not reduce
  the MDE directly, but it means that we would expect a larger mortality effect, reducing the
  overall probability of a false negative result." Development Innovation Lab, Proposal for water treatment study, 2022, p. 13.

• 34

  We have tried to address these reservations with DIL by working with their team to (i) focus on a study design that gives vouchers the best chance of detecting an effect (i.e., targeting regions of Kenya with high baseline mortality rates and lower household chlorination rates, targeting vouchers to pregnant women and measuring under-two mortality as primary outcome); and (ii) working to expand the study to a multi-site trial that tests vouchers and in-line chlorination in different countries, and pooling mortality effects across sites/interventions. Since our understanding is that in-line chlorination is a more potent intervention that requires less behavior change on behalf of the user, take-up and therefore effect sizes are likely to be higher relative to vouchers.

• 35

  If the study finds a null effect on all-cause mortality because take-up of vouchers was too low, it would be difficult to determine whether vouchers are ineffective in reducing mortality or the study was just underpowered to detect an effect. This would likely present challenges in interpreting and communicating the findings with the broader research community and general public.

• 36

  Development Innovation Lab, Proposal for water treatment study, 2022, “Phase 0: Update on 2022 activities so far (DIL-funded).” pp. 4-6.

• 37

  We account for this possibility with a -10% adjustment in our value of information model. See the calculations here.

• 38

  “From mid to late 2023, we will have better information about a) whether it is plausible to run a large-scale study in each country, b) factors affecting the power of such a study (expected take- up, cost of implementation and data collection, intra-cluster correlation for ILC), and c) how quickly such a study could be launched (e.g, established partnerships, required permissions and processes, etc), and d) the likelihood of scale in each context (for example, if the government of Nigeria were enthusiastic about scaling a coupon program, that might raise the value of working in Nigeria).” Development Innovation Lab, Proposal for water treatment study, 2022, p. 9.

• 39

  “Based on this, we would discuss the options with GiveWell. There may be trade-offs here between different objectives (estimating the local treatment effect, estimating the global treatment effect, positioning the program for scale-up). We would then make a collective judgment on how to spend the remaining $8.2m. This would include whether to run a single- or multi-country study, choosing countries and study sites, and deciding on the design of implementation and data collection.
  We would then launch larger scale studies as early as possible. We understand that GiveWell puts substantial value on having answers as soon as possible, as well as on conducting the best possible study.
  We're fairly confident that we could launch a larger scale study in Kenya by mid 2023. Three to four months after the launch in HDSS sites, we should have sufficient information to settle on a final version of the program (clinic vs kiosk redemption, water-guard vs aqua-tabs, etc.), and launch at a larger scale.
  We can provide estimates for other countries, but they're subject to some uncertainty. In both India and Nigeria, we should be able to conduct the research activities described above and be ready to launch larger studies between October 2023 and January 2024. However, timelines will depend on local partners, including state and national governments and implementing partners. We are continuing to liaise with government partners to try to address this, for example, Elisa Maffioli just traveled to Nigeria and Michael Kremer is going to India in January. They both met (or plan to meet with) the relevant ministers and officials. By Fall 2023, we will have a much better understanding of timelines in each country, and this can be taken into account when choosing where to conduct larger-scale research. Development Innovation Lab, Proposal for water treatment study, 2022, pp. 9-10

• 40

  “Frequentist and Bayesian methods were used to estimate the effect of water treatment on child mortality among included studies. We estimated a mean cross-study reduction in the odds of all-cause under-5 mortality of about 30% (Peto odds ratio, OR, 0.72; 95% CI 0.55 to 0.92; Bayes OR 0.70; 95% CrI 0.49 to 0.93). The results were qualitatively similar under alternative modeling and data inclusion choices. Taking into account heterogeneity across studies, the expected reduction in a new implementation is 25%.” Kremer et al. 2023, p. 1.