Water Quality Interventions

Published: April 2022 (November 2013 version of page here)

What is the impact of water quality interventions on mortality?

Overall, we believe the evidence is moderately strong that water quality interventions such as chlorination reduce all-cause mortality in children under 5, and probably to a small degree in people five and over as well, but we have high uncertainty about the size of the effect.

Water chlorination is effective for controlling waterborne bacteria and viruses, but it has limited effectiveness against the protozoan parasite Cryptosporidium, a common cause of diarrhea in children in low-income settings.

We pool mortality data from five randomized controlled trials of chlorination interventions in children under five years old in low-income settings to estimate the impact of chlorination on all-cause mortality in this demographic. This estimate suggests that chlorination reduces all-cause mortality in children under five by about 14%, although our estimates of mortality reduction in specific charity contexts are lower. This is unexpectedly large relative to the mortality impact one would expect based on the estimated share of all-cause mortality caused by diarrhea and the impact of chlorination on diarrheal illness.

Other sources of evidence, including studies on the mortality impact of historical water quality improvements, suggest that water quality interventions may reduce mortality from non-waterborne diseases. This provides a plausible explanation for the unexpectedly large estimate of the impact of chlorination on mortality in children under five.

Mechanism of action

Chlorination is a well-characterized method of disinfecting drinking water. When chlorine is added to water, it reacts with organic matter,1 inactivating microorganisms like viruses and bacteria.2 The rate and efficacy of inactivation of microorganisms varies, and depends on pH, chlorine concentration, and chlorine demand.3 Additionally, microorganisms may be protected from inactivation by chlorine if they are attached to or within particulate matter in the water; therefore, chlorination is less effective in water with more particulate matter.4

Some microorganisms—including the cysts of diarrhea-causing protozoa such as Cryptosporidium and the eggs of parasitic worms such as Schistosoma—are resistant to chlorine.5 A multi-country study suggests that Cryptosporidium is a common cause of diarrhea in children in low-income settings.6 After disinfection, excess chlorine in the water may remain available to prevent recontamination, depending on storage conditions.7

The ability of chlorine to inactivate waterborne bacteria and viruses is consistent with the reduction in diarrhea risk reported in chlorination trials in settings with a high prevalence of diarrhea.8

Global Burden of Disease (GBD) estimates that enteric infections cause 15% of all deaths in children under five in sub-Saharan Africa,9 and the World Health Organization (WHO) states that “diarrhoeal disease is the second leading cause of death in children under five years old.”10 Because chlorination reduces the concentration of diarrheal pathogens and protects against diarrheal illness, we believe this provides moderately strong evidence that it reduces all-cause mortality in children under five years of age.

Randomized controlled trials

At least 17 randomized controlled trials (RCTs) have collected data on the impact of water quality interventions on all-cause mortality in children under five in settings without safe water. To derive our estimate of the mortality impact of chlorination in low-income settings, we pool data from five RCTs that tested chlorination interventions similar to those we are evaluating and which we believe are the least susceptible to publication bias and other limitations. Our estimate suggests that chlorination interventions reduce all-cause mortality in children under five by about 14% in low-income settings.11 This estimate is imprecise and we do not view the evidence underlying it as strong, but we nevertheless believe it is the most informative estimate available due to its directness.

Most RCTs of water quality interventions were not designed to determine the impact of the intervention on mortality, and most publications do not report this outcome. Michael Kremer and colleagues collected published and unpublished mortality data from randomized controlled trials of water quality interventions, then pooled them in a meta-analysis, Kremer et al. 2022 (working paper).12 An advantage of this approach is that it can yield statistically persuasive conclusions from trials that individually do not have enough statistical power to do so.

The meta-analysis identified 17 water quality trials with under-5 mortality data and pooled 15 of them in its main analysis.13 Twelve of these 15 trials involved chlorination interventions,14 although only 4 of the trials involving chlorination used interventions that were limited to chlorination.15 Four of the chlorination trials included flocculation,16 which removes particulate matter and chlorine-resistant pathogens.17 Six of the chlorination trials included safe storage vessels intended to limit recontamination of chlorinated water after collection.18 Three of the chlorination trials included hygiene interventions in addition to water treatment, such as handwashing education and supplies.19

Of the three trials that did not test chlorination, two tested filtration20 and one tested spring protection.21

Depending on the method used to pool results, the analysis reports that water quality interventions reduce the odds of all-cause mortality in children under five by 28% (95% confidence interval, 8% to 45%) or 30% (95% credible interval, 8% to 51%).22

We have reviewed Kremer et al. 2022 (working paper) internally and had it reviewed externally by a water quality trial expert (Thomas Clasen) and a statistical methods expert (Megan Higgs).23 This process identified several important limitations:

• Quantified uncertainty. The size of the effect as reported in the paper is quite uncertain, with the 95% confidence interval of the 28% odds of mortality reduction estimate ranging from 8% to 45%.24
• Unquantified uncertainty. The 95% confidence interval of a meta-analytic estimate accurately reflects uncertainty when the assumptions underlying the method are fully satisfied. This is typically not the case, and Higgs believes uncertainty is greater than reflected in the 95% confidence interval associated with the estimate, particularly when using the estimate to predict future outcomes (even in similar settings) rather than quantify variation within the included studies.25
• Limitations of preregistration. Preregistration of studies reduces the risk of bias.26 Although Kremer et al. 2022 (working paper) has a public registry page, it was created in July 2020,27 after initial literature searches and analyses had been conducted.28
• Possible publication bias. Kremer and colleagues were unable to obtain mortality data from the majority of water quality trials that met their criteria.29 It is possible that the trials included in the meta-analysis are not representative of all the trials that were conducted, biasing the effect size estimate. Using two tests of publication bias, the authors did not identify statistically significant evidence of such bias.30 However, statistical power to detect publication bias was limited, and the paper reports that if one assumes publication bias occurred, adjusting for it reduces the effect size to a 17% reduction in the odds of mortality (95% confidence interval, 8% to 26%).31 The paper also demonstrates that the reduction in diarrhea risk reported in included trials is similar to that in excluded trials, which provides some evidence against publication bias.32 Nevertheless, we remain somewhat concerned about publication bias.
• Bundled treatments. Most of the trials underlying this meta-analysis included interventions in addition to water quality improvement, such as the provision of safe water storage vessels and hand washing education and supplies.33 The results are therefore likely to overestimate the impact of isolated chlorination interventions such as those we are evaluating. We apply a downward adjustment in our cost-effectiveness analysis to account for this.34
• External validity. Some of the trials used water quality interventions that are dissimilar to chlorination, such as spring protection and water filtration.35 The external reviewers questioned the relevance of these trials to simple chlorination interventions such as chlorine dispensers and in-line chlorination.36
• Mortality assessment methods. These trials were not designed to measure mortality impacts as a primary outcome and they may have used less reliable methods to do so.37

• The water treatment method is chlorination, without additional treatments like flocculation or filtration. This excludes water quality interventions that were less similar to those we are evaluating.
• Follow-up length of one year or greater. This tends to exclude small trials that are more susceptible to publication bias.38

We also exclude Haushofer et al. 2021 (working paper), a follow-up study of a RCT of chlorine dispensers, because we believe the effect size it reports is implausibly large, and it has a substantial impact on the pooled estimate.39

This leaves five trials, Reller et al. 2003, Boisson et al. 2013, Luby et al. 2018, Null et al. 2018, and Humphrey et al. 2019. We weight them using inverse variance, a common meta-analysis method that minimizes variance of the mean, and also by their similarity to the simple chlorination interventions we are evaluating. We then pool individual trial estimates of mortality reduction using these weights.40

The resulting estimate suggests that chlorination interventions reduce all-cause mortality in children under five by approximately 14% in low-income settings (95% confidence interval, 32% reduction to 10% increase). Three large, recent water quality trials contribute 97% of study weight: Luby et al. 2018, Null et al. 2018, and Humphrey et al. 2019. We believe our pooled estimate is resistant to publication bias because large, recent water quality trials have all reported mortality outcomes, and trials this size are less likely to go unpublished.41 In addition, the three large trials all report similar effect size estimates (6% to 17% reduction in all-cause mortality), with the less precise estimates of the two smaller trials lying on either side of this cluster.42

We note that our pooled estimate is imprecise and not statistically significant by conventional standards. We nevertheless use it as the base effect size estimate in our cost-effectiveness analysis, for the following reasons:

• We are not using the estimate to determine whether chlorination reduces mortality, but rather to estimate effect size as reliably as we can. Stronger evidence suggests that chlorination reduces the risk of experiencing diarrhea in children under five.43 Since diarrhea is a major cause of mortality in this age group,44 we are fairly confident that chlorination reduces all-cause mortality to some degree.
• Meta-analysis of RCTs is the most direct estimate available of the impact of water chlorination on all-cause mortality in children under five in low-income settings.
• Although our estimate is uncertain, this degree of uncertainty does not stand out relative to our other cost-effectiveness analyses.45

We consider the mortality reduction effect reported in Kremer et al. 2022 (working paper) and estimated by our pooling method to be surprisingly large, given the impact of chlorination interventions on the risk of diarrheal illness. The most recent Cochrane meta-analysis of water quality interventions reports that chlorination interventions reduce the risk of diarrheal illness by 23% in low-income settings.46 GBD estimates that 14.5% of all-cause mortality in children under age five in countries with low socio-demographic index is caused by enteric infections.47 Together, these estimates imply that chlorination might reduce all-cause mortality by 3.3%, about one-eighth of the Kremer et al. 2022 (working paper) estimate and about one-quarter of our estimate.48 However, after further adjustments in our cost-effectiveness analysis, the discrepancy is smaller.49

We are uncertain about our central estimate of the mortality impact of chlorination, but we believe the impact of water quality interventions on all-cause mortality is probably larger than predicted from their impact on diarrheal illness, for the following reasons:

• Mortality caused by diarrhea may be underestimated. The GBD estimate of diarrhea mortality in children under five assumes that each death has exactly one cause.50 However, deaths often have more than one contributing cause, such that averting any single contributing cause would avert death.51 Therefore, we expect this method to underestimate the total mortality impact of averting a specific disease such as diarrhea.
• Related to the previous point, water quality interventions probably avert deaths attributed to causes other than diarrhea. Observational studies suggest that diarrhea may be a significant cause of malnutrition,52 and that malnutrition in turn increases the risk of dying from several major infectious diseases.53 Five of six RCTs of water quality interventions included in Kremer et al. 2022 (working paper) that report growth outcomes did not report an impact of the intervention on weight or height growth; however, four of five reporting no effect on growth also reported no effect on diarrhea, suggesting that the interventions were not very effective.54 There is a substantial body of research suggesting that historical municipal water quality improvements in the US and Europe reduced mortality to a larger degree than can be explained by direct mortality from waterborne diseases.55 These interventions apparently reduced mortality from causes not directly related to waterborne diseases, such as respiratory infections.56 We review these studies in the next section. In support of this, more recent observational studies suggest that diarrheal infections may leave people more susceptible to respiratory infections.57 Evidence from water quality intervention trials is partially supportive and partially unsupportive of this mechanism.58
• The effect size we estimate and that reported in Kremer et al. 2022 (working paper) are roughly consistent with the estimated impact of historical municipal water quality improvements on child mortality in the US and Europe.59 We review these studies in greater detail in the next section.

Conversations with researchers

We spoke with three researchers about the link between water quality and mortality: Stephen Luby, professor of medicine at Stanford University and a co-author of the Kremer et al. 2022 (working paper) meta-analysis;60 Robert Black, professor at the Johns Hopkins Bloomberg School of Public Health;61 and Jay Berkley, professor of pediatric infectious diseases at the University of Oxford.62 Dr. Luby argued that changes in diarrhea morbidity are a poor proxy for changes in all-cause mortality, possibly explaining the apparent discrepancy between the impact of water quality interventions on the two outcomes.63 All three believe the impact of water quality interventions on all-cause mortality may exceed what would be predicted from their impact on diarrhea alone,64 and all three believe the magnitude of all-cause mortality reduction in children under five implied by our cost-effectiveness analysis of in-line chlorination and chlorine dispensers (6-11% reduction) is plausible, although Dr. Black views our estimates in two locations (11%) to be near the upper limit of plausibility.65

Studies of historical water quality improvements

A substantial body of research has estimated the mortality impact of historical water quality improvements, primarily in the US, Europe, and Japan. Overall, this research provides fairly consistent evidence that major municipal water quality improvements reduced all-cause mortality in children and probably adults. These studies also provide fairly consistent evidence that improving water quality reduces mortality from causes not directly linked to water quality, such as respiratory infections. We are uncertain of the relevance of these estimates to modern charity settings, but the effect sizes they report are roughly consistent with our estimates, and they support the hypothesis that reducing the risk of waterborne disease also reduces the risk of non-waterborne diseases.

In the late 19th and early 20th centuries, cities in several nations implemented major municipal water quality improvements.66 These often involved discrete events such as the chlorination or filtration of an entire municipal water supply, making them well suited for impact evaluation.67 Researchers at the time, and more recently, have estimated the health and mortality impact of these interventions. We focused our searches on the economics literature since our initial searches suggested this is primarily where these studies are published.68 We focused on studies that estimate the degree of all-cause mortality reduction caused by a major water quality improvement or that evaluate the hypothesis that improving water quality impacts causes of death not directly related to waterborne disease. This latter hypothesis is called the Mills-Reincke phenomenon.69 We have only examined these studies lightly, except the most influential study in the literature on this topic,70 Cutler and Miller 2005, which we conclude probably overestimates the impact of water quality improvements on mortality.71

We identified eleven studies that report the effect of water quality improvements on mortality, most of which are natural experiment studies.72

The studies we identified report a range of effect sizes of water quality improvements on population-wide all-cause mortality, from +1 to -58 percent, with a median effect size of -19 percent.73 They also report a range of effect sizes for under-five all-cause mortality, from +8 percent to -54 percent, with a median effect size of about -12 percent.74 Studies with the strongest evidence of large improvement in water quality report a range of effect sizes on all-cause mortality of +1 to -58 percent, with a median effect size of -19 percent.75 This same subset of studies estimates that water quality improvements altered under-five all-cause mortality by +8 to -54 percent, with a median effect size of -11 percent.76

We identified eight studies that report outcomes relevant to the Mills-Reincke phenomenon.77 Note that six of these studies overlap with those mentioned previously. All studies except one provide evidence that supports the Mills-Reincke phenomenon.78 Overall, we believe these studies provide fairly consistent evidence in support of the Mills-Reincke phenomenon, either because they report reductions in mortality from non-waterborne diseases, or because the degree of reduction in all-cause mortality is larger than can be explained by changes in waterborne disease incidence or mortality. Respiratory diseases such as pneumonia and tuberculosis are the most consistent non-waterborne diseases to be prevented by water quality improvements in these studies.79

These studies provide a mechanism that could contribute to explaining the surprisingly large reduction in mortality observed in the Kremer et al. 2022 (working paper) meta-analysis: water quality improvements may reduce mortality from infectious diseases that are not classified as waterborne, in addition to waterborne diseases.

Several factors lead us to be uncertain about the relevance of these studies to modern interventions that target waterborne disease in low-income settings:

• Most were conducted in contexts that differ substantially from the contexts in which water quality interventions would be applied today. This includes differences in location, infectious disease profile, baseline mortality rates, and other factors.80
• The water quality interventions represented in these studies vary, and none is identical to the interventions to which we might direct funding.81
• These studies are observational and used different strategies to attempt to isolate the causal impact of water quality on mortality. We believe some of these strategies are more convincing than others, but all of them may be susceptible to some degree of confounding from other variables that changed alongside water quality.
• We do not know whether unsupportive studies went unpublished, potentially creating publication bias. This would make the overall literature appear more supportive than it would otherwise be.
• None of these studies were preregistered, increasing the risk of bias.

For these reasons, we view these studies as providing rough triangulation for our mortality reduction estimate and a plausible mechanism to explain its unexpectedly large effect size. We do not see them as providing precise effect size estimates for the impact of water quality interventions on mortality.

Over-five mortality

Although Kremer et al. 2022 (working paper) focuses on the mortality impact of water quality improvement on children under five years of age, we believe its benefits are unlikely to be limited to this age group, for the following reasons:

• GBD data attribute some over-five mortality to waterborne disease. In Kenya, for example, these data suggest that 5.5 percent of over-five mortality is caused by diarrheal diseases, compared with 17 percent of under-five mortality.82
• Natural experiment studies suggest that historical municipal water quality improvements led to reductions in mortality that were not limited to children under five. See our summary of this evidence here.

Other benefits of water quality interventions

Morbidity reduction

There is fairly strong evidence that water quality interventions reduce the risk of diarrhea morbidity in low-income settings. Clasen et al. 2015, the most recent Cochrane Collaboration meta-analysis on the subject, reports that water disinfection at the household level reduces diarrhea risk by about one quarter, and other types of water quality interventions are also effective. Evidence certainty ranged from low to moderate, depending on the intervention.83 Chlorination’s well-characterized and straightforward mechanism of protection against diarrheal pathogens adds to the case that it is effective.

Development effects

We tentatively believe that water quality interventions probably increase later-life income, which we call “development effects.” We remain very uncertain about this benefit, but include it because other interventions that provide health benefits to children appear to increase adult income. Bleakley 2010 and Cutler et al. 2010 are natural experiment studies that suggest that malaria eradication programs in India, South America, and North America increased the eventual adult income of children that benefited from them.84 Baird et al. 2016 is a follow-up to a randomized controlled trial that suggests that a deworming program in Kenya increased the eventual adult income of children that benefited from it.85

In addition, in a medium-depth search of the biomedical and economics literature,86 we identified one study that provides direct evidence on this question. Beach et al. 2016 estimates the impact of waterborne disease exposure in early life on adult income in men in the early 20th-century United States.87 The study estimates that a marked improvement in water quality around the time of birth increased eventual adult income by 1% to 9%, depending on how it is measured.88 Although this finding is broadly consistent with the studies previously discussed, we do not place much weight on it due to its limitations.89

Because the only piece of direct evidence on development effects we are aware of is observational and has substantial limitations, we remain very uncertain about the existence of this benefit and its possible size.

Medical costs averted

Parents often seek medical care for children who have diarrhea or other illnesses. In Kenya, Uganda, and Malawi, three locations we model in our cost-effectiveness analysis of in-line chlorination and chlorine dispensers, survey data suggest that parents seek care for diarrhea episodes in their children under five years old 58% to 71% of the time.90 Estimated care-seeking rates for respiratory tract infections and fever are similar to those for diarrhea.91

The costs of such care are substantial. We roughly estimate that in low- and lower-middle-income sub-Saharan Africa:

• About 6% of total diarrhea cases are admitted to a hospital and receive inpatient care.92 An inpatient care episode costs an average of $44-117 USD, depending on whether a country is low-income or lower-middle-income.93
• About 60% of total diarrhea cases receive outpatient care.94 An outpatient care episode costs an average of $14-33, depending on whether a country is low-income or lower-middle-income.95

We further estimate that among beneficiaries of chlorine dispensers in Kenya, Uganda, and Malawi, children under five years old experience about four episodes of diarrhea per year.96 Together, these figures imply that parents spend $41-95 USD per year on diarrhea care for each child under five in these contexts, with additional expenditures for medical care of other diseases whose risk may be reduced by water quality interventions.97

We conclude that averting diarrheal illness has the potential to meaningfully reduce medical costs in households with children under five years old.

Potential offsetting/negative effects of water quality interventions

Chlorine itself is toxic to humans in concentrated form,98 but at lower concentrations water chlorination has a long history of apparently safe municipal use.99

Chlorine reacts with substances present in water to create harmful by-products. The concentration of these by-products depends on the concentration of chlorine, organic and inorganic substances present in the water, and time.100 WHO states that “the risks to health from disinfection by-products are extremely small in comparison with the risks associated with inadequate disinfection,”101 but we have not evaluated this statement.

For these reasons, we believe the risk of substantial harm is probably low, and we have not included offsetting effects in our cost-effectiveness analysis.

How we incorporate the evidence into our cost-effectiveness analyses

Where applicable, our cost-effectiveness analyses of water quality interventions include reductions in under-five mortality, reductions in over-five mortality, reductions in diarrhea morbidity, averted medical costs, and the impact of receiving the intervention in childhood on adult income (“development effects”).102

Mortality reduction is the single largest driver of the cost-effectiveness of water quality interventions in our models. Our estimate is based on RCTs of water quality interventions, adjusted for limitations of the studies and biological plausibility (internal validity) and for major differences between the context of the included trials and the beneficiary context (external validity).

We estimate diarrhea morbidity reduction by multiplying the morbidity burden of diarrhea in the beneficiary context by the degree of reduction caused by the intervention. To estimate medical costs averted, we multiply the annual costs of care for diarrhea and other diseases thought to respond to water quality by the annual reduction in diarrhea cases caused by the intervention. To estimate development effects of water quality interventions, we use a formal method we developed for inferring development effects from the impacts of an intervention on measures of early-life health and growth.

Mortality reduction

Mortality reduction is the single largest driver of the cost-effectiveness of water quality interventions in our models. Our estimate is based on RCTs of water quality interventions because they are the only published direct estimates of the impact of water quality interventions on mortality across multiple low-income contexts.103

For chlorination interventions, we begin with the pooled mortality reduction estimate from five chlorination RCTs, which suggests that the intervention reduces all-cause mortality in children under five years old by about 14%.104

We do not have a direct estimate of over-five mortality reduction. Where applicable, we derive estimates of over-five mortality reduction from under-five mortality reduction by using the proportion of all-cause mortality attributed to enteric diseases in each age group,105 and adjusting for our belief that the Mills-Reincke phenomenon is less applicable to people over five.106

We further adjust these estimates for the fact that some of the RCTs included interventions other than chlorination. We limit the size of the mortality effect based on our assessment of its plausibility, given the estimated causes of death in each location we model. Finally, we adjust for major differences between the context of the included trials and the beneficiary context (external validity), especially differences in intervention adherence between the RCTs and the beneficiary context.

One of the most impactful adjustments we apply is an effect size limit based on biological plausibility.107 Based on the mechanisms we believe are most likely to mediate the relationship between water quality improvement and mortality reduction, we generate an estimate of mortality reduction that is the maximum we believe is biologically plausible in the beneficiary context.108 We use this to constrain mortality reduction estimates in our model.

To generate the plausibility adjustment, we investigated several possible mechanisms that could explain a larger-than-expected impact of water quality interventions on all-cause mortality.109 Of these, the only mechanism that was substantially supported and could explain a large effect is that water quality interventions reduce the risk of non-waterborne infectious and nutritional diseases.110 To generate an upper bound of biological plausibility for each beneficiary setting that is independent of the estimates from RCTs, we assume that mortality from all nutritional and infectious diseases is reduced by a percentage equal to the percent reduction in diarrhea morbidity resulting from the water quality intervention.111 For example, we estimate that in-line chlorination reduces diarrhea morbidity by 17% in children under five in Kenya,112 so we assume for the plausibility adjustment that the risk of death from all infectious and nutritional causes is reduced by 17%. This implies a maximum plausible reduction in all-cause mortality in children under five of 11%.113

We believe an effect size larger than implied by this method would be difficult to justify, and even this effect size requires optimistic assumptions. We emphasize that the plausibility limit is not intended to be our best guess of effect size, but rather to exclude implausible values. We believe RCTs of water quality trials offer the best available estimate of the impact of water quality interventions on mortality in children under five, but our estimate derived from RCTs is imprecise so we use the plausibility limit in an attempt to limit the risk of overestimation.

We also apply an internal validity adjustment to account for the fact that some of the RCTs that underlie our mortality reduction estimate included interventions in addition to chlorination, such as the provision of safe water storage vessels.114

When adjusting for external validity, we consider the following major factors:

• Proportion of deaths linked to water quality in trial versus beneficiary contexts. Some locations have a higher prevalence of waterborne disease than others. We use the prevalence of enteric infections in the times/locations of the trials that underlie our mortality reduction estimate, and the prevalence in beneficiary contexts, to derive an adjustment factor.115
• Intervention coverage/adherence. Some interventions achieve higher coverage than others. We use the coverage rates in the trials included in Kremer et al. 2022 (working paper) and the interventions of interest to derive an adjustment factor.116

As an example, after all adjustments we estimate that in-line chlorination in Kenya reduces all-cause mortality by 11% in children under five and 2% in people five and over.117 In contrast, an estimate obtained by multiplying the reduction in diarrhea risk caused by in-line chlorination by the GBD estimate of under-five deaths caused by enteric infections in Kenya is 2.9%, suggesting that our estimate is about 3.7 times as large as expected based on indirect estimation methods.118 This implies that for each enteric infection death averted in children under five, 2.7 deaths are averted from other causes.

For a second intervention, chlorine dispensers, we estimate that it reduces all-cause mortality by 6-11% in children under five and 1-4% in people five and over, with estimates varying by location.119

Because of the discordance between estimates generated by different methods, the statistical uncertainty of the RCT-based estimate, and limited information on the mechanisms that may account for a larger mortality effect, we remain very uncertain about the size of the mortality reduction effect. A more precise effect size estimate from meta-analyses of water quality trials or a more complete understanding of the mechanisms underlying the mortality reduction effect would be the most likely updates to change our effect size estimate.

Diarrhea morbidity reduction

Averting diarrhea morbidity accounts for a small share of benefits in our cost-effectiveness analyses of interventions that impact diarrhea risk. We estimate the benefit of diarrhea morbidity reduction by multiplying the morbidity burden of diarrhea in the beneficiary context, measured in years lost due to disability (YLDs), by the degree of reduction caused by the intervention. We then apply internal and external validity adjustments.120

Development effects

We estimate the development effects of water quality interventions using a formal method we developed for inferring development effects from the impacts of an intervention on measures of early-life health and growth.121 We then apply the same internal and external validity adjustments that we applied to our mortality estimates.122

Examples

Below, we briefly describe how we model mortality reduction in our cost-effectiveness analyses of interventions that impact waterborne pathogens.

In-line chlorination

In-line chlorination is a technology for automatically disinfecting water at shared water collection points in low-income settings with unsafe water. We use the findings of water quality RCTs, adjusted for internal and external validity, to estimate reductions in all-cause mortality in children under five and people five and older.123 Pickering et al. 2019, the only published RCT of in-line chlorination with a diarrhea outcome, forms the basis of our adherence adjustment,124 a key input of our external validity adjustment. Clasen et al. 2015, the most recent Cochrane meta-analysis of water treatment trials, forms the basis for the morbidity reduction estimate125 and the plausibility limit derived from it.126 After adjustments, we estimate that in-line chlorination in Kenya reduces all-cause mortality by 11% in children under five and 2% in people five and over.127 We further estimate that development effects and medical costs averted account for 35% and 19% of the total benefit of the intervention, respectively.128

Chlorine dispensers

Dispensers for Safe Water is a program that installs and maintains chlorine dispensers at shared water collection points with unsafe water in Kenya, Uganda, and Malawi.129 Since chlorine dispensers and in-line chlorination share the same mechanism of action, we model their cost-effectiveness using a nearly identical method.130 However, because the program is currently in operation, we use adherence data from surveys of the program itself rather than from RCTs.131 After adjustments, we estimate that chlorine dispensers in Kenya reduce all-cause mortality by 6% in children under five and 1% in people five and over.132 The smaller estimated impact on mortality compared to in-line chlorination is primarily due to the fact that chlorine dispensers do not impact chlorination rates as much as in-line chlorination does.133 We further estimate that development effects and medical costs averted account for 25% and 27% of the total benefit of the intervention, respectively.134

Questions for further investigation

The following are key sources of uncertainty that we will prioritize for future work.

• Effect size of chlorination on all-cause mortality. We believe we will need additional evidence to significantly reduce our uncertainty around this estimate. This could be in the form of empirical effect size estimates, or a more complete understanding of the mechanisms that link water quality to mortality.
• Adherence to chlorination interventions. Our cost-effectiveness analysis of chlorination interventions is very sensitive to adherence. Our estimate of adherence to in-line chlorination is particularly uncertain because we do not have direct estimates from a charity implementation context. We will update this value with direct estimates when available.
• Baseline water treatment rates. Our cost-effectiveness analysis of chlorination interventions is very sensitive to baseline water treatment rates, and our current estimates are quite uncertain.135
• Health expenditures. We have general estimates for the cost of medical treatment for diarrhea in low- and lower-middle-income countries, but we do not have direct estimates of medical treatment costs for the specific populations benefited by the interventions we are evaluating. Further desk research may allow us to refine these estimates somewhat, but direct estimates from implementing organizations would be ideal.
• Baseline consumption in beneficiary populations. Our estimate of the benefit of averting medical costs is sensitive to our estimate of baseline consumption. We currently use nation-level data from the World Bank, but these estimates are not specific to beneficiary populations. Further desk research may allow us to refine these estimates somewhat, but direct estimates from implementing organizations would be ideal.
• Development effects. We currently do not have strong direct evidence on the impact of water quality interventions on income later in life, and our estimate of this benefit is therefore highly uncertain. We believe we will need additional evidence to refine our estimate of this parameter, ideally direct evidence from long-term follow-up studies of water quality RCTs. We have not yet investigated the feasibility of such a study.
  

  Sources

  Document	Source
  AEA RCT Registry, "Water treatment and child survival: A meta-analysis," 2020	Source (archive)
  Alsan and Goldin 2019	Source
  Anderson, Charles, and Rees 2018 (working paper)	Source (archive)
  Anderson, Charles, and Rees 2019 (working paper)	Source (archive)
  Ashraf et al. 2013	Source
  Ashraf et al. 2020	Source (archive)
  Baird et al. 2016	Source
  Baral et al. 2020	Source (archive)
  Beach et al. 2016	Source
  Bleakley 2010	Source
  Boisson et al. 2013	Source (archive)
  Breiman et al. 2011	Source
  Burton et al. 2011	Source
  Caulfield et al. 2004	Source
  Centers for Disease Control and Prevention, "Free Chlorine Testing," 2020	Source (archive)
  Checkley et al. 2008	Source
  Cheung 2019	Source (archive)
  Chiller et al. 2006	Source (archive)
  Clasen et al. 2015	Source (archive)
  Clasen, "Report on independent review," 2021	Source
  Coles et al. 2005	Source
  Crump et al. 2005	Source
  Cutler and Miller 2005	Source
  Cutler and Miller 2020 (working paper)	Source (archive)
  Cutler et al. 2010	Source
  Dupas et al. 2020 (working paper)	Source
  Ferrie and Troesken 2005 (working paper)	Source
  Fischer Walker et al. 2013	Source (archive)
  Gallardo-Albarran 2020	Source (archive)
  Gamper-Rabindran, Khan, and Timmins 2010	Source
  GBD 2015 and Causes of Death Collaborators 2016	Source (archive)
  GiveWell, "Deworming might have huge impact, but might have close to zero impact," 2017	Source
  GiveWell, "Evidence Action's Dispensers for Safe Water program – December 2018 version," 2018	Source
  GiveWell, "Seasonal malaria chemoprevention," 2018	Source
  GiveWell, A method for estimating adult consumption effects of interventions for which we do not have direct evidence, 2020	Source
  GiveWell, Calculations for econometric studies of water quality and mortality, 2022	Source
  GiveWell, Could water quality have a larger impact on child mortality than implied by its impact on diarrhea incidence?, 2022	Source
  GiveWell, Econometric studies of the impacts of water quality improvements on mortality, 2022	Source
  GiveWell, ILC development effects, 2021	Source
  GiveWell, Modeling the plausibility of the mortality effect size of ILC, 2022	Source
  GiveWell, Mortality plausibility modeling, 2022	Source
  GiveWell, Water quality CEA (DSW and ILC), 2021	Source
  GiveWell's non-verbatim summary of a conversation with Dr. Jay Berkley, April 29, 2021	Source
  GiveWell's non-verbatim summary of a conversation with Dr. Robert Black, March 4, 2021	Source
  GiveWell's non-verbatim summary of a conversation with Dr. Stephen Luby, December 18, 2020	Source
  Global Burden of Disease, GBD Results tool, Deaths in countries with low SDI from enteric infections in children under 5, 2019	Source (archive)
  Global Burden of Disease, GBD Results tool, Deaths in Kenya from enteric infections in children under 5, 2019	Source (archive)
  Global Burden of Disease, GBD Results tool, Deaths in sub-Saharan Africa from enteric infections in children under 5, 2019	Source (archive)
  Gunther and Fink 2010 (working paper)	Source (archive)
  Haushofer et al. 2021 (working paper)	Source
  Higgs, "Summary of Evaluation of Kremer et al. (2021)," 2021	Source
  Humphrey et al. 2019	Source (archive)
  Imdad et al. 2017	Source
  Inoue and Ogasawara 2020	Source
  Kenya Demographic and Health Survey 2014	Source (archive)
  Kirby et al. 2019	Source (archive)
  Knutsson 2017	Source (archive)
  Kotloff et al. 2013	Source
  Kremer et al. 2011	Source
  Kremer et al. 2022 (working paper)	Source (archive)
  Kremer et al. 2022 (working paper), supplementary materials	Source (archive)
  Luby et al. 2006	Source
  Luby et al. 2018	Source (archive)
  Malawi Demographic and Health Survey 2015-16	Source (archive)
  Manohar et al. 2020	Source (archive)
  Newman et al. 2020	Source (archive)
  Nosek et al. 2018	Source
  Null et al. 2018	Source (archive)
  Ogasawara and Inoue 2016 (working paper)	Source (archive)
  Ogasawara and Matsushita 2018	Source
  Omore et al. 2013	Source (archive)
  Page et al. 2011	Source (archive)
  Peletz et al. 2012	Source (archive)
  Pickering et al. 2019	Source (archive)
  Quick et al. 2002	Source
  Reller et al. 2003	Source
  Sedgwick and MacNutt 1910	Source
  Semenza et al. 1998	Source
  Sterne et al. 2011	Source
  Swarthout et al. 2020	Source (archive)
  Uganda Demographic and Health Survey 2016	Source (archive)
  Watson 2006	Source
  Wikipedia, "Publication bias," 2021	Source (archive)
  World Bank, "Data: Kenya"	Source (archive)
  World Bank, "Data: Malawi"	Source (archive)
  World Bank, "Data: Uganda"	Source (archive)
  World Health Organization, "Diarrhoeal Disease," 2017	Source (archive)
  World Health Organization, "Fact Sheet 2.17: Inactivation of microbes by chlorine"	Source (archive)
  World Health Organization, "Measuring chlorine levels in water supplies," 2011	Source (archive)
  World Health Organization, Principles and practices of drinking-water chlorination, 2017	Source (archive)

  • 1

    "When chlorine is added to water, it is involved in three types of reaction. These affect the availability of chlorine and its efficiency as a disinfectant.
    First, substances such as manganese, iron, and hydrogen sulphide dissolved in the water will react irreversibly with chlorine. This reaction removes these substances, thereby improving water quality and taste. Chlorine, which reacts in this way is, however, lost and does not contribute to disinfection.
    Secondly, chlorine may react reversibly with organic matter and ammonia in water. The compounds formed are weak disinfectants. The products are referred to as combined chlorine or residual combined chlorine.
    Thirdly, the chlorine may react with and dissociate in water. The products are efficient disinfectants unless the water is alkaline and are referred to as free chlorine or free residual chlorine.
    The total amount of chlorine which will react both with compounds like iron and manganese and with organics and ammonia is referred to as the chlorine demand. The chlorine demand of different waters can vary widely.
    Chlorine demand is therefore the difference between the amount of chlorine added to the water (the chlorine dose) and the free chlorine detectable in the water."
    World Health Organization, "Fact Sheet 2.17: Inactivation of microbes by chlorine" Pgs. 117-18.

  • 2

    "When chlorine is added to water, it destroys the membrane of microorganisms and kills them." World Health Organization, "Measuring chlorine levels in water supplies," 2011, Pg. 1.

  • 3
    • "The rate of inactivation varies widely, but is more rapid when more chlorine is present in the water. . . . The efficiency of inactivation of microbes by chlorine is affected by a number of factors including pH, contact time and the reactions of chlorine with the water." World Health Organization, "Fact Sheet 2.17: Inactivation of microbes by chlorine", Pg. 117.
    • "When chlorine is added to water, some of the chlorine reacts first with inorganic and organic materials and metals in the water and is not available for disinfection (this is called the chlorine demand of the water). . . . Free chlorine, which is the chlorine that is left over and is available to inactivate disease-causing organisms . . . is a measure of the potability of the water. For example, if using completely clean water with no contaminants, the chlorine demand will be zero, and since there will be no inorganic or organic material present, no combined chlorine will be present. Thus, the free chlorine concentration will be equal to the concentration of chlorine initially added. In natural waters, especially surface water supplies such as rivers, organic material will exert a chlorine demand, and inorganic compounds like nitrates will form combined chlorine. Thus, the free chlorine concentration will be less than the concentration of chlorine initially added (Free chlorine = Total chlorine measurement – Combined chlorine measurement)." Centers for Disease Control and Prevention, "Free Chlorine Testing," 2020,
    • "The time taken for different types of microbes to be killed varies widely. In general, amoebic cysts are very resistant and require most exposure. Bacteria, including free-living Vibrio cholerae are rapidly inactivated by free chlorine under normal conditions. For example, a chlorine residual of 1mg/l after 30 minutes will kill schistosomiasis cercariae, while 2mg/l after 30 minutes may be required to kill amoebic cysts. Thus it is important to ensure that adequate contact time is available before water enters a distribution system or is collected for use." World Health Organization, "Fact Sheet 2.17: Inactivation of microbes by chlorine", Pg. 119.

  • 4

    "Nevertheless, microbes may be protected from chlorine if they are attached to or within particles in the water. For this reason, water to be chlorinated must be clear. It should always have a turbidity of less than five turbidity units and ideally less than one turbidity unit." World Health Organization, "Fact Sheet 2.17: Inactivation of microbes by chlorine", Pg. 117.

  • 5
    • "For practical purposes, cysts and eggs of protozoa and helminths [parasitic worms] may be considered resistant to disinfection with chlorine. They are killed at high doses or after prolonged contact times, but these are often impractical. Cysts and eggs of protozoa and helminths should be removed by filtration prior to disinfection or, in the case of groundwaters (springs and wells), excluded by source protection." World Health Organization, "Fact Sheet 2.17: Inactivation of microbes by chlorine" Pg. 117.
    • "The time taken for different types of microbes to be killed varies widely. In general, amoebic cysts are very resistant and require most exposure. Bacteria, including free-living Vibrio cholerae are rapidly inactivated by free chlorine under normal conditions. For example, a chlorine residual of 1mg/l after 30 minutes will kill schistosomiasis cercariae, while 2mg/l after 30 minutes may be required to kill amoebic cysts. Thus it is important to ensure that adequate contact time is available before water enters a distribution system or is collected for use." World Health Organization, "Fact Sheet 2.17: Inactivation of microbes by chlorine" Pg. 119.

  • 6

    We enrolled 9439 children with moderate-to-severe diarrhoea and 13 129 control children without diarrhoea. By analysing adjusted population attributable fractions, most attributable cases of moderate-to-severe diarrhoea were due to four pathogens: rotavirus, Cryptosporidium, enterotoxigenic Escherichia coli producing heat-stable toxin (ST-ETEC; with or without co-expression of heat-labile enterotoxin), and Shigella. . . . Pathogens associated with increased risk of case death were ST-ETEC (hazard ratio [HR] 1·9; 0·99–3·5) and typical enteropathogenic E coli (HR 2·6; 1·6–4·1) in infants aged 0–11 months, and Cryptosporidium (HR 2·3; 1·3–4·3) in toddlers aged 12–23 months.” Kotloff et al. 2013, abstract.

  • 7

    "Chlorine persists in water as residual chlorine after dosing and this helps to minimize the effects of re-contamination by killing or inactivating microbes which may enter the water supply after chlorination. It is important to take this into account when estimating requirements for chlorination in order to ensure that residual chlorine is always present.
    The level of chlorine residual required varies with the type of water supply and local conditions. In water supplies which are chlorinated there should always be a minimum of 0.5 mg/l residual chlorine after 30 minutes contact time in water. . . .
    Chlorine residual is readily and rapidly lost, particularly in open or regularly opened storage containers. Good household storage and handling practice are therefore vital to ensure good quality water in the home, and reliance should not be placed on residual disinfectant effect." World Health Organization, "Fact Sheet 2.17: Inactivation of microbes by chlorine" Pgs. 120, 122.

  • 8

    On average, distributing water disinfection products for use at the household level may reduce diarrhoea by around one quarter (Home chlorination products: RR 0.77, 95% CI 0.65 to 0.91; 14 trials, 30,746 participants, low quality evidence; flocculation and disinfection sachets: RR 0.69, 95% CI 0.58 to 0.82, four trials, 11,788 participants, moderate quality evidence).” Clasen et al. 2015, abstract.

  • 9

    15.31% in 2019. See GBD Results Tool data here.

  • 10

    World Health Organization, "Diarrhoeal Disease," 2017.

  • 11

    See GiveWell, Water quality CEA (ILC and DSW), 2021, "Mortality effect size" tab, "Pooled relative risk" row.

  • 12

    “Randomized controlled trials (RCTs) of water treatment are typically powered to detect effects on caregiver-reported diarrhea but not child mortality, as detecting mortality effects requires prohibitively large sample sizes. To increase statistical power, we conducted a meta-analysis that combined available RCT evidence on child mortality with new evidence on mortality obtained from authors of studies reporting only diarrhea outcomes.” Kremer et al. 2022 (working paper), abstract.

  • 13

    “The sample of 17 studies with mortality data is summarized in Table 1, based on information from manuscripts, aggregation of microdata, and correspondence with authors. Two of them were excluded from the main analysis due to contamination in the control group.” Kremer et al. 2022 (working paper), Pg. 6.

  • 14

    “Twelve examined water chlorination, two examined water filtration, and one examined spring protection.” Kremer et al. 2022 (working paper), Pg. 7.

  • 15
    • The following four studies involved interventions that were limited to chlorination: Boisson et al. 2013, Null et al. 2018, Haushofer et al. 2021 (working paper), Dupas et al. 2020 (working paper).
    • Table 1 of Kremer et al. 2022 lists study characteristics, including intervention type. Chlorination interventions often include additional elements such as flocculation or provision of safe storage vessels. Kremer et al. 2022 (working paper), table 1, Pg. 35-43.

  • 16

    Reller et al. 2003, Crump et al. 2005, Luby et al. 2006, Chiller et al. 2006

  • 17

    “Bleach is less effective if water is turbid or contaminated with chlorine-resistant pathogens… In response to these limitations and the persistent unmet need for water treatment, the Procter & Gamble Company (Cincinnati, OH) developed a new flocculant-disinfectant technology for treating water in the home that incorporates techniques used in municipal water purification. The product is a powder that is added to water; uses precipitation, coagulation, and flocculation to remove heavy metals, organic matter, and microorganisms; and leaves a free chlorine residual. After decanting, the treated water is microbiologically and chemically cleaner and looks clearer.” Reller et al. 2003, Pg. 411.

  • 18

    Semenza et al. 1998, Reller et al. 2003, Luby et al. 2006, Chiller et al. 2006, Luby et al. 2018, Quick et al. 2002.

  • 19

    Luby et al. 2006, Humphrey et al. 2019, Dupas et al. 2020 (working paper).

  • 20

    Peletz et al. 2012 and Kirby et al. 2019.

  • 21

    Kremer et al. 2011.

  • 22

    “In the full set of 15 studies we estimated an average reduction in odds of all-cause child mortality of 28% (Peto OR 0.72; CI 95% 0.55, 0.92) or 30% (Bayes OR 0.70; CrI 95% 0.49, 0.92), depending on the model (see Figure 2, Table S3).” Kremer et al. 2022 (working paper), pg 8.

  • 23

    Dr. Thomas Clasen is Rose Salamone Gangarosa Chair in Sanitation and Safe Water at the Rollins School of Public Health, Emory University. Dr. Megan Higgs is a statistics consultant and owner of Critical Inference. We have summarized their analyses in the section that follows. The conclusions described in this page are our own.

  • 24

    “The resulting meta-analysis of the full sample estimated a mean (cross-study) reduction in the odds of all-cause child mortality of about 30% (Peto odds ratio, OR, 0.72; 95% CI 0.55 to 0.92; Bayes OR 0.70; 95% CrI 0.49 to 0.92).” Kremer et al. 2022 (working paper), abstract.

  • 25

    Higgs has expressed concern about unquantified sources of uncertainty caused by different follow-up lengths in the underlying trials (below), and other sources, such as those associated with the particular subset of studies selected for analysis (the effect size estimates included meta-analysis may not accurately reflect the center or variance of some overall population of similar studies; personal communication from Megan Higgs), as well as sources of uncertainty associated with predicting effects in similar contexts at future times (as opposed to estimating the mean and uncertainty of the subset of studies used in the analysis; personal communication from Megan Higgs). We believe the latest version of the paper describing the meta-analysis addresses the issue of follow-up length adequately in sensitivity analysis, but the general concern of unaccounted-for sources of uncertainty remains.

    • “The uncertainty expressed in confidence/credible intervals is likely substantially under-stated given the heterogeneity among the included studies, especially in the time period over which children were monitored post-intervention” Higgs, "Summary of Evaluation of Kremer et al. (2021)," 2021, Pg. 2.
    • “We conducted an additional check of whether short studies may be unduly impacting the model. We started from 10 studies in the dataset that include one year or more of follow-up data and fit the Peto OR model. Then, we considered a hypothetical short study of 13 weeks (3 months), where the death risk is supposed to (crudely) approximate event rates in the dataset, 0.4%, and the size of the control arm is same as average size of control in the dataset, 1189. We assumed 1:1 randomisation and that the true OR is the same as in the model of 10 long studies (0.80). We then simulated a growing number of short studies, 1, 2, 3, …, 10, in each case conducting 100 replications. We examined the behavior of mean and 95% intervals. Predictably, the mean was not affected and the intervals shrank only slightly: in the model of only 10 long studies the 95% interval was 66.0% to 97.2%. In the model with 10 long and 10 simulated short studies the 95% interval was 66.9% to 95.6% (averaged over 100 replications). This suggests that including short studies has a negligible impact on precision of the estimate, unless they have high event rates.” Kremer et al. 2022 (working paper), supplementary materials, Pg. 6.

  • 26

    “Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings. However, ordinary biases in human reasoning, such as hindsight bias, make it hard to avoid this mistake. An effective solution is to define the research questions and analysis plan before observing the research outcomes—a process called preregistration.” Nosek et al. 2018.

  • 27

    AEA RCT Registry, "Water treatment and child survival: A meta-analysis," 2020.

  • 28
    • “The first step was a review of all studies identified by previous meta-analyses (8, 9) of studies from 1970 to February 2016 examining the impact of water quality interventions on diarrhea. Next, the search procedure and selection criteria followed by a previous meta-analysis (9) were replicated for the period from February 2016 to May 2020 to add more recent studies (last date search was conducted is April 20, 2020).” Kremer et al. 2022 (working paper), Pg. 4.
    • Kremer et al. also conducted a precursor of this meta-analysis using published mortality data only, indicating that the methods underlying the meta-analysis evolved over time and were initially not constrained by a pre-analysis plan.
    • “At the beginning of this study, five RCTs were identified which reported mortality outcomes as part of their analysis (23, 29, 30, 53, 19). The estimates from restricting the analysis to only the five studies which published mortality outcomes were similar in magnitude to that of the full sample though insignificant at the 95% confidence level (Peto OR 0.67; CI 95% 0.41, 1.11; Bayes OR 0.73; CrI 95% 0.28, 1.44).” Kremer et al. 2022 (working paper), Pg. 16.

  • 29

    “52 studies matched the inclusion criteria and we requested child mortality from the authors of each study. Twenty five authors reported that they did not collect mortality data or that the data was no longer available. The author of one study died and the authors of nine studies did not reply. Excluded studies are given in Table S2. The sample of 17 studies with mortality data is summarized in Table 1, based on information from manuscripts, aggregation of microdata, and correspondence with authors.” Kremer et al. 2022 (working paper), Pg. 6.

  • 30

    “Neither Begg’s or Andrews and Kasy’s tests provided evidence of publication bias (see Figure S3 and Table S8) (27).” Kremer et al. 2022 (working paper), Pg. 7.

  • 31

    “An adjusted estimate of the OR obtained using Andrews and Kasy method was OR = 0.83 (CI 95% 0.74, 0.92). Since the power of these tests may be limited when applied to our sample of 15 studies, we also consider a post hoc simulation-based exploration of small-study bias; see Discussion.” Kremer et al. 2022 (working paper), Pg. 7.

  • 32

    “The distribution of effect estimates of water treatment on diarrhea and compliance rates are similar across included and excluded data (see Fig. S4).” Kremer et al. 2022 (working paper), supplementary materials, Pg. 10.

  • 33
    • The interventions are summarized in Kremer et al. 2022 (working paper), table 1, Pg. 35-43.
    • "We randomly assigned 492 rural Guatemalan households to five different water treatment groups: flocculant-disinfectant, flocculant-disinfectant plus a customized vessel, bleach, bleach plus a vessel, and control." In our analysis, we only include the treatment groups in Reller et al. 2003 that used bleach or bleach plus a safe storage vessel. Bleach is a means of chlorination that is safe at the appropriate dose. Reller et al. 2003, abstract.
    • "The water intervention, which was modelled on a successful intervention from a previous trial, provided a 10 L vessel with a lid, tap, and regular supply of sodium dichloroisocyanurate tablets (Medentech, Wexford, Ireland) to the household of index children." Luby et al. 2018, Pg. e305.
    • "The WASH intervention included a play space to minimise geophagia and ingestion of chicken faeces by children in addition to conventional WASH interventions (sanitation, water treatment, handwashing, and hygienic preparation of food)." Humphrey et al. 2019, Pg. e133.

  • 34

    See GiveWell, Water quality CEA (DSW and ILC), 2021, “Internal validity adjustment” tab, "Adjustment for bundled treatments" row.

  • 35

    For example, Peletz et al. 2012 and Kirby et al. 2019 used water filtration, and Kremer et al. 2011 used spring protection.

  • 36

    “The scope of the review is confusing. The title—'Water quality and child survival' implies a vast range of possible interventions, including conventional municipal systems that combine various methods. In the Summary, this is narrowed to 'the effect of chlorination'. However, in at least three included papers (Peletz 2012, Kirby 2019, Kremer 2011) the intervention used filters or spring protection and no chlorine at all. Moreover, in at least four other papers (Reller 2003, Luby 2006, Crump 2005, and Chiller 2006), investigators tested a product that included both chlorine and a flocculant--an essential antimicrobial agent that not only enhances chlorination but is necessary to address the chlorine-resistant microbes such as Cryptosporidium that is a common cause of moderate to severe diarrhea (Kotloff et al. 2013). Even the type of chlorine is different, ranging from NaDCC tablets to calcium hypochlorite to different dilutions of sodium hypochlorite. It would be misleading to infer an “effect of chlorination” by treating this disparate group of studies as a homogeneous intervention.” Clasen, "Report on independent review," 2021, Pg. 2.

  • 37

    “Although mortality is presented here as an objective outcome, there is considerable heterogeneity in the way it is collected especially in studies, like these, where it is not a primary outcome. Curiously, while Table 1 describes extracted study data such as contamination levels and diarrhea rates that are at best indirectly relevant to the aims of the review, it does not describe how mortality was ascertained. Best practices would require reviewing death certificates or hospital records or conducting a verbal autopsy. However, it’s unlikely that any of that was done in the included studies given the secondary need for these data and the expense of doing so. More likely it was reported by the field workers based on the householder reports. This is another source of measurement bias and limitation that the review should acknowledge.” Clasen, "Report on independent review," 2021, Pg. 3.

  • 38

    Our primary rationale for this criterion was initially that small studies may have been introducing excess unquantified uncertainty. However, we believe that Kremer et al. have addressed this concern adequately in the current version of the manuscript.

    • “We conducted an additional check of whether short studies may be unduly impacting the model. We started from 10 studies in the dataset that include one year or more of follow-up data and fit the Peto OR model. Then, we considered a hypothetical short study of 13 weeks (3 months), where the death risk is supposed to (crudely) approximate event rates in the dataset, 0.4%, and the size of the control arm is same as average size of control in the dataset, 1189. We assumed 1:1 randomisation and that the true OR is the same as in the model of 10 long studies (0.80). We then simulated a growing number of short studies, 1, 2, 3, …, 10, in each case conducting 100 replications. We examined the behavior of mean and 95% intervals. Predictably, the mean was not affected and the intervals shrank only slightly: in the model of only 10 long studies the 95% interval was 66.0% to 97.2%. In the model with 10 long and 10 simulated short studies the 95% interval was 66.9% to 95.6% (averaged over 100 replications). This suggests that including short studies has a negligible impact on precision of the estimate, unless they have high event rates.” Kremer et al. 2022 (working paper), supplementary materials, Pg. 6.

  • 39

    This trial estimates that the intervention reduced all-cause mortality in children under five by 1.5 percentage points (95% confidence interval, 0.3 to 2.5%), which represents a 63% relative reduction in mortality.

    • "Comparison of post-intervention mortality risks between treatment and control areas using an ANCOVA analysis (i.e. controlling for baseline mortality risks) suggests that the communitywide provision of dilute chlorine solution through dispensers and WaterGuard reduced all-cause under-5 mortality by 1.41 p.p. (95% CI: 0.27 p.p., 2.55 p.p.), a 63% reduction relative to control, four years after the start of the intervention." Haushofer et al. 2021 (working paper), Pg. 2.

    If we assume the same confidence range around the relative reduction figure, this implies a confidence interval of 13 to 105% reduction in mortality. Table 3 reports several estimates of the between-group difference in adherence to the intervention. For these calculations, we assume the difference was 18%, which is on the upper end of the estimates provided. This implies (assuming no spillover effects) that in the subset of households that chlorinated their water as a result of the intervention, all-cause mortality in children under five declined by 72% to 583%, with a central estimate of 350%. We believe that the entire range of this 95% confidence interval is implausible, because even its lowest value implies that most causes of death are eliminated among children who drink clean water, and it is also a much larger effect size than reported by several large RCTs that report mortality outcomes. Very large spillover effects would have to be invoked to justify an effect size this large. We are not aware that water quality interventions cause large spillover effects, and this has not been observed in the morbidity or mortality outcomes of other water quality trials.

  • 40

    See our calculations here.

  • 41

    “These biases may cause funnel plot asymmetry if statistically significant results suggesting a beneficial effect are more likely to be published than non-significant results. Such asymmetry may be exaggerated if there is a further tendency for smaller studies to be more prone to selective suppression of results than larger studies. This is often assumed to be the case for randomised trials. For instance, it is probably more difficult to make a large study disappear without trace, while a small study can easily be lost in a file drawer.” Sterne et al. 2011, Pg. 2.

  • 42

    See the relative risk for each trial in GiveWell, Water quality CEA (ILC and DSW), 2021, “Mortality effect size” tab, "Mortality RR" column.

  • 43
    • According to the most recent Cochrane meta-analysis of water quality trials, chlorination interventions reduce the risk of diarrhea by 23% (95% confidence interval, 9% to 35%). The evidence is described as “low quality.” However, flocculation and disinfection also reduce diarrhea rates, and this evidence is described as “moderate quality.” These disinfection methods have substantial, but not complete, overlap in mechanism of action with chlorination. In conjunction with the well-demonstrated ability of chlorine to inactivate viral and bacterial pathogens in water, we consider these findings overall to support the effectiveness of chlorination for reducing diarrhea risk.
    • “On average, distributing water disinfection products for use at the household level may reduce diarrhoea by around one quarter (Home chlorination products: RR 0.77, 95% CI 0.65 to 0.91; 14 trials, 30,746 participants, low quality evidence; flocculation and disinfection sachets: RR 0.69, 95% CI 0.58 to 0.82, four trials, 11,788 participants, moderate quality evidence). However, there was substantial heterogeneity in the size of the effect estimates between individual studies. Point-of-use filtration systems probably reduce diarrhoea by around a half (RR 0.48, 95% CI 0.38 to 0.59, 18 trials, 15,582 participants, moderate quality evidence).” Clasen et al. 2015, abstract.

  • 44

    GBD data suggest that enteric infections account for 15% of all mortality in children under five in sub-Saharan Africa. Global Burden of Disease, GBD Results tool, Deaths in sub-Saharan Africa from enteric infections in children under 5, 2019.

  • 45

    For example, we have high uncertainty about the impact of deworming on adult income, and we do not have direct evidence on the impact of seasonal malaria chemoprevention on malaria mortality. See this blog post and our report on seasonal malaria chemoprevention.

  • 46

    “On average, distributing water disinfection products for use at the household level may reduce diarrhoea by around one quarter (Home chlorination products: RR 0.77, 95% CI 0.65 to 0.91; 14 trials, 30,746 participants, low quality evidence; flocculation and disinfection sachets: RR 0.69, 95% CI 0.58 to 0.82, four trials, 11,788 participants, moderate quality evidence).” Clasen et al. 2015, abstract.

  • 47

    14.49% in 2019. See Global Burden of Disease, GBD Results tool, Deaths in countries with low SDI from enteric infections in children under 5, 2019.

  • 48

    0.145 X 0.23 = 0.033. 0.26 / 0.033 = 7.9. 0.14 / 0.033 = 4.2

  • 49

    GiveWell, Water quality CEA (ILC and DSW), 2021, "Internal validity adjustments" tab.

  • 50

    “The GBD cause list relies on categorical attribution of deaths to a single underlying cause in accordance with the principles outlined in the ICD. The core principle of the ICD is to assign each death to only the underlying cause of death; ie, the cause that initiated the series of events leading to death.” GBD 2015 and Causes of Death Collaborators 2016, Pg. 1470.

  • 51

    For example, vitamin A supplementation reduces the risk of dying from diarrhea in low-income settings with a high prevalence of vitamin A insufficiency. This implies that these deaths require the presence of both vitamin A insufficiency and a diarrheal pathogen, and averting either one would avert death. Another example is that people often die of “secondary infections” acquired after having been weakened by a primary infection.

    • “Nine trials reported mortality due to diarrhoea and showed a 12% overall reduction for VAS (RR 0.88, 95% CI 0.79 to 0.98; 1,098,538 participants; high-quality evidence).” Imdad et al. 2017, abstract.
    • “Where data was provided for COVID-19 patients, a mortality rate of 15.2% due to secondary bacterial infections was observed for patients with pneumonia (41 of 268). Most clinicians treated patients with SARS-CoV-2 infections with prophylactic antibiotics (63.7%, n = 1,901), compared to 73.5% (n = 3,072) in all clinical reports of viral pneumonia included in this review. For all cases of viral pneumonia, a mortality rate of 10.9% due to secondary infections was observed (53 of 482).” Manohar et al. 2020, abstract.

  • 52

    “The odds of stunting at age 24 months increased multiplicatively with each diarrhoeal episode and with each day of diarrhoea before 24 months (all P &lt 0.001). The adjusted odds of stunting increased by 1.13 for every five episodes (95% CI 1.07-1.19), and by 1.16 for every 5% unit increase in longitudinal prevalence (95% CI 1.07-1.25). In this assembled sample of 24-month-old children, the proportion of stunting attributed to >or=5 diarrhoeal episodes before 24 months was 25% (95% CI 8-38%) and that attributed to being ill with diarrhoea for >or=2% of the time before 24 months was 18% (95% CI 1-31%). These observations are consistent with the hypothesis that a higher cumulative burden of diarrhoea increases the risk of stunting.” Checkley et al. 2008, abstract.

  • 53

    Caulfield et al. 2004 estimates that 45% of measles mortality and 57% of malaria mortality is attributable to undernutrition. Caulfield et al. 2004, table 4, Pg. 196.

  • 54
    • Five out of six water quality RCTs in the Kremer et al. 2022 (working paper) meta-analysis that report growth outcomes did not identify a growth effect, while one (Kremer et al. 2011) reports that BMI was increased (the latter finding was only significant at the p < 0.10 level, and the effect on body weight was not significant). However, of the five trials that did not identify a growth effect, four also did not impact diarrhea risk.
    • Kremer et al. 2011 reports that spring protection reduces diarrhea risk and has a marginally significant impact on BMI growth. “Spring infrastructure investments reduce fecal contamination by 66%, but household water quality improves less, due to recontamination. Child diarrhea falls by one quarter.” Kremer et al. 2011, abstract. “There are no statistically significant impacts on child weight but impacts are positive and marginally significant for body mass index (BMI) in the three follow-up surveys (Table IV , regressions 7–10).” Kremer et al. 2011, Pg. 171.
    • Peletz et al. 2012 reports that provision of water filters and a safe water storage container substantially reduced diarrhea risk in children younger than two over 12 months, but had no detectable impact on weight-for-age z-score. However, diarrhea itself was associated with lower weight-for-age z-score. “The intervention was associated with reductions in the longitudinal prevalence of reported diarrhea of 53% among children <2 years (LPR = 0.47, 95% CI: 0.30–0.73, p = 0.001) and 54% among all household members (LPR = 0.46, 95% CI: 0.30–0.70, p < 0.001). While reduced WAZ was associated with reported diarrhea (-0.26; 95% CI: -0.37 to -0.14, p < 0.001), there was no difference in WAZ between intervention and control groups.” Peletz et al. 2012, abstract.
    • Boisson et al. 2013 reports that provision of household chlorination products did not impact diarrhea risk or weight-for-age z-score in children under five over 12 months. “The longitudinal prevalence of diarrhoea among intervention children was 1.69% compared to 1.74% among controls. After adjusting for clustering within household, the prevalence ratio of the intervention to control was 0.95 (95% CI 0.79–1.13). The mean WAZ was similar among children of the intervention and control groups (-1.586 versus -1.589, respectively).” Boisson et al. 2013, abstract.
    • Null et al. 2018 reports that a water chlorination intervention did not impact diarrhea risk or length-for-age z-score in children over a two-year period. Adherence was relatively low by year two. “Adherence indicators for sanitation, handwashing, and nutrition were more than 70% at year 1, handwashing fell to less than 25% at year 2, and for water was less than 45% at year 1 and less than 25% at year 2; combined groups were comparable to single groups. None of the interventions reduced diarrhoea prevalence compared with the active control. Compared with active control (length-for-age Z-score –1·54) children in nutrition and combined water, sanitation, handwashing, and nutrition were taller by year 2 (mean difference 0·13 [95% CI 0·01–0·25] in the nutrition group; 0·16 [0·05–0·27] in the combined water, sanitation, handwashing, and nutrition group). The individual water, sanitation, and handwashing groups, and combined water, sanitation, and handwashing group had no effect on linear growth.” Null et al. 2018.
    • Luby et al. 2018 reports that a water chlorination intervention did not impact diarrhea risk or length-for-age z-score in children over a two-year period. “Compared with a prevalence of 5·7% (200 of 3517 child weeks) in the control group, 7-day diarrhoea prevalence was lower among index children and children under 3 years at enrolment who received sanitation (61 [3·5%] of 1760; prevalence ratio 0·61, 95% CI 0·46–0·81), handwashing (62 [3·5%] of 1795; 0·60, 0·45–0·80), combined water, sanitation, and handwashing (74 [3·9%] of 1902; 0·69, 0·53–0·90), nutrition (62 [3·5%] of 1766; 0·64, 0·49–0·85), and combined water, sanitation, handwashing, and nutrition (66 [3·5%] of 1861; 0·62, 0·47–0·81); diarrhoea prevalence was not significantly lower in children receiving water treatment (90 [4·9%] of 1824; 0·89, 0·70–1·13). Compared with control (mean length-for-age Z score –1·79), children were taller by year 2 in the nutrition group (mean difference 0·25 [95% CI 0·15–0·36]) and in the combined water, sanitation, handwashing, and nutrition group (0·13 [0·02–0·24]). The individual water, sanitation, and handwashing groups, and combined water, sanitation, and handwashing group had no effect on linear growth.” Luby et al. 2018, abstract.
    • Humphrey et al. 2019 reports that a chlorination and hygiene intervention did not impact diarrhea risk or length-for-age z-score in children over an 18-month period. “The WASH intervention had no effect on either primary outcome [one of which was length-for-age z-score]. Neither intervention reduced the prevalence of diarrhoea at 12 or 18 months.” Humphrey et al. 2019, abstract.

  • 55

    See our discussion of this evidence here.

  • 56

    See our discussion of this evidence here.

  • 57
    • “Children had an increased risk of pneumonia for every additional day of diarrhoea in the 2 weeks (1.06, 95% CI: 1.03–1.09) and 4 weeks (1.04, 95% CI: 1.03–1.06) prior to the week of pneumonia onset. The attributable risk of pneumonia cases due to recent exposure to diarrhoea was 6%.” Ashraf et al. 2013, abstract.
      
    • Fischer Walker et al. 2013, abstract.
    • “Logistic regression models were used to evaluate associations of CAAP [community-acquired alveolar pneumonia] with nutritional status and recent diarrhea experience. Anemia (adjusted odds ratio (AOR) = 3.32, 95% confidence interval (CI): 2.24, 4.94; p < 0.001), low birth weight (AOR = 2.16, 95% CI: 1.32, 3.54; p = 0.002), stunting (AOR = 2.22, 95% CI: 1.31, 3.78; p = 0.004), serum retinol concentration (AOR = 1.03 per microg/dl, 95% CI: 1.02, 1.05; p < 0.001), and having >or=1 diarrhea episodes within 31 days prior to enrollment (AOR = 2.30, 95% CI: 1.26, 4.19; p = 0.007) were identified as risk factors for CAAP.” Coles et al. 2005, abstract.

  • 58

    We identified four water quality trials with respiratory illness outcomes. Of those, two report a reduction in respiratory illness, while two do not. Of the two that report no change in the risk of respiratory illness, one (WASH Benefits Kenya; Swarthout et al. 2020 and Null et al. 2018) also reported no change in the risk of diarrhea, suggesting that the intervention may not have been very effective. Of the two that report a benefit for respiratory illness, one (Kirby et al. 2019) also involved a clean cookstove intervention. Although it reports no significant impact on fine particulate exposure, we still regard this as a possible confounding factor. Ashraf et al. 2020 and Luby et al. 2018 are from the WASH Benefits Bangladesh trial; Swarthout et al. 2020 and Null et al. 2018 are from the WASH Benefits Kenya trial.

    • “Compared with children in the control group ([acute respiratory infection] prevalence, P: 8.9%), caregivers of index children reported significantly lower [acute respiratory infection] in the water (P: 6.3%, prevalence ratio (PR): 0.71; 95% CI: 0.53, 0.96), sanitation (P: 6.4%, PR: 0.75, 95% CI: 0.58, 0.96), handwashing (P: 6.4%, PR: 0.68, 95% CI: 0.50, 0.93), and the combined WSH+N arms (P: 5.9%, PR: 0.67, 95% CI: 0.50, 0.90). Those in the nutrition (P: 7.4%, PR: 0.84, 95% CI: 0.63, 1.10) or the WSH arm (P: 8.9%, PR: 0.99, 95% CI: 0.76, 1.28) reported similar [acute respiratory infection] prevalence compared with control children.” Ashraf et al. 2020, abstract.
    • “Compared with a prevalence of 5·7% (200 of 3517 child weeks) in the control group, 7-day diarrhoea prevalence was lower among index children and children under 3 years at enrolment who received sanitation (61 [3·5%] of 1760; prevalence ratio 0·61, 95% CI 0·46–0·81), handwashing (62 [3·5%] of 1795; 0·60, 0·45–0·80), combined water, sanitation, and handwashing (74 [3·9%] of 1902; 0·69, 0·53–0·90), nutrition (62 [3·5%] of 1766; 0·64, 0·49–0·85), and combined water, sanitation, handwashing, and nutrition (66 [3·5%] of 1861; 0·62, 0·47–0·81); diarrhoea prevalence was not significantly lower in children receiving water treatment (90 [4·9%] of 1824; 0·89, 0·70–1·13).” Luby et al. 2018, abstract.
    • “Water, sanitation, and handwashing interventions with behavior change messaging did not reduce ARIs.” Swarthout et al. 2020, abstract. “None of the interventions reduced diarrhoea prevalence compared with the active control.” Null et al. 2018, abstract.
    • “Respiratory illness and negative control outcomes were very similar between groups, and there were no significant differences between groups in weight-for-age, height-for-age Z scores, or blood in stool (table 2).” Pickering et al. 2019, Pg. e1253. https://doi.org/10.1016/S2214-109X(19)30315-8
    • “Children in the treatment group had less WHO-defined diarrhoea than did children in the control group (control 216 [10·0%] of 2154; treatment 156 [7·5%] of 2073; prevalence ratio 0·77, 95% CI 0·65–0·91).” Pickering et al. 2019, abstract.
    • “The intervention reduced the prevalence of reported child diarrhea by 29% (prevalence ratio [PR] 0.71, 95% confidence interval [CI] 0.59–0.87, p = 0.001) and reported child [acute respiratory infection] by 25% (PR 0.75, 95% CI 0.60–0.93, p = 0.009)... The intervention reduced the prevalence of households with detectable fecal contamination in drinking water samples by 38% (PR 0.62, 95% CI 0.57–0.68, p < 0.0001) but had no significant impact on 48-hour personal exposure to log-transformed fine particulate matter (PM2.5) concentrations among cooks (β = −0.089, p = 0.486) or children (β = −0.228, p = 0.127).”
      Kirby et al. 2019, abstract.

  • 59

    See our discussion of this evidence here.

  • 60

    GiveWell's non-verbatim summary of a conversation with Dr. Stephen Luby, December 18, 2020.

  • 61

    GiveWell's non-verbatim summary of a conversation with Dr. Robert Black, March 4, 2021.

  • 62

    GiveWell's non-verbatim summary of a conversation with Dr. Jay Berkley, April 29, 2021.

  • 63
    • “Changes in diarrhea morbidity are unlikely to be a good proxy for changes in all-cause mortality. All-cause mortality reductions from water quality interventions may exceed the level predicted from the reduction in diarrhea morbidity.”
    • “Diarrheal deaths tend to occur in children with additional risk factors, such as poverty, malnutrition, and respiratory disease. Some research suggests that diarrhea can increase the risk of contracting respiratory diseases. Therefore, it's possible that preventing diarrheal disease could also reduce deaths attributed to other causes, such that water quality interventions would have a larger effect on all-cause mortality than predicted from their impact on diarrhea morbidity.”
    • “Water contaminated by fecal matter can also contain pathogens like Salmonella typhi and Salmonella paratyphi, both of which can cause typhoid fever, as well as the Hepatitis A and Hepatitis E viruses. Ingesting these pathogens can lead to deadly disease without causing diarrhea, such that water quality improvements could have a disproportionate effect on all-cause mortality by preventing deaths from non-diarrheal waterborne diseases in addition to diarrheal deaths.”
    • “It's possible that water quality interventions disproportionately reduce severe cases of diarrheal disease. If this were the case, a relatively modest reduction in overall diarrhea morbidity could be associated with a much larger reduction in diarrhea mortality, because severe cases are more likely to be fatal than mild ones.” GiveWell's non-verbatim summary of a conversation with Dr. Stephen Luby, December 18, 2020, Pg. 2.

  • 64
    • “All-cause mortality reductions from water quality interventions may exceed the level predicted from the reduction in diarrhea morbidity. This is what some econometric studies of historical water quality improvements have reported. These studies have also reported reductions in mortality from non-waterborne causes.” GiveWell's non-verbatim summary of a conversation with Dr. Stephen Luby, December 18, 2020, Pg. 2.
    • “It seems plausible that water quality interventions have an effect on overall mortality greater than that explained by their impact on diarrhea alone, but Dr. Black is skeptical that it's a very high magnitude greater.” GiveWell's non-verbatim summary of a conversation with Dr. Robert Black, March 4, 2021, Pg. 1.
    • “Professor Berkley thinks it plausible that about half of the effect from water quality interventions on all-cause mortality could be accounted for by a direct effect on diarrhea. However, he would also not be surprised if effects on diarrhea were responsible for less than half of this impact. The pathway may depend in part on the ages of the children studied and causes of death.” GiveWell's non-verbatim summary of a conversation with Dr. Jay Berkley, April 29, 2021.

  • 65
    • For our estimates of mortality reduction, see GiveWell, Water quality CEA (ILC and DSW), 2021, "DSW" tab, "Percent reduction in under-5 all-cause mortality, final estimate" row. Dr. Luby is a co-author on the Kremer et al. 2022 (working paper) meta-analysis, whose main finding is that water quality interventions reduce all-cause mortality in children under five by 28% or 30%, depending on the specification. This is a larger effect size than implied by our cost-effectiveness analysis, so we infer that he finds the effect size in our cost-effectiveness analysis to be plausible.
    • “The resulting meta-analysis of the full sample estimated a mean (cross-study) reduction in the odds of all-cause child mortality of about 30% (Peto odds ratio, OR, 0.72; 95% CI 0.55 to 0.92; Bayes OR 0.70; 95% CrI 0.49 to 0.92).” Kremer et al. 2022 (working paper), abstract.
    • See the effect size implied by our cost-effectiveness analysis of in-line chlorination and chlorine dispensers (here and here).
    • “Professor Berkley believes an 18% reduction in all-cause mortality from that baseline, in a trial setting, is a plausible effect size, in part because it includes perinatal deaths.” GiveWell's non-verbatim summary of a conversation with Dr. Jay Berkley, April 29, 2021, Pg. 6.
    • “In settings where people use a common water source, which may be relatively high-mortality settings, Dr. Black would expect chlorination to lead to, at most, a 10-15% reduction in overall mortality for children 1-59 months old (assuming chlorination eliminated essentially all diarrhea deaths related to water quality and increased risks of death due to malnutrition).” GiveWell's non-verbatim summary of a conversation with Dr. Robert Black, March 4, 2021, Pg. 3.

  • 66
    • "In this article, we respond to these problems by examining the introduction of a major class of discrete public health interventions-clean water technologies-in large American cities in the early twentieth century." Cutler and Miller 2005, Pg. 2.
    • "Following the demonstration of chlorine's use for disinfection in 1908, most major cities began water chlorination within the next decade." Cutler and Miller 2005, Pg. 5
    • “This article studies the impact of waterworks and sewerage on mortality in German cities during the period 1877–1913.” Gallardo-Albarran 2020, abstract.
    • “The construction of modern water-supply systems, wherein water was purified at a filtration plant and delivered via steel pipes, in Japanese cities gathered pace during the interwar period (Appendix A.1). In 1918, the national government committed to subsidizing the construction of systems at the town and village levels. The scope of the subsidy was expanded in 1921 to include not only city but also town and village operations. Many modern water supply systems were installed between the early 1920s until the weakening of Japan's wartime regime in 1940. It is reported that the systems had been installed in 55 locations in 1921. By 1940, this figure had increased to 345 locations. At the same time, the maximum average amount of portable water per person per day was 0.023 cubic meters in 1921, increasing to 0.061 cubic meters in 1940. In summary, the introduction of systems began in earnest from the early 1920s, and this was followed by a gradual increase in water consumption.” Ogasawara and Matsushita 2018, Pg. 199.

  • 67

    For example, in both Hamburg, Germany, and Lawrence, Massachusetts, municipal water filtration was implemented in a single year (1893).
    “Shortly after the introduction of a filtered and purified water supply into Lawrence, Massachusetts, in September, 1893, it was observed by Mr. Hiram F. Mills, C.E., a member of the State Board of Health of Massachusetts, then a resident of the city of Lawrence and chief engineer of the company controlling the water power of that city, that a marked decrease in the general death-rate of the city, and not merely in the death-rate from typhoid fever, was taking place. A few months earlier (May, 1893) filtration of the public water-supply had likewise been established for the city of Hamburg, Germany, and there also it was observed by Dr. J. J. Reincke, health officer of that city, that the general death-rate was declining more rapidly than could possibly be accounted for by the deaths from typhoid fever alone.” Sedgwick and MacNutt 1910, Pg. 491.

  • 68

    We began by performing a backward literature search, drawing references from papers cited in Kremer et al. 2022 (working paper). This identified a number of papers not cited in Kremer et al. 2022 (working paper) itself. We then performed a literature search in the EconPapers database on October 8, 2020, using the following search terms: ("water quality" OR "piped water" OR "clean water") AND (mortality OR death*). This search returned 168 records and appeared to have good sensitivity and specificity. We reviewed all records for relevance and identified a number of studies not identified by the backward literature search. We also performed a search in EconPapers on October 8, 2020, using the search term “Mills-Reincke”; this search returned three records, including one not identified by the other search methods.

  • 69

    “Shortly after the introduction of a filtered and purified water supply into Lawrence, Massachusetts, in September, 1893, it was observed by Mr. Hiram F. Mills, C.E., a member of the State Board of Health of Massachusetts, then a resident of the city of Lawrence and chief engineer of the company controlling the water power of that city, that a marked decrease in the general death-rate of the city, and not merely in the death-rate from typhoid fever, was taking place. A few months earlier (May, 1893) filtration of the public water-supply had likewise been established for the city of Hamburg, Germany, and there also it was observed by Dr. J. J. Reincke, health officer of that city, that the general death-rate was declining more rapidly than could possibly be accounted for by the deaths from typhoid fever alone. To this important discovery, made thus independently by Mr. Mills in Lawrence and Dr. Reincke in Hamburg, we have, because of its fundamental and far-reaching significance, applied the name of The Mills-Reincke Phenomenon.” Sedgwick and MacNutt 1910, Pgs. 491-492.

  • 70

    “Although water filtration is associated with a (statistically insignificant) 1-2 percent decrease in total mortality and an 11-12 percent decrease in infant mortality, these estimates are considerably smaller than those found by previous researchers, including Cutler and Miller (2005), the authors of the most influential study in this literature.” Anderson, Charles, and Rees 2018 (working paper), Pg. 2.

  • 71
    • Cutler and Miller (2005) report that water quality improvements in US cities substantially reduce population-wide, child, and infant all-cause mortality. However, its findings have been credibly challenged by an independent study and reanalysis, Anderson, Charles, and Rees 2018 (working paper). After considering the arguments of both parties, we believe the original analysis by Cutler and Miller 2005 substantially overestimates the mortality impact of water quality improvements in this context.
    • Anderson, Charles, and Rees (2018) point out a number of features of the Cutler and Miller 2005 analysis that it argues are erroneous or suboptimal. In a reply to this reanalysis, Cutler and Miller (2020, originally 2018) concede several points, particularly that correcting errors produces an effect size of water filtration on infant mortality much smaller than their original estimate. However, Cutler and Miller (2020, originally 2018) argue that the effect size of water filtration on population-wide all-cause mortality is similar to their original estimate. In a reply to this,
    • Anderson, Charles, and Rees (2019) point out, among other things, that the method used by Cutler and Miller to calculate mortality rates yields implausible estimates. Using a more accurate method reduces the effect size on population-wide all-cause mortality by approximately half, even when other details of the specification remain those preferred by Cutler and Miller (and after correcting mutually-agreed-upon errors). For this reason, we take the original Cutler and Miller 2005 estimate to imply a reduction in population-wide all-cause mortality of 8 percent (for filtration, 4% for chlorination) rather than the original effect size of 13 percent.
    • “On average, filtration and chlorination together reduced typhoid fever mortality by 25%, total mortality by 13%, infant mortality by 46%, and child mortality by 50%.” Cutler and Miller 2005, Pgs. 11-12l; also see table 5, Pg. 13.
    • “Using their original data and specification, we find that the estimated effect of filtration on total mortality shrinks by half, from -16 log points to -8 log points, when we correct a handful of transcription errors and use U.S. Bureau of the Census population estimates to consistently calculate the total mortality rate for the entire period under study, 1900-1936. Correcting a series of transcription errors in their infant mortality counts (79 of 410 infant mortality counts were incorrectly transcribed) reduces the estimated effect of filtration on infant mortality by two-thirds, from -43 log points to -13 log points.” Anderson, Charles, and Rees 2018 (working paper), Pg. 27.
    • The reanalysis of Cutler and Miller 2005 is on pages 21-24 of Anderson, Charles, and Rees 2018 (working paper), and is too detailed to summarize here. Anderson, Charles, and Rees 2018 (working paper), Pg. 21-24.
    • “We are very grateful to ACR for the careful re-analysis of our earlier paper and deeply appreciate both the constructive nature of our exchanges with them and the identification of several mistakes in our original paper.” Cutler and Miller 2020 (working paper), Pg. 10.
    • “However, effects on infant mortality rates appear more sensitive to these adjustments and markedly smaller than in our original analysis – although we believe the evidence still supports significant, and quantitatively meaningful, effects of clean water on infant mortality as well.” Cutler and Miller 2020 (working paper), Pg. 10-11.
    • “Although the CM-Comment does not present new estimates after correcting the coding errors identified by ACR, we read their language (specifically, that the point estimate for infant mortality is “markedly reduced”) as agreeing with the point estimate reported in ACR. We, however, disagree adamantly with CM’s contention that this reduction in the point estimate to nearly one-quarter of its original size should not be seen as a departure from the consensus view in the literature.” Anderson, Charles, and Rees 2019 (working paper), Pg. 4.
    • “After correcting unambiguous data mistakes, our revised estimates suggest that municipal water disinfection (filtration) explains 38% of the total mortality rate decline in our sample cities and years – a result not very different from our 43% original estimate.” Cutler and Miller 2020 (working paper), abstract.
      Anderson, Charles, and Rees (2019) argue on pages 7-9 that the method used by Cutler and Miller to calculate mortality rates yields implausible estimates, and they provide charts convincingly illustrating this point on pages 13-19. “For some cities, the population estimates used by CM are essentially equivalent to the linearly interpolated estimates (e.g., Chicago and Philadelphia). For other cities, however, the estimates used by CM are clearly inaccurate (e.g., Baltimore, Cincinnati and Detroit). For instance, CM’s population estimate for Detroit is slightly over 600,000 in 1917, while the linearly interpolated estimate would put Detroit’s population at over 800,000 in 1917. To take another example, CM’s population estimate for Jersey City in 1917 is over 310,000, yet Jersey City’s 1920 census population was less than 300,000.” Anderson, Charles, and Rees 2019 (working paper), Pg. 7-9 and 13-19.
    • Table 3, column 4 of Cutler and Miller 2020 provides their estimate of the all-cause mortality impact of water chlorination and filtration after applying the more accurate method of estimating mortality rates. The estimate is -0.04 log points for chlorination and -0.08 log points for filtration, which equals an approximately 4 and 8 percent reduction, respectively. Cutler and Miller 2020 (working paper), table 3, Pg. 19.

  • 72

    See this document for tables containing the studies we identified.

  • 73
    • For studies that reported mortality reduction per unit coverage of water quality improvement, or that reported estimates representing low levels of coverage, we normalized the figures to represent 75 percent coverage. For Cutler and Miller 2005, we used our preferred effect size estimate derived from an exchange between the authors and Anderson, Charles, and Rees, who critiqued the original analysis. Please refer to the footnote above discussing this issue.
    • See this document for tables containing the studies we identified. We compiled estimates of the mortality impact of water quality interventions in this spreadsheet.

  • 74
    • For studies that reported mortality reduction per unit coverage of water quality improvement, or that reported estimates representing low levels of coverage, we normalized the figures to represent 75 percent coverage. We do not emphasize values from Gamper-Rabindran, Khan, and Timmins 2010 because extrapolation to 75 percent coverage results in an implausibly large effect size estimate. See the calculations and discussion for Gamper-Rabindran, Khan, and Timmins 2010 here.
    • See this document for tables containing the studies we identified. We compiled estimates of the mortality impact of water quality interventions in this spreadsheet.

  • 75
    • Anderson, Charles, and Rees 2018 (working paper), Cutler and Miller 2005, Knutsson 2017, Ogasawara and Inoue 2016 (working paper), Ogasawara and Matsushita 2018.
    • See this document for tables containing the studies we identified. We compiled estimates of the mortality impact of water quality interventions in this spreadsheet.

  • 76
    • Anderson, Charles, and Rees 2018 (working paper), Cutler and Miller 2005, Knutsson 2017, Ogasawara and Inoue 2016 (working paper), Ogasawara and Matsushita 2018.
    • See this document for tables containing the studies we identified. We compiled estimates of the mortality impact of water quality interventions in this spreadsheet.

  • 77

    See this document for tables containing the studies we identified.

  • 78

    The study that does not support the hypothesis is Gallardo-Albarran 2020. See Gallardo-Albarran 2020, table 4, Pg. 749. We are uncertain how to interpret the findings of the single unsupportive study because it does not provide information on the degree of improvement in water quality. However, it is worth noting that the same study does report that sewer infrastructure improvements reduced the risk of death from airborne diseases: "The joint effect of β and γ is not statistically significant, although the findings show that improvements in sewage removal are associated with lower deaths due to respiratory conditions." Gallardo-Albarran 2020, Pg. 749.

  • 79

    It is possible that this is because respiratory diseases are one of the most common causes of infectious disease mortality, so changes in mortality are easier to detect. In several studies, reduction in deaths from waterborne diseases and respiratory infections together do not fully explain the reduction in all-cause mortality, so the reduction in all-cause mortality may also be expressed via other causes that are more difficult to identify. Alternatively, some or all of this discrepancy could be due to misdiagnosis.

    • “Of the total reduction, i.e., in the general death-rate, we find that the chief components for specific causes of death are as follows: 19 percent contributed by diarrhea and gastro-intestinal diseases; 15 per cent by inflammatory diseases of the respiratory organs; 12 per cent by phthisis; 8 per cent by diphtheria; 6 per cent by typhoid fever; and 40 per cent by other causes of death.” Sedgwick and MacNutt 1910, Pg. 519.
    • “Table 6 presents regression results using the same specification with other causes of death as the dependent variable. Among the causes of death that were reported consistently from 1900 to 1936, the other diseases that responded to clean water were infectious diseases: pneumonia, tuberculosis, meningitis, and diphtheria/croup. As is shown in the last column of Table 6, reductions in pneumonia, meningitis, tuberculosis, and diphtheria/croup account for 9%, 5%, 6%, and 4%, respectively, of the total mortality reduction. Together with typhoid fever and an assumption about unobserved reductions in diarrhea and enteritis, we can identify specific causes of death for 32 percentage points of the 43% decline in total mortality that is attributable to clean water.” Cutler and Miller 2005, Pg. 14.
    • “About three-quarters of the observed effect of sanitation on infant mortality can be attributed to diseases that one would expect to be related to sanitation a priori. Sanitation has a sizeable effect on both gastrointestinal disease and infectious respiratory disease. The implied elasticities are just over 0.5 for both categories evaluated at weighted sample means. Similar effects are found for the MacDorman and Rosenberg categories 'certain gastrointestinal' and 'pneumonia and influenza'. As expected, most other disease categories show no significant correlation with the timing of sanitation projects, and there is no significant effect of sanitation on mortality from all causes other than GI and infectious respiratory diseases.” Watson 2006, Pg. 1551.

  • 80

    For example, nearly all of the studies focus on Europe, the United States, and Japan in the late 19th and early 20th centuries. One (Gunther and Fink 2010 (working paper)) looks at water quality improvements in sub-Saharan Africa, and one (Gamper-Rabindran, Khan, and Timmins 2010) examines water quality improvements in Brazil.

  • 81

    For example, most of the studies focus on municipal water improvements, including chlorinated or filtered piped water, differ from the chlorine dispensers or in-line chlorination to which we might direct funding.

  • 82

    Data and calculations supporting this statement can be found in GiveWell, Water quality CEA (ILC and DSW), 2021, “Adult mortality scaling factor” tab.

  • 83

    “On average, distributing water disinfection products for use at the household level may reduce diarrhoea by around one quarter (Home chlorination products: RR 0.77, 95% CI 0.65 to 0.91; 14 trials, 30,746 participants, low quality evidence; flocculation and disinfection sachets: RR 0.69, 95% CI 0.58 to 0.82, four trials, 11,788 participants, moderate quality evidence). However, there was substantial heterogeneity in the size of the effect estimates between individual studies. Point-of-use filtration systems probably reduce diarrhoea by around a half (RR 0.48, 95% CI 0.38 to 0.59, 18 trials, 15,582 participants, moderate quality evidence). Important reductions in diarrhoea episodes were shown with ceramic filters, biosand systems and LifeStraw® filters; (Ceramic: RR 0.39, 95% CI 0.28 to 0.53; eight trials, 5763 participants, moderate quality evidence; Biosand: RR 0.47, 95% CI 0.39 to 0.57; four trials, 5504 participants, moderate quality evidence; LifeStraw®: RR 0.69, 95% CI 0.51 to 0.93; three trials, 3259 participants, low quality evidence). Plumbed in filters have only been evaluated in high-income settings (RR 0.81, 95% CI 0.71 to 0.94, three trials, 1056 participants, fixed effects model). In low-income settings, solar water disinfection (SODIS) by distribution of plastic bottles with instructions to leave filled bottles in direct sunlight for at least six hours before drinking probably reduces diarrhoea by around a third (RR 0.62, 95% CI 0.42 to 0.94; four trials, 3460 participants, moderate quality evidence).” Clasen et al. 2015, abstract.

  • 84
    • "We examine the effects of exposure to malaria in early childhood on educational attainment and economic status in adulthood by exploiting geographic variation in malaria prevalence in India prior to a nationwide eradication program in the 1950s. We find that the program led to modest increases in household per capita consumption for prime age men, and the effects for men are larger than those for women in most specifications.” Cutler et al. 2010, abstract.
    • “This study uses the malaria-eradication campaigns in the United States (circa 1920), and in Brazil, Colombia and Mexico (circa 1955) to measure how much childhood exposure to malaria depresses labor productivity. The campaigns began because of advances in health technology, which mitigates concerns about reverse causality. Malarious areas saw large drops in the disease thereafter. Relative to non-malarious areas, cohorts born after eradication had higher income as adults than the preceding generation. These cross-cohort changes coincided with childhood exposure to the campaigns rather than to pre-existing trends.” Bleakley 2010, abstract.

  • 85

    “This study estimates long-run impacts of a child health investment, exploiting community-wide experimental variation in school-based deworming. The program increased labor supply among men and education among women, with accompanying shifts in labor market specialization. Ten years after deworming treatment, men who were eligible as boys stay enrolled for more years of primary school, work 17% more hours each week, spend more time in non-agricultural self-employment, are more likely to hold manufacturing jobs, and miss one fewer meal per week. Women who were in treatment schools as girls are approximately one quarter more likely to have attended secondary school, halving the gender gap. They reallocate time from traditional agriculture into cash crops and non-agricultural self-employment. We estimate a conservative annualized financial internal rate of return to deworming of 32%, and show that mass deworming may generate more in future government revenue than it costs in subsidies.” Baird et al. 2016, abstract.

  • 86
    • We searched the PubMed database using the following search terms: water AND (chlorin* OR filtration OR purification OR disinfec*) AND (income OR earnings OR econom*). We activated the “humans” filter. Due to the large number of results (1,045), we used three strategies to narrow the search. First, we activated the “best match” function and considered the first 200 results. Then, we activated the “clinical trials” filter and considered all 53 results. Then, we repeated the search, adding “AND (long-term OR long-run OR follow-up)” to the search terms, and considered all 77 results.
    • We searched the EconPapers database using the following search terms: (free text search): water AND (chlorin* OR filtration OR purification OR disinfec*) AND (income OR earnings OR econom*). This search returned 166 results, all of which we considered. We then performed an additional search in EconPapers for diarrhea-specific studies using the following search terms: (diarrhea OR typhoid) AND (income OR earnings OR econom*). This search returned 157 results, all of which we considered.
    • We also searched for additional references in the papers we identified.

  • 87

    “New water purification technologies led to large mortality declines by helping eliminate typhoid fever and other waterborne diseases. We examined how this affected human capital formation using early-life typhoid fatality rates to proxy for water quality.” Beach et al. 2016, abstract.

  • 88

    The study used typhoid fever rates as a proxy for waterborne disease exposure. “We examine how this affected human capital formation using early-life typhoid fatality rates to proxy for water quality. We merge city-level data to individuals linked between the 1900 and 1940 Censuses. Eliminating early-life exposure to typhoid fever increased later-life earnings by one percent and educational attainment by one month. Instrumenting for typhoid fever using typhoid rates from cities that lie upstream produces results nine times larger. The increase in earnings from eliminating typhoid fever more than offset the cost of elimination.” Beach et al. 2016.

  • 89

    See our assessment of this study: GiveWell, ILC development effects, 2021.

  • 90
    • Kenya: "Among children under age five who had diarrhoea in the two weeks preceding the survey, the percentage for whom advice or treatment was sought from a health facility or provider: Total 57.6%." Kenya Demographic and Health Survey 2014, table 10.7, Pg. 152.
    • Uganda: "Among children under age 5 with diarrhoea: Percentage for whom advice or treatment was
      Sought: Total: 70.5%." Uganda Demographic and Health Survey 2016, table 10.8, Pg. 181.
    • Malawi: "Among children under age 5 with diarrhoea: Percentage for whom advice or treatment was
      Sought: Total: 65.8%." Malawi Demographic and Health Survey 2015-16, table 10.7, Pg. 154.

  • 91

    For example, in Malawi, care-seeking rates for children under five with acute respiratory tract infections, fever, and diarrhea are 78%, 67%, and 66%, respectively. Malawi Demographic and Health Survey 2015-16, tables 10.5, 10.6, and 10.7, Pgs. 152-54.

  • 92

    This is based on a simple mean of the findings of four studies identified by a quick literature search for data on the proportion of under-five diarrhea cases that are hospitalized in sub-Saharan Africa. We excluded studies published more than 10 years ago and those conducted in the context of free healthcare. This identified Burton et al. 2011, Page et al. 2011, Breiman et al. 2011, and Omore et al. 2013. See our calculations in GiveWell, Water quality CEA (ILC and DSW), 2021, “Costs averted supp calculations” tab.

  • 93
    • This estimate comes from Baral et al. 2020, a meta-analysis of studies on the medical costs of diarrhea in low- and middle-income countries. Baral et al. 2020, table 3.
    • Kenya is classified by the World Bank as a lower-middle-income country, while Uganda and Malawi are classified as low-income countries. See World Bank, "Data: Kenya", World Bank, "Data: Uganda", and World Bank, "Data: Malawi".

  • 94

    This is calculated as the percentage of diarrhea episodes for which care is sought, minus the percentage that receive inpatient care. See our calculations in GiveWell, Water quality CEA (ILC and DSW), 2021, “Costs averted supp calculations” tab.

  • 95
    • This estimate comes from Baral et al. 2020, a meta-analysis of studies on the medical costs of diarrhea in low- and middle-income countries. Baral et al. 2020, table 3.
    • Kenya is classified by the World Bank as a lower-middle-income country, while Uganda and Malawi are classified as low-income countries. See World Bank, "Data: Kenya", World Bank, "Data: Uganda", and World Bank, "Data: Malawi".

  • 96

    See our data and calculations in GiveWell, Water quality CEA (ILC and DSW), 2021, “Costs averted supp calculations” tab.

  • 97

    See our calculations in GiveWell, Water quality CEA (ILC and DSW), 2021, “Costs averted supp calculations” tab.

  • 98

    “As discussed in Section 1.2, chlorine is a hazardous substance. Chlorine disinfection requires that water treatment plant staff work in contact with, and in proximity to, high strength forms of chlorine. The health and safety of staff is critical at all times. All staff in contact with chlorine should receive basic training on the dangers of chlorine, how to handle and store it safely and basic first-aid measures in the event of accidental contact (Textbox K).” World Health Organization, Principles and practices of drinking-water chlorination, 2017, Pg. 21.

  • 99

    Table 3 of Cutler and Miller 2005 lists the dates of water quality improvements for 13 US cities; 12 of 13 had implemented water chlorination by 1919. Cutler and Miller 2005, table 3, Pg. 8.

  • 100

    “Disinfection by-products (or DBPs) result from the reaction of chlorine with organic and inorganic
    material present in the drinking-water. Some of these compounds have been linked to public
    health concerns. Examples of disinfection by-products include: trihalomethanes (THMs); haloacetic acids (HAAs); chlorate; chlorite. Strategies to control disinfection by-product formation include: optimizing water treatment processes to remove organic and inorganic material (i.e., disinfection by-product precursors); optimizing the chlorine dose to ensure adequate disinfection without adding too much chlorine (i.e., over-dosing chlorine); optimizing the chlorine dose point to ensure that only treated water is dosed (without compromising contact times); avoiding pre-chlorination (only if possible without compromising other water quality considerations; and reducing the water age in the distribution network (as time is an important factor for disinfection by-product formation).” World Health Organization, Principles and practices of drinking-water chlorination, 2017, Pg. 18.

  • 101

    World Health Organization, Principles and practices of drinking-water chlorination, 2017, Pg. 18.

  • 102

    For an example, see our cost-effectiveness analysis of Dispensers for Safe Water and in-line chlorination: GiveWell, Water quality CEA (ILC and DSW), 2021.

  • 103

    See our discussion of this evidence here.

  • 104

    See our calculations here. For details on the method and why we selected it, see our discussion here.

  • 105

    See our calculations here: GiveWell, Water quality CEA (DSW and ILC), 2021, “Adult mortality scaling factor” tab.

  • 106
    • Based on a conversation with Dr. Jay Berkley, we believe a likely mechanism by which the Mills-Reincke phenomenon operates is that enteric diseases impair nutritional status and body energy reserves, increasing risk from subsequent infectious diseases: "Professor Berkley believes there is an identifiable, though complex, mechanism by which poor nutrition and stunting are linked to diarrhea and increased risk of infectious diseases. . . . The increased infectious disease risk from recurring diarrhea or undernutrition may be a function of metabolic fragility as well as immunological weakness." GiveWell's non-verbatim summary of a conversation with Dr. Jay Berkley, April 29, 2021.
    • This mechanism is expected to affect young children the most, since they have smaller body reserves of energy, protein, and other nutrients. We identified one study that addresses this hypothesis empirically in people under five years old and people over five years old side-by-side. Newman et al. 2020 measured the risk of respiratory infection following a bout of diarrhea in infants and mothers. Infants had a substantially elevated risk of respiratory infection following diarrhea, while mothers did not. This provides some evidence that the Mills-Reincke phenomenon may not operate in adults. It is worth noting that the “over five” category is not composed solely of adults, and children over five may still be susceptible to the Mills-Reincke phenomenon.
    • To derive our adjustment, we roughly assume that Mills-Reincke has an impact half as large in over-fives as in under-fives, meaning that mortality reduction attributed to non-enteric diseases is reduced by half. (“Diarrheal illness was a significant risk factor for subsequent respiratory illness in infants but not in women during pregnancy or in women up to six months postpartum.” Newman et al. 2020.)
    • See our adjustment here: GiveWell, Water quality CEA (DSW and ILC), 2021, “Internal validity adjustment” tab.

  • 107

    For example, our model of in-line chlorination in Kenya suggests that the intervention reduces under-five mortality by 13%, but we constrain this to 11% as a result of plausibility modeling. See here.

  • 108

    We describe our methods for this in this document. The accompanying spreadsheet contains the calculations.

  • 109

    These include the possibilities that diarrhea-related mortality may be underestimated, water quality interventions may impact more severe cases of diarrhea disproportionately, water quality interventions may reduce death from non-waterborne diseases, and water quality interventions may directly prevent waterborne diseases that do not manifest as diarrhea. These are discussed here.

  • 110

    See our initial investigation of this issue in this document. See additional evidence and discussion in our plausibility adjustment document. See our discussion of this mechanism in the context of our evaluation of the RCT evidence.

  • 111

    This spreadsheet illustrates an example of how we apply this method to in-line chlorination in Kenya.

  • 112

    See our calculations for in-line chlorination here: GiveWell, Water quality CEA (DSW and ILC), 2021, “ILC Kenya” tab.

  • 113

    See this sheet for calculations.

  • 114

    Some of the trials included in Kremer et al. 2022 (working paper) included interventions other than direct improvement of water quality. Since these other interventions are not included in the programs we are evaluating, their contribution to effect size must be subtracted from our estimate. For these trials, we exclude the portion of the effect size that we believe would not be replicated by a simple chlorination intervention. See this adjustment in our cost-effectiveness analysis of Dispensers for Safe Water and in-line chlorination. GiveWell, Water quality CEA (DSW and ILC), 2021, “ILC internal validity adjustment” tab.

  • 115

    See an example of this in our cost-effectiveness analysis of Dispensers for Safe Water and in-line chlorination: GiveWell, Water quality CEA (DSW and ILC), 2021, “Deaths linked to water quality” tab.

  • 116

    See an example of this in our cost-effectiveness analysis of Dispensers for Safe Water and in-line chlorination: GiveWell, Water quality CEA (DSW and ILC), 2021, “Adherence adjustments” tab.

  • 117

    GiveWell, Water quality CEA (DSW and ILC), 2021, “ILC Kenya” tab, "Percent reduction in under-5 all-cause mortality, final estimate" and "Implied reduction in over-5 all-cause mortality, for illustration only" rows.

  • 118

    See our calculations related to the Mills-Reincke adjustment in the “Internal validity adjustment” tab of our water quality CEA. GiveWell, Water quality CEA (DSW and ILC), 2021, “Internal validity adjustment” tab.

  • 119

    GiveWell, Water quality CEA (DSW and ILC), 2021, “DSW” tab, "Percent reduction in under-5 all-cause mortality, final estimate" and "Implied reduction in over-5 all-cause mortality, for illustration only" rows.

  • 120

    See an example of this in our cost-effectiveness analysis of Dispensers for Safe Water and in-line chlorination. GiveWell, Water quality CEA (DSW and ILC), 2021, “ILC Kenya” tab.

  • 121

    GiveWell, A method for estimating adult consumption effects of interventions for which we do not have direct evidence, 2020

  • 122

    See an example of this in our cost-effectiveness analysis of Dispensers for Safe Water and in-line chlorination: GiveWell, Water quality CEA (DSW and ILC), 2021, “ILC Kenya” tab.

  • 123

    See our model here: GiveWell, Water quality CEA (DSW and ILC), 2021, “ILC Kenya” tab.

  • 124

    See our calculations here: GiveWell, Water quality CEA (DSW and ILC), 2021, “Adherence adjustments” tab.

  • 125

    See our calculations here: GiveWell, Water quality CEA (DSW and ILC), 2021, “Morbidity effect size” tab.

  • 126
    • See our plausibility limits here: GiveWell, Water quality CEA (DSW and ILC), 2021, “Internal validity adjustment” tab.
    • See how our plausibility limits are calculated in this document and accompanying spreadsheet.

  • 127

    See our cost-effectiveness analysis of in-line chlorination here: GiveWell, Water quality CEA (DSW and ILC), 2021, “ILC Kenya” tab.

  • 128

    See our cost-effectiveness analysis of in-line chlorination here: GiveWell, Water quality CEA (DSW and ILC), 2021, “ILC Kenya” tab, "Contribution of each outcome to overall cost-effectiveness" section.

  • 129

    “Dispensers for Safe Water provides chlorine dispensers at wells and other water sources in rural and remote communities of Kenya, Uganda, and Malawi. Community members can dispense a pre-measured amount of diluted chlorine into their water container prior to collecting water from the source.” Evidence Action's Dispensers for Safe Water program – December 2018 version.

  • 130

    To see the similarities between the models, compare the "ILC Kenya" and "DSW" tabs of our cost-effectiveness analysis: GiveWell, Water quality CEA (DSW and ILC), 2021.

  • 131

    GiveWell, Water quality CEA (DSW and ILC), 2021, “Adherence adjustments” tab.

  • 132

    See our cost-effectiveness analysis of Dispensers for Safe Water here: GiveWell, Water quality CEA (DSW and ILC), 2021, “DSW” tab.

  • 133

    For example, we estimate that chlorine dispensers would increase water treatment in Kenya by about 25%, and we estimate that in-line chlorination would increase water treatment in Kenya by about 60%. See our figures and data sources here: GiveWell, Water quality CEA (DSW and ILC), 2021, “Adherence adjustments” tab.

  • 134

    See our cost-effectiveness analysis of in-line chlorination here: GiveWell, Water quality CEA (DSW and ILC), 2021, “ILC Kenya” tab.

  • 135

    Evidence Action shared baseline water treatment data for Dispensers for Safe Water with us, but the data only represent a fraction of the areas served by the intervention. We supplement this with Demographic and Health Survey data on water treatment rates in an effort to improve our estimate, but these estimates are at the national level and are therefore limited in their specificity for beneficiary populations. See our data and calculations on the “Adherence adjustments” tab in our water quality CEA: GiveWell, Water quality CEA (DSW and ILC), 2021.