Note: This page summarizes the rationale behind a GiveWell Incubation Grant to Tufts University. Amy Pickering and Brandon Joel Tan, two of the researchers whose work this grant is intended to support, reviewed this page prior to publication.
Summary
In September 2020, Tufts University received a GiveWell Incubation Grant of $203,796 to support an expansion of the Kenya Study of Water Treatment and Child Survival (KSWTCS). The Principal Investigator team includes Michael Kremer, Amy Pickering, Ricardo Maertens, Julie Powers, and Brandon Joel Tan. The study is a follow-up of an earlier randomized controlled trial, and it aims to estimate the effect of water chlorination on mortality rates of children under five years old. We expect the study to be a major input into our cost-effectiveness analysis of Evidence Action's Dispensers for Safe Water program, a current standout charity that we are evaluating as a potential top charity. KSWTCS is partly funded by another donor, which is covering the costs associated with estimating the program’s effect on child mortality. This grant is intended to fund (a) additional analyses, testing specific mechanisms through which water chlorination decreases child mortality, and (b) additional survey checks, to improve the quality of the data collected. We expect these additions to increase our confidence in the study results.
Published: April 2021
Table of Contents
Background
The Kenya Study of Water Treatment and Child Survival (KSWTCS) is a follow-up to the WASH Benefits study, and is designed to estimate the effect of point-of-collection water chlorination on mortality rates of children under five years old. The WASH (Water, Sanitation, and Hygiene) Benefits study in Kenya was a randomized controlled trial (RCT) that examined the effects of a variety of water, sanitation, handwashing, and nutrition interventions on early child development. The WASH Benefits study in Kenya enrolled pregnant women and tested their children's development outcomes one and two years after the interventions began.1 One of the interventions tested by the RCT was a water quality intervention, which consisted of providing chlorine dispensers at water collection sites as well as providing bottled chlorine directly to households.2
After the trial, distribution of chlorine bottles was discontinued, but Evidence Action maintained the dispensers that had been installed in most of the RCT's treatment group villages and did not install dispensers in the control group villages.3 An initial, small follow-up analysis was conducted using data collected during an unrelated psychology study run in the area.4 This analysis took advantage of the preserved randomization to estimate the effect of water chlorination on child mortality rates. The preliminary results of this analysis suggest a significantly larger effect of chlorination on child mortality rates than we had previously estimated in our water quality interventions report and review of Dispensers for Safe Water (an Evidence Action program and GiveWell standout charity).5 However, the confidence interval of this result is wide because the study's sample size was small.6 KSWTCS, a separate follow-up study, will have a larger sample size.7 It will also collect information on cause of death, with the goal of exploring the mechanisms and plausibility of the measured effect on child mortality rates.8
The study's sample consists of women who have given birth since chlorine dispensers for the Kenya WASH Benefits study were installed.9 KSWTCS will have the opportunity to achieve greater statistical power to measure child mortality than WASH Benefits. The WASH Benefits study covered only women who were in their last two trimesters of pregnancy at the start of the study, and followed their children for two years. KSWTCS will include all children who have been exposed to the intervention over the six years since dispensers were installed (or are in the control group).10
Planned activities
The primary funding for KSWTCS is from Sint Antonius Stichting Projecten (SAS-P), a Dutch foundation.11 This GiveWell Incubation Grant will be used to:
- Collect and analyze additional data on water sources and the people who live near them in treatment and control villages. Additional data will be collected to identify populations with high potential chlorination rates—i.e., those who are most likely to chlorinate their water, primarily based on proximity to a water source that either has a chlorine dispenser or would be eligible for one—in both the treatment and the control groups. The researchers expect this to increase the study's power to detect an effect of water chlorination on child mortality and the specific mechanisms explaining this effect.12
- Collect data on the effect the intervention had on maternal health. Data will be collected on the prevalence of gastrointestinal and respiratory symptoms among the studied mothers, which the researchers expect could inform our understanding of a potential mechanism of the child mortality effect, since improved maternal health might increase infants’ survival rates.13
- Fund staff time to analyze the effect of distributing chlorine bottles for point-of-use chlorination. This information will allow the researchers to distinguish effects from the initial intervention implemented in the WASH Benefits study (which included chlorine dispensers and bottled chlorine) from the effects of the intervention currently implemented by Dispensers for Safe Water (which includes chlorine dispensers only).14
- Improve the quality of the data collected by increasing the number of survey checks performed.
The researchers will increase back-checks of the surveys conducted, re-do surveys from enumerators with poor data quality, visit a subset of women a second time who were not available for interview during the first survey, and repeat a subset of chlorination tests.15
Details of the grant’s budget can be found here.
Case for the grant
Our current model of Dispensers for Safe Water's cost-effectiveness estimates the program’s effect on child mortality indirectly, based on our estimate of the program’s effect on diarrhea rates, and our best guess of diarrhea case fatality rates in the African countries where it operates.16 Moreover, our estimate of the program’s effect on diarrhea rates relies on a combination of studies that assessed direct provision of chlorine to households (point-of-use chlorination) and studies that assessed provision of chlorine dispensers near water sources (point-of-collection chlorination).17 Dispensers for Safe Water provides the latter. We are unsure of the extent to which evidence on point-of-use chlorination generalizes to point-of-collection chlorination. We also have concerns about the robustness of some of the studies we rely on in our current review.18
KSWTCS is the first major study we know of that is evaluating an intervention very similar to Dispensers for Safe Water's, and it is rigorously designed to directly measure the impact of the intervention on child mortality, which is the outcome we care most about. The small-scale follow-up to WASH Benefits found an effect on child mortality that seems implausibly high to us, but if we became more confident in these results, we would believe that Dispensers for Safe Water is one of the most cost-effective opportunities we've found, and would likely add it to our list of top charities and seek to direct funding to it.19 We believe that this grant will increase the likelihood that the study gives us information that can help us make a more confident assessment of Dispensers for Safe Water's impact (and therefore its cost-effectiveness), in three main ways:
- By enabling the researchers to collect additional data that will increase the study's power to detect a result.20
- By enabling the researchers to collect additional information about the mechanisms through which water chlorination averts deaths. One reason we're currently skeptical of the results of the small-scale follow-up is because these results are inconsistent with our current assumption that the deaths prevented via water chlorination are only those directly caused by diarrhea.21 Evidence that there are other ways in which this intervention averts deaths22 would give us more confidence in the study results, in the case that the study finds a larger effect than can be explained by preventing deaths from diarrhea alone.
- By reducing the risk of bias in the results by increasing the number of survey checks performed.23
Given the relatively small size of this grant, our assessment of the case for the grant relies primarily on qualitative considerations; we have not estimated the cost-effectiveness of this grant.
Our process
The researchers sent us a funding proposal for the KSWTCS study extension. We had spoken with the researchers before in the context of the initial, smaller study that followed up on WASH Benefits, and they were aware of our interest in the larger study. We reviewed the proposal and had two follow-up calls with the researchers.
Risks and reservations
- The researchers are collecting additional data that could inform our understanding of the mechanism through which the intervention has impact, and we aren't confident that there will be sufficient statistical power to be confident in what we learn about the mechanism. We did not do an in-depth analysis of how much statistical power the study will have to detect specific mechanisms by which water chlorination reduces child mortality. The data on mechanisms is of interest to us, but is not critical to our conclusions about the cost-effectiveness of the program.
- The study has been delayed due to the COVID-19 pandemic, and we are unsure when we'll see results. Data collection was interrupted in March 2020, after 70% of surveys had been completed.24 The researchers resumed data collection in January 2021.25 (Though this grant was made in September 2020, GiveWell formally committed to make the grant on March 4, 2020, and had been in discussions about the grant with the researchers for several months before then.)
- We plan to further refine our cost-effectiveness analysis of Dispensers for Safe Water's program, beyond updating it with the results of this study. It is possible that had we spent more time refining our model before making this grant, we might have concluded that the results from this study were very unlikely to change our conclusions about whether or not to recommend funding for Dispensers for Safe Water.
Sources
- 1
"We aimed to assess whether water, sanitation, handwashing, and nutrition interventions reduced diarrhoea or growth faltering.(...) The WASH Benefits cluster-randomised trial enrolled pregnant women from villages in rural Kenya and evaluated outcomes at 1 year and 2 years of follow-up.(...) Primary outcomes were caregiver-reported diarrhoea in the past 7 days and length-for-age Z score at year 2 in index children born to the enrolled pregnant women." Null et al. 2018, p. 316.
- 2
"Chlorine dispensers for convenient water treatment at the point of collection were installed at an average of five communal water sources in the cluster and refilled as needed. Every 6 months, households in study compounds were given a 1 L bottle of chlorine for point-of-use water treatment in case households collected rainwater or used a source without a dispenser. Promoters used chlorine test strips during their regular visits to determine if the household was using chlorine, and negative results stimulated conversation about addressing barriers to chlorination." Null et al. 2018, p. 319.
- 3
- "Evidence Action continued to provide access to dispensers in treatment areas following the end of the WASH Benefits study." Funding Proposal for KSWTCS, October 17, 2019, p. 5.
- "Evidence Action continued to refill dispensers after the WASH Benefits Kenya study was finished in Bungoma and Kakamega counties, but not in Vihiga county." Funding Proposal for KSWTCS, October 17, 2019, p. 10.
- "The Kenya WASH Benefits study was a cluster-randomised trial done in rural villages in Bungoma, Kakamega, and Vihiga counties in Kenya's western region." Null et al. 2018, p. 317.
- In a conversation on November 4, 2019, the research team told us that Vihiga includes ~10% of the study population in the original WASH Benefits study. Michael Kremer, Ricardo Maertens, and Amy Pickering, conversation with GiveWell, November 4, 2019
- 4
The psychology study is Haushofer, John, and Orkin 2018 (working paper). We have not received permission to cite the follow-up analysis, which is unpublished, so we are not naming it in this report.
- 5
Funding Proposal for KSWTCS, October 17, 2019, p. 3. Because we do not have permission to publish this analysis, we are not publishing the effect size that it found or other specific data from it.
- 6
Funding Proposal for KSWTCS, October 17, 2019, p. 3.
- 7
"We note that by covering all the areas of the Null et al. (2018) study, we expect to obtain a sample about 10 times as large as the one covered by Haushofer et al. (2018). Further, by including women who were not enrolled in Null et al. (2018), we expect to cover over twice as many women as initially examined by Null et al. (2018)." Michael Kremer, concept note on water treatment and child mortality, February 6, 2019, p. 7.
- 8
"In addition, data on cause of death will be collected through verbal autopsy. Conducting verbal autopsies can provide evidence of the mechanisms through which water treatment promotes survival and whether potential child survival effects are consistent with the known biological pathways through which water treatment affects health (e.g., death by diarrhea or other infectious diseases)." Michael Kremer, concept note on water treatment and child mortality, November 17, 2018, pp. 7-8.
- 9
- "The main aim of the proposed study is to survey a large enough number of women from the villages included in the study by Null et al. (2018), with a long enough follow-up period, to attain adequate power to examine the effect of the community-wide provision of water treatment solution on child survival. Such a sample can be obtained by surveying all women under 35, who had a child in the past 10 years, and who live in a village which was randomized to water treatment or control arms. This would involve surveying both women who were and who were not “enrolled” in the Null et al. (2018) study. The latter group is made up of women who were pregnant before the Null et al. (2018) study, women who were in their first trimester at the time of enrollment, and women who got pregnant after study enrollment." Michael Kremer, concept note on water treatment and child mortality, February 6, 2019, p. 7.
- "Based on the data we have collected thus far, we now project there are approximately 44,000 women eligible to participate in our study in the WASH Benefits study villages (i.e. who have given birth in the relevant time period)." Michael Kremer, email to GiveWell, August 2019
- 10
“The WASH Benefits study covered only women in their last two trimesters of pregnancy and followed their children for two years. However, it provided dispensers that could be used by others in the community. Moreover, Evidence Action continued to provide access to dispensers in treatment areas following the end of the WASH Benefits study. Approximately 60% of those in treatment communities use water sources with dispensers. This creates an opportunity to measure water treatment effects on child survival in a very large sample, encompassing 55,000 children with up to six years of exposure to water treatment.” Funding Proposal for KSWTCS, October 17, 2019, p. 5.
- 11
Michael Kremer, email to GiveWell, October 22, 2019
- 12
"Because most households use the closest water source, one can attain higher power by focusing on households that live close to sources that would have been eligible for dispensers had the village been assigned to treatment. Similarly, one can focus on households with high potential chlorination rates for reasons other than their distance to a dispenser-eligible water source, e.g., households where the most senior woman has a high level of education, or households with young children. (...)An analysis focusing on the comparison of mortality rates among communities with high potential chlorination rates (e.g., who live close to a dispenser-eligible water source) not only increases the power to pick up average all-cause mortality effects, but creates sufficient power to examine specific subpopulations and thus to identify potential mechanisms which might resolve the puzzle of larger survival estimates than predicted with the linear model." Funding Proposal for KSWTCS, October 17, 2019, pp. 6-8.
- 13
“We can also examine whether water treatment improves maternal health, which could also help explain the very large survival effects with the Haushofer et al. (2018) data. To do this, we would develop a maternal health module, which would be incorporated in our survey moving forward for all eligible pregnant women. The module would record prevalence of gastrointestinal and respiratory illness symptoms, including loose or watery stool, bloody stool, fever, cough, and difficulty breathing. For children with available clinic cards, we would also be able to record weight at birth.” Funding Proposal for KSWTCS, October 17, 2019, p. 9.
- 14
"The original WASH Benefits intervention installed chlorine dispensers but also distributed bottled chlorine. However, since Evidence Action only continued with the distribution of chlorine through dispensers, there is an opportunity to exploit the time variation in which bottled chlorine stopped being distributed to examine the effect of dispensers separately from that of bottled chlorine. While all households with children under 5 received 1-liter bottles of dilute chlorine solution, every 6 months, for the first year of the study, only enrolled households continued to receive bottled chlorine during the second year. We will, therefore, examine mortality impacts before and after these changes in the provision of bottled chlorination among the relevant groups." Funding Proposal for KSWTCS, October 17, 2019, p. 9.
- 15
- "[W]e think it makes sense to strengthen our data quality by increased back-checks of surveys conducted, repeated chlorine residual testing, and conducting revisits for a subset of women we were not able to interview. Scaling up our back-checks would allow us to further improve our QA/QC and more finely track enumerator data quality." Funding Proposal for KSWTCS, October 17, 2019, p. 10.
- "Please find a breakdown of our data collection direct costs below: (...)
- Water source census: $20,000
- Backchecks: $10,000
- Re-doing surveys from enumerators with poor data quality: $10,000
- Repeated chlorination tests: $15,000
- Mother Module: 25,000
- Revisits to women who are not home at the time of the first visit: $25,000"
Ricardo Maertens, emails to GiveWell, November 4 and 12, 2019
- 16
See here for details. Note that our cost-effectiveness analyses are simplified models that do not take into account a number of factors. There are limitations to this kind of cost-effectiveness analysis, and we believe that cost-effectiveness estimates such as these should not be taken literally, due to the significant uncertainty around them. We provide these estimates (a) for comparative purposes and (b) because working on them helps us ensure that we are thinking through as many of the relevant issues as possible.
- 17
Our 2017 review of Dispensers for Safe Water mainly relied on Clasen et al. 2015, a Cochrane meta-analysis of the effect point-of-use chlorination has on diarrhea, and Null et al., an unpublished study on point-of-collection chlorination.
In our review of Dispensers for Safe Water, we wrote: "Our best guess is based on results from the studies identified in the Cochrane review [Clasen et al. 2015], and adjusts for our expectation of results from the Null et al. working paper."
- 18
We wrote, "Our best guess is that water chlorination interventions prevent diarrhea in children under five years of age. However, a number of factors complicate the interpretation of this body of evidence: self-report of diarrhea, risk of bias from lack of blinding, and variable adherence to treatment."
- 19
See here for our best-guess estimate of Dispensers for Safe Water's cost-effectiveness before seeing the preliminary results of the WASH Benefits follow-up. See here for more details on how our cost-effectiveness estimate would change as our estimate of the program's effect increases. Since we are unable to cite the paper draft reporting the preliminary results of the WASH Benefits follow-up, we are not reporting our current best guess.
- 20
See "Collect and analyze additional data on water sources and the people who live near them in treatment and control villages," above.
- 21
See our 2013 report on water quality interventions and our review of Dispensers for Safe Water for more detail.
- 22
Some additional mechanisms through which water chlorination might decrease child mortality are: (a) decreasing diarrhea rates might improve children’s immune systems, thus indirectly decreasing mortality from other infectious diseases; (b) improving water quality might affect maternal health, which, in turn, might improve rates of infant survival; (c) improving water quality might improve hygienic practices during birth, also increasing infant survival.
- “There are several potential reasons why true effects could differ from those predicted in the linear model. One is the Mills-Reincke phenomenon in which interventions to improve water quality also reduce mortality rates from other infectious diseases (Sedgwick and MacNutt, 1910). A leading explanation for the Mills-Reincke phenomenon is that diarrheal disease increases the risk of other diseases. Therefore, by reducing the risk of diarrheal disease, water treatment would be able to avert deaths from diarrheal and non-diarrheal, infectious diseases.” Funding Proposal for KSWTCS, October 17, 2019, pp. 3-4.
- “[A]ccess to chlorinated water for handwashing and cleaning during the birth process could reduce neonatal exposure to bacterial pathogens (Blencowe et al., 2011), potentially reducing the risk of sepsis.” Funding Proposal for KSWTCS, October 17, 2019, p. 4.
- “Fifth, water treatment has the potential to improve maternal health which, in turn, could improve fetal health and survival after birth. For example, maternal enteric infections could lead to premature delivery or low birth weight.” Funding Proposal for KSWTCS, October 17, 2019, p. 5.
- 23
See "Improve the quality of the data collected by increasing the number of survey checks performed," above.
- 24
Amy Pickering, emails to GiveWell, March 23 and April 27, 2020
- 25
Amy Pickering, email to GiveWell, January 13, 2021