Evidence Action — Scale-Up of In-Line Chlorination in India (September 2023)

Evidence Action is a longtime recipient of GiveWell grants.2 Its Safe Water Now program operates two water chlorination interventions, chlorine dispensers (former program name Dispensers for Safe Water) in Uganda, Kenya, and Malawi and in-line chlorination in Malawi and Uganda. It also runs two technical assistance programs in India that are supported by GiveWell: Deworm the World and Equal Vitamin Access, an iron and folic acid supplementation program.

The opportunity

The goal of this grant is to support the government of India to introduce ILC and to provide the tools needed to overcome any barriers to doing so. Below, we walk through some key background points:

• The Indian government is prioritizing delivering piped water to all rural Indian households. The Government of India is undertaking a major push to get piped water to all rural households in India. This effort is called the Jal Jeevan Mission (JJM). Our understanding is that its structure as a "mission" indicates a high level of government support. ILC can be installed at water points that rely on pipes, so there is an opportunity to leverage the infrastructure that is being built under JJM.
• The Indian government is currently prioritizing improving water access, rather than water chlorination. While increasing access to piped water has been a priority, our understanding, based on conversations with government officials and experts, is that water chlorination has been a lower priority to date.3
• We think the government would be unlikely to pay for technical assistance to support water chlorination programs itself. Our understanding from conversations with Evidence Action and a consultant with whom we work closely is that it would be very unusual for a government to pay an NGO to provide this type of support. We did not directly ask the government officials we spoke with about this possibility. However, when speaking with external experts about the grant, none proactively raised the possibility that the government would pay an NGO directly to provide this assistance.

Cost-effectiveness

We think there's a plausible, intuitive case for the grant being cost-effective that is based on empirical evidence and conversations with Evidence Action, government officials in India, and other experts. We quantify the intuitive case for the grant by creating a rough cost-effectiveness estimate, which indicates that our best guess is that the program is approximately 22 times as cost-effective as unconditional cash transfers.

This estimate is significantly more uncertain than our typical cost-effectiveness estimates. We view it as a rough check on our intuition, rather than a precise estimate of impact. Below, we walk through the intuition for the grant's cost-effectiveness.

We think chlorination can be effective at reducing deaths.

Our estimate of the reduction in deaths in children under 5 is based on five randomized controlled trials. Based on this analysis, we estimate that chlorination interventions reduce all-cause mortality in children under 5 by ~12% in low-income settings (95% confidence interval, -31% to +13%).15 We apply a 25% discount to this effect size to account for bundling of chlorination with other interventions in the trials, resulting in a ~9% effect of chlorination on reducing all-cause mortality in children under 5.16

We expect chlorination to have a large impact on mortality in India.

Our best guess is that chlorination will reduce all-cause mortality by ~4%, averaged across age groups, in the program's target populations in India.

To estimate the effect chlorination has in India, we start from the estimates discussed above and then account for three factors:

• How water contamination in India compares to water contamination in the trials.
• The proportion of deaths linked to water quality in India compared to the trials.
• Intervention coverage and adherence.

We discuss these in turn below.

We think there is a large degree of water contamination.

Available evidence suggests that contamination of water with fecal pathogens is likely a major problem in India, even within piped water infrastructure.18 We combine three indirect sources of evidence to come up with an assessment:

• The first input is E. coli data from Multiple Indicator Cluster Surveys (MICS). We consider MICS results to be high quality. MICS data is not available for the Indian states we're interested in, so we use MICS surveys from piped water systems in neighboring countries with comparable economic status and water infrastructure to make an informed guess.
• The second input is 2019-21 Demographic and Health Survey (DHS) reports on how people typically treat their water in India, such as through boiling, filtering, or chlorination.
• The third input is research on the intermittency of water pressure in South Asia. Intermittent water pressure is correlated with higher rates of contamination because people are more likely to store water when it's not continuously available, and water is often contaminated during storage.

Based on these sources, we guess that the level of water contamination in the households served by the program would be similar to the level of contamination in the trials.

We used this information to make a "water contamination adjustment" score for three states to which we considered recommending support during our grant investigation (Andhra Pradesh, Madhya Pradesh, and Rajasthan), which we use to adjust the effect we expect chlorination to have on mortality in the grant states relative to the chlorination trials. We created simplified scores for other Indian states by comparing DHS data on water infrastructure in these states with data from Andhra Pradesh, Madhya Pradesh, and Rajasthan. Our adjustments range from -7% to +17%.

Some locations have a higher prevalence of waterborne disease than others. We compare the prevalence of enteric (intestinal) infections in the times/locations of the trials that underlie our mortality reduction estimate and the prevalence of enteric infections in India to assess the likelihood that chlorination will be effective in the program context.

Our estimate of the extent to which contaminated water leads to deaths in India is based on data on the share of deaths due to enteric infection from the Institute for Health Metrics and Evaluation (IHME). We sense-checked these data with Indian government data from nationally representative verbal autopsies. We further sense-checked the IHME data with three experts, who expressed skepticism about the IHME figures for people over 5 and provided specific reasons why they could be inflated.19 We make a -31% adjustment to the IHME estimates of enteric infection mortality among people over 5 to account for this.

In-line chlorination is a way to deliver chlorine to water sources that have a tank fed by a pipe. ILC is set up by attaching a small device to the pipe. As water passes through the pipe, it is automatically disinfected before entering the tank.20 We think ILC devices are an effective way to increase uptake of chlorination because they don't require action on the part of the consumer to chlorinate the water. The water is chlorinated by the time it comes out of the tap.

We estimate baseline water treatment rates in India on the basis of DHS data, which suggest water disinfection rates of 1%-62% after accounting for some upward bias due to self-reporting, depending on the state.21 We estimate chlorination rates for households served by ILC devices based on data from Pickering et al. 2019 (a randomized controlled trial of ILC in Bangladesh) and the ILC Kenya pilot data from 2022,22 which together suggest ~67% water chlorination among targeted households.

We expect our grant to lead to more households in rural India receiving chlorination.

We estimate that our grant will lead to approximately 260 million additional person-years of ILC coverage.

We think the grant will increase the number of people served by ILC devices:

1. By providing advice and additional staff capacity to state governments in Andhra Pradesh and Madhya Pradesh.

Evidence Action plans to provide technical assistance to support setting up an ILC program in Andhra Pradesh and Madhya Pradesh. We estimate that this work will lead to an additional ~33 million person-years of ILC coverage in Andhra Pradesh and Madhya Pradesh.

Theory of change

We think ILC's work will increase the number of person-years of ILC coverage in Andhra Pradesh and Madhya Pradesh through two paths:

1. Increasing the likelihood that Andhra Pradesh and Madhya Pradesh implement ILC.

• We think Evidence Action will make it more likely for the federal government to recommend and support ILC by keeping political attention on the program and by providing technical guidance that the federal government can share with states on how to budget for, tender, and run ILC programs.23 Evidence Action will also directly support officials in Andhra Pradesh and Madhya Pradesh in budgeting for, tendering, and running ILC programs.24 We expect this to remove barriers to implementing the program.

2. Increasing the quality of Andhra Pradesh and Madhya Pradesh's ILC implementation.

• Once Andhra Pradesh and Madhya Pradesh have launched their tenders and selected contractors to install and maintain devices, Evidence Action will provide support during device installations.25 It will troubleshoot when issues arise and provide materials to support community sensitization, so that people are less likely to reject chlorinated water.26
• Once devices are installed, Evidence Action will help the government set up strong monitoring and accountability for the program at the federal and state level.27 It will also conduct independent checks to see how well the program is working. If the monitoring turns up any issues, Evidence Action staff can help troubleshoot.28

Evidence Action will work with the government to entrench ILC by standardizing operating procedures, generating attention for the program, and building government capacity to maintain the program.29 Evidence Action will eventually wind down its support as the program shifts from the phase where contractors are installing and maintaining devices to the longer-term maintenance plan, where state and local officials are responsible for maintaining the program.30 We think it's plausible that the systems and processes that Evidence Action has put in place will continue to raise the quality of ILC after its direct support ends.

We estimate the number of person-years of additional ILC coverage in Andhra Pradesh and Madhya Pradesh based on:

• The number of person-years of ILC coverage with Evidence Action
  • We estimate this on the basis of:
    • The number of devices that will be installed during the program.
      • We base this on Evidence Action’s estimate that ~13,500 devices will be installed across the two states during the five-year grant period,31 which we then adjust by assuming the scale-up will be six months slower than Evidence Action predicts for the five-year grant and 12 months slower for a potential follow-up grant (see below for details on this grant).32 We sense-checked our estimates by:
        • Comparing them to the speed of scale-up of Evidence Action’s water programs in Africa.
        • Checking what they imply in terms of coverage of the target population. Overall, we assume the program will be targeting ~45 million people per year by year five.
        • We asked experts and stakeholders whether the implied coverage estimates and our estimates of target population are plausible. The majority of stakeholders said they seem plausible.
    • The number of people reached per device (~2,000 in Andhra Pradesh and ~1,000 in Madhya Pradesh). Our estimate is based on government and DHS survey data.33
    • We model continued (and decreasing) benefits for five years following each grant period. We expect benefits to last after Evidence Action concludes the program because it plans to set up systems and processes for the government to continue to run the program (see timeline above for details). This is consistent with how we've modeled other technical assistance grants, but is a subjective choice.
    • We apply a cap to adjust for our expectation that no more than 30% of the target population could be reached in any given year under the initial grant. These are the maximum coverage percentages that Evidence Action reported it would achieve over each respective grant period.34
    • Overall, we estimate roughly 96 million people-years of ILC coverage over 10 years in Andhra Pradesh and Madhya Pradesh with Evidence Action’s support.
• The number of person-years of ILC coverage without Evidence Action
  • We take at face value Evidence Action's estimates of the percentage of the target population in each state that would be reached with water treatment in years 1-7 in the absence of its support.35 Evidence Action's estimates are based on available state-level data on past rates of water treatment coverage.36 Its estimates assume a roughly linear year-on-year trendline.
  • We model benefits beyond year 7 by continuing to assume a linear increase in counterfactual coverage but at about half the magnitude as in years 1-7.37 This is because our understanding is that improving water quality is currently a higher priority for government officials due to JJM, but our expectation is that this interest could wane in the event that JJM were to end.
  • We model coverage with Evidence Action as higher quality and more effective.38
  • We apply a cap to adjust for our belief that it would be implausible for coverage without Evidence Action to reach more than 80% of coverage with Evidence Action's support in any given year.
  • Overall, we estimate roughly 32 million person-years of ILC coverage over 10 years in Andhra Pradesh and Madhya Pradesh without Evidence Action’s support.39
• We estimate that there’s a ~50% chance of realizing program benefits over 10 years for the first grant. The majority of this risk is due to the likelihood that Evidence Action's program could be discontinued entirely at different points in time.
  • We estimate the chances of discontinuation to be higher in the first two years (as the program is set up), decrease in years 3-7 (once the government program is more established), and increase after year 7 (once responsibility for the program transitions to the community).
  • These estimates are based on our understanding that there is a reasonably high risk of discontinuation for NGOs implementing technical assistance programs.

If the first grant is successful, GiveWell or other funders could decide to fund a second grant, covering a larger proportion of the population in Andhra Pradesh and Madhya Pradesh. We think that there could be another ~99 million person-years of ILC coverage in these states with potential future funding.

The theory of change and calculations are similar to the ones outlined above. We estimate the program would install roughly 32,000 devices and reach ~70% coverage by year 7, with around 400 million person-years of ILC coverage over 13 years. We estimate that without Evidence Action, there would have been 135 million person-years of ILC coverage. We estimate a ~35% chance of realizing program benefits over 13 years for a second grant.40

3. By causing other Indian states to take up the ILC program due to seeing the success of the program in Andhra Pradesh and Madhya Pradesh or as a result of the federal government providing additional encouragement and guidance.

We estimate this grant will lead to ~60 million additional person-years of ILC coverage in states outside of Andhra Pradesh and Madhya Pradesh.

Theory of change

The technical assistance provided by Evidence Action to the federal government and Andhra Pradesh and Madhya Pradesh may lead to this outcome by:

1. Making it more likely for a new state to become interested in ILC.41

• With Evidence Action's support, we expect the federal government to provide guidance on how to budget for ILC and implement the program successfully (for example, by sharing standard operating procedures, draft tenders, community sensitization plans, and MLE plans).42 This guidance could lower the cost for a new state to set up an ILC program.
• With Evidence Action's support, the federal government may remove less effective water disinfection technologies from the list of recommended options, making it more likely states converge towards ILC.
• Evidence Action plans to work with the federal government to set up meetings to hold states accountable for progress in water treatment.43
• The federal government may explicitly advocate that states adopt ILC because the program has gone well in Andhra Pradesh or Madhya Pradesh.
• A new state may see the state(s) that are implementing the program well as a model that it wants to copy.44
• Evidence Action may increase public attention on water chlorination, increasing the likelihood that other states take interest.

2. Making it easier for that state to adopt the program and do it well.

• Evidence Action will increase the amount of technical support that the federal government can provide to states that want to implement ILC programs.
• Through its work in Andhra Pradesh and Madhya Pradesh, Evidence Action plans to develop a "playbook" for how to run the ILC program well, which the national government can make available to states.45
• Evidence Action plans to staff a flexible, 10-person team that it can deploy to states where it's not directly working in order to provide support.46

Based on our research during the grant investigation, we think that without this grant, there would be a lower likelihood that other states would be interested in taking up, or would effectively implement, an ILC program.

We estimate the number of person-years of additional ILC coverage provided due to this grant's indirect effect on government scale-up of ILC in other states based on:

• Our estimate of the target population
  • We estimate the number of people who are currently reached by tap water by using government estimates and discounting them by 25% to account for GiveWell's skeptical prior.
    • We then estimate how access to tap water will grow as JJM expands. Based on conversations with experts, we think it's likely that JJM will be extended until 2028. We therefore model continued growth in household tap connections until then. We model this growth by assuming JJM reaches the average number of new households it reached per year in 2020-2023 from 2024-2028.
• Our estimate of coverage reached by the program
  • We outline four scenarios of varying degrees of optimism and assign them each a probability. Both the scenarios and the probabilities we assigned were based on Evidence Action’s input. Evidence Action's assessment is informed by its experience running another technical assistance program in India, Deworm the World.47 We're uncertain whether this experience is predictive. We asked a number of experts about the likelihood that Evidence Action's program in two states will lead to higher ILC coverage in other states. We broadly heard that this was a plausible story.
    • We estimate that the program will increase coverage of ILC for the total population served by JJM's single-village-scheme (SVS) sites by as little as 0 percentage points and as much as ~4 percentage points per year across scenarios, depending on location.48 We also think the program will have some effect on coverage of people living in JJM's multi-village-scheme (MVS) sites. Since MVS is not the focus of Evidence Action's program, we expect half the percentage point increase in coverage for this population compared to those living in sites served by SVS.
  • Overall, we estimate roughly 130 million additional person-years of ILC coverage over 12 years in states beyond Andhra Pradesh and Madhya Pradesh with Evidence Action’s support.
• Our estimate that there’s an approximately 50% chance of program success over 14 years.49
  • We based our estimate on the information reported above, using Andhra Pradesh and Madhya Pradesh as benchmarks.

3. By causing other Indian states to request support from Evidence Action.

Theory of change

Some states outside of Andhra Pradesh and Madhya Pradesh that become interested in ILC may seek Evidence Action's support to scale up the program. If GiveWell or another funder supported Evidence Action, it could further increase chlorination coverage in additional states, leading to ~70 million person-years of ILC coverage.

We estimate the number of person-years of additional ILC coverage based on:

• The number of person-years of ILC coverage with Evidence Action
  • We estimate this on the basis of our estimate of target population, described here, and our assumption of coverage reached with Evidence Action’s support, which takes the yearly coverage rates Evidence Action estimates it will achieve in Andhra Pradesh and Madhya Pradesh and adjusts for scale-up to occur one year more slowly.
• The number of person-years of ILC coverage without Evidence Action
  • We base our estimate on the information reported above, using Andhra Pradesh and Madhya Pradesh as benchmarks.
• We estimate that there’s an approximately 20% chance of program success over 12 years.50
  • We base our estimate on the information reported above, using Andhra Pradesh and Madhya Pradesh as benchmarks. We also account for the likelihood that additional states are interested in Evidence Action's support for ILC scale-up, which we estimate on the basis of Evidence Action’s input.

In-line chlorination is inexpensive to deliver.

We estimate that in-line chlorination costs approximately $1-3 per additional person-year of ILC coverage provided in India. This includes costs from Evidence Action’s technical assistance and device installations (including GiveWell-provided and other funding sources) as well as government costs. We estimate this on the basis of Evidence Action’s budget and Evidence Action's estimate of government costs.

Our estimate includes the costs to procure, install, and maintain the device (for example, refilling it with chlorine tablets and providing electricity). In-line chlorination is inexpensive because chlorine is cheap, the one-time procurement and installation costs (about $1,500 per device)51 are spread over multiple years (the lifespan of the device, which can vary based on model)52 and because each device can serve many people (~1,000 to 2,000 people on average in Andhra Pradesh and Madhya Pradesh).

A large share of program costs will be borne by the government, and we think this program is more cost-effective than what the government would otherwise spend money on.

Since Evidence Action's role is to provide technical assistance to the government, we expect the funding for the ILC program to largely come from the government (~69% of expected costs).53 The grant's high estimated cost-effectiveness is in large part due to our assumption that it will cause a large amount of government funding to be redirected from less cost-effective programs to ILC.

Our understanding is that, due to its nature as a "mission," the funding for JJM comes from a general pool of funding that could be spent on programs across a variety of sectors, rather than funding that is more narrowly earmarked (for example, for water programs).54 Our estimate of how cost-effectively those funds would otherwise be spent is based on Evidence Action's calculation, using its understanding of GiveWell’s estimation methodology, of the expected value of government spending in India.55 According to this method, counterfactual government spending is ~one-third as cost-effective as cash transfers.56 We take Evidence Action's estimate at face value because it seems reasonable, but we have not thoroughly vetted its calculation, and we have low confidence in it.

What is the effect of chlorination on mortality?

We are uncertain about the effect of chlorination on mortality of people under 5 and over 5. Our 25th - 75th percentile confidence interval for the effect that chlorination has on mortality for all ages is 1%-7%, which implies a cost-effectiveness range of 5x-37x cash.

We describe our method to estimate this input above.

We remain very uncertain about the size of the mortality reduction effect on people under 5. This is due to the discordance between estimates generated by different methods, the wide confidence interval of the trial-based estimate, and limited information on the mechanisms that may account for such a large mortality effect. We discuss this in more detail in our water quality report here.

We don't expect to get information from this grant to improve our estimates of the effect of chlorination on mortality in people under age 5.

We describe our method to estimate this input above.

In the absence of experimental evidence, we're very uncertain about the size of the effect. In particular, a large portion of the mortality benefits we model are from chlorination's effect on reducing non-waterborne disease (the Mills-Reincke phenomenon). While we're uncertain about the mechanism by which this occurs, we think a likely explanation is that enteric diseases increase the risk from infectious disease by impairing nutritional status and body energy reserves. We think this mechanism is likely to affect younger children more than older population groups.66 We make an adjustment to reflect our best guess that the impact of Mills-Reincke phenomenon is weaker on people over 5 than under 5, but we're highly uncertain about its size.

We're also uncertain about the effect that chlorination will have on people over 5 in India, specifically. To estimate the effect chlorination has in India, we start from the pooled estimate discussed above and account for the proportion of deaths linked to water quality in India. To estimate the latter, we use IHME data and adjust it based on the results of our sense-check and expert input. However, we're uncertain about the size of that adjustment.

The reduction in mortality for people over age 5 accounts for approximately 40% of the benefits in our cost-effectiveness model.67

We don't expect to get information from the grant to improve our estimates of the effect of chlorination on mortality in people over age 5.

We describe our method to estimate this input above.

We believe piped water schemes in India possess microbiological contamination, based on data from neighboring countries and information on risk factors for contamination. However, this is only indirect information, so we could easily be wrong.

How many additional people will receive chlorination in Andhra Pradesh and Madhya Pradesh as a result of Evidence Action's program?

We describe our method to estimate this input above.

We're unsure about the number of person-years of ILC coverage with Evidence Action. This is especially due to uncertainty about the speed of scale-up of the ILC program (since we don’t know whether Evidence Action’s water programs in Africa are an apt comparison), uncertainty about how long benefits will persist after Evidence Action leaves the program (since we don’t have data on this), and uncertainty about the risk of program discontinuation (since we have heard disparate opinions from experts). We are particularly uncertain about the latter. For example, we could be underestimating the risk that the government withdraws its support to Evidence Action, or the ILC technology does not work well in India, or communities don't accept the intervention. Alternatively, we could be overestimating Evidence Action’s ability to run a project at this scale.

During the grant, we expect to learn about the speed of scale-up of ILC in Andhra Pradesh and Madhya Pradesh, which will enable us to estimate the number of people reached by this grant.

We're unsure about the number of person-years of ILC coverage without Evidence Action, which we use to estimate the counterfactual impact of Evidence Action's program. This is especially due to uncertainty about using past water treatment rates to predict future rates. It’s possible that interest in water chlorination will grow faster in future years, for example due to people learning about the results of Kremer et al. 2022 (working paper), a recent meta-analysis that found a significant effect of water treatment, including chlorination, on mortality.

We estimate the initial grant will lead to ~33 million additional person-years of ILC coverage in Andhra Pradesh and Madhya Pradesh and that additional grants will lead to ~99 million additional person-years of ILC coverage. Our 25th - 75th percentile confidence interval is that:

• The initial grant will lead to 15-45 million additional person-years of ILC coverage, which implies a cost-effectiveness range of 20x-23x cash.
• Additional grants will lead to 40-160 million additional person-years of ILC coverage, which implies a cost-effectiveness range of 17x-27x cash.

How many additional people will receive chlorination in other states as a result of Evidence Action's program?

​​We describe our method to estimate this input above: here for states that take up the ILC program without Evidence Action's direct support, and here for states that request support from Evidence Action.

We're unsure about the number of additional person-years of ILC coverage with Evidence Action. This is because we're unsure whether Evidence Action’s experience with Deworm the World is predictive of what might happen with ILC, since the programs are very different. For example, ILC requires setting up physical infrastructure and requires the government to run very large tenders, while deworming does not. This could make ILC more difficult to scale up in other states than deworming was. During the grant, we expect to learn about the scale-up of ILC and interest in scaling ILC in states beyond Andhra Pradesh and Madhya Pradesh.

We take Evidence Action's estimates of ILC coverage at face value, but we assume benefits would materialize a year later than Evidence Action estimates in states we think are likely to be quick adopters of ILC and four years later in states we think are likely to be slower adopters of ILC.68 We're highly unsure about this timeline.

Since Evidence Action's estimates of its policy impact were conceived of with a three-state program in mind (rather than the two we ultimately funded), we also adjust downwards to account for the lower likelihood that a two-state program will catalyze policy change. We're very uncertain about these adjustments, as they're highly subjective.

More broadly, the grant's effect beyond Andhra Pradesh and Madhya Pradesh relies on complex causal chains that are hard to forecast. While experts have told us this type of effect is plausible, we have not been able to elicit quantitative estimates, which leaves us highly unsure about the likelihood and size of these effects.

We estimate the program will lead to an additional ~60 million person-years of ILC coverage in states that don't seek Evidence Action's direct support. Our 25th - 75th percentile confidence interval for this parameter is that this grant will lead to 2-100 million additional person-years of ILC coverage in these states, which implies a cost-effectiveness range of 17x-25x cash.

We estimate the program will lead to an additional ~70 million person-years of chlorination if GiveWell or another funder supported Evidence Action to work in additional states. Our 25th - 75th percentile confidence interval for this parameter is that this grant will lead to 10-140 million additional person-years of ILC coverage by opening up funding opportunities to scale ILC in other states, which implies a cost-effectiveness range of 17x-27x cash.

During the grant, we expect to learn about the scale-up of ILC, or interest in scaling ILC, in states beyond Andhra Pradesh and Madhya Pradesh.

How would the government otherwise spend funding?

​​We describe our method to estimate this input above. We have very little confidence in it, since we have not triangulated it with local stakeholders or experts. We don't expect to learn about the counterfactual value of government spending via this grant, though we may decide to conduct additional desk research to improve this estimate in the future.

Estimating the 25th - 75th percentile confidence interval for this input implies a cost-effectiveness range of -4x-24x cash.69

We're unsure whether the states we have selected optimize the program’s cost-effectiveness.

We don't believe that Andhra Pradesh and Madhya Pradesh are the most cost-effective states in which to implement the program, when we only consider direct benefits, and exclude benefits deriving from chlorination being taken up in other states. This is because in other Indian states, mortality rates are higher and the proportion of deaths caused by enteric diseases is larger (and those are two key drivers of cost-effectiveness).70

However, our understanding is that Andhra Pradesh and Madhya Pradesh provide a good chance of unlocking policy benefits in other states (which make up a significant percentage of overall program benefits).71 This relies on Evidence Action’s assessment, which is in turn based on its understanding of chances of program success across states and the fact that these two states encompass different geographies and demographics, which makes their policy experiences relevant to a wide set of Indian states.

This picture makes sense to us, though we have not systematically considered whether these two states provide the optimal combination of direct and indirect benefits.

We haven't considered other possible implementers for this program.

As discussed above, we believe Evidence Action is well-placed to implement this program. However, we have not actively looked for alternative implementers.

We're not sure if Evidence Action is excited about the grant for the same reasons we are.

​​We describe how Evidence Action's view of this grant as an exceptional opportunity contributes to the qualitative case for the grant above.

We're uncertain whether Evidence Action is excited about this opportunity because of its high potential cost-effectiveness or because of other ways it could have impact, such as the large number of people reached or the potential for long-term sustainability of the program.

We're unsure how much we will learn from this grant and how our learnings will generalize to other grants.

We describe how this grant could improve our understanding of technical assistance grants above.

Although we're taking steps to increase the learnings from this grant, we don't expect to generate a quantitative estimate of counterfactual coverage (see above). It's also not yet clear to what extent we will see the learning from this grant as relevant for future TA programs we're evaluating. We expect technical assistance grants to be highly context-specific, which can limit the applicability of learnings from one grant to another.

Monitoring and evaluation

There are four key areas about which we hope to learn during this grant.

1. Baseline water microbiological quality and water treatment

We're considering collecting baseline data on water microbiology and treatment. This baseline data would help us refine our estimates of baseline water contamination rates and chlorination rates. This is valuable because we're very uncertain about our current baseline estimates for water contamination and somewhat uncertain about our estimate of baseline chlorination rates.72

We're very uncertain about our estimates of the number of person-years of ILC coverage in Andhra Pradesh and Madhya Pradesh with Evidence Action’s support. During the grant, we expect to get information from Evidence Action that will help us assess the accuracy of our estimates:

• Number of functioning devices73
• Number of people per device74
• Percentage of people served by a device who have chlorinated water75

Our estimate of the target population to be reached with in-line chlorination also makes assumptions about whether funding for JJM is renewed and the speed of scale-up of tap water connections. Going forward, we expect to learn about JJM funding and to review government data on the progress of scale-up.

3. What impact does the program have on chlorination rates in the target population in Andhra Pradesh and Madhya Pradesh, as well as in states where Evidence Action does not plan to work directly?

Evidence Action is planning to collect data on chlorination rates as part of its routine monitoring.76

We're also exploring the possibility of funding an independent before-and-after impact evaluation to estimate the effect the program has on chlorination rates. We're unsure about the likely design. At the moment, we think it will consist of before and after surveys for chlorination rates. Because a before-and-after design would not allow us to distinguish between changes brought about by Evidence Action and changes that would have happened anyway, we also plan to conduct interviews with stakeholders and collect data on process outputs in order to understand if and how Evidence Action helped release the bottlenecks to scaling ILC.77

This would take place in Andhra Pradesh and Madhya Pradesh, as well as a number of additional states where Evidence Action does not plan to directly work (though Evidence Action noted that government approvals could be a potential bottleneck outside of Andhra Pradesh and Madhya Pradesh).78 The impact evaluation is not included in the overall grant budget.79

4. What are Evidence Action and the government's costs to run the program?

We expect to receive updated information about the costs to run this program from Evidence Action during the course of the grant. We also expect to be able to review government budgets.

For this grant, we record the following forecasts. Note that a number of our forecasts are conditional on Evidence Action's program not being discontinued.

Because of the size of the grant and the considerable uncertainty we felt about the program, we conducted a particularly extensive grant investigation.

• We spoke with Evidence Action ~30 times.
• We traveled to India to meet with key stakeholders, including government officials.
• We spoke with more than 20 experts, including several recommended by Evidence Action, to discuss the grant's theory of change, the plausibility of some of the estimates in our cost-effectiveness model, and other topics.
• We put the grant recommendation through an unusually intensive peer review process, during which GiveWell's internal peer reviewers provided feedback approximately weekly for the final two months of the grant investigation.