Evidence Action — India In-Line Chlorination Pilot Expansion (July 2023)

Note: This page summarizes the rationale behind a GiveWell grant to Evidence Action and reflects our rationale at the time the grant was recommended. Evidence Action staff reviewed this page prior to publication.

Published: March 2024

Summary

What we think this grant will do

In March 2023, we recommended a ~$1.8 million grant to Evidence Action to pilot in-line chlorination in India by installing 18 in-line chlorination devices across three states: Andhra Pradesh, Madhya Pradesh, and Rajasthan.1 The goal of the 14-month pilot was to learn about the implementation feasibility of different in-line chlorination devices in these Indian states and to build state government buy-in around in-line chlorination as a promising water safety technology to scale up.

This $1.03 million grant will expand the original pilot from 18 to 50 devices and will enable Evidence Action to test the devices' applicability across a wider range of scenarios. The grant budget will primarily support in-line chlorination devices and monitoring sensors for devices. (More)

Why we recommended this grant

We recommended this grant because we think the pilot expansion will:

• Increase the likelihood of success of an Evidence Action program we're considering funding. We're considering a large grant (expected value $27 million) to Evidence Action to provide technical assistance to the government of India to support the scale-up of in-line chlorination at single-village schemes (SVS). We're considering supporting that program because we think it is likely to be cost-effective. This is because we think chlorination is inexpensive and has a substantial effect on mortality. We also think that Evidence Action’s technical assistance will lead to greater adoption of chlorination. Note: for more details, see our write-up on the larger grant, which we recommended in September 2023.

  One of our uncertainties about the program is that the technology has not been tested at scale, which makes us concerned the program might fail due to unforeseen technical problems. We think that testing devices' applicability across a wider range of scenarios will enable Evidence Action to troubleshoot technical problems in advance of the scale-up. We think this decreases the risk that the program will ultimately fail. (More)

• Open up additional funding opportunities by improving our and Evidence Action's understanding of the technology needed to provide in-line chlorination across water points that serve multiple villages. In-line chlorination for multi-village schemes (MVS) is promising, since its wider reach might lead to lower cost per person than SVS. However, Evidence Action has limited experience running this technology. The pilot would give Evidence Action additional experience running in-line chlorination for MVS, improving their and our learning about this technology, and the likelihood we have enough information to decide to fund an MVS ILC program. (More)

We have developed a rough model to quantify the above benefits. Our best guess is that this grant is ~11 times as cost-effective as cash transfers to GiveDirectly, which is higher than our current bar of 10x cash. However, this analysis is very uncertain. As a result, we don’t put much stock in the precise number, but we take it as an indication that the grant is around our funding bar. A simplified version of our model is below:

Main reservations

• We might be overestimating the likelihood that we will fund Evidence Action's SVS ILC program, the grant size, or the program's cost-effectiveness, since our estimates come from an ongoing investigation. (More)
• We might be overestimating how much this pilot expansion decreases the risk that the SVS program will fail. For example, it's possible that the program variations Evidence Action is planning to test are not highly applicable in the states in which we're considering funding the program. (More)
• We might be overestimating the size or cost-effectiveness of the funding gap for the MVS ILC program, since our estimates benchmark on estimates for the SVS program, which are uncertain (see above). (More)
• We might be overestimating how much this pilot expansion increases the likelihood that we decide to fund the MVS ILC program, since the scale of the pilot might still be too limited to meaningfully improve our and Evidence Action’s understanding of the technology. (More)
• We spent limited amounts of time on this investigation, due to the limited size of the grant and time constraints coming from the investigation for the larger grant. (More)

The organization

Evidence Action is an organization with which we have a long-standing relationship and to which we have provided significant support.2 As of the time we recommended this grant, Evidence Action operated two water chlorination programs: Dispensers for Safe Water in Uganda, Kenya, and Malawi and in-line chlorination in Malawi and Uganda.

The intervention

In-line chlorination is a water disinfection method that can be added to any water source that has a tank fed by a pipe. In-line chlorination is done by attaching a small device to the pipe; as water passes through the pipe, it is automatically disinfected by the device before entering the tank.3

Evidence Action's pilot incorporates in-line chlorination at two different kinds of water points:4

• Single village schemes, which serve an average of 400 households per tank5
• Multi-village schemes, which serve an average of 3,700 households per tank6

Multi-village schemes have two sites for in-line chlorination: at village-level water tanks and at intermediary pumping stations.7 Evidence Action plans to test devices in each location.8

Does in-line chlorination work?

We believe that in-line chlorination can be effective at saving and improving lives. Our full analysis of the benefits of water chlorination on mortality, including in-line chlorination, can be found in our water quality report.

The grant

In July 2023, we recommended a $1.03 million grant from the All Grants Fund to Evidence Action to expand its 14-month pilot of in-line chlorination in three states in India: Andhra Pradesh, Madhya Pradesh, and Rajasthan.

• Equipment variables, such as solar-powered devices.
• Demographic variables, such as working in lower socioeconomic areas, varying the distance from the capital, and working in tribal areas.
• Environmental variables, such as setting up devices in different agro-climatic and ecological zones.11

The additional 32 devices will be divided as follows:12

• Single-village schemes: 45% (~15 devices)
• Multi-village scheme installed at the village levels: 45% (~14 devices)
• Multi-village scheme intermediary level, where the intervention is applied at a more centralized location in the system: 10% (~3 devices).

Budget for grant activities

We have not seen a breakdown of costs for the $1.03 million budget. Evidence Action reported that the budget's main cost-drivers are the 32 in-line chlorination devices and monitoring sensors.13 The sensors track and report chlorine doses in the water.14

The case for the grant

This grant will increase the likelihood of success of an Evidence Action program we're considering funding

We're considering recommending a large grant to Evidence Action to provide technical assistance to the government of India to support the scale-up of in-line chlorination at single-village-scheme sites in two to three states (Andhra Pradesh, Madhya Pradesh, and Rajasthan). We're considering supporting that program because we think it is likely to be cost-effective. This is because we think chlorination is inexpensive and has a substantial effect on mortality. We also think that Evidence Action's technical assistance will lead to greater adoption of chlorination, including both in the states they're planning to work in and in additional states. Note: for more details, see our write-up on the larger grant, which we recommended in September 2023.

One of our uncertainties about the program is that the technology has not been tested at scale, which makes us concerned the program might fail due to unforeseen technical problems. The expanded pilot will allow Evidence Action to test in-line chlorination devices across a larger number of variables, described above. We think this will enable Evidence Action to troubleshoot technical problems in advance of the scale up and that this decreases the risk that the program will fail due to technical problems.

In our cost-effectiveness analysis, we assume:

• The expected value of the SVS grant is ~$27 million. This incorporates our best guess of the grant size ($90 million) and the probability we recommend the grant (30%).15
• The grant will be around 15 times as cost-effective as unconditional cash transfers (“15x”). Since our funding bar is 10x, this implies a 5x increase in cost-effectiveness gained by funding the SVS grant instead of GiveWell's next-best funding opportunity.16
• The pilot will reduce the risk of failure of Evidence Action's technical assistance program by 5%, based on Evidence Action's estimate.17

All these assumptions are highly speculative, and we are very uncertain about them.

This grant could open up additional funding opportunities by increasing our understanding of how well in-line chlorination works at multi-village schemes

The larger funding opportunity we're considering is limited to single-village scheme sites.
Implementing in-line chlorination through multi-village schemes may be more cost-effective than single-village schemes because each device may reach more households.18 However, multi-village scheme sites may present different challenges that will still need to be evaluated. For example, they may have a higher baseline treatment rate than single-village schemes (see below for one possible reason) and a potentially higher technological complexity and cost compared to single-village schemes.19

The in-line chlorination technology for single-village schemes is similar to that used in Evidence Action's existing in-line chlorination programs in East Africa.20 However, Evidence Action's experience running the technology for multi-village schemes is limited to the existing pilot. The pilot expansion will increase that experience: Evidence Action plans to install 55% of the additional devices from the expanded pilot (~17 devices) at multi-village scheme tanks and intermediary pumping stations.21 We expect this to inform us about the effectiveness of in-line chlorination in these contexts. If it looks promising, this could lead to future cost-effective funding opportunities to implement in-line chlorination across multi-village schemes at scale.

In our cost-effectiveness analysis, we assume:

• The expected value of the MVS funding gap, assuming the pilot provides us with enough information to make a decision to fund an MVS TA program, is around $27 million.22 Our calculation for this value incorporates our estimate that the funding gap for SVS ILC is around $150 million, Evidence Action’s estimate that 20-40% of the population is served by multi-village schemes,23 and our subjective guess that there’s a ~50% chance we decide to fund the MVS program (assuming the pilot provides us with enough information to make this decision).24
• If we fund an MVS technical assistance program, the program will be around 16 times as cost-effective as unconditional cash transfers (“16x”).This is based on our assumption that the cost-effectiveness of technical assistance for in-line chlorination at multi-village schemes would be 10% higher than our estimate for single-village schemes. In turn, this is based on our understanding that multi-village schemes would reach more people per device, leading to a lower cost per person reached.25 Since our funding bar is 10x, this implies a 6x increase in cost-effectiveness gained by funding an MVS technical assistance program instead of GiveWell's next-best funding opportunity.
• The pilot will lead to a 5% increase in the likelihood that we have enough information to make a decision on whether to fund scale up of ILC at MVS sites.26

All these assumptions are highly speculative, and we are very uncertain about them.

Main reservations

We might be overestimating the chance that we will fund Evidence Action’s SVS ILC program, the grant size, or the program's cost-effectiveness.

We are in the middle of the investigation for the SVS ILC program, and we are highly uncertain about the program’s cost-effectiveness. For example, the program's success relies on a number of assumptions we are very uncertain about, including the likelihood that the government will support the program and communities will accept the intervention of chlorine. We're also discussing a wide range of potential grant sizes with Evidence Action and are uncertain where the final budget will land. As a result of these uncertainties, we're also unsure about the likelihood we will recommend a grant to the program.

The lower the likelihood we recommend the grant, the grant size, and the program's expected cost-effectiveness, the lower the value of increasing the program's chances of success (since less impact will be affected).

We might be overestimating how much this pilot expansion decreases the risk that the SVS program will fail.

Our estimate that the pilot expansion will decrease risk of program failure by 5% is a subjective guess based on Evidence Action's own best guess, but we feel very uncertain about it. For example, it’s possible that the variables added to the pilot (such as testing solar-powered devices or setting up devices in different agro-climatic and ecological zones) are not very common in the states in which we're considering funding the program, and therefore learning about those won’t significantly affect the risk of program failure.

The lower the amount by which the pilot expansion decreases the risk of failure, the lower the value of this grant.

We might be overestimating the size or cost-effectiveness of the funding gap for the MVS ILC program.

Our estimates of the funding gap and cost-effectiveness of the MVS ILC program benchmark on our estimates for the SVS ILC program. However, we are very uncertain about the latter, since our investigation of the SVS ILC program is still ongoing.

The smaller the amount of cost-effective funding opportunities to scale in-line chlorination at multi-village schemes, the lower the value of scoping multi-village schemes.

We might be overestimating how much this pilot expansion increases the likelihood that we decide to fund the MVS ILC program.

Our estimate is a subjective guess, and we're very uncertain about it. In particular, it’s possible that the scale of the pilot will still be too limited to meaningfully improve our and Evidence Action’s understanding of the technology.

The less we learn about ILC at MVS sites as a result of this pilot expansion, the lower the value of this grant.

We spent limited amounts of time on this investigation.

Due to the limited size of the grant and time constraints coming from the investigation for the larger grant, we only spent a limited amount of time on this investigation. As a result, it is possible there are additional uncertainties we have not identified.

Plans for follow-up

Evidence Action will provide updates on the progress of pilot activities, including identifying sites to install in-line chlorination devices, device installation, and community sensitization activities, which are intended to familiarize communities with in-line chlorination. Evidence Action also plans to share its data on indicators such as dosing levels, refill rates, device breakage, supply chain issues, and baseline water treatment rates at the end of the pilot in April 2024.29

Internal forecasts

For this grant, we recorded the following forecasts.

Our process

• Evidence Action expressed interest in expanding the pilot on a call on June 1.
• Evidence Action sent us an email outlining the reasons for expanding the pilot.
• We had an additional call with Evidence Action to discuss our questions on the pilot expansion.

Sources

• 1

  This grant was funded by the Weiss Asset Management Foundation and the Rauch Family Foundation.

• 2

  Evidence Action’s Deworm the World Initiative was one of GiveWell’s top charities from 2013 to 2022 and Evidence Action's No Lean Season was a top charity from 2017-2018. We have also supported the Evidence Action Accelerator, as well as programs such as syphilis screening and treatment during pregnancy in Liberia, Zambia, and Cameroon, Dispensers for Safe Water in Kenya, Uganda, and Malawi, and in-line chlorination in Malawi.

• 3

  "In-line chlorination (ILC) is a method for automatically treating drinking water. A small device is attached on a pipe near the point of water collection which dispenses chlorine into the water as the water is distributed." Evidence Action, in-line chlorination Kenya pilot results report, 2022, p. 1.

• 4

  Evidence Action, India Safe Water Opportunity, February 2023 (unpublished).

• 5

  Evidence Action, India Safe Water Proposal, 2023, p. 8.

• 6

  Evidence Action, India Safe Water Proposal, 2023, p. 10.

• 7

  "There are two identified water treatment intervention points in MVS, Village-Level Water Tanks and Intermediary Water Tanks." Evidence Action, India Safe Water Proposal, 2023, p. 9.

• 8

  "System Types: 45% single village schemes, 45% multi village scheme village-level, 10% multi village scheme intermediary level" Evidence Action, email to GiveWell, June 6, 2023 (unpublished).

• 9

  "Sites: ~32 additional sites split across all three states" Evidence Action, email to GiveWell, June 6, 2023 (unpublished).

• 10

  "Evaluate in-line chlorination technologies across additional variables to identify applicability across a wider range of scenarios. These include equipment variables (lower-range dosing pumps, solar powered devices, etc), demographic variables (lower socioeconomic areas, distance from the capital, tribal areas, etc), environmental variables (agro climatic and ecological zones), and others." Evidence Action, email to GiveWell, June 6, 2023 (unpublished).

  Source for the claim that the original pilot planned to distribute 18 devices: Evidence Action, India Safe Water Opportunity Feb 2023 (unpublished).

• 11

  See Evidence Action, Pilot expansion variables, 2023 for the full list of variables they plan to test.

• 12

  "System Types: 45% single village schemes, 45% multi village scheme village-level, 10% multi village scheme intermediary level" Evidence Action, email to GiveWell, June 6, 2023 (unpublished).

• 13

  Evidence Action, email to GiveWell, June 6, 2023 (unpublished):

  • "Budget Request: Approx $1.03 million"
  • "Sites: ~32 additional sites split across all three states"
  • "The budgets’ main cost drivers are water treatment devices and IoT monitoring sensors."

• 14
  • "The MVS pilot expansion sites will include all of the same MLE protocols, data collection, and IoT device monitoring as outlined in the initial pilot grant request." Evidence Action, email to GiveWell, June 6, 2023 (unpublished).
  • "Automated dose sensors - IoT chlorine sensors will be installed along with the water treatment devices that will provide real-time information on chlorine dosing in each system." Evidence Action, India Safe Water Pilot MLE Overview, 2023, p. 3.

• 15

  Our best guess of the grant size was based on Evidence Action's budget of $140 million over 7 years; we thought we were likely to provide a smaller grant.
  To calculate the expected value of $27 million, we multiplied $90 million by 30%.

• 16

  Read more about how we use cost-effectiveness estimates in our grantmaking here.

• 17

  Evidence Action, conversation with GiveWell, June 6, 2023 (unpublished).

• 18

  Evidence Action, conversation with GiveWell, June 16, 2023 (unpublished).

• 19

  Evidence Action indicated these challenges in written feedback on this grant page, February 22, 2024.

• 20

  Evidence Action, conversation with GiveWell, February 21, 2023 (unpublished).

• 21

  "System Types: 45% single village schemes, 45% multi village scheme village-level, 10% multi village scheme intermediary level" Evidence Action, email to GiveWell, June 6, 2023 (unpublished).

  • 45% + 10% = 55% multi-village scheme sites
  • 32 additional devices * 55% = ~17 devices

• 22

  We calculate this as a weighted average of our estimates of the following under scenarios in which GiveWell has a cost-effectiveness bar for funding grant opportunities of 8x cash, 10x cash, and 12x cash, and the probability we assign each of those scenarios (row 7, "Main CEA" tab):

  • MVS annual RFMF,$m (row 29, "Main CEA" tab)
  • Probability that we conclude program is above bar, assuming we have enough information to make a decision on MVS scale up after the pilot (row 31, "Main CEA" tab)
  • How long we would fund program if above bar after the pilot (row 36, "Main CEA" tab)

  GiveWell, India Water Pilot Expansion BOTEC, 2023

• 23

  Evidence Action, conversation with GiveWell, June 16, 2023 (unpublished).

• 24

  (15% * 70%) + (50% * 50%) + (35% * 30%) = 46% chance we would fund the MVS program if the pilot is sufficiently informative. For more details about how we calculate the MVS funding gap, see this cell of our BOTEC.

• 25

  Evidence Action, conversation with GiveWell, June 16, 2023 (unpublished).

• 26

  This is a subjective guess.

• 27
  • "While filtration and silver ionization are both JJM approved best practices and are promising at certain scale and situations, chlorine serves as the most applicable technology field when considering the water tank capacity, distance water travels, applicability across Total Dissolved Solid (TDS) ranges, residual chlorine benefit, and that MVS are already treated with chlorine [bold ours]." Evidence Action, India Safe Water Proposal, 2023, p. 6.
  • Evidence Action, conversation with GiveWell, June 16, 2023 (unpublished).

• 28

  Evidence Action, conversation with GiveWell, June 16, 2023 (unpublished).

• 29
  • "The MVS pilot expansion sites will include all of the same MLE protocols, data collection, and IoT device monitoring as outlined in the initial pilot grant request. The SVS pilot expansion sites will follow the same MLE protocols and data collection." Evidence Action, email to GiveWell, June 6, 2023 (unpublished).

    The inital's pilot's planned MLE outcomes are here.

  • "We would recommend we consolidate both pilot and pilot expansion reporting to GW in April 2024." Evidence Action, email to GiveWell, October 6, 2023 (unpublished).