Deworm the World Initiative, Led By Evidence Action - 2014 Review

We have published a more recent review of this organization. See our most recent report on Deworm the World.

The Deworm the World Initiative, led by Evidence Action, is one of our top-rated charities, and an organization that we feel offers donors an outstanding opportunity to accomplish good with their donations.

More information:


Published: December 2014

Summary

What do they do? The Deworm the World Initiative (DtWI), led by Evidence Action, advocates for, supports, and evaluates government-run school-based deworming programs. (evidenceaction.org/deworming)

Does it work? We believe that deworming is a program backed by relatively strong evidence. We also believe that DtWI raises the probability that government-run deworming programs go forward. This view is based on highly limited evidence; mostly conversations with Indian state government officials and the fact that DtWI has been involved in most statewide deworming programs currently operating in India as well as the new national deworming day. We further think that the deworming programs in which DtWI is involved are reasonably well executed, although our view is based on monitoring reports collected by DtWI, in which we have limited confidence. DtWI has made recent changes to its monitoring process that it believes addresses some of our concerns. DtWI shared information about those programs in November 2014 and we will update this review when we have finished evaluating it. (more)

What do you get for your dollar? We estimate that children are dewormed for a total of about $0.30 per child, or $0.09 per child excluding the value of teachers’ and principals’ time spent on the program. A small portion of these costs are paid for by DtWI; we generally include costs paid by all partners in our calculation. (more)
We believe the per-child positive impact of deworming is likely smaller than the impact implied by the strongest studies on deworming, because compared to those studies, the areas DtWI is working in have lower prevalence and intensity of hookworm and schistosomes, which are the worms with the strongest evidence of harmful long-term effects.

Is there room for more funds? DtWI has told us that it has a funding gap of $1.3 million for 2015 and 2016. DtWI seeks unrestricted funds to evaluate new evidence-based programs that leverage deworming, as well as support direct deworming scaleup, particularly in new countries in Southeast Asia and Sub-Saharan Africa. Quantifying the marginal impact of these funds is extremely difficult. In addition, donors should note that DtWI is only one initiative of an organization, Evidence Action, that has other work we have not evaluated as closely.

DtWI is recommended because of its:

  • Focus on a program with a strong track record and excellent cost-effectiveness. (more)
  • Standout transparency – it has shared significant, detailed information about its programs with us.
  • Room for more funding – DtWI appears to need a modest amount of additional funding. (more)

Major unresolved issues include:

  • The difficulty of assessing the impact of an organization whose role is advocacy and technical assistance, rather than direct implementation. There are substantial potential advantages to supporting such an organization, as it may be able to have more impact per dollar by influencing government policy than by simply carrying out programs on its own, but this situation also complicates impact assessment. While we believe DtWI to be impactful, our evidence is limited. In addition, there is always a risk that future expansions will prove more difficult than past ones.
  • The marginal value of additional unrestricted funds for DtWI, particularly as a significant portion of marginal funds are likely to be used on policy-oriented research, rather than on scaling up current programs.
  • The fact that we find the data we have had on the quality of the deworming programs that DtWI has been involved in to be relatively limited. DtWI shared information about those programs that it believes addresses some of our concerns in November 2014 and we will update this review when we have finished evaluating it. (more)

Table of Contents

Our review process

Our review process has consisted of:

  • Conversations with Evidence Action Executive Director Alix Zwane, DtWI Director Grace Hollister, DtWI Associate Director Jessica Harrison, Evidence Action Director of Strategic Initiatives Christina Riechers, DtWI India Country Director Priya Jha, DtWI India Operations Director Ayan Chatterjee, DtWI Strategy and M&E Associate Director Sharad Barkataki in 2013 and 2014, and Deworm the World’s Managing Director Alissa Fishbane and Regional Director Karen Levy in 2012.1
  • Conversations with the Children's Investment Fund Foundation (CIFF), a funder of DtWI, about DtWI.
  • Reviewing documents DtWI sent in response to our queries.
  • Site visits:
    • In November 2012, we visited DtWI's office in Nairobi, Kenya and met its staff there.
    • In September 2013, we visited DtWI's operations in Rajasthan, India, where we met with its local staff and with government officials who had worked with DtWI.

Older content on DtWI:

What do they do?

The Deworm the World Initiative is led by Evidence Action; its aim is to advocate for and provide technical assistance for the implementation of deworming programs for preschool- and school-aged children.

DtWI was founded in 2007,2 and is currently active in Kenya, India, and with a nascent program in Vietnam, while looking to expand to other countries in Sub-Saharan Africa and Southeast Asia.3
Its program in Kenya is fully supported by the Children’s Investment Fund Foundation (CIFF) and the Ending Neglected Diseases (END) Fund through 2016,4 so we focus this review on its activities in India – the primary program that DtWI has been developing for the last couple years and the model for its work in new countries in the future.

Since 2009, DtWI has worked to:

  • Encourage national and Indian state governments to implement mass school-based deworming programs5
  • Design and conduct prevalence and intensity surveys to determine the need for deworming in specific areas6
  • Provide technical assistance to governments’ deworming programs
  • Provide additional support for the monitoring and evaluation of the programs7
  • Evaluate new evidence-based programs that leverage deworming8

In India, the bulk of total program costs are paid for by Indian state governments (mostly by way of spending teacher and other employee time rather than financial costs); DtWI primarily pays for the activities listed above. We estimate the following breakdown for DtWI’s spending in India (note that all personnel costs are classified as program management, so non-program management lines represent non-personnel direct costs):

DtWI costs in India9
Program management: India personnel 30.6%
Program management: Global personnel 22.9%
Program management: Travel and other mgmt 18.9%
Prevalence survey 9.7%
Training 6.9%
Program management: Indirect costs 4.4%
Monitoring and evaluation 4.1%
Awareness and mobilization 1.9%

Activities by state in India

Historically, DtWI has supported deworming programs in Andhra Pradesh, Bihar, Delhi, Rajasthan, and Madhya Pradesh.10 As of October 2014, DtWI was actively supporting programs in Bihar, Rajasthan, and Madhya Pradesh (Madhya Pradesh is currently planning to complete its first statewide deworming in early 2015), while also in conversations with additional states for possible future partnerships.11 Much of our review focuses on the four deworming rounds that DtWI assisted with in 2011 and 2012 because those are the only rounds for which we had seen complete monitoring and budget information until recently. DtWI sent us information for the rounds in 2013 and 2014 in November 2014, which it believes demonstrates improved monitoring quality and budget.12 We will update this review when we have finished evaluating it.

DtWI and Evidence Action

In early 2013, Innovations for Poverty Action announced the formation of Evidence Action to take over programs implemented at scale from Innovations for Poverty Action. Evidence Action now houses two such programs: DtWI and Dispensers for Safe Water.13 We focus this review on DtWI and discuss the room for more funding implications of DtWI’s being a program of a larger organization below.

Evidence Action projects a total budget of $17.2 million for 2015, of which $10.1m is DtWI.14 Of this, about $2.5 million is allocated to Kenya (which is fully supported by CIFF and the END Fund, see above), about $6.0 million to India (of which the biggest funders will be CIFF and the United States Agency for International Development (USAID)), and about $0.8 million is allocated to engaging new countries.15

Does it work?

DtWI supports mass school-based deworming programs, the independent evidence for which we discuss extensively in our intervention report on deworming programs. There is a very strong case that mass deworming programs in general are effective in reducing infections. The evidence on the connection to positive quality-of-life impacts is less clear, but there is a possibility that deworming is strongly beneficial.

Note that our intervention report focuses on combination deworming (i.e., programs that treat both schistosomiasis and soil-transmitted helminths (STH)). DtWI is generally expanding in countries where it treats only STH due to the extremely limited presence of schistosomiasis.16 Below, we discuss the considerations relevant to comparing deworming in the places DtWI works or hopes to work (India, Southeast Asia, and Sub-Saharan Africa) to the places where the studies that form the strongest basis for the life impacts of deworming were conducted.

Here, we focus on the following questions about the deworming programs in which DtWI has already been involved in a way it would like to scale up. All such deworming programs so far are in Indian states:17

  • What proportion of children in areas DtWI works are infected? What is the intensity of these infections? The areas in which DtWI works appear to qualify for deworming under WHO guidelines, though the prevalence and intensity of infections is generally lower than it was in the studies that serve as the best evidence of deworming's impact. More in our section on cost-effectiveness.
  • Do the programs DtWI supports successfully deworm children? What evidence of impact for its programs has DtWI shared? Until recently, the best data we have on the quality of deworming programs comes from reports submitted by DtWI-compensated monitors, and suggests fairly strong (though far from perfect) implementation. However, we have substantial reservations about this data, as well as about the reliability of the government data we are using to estimate the number of children treated. (We understand that collecting reliable data at scale can be quite costly and that there is a trade-off between evidence quality and cost-effectiveness.) DtWI has made recent changes to its monitoring process that it believes addresses some of our concerns. DtWI shared information about those programs in November 2014 and we will update this review when we have finished evaluating it.18
  • DtWI is an advocacy/technical assistance organization, not a direct service provider. What impact does DtWI’s involvement have on program implementation? We have limited evidence with which to answer this question. We believe it is highly likely that DtWI's work improves the likelihood of a given Indian state's going forward with deworming, based on speaking with government officials, speaking with a source on the general dynamics of Indian school health programs, and the observation that DtWI has been involved in most of the Indian statewide deworming programs of which we are aware, as well as the planned national deworming program. We are less certain about how much DtWI improves the quality of deworming programs due to very limited evidence, although we will update this review when we have evaluated the information DtWI sent in November 2014.19

What proportion of children are infected? What is the intensity of these infections?

DtWI aims to conduct prevalence surveys for each region in which it works before the first round of treatment to determine the prevalence and intensity of infections from each STH and therefore inform appropriate treatment strategies.20

These aim to survey a representative sample of the population. DtWI has shared the technical details of several of these surveys with us, and we have spoken with the researchers who advised DtWI and its partners on the second Rajasthan prevalance survey; methodological details in this footnote.21

The following table provides results from the original prevalence surveys DtWI conducted in Bihar and Delhi, and Rajasthan’s second prevalence survey in 2013 (DtWI conducted a second survey in Rajasthan after the first year of treatment due to problems with the first prevalence survey). This footnote provides additional detail on unique circumstances for each state’s survey.22

Worm prevalence and intensity in DtWI-supported Indian states
STH (intensity) Bihar Delhi Rajasthan
Hookworm prevalence (moderate/high) 0.7% 0.0% Not measured
Ascaris prevalence (moderate/high) 8.8% 1.2% Not measured
Trichuris prevalence (moderate/high) 0.2% 0.3% Not measured
Any STH prevalence (moderate/high) 9.7% 1.5% Not measured
Hookworm prevalence (any) 42.2% 1.3% 1.0%
Ascaris prevalence (any) 52.1% 11.6% 20.2%
Trichuris prevalence (any) 0.2% 5.6% 0.2%
Any STH prevalence (any intensity) 67.5% 15.8% 21.1%

Note that the WHO recommends the following treatment frequencies based on prevalence of any STH, which DtWI advises governments to follow:23

  • Less than 20%: no mass treatment
  • 20%-49%: annual treatment
  • Greater than 50%: biannual treatment24

How many children were reached?

As of October 2014, DtWI had assisted with eight rounds of deworming in four states in India. We have seen data on how many children received treatment in seven of these rounds. We did not ask to see coverage data from the first round of treatment with which DtWI assisted (in Andhra Pradesh in 2009).

All coverage figures are reported by the state governments. We report state-level figures here, though DtWI also shared more granular data for some of the rounds (for example, at the district and block level, which each contain hundreds of schools) and hires data entry firms to check consistency between the state figures and school or block level figures.25

People dewormed by DtWI-supported programs in Indian states
State Year # of school-aged children # of pre-school children # of adults % reporting Notes
Bihar26 2011 16,775,003 - 361,408 Not reported -
Bihar27 2012 16,265,432 - 601,956 97% of blocks -
Bihar28 2014 16,225,546 - 1,244,973 92% of schools29 Delayed 4 months from annual cycle
Delhi30 2012 1,954,155 756,266 159,235 82% of schools and pre-schools Adults figure includes teen girls
Delhi31 2013 1,704,361 678,156 None reported 80% of schools32 -
Rajasthan33 2012 6,950,363 3,866,056 None reported 99% of blocks -
Rajasthan34 2013 6,691,558 4,151,147 None reported Not reported -

Uncertainty in coverage figures35

The coverage figures above may over- or underestimate actual coverage. The number of children treated in each school is reported by school staff to government officials who aggregate data from many schools and pass a tally for their administrative area to a higher level of government.36 This process is repeated several times, and the lowest level at which the disaggregated data is preserved is generally blocks, which each contain hundreds of schools.37 It is not clear what error-checking, if any, is conducted by government officials at each level of aggregation, although some blocks may be erroneously counted as having done no deworming since the tally was never reported. Errors could theoretically be introduced at any stage, either inadvertently or purposefully, such as by officials inflating numbers or guessing when reports are low or missing.

To monitor the quality of coverage figures, DtWI sends monitors to audit a sample of schools, generally a few days after the deworming days, and asks them to record the number of students dewormed according to class registers (on which teachers mark off students’ names when they receive pills) and according to school-level summaries of how many children were dewormed in each class (which are then used to create the block or district tallies).38 However, it seems to us that these audits cannot tell us about the quality of state-level data, because audits are performed at the school level while all the government data we have seen is at the block level or higher, so we have no information about errors between the school and block level. More details on our review of the auditing data are in this footnote.39

Note that collecting reliable data at scale may be quite costly; this may present a trade-off between evidence quality and cost-effectiveness.

Were deworming programs of high quality?

The information below focuses on DtWI’s four deworming programs in 2011 and 2012. DtWI has made recent changes to its monitoring process that it believes addresses some of the concerns expressed below. DtWI shared information about those programs in November 2014 and we will update this review when we have finished evaluating it.40

In each of the four deworming programs in 2011 and 2012, monitors visited randomly selected schools on the day of deworming and on the mop-up day (when children who were absent on the first deworming day are treated) to observe how the program was carried out.

Monitoring procedures

Monitoring procedures varied somewhat among the states and rounds of deworming, but in most cases the process included:41

  • Pre-program: Monitors visit a selection of schools to observe whether hygiene facilities are available, principal and teachers are prepared for deworming day, drugs and supplementary materials are available, teachers are knowledgeable on topics related to deworming, and students are aware of the upcoming deworming day.
    This part of the process is a more recent addition. To date, we have only seen data from pre-program monitoring from one deworming round, in Rajasthan in 2012 (see table below).42
  • Deworming day and mop-up day: On the days in which deworming takes place, monitors visit schools to observe whether the deworming is taking place, the principal and teacher are prepared and knowledgeable on topics related to deworming, the drugs are stored properly, the teachers are adhering to deworming procedures (such as administering the drugs after the students have eaten and not giving drugs to children who appear sick), and the children are chewing and swallowing the pills.
  • Post-deworming: Monitors are asked to visit schools a few days after deworming has concluded to check whether class registers match summary forms (discussed above) and whether students report having been given a tablet recently, what color it was and what it tasted like, and whether they swallowed it.

In Bihar, monitors visited 5% of schools in 201143 and 1.75% of schools in 2012.44 In Delhi, monitors visited 2.2% of schools and pre-schools.45 In Rajasthan, monitors visited 1.1% of schools.46 Results of the monitoring are discussed below.

Monitor selection and training

  • Bihar: Monitors for the 2011 program included both district-level officials and “the existing cadre of external monitors of the WHO’s polio programme.”47 The latter were "trained WHO monitors who DtWI recruited through a database of independent monitors provided by the WHO."48 In Bihar's 2012 program, most of the monitors came from the same database and "14 of the monitors were non-WHO monitors who [DtWI] contracted through recommendations from partners."49
  • Delhi: Monitors for the 2012 program were senior "government officials and monitors from the Health Department who had worked as monitors in the polio program."50 In Delhi's 2013 program, the "monitors were hired through a vendor. Each [2013] monitor had prior non-profit field experience, and was interviewed by [DtWI's] Delhi Team prior to selection."51
  • Rajasthan: Monitors for the 2012 and 2013 program were "independent monitors… hired through a vendor with 15 years of experience doing similar work for NGOs... such as UNICEF."52

Monitoring results

Monitors recorded observations and responses on a large number of questions. For complete results, refer to the following documents:

Monitoring reports discussed in this review:

Monitoring information from 2013 and 2014 that has not yet been incorporated into this review:

DtWI shared the more recent monitoring information with us in November 2014, and described how the monitoring process has improved in this document: DtWI Monitoring Improvements 2014.

Here we present data from a small subset of questions. We did not choose these indicators systematically. Instead, we chose a few indicators from the 2012 deworming rounds that were fairly easy to interpret, represented examples from each stage of the monitoring process, and directly addressed the question of how well the program was implemented (sources in this footnote).53

Bihar 2012 Delhi 2012 Rajasthan 2012 Sample of question asked
Pre-deworming: percent of schools that report having received drugs54 96% (on deworming day) N/A 95% of schools, 55% of pre-schools "Did the school receive the Deworming tablets?"55
Pre-deworming: percent of teachers that correctly answer whether it is safe to give deworming medicine to children who may not have worms N/A N/A 49% in schools, 44% in pre-schools "Is it dangerous to treat children who are not infected?"56
Deworming day: monitors that observe deworming taking place at this portion of schools 90% of schools 74% of schools, 86% of pre-schools 83% of schools, 69% of pre-schools "Is the deworming activity clearly visible in the school?"57
Deworming day: monitors observe drugs administered after a meal at this portion of schools 98% of schools 97% of schools, 100% of pre-schools 97% of schools, 96% of pre-schools "Did the students eat tiffin/mid-day meal before deworming?"58
Deworming day: monitors do not observe drugs administered to any sick children at this portion of schools59 89% of schools 79% of schools, not reported for pre-schools 93% of schools, 90% of pre-schools "Did you observe any sick child eating the tablet?"60
Follow-up: percent of students that report receiving treatment 98% of schools 99% in schools, 97% in pre-schools 98% in schools, 95% in pre-schools "When were you given the tablet?"61
Follow-up: percent of students that correctly report the color of tablet 67% of schools 92% in schools, n/a for pre-schools 94% in schools, n/a for pre-schools "What was the colour of the tablet?"62

For Bihar’s 2011 deworming, we have not seen aggregated results (only raw data which we have not analyzed).63

We believe there are a number of reasons to believe that monitoring data could be inaccurate, including:

  • Data is missing for many questions from many schools, presumably because monitors did not report this data (We have not analyzed how much data is missing, but looking through the data sets, there appears to be a large number of blank cells).64 It is our understanding that missing data has been excluded from the summarized results.
  • Monitors may be motivated to report more favorable results than they observe. We have limited information on who the monitors are and what their motivations are. It is our understanding that monitors receive minimal training for their role. We have not seen audits of monitors' work.
  • While some results are directly observed by monitors, some results are reported by principals, teachers, and students. Principals, teachers, and students may seek to reflect positively on their schools by hiding problems with the program.
  • Principals and teachers that are monitored on the deworming days may conduct higher quality programs because they know they are being monitored.

We also note that we have little information about procedures for storing drugs, ensuring that they do not expire, etc. The bottles of albendazole we examined on our site visit had expiration dates five years into the future, and their only storage instructions were to avoid direct sunlight and high temperatures, so we consider this a relatively minor concern.

Did worm infection and intensity rates decline after deworming?

If pills were stored and administered properly, we would expect, based on independent evidence of the efficacy of deworming drugs, that worm burdens would decrease following treatment. However, we do not have direct evidence of this for the programs that DtWI has assisted in India. DtWI plans to measure worm infection and intensity rates again after three years of deworming treatment in each Indian state, with the first such followup survey underway in Bihar as of November 2014.65 DtWI has completed two prevalence surveys in Rajasthan, which actually showed an increase in worm prevalence in the state after one year of treatment.66 However, the second survey (which showed a higher worm prevalence than the first survey) was only commissioned due to problems with the first one; DtWI believes the first survey was not very accurate and that the surveys together do not imply an increase in worm prevalence.67

What impact do DtWI’s activities have on deworming programs?

Because it is a technical assistance and advocacy organization (not a direct service organization), evaluating DtWI’s impact is not straightforward. There are substantial potential advantages to supporting such an organization, as it may be able to have a greater impact per dollar by influencing government policy than it would have by simply carrying out programs on its own. However, this situation complicates impact assessment.

DtWI may be having an impact in the following ways:

  1. It may increase the likelihood that a government implements a deworming program.
  2. It may improve the quality of the deworming program (leading to more children dewormed).

While we have seen limited evidence to support the above, we would guess that DtWI likely has some impact on the likelihood that governments implement deworming and on the quality of the deworming it implements. We find the former effect to be more important, both in terms of the strength of the case for DtWI’s impact and in terms of the magnitude of DtWI’s impact (since a modest impact on the likelihood that a state implements deworming could provide good value for money, due to the scale of the programs).

Does DtWI increase the likelihood that governments implement deworming programs?

We have seen limited evidence supporting the conclusion that DtWI has directly caused Indian states to implement deworming programs. In a nutshell, (a) DtWI has played a role in most of the mass school-based deworming programs currently in operation in India as well as the planned national deworming day; (b) our limited understanding of how Indian states decide to implement and continue new programs points to a potentially important role for the type of work DtWI does; (c) the conversations we’ve had with government officials are broadly (though not unequivocally) consistent with the idea that DtWI has raised the probability of dewormings being implemented.

Details on what we know:

  • Existing deworming programs in India and DtWI’s role.
    • DtWI and the Children’s Investment Fund Foundation (CIFF) have told us that the Indian government mandates that all states provide school-based deworming through the larger school-based Weekly Iron and Folic Acid Supplementation program, and that states can request funding for deworming through this program.68 At the end of 2013, the only states in India that appear to have been implementing school-based deworming programs were Andhra Pradesh, Bihar, Delhi, Jharkhand, Punjab, and Rajasthan.69 Excluding Punjab and Jharkhand (where it was not involved), DtWI states that it (or others affiliated with it) played a key role in each of these states’ decisions to implement deworming.70 In 2014, DtWI believes that more states are starting to launch their own deworming programs without assistance from DtWI, but is unsure how many have successfully done so.71
    • Without involvement from DtWI, Assam initiated a vitamin A supplementation and deworming mass drug administration (MDA) in the 2010-2011 school year, but reported low and conflicting coverage for that year and no apparent deworming MDAs subsequently.72 In the 2012-2013 school year Assam planned to implement a deworming program through the School Health Program, although it was delayed until at least March 2013.73 However, Assam did approach DtWI in late 2013 to begin a program with DtWI's assistance; we are not aware of any further deworming progress in Assam.74
    • On our site visit to DtWI in Rajasthan, India, we spoke with three government officials who were involved in the deworming program.75 Of these, two stressed the importance of DtWI’s technical assistance, and the person we perceive as having been most responsible (of the three) for the decision to go forward with deworming gave the impression that the availability of this technical assistance had been a key factor in deciding to go forward. It should be noted that one of the three gave the impression that DtWI’s help was not needed, and all three conversations took place with multiple DtWI representatives present. With that said, the highest ranking of the three officials gave what we felt to be a nuanced and realistic picture of DtWI’s impact that implied a substantial (while not determinative) role. She stated that the immediate availability of technical assistance improved her confidence that the program would proceed quickly and smoothly; that she wasn’t sure whether the program would have proceeded if not for DtWI; and that there is at least one category of other program that she would very much like to find a nonprofit technical assistance partner for, supporting the idea that nonprofit technical assistance can be a key factor in getting a program to go forward.76
  • Reasons new health programs may be started or halted in India.
    • DtWI and CIFF have told us that Indian state governments often receive funds earmarked for broader health programs, but often fail to spend these funds.77
    • We asked the Center for Global Development for a recommendation of someone to talk to in order to understand states’ decisions to initiate deworming programs, and CGD referred us to Professor Devesh Kapur.78 Dr. Kapur asserted that it takes significant internal political will or external stimulus (such as from a non-profit) to sufficiently overcome general bureaucratic inertia in India to sustain a new health program. This holds true even in cases where a national mandate exists for a program (as it does for school-based deworming). Dr. Kapur felt that in the majority of cases, nonprofit technical assistance was likely to increase the probability of a program’s going forward.79
    • DtWI told us that it believes that some school health programs have been stalled in Indian states due to negative media attention;80 one of DtWI’s goals is to prevent these reports in the states in which it works.81 The documentation that DtWI has sent us supports -- but does not fully demonstrate -- this interpretation of events in Assam. We have not attempted to independently verify it because we do not think we would likely be able to do so effectively. Nonetheless, it is plausible that were decision-makers in Indian states to have the impression that negative reports could cause a program to be halted, they might be less willing to move ahead, and DtWI’s assurances that it would help prevent these reports could increase the likelihood that a state agrees to implement deworming.

Does DtWI’s work increase the quality of deworming programs?

We have seen limited evidence to convince us that DtWI’s work directly improves the quality of the programs it supports. However, DtWI has made recent changes to its monitoring process that it believes addresses some of our concerns. DtWI shared information about those programs in November 2014 and we will update this review when we have finished evaluating it.82

DtWI may improve program quality by:

  • Increasing the chances that the first deworming round in each state begins earlier than it otherwise would have and that subsequent rounds occur on schedule.83
  • Increasing the training quality by simplifying training material and creating a more robust training program for those who train representatives from each school.84
  • Ensuring that support roles are staffed.85
  • Improving focus and attention to detail, possibly increasing the likelihood that schools receive the materials and instructions necessary to implement the deworming program.86
  • Expanding the scope of the program to a broader age group.87
  • Increasing community acceptance of mass treatment and the ability of a program to withstand publicity associated with adverse events or seemingly related adverse events.88
  • Implementing monitoring systems for both worm prevalence surveying and drug coverage which may improve program quality by creating a mechanism through which implementers are held accountable or by generating lessons that DtWI and the government can use in future rounds of treatment.89

We believe our strongest piece of evidence in favor of the idea that DtWI improves quality is the conversation we had with a Rajasthan nodal officer, who cited many of the points above and made the case that DtWI had played an important role in improving the quality of deworming, backing up an intuition that without external support, such programs would often be significantly disadvantaged. On the other hand, (a) this conversation took place with multiple DtWI representatives present, and the nodal officer had worked closely with DtWI; (b) another government official (on the same visit) stated that he believed DtWI’s technical help was not needed and did not add value, as the state was accustomed to running school health programs and deworming is a relatively simple one. In addition, we have limited information on the quality of India’s deworming programs, and we felt that the training we attended (the only part of DtWI’s work that we observed directly) had major limitations in terms of potential to improve program quality - much of our view that the dewormings are likely carried out appropriately comes down to a belief that the program is relatively simple to execute (we do not put strong weight on (b) alone, since the government official may have had other incentives to give the message he did and had not worked as closely with the program as the Nodal Officer, but in context of the other observations we find (b) worth noting).

Possible negative and offsetting impact

  • Replacement of government funding: We have limited information about whether governments would pay for the parts of the program paid for by DtWI in its absence, though our impression is that they would not.90 We also have little information about what governments would use deworming funds for if they did not choose to implement deworming programs. In India, our understanding is that they might spend those funds on other state-government expenses (not necessarily health programs) or return them to the central government.91
  • Diversion of skilled labor: Drug distribution occurs only once or twice per year and is conducted by teachers in schools. Based on our site visit in Rajasthan, our impression is that the Nodal Officer (the state official who manages all state school-based programs), the Nodal Officer’s staff, and the people that the Nodal Officer manages throughout the state (Resource Persons and Community Development Project Officers) have significant capacity to take on additional programs, so their taking on this program doesn't impose a significant burden on their time.92 On the other hand, a principal we spoke with commented that he would prefer fewer school-based health programs because they take focus away from the school day, and teachers may not have as much capacity to add deworming to their school schedule in the countries to which DtWI expands.93
  • Adverse effects and unintended consequences of taking deworming drugs: Our understanding is that expected side effects are minimal and there is little reason to be concerned that drug resistance is currently a major issue (more information from our report on deworming).

What do you get for your dollar?

This section addresses the following questions:

  • What is the cost per child treated of the deworming programs, and what percentage of this does DtWI bear? We estimate that children are dewormed for a total of about $0.30 per child, or $0.09 per child excluding the value of teachers’ and principals’ time spent on the program. DtWI itself pays about $0.026 per child treated.
  • How much impact should DtWI-supported deworming programs be expected to have (per person treated) compared with the ones on which the evidence for deworming's positive impact on life outcomes is based? We believe that the programs DtWI is involved with are likely to have less impact on a per-person basis, due to relatively low prevalence and intensity of infections where DtWI works.
  • Should we think of DtWI as "leveraging" government funds, such that each dollar it spends mobilizes multiple dollars from other players? We could imagine that DtWI's funds have substantial leverage, but could also imagine that DtWI is covering costs that aren't crucial to the success (in terms of quality, sustainability, and coverage) of deworming and are thus less impactful on a per-dollar basis than the average dollar spent on deworming. Because we believe that DtWI likely increases the probability of a given program's going forward, we believe that it probably does leverage other funds to some extent, though this is difficult to quantify.

For a discussion on how the impact of unrestricted giving to DtWI is expected to differ from simply scaling up its past activities, see our discussion below.

What is the cost per child treated in India?

DtWI has estimated the total cost of each of the last seven deworming rounds in India (three in Bihar and two each in Delhi and Rajasthan). These estimates include the costs listed in the following table. Each of the costs were paid by a combination of government, DtWI, and other partners (percentage breakdown only includes the most recent round of deworming in Bihar, Delhi, and Rajasthan):94

DtWI’s and others’ costs in Indian states’ deworming programs
Cost category Percentage of total costs % paid by DtWI % paid by others
School staff time in training and deworming day 71% 0% 100%
Drug procurement & management 12% 2% 98%
Training & distribution financial costs 7% 8% 92%
Program management 6% 100% 0%
Public mobilization & community sensitization 2% 9% 91%
Prevalence surveys 1% 97% 3%
Monitoring & evaluation 1% 43% 57%
Policy & advocacy 0% 100% 0%
Total 100% 9% 91%

DtWI has estimated some of the above costs because it did not have access to full cost data for governments and other partners, and boundaries between the cost categories are not always clear.95 In particular, the estimated costs of “School staff time in training and deworming day” are a GiveWell-added cost designed to represent the value of the teacher and principal time that is used from the deworming program, despite not creating an additional financial cost for the government (since they are already paid on salary).96

The table below shows the costs of DtWI, its partners, and the governments in Bihar, Delhi, and Rajasthan, as well as estimates of the cost per child treated for each.97 About two thirds of the total cost comes from contributions of time from government employees – which have been monetized according to salary levels – rather than financial costs. We would guess that appropriately valuing that time spent, and estimating how much time is spent by teachers and others due to the deworming program, is the largest source of uncertainty in the cost-per-child-treated calculation.98

Total cost per child treated
Expense category Recent deworming rounds Cost per child
DtWI $771,371 $0.03
Partners $564,050 $0.02
Government financial costs $1,293,090 $0.04
Government staff time value $6,326,595 $0.21
Subtotal without staff time $2,628,511 $0.09
Total costs $8,955,106 $0.30

DtWI estimated that its Kenya deworming program costs around $0.40 per child treated.99 We have not examined the details behind that calculation, since our focus here is on the India program.

DtWI deworming locations vs. key deworming study locations

In this section, we discuss two key differences between DtWI’s deworming programs and the deworming programs in previous studies: (a) the prevalence of particular worms and the prevalence of high intensity infections and (b) the existence of other neglected tropical disease programs that may treat the same condition.

Difference in worm prevalence and intensity

Key pieces of evidence that we discuss in our report on deworming (Miguel and Kremer 2004, Baird et al 2012, and Croke 2014) are from deworming experiments conducted in Kenya and Uganda in the late 1990s and early 2000s. The participants in those studies had relatively high rates of moderate-to-heavy infections of schistosomes or hookworm.100

The situation in India is different. Schistosomiasis is not present in the vast majority of India,101 and hookworm prevalence is minimal in two of the three states in which DtWI has supported deworming programs.102 In the third state, Bihar (more populous than the other two combined), the prevalence and intensity of hookworm infections is still lower than it was in the two key studies.103 In the table below, we compare the total prevalence and moderate-or-heavy prevalence (using definitions from Miguel and Kremer 2004) of Rajasthan, Bihar, and the populations in the Croke experiment and the Miguel and Kremer experiment.104

Worm infections in representative DtWI regions and key deworming studies
Worm (intensity) Rajasthan Bihar Croke 2014 Miguel and Kremer 2004 year 1
Schistosome prevalence (moderate/high) 0% 0% 0% 7.12%
Hookworm prevalence (moderate/high) Not measured 3.70% Not reported 15.4%
Ascaris prevalence (moderate/high) Not measured 8.80% Not reported 15.7%
Trichuris prevalence (moderate/high) Not measured 0.40% Not reported 9.8%
Schistosome prevalence (any) 0.0% 0.0% 0.03% 21.8%
Hookworm prevalence (any) 1.0% 42.2% 55.1% 77.3%
Ascaris prevalence (any) 20.2% 52.1% 3.5% 42.4%
Trichuris prevalence (any) 0.2% 0.2% 1.4% 55.2%

Difference in pre-existing programs

India has an existing program to treat lymphatic filariasis (LF).105 Albendazole, the same drug used to treat STH, is sometimes used to treat LF.106 The treatment for LF in India is annual administration of DEC or DEC and Albendazole107 and the WHO recommends annual albendazole treatment for areas with 20-50% prevalence of STH.108 According to the Indian government's program website, the LF treatment program has high coverage across many states (e.g., 80-90% in states carrying out the program), including Bihar.109 Therefore, it appears that in some cases the effect of DtWI's work may be to transition from annual to biannual treatment in appropriate areas, and we are unsure of how the impact of a such a transition would compare to the impact of transitioning from no treatment to some treatment. Additionally, if the LF treatment and DtWI’s deworming treatment occur within less than six months of each other, leaving a majority of the year with no deworming administration, the effect of the combined programs may be reduced. This issue appears likely in DtWI’s 4th round of treatment in Bihar, which is currently scheduled for February 2015, just two months after the last LF treatment, in order to align with the new national deworming day.110

How leveraged are DtWI funds used for scaleup?

We have written before about the complexity involved in trying to understand leverage in charity, and below we lay out the relevant issues worth considering in the case of DtWI.

Donors to DtWI support only the costs paid by DtWI, which are less than 10% of the overall cost of the program. The role these funds play in the program is an important consideration in determining the cost-effectiveness of donations.

The role DtWI funds play, and the resulting cost-effectiveness, could range between:

  • High leverage, high cost-effectiveness. If DtWI leverages government funds or teacher time that otherwise would not have been spent or would have been spent on a less valuable program by causing the government to run deworming programs or by making those programs significantly better than they otherwise would be. In that case a small contribution from DtWI has a large impact on the number of children receiving treatment and DtWI is highly cost-effective.
  • Small marginal contribution to the quality of the program. If DtWI's involvement doesn't cause more infected children to be reached, then DtWI’s contribution is more appropriately considered a small marginal contribution to the program that would have happened without its help. It is possible in that case that DtWI contributes little to increasing the number of children who receive treatment, but may improve the functioning of the program in less obvious ways.

We would guess that DtWI has played a role in increasing the likelihood that governments conduct deworming programs, while also contributing to the quality of programs. We also believe that it is very unlikely that DtWI was causally necessary for each of the programs it has assisted.

We believe that DtWI's leverage on funds used for scaleup could reasonably range from less than 1x (i.e., programs would take place without DtWI's involvement) to 4-5x. We do not think we have a reasonable way of refining this estimate further, but this footnote provides individual GiveWell staff estimates of the leverage they would guess DtWI has with scaleup funds in India.111

Is there room for more funding?

See our blog post for how DtWI has spent unrestricted money directed to them by GiveWell’s recommendation so far.

The Deworm the World Initiative seeks an additional $1.3 million to support its activities in 2015 and 2016. DtWI expects to spend $377,000 of the $1.3 million (29%) it seeks on work related to expanding school-based mass deworming programs and funding related operating expenses (including impact evaluation related expenses). More specifically, these activities would be:

  • $230,000: staff to support expansion in India, new countries, and related operating and evaluation expenses.
  • $144,000: DtWI overhead. These funds support DtWI as an organization but are not directly programmed (e.g., a portion of Alix Zwane's, the Executive Director of Evidence Action salary, Evidence Action financial staff, etc.).
  • $170,000: implementation support for the integrated deworming, sanitation and hygiene education program in Vietnam, in partnership with Thrive Networks.
  • $500,000: evaluation of new evidence-based programs that leverage deworming. We have limited detail about what this would entail. One idea that DtWI has investigated is the possibility of distributing bednets along with deworming pills in schools as an alternative distribution mechanism to national net distributions. Another is including hand-washing educational programming alongside deworming days. This line item includes $50,000 to support DtWI's evaluation of its hygiene and deworming program funded by Dubai Cares and $50,000 to enable DtWI to hire a senior epidemiologist.
  • $230,000: staff to support evaluation of DtWI’s work in Kenya. This work is primarily funded by CIFF. DtWI believes that additional resources can improved significantly the quality of the analysis done regarding the cost effectiveness of breaking transmission.

Why is DtWI seeking additional funds primarily to support research and evaluation rather than scale-up? What changed in the past year?

In 2014, two events affected DtWI's projection of the additional funding it would require to scale up in India:

  1. CIFF, a major foundation that had supported DtWI's programs in Kenya, agreed to a 6-year, $17.7 million grant to support DtWI's expansion to additional states in India and technical assistance to the Government of India for a national deworming program. At the end of 2013, DtWI believed it was reasonably likely that it would not receive this grant and had not anticipated how quickly it would come through. With these funds, DtWI does not require significant additional funding to support its India expansion.
  2. The new Indian government expressed interest in conducting a single deworming day nationally with increased national attention and resources. Advocating for such a policy and assisting the national government in creating a plan became the major focus of DtWI’s work in India in 2014, which both reduced the amount of time it was able to spend generating interest in heavy DtWI involvement in new states, and also required little funding since there were few costs of that project aside from staff time. DtWI believes that the first national deworming day will likely happen in February 2015.

Together, these changes led DtWI to conclude that funding is no longer the bottleneck to reaching more people in India.

Dr. Zwane believes that DtWI's research agenda is important for two reasons:

  1. She believes it is possible that this research will demonstrate that other approaches to deworming are more cost-effective, such as eliminating worms from areas to avoid the need for mass treatments, or combining deworming with other interventions such as bednet distributions or hygiene education.
  2. She would like DtWI to consistently provide useful information to funders and policymakers and undertaking this research will enable it to continue doing so.

What is the marginal impact of these uses for funds?

Quantifying the marginal impact of such funds would largely be an exercise in guesswork. For funding used for scale-up, we could see a case that most of DtWI’s impact has come from sparking interest in deworming in a few initial states, which has led to more interest over time (in Rajasthan, our impression is that the largest factor in the deworming program’s being started was media coverage of the program in Delhi),112 and that further funds will be far less impactful. We could also see a case that by allowing DtWI flexibility while it scales, unrestricted funds have and will continue to allow DtWI to be more strategic in its scale-up, such as being able to focus on the national program in 2014.

Similarly, for funds DtWI would like to use for its research agenda, we could see a case that not enough will be learned from such projects to significantly improve future deworming activities, making those funds much less cost-effective than scaling up DtWI’s current programs. We could also see a case that by demonstrating the existence (or difficulty) of a higher efficiency deworming model, those funds would leverage future scale-up or funding.113

Deworm the World Initiative and Evidence Action

The Deworm the World Initiative is an initiative led by Evidence Action, run by Executive Director, Alix Zwane. Evidence Action includes other programs, of which the only other large program is Dispensers for Safe Water (DSW).114

Donations to DtWI, even if restricted to DtWI, can change the actions staff take to fundraise (i.e., which grants they pursue, what type of funding they ask for). Therefore, we do not believe donors can realistically donate to DtWI and support only DtWI and not DSW or Evidence Action more broadly. Dr. Zwane has told us that DSW has a funding gap of $2.1 million for 2015.115 Dr. Zwane has told us that she is currently in conversations with donors to close this gap, but those donations are not assured.116

We recommend that donors recognize this fact and support Evidence Action with unrestricted gifts (to reduce the administrative burden placed on Evidence Action); Dr. Zwane has told us that Evidence Action recognizes that funds Evidence Action receives via GiveWell are due to DtWI and are aimed at supporting it to the extent possible.

As with all our recommendations, we will follow Evidence Action’s progress in the future and report publicly on how it spends all funds it receives.

DtWI as an organization

We believe the Deworm the World Initiative, led by Evidence Action to be a strong organization:

  • Track record: DtWI has a track record helping to start deworming programs where they do not already exist and providing ongoing support to programs in India and Kenya.
  • Self-evaluation: We believe DtWI’s self-evaluation to have been less rigorous than that of our other top charities to date, though we are hopeful that it will improve going forward.
  • Communication: DtWI has communicated clearly and directly with us, has given thoughtful answers to our critical questions, and has shared significant, substantive information.
  • Transparency: DtWI is highly transparent. We have not seen it hesitate to share information publicly (unless it had what we consider a good reason).

More on how we think about evaluating organizations at our 2012 blog post.

Sources

Document Source
Alderman et al. 2006 Source (archive)
Alix Zwane conversation August 30th 2013 Unpublished
Alix Zwane conversation June 4th 2013 Source
Alix Zwane, DtWI Executive Director, email exchange with GiveWell, November 2013 Unpublished
Alix Zwane, DtWI Executive Director, phone call with GiveWell, November 2013 Unpublished
Assam 2010 guidelines for deworming Source
Assam midday meal report 2013 Source (archive)
Assam reproductive and child health 2011-2012 Source (archive)
Assam state programme implementation plan 2011-2012 Source (archive)
Baird et al 2012 Source
CIFF conversation September 10th 2013 Source
Croke 2014 Source (archive)
Devesh Kapur conversation October 14th 2013 Source
DSW 2012 GiveWell site visit Source
DtWI 2013 GiveWell government interviews Source
DtWI 2013 GiveWell site visit Source
DtWI Assam research 2013 Source
DtWI Bihar 2011 cost data Source
DtWI Bihar 2011 coverage data Source
DtWI Bihar 2011 monitoring data for deworming day Source
DtWI Bihar 2011 monitoring data for mop-up day Source
DtWI Bihar 2011 Monitoring Form for Deworming Day Source
DtWI Bihar 2011 monitoring form for mop-up day Source
DtWI Bihar 2011 prevalence survey report Source
DtWI Bihar 2011 program report Source
DtWI Bihar 2012 cost data Source
DtWI Bihar 2012 coverage data Source
DtWI Bihar 2012 monitoring data for coverage validation Source
DtWI Bihar 2012 monitoring report Source
DtWI Bihar 2014 cost data Unpublished
DtWI Bihar 2014 program report Source
DtWI class register audits 2013 Source
DtWI cost narrative 2013 Source
DtWI cost per treatment summary 2013 Source
DtWI coverage data 2013 - 2014 Source
DtWI Delhi 2012 cost data Source
DtWI Delhi 2012 coverage data by anganwadi Source
DtWI Delhi 2012 coverage data by school Source
DtWI Delhi 2012 coverage report Source
DtWI Delhi 2012 monitoring data Source
DtWI Delhi 2012 monitoring form deworming day Source
DtWI Delhi 2012 prevalence survey design Source
DtWI Delhi 2012 prevalence survey report Source
DtWI Delhi 2012 program report Source
DtWI Delhi 2013 cost data Unpublished
DtWI Delhi 2013 program report Source
DtWI Monitoring Improvements 2014 Source
DtWI Rajasthan 2012 cost data Source
DtWI Rajasthan 2012 coverage data for anganwadi Source
DtWI Rajasthan 2012 coverage data for schools Source
DtWI Rajasthan 2012 monitoring data for coverage validation in anganwadis Source
DtWI Rajasthan 2012 monitoring data for coverage validation in schools Source
DtWI Rajasthan 2012 monitoring form coverage day Source
DtWI Rajasthan 2012 monitoring form pre-deworming day Source
DtWI Rajasthan 2012 monitoring report Source
DtWI Rajasthan 2012 prevalence survey report Source
DtWI Rajasthan 2013 cost data Unpublished
DtWI Rajasthan 2013 prevalence survey report Source
DtWI Rajasthan 2013 program report Source
Evidence Action 2014 budget Unpublished
Evidence Action 2015 draft budget Unpublished
Evidence Action cover letter 2013 Source
Evidence Action launch announcement 2013 Source (archive)
Evidence Action website 2013 Source (archive)
Evidence Action website announcement April 2014 Source (archive)
GiveWell DtWI 2013-2014 cost data summary Source
GiveWell non-verbatim summary of a conversation with Alix Zwane and Jessica Harrison on November 4th, 2014 Unpublished
GiveWell non-verbatim summary of a conversation with Alix Zwane and Karen Levy on May 14, 2013 Source
GiveWell’s non-verbatim summary of a conversation with Alix Zwane and Grace Hollister on March 17th, 2014 Unpublished
GiveWell’s non-verbatim summary of a conversation with Alix Zwane on December 20th, 2013 Source
GiveWell’s non-verbatim summary of a conversation with Alix Zwane on February 18th, 2014 Source
GiveWell’s non-verbatim summary of a conversation with Alix Zwane on October 23rd, 2014 Unpublished
GiveWell’s non-verbatim summary of a conversation with Grace Hollister on June 24th, 2014 Source
Grace Hollister conversation June 19th 2013 Source
Grace Hollister, DtWI Director, email exchange with GiveWell, October 2013 Unpublished
Harvard Business School Kenya Case Study A 2010 Unpublished
Jessica Harrison, DtWI Associate Director, email exchange with GiveWell, November 2014 Unpublished
JPAL CEAs in education 2011 Source
Kabatereine et al. 2001 Source (archive)
LF treatment coverage 2012 Source (archive)
LF treatment drugs 2012 Source (archive)
Miguel and Kremer 2004 Source
Preventive chemotherapy in human helminthiasis 2006 Source (archive)
Professor Devesh Kapur Biography 2013 Source (archive)
STH coalition framework for action November 2014 Source (archive)
WHO soil-transmitted helminthiases 2012 Source (archive)
World Schistosomiasis Risk Chart 2012 Source

Uncited supplementary materials

Document Source
DtWI Bihar 2011 monitoring forms phase I Source
DtWI Bihar 2012 monitoring data deworming day anon Source
DtWI Bihar 2012 monitoring data mop-up anon Source
DtWI Bihar 2014 program report annex 1 Source
DtWI Bihar 2014 program report annex 2 Source
DtWI Bihar 2014 program report annex 3a Source
DtWI Bihar 2014 program report annex 3b Source
DtWI Bihar 2014 program report annex 3c Source
DtWI Bihar 2014 program report annex 3d Source
DtWI Bihar 2014 program report annex 4a Source
DtWI Bihar 2014 program report annex 4b Source
DtWI Bihar 2014 program report annex 4c Source
DtWI Bihar 2014 program report annex 5a Source
DtWI Bihar 2014 program report annex 5b Source
DtWI Bihar 2014 program report annex 6a Source
DtWI Bihar 2014 program report annex 6b Source
DtWI Bihar 2014 program report annex 7 Source
DtWI Bihar 2014 program report annex 8 Source
DtWI cost per child description Source
DtWI Kenya site visit 2012 Source
DtWI Rajasthan 2012 monitoring data anganwadi deworming day anon Source
DtWI Rajasthan 2012 monitoring data anganwadi mop-up anon Source
DtWI Rajasthan 2012 monitoring data anganwadi prep anon Source
DtWI Rajasthan 2012 monitoring data schools deworming day anon Source
DtWI Rajasthan 2012 monitoring data schools mop-up anon Source
DtWI Rajasthan 2012 monitoring data schools prep anon Source
  • 1

  • 2

    “In January 2007, the [Young Global Leaders] launched the Deworm the World campaign with the goal of improving children’s health and education by massively expanding deworming programs.” Harvard Business School Kenya Case Study A 2010 Pg 7.

  • 3

    The Vietnam program is operated in partnership with another non-governmental organization (NGO), Thrive Networks.

  • 4

    “Deworm the World (DtW) is funded through 2015-2016 in Kenya, thanks to the Children’s Investment Fund Foundation (CIFF), and the Ending Neglected Diseases (END) Fund.”
    Alix Zwane conversation June 4th 2013, Pg 1.

  • 5

    “As such, the states themselves have to make the decision to conduct a deworming campaign; DtW can only encourage that decision by showing that it can be done and offering assistance to help implement the program in a robust fashion that involves intensive monitoring of the program.”
    @Alix Zwane Conversation June 4th 2013@, Pg 2.

  • 6
    • “Six districts were identified within [Andhra Pradesh] where deworming would be piloted. DtW conducted prevalence surveys in those six districts, finding that worms existed in less than 20% of the population in the districts, which is the World Health Organization-recommended threshold for treating all children.” Grace Hollister conversation June 19th 2013, Pg 1.
    • “DtW did two stages of prevalence surveys between August 2010 and February 2011. They found that over 50% of school-aged children had worms, a level at which the World Health Organization (WHO) recommends deworming twice a year, rather than just once a year. Bihar already had a statewide albendazole treatment.” Grace Hollister conversation June 19th 2013, Pg 2.
    • “In 2011 DtW conducted a prevalence survey throughout the National Capital Territory. The average infection rate was below the 20% threshold, although there were large disparities in prevalence between different areas of the city.” Grace Hollister conversation June 19th 2013, Pg 3.
    • “DtW’s prevalence survey found that around 20% of the children were infected with at least one type of STH, particularly in the Western part of the state. Based on elevations and other climatic factors, it is estimated that hookworm is a lot more prevalent in the Eastern part of the state. Taken together, the data led DtW to recommend a mass treatment for the whole state once a year.” Grace Hollister conversation June 19th 2013, Pg 4.

  • 7

    “Andhra Pradesh...
    Deworm the World [Initiative]’s contributions

    • Prevalence survey...
    • Operational support
      • Helped government develop operational plans and budgets
      • Coordinated cross-sectoral partners through the establishment of a State School Health Coordination Committee, bringing together health and education departments and other stakeholders (such as the microfinance partner SKS)
      • Coordinated drug donation made by Feed the Children
      • Designed a monitoring and evaluation (M&E) system
      • Created government tableau for community awareness
    • Trainings
      • Conducted a master training session for program
      • Designed training cascade for the master trainees to train the rest of the implementors
      • Designed training materials
      • Developed materials and campaigns for community sensitization

    …

    Bihar…
    Deworm the World’s contributions to the deworming program in Bihar were similar to those in Andrha Pradesh (see above). In Bihar, DtW coordinated drug donations for Rounds 2 and 3 of the program through the WHO…

      Delhi…
      In addition to the standard contributions (see Andrha Pradesh, above), DtW helped set up a technical secretariat within the School Health Scheme of the Delhi government to support program monitoring. In Delhi, DtW coordinated drug donations for school-age children through Feed the Children.

        Rajasthan…
        DtW’s prevalence survey and recommendation to treat annually thus increased the efficiency of the program significantly, as well as decreasing the required government funding contribution. Additionally DtW successfully encouraged the government to include preschoolers in the program as well. DtW coordinated drug donations for school-age children through the WHO.” @Grace Hollister Conversation June 19th 2013@, Pg 1-4.

      • 8

        GiveWell’s non-verbatim summary of a conversation with Alix Zwane on October 23rd, 2014

      • 9
        • This information is based on deworming programs DtWI ran in 2011 and 2012. Note that in November 2014 we received budgets from deworming programs in 2013 and 2014, but have not included those here yet: GiveWell DtWI 2013-2014 cost data summary
        • Data summary and primary sources in DtWI cost per treatment summary 2013, see DtWI v partners sheet. Includes costs from three rounds of deworming: Bihar 2012, Delhi 2012, and Rajasthan 2012. Excludes data from Bihar 2011 because we do not have a budget breakdown for DtWI for that deworming round.
        • Descriptions of the categories are available in DtWI cost narrative 2013.
        • DtWI notes that these campaign-oriented costs don't include any costs of running DtWI as a whole where they don't pertain to a particular campaign.

      • 10
        • ”DtW has been involved in deworming programs in four different states, and is still actively involved of three of those. Of the states DtW has worked with in the past, none of them had school-based deworming programs before DtW's involvement… Andhra Pradesh… Bihar… Delhi… Rajasthan” Grace Hollister conversation June 19th 2013, Pg 1-4.
        • GiveWell non-verbatim summary of a conversation with Alix Zwane and Jessica Harrison on November 4th, 2014

      • 11
        • ”DtW has been involved in deworming programs in four different states, and is still actively involved of three of those. Of the states DtW has worked with in the past, none of them had school-based deworming programs before DtW's involvement… Andhra Pradesh… Bihar… Delhi… Rajasthan” Grace Hollister conversation June 19th 2013, Pg 1-4.
        • GiveWell non-verbatim summary of a conversation with Alix Zwane and Jessica Harrison on November 4th, 2014

      • 12

        Here is the updated information we received in November 2014

      • 13

        “Innovations for Poverty Action (IPA) is pleased to announce the launch of a new organization created with IPA’s support and dedicated to taking cost-effective programs to scale to improve the lives of millions in Africa and Asia. Evidence Action has been created to bridge the gap between evidence about what works to alleviate poverty around the world and what is actually implemented. The organization scales interventions based on rigorous evidence and crafts resilient business models for long run success.

        Two IPA initiatives that touch millions of people in Africa and Asia – Dispensers for Safe Water and the Deworm the World Initiative – will spin off from IPA to be managed by Evidence Action.” Evidence Action launch announcement 2013

      • 14

        Evidence Action 2015 draft budget

      • 15

        Evidence Action 2015 draft budget

      • 16
        • “INDIA: Risk is limited to the area around Gimvi in Ratnagiri district (Maharashtra) in the hills along the Konkan coast south of Mumbai (approximately 16km from shore).” World Schistosomiasis Risk Chart 2012, Pg 3.
        • GiveWell non-verbatim summary of a conversation with Alix Zwane and Jessica Harrison on November 4th, 2014

      • 17

        Kenya is not included here as it uses a different model that DtWI is not intending to scale up with unrestricted funding.

      • 18

      • 19

      • 20

        "Prevalence and intensity mapping to target treatment is what is recommended by the WHO and what is encouraged by DtW." Alix Zwane conversation June 4th 2013, Pg 3.

      • 21

        Delhi:
        DtWI Delhi 2012 prevalence survey report

        • School and participant selection: "Pots were distributed to 5240 children aged 1-20. The sample was stratified into three groups MCD schools (1600), Government Schools (1600) and Slums (2040) and weighted in its distribution according to the populations and numbers of schools and slums in each district (and zone where appropriate). At all sites 40 children were sampled (with a target of 30 samples returned from each site and a total sample size of 3870)." Pg 1. We are unsure of whether the 40 students in each school were selected randomly in Delhi or by some other process.
        • Participation rate: Table One, on Pg 1, shows that 3668 out of 5240 pots were returned, a participation rate of 70%. In addition, "Of the 3668 children remaining, 441 were dropped from the data set due to missing parasitological data and a further 3 who lacked age data," leaving a sample of 3224 out of 5240 (62%). Pg 2.
        • The report did not provide details of the technique used to analyze the samples.



        Bihar
        DtWI Bihar 2011 prevalence survey report

        • Selection of districts and schools: "Four districts of Bihar were selected to complement the existing STH model and twenty schools, five from each district were randomly selected from the state schools database." Pg 9. For more details see DtWI Delhi 2012 prevalence survey design
        • Selection of students: "Within each school 65 children aged 6 and above representing both sexes equally from class 1 to 6 were randomly selected, class by class, using random number tables." Pg 9.
        • Technique: "Screening of infection for STH was based on a double Kato-Katz smear of 41.7 mg prepared from fresh stool samples." Pg 11.
        • Participation rate: "From a total of 1,281 school children registered in the survey and provided with pots, 1,159 returned samples." (90.5%) Pg 11.



        Rajasthan:
        DtWI Rajasthan 2012 prevalence survey report

        • District selection: "Thirteen districts were randomly selected and compared with secondary data collected for all districts. This information was circulated to the task force constituted to oversee the prevalence survey, who noted a good spread of key factors expected to influence worm prevalence: elevation, population density, and climate.” Pg 6.
        • School selection: “Four schools were randomly selected per district from a complete list provided by the RCEE, excluding those with enrolment lower than 30 students, for a total of 52 schools… The targeted anganwadis were drawn from the one or two anganwadis nearest to the randomly selected school, as a comprehensive database of anganwadis was unavailable.” Pg 6.
        • Participant selection: “35 children (aged 6-15) were selected per school. Selection was undertaken by having children from each grade line up, separated by gender, and selecting students randomly from each queue while attempting to have a 50-50 male-female split. From each anganwadi, 24 children (aged 2-5) were randomly selected and invited to participate.” Pg 7.
        • Technique: “Samples were analyzed by Kato Katz technique using a World Health Organization (WHO) standard template.” Pg 9.
        • Participant selection:The report did not provide details on participation rate, but DtWI reports that it was approximately 56% and is verifying that number. Grace Hollister, DtWI Director, email exchange with GiveWell, October 2013

      • 22
        • This table uses WHO definitions of moderate intensity:
          • "Ascarsis [sic] lumbricoides: 1 - 4,999 epg: light-intensity infections; 5,000 - 49,999: moderate-intensity infections; >50,000 epg: heavy intensity infections
          • Trichuris trichiura: 1 - 999 epg: light-intensity infections; 1,000 - 9,999: moderate-intensity infections; >10,000: heavy-intensity infections
          • Hookworms: 1 - 1,999: light-intensity infections; 2,000 - 3,999: moderate-intensity infections; >4,000: heavy-intensity infections"
          • WHO soil-transmitted helminthiases 2012, Pg viii.
        • Bihar
          • There is a pre-existing LF program that provides albendazole to a large portion of the population in Bihar annually.
          • Based on timing of the decision to deworm in Bihar, there was only time to collect prevalence survey from two districts before the annual LF treatment in 2010. Results from that first stage of surveying were sufficient to convince the state to carry out a full deworming program 6 months after the LF treatment, although the methodology did not meet DtWI's typical standard of quality.
          • Before the DtWI-managed deworming program in early 2011, a second stage prevalence survey was conducted in an additional four states. These states were selected based in part on their complementarity with the original two states. DtWI Bihar 2011 prevalence survey report
          • Since the quality of the second stage was higher, our analysis excludes results from the first stage of the survey. Since the second stage districts were selected in part based on complementarity with the first stage, there may be bias introduced into the statewide estimates from lack of randomness.
          • Grace Hollister, DtWI Director, email exchange with GiveWell, October 2013
        • Delhi
        • Rajasthan
          • The first prevalence survey in Rajasthan was unable to measure prevalence or intensity of hookworm infections. Hookworm prevalence is expected to be significant in Rajasthan, and DtWI conducted a second prevalence survey in the state in September 2013 which is intended to more accurately measure hookworm prevalence. DtWI Rajasthan 2012 prevalence survey report.
          • The results presented here are from DtWI’s second prevalence survey, one year after the first statewide round of deworming. DtWI Rajasthan 2013 prevalence survey report

      • 23

        Preventive chemotherapy in human helminthiasis 2006, Pg 41.

      • 24

        With respect to prevalence greater than 50%, The WHO adds that "If resources are available, a third drug distribution intervention might be added. In this case the appropriate frequency of treatment would be every 4 months" Preventive chemotherapy in human helminthiasis 2006, Pg 41.

      • 25

        See:

      • 26DtWI Bihar 2011 coverage data
      • 27DtWI Bihar 2012 coverage data
      • 28DtWI coverage data 2013 - 2014
      • 29 "In total, 64,724 schools out of the 70,675 targeted schools submitted their summary forms." DtWI Bihar 2014 program report, Pg 27.
      • 30DtWI Delhi 2012 coverage report
      • 31DtWI coverage data 2013 - 2014
      • 32 Also over 100% of pre-schools: "The program targeted 3,032 schools and 10,500anganwadis. As on the cutoff date for report collection, 15 December 2013, data from 603 schools was pending. The above data is based on a dataset comprising 2,417 schools and 10,591 anganwadis." DtWI Rajasthan 2013 program report, Pg 2.
      • 33DtWI Rajasthan 2012 coverage data for schools DtWI Rajasthan 2012 coverage data for anganwadi
      • 34DtWI coverage data 2013 - 2014
      • 35

        The information below focuses on DtWI’s four deworming programs in 2011 and 2012. DtWI has made recent changes to its coverage validation process that it believes addresses some of our concerns. DtWI shared information about those programs in November 2014 and we will update this review when we have finished evaluating it.

      • 36

        Our understanding of how the coverage estimates are created come from rounds in 2012 and may not reflect more recent changes. In Rajasthan, for example, there are 257 blocks for 80,000 schools: "There are 257 blocks in Rajasthan." DtWI 2013 GiveWell site visit, Pg 3. "There are 80,000 public schools in Rajasthan." DtWI 2013 GiveWell site visit, Pg 4.

      • 37

        Our understanding of how the coverage estimates are created come from rounds in 2012 and may not reflect more recent changes. In Rajasthan, for example, there are 257 blocks for 80,000 schools: "There are 257 blocks in Rajasthan." DtWI 2013 GiveWell site visit, Pg 3. "There are 80,000 public schools in Rajasthan." DtWI 2013 GiveWell site visit, Pg 4.

      • 38
        • In Bihar in 2011, monitors visited a randomly selected set of schools on deworming day and mop up day. In one of the three phases of the program, monitors were asked to record the number of children dewormed on that day by asking the teacher conducting the deworming and checking the school’s forms. They were not asked to compare two sources of data. See @DtWI Bihar 2011 monitoring form for deworming day@, DtWI Bihar 2011 monitoring form for mop-up day, DtWI Bihar 2011 monitoring data for deworming day, and DtWI Bihar 2011 monitoring data for mop-up day. DtWI shared the data from this monitoring process with us, but there appears to be a problem with the data we received (the data in the column for “How many children were dewormed today” is yes/no rather than numerical). DtWI Bihar 2011 monitoring data for deworming day, worksheet "Phase 3".
        • In Bihar in 2012, monitors visiting randomly selected schools after the deworming days were asked to record the number of children dewormed according to the class registers and according to the school summary report. The class registers were available for examination in “almost all schools,” while the summary form was available in only 69% of schools. DtWI Bihar 2012 monitoring report, Pg 11.
        • In Delhi in 2012, monitors visiting randomly selected schools after the deworming days were asked to record the number of children dewormed according to the class registers and according to the school summary report. Data was available for comparison for 65% of classes visited. DtWI Delhi 2012 monitoring data
        • In Rajasthan in 2012, monitors visiting randomly selected "independent monitors visited a randomly selected sample of schools and anganwadis over five days – one day before deworming for preparation monitoring, on Deworming Day, on Mop-Up Day, and two days post-deworming allocated for coverage validation." DtWI Rajasthan 2012 monitoring report, Pg 1.

      • 39
        • In the monitoring report on Bihar’s 2012 deworming day, the median number of children dewormed per class according to the class registers was 35, while the median number according to the school summary forms was 38. DtWI Bihar 2012 monitoring report. Our own analysis finds a median of 38 for both, an average of 49 according to the class registers and 50 according to the summary reports, and an inflation rate of 1.5%. (Data missing for 31% of classes visited.)
        • In Delhi in 2012, the median number of children dewormed per class according to the class registers was 88, while the median number according to the school summary forms was 89.5. The average number from the registers was very slightly higher than from the summary forms (128 versus 127). The inflation rate on the summary forms was 1%.(Data missing for 38% of classes visited.)
        • In Rajasthan in 2012, the median number of children dewormed per class according to the class registers was 17, while the median number according to the school summary forms was 18. The average number from the registers was 25 and from the summary forms was 28. The inflation rate on the summary forms was 11.3%. The vast majority of the inflation comes from schools in which the inflation rate is several hundred percent. (Data missing for 20% of classes visited.)
        • Data summarized in DtWI class register audits 2013.
        • DtWI explains, "In those small percentage of cases where [class summaries report significantly higher coverage numbers than school attendance registers], we believe it is not monitor error, but a failure by the school teachers to tick the attendance register. On deworming day Round 2 in Rajasthan, DtWI staff observed that a teacher had directly filled out the school summary form with the number of children dewormed rather than entering it on the register. We speculate, based on our own experiences monitoring the process across all the states, that the main reason for this difference is the teachers either partially fill out the register and then go directly to the summary form, or they simply do not fill out the register on one of the two days. This is a common problem that we try to rectify through the training. In Bihar Round 2, similar to Rajasthan Round 1, there were some schools where the summary form totals were greater than the attendance registrar ticking. Given that the percentage of these errors is not large, and our own staff’s experiences of visiting schools where attendance registers were not ticked (rather the school form was filled out that day itself), we do not find it surprising that this discrepancy exists. We do not believe this discrepancy indicates a systematic bias in the reported deworming numbers." Grace Hollister, DtWI Director, email exchange with GiveWell, October 2013

      • 40

      • 41
        • Bihar 2011:
          • Process: “Moreover, independent monitors were mobilized in each block (560 total) to visit 5% of all government schools on deworming and mop-up days to ensure that an adequate quantity of drugs were available, trained teachers were administrating drugs according to protocol, and community sensitization initiatives were successful in mobilizing non-enrolled children to attend. One auditor per district was in charge of supervising monitors to ensure quality monitoring of all schools.” DtWI Bihar 2011 program report, Pg 9.
          • What we’ve seen: Monitoring forms used for data collection and full monitoring data from deworming day and mop-up day. We have not seen a summary of this data and have not analyzed it ourselves.
        • Bihar 2012:
          • Process: “Independent monitors visited a randomly selected sample of schools over four days – Deworming Day, Mop-Up Day, and two days post-deworming allocated for coverage validation. A multi-stage sampling strategy was used to select the 1216 schools (1.75% of 69,299 schools in Bihar) targeted for monitoring. From each of the 38 districts in Bihar, 2 blocks were selected by simple random sampling, for a total of 76 blocks. 2 clusters were randomly selected from each of the 76 blocks, excluding clusters with fewer than 8 schools. In each of these 152 clusters, 8 schools were randomly selected for a total of 1216 schools. Each monitor was assigned a block comprising 16 schools to be monitored. From this list of 16 schools, the monitor could visit any 4 schools on Deworming Day, another 4 on Mop-Up Day, and 8 more schools over the two Coverage Validation days. Out of the total sample of 1216 schools, monitors were able to survey 1196 schools comprising 302 schools on Deworming Day, 296 on Mop-Up Day, and 598 on Coverage Validation days.” DtWI Bihar 2012 monitoring report, Pg 1.
          • What we’ve seen: Monitoring forms used for data collection, full monitoring data from deworming day, mop-up day, and coverage validation days following deworming, and a summary report of results (DtWI Bihar 2012 monitoring report).
        • Delhi 2012:
          • Process: “Monitoring visits were conducted by senior government officials. Random site visits by independent auditors occurred at a subset of training sessions and participating institutions. Coverage validation engaged independent auditors who conducted random site visits at a representative sample of schools and anganwadis to validate coverage statistics. This source of information was carefully compared with programme reports collated from each school and anganwadi to arrive at an accurate assessment of programme coverage.” DtWI Delhi 2012 program report, Pgs 7-8.
          • What we’ve seen: Monitoring forms used for data collection, full monitoring data from deworming day, mop-up day, and coverage validation days following deworming, and a summary report of results (DtWI Delhi 2012 program report, Pgs 17-22).
        • Rajasthan 2012:
          • Process: “In order to evaluate the efficacy of the deworming protocol and process, independent monitors visited a randomly selected sample of schools and anganwadis over five days – one day before deworming for preparation monitoring, on Deworming Day, on Mop-Up Day, and two days post-deworming allocated for coverage validation. A multi-stage sampling strategy was used to select the 990 schools (1.1% of 90,488 schools in Rajasthan) targeted for monitoring. From each of the 33 districts in Rajasthan, 2 blocks were selected by simple random sampling. In each of these 66 blocks, 15 schools were randomly selected for a total of 990 schools. One monitor was assigned to each of these blocks. From the list of 15 schools in a block, the monitor could visit any 3 schools before Deworming Day for Preparation Monitoring, another 2 schools on Deworming Day, 2 more schools on Mop-Up Day, and 4 more schools over the two Coverage Validation days. The remaining 4 schools served as a buffer in case a particular school could not be visited. Hence, the actual sample size was 726 schools (0.8% of all schools).” DtWI Rajasthan 2012 monitoring report, Pg 1.
          • What we’ve seen: Monitoring forms used for data collection, full monitoring data from deworming day, mop-up day, and coverage validation days following deworming, and a summary report of results (DtWI Rajasthan 2012 monitoring report).

      • 42

        DtWI Rajasthan 2012 monitoring report

      • 43

        DtWI Bihar 2011 program report, Pg 9.

      • 44

        “A multi-stage sampling strategy was used to select the 1216 schools (1.75% of 69,299 schools in Bihar) targeted for monitoring.” DtWI Bihar 2012 monitoring report, Pg 1.

      • 45

        Total schools: “The programme targeted 3,032 schools and 9,934 anganwadis [pre-schools].“ Pg 11.
        Deworming day: “This data analysis is based on 45 schools and 15 anganwadis." DtWI Delhi 2012 program report, Pg 17.

        Mop-up day: “This data was collected from 49 schools and 14 anganwadis." DtWI Delhi 2012 program report, Pg 19.
        Coverage validation: “This data was collected from 80 schools and 87 anganwadis.” DtWI Delhi 2012 program report, Pg 20.

      • 46

        “A multi-stage sampling strategy was used to select the 990 schools (1.1% of 90,488 schools in Rajasthan) targeted for monitoring.” DtWI Rajasthan 2012 monitoring report, Pg 1.

      • 47

        Grace Hollister, DtWI Director, email exchange with GiveWell, October 2013

        “On deworming days, district-level officials from both the Health and Education Departments visited randomly selected schools to confirm that children were being dewormed in accordance with protocols. Both government monitors and the existing cadre of external monitors of the WHO’s polio programme were trained to monitor the school-based deworming programme in Bihar.” DtWI Bihar 2011 program report, Pg 9.

      • 48

        “On deworming days, district-level officials from both the Health and Education Departments visited randomly selected schools to confirm that children were being dewormed in accordance with protocols. Both government monitors and the existing cadre of external monitors of the WHO’s polio programme were trained to monitor the school-based deworming programme in Bihar.” DtWI Bihar 2011 program report, Pg 9.

      • 49

        Grace Hollister, DtWI Director, email exchange with GiveWell, October 2013

      • 50

        "Delhi: For Round 1, we used government officials and monitors from the Health Department who had worked as monitors in the polio program (mostly medical doctors)." Grace Hollister, DtWI Director, email exchange with GiveWell, October 2013

        “Monitoring visits were conducted by senior government officials. Random site visits by independent auditors occurred at a subset of training sessions and participating institutions. Coverage validation engaged independent auditors who conducted random site visits at a representative sample of schools and anganwadis to validate coverage statistics.” DtWI Delhi 2012 program report, Pg 7.

      • 51

        Grace Hollister, DtWI Director, email exchange with GiveWell, October 2013

      • 52

        Grace Hollister, DtWI Director, email exchange with GiveWell, October 2013

        "In order to evaluate the efficacy of the deworming protocol and process, independent monitors visited a randomly selected sample of schools and anganwadis over five days – one day before deworming for preparation monitoring, on Deworming Day, on Mop-Up Day, and two days post-deworming allocated for coverage validation." DtWI Rajasthan 2012 monitoring report, Pg 1.

      • 53

      • 54Monitors asked the principal of each school whether they had received the deworming tablets or not.
      • 55DtWI Rajasthan 2012 monitoring form pre-deworming day, Pg 2.
      • 56DtWI Rajasthan 2012 monitoring form pre-deworming day, Pg 4.
      • 57DtWI Delhi 2012 monitoring form deworming day, Pg 1.
      • 58DtWI Delhi 2012 monitoring form deworming day, Pg 4.
      • 59Monitors were instructed to visit one or more classrooms while they were administering the deworming tablets and answer a handful of questions about the process. The monitors were not required to watch the entire process for each class visited, or every class in each school, so when the monitors report that they did not observe any sick children receiving the tablet, they are not reporting that no sick children received the tablet in that class or school.
      • 60DtWI Delhi 2012 monitoring form deworming day, Pg 5.
      • 61DtWI Rajasthan 2012 monitoring form coverage day, Pg 5.
      • 62DtWI Rajasthan 2012 monitoring form coverage day, Pg 5.
      • 63

        DtWI Bihar 2011 monitoring data for deworming day

      • 64

      • 65
        • Grace Hollister, DtWI Director, email exchange with GiveWell, October 2013
        • GiveWell non-verbatim summary of a conversation with Alix Zwane and Jessica Harrison on November 4th, 2014

      • 66
        • Due to a problem with the first prevalence survey in Rajasthan (conducted before the first state-wide round of deworming), there have now been two surveys completed there, with the second survey (conducted a year later, right before the second round of deworming) showing substantially higher prevalence of the primary worm in the state, Ascaris. DtWI believes this does not represent an increase of worm prevalence in the state but instead reflects the poor quality of the first survey.
        • DtWI Rajasthan 2013 prevalence survey report

      • 67
        • Due to a problem with the first prevalence survey in Rajasthan (conducted before the first state-wide round of deworming), there have now been two surveys completed there, with the second survey (conducted a year later, right before the second round of deworming) showing substantially higher prevalence of the primary worm in the state, Ascaris. DtWI believes this does not represent an increase of worm prevalence in the state but instead reflects the poor quality of the first survey.
        • GiveWell non-verbatim summary of a conversation with Alix Zwane and Jessica Harrison on November 4th, 2014
        • DtWI Rajasthan 2013 prevalence survey report

      • 68

        "National funds for deworming are part of the [Weekly Iron Fortification Supplements] WIFS program and the national government mandates that states should adopt the WIFS program. Indian national policy indicates that deworming should take place biannually." DtWI 2013 GiveWell site visit, Pg 12.

          "Wherever possible, DtWI works to get deworming added as a line item in the budget. Otherwise, it is difficult to ensure that long-term funding will exist for the program." DtWI 2013 GiveWell site visit, Pg 10.

            "India is an appealing place to implement a deworming program because the Indian government has policies in place mandating deworming and makes money available to states to implement deworming programs, but many states don't have the background or the expertise to implement deworming programs effectively." CIFF conversation September 10th 2013, Pg 2.

          • 69
            • Alix Zwane conversation August 30th 2013
            • “Last year Jharkhand launched a deworming as part of the WIFS Program, without any direct advocacy from DtWI. The program had no formalized protocols, didn’t measure coverage, and focused on children aged 10 to 19. In 2014 DtWI has been meeting with relevant officials in the state to see if there is an opportunity to improve the quality of the program with technical assistance. DtWI is hoping to sign a Memorandum of Understanding (MOU) with the state soon, and its work there would be funded by a grant from USAID.” GiveWell’s non-verbatim summary of a conversation with Alix Zwane and Grace Hollister on March 17th, 2014, Pg 3.
            • Assam has also announced its intentions to move forward with its deworming program in March 2013: "Status of School Health Programme with special focus on provision of micronutrients, Vitamin-A, de-worming medicine, Iron and Folic acid, Zinc, distribution of spectacles to children with refractive error and recording of height, weight etc. As reported by NRHM, Assam any [sic] school has not been covered under School Health Programme during 2012-13. The programme is being implemented from March/2013. They have completed the training of Multipurpose Worker (MPW) and Lady Health Visitor (LHV) for the purpose. Recruitment of dedicated Medical officer, Dental Surgeon and Block Health Programme officer have been made for implementation of the programme. The weekly Iron and Folic Acid Supplementation Programme among the adolescent students of Class VI to VIII is also being implemented from the March/ 2013. The training programme for District trainers of all the districts have been completed in Dec./12." @Assam mid day meal report 2013@, Pg 11.
            • According to DtWI, "No schools in Assam have been covered under the School Health Program to provide deworming in 2012/13. The programme was then re-scheduled to begin in March 2013. No evidence of this actually having taken place." DtWI Assam research 2013, Pg 1.
            • "[DtWI] are also widely acknowledged by the deworming community to be the only technical assistance available in India." CIFF conversation September 10th 2013, Pg 2.
            • Originally DtWI didn’t believe that Jharkhand conducted a school-based deworming MDA in 2013, but learned of it by early 2014.
              • Alix Zwane conversation August 30th 2013
              • GiveWell’s non-verbatim summary of a conversation with Alix Zwane and Grace Hollister on March 17th, 2014, Pg 3.

          • 70
            • “Last year Jharkhand launched a deworming as part of the WIFS Program, without any direct advocacy from DtWI. The program had no formalized protocols, didn’t measure coverage, and focused on children aged 10 to 19. In 2014 DtWI has been meeting with relevant officials in the state to see if there is an opportunity to improve the quality of the program with technical assistance. DtWI is hoping to sign a Memorandum of Understanding (MOU) with the state soon, and its work there would be funded by a grant from USAID.” GiveWell’s non-verbatim summary of a conversation with Alix Zwane and Grace Hollister on March 17th, 2014, Pg 3.
            • "DtW has been involved in deworming programs in four different states, and is still actively involved of three of those. Of the states DtW has worked with in the past, none of them had school-based deworming programs before DtW's involvement." Grace Hollister conversation June 19th 2013, Pg 1.
            • "In 2009, DtW and the World Bank had conversations with the Chief Minister of Andhra Pradesh, in which they advocated for a broad school-based deworming program, which hadn't happened before in the state. In a public announcement with health and education ministers following this interaction, the Chief Minister announced the plan to do so, and deworming became the flagship of the state’s school health program." Grace Hollister conversation June 19th 2013, Pg 1.
            • "In January 2010 the Jameel Poverty Action Lab (J-PAL) hosted a regional development and policy conference, at which evidence on school-based deworming was presented, as well as experiences from Andhra Pradesh. Immediately following the conference, discussions started among the state of Bihar, J-PAL, led by members of the DtW Board of Directors, and DtW about the possibility of a deworming initiative there. In August a memorandum of understanding (MoU) was formalized between DtW and the relevant players in Bihar (School Health Society Bihar and Bihar Education Project Council) for program implementation." Grace Hollister conversation June 19th 2013, Pg 2.
            • "DtW leveraged its networks to engage the support of the Minister of Health, Women, and Child Welfare [in Delhi]. Education stakeholders were also brought in so that the program could reach beyond preschool to older grades as well. DtW conducted a great deal of advocacy, maintaining continuous interactions and significant support among all relevant departments to bring the program to fruition Due to the complicated agency system, there were many other government stakeholders as well, requiring a high degree of coordination by DtW to ensure program objectives could be achieved." Grace Hollister conversation June 19th 2013, Pg 3.
            • "After the deworming program launched in Delhi, Rajasthan saw the results generated by the DtW- supported program in Delhi and committed to doing a deworming program, allocating funding for it in their budget. In March 2012 they brought DtW in to help. In this case, the state already knew what it wanted and already had a school health program. They sought DtW's technical expertise, mapping ability, general program support, stakeholder coordination, etc. DtWI coordinated signature of a MoU between the Departments of Women and Child Development, Education, and Health, UNICEF and DtW to guide program implementation, and helped establish of a technical secretariat housed within the Education Department." Grace Hollister conversation June 19th 2013, Pg 3.
            • "DtW has worked in Delhi, Bihar, Rajasthan, and Andhra Pradesh (AP).
              • In Bihar, according to documentation provided by DtW, the program began because of previous Poverty Action Lab (J-PAL) work in Bihar. DtW also noted that Rajasthan approached them because of the success (as highlighted in the media) of the Delhi program. DtW supported a pilot program and prevalence survey in AP, and though DtW didn't continue to engage with AP, the AP government seems to have recently conducted a school-based deworming program.
              • More recently, Punjab approached DtW, but DtW didn't have the funding to move forward with Punjab, and Punjab is now implementing deworming on its own.

              It's not possible to be certain whether or not DtW has caused deworming to happen that otherwise would not have, but it's important to note that DtW took the opportunities presented to help run better programs where possible as well as improve data quality and did so on a limited budget. They are also widely acknowledged by the deworming community to be the only technical assistance available in India." CIFF conversation September 10th 2013, Pg 2.

          • 71

            GiveWell non-verbatim summary of a conversation with Alix Zwane and Jessica Harrison on November 4th, 2014

          • 72

            "Keeping in view the problem of worm infestation of children in Assam it has been decided to conduct two rounds of De-worming every year in the month of March and September. The first round will be conducted during March’ 2010. On every Wednesday (VHND/Immunization Day) medicine for De-worming will be given to all children between the age group 1 to 5 years along with vitamin A supplementation." Assam 2010 guidelines for deworming, Pg 1.

              976,192 "children below 5 years provided vitamin A syrup" as of November/December 2010 Assam reproductive and child health 2011-2012, Pg 73.

                The National Rural Health Mission reported the progress for 2010-2011 as 92,957 "Students given IFA/ de-worming tablets". Assam state programme implementation plan 2011-2012, Pg 21.

                  Assam is "supposed to provide deworming to all children, but AWC services provided in Assam are some of the worst performers for deworming coverage."
                  DtWI Assam research 2013, Pg 1.

                • 73

                  "Status of School Health Programme with special focus on provision of micronutrients, Vitamin-A, de-worming medicine, Iron and Folic acid, Zinc, distribution of spectacles to children with refractive error and recording of height, weight etc. NB. As reported by NRHM, Assam any school has not been covered under School Health Programme during 2012-13. The programme is being implemented from March/2013. They have completed the training of Multipurpose Worker (MPW) and Lady Health Visitor (LHV) for the purpose. Recruitment of dedicated Medical officer, Dental Surgeon and Block Health Programme officer have been made for implementation of the programme. The weekly Iron and Folic Acid Supplementation Programme among the adolescent students of Class VI to VIII is also being implemented from the March/ 2013. The training programme for District trainers of all the districts have been completed in Dec./12." @Assam mid day meal report 2013@, Pg 11.

                • 74

                  Grace Hollister, DtWI Director, email exchange with GiveWell, October 2013

                • 75DtWI 2013 GiveWell government interviews
                • 76

                  "If DtWI had not been there, how would the deworming program be different?

                  • Since DtWI have already implemented deworming programs in Delhi and Tamil Nadu, they brought experience.
                  • If a state government decides to do something, nothing is impossible. However, DtWI's presence helped Rajasthan to roll out the program quickly and error-free. If the government did not have DtWI’s experience, there could have been more problems.
                  • The government received important support from DtWI, particularly with distributing the deworming tablets, implementing the trainings, and developing training materials."

                  "Have you ever wanted to do a program but been unable to find a technical partner?

                  • Ms. Gupta is still looking for a technical partner on specific issues. For example, Rajasthan has a large population with special needs. It has generally been able to find partners for helping the visually impaired, but very few organizations work with autistic children, children with cerebral palsy, or children with mental retardation. If there were a partner to support the Rajasthan government in this area, this would be very welcome. The Rajasthan government has funds for this type of program, but are looking for a technical partner.
                  • An NGO called Sight Savers works with blind children. Before Sight Savers, she was not aware of problems with low vision. Many children need magnifying glasses and large print books. Sight Savers helped them to identify doctors and hospitals to work with such children."

                  DtWI 2013 GiveWell government interviews, Pg 7-9.

                • 77

                  "In India, DtW facilitates the expenditure of resources that the states already have available." Alix Zwane conversation June 4th 2013, Pg 2.

                  "India is an appealing place to implement a deworming program because the Indian government has policies in place mandating deworming and makes money available to states to implement deworming programs, but many states don't have the background or the expertise to implement deworming programs effectively." CIFF conversation September 10th 2013, Pg 2.

                • 78

                  Professor Devesh Kapur Biography 2013

                • 79

                  Devesh Kapur conversation October 14th 2013

                • 80

                  "Some school health programs, such as the WIFS program in Delhi, are rolled out too quickly and their trainings are carried out poorly. Then they receive negative media attention because of students’ adverse reactions to treatments. Negative media can cause delays or cancellation of school health programs. The WIFS program has also received negative media attention in the states of Haryana and Odisha." DtWI 2013 GiveWell site visit, Pg 10.

                • 81

                  "There have been very few adverse reactions to deworming treatments in DtWI-supported states because of the quality of DtWI’s trainings." DtWI 2013 GiveWell site visit, Pg 10.

                • 82

                • 83

                  "DtWI's presence helped Rajasthan to roll out the program quickly and error-free. If the government did not have DtWI’s experience, there could have been more problems." Veenu Gupta, Principal Secretary to School Education Department, Government of Rajasthan DtWI 2013 GiveWell government interviews, Pg 7.

                • 84

                  "DtWI has helped to improve deworming trainings. Last year, DtWI helped to develop the content for the trainings. DtWI made the content more concise and easy to understand than the government would have done on its own. DtWI also improves trainings by tracking when people do not show up to trainings and following up with them and by determining aspects of training that could be improved in future years." Girish Bharbwag, Nodal Officer in Rajasthan DtWI 2013 GiveWell government interviews, Pg 4.

                • 85

                  The District Coordinator position, for example, is not always staffed in other school-based health programs. "The state government paid for District Coordinators for the hand washing program in 2008. However, the payment for this position was low. In the first year, there were 15-16 independent monitors. This year, there are only 7 District Coordinators. The government is finding it difficult to fill these vacancies." DtWI 2013 GiveWell government interviews, Pg 5.

                  "District Coordinators (temporary DtWI employees that play a monitoring and evaluation role) are important because they provide reliable feedback to the government about any problems with the deworming program. Typically, the government must rely on government officers to monitor school health programs. However, these officers often fix any problems that they see and then do not report them to the state government because they are worried that the existence of problems will reflect negatively on them. District Coordinators hired and managed by non-governmental organizations are more likely to report problems.
                  The presence of District Coordinators, combined with the independent monitors hired by DtWI that were known to show up unannounced to inspect the program, makes everyone more careful and more likely to implement the program properly because they know that people are paying attention and that they will receive feedback about any mistakes that they make." DtWI 2013 GiveWell government interviews, Pg 4.

                • 86

                  "The presence of District Coordinators, combined with the independent monitors hired by DtWI that were known to show up unannounced to inspect the program, makes everyone more careful and more likely to implement the program properly because they know that people are paying attention and that they will receive feedback about any mistakes that they make." DtWI 2013 GiveWell government interviews, Pg 5.

                • 87
                  • In Bihar, DtWI says that it has convinced the state to expand its program to include preschool-aged children in the next round of deworming, set for early 2015. GiveWell non-verbatim summary of a conversation with Alix Zwane and Jessica Harrison on November 4th, 2014
                  • The MDA that Jharkhand implemented before it was in contact with DtWI only targeted 10-19 year olds: “Last year Jharkhand launched a deworming as part of the WIFS Program, without any direct advocacy from DtWI. The program had no formalized protocols, didn’t measure coverage, and focused on children aged 10 to 19. In 2014 DtWI has been meeting with relevant officials in the state to see if there is an opportunity to improve the quality of the program with technical assistance. DtWI is hoping to sign a Memorandum of Understanding (MOU) with the state soon, and its work there would be funded by a grant from USAID.” GiveWell’s non-verbatim summary of a conversation with Alix Zwane and Grace Hollister on March 17th, 2014, Pg 3.

                • 88

                  Grace Hollister, DtWI Director, email exchange with GiveWell, October 2013

                • 89

                • 90
                  • "There's limited data available on current access to deworming in India because very few prevalence surveys have been done and because the deworming that does occur is not always reported, or, if it is, state-wide data is difficult to access. The poorest states are unlikely (in CIFF's view) to have the capacity to implement evidence-based statewide deworming programs on their own. CIFF notes that many parts of India are extremely poor with high percentages (60%) of the population practicing open defecation; limited access to sanitation services makes it likely that deworming is needed." CIFF conversation September 10th 2013, Pg 2.
                  • "District Coordinators (temporary DtWI employees that play a monitoring and evaluation role) are important because they provide reliable feedback to the government about any problems with the deworming program. Typically, the government must rely on government officers to monitor school health programs. However, these officers often fix any problems that they see and then do not report them to the state government because they are worried that the existence of problems will reflect negatively on them. District Coordinators hired and managed by non-governmental organizations are more likely to report problems. The presence of District Coordinators, combined with the independent monitors hired by DtWI that were known to show up unannounced to inspect the program, makes everyone more careful and more likely to implement the program properly because they know that people are paying attention and that they will receive feedback about any mistakes that they make. The District Coordinators and DtWI’s tele-callers were valuable because they were able to confirm that schools received the appropriate amount of drugs and that teachers had been trained. DtWI called a random sample of 8,000 schools.
                    The prevalence survey would not have happened without DtWI’s support." DtWI 2013 GiveWell government interviews, Pg 5.

                • 91

                  "India is an appealing place to implement a deworming program because the Indian government has policies in place mandating deworming and makes money available to states to implement deworming programs, but many states don't have the background or the expertise to implement deworming programs effectively. It is possible that DtW brings its expertise to the table and leverages these available resources." CIFF conversation September 10th 2013, Pg 2.

                • 92

                  "RPs tend to have enough capacity that adding further school health programs would not take away from the work they do for other school-based health programs." DtWI 2013 GiveWell site visit, Pg 3.

                • 93

                  "[The Nodal headmaster said] that most aspects of the program are excellent, but he had 2 suggestions:

                  • Deworm students in private schools as well (even though they have more money and can often buy treatment, they will often not do so)
                  • Reduce the number of health programs throughout the year; it takes away from teaching time. His school has school health programs on 40 to 42 days each year." DtWI 2013 GiveWell site visit, Pg 6.

                • 94

                  GiveWell DtWI 2013-2014 cost data summary

                • 95

                  Alix Zwane conversation August 30th 2013

                • 96

                  We estimated the staff time costs based on DtWI’s estimate of similar costs from the same states in 2012. DtWI had imputed those costs based on estimates of government employee salaries, for example, in Rajasthan it estimated 300,000 teacher- and principal-days were used in deworming day and mop-up day, and valued that time at 150 rupees per day (about $2.50).

                • 97
                  • For DtWI’s 2013 and 2014 budgets, DtWI stopped estimating the financial value of government employees’ time spent on the deworming program after the deworming rounds in 2012: “We have consulted with JPAL on the costing model approach and they have suggested, in line with papers they've done, not to quantify teachers' time nor principals' time for the actual deworming day; we still have included the cost of teachers attending the training since there is a direct cash transfer to the teachers for their coming to the training.” Jessica Harrison, DtWI Associate Director, email exchange with GiveWell, November 2014
                  • DtWI cited JPAL CEAs in education 2011 for this approach.
                  • The change is most relevant for teachers and principals, large numbers of whom attend a training and administer the pills to children. While we have seen direct costs of paying teachers and principals that attend deworming training, we are unsure whether those costs should be counted as per-diems (perhaps necessary for some trainees to pay for transport, but not accounting for the lost work time at school) or as optional extra wages (that don’t result in lost work time at school and are sufficient to incentivize the necessary labor), or something in between.
                  • In order to include the value of teachers and principals’ time for the three recent rounds, we have copied the relevant expenses from the prior year’s budget of the same state (which were imputed costs) and replaced the allowances that were paid directly to teachers and principals during trainings. We believe this methodology is likely to result in some inaccuracies and may revise it as we become confident of a more accurate method.

                • 98

                  We estimated the staff time costs based on DtWI’s estimate of similar costs from the same states in 2012. DtWI had imputed those costs based on estimates of government employee salaries, for example, in Rajasthan it estimated 300,000 teacher- and principal-days were used in deworming day and mop-up day, and valued that time at 150 rupees per day (about $2.50).

                • 99

                  "We note that preliminary estimates of program costs in Kenya, which we can estimate with far greater precision, suggest programming costs there of about $0.40 per child." Evidence Action cover letter 2013, Pg 2.

                • 100

                • 101

                  "…absent from most of the country, risk exists only in restricted areas." World Schistosomiasis Risk Chart 2012, Pg 1.

                    "INDIA - Risk is limited to the area around Gimvi in Ratnagiri district (Maharashtra) in the hills along the Konkan coast south of Mumbai (approximately 16km from shore)." World Schistosomiasis Risk Chart 2012, Pg 3.

                  • 102

                  • 103

                  • 104
                    • Data from Miguel and Kremer 2004 (Table II, Pg 168.) and Croke 2014 is provided in this table for comparison, as these papers represent what we believe to be the strongest pieces of evidence of long term damages from intestinal parasites (see discussion in this section). We provide data from year one of the Miguel and Kremer study, which is the time period before el nino weather conditions set in and increased the prevalence and intensity of local infections.
                    • The table uses Miguel and Kremer 2004 definitions of moderate intensity, which are not the same epg levels as WHO definitions for all worms.
                    • "Following Brooker, Miguel, et al. (2000), thresholds for moderate infection are 250 epg for Schistosomiasis. mansoni and 5,000 epg for Roundworm, the WHO standards, and 750 epg for Hookworm and 400 epg for Whipworm, both somewhat lower than the WHO standard." Miguel and Kremer 2004, Pg. 167.
                    • We also have worm infection data from Delhi, but do not include it here as DtWI has told us that Delhi had lower prevalence than a region it would normally prioritize for mass school-based deworming, but that it has particular political and cultural significance as the capital of India. Additionally, the deworming round in 2013 is the last for which DtWI is expecting to assist Delhi significantly.
                    • Croke 2014:
                      • Kabatereine et al. 2001, Pg 284, Table 1, Average for eastern districts.
                      • Kabatereine et al. 2001 was a disease mapping study conducted in southern Uganda in 1998, two years before the study underlying Croke 2014 began, i.e. these are not results from the individuals enrolled in the study underlying Croke 2014.
                      • Note that the districts in the eastern district average in Kabatereine et al. 2001 are not an exact match for the districts included in Croke 2014, though four of the five districts included in Croke 2014 (Busia, Iganga, Mbale, and Tororo) are included in the Eastern district average.
                      • The districts for Croke 2014 were selected because Kabatereine et al. 2001 had found high worm prevalence in eastern districts: "Five districts in the eastern region of Uganda were selected (Busia, Iganga, Mbale, Palissa, and Tororo) because a survey had indicated that about 60% of children aged 5-10 years were infected with intestinal nematodes, most commonly hookworm." Alderman et al. 2006, pg 1.
                    • Bihar
                      • DtWI Bihar 2011 prevalence survey report
                      • There is a pre-existing LF program that provides albendazole to a large portion of the population in Bihar annually.
                      • Based on timing of the decision to deworm in Bihar, there was only time to collect prevalence survey from two districts before the annual LF treatment in 2010. Results from that first stage of surveying were sufficient to convince the state to carry out a full deworming program 6 months after the LF treatment, although the methodology did not meet DtWI's typical standard of quality.
                      • Before the DtWI-managed deworming program in early 2011, a second stage prevalence survey was conducted in an additional four states. These states were selected based in part on their complementarity with the original two states. DtWI Bihar 2011 prevalence survey report, Pg 9.
                      • Since the quality of the second stage was higher, our analysis excludes results from the first stage of the survey. Since the second stage districts were selected in part based on complementarity with the first stage, there may be bias introduced into the statewide estimates from lack of randomness. (Grace Hollister, DtWI Director, email exchange with GiveWell, October 2013)
                    • Rajasthan

                  • 105

                    The National Vector Borne Disease Control Programme LF treatment coverage 2012

                  • 106

                    "DEC + Albendazole in selected distt & DEC in other distt" LF treatment drugs 2012.

                  • 107

                    "Annual Mass Drug Administration with single dose of DEC was taken up as a pilot project covering 41 million population in 1996-97 and extended to 74 million population. This strategy was to be continued for 5 years or more to the population excluding children below two years, pregnant women and seriously ill persons in affected areas to interrupt transmission of disease.
                    Interruption of transmission of filariasis by Annual MDA for 5 years or more to the population except:
                    - children below 2 years
                    - pregnant women
                    - seriously ill persons
                    (DEC + Albendazole in selected distt & DEC in other distt)" LF treatment drugs 2012.

                  • 108

                    Preventive chemotherapy in human helminthiasis 2006, Pg 41.

                  • 109

                    The LF program has recently been active in Bihar, but not Rajasthan or Delhi. LF treatment coverage 2012

                  • 110
                    • GiveWell non-verbatim summary of a conversation with Alix Zwane and Jessica Harrison on November 4th, 2014
                    • DtWI notes that it is already in conversations to attempt to arrange for the next LF treatment day in Bihar to be in October 2015 to avoid a long stretch without any deworming in the state. (GiveWell non-verbatim summary of a conversation with Alix Zwane and Jessica Harrison on November 4th, 2014)

                  • 111
                    • Timothy Telleen-Lawton: 3x. I expect that DtWI's leverage value for some states will be about 1x, and for some states will be close to 10x (when they are a critical factor in the program happening at all in a given year). With large uncertainties about how much leverage they deserve credit for even within the states they operate currently, I would guess a geometrically central estimate of 3x.
                    • Elie Hassenfeld: 3x. Very few deworming programs exist in India without DtWI's involvement, and I suspect we'll see the pattern continue in the future. I recognize that this may only be a reflection of the fact that states that decide to deworm seek assistance from DtWI even though they might have chosen to deworm without DtWI's help.
                    • Holden Karnofsky: 3x. I feel roughly the same way as Elie and Tim.
                    • Alexander Berger: 2x. I agree with Tim's general reasoning but my guess is that in most cases, the impact of DtWI funding is less than its proportion (i.e. a multiplier below 1x), but that in some states it garners considerable leverage. I also think that the degree of leverage is likely to decline relatively quickly as the organization grows, and may be less than one at the current margin (i.e. DtWI may be "being leveraged" at this point instead of "leveraging additional funding").

                  • 112

                    "How the deworming program in Rajasthan began:

                    • Ms. Gupta was a Commissioner Associate when she first heard about deworming, via media coverage of the deworming program in Delhi.
                    • After she heard about deworming, she spoke with other staff members about the possibility of doing deworming in Rajasthan. Then they got in touch with DtWI, and DtWI was very responsive." DtWI 2013 GiveWell government interviews, Pg 6.

                  • 113

                    DtWI has told us that there is a trend in the global aid community to transition interventions into more horizontally-integrated programs that implement multiple programs at once. Given the interest in that strategy, demonstrating the efficacy (or lack thereof) of combining mass deworming with complementary programs could be more valuable than it otherwise would be. (GiveWell’s non-verbatim summary of a conversation with Alix Zwane on October 23rd, 2014)

                  • 114

                    "Two programs evaluated and incubated within Innovations for Poverty Action which are currently making a difference in the lives of millions of people in Africa and Asia – Dispensers for Safe Water and the Deworm the World Initiative - are transitioning to Evidence Action." Evidence Action website 2013, homepage.

                      GiveWell conducted a site visit to DSW in November 2012 and published notes. DSW 2012 GiveWell site visit

                        See GiveWell's review on water quality interventions.

                      • 115

                        Evidence Action 2015 draft budget

                      • 116

                        Evidence Action 2015 draft budget

                      Source URL: https://www.givewell.org/charities/deworm-world-initiative/November-2014-review