Evidence Action Beta — Iron and Folic Acid Supplementation ("Phase 2")

Published: April 2019

Note: This page summarizes the rationale behind a GiveWell Incubation Grant to Evidence Action Beta. Evidence Action staff reviewed this page prior to publication.

Background

Evidence Action is the parent organization of one current (as of December 2018) top charity (Deworm the World Initiative) and one standout charity (Dispensers for Safe Water).

Evidence Action has built relationships with Indian policymakers through its work supporting the scale-up of deworming in India.1 This grant would enable Evidence Action to pilot technical assistance for a different government program: weekly iron and folic acid (IFA) supplementation for school-aged children (the Government of India program name is “WIFS”). Evidence Action previously received a $320,000 Incubation Grant for "Phase 1" of this project.

Activities

This $3,408,259 grant would support "Phase 2" of this project, in which Evidence Action would provide technical assistance to one region in each of two states in India over the course of two years, with the goal of substantially increasing coverage in those regions.2

The specific nature of Evidence Action's technical assistance will be determined based on what it learns in the early stages of this grant, but Evidence Action expects that it will involve activities such as aiming to ensure that schools consistently have sufficient supplies of iron and folic acid supplements to implement the program. For more details on Evidence Action's proposed activities, see the following footnote.3 For further details on how Evidence Action chose the states in which to work, see the following footnote.4

Budget

A rough breakdown of Evidence Action's budget for this grant is (quoting from Evidence Action):5

• "Personnel (23%), including team time for project management, monitoring and data collection, technical assistance in two states, state and national government partnership management, and cost-effectiveness analysis
• Data collection, travel and other program costs (12%), including baseline and endline coverage data collection, program monitoring data collection, team travel, printing etc.
• India-related taxes and program administrative costs (17%), including taxes charged at 18% of total direct costs, office utilities, rent, etc.
• Reserves (39%), [Added by GiveWell: one year of reserves was provided to account for a) six months of "bridge" funding while GiveWell collects results from Phase 2 and makes a decision about whether to renew this grant, and b) if GiveWell does not renew, six months of annual budget are included in the current grant amount as an "exit" grant.]
• Indirect costs (9%), consistent with Evidence Action’s organizational policy for core infrastructure costs."

This grant size was chosen with the assumption that a co-funder would be likely to provide an additional $600,000.6

Note that GiveWell may provide some additional funding beyond the above budget to support additional monitoring and evaluation of this grant. GiveWell expects the detailed evaluation plan for this grant to be finalized in early 2019.

Case for the grant

The overall case for this grant is:

• This seems like a strong opportunity for potential large-scale impact. Evidence Action told us that the Indian government expressed interest in having Evidence Action’s technical assistance to improve the IFA program. The IFA program targets a large population and receives substantial resources from the Indian government, so any improvements to the program could help a large number of people.
• There seem to be major issues in the current implementation of the IFA program such as some schools being unaware of the program and not having a sufficient supply of iron and folic acid treatments. Evidence Action has told us that the Indian government is facing challenges in fixing these problems, which could be addressed with its technical assistance.
• Weekly iron and folic acid supplementation in India seems to be in the range of cost-effectiveness of other priority programs in global health and development. Increasing coverage of this program via technical assistance has the potential to be substantially more cost-effective than GiveWell's current top charities.
• We believe that with some additional work we will be in a good position to evaluate whether Evidence Action's technical assistance had an impact at the end of the grant.

More details follow.

Is iron supplementation effective?

We believe there is strong evidence that iron supplementation reduces anemia and weak to moderate quality evidence that it causes cognitive benefits. For more information, see our interim intervention report on this program.

We make a variety of adjustments to apply the independent evidence on the effectiveness of iron supplementation to the IFA program in India. For more details, see below and our cost-effectiveness model.

External validity

The main ways that the programs and contexts supported by Evidence Action seem to differ from the studies covered in our intervention report are:

• India's IFA program is a weekly supplementation program whereas some of the evidence base discussed in our interim intervention report assesses the effects of daily supplementation programs.7 We believe this difference is unlikely to significantly affect our bottom line for the reasons laid out in the following footnote.8
• Malaria prevalence in India is substantially lower than in sub-Saharan Africa, so potential concerns about negative effects of iron on malaria risk seem not to be highly relevant to our cost-effectiveness estimate in India.9
• There are a variety of other factors we have considered when applying the iron supplementation evidence base to estimating the cost-effectiveness of the IFA program in India, such as differences in anemia rates between the relevant populations in India and the participants in studies. For more details, see our cost-effectiveness model.

Note on folic acid supplementation

In this report, we primarily discuss the effects of iron rather than folic acid.

We have not carefully considered the potential benefits of folic acid as distinct from iron. Our understanding is that in some of the iron supplementation studies we rely on, folic acid is also part of the treatment. However, our rough impression from reviewing the folic acid literature is that most of the potentially important public health impacts of folic acid come from providing supplements to pregnant women to reduce birth defects,10 and we expect that pregnant women would be a small portion of beneficiaries targeted by this program.11

To simplify our analysis, we have focused on the effects of iron. However, our analysis would be conservative if the public health effects of folic acid in this population are larger than we currently expect.

How much will Evidence Action increase IFA coverage?

In our cost-effectiveness analysis, our current best guess is that coverage of IFA supplementation will increase by about 6 percentage points per year with Evidence Action support and by about 3 percentage points per year without it. However, this guess is based on qualitative information that could justify a wide range of guesses (more below). Therefore, we see our current best guess as indicative that high cost-effectiveness is possible if Evidence Action succeeds to a plausible degree, but we expect to revise our views based on substantially better information at the conclusion of this pilot project in about two years.

Information that informed our best guess includes:

• Current coverage rates of the IFA program seem to be low (reported to be about 37% at the national level for children in school, and lower than this for children out of school and in many states, according to one government data source),12 and there is some evidence of major issues that could be contributing to low coverage, such as (a) some schools being unaware of the program or having never implemented it (about 60% of schools according to one survey), and (b) not having a sufficient supply of iron and folic acid tablets (about 60% of schools in the same survey).13
• It seems plausible that health programs can achieve high coverage in contexts like India with sufficient support and attention. Estimated coverage rates of deworming in India are currently high (over 80% in all states for which we have assessed data from 2015-2016).14 Coverage rates of all basic vaccinations were estimated as >60% at the national level.15 However, weekly iron and folic acid supplementation may be a relatively challenging program to implement because it is delivered more often than many health programs.
• Evidence Action has a strong track record as an organization overall (as the parent organization of one top charity and one standout charity), so it seems plausible to us that its work in this case could have a large impact. We believe that Evidence Action played a meaningful role in getting the Indian government to implement deworming, so it has relevant experience with technical assistance.16 However, our case study of Evidence Action's impact on deworming is fairly different from the proposed work of improving coverage of the IFA program, so we do not have a strong sense of the track record of this particular kind of technical assistance work.
• Evidence Action has told us that the Indian government is facing challenges in fixing these problems which could be addressed with its technical assistance, and that the Indian government has expressed interest in having Evidence Action's support. We have not independently vetted these statements or spoken with relevant officials, but it seems plausible that government capacity constraints would hinder improvements to the program, and we hope to learn more via later evaluation of this pilot project.

A few reasons to believe that coverage will increase to some extent without technical assistance are:

• Evidence Action told us that this program began in 2013 and that it is receiving a further substantial injection of funds in 2018-2019 as part of a national initiative to reduce anemia in India ("Anemia-Free India").17 The fact that this program has achieved some coverage prior to Evidence Action involvement and that it is receiving additional investment and attention over time suggests to us that various actors in India would increase the coverage of the program to some extent without further external assistance.
• Our rough impression is that, in general, coverage of health programs is improving in developing countries over time, so some degree of continued progress seems likely.18
• We unfortunately do not have detailed estimates of trends in IFA coverage rates since the program launch in 2013-14 that we could use to extrapolate a rough counterfactual. We also have not yet carefully examined pre-existing trends in anemia reduction over time in India, which could be an imperfect proxy for coverage of IFA programs.19

Cost-effectiveness

Our current rough best guess is that, at scale, this technical assistance intervention would be about 15x as cost-effective as cash transfers. Our cost-effectiveness analysis is here.

Note that our cost-effectiveness analyses are simplified models that do not take into account a number of factors. There are limitations to this kind of cost-effectiveness analysis, and we believe that cost-effectiveness estimates such as these should not be taken literally due to the significant uncertainty around them. We provide these estimates (a) for comparative purposes and (b) because working on them helps us ensure that we are thinking through as many of the relevant issues as possible.

Key judgment calls in our analysis include:

• There are a number of difficult judgment calls relevant to the cost-effectiveness of iron supplementation itself (as distinct from technical assistance), such as the magnitude of long-term benefits due to cognitive effects of iron, how valuable it is to avert cases of anemia, and how to account for costs such as teachers' administration of the program. More discussion of these judgment calls is in our interim intervention report, here, and an accompanying document, here.
• Regarding our technical assistance cost-effectiveness model specifically, difficult judgment calls include:
  • Over what timeline should we estimate benefits? Our current model assumes that benefits from technical assistance will accrue over a 6-year time period. This is a fairly arbitrary assumption that is based on the idea that the policy environment may change substantially over the next decade. By funding this intervention soon, we assume that we effectively "speed up" major coverage increases of this program by about six years.20
  • How likely is it that other programs (e.g. iron fortification) lead to major reductions in anemia? It seems reasonable to expect that programs such as iron fortification may substantially reduce anemia, which would reduce the potential impact of the IFA program. We have roughly taken this possibility into account when considering the likely "counterfactual" coverage rate in the absence of technical assistance, but we have not modeled it in detail.
  • Which factors should be included vs. excluded from the model? We have made a number of simplifying assumptions at this stage which could affect our estimates if fully incorporated. For example, we have not included any potential benefits of Evidence Action's technical assistance at the national level, even though a portion of Evidence Action's proposed budget is earmarked for such work. We also have not carefully extrapolated any trends in anemia over time, which may fall in the absence of any intervention.21 For now, we are assuming that these factors are sufficiently likely to offset each other that they would not have likely changed our decision on whether to make this grant.

Note: This model is rough and explicitly does not include some factors which would improve accuracy and consistency in order to save time. For example, it does not apply discounting for costs or benefits that would occur in the future; we expect fixing this would have small and offsetting impacts on our bottom line.

Room for more funding

We have not carefully considered the scale of additional funding that this project could absorb if it is successful, but our rough impression is that there are at least five Indian states which may benefit from technical assistance and that this could cost at least roughly ~$1 million per year per state.22 If five states receive assistance for four years, this work could absorb at least $20 million in additional funding.

GiveWell strategic considerations

We believe that it is highly probable that the funding gaps at highly cost-effective top charities (i.e., 5x as cost-effective as cash transfers as of October 2018)23 will outpace the amount of funding that we are able to direct to top charities in the next few years. Therefore, in order to have more impact, our research will need to find giving opportunities that are more cost-effective than marginal spending at our current top charities. This makes giving opportunities with high upside potential, such as this one, relatively more valuable.

We have formalized the above considerations to some extent in a "value of information" model, suggesting that this grant is roughly 10x as cost-effective as cash transfers.24 However, there are a number of limitations of this model, such as that additional spending may be necessary before we have established that scaling up these activities is highly cost-effective.

Plans for follow-up

Impact evaluation

We recommended this grant with the expectation that we and Evidence Action will be able to agree to a strong plan for evaluating its impact. We are early in the process of determining how best to evaluate technical assistance-oriented grants such as this one. We expect to agree to a final impact evaluation plan in the months following this grant recommendation.

Update (April 2019): GiveWell recommended that Good Ventures make an additional grant of $800,000 to support evaluation of this grant. For more details, see this page.

Other

Other follow-up that we expect to conduct on this grant includes:

• Making a decision about whether to renew and increase support for this work in about two years. This would likely involve improving our understanding of where Evidence Action would do additional work and how it would relate to the activities conducted as part of this grant.
• Based on our analysis of the impact of this grant, we will create a revised forward-looking cost-effectiveness model to estimate the value of giving additional funds.
• We will consider doing a project to assess other efforts to improve coverage of health programs via technical assistance, in order to inform our understanding of the general plausibility of the impact of these kinds of programs. This may involve further evaluation of whether Deworm the World Initiative has increased coverage of deworming in India via its technical assistance.

Major ways we could be wrong

• Our current cost-effectiveness model does not capture the cost of teachers' time in implementing this program, under the assumption that the opportunity cost of that time is likely very low. This may be incorrect, e.g. because supporting IFA supplementation may trade off with delivering other health programs via schools (see Berry et al. 2018).25 Under some assumptions, teachers' time may dominate the costs of implementing this program.26 If we ultimately see this cost as meaningful, this could dramatically change our view on the cost-effectiveness of IFA supplementation.
• Other programs such as iron fortification may scale up in the relevant regions and substantially reduce the impact of the IFA supplementation program. The recent apparent large injection of funds for "Anemia-Free India" may lead to large increases in coverage in the absence of additional support.27 We have not yet examined other efforts to reduce anemia in the relevant regions.
• Our cost-effectiveness model assumes that Evidence Action's projections for its long-run technical assistance budget are correct. Phase 2 of the program seems to have a cost of about $0.65 per person targeted by technical assistance per year while the assumed long-run costs imply spending $0.08 per person targeted by technical assistance per year; therefore the expected direct impacts from Phase 2 of this grant are not cost-effective in their own right (about 2x cash).28 Evidence Action gave us some reasons that it expects costs to fall dramatically without impact falling accordingly (see following footnote).29 We have not yet carefully vetted these points.
• Our current estimates imply that direct delivery of IFA supplementation is extremely cost-effective (about 12x cash).30 Since this is still an early-stage model, we should probably expect some mean reversion to lower levels of cost-effectiveness. Potential sources of error are that we have not yet deeply vetted the cost estimates for the program or the estimates of anemia prevalence in the relevant population.
• Our cost-effectiveness model's best guess implies that achieving an additional person-year of coverage of IFA supplementation via technical assistance costs roughly $0.80 while simply paying for direct delivery of a person-year of IFA supplementation costs roughly $1.00.31 If technical assistance is not much more cost-effective, perhaps it would be more straightforward to instead pay for direct delivery of IFA supplementation for children who are not yet receiving it.

Internal forecasts

For this grant, we are recording the following forecasts:

Confidence	Prediction	By Time
60%	GiveWell’s best guess is that Evidence Action’s intervention increases coverage relative to the counterfactual in the first year of Phase 2 of the program by at least 4 percentage points	December 2021
50%	GiveWell’s best guess is that Evidence Action’s intervention increases coverage relative to the counterfactual in the second year of Phase 2 of the program by at least 8 percentage points (cumulatively)	December 2021
75%	Evidence Action requests funding for Phase 3 of this program because it believes Phase 2 to have been successful	December 2021
80%	Estimates of anemia rates from the India National Family Health Survey in an average of 5 randomly chosen non-Evidence Action-supported states do not show anemia declining by more than 2 percentage points per year over the last 5 years (e.g., due to iron fortification or other changes)	January 2024
35%	Evidence Action ultimately spends at least $15 million total on IFA technical assistance that we retrospectively model as 10x as effective (or more) than cash transfers (using our January 2018 CEA as a baseline)	January 2025

Sources

Document	Source
Anemia-Free India, Operational guidelines	Source (Archive)
Berry et al. 2018	Source (Archive)
Centers for Disease Control and Prevention page on folic acid	Source (Archive)
De-Regil et al. 2011	Source
Evidence Action, IFA cost-effectiveness model	Unpublished
Evidence Action, IFA Data sources and quality	Unpublished
Evidence Action, IFA Phase 2 budget	Unpublished
Evidence Action, IFA Phase 2 timeline	Unpublished
Evidence Action, IFA steady state budget narrative	Unpublished
Evidence Action, IFA steady state government cost narrative	Unpublished
Evidence Action, IFA steady-state budget	Unpublished
Evidence Action, Overview of IFA documents	Unpublished
Evidence Action, State selection note	Unpublished
Evidence Action, unpublished email to GiveWell, October 2, 2018	Unpublished
Evidence Action, WIFS National Concept Note	Unpublished
GiveWell, Deworm the World monitoring summary (2017)	Source
GiveWell, Phase 2 IFA cost-effectiveness model	Source
Government of India, presentation on coverage of IFA program	Unpublished
India National Family Health Survey (NFHS-4) 2015-16	Source
Institute of Medicine, Recommended Dietary Allowances	Source (Archive)
Linus Pauling Institute, Iron	Source (Archive)
Our World in Data, Chart of Child mortality by income level of country	Source (Archive)
Our World in Data, Chart of Prevalence of anemia in children	Source (Archive)
USAID, Idea to Impact report 2015	Source (Archive)

• 1

  More information in our review of the Deworm the World Initiative.

• 2

  For more details on Evidence Action's planned activities under this grant, see "Request for Phase 2 support," Pgs. 7-8, in Evidence Action, Overview of IFA documents

• 3
  • "Evidence Action’s Deworm the World Initiative is the technical assistance provider to India’s National Deworming Day (NDD) programme. NDD was launched in 2015 and scaled rapidly to treat 260 million children by 2017. We propose that, with our technical assistance, the WIFS programme for children and adolescents can reach similar scale. Our proposed model of technical assistance for WIFS includes:
    • Strengthening operational and financial guidelines to increase understanding and adherence among stakeholder departments and provide a clear implementation framework for state governments.
    • Supporting inter-ministerial coordination. The WIFS programme has multiple stakeholders in the Departments of Health, Education, and Women & Child Development. Evidence Action will support the strengthening of existing platforms to allow these stakeholders to work more collaboratively and have greater collective ownership over the program.
    • Strengthening supply chain management.Managing the WIFS programme supply chain is a challenging task, requiring frequent last-mile distribution of large quantities of tablets. Evidence Action will work with stakeholder departments and functionaries to develop a strong supply chain that ensures the regular and sufficient supply of tablets to all schools and AWCs.
    • Streamlining monitoring and supervision to ensure the availability of high-quality data to inform planning, decision-making and programme improvements.●Supporting improvements in areas such as training, day-to-day programme implementation, and community awareness and mobilisation to improve programme delivery and coverage.●Conducting a cost-effectiveness analysis to estimate the cost of the WIFS programme relative to its estimated impact.
    • Leveraging lessons and resources from the national deworming platform, including Evidence Action’s existing relationships and expertise, for the benefit of the WIFS programme.This technical assistance will support the effective delivery and scaling of the WIFS programme to reduce the prevalence of iron deficiency anaemia among hundreds of millions of children and adolescents in India."

    Pg. 1, Evidence Action, WIFS National Concept Note

  • For further details on the above activities, see pgs. 6-10, Evidence Action, WIFS National Concept Note

• 4

  In brief:

  • Evidence Action chose to work in 2 states because it wants breadth of experience and to mitigate risk in case it is unable to operate in one state.
  • It narrowed down from the 29 states and 7 union territories in India to 7 states using 3 criteria: 1) existing Evidence Action technical assistance presence due to deworming programs; 2) existing implementation of WIFS; 3) high population and/or population density (for operations reasons).
  • It narrowed down from 7 states to 4 states based on anemia prevalence data (choosing higher prevalence states).
  • It narrowed down from 4 states to 2 states based on (a) availability of team capacity and strong government relationships, and (b) additional benefit that it will be able to see how deworming coverage changes after it stops providing technical assistance (Rajasthan and Madhya Pradesh have been sufficiently successful that Evidence Action will potentially be phasing out its technical assistance for deworming in those states, pending the results of deworming prevalence surveys and assessment of state capacity for continued program delivery, which will take place in early 2019).

  For more details on the above, see Evidence Action, State selection note.

• 5

  For additional details see Evidence Action, IFA Phase 2 budget and Evidence Action, IFA Phase 2 timeline.

• 6

  We may release further information on this potential co-funder in the future, with permission.

• 7
  • "The 2017 National Health Policy and National Nutrition Strategy build on the existing National Iron Plus Initiative (NIPI), which was launched in 2013. The NIPI holistically addresses both preventive and curative aspects of iron deficiency anaemia across all life stages and at various levels of care.

    The NIPI for children and adolescents aged 5 to 19 is the Weekly Iron and Folic Acid Supplementation (WIFS) programme, which provides weekly supplements of 45 mg of elemental iron and 400 mcg of folic acid for 5 to 9 year-olds, and 100 mg of elemental iron and 500 mcg of folic acid for 10 to 19 year-olds." Pg. 4, Evidence Action, WIFS National Concept Note

  • For more on the evidence base for iron supplementation, see our interim intervention report on this program.

• 8
  • Total body iron content is substantially larger than the daily iron requirement. Once absorbed, iron is recycled and remains in the body for a long period of time. For these reasons, iron status is more sensitive to average intake over weeks and months than day-to-day intake, suggesting that daily and weekly iron supplementation should be similarly effective over periods of months to years, given equal amounts of total absorbed iron. “Total body content of iron in adults is estimated to be 2.3 g in women and 3.8 g in men. The body excretes very little iron; basal losses, menstrual blood loss, and the need of iron for the synthesis of new tissue are compensated by the daily absorption of a small proportion of dietary iron (1 to 2 mg/day).” Linus Pauling Institute, Iron
  • The quantity of iron provided by the IFA program, averaged over the week, is close to the total daily dietary iron requirement (Recommended Dietary Allowance) for the targeted age group. Institute of Medicine, Recommended Dietary Allowances
  • A meta-analysis of randomized and quasi-randomized controlled trials suggests that intermittent iron supplementation, in most cases weekly, is effective for reducing anemia and iron deficiency, although its effectiveness against anemia is lower than daily supplementation. We have not examined the underlying trials to determine if average daily intake was the same in intermittent vs. daily supplementation interventions. “In comparison with receiving no intervention or a placebo, children receiving iron supplements intermittently have a lower risk of anaemia (average risk ratio (RR) 0.51, 95% confidence interval (CI) 0.37 to 0.72, ten studies) and iron deficiency (RR 0.24, 95% CI 0.06 to 0.91, three studies) and have higher haemoglobin (mean difference (MD) 5.20 g/L, 95% CI 2.51 to 7.88, 19 studies) and ferritin concentrations (MD 14.17 µg/L, 95% CI 3.53 to 24.81, five studies).Intermittent supplementation was as effective as daily supplementation in improving haemoglobin (MD -0.60 g/L, 95% CI -1.54 to 0.35, 19 studies) and ferritin concentrations (MD -4.19 µg/L, 95% CI -9.42 to 1.05, 10 studies), but increased the risk of anaemia in comparison with daily iron supplementation (RR 1.23, 95% CI 1.04 to1.47, six studies).” De-Regil et al. 2011, Abstract.

• 9See sections on "Malaria risk" in Sheet "Iron CEA, direct impact" in GiveWell, Phase 2 IFA cost-effectiveness model. For example, in the current iteration of the model (as of October 10, 2018) there are 172.97 "units of value per $10,000" due to iron supplementation, and increases in malaria risk account for -3.7 units of value per $10,000, so affect the bottom line by less than 3%. See Rows 93-99 in the aforementioned sheet.
• 10

  This impression comes from general reading that is not easily cited. However, as corroboration, the Centers for Disease Control and Prevention page on folic acid highlights birth defects as the key health benefit of folic acid: "Folic acid is very important because it can help prevent some major birth defects of the baby’s brain and spine (anencephaly and spina bifida)."

• 11

  According to Evidence Action, this program targets people ages 5-19 years old. According to Evidence Action’s internal calculations, in Rajasthan, there are roughly 60,000 pregnant women 11-19 years old in a total target population of about 15 million people. This was estimated using 2017 UN population data on the numbers of births among women in India aged 15-19. The total in-school target population was estimated using U-DISE data, including only public schools for children ages 5-19 years old. Since the program targets out-of-school girls only, this population was estimated by subtracting census data from school enrollment data.

• 12

  See Slide 13, Government of India, presentation on coverage of IFA program. Though, note that we know very little about the data quality of these coverage estimates. Evidence Action wrote that the quality of this data is "Low" and that it does not know core aspects of the methodology: "Generally, we are not confident in the quality of the data in this presentation, and have been unable to confirm how the government arrived at numerators or denominators for these calculations." See Row 17, Evidence Action, IFA Data sources and quality.

• 13
  • See survey discussed in Evidence Action, WIFS National Concept Note (quotes inserted below). Note that we have not yet assessed the methodology of this survey and do not have a strong sense of its representativeness, though it seems to have covered a large number of schools (over 4,000), and it would need to be highly nonrepresentative to affect the bottom line that a substantial proportion of schools are not implementing the program. We also have not attempted to weight the schools by number of students.
    • Evidence Action noted the following in its comments on this grant writeup (unpublished email from December 10, 2018): "This data is from a survey conducted by Evidence Action. According to Evidence Action, in 2018, it collected data on the WIFS program as part of an independent assessment survey for National Deworming Day (NDD) in India, which it supports through its Deworm the World Initiative. Data was collected from 10 states with consultation and approval from the Child Health Division of the Ministry of Health and Family Welfare. Independent monitors visited schools and anganwadi centres to conduct the survey. The survey was conducted using the NDD coverage validation sampling methodology and could therefore not be representative of WIFS coverage in the state."
  • "To assess current coverage of the WIFS programme, we asked schools about their awareness of the programme and the regularity of implementation. Of the 4,145 schools surveyed, only 16% had been implementing the programme regularly with sufficient tablet supply. A further 6% had been implementing the programme, but without always having sufficient supply. Nine per cent had not implemented the programme at all in the three months prior to the survey, and 8% had never implemented the programme. The remaining 61% of schools were unaware of the programme. The situation is similar with AWCs delivering the WIFS programme to out-of school girls aged 5 to 19 years delivering the programme to out-of-school girls aged 5 to 19: of the 5,291 centres surveyed, 20% had been implementing regularly with sufficient tablet supply. Six per cent were implementing the programme, but without always having sufficient supply; 8% had not implemented the programme in the three months prior to the survey; and 8% had never implemented the programme. The remaining 58% of centres were unaware of the programme." Pg. 5, Evidence Action, WIFS National Concept Note.
  • "We also asked all schools and AWCs to specify which components they perceived as ‘bottlenecks’ in the smooth delivery of the WIFS programme. Most schools and AWCs cited the supply of tablets (64%), followed by trainings (58%) and tablet storage (37%). Other challenges reported were screening children for anaemia symptoms (33%), supervising consumption (26%), managing tablet inventory (29%), and convincing children to consume the tablets (26%). The low awareness and coverage of WIFS across high-burden states indicates a need for further sensitisation to and strengthening of the programme. The survey also highlights challenges with sufficient tablet supply and the need for stronger supply chain management, and points to other areas that need further support." Pg. 5, Evidence Action, WIFS National Concept Note.
  • We have also seen results from this survey at a state-by-state level and did not see substantial differences between the overall figures cited above and the estimates of similar figures in Rajasthan and Madhya Pradesh, the target states for Phase 2 of this grant.

• 14

  See Columns P to R for Indian states in GiveWell, Deworm the World monitoring summary (2017), Sheet "Monitoring methods and results."

• 15
  • Coverage rates of all basic vaccinations were estimated as >60% at the national level. See Figure 9.1, Pg. 253, India National Family Health Survey (NFHS-4) 2015-16. We have not vetted this estimate.
  • External actors may have been involved in supporting the scale-up of many treatments, and in some cases coverage of drugs and vaccines may be low and slow to increase. For examples of scale-up of coverage trends in various global health treatments, see Figure 1, Pg. 5, USAID, Idea to Impact report 2015. ("As a recent Bill & Melinda Gates Foundation, Dalberg Global Development Advisors, and Boston Consulting Group analysis shows, global health launches can sometimes take decades to reach intended users (Figure 1). This is in comparison to “typical,” or average, launches in the United States, which can reach target users in about five years." Pg. 5, USAID, Idea to Impact report 2015.) We have not vetted this analysis.

• 16

  For more on our understanding of the impact of Evidence Action's deworming technical assistance in India, see these sections of our Deworm the World Initiative review: Does Deworm the World increase the likelihood that governments implement deworming programs? and Does Deworm the World's work increase the quality of deworming programs?.

• 17
  • Evidence Action told us by email that:
    • "The national IFA/WIFS budget for financial year 2017-18 is approx. $28 million
    • "On AMB, we have gone through the recently launched operational and financial guidelines on AMB (Anemia-Free India, Operational guidelines) and they provide a budget of approx. $160k per district per year. As there are approx. 700 districts in the whole country, the total annual budget of AMB comes out to be $112 million.
    • "It is important to note (as you will notice in the attachments as well), this budget encompasses NDD and WIFS budget as both are part of overarching anaemia control strategy under AMB.
    • For reference, the current per district budget for deworming/NDD is approx. $26k" Evidence Action, unpublished email to GiveWell, October 2, 2018
  • We have not yet discussed the above with Evidence Action in detail. This appears to be a large increase in funding for IFA that we plan to understand better.

• 18
  • For example, see Our World in Data, Chart of Child mortality by income level of country.
  • Coverage rates of all basic vaccinations were estimated as >60% at the national level. See Figure 9.1, Pg. 253, India National Family Health Survey (NFHS-4) 2015-16. We have not vetted this estimate.
  • However, external actors may have been involved in supporting the scale-up of many treatments, and in some cases coverage of drugs and vaccines may be low and slow to increase. For examples of scale-up of coverage trends in various global health treatments, see Figure 1, Pg. 5, USAID, Idea to Impact report 2015. ("As a recent Bill & Melinda Gates Foundation, Dalberg Global Development Advisors, and Boston Consulting Group analysis shows, global health launches can sometimes take decades to reach intended users (Figure 1). This is in comparison to “typical,” or average, launches in the United States, which can reach target users in about five years." Pg. 5, USAID, Idea to Impact report 2015.) We have not vetted this analysis.

• 19

  However, see Our World in Data, Chart of Prevalence of anemia in children, which suggests that anemia in South Asia has fallen by roughly 15 percentage points over the course of about two decades, or less than 1 percentage point per year. We have not vetted this analysis.

• 20

  We have not currently modeled this in the ideal way (e.g., we could factor this in to the counterfactual scenario by showing steady "catch-up" coverage increases), but have used a simpler form of modeling since this is a preliminary model.

• 21

  However, see Our World in Data, Chart of Prevalence of anemia in children, which suggests that anemia in South Asia has fallen by roughly 15 percentage points over the course of about two decades, or less than 1 percentage point per year. We have not vetted this analysis.

• 22

  For further details, see Evidence Action, State selection note and Evidence Action, IFA steady-state budget. Evidence Action notes at least nine states that could potentially benefit from technical assistance. We conservatively assume that about half of these states may not ultimately be prioritized, even if the initial project is successful. This was not a careful analysis and was mainly intended to provide a rough sense of the lower bound of additional states that may benefit.

• 23

  For more, see our core cost-effectiveness analysis.

• 24

  See Rows 78-98, under "CEA of Phase 2 grant, indirect effects on learning" in Sheet "Beta IFA CEA" in GiveWell, Phase 2 IFA cost-effectiveness model

• 25

  "Governments often rely on school infrastructure to implement programs targeting children, but whether and to what extent managerial capacity constraints affect the implementation of these programs are important, open questions. We consider these questions in the context of India’s school meals program and iron and folic acid (IFA) supplementation program. Using a randomized controlled trial in a rural district in the state of Odisha, we evaluate the impact of two interventions on child health and on how these government-run programs are implemented. First, we distribute a micronutrient mix (MNM) to be added to the school meal to complement the existing IFA program. Second, we monitor school meals with increased frequency early in the intervention. While we find significant positive effects of distributing the MNM on micronutrient levels in the meals, we find no detectable effects on child health. Increased monitoring of school meals, on the other hand, does improve hemoglobin levels. Monitoring did not affect take-up of the MNM, but it did improve implementation of the government’s IFA program. We also find significant negative spillovers of the MNM intervention on how well the IFA program was implemented, suggesting that effort by school officials was crowded out by the introduction of the new MNM program. We present additional evidence suggesting that these effects are driven by managerial capacity constraints." Abstract, Berry et al. 2018.

• 26

  See "Cost of Indian government implementation, including govt staff (e.g. teacher) time," Rows 120-127, Sheet "Beta IFA CEA," in GiveWell, Phase 2 IFA cost-effectiveness model.

• 27
  • Evidence Action told us by email that:
    • "The national IFA/WIFS budget for financial year 2017-18 is approx. $28 million
    • "On AMB, we have gone through the recently launched operational and financial guidelines on AMB (Anemia-Free India, Operational guidelines) and they provide a budget of approx. $160k per district per year. As there are approx. 700 districts in the whole country, the total annual budget of AMB comes out to be $112 million.
    • "It is important to note (as you will notice in the attachments as well), this budget encompasses NDD and WIFS budget as both are part of overarching anaemia control strategy under AMB.
    • For reference, the current per district budget for deworming/NDD is approx. $26k" Evidence Action, unpublished email to GiveWell, October 2, 2018
  • We have not yet discussed the above with Evidence Action in detail. This appears to be a large increase in funding for IFA that we plan to understand better.

• 28

  See "CEA of Phase 2 grant, direct impacts only," Rows 42-75 and “CEA of steady-state grant, EvAc TA costs only; TA marginal impact, excluding any additional costs incurred by govt,” Row 22 in Sheet "Beta IFA CEA" in GiveWell, Phase 2 IFA cost-effectiveness model.

• 29
  • GiveWell asked:
    • "It appears that in Phase 2 of this project you expect to spend ~$0.50 on TA per child targeted per year,[1] while in the "steady-state" phase of the project you expect to spend ~$0.08 on TA per child targeted per year.[2] Is this magnitude of cost decrease in line with other projects where Evidence Action has provided TA? Are there any parts of the TA that require staff to roughly scale with the number of people you're trying to reach? (E.g., if you're providing detailed support to administrators to improve supply chains in phase 2, will you need proportionally more TA staff to provide support to a much larger number of administrators in steady-state?)
      • [1] ~$3 million phase 2 grant for a target population of ~3 million children over 2 years. $3m / (3m*2) = ~$0.50.
      • [2] ~$2.5 million annual steady-state budget for a target population of ~30 million children (pg. 6) per year. $2.5m / 30 million = ~$0.08.”
  • Evidence Action responded:
    • "On your first question asking whether this magnitude of cost decrease is in line with other Evidence Action programs, we do not have data from when deworming was in ‘pilot’ stage to say with reasonable certainty if the magnitude of cost decrease was similar. We also do not have any other project that would be a good comparator for the cost and scale economies we see in India or the way in which we are supporting a government in improving program delivery.

      In IFA (as with deworming), because the model works within the government systems and is built around the school-based platform, we would fully expect the cost per child targeted to decrease significantly as programs grow in scale with economies of scale. For example, in 2014, Deworm the World provided technical assistance to Rajasthan with a TA budget of $358,191, reaching 10.8 million children at a per-child treated TA cost of $0.03. By 2017, with a TA budget of $384,430, our reach in Rajasthan increased to 18.6 million children at a per-child treated TA cost of $0.02. Even though our reach increased by 72%, our TA costs increased by only ~7%.

      The [Evidence Action, IFA steady state budget narrative] details out the components of the TA budget. There is an increase in state-level personnel costs and travel especially for our last-mile support through district coordinators (this increases from 4 to 33 district coordinators in Rajasthan and 5 to 51 district coordinators in MP). While this cost will increase with scale, the increase in cost is not proportional to the target population reached. In steady state, we also offset these implementation costs with some of the larger pilot stage costs, such as team time and travel for TA model design and collecting additional rounds of data, etc.

      We have put the Phase 2 and Steady State budget summaries side by side to more easily highlight differences in the two budgets. [excluded due to difficult of copy-pasting.] As mentioned in the budget narrative note, the steady state budget does not include Evidence Action global team. We estimate the global team support cost at 8-10% of the total direct cost of technical assistance. It also does not include reserves."

• 30

  See Sheet "Iron CEA, direct impact," GiveWell, Phase 2 IFA cost-effectiveness model.

• 31
  • For technical assistance cost per person, see "Cost per person-year of coverage due to TA," Row 18, Sheet "Beta IFA CEA," GiveWell, Phase 2 IFA cost-effectiveness model.
  • For government cost per person, see Evidence Action estimates with explanation in pgs. 5-9, Evidence Action, IFA steady state government cost narrative.