Evidence Action — Impact Evaluation of Iron and Folic Acid Supplementation ("Phase 2")

Published: April 2019

Note: This page summarizes the rationale behind a grant to Evidence Action that GiveWell has recommended to Open Philanthropy. Evidence Action staff reviewed this page prior to publication.

Background

In December 2018, Evidence Action's Beta Incubator received a GiveWell Incubation Grant to pilot a project to provide technical assistance to the Indian government's large-scale school-based IFA supplementation targeting children and adolescents. For more details on that grant, see this page.

About the grant

In March 2019, GiveWell recommended that Open Philanthropy make a grant of $800,000 to Evidence Action for the monitoring and evaluation of the Phase 2 pilot stage of its technical assistance to the Government of India's IFA supplementation program.

Evidence Action told us that it will undertake the following key monitoring activities with this grant. This is a tentative plan that is subject to change, but we believe that it is indicative of the type of activities that it will undertake:

• Monitor the change in coverage caused by Evidence Action's technical assistance to the IFA program. Evidence Action will try to do this in a few main ways:
  • Difference-in-differences evaluation at the state and district level. Evidence Action will measure and compare changes in IFA supplementation coverage between baseline and follow up in a sample of schools between treatment and control districts in Rajasthan and Madhya Pradesh, the two states in which Evidence Action will pilot its technical assistance. It will randomly choose (within implementation feasibility constraints) which districts are treatment and control.1
  • High frequency process monitoring. Evidence Action will also conduct weekly surveys in a random sample of schools to measure several other outcomes that may drive changes in coverage. For example, this includes drug availability, and teachers' and children's awareness and knowledge of the program.2
  • Qualitative information from key stakeholders. Evidence Action and/or GiveWell will conduct interviews with key stakeholders, such as block or district government officials, to collect additional information on what level of coverage might have otherwise been achieved in the counterfactual, for example what other technical assistance partners might have worked with the Government of India in the treatment districts had Evidence Action not provided technical assistance.3
• Monitor the change in rates of anemia. Increasing coverage of IFA is intended to decrease rates of anemia. Evidence Action will directly measure anemia rates through low-cost haemoglobin tests at baseline and follow-up in both treatment and control districts using HemoCue machines.4
• Track government costs. Evidence Action will use a "government cost tracker" tool to collect information on the costs of the program.5

Evidence Action is still finalising many of the details of its monitoring and evaluation plans, particularly the approach to the difference-in-differences (including the sample size in terms of both numbers of districts and schools). To do so, it is working with Kartini Shastry, Associate Professor of Economics at Wellesley College, and IDinsight. GiveWell expects to receive further details about these plans in April 2019. Evidence Action intends to conduct the baseline monitoring in July or August 2019.

Risks and reservations

The main risk of this grant is that we don't learn much about whether Evidence Action's technical assistance increased coverage of the Government of India's program. This might occur for a few reasons:

• Small sample size for the difference-in-differences. It is our understanding that only a small number of districts will be included in the difference-in-differences analysis. Randomising across a small number of districts, it is unlikely that systematic differences between the treatment and control groups wash out. We may have lasting doubts about whether any effect estimated in the difference-in-differences analysis is a true causal effect of the program as opposed to just picking up different pre-program trends in coverage between the treatment and control districts caused by other factors.
• Selection of control districts for the difference-in-differences. For the difference-in-differences estimates to tell us the effect of the program, coverage in the control group should be representative of what would have happened in the counterfactual in the districts in which Evidence Action intervened. Evidence Action currently plans to drop some districts from selection into the study (and therefore the control group), e.g. districts in which there are many other technical assistance partners.6 We hope to work out a plan with Evidence Action to mitigate any selection effects and help us best understand the counterfactual of Evidence Action's activities, but this plan has not been finalized.
• Measure of coverage. We don't yet have details on the methodology that will be used to measure coverage. For example, we do not know whether enumerators will directly observe children taking supplements, or whether Evidence Action will rely on answers to survey questions.

• 1

  Evidence Action told us this in a conversation on 4th February 2019. We do not yet have details on the implementation feasibility constraints.

• 2

  "Monitoring survey conducted in random and targeted of sample schools and AWCs [anganwadi centers] every week on drug availability and sufficiency, regularity of administration and consumption of drugs, knowledge of teachers and children, IEC [information, education and communication] materials/activities etc", slide 13 in Evidence Action, IFA Phase 2 Monitoring Considerations slide deck.

• 3

  "Qualitative insights would include:

  • Informal interviews with district/block officials conducted by EA staff at various levels (frequency TBD)
  • Structured/Formal exploratory discussions and interviews
    • Interviews with key officials
    • Structured group discussion with stakeholders (teachers, AWWs, block officials etc.)
    • Observational and informal insights"

  Slide 14 in Evidence Action, IFA Phase 2 Monitoring Considerations slide deck.

• 4

  "Additional costs... for two rounds in two states for basic Hb level testing using HemoCue machines", slide 7 in Evidence Action, IFA Phase 2 Monitoring Considerations slide deck.

• 5

  "Cost data collected quarterly by State Project Co-od. (M&E) from districts and state using a “government cost tracker” tool using similar channels to DtW cost data collection", slide 14 in Evidence Action, IFA Phase 2 Monitoring Considerations slide deck.
  

• 6

  "Drop districts due to security, terrain, density, districts with extensive partner presence - to check data on which point in the process it makes most sense to drop, districts where there may be significant changes in boundaries planned", slide 15 in Evidence Action, IFA Phase 2 Monitoring Considerations slide deck.