Note: This page summarizes the rationale behind a GiveWell-recommended grant to Evidence Action for its Accelerator program. Evidence Action staff reviewed this page prior to publication.
Summary
We are recommending a $14,006,277, four-year renewal of our 2018 Evidence Action Accelerator grant. The Accelerator is a program housed at Evidence Action whose goal is to scope and scale potentially cost-effective interventions that don’t have clear existing implementers. We expect that our continued partnership with the Accelerator will help us direct more money to cost-effective programs. This grant was funded by Open Philanthropy.
Published: March 2022; Last Updated: August 2022
Table of Contents
Background
At GiveWell, we aim to support the implementation of evidence-backed, cost-effective programs. However, we sometimes find programs that seem potentially cost-effective, but for which there is no obvious organizational partner to implement them at a large scale.
In 2018, GiveWell recommended a 2.5-year grant of $5,069,866 to Evidence Action to build the Evidence Action Accelerator (then called Evidence Action Beta), to assess, pilot, and scale interventions that GiveWell and Evidence Action believe are promising but that lack existing organizations to scale them.
We are recommending a renewal of this partnership and an expansion of the Accelerator team. Our continued aim is for programs scaled up through this partnership to eventually become GiveWell top charities.
Planned activities
The work of the Accelerator will occur in several stages, drawing on the team’s experience and approach during the previous grant. The stages are decision-focused, such that each intervention must pass certain criteria before advancing to the next stage, allowing the Accelerator to focus only on the most promising interventions.1
- Stages 1 and 2 – Screening and rapid review: Evidence Action will conduct quick assessments (screening2 ) and further desk research (rapid review3 ) of an intervention’s evidence of effectiveness, cost-effectiveness, room for more funding, and feasibility. We don’t expect substantial Accelerator capacity to be devoted to these stages relative to subsequent stages,4 and are unsure how much value they will add relative to GiveWell’s internal intervention identification process.
- Stage 3 – Deep dive: Evidence Action will work to determine whether it should expend resources scoping feasibility and what program delivery might look like at scale. At this stage, Evidence Action and GiveWell will work together to develop a shared understanding of evidence of effectiveness, cost-effectiveness, key questions for consideration, and the suitability of the program for Evidence Action implementation.5
- Stage 4 – Scope and design: During this stage, Evidence Action will identify target countries for program implementation, design the program in detail, build a monitoring and evaluation framework, conduct country visits, conduct any necessary field activities or data collection, and prepare a written report for GiveWell describing its findings.6 Evidence Action and GiveWell will work together closely to ensure that the findings from this stage are maximally informative to the decision of launching a program at scale.
- Stage 5 – Launch: During program launch, operational and administrative requirements will be put into place to begin implementation.7
- Stage 6 – Test at scale: At this stage, Evidence Action will deliver the program at units of substantial scale (district, state, country-wide), alongside strong monitoring and evaluation that will continue to inform the GiveWell cost-effectiveness analysis.8
This grant is only intended to cover stages 1-4. When Evidence Action has scoped a promising opportunity that it would like to bring to scale, GiveWell will decide whether to make additional grants to fund work in stages 5 and 6.
During the initial Accelerator grant, Evidence Action told us that it had insufficient staff capacity to continue earlier-stage work while designing and piloting a single promising intervention.9 This grant renewal is intended to grow the Accelerator team so that it will have adequate capacity to continue building out earlier-stage opportunities while investing in later-stage program development for more than one intervention at a time.
The expanded Accelerator team may include up to four new “program pillar” teams focusing on particular promising intervention areas,10 three new dedicated monitoring and evaluation roles,11 and dedicated technical review capacity.12 Evidence Action intends for some of these roles to be located in the countries where it works to support more rapid scoping of opportunities.13
Evidence Action aims to focus its hiring primarily on generalists that can adapt to new program areas as the pipeline of promising opportunities evolves.14
Throughout the grant, GiveWell and Evidence Action will regularly communicate to ensure both organizations’ priorities are aligned. We expect Evidence Action to prioritize interventions that are cost-effective relative to our marginal funding and that could potentially absorb substantial additional funding in the future (“room for more funding”) as interventions proceed through the stages above.15
Budget
Evidence Action’s 4-year budget for this program totals $16,300,366 and breaks down as follows:16
- Personnel: $9,327,358
- Travel, office, and operating expenses: $1,759,950
- Field Activities: $2,726,563
- Indirect costs (at 18%): $2,486,49717
Evidence Action has $2,294,089 in funds remaining from the original grant. It will carry forward this projected balance under this renewal, resulting in a funding recommendation of $14,006,279, or three years’ equivalent funding.18
Case for the grant
We recommended this grant because:
- We think it has potential to increase the amount of money we direct to promising programs. We aim to move significant amounts of funding each year to support the implementation of highly cost-effective programs. To do this, our research staff identify promising programs, investigate them, and recommend grants for their implementation. However, this process is time-intensive, and even more so when we identify a promising program that does not have an existing implementing partner, because it involves identifying a partner and then making individual grants to scope, pilot and implement a program. The Evidence Action Accelerator team has the resources to do the work associated with new program inception, requiring less GiveWell staff capacity to incubate new programs. We expect that our partnership with the Accelerator will help us direct more money to promising programs as a result. This is based on the success of the original grant and our perception that there are already several promising interventions in the Accelerator pipeline that it has not yet had sufficient capacity to explore that we are unlikely to make independent progress on.19
- We think this grant will be cost effective. In expectation, we think that this grant will yield roughly $40 million per year in room for more funding identified for new programming by the end of 2025.20
Based on this, we have built a back of the envelope model to assess the promisingness of this grant. That rough model indicates that this grant likely compares favorably to other giving opportunities this year. Key factors influencing our cost-effectiveness model include:
- the number of cost-effective annual spending opportunities that this partnership identifies by the end of 2025,21
- the cost-effectiveness of Accelerator-incubated programs at that time, and the marginal cost-effectiveness of spending on other opportunities in the future,22
- the chance that the programs implemented via the Accelerator would have been implemented without the Accelerator,23 and
- the number of years we expect to fund each identified program.24
We also believe there are a number of potential scenarios in which this grant could have upside impact that we have not factored into our cost-effectiveness analysis. For example, Accelerator-incubated programs may increase global attention towards otherwise neglected cause areas.
- We think Evidence Action is a strong partner. We previously made a 2-3 year grant of $5 million to establish the Accelerator. We think that the results thus far are promising.25 Over the course of that grant, we have built a strong relationship with the Accelerator team and have found it to be generally aligned with GiveWell’s priorities.
More broadly, Evidence Action has a strong track record of scaling up cost-effective programs with high-quality monitoring.26 It is the parent organization of Deworm the World Initiative, one of our top charities, and Dispensers for Safe Water.
Plans for follow up
As with the initial grant, we plan to engage closely with Evidence Action on the progress of this work via regular meetings, annual check-ins on expected room for more funding in the Accelerator pipeline, and project-specific calls for any interventions that are at or beyond the Stage 4 scoping and design stage.
Risks and reservations
Our primary reservations about this grant are documented below.
Evidence Action seems relatively risk averse with respect to growth. In recent interactions, Evidence Action has expressed an unwillingness to grow its organizational revenue quickly without substantial institutional overhead support beyond its typical budgeted overhead.27
Evidence Action has indicated that growing while maintaining high program quality will require additional investments in leadership, in expanding their operational footprint into new countries, and in expanded internal support functions such as operations, human resources, and finance.28 We plan to investigate ways to support Evidence Action as an organization outside of the context of this and other grants to ensure key leadership and operational support functions are able to be maintained through this period of growth.
At the same time, it is our impression that Evidence Action is more risk averse with respect to growth than some other organizations we work with. As a result, we seek to identify opportunities to partner with other organizations that have core competencies that we think likely align with cost-effective giving opportunities in our pipeline and that are willing to grow more quickly in the near-term.
We may be over-invested in Evidence Action. We have recently recommended grants to Evidence Action for its Deworm the World Initiative, Dispensers for Safe Water, Syphilis Screening and Treatment in Pregnancy, and Iron and Folic Acid Supplementation programs. If Evidence Action constitutes an increasing portion of our money moved, Evidence Action might have undue influence on how GiveWell makes decisions. We hope to mitigate this risk by diversifying our portfolio of partners with whom we are able to build new programs.
An expanded Accelerator team may be less aligned with GiveWell values. There is a risk that growth of the Accelerator could lead to a team that is less aligned with GiveWell’s values. Evidence Action has said that its approach to maintaining the quality and performance of its team includes focusing on whether the team is succeeding at identifying and bringing to scale opportunities that meet its primary criteria, and the criteria for GiveWell to fund.29
There may not be enough cost-effective interventions for the Accelerator to scale. The Accelerator team and GiveWell will both be working to identify large, cost-effective interventions suitable for Evidence Action. If we are unable to identify sufficient opportunities, then this grant will be less likely to be cost-effective. On the other hand, there are already interventions in the Accelerator’s pipeline that GiveWell believes are probably cost-effective when implemented by Evidence Action, such as programs that leverage Evidence Action’s existing community distribution networks in Kenya, Uganda, and Malawi30 or its school-based health programming in India and Nigeria.31
Evidence Action may not be the right organization for a given intervention of interest. It’s possible that GiveWell and Evidence Action might disagree on whether Evidence Action is best suited to implement a particular program. In that case, GiveWell may prefer to bring a program identified and/or scoped by Evidence Action to another implementing organization. We will try to mitigate this risk by maintaining alignment at a very early stage of investigation around the promisingness of opportunities and their fit for Evidence Action specifically.32
We may not need to make a four-year commitment. Evidence Action has said that a four-year grant will allow them to attract more senior staff and plan more efficiently for growth,33 but we remain uncertain about the benefits of a longer grant.
Forecasts
By the end of 2025, we think there is a 70% chance that we will have identified $40 million or more in room for more funding for grantmaking opportunities via the Accelerator that we estimate to be at least 8 times as cost-effective as GiveDirectly’s unconditional cash transfer program.34
Our process
In the course of investigating this grant, GiveWell:
- Had a series of conversations with Evidence Action
- Agreed on targets with Evidence Action
- Received a budget and plan from Evidence Action
- Made a rough model of cost-effectiveness (back-of-the-envelope calculations)
- Held a follow-up call with Evidence Action on November 30, 2021
- Held calls with some of Evidence Action’s partners in low- and middle-income countries
- A Senior Researcher and a Program Officer reviewed the case for this grant and provided feedback.
Sources
- 1
Evidence Action, New Program Development Process slides, 2021, page 1
- 2
“Stage 1: Screening: Is the intervention effective, low-cost, and supported by robust evidence?” Evidence Action, New Program Development Process slides, 2021, page 1
- 3
“Stage 2: Rapid Review Is the intervention worth serious consideration for implementation?
- Evidence Review, including Theory of Change and assessment of burden
- Research agenda
- BOTEC CEA"
Evidence Action, New Program Development Process slides, 2021, page 1
- 4
“Currently, the Accelerator’s initial stages of review comprise a small proportion of both staff time and the budget. Going forward, Evidence Action expects that the initial stages would continue to take a minority of its total level of effort, but not an insignificant amount. In general, the most expensive part of the Accelerator’s process tends to be the field activities, including scoping, working with partners, and launching and testing the intervention.” GiveWell's non-verbatim summary of a conversation with Evidence Action, November 30, 2021
- 5
“Stage 3: Deep dive: Should the intervention be scoped? If so, where?
- In-depth evidence review
- Expanded CEA
- Intervention strawman
- Project scorecard
- Scoping trip preparation”
Evidence Action, New Program Development Process slides, 2021, page 1
- 6
“Stage 4: Scope & Design. Should the intervention be launched? If so, where?
- Program design
- Trip report(s)
- Updated CEA
- Monitoring framework
- Evaluation design outline”
Evidence Action, New Program Development Process slides, 2021, page 1
- 7
“Stage 5: Launch: Put all operational and administrative requirements in place and begin implementing.” Evidence Action, New Program Development Process slides, 2021, page 1
- 8
“Stage 6: Test at scale: Is the program cost effective at scale?” Evidence Action, New Program Development Process slides, 2021, page 1
- 9
“The Accelerator has a list of promising interventions that it is investigating. In the past year, it has been focusing on field testing its in-line chlorination (ILC) program. The focus on that program has diverted limited staff capacity from refining the pipeline of early-stage investigations or pursuing more intense scoping of interventions already identified as potentially promising...The Accelerator is fairly confident that it would be able to identify valuable opportunities. As it hires more staff, it would be able to investigate more interventions more deeply, compared to the last year, in which it spent very little staff time on sourcing new interventions.” GiveWell's non-verbatim summary of a conversation with Evidence Action, November 30, 2021
- 10
“Program pillars are illustrative and will be developed around the most promising programmatic areas
- Notionally, we plan for four pillars: Water; Nutrition; Maternal/Neonatal Health; New Program (pending identification of a promising new area)” Evidence Action, Accelerator Staffing Models, 2021, pg 1
- 11
“MLE (3)”, Evidence Action, Accelerator Staffing Models, 2021, pg 2
- 12
“Robust technical team, with increased capacity for research and MLE,” Evidence Action, Accelerator Staffing Models, 2021, pg 1
- 13
“Overarching goal: More bandwidth to develop and launch evidence-based, large-scale, CE programs
- Program pillar teams to deepen sector expertise and drive growth within in highly promising program areas
- Robust technical team, with increased capacity for research and MLE
- Integrated teams, to ensure that new programs leverage the knowledge of mature programs (eg ILC leveraging DSW)
- There will be some staff integrated into the team working on early-stage programs (funded through other grants and not presented in this org chart)
- We will support small amounts of key strategic regional leaders, to harness expertise and govt/partner relationships of regional leadership
- More in-country capacity: more staff based in countries where we work
- More flex capacity, to be rapidly respond to new opportunities that arise”
Evidence Action, Accelerator Staffing Models, 2021, pg. 1
- 14
“To retain flexibility as the Accelerator grows, Evidence Action plans to focus on hiring generalist staff, though it may hire some sector experts. Generalist staff would likely have an easier time pivoting as priorities change. Evidence Action plans to develop mechanisms to ensure that different interventions across sectors are being evaluated consistently. Mechanisms could include peer review or committees to ensure that staff receive feedback from people on other teams.” GiveWell's non-verbatim summary of a conversation with Evidence Action, November 30, 2021
- 15
“Under the current grant, the Accelerator finds interventions according to the following criteria: potential scale of impact, strength of evidence, and cost-effectiveness. Within the context of this grant, it would add the criterion of room for more funding, which often, but not always, correlates with its current primary criterion of scale. Adding this criterion would allow Evidence Action to align more closely with GiveWell’s current priorities. However, Evidence Action believes there is still benefit in identifying opportunities with less room for funding that are highly cost-effective.” GiveWell's non-verbatim summary of a conversation with Evidence Action, November 30, 2021
- 16
- 17
Indirect costs include: finance and administration, human resources, and fundraising and communications expenses GiveWell's non-verbatim summary of a conversation with Evidence Action, November 30, 2021
- 18
Total Budget: $16,300,366
12/31/2021 projected balance on existing grant*: $2,294,089
Total Ask, Less Projected Balance: $14,006,277.
Evidence Action, Accelerator budget 2022-2025
- 19
“This list would likely change as the Accelerator reprioritizes its search criteria and adds staff research capacity, but currently includes:
- Water treatment – Coupons for water treatment are high on the Accelerator’s list of priority programs.
- Maternal and neonatal health – The Accelerator is investigating antenatal care-based programs such as multiple micronutrient supplements for pregnant women, which currently looks quite cost-effective and is probably its second-most promising intervention after coupons for water. Other promising antenatal care-based interventions include calcium supplementation, balanced energy-protein supplementation, chlorhexidine for umbilical cord care, and home- or community-based kangaroo mother care (KMC), which encourages skin-to-skin contact and early initiation of breastfeeding to reduce mortality in preterm or low birth weight babies.
- Nutrition – The Accelerator is also looking into nutrition interventions, such as micronutrient powders and lipid-based nutrient supplements.
- School-based programs – The Accelerator is investigating programs for treating children in schools, such as intermittent preventive treatment for malaria and anemia testing and treatment.” GiveWell's non-verbatim summary of a conversation with Evidence Action, November 30, 2021
- 20
This is in expectation; see here. Note that this is not total room for more funding for all GiveWell-funded programs but new programming being implemented in 2025 by Evidence Action as a result of this renewal.
These estimates were developed in collaboration with Evidence Action. For example, Evidence Action estimated a 25% chance the Accelerator would lead to significantly less than $40 million in new programming per year. Jeff Grosz, Senior Director Evidence Action Accelerator, Email to GiveWell, October 26, 2021 (unpublished)
- 21
We guess there’s a 65% chance that this grant will yield roughly $40 to $65 million per year of room for more funding identified by the end of 2025 at cost-effectiveness equal to or greater than our marginal dollar. See here. Evidence Action expects about 25% of eventual funding for scaled opportunities identified by the end of 2025 to come from sources other than GiveWell.
- 22
We guess that the opportunities Evidence Action identifies will be 8x more cost-effective than cash transfers on average, and that without the Accelerator we would spend that cash on opportunities that are 6x as cost-effective as cash transfers. See here.
- 23
We guess there’s a 50% chance that opportunities scaled up via the Accelerator would have been scaled up without the Accelerator within 5 years. See here.
- 24
We’ve assumed that programs are funded for ten years in this model.
- 25
In 2.5 years, the Accelerator has launched the following programs:
- Iron and folic acid (IFA) supplementation in India: “With the government’s guidance and the support of charity evaluator GiveWell, we spent 2018 conducting a light-touch assessment of the potential of this intervention and identifying key questions for further exploration as part of phase 1 of our Beta incubation process. In 2019, with the continued support of GiveWell, we launched phase 2 of the incubation process, centered on “prototyping” our proposed state-level technical assistance model for IFA delivery through schools and anganwadis, which is being tested in four states in India.” Evidence Action, “Targeting Our Existing Government Partnership in India to Tackle Anemia,” 2019.
- Syphilis screening and treatment for pregnant people in Liberia: “This intervention was identified and selected by Evidence Action’s Accelerator under our New Program Development process, where the evidence for interventions is rigorously evaluated so that only those with the greatest potential for cost-effective impact are scaled up.” "Evidence Action, Syphilis Screening and Treatment for Pregnant Women"
- Evidence Action is currently field testing a water quality intervention called in-line chlorination (ILC): “We are currently designing and testing an in-line chlorination intervention to reach populations in urban and peri-urban areas, leveraging our Dispensers for Safe Water network that serves nearby rural communities. This is under the “Scope and Design” Stage of our new program development process, through which we have rigorously tested various models of in-line chlorination devices to inform selection of a device that is consistent, durable, and suited to community needs.” "Evidence Action, Evidence Action Accelerator"
- We will know more about the likely cost-effectiveness of ILC in 2022, and expect to decide on an initial grant at that time to scale ILC in one or more countries.
- 26
See, for example, our report on Evidence Action’s Deworm the World Initiative.
- 27
Based on several unpublished conversations with Evidence Action.
- 28
“As Evidence Action scales up rapidly, it would need to increase its support structures that are external to the Accelerator team, including finance, human resources, and operations.”GiveWell's non-verbatim summary of a conversation with Evidence Action, November 30, 2021
- 29
“Evidence Action’s approach to maintaining the quality and performance of their team includes focusing on whether the team is succeeding at identifying and bringing to scale opportunities that GiveWell will fund.” GiveWell's non-verbatim summary of a conversation with Evidence Action, November 30, 2021
- 30
“Evidence Action already has a large community-level workforce in Kenya, Uganda, and Malawi, with local government relationships and regular touchpoints, which it used to scale up its Dispensers for Safe Water program, and will likely use to scale up its in-line chlorination project (ILC).” GiveWell's non-verbatim summary of a conversation with Evidence Action, November 30, 2021
- 31
School-based health programs in India include deworming and iron and folic acid supplementation:
- “Our Deworm the World Initiative helps bring the treatment for free to children at their schools instead of placing the burden on the families to obtain it. We partner with governments to regularly treat all at-risk children in places where at least 20% are infected with worms, as recommended by the World Health Organization. School-based deworming works by freeing children from worm infection, improving their health and enabling them to attend school regularly.” Evidence Action,“Our Fight Against Worms”
- “Rigorous evidence shows regular iron and folic acid (IFA) supplementation significantly reduces the risk of this condition. Beginning in India, we’re testing how this intervention can be delivered cost-effectively to millions of children through schools.” Evidence Action,“Iron and Folic Acid Supplementation”
- 32
“Should Evidence Action identify a promising intervention, and GiveWell think that it would be best implemented by another organization, the organizations would discuss this possibility at an early stage. Evidence Action and GiveWell both prioritize delivery of effective implementation of programs, so it is unlikely that the choice to fund a different organization from Evidence Action for an intervention would cause problems, so long as there was upfront and open communication.” GiveWell's non-verbatim summary of a conversation with Evidence Action, November 30, 2021
- 33
“Evidence Action proposes a 4-year grant period for the Accelerator, but is only asking for new funds for 3 of those years. One of the years would be funded with existing funds from the current grant. This relatively long funding period would allow the Accelerator to attract high quality candidates and build expertise around specific promising interventions without worrying about a funding cliff. It also allows Evidence Action to invest time in researching a variety of promising interventions, rather than being more risk averse on a shorter timeline. In addition, a longer project period gives the Accelerator more leeway to pilot projects with governments, because it does not commit to running projects until it has gained sufficient confidence around the project design.” GiveWell's non-verbatim summary of a conversation with Evidence Action, November 30, 2021
- 34
As of early 2022, our bar for directing funding is about 6x as cost-effective as GiveDirectly. For an example of the cost-effectiveness of our recommendations, see this page.