Evidence Action Beta — Incubator Program

Published: September 2018; Last updated: December 2021

Note: This page summarizes the rationale behind a GiveWell Incubation Grant to Evidence Action Beta. Evidence Action staff reviewed this page prior to publication.

We spoke to Jeff Grosz and Kevin Kelsey about a possible renewal of this grant on November 30, 2021.

Background and case for the grant

A key part of GiveWell’s research process is trying to identify evidence-backed, cost-effective programs. GiveWell sometimes finds programs that seem potentially highly impactful based on academic research, but for which there is no obvious organizational partner that could scale up and test them. This grant will fund Evidence Action Beta to create a dedicated portfolio within its overall incubator portfolio focused on interventions that GiveWell and Evidence Action believe are promising but that lack existing organizations to scale them.

We have found that which program a charity works on is generally the most important factor in determining its overall cost-effectiveness. Through partnering with Evidence Action Beta to test programs that we think have the potential to be very cost-effective, there is a chance that Evidence Action will incubate charities that are more cost-effective than our current top charities, or similarly cost-effective (which would be beneficial if our top charities' current funding gaps are filled in the future). Our hope is that programs tested and scaled up through this partnership may eventually become GiveWell top charities.

We believe this incubator has the potential to fill a major gap in the nonprofit world by providing a well-defined path for testing and potentially scaling any promising idea for helping the global poor that does not have an existing organization to implement it.

This grant initiates a partnership with Evidence Action toward which we might contribute substantial additional GIG funding in the future (more below).

Proposed activities

Evidence Action Beta plans for the work to occur in several "phases," drawing on its experience and approach over the past several years in incubating other interventions:

• Phase 1: A "diagnostic" phase consisting mainly of research to assess the feasibility of promising interventions and the Beta incubator's potential value-add.
• Phase 2: A "prototype/pilot" phase designing and implementing interventions at a small scale in a relatively narrow context.
• Phase 3: Scaled-up testing of interventions that seem promising after Phase 2, including potentially running randomized controlled trials (RCTs) and optimizing the program design and operational, organizational, and financial models necessary to support potential future scaled implementation.
• Phase 4: Successful interventions become GiveWell top charities.

This grant is only intended to cover Phases 1 and 2, allowing Evidence Action Beta to do small-scale testing of about six of the most promising programs that we're aware of at Phase 2 over the next two to three years to determine whether it makes sense to scale those programs up further. Over the course of the two- to three-year grant period, GiveWell will need to decide whether to make additional Incubation Grants to fund Phase 3 work on a rolling basis and, at the end of the grant, will make a decision about whether to renew funding for continued Phase 1 and 2 work. Our cost-effectiveness model is intended to capture the full cost of advancing a program through the incubator to the point where it could become a new top charity (i.e., including potential future grants for Phase 3). We think our current assumptions likely overestimate this total cost somewhat. (More on our estimate of this grant's cost-effectiveness below.)

Cost-effectiveness

GiveWell is currently aware of about 15 programs that we think might be worth testing through this dedicated portfolio in the Beta incubator. Our best guess is that only about a third of these programs will still appear promising after further analysis. We believe there is a significant chance (~25%) that only one to three programs will turn out to be worth testing; that risk seems acceptable to us given the potential upsides of this grant.

Reasons we think this grant has the potential to be highly cost-effective include:

• Evidence Action has a strong track record of scaling up cost-effective programs with high-quality monitoring. It is the parent organization of two of our top charities (Deworm the World Initiative and No Lean Season) and one standout charity (Dispensers for Safe Water). Evidence Action is one of the strongest nonprofits that we are aware of, and we believe it is among the most well-positioned organizations to run a partnership like this that is focused on filling the gap when other organizations are not well-positioned to scale a cost-effective, evidence-backed intervention.
• Our rough cost-effectiveness analysis suggests this grant is ~9x as cost-effective as cash transfers to people living in extreme poverty.1 (As of this writing, we estimate that giving to the Against Malaria Foundation, the most cost-effective marginal funding opportunity among our current top charities, is about ~5x as cost-effective as cash (see "Results" tab in Version 6 of our 2018 cost-effectiveness model)).
• We also believe there are a number of potential scenarios in which this grant could have particularly high upside impact that we have not fully and explicitly factored into our cost-effectiveness analysis, e.g. if this partnership identifies cost-effective programs that are then taken up and implemented at scale by governments, or by improving global incentives for generating ideas for cost-effective programs. We included a rough estimate for potential upside impact in our cost-effectiveness model, but we have not carefully modeled this scenario and our estimate could change substantially with additional research.2

Key factors impacting our cost-effectiveness model include:

• how successful this partnership is at scaling up cost-effective programs;
• if any of the incubated programs become GiveWell top charities, how cost-effective they are at that time; and
• the marginal cost-effectiveness of spending on our current top charities (or other cost-effective opportunities that we identify through avenues other than this partnership) in the future (i.e., whether programs such as bed nets, seasonal malaria chemoprevention, and deworming continue to have funding gaps that are ~5x as cost-effective as cash transfers and that outpace GiveWell's money moved).

Risks and reservations

Reasons this grant may turn out to be ineffective include:

• It might turn out to be very difficult for programs to be more cost-effective than our current top charities, and/or our top charities may have more room for more funding than GiveWell-directed funds can fill for the foreseeable future. If this partnership does not identify programs that are more cost-effective than our current top charities, or only identifies similarly cost-effective opportunities while our current top charities still have funding gaps, then – barring any of the other significant upside impacts such as those discussed above – this grant will have failed to have an impact.
• We expect our ongoing partnership with Evidence Action around this sub-portfolio to require a fair amount of GiveWell staff capacity, which might trade off with our other work to identify new cost-effective giving opportunities (such as our work to evaluate policy opportunities). That capacity may turn out not to have been used well if other avenues for identifying cost-effective opportunities end up seeming more promising.

Budget

Evidence Action has a total proposed budget of $5,069,866 over two-and-a-half years of operations for this GiveWell-aligned sub-portfolio in its Beta incubator. This breaks down roughly as follows:

• ~50% (~$2.6 million): Funding for Phase 2 (prototyping/piloting work). Evidence Action plans to test six programs in the field (for roughly $450,000 each) to evaluate their feasibility, better understand their costs, and help determine whether they merit a Phase 3 grant from GiveWell for further scale up.
• ~30% (~$1.5 million): "Cross-cutting Leadership & Support." This includes contributions to salaries for a Chief Innovation Officer and another manager, a cost-effectiveness modeler, an economist, and various operational support staff.
• ~15% (~$725,000): Reserves (per Evidence Action's organizational policy).
• ~5% (~$220,000): Funding for Phase 1 (diagnostic work). Evidence Action plans to scope about 11 interventions (for roughly $20,000 each) to learn which other actors are already working on the program and decide whether it is worthwhile to run a Phase 2 pilot version.

Key questions for follow-up

We plan to engage with Evidence Action about this work on an ongoing basis. To assess the success of this grant, we expect to regularly revisit a wide variety of questions (organized roughly by topic below).

Questions about this partnership’s programs and their progression through the incubation pipeline include:

• How large is the pipeline of programs that we would like the Beta incubator to investigate at Phase 1, and are we steadily adding programs to that pipeline?
• What percentage of programs that enter Phase 1 seem promising enough to move to Phase 2?
• What percentage of programs that enter Phase 2 seem promising enough for us to provide an additional GIG for Phase 3?
• How many Phase 3 programs become GiveWell top charities?

Questions about the impact of this partnership include:

• Throughout the process of testing programs in this portfolio in the Beta incubator, what are our estimates of their cost-effectiveness? Is the incubator finding programs that seem likely to be more cost-effective than our current top charities?
• Are other actors interested in testing or funding programs that are part of the Beta incubator?
• Do Beta incubator-tested programs (and Evidence Action programs generally) get taken up by governments or funded by other actors?
• Are other actors generating research to try to influence the incubator's activities (e.g., suggesting ideas for programs that have been tested in an academic study but have not been scaled up)?

Questions about the Beta incubator's operations include:

• Has Evidence Action hired a strong team to run the incubator?
• Does the amount of money that Evidence Action is spending at Phase 2 seem reasonable given the knowledge it is producing? Does Phase 2 seem to be producing sufficient information to facilitate good decisions about which programs should move to Phase 3?

Questions pertaining to GiveWell's broader strategy include:

• What is the marginal cost-effectiveness of giving to our existing top charities, and how large are their funding gaps? What do we expect the marginal cost-effectiveness of giving to our top charities to be in the future, taking into account potential GIG opportunities?
• Is engaging with the Beta incubator an effective use of GiveWell staff capacity? How much staff capacity is it using, and what are the alternative uses of that capacity?

Internal forecasts

For this grant, we are recording the following forecasts:

Confidence	Prediction	By Time
30%	This grant does not lead to any new top charities.	December 2023
55%	The Beta incubator leads to a new top charity that is 1-2x the cost-effectiveness of our marginal spending on current top charities.	December 2023
10%	The Beta incubator leads to a new top charity that's >2x as cost-effective as our marginal spending on current top charities	December 2023
5%	The Beta incubator program has impacts that lead us to make a public case that it was extremely cost-effective overall (i.e., it resulted in at least $10 million in spending at 15x the cost-effectiveness of cash transfers or more).	December 2023
15%	Our marginal spending on top charities will be 2.5x as cost-effective as cash or less (using our current cost-effectiveness estimate for cash)	December 2023

Sources

Document	Source
Evidence Action Beta, Early-stage funding request	Source
GiveWell, Evidence Action Beta BOTEC	Source

• 1

  See GiveWell, Evidence Action Beta BOTEC, "EAB BOTEC" sheet, cell B33.

• 2

  See "Scenario 4," Row 9, on the "Misc inputs" sheet in the model.