How We Produce Impact Estimates – November 2021 version

We have published a more recent version of this page. See our most recent version of this page.

Published: November 2021

General philosophy

When we’re communicating impact, we strive to adhere to the following principles:

• Be accurate. Don’t misrepresent or imply that a donation will accomplish more (or less) than we expect it to.
• Be clear. To the extent possible, make straightforward claims that are easy to understand.
• Don’t imply a false level of precision. While models may provide estimates down to the dollar, round them so they don’t suggest we believe in that level of specificity.
• Share the work with those who want it. Make the detail behind impact estimates available for anyone who wants it.

How we generate impact estimates

Impact metrics for GiveWell’s top charities

We use the inputs below on our top charities page and other estimates of impact. You can see more detail in this spreadsheet.

For live-saving charities, we present two cost-effectiveness estimates: one based on the cost of each output and one based on the cost of each outcome. Outcomes refer to the ultimate impact of the donation, such as saving a life, that results from the outputs. Outputs refer to the measurable activities a donation generates, such as treating a child with a full course of malaria medication, which in turn lead to the outcomes.

We don’t provide both outputs and outcomes for the life-improving charities we recommend:

• For GiveDirectly, we don't present separate outputs and outcomes — the final result is an increase in wealth.
• We believe deworming may lead to an increase in lifetime earnings, but it is especially complex. You can read more in the separate section on deworming impact below.

Our cost-per-output estimates are based on financial and programmatic information from our top charities. When estimating total outputs attributed to a specific amount of funding (e.g., how many nets will $100 deliver?), we divide the total funding by our estimate of how much it cost to deliver that output (e.g., how much it costs to deliver an insecticide-treated net).

For both outcome and output estimates, we use a weighted average of the cost-effectiveness for funds we directed to specific funding opportunities in the most recent calendar year. We use a weighted average because both cost per output and cost per outcome vary among programs we recommend funding for. For example, we made both of the grants below to the Malaria Consortium’s seasonal malaria chemoprevention program in March 2020:

• $2.8M to distribute malaria medication in Chad at a cost of $8 per child treated with a full course of medicine, resulting in an estimated cost-effectiveness of $8,000 per life saved
• $3.4M to distribute malaria medication in Burkina Faso at a cost of $6 per child treated with a full course of medicine, resulting in an estimated cost-effectiveness of $2,000 per life saved

Cost-per-life-saved estimates have been rounded up to the nearest $500, and outputs have been rounded to the nearest dollar. Our impact estimate for this program includes a weighted average of all grants (including the two above) we made to this program. You can see a list of the specific funding opportunities used in these calculations by looking at grants made in 2020 in this spreadsheet. Outcome numbers don’t include all funds we direct to our top charities. More detail on this below.

What funding does GiveWell look at to calculate cost-effectiveness?

Funds directed to our top charities by GiveWell can be grouped in three buckets:

1. Grants we make from our Maximum Impact Fund to specific funding opportunities [Note added October 2022: We updated the name of this fund in September 2022; more information here.]
2. Grants we recommend to the Open Philanthropy Project
3. Donations that donors make to an organization we recommend, but that aren’t directed to a specific funding opportunity. This includes donations made to charities on our checkout page, donations made directly to the organizations, and donations through other third-party organizations that share GiveWell’s recommendations (e.g., One for the World).

We calculate the cost-effectiveness of specific funding opportunities, not the cost-effectiveness of all the funding an organization uses. Because of this, we are only using buckets 1 and 2 as the basis for calculating the cost-effectiveness numbers we’re using in our impact estimates. This ranges between 36% to 94% of the total funding we direct to each top charity (source).

We exclude the funding in bucket 3 because we don’t have estimates of the cost-effectiveness of those donations. We only calculate the cost-effectiveness for specific funding opportunities. Funding directed to our top charities could be either more or less cost-effective than the funding directed to specific funding gaps. Rather than guessing at a number and using it for our calculations, we’ve excluded this funding.

We also exclude grants for which we don’t generate a cost-effectiveness estimate. For example, we do not estimate the cost-effectiveness of grants for monitoring surveys to assess the success of programs. These improve our decision-making ability, but don’t have a direct impact.

What are funding opportunities?

GiveWell makes its decisions about where to recommend funding based on estimates of the cost-effectiveness of specific "funding opportunities" rather than the entire portfolio of a charity's work.

A funding opportunity is generally defined as a specific intervention (e.g., seasonal malaria chemoprevention) in a specific country, although funding opportunities may also be defined at the subnational level.

Our cost-effectiveness estimates can differ between funding opportunities. For example, in 2021, we estimated that Malaria Consortium's seasonal malaria chemoprevention program in Burkina Faso was roughly four times as cost-effective as its work in Chad, largely driven by higher rates of malaria mortality in Burkina Faso.

Why does GiveWell calculate cost-effectiveness at the funding opportunity level?

We analyze individual funding opportunities rather than charities because we believe it allows us to better distinguish between different uses of marginal funding. If we directed funding based solely on a single charity-level cost-effectiveness estimate, our analysis would result in recommending funding to only one charity.

We believe that would be misleading. By estimating cost-effectiveness at a more granular level, we are better able to recommend funding to multiple charities and direct funding to the most cost-effective opportunities on the margin. We use these estimates to decide how to allocate funding from the Maximum Impact Fund. [Note added October 2022: We updated the name of this fund in September 2022; more information here.]

As well as estimating the cost-effectiveness of specific funding opportunities, we also present aggregate cost-effectiveness estimates for a particular charity on our top charities page.

How are numbers rounded?

You can see a list of the specific funding opportunities used in these calculations here.

The cost-effectiveness estimates we share for specific funding opportunities don’t exactly match the numbers we’ve shared in the table above. We’ve rounded the numbers to avoid giving a false sense of precision.

• We’ve rounded life-saving estimates up to the nearest $500.
• We’ve rounded output numbers to the nearest dollar (up or down).

For example, you can see in the linked sheet that our life-saving charities can vary by hundreds of dollars in cost-effectiveness. We don’t think our models are precise enough to actually differentiate between differences in cost-effectiveness of only a few hundred dollars, so we’ve rounded estimates up to the nearest $500 to better convey this.

How often does GiveWell update impact numbers?

We update our impact estimates annually in January.

Does GiveWell recalculate past cost-effectiveness estimates regularly?

We finalize our cost-effectiveness estimate for any given funding opportunity shortly before we make the grant to that opportunity (or recommend it to a third party). We don’t go back and update these estimates when our cost-effectiveness model changes. We think it would impose a large administrative burden without substantially improving our decisions for upcoming grants.

This means that we may reach a different estimate of cost-effectiveness for some past grants if we were to re-evaluate them now. You can see more information on changes we've made to our cost-effectiveness analyses here.

Guidance for interpreting impact estimates

Why is the cost to save a life so high?

We often get questions about why we estimate it costs thousands of dollars to save a life if, for example, it only costs approximately $5 to distribute a malaria net, or $7 to distribute antimalarial medicine, or $1 to provide a vitamin A supplement—the interventions we recommend to save lives. It's not intuitive!

Here, we'll briefly walk through why our estimate of the cost to save a life is higher than our estimate of the cost per treatment or item. We'll use the example of the Against Malaria Foundation (AMF), a GiveWell top charity.

AMF supports the distribution of insecticide-treated nets to prevent malaria. These nets are hung over sleeping spaces and block and kill malaria-carrying mosquitoes. We estimate that it costs about $5 to purchase and distribute one of these nets.1

Not every person who receives a net from an AMF-supported distribution would have otherwise died of malaria. This is why the cost per life saved is not $5.

In addition, although the nets are highly effective, the nets do not prevent malaria 100% of the time. They wear out over time, developing holes that reduce their effectiveness.2 In addition, mosquitoes have developed some resistance to the insecticide that the nets are treated with, also reducing the nets' effectiveness.3 In addition to factors related to the nets themselves, some people will choose not to use their nets, will not use them consistently, or may use them incorrectly (for example, leaving gaps between the net and their sleeping space).4 People can also be bitten by infected mosquitoes when they are not in bed.5 We adjust our estimate of nets' effectiveness against malaria to account for these factors.

We also incorporate the local malaria mortality rates where AMF works. While nets are distributed to people who are at risk of malaria, even without nets a fairly small portion of them would die due to the disease. Many would, by chance, avoid infection. Many who do get infected would survive the infection with treatment or without. We estimate the cost per life saved for each location based on the likelihood that a person who receives a net would have died from malaria without the net.

In addition, we include whether we believe that other actors would distribute nets if AMF didn't, so that we can estimate AMF's true impact—and thus our best guess of the true impact of donations to AMF.

See here for a step-by-step walkthrough using a specific AMF distribution in Guinea.

What are marginal funding and marginal impact?

Marginal funding is the last funding that goes to a program. It’s important because the impact of marginal funding is often different from the average impact of funding to a program.

For example, imagine that the Against Malaria Foundation is considering funding the distribution of insecticide-treated nets in Nigeria. They have identified two areas as excellent candidates for distribution:

• They could distribute $1 million worth of nets in areas with the highest rates of malaria. This would save 200 lives at the cost of $5,000 per life saved.
• They could distribute an additional $2 million worth of nets in areas with lower rates of malaria. There is less malaria to prevent, so it costs a bit more to save lives. This distribution will save 250 lives at the cost of $8,000 per life saved.

The average cost to save a life with this program if fully funded with $3 million would be about $6,700 per life saved.

But that wouldn’t be the impact of your donation. Your donation would go to one area or the other. If AMF had already raised $1 million before you gave, they would spend it in the areas with the highest malaria prevalence. If you gave after that, your donation would go to the areas with lower malaria prevalence where it cost $8,000 to save a life.

We call this last donation marginal funding. The marginal impact is what your donation actually causes to happen.

Rather than trying to estimate the marginal impact of each dollar, GiveWell typically estimates the marginal impact of giving to a charity’s funding opportunities. These typically refer to a pool of funding directed to a charity's activities in a single country or subnational region. We estimate the cost-effectiveness of each funding opportunity, and regularly reassess what marginal donations will accomplish based on which funding opportunities a charity will spend on next.

What is funging, and how does it affect impact?

When you give, you impact how GiveWell allocates funding from other donors. So even when you give to a specific charity, the impact of your donation may be felt elsewhere based on how GiveWell ultimately allocates its full pool of funding. We call this funging (from fungibility).

For example, imagine that GiveWell has identified two excellent funding opportunities:

• An opportunity with Helen Keller International is the most cost-effective, and needs $5M to distribute vitamin A supplements.
• An opportunity with Against Malaria Foundation (AMF) is the next most cost-effective, and needs $20M to distribute malaria nets.

GiveWell has $15M available to direct wherever it deems best. You decide to donate $50 to Helen Keller International vitamin A supplements.

Because GiveWell will fully fund the most cost-effective opportunity first, your donation effectively moves $50 of GiveWell’s dollars from Helen Keller International to AMF. GiveWell doesn’t have to give that $50 to Helen Keller International — you’ve already provided it.

In 2020, GiveWell granted (or recommended grants) totaling over $100 million to specific funding opportunities that GiveWell chose. With that much funding, many donors who choose to give to a specific charity will effectively have their donation increase the budget of a different organization.

Note that there are different kinds of funging. The above only refers to how donations directed to an individual top charity displace other GiveWell-directed funds. We do attempt to incorporate the possibility that our top charities displace other funds for an intervention from other sources (e.g., governments) in our estimates.

Why does GiveWell share the impact of grants rather than donations?

Sharing the impact of donations is challenging because individual donations to particular charities can be funged by GiveWell (see above). Because we don’t know the ultimate impact that a given donation will cause through funging, any estimate we shared based on the immediate recipient of the donation would likely be inaccurate.

To avoid sharing potentially misleading estimates, we share the impact of grants from the pool of funding we direct to funding opportunities. Because grants don't displace other GiveWell-directed funding, this allows us to share estimates we’re more confident in. It also gives donors the estimate we think they’re looking for — the impact a certain amount of funding has with that program (without taking funging by GiveWell into account).

Why does GiveWell show past cost-effectiveness rather than projected cost-effectiveness?

Because we finalize the cost-effectiveness estimates of funding opportunities at the time we recommend a grant, we don’t have estimates for the cost-effectiveness of future grants. Our best guess is that they’ll be similar to past grants, but will become less cost-effective over time. This may happen quickly if GiveWell sees a rapid increase in funding we direct, as we will fully fund the most cost-effective opportunities first.

We believe sharing the direct past estimated impact gives donors a useful proxy for what their impact may be without making potentially misleading statements about the future.

An idealized version of the top charities page would present an aggregated (charity-level) cost-effectiveness estimate based on an up-to-date best guess of how incoming funds would be allocated on the margin. For example, if we believed the next $5 million of funding for Malaria Consortium would be spent in Burkina Faso, and the $5 million after that would be spent in Chad, we would present the cost-effectiveness estimate for Burkina Faso until $5 million was donated, and then present the cost-effectiveness estimate for Chad until the next $5 million was donated, and so on.

However, we believe this approach would be impractical:

• It would be confusing if our impact estimates frequently changed.
• Keeping an up-to-date estimate of how marginal funds would be spent would create an unnecessary administrative burden. Instead, we make decisions about how to allocate funds quarterly, and prioritize a detailed understanding of how we expect marginal funds to be spent for the most decision-relevant funding opportunities.
• It can be challenging to communicate clearly about what we mean by "the margin." When making decisions about how to allocate funding, we estimate the marginal dollar based on a projection of how much funding charities will receive from other sources (including donors who give to a specific charity on the basis of GiveWell's recommendation) between now and when the funds will be needed, as well as in the past. For example, based on the example above, if we projected that donors would give $7.5 million to Malaria Consortium, we would expect the marginal additional dollar to be spent in Chad, rather than Burkina Faso. However, that would mean never presenting the cost-effectiveness estimate of Burkina Faso, which we believe would be misleading.

Because of these difficulties, we use a weighted average of programs that GiveWell has directed funds toward in the past calendar year rather than the cost-effectiveness of the specific program we expect would be funded by a marginal donation. We think this is a clearer way to present a rough and intuitive estimate of how much good a donation can accomplish (which is the goal of the top charities page).

Do these cost-effectiveness estimates include all factors that are relevant for choosing which programs to fund?

The purpose of our cost-effectiveness analyses is to help GiveWell make better decisions about how best to allocate funds between different programs.

These estimates abstract from certain features of a program that we think are relevant for choosing which programs to fund. For example, our estimates do not include:

• our assessment of an organization's ability to respond effectively to unexpected events
• the costs and benefits of delaying a decision to preserve the value of retaining future options

Because of these limitations, a cost-effectiveness analysis is not the only tool we use when we decide whether to allocate funding to a program. You can see the principles we follow to allocate funding on this page.

In addition to excluding certain factors that we take into account in our decision-making, our cost-effectiveness estimates are often highly uncertain.

While we make our best effort to produce a reliable estimate, it can be challenging to ensure internal consistency between different estimates. For example, for practical reasons we restrict the scope of our estimates to focus on the most important benefits of a particular program.

You can find more information on what goes into our cost-effectiveness estimates here.

Why does GiveWell compare programs to cash transfers?

When we're making decisions about which programs to fund, we use a single unified metric that compares cost-effectiveness to GiveDirectly's cash transfer program. For example, in 2021, we estimated that some funding opportunities with AMF in the Democratic Republic of the Congo were approximately 10x as cost-effective as cash.

We often model different types of benefits from an intervention. For example, in 2021 we estimated that about 30% of the benefits of AMF's programs are the result of improved long-term economic outcomes for children who are protected from malaria. The "x cash framework" allows us to aggregate a program with different types of modeled benefits into a single metric, which is useful for our decision-making.

An additional benefit of using this metric is that we could spend large amounts of money on direct cash transfers, as the program is relatively straightforward to implement and GiveDirectly has the infrastructure to put significant increases in funding to use. Because of this, cash transfers to GiveDirectly set a clear lower bound on cost-effectiveness that GiveWell should consider (although in practice we aim to fund programs that we believe are substantially more cost-effective than cash transfers).

Despite the benefits of comparing programs to cash transfers, this metric is especially unintuitive. For ease of interpretation, we therefore present our cost-effectiveness analyses in terms of the main benefit from a charity's program (excluding deworming — see more below).

Questions about specific programs

Is vitamin A supplementation the best program GiveWell recommends?

Our 2020 estimates suggest that Hellen Keller International can save a life for $3,000, and our other top charities for $4,500. We don't believe that Helen Keller International is a better giving opportunity than our other recommendations, even though this estimate might suggest that. To generate our impact numbers, we look for specific funding opportunities within organizations and determine their cost-effectiveness.

We identified and directed funds to $9.6M in very cost-effective funding opportunities at Helen Keller International available in 2020. We were unable to identify additional funding opportunities at Helen Keller International that we thought were more cost-effective than (or as cost-effective as) the funding opportunities available at other top charities.

Helen Keller International appears to be especially cost-effective because the average program we funded there in 2020 was especially cost-effective. But in practice, Helen Keller International had a limited capacity to absorb funding for additional opportunities that were as cost-effective (or more cost-effective) than those we had already funded. For this reason, we don’t expect additional donations to Helen Keller International to be more cost-effective than donations to our other top charities.

Why doesn't GiveWell share outcome estimates for deworming programs?

Deworming is especially complicated. We recommend deworming because of its potential economic impacts rather than its impact on health. Deworming is extremely cheap and may provide program participants with substantial increases in lifetime income relative to the amount donated. See our intervention report on deworming for more details.

Unlike direct cash transfers, we think it is possible that deworming has no effect or only a small effect. The evidence that deworming increases lifetime incomes relies primarily on one study (which we find credible) with a large central estimate but wide confidence intervals.

Because there is a chance that deworming has a huge impact, we believe the expected value of deworming is still high. Internally, we model this impact as a multiple of direct cash transfers through GiveDirectly. For example, we think that providing deworming medication may be many times more effective than giving cash through GiveDirectly.

We don’t have a way to share that impact of deworming that effectively communicates the uncertainty around our estimates. An estimate for the increase in lifetime income would convey a sense of certainty and precision that misrepresents our beliefs.

Our current internal metric for the impact of deworming, “multiples of cash transfers through GiveDirectly,” is also unintuitive and likely to confuse. Instead, we’ve decided to share our estimate for the number of children dewormed. This is a concrete estimate that we’re relatively confident in, and still gives donors a rough idea of how their donation helps.

For donors who would like the details of our cost-effectiveness estimate for deworming, our calculations are available here.

• 1

  We estimate that nets delivered by Against Malaria Foundation cost $4.95 each.

• 2
  • "Post-distribution net loss occurs for multiple reasons: attrition (i.e., nets were discarded due to damage, appropriated for other uses, given away, moved, or stolen), extensive holes in the net that make it permeable to mosquitoes, and decay of the net's insecticide component to the point that it is no longer effective." GiveWell: Estimating Equivalent Coverage Years for Long-Lasting Insecticide-Treated Nets (LLIN), section on "Measuring Net Durability."
  • Our estimate of the duration of insecticide-treated nets is incorporated into our cost-effectiveness model here: GiveWell: 2021 GiveWell cost-effectiveness analysis — version 3, "AMF" sheet, cell A30.

• 3
  • "Insecticide resistance (defined broadly as any ways in which populations of mosquitoes adapt to the presence of insecticide-treated nets (ITNs) in order to make them less effective) is a major threat to the effectiveness of ITNs." GiveWell: Insecticide Resistance and Malaria Control.
  • We adjust our cost-effectiveness model to account for insecticide resistance here: GiveWell: 2021 GiveWell cost-effectiveness analysis — version 3, "AMF" sheet, cell A57.

• 4
  • "We define LLIN coverage as the proportion of people living in households that were targeted by an AMF distribution who subsequently sleep under LLINs, thereby benefiting from the protective effect against malaria. . . . With one exception, the PDMs [post-distribution monitoring studies] we focus on have found reported coverage rates of between 75% and 90%." GiveWell: Against Malaria Foundation, section on "What proportion of targeted recipients use LLINs over time?"
  • "Interestingly, nets with no holes had an average of just over one mosquito per net. Presumably, this was due to improper usage and residents should be instructed on how to tuck their nets in to prevent mosquitoes from entering them." Ochomo et al. 2013, pg. 7.
  • We adjust our cost-effectiveness estimate for net use here: GiveWell: 2021 GiveWell cost-effectiveness analysis — version 3, "AMF" sheet, cell A51.

• 5

  For example, one study found that ITNs are less effective in some locations, such as Myanmar, "where the principal malaria vectors bite outdoors early in the evening, before people go in or near their ITN." GiveWell: Mass Distribution of Long-Lasting Insecticide-Treated Nets (LLINs), section on "How might the effectiveness of ITNs vary across settings with different malaria transmission patterns?," footnote 45.