2019 Cost-Effectiveness Analysis Changelog

This page provides details about changes that were made to our cost-effectiveness analysis (CEA) in 2019. Below each change, we share a table indicating how the change impacted our cost-effectiveness estimates for each charity. (If a charity doesn't appear in the table, it was not impacted by the change.) For past versions of our CEA, see this page.

Table of Contents

Version 6 — Published November 25, 2019

Link to the cost-effectiveness analysis file: 2019 CEA — Version 6

Change 1: Applied country-level leverage and funging adjustments

Our cost-effectiveness analyses (CEAs) use leverage and funging adjustments to account for the impact we expect GiveWell-directed funding to have on the actions of other funders. We discuss our overall approach to leverage and funging adjustments in this blog post.

Earlier this year, we restructured our CEAs to calculate country-level cost-effectiveness estimates for each charity (see below). With this update, we began applying leverage and funging adjustments separately by country as well.

Applying leverage and funging adjustments separately by country leads to a large increase in our estimate of Deworm the World's cost-effectiveness. This is because Deworm the World covers a lower proportion of the total cost of deworming programs in India than in other countries (details in footnote).1



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Deworm the World 17.5x 42.4x 142.2%
Malaria Consortium 8.5x 8.6x 0.8%

Change 2: Updated quantitative judgments used in leverage and funging adjustments

For our leverage and funging adjustments (see above), we make quantitative judgments about the probability that our charities are causing governments and multilateral aid agencies to spend more or less on a program than they otherwise would have.

We updated these quantitative judgments for each of our top charities (see footnote for details).2



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Deworm the World 42.4x 42.6x 0.4%
SCI Foundation 11.1x 10.5x -5.4%
Sightsavers 10.2x 9.7x -5.4%
Against Malaria Foundation 7.4x 7.6x 2.7%
Malaria Consortium 8.6x 8.1x -5.9%
The END Fund 6.1x 5.8x -5.1%

Change 3: Removed Mozambique from our CEA of Helen Keller International (HKI)

In July 2018, HKI told us that it was seeking additional funding to expand its work on vitamin A supplementation (VAS) mass campaigns to Mozambique.3 In July 2019, HKI told us that it was no longer seeking funding to expand to Mozambique, so we removed the country from our CEA.4



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Helen Keller International 5.4x 5.6x 4.9%

Change 4: Updated HKI external validity adjustment

We use external validity adjustments in our CEAs to account for the differences between the populations studied in the evidence base for interventions implemented by our top charities and the populations reached by our top charities today.5

Our external validity adjustment for HKI in 2018 was 23%, but with our updated calculations for 2019 it is 39%. This change increases HKI's cost-effectiveness by 69%. Our external validity adjustment for HKI takes two factors into account:

  • An estimate of the proportion of children under 5 with vitamin A deficiency (VAD) in countries where HKI works (as compared to our estimate of the proportion of children in the vitamin A supplementation randomized controlled trials (RCTs) with VAD).
  • The proportion of under-5 child mortalities we expect to be "vitamin A susceptible" today in countries where HKI works (compared to the proportion of under-5 mortalities we expect would have been "vitamin A susceptible" in the same countries in 1990, roughly around the time of vitamin A supplementation RCTs).

For additional details, see the How prevalent is vitamin A deficiency in areas where HKI works? and How high are child mortality rates in areas where HKI works? sections of our review of HKI.

Our updates to our external validity adjustment calculation:

  • In 2019, following a conversation with the Institute for Health Metrics and Evaluation (IHME), we decided to use VAD prevalence estimates from its Global Burden of Disease (GBD) project. IHME's estimates of VAD prevalence were higher than our previous estimates.6
  • We began using 2017 GBD data (which was the latest available) to estimate the proportion of under-5 child mortalities we expect to be "vitamin A susceptible" today in countries where HKI works.7 We had previously used a combination of 2016 GBD data and 2016 data from the UN Inter-agency Group for Mortality Estimation (UN IGME).8

We decided to use a single data source for estimates of mortality rates in our CEAs this year, in part because using multiple data sources for mortality rates made our models more difficult to update and understand. We do not have strong reasons for believing that GBD mortality estimates are more accurate than UN IGME estimates, but in some cases GBD provides more granularity. For example, the GBD results tool includes more granular age bracketing than the UN IGME database (e.g., "early neonatal" and "late neonatal" vs. "neonatal").



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Helen Keller International 5.6x 9.5x 69.3%

Change 5: Used 2017 Global Burden of Disease (GBD) data for baseline mortality rates for HKI

As described above, we began using 2017 mortality rate data from GBD in our CEAs this year. For HKI, mortality rate data affects two inputs into our CEA: the external validity adjustment (described above) and the baseline mortality rates among 6- to 59-month-old children in countries where HKI works.

Updating baseline mortality rates led to a small decrease in HKI's cost-effectiveness.



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Helen Keller International 9.5x 9.1x -4.8%

Change 6: Updated HKI internal validity adjustment

We use internal validity adjustments in our CEAs to account for the possibility that results presented in academic research may not reflect the actual effect an intervention had on the populations studied.9

Our CEA for HKI relies on Imdad et al. 2017, a Cochrane systematic review of RCTs of vitamin A supplementation (VAS) programs. Imdad et al. 2017's fixed-effect meta-analysis finds that VAS causes a 12% reduction in child mortality (95% confidence interval 7% to 17% reduction) and its random-effects meta-analysis finds that VAS causes a 24% reduction in child mortality (95% confidence interval 17% to 31% reduction).10

We use the random-effects estimate from Imdad et al. 2017 in our CEA of HKI. Our decision to rely on the random-effects model is supported by an unpublished work that we have not discussed publicly.

We updated our internal validity adjustment for HKI from 95% to 85%. This update reflects that we are moderately less confident in the random-effects point estimate from Imdad et al. 2017 compared to the evidence we rely on for our CEAs of seasonal malaria chemoprevention (SMC) and long-lasting insecticide-treated nets (LLINs).11



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Helen Keller International 9.1x 8.1x -10.7%

Change 7: Added external validity adjustment for Against Malaria Foundation (AMF)

We use external validity adjustments in our CEAs to account for the differences between the populations studied in the evidence base for programs implemented by our top charities and the populations reached by our top charities today.12 We had previously not included an external validity adjustment in our CEA of Against Malaria Foundation (AMF) because we had already accounted for the major considerations that we believe drive external validity separately in this model: net usage, country-specific mortality rates, and insecticide resistance. We added a small additional external validity adjustment (95%) to account for other possible differences between populations studied in the evidence base for LLINs and populations reached by AMF.

Changes to AMF's cost-effectiveness lead to small changes in the cost-effectiveness of other organizations (as displayed in the table below) because we use our cost-effectiveness estimates of AMF as an input into the "counterfactual non-philanthropic" sheet as part of our leverage and funging adjustments.



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Deworm the World 42.6x 42.6x 0.0%
Schistosomiasis Control Initiative 10.5x 10.5x 0.4%
Sightsavers 9.7x 9.7x 0.3%
Against Malaria Foundation 7.6x 7.2x -5.0%
Malaria Consortium 8.1x 8.0x -0.8%
The END Fund 5.8x 5.8x 0.1%

Change 8: Updated cost per seasonal malaria chemoprevention (SMC) treatment for Malaria Consortium

We used information Malaria Consortium shared with us this year to update our cost per SMC treatment analysis.13 The new estimates are similar to the previous estimates.



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Malaria Consortium 8.0x 8.0x 0.5%

Change 9: Updated malaria mortality and prevalence estimates for AMF

Our CEA of AMF relies on estimates of malaria prevalence and malaria mortality rates. We used updated estimates for these parameters for 2017 (the most recent data available) from the Global Burden of Disease (GBD)—we had previously been using GBD estimates for 2016.

Changes to AMF's cost-effectiveness lead to small changes in the cost-effectiveness of other organizations (as displayed in the table below) because we use our cost-effectiveness estimates of AMF as an input into the "counterfactual non-philanthropic" sheet as part of our leverage and funging adjustments.



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Deworm the World 42.6x 42.6x 0.0%
Schistosomiasis Control Initiative 10.5x 10.5x -0.4%
Sightsavers 9.7x 9.7x -0.2%
Against Malaria Foundation 7.2x 7.5x 5.0%
Malaria Consortium 8.0x 8.1x 0.8%
The END Fund 5.8x 5.8x -0.1%

Change 10: Updated model of SMC spillover developmental effects

Treating children with SMC may disrupt malaria transmission, reducing incidence of malaria in the wider (untreated) population. We discuss an RCT that estimates benefits for untreated populations in this section of our SMC intervention report.

Reducing exposure to malaria during childhood may have an effect on long term productivity and earnings ("developmental effects"). We have reviewed the evidence for the developmental effects of malaria prevention here.

Malaria Consortium's SMC programs target children aged 3 to 59 months.14 Our CEA of Malaria Consortium includes an estimate of the gains to long term productivity and earnings for children aged 14 and under (including both direct and indirect beneficiaries of SMC).

Previously, for the purpose of calculating developmental effects, we assumed that the untreated population of children under 14 years old was the same size as the treated population of 3- to 59-month-old children. We now explicitly calculate the size of the untreated population.15 This update led to a small increase in our cost-effectiveness estimate for Malaria Consortium.



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Malaria Consortium 8.1x 8.4x 3.6%

Change 11: Updated AMF insecticide resistance adjustment

Our CEA of AMF uses an adjustment to account for the impact of insecticide resistance on the effectiveness of LLINs. We updated our spreadsheet calculating insecticide resistance adjustments by country here. We updated our estimates of the proportion of nets AMF plans to purchase that incorporate piperonyl butoxide (PBO nets) and added information about insecticide resistance in Guinea and Nigeria.16 We later added Guinea and Nigeria to our CEA of AMF—see change 22 and change 35.

Changes to AMF's cost-effectiveness lead to small changes in the cost-effectiveness of other organizations (as displayed in the table below) because we use our cost-effectiveness estimates of AMF as an input into the "counterfactual non-philanthropic" sheet as part of our leverage and funging adjustments.



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Deworm the World 42.6x 42.6x 0.0%
Schistosomiasis Control Initiative 10.5x 10.4x -0.2%
Sightsavers 9.7x 9.6x -0.2%
Against Malaria Foundation 7.5x 7.8x 3.1%
Malaria Consortium 8.4x 8.4x 0.5%
The END Fund 5.8x 5.8x -0.1%

Change 12: Updated AMF cost per net

In 2018, our overall cost per net estimate for AMF was $4.53.17 In 2019, we updated our analysis of AMF's cost per net but did not publish an updated version of the spreadsheet with the data we used and our calculations because we have not received permission to publish country-specific cost estimates we received from the Global Fund to Fight AIDS, Tuberculosis, and Malaria, which our calculations rely on.18 Country-level cost per net estimates we updated for this change are available here.

Our overall cost per net estimate for AMF for 2019 is $4.59.19

Changes to AMF's cost-effectiveness lead to small changes in the cost-effectiveness of other organizations (as displayed in the table below) because we use our cost-effectiveness estimates of AMF as an input into the "counterfactual non-philanthropic" sheet as part of our leverage and funging adjustments.



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Deworm the World 42.6x 42.6x 0.0%
Schistosomiasis Control Initiative 10.4x 10.4x -0.5%
Sightsavers 9.6x 9.6x -0.3%
Against Malaria Foundation 7.8x 8.5x 9.0%
Malaria Consortium 8.4x 8.5x 1.0%
The END Fund 5.8x 5.8x -0.2%

Change 13: Used 2017 Global Burden of Disease (GBD) data for baseline mortality rates for Malaria Consortium

Our CEA for Malaria Consortium relies on estimates of baseline mortality rates for populations Malaria Consortium targets for SMC programs.

We previously used an average of 2016 GBD data and 2016 data from the World Bank and World Health Organization (WHO) to estimate all-cause mortality and the fraction of all-cause mortality caused by malaria.20 We updated our SMC CEA to use only 2017 GBD data, and to estimate malaria mortality directly (rather than estimating all-cause mortality and the fraction of all-cause mortality caused by malaria).21

We decided to use a single data source for estimations of mortality rates in our CEAs this year, in part because using multiple data sources for mortality rates made our models more difficult to update and understand. We do not have strong reasons for believing that GBD mortality estimates are more accurate than WHO and World Bank estimates, but in some cases GBD provides more granularity.22



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Malaria Consortium 8.5x 9.6x 12.3%

Change to the way we track changes

For the changes tracked above, we reported how the changes impacted our top charities' overall cost-effectiveness. Following change 13, we began to also track the impact of changes to our CEA on country-level cost-effectiveness estimates for each of our top charities. Each change below includes a link to a spreadsheet displaying our updated country-level and overall cost-effectiveness estimates.

Change 14: Updated estimates of where AMF will spend additional funding

We calculate both country-level cost-effectiveness estimates and overall cost-effectiveness estimates for our top charities. Our overall analyses use weighted averages of country-level parameters, weighted by the amount of funding we expect our top charities to spend in each country with marginal donations.

We updated our estimates of how AMF would allocate additional funding by country, which led to changes in our overall cost-effectiveness estimates for AMF. At the time of this change, we had estimated that AMF would spend a large majority (75%) of its funding going forward in the Democratic Republic of the Congo. This estimate was based on information AMF shared with us about its plans and on our past observations of how AMF's preliminary plans have materialized into firm commitments. Previously, our CEA used AMF's historical spending by country.23

We use our overall cost-effectiveness estimate of AMF as an input into the "counterfactual non-philanthropic" sheet as part of our leverage and funging adjustments, so this change had a small impact on our cost-effectiveness estimates for other charities as well.

For the impact of this change on our cost-effectiveness estimates, see this spreadsheet.

Change 15: Updated treatment effect for deworming

Our recommendation of deworming programs relies on a series of follow-ups to one experiment (Miguel and Kremer 2004), which find that reducing worm infection loads during childhood can have a significant impact on income later in life.

We previously relied on the results of the 10-year and 15-year follow-up studies—the Kenya Life Panel Survey 2 (KLPS-2) and Kenya Life Panel Survey 3 (KLPS-3)—to estimate the long-term impact of deworming on income in the population studied.24

Our new calculation uses new results from the fourth round of the Kenya Life Panel Survey (KLPS-4), a 20-year follow-up survey on long-run economic outcomes in the cohort from the original Miguel and Kremer deworming randomized controlled trial (RCT), which was published in 2004. These results have been provided to GiveWell by Professor Ted Miguel and the Kenya Life Panel Survey team. This work on KLPS-4 is ongoing and not yet published, so these results are subject to change. The authors have made preliminary figures and tables available for transparency. We expect to revisit this input if and when we receive updated information.

We now have three rounds of measurements for the economic outcomes of deworming from KLPS-2, -3, and -4, which surveyed RCT participants 10, 15, and 20 years after initial deworming treatment, respectively. KLPS-4 is the first survey round that collected data on per capita household consumption for the full sample in addition to individual earnings for the RCT study participant. Now that these results are available, we have chosen to factor both earnings and consumption results into our best-guess input. We include earnings because it is the most comparable measure between all KLPS rounds, and it is isolated to the individual contribution of the study participant. We also include per capita consumption because our impression is that it is likely to be a more comprehensive measure of economic benefits, particularly in locations where a high proportion of income-earners work in informal and varied occupations.

KLPS-4 found positive average treatment group differences on all measurements of economic life: income, consumption, and wealth, which is consistent with deworming boosting economic outcomes over the course of a career. However, the KLPS-4 results are smaller in magnitude (on a percentage increase basis) and higher variance than earlier survey rounds. Consumption benefits are larger than income benefits, although these results are not statistically significantly different from each other. The fall in observed benefits in KLPS-4 was the main driver of a roughly 25% decrease in this input.

For the impact of this change on our cost-effectiveness estimates, see this spreadsheet.

Change 16: Updated analysis of cost per child dewormed per year and changed which locations are modeled for Deworm the World

We incorporated information Deworm the World shared with us in 2019 into our cost per child dewormed per year analysis.25 These updates did not lead to much change in our cost per child dewormed per year estimates for India and Kenya, but caused decreases in our estimates for Nigeria, Pakistan, and Vietnam.26

We also removed the Indian union territory of Delhi and the Nigerian states of Bayelsa and Osun from our CEA. We added the Indian states of Haryana, Karnataka, and Uttarakhand. We made these changes based on where we expected Deworm the World to spend funding going forward.27

For the impact of the changes on our cost-effectiveness estimates, see this spreadsheet.

Change 17: Updated our analysis of HKI's cost per vitamin A supplement delivered

We incorporated information HKI shared with us in 2019 about its programs to update our cost per vitamin A supplement delivered estimate. Our updates led to a decrease in our overall estimate from $1.35 to $1.23.28

For the impact of this change on our cost-effectiveness estimates, see this spreadsheet.

Change 18: Updated analysis of cost per child dewormed per year and changed which locations are modeled for Sightsavers

We incorporated information Sightsavers shared with us in 2019 about its programs to update our cost per child dewormed estimate. We also began calculating cost per child dewormed per year estimates for Sightsavers separately by location.29 We also added Senegal to our analysis and changed the regional breakdown of Cameroon in our model. We made this update based on changes in Sightsavers' prioritization of additional spending opportunities.30

For the impact of the changes on our cost-effectiveness estimates, see this spreadsheet.

Change 19: Corrected external validity adjustment and baseline mortality estimates for HKI

We corrected an error in our external validity adjustment and baseline mortality estimates for HKI, which we had updated for change 4 and change 5 above.

Our 2019 external validity calculation spreadsheet previously included an error in our calculations on the "Child mortality rates by country" sheet. On the "External validity by country" sheet, we also began estimating VAD prevalence among children 6 to 59 months (the target population for VAS programs), rather than children under 5.

These changes led to a small increase (~0.4x cash) in our estimate of HKI's overall cost-effectiveness.

For the impact of the changes on our cost-effectiveness estimates, see this spreadsheet.

Change 20: Updated our cost breakdown among actors for Malaria Consortium

For this change, we updated the proportion of costs covered by Malaria Consortium and other actors based on our 2019 cost per SMC treatment analysis.

The proportions of total costs covered by Malaria Consortium and other actors are an input in our leverage and funging adjustments. This update led to minimal changes in our cost-effectiveness estimates for Malaria Consortium.

For the impact of the changes on our cost-effectiveness estimates, see this spreadsheet.

Change 21: Added Togo to our Malaria Consortium CEA

Malaria Consortium told us it is exploring the possibility of expanding its SMC program to Togo.31

We added Togo to our CEA of Malaria Consortium, but it did not receive any weight in our overall cost-effectiveness estimate as of this update.

For the impact of the change on our cost-effectiveness estimates, see this spreadsheet.

Change 22: Added Nigeria to our AMF CEA

We added Nigeria to our CEA of AMF. Nigeria is the country with the largest funding gap for nets, so we use it as a reference for estimating the cost-effectiveness of additional funding for AMF.32 Nigeria does not receive any weight in our overall cost-effectiveness estimate of AMF.

For the impact of the change on our cost-effectiveness estimates, see this spreadsheet.

Change 23: Removed errors in sums across countries

To calculate the impact of a hypothetical donation to a charity, we split the donation across different countries where we expect the charity to operate with additional funding, and sum the impacts from each of the countries. We discovered and fixed a small error in how these sums were calculated.

For the impact of this change on our cost-effectiveness estimates, see this spreadsheet.

Change 24: Changed the structure of our correction for imperfect compliance in deworming experiment

We noticed and fixed an error in how we account for the fact that there was imperfect compliance in the experiment we rely on to estimate the impact of deworming. This change was later superseded by change 29.

For the impact of the change on our cost-effectiveness estimates, see this spreadsheet.

Change 25: Restructured to calculate country-level cost-effectiveness for the END Fund

Earlier this year, we restructured our models of all of our top charities except the END Fund and GiveDirectly to calculate country-level cost-effectiveness estimates (see below). We now also calculate country-level cost-effectiveness estimates for the END Fund.

For the impact of the changes on our cost-effectiveness estimates, see this spreadsheet.

Change 26: Correction to analysis of cost per child dewormed per year for Sightsavers

We updated our analysis of Sightsavers’ cost per child dewormed per year with additional information Sightsavers shared with us on the costs of its program in Cameroon.33

For the impact of the changes on our cost-effectiveness estimates, see this spreadsheet.

Change 27: Updated user inputs

We removed user inputs from some staff who left GiveWell and added estimates from new staff. Staff who contributed inputs in previous years had the opportunity to update their inputs. Parameters that were updated were: discount rate, replicability adjustment for deworming, percent of VAS benefits coming from development effects, and duration of investment benefits (in years) for GiveDirectly's cash transfer programs.34

For the impact of the changes on our cost-effectiveness estimates, see this spreadsheet.

Change 28: Corrected Uttarakhand and Haryana cost figures in cost per child dewormed estimates for Deworm the World

We corrected errors in our cost per child dewormed estimates for Deworm the World in Uttarakhand and Haryana states in India. These corrections made a very small difference in our cost-effectiveness estimates for Deworm the World.

For the impact of the changes on our cost-effectiveness estimates, see this spreadsheet.

Change 29: Corrected adjustments for imperfect compliance in deworming experiment

We noticed and fixed an error we made in change 24 above on how we account for the fact that there was imperfect compliance in the experiment we rely on to estimate the impact of deworming.

We now explicitly calculate the treatment-on-the-treated effect size.35

For the impact of the changes on our cost-effectiveness estimates, see this spreadsheet.

Change 30: Corrected cost per child dewormed per year estimate for Sightsavers in Guinea

We noticed and fixed an error in our estimate of Sightsavers' cost per child dewormed per year in Guinea. We had previously used our overall cost per child dewormed per year estimate for Guinea ($0.73). With this update, we switched to using the estimate we calculated for Guinea specifically ($2.82).36

For the impact of the change on our cost-effectiveness estimates, see this spreadsheet.

Change 31: Updated our estimates of where AMF will spend additional funding

We again updated our estimates of where AMF will spend marginal funding (we describe our first update in change 14 above).

We calculate our updated estimates in our 2019 room for more funding analysis of AMF.

For the impact of the changes on our cost-effectiveness estimates, see this spreadsheet.

Change 32: Update to moral weights

GiveWell uses moral weights to make cost-effectiveness comparisons between interventions achieving different types of outcomes (e.g., averting the death of a child from malaria vs. doubling consumption for a year for a very poor household). See this page for a discussion of how GiveWell uses moral weights.

In March 2019, GiveWell made an Incubation Grant to IDinsight to support a scaled-up study surveying individuals broadly representative of the populations our top charities aim to serve about the relative value they place on different good outcomes.

IDinsight has shared the results of its survey with us. We plan to deeply vet this study and revisit our framework for moral weights over the next year, so we do not yet have robust conclusions on what our moral weights will be going forward.

In the interim, we are planning to use the following moral weights. The direction of these updates was driven by IDinsight's survey and additional arguments from staff in favor of putting more weight on health relative to income.


2019 moral weights GW staff median 2018
Doubling consumption 1 1
Averting an <5 death 100 48
Averting an >5 death 100 85

For the impact of the changes on our cost-effectiveness estimates, see this spreadsheet.

Change 33: Fixed an error in our relative baseline mortality calculation for AMF

Our CEA for AMF includes an adjustment for relative mortality rates in countries where AMF works today and the populations studied in Lengeler 2004 (the primary meta-analysis we rely on in our intervention report on LLIN distributions). We began using mortality rates in countries where AMF works today for 1- to 59-month-old children, rather than mortality rates for children under 5 years old, in order to match the age range of mortality rates reported in Lengeler 2004.

For the impact of the changes on our cost-effectiveness estimates, see this spreadsheet.

Change 34: Revised percentage of VAS benefits coming from development effects for HKI

We revised the parameter "percent of VAS benefits coming from development effects" downwards (from 25% to 22.5%) to account for decreases in the percentage of benefits coming from development effects in our CEAs for AMF and Malaria Consortium.

See this document for an explanation of how we arrived at our estimate for this parameter.

For the impact of this change on our cost-effectiveness estimates, see this spreadsheet.

Change 35: Added Guinea to our CEA of AMF

We added Guinea to our CEA of AMF. AMF funded a distribution in Guinea in 201937 and, in expectation, we believe that AMF will use a portion of marginal funding to support the next scheduled distribution in Guinea in 2022.

For the impact of this change on our cost-effectiveness estimates, see this spreadsheet.

Change 36: Updated treatment effect for deworming

We discuss a major update to our estimate of the treatment effect for deworming above. We updated our estimate of the treatment effect of deworming on ln(income) from 0.107 to 0.109, a 2% increase.

We revised this estimate based on our discussions with authors of KLPS-4.

To see the impact of this change on our cost-effectiveness estimates, see this spreadsheet.

Change 37: Corrected cost per child dewormed per year for Deworm the World in Nigeria

We noticed and fixed an error in our cost per child dewormed per year estimates for Cross River and Rivers states in Nigeria. We had previously used cost per child dewormed estimates only for areas of Cross River and Rivers states with one deworming treatment round per year (some areas of both states implement two deworming rounds per year).38

To see the impact of these changes on our cost-effectiveness estimates, see this spreadsheet.

Change 38: Fixed an error in our relative baseline mortality calculation for AMF in Nigeria

We noticed and fixed a spreadsheet error in our calculation of relative baseline mortality for Nigeria in our AMF CEA. For more about this parameter, see change 33 above.

To see the impact of this change on our cost-effectiveness estimates, see this spreadsheet.

Change 39: Set country weights for additional donations following our November 2019 recommendations to Open Philanthropy and allocation of Q3 discretionary funding

In November 2019, we recommended grants of $57.3 million to our top charities and standout charities, composed of a recommendation to Open Philanthropy to grant $54.6 million to our top charities, and $2.6 million in grants to top charities at GiveWell's discretion. See this page for additional details.

In our CEAs, we calculate both country-level cost-effectiveness estimates and overall cost-effectiveness estimates for our top charities. Our overall analyses use weighted averages of country-level parameters, weighted by the amount of funding we expect our top charities to spend in each country with marginal donations.

For this update, we set country-level weights to reflect our expectations of where our top charities would spend additional funding after taking our November 2019 recommendations to Open Philanthropy and allocation of discretionary funding into account.

To see the impact of these changes on our cost-effectiveness estimates, see this spreadsheet.

Version 5 — Published August 7, 2019

Link to the cost-effectiveness analysis file: 2019 CEA — Version 5

Change 1: Restructured to calculate country-level cost-effectiveness estimates and to use an outcomes-per-dollar structure

Background

Every year, we use our cost-effectiveness model as one of the primary inputs into our decisions about where to recommend that Good Ventures allocate funding among our top charities.

In past years, the structure of the model has made it difficult to use, and therefore more prone to error. A few problems we ran into last year included:

  1. The only way to look at the cost-effectiveness of individual programs was to make a separate copy of the model for each program.
  2. The main cost-effectiveness model didn't necessarily give the same result as the individual charity programs (due to different weightings), which made it more difficult to vet for errors.
  3. For the parameters for which we have country-specific data (e.g. baseline mortality rates), we were aggregating the country-specific data into a single figure, weighted by e.g. the proportion of the charity's funding that was spent in each country over the past few years. The country weightings for different parameters were calculated on separate spreadsheets, which made it possible for us to use different country weights for different parameters for the same charity. In putting together this update, we found that we had in fact used different country weights for the cost per treatment and deworming intensity adjustment parameters for Deworm the World.
  4. It was difficult to track the total good outcomes we expected to accomplish with a given grant.

Description of changes

To address these problems, we made two major changes to the structure of the model:

  1. Adding country-specific information to columns of intervention sheets: We are now using the columns in the intervention sheets to calculate the cost-effectiveness of programs by country (or by state, in some cases), addressing the first three problems listed above.
  2. Moving from a dollars-per-outcome structure to an outcomes-per-dollar structure: Previously, the CEA was structured to calculate the cost per outcome (for example, the cost per death averted). We restructured the model to start by inputting an arbitrary donation amount and modeling how that donation would be split across countries, and what outcomes would be achieved in each country with that amount of funding (e.g. how many deaths would be averted in each country, as well as in total). This allows us to consistently and easily model the total good outcomes that we expect a given grant to achieve, addressing the fourth problem listed above.

We also made some formatting changes to make the model easier to read, as well as making a few minor changes to address errors or areas of improvement that we found.

Impact of these changes on bottom-line cost-effectiveness estimates

Since this was primarily a structural update, we did not expect significant changes to the bottom line. These changes were mainly due to errors or inconsistencies that we identified and corrected.

The overall changes in cost-effectiveness between version 4 and version 5 are depicted in the table below. We include additional tables below for a breakdown of the impact of individual changes.



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Deworm the World 19.2x 17.5x -8.9%
Schistosomiasis Control Initiative 9.8x 11.1x 12.5%
Sightsavers 10.3x 10.2x -0.6%
Against Malaria Foundation 6.4x 7.4x 14.2%
Malaria Consortium 8.5x 8.5x 0.0%
Helen Keller International 5.8x 5.4x -7.6%
The END Fund 5.8x 6.1x 5.2%

Total change to the bottom line that resulted from the structural updates (rather than the other changes listed below)



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Sightsavers 10.3x 10.2x -1.0%
Against Malaria Foundation 6.4x 6.7 4.7%
Malaria Consortium 8.5x 8.4 -1.2%
Helen Keller International 5.8x 5.9 1.7%

Moved from a single overall insecticide resistance adjustment to country-specific adjustments



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Sightsavers 10.2x 10.3x 1.0%
Against Malaria Foundation 6.7x 6.6x -1.5%

Reduced the expected lifespan of an LLIN in DRC by 10%

The data we have seen from AMF's distributions in DRC provide some evidence that LLINs decayed considerably more quickly than expected. We added an adjustment of 10% to account for this.



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Schistosomiasis Control Initiative 9.8x 9.9x 1.0%
Against Malaria Foundation 6.6x 6.5x -1.5%

Moved from an overall estimate of the proportion of deworming going to children in SCI's programs to country-specific estimates

Since we don't have good data on the proportion of deworming treatments going to children in the END Fund's programs, we guess that the proportion is roughly similar to that in SCI's programs. Accordingly, changing the estimate for SCI also changed our estimate of the END Fund's cost-effectiveness.



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Schistosomiasis Control Initiative 9.9x 11.2x 13.1%
The END Fund 5.8x 6.1x 5.2%

Updated estimates of malaria prevalence in AMF context (Age <5)

We noticed that our estimates of malaria prevalence for children under age 5 in the countries where AMF works were out of date, so we updated this to use more recent data.



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Schistosomiasis Control Initiative 11.2x 11.1x -0.9%
Sightsavers 10.3x 10.2x -1.0%
Against Malaria Foundation 6.5x 6.8x 4.6%

Updated estimates of malaria prevalence in AMF context (Age 5-14)

We noticed that our estimates of malaria prevalence for children aged 5-14 in the countries where AMF works were out of date, so we updated this to use more recent data.



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Against Malaria Foundation 6.7x 7.0x 2.9%

Removed a 5% wastage adjustment for AMF

We removed this adjustment for consistency with our other charities. We plan to add this adjustment to the spreadsheet that we use at the end of the year to help with our decisions about what funding allocation to recommend to Good Ventures.



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Against Malaria Foundation 7.0x 7.4x 5.7%
Malaria Consortium 8.4x 8.5x 1.2%

Updated country weightings for Deworm the World

We noticed that for Deworm the World, we were previously using two different sets of country weights for different parameters. Specifically, the calculations of a) an overall cost per treatment and b) an overall deworming intensity adjustment both weighted countries based on the proportion of treatments that went to each country, but were looking at different time periods. We chose one of these time periods to use for both parameters.



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Deworm the World 19.2x 17.5x -8.9%

Updated the proportion of total benefits attributed to development effects for HKI

We were previously benchmarking our estimate of HKI's development effects off of our estimate of the development effects of deworming. We moved to using the median of staff estimates of the percentage of total benefits of vitamin A supplementation that come from development effects, which changed this percentage from 31% to 24%.



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Helen Keller International 5.9x 5.4x -8.5%

Version 4 — Published May 29, 2019

Link to the cost-effectiveness analysis file: 2019 CEA — Version 4

Change 1: Fixed an error in which moral weights were being used in the "Leverage/funging" sheet

It was brought to our attention that in some places in the "Leverage/funging" sheet, our calculations incorporated a single staff member's moral weights rather than our staff aggregate moral weights. We fixed the relevant calculations.



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Deworm the World 19.3x 19.2x -0.3%
Schistosomiasis Control Initiative 9.9x 9.8x -0.6%
Sightsavers 10.3x 10.3x -0.4%
Against Malaria Foundation 6.4x 6.4x 0.3%
Malaria Consortium 8.5x 8.5x -0.7%
Helen Keller International 5.9x 5.8x -0.9%
The END Fund 5.8x 5.8x -0.5%

Change 2: Added a parameter to the "Inclusion/exclusion" sheet

We added "Serious adverse events due to side effects of SMC treatment" to the "Inclusion/exclusion" sheet as a parameter that we have not included in the CEA. For more detail, see row 51 here.

Version 3 — Published March 21, 2019

Link to the cost-effectiveness analysis file: 2019 CEA — Version 3

Change 1: Fixed errors in calculation of insecticide resistance adjustment

We found and fixed some errors in the way we were calculating our estimate of the reduction in the effectiveness of anti-malarial bednets due to insecticide resistance.



Charity		 Median [Charity]
vs. cash before		 Median [Charity]
vs. cash after		 Percent
change
Deworm the World 19.3x 19.3x -0.1%
Schistosomiasis Control Initiative 10.0x 9.9x -1.0%
Sightsavers 10.4x 10.3x -0.6%
Against Malaria Foundation 5.6x 6.4x 15.5%
Malaria Consortium 8.5x 8.5x 1.1%
The END Fund 5.8x 5.8x -0.3%

Version 2 — Published January 25, 2019

Link to the cost-effectiveness analysis file: 2019 CEA — Version 2

Change 1: Moved from aggregating outputs of staff values to aggregating inputs

For some of the most uncertain and debatable parameters in our cost-effectiveness estimates, we attempt to use the wisdom of the crowds to reduce bias. That is, we allow multiple staff members to input their own best guesses, and then aggregate those answers to arrive at a central estimate. In this version of the CEA, we made a change to the way we aggregate staff estimates.

Previously, for each of our charities, each staff member would calculate their own cost-effectiveness results based on their own inputs. We took a median of these results to arrive at the final output of our CEA. We changed this to instead aggregate each input and feed these aggregate inputs through the model to arrive at the final output of our CEA.

The aggregated inputs can be seen on the User inputs and Moral weights sheets.

The primary advantages we see to this structure are:

  • It is easier to track how changes in staff inputs (including adding and removing staff members from the model) lead to changes in our final results, particularly when we make several changes simultaneously.
  • We believe there are theoretical reasons to aggregate staff inputs rather than aggregating outputs of the model, though we are not certain we fully understand every argument for this. In brief, this is because when aggregating outputs, we are taking into account information about who contributed which inputs (which in the majority of cases we believe is irrelevant) and disregarding other information that may be relevant. For example, if a staff member had a particularly low estimate for the value of preventing deaths relative to increasing consumption, it is likely that they would be towards the lower end of estimates for the cost-effectiveness of life-saving interventions, such as distributing anti-malaria bed nets. Their empirical estimates would therefore be disregarded, because they would be unlikely to influence which value is taken as the median for a given parameter.
  • We think that by making it clearer which values are being fed into the model, this structure makes it easier for others to productively disagree with the values we are using.

The primary disadvantages we see to this structure are:

  • As noted above, we are not taking into account any relevant information that may be communicated by which staff member contributed which inputs (for example, in cases where it would make sense for someone with a low value for one parameter to also have a low value for another parameter). For the most part we believe that the values staff use for each parameter should be independent of the values they use for others, but there may be a small number of cases where we are disregarding relevant information.
  • It is not clear to us what is the best way to aggregate moral weights. The primary difficulty comes from the fact that staff use different units to represent their moral weights, and we are therefore aggregating the ratios between staff values for different outcomes. When aggregating more than two ratios, the aggregate result changes depending on which outcome we normalize our results to (i.e. the outcome to which we arbitrarily assign a value of 1). To mitigate the effect of this arbitrary decision, we have normalized the moral weights to three different outcomes and taken the mean of the results of the three outcomes. We are not confident that this is the theoretically best approach, but have not prioritized further work as we believe it is unlikely to materially change our conclusions.

      Our bottom-line cost-effectiveness estimates for each charity changed slightly as a result of the new structure.



      Charity      		 Median [Charity]
      vs. cash before      		 Median [Charity]
      vs. cash after      		 Percent
      change
      Deworm the World 19.1x 19.3x 1.3%
      Schistosomiasis Control Initiative 9.6x 10.0x 4.3%
      Sightsavers 10.1x 10.4x 2.8%
      Against Malaria Foundation 5.8x 5.6x -3.8%
      Malaria Consortium 9.0x 8.5x -6.6%
      Helen Keller International 6.1x 5.9x -4.5%
      The END Fund 5.7x 5.8x 2.3%

      Version 1 — Published January 3, 2019

      Link to the cost-effectiveness analysis file: 2019 CEA — Version 1

      Change 1: Removed suggested inputs from "User inputs" sheet

      We removed the suggested values from the "User inputs" sheet because these values no longer represent our best guesses. The median result of the CEA was not affected by this change.

      Change 2: Removed two former staff members

      We removed two former staff members from the cost-effectiveness model.



      Charity      		 Median [Charity]
      vs. cash before      		 Median [Charity]
      vs. cash after      		 Percent
      change
      Deworm the World 18.3x 19.0x 3.9%
      Schistosomiasis Control Initiative 9.4x 9.6x 2.1%
      Sightsavers 9.8x 10.1x 3.3%
      The END Fund 5.5x 5.7x 3.9%

      Change 3: Fixed a small error in malaria development effects calculation

      We found and fixed a small error in the way we were calculating malaria development effects.



      Charity      		 Median [Charity]
      vs. cash before      		 Median [Charity]
      vs. cash after      		 Percent
      change
      Deworm the World 19.0x 19.1x 0.0%
      Schistosomiasis Control Initiative 9.6x 9.6x 0.2%
      Sightsavers 10.1x 10.1x 0.1%
      Against Malaria Foundation 5.9x 5.8x -2.4%
      Malaria Consortium 9.3x 9.0x -3.0%
      The END Fund 5.7x 5.7x 0.1%

      • 1
        • We estimate that philanthropic actors (including Deworm the World) cover 11% of the total cost of deworming programs Deworm the World supports in India. For Kenya, Nigeria, and Pakistan, we estimate that philanthropic actors cover 63% of total costs, and for Vietnam we estimate that philanthropic actors cover 34% of total costs. See our 2019 cost per child dewormed per year analysis for Deworm the World for additional details.
        • Before this update, we calculated a single leverage and funging adjustment for all countries Deworm the World works in, based on the average costs covered by different actors across all countries Deworm the World works in. See this section of 2019 GiveWell Cost-effectiveness Analysis — Version 5.
        • When calculating leverage and funging adjustments separately by country, we estimate large increases in cost-effectiveness for India, where Deworm the World and other philanthropic actors cover a relatively small proportion of the total cost. See this section of version 6 of our CEA.

      • 2

        See this spreadsheet for a comparison between our previous and updated quantitative judgments.

      • 3

        HKI room for more funding report July 2018, Pg 12-13.

      • 4

        Mozambique is not listed in HKI room for more funding report July 2019 redacted, Pg 8, Table 4.

      • 5

        See this section of our guide to GiveWell CEAs for a general discussion of external validity adjustments.

      • 6

        See this section of our HKI review for more details.

      • 7

        See the final bullet point of this section of our review of HKI for details on our estimation of "vitamin A susceptible" mortalities.

      • 8

        Our 2018 external validity spreadsheet for HKI includes 2016 GBD data and 2016 data from UN IGME. The calculations used an average of GBD and UN IGME data.

      • 9

        See this section of the guide to GiveWell CEAs for a general discussion of internal validity adjustments.

      • 10

        See this section of our review of HKI for additional details on Imdad et al. 2017 and fixed-effect and random-effects meta-analyses.

      • 11

        We use 95% internal validity adjustments for our CEAs of SMC and LLINs. See here and here in our 2019 CEA — version 6.

      • 12

        See this section of our guide to GiveWell CEAs for a general discussion of external validity adjustments.

      • 13
        • Our 2019 cost per SMC treatment spreadsheet incorporates information Malaria Consortium shared with us on its costs in Nigeria, Burkina Faso, and Chad in 2018. See "2018 costs" sheet.
        • We also incorporated data from Malaria Consortium's coverage surveys in 2018. See "Coverage" sheet in the spreadsheet linked above.
        • For comparison, the cost per SMC treatment analysis we completed in 2018 is available here.

      • 14

        See this section of our review of Malaria Consortium for additional details.

      • 15

        Compare the "Income increases — age 14 and under" section of the 2019 version 5 CEA to the same section of our 2019 version 6 CEA.

      • 16

        In the 2019 version 5 of our CEA, we used insecticide resistance adjustments calculated on an internal spreadsheet. Our updated insecticide resistance adjustment spreadsheet is here.

      • 17

        See our 2018 cost per net analysis.

      • 18

        See this section of our November 2019 review of AMF for additional details.

      • 19

        Our cost-effectiveness table for this change indicates that AMF's overall cost-effectiveness increased by 9%, even though our overall cost per net estimate for AMF rose slightly (from $4.53 to $4.59). This is because Guinea is included in our overall cost per net estimate, but is not yet included in our cost-effectiveness estimate. We add Guinea to our AMF CEA in change 35.

      • 20

        See the "Baseline mortality rate" section in this version of our Malaria Consortium CEA.

      • 21

        See the "Baseline mortality rate" section in the 2019 GiveWell CEA — version 6.

      • 22
        • For example, the GBD results tool includes more granular age bracketing than the UN IGME database (e.g., "early neonatal" and "late neonatal" vs. "neonatal").
        • Note that UN IGME child mortality estimates are created by WHO, the World Bank, and other groups:
          • "The United Nations Inter-agency Group for Child Mortality Estimation (UN IGME) was formed in 2004 to share data on child mortality, improve methods for child mortality estimation, report on progress towards child survival goals, and enhance country capacity to produce timely and properly assessed estimates of child mortality. The UN IGME is led by the United Nations Children’s Fund (UNICEF) and includes the World Health Organization (WHO), the World Bank Group and the United Nations Population Division of the Department of Economic and Social Affairs as full members." UN IGME about page, accessed November 2019.

      • 23

        See our 2019 version 5 CEA.

      • 24
        • Roughly 10 years after the initial experiment described in Miguel and Kremer 2004, the Kenya Life Panel Survey 2 (KLPS-2) measured earnings in the Miguel and Kremer 2004 study population. This was followed by a 15-year follow-up survey, KLPS-3.
        • We estimate that the average treatment household experienced a 15.4% increase in earnings. This corresponds to an increase in ln(earnings) of 0.143. This estimate is based on a preliminary, confidential analysis by Ted Miguel and others that pools earnings data from KLPS-2 and KLPS-3. The estimate is formed using the total earnings findings across the whole sample (including individuals with zero earnings) and trimming the top 1% of earners in both the treatment and control groups. We take the log effect of earnings at the sample mean. The p-value for this result is .019. The authors have graciously shared their preliminary estimates with us, but we cannot yet publish the full details.
        • Note that this is an intention-to-treat effect. Not all individuals in the study population received treatment. We adjust this estimate for treatment compliance later using a parameter that captures the additional years of deworming received by the treatment group.

      • 25

        See this section of our November 2019 review of Deworm the World for details.

      • 26

        Compare our 2019 cost per child dewormed per year estimates on the "Summary" sheet here to the "Total cost per child dewormed per year" estimates from 2018 on the "Weighted average of country-level estimates" sheet in our 2018 analysis.

      • 27

        See our 2019 room for more funding analysis of Deworm the World for more details.

      • 28

      • 29

        Compare our 2019 cost per child dewormed per year analysis for Sightsavers to our 2018 cost per child dewormed per year analysis.

      • 30
        • Prior to this update, we included three areas of Cameroon, based on Sightsavers' priority levels for expanding to each area. We now include Cameroon Littoral and Cameroon Sud.
        • We made this update based on changes in Sightsavers’ prioritization of additional spending opportunities in Cameroon between our 2018 room for more funding analysis and our 2019 room for more funding analysis.
        • We added Senegal to the model because Sightsavers is seeking additional funding to expand to Senegal. See our 2019 room for more funding analysis.

      • 31

        Christian Rassi, SMC Programme Director, Malaria Consortium, email to GiveWell, November 19, 2019.

      • 32

        "The largest absolute funding gaps are in Nigeria ($330 million), DRC ($95 million), Uganda ($42 million), and Kenya ($34 million)." GiveWell's November 2018 page on Estimating the Funding Gaps for Distribution of Antimalarial Nets and Seasonal Malaria Chemoprevention.

      • 33

        See this cell in our cost per child dewormed per year analysis of Sightsavers.

      • 34

        Compare our previous "User inputs" sheet here to our updated sheet here.

      • 35

        See the "Estimated treatment effect on the treated" calculation in version 6 of our 2019 CEA here. See version 5 of our 2019 CEA for comparison.

      • 36

        See our 2019 cost per child dewormed per year analysis for Sightsavers for additional information.

      • 37

        See our 2019 spreadsheet summarizing AMF's distributions.

      • 38

        Our corrected cost per child dewormed estimates for Cross River and Rivers states are here.