What We Learned From Red Teaming Our Top Charities

Published: September 2024; Last updated: November 2024

Summary

What we did

From August to November 2023, we conducted "red teaming" to pressure-test the research underlying these four top charities. A team of about five research staff spent two weeks on each charity, with additional support from 5-7 researchers and grantmakers. We also got feedback from 1-4 external experts per charity. Our goal was not to produce a comprehensive scorecard of our grantmaking, but rather to surface the highest priority issues for further investigation. We asked: How could we be wrong? What potential problems are most likely to worsen our grantmaking or the credibility of our research? (more)

Main issues we found

We identified several issues. We think some of these are mistakes (either because they led to important errors or because they worsened the credibility of our research) that we should have caught already. We think the rest are issues that we had been aware of and had deprioritized, but are worth looking into now. (These findings are based both on internal analysis and feedback from external experts we spoke to. A summary of external reviewers’ comments is here.)

Mistakes that we think we should have already addressed

• Failure to fully account for individuals receiving interventions from other sources. For example, we’ve typically assumed only 5% of the population in countries where we fund nets campaigns would obtain nets from non-campaign sources, in line with the ~30-year-old randomized controlled trials of net distribution we rely on. However, we’ve reviewed recent evidence suggesting this may be higher in current distributions. If this and similar issues hold up after further investigation, it could mean we’re overestimating the effect of top charities on coverage and, more broadly, need to better understand interaction with other programs and alternative distribution approaches (like routine distribution). (more)
• Failure to sense check raw data in burden calculations. We’ve found surprising differences in all-cause mortality estimates in different states in Nigeria between the Institute for Health Metrics and Evaluation’s (IHME) Global Burden of Disease data (our preferred source for mortality data) and other sources, which could impact which states we decide to fund New Incentives in, and surprisingly low malaria-specific mortality in some countries. We think we’ve taken this burden of disease data too much at face value, which makes us vulnerable to letting noisy data drive what grants we make. (more)
• Failure to estimate the interactions and overlap between programs. In regions like Northern Nigeria, many programs are being delivered at once (e.g., ITNs, SMC, vaccines, ORS, azithromycin). We’re unsure if we’re modeling the impact of our top charities correctly, given these overlapping programs. For example, we think failing to account for the interaction between vitamin A supplementation and azithromycin distribution could lead to a ~20% overestimation of VAS cost-effectiveness. (more)
• Failure to more frequently engage with outside experts. We've identified some issues in our analyses through recent interactions with external experts (e.g., overestimation of net durability and underestimation of routine net distribution). This suggests we may be missing important perspectives by not engaging with implementation experts, researchers, and other stakeholders more frequently. (more)

Potential issues that we deprioritized previously but that we think are worth looking into now

• Insufficient consideration of other ways to support uptake of effective interventions. We have focused mainly on direct delivery methods for interventions like vitamin A supplementation and nets. However, fortification may be a more effective way to increase vitamin A uptake in some regions, or there may be other opportunities to increase the use of nets (e.g., market shaping to lower prices, research and development into new types of nets). We haven’t investigated these in-depth, which means we may be missing out on more cost-effective funding opportunities. (more)
• Insufficient investigation of how our funding decisions affect other funders. We think we could do more to understand how much our funding of top charities crowds in or out other funders. We might do this by, for example, commissioning an external review of our assumptions to get a more objective view on these questions, looking at how long it took for programs we decided not to fund to get funded, or trying to find specific funding opportunities where we’re especially likely to crowd in funding from others. (more)
• Insufficient attention to some fundamental drivers of intervention efficacy. For example, we think our assumptions about insecticide resistance and net durability are outdated, and we’ve postponed research on vaccine efficacy in Nigeria and other countries we fund. Doing these checks not only informs cost-effectiveness estimates but may also reveal new grant opportunities (e.g., support for improving vaccine efficacy if we find it’s low, support for next-generation nets less affected by insecticide resistance). (more)
• Insufficient attention to inconsistency across CEAs. For example, our assumptions about “indirect deaths” (deaths prevented by, say, a malaria prevention program that aren’t directly attributable to malaria) range from 0.75 for SMC and nets to over 2 for water chlorination programs. We haven’t done the work to think through whether these differences are plausible. (more)
• Insufficient focus on simplicity in cost-effectiveness models. Our CEAs have become too complex, with models like the main CEA for ITNs spanning 225 lines. We think this complexity led to at least two errors. More broadly, high complexity makes it difficult to understand and critique our models. (more)
• Insufficient investigation of some factors our models may miss. Our models include only rough, ad hoc adjustments for issues like transmission dynamics, and it's possible more complex models by epidemiologists would show meaningfully different cost-effectiveness estimates. If we were to add these types of considerations, we'd want to do it in a way that still preserves legibility and consistency across our models. (more)
• Insufficient consideration of what happens when conditional cash transfers for vaccination are stopped. At the time of red teaming, we had deprioritized investigation of what would happen if we stopped funding New Incentives’ program in a given area. I.e., what would happen to vaccination rates in Northern Nigeria if we removed conditional cash transfers for vaccination? We now think this was a mistake and that we should look more into the potential effects of ending a program we fund. (more)
• Insufficient follow-up on potentially concerning monitoring and costing data. For example, in our VAS cost-effectiveness analysis, we used aggregate data instead of country-specific costs for four countries where the data seemed implausibly low. While this conservative approach avoids relying on potentially unreliable data, it misses an opportunity to understand how some countries may be achieving lower costs or whether this is indicative of lower quality monitoring and program implementation in these countries. (more)
• Insufficient sideways checks on coverage, costs, and program impact. We haven't regularly cross-checked our data on program coverage, costs, and impact against external sources like Demographic and Health Surveys (DHS) or other organizations' reports. While initial checks during red teaming didn't reveal major discrepancies, we think more regularly doing these checks could increase confidence in our work and surface errors. (more)
• Insufficient attention to high uncertainty regarding VAS. We think the case for VAS is categorically more uncertain than our top charities, given fundamental questions about VAS’s effect on child mortality. We need to consider how this uncertainty will influence our grantmaking. (more)

We haven’t thoroughly investigated all of these issues yet, and our conclusions could change as we look into these further. This should be viewed more as a prioritized to-do list than a set of final determinations.

Implications for grantmaking and immediate next steps

Overall, we continue to think that our top charities are among the most impactful opportunities that donors can support. However, we expect that addressing the issues we identified during red teaming will lead to changes in our grant decisions this year.

Our best guess is that these changes will result in:

• $5m-$40m more in grants we wouldn’t have made otherwise (e.g., due to learning that disease burden is higher in some areas than we previously thought)
• $5m-$40m less in grants we would’ve made otherwise (e.g., because we decide to explore other ways to promote vitamin A beyond supplementation campaigns or because conversations with experts show that certain opportunities are less promising than we thought).3

We currently expect that we’ll make ~$325 million in grants in 2024. These predictions suggest changes from red teaming can lead to “replacing” 2-12% of our grantmaking over this year.4 In 2025, we plan to look back at all of our work in 2024 and estimate the impact red-teaming had on our grantmaking against these forecasts.

We’ve already begun looking further into these issues, and in grants we've made recently, we've already adjusted our process to account for several of them.

Because we haven’t fully investigated these issues yet, we’re uncertain about their impact on our grantmaking, and their impact could end up being more or less significant than we currently anticipate. We also think it's possible that further work on all of these issues could lead to more substantial changes to our grantmaking or cause us to reclassify “potential issues we should look more into” as “mistakes.”

Our next steps:

• We expect to continue making grants to our top charities this year, and we will prioritize investigation of these issues as we evaluate specific grants. For example, we’ve started accounting for higher-than-expected use of routine distribution of nets and concerns about IHME data on malaria mortality as part of our investigation into grants to AMF.
• We're implementing immediate changes to our research process, including more frequent engagement with outside experts and additional cross-checks on key data points.
• We plan to follow up on these issues and report back next year on what we've found, including any significant updates to our analyses or recommendations.
• We plan to red-team 1-2 additional grantmaking areas in 2024 to ensure we're applying this critical approach beyond just our top charities.
• In the future, we’ll consider how to apply these same lessons to programs beyond our top charities, too. (more)

Main reservations about findings from red teaming and limitations

• Our red teaming did not comprehensively vet the research behind our top charities, and we think it’s likely we’ve missed some issues. For example, following red teaming, we received feedback arguing that the Cochrane meta-analysis we rely on in our VAS cost-effectiveness analysis shows evidence of substantial publication bias, which we had not flagged during the red-teaming process.
• We haven’t thoroughly investigated all of the issues above yet. It’s possible that, after further investigation, we find they have a much larger impact on our grantmaking than we expect.
• It’s possible that the issues we did identify may end up having a limited impact on our grantmaking or end up being intractable.
• The issues we identified may inadvertently skew negative overall, though we tried to correct for that potential bias and note some factors that could be positive.
• We had limited back-and-forth with external experts during the red teaming process, and we think more engagement with individuals outside of GiveWell could improve the process.
• While we benefited from having more legible intervention reports during red teaming, there were some areas where legibility was a roadblock (e.g., see our discussion of the timing of nets campaigns parameter here).

What did we do?

We conducted a “red team” process to critically examine the case for our four top charities:

• Malaria Consortium’s seasonal malaria chemoprevention (SMC) program5
• Against Malaria Foundation’s mass distribution of insecticide-treated nets (ITNs)6
• Helen Keller International’s vitamin A supplementation (VAS)7
• New Incentives’ conditional cash transfers to increase infant vaccination8

We dedicated a team of ~5 research staff to work full-time for 2 weeks for each top charity. The majority of this dedicated team did not have significant experience investigating these areas, giving them the advantage of a relatively unbiased starting point. An additional 5-7 researchers with more expertise in these areas of our research provided support by answering questions and assessing the red team’s arguments.

The goal was to answer the following questions:

• What do you think are the top issues we should look into across our top charities?
• What are the ways we could be over- or underestimating the impact of our top charities?
• Are there ways we could approach our grantmaking for these top charities differently? If we were investigating this program for the first time today, would we approach it in a new way?

In parallel, we asked for feedback on our reports from 1-4 external experts per intervention, including researchers, academics, and leaders at NGOs.

We asked experts:

• Do you understand why we believe this program is an effective giving opportunity? Which parts are unclear?
• What are the ways you think we could be wrong? Which assumptions or claims seem off to you? Why?

They provided written briefs and, when possible, met with our internal red team to discuss their feedback. We’ve incorporated findings from these experts into conclusions in the following section on the top issues identified during red teaming. We also provide more detail on the expert reviews in this section.

What were the top issues we identified? What do we plan to do about them?

Failure to fully account for individuals receiving interventions from other sources

One of the reasons we think our top charities are cost-effective is because they cause a large uptake of interventions. For example, we think free net distribution via campaigns leads to many more individuals receiving nets, relative to if there were no campaigns and people had to get nets from other sources.9

We found a few cases where we think we’ve done insufficient research to understand whether individuals receiving nets, vaccines, SMC, or vitamin A supplements as a result of our top charities’ programs would’ve received those interventions from other sources:

• Nets received from sources other than campaigns. In our analysis of insecticide-treated net (ITN) distribution campaigns in the Democratic Republic of the Congo (DRC), for example, we assume that only 5% of the population would obtain nets from alternative (non-campaign) sources based on trial data.10 However, these trials were conducted ~30 years ago,11 and we think it’s possible that in more recent distributions, this figure is higher. For example, DRC’s Global Fund application included requests for over 30 million nets for routine distribution, targeted at young children (via immunization visits) and expecting mothers (via antenatal care).12 Our initial best guess was that this would imply that around 25-50% of children under 3 would be covered by routine distribution,13 which would lower cost-effectiveness by 15-30%.14 We are updating our analysis to account for this concern in DRC in 2024 and are planning to look into counterfactual coverage in other countries.
• Vitamin A deficiency. We describe our concern about vitamin A deficiency (VAD) rate estimates in our VAS intervention report: “Our analysis is very sensitive to vitamin A deficiency rates today, but our estimates are based on information we have low confidence in (10- to 20-year-old surveys of deficiency, updated for change over time, and modeled estimates from the Global Burden of Disease Project whose methodology we do not fully understand). Since these surveys were conducted, many countries have introduced vitamin A fortification programs, and we’re unsure how effective these have been at reducing deficiency rates.” We think there’s additional work that could change our estimates of vitamin A deficiency (VAD) in settings where we fund VAS campaigns that we’ve deprioritized (e.g., funding additional surveys of VAD or better understanding fortification efforts happening alongside VAS campaigns).
• Counterfactual vaccination rates in Nigeria. When estimating the effectiveness of New Incentives' conditional cash transfer program for increasing vaccination rates, we rely on New Incentives’ coverage surveys for counterfactual coverage15 and our adjustment is equivalent to assuming that coverage increases by roughly 1.5 percentage points per year.16 However, Demographic and Health Surveys (DHS) and Multiple Indicator Cluster Surveys (MICS) indicate that counterfactual vaccine coverage was increasing by 5 percentage points per year in several Nigerian states prior to New Incentives' entry.17 This suggests we could be overestimating the cost-effectiveness of New Incentives’ program by ~10%.18

These examples make us think this is an issue we’ve neglected more broadly. As a result, we may not only misestimate cost-effectiveness (e.g., by overstating how much our programs increase coverage) but may also fail to understand how campaigns we fund interact with clinics and routine distribution of health programs (e.g., should we be considering supporting routine distribution, rather than campaigns, in some areas?).

To address this, we plan to:

• Understand how common it is for individuals to receive nets outside of campaigns in DRC and Nigeria, two countries where we’re considering large nets grants. What is our best guess for coverage from continuous distribution or other channels currently in both DRC and Nigeria?
• Explore whether and how to fund additional surveys of Vitamin A deficiency in countries where we expect to consider large VAS grants.
• Update our estimates of counterfactual vaccination coverage for New Incentives' program to account for the increase in vaccination rates over time. We will compare our revised estimates to the actual coverage achieved in areas where New Incentives operates to check the accuracy of our assumptions.
• Explicitly state our assumptions on the percentage of individuals who would receive interventions like nets from other sources without campaigns. (We’ve already incorporated this into our updated top charity pages.)19
• Talk to experts about the extent to which campaigns for health commodities we fund interact with routine distribution, and consider whether we should be supporting routine distribution, at least in some areas. Is our funding inadvertently leading to the weakening of routine coverage systems? What would the consequences of this be? Are there areas where supporting routine distribution would be cost-effective?

Failure to sense check raw data in burden calculations

We identified two cases where we used data inputs in our cost-effectiveness analyses without sufficiently examining their accuracy and consistency with other data sources.

• All-cause mortality across Nigerian states. Our estimates of under-5 all-cause mortality rates in Nigerian states, which are a key driver of our decisions about where to fund New Incentives' conditional cash transfer program, rely heavily on data from the Institute for Health Metrics and Evaluation (IHME).20 However, we found that IHME's estimates are negatively correlated with estimates from the United Nations Interagency Group for Child-Mortality Estimation (UN IGME) across the states where New Incentives operates.21 If UN IGME estimates are more accurate, relying solely on IHME could lead us to fund New Incentives’ program in less cost-effective states.22

  Additionally, the UN IGME’s estimates are, on average, 35-40% higher than IHME's estimates in states where New Incentives operated as of January 2024.23 If we come to believe that UN IGME’s results are more reliable, or it is best to take an average of the two, then relying on IHME alone is causing us to underestimate overall disease burden and, consequently, underestimate the potential impact of New Incentives' program.

• Malaria mortality in Chad. Our estimates of the malaria burden in Chad are significantly lower than in other countries where we fund malaria interventions, but we have not thoroughly investigated the reasons for this discrepancy. A comparison of under-5 malaria mortality estimates from the UN IGME and IHME across several African countries reveals substantial variation. For example, the UN IGME estimate for under-5 malaria mortality in Chad is ~2.5 times higher than IHME's estimate, while in Guinea, the estimates are virtually the same, and in Uganda, the UN IGME estimate is only ~45% as high as IHME's.24 These inconsistencies raise questions about the reliability of the data we use to estimate the malaria burden and, consequently, the cost-effectiveness of malaria interventions in different countries.

We discuss other examples on this page, which covers our approach to uncertainty in our modeling, as well as on our Mistakes page.

Using inconsistent or potentially inaccurate data inputs can lead to under- or over-estimation of the impact of the programs we fund. For example, we may be underfunding states where IHME data has a low mortality estimate due in part to noise, or overfunding states where IHME data shows higher burden.

To address these issues, we plan to:

• Audit analyses of other programs to understand where else this issue is present. We guess this is much more prevalent than just the examples we’ve cited.
• Investigate the discrepancies between IHME and UN IGME estimates of under-5 mortality rates in Nigerian states. We will consult with experts in the field and examine the methodologies used by each organization to better understand the reasons for the differences. In the meantime, we will consider using an average of state-level and zonal estimates to reduce the impact of potential noise in the data.
• Conduct a systematic review of the data sources and methodologies used to estimate the malaria burden across the countries where we fund malaria interventions. We will engage with experts and compare our estimates to those from other reputable organizations to identify any inconsistencies and potential areas for improvement. In the meantime, we’ll include a note in our nets and SMC CEAs to explain how to deal with specific countries like Chad with burden data that’s very different from other data sources.
• Develop a standardized process for scrutinizing key data inputs in our cost-effectiveness analyses, including comparing estimates from multiple sources, engaging with experts, and documenting any discrepancies and the reasons for our chosen inputs.

Failure to estimate the interactions and overlap between programs

By focusing on the delivery of a specific health program, we’re susceptible to ignoring interactions or overlap with other programs. We found a few cases where we think this could be an issue:

• Overlapping programs in places like Northern Nigeria. In areas like Northern Nigeria, GiveWell or others are funding insecticide-treated nets (ITNs), seasonal malaria chemoprevention (SMC), vaccines, oral rehydration solution (ORS), increased access to clean water, azithromycin, and other programs.25 If these programs interact or address the same health issues, we could be incorrectly estimating their combined impact, potentially leading to, for example, fewer child deaths averted than we estimate.
• Interaction between vitamin A supplementation (VAS) and azithromycin distribution. In 2020, WHO issued a conditional recommendation for mass distribution of azithromycin in high-mortality settings,26 and we expect that it will scale up significantly in locations where GiveWell funds VAS (which have very high child mortality rates) in the near future. If we do not account for this, we might overestimate the effectiveness of VAS by approximately 20%.27 On the other hand, delivering azithromycin alongside VAS could achieve the benefits of both treatments with only a minor increase in costs. Failing to consider overlaps could mean missing out on ways to deliver similar impact at much lower cost.

More broadly, we think it’s possible we’re devoting insufficient attention to the impact that funding vertical programs (e.g. programs that only deliver a specific drug) can have on health systems. Critics argue that by focusing on vertical programs, we may be neglecting the importance of building strong, sustainable health systems that can effectively deliver a variety of essential health services. Investing in horizontal programs (e.g., health systems strengthening, broader support for clinics or community-based programs beyond specific campaigns) could lead to more long-term, systemic improvements in health outcomes.

To address these issues, we plan to:

• Develop an approach to model overlapping effects of programs and address this issue in upcoming grant investigations where overlap is most likely (e.g., azithromycin and VAS).
• In the future, publish our view on why we typically support vertical over horizontal programs in order to solicit feedback and pushback.

Failure to more frequently engage with outside experts

Some of the issues we identified through red-teaming came, at least in part, from engaging with people outside of GiveWell. This suggests we might be missing important issues by not reaching out to outside experts (e.g., implementation experts, researchers, individuals with in-country experience, fellow funders, etc.) more frequently.

Two examples:

• Net durability issues: Two experts we engaged with during red teaming, Justin Cohen, Vice President of Malaria and Neglected Tropical Disease at the Clinton Health Access Initiative, and David McGuire, Director of Access and Country Engagement at the Innovative Vector Control Consortium (IVCC), suggested we may be using overly optimistic or outdated assumptions on net durability. (See discussion of net durability below.)
• Routine distribution of nets: In parallel with red teaming, our malaria team spoke to malaria experts and program implementers about the routine distribution of nets. These conversations increased our concern that more individuals were receiving nets via routine distribution than we’d previously estimated. (See discussion of routine distribution of nets above.)

To address this, we plan to:

• More regularly attend conferences with experts in areas in which we fund programs (malaria, vaccination, etc.).
• Reach out to experts more regularly as part of grant investigations and intervention research. We’ve always consulted with program implementers, researchers, and others through the course of our work, but we think we should allocate more relative time to conversations over desk research in most cases.
• Experiment with new approaches for getting feedback on our work.

Insufficient consideration of other ways to support uptake of effective interventions

Our top charities aim to increase uptake of effective interventions like vitamin A supplementation, SMC, vaccines, and insecticide-treated nets primarily through direct delivery of the intervention. However, there may be other approaches to increasing uptake of these interventions that could be as or more cost-effective. We may miss these if we’re narrowly focused on campaigns or traditional funding opportunities.

We think there are at least two cases where we’ve under-investigated other potential opportunities to support uptake of effective interventions:

• Vitamin A. GiveWell directs a significant amount of funding toward supporting VAS through Helen Keller International and Nutrition International. However, we have not historically supported fortification campaigns, which one external expert suggested may be a cost-effective alternative in some regions.28 During and following red teaming, we conducted an initial unpublished analysis that suggests vitamin A fortification could be a promising opportunity.29
• Nets. Over the past several years, GiveWell has directed hundreds of millions of dollars to a single partner, the Against Malaria Foundation (AMF), and, more recently, to Malaria Consortium, for ITN procurement and distribution.30 However, we think it’s possible there are alternative approaches to improving the use of nets that could be as or more impactful. This could include market shaping (to increase access by lowering prices), investment in more durable or effective nets, behavior change programs to increase uptake of nets in cases where use is low even among those who receive nets, or alternative grantees for nets campaigns.31

It’s possible that these alternative approaches are not as cost-effective as our current top charities or that there aren’t genuine funding gaps for them, but we think it’s worth exploring further.

To address these concerns, we plan to:

• Assess the cost-effectiveness and funding gaps for vitamin A fortification programs, to determine whether this approach could complement or partially replace vitamin A supplementation in some regions.
• Learn more about other ways philanthropic funding could support more uptake of nets or more effective nets by talking to other funders, malaria experts, and implementers.
• In the future, beyond our top charities, look for additional ways to support effective programs beyond just, for example, direct delivery. Are there alternative ways to support uptake of oral rehydration solution (ORS), chlorination, etc., via technical assistance, evidence generation, or market shaping? Where is direct delivery the bottleneck, and where are other approaches likely to be more impactful?

Insufficient investigation of how our funding decisions affect other funders

In our analysis, we try to understand how our funding decisions affect other funders.32 We think our funding might cause other actors to spend more (we refer to this as "leveraging,” or “crowding in”) or less (we refer to this as "funging," from “fungibility,” or “crowding out”) on the program than they otherwise would. For example, if we didn’t fund a nets, SMC, or VAS campaign, would another funder step in? Have we set an expectation that we’ll fill funding gaps for nets, SMC, or VAS campaigns so that some countries stop including these in their budgets or their applications to other funders? If so, the impact of our funding would be reduced.33 Similarly, are there cases where our funding caused other funders to direct more to our top charities’ programs, increasing the impact of our funding?

While we’ve done a lot of work to understand how our funding impacts other funders,34 there’s some additional research that we think is worth exploring further.

• External reviews of our funging assumptions. We’ve typically relied on conversations with other funders (e.g., Global Fund, PMI, and national malaria control programs) and our top charities to understand whether others would fill a funding gap if we didn’t. Because we’re a funder, it’s possible people talking to us about this are more likely to inflate the chance that a program wouldn’t get filled without our support. Having an external expert speak to key funders about why these funding gaps for our top charities persist could update us on how much we influence other funders.
• Lookbacks to understand what happened when we didn’t fund a program. A potentially more “objective” test of our funding assumptions would be to see, for programs we decided not to fund, how long it took for them to be funded. We’ve done initial passes at this question,35 but we think that collecting information for all recent top charity-country gaps we’ve chosen not to fund could update us on the extent to which we’re filling funding gaps that wouldn’t have been filled otherwise.
• Opportunities where we’re especially likely to crowd in other funders. For example, we funded the implementation of SMC programs in Uganda, a country that had never implemented SMC campaigns before.36 Based on the results of the pilot and other evidence, the Ugandan Ministry of Finance, Planning and Economic Development expanded SMC across the Karamoja region, requesting funding from the Global Fund for SMC in four of the nine districts.37 By not accounting for this potential “crowding-in” effect, we may be underestimating the benefits of our investments in new areas.
• Advocacy. We have not looked in depth at whether GiveWell could support increased coverage of programs like VAS or nets by advocating for increased funding from others. This could include advocating for more spending from existing funders or from funders that have not traditionally funded the programs implemented by our top charities. We think this area is more speculative but potentially worth taking some initial steps on (e.g., by understanding how other funders choose which programs to fund).
• Modeling “long-term funging” in our CEAs. In our analysis of our top charities, we incorporate adjustments for “short-term funging” (the chance that another funder would support, e.g., the SMC distribution we’re funding over the next 3 years).38 However, we’re less consistent about incorporating “long-term funging”: the chance that our funding creates the expectation that we’ll continue funding these programs into the future. We think we’re being inconsistent about how we model this risk across programs. In our SMC and nets CEAs for Nigeria, we include the risk that “other funders are taking Malaria Consortium and AMF into account” as one of the considerations in estimating short-term funging risk.39 In our VAS and New Incentives CEAs, we don’t incorporate an adjustment for long-term funging.40

These issues could influence which programs we fund. We may be underfunding programs that have a likelihood of crowding in other funders longer-term, or missing out on opportunities to advocate for additional funding to our top charities from other donors. On the other hand, we may be overfunding programs where there is a risk that we’re crowding out other funders. We’d guess there are broader benefits as well to better understanding the funding landscape and how we fit into it.

To address these issues, we plan to:

• Commission an external review of our short- and long-term funging assumptions. Since this red-teaming exercise, we’ve made a grant to the Center for Global Development for Justin Sandefur to talk to other funders about how they decide which programs to fund and how GiveWell’s funding may influence their decisions. This project will include speaking to Ministry of Health officials in countries where GiveWell’s top charities operate, multilateral and bilateral funders, and other large philanthropic funders.41
• Look back at past “close calls” (i.e. cases where we strongly considered funding a campaign, but didn’t end up funding it) to better understand what happens when we don’t fund campaigns. Did these eventually get funded? How long did it take?
• Look for opportunities where we think there’s a good chance our funding could crowd in other funders. Are there new programs we could support that, if successful, would be a natural fit for funders like the Global Fund or Gavi? Is it possible to distinguish these cases from those where we’re likely to crowd out other funders in the long term?
• Decide whether and how to account for long-term funging explicitly in our cost-effectiveness estimates. How much does this affect programs? How should this change the cost-effectiveness of our top charities relative to other programs we might fund?
• In the future, consider other approaches to learn more about how we influence funding from other donors. This could include:
  • Analysis of how other funders’ support of our top charities’ programs has changed once we start funding them. We’ve done some initial, unpublished analysis to understand how Global Fund support for nets has changed since we started funding nets in specific countries. We could extend this analysis to other interventions and keep it up-to-date as we begin funding in new countries.
  • Historical case studies on funding in specific countries or by other funders. E.g., how has malaria funding evolved in Nigeria over the past 10 years? What is the history of funding for SMC programs in Sub-Saharan Africa?

Insufficient attention to some fundamental drivers of intervention efficacy

We want to make sure we’re doing basic checks to ensure that nets, vitamin A supplements, vaccines, and SMC are having intended efficacy. Are they meeting quality standards? Are they still as effective as they were when evidence was collected on them?

We found two cases where we think we did not prioritize research on key indicators of program effectiveness:

• Durability of nets. We think our assumptions about net durability may be out of date. These assumptions are primarily based on the durability of one particular “standard” (pyrethroid) net, PermaNet 2.0,42 while most of the nets procured for campaigns we’ve funded recently have been newer nets using piperonyl butoxide (PBO) synergists or pyrroles.43 We were aware of potential differences in durability for PBO nets and had commissioned more research in the past,44 but we hadn't yet incorporated evidence on PBO net durability into our bottom line, leaving our cost-effectiveness estimates somewhat out-of-date. David McGuire, Director of Access and Country Engagement at the Innovative Vector Control Consortium (IVCC), also highlighted in his review of our nets CEA that PBO nets may not last as long45 and pointed us to a recent modeling study estimating median net retention of around ~1.6 years.46 If we take this study at face value, this would imply ~20% lower cost-effectiveness.47 While investigating grants in DRC post-red-teaming, we updated some of these assumptions: we now include monitoring studies of other nets instead of only the PermaNet 2.0 when estimating net durability (-10% adjustment to cost-effectiveness), and we added an adjustment for chemical decay of PBO synergists (-10% adjustment to cost-effectiveness).48 We plan to continue investigating net quality and retention further, including differences between our updated assumptions and estimates from modeling studies.
• Insecticide resistance. We think our assumptions about insecticide resistance may also be outdated. Our assumption at the time of red teaming that resistance to pyrethroid nets increases by ~1 percentage point per year was based on a single insecticide (permethrin) in a 2018 WHO report.49 However, a review of more recent studies published in 2020 suggests that resistance may be increasing faster in some areas where we distribute nets.50 Updating this parameter to a more conservative ~4 percentage points per year in western Africa51 would lead to an estimated 10% lower cost-effectiveness for pyrethroid nets and potentially increase the estimated cost-effectiveness of new net technologies, such as chlorfenapyr nets, which use a different class of insecticides (pyrroles) to combat resistance.52 We think this also indicates that we should check on the accuracy of fast-moving parameters, like insecticide resistance, more regularly. Since red-teaming, we have begun a broader update of our insecticide-resistance adjustment, but we are still using the assumption of a ~1 percentage point increase in pyrethroid resistance per year, which was flagged as concerning by the red team. For recent grants in Nigeria and DRC, Against Malaria Foundation and Malaria Consortium plan to procure only PBO or chlorfenapyr nets, which we expect to significantly reduce the impact of insecticide resistance.
• Vaccine efficacy. We’ve seen some evidence that the measles vaccine might be less effective in Nigeria than would be expected based on published literature.53 We adjust for this risk in our analysis (reducing our estimate of vaccine efficacy by around 20%),54 but we think a range of 40-60% is plausible.55 We originally flagged concerns about this issue in November 2020.56 While we’ve conducted some desk research on other studies of vaccine efficacy, we’ve yet to fund any follow-up research on this question or meaningfully narrow our uncertainty on this question.
• Drug resistance for SMC. While we think our current adjustments for resistance to SP and AQ, the antimalarial drugs used in SMC, are not currently off, this is an area that may change quickly, and we should continue to monitor it.57

We think this issue is especially likely to occur for parameters that are likely to change a lot over time (such as insecticide or drug resistance).

Failing to accurately assess these fundamental indicators of program effectiveness could mean programs we’re funding are much less impactful than we think. We may also miss opportunities to improve impact (e.g., by funding improvements to vaccine quality, research into the efficacy of different insecticides, or next-generation nets that are less impacted by insecticide resistance).

To address these issues, we plan to:

• Update our assumptions about insecticide resistance and net durability by reviewing the most recent available evidence and incorporating up-to-date estimates into our cost-effectiveness analyses. We will also consider whether to prioritize the adoption of new net technologies in areas with high levels of resistance.
• Prioritize research on vaccine efficacy in the contexts where we fund vaccination programs. This may include commissioning additional biomarker studies for a range of diseases and engaging with experts and other organizations to better understand potential explanations for the observed discrepancies in efficacy.
• Consider funding monitoring of these types of issues alongside upcoming grants. For example, talk to Malaria Consortium about funding studies of drug resistance to SMC alongside upcoming SMC distributions.
• Consider whether this broader issue applies to other aspects of our top charity interventions. Who reviews the quality of SMC or VAS? Who checks whether nets, SMC, or VAS have reached their expiration dates?
• In the future:
  • Develop a process for regularly reviewing and updating key parameters in our cost-effectiveness analyses, particularly those that are likely to change rapidly over time (like increases in insecticide resistance or the spread of the anopheles stephensi mosquito58 ), to ensure that our assumptions remain accurate and up-to-date.
  • Look for cases where program effectiveness is likely to change from one grant to the next and prioritize research on those cases that are likely to be decision-relevant.
  • Consider funding opportunities that would keep us more up-to-date on these parameters.

Insufficient attention to inconsistency across CEAs

If we’re inconsistent in how we assess different programs, we’ll overfund less cost-effective programs and underfund more cost-effective ones. We found two examples of where we’re inconsistent in our cost-effectiveness estimates. We think there are likely more.

• Indirect deaths. “Indirect deaths” are deaths that wouldn’t have occurred without the incidence of a disease, but aren’t directly attributable to that disease in disease burden data. For example, someone who had contracted malaria might die from an unrelated disease because malaria weakened their immune system. We account for the indirect deaths averted by a program by estimating how many indirect deaths are averted for every direct death averted.59 Our assumptions about indirect deaths vary significantly across programs, ranging from ~5 indirect deaths averted for each direct death averted for vitamin A supplementation (VAS), 0.75 for malaria and vaccines, and ~2 for water chlorination.60 We have not thoroughly assessed whether these magnitudes, or their relative sizes across top charities, are plausible.
• Long-term income effects. We think our top charities lead to improvements in long-term income via improved health during childhood.61 At the time of red-teaming, we were estimating income effects for top charities using a qualitative rubric that benchmarks interventions against SMC. This led to large differences in the proportion of benefits attributed to development effects across our top charities, ranging from 10% on average for current New Incentives sites to 43% on average for nets.62 We don’t think these differences seem plausible and would guess the share of benefits due to development effects should be roughly similar across our top charities.

To address these issues, we plan to:

• Conduct more research on indirect effects. We’re uncertain about these estimates in general, both because they vary across programs and because the evidence we have for them has substantial limitations.63 It’s possible we end up concluding that it’s plausible indirect deaths are meaningfully different across programs.
• Correct our development effects estimates to be more consistent across programs. (We’ve completed this update since red-teaming: our current method for estimating development effects is to use our estimate of the development effects of malaria, calculated here, and adjust it based on how the current program compares to an anti-malaria program. See our VAS estimate and New Incentives estimate. Development effects now account for 20-32% of benefits across Top Charities, taking averages across locations.64 )
• Conduct more regular consistency checks across our programs to ensure that, at a high level, we understand why programs are more or less cost-effective and, for certain key parameters, we understand whether they differ in plausible ways across programs. (Since red teaming, we’ve begun implementing consistency checks as a regular part of reviewing CEAs. We describe these checks in more detail in our CEA consistency guidelines.)

Insufficient focus on simplicity in cost-effectiveness models

Our cost-effectiveness analyses (CEAs) have become increasingly complex, making them difficult to understand, critique, and update. This complexity also increases the risk of errors and diverts time and resources from other important work.

For example, our main CEA for insecticide-treated nets is 225 lines long and relies on nine additional tabs for calculations, many of which reference separate standalone analyses.65 This intricate structure can make it challenging for both internal and external stakeholders to fully comprehend and provide feedback on our models.

Two specific examples of where this complexity has led to either a meaningful mistake or makes us more uncertain about our conclusions:

• SMC cycles adjustment. Our CEA for seasonal malaria chemoprevention (SMC) includes two parameters: "number of SMC cycles" and "percent of malaria transmission occurring while people are being treated with SMC." These parameters are inherently linked: a higher number of SMC cycles would lead to a higher percentage of malaria mortality occurring during treatment. However, when evaluating a grant in Uganda, we only adjusted the "number of cycles" parameter without updating the "percent of transmission" parameter accordingly. This led to an underestimation of the program's cost-effectiveness by ~15% (8.8x instead of ~10.3x) because we accounted for the costs of an additional cycle but not its benefits. We’ve fixed this.66
• Timing of nets campaigns parameter. We make an adjustment to our model based on the possibility that a given AMF campaign may reduce the time interval between campaigns compared to what would happen if they were not funded.67 This has a large effect on cost-effectiveness, but we think this adjustment is hard to follow and that it behaves in counterintuitive ways.68

To address this issue, we plan to:

• Develop simplified, 15-25 line CEAs as a starting point when creating or reviewing cost-effectiveness models. This will help ensure that the core assumptions and calculations are easily understandable and can be quickly assessed for accuracy. (We’ve now incorporated these into our top charity reports69 and all new grant pages and intervention reports have these simple CEAs.70 )
• Re-examine our assumption about reducing the interval between campaigns, and explain this adjustment and the key assumptions underlying it more legibly.
• In the future, we plan to apply these same legibility principles and level of scrutiny to all important and complex parameters.

Insufficient investigation of some factors our models may miss

While we’d like to simplify our models, we want to make sure they’re not missing key considerations that meaningfully affect the relative cost-effectiveness of programs we fund.

One example we identified during red teaming:

• Transmission effects in our models. In our cost-effectiveness analysis of New Incentives, we account for the transmission effects of vaccines using a simple 20% upward adjustment.71 However, a "Change Our Mind" entry from Dr. Yannish Naik and Dr. Samantha Field highlighted a review suggesting that the dynamic impacts of vaccination programs may be larger than our model assumes, which they suggested would increase the program's cost-effectiveness by more than 10%.72 By relying on an ad hoc adjustment rather than engaging with more sophisticated models used by vaccine experts, we may be missing important factors that could significantly influence our estimates of the program's impact. We include this concern in our report on New Incentives’ here.

We’re unsure if further work on this will change our bottom line. But we think this is worth looking into, and we would guess there are more examples where further engagement with others who have modeled global health and development programs could change our mind.

If we were to include these types of considerations, we'd want to do it in a way that still preserves legibility (i.e., it’s easy to understand the main drivers of cost-effectiveness and spot major errors) and consistency across our models (e.g., if we incorporate more complex transmission dynamics in our vaccines model, should we expect similar effects for malaria prevention programs?).

To address this issue, we plan to:

• Reach out to vaccine experts, including epidemiologists and disease modelers, to discuss our current approach to modeling the transmission effects of vaccines in the New Incentives CEA. We will seek their input on how we can improve our model to better capture the dynamic impacts of vaccination programs.
• Consider whether similar transmission questions apply to other programs. Are transmission effects of malaria or diseases averted by VAS similar?
• In the future, get more regular feedback from experts on our modeling assumptions. This will require making our models legible to these experts.

Insufficient consideration of what happens when conditional cash transfers for vaccination are stopped

From our April 2024 intervention report for New Incentives:

“In conversations about New Incentives’ program, we have heard concerns that New Incentives’ program could lead to harmful effects if it is discontinued in a given area. For example, it is possible that, by creating a financial motivation to vaccinate infants, New Incentives' program "crowds out" intrinsic motivations to vaccinate infants. This might potentially lead to lower vaccination rates after the program is discontinued in an area than there would have been if the program had never been implemented.

“Our bottom line: We don’t currently account for this concern in our analysis. Our best guess is that this does not significantly undermine the case for New Incentives’ program, but we’re unsure about this and see learning more about it as an important future priority.”

We’ve recently updated our model with a ~10%-30% downward adjustment to address this concern. See our current page for more details.

In the future, we plan to consider other opportunities to learn about what happens after our top charities (or other major programs we fund) stop delivering their programs. Three examples:

• We were funding SMC in Chad through Malaria Consortium and recently decided to provide “exit funding” to transition out of these programs by 2024.73 A few years from now, we could look back to ask: What happened after we exited? Is there anything we could’ve done differently to make for a smoother transition? Do we still think exiting was the right choice?
• If there are negative effects of removing cash incentives for vaccinations, are there approaches that can mitigate these negative effects?
• Are there negative effects of removing other programs like vouchers for chlorination, free ORS and zinc, and mobile conditional cash transfers for vaccination programs?

Insufficient follow-up on potentially concerning monitoring and costing data

We found a few cases where we think it’s possible we did not sufficiently investigate concerning monitoring and costing data:

• AMF campaign in Zambia. Following a 2017-2018 distribution campaign in Zambia conducted by AMF, post-distribution monitoring suggested relatively poor long-term coverage of 21-29% at 24-30 months post-distribution, compared to an average of 56-60% across all countries in the relevant report.74 Additionally, a Demographic and Health Surveys (DHS) Program survey conducted in mid-2018 indicated that only about 60% of the population in the targeted provinces in Zambia had access to nets,75 much lower than our expected figure of 85-90%.76

  At the time, we briefly investigated AMF’s monitoring from this campaign and were concerned that there may have been some irregularities. Our understanding is that net rationing due to insufficient nets may have contributed to the relatively low coverage.77 We didn’t reach a conclusion on how convincing this explanation was, how cost-effective we thought this grant was in hindsight, or how likely this issue is to occur in other countries. We have not supported additional campaigns in Zambia.

• Helen Keller costing. In our evaluation of Helen Keller’s vitamin A supplementation program, we use country-specific distribution costs for most countries. However, in four countries, we disregarded the country-specific cost data because it appeared implausibly low, opting to use aggregate data instead.78 We did not deeply investigate the differences in costs across locations or the potential reasons for the implausibly low figures,79 despite our understanding that Helen Keller employs somewhat different distribution models in different countries.80 Ignoring the country-specific data in favor of an average is conservative if the countries truly have lower costs, but we think this could be concerning because:
  • The implausibly low costs may indicate generally unreliable costing data and the costs in these countries could be higher than our “conservative” estimate. We also aren’t sure whether we might expect lower quality monitoring and distribution in countries with unreliable cost data.
  • We aren’t sure if we should be modeling the impact of grants differently in countries where Helen Keller uses different distribution strategies.
  • If the countries truly have lower costs per supplement delivered, then we’re missing an opportunity to understand how those low costs are achieved.

We’re unsure about whether further investigation of these two examples would lead to meaningful updates to our view of how cost-effective these campaigns were. But we think that they’re indicative of a need for marginally more attention on monitoring and costing data we receive from grantees. Looking into these data, especially in cases where we get unusual results, can help us learn whether programs we fund are having the impact we expect or if there are problems we should be on the lookout for in future grant investigations.

To address this, we plan to:

• More regularly review monitoring data we receive from grantees and ask questions about any irregularities or lower-than-expected coverage figures. For example, are potential issues in the Zambia nets campaign likely to apply to other campaigns? What drove lower-than-expected coverage in these cases?
• Publish updates on how successful previous programs we’ve funded were. For example, for large nets campaigns, how many nets were distributed, compared to expectations? What can we learn about the successes or failures of previous net distributions?
• Talk to Helen Keller about the extent to which program costs vary across countries and whether variation in costs capture genuine differences or just noise.
• In the future, publish more regularly on how the actual costs of distributions compared to the costs we’ve budgeted. How far off were our estimates? If they changed, what drove that? How can we apply this to future distributions?

Insufficient sideways checks on coverage, costs, and program impact

Our cost-effectiveness analyses often require complex calculations and rely on specific data points (e.g., effect sizes from randomized controlled trials, estimates of burden from IHME, cost estimates from organizations we fund). We think checking these numbers from outside data sources can strengthen confidence in our analysis and potentially reveal ways we could be wrong.

We found a couple examples during red teaming of cases where we haven’t done these checks:

• Confirming VAS coverage data. We had not previously conducted a side-by-side comparison of coverage reported by recent Helen Keller campaigns to Demographic and Health Surveys (DHS) data collected around the same time as those campaigns. During red teaming, we did a quick initial analysis that suggested DHS data roughly matched our estimates based on Helen Keller’s data.81
• Confirming nets coverage data. We had not previously conducted a side-by-side comparison of coverage reported by AMF to DHS surveys conducted at the same time as those distributions. During red teaming, we did a quick initial analysis that suggested DHS surveys matched AMF’s data.82 One of the reviewers of our nets report also suggested comparing our estimates to DHS’ Malaria Indicators Survey (MIS) surveys (see below).
• Triangulating cost estimates. We often rely on grantee reports to estimate program costs, but we received some feedback that our cost estimates for mosquito nets appeared to be optimistic.83 The red team compared the costs paid per net by AMF to costs paid per net by other procurers of nets (e.g. Malaria Consortium, the Global Fund). In this case, we believe that our cost estimates broadly check out: AMF’s per net costs are not broadly out of line with those of other organizations, so it increased our confidence in our analysis.84
• Sense checking against alternative sources on burden of disease. We discuss this above.

We’re unsure if these additional checks will meaningfully update our estimates, but we think they’re an important check on our work. Burden, costs, and coverage are key pieces of the case for these programs, and it's worth investing extra time validating them rather than relying too heavily on a single source of information.

To address this, we plan to:

• Look for additional opportunities to use DHS surveys or other sources to confirm coverage estimates.
• Look for additional opportunities to sense check other parameters, such as burden estimates, program effectiveness, or costs.

Insufficient attention to high uncertainty regarding VAS

Looking across our top charities, we think the case for vitamin A supplementation (VAS) is the most uncertain. That’s because there are major uncertainties about the effect of VAS on mortality.

From our current VAS intervention report:85

• What was the true impact of VAS at the time the studies we rely on were conducted? Our analysis of the impact of VAS relies on a Cochrane meta-analysis of VAS RCTs (more) and a -25% adjustment to account for ways the meta-analysis estimate could be biased upwards (more). But we have a number of uncertainties about this:
  • We have received some expert feedback that the meta-analysis shows evidence of substantial publication bias, implying a smaller effect size on mortality than reported. Our internal validity adjustment intends to account for some possibility of publication bias, but we don’t use any statistical methods to estimate this and we have not systematically investigated this question. It’s possible that doing so would lead to a larger downward adjustment. (More)
  • Studies in the meta-analysis vary widely in their estimate of the impact of VAS on mortality (more). We’re unsure how to explain this variation, and the factors that we currently assume are most important in our analysis for determining the effectiveness of VAS across locations (vitamin A deficiency rates and diarrhea and measles rates) do not appear to explain it. (More)
  • The main finding in the VAS meta-analysis is sensitive to the choice of analysis (either 12% or 24% reduced mortality overall depending on the analytical approach). We use the larger of these two estimates (24%). This is a judgment call that we have thought through extensively, but if we are wrong it could imply we are significantly overestimating cost-effectiveness. (More)
  • The largest and one of the most recent trials of VAS found an effect size that is small and not statistically significant, and we’re not sure what is driving this. (More)
• How effective is VAS in modern contexts? The main studies we rely on were primarily conducted in the 1980s and 1990s, when the infectious disease landscape was different and child health was significantly worse than today. It’s likely that VAS delivered today would result in a significantly smaller reduction in mortality. We attempt to account for this with a -41% to -79% adjustment, but we’re uncertain about this for a number of reasons:
  • We don’t have a strong understanding of what mediates the impact of VAS on mortality, meaning we’re unsure what factors to adjust for. (More)
  • Our analysis is very sensitive to vitamin A deficiency rates today, but our estimates are based on information we have low confidence in (10 to 20 year old surveys of deficiency, updated for change over time, and modeled estimates from the Global Burden of Disease Project whose methodology we do not fully understand) (more). Since these surveys were conducted, many countries have introduced vitamin A fortification programs, and we’re unsure how effective these have been at reducing deficiency rates. Our understanding is also that measurement of vitamin A deficiency is sensitive to the choice of test used, reducing our confidence in our estimates (more).
  • We’re unsure how changes to health environments in the future will affect VAS. Our biggest uncertainty is whether the scale-up of another child health program, azithromycin distribution, will avert deaths that would have been averted by VAS, implying the additional benefit of delivering VAS could be lower. (more).

This uncertainty doesn’t make us think we’re necessarily over- or underestimating the cost-effectiveness of VAS programs because (a) we've incorporated some of these concerns into our analysis and (b) we'd roughly guess that the sources of uncertainty that we don't model would not move cost-effectiveness in one direction over the other. Though this uncertainty doesn’t make us think we’re over- or underestimating the cost-effectiveness of VAS programs, it could still impact our grantmaking. This impact could include (a) waiting to fund more VAS grants until we’ve resolved some of these uncertainties or (b) limiting our funding as a result of this uncertainty (and adopting a more “risk-averse” stance).

To address this issue, we plan to:

• Consider how much funding we’re willing to direct to VAS programs in light of these uncertainties.
• Publish a stance on how we approach this type of uncertainty in our grantmaking.

What did expert reviewers say?

This section provides more detail on feedback from expert reviewers on each of our four top charities. We reference these in the section above on main issues but provide a more comprehensive summary, including links to full notes, below.

Against Malaria Foundation’s (AMF) insecticide-treated net (ITN) distribution

Two experts provided feedback on our report on mass distribution of ITNs:

• Dr. Justin Cohen, Vice President of Malaria and Neglected Tropical Disease at the Clinton Health Access Initiative. His written review of our report can be read in full here. Disclosure: The Clinton Health Access Initiative has received funding from GiveWell (for work unrelated to providing this feedback). See the grants made to the Clinton Health Access Initiative here.
• David McGuire, Director of Access and Country Engagement at the Innovative Vector Control Consortium (IVCC). His written review of our report can be read in full here.

Both experts agreed that ITNs are an effective intervention.86 They also gave the following feedback:

• Dr. Cohen and Mr. McGuire said that the type of net and the environment in which it’s used significantly affect efficacy and that we should incorporate that information in our cost-effectiveness analysis. While we were accounting for the impacts of chemicals like PBO and chlorfenapyr on insecticide resistance,87 we had not explicitly examined the durability of these nets, or the possibility of PBO washing out of nets. Both suggested using the Malaria INtervention Tool developed by Professor Tom Churcher at Imperial College London,88 which models the epidemiological impact of deploying different malaria-control tools in a specified geography based on a set of variables including the type of net and transmission setting.89 We have since spoken with Dr. Churcher, and have updated our durability estimates to include studies of newer nets, and incorporated a specific update for PBO chemicals decaying from nets.90
• Dr. Cohen pointed out several areas in which it would be helpful to compare our estimates to the surveys from the Demographic and Health Surveys (DHS) Malaria Indicators Survey (MIS) program.91 We did some cross-checking during red teaming and did not surface any major inconsistencies (see above). We may use these resources to cross-check AMF’s monitoring data further in the future.
• We included some concerns Mr. McGuire shared in our report on nets. We haven’t considered these criticisms in detail, or thought through how they should affect our funding recommendations (with the exception of the final bullet, given we make explicit assumptions about net usage). We summarized feedback from Mr. McGuire in the report as follows:92
  • “External donors have been focused on ITNs as the primary intervention for vector control, and continue to shift resources away from other effective interventions such as Indoor Residual Spraying (IRS), despite evidence that removal of IRS often leads to increased malaria incidence.93
  • Funders of ITNs have historically relied too little on advice from malaria experts in Africa when making decisions about how to best tailor vector control strategies using a broader mix of targeted interventions based on local context.94
  • ITNs are an excellent tool to control malaria in high-transmission settings, but no country has used ITNs as a key intervention in the final stage of successful elimination.95
  • ITNs generate large amounts of plastic waste.96
  • Many people do not like using ITNs, meaning it is challenging to get mass uptake. This can be for various reasons including discomfort on hot nights, and standardized net designs not being adapted for people’s preferences.97 ”
• Dr. Cohen also made the same point as one reviewer of our report on SMC, Dr. André Tchouatieu: they both argued that our cost-effectiveness estimate of money spent by the Global Fund98 is inaccurate because of the structure of the Global Fund’s spending. Our cost-effectiveness estimate of the Global Fund’s alternative funding includes their work on HIV and TB. Dr. Cohen and Dr. Tchouatieu argue that this is incorrect because the Global Fund allocates funding by cause area, so any money that we “free up” for the Global Fund by funding nets programs they might have funded would only go to a malaria program, rather than to HIV or TB.99 While we guess that addressing this concern would have a limited effect on cost-effectiveness, we think it points to a need for us to understand better how our funding might affect others’ funding decisions. We may prioritize further work on this in the future (see above).

Malaria Consortium’s seasonal malaria chemoprevention (SMC)

Two experts provided feedback on our SMC report:

• Dr. André Tchouatieu, Director of Access & Product Management at Medicines for Malaria Venture. His written review of our report can be read in full here.
• The second reviewer did not respond to our request for permission to publish their feedback, so will be referred to here as “the second reviewer”.

Broadly, both experts agreed that SMC is a good use of funding.100 They also gave the following feedback:

• Dr. Tchouatieu said equity was a concern for SMC given that, as children under age five are increasingly protected via SMC, the malaria burden will shift to children over age 5 and children in areas that don’t receive SMC.101 While we are also concerned about this issue, we do not see it as a negative reflection on SMC, but a reflection of insufficient funding for malaria control in general. We have also briefly looked into programs to provide seasonal malaria chemoprevention to children 5-14-years-old but have deprioritized further investigation because we think these programs are below our cost-effectiveness bar.
• Dr. Tchouatieu stated that connecting averted malaria to long-term income increases for the child affected was tenuous and difficult to support.102 On the other hand, the second reviewer pointed out that there is growing evidence of the effect of malaria on early childhood development, which tends to be a strong predictor of long-term development and educational attainment. We’ve done considerable research looking into how averting malaria can lead to long-term increases in income (see this section of our SMC report). We also have uncertainties (e.g., on the quality of the evidence) and incorporate these into our analysis by downweighting the effect size.
  • Dr. Tchouatieu suggested that averting malaria cases could increase household income in the short term through averted medical costs and that this would be easier to incorporate and support in our analysis.103 We incorporate “treatment costs averted from prevention” as a supplemental adjustment in our cost-effectiveness analysis but have not explored this parameter in depth.104
• Dr. Tchouatieu also brought up the same criticism as Dr. Cohen, a reviewer of our nets report. They both said that our estimation of Global Fund spending was flawed; we describe their feedback above.

New Incentives’ conditional cash transfers to increase infant vaccination

One expert provided feedback on our New Incentives report:

• Dr. Jessica Cohen, Bruce A. Beal, Robert L. Beal and Alexander S. Beal Associate Professor of Global Health at Harvard University. Her written review of our report can be read in full here.

Dr. Cohen generally agreed with the argument behind New Incentives’ programs.105 She also gave the following feedback:

• Dr. Cohen was concerned by the increase in vaccination rates in the control group of the RCT run by IDinsight on New Incentives program in three states.106 We also had this concern, but our confidence in the RCT increased after we later compared the increase in the vaccination rates of the control group to independent vaccination coverage surveys. We found that at least half of the control-group increase could be attributed to regional increases in vaccination rates.107
• Dr. Cohen also pointed out that we could be underestimating the impact of New Incentives’ program because we don’t account for the potential spillover benefits of increased clinic visits for children.108 We currently exclude these benefits109 because the increase in clinic utilization that IDinsight found when they tested this in an RCT was small (5 percentage points),110 but we think it’s possible that our focus on the intensive margin of clinic utilization may miss other important effects.

Helen Keller International’s (Helen Keller) vitamin A supplementation (VAS)

Four experts provided feedback on our updated VAS analysis:

• Dr. Christine Benn, Professor in Global Health at the University of Southern Denmark. Her written review of our report can be read in full here.
• Dr. Kenneth Brown, Distinguished Professor Emeritus in the Department of Nutrition at the University of California, Davis. His written review of our report can be read in full here.
• Dr. Sherry Tanumihardjo, Professor of Nutritional Sciences at the University of Wisconsin-Madison. Her written responses to our questions on VAS can be read in full here.
• Dr. Keith West, Professor of Infant and Child Nutrition at Johns Hopkins University. Dr. West answered our specific questions on VAS and there is no full writeup to share.

Dr. Benn and Dr. Brown each reviewed a draft of our report on VAS and had different perspectives on the importance and effectiveness of VAS. While Dr. Brown believes that VAS is a cost-effective intervention in areas with VAD,111 Dr. Benn said that the potential harms of VAS mass campaigns counterbalance the benefits and that they should not be a top priority for GiveWell.112 Dr. Tanumihardjo and Dr. West were asked specific questions about our analysis of VAS and didn’t give their overall perspective on VAS as an intervention.

We received the following additional feedback from the four experts:

• Both Dr. Benn and Dr. Brown brought up the possibility that the populations studied in the original trials from the 1980s and 1990s are different enough from today’s affected populations that the effect found in the studies could be reduced. While Dr. Benn said the evidence indicated the declining effect of VAS in these modern contexts, Dr. Brown said that the evidence is more mixed.113 We account for the declining effect of VAS on mortality in our report, though we are highly uncertain about the magnitude of the effect. See this section of our VAS report.
• Dr. Benn said that VAS’s impact is partly through modifying immune response and therefore could affect non-vitamin A deficient (VAD) children as well.114 She argues that this impact might be modified by other factors, such as sex and vaccination status, and may lead to negative outcomes (e.g. increased mortality among girls whose most recent vaccine was an inactivated vaccine, like DTP). Dr. Benn shared with us an unpublished meta-analysis that found no evidence that layering VAS onto vaccination campaigns leads to a reduction in mortality, and found it may actually increase mortality in some subgroups.115 We’ve chosen not to incorporate this into our analysis for a few reasons which we include in our report on VAS:116
  • “The underlying data are observational and therefore at higher risk of bias than the evidence from RCTs. We’ve received feedback from the author that she believes confounding is unlikely to be driving the results, but we haven’t deeply investigated or tried to corroborate this.
  • The confidence interval for the random-effects analysis (VAS is associated with a 14% decrease to a 47% increase in mortality) encompasses our estimate of the effect of VAS (~4% to ~12% reduction in mortality across locations, more here).
  • The analysis has not yet been published or peer reviewed.
  • The analysis is based on only three locations, with substantial heterogeneity across sites.”
• Dr. Brown also gave the following feedback:
  • He argues that one of the studies included in a meta-analysis we use to estimate VAS’s effect on mortality is affected by ascertainment bias (i.e., mismeasurement of deaths) and that we should exclude it from our work.117 We’ve chosen not to incorporate this into our analysis because we wouldn’t expect this bias to apply more to children in areas receiving VAS than control areas, so it doesn’t significantly affect our interpretation of the study results.118
  • He raised the possibility of excessive vitamin A intake leading to harm in locations where multiple vitamin A interventions overlap, although he added that this was unlikely to be “an important issue for public health in most of sub-Saharan Africa.” We don’t currently include potential negative health impacts from VAS campaigns. We still have relatively low concern, but may look more into the issue in the future.119
  • He said we should be cautious about conclusions that VAS might increase respiratory disease risk, due to uncertainties about the methods used in field studies to detect respiratory disease.120 We don’t currently include the potential negative effects of VAS on risk of respiratory infection, so haven’t incorporated this feedback into our analysis or deeply investigated it.121
• When we asked whether VAS could be harmful to non-VAD children, Dr. Tanumihardjo said that excessive vitamin A intake can lead to harmful effects on the bone.122 She also shared studies done in South Africa and Malawi that some children are consuming an excess of vitamin A in areas where they are receiving it from multiple sources.123 As mentioned above, we don’t currently include potential negative health impacts from VAS campaigns. We still have a relatively low level of concern, but may look more into the issue in the future.124
• Dr. West gave feedback on the interpretation of study results, pointing out that “smaller trials not intended, designed, powered or executed” to detect effects on mortality are not likely to be helpful in assessing the impact of VAS on mortality.125 We’re considering (but haven’t yet prioritized) alternative analyses that exclude these smaller trials.126

What are our next steps?

Next steps for red teaming our top charities:

• Our team leads for malaria (SMC and nets), nutrition (VAS), and vaccines (New Incentives) are prioritizing follow-up work on the issues above that relate to their team.
• Our cross-cutting team is planning to (a) work with team leads to make sure the issues surfaced during red teaming are addressed and (b) investigate issues that cut across these areas (e.g., the reliability of IHME data, how to model overlapping programs, checking for consistency across grantmaking areas).
• As we investigate new top charity grants this year, we plan to prioritize research on relevant issues raised by red teaming (e.g., better understanding the prevalence of routine distribution of nets and concerns about IHME data on malaria mortality as part of our investigation into grants for nets in DRC and Nigeria).

This year, we also plan to:

• Red team an additional 1-2 grantmaking areas beyond our top charities. As of August 2024, we’ve red-teamed our analysis of iron supplementation and fortification programs.
• Experiment with additional approaches to soliciting feedback on our grantmaking and research. One way in which we’ve done this is to make a grant to the Center for Global Development for Justin Sandefur to do a research project on why other funders don’t already fund the programs GiveWell does.127

We plan to report back next year on what we’ve found and how our research and grantmaking have shifted as a result.

In the future, we’ll consider how these issues apply to programs beyond our top charities. For example, what happens when mobile-based conditional cash transfers are removed (see above), or how long does it take for other funders to fund other programs we decided not to fund (see above)?

Sources

• 1

  This only includes funding to our current top charities through the end of our 2022 metrics year (which ended January 31, 2023). See the totals here.

• 2

  We’ve done a rough internal analysis that found GiveWell has done over 300,000 hours of research since 2007 and currently conducts around 50,000 hours of research each year. Though we haven’t divided this research time by charity, we would estimate that a substantial portion of this time has been spent on our current top charities.

• 3

  These anticipated changes in our grantmaking are a rough guess. They’re based on considering potential 2024 grants where these updates could be decisive and estimating (a) size of that grant, (b) probability we’d overturn the decision to make that grant or not make that grant, and (c) “credit” due to issues flagged by red teaming (i.e., because grant decisions come down to many factors, in many cases an issue flagged by red teaming that led to a positive or negative update may not clearly be the one deciding factor).
  We haven’t published the explicit rationale for these guesses, since these grant decisions are still in progress. We expect to have a clearer sense of the impact of issues identified by red teaming once we’ve made decisions this year and will publish what we find next year.

• 4

  We estimate that we’d make $5m-$40m in grants we wouldn’t have otherwise and not make $5m-$40m in grants we would have otherwise. $5m is ~2% of $325m (5m/325m=~2%), and $40m is ~12% of $325m (40m/325m=~12%).

• 5

  For our intervention report on seasonal malaria chemoprevention, see this page. For our charity report on Malaria Consortium, see this page.

• 6

  For our intervention report on the mass distribution of insecticide-treated nets, see this page. For our charity report on the Against Malaria Foundation, see this page.

• 7

  For our intervention report on vitamin A supplementation, see this page. For our charity report on Helen Keller International, see this page.

• 8

  For our intervention report on conditional cash transfers for routine vaccination (New Incentives), see this page. For our charity report on New Incentives, see this page.

• 9
  • Nets:
    • We discuss how GiveWell funding affects access to nets in the "How cost-effective is it?" section of our report on nets: “Without GiveWell funding, fewer children would have access to ITNs:
    • Without campaigns, people would be unlikely to access ITNs. Our understanding is that these campaigns are the main way people access ITNs in malaria-endemic countries. We think that relatively few people would access nets through other sources, such as “continuous distribution” of nets (which only targets certain groups and we think is underfunded in most countries), or purchasing nets on their own. We currently assume that people are equally likely to obtain nets from other sources as they were in the original studies measuring the impacts of ITNs on mortality (i.e., roughly 5% to 10% of people would get nets in the absence of campaigns). (More)
    • Without our funding, campaigns would be unlikely to occur. AMF works in countries that don't have enough money from the other big funders of net campaigns (the Global Fund and President’s Malaria Initiative) to pay for high-quality mass campaigns (i.e., those that occur at least every three years, achieve a high degree of coverage, and distribute nets that are effective given local resistance profiles). We currently estimate that there is a 20% to 60% chance that other funders would replace GiveWell’s ITN funding in our absence (depending on the specific country and campaign). Our adjustment for “diverting other actors spending away from net campaigns” lowers cost-effectiveness by approximately 15% to 40% depending on the location. This is high relative to some other programs we fund, but we think ITNs are still a good investment even after accounting for this. (More)”
    • For more detail on the likelihood that people would receive nets without mass distribution campaigns, see the “How many people would access nets from other sources?” section of our report on nets.
  • SMC:
    • We discuss how GiveWell funding affects the uptake of SMC in the "How cost-effective is it?" section of our report on SMC: “Without campaigns like the ones GiveWell funds, we estimate that virtually no children would get SMC from other sources. This is because campaigns are the only form of SMC delivery we have heard about, and our understanding is that WHO recommends that national governments prohibit the private sale of the SMC drugs in areas implementing SMC (more).”
    • For more detail on the likelihood that people would receive SMC without a campaign, see the “Risk of wastage” section of our report on SMC.
  • VAS:
    • We discuss how GiveWell funding affects uptake of VAS for children in the “How cost-effective is it?” section of our VAS report:
    • “Without GiveWell funding, fewer children would receive VAS. That’s because we think:
    • It’s unlikely that a high proportion of children would be reached without mass distribution campaigns. In most countries where GiveWell funds VAS campaigns, children can also access VAS through other sources (e.g., through routine immunization appointments). Our current best guess is that 25% of children would receive VAS through other sources besides campaigns, although this is based on limited data. (More)
    • Other funders are unlikely to replace GiveWell’s funding for campaigns. Both Helen Keller International and Nutrition International work in locations where we do not believe there is sufficient funding from other sources to support the whole country with regular, high-quality VAS campaigns. We currently estimate that there is a 20% to 50% chance (varying by location) that other funders, primarily UNICEF, would replace GiveWell’s funding for VAS in our absence. Our adjustment to account for this lowers cost-effectiveness by 18% to 46% depending on the location. This is high relative to some other programs we fund, but we think VAS is still cost-effective even after accounting for this. (More)”
    • For more detail on the proportion of children we think would receive VAS without mass distribution campaigns, see the “How many children would have received VAS without campaigns?” section of our VAS report.
  • Conditional cash transfers to increase infant vaccination (New Incentives):
    • We discuss how New Incentives’ program affects the number of children who receive vaccination in the "How cost-effective is it?" section of our report on its program: “Vaccination rates in northern Nigeria are low (~25% to ~65% in locations where New Incentives works), and we think New Incentives’ program addresses some of the barriers to vaccination (e.g., opportunity costs of taking children to clinics and perceived low value of vaccinations). (More) Extrapolating from the results of a randomized controlled trial (RCT) of New Incentives' program and triangulating against other studies of vaccine incentive programs, we estimate the program increases vaccination rates by approximately 11 - 22 percentage points in areas where it now works. (More)”
    • For more detail on how New Incentives’ program affects vaccination rates, see the "How much does the program increase vaccination rates?" section of our report on New Incentives’ program.

• 10

  See our discussion in the “How many people would access nets from other sources?” section of our report on nets: “We currently have no adjustments in the model to account for concern 4 [‘We don’t fund a mass distribution campaign and neither does another funder. However, people get nets anyway because they buy them from shops, or access them through routine distribution channels (e.g., via health clinic visits).’]. Effectively, this amounts to assuming that people today are no more or less likely to obtain nets from other sources than they were during original studies testing the impact of ITNs on malaria (more below), which were predominantly conducted in the 1980s and 1990s. In these trials, around 5% to 10% of people had access to nets in the control group. By using these control groups as a stand-in for the counterfactual today, we effectively assume that 5% to 10% of people would get nets in the absence of any mass distribution campaign.”

• 11

  See GiveWell, Summary of ITN RCTs (March 2023), "1. Studies from Pryce et al. 2018" sheet, "When was the study conducted?" column. Twenty of the twenty-three studies were conducted in the 1990s or before.

• 12

  “b. Continuous distribution - Antenatal consultations.
  As with mass campaigns, the same types of LLINs will be used in the same provinces: LLIN PBO 19 Provinces and Dual-Processed LLIN in seven provinces. A LLIN will be distributed to all women attending their first prenatal visit. A total of 17,054,542 pregnant women are expected to attend the antenatal consultation during the period 2024-2026, including 9,758,637 in the 16 provinces of Global Fund intervention plus Kasai.
  c. Continued distribution - Expanded Program of Immunization (SMC/ENP).
  As for prenatal consultation, 19 provinces will use PBO LLINs and 7 provinces will use Bi-treated LLINs. Every child who comes to the preschool consultation will receive a net, which represents a national need for 14,880,088 LLINs over the intervention period, including 634,386 in the 17 provinces of intervention of the Global Fund.”
  Global Fund, “DRC Funding Request Malaria, Program Continuation, Allocation Period 2023-2025”, March 2023, emphasis added.
  (17,054,542 LLINs for antenatal consultations + 14,880,088 LLINs for preschool consultations = 31,934,630 LLINs via routine distribution.)

• 13

  These estimates are based on an unpublished internal analysis.

• 14

  We came to this estimate via a very rough calculation. We make the following assumptions:

  • Instead of assuming 5% of people who receive nets from mass distribution would receive nets from routine distribution in the absence of mass campaigns (as we currently do), we assume:
    • 25-50% of children under 3 years old would receive nets from routine distribution in the absence of mass distribution campaigns
    • and 15% of people over 3 years old would receive nets from routine distribution in the absence of mass distribution campaigns.
  • Half of the benefits of nets campaigns (including from averted mortality and income effects later in life) are from increasing the number of children under 3 years old who receive nets and half are from increasing the number of people over 3 years old who receive nets. This is a very rough assumption we make because in our nets CEA we often primarily distinguish between under- and over-5-year-olds (or 15-year-olds in calculating long-term increases in income), rather than between under- and over-3-year-olds as these estimates of routine coverage do.
  • Our current cost-effectiveness estimate implicitly assumes that 5% of people would receive nets from routine distribution (see the “How many people would access nets from other sources?” section of our report on nets). We therefore subtract 5% from the numerator and denominator of the “share of over-/under-3 benefits lost due to nets from routine coverage” value to account for this percentage that is already not included in the cost-effectiveness estimate.

  Given these assumptions, we calculate the following potential changes in cost-effectiveness:
  More conservatively, assuming routine distribution of nets would reach 50% of under-3s:
  50% (proportion of benefits from under-3s) * ((50% - 5%) / (100% - 5%)) (share of under-3 benefits lost due to nets from routine coverage) + 50% (proportion of benefits from over-3s) * ((15% - 5%) / (100% - 5%)) (share of over-3 benefits lost due to nets from routine coverage) = ~29% reduction in cost-effectiveness
  Less conservatively, assuming routine distribution of nets would reach 25% of under-3s:
  50% (proportion of benefits from under-3s) * ((25% - 5%) / (100% - 5%)) (share of under-3 benefits lost due to nets from routine coverage) + 50% (proportion of benefits from over-3s) * ((15% - 5%) / (100% - 5%)) (share of over-3 benefits lost due to nets from routine coverage) = ~16% reduction in cost-effectiveness

• 15

  See this cell of our 2023 New Incentives CEA, which explains how we use New Incentives’ coverage surveys to estimate counterfactual coverage.

• 16

  We estimate that our adjustment is equivalent to coverage increasing by 1.5 percentage points per year in an unpublished internal analysis.

• 17

  This calculation was part of an unpublished internal analysis of three surveys:

  • UNICEF, "Nigeria, Multiple Indicator Cluster Survey 2016-2017, National Survey Finding Report", February, 2018
  • DHS Program, "Nigeria Demographic and Health Survey, 2018", October 2019
  • UNICEF, "Nigeria 2021 Multiple Indicator Cluster Survey (MICS) & National Immunization Coverage Survey (NICS), Survey Findings Report August 2022"

• 18

  This estimate is from an unpublished internal analysis.

• 19

  For examples of explicitly stating our assumptions about underlying coverage rates, see this section of our report on mass distribution of ITNs, this section of our report on seasonal malaria chemoprevention (SMC), this section of our report on conditional cash transfers for routine vaccination, and this section of our report on vitamin A supplementation.

• 20

  E.g., we fund New Incentives in Zamfara state, where, per the IHME, 17.4% of children die before reaching age 5, but not in Niger state, which has 8.8% under-5 mortality and therefore presumably sees less benefit from increases in vaccination. See the Zamfara and Niger under-5 mortality rates from the GBD 2021 results, used with permission. See the Zamfara and Niger columns of our cost-effectiveness analysis of New Incentives for their estimated cost-effectivenesses.

• 21

  A rough, unvetted analysis is here.

• 22

  For example, we currently estimate Niger state’s cost-effectiveness at ~8.5x, using the IHME under-5 mortality rate of 8.8%. If we instead used the UN IGME estimate of 13.7%, we think Niger state might be brought above our 10x bar for funding, though we haven’t done a detailed analysis and there could be other factors that inform whether we fund New Incentives in a particular state beyond bottom line cost-effectiveness.

• 23

  See this cell of our rough, unvetted analysis where we calculate the relationship between IHME and UN IGME estimates.

• 24

  Chad: The UN IGME estimate for under-5 mortalities from malaria in Chad in 2019 is 11,076 and the IHME estimate is 4,526.
  11,076 / 4,526 = 2.44, indicating the UN IGME estimate is nearly 2.5x as high as the IHME estimate.
  Guinea: The UN IGME estimate for under-5 mortalities from malaria in Guinea in 2019 is 8,848 and the IHME estimate is 8,181.
  8,848 / 8,181 = 1.08, or a roughly 8% difference, indicating a similar estimate from both sources.
  Uganda: The UN IGME estimate for under-5 mortalities from malaria in Uganda in 2019 is 7,745 and the IHME estimate is 17,386.
  7,745 / 17,386 = 0.45, indicating the UN IGME estimate is 45% as large as the IHME estimate.
  For the IHME figures, see these cells for Chad, these cells for Guinea, and these cells for Uganda in our CEA of nets. For UN IGME figures, see this page for Chad, this page for Guinea, and this page for Uganda. UN Inter-agency Group for Child Mortality Estimation, Causes of Death by Country, accessed 7/15/2024.

• 25

  Examples of GiveWell grants in northern Nigeria: ITN distribution through Against Malaria Foundation, co-delivery of vitamin A supplementation and SMC through Malaria Consortium, New Incentives’ work on vaccination, ORS through the Clinton Health Access Initiative. For more examples of GiveWell grants in northern Nigeria, see our grants database.
  Azithromycin in Nigeria is being supported by the Gates Foundation. Gates Foundation, Committed Grants: Crown Agents Limited, accessed July 19th, 2024.
  The World Bank is funding the Sustainable Urban and Rural Water Supply, Sanitation and Hygiene Program-for-Results (SURWASH) in Nigeria, which aims to “increase access to water, sanitation, and hygiene services and to strengthen sector institutions in participating states of Nigeria.”
  World Bank, “Nigeria Sustainable Urban and Rural Water Supply, Sanitation and Hygiene Program-for-Results” page, accessed 8/5/2024.

• 26

  “WHO recommends that consideration be given to mass drug administration of azithromycin to children 1 to 11 months of age for prevention of childhood mortality in sub-Saharan African settings in which:

  • infant mortality is &gt 60 per 1000 live births or under-five mortality is &gt 80 per 1000 live births respectively, and
  • infant and under-five mortality rates, adverse effects and antibiotic resistance (AMR) are continuously monitored, and
  • implementation of existing child survival interventions, including seasonal malaria chemoprophylaxis where recommended, is concurrently strengthened.”

  WHO, “WHO Guideline on mass drug administration of azithromycin to children under five years of age to promote child survival”, August 27th, 2020.

• 27

  We think the cost-effectiveness of VAS campaigns could be reduced by increased azithromycin distribution because we think both interventions may decrease mortality from the overlapping causes, non-malaria infectious disease. (For more information, see the “Will VAS remain impactful in the future?” section of our report on VAS (footnote 339 in particular). Note that we think azithromycin may also affect malaria mortality.) If azithromycin prevents deaths that VAS would have averted, the impact of a VAS campaign would be reduced. We very roughly calculate a potential 20% downward adjustment in the cost-effectiveness in VAS based on this possible overlap:

  • Based on data from the Institute for Health Metrics and Evaluation (IHME), we roughly calculate that in Burkina Faso, approximately 55% of all mortality among children aged 1 month to four years stems from non-malaria infectious disease. In 2019, there were ~71,900 total deaths in this cohort, with ~15,700 malaria deaths, and ~55,000 deaths from all infectious disease. (55,000 - 15,700) / 71,900 = ~55%. Source: Institute for Health Metrics and Evaluation (IHME). Used with permission. All rights reserved.
  • One study found that azithromycin leads to a ~14% reduction in all-cause mortality.
  • Assuming that VAS and azithromycin primarily treat the ~55% of deaths that are due to non-malaria infectious disease and roughly adjusting by 20% for the assumption that azithromycin and VAS might treat slightly different subsets of that 55%, we estimate the total reduction in VAS’s effectiveness to be -20%: (share of non-malaria deaths averted by azithromycin) * (adjustment for VAS and azithromycin treating different groups) = (14% / 55%) * (100% - 20%) = ~20%.

• 28

  Vitamin A fortification is the process of increasing the amount of vitamin A in a food, usually staple foods, to increase intake of vitamin A.
  “Fortification has an added advantage that small amounts are consumed on most days if the appropriate vehicle is used… Small amounts of vitamin A everyday, even it if does not meet requirements, are beneficial to immunity.”
  Dr. Sherry Tanumihardjo, Review of GiveWell VAS report, December 2023.

• 29

  We discuss our unpublished analysis of vitamin A fortification in our report on vitamin A supplementation here.

• 30

  As of 8/5/2024, GiveWell has directed $193,875,033 to the Against Malaria Foundation (AMF) for nets campaigns since 2014 and $32,476,786 to Malaria Consortium for nets campaigns ($226,351,819 combined). See our grants database for more information. AMF primarily identifies funding gaps, procures nets, and conducts monitoring and evaluation on the nets campaigns (see this section of our report on AMF), while Malaria Consortium has worked on both procurement and distribution of nets (see this grant page).

• 31

  We raise these alternative avenues in the “Would other ways of funding nets be effective?” section of our report on nets.

• 32

  For more detail on how we incorporate how our programs affect other actors’ spending into our analysis, see this section of our report on mass distribution of ITNs, this section of our report on seasonal malaria chemoprevention (SMC), this section of our report on conditional cash transfers to increase infant vaccination, and this section of our report on vitamin A supplementation.

• 33

  For more detail on how this might limit the impact of our funding, see this blog post on fungibility.

• 34

  For more detail on how we incorporate how our programs affect other actors’ spending into our analysis, see this section of our report on mass distribution of ITNs, this section of our report on seasonal malaria chemoprevention (SMC), this section of our report on conditional cash transfers to increase infant vaccination, and this section of our report on vitamin A supplementation.

• 35

  One example of this sort of lookback is this page from 2016, looking back at areas where AMF considered funding a nets distribution but did not. We also keep a rough internal record of grants we declined to fund so that we can look back and see whether the funding gap was ultimately filled.

• 36

  "In 2020, the National Malaria Control Division (NMCD) approached Malaria Consortium with a request to support an SMC implementation study in Karamoja (Figure 13) to investigate the feasibility, acceptability, and impact of SMC... This was the first time SMC was implemented in the country."
  Malaria Consortium’s seasonal malaria chemoprevention program: Philanthropy report 2021, p. 37.
  “Uganda: Malaria Consortium supports SMC in Karamoja region, where the story has been similar to the story in Nampula. GiveWell funded Malaria Consortium to pilot SMC in Karamoja in 2021 and 2022 and co-funded a similar two-year implementation study, (including an RCT), with BMGF.”
  GiveWell, report on SMC, “What SMC programs has Malaria Consortium delivered?” section.

• 37

  “In line with the priorities in the UMRESP (p. 55), the NMCD in collaboration with Malaria Consortium assessed the protective effectiveness of seasonal malaria chemoprevention - monthly administration of SP and amodiaquine (SPAQ) - to children aged 3–59 months in Karamoja. The study applied the standard door-to-door distribution model using village health teams (VHTs) members and demonstrated U5 children in intervention districts had a 92.2 percent lower risk of developing confirmed malaria in the five-month follow-up versus those in the control district. This study also found that SMC using SPAQ was highly feasible, acceptable, and safe. Similarly, a systematic analysis of routine health facility data revealed substantial reduction in malaria incidence among under five before and after SMC in intervention districts compared to control. Uganda is thus among the 1st countries outside the Sahel region to successfully test SMC implementation.
  Based on this evidence, the SMC intervention was expanded from 2 to 8 districts out of the 9 in Karamoja in 2022.
  For this GC7 malaria funding request, given the limited funds, we have prioritized SMC for all the 9 districts in Karamoja, and allocated funds within funds to continue SMC implementation in 4 districts leaving costs for five SMC in five districts prioritized as above allocation request. NMCD will engage with Givewell, an existing health sector partner, to mobilise resources to cover this gap.”
  Uganda Funding Request Form, Allocation Period 2023-2025, Global Fund.

• 38

  For more detail on how this might limit the impact of our funding, see the “What is funging, and how does it affect impact?” section of our page on impact estimates.

• 39

  The “Other funders take AMF or MC into account” rows (e.g., the nets row) are in our separate spreadsheet Likelihood of crowding out GFATM/PMI, which feeds into our CEAs for SMC and nets. For example, values from this spreadsheet are pulled into the SMC CEA here.

• 40

  See our VAS CEA sheet on leveraging and funging here and our New Incentives CEA sheet on leveraging and funging here.

  • For New Incentives, see the Funging section; in it we only address the likelihood that the grant immediately at hand might be funging, rather than including the possibility that creating the expectation of future funding is pushing out other actors' spending.
  • Similarly for vitamin A, we don’t include long-term funging concerns in the Funging section of the report.

• 41

  For more information about this grant, see this grant page.

• 42

  See the Data section of our report on “Estimating Equivalent Coverage Years for Long-Lasting Insecticide-Treated Nets (LLINs)”: “Our current input starts with a durability estimate for Vestergaard's PermaNet 2.0 LLIN, which is based on all field monitoring studies of PermaNet 2.0 that we are aware of that present results compliant with WHOPES monitoring guidelines. We then make rough adjustments for other types of nets that AMF purchases that may have more limited performance data.”

• 43

  See the following rows of our CEA of ITN distribution, showing the types of nets used in each country. N.B. that we have not made ITN grants in Chad or South Sudan and think the other six countries are more representative of our grantmaking.

• 44

  We recommended a grant in 2021 in part to learn more about the durability of the DuraNet Plus, a PBO-treated net. We haven’t yet received the results of this monitoring and evaluation.
  “We have relied on a large number of data points in estimating the effective lifespan of a standard LLIN (without PBO). There appears to be significant variation found across studies and locations. We are unsure about the durability of the PBO component when it is added to nets, and our impression is that limited evidence exists on this subject. A similar test will be done in Ondo. Our understanding is that there are not yet any durability studies specifically of the DuraNet Plus net, the type of net that Malaria Consortium will distribute in Ondo and Anambra, which contains PBO… [We think this grant will:] Improve the accuracy of parameters in our cost-effectiveness analyses for future LLIN campaigns, particularly for campaigns that will use the DuraNet Plus net and somewhat for other PBO net campaigns.”
  GiveWell, Malaria Consortium — Monitoring and Evaluation of Net Distribution in Anambra, Nigeria (November 2021) grant page.

• 45

  “PBO and PPF nets have shorter durability than LLINs such as PermaNet 2.0.”
  David McGuire, Comments on a draft of GiveWell’s nets report 2023 (unpublished).

• 46

  Another major issue with campaigns is that they only happen every three years and there is increasing evidence that nets are not lasting that long, thereby leaving people exposed in the meantime. According to a 2021 publication, “Of the 40 countries in this analysis, 35 show LLIN median retention times of under 3 years, with an overall median value of 1.64 years (IQR 1.33–2.37).” Maps and metrics of insecticide-treated net access, use, and nets-per-capita in Africa from 2000-2020 | Nature Communications”
  David McGuire, “Review of GiveWell LLIN report,” 2023, p. 8.
  “Of the 40 countries in this analysis, 35 show LLIN median retention times of under 3 years, with an overall median value of 1.64 years (IQR 1.33–2.37).”
  Bertozzi-Villa et al. 2021, p. 7.

• 47

  While we are unsure of the best way to compare the results of the study to our estimate of ~2.1 “effective coverage years,” simply comparing these two numbers would suggest ~20-25% (1.64 / 2.11) lower net durability than we assume, which could lead to a similarly sized decrease in cost-effectiveness.

• 48

  These calculations were done in an unpublished analysis.

• 49

  See the cell note on this cell of our nets CEA:
  “We use the estimated increase in median mortality to permethrin (5% increase over 6 years, see cell formulas).
  ‘Trends analyses were conducted to determine whether there were any significant changes between 2010 and 2016 in malaria vector resistance to specific insecticides, and in specific vector groups (Fig. 4.4 and Fig. 4.5). A global increase in resistance frequency was observed for all pyrethroid insecticides tested. Increases were greatest for etofenprox (44% rise, from 7% to 51%), alphacypermethrin (40% rise, from 10% to 50%) and cyfluthrin (28% rise, from 4% to 32%). The increase was less pronounced for the other pyrethroids although these also had a higher initial resistance frequency in 2010: deltamethrin (14% rise, from 20% to 34%), permethrin (5% rise, from 40% to 45%) and lambda-cyhalothrin (3% rise, from 33% to 36%). This indicates that increasing resistance is an issue for all pyrethroids, and that reductions in susceptibility are most marked for those insecticides for which susceptibility was highest in 2010. Further evaluations will be undertaken to identify whether there are differences in resistance frequency and trends over time between insecticides of the pyrethroid class, in order to guide requirements for insecticide resistance monitoring.’ WHO, “Global Report on insecticide resistance 2010-2016”, p. 20.”

• 50

  The studies reviewed were Hancock et al., 2020 and Moyes et al., 2020. Hancock et al., 2020 observed increases of more than 5 percentage points per year from 2005-2017 in some areas: "Predicted mean proportional mortality to deltamethrin was below 0.9 (the WHO threshold for confirmed resistance) across 15% (95% credible interval [CI] 13%–17%) of the west region in 2005, and across 98% (CI 96.6%–98.7%) of the region in 2017."
  (98% - 15%) / 12 years = 6.9 ppts / year

• 51

  This was the average increase in resistance across insecticides in western Africa cited in the original WHO report, which appears more consistent with the more recent studies:
  “This may indicate a steady overall increase in pyrethroid resistance frequency, irrespective of the pre-existing frequency. When examining data for the three African subregions, an increase from 2010 was most notable in west Africa (22% rise, from 13% to 35%), and was similar for the two other WHO subregions of central Africa (14% rise, from 16% to 30%) and east and southern Africa (11% rise, from 42% to 53%).”
  WHO, “Global Report on insecticide resistance 2010-2016”, p. 20.
  22% / 6 years = 3.7 percentage points per year.

• 52

  This is based on an internal, unpublished analysis.

• 53

  See the ”Adjustment for lower vaccine efficacy in Nigeria” section of our report on New Incentives: “This [vaccine efficacy] effect size is based on four sources of evidence on measles vaccine efficacy (each discussed below):

  • The main estimate of measles vaccine efficacy from the LiST meta-analysis (discussed above).
  • Results from biomarkers pilots of measles immunity in Nigeria, which found lower-than-expected agreement between reported vaccination and detection of antibodies (more).
  • Other studies of immunity biomarkers in Nigeria (more).
  • Meta-analyses of the efficacy of the measles vaccine in Africa compared to vaccine efficacy overall (more).”

• 54

  To read more about our current adjustment, see the ”Adjustment for lower vaccine efficacy in Nigeria” section of our report on New Incentives: “We apply a -19% to -20% adjustment to our vaccine efficacy estimates to account for lower vaccine efficacy in New Incentives’ program than implied by the vaccine meta-analyses, with a 25th to 75th percentile confidence interval of -35% to -10%.”

• 55

  The red team estimated that a 40-60% downward adjustment might be plausible given that one study found only 22% of children reported by their parents to be vaccinated for measles tested positive for measles antibodies. As we write in our report on New Incentives, this implies a vaccine efficacy of 26%, much lower than our starting-point estimate of 85% for measles.
  As we note in the report, this biomarkers study was not designed to assess vaccine efficacy and there may be other explanations for this low proportion of children with antibodies. However, if we assume that implied measles vaccine efficacy (26%) is accurate, then our adjustment for lower vaccine efficacy for measles should be -70%:
  85% * (100% + [Adjustment for lower measles vaccine efficacy in Nigeria]) = 26%
  [Adjustment for lower measles vaccine efficacy in Nigeria] = 26% / 85% - 1 = 30% -1 = -70%
  If we assume, as we do in the ”Adjustment for lower vaccine efficacy in Nigeria” section of our report on New Incentives, that 75% of measles-specific adjustment applies to other vaccines, then we get a total adjustment of 75%*(-70%) = -52.5%, hence the suggested 40-60% downward adjustment.
  “Biomarker pilots of measles immunity in Nigeria. IDinsight conducted two pilots to test the feasibility of using oral fluid measles antibody tests to validate caregiver-reported vaccination status. Both of these pilots found low agreement between the test and caregiver-reported vaccination status, as well as administrative records (e.g., child health cards). Among children reported to be vaccinated, just 22% tested positive for measles antibodies.
  These pilots were not designed to test for vaccine efficacy, and we cannot definitively rule out alternative explanations for this lower-than-expected agreement. However, our interpretation of these results is that they imply lower vaccine efficacy than our main estimate of 85% for measles. We estimate (based on a number of uncertain assumptions, discussed in footnote) that measles vaccine efficacy would need to be 26% rather than 85% to explain the results, assuming that the results were entirely explained by vaccine efficacy, and that the sensitivity of the test is the same as reported.”
  GiveWell, report on New Incentives, “Biomarker pilots of measles immunity in Nigeria” section.

• 56

  “We view the biomarkers results in particular as somewhat concerning, and we are continuing to investigate this issue. We may decide to conduct follow-up research to understand biomarkers for infants vaccinated through New Incentives' program. We also have not deeply reviewed seroconversion studies or meta-analyses that separate effects by low-income vs. non-low-income countries, and we have not fully explored all potential explanations for low vaccine efficacy.” GiveWell, "New Incentives (Conditional Cash Transfers to Increase Infant Vaccination)", 2020 version.

• 57

  For our current approach to drug resistance in SMC, see this section of our report on SMC.

• 58

  “Anopheles stephensi is a mosquito species that is capable of transmitting both Plasmodium falciparum and P. vivax malaria parasites. Unlike the other main mosquito vectors of malaria, it thrives in urban and man-made environments. Originally native to parts of South Asia and the Arabian Peninsula, An. stephensi has been detected, to date, in 7 countries in the African continent.”
  WHO initiative to stop the spread of Anopheles stephensi in Africa, 2023 update.

• 59

  See the indirect deaths section of our internal CEA consistency guidance. This document was made for an internal audience and published for the sake of transparency. It’s not thoroughly vetted.

• 60

  See the indirect deaths section of our internal CEA consistency guidance. This document was made for an internal audience and published for the sake of transparency. It’s not thoroughly vetted.
  Note that the indirect deaths estimate for VAS is implicit in the VAS CEA because it calculates mortality effects based on effects on all-cause mortality. See these rows in the VAS CEA.

• 61

  See the “Long-term income increases” sections of our report on SMC, our report on nets, our report on VAS, and our report on New Incentives’ program.

• 62

  This is based on an unpublished internal analysis of our cost-effectiveness estimates at the time of red-teaming.

• 63

  See the “Indirect malaria mortality” section of our report on ITNs, the
  “Non-malaria deaths indirectly averted” section of our report on SMC, and the “Adjustment for all-cause mortality effect” section of our report on New Incentives.

• 64

  We estimated the percentage of benefits from development effects for each top charity intervention by averaging across relevant locations included in our analysis. However, this does not always cover every location we analyze for that intervention. This is because not all locations are representative: some locations have not received funding from us, or we decided to discontinue funding in those areas. For clarity, we rely on our intervention reports to determine the relevant locations and provide links to explanations in those reports.
  For mass distribution of nets, 31.5% of benefits come from development effects on average across DRC, Guinea, Nigeria (Global Fund states), Nigeria (PMI states), Togo, and Uganda. See this row of our nets CEA for the figures averaged and footnote 23 of our report on nets for the reasoning behind using these locations.
  For SMC, 26% of benefits come from development effects on average across Burkina Faso, Togo, and various states in Nigeria. See this row of our SMC CEA for the figures averaged and footnote 25 of our report on SMC for the reasoning behind using these locations.
  For VAS, 20% of benefits come from development effects on average across Burkina Faso, Cameroon, Côte d'Ivoire, DRC, Guinea, Madagascar, Mali, Niger, and five states in Nigeria: Adamawa, Benue, Ebonyi, Nasarawa, and Taraba (for Helen Keller) and Chad (for Nutrition International). See this row of our VAS CEA for the figures averaged and footnote 10 of our report on VAS for the reasoning behind using these locations. NB: We use an estimate of 20% across the board for VAS.
  For New Incentives, 20% of benefits come from development effects on average across Bauchi, Gombe, Jigawa, Kano, Katsina, Kebbi, Sokoto, and Zamfara. See this row of our New Incentives CEA for the figures averaged and footnote 14 of our report on New Incentives for the reasoning behind using these locations.

• 65

  See our nets CEA here.

• 66

  See the Number of SMC cycles per year and Proportion of annual malaria mortality that occurs during the SMC season rows in our SMC CEA. Before updating the “proportion of annual malaria mortality” figure, this cell read “52%” rather than the current “62%”.

• 67

  See the “Changes in net distribution intervals” section of our nets CEA.

• 68

  Specifically, the red team noticed that the model appears to suggest that delaying a net distribution will increase cost-effectiveness relative to a fixed counterfactual distribution. More specifically, our model seemed to suggest that it may be more cost-effective to reduce the gap between campaigns from 45 months to 42 months instead of from 45 months to 36 months. We think this is because the model implicitly assumes that it takes fewer nets to reduce the campaign spacing by less time, so that the smaller reduction in time between campaigns has added benefit with lower costs. We don't discuss this feature of the model in our intervention report, and the red team found it difficult to evaluate the accuracy of this assumption.

• 69

  The simple CEAs are included in the following linked sections of our nets report, SMC report, New Incentives report, and VAS report. The simple CEAs are also included in the broader CEA spreadsheets: nets simple CEA, SMC simple CEA, New Incentives simple CEA, and VAS simple CEA.

• 70

  For examples of simple CEAs in new non-top charity grant pages, see our grant to IRD Global for a breastfeeding reminders pilot, our grant to ALIMA for a malnutrition treatment in Chad, and our grant to the Clinton Health Access Initiative for oral rehydration solution/zinc co-delivery.
  For an example of simple CEAs in our intervention reports, see our intervention report on oral rehydration solution and zinc. For all intervention reports, see this table.

• 71

  The 20% adjustment for transmission effects of vaccines referred to here is a combination of the “Lower likelihood of infecting others” and “Herd immunity” factors, included in our New Incentives CEA here and described in our report on New Incentives here:

  • Lower likelihood of infecting others (+6%): People who are vaccinated may be less likely to spread infection to others. The size of this effect will depend on (1) how much vaccination reduces transmission, and (2) the proportion of the population that is unvaccinated (the higher this proportion, the greater the benefit would be). This effect is included in more complicated models of vaccine efficacy, and it is possible that ignoring this effect leads us to underestimate vaccine efficacy, especially for diseases with high transmissibility (e.g., measles). We have not yet sought to model these effects, since doing so would add a substantial degree of complexity to our model, but we believe this factor may be worth further investigation, since it is a key component of vaccines' impact.
  • Herd immunity (+13%): High overall vaccination levels in treatment groups suggest that it is possible that New Incentives' program could lead at least some areas to achieve herd immunity as the program scales up or in specific areas with high vaccination rates. We have not yet sought to model these effects, since doing so would add a substantial degree of complexity to our model, but we believe this factor may be worth further investigation, since it is a key component of vaccines' impact.

• 72

  “‘Change Our Mind’ entry” refers to the Change Our Mind contest that GiveWell ran in 2022. The contest was a call for critiques of GiveWell’s cost-effectiveness analyses that could lead to substantial improvements in our grantmaking. In December 2022, we announced the winners of the contest and reflected on their and others’ submissions.
  “Our first critique is the core modelling approach taken for this cost-effectiveness model. We understand the model developed by GiveWell to be a static model; the gold standard of models for vaccination cost effectiveness are dynamic models (1). Models such as Susceptible-Infected-Recovered (SIR) models may be especially suited. However, we understand from the literature that it is not straightforward to estimate the extent to which these types of models provide differing estimates to static models….Our assessment is that even if the core modelling strategy is unchanged but the latest evidence in the field is taken into account, this is sufficient to affect GiveWell’s estimates of cost-effectiveness by well over 10%.”
  Dr. Yannish Naik & Dr. Samantha Field, “A critique of GiveWell’s CEA model for Conditional Cash Transfers for vaccination in Nigeria (New Incentives)”, October 2022, pp 1-2.

• 73

  “This new $87.5 million grant extends the program’s funding runway in all four countries. It consists of… $6.5 million for Chad (7% of the grant). This amount, intended to be exit funding, is based on adding one additional year of funding at the current scale (as of 2022) of Malaria Consortium's support to the program in Chad.” GiveWell, “Malaria Consortium — SMC Renewal in Nigeria, Burkina Faso, Chad, and Togo (January 2023)”, April 2023.
  See also the “Risks of discontinuing funding in Chad” section of the same page.

• 74

  See these cells for Zambia’s long-term coverage and these cells for the average long-term coverage in GiveWell’s Summary of AMF PDM monitoring.
  We also flag this relatively low usage in our report on AMF here.

• 75

  AMF distributed nets in “four of Zambia's ten provinces, specifically Central, Eastern, North-Western and Western.” Against Malaria Foundation, “C + E + NW + W Provinces, 2017, Zambia”
  Average access across provinces found by the DHS Program in 2018: (53.3% (access in Central) + 62.9% (access in Eastern) + 60.2% (access in Noth-Western) + 56.7% (access in Western)) / 4 (total provinces) = 58.3% average access across targeted provinces.
  See the rows on Central, Eastern, North-Western, and Western districts on this page on Zambia. Statcompiler, the DHS Program, “Persons with access to an insecticide-treated mosquito net”, Zambia, accessed 7/31/2024.

• 76

  We have significant uncertainty about this estimate and it is based on a rough analysis. If we assume:

  • Enough nets were supplied to achieve universal coverage in the four provinces targeted. (“3 million nets will be distributed in four of Zambia's ten provinces, specifically Central, Eastern, North-Western and Western, to achieve universal coverage - all sleeping spaces covered.” Against Malaria Foundation, “C + E + NW + W Provinces, 2017, Zambia”
  • 5% of nets will be purchased but not delivered.
  • There will be 5% attrition in net usage given the survey took place less than a year after AMF reports delivering the nets. (Survey timing: “Field work: July 2018 - January 2019.” DHS Program, “Zambia: Standard DHS, 2018,” accessed 8/7/2024. Net delivery: “When: Jan-May 18.” Against Malaria Foundation, “C + E + NW + W Provinces, 2017, Zambia.”)
  • We should include our usual adjustments for misappropriation (-2%) or false monitoring (-2%).

  Given the above assumptions, we would estimate that net access would be around 85-90% in the targeted areas:

  (100% - 5% (purchased but not delivered)) * (100% - 5% (attrition)) * (100% - 2% (misappropriation) - 2% (false monitoring)) = 86.6% access to nets

• 77

  “AMF told us that the Zambia campaign had insufficient ITNs, and thus the number of ITNs that could be distributed to households was capped (Against Malaria Foundation, conversation with GiveWell, November 20, 2019 [unpublished]). If households did not receive enough ITNs to cover all sleeping spaces, this may have contributed to the relatively low coverage rate measured in Zambia.” Footnote 112 of our report on Against Malaria Foundation. It was unclear to the red team what led to this lack of nets (e.g. mis-estimation of target population?), whether we could have foreseen it, and whether or how these issues would affect cost-effectiveness.

• 78

  “We calculate surprisingly low cost per supplement estimates (varying between $0.53 and $0.72) for Helen Keller-supported programs in Côte d’Ivoire, the Democratic Republic of the Congo (DRC), Kenya, and Nigeria. Our best guess is that these figures are too low to be plausible. One reason they are so low may be that some other actors’ costs are not included in our analysis, although Helen Keller has suggested reasons for these lower costs that we also think are plausible (details in footnote). Because of our uncertainty, we use an average cost per supplement figure of $1.02 (the weighted average of all other countries’ cost per supplement in our analysis) for these four countries.”
  GiveWell, report on vitamin A supplementation, “Shortcomings and uncertainties” section.
  We further describe our reasoning behind the cost-per-supplement adjustment used for each country in these explanatory notes.

• 79

  These vary from $0.53 per child treated in a supplementation round in Cote d’Ivoire to $1.39 in Guinea. GiveWell, Helen Keller International cost per supplement, 2022.
  The reasoning behind our cost-per-supplement estimate for each country can be read in this document.

• 80

  These distribution methods include door-to-door, community, school-based, routine health facility-based, and others. See the “Distribution method” column in GiveWell, Helen Keller distribution methods for VAS mass distribution campaigns [2022].

• 81

  This is based on unpublished internal analysis.

• 82

  This is based on unpublished internal analysis.

• 83

  “Are these recent, or historical? [The cost per net] seems too low for today given recent inflation in manufacturing and shipping costs.” Dr. Justin Cohen, “ITN report review,” 2023, p. 2.

• 84

  This assessment is based on an internal, unpublished analysis. As an illustration of what this comparison consisted of, if we had looked at the prices paid for a PBO net and found that AMF paid $1.25 per net and Global Fund paid $8 per net, that might have suggested to us that our costs were strongly out of line with plausible estimates.
  Our analysis is unpublished because we don’t have permission from the Global Fund or AMF to publish their costs per net. The reasoning behind this is explained further in the “How much does each net cost?” section of our nets report: “Unlike most of our analysis, our estimates of cost per net are unpublished and we do not break down our calculations in detail in this report. This is because we have not received permission to publish country-specific cost estimates we received from the Global Fund or AMF’s projections for the costs it expects to pay to procure each net by country and net type, which our calculations rely on. AMF notes that, while they would like to allow the publishing of net costs, in keeping with their approach to transparency, doing so has the risk of compromising sensitive net price negotiations with manufacturers.”

• 85

  This text is pulled from the “How could we be wrong?” section of our report on VAS. The hyperlinked “(more)”s throughout the quote will lead to the report on VAS.

• 86

  “[GiveWell:] ‘We think our funding of nets from AMF causes mass campaigns to achieve higher coverage… these cause people to have access to more protection from nets than they otherwise would have.’ [Cohen:] Despite my quibbles above, I still 100% agree with this statement. My comments suggest only that perhaps even higher coverage is attainable with the same resources.” Dr. Justin Cohen, “ITN report review,” 2023, p. 2.

  “I… fully support your conclusion that ITNs are a cost-effective and critical tool. However, I will provide some of my thoughts on their limitations, suggest alternative perspectives on how to assess their cost-effectiveness and what other complementary investments in VC GiveWell might consider in the future.” David McGuire, “Review of GiveWell LLIN report,” 2023, p. 1.

• 87

  PBO and chlorfenapyr are each used in different types of “next-generation” nets, which aim to combat resistance to pyrethroid (the main insecticide used in ITNs) in mosquitos. For more information on types of nets, see the “What are ITNs?” section of our report on nets. For more information on how we account for the impacts of PBO and chlorfenapyr on insecticide resistance, see the “Insecticide resistance” section of our report on nets.

• 88

  “Not all ITNs are created equally, and generalizations should not be made. You should conduct your assessments based on the specific type of ITN being used and the transmission setting being targeted. I would encourage you to reach out to Prof Tom Churcher at Imperial College to check your estimated impact of nets on morbidity and mortality based on these variables. In my opinion he has the best data and modelling on ITNs and particularly dual-AI nets.”
  David McGuire, “Review of GiveWell LLIN report,” 2023, pp. 5-6.
  “I find this problematic as the decision to fund and purchase chlorfenapyr nets vs PBO nets vs pyrethroid nets is not a future problem but very much the key question today that determines the impact of a dollar spent on nets. Based on the available RCTs and mathematical modeling based on them (you might check out Imperial’s MINT tool).”
  Dr. Justin Cohen, “ITN report review,” 2023, p. 2.

• 89

  “The user-friendly interface allows access to results from a malaria transmission model that explores the potential epidemiological impact of deploying multiple vector control tools… Local human, mosquito and cost data are used to characterise the setting of interest (referred to as a ‘region’ in the tool). Version 2.0 of the tool has all the functionality of version 1.0 but with the addition of new ITNs recently recommended by the WHO.”
  MINT (Malaria Intervention Tool) User Guide, accessed 8/1/2024, p. 4.

• 90

  Both the updated durability estimates and the guidance for adjusting for PBO content decline in our analyses are unpublished as of 8/2/2024, but are in use for current grant investigations.

• 91

  “The footnote #21 on p13 saying AMF finds most nets in places where it conducts campaigns are AMF nets feels circular: would suggest using MIS/DHS survey data instead – this still shows that most nets come from campaigns, but it isn’t AMF specific.” (p. 1)
  “I note that here and elsewhere you rely on AMF surveys – it is worth noting that DHS/MIS surveys ask the same questions and would I guess provide a much richer dataset of usage patterns across a wider set of geographies, implementers, and time points.” (p. 3)
  “Usage by age. [GiveWell:] ‘We assume that people of different ages sleep under LLINs in equal proportion to their share of the population’. I think this is probably not a good assumption. It can be evaluated from DHS/MIS surveys, which compile a complete net roster of all LLINs in the house and ask about where each was received from and who used it.” (p. 3)
  Dr. Justin Cohen, “ITN report review,” 2023.

• 92

  We include this list of concerns in the “Do experts and practitioners see ITNs as an effective investment?” section of our report on nets.

• 93

  “Historically I believe that funders have relied too much on the so-called experts in North America and Europe (like myself) without sufficiently consulting with those in Africa tasked to make it all work. Many of these managers are frustrated by the fact that donors are prioritizing nets over other VC [vector control] tools rather than allowing them to tailor their VC strategies for optimal impact based on what they think will work best within their specific transmission and socio-cultural settings. Malaria managers are enthusiastic about ITNs, especially new dual-AI nets, but they are also quite vocal about the challenges with uptake and attrition and the need for complementary interventions.” (p. 2)
  “GiveWell would also be wise to consider future investment in other vector control tools that are currently available or under development in settings where they may be more appropriate, preferred by NMCPs/users [national malaria control programs], or provide significant additional impact when layered on top of ITNs. For example, there are many countries, particularly in East and Southern Africa where there is a strong preference for and long history of Indoor Residual Spraying (IRS).” (p. 3)
  “NMPs [national malaria programs], including Uganda's NMCD, have expressed concerns for many years about the potential impact of reducing IRS coverage in high transmission areas. This is based on very bad experiences they have had in Uganda when donors have withdrawn IRS funding. More broadly, data generated under the NgenIRS project showed spikes in malaria incidence when IRS was removed from areas where nets are deployed.” (p. 7)
  David McGuire, “Review of GiveWell LLIN report,” 2023.

• 94

  “Historically I believe that funders have relied too much on the so-called experts in North America and Europe (like myself) without sufficiently consulting with those in Africa tasked to make it all work. Many of these managers are frustrated by the fact that donors are prioritizing nets over other VC [vector control] tools rather than allowing them to tailor their VC strategies for optimal impact based on what they think will work best within their specific transmission and socio-cultural settings.” David McGuire, “Review of GiveWell LLIN report,” 2023, p. 2.

• 95

  “It is noteworthy that no country that has successfully eliminated malaria has done so with nets. Nets are an effective control tool in medium to high-transmission areas, but as countries progress towards elimination they must rely more on surveillance, case management, focal IRS [indoor residual spraying], larviciding, etc. And, as malaria rates come down, cost-effectiveness becomes a less useful metric since nothing meets traditional cost-effectiveness thresholds in low transmission settings.” David McGuire, “Review of GiveWell LLIN report,” 2023, p. 3.

• 96

  “With between 200-300 million ITNs being distributed each year, countries and their donor and implementing partners should also consider the environmental impact of accumulating plastic waste. There are pilot efforts underway to minimize the amount of plastic packaging and to implement recovery and recycling of ITNs. However, looking ahead it would be preferable to deploy VC tools that would help enable more significant reduction of plastic waste and easier and more environmentally friendly disposal.” David McGuire, “Review of GiveWell LLIN report,” 2023, p. 2.

• 97

  “Furthermore, to maximize the ability of donors to procure as many nets as possible, there has been a move to standardize the size, shape and color of ITNs. This makes perfect economic sense, but research has shown great variability in user preferences and, therefore, this one-size-fits-all approach may be a contributing factor to reduced uptake of ITNs. For example, many people prefer conical nets since they are much easier to hang and tie up during the day, and in some predominantly Muslim communities there is an aversion to white nets because of their resemblance to traditional funeral shrouds.” David McGuire, “Review of GiveWell LLIN report,” 2023, p. 2.

• 98

  We estimate the cost effectiveness of Global Fund spending to calculate our leveraging and funging adjustments. This is necessary because our leveraging and funging adjustments try to account for the possibility that our funding a program affects other actor’s spending. For example, if we think the Global Fund might have funded a program we end up funding, part of the effect of our funding is to free up funds for the Global Fund. In order to incorporate that effect into our analysis, we have to estimate how cost-effectively the newly freed funds will be spent.

• 99

  “I’d have some concerns given the structure of GF allocations – a country has a malaria-specific envelope that they allocate across malaria activities. If funds currently intended to distribute nets went to something else, it would be malaria case management, chemoprevention, surveillance, etc – not HIV care.”
  Dr. Justin Cohen, “ITN report review,” 2023, p. 6.
  “The assumption that all funding that the Global Fund could use for SMC in Burkina Faso but uses for other activities is spent on programs combating these diseases could be challenged as TGF envelopes are fixed per disease area. Therefore any diversion of funding from SMC should be reconverted in a different malaria-related activity but not in either TB or HIV AIDS.”
  Dr. André Tchouatieu, “GiveWell SMC report: Feedback from an independent review”, 2023, p. 3.

• 100

  “Overall SMC is and will remain for many years one of the most cost-effective interventions to prevent malaria. Investment towards such intervention is valuable in all senses, despite some of the criteria used to assess the worth of the investment remaining unclear.”
  Dr. André Tchouatieu, “GiveWell SMC report: Feedback from an independent review”, 2023, pp. 4-5.

• 101

  “The clear success of SMC is however now posing issues of equity. In SMC areas, children up to 5 years of age are highly protected against the disease, anticipating a shift of the burden of the disease to older age groups and this needs to be addressed. In non-SMC areas, children in the same age group as those living in the SMC areas are not benefiting from any sort of protection apart from the LLINs which are also distributed in SMC areas and this can be considered inequitable. And worst is that this situation can happen within the same country, Nigeria being an example.”
  Dr. André Tchouatieu, “GiveWell SMC report: Feedback from an independent review”, 2023, p. 5.

• 102

  “The notion of increased income at a later age for children benefiting from SMC is ambiguous. The consideration of doubling consumption as a key determinant of moral values seems unclear and very subjective as it depends on several other non-malaria-related factors. Although it is understandable to include such criteria in establishing the moral weight of SMC impacts (e.g. increased income vs. reduced deaths), it is however difficult to relate any increased income to SMC alone as there are several factors and a considerable duration of time before this can be evaluated, as the concept is considering later years in children’s life.”
  Dr. André Tchouatieu, “GiveWell SMC report: Feedback from an independent review”, 2023, p. 2.

• 103

  “This impact on household purchase power is a more direct and obvious impact than the increased income at a later age. In Mozambique as an example, it was estimated that the median household costs associated with uncomplicated malaria among individuals of all ages were US$ 3.46 (IQR US$ 0.07–22.41) and US$ 81.08 (IQR US$ 39.34–88.38) per severe case6. In such a context, preventing malaria cases would generate savings for the household, but not automatically translate into long-term income benefits for the child.”
  Dr. André Tchouatieu, “GiveWell SMC report: Feedback from an independent review”, 2023, p. 2.

• 104

  See the “Treatment costs averted from prevention” cell of our SMC CEA.

• 105

  Responding to the question, “Do you understand why we believe cash incentives for immunization are an effective giving opportunity? Which parts are unclear?”, Dr. Cohen wrote, “Yes, this is clear and persuasive. In particular, the model of working in low-baseline vaccination rate areas and offering a strong incentive (that helps overcome multiple barriers to vaccination) for vaccines for diseases that are important causes of mortality is persuasive.”
  Dr. Jessica Cohen, “Comments on GiveWell’s draft New Incentives Report”, November 2023, p. 1.

• 106

  “I don’t understand the reason for the huge increases in vaccination rates in the control areas in the RCT. I agree with your conclusion that those effects are unlikely to change the determination that NI’s program significantly increased vaccinations but I have two reservations. First, it seems important to determine what it was that increased vaccination rates in the control areas so dramatically, because that change was bigger than the treatment effect from NI. My hunch is that it was survey effects or spillovers that weren’t totally captured.”
  Dr. Jessica Cohen, Comments on GiveWell’s draft New Incentives Report, November 2023, p. 1. Note that Dr. Cohen’s response was shared before we compared the control group results to independent vaccine coverage surveys.

• 107

  “In 2023, we also received feedback from Dr. Jessica Cohen, Bruce A. Beal, Robert L. Beal and Alexander S. Beal Associate Professor of Global Health at Harvard University, expressing some uncertainty about the control group increase along with possible explanations. To investigate this question in more detail, we compared the RCT results against independent vaccine coverage surveys in 2016-2017, 2018, and 2021. While these survey results are noisy, we found that perhaps 40%, and plausibly substantially more, of this increase can be explained by a wider regional trend of increasing vaccine coverage around the time the RCT was conducted (details on our method in footnote). We discuss how we account for this trend in our analysis elsewhere in the report. Broad alignment between the RCT results and general regional trends slightly increases our confidence in the RCT.”
  GiveWell, technical report on New Incentives, “What was baseline vaccination coverage in the RCT?” section.

• 108

  “Not including potential benefits of clinic visits for young children (where health concerns can be addressed in addition to the receipt of vaccinations) could cause you to underestimate impact.”
  Dr. Jessica Cohen, Comments on GiveWell’s draft New Incentives Report, November 2023, p. 2.

• 109

  See the cell labeled “Increased clinic utilization” in our New Incentives CEA, which as of August 2024 is set to 0%.

• 110

  "Outcome: Ever visited clinic… Adjusted OLS results: 0.05 ([95% CI] 0.03, 0.08)"
  IDinsight, Impact evaluation of New Incentives, final report, 2020, Table 13, p. 36.

• 111

  “In response to the specific questions concerning why GiveWell believes VAS is an effective giving opportunity and possible modifications of the GiveWell C-E analyses, I conclude that VAS (delivered either through campaigns or routine, facility-based health care services) is still a cost-effective intervention in settings where there is an elevated prevalence of VAD.”
  Dr. Kenneth Brown, Comments on GiveWell Vitamin A Supplementation cost-effectiveness analyses, November 2023, pp. 1-2.

• 112

  “I do not see blanket VAS campaigns as one of four main GiveWell priorities. There may be huge benefits of VAS in some situations, but there can also be harms, and per current evidence, the harms counterbalance the benefits in the blanket VAS distributions.”
  Dr. Christine Benn, Feedback on GiveWell vitamin A supplementation research, December 2023, p. 5.

• 113

  “The results are compatible with the interpretation that the overall mortality effect of VAS observed in the old trials is no longer representative. The declining effect of VAS over calendar time has been interpreted as being due to less VAD, or due to other interventions such as measles vaccine attenuating any effect of VAS on other infections, as also mentioned in the GiveWell report.”
  Dr. Christine Benn, Feedback on GiveWell vitamin A supplementation research, December 2023, p. 3.
  “In principle, the potential effect of VAS campaigns on child mortality can be estimated from the impact of VAS on mortality, as demonstrated by carefully controlled efficacy trials, and the coverage of the VAS campaign. However, if the target populations of current-day interventions differ substantially from the populations included in previously completed mortality trials (in terms of the prevalence and severity of VAD, the overall mortality rate, and the causes of mortality), the estimated impact of VAS may need to be adjusted for these trends.”
  Dr. Kenneth Brown, Comments on GiveWell Vitamin A Supplementation cost-effectiveness analyses, November 2023, p. 3.
  “With regard to the prevalence of VAD, the analyses by Stevens et al (2015) indicate that the overall prevalence of VAD in low and middle-income countries declined from 39% in 1991 to 29% in 2013. However, in sub-Sahara Africa, where GiveWell supports VAS campaigns, the prevalence of VAD did not change significantly over that period of time and remained at ~40%. Admittedly, the prevalence of VAD may have declined since 2013 with the introduction of VA-fortification programs, but the actual coverage and impact of these programs on young children’s VA status have not been well documented. Another issue that undermines comparisons of VAD prevalence over time is that fact that earlier studies did not adjust serum retinol concentration for the presence of infection/inflammation, whereas more recent studies generally do, thereby reducing the apparent prevalence of VAD relative to the earlier studies.”
  Dr. Kenneth Brown, Comments on GiveWell Vitamin A Supplementation cost-effectiveness analyses, November 2023, p. 4.

• 114

  “Rather, many studies have suggested that preventing VAD is just one of several functions of high-dose VAS (see figure below, from [4]). VAS is an immune-modulator, and its effect may thus be affected by other interventions/factors, which impact the immune system. Among such potential effect modifiers are sex, age group, and vaccines.”
  Dr. Christine Benn, Feedback on GiveWell vitamin A supplementation research, December 2023, p. 2.

• 115

  “We are in the process of conducting a meta-analysis of the existing studies assessing the mortality effect of VAS campaigns, based on these data. We have confidentially shared the preliminary results. The meta-analysis indicates no overall benefit, and in fact a signal of harm of VAS campaigns, given alone or in combination with other interventions.”
  Dr. Christine Benn, Feedback on GiveWell vitamin A supplementation research, December 2023, p. 4.

• 116

  The reasons listed above are pulled from the “What is the observational evidence on VAS and mortality?” section of our report on VAS.

• 117

  “However, the GiveWell document fails to consider the problem of ascertainment bias, which may be the most important concern if substantial mortality went unreported, as is likely the case given the data collection methods. Because of the small number of research staff members, the limited training of the child-care center staff, the small proportion of reported mortality that was confirmed, and the lack of study population enumeration data and population-based mortality assessment, the methods do not seem to be sufficiently rigorous for the results to be plausible (see, for example, comments by Sommer et al, 2013).”
  Dr. Kenneth Brown, Comments on GiveWell Vitamin A Supplementation cost-effectiveness analyses, November 2023, p. 4.

• 118

  “In 2023, we received feedback on our analysis of VAS from Kenneth Brown, Distinguished Professor Emeritus at UC Davis, arguing that the results of DEVTA might be biased because of ascertainment bias (i.e., mismeasurement of deaths). Similar points have been leveled by other researchers in the past. While we would expect the study’s methods for measuring mortality to be unreliable (details in footnote), we don’t have any reason to think this would apply more to children in areas receiving VAS than control areas, and so this does not significantly affect our interpretation of the study results.”
  GiveWell, report on vitamin A supplementation, “Incomplete measurement of mortality” section.

• 119

  “We don’t currently account for any negative health impacts from VAS campaigns. Before 2023, we assessed the risk as low, although we have recently received expert feedback that we could be underestimating the potential harms. Our level of concern on this question remains fairly low, but we may do more work to investigate this in the future.”
  GiveWell, report on vitamin A supplementation, “Does VAS have unintended negative consequences?” section.

• 120

  "With regard to the statements on possible adverse effects of VAS on the incidence of respiratory infections, I believe it is important to recognize that it is notoriously difficult in community-based studies to assess the presence of lower respiratory tract infections (ALRI -- bronchitis and pneumonia), which are much more important causes of mortality than upper respiratory tract infections. In most community-based studies, the diagnosis of ALRI depends on the (reported or observed) presence of cough and age-specific elevated respiratory rate. If VAS results in a more vigorous (protective) cough response, ALRI may be diagnosed more frequently, while this (i.e., the presence of cough) actually might be a beneficial effect of VAS. Thus, caution is needed in the interpretation of these findings, and any studies that included radiographic confirmation of pneumonia should be considered more reliable.”
  Kenneth Brown, email to GiveWell, February 8th 2024 (unpublished).

• 121
  • “In 2023, we also reviewed additional scientific literature critical of VAS. A 2010 commentary by Michael Latham argues that VAS may increase the risk of respiratory infection among children with adequate vitamin A status. This is potentially concerning, since respiratory infections are a leading cause of under-5 mortality in low-income settings. The Cochrane meta-analysis of VAS trials does not report a significant overall impact of VAS on respiratory infection incidence or mortality risk in children under 5. However, we have not yet investigated whether the response may be different among the subset of children who do not have vitamin A deficiency.” GiveWell, report on vitamin A supplementation, “Does VAS have unintended negative consequences?” section.
  • Footnote 319: “Note that we have received feedback from a VAS Expert, Dr. Kenneth Brown, that we should be cautious about conclusions that VAS might increase respiratory disease risk, due to uncertainties about the methods used in field studies to detect respiratory disease. We have not deeply investigated or tried to corroborate this.” GiveWell, report on vitamin A supplementation, footnote 319.

• 122

  “The relationship of health with vitamin A status is a U shape. Too little and there are bad effects leading to death. Too much, and there are effects on bone.”
  Dr. Sherry Tanumihardjo, Review of GiveWell VAS report, December 2023, p. 3.

• 123

  “In some cases, VAS might interrupt growth if children are hypervitaminotic. In this case, we think it is because of acute toxicity causing the child to feel nausea and then not eating for several days. We have evidence of this in South African (van Stuijvenberg et al. 2019; Sheftel et al., 2022) and Malawian children (Williams et al., 2021). Below are two graphs that we published in South African children who regularly consume liver (attached). On the background of good dietary intake of vitamin A, the supplements are mostly stored and do accumulate causing the children to have hypervitaminosis A… In the Malawian school-age children we saw elevated retinyl ester concentrations. Serum retinyl esters will only be elevated if there are very high stores of vitamin A to the point of toxicity.”
  Dr. Sherry Tanumihardjo, Review of GiveWell VAS report, December 2023, p. 1.

• 124

  “We don’t currently account for any negative health impacts from VAS campaigns. Before 2023, we assessed the risk as low, although we have recently received expert feedback that we could be underestimating the potential harms. Our level of concern on this question remains fairly low, but we may do more work to investigate this in the future.”
  GiveWell, report on vitamin A supplementation, “Does VAS have unintended negative consequences?” section.

• 125

  “Dr. West told us that "smaller trials not intended, designed, powered or executed to enable the efficacy of vitamin A on child mortality to be assessed are unlikely to be informative for this outcome."
  Dr. Keith West, email to GiveWell, February 25th, 2024. (unpublished)

• 126

  “We note that the meta-analysis of mortality includes trials of various sizes and our understanding is that not all these trials were designed to measure the impact of VAS on mortality. We’ve received feedback from Dr. Keith West, Professor of Infant and Child Nutrition at Johns Hopkins University, that smaller trials may not be as informative for assessing the impact of VAS on mortality (details in footnote). We’re considering (but haven’t yet prioritized) alternative analyses that exclude these trials.”
  GiveWell, report on vitamin A supplementation, “Reduced mortality for young children” section.

• 127

  For more information, see the “Center for Global Development — Research on why other funders don't fund opportunities GiveWell does (May 2024)” grant page.