Aggregator

This is the first of three posts that form our annual review and plan for the following year. This post reviews and evaluates last year’s progress on our work of finding and recommending evidence-based, thoroughly-vetted charities that serve the global poor. The following two posts will cover (i) our plans for GiveWell’s research in 2018 and (ii) GiveWell’s progress and plans as an organization. We aim to release our metrics on our influence on donations in 2017 by the end of June 2018.

Summary

We believe that 2017 was a successful year for GiveWell’s research. We met our five primary goals for the year, as articulated in our plan post from the beginning of the year:

Our primary research goals for 2017 are to:

1. Speed up our output of new intervention assessments, by hiring a Senior Fellow and by improving our process for reviewing interventions at a shallow level.
2. Increase the number of promising charities that apply for our recommendation. Alternatively, we may learn why we have relatively few strong applicants and decide whether to change our process as a result. Research Analyst Chelsea Tabart will spend most of her time on this project.
3. Through GiveWell Incubation Grants, fund projects that may lead to more top charity contenders in the future and consider grantees No Lean Season and Zusha! as potential 2017 top charities.
4. Further improve the robustness and usability of our cost-effectiveness model.
5. Improve our process for following the progress of current top charities to reduce staff time, while maintaining quality. We also have some specific goals (discussed below) with respect to answering open questions about current top charities.

We achieved our five primary goals for the year:

1. Our intervention-related output was greater than in any past year, although we still see room for improvement in the pace with which we complete and publish this work (more). We hired a Senior Fellow and published nine full or interim intervention reports in 2017, compared to four in 2016.
2. We increased the number of promising charities that applied for our recommendation (more).
3. We added two new top charities: Evidence Action’s No Lean Season (the first top charity to start as a GiveWell Incubation Grant recipient) and Helen Keller International’s vitamin A supplementation program (which joined our list as a result of our charity outreach work). We continued to follow our current Incubation Grant recipients and made several new Incubation Grants to grow the pipeline of new top charities (more).
4. We made substantial improvements to our cost-effectiveness analysis (more).
5. We reduced the amount of staff time spent on following our current top charities. We also completed 17 of the 19 activities outlined in last year’s plan (more).

We discuss progress on each of our primary goals below. For each high-level goal, we include (i) the subgoals we set in our last annual review, (ii) an evaluation of whether we met those subgoals, and (iii) a summary of key activities completed last year.

Goal 1: Speed up intervention assessments

In early 2017, we wrote:

In recent years, we have completed few intervention reports, which has limited our ability to consider new potential top charities. We plan to increase the rate at which we form views on interventions this year by:

• Hiring a Senior Fellow (or possibly more than one). We expect a Senior Fellow to have a Ph.D. in economics, public health, or statistics or equivalent experience and to focus on in-depth evidence reviews and cost-effectiveness assessments of interventions that appear promising after a shallower investigation. In addition, Open Philanthropy Project Senior Advisor David Roodman may spend some more time on intervention related work.
• Doing low-intensity research on a large number of promising interventions. We generally start with a two to four hour “quick intervention assessment,” and then prioritize interventions for a 20-30 hour “interim intervention report” (example). We don’t yet have a good sense of how many of these of these we will complete this year, because we’re unsure both about how much capacity we will have for this work and about how many promising interventions there will be at each step in the process.
• Continuing to improve our systems for ensuring that we become aware of promising interventions and new relevant research as it becomes available. We expect to learn about additional interventions by tracking new research, particularly randomized controlled trials, in global health and development and by talking to select organizations about programs they run that they think we should look into.

How did we do? Achieved our goal.

Due to our uncertainty about the capacity we could devote to intervention assessments, we did not have an explicit target for how many reports we expected to complete. In 2017, we published seven interim intervention reports, two full intervention reports, and completed ~30 quick evidence assessments (defined below). Our research output for 2017 was higher than 2016, when we published one full intervention report, three interim intervention reports, and completed 30 quick evidence assessments.

What did we do?

• Hired Caitlin McGugan as a full-time Senior Fellow to work on intervention reports.
• Completed quick evidence assessments (two to four hours spent reviewing the evidence base) for ~30 programs. (We plan to write more about this process, what we found, and how it informs our work in a future blog post.)
• Published two full intervention reports: (i) conditional subsidies for seasonal labor migration in northern Bangladesh (which led to a new top charity, No Lean Season) and (ii) surgery to repair obstetric fistula.
• Completed seven interim intervention reports, which are less intensive than full intervention reports, on antiretroviral therapy to treat HIV/AIDS, oral rehydration solution, Sayana® Press, SMS reminders for vaccination, distribution of eyeglasses in developing countries, interventions to promote handwashing, and oral pre-exposure prophylaxis (PrEP) for HIV.
• Substantially updated intervention reports on two of our priority programs: seasonal malaria chemoprevention (previously an interim intervention report) and vitamin A supplementation.
• Developed and implemented a system for (i) tracking new published research relevant to our priority programs and (ii) identifying new programs for evidence assessments.

Goal 2: Increase the pipeline of promising charities applying for our recommendation

In early 2017, we wrote:

We would like to better understand whether we have failed to get the word out about the potential value we offer or communicate well about our process and charities’ likelihood of success, or, alternatively, whether charities are making well-informed decisions about their fit with our criteria. (More on why we think more charities should consider applying for a GiveWell recommendation in this post.)

This year, we have designated GiveWell Research Analyst Chelsea Tabart as charity liaison. Her role is to increase and improve our pipeline of top charity contenders by answering charities’ questions about our process and which program(s) they should apply with, encouraging promising organizations to apply, and, through these conversations, understanding what the barriers are to more charities applying.

We aim by the end of the year to have a stronger pipeline of charities applying, have confidence that we are not missing strong contenders, or understand how we should adjust our process in the future.

How did we do? Achieved our goal.

More charities entered our top charity review process in 2017, although it’s unclear whether this was due to our charity liaison activities. Five charities formally applied in 2017, compared to two in 2016, and four in 2015. One of those charities, Helen Keller International’s vitamin A supplementation program, became a top charity.

While we feel our relationships with well-regarded global health and development implementers and funders have improved, we continue to feel limited in our ability to understand whether there are funding gaps for evidence-backed, highly cost-effective work within large international NGOs and multilateral aid organizations such as the Global Fund to Fight AIDS, Tuberculosis and Malaria.

What did we do?

• We had at least one conversation with 42 organizations to introduce them to GiveWell’s work in 2017, compared to 16 in 2016.
• Where organizations running multiple programs expressed interest in applying for our recommendation, we had several calls with them to help determine whether they should apply and which of their programs would be the most promising fit for a top charity evaluation. We had not offered this proactive support to organizations in the past.
• We hosted two charity-focused events: (i) a conference call for charities with GiveWell senior staff to present an update on our work as it relates to charities and to give them a chance to ask questions directly of our senior team and (ii) a networking event for our recommended organizations in London.
• We attended seven conferences on global health and development issues to broaden our network and perspective in subject-matter areas that GiveWell has not historically worked on.

Goal 3: Maintain Incubation Grants

In early 2017, we wrote:

We made significant progress on Incubation Grants in 2016 and plan in 2017 to largely continue with ongoing engagements, while being open to new grantmaking opportunities that are brought to our attention.

Among early-to-mid stage grants, we plan to spend the most time on working with IDinsight and New Incentives (where our feedback is needed to move the projects forward), and a smaller amount of time on Results for Development and Charity Science: Health (where we are only following along with ongoing projects).

Another major priority will be following up on two later-stage grantees, No Lean Season and Zusha!, groups that are contenders for a top charity recommendation in 2017. For No Lean Season, a program run by Evidence Action, our main outstanding questions are whether the program will have room for more funding in 2018 and whether monitoring will be high quality as the program scales. We have similar questions about Zusha! and in addition are awaiting randomized controlled trial results that are expected later this year.

How did we do? Exceeded goal.

As expected, our work last year focused on following up on current grantees. No Lean Season, one of our later-stage grantees, graduated to top charity status and we made one grant to a new grantee, the Centre for Pesticide Suicide Prevention. We also made a number of grants to improve our understanding of the evidence base for our priority programs and deepened our partnership with IDinsight.

What did we do?

• Completed a full review of No Lean Season and named it a top charity. We are currently reviewing Zusha! in greater depth.
• Made Incubation Grants totaling $10.8 million in 2017:
  • $7.8 million to support the continued operation of previous grantees: Evidence Action, Zusha!, New Incentives, Charity Science Health, and the IDinsight embedded GiveWell team.
  • $3.0 million to new Incubation Grant recipients: the Centre for Pesticide Suicide Prevention, UC Berkeley for a long-run follow-up survey on the effect of deworming on consumption, and the Center for Effective Global Action at UC Berkeley to scope other randomized controlled trials (RCTs) for long-term follow-ups.

Goal 4: Improve our cost-effectiveness analysis

In early 2017, we wrote:

We plan to continue making improvements to our cost-effectiveness model and the data it draws on (separate from adding new interventions to the model, which is part of the intervention report work discussed above). Projects we are currently prioritizing include:

• Making it more straightforward to see how personal values are incorporated into the model and what the implications of those values are.
• Revisiting the prevalence and intensity adjustment that we use to compare the average per-person impact of deworming in places that our top charities work to the locations where the studies that found long-term impact of deworming were conducted. More in this post.
• Improving the insecticide-treated nets model by revisiting how it incorporates effects on adult mortality and adjustments for regions with different malaria burdens and changes in malaria burden over time.

How did we do? Achieved goal.

We made substantial progress on improving our cost-effectiveness analysis in 2017.

What did we do?

• Moved to a system of making more frequent updates to our cost-effectiveness analysis. This has made it easier to identify which specific factors are driving changes in the estimated cost-effectiveness of our top charities.
• Revisited how we think about leverage and funging (how donating to our top charities influences how other funders spend their money) and updated our cost-effectiveness analysis accordingly.
• Published a report on how other global actors approach the difficult moral tradeoffs we face.
• Prior to announcing our 2017 recommendations, we performed a sensitivity check on our cost-effectiveness analysis to identify how sensitive our final outputs were to different uncertain inputs. This has helped us identify which inputs we should prioritize additional research on, and we believe it has made our communication more transparent, particularly around our personal values.
• Revisited and updated our prevalence and intensity adjustments for deworming.
• Deprioritized improving how our insecticide-treated net model incorporates effects on adult mortality. A limited number of conversations with malaria experts made us less confident that there was informative research on the question that would improve the accuracy of our models.
• Deprioritized making adjustments for subnational regions with different malaria burdens because it would take substantial time to deeply understand the assumptions informing the subnational models we have seen. We believe this remains an important weakness of our model and that it limits our ability to make high-quality decisions about prioritization among different regional funding gaps.

Goal 5: Improve our process for following top charities

In early 2017, we wrote:

“In 2017, we plan to have a single staff member do most of this work and expect it to take a half to two-thirds of a full-time job. Three other staff will spend a small portion of their time, totaling approximately the equivalent of one full-time job, on this work.”

How did we do? Achieved goal.

We estimate that it took about 40 percent of the staff member’s time who focused on this work plus a small portion of four other staff members’ time, totaling at most and likely somewhat less than the equivalent of a full-time job (roughly half the time we dedicated to top charity updates in 2016).

We believe we maintained or increased the quality of the top charity updates, as we completed or made major progress on all but two of the activities and questions outlined in last year’s plan.

What did we do?

The table below summarizes our progress on each of the activities and open questions outlined in last year’s plan.

Charity Goals and open questions from 2017 plan Did we meet our goal? What did we do? Evidence Action’s Deworm the World Initiative “We have now followed these groups for several years and do not have major outstanding questions about them. We plan to ask for updates on financial information, monitoring results, and room for more funding and have regular phone calls with them to learn about operational changes that might lead us to ask additional questions.” Yes We had regular phone calls, received up-to-date financial information, updated room for more funding, and reviewed new monitoring information from Nigeria, Vietnam, Kenya, and Ethiopia (see rows 11-20 and an overview of what we learned). GiveDirectly “We have now followed these groups for several years and do not have major outstanding questions about them. We plan to ask for updates on financial information, monitoring results, and room for more funding and have regular phone calls with them to learn about operational changes that might lead us to ask additional questions.” Yes We had regular phone calls, received up-to-date financial information, updated room for more funding, and reviewed new monitoring information from Kenya (1, 2). (Overview of what we learned.) Schistosomiasis Control Initiative “We have now followed these groups for several years and do not have major outstanding questions about them. We plan to ask for updates on financial information, monitoring results, and room for more funding and have regular phone calls with them to learn about operational changes that might lead us to ask additional questions.” Yes We had regular phone calls, received up-to-date financial information, updated room for more funding, and reviewed new monitoring information from 2016 programs in a number of countries. (Monitoring information, overview of what we learned.) Against Malaria Foundation (AMF) “Will AMF’s monitoring processes be high quality?” Yes We commissioned IDinsight, an organization with which we are partnering as part of our Incubation Grants program, to observe post-distribution surveys in Malawi and Ghana and report their findings. “Going forward, AMF aims to fund larger distributions and commit funding further ahead of when a distribution is scheduled to occur than it has, for the most part, done in the past. Will this increase the extent to which AMF funds displace funds from other sources, or will there continue to be evidence that AMF’s funds are largely adding to the total number of nets distributed?” Partial We learned relatively little about the displacement/fungibility question because AMF signed relatively few new agreements to fund long-lasting insecticide-treated net distributions in 2017. There was an update to how AMF will be tracking displacement, described in the second paragraph here. “In order to estimate AMF’s room for more funding, we will seek out information on the location and size of funding gaps for mass net distribution campaigns from AMF, the African Leaders Malaria Alliance, and possibly other funders of nets. As we have in the past, we will use this information in conjunction with conversations with AMF about non-funding bottlenecks to its ability to fill various gaps.” Yes We got updates on AMF’s room for more funding, as summarized in this post. The END Fund’s deworming program “We have not yet seen monitoring on par with that from our other top charities from the END Fund. We expect results from coverage surveys from END Fund programs this year. Will these surveys be high quality and demonstrate that the END Fund is funding successful programs?” Yes We saw some monitoring from END Fund programs; previously our recommendation of the END Fund was based on specific monitoring plans that we found credible (more here). “We have not yet tried to compare the cost-effectiveness of the END Fund to our other top charities in our cost-effectiveness model. We will be seeking additional information from the END Fund about cost per treatment and baseline infection rates” Yes We significantly improved our understanding of the END Fund’s cost per treatment and the baseline prevalence in areas where the END Fund works. We completed a cost-effectiveness analysis, though we continue to have lower confidence in our estimates than we do for the deworming organizations that we have recommended for several years. “Questions around room for more funding: the extent to which funding due to GiveWell’s recommendation increases the amount that the END Fund spends on deworming versus other programs, actual and projected revenue from other sources, and what deworming grantmaking opportunities the END Fund expects to have.” Yes We estimated the extent to which funding due to GiveWell’s recommendation increases the amount that the END Fund spends on deworming versus other programs, discussed here. “We visited the END Fund’s programs in Rwanda and Idjwi island, DRC in January 2017 and will publish notes and photos from our visit shortly.” Yes We posted notes and photos from our site visit here. Malaria Consortium’s seasonal malaria chemoprevention program “Further research on the evidence of effectiveness, cost-effectiveness, and potential downsides of seasonal malaria chemoprevention (SMC) (due to time constraints we have not yet completed a full intervention report, though we felt sufficiently confident in the intervention to recommend Malaria Consortium).” Yes We reviewed each of the RCTs included in the Cochrane review for seasonal malaria chemoprevention, and possible negative/offsetting impacts. We updated our interim intervention report to a full intervention report and added new information to our cost-effectiveness analysis. Our key conclusions did not change substantially and SMC remains a priority program. “Getting a better understanding of the methodology Malaria Consortium uses for estimating coverage rates.” Yes We spoke with Malaria Consortium to understand how they measure coverage and updated our cost-effectiveness analysis to account for different levels of coverage in the Malaria Consortium program relative to the headline results of the RCTs in the Cochrane review (conversation notes here). “Completing a more in-depth room for more funding analysis for the program for 2018 than we did for 2017.” Yes We completed a significantly more in-depth room for more funding analysis than we had previously (more here). “We may visit a Malaria Consortium seasonal malaria chemoprevention program in summer 2017.” No We did not conduct a site visit. Sightsavers’ deworming program “We expect to make limited progress this year because the first deworming mass drug administration funded with GiveWell-influenced funds is not expected to take place until September at the earliest and monitoring results aren’t expected until early 2018. Because Sightsavers has done fairly little deworming in the past year, we don’t expect to be able to learn much from its ongoing programs.” Exceeded In 2017, as expected, we learned relatively little about the performance of Sightsavers’ deworming programs, because programs funded with GiveWell-directed funds were at early stages. We did not expect to receive any monitoring results from programs funded with GiveWell-directed funds; however, Sightsavers shared a coverage survey from Guinea with us earlier than expected. The survey found middling coverage results. “Getting more information from Sightsavers about baseline prevalence and intensity of worm infections in the areas it is working, to inform our cost-effectiveness analysis.” Yes We significantly improved our understanding of Sightsavers’ cost per treatment and the baseline prevalence in areas where Sightsavers works (which is used in our cost-effectiveness analysis). “Using Sightsavers’ budget for the projects and planned treatment numbers to improve our estimate of the cost per treatment – another input into our cost-effectiveness analysis.” Yes We significantly improved our understanding of Sightsavers’ cost per treatment and the baseline prevalence in areas where Sightsavers works (which is used in our cost-effectiveness analysis). “Completing a room for more funding analysis for 2018.” Yes We completed a room for more funding analysis (more here). Standout charities “We plan to have at least one phone call with each of these groups to discuss whether anything has changed that might lead us to reopen consideration of the organization as a potential top charity” Yes We spoke with each standout charity. Conversation notes here: Development Media International, Food Fortification Initiative, Global Alliance for Improved Nutrition’s Universal Salt Iodizational program, Iodine Global Network, Living Goods, and Project Healthy Children.

The post Review of our research in 2017 appeared first on The GiveWell Blog.

In the fourth quarter of 2017, we received $5.6 million in funding for making grants at our discretion. In this post we discuss:

• The decision to allocate the $5.6 million to the Schistosomiasis Control Initiative (SCI).
• Our recommendation that donors give to GiveWell for granting to top charities at our discretion so that we can direct the funding to the top charity or charities with the most pressing funding need. For donors who prefer to give directly to our top charities, we continue to recommend giving 70 percent of your donation to AMF and 30 percent to SCI to maximize your impact.

We noted in November that we would use funds received for making grants at our discretion to fill the next highest priority funding gaps among our top charities. We also noted that our best guess at the time was that we would give 70 percent to the Against Malaria Foundation (AMF) and 30 percent to SCI.

Based on information received since November, described below, we allocated the $5.6 million to SCI, rather than dividing these funds between AMF and SCI, as previously expected. GiveWell’s Executive Director, Elie Hassenfeld, the fund advisor on the Effective Altruism Fund for Global Health and Development, also recommended that the fund grant out the $1.5 million that it held to SCI.

Update on AMF

AMF has been somewhat slower to make commitments to fund distributions of insecticide-treated nets than we expected and our best guess is that its currently available funding will be sufficient to fund all distributions that it is likely to commit to before our next major round of funding allocations in November. Notwithstanding that fact, we continue to believe that AMF has room for more funding. Additional funds would reduce the risk that AMF’s progress will be slowed if it is able to sign several major agreements in the next few months, which, while somewhat unlikely in our estimation, remains a possibility.

We wrote in November 2017:

Progress at signing new agreements was slow in 2017, leaving AMF with a large amount of funds on hand. We attribute this to the fact that countries spent much of 2017 applying for Global Fund funding and decisions about how much funding would be allocated to LLIN distributions for 2018-2020 and what the funding gaps would be for LLINs were being finalized in many countries as of October 2017. AMF noted that it did not commit to funding distributions earlier in part because GiveWell had asked AMF not to make funding commitments until the size of funding gaps were known.

Our expectation had been that the last couple months of 2017 and first months of 2018 would be a period in which AMF would commit a significant portion of its available funding to help fill these gaps because we expected countries to have more visibility into their funding gaps following finalization of Global Fund commitments around October 2017. This has not been the case. AMF recently told us that most of the countries that it was in discussions with did not have visibility into their funding gaps until December 2017, and in some cases it has taken longer than that. In making the decision regarding the fourth quarter discretionary funds, we relied on a document from AMF detailing its signed and potential agreements as of early February. The document noted that AMF had committed to one new distribution since October, in Ghana in 2018. This distribution will cost about $8 million. (We have since learned that AMF has also committed to additional distributions in Papua New Guinea in 2019 and 2020, costing $5.2 million and signed in November 2017, and in Malawi in 2018, costing $10.1 million and signed in mid February.)

AMF’s pipeline of potential future distributions includes both repeat distributions with partners and in countries it has worked with in the past and distributions with new potential partners. AMF has decided to move somewhat slowly with both types of partners. In the case of repeat partners, for several distributions, AMF is waiting to verify that the partner is able to deliver all requested data from distributions that took place in 2017 (and the monitoring that follows each distribution) before agreeing to fund the next round of nets to be delivered in 2020. These decisions seem very reasonable to us, but do result in a short-term decrease in the amount of funding we expect AMF to be able to absorb. When it is ready to do so, AMF could potentially commit up to $50 million to distributions in this category. For the largest potential new partnership that AMF is considering, there are some concerns about in-country capacity and AMF expects to to commit to a smaller-scale distribution (with an estimated cost of $5 million) with the partner and assess the results of that distribution before committing to a larger-scale distribution. AMF is also considering two additional opportunities to commit $5 to $7 million each to distributions with new partners. It could potentially commit tens of millions of dollars to one or more of these countries in future rounds if the initial engagements go well. AMF is also in several early stage conversations about potential distributions with new partners.

According to the document that we relied on for this decision, AMF held $64 million in uncommitted funds, of which $15 million was set aside for “agreement imminent” distributions, leaving $49 million “available to allocate.” Accounting for the additional agreements for Papua New Guinea and Malawi noted above, we estimate that AMF had $49 million in uncommitted funds and $45 million available to allocate as of late February.

The combination of somewhat slower progress in signing distributions than expected and our updated understanding of AMF’s pipeline led us to conclude that AMF continues to have room for more funding, but that SCI’s funding needs were more urgent. Our best guess was that the $5.6 million from GiveWell discretionary funds and $1.5 million from the Effective Altruism Fund would have a greater impact if allocated to SCI.

Update on SCI

In November, we recommended that donors give 30 percent to SCI because SCI had additional room for more funding to sustain its work in its current countries of operation and would need to scale down without additional funding. SCI recently confirmed to us that it would need to cut budgets if it did not receive additional funds before setting its annual budget for April 2018 to March 2019 in March 2018. With AMF having a less urgent funding need than previously expected, we concluded that the best use of the fourth quarter discretionary funds would be to allocate them to SCI.

It is also the case that in the last few months of 2017 SCI received less funding than we projected, both from donors influenced by GiveWell’s research and other donors.

We believe that SCI will continue to have room for more funding after the two grants totaling about $7 million. Recently, SCI sent us an early version of a budget for its 2018-19 budget year. It includes funding requests from each country program, estimates of country program requests in cases where the country has not yet submitted a request, and estimates of SCI spending on central costs and research costs. We estimate that, assuming the same budget in each of the next three years, SCI’s funding gap for that period, after receiving the grants discussed above, is about $9 million. SCI could likely absorb funding beyond that level, as the budget does not include opportunities it has to expand to additional countries. It also assumes that SCI’s other major funders will continue their support at the same level, and some of this funding may be in doubt. We note that about 13 percent of treatments that would be delivered at this scale would be for adults (discussion of this here).

Other possibilities that we decided against

Helen Keller International (HKI) for stopgap funding in one additional country

In December, Good Ventures, on GiveWell’s recommendation, provided HKI with funding for vitamin A supplementation (VAS) programs in Burkina Faso, Mali, and Guinea. Since then, HKI has learned about an unanticipated funding gap for VAS in another country. As a result, a planned VAS distribution in September may not reach national scale and/or may not include deworming (as is common for VAS campaigns). We are in ongoing conversations with HKI about either HKI allocating some of the Good Ventures funding to this country, or GiveWell providing additional funding to cover the gap. We plan to consider this funding opportunity when allocating discretionary funds from the first quarter of 2018. We expect to hold more than enough in discretionary funds (received in the first quarter of 2018) to fill the potential gap and HKI has told us that more information about the gap will be available in time for that decision. (We grant out funds from the previous quarter about two months after the end of that quarter, after we have fully checked the accuracy of our data and the size of grants).

Evidence Action’s Deworm the World for Nigeria

The grant that Good Ventures made to Evidence Action for Deworm the World in December 2017, based on our recommendation, did not include sufficient funds to fund expansion of Deworm the World’s work in Nigeria. Deworm the World sought funding for this work and we prioritized other charities’ funding gaps ahead of this work because we modeled the cost-effectiveness of this work as being lower. We noted in November, “its planned work in Nigeria is around three times as cost-effective as cash transfers (though this estimate is based on low-quality information).” We continue to think that AMF and SCI’s marginal uses of funding are likely more cost-effective than Deworm the World’s potential work in Nigeria, but this conclusion is highly dependent on a model that incorporates many highly uncertain values.

Malaria Consortium for seasonal malaria chemoprevention (SMC)

Our recommendation of Malaria Consortium has resulted in about $30 million in funding for its SMC program since November; however, we believe that there will still be a large funding gap for the program over the next three years. We decided against providing additional funding to Malaria Consortium at this time because of worries about increasing our already very large bet on a program that’s relatively new to us. We are not opposed to increasing this funding level in the future but on balance believe that granting additional funds to SCI is a stronger option at current levels. We’d also note that we’d expect additional funding at this time to go to funding SMC in 2019 and beyond (given the time needed to order drugs and plan programs for the 2018 SMC season) and there is some uncertainty as to the size of the funding gap for SMC in 2019. The program is in a scale-up phase globally and other major funders may increase their contributions to SMC starting in 2019.

What is our recommendation to donors?

We continue to recommend that donors give to GiveWell for granting to top charities at our discretion so that we can direct the funding to the top charity or charities with the most pressing funding need. For donors who prefer to give directly to our top charities, we are continuing to recommend giving 70 percent of your donation to AMF and 30 percent to SCI to maximize your impact.

As part of the process we went through to decide where to allocate these funds, we also discussed whether we should update our recommendation for donors who prefer to give directly to our top charities. We ultimately decided that because updating that recommended allocation is a difficult and time-consuming process because of the additional research and internal discussions involved and because, relatively speaking, few dollars follow this recommendation outside of giving season, we plan to update that allocation only once each year (in November) unless we believe our previously recommended allocation is clearly suboptimal.

In this case, we believe that the $7 million in grants to SCI roughly brings the situation back in line with where it was in November, with AMF and SCI having the next most impactful funding gaps and it being difficult to distinguish on the margin between the quality of AMF and SCI’s funding gaps. SCI has better modeled cost-effectiveness, while AMF appears to be better on several qualitative factors, including monitoring of program performance.

The post Allocation of discretionary funds from Q4 2017 appeared first on The GiveWell Blog.

In the fourth quarter of 2017, we received $5.6 million in funding for making grants at our discretion. In this post we discuss:

• The decision to allocate the $5.6 million to the Schistosomiasis Control Initiative (SCI).
• Our recommendation that donors give to GiveWell for granting to top charities at our discretion so that we can direct the funding to the top charity or charities with the most pressing funding need. For donors who prefer to give directly to our top charities, we continue to recommend giving 70 percent of your donation to AMF and 30 percent to SCI to maximize your impact.

We noted in November that we would use funds received for making grants at our discretion to fill the next highest priority funding gaps among our top charities. We also noted that our best guess at the time was that we would give 70 percent to the Against Malaria Foundation (AMF) and 30 percent to SCI.

Based on information received since November, described below, we allocated the $5.6 million to SCI, rather than dividing these funds between AMF and SCI, as previously expected. GiveWell’s Executive Director, Elie Hassenfeld, the fund advisor on the Effective Altruism Fund for Global Health and Development, also recommended that the fund grant out the $1.5 million that it held to SCI.

Update on AMF

AMF has been somewhat slower to make commitments to fund distributions of insecticide-treated nets than we expected and our best guess is that its currently available funding will be sufficient to fund all distributions that it is likely to commit to before our next major round of funding allocations in November. Notwithstanding that fact, we continue to believe that AMF has room for more funding. Additional funds would reduce the risk that AMF’s progress will be slowed if it is able to sign several major agreements in the next few months, which, while somewhat unlikely in our estimation, remains a possibility.

We wrote in November 2017:

Progress at signing new agreements was slow in 2017, leaving AMF with a large amount of funds on hand. We attribute this to the fact that countries spent much of 2017 applying for Global Fund funding and decisions about how much funding would be allocated to LLIN distributions for 2018-2020 and what the funding gaps would be for LLINs were being finalized in many countries as of October 2017. AMF noted that it did not commit to funding distributions earlier in part because GiveWell had asked AMF not to make funding commitments until the size of funding gaps were known.

Our expectation had been that the last couple months of 2017 and first months of 2018 would be a period in which AMF would commit a significant portion of its available funding to help fill these gaps because we expected countries to have more visibility into their funding gaps following finalization of Global Fund commitments around October 2017. This has not been the case. AMF recently told us that most of the countries that it was in discussions with did not have visibility into their funding gaps until December 2017, and in some cases it has taken longer than that. In making the decision regarding the fourth quarter discretionary funds, we relied on a document from AMF detailing its signed and potential agreements as of early February. The document noted that AMF had committed to one new distribution since October, in Ghana in 2018. This distribution will cost about $8 million. (We have since learned that AMF has also committed to additional distributions in Papua New Guinea in 2019 and 2020, costing $5.2 million and signed in November 2017, and in Malawi in 2018, costing $10.1 million and signed in mid February.)

AMF’s pipeline of potential future distributions includes both repeat distributions with partners and in countries it has worked with in the past and distributions with new potential partners. AMF has decided to move somewhat slowly with both types of partners. In the case of repeat partners, for several distributions, AMF is waiting to verify that the partner is able to deliver all requested data from distributions that took place in 2017 (and the monitoring that follows each distribution) before agreeing to fund the next round of nets to be delivered in 2020. These decisions seem very reasonable to us, but do result in a short-term decrease in the amount of funding we expect AMF to be able to absorb. When it is ready to do so, AMF could potentially commit up to $50 million to distributions in this category. For the largest potential new partnership that AMF is considering, there are some concerns about in-country capacity and AMF expects to to commit to a smaller-scale distribution (with an estimated cost of $5 million) with the partner and assess the results of that distribution before committing to a larger-scale distribution. AMF is also considering two additional opportunities to commit $5 to $7 million each to distributions with new partners. It could potentially commit tens of millions of dollars to one or more of these countries in future rounds if the initial engagements go well. AMF is also in several early stage conversations about potential distributions with new partners.

According to the document that we relied on for this decision, AMF held $64 million in uncommitted funds, of which $15 million was set aside for “agreement imminent” distributions, leaving $49 million “available to allocate.” Accounting for the additional agreements for Papua New Guinea and Malawi noted above, we estimate that AMF had $49 million in uncommitted funds and $45 million available to allocate as of late February.

The combination of somewhat slower progress in signing distributions than expected and our updated understanding of AMF’s pipeline led us to conclude that AMF continues to have room for more funding, but that SCI’s funding needs were more urgent. Our best guess was that the $5.6 million from GiveWell discretionary funds and $1.5 million from the Effective Altruism Fund would have a greater impact if allocated to SCI.

Update on SCI

In November, we recommended that donors give 30 percent to SCI because SCI had additional room for more funding to sustain its work in its current countries of operation and would need to scale down without additional funding. SCI recently confirmed to us that it would need to cut budgets if it did not receive additional funds before setting its annual budget for April 2018 to March 2019 in March 2018. With AMF having a less urgent funding need than previously expected, we concluded that the best use of the fourth quarter discretionary funds would be to allocate them to SCI.

It is also the case that in the last few months of 2017 SCI received less funding than we projected, both from donors influenced by GiveWell’s research and other donors.

We believe that SCI will continue to have room for more funding after the two grants totaling about $7 million. Recently, SCI sent us an early version of a budget for its 2018-19 budget year. It includes funding requests from each country program, estimates of country program requests in cases where the country has not yet submitted a request, and estimates of SCI spending on central costs and research costs. We estimate that, assuming the same budget in each of the next three years, SCI’s funding gap for that period, after receiving the grants discussed above, is about $9 million. SCI could likely absorb funding beyond that level, as the budget does not include opportunities it has to expand to additional countries. It also assumes that SCI’s other major funders will continue their support at the same level, and some of this funding may be in doubt. We note that about 13 percent of treatments that would be delivered at this scale would be for adults (discussion of this here).

Other possibilities that we decided against

Helen Keller International (HKI) for stopgap funding in one additional country

In December, Good Ventures, on GiveWell’s recommendation, provided HKI with funding for vitamin A supplementation (VAS) programs in Burkina Faso, Mali, and Guinea. Since then, HKI has learned about an unanticipated funding gap for VAS in another country. As a result, a planned VAS distribution in September may not reach national scale and/or may not include deworming (as is common for VAS campaigns). We are in ongoing conversations with HKI about either HKI allocating some of the Good Ventures funding to this country, or GiveWell providing additional funding to cover the gap. We plan to consider this funding opportunity when allocating discretionary funds from the first quarter of 2018. We expect to hold more than enough in discretionary funds (received in the first quarter of 2018) to fill the potential gap and HKI has told us that more information about the gap will be available in time for that decision. (We grant out funds from the previous quarter about two months after the end of that quarter, after we have fully checked the accuracy of our data and the size of grants).

Evidence Action’s Deworm the World for Nigeria

The grant that Good Ventures made to Evidence Action for Deworm the World in December 2017, based on our recommendation, did not include sufficient funds to fund expansion of Deworm the World’s work in Nigeria. Deworm the World sought funding for this work and we prioritized other charities’ funding gaps ahead of this work because we modeled the cost-effectiveness of this work as being lower. We noted in November, “its planned work in Nigeria is around three times as cost-effective as cash transfers (though this estimate is based on low-quality information).” We continue to think that AMF and SCI’s marginal uses of funding are likely more cost-effective than Deworm the World’s potential work in Nigeria, but this conclusion is highly dependent on a model that incorporates many highly uncertain values.

Malaria Consortium for seasonal malaria chemoprevention (SMC)

Our recommendation of Malaria Consortium has resulted in about $30 million in funding for its SMC program since November; however, we believe that there will still be a large funding gap for the program over the next three years. We decided against providing additional funding to Malaria Consortium at this time because of worries about increasing our already very large bet on a program that’s relatively new to us. We are not opposed to increasing this funding level in the future but on balance believe that granting additional funds to SCI is a stronger option at current levels. We’d also note that we’d expect additional funding at this time to go to funding SMC in 2019 and beyond (given the time needed to order drugs and plan programs for the 2018 SMC season) and there is some uncertainty as to the size of the funding gap for SMC in 2019. The program is in a scale-up phase globally and other major funders may increase their contributions to SMC starting in 2019.

What is our recommendation to donors?

We continue to recommend that donors give to GiveWell for granting to top charities at our discretion so that we can direct the funding to the top charity or charities with the most pressing funding need. For donors who prefer to give directly to our top charities, we are continuing to recommend giving 70 percent of your donation to AMF and 30 percent to SCI to maximize your impact.

As part of the process we went through to decide where to allocate these funds, we also discussed whether we should update our recommendation for donors who prefer to give directly to our top charities. We ultimately decided that because updating that recommended allocation is a difficult and time-consuming process because of the additional research and internal discussions involved and because, relatively speaking, few dollars follow this recommendation outside of giving season, we plan to update that allocation only once each year (in November) unless we believe our previously recommended allocation is clearly suboptimal.

In this case, we believe that the $7 million in grants to SCI roughly brings the situation back in line with where it was in November, with AMF and SCI having the next most impactful funding gaps and it being difficult to distinguish on the margin between the quality of AMF and SCI’s funding gaps. SCI has better modeled cost-effectiveness, while AMF appears to be better on several qualitative factors, including monitoring of program performance.

The post Allocation of discretionary funds from Q4 2017 appeared first on The GiveWell Blog.

In the fourth quarter of 2017, we received $5.6 million in funding for making grants at our discretion. In this post we discuss:

• The decision to allocate the $5.6 million to the Schistosomiasis Control Initiative (SCI).
• Our recommendation that donors give to GiveWell for granting to top charities at our discretion so that we can direct the funding to the top charity or charities with the most pressing funding need. For donors who prefer to give directly to our top charities, we continue to recommend giving 70 percent of your donation to AMF and 30 percent to SCI to maximize your impact.

We noted in November that we would use funds received for making grants at our discretion to fill the next highest priority funding gaps among our top charities. We also noted that our best guess at the time was that we would give 70 percent to the Against Malaria Foundation (AMF) and 30 percent to SCI.

Based on information received since November, described below, we allocated the $5.6 million to SCI, rather than dividing these funds between AMF and SCI, as previously expected. GiveWell's Executive Director, Elie Hassenfeld, the fund advisor on the Effective Altruism Fund for Global Health and Development also recommended that the fund grant out the $1.5 million that it held to SCI.

Read More

The post Allocation of discretionary funds from Q4 2017 appeared first on The GiveWell Blog.

In the fourth quarter of 2017, we received $5.6 million in funding for making grants at our discretion. In this post we discuss:

• The decision to allocate the $5.6 million to the Schistosomiasis Control Initiative (SCI).
• Our recommendation that donors give to GiveWell for granting to top charities at our discretion so that we can direct the funding to the top charity or charities with the most pressing funding need. For donors who prefer to give directly to our top charities, we continue to recommend giving 70 percent of your donation to AMF and 30 percent to SCI to maximize your impact.

We noted in November that we would use funds received for making grants at our discretion to fill the next highest priority funding gaps among our top charities. We also noted that our best guess at the time was that we would give 70 percent to the Against Malaria Foundation (AMF) and 30 percent to SCI.

Based on information received since November, described below, we allocated the $5.6 million to SCI, rather than dividing these funds between AMF and SCI, as previously expected. GiveWell’s Executive Director, Elie Hassenfeld, the fund advisor on the Effective Altruism Fund for Global Health and Development, also recommended that the fund grant out the $1.5 million that it held to SCI.

Update on AMF

AMF has been somewhat slower to make commitments to fund distributions of insecticide-treated nets than we expected and our best guess is that its currently available funding will be sufficient to fund all distributions that it is likely to commit to before our next major round of funding allocations in November. Notwithstanding that fact, we continue to believe that AMF has room for more funding. Additional funds would reduce the risk that AMF’s progress will be slowed if it is able to sign several major agreements in the next few months, which, while somewhat unlikely in our estimation, remains a possibility.

We wrote in November 2017:

Progress at signing new agreements was slow in 2017, leaving AMF with a large amount of funds on hand. We attribute this to the fact that countries spent much of 2017 applying for Global Fund funding and decisions about how much funding would be allocated to LLIN distributions for 2018-2020 and what the funding gaps would be for LLINs were being finalized in many countries as of October 2017. AMF noted that it did not commit to funding distributions earlier in part because GiveWell had asked AMF not to make funding commitments until the size of funding gaps were known.

Our expectation had been that the last couple months of 2017 and first months of 2018 would be a period in which AMF would commit a significant portion of its available funding to help fill these gaps because we expected countries to have more visibility into their funding gaps following finalization of Global Fund commitments around October 2017. This has not been the case. AMF recently told us that most of the countries that it was in discussions with did not have visibility into their funding gaps until December 2017, and in some cases it has taken longer than that. In making the decision regarding the fourth quarter discretionary funds, we relied on a document from AMF detailing its signed and potential agreements as of early February. The document noted that AMF had committed to one new distribution since October, in Ghana in 2018. This distribution will cost about $8 million. (We have since learned that AMF has also committed to additional distributions in Papua New Guinea in 2019 and 2020, costing $5.2 million and signed in November 2017, and in Malawi in 2018, costing $10.1 million and signed in mid February.)

AMF’s pipeline of potential future distributions includes both repeat distributions with partners and in countries it has worked with in the past and distributions with new potential partners. AMF has decided to move somewhat slowly with both types of partners. In the case of repeat partners, for several distributions, AMF is waiting to verify that the partner is able to deliver all requested data from distributions that took place in 2017 (and the monitoring that follows each distribution) before agreeing to fund the next round of nets to be delivered in 2020. These decisions seem very reasonable to us, but do result in a short-term decrease in the amount of funding we expect AMF to be able to absorb. When it is ready to do so, AMF could potentially commit up to $50 million to distributions in this category. For the largest potential new partnership that AMF is considering, there are some concerns about in-country capacity and AMF expects to to commit to a smaller-scale distribution (with an estimated cost of $5 million) with the partner and assess the results of that distribution before committing to a larger-scale distribution. AMF is also considering two additional opportunities to commit $5 to $7 million each to distributions with new partners. It could potentially commit tens of millions of dollars to one or more of these countries in future rounds if the initial engagements go well. AMF is also in several early stage conversations about potential distributions with new partners.

According to the document that we relied on for this decision, AMF held $64 million in uncommitted funds, of which $15 million was set aside for “agreement imminent” distributions, leaving $49 million “available to allocate.” Accounting for the additional agreements for Papua New Guinea and Malawi noted above, we estimate that AMF had $49 million in uncommitted funds and $45 million available to allocate as of late February.

The combination of somewhat slower progress in signing distributions than expected and our updated understanding of AMF’s pipeline led us to conclude that AMF continues to have room for more funding, but that SCI’s funding needs were more urgent. Our best guess was that the $5.6 million from GiveWell discretionary funds and $1.5 million from the Effective Altruism Fund would have a greater impact if allocated to SCI.

Update on SCI

In November, we recommended that donors give 30 percent to SCI because SCI had additional room for more funding to sustain its work in its current countries of operation and would need to scale down without additional funding. SCI recently confirmed to us that it would need to cut budgets if it did not receive additional funds before setting its annual budget for April 2018 to March 2019 in March 2018. With AMF having a less urgent funding need than previously expected, we concluded that the best use of the fourth quarter discretionary funds would be to allocate them to SCI.

It is also the case that in the last few months of 2017 SCI received less funding than we projected, both from donors influenced by GiveWell’s research and other donors.

We believe that SCI will continue to have room for more funding after the two grants totaling about $7 million. Recently, SCI sent us an early version of a budget for its 2018-19 budget year. It includes funding requests from each country program, estimates of country program requests in cases where the country has not yet submitted a request, and estimates of SCI spending on central costs and research costs. We estimate that, assuming the same budget in each of the next three years, SCI’s funding gap for that period, after receiving the grants discussed above, is about $9 million. SCI could likely absorb funding beyond that level, as the budget does not include opportunities it has to expand to additional countries. It also assumes that SCI’s other major funders will continue their support at the same level, and some of this funding may be in doubt. We note that about 13 percent of treatments that would be delivered at this scale would be for adults (discussion of this here).

Other possibilities that we decided against

Helen Keller International (HKI) for stopgap funding in one additional country

In December, Good Ventures, on GiveWell’s recommendation, provided HKI with funding for vitamin A supplementation (VAS) programs in Burkina Faso, Mali, and Guinea. Since then, HKI has learned about an unanticipated funding gap for VAS in another country. As a result, a planned VAS distribution in September may not reach national scale and/or may not include deworming (as is common for VAS campaigns). We are in ongoing conversations with HKI about either HKI allocating some of the Good Ventures funding to this country, or GiveWell providing additional funding to cover the gap. We plan to consider this funding opportunity when allocating discretionary funds from the first quarter of 2018. We expect to hold more than enough in discretionary funds (received in the first quarter of 2018) to fill the potential gap and HKI has told us that more information about the gap will be available in time for that decision. (We grant out funds from the previous quarter about two months after the end of that quarter, after we have fully checked the accuracy of our data and the size of grants).

Evidence Action’s Deworm the World for Nigeria

The grant that Good Ventures made to Evidence Action for Deworm the World in December 2017, based on our recommendation, did not include sufficient funds to fund expansion of Deworm the World’s work in Nigeria. Deworm the World sought funding for this work and we prioritized other charities’ funding gaps ahead of this work because we modeled the cost-effectiveness of this work as being lower. We noted in November, “its planned work in Nigeria is around three times as cost-effective as cash transfers (though this estimate is based on low-quality information).” We continue to think that AMF and SCI’s marginal uses of funding are likely more cost-effective than Deworm the World’s potential work in Nigeria, but this conclusion is highly dependent on a model that incorporates many highly uncertain values.

Malaria Consortium for seasonal malaria chemoprevention (SMC)

Our recommendation of Malaria Consortium has resulted in about $30 million in funding for its SMC program since November; however, we believe that there will still be a large funding gap for the program over the next three years. We decided against providing additional funding to Malaria Consortium at this time because of worries about increasing our already very large bet on a program that’s relatively new to us. We are not opposed to increasing this funding level in the future but on balance believe that granting additional funds to SCI is a stronger option at current levels. We’d also note that we’d expect additional funding at this time to go to funding SMC in 2019 and beyond (given the time needed to order drugs and plan programs for the 2018 SMC season) and there is some uncertainty as to the size of the funding gap for SMC in 2019. The program is in a scale-up phase globally and other major funders may increase their contributions to SMC starting in 2019.

What is our recommendation to donors?

We continue to recommend that donors give to GiveWell for granting to top charities at our discretion so that we can direct the funding to the top charity or charities with the most pressing funding need. For donors who prefer to give directly to our top charities, we are continuing to recommend giving 70 percent of your donation to AMF and 30 percent to SCI to maximize your impact.

As part of the process we went through to decide where to allocate these funds, we also discussed whether we should update our recommendation for donors who prefer to give directly to our top charities. We ultimately decided that because updating that recommended allocation is a difficult and time-consuming process because of the additional research and internal discussions involved and because, relatively speaking, few dollars follow this recommendation outside of giving season, we plan to update that allocation only once each year (in November) unless we believe our previously recommended allocation is clearly suboptimal.

In this case, we believe that the $7 million in grants to SCI roughly brings the situation back in line with where it was in November, with AMF and SCI having the next most impactful funding gaps and it being difficult to distinguish on the margin between the quality of AMF and SCI’s funding gaps. SCI has better modeled cost-effectiveness, while AMF appears to be better on several qualitative factors, including monitoring of program performance.

The post Allocation of discretionary funds from Q4 2017 appeared first on The GiveWell Blog.

In the fourth quarter of 2017, we received $5.6 million in funding for making grants at our discretion. In this post we discuss:

• The decision to allocate the $5.6 million to the Schistosomiasis Control Initiative (SCI).
• Our recommendation that donors give to GiveWell for granting to top charities at our discretion so that we can direct the funding to the top charity or charities with the most pressing funding need. For donors who prefer to give directly to our top charities, we continue to recommend giving 70 percent of your donation to AMF and 30 percent to SCI to maximize your impact.

We noted in November that we would use funds received for making grants at our discretion to fill the next highest priority funding gaps among our top charities. We also noted that our best guess at the time was that we would give 70 percent to the Against Malaria Foundation (AMF) and 30 percent to SCI.

Based on information received since November, described below, we allocated the $5.6 million to SCI, rather than dividing these funds between AMF and SCI, as previously expected. GiveWell’s Executive Director, Elie Hassenfeld, the fund advisor on the Effective Altruism Fund for Global Health and Development, also recommended that the fund grant out the $1.5 million that it held to SCI.

Update on AMF

AMF has been somewhat slower to make commitments to fund distributions of insecticide-treated nets than we expected and our best guess is that its currently available funding will be sufficient to fund all distributions that it is likely to commit to before our next major round of funding allocations in November. Notwithstanding that fact, we continue to believe that AMF has room for more funding. Additional funds would reduce the risk that AMF’s progress will be slowed if it is able to sign several major agreements in the next few months, which, while somewhat unlikely in our estimation, remains a possibility.

We wrote in November 2017:

Progress at signing new agreements was slow in 2017, leaving AMF with a large amount of funds on hand. We attribute this to the fact that countries spent much of 2017 applying for Global Fund funding and decisions about how much funding would be allocated to LLIN distributions for 2018-2020 and what the funding gaps would be for LLINs were being finalized in many countries as of October 2017. AMF noted that it did not commit to funding distributions earlier in part because GiveWell had asked AMF not to make funding commitments until the size of funding gaps were known.

Our expectation had been that the last couple months of 2017 and first months of 2018 would be a period in which AMF would commit a significant portion of its available funding to help fill these gaps because we expected countries to have more visibility into their funding gaps following finalization of Global Fund commitments around October 2017. This has not been the case. AMF recently told us that most of the countries that it was in discussions with did not have visibility into their funding gaps until December 2017, and in some cases it has taken longer than that. In making the decision regarding the fourth quarter discretionary funds, we relied on a document from AMF detailing its signed and potential agreements as of early February. The document noted that AMF had committed to one new distribution since October, in Ghana in 2018. This distribution will cost about $8 million. (We have since learned that AMF has also committed to additional distributions in Papua New Guinea in 2019 and 2020, costing $5.2 million and signed in November 2017, and in Malawi in 2018, costing $10.1 million and signed in mid February.)

AMF’s pipeline of potential future distributions includes both repeat distributions with partners and in countries it has worked with in the past and distributions with new potential partners. AMF has decided to move somewhat slowly with both types of partners. In the case of repeat partners, for several distributions, AMF is waiting to verify that the partner is able to deliver all requested data from distributions that took place in 2017 (and the monitoring that follows each distribution) before agreeing to fund the next round of nets to be delivered in 2020. These decisions seem very reasonable to us, but do result in a short-term decrease in the amount of funding we expect AMF to be able to absorb. When it is ready to do so, AMF could potentially commit up to $50 million to distributions in this category. For the largest potential new partnership that AMF is considering, there are some concerns about in-country capacity and AMF expects to to commit to a smaller-scale distribution (with an estimated cost of $5 million) with the partner and assess the results of that distribution before committing to a larger-scale distribution. AMF is also considering two additional opportunities to commit $5 to $7 million each to distributions with new partners. It could potentially commit tens of millions of dollars to one or more of these countries in future rounds if the initial engagements go well. AMF is also in several early stage conversations about potential distributions with new partners.

According to the document that we relied on for this decision, AMF held $64 million in uncommitted funds, of which $15 million was set aside for “agreement imminent” distributions, leaving $49 million “available to allocate.” Accounting for the additional agreements for Papua New Guinea and Malawi noted above, we estimate that AMF had $49 million in uncommitted funds and $45 million available to allocate as of late February.

The combination of somewhat slower progress in signing distributions than expected and our updated understanding of AMF’s pipeline led us to conclude that AMF continues to have room for more funding, but that SCI’s funding needs were more urgent. Our best guess was that the $5.6 million from GiveWell discretionary funds and $1.5 million from the Effective Altruism Fund would have a greater impact if allocated to SCI.

Update on SCI

In November, we recommended that donors give 30 percent to SCI because SCI had additional room for more funding to sustain its work in its current countries of operation and would need to scale down without additional funding. SCI recently confirmed to us that it would need to cut budgets if it did not receive additional funds before setting its annual budget for April 2018 to March 2019 in March 2018. With AMF having a less urgent funding need than previously expected, we concluded that the best use of the fourth quarter discretionary funds would be to allocate them to SCI.

It is also the case that in the last few months of 2017 SCI received less funding than we projected, both from donors influenced by GiveWell’s research and other donors.

We believe that SCI will continue to have room for more funding after the two grants totaling about $7 million. Recently, SCI sent us an early version of a budget for its 2018-19 budget year. It includes funding requests from each country program, estimates of country program requests in cases where the country has not yet submitted a request, and estimates of SCI spending on central costs and research costs. We estimate that, assuming the same budget in each of the next three years, SCI’s funding gap for that period, after receiving the grants discussed above, is about $9 million. SCI could likely absorb funding beyond that level, as the budget does not include opportunities it has to expand to additional countries. It also assumes that SCI’s other major funders will continue their support at the same level, and some of this funding may be in doubt. We note that about 13 percent of treatments that would be delivered at this scale would be for adults (discussion of this here).

Other possibilities that we decided against

Helen Keller International (HKI) for stopgap funding in one additional country

In December, Good Ventures, on GiveWell’s recommendation, provided HKI with funding for vitamin A supplementation (VAS) programs in Burkina Faso, Mali, and Guinea. Since then, HKI has learned about an unanticipated funding gap for VAS in another country. As a result, a planned VAS distribution in September may not reach national scale and/or may not include deworming (as is common for VAS campaigns). We are in ongoing conversations with HKI about either HKI allocating some of the Good Ventures funding to this country, or GiveWell providing additional funding to cover the gap. We plan to consider this funding opportunity when allocating discretionary funds from the first quarter of 2018. We expect to hold more than enough in discretionary funds (received in the first quarter of 2018) to fill the potential gap and HKI has told us that more information about the gap will be available in time for that decision. (We grant out funds from the previous quarter about two months after the end of that quarter, after we have fully checked the accuracy of our data and the size of grants).

Evidence Action’s Deworm the World for Nigeria

The grant that Good Ventures made to Evidence Action for Deworm the World in December 2017, based on our recommendation, did not include sufficient funds to fund expansion of Deworm the World’s work in Nigeria. Deworm the World sought funding for this work and we prioritized other charities’ funding gaps ahead of this work because we modeled the cost-effectiveness of this work as being lower. We noted in November, “its planned work in Nigeria is around three times as cost-effective as cash transfers (though this estimate is based on low-quality information).” We continue to think that AMF and SCI’s marginal uses of funding are likely more cost-effective than Deworm the World’s potential work in Nigeria, but this conclusion is highly dependent on a model that incorporates many highly uncertain values.

Malaria Consortium for seasonal malaria chemoprevention (SMC)

Our recommendation of Malaria Consortium has resulted in about $30 million in funding for its SMC program since November; however, we believe that there will still be a large funding gap for the program over the next three years. We decided against providing additional funding to Malaria Consortium at this time because of worries about increasing our already very large bet on a program that’s relatively new to us. We are not opposed to increasing this funding level in the future but on balance believe that granting additional funds to SCI is a stronger option at current levels. We’d also note that we’d expect additional funding at this time to go to funding SMC in 2019 and beyond (given the time needed to order drugs and plan programs for the 2018 SMC season) and there is some uncertainty as to the size of the funding gap for SMC in 2019. The program is in a scale-up phase globally and other major funders may increase their contributions to SMC starting in 2019.

What is our recommendation to donors?

We continue to recommend that donors give to GiveWell for granting to top charities at our discretion so that we can direct the funding to the top charity or charities with the most pressing funding need. For donors who prefer to give directly to our top charities, we are continuing to recommend giving 70 percent of your donation to AMF and 30 percent to SCI to maximize your impact.

As part of the process we went through to decide where to allocate these funds, we also discussed whether we should update our recommendation for donors who prefer to give directly to our top charities. We ultimately decided that because updating that recommended allocation is a difficult and time-consuming process because of the additional research and internal discussions involved and because, relatively speaking, few dollars follow this recommendation outside of giving season, we plan to update that allocation only once each year (in November) unless we believe our previously recommended allocation is clearly suboptimal.

In this case, we believe that the $7 million in grants to SCI roughly brings the situation back in line with where it was in November, with AMF and SCI having the next most impactful funding gaps and it being difficult to distinguish on the margin between the quality of AMF and SCI’s funding gaps. SCI has better modeled cost-effectiveness, while AMF appears to be better on several qualitative factors, including monitoring of program performance.

The post Allocation of discretionary funds from Q4 2017 appeared first on The GiveWell Blog.

In September 2017, we posted an interim update on GiveWell’s 2016 money moved and web traffic. This post summarizes the key takeaways from our full 2016 money moved and web traffic metrics report. Note that some of the numbers, including the total headline money moved, have changed since our interim report. Since then, we decided to exclude some donations from our headline money moved figure (details in the full report), and we corrected some minor errors.

This report was highly delayed (as discussed in the interim update). We expect to publish our report on GiveWell’s 2017 money moved and web traffic much more quickly; our current expectation is that we will publish that report by the end of June.

GiveWell is dedicated to finding outstanding giving opportunities and publishing the full details of our analysis. In addition to evaluations of other charities, we publish substantial evaluation of our own work. This post lays out highlights from our 2016 metrics report, which reviews what we know about how our research impacted donors. Please note:

• We report on “metrics years” that run from February through January; for example, our 2016 data cover February 1, 2016 through January 31, 2017.
• We differentiate between our traditional charity recommendations and our work on the Open Philanthropy Project, which became a separate organization in 2017 and whose work we exclude from this report.
• More context on the relationship between Good Ventures and GiveWell can be found here.

Summary of influence: In 2016, GiveWell influenced charitable giving in several ways. The following table summarizes our understanding of this influence.

Headline money moved: In 2016, we tracked $88.6 million in money moved to our recommended charities. Our money moved only includes donations that we are confident were influenced by our recommendations.

Money moved by charity: Our seven top charities received the majority of our money moved. Our six standout charities received a total of $2.9 million.

Money moved by size of donor: In 2016, the number of donors and amount donated increased across each donor size category, with the notable exception of donations from donors giving $1,000,000 or more. In 2016, 93% of our money moved (excluding Good Ventures) came from 19% of our donors, who gave $1,000 or more.

Donor retention: The total number of donors who gave to our recommended charities or to GiveWell unrestricted increased about 16% year-over-year to 17,834 in 2016. This included 12,461 donors who gave for the first time. Among all donors who gave in the previous year, about 35% gave again in 2016, down from about 40% who gave again in 2015.

Our retention was stronger among donors who gave larger amounts or who first gave to our recommendations prior to 2014. Of larger donors (those who gave $10,000 or more in either of the last two years), about 77% who gave in 2015 gave again in 2016.

GiveWell’s expenses: GiveWell’s total operating expenses in 2016 were $5.5 million. Our expenses increased from about $3.4 million in 2015 as the size of our staff grew and average seniority level rose. We estimate that about one-third of our total expenses ($2.0 million) supported our traditional top charity work and about two-thirds supported the Open Philanthropy Project. In 2015, we estimated that expenses for our traditional charity work were about $1.1 million.

Donations supporting GiveWell’s operations: GiveWell raised $5.6 million in unrestricted funding (which we use to support our operations) in 2016, compared to $5.1 million in 2015. Our major institutional supporters and the five largest individual donors contributed about 70% of GiveWell’s operational funding in 2016. This is driven in large part by the fact that Good Ventures funded two-thirds of the costs of the Open Philanthropy project, in addition to funding 20% of GiveWell’s other costs.

Web traffic: The number of unique visitors to our website was down very slightly (by 1%) in 2016 compared to 2015 (when excluding visitors driven by AdWords, Google’s online advertising product).

For more detail, see our full metrics report (PDF).

The post GiveWell’s money moved and web traffic in 2016 appeared first on The GiveWell Blog.

In September 2017, we posted an interim update on GiveWell’s 2016 money moved and web traffic. This post summarizes the key takeaways from our full 2016 money moved and web traffic metrics report. Note that some of the numbers, including the total headline money moved, have changed since our interim report. Since then, we decided to exclude some donations from our headline money moved figure (details in the full report), and we corrected some minor errors.

This report was highly delayed (as discussed in the interim update). We expect to publish our report on GiveWell’s 2017 money moved and web traffic much more quickly; our current expectation is that we will publish that report by the end of June.

GiveWell is dedicated to finding outstanding giving opportunities and publishing the full details of our analysis. In addition to evaluations of other charities, we publish substantial evaluation of our own work. This post lays out highlights from our 2016 metrics report, which reviews what we know about how our research impacted donors. Please note:

• We report on “metrics years” that run from February through January; for example, our 2016 data cover February 1, 2016 through January 31, 2017.
• We differentiate between our traditional charity recommendations and our work on the Open Philanthropy Project, which became a separate organization in 2017 and whose work we exclude from this report.
• More context on the relationship between Good Ventures and GiveWell can be found here.

Summary of influence: In 2016, GiveWell influenced charitable giving in several ways. The following table summarizes our understanding of this influence.

Headline money moved: In 2016, we tracked $88.6 million in money moved to our recommended charities. Our money moved only includes donations that we are confident were influenced by our recommendations.

Money moved by charity: Our seven top charities received the majority of our money moved. Our six standout charities received a total of $2.9 million.

Money moved by size of donor: In 2016, the number of donors and amount donated increased across each donor size category, with the notable exception of donations from donors giving $1,000,000 or more. In 2016, 93% of our money moved (excluding Good Ventures) came from 19% of our donors, who gave $1,000 or more.

Donor retention: The total number of donors who gave to our recommended charities or to GiveWell unrestricted increased about 16% year-over-year to 17,834 in 2016. This included 12,461 donors who gave for the first time. Among all donors who gave in the previous year, about 35% gave again in 2016, down from about 40% who gave again in 2015.

Our retention was stronger among donors who gave larger amounts or who first gave to our recommendations prior to 2014. Of larger donors (those who gave $10,000 or more in either of the last two years), about 77% who gave in 2015 gave again in 2016.

GiveWell’s expenses: GiveWell’s total operating expenses in 2016 were $5.5 million. Our expenses increased from about $3.4 million in 2015 as the size of our staff grew and average seniority level rose. We estimate that about one-third of our total expenses ($2.0 million) supported our traditional top charity work and about two-thirds supported the Open Philanthropy Project. In 2015, we estimated that expenses for our traditional charity work were about $1.1 million.

Donations supporting GiveWell’s operations: GiveWell raised $5.6 million in unrestricted funding (which we use to support our operations) in 2016, compared to $5.1 million in 2015. Our major institutional supporters and the five largest individual donors contributed about 70% of GiveWell’s operational funding in 2016. This is driven in large part by the fact that Good Ventures funded two-thirds of the costs of the Open Philanthropy project, in addition to funding 20% of GiveWell’s other costs.

Web traffic: The number of unique visitors to our website was down very slightly (by 1%) in 2016 compared to 2015 (when excluding visitors driven by AdWords, Google’s online advertising product).

For more detail, see our full metrics report (PDF).

The post GiveWell’s money moved and web traffic in 2016 appeared first on The GiveWell Blog.

In September 2017, we posted an interim update on GiveWell’s 2016 money moved and web traffic. This post summarizes the key takeaways from our full 2016 money moved and web traffic metrics report. Note that some of the numbers, including the total headline money moved, have changed since our interim report. Since then, we decided to exclude some donations from our headline money moved figure (details in the full report), and we corrected some minor errors.

This report was highly delayed (as discussed in the interim update). We expect to publish our report on GiveWell’s 2017 money moved and web traffic much more quickly; our current expectation is that we will publish that report by the end of June.

GiveWell is dedicated to finding outstanding giving opportunities and publishing the full details of our analysis. In addition to evaluations of other charities, we publish substantial evaluation of our own work. This post lays out highlights from our 2016 metrics report, which reviews what we know about how our research impacted donors. Please note:

• We report on “metrics years” that run from February through January; for example, our 2016 data cover February 1, 2016 through January 31, 2017.
• We differentiate between our traditional charity recommendations and our work on the Open Philanthropy Project, which became a separate organization in 2017 and whose work we exclude from this report.
• More context on the relationship between Good Ventures and GiveWell can be found here.

Summary of influence: In 2016, GiveWell influenced charitable giving in several ways. The following table summarizes our understanding of this influence.

Headline money moved: In 2016, we tracked $88.6 million in money moved to our recommended charities. Our money moved only includes donations that we are confident were influenced by our recommendations.

Money moved by charity: Our seven top charities received the majority of our money moved. Our six standout charities received a total of $2.9 million.

Money moved by size of donor: In 2016, the number of donors and amount donated increased across each donor size category, with the notable exception of donations from donors giving $1,000,000 or more. In 2016, 93% of our money moved (excluding Good Ventures) came from 19% of our donors, who gave $1,000 or more.

Donor retention: The total number of donors who gave to our recommended charities or to GiveWell unrestricted increased about 16% year-over-year to 17,834 in 2016. This included 12,461 donors who gave for the first time. Among all donors who gave in the previous year, about 35% gave again in 2016, down from about 40% who gave again in 2015.

Our retention was stronger among donors who gave larger amounts or who first gave to our recommendations prior to 2014. Of larger donors (those who gave $10,000 or more in either of the last two years), about 77% who gave in 2015 gave again in 2016.

GiveWell’s expenses: GiveWell’s total operating expenses in 2016 were $5.5 million. Our expenses increased from about $3.4 million in 2015 as the size of our staff grew and average seniority level rose. We estimate that about one-third of our total expenses ($2.0 million) supported our traditional top charity work and about two-thirds supported the Open Philanthropy Project. In 2015, we estimated that expenses for our traditional charity work were about $1.1 million.

Donations supporting GiveWell’s operations: GiveWell raised $5.6 million in unrestricted funding (which we use to support our operations) in 2016, compared to $5.1 million in 2015. Our major institutional supporters and the five largest individual donors contributed about 70% of GiveWell’s operational funding in 2016. This is driven in large part by the fact that Good Ventures funded two-thirds of the costs of the Open Philanthropy project, in addition to funding 20% of GiveWell’s other costs.

Web traffic: The number of unique visitors to our website was down very slightly (by 1%) in 2016 compared to 2015 (when excluding visitors driven by AdWords, Google’s online advertising product).

For more detail, see our full metrics report (PDF).

The post GiveWell’s money moved and web traffic in 2016 appeared first on The GiveWell Blog.

In August 2017, GiveWell recommended a grant of $1.3 million to the Centre for Pesticide Suicide Prevention (CPSP). This grant was made as part of GiveWell’s Incubation Grants program to seed the development of potential future GiveWell top charities and to grow the pipeline of organizations we can consider for a recommendation. CPSP implements a different type of program from work GiveWell has funded in the past. Namely, CPSP identifies the pesticides which are most commonly used in suicides and advocates for governments to ban the most lethal pesticides.

Because CPSP's goal is to encourage governments to enact bans, its work falls into the broader category of policy advocacy, an area we are newly focused on. We plan to investigate or are in the process of investigating several other policy causes, including tobacco control, lead paint regulation, and measures to improve road traffic safety.

Summary

This post will discuss:

• GiveWell's interest in researching policy advocacy interventions as possible priority programs. (More)
• Why CPSP is promising as a policy advocacy organization and Incubation Grant recipient. (More)
• Our plans for following CPSP's work going forward. (More)

Read More

The post Considering policy advocacy organizations: Why GiveWell made a grant to the Centre for Pesticide Suicide Prevention appeared first on The GiveWell Blog.

In August 2017, GiveWell recommended a grant of $1.3 million to the Centre for Pesticide Suicide Prevention (CPSP). This grant was made as part of GiveWell’s Incubation Grants program to seed the development of potential future GiveWell top charities and to grow the pipeline of organizations we can consider for a recommendation. CPSP implements a different type of program from work GiveWell has funded in the past. Namely, CPSP identifies the pesticides which are most commonly used in suicides and advocates for governments to ban the most lethal pesticides.

Because CPSP’s goal is to encourage governments to enact bans, its work falls into the broader category of policy advocacy, an area we are newly focused on. We plan to investigate or are in the process of investigating several other policy causes, including tobacco control, lead paint regulation, and measures to improve road traffic safety.

Summary

This post will discuss:

• GiveWell’s interest in researching policy advocacy interventions as possible priority programs. (More)
• Why CPSP is promising as a policy advocacy organization and Incubation Grant recipient. (More)
• Our plans for following CPSP’s work going forward. (More)

Policy advocacy work

One of the key criteria we use to evaluate potential top charities is their cost-effectiveness—how much good each dollar donated to that charity can accomplish. In recent years, we’ve identified several charities that we estimate to be around 4 to 10 times as cost-effective as GiveDirectly, which we use as a benchmark for cost-effectiveness. Our top charities are extremely cost-effective, but we wonder whether we might be able to find opportunities that are significantly more cost-effective than the charities we currently recommend.

Our current top charities largely focus on direct implementation of health and poverty alleviation interventions. One of our best guesses for where we might find significantly more cost-effective charities is in the area of policy advocacy, or programs that aim to influence government policy. Our intuition is that spending a relatively small amount of money on advocacy could lead to policy changes resulting in long-run benefits for many people, and thus could be among the most cost-effective ways to help people. As a result, researching policy advocacy interventions is one of our biggest priorities for the year ahead.

Policy advocacy work may have the following advantages:

• Leverage: A relatively small amount of spending on advocacy may influence larger amounts of government funding;
• Sustainability: A policy may be in place for years after its adoption; and
• Feasibility: Some effective interventions can only be effectively implemented by governments, such as increasing taxes on tobacco to reduce consumption.

Policy advocacy also poses serious challenges for GiveWell when we consider it as a potential priority area:

• Evidence of effectiveness will likely be lower quality than what we’ve seen from our top charities, e.g. it may involve analyzing trends over time (where confounding factors may complicate analysis) rather than randomized controlled trials or quasi-experimental evidence;
• Causal attribution will be challenging in that multiple players are likely to be involved in any policy change and policymakers are likely to be influenced by a variety of factors;
• There may be a substantial chance of failure to pass the desired legislation; and
• Regulation may have undesirable secondary effects.

Overall, evaluating policy advocacy requires a different approach to assessing evidence and probability of success than our top charities work has in the past.

Incubation Grant to the Centre for Pesticide Suicide Prevention

CPSP began work in 2016 and aims to reduce deaths due to deliberate ingestion of lethal pesticides. With this Incubation Grant, which is intended to cover two years of expenses, CPSP expects to collect data on which pesticides are most often used in suicide attempts and which are most lethal, and then to use this data to advocate to the governments of India and Nepal to implement bans of certain lethal pesticides.

Research suggests that worldwide, approximately 14% to 20% of suicides involved the deliberate ingestion of pesticides. This method of suicide may be particularly common in agricultural populations. The case we see for this grant relies largely on data from Sri Lanka, where bans on the pesticides that were most lethal and most commonly used in suicide coincided with a substantial decrease in the overall suicide rate; we find the case that the decline in suicides was primarily caused by the pesticide bans reasonably compelling. CPSP’s director, Michael Eddleston, was involved in advocating for some of those bans. Read more here.

GiveWell learned of CPSP’s work through James Snowden, who joined GiveWell as a Research Consultant in early 2017. We decided to recommend support to CPSP based on the evidence that pesticide regulation may reduce overall suicide rates, our impression that an advocacy organization could effect changes in regulations, our view that Michael Eddleston and Leah Utyasheva (the co-founders) are well-positioned to do this type of work, and our expectation that we would be able to evaluate CPSP’s impact on pesticide regulation in Nepal and India over the next few years. We thus think CPSP is a plausible future GiveWell top charity and a good fit for an Incubation Grant.

While deciding whether to make this grant, GiveWell staff discussed how to think about the impact of preventing a suicide. Thinking about this question depends on limited empirical information, and staff did not come to an internal consensus. Our best guess at this point is that CPSP generally prevents suicide by people who are making impulsive decisions.

We see several risks to the success of this grant:

• Banning lethal pesticides may be ineffective as a means of preventing suicide, in India and Nepal or more broadly. The case for this area of policy advocacy relies largely on the observational studies from Sri Lanka mentioned above, supported by Sri Lankan medical records suggesting the decline is partially explained by a shift to less lethal pesticides in suicide attempts.
• CPSP may not be able to translate its research into policy change. This risk of failure to achieve legislative change characterizes policy advocacy work in general, to some extent, and requires us to make a type of prediction that is not needed when evaluating a charity directly implementing a program.
• Banning pesticides could lead to offsetting effects in agricultural production. The limited evidence we have seen on this question suggests that past pesticide bans have not led to notable decreases in agricultural production, but we still believe this is a risk.
• CPSP is a new organization, so it does not have a track record of successfully conducting this type of research and achieving policy change.

To quantify the risks above, GiveWell Executive Director Elie Hassenfeld and James Snowden each recorded predictions about the outcomes of this grant at the time the grant was made. Briefly (more predictions here), Elie and James predict with 33% and 55% probability, respectively, that Nepal will pass legislation banning at least one of the three pesticides most commonly used in suicide by July 1, 2020, and with 15% and 35% probability, respectively, that at least one state in India will do so.

Going forward

We plan to continue having regular conversations with CPSP, and a more substantial check-in one year after the grant was made. At that point, we intend to assess whether CPSP has been meeting the milestones it expected to meet and decide whether to provide a third year of funding. If this grant is successful, we hope we may be able to evaluate CPSP as a potential top charity.

The post Considering policy advocacy organizations: Why GiveWell made a grant to the Centre for Pesticide Suicide Prevention appeared first on The GiveWell Blog.

In August 2017, GiveWell recommended a grant of $1.3 million to the Centre for Pesticide Suicide Prevention (CPSP). This grant was made as part of GiveWell’s Incubation Grants program to seed the development of potential future GiveWell top charities and to grow the pipeline of organizations we can consider for a recommendation. CPSP implements a different type of program from work GiveWell has funded in the past. Namely, CPSP identifies the pesticides which are most commonly used in suicides and advocates for governments to ban the most lethal pesticides.

Because CPSP’s goal is to encourage governments to enact bans, its work falls into the broader category of policy advocacy, an area we are newly focused on. We plan to investigate or are in the process of investigating several other policy causes, including tobacco control, lead paint regulation, and measures to improve road traffic safety.

Summary

This post will discuss:

• GiveWell’s interest in researching policy advocacy interventions as possible priority programs. (More)
• Why CPSP is promising as a policy advocacy organization and Incubation Grant recipient. (More)
• Our plans for following CPSP’s work going forward. (More)

Policy advocacy work

One of the key criteria we use to evaluate potential top charities is their cost-effectiveness—how much good each dollar donated to that charity can accomplish. In recent years, we’ve identified several charities that we estimate to be around 4 to 10 times as cost-effective as GiveDirectly, which we use as a benchmark for cost-effectiveness. Our top charities are extremely cost-effective, but we wonder whether we might be able to find opportunities that are significantly more cost-effective than the charities we currently recommend.

Our current top charities largely focus on direct implementation of health and poverty alleviation interventions. One of our best guesses for where we might find significantly more cost-effective charities is in the area of policy advocacy, or programs that aim to influence government policy. Our intuition is that spending a relatively small amount of money on advocacy could lead to policy changes resulting in long-run benefits for many people, and thus could be among the most cost-effective ways to help people. As a result, researching policy advocacy interventions is one of our biggest priorities for the year ahead.

Policy advocacy work may have the following advantages:

• Leverage: A relatively small amount of spending on advocacy may influence larger amounts of government funding;
• Sustainability: A policy may be in place for years after its adoption; and
• Feasibility: Some effective interventions can only be effectively implemented by governments, such as increasing taxes on tobacco to reduce consumption.

Policy advocacy also poses serious challenges for GiveWell when we consider it as a potential priority area:

• Evidence of effectiveness will likely be lower quality than what we’ve seen from our top charities, e.g. it may involve analyzing trends over time (where confounding factors may complicate analysis) rather than randomized controlled trials or quasi-experimental evidence;
• Causal attribution will be challenging in that multiple players are likely to be involved in any policy change and policymakers are likely to be influenced by a variety of factors;
• There may be a substantial chance of failure to pass the desired legislation; and
• Regulation may have undesirable secondary effects.

Overall, evaluating policy advocacy requires a different approach to assessing evidence and probability of success than our top charities work has in the past.

Incubation Grant to the Centre for Pesticide Suicide Prevention

CPSP began work in 2016 and aims to reduce deaths due to deliberate ingestion of lethal pesticides. With this Incubation Grant, which is intended to cover two years of expenses, CPSP expects to collect data on which pesticides are most often used in suicide attempts and which are most lethal, and then to use this data to advocate to the governments of India and Nepal to implement bans of certain lethal pesticides.

Research suggests that worldwide, approximately 14% to 20% of suicides involved the deliberate ingestion of pesticides. This method of suicide may be particularly common in agricultural populations. The case we see for this grant relies largely on data from Sri Lanka, where bans on the pesticides that were most lethal and most commonly used in suicide coincided with a substantial decrease in the overall suicide rate; we find the case that the decline in suicides was primarily caused by the pesticide bans reasonably compelling. CPSP’s director, Michael Eddleston, was involved in advocating for some of those bans. Read more here.

GiveWell learned of CPSP’s work through James Snowden, who joined GiveWell as a Research Consultant in early 2017. We decided to recommend support to CPSP based on the evidence that pesticide regulation may reduce overall suicide rates, our impression that an advocacy organization could effect changes in regulations, our view that Michael Eddleston and Leah Utyasheva (the co-founders) are well-positioned to do this type of work, and our expectation that we would be able to evaluate CPSP’s impact on pesticide regulation in Nepal and India over the next few years. We thus think CPSP is a plausible future GiveWell top charity and a good fit for an Incubation Grant.

While deciding whether to make this grant, GiveWell staff discussed how to think about the impact of preventing a suicide. Thinking about this question depends on limited empirical information, and staff did not come to an internal consensus. Our best guess at this point is that CPSP generally prevents suicide by people who are making impulsive decisions.

We see several risks to the success of this grant:

• Banning lethal pesticides may be ineffective as a means of preventing suicide, in India and Nepal or more broadly. The case for this area of policy advocacy relies largely on the observational studies from Sri Lanka mentioned above, supported by Sri Lankan medical records suggesting the decline is partially explained by a shift to less lethal pesticides in suicide attempts.
• CPSP may not be able to translate its research into policy change. This risk of failure to achieve legislative change characterizes policy advocacy work in general, to some extent, and requires us to make a type of prediction that is not needed when evaluating a charity directly implementing a program.
• Banning pesticides could lead to offsetting effects in agricultural production. The limited evidence we have seen on this question suggests that past pesticide bans have not led to notable decreases in agricultural production, but we still believe this is a risk.
• CPSP is a new organization, so it does not have a track record of successfully conducting this type of research and achieving policy change.

To quantify the risks above, GiveWell Executive Director Elie Hassenfeld and James Snowden each recorded predictions about the outcomes of this grant at the time the grant was made. Briefly (more predictions here), Elie and James predict with 33% and 55% probability, respectively, that Nepal will pass legislation banning at least one of the three pesticides most commonly used in suicide by July 1, 2020, and with 15% and 35% probability, respectively, that at least one state in India will do so.

Going forward

We plan to continue having regular conversations with CPSP, and a more substantial check-in one year after the grant was made. At that point, we intend to assess whether CPSP has been meeting the milestones it expected to meet and decide whether to provide a third year of funding. If this grant is successful, we hope we may be able to evaluate CPSP as a potential top charity.

The post Considering policy advocacy organizations: Why GiveWell made a grant to the Centre for Pesticide Suicide Prevention appeared first on The GiveWell Blog.

Our goal with hosting quarterly open threads is to give blog readers an opportunity to publicly raise comments or questions about GiveWell or related topics (in the comments section below). As always, you’re also welcome to email us at info@givewell.org or to request a call with GiveWell staff if you have feedback or questions you’d prefer to discuss privately. We’ll try to respond promptly to questions or comments.

You can view our December 2017 open thread here.

The post March 2018 open thread appeared first on The GiveWell Blog.

Our goal with hosting quarterly open threads is to give blog readers an opportunity to publicly raise comments or questions about GiveWell or related topics (in the comments section below). As always, you’re also welcome to email us at info@givewell.org or to request a call with GiveWell staff if you have feedback or questions you’d prefer to discuss privately. We’ll try to respond promptly to questions or comments.

You can view our December 2017 open thread here.

The post March 2018 open thread appeared first on The GiveWell Blog.

Many charities aim to influence how others (other donors, governments, or the private sector) allocate their funds. We call this influence on others “leverage.” Expenditure on a program can also crowd out funding that would otherwise have come from other sources. We call this “funging” (from “fungibility”).

In GiveWell’s early years, we didn’t account for leverage in our cost-effectiveness analysis; we counted all costs of an intervention equally, no matter who paid for them.1For example, see row 3 of our 2013 cost-effectiveness analysis for Against Malaria Foundation. jQuery("#footnote_plugin_tooltip_1").tooltip({ tip: "#footnote_plugin_tooltip_text_1", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); For example, for the Schistosomiasis Control Initiative (SCI), a charity that treats intestinal parasites (deworming), we counted both drug and delivery costs, even when the drugs were donated. We did this because we felt it was the simplest approach, least prone to significant error or manipulation.

Over the last few years, our approach has evolved, and we made some adjustments for leverage and funging to our cost-effectiveness analyses where we felt they were clearly warranted.

In our top charities update at the end of 2017, we made a major change to how we dealt with the question of leverage by incorporating explicit, formal leverage estimates for every charity we recommend.

This change made our cost-effectiveness estimates of deworming charities (which typically leverage substantial government funding) look more cost-effective than our previous method. For example, our new method makes SCI look 1.2x more cost-effective than in the previous cost-effectiveness update. More details are in the table at the end of this post.

We also think the change makes our reasoning more transparent and more consistent across organizations.

In this post, we:

• Describe how our treatment of leverage and funging has evolved.
• Highlight two major limitations of our current approach.
• Present how much difference leverage and funging make to our cost-effectiveness estimates.

Details follow.

How our thinking has evolved

We last wrote about our approach to leverage and funging in a 2011 blog post. In short, we didn’t explicitly account for leverage in our cost-effectiveness analysis, counting costs to all entities equally. We concluded:

When we do cost-effectiveness estimates (e.g., “cost per life saved”) we consider all expenses from all sources, not just funding provided by GiveWell donors. For SCI, we count both drug and delivery costs, even when drugs are donated. (Generally, we try to count all donated goods and services at market value, i.e., the price the donor could have sold them for instead of donating them.) For [the Against Malaria Foundation (AMF)], we count net costs and distribution costs, even though AMF pays only for the former. In the case of VillageReach, we even count government costs of delivering vaccines, even though VillageReach works exclusively to improve the efficiency of the delivery system.

We consider this approach the simplest approach to dealing with the issues discussed here, and given our limited understanding of how “leverage” works, we believe that this approach minimizes the error in our estimates that might come from misreading the “leverage” situation. As our understanding of “leverage” improves, we may approach our cost-effectiveness estimates differently.

Since 2011, our thinking changed. Over time, we started applying some adjustments to our cost-effectiveness model to account for leverage and funging when it seemed important to our bottom line and fairly clear that some adjustment was warranted:

• We applied discounts to costs incurred by certain entities. For example, we applied a 50% discount to the value of teacher time spent distributing deworming tablets, and excluded the costs to pharmaceutical companies donating these tablets.2See our May 2017 cost-effectiveness analysis. jQuery("#footnote_plugin_tooltip_2").tooltip({ tip: "#footnote_plugin_tooltip_text_2", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); Our rationale was that without our top charities, these resources would likely otherwise been used less productively.
• We applied ‘alternative funders adjustments’ to account for the possibility that we were crowding out other funders. For example, some of the distributions that AMF considered funding, but didn’t ultimately fund, were picked up by other funders (more).

This helped us explicitly think through considerations relevant to our top charities. But by the end of 2016, our model had a handful of ad hoc adjustments that were difficult to identify, understand, and vet. For example, the discounts we applied to costs incurred by certain entities were ‘baked in’ to our estimates of cost per treatment, rather than explicit on the main spreadsheet of our cost-effectiveness analysis.

Changes to how we incorporate leverage and funging into our cost-effectiveness analysis

We revisited the way we thought about leverage and funging in preparation for our 2017 top charities decision. We wanted to make sure our adjustments were transparent and consistent across all charities.

We now explicitly make quantitative judgments about (i) the probability that our charities are causing governments and multilateral aid agencies to spend more or less on a program than they otherwise would have and (ii) the value of what those funds would otherwise have been spent on.3Our current best guess of a reasonable benchmark for the counterfactual value of government funds is ~75% as cost-effective as GiveDirectly (discussed later in the post). We view this is a very rough guess. jQuery("#footnote_plugin_tooltip_3").tooltip({ tip: "#footnote_plugin_tooltip_text_3", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

Here’s an exercise that some GiveWell staff have found helpful for getting a more intuitive feel for different ways of treating leverage.

Suppose a charity pays $5,000 to purchase magic pills. This would cause (with 100% certainty) the government to spend another $5,000 distributing those pills. The pill distribution saves 1,000 lives in total. If the government didn’t fund the pill distribution, it would have spent $5,000 on something that would have saved 250 lives.

How should a philanthropist think about the cost-effectiveness of this charity?

1. One option is to include all costs to all actors on the cost side of the cost-effectiveness ratio. Total costs are $10,000 to save 1,000 lives and cost-effectiveness is $10 / life saved. This was GiveWell’s approach in 2011.
2. Another option is to discount government costs by 50%, because the government would otherwise have spent the funds on something 50% as effective. So total costs are $5,000 + (50% x $5,000) = $7,500. 1,000 lives are saved and cost-effectiveness is $7.50 / life saved. This was GiveWell’s approach from 2014 through 2016.
3. A third option is to include only the costs to the charity on the ‘cost’ side. The charity causes the magic pill distribution to happen, saving 1,000 lives. But it also causes the government to spend $5,000, which otherwise would have been used to save 250 lives. So the total costs are $5,000, and 1,000 – 250 = 750 lives are saved. Cost-effectiveness is $6.66 / life saved. This is GiveWell’s approach now.4In order to isolate the effect that leverage/funging has, we first calculate the impact of the program using the first method (including all costs equally), then apply a “leverage/funging” adjustment to transform the answer to the third method. jQuery("#footnote_plugin_tooltip_4").tooltip({ tip: "#footnote_plugin_tooltip_text_4", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

We believe the third way of treating leverage best reflects the true counterfactual impact of a charity’s activities. It also makes charities that are leveraging other funders look substantially more cost-effective than we previously thought.

Limitations of our approach

There are two important limitations to the way we account for leverage and funging.

First, these estimates rely on more guesswork than most of our cost-effectiveness analysis, reflecting a fundamental tradeoff we face in deciding which considerations to explicitly quantify. Quantification forces us to think through not just whether a particular consideration matters, but how much it matters relative to other factors, and to be explicit about that. On the other hand, incorporating very uncertain factors into our analysis can reduce its reliability, give a false impression of certainty, and make it difficult for others to engage with our work. In this case, we thought the benefits of explicit quantification outweighed the costs.

Two examples of assumptions going into our leverage and funging adjustments that we’re highly uncertain about:

1. Our best guess is that the average counterfactual use of domestic government spending that could be leveraged by our top charities is ~75% as cost-effective as GiveDirectly. We think using this figure is a useful heuristic, which roughly accords with our intuitions (and ensures we’re being consistent between charities), but we don’t feel confident that we have a good sense of what governments would counterfactually spend their funds on, or how valuable those activities might be.
2. We estimate there is a ~70% chance that, without Malaria Consortium funding, the marginal seasonal malaria chemoprevention (SMC) program would go unfunded, but only a ~40% chance that, without Against Malaria Foundation funding, the marginal bednet distribution would go unfunded. Estimating these probabilities is challenging, but taking our best guess forces us to evaluate how much weight to place on the qualitative consideration that there are more alternative funders for bednet distribution than SMC.

Second, we don’t explicitly model the long-term financial sustainability of a program. One worldview we find plausible for the role of effective philanthropy is in demonstrating the effectiveness of novel projects that, in the long run, are taken up by governments. This is not captured within our current model, which only looks at the effects of leverage and funging in the short term. Due to the difficulty of explicitly modelling this consideration, we take it into account qualitatively.5For example, we allocated more discretionary funding than we would have on the basis of cost-effectiveness alone to No Lean Season in 2017 due to our view that it was demonstrating the effectiveness of a novel program, which may have long-run funding implications. jQuery("#footnote_plugin_tooltip_5").tooltip({ tip: "#footnote_plugin_tooltip_text_5", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

 

How much of a difference do leverage and funging make?

In the table below, we present how our new method of accounting for leverage and funging compares to (i) counting all costs equally and (ii) our previous method of accounting for leverage and funging.

Adjustments range between a modest penalty for AMF (because we expect AMF crowds out some funds from other sources) to a large boost to SCI (because the cost to pharmaceutical companies of manufacturing donated drugs comprises a substantial proportion of cost per treatment in SCI distributions, and we expect that without SCI, these resources would have been put to less valuable uses).

Note: 1.2x implies the adjustment makes the charity look 20% more cost-effective; 0.8x implies the adjustment makes the charity look 20% less cost-effective. All charities listed are GiveWell top charities as of November 2017.

Charity Versus counting all costs equally6Calculations here. “N/A” refers to charities for which we had not completed a cost-effectiveness analysis before October 2017. jQuery("#footnote_plugin_tooltip_6").tooltip({ tip: "#footnote_plugin_tooltip_text_6", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); Versus our 2014-16 methodology Commentary Against Malaria Foundation 0.8x 1.1x Government costs represent a small proportion of funding for AMF programs. Our analysis of distributions that AMF considered, but did not fund, suggests that some of these distributions are covered by alternative funders, who would otherwise have supported less valuable programs. Schistosomiasis Control Initiative 2x 1.2x We estimate ~60% of the costs of SCI-supported deworming programs are incurred by either governments or pharmaceutical companies. We expect that without SCI, most of these resources would have been used on less valuable programs. Evidence Action’s Deworm the World Initiative 1.4x 1.1x We estimate ~40% of the costs of Deworm the World-supported deworming programs are incurred by either governments or pharmaceutical companies. We expect that without Deworm the World, most of these resources would have been used on less valuable programs. Sightsavers’ deworming program 1.6x 1.3x We estimate ~50% of the costs of deworming in Sightsavers supported programs are from governments or donated drugs from pharmaceutical companies. We expect that without Sightsavers, most of these resources would have been used on less valuable programs. END Fund’s deworming program 1.3x N/A We estimate ~40% of the costs of END Fund-supported deworming programs are incurred by either governments or pharmaceutical companies. We expect that without the END Fund, most of these resources would have been used on less valuable programs. Helen Keller International (HKI)’s vitamin A supplementation (VAS) program 1.1x N/A We estimate ~25% of the costs of HKI-supported VAS programs are covered by governments. We expect that without HKI, most of these resources would have been used on less valuable programs. GiveDirectly 1x 1x Due to the scalability of GiveDirectly’s program, we believe it is unlikely that GiveDirectly crowds out funding from other sources. GiveDirectly does not leverage funds from other sources. Malaria Consortium’s seasonal malaria chemoprevention program .98x 1.04x Government costs represent a small proportion of funding for Malaria Consortium programs. We believe it is possible but unlikely that Malaria Consortium crowds out additional government funding. Evidence Action’s No Lean Season 1x N/A No Lean Season is a novel program, and we think it’s unlikely to be crowding out funding from other sources. No Lean Season does not leverage substantial funding from other sources.

Notes   [ + ]

1. ↑ For example, see row 3 of our 2013 cost-effectiveness analysis for Against Malaria Foundation. 2. ↑ See our May 2017 cost-effectiveness analysis. 3. ↑ Our current best guess of a reasonable benchmark for the counterfactual value of government funds is ~75% as cost-effective as GiveDirectly (discussed later in the post). We view this is a very rough guess. 4. ↑ In order to isolate the effect that leverage/funging has, we first calculate the impact of the program using the first method (including all costs equally), then apply a “leverage/funging” adjustment to transform the answer to the third method. 5. ↑ For example, we allocated more discretionary funding than we would have on the basis of cost-effectiveness alone to No Lean Season in 2017 due to our view that it was demonstrating the effectiveness of a novel program, which may have long-run funding implications. 6. ↑ Calculations here. “N/A” refers to charities for which we had not completed a cost-effectiveness analysis before October 2017. function footnote_expand_reference_container() { jQuery("#footnote_references_container").show(); jQuery("#footnote_reference_container_collapse_button").text("-"); } function footnote_collapse_reference_container() { jQuery("#footnote_references_container").hide(); jQuery("#footnote_reference_container_collapse_button").text("+"); } function footnote_expand_collapse_reference_container() { if (jQuery("#footnote_references_container").is(":hidden")) { footnote_expand_reference_container(); } else { footnote_collapse_reference_container(); } } function footnote_moveToAnchor(p_str_TargetID) { footnote_expand_reference_container(); var l_obj_Target = jQuery("#" + p_str_TargetID); if(l_obj_Target.length) { jQuery('html, body').animate({ scrollTop: l_obj_Target.offset().top - window.innerHeight/2 }, 1000); } }

The post Revisiting leverage appeared first on The GiveWell Blog.

Many charities aim to influence how others (other donors, governments, or the private sector) allocate their funds. We call this influence on others “leverage.” Expenditure on a program can also crowd out funding that would otherwise have come from other sources. We call this “funging” (from “fungibility”).

In GiveWell’s early years, we didn’t account for leverage in our cost-effectiveness analysis; we counted all costs of an intervention equally, no matter who paid for them.1For example, see row 3 of our 2013 cost-effectiveness analysis for Against Malaria Foundation. jQuery("#footnote_plugin_tooltip_1").tooltip({ tip: "#footnote_plugin_tooltip_text_1", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); For example, for the Schistosomiasis Control Initiative (SCI), a charity that treats intestinal parasites (deworming), we counted both drug and delivery costs, even when the drugs were donated. We did this because we felt it was the simplest approach, least prone to significant error or manipulation.

Over the last few years, our approach has evolved, and we made some adjustments for leverage and funging to our cost-effectiveness analyses where we felt they were clearly warranted.

In our top charities update at the end of 2017, we made a major change to how we dealt with the question of leverage by incorporating explicit, formal leverage estimates for every charity we recommend.

This change made our cost-effectiveness estimates of deworming charities (which typically leverage substantial government funding) look more cost-effective than our previous method. For example, our new method makes SCI look 1.2x more cost-effective than in the previous cost-effectiveness update. More details are in the table at the end of this post.

We also think the change makes our reasoning more transparent and more consistent across organizations.

In this post, we:

• Describe how our treatment of leverage and funging has evolved.
• Highlight two major limitations of our current approach.
• Present how much difference leverage and funging make to our cost-effectiveness estimates.

Details follow.

How our thinking has evolved

We last wrote about our approach to leverage and funging in a 2011 blog post. In short, we didn’t explicitly account for leverage in our cost-effectiveness analysis, counting costs to all entities equally. We concluded:

When we do cost-effectiveness estimates (e.g., “cost per life saved”) we consider all expenses from all sources, not just funding provided by GiveWell donors. For SCI, we count both drug and delivery costs, even when drugs are donated. (Generally, we try to count all donated goods and services at market value, i.e., the price the donor could have sold them for instead of donating them.) For [the Against Malaria Foundation (AMF)], we count net costs and distribution costs, even though AMF pays only for the former. In the case of VillageReach, we even count government costs of delivering vaccines, even though VillageReach works exclusively to improve the efficiency of the delivery system.

We consider this approach the simplest approach to dealing with the issues discussed here, and given our limited understanding of how “leverage” works, we believe that this approach minimizes the error in our estimates that might come from misreading the “leverage” situation. As our understanding of “leverage” improves, we may approach our cost-effectiveness estimates differently.

Since 2011, our thinking changed. Over time, we started applying some adjustments to our cost-effectiveness model to account for leverage and funging when it seemed important to our bottom line and fairly clear that some adjustment was warranted:

• We applied discounts to costs incurred by certain entities. For example, we applied a 50% discount to the value of teacher time spent distributing deworming tablets, and excluded the costs to pharmaceutical companies donating these tablets.2See our May 2017 cost-effectiveness analysis. jQuery("#footnote_plugin_tooltip_2").tooltip({ tip: "#footnote_plugin_tooltip_text_2", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); Our rationale was that without our top charities, these resources would likely otherwise been used less productively.
• We applied ‘alternative funders adjustments’ to account for the possibility that we were crowding out other funders. For example, some of the distributions that AMF considered funding, but didn’t ultimately fund, were picked up by other funders (more).

This helped us explicitly think through considerations relevant to our top charities. But by the end of 2016, our model had a handful of ad hoc adjustments that were difficult to identify, understand, and vet. For example, the discounts we applied to costs incurred by certain entities were ‘baked in’ to our estimates of cost per treatment, rather than explicit on the main spreadsheet of our cost-effectiveness analysis.

Changes to how we incorporate leverage and funging into our cost-effectiveness analysis

We revisited the way we thought about leverage and funging in preparation for our 2017 top charities decision. We wanted to make sure our adjustments were transparent and consistent across all charities.

We now explicitly make quantitative judgments about (i) the probability that our charities are causing governments and multilateral aid agencies to spend more or less on a program than they otherwise would have and (ii) the value of what those funds would otherwise have been spent on.3Our current best guess of a reasonable benchmark for the counterfactual value of government funds is ~75% as cost-effective as GiveDirectly (discussed later in the post). We view this is a very rough guess. jQuery("#footnote_plugin_tooltip_3").tooltip({ tip: "#footnote_plugin_tooltip_text_3", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

Here’s an exercise that some GiveWell staff have found helpful for getting a more intuitive feel for different ways of treating leverage.

Suppose a charity pays $5,000 to purchase magic pills. This would cause (with 100% certainty) the government to spend another $5,000 distributing those pills. The pill distribution saves 1,000 lives in total. If the government didn’t fund the pill distribution, it would have spent $5,000 on something that would have saved 250 lives.

How should a philanthropist think about the cost-effectiveness of this charity?

1. One option is to include all costs to all actors on the cost side of the cost-effectiveness ratio. Total costs are $10,000 to save 1,000 lives and cost-effectiveness is $10 / life saved. This was GiveWell’s approach in 2011.
2. Another option is to discount government costs by 50%, because the government would otherwise have spent the funds on something 50% as effective. So total costs are $5,000 + (50% x $5,000) = $7,500. 1,000 lives are saved and cost-effectiveness is $7.50 / life saved. This was GiveWell’s approach from 2014 through 2016.
3. A third option is to include only the costs to the charity on the ‘cost’ side. The charity causes the magic pill distribution to happen, saving 1,000 lives. But it also causes the government to spend $5,000, which otherwise would have been used to save 250 lives. So the total costs are $5,000, and 1,000 – 250 = 750 lives are saved. Cost-effectiveness is $6.66 / life saved. This is GiveWell’s approach now.4In order to isolate the effect that leverage/funging has, we first calculate the impact of the program using the first method (including all costs equally), then apply a “leverage/funging” adjustment to transform the answer to the third method. jQuery("#footnote_plugin_tooltip_4").tooltip({ tip: "#footnote_plugin_tooltip_text_4", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

We believe the third way of treating leverage best reflects the true counterfactual impact of a charity’s activities. It also makes charities that are leveraging other funders look substantially more cost-effective than we previously thought.

Limitations of our approach

There are two important limitations to the way we account for leverage and funging.

First, these estimates rely on more guesswork than most of our cost-effectiveness analysis, reflecting a fundamental tradeoff we face in deciding which considerations to explicitly quantify. Quantification forces us to think through not just whether a particular consideration matters, but how much it matters relative to other factors, and to be explicit about that. On the other hand, incorporating very uncertain factors into our analysis can reduce its reliability, give a false impression of certainty, and make it difficult for others to engage with our work. In this case, we thought the benefits of explicit quantification outweighed the costs.

Two examples of assumptions going into our leverage and funging adjustments that we’re highly uncertain about:

1. Our best guess is that the average counterfactual use of domestic government spending that could be leveraged by our top charities is ~75% as cost-effective as GiveDirectly. We think using this figure is a useful heuristic, which roughly accords with our intuitions (and ensures we’re being consistent between charities), but we don’t feel confident that we have a good sense of what governments would counterfactually spend their funds on, or how valuable those activities might be.
2. We estimate there is a ~70% chance that, without Malaria Consortium funding, the marginal seasonal malaria chemoprevention (SMC) program would go unfunded, but only a ~40% chance that, without Against Malaria Foundation funding, the marginal bednet distribution would go unfunded. Estimating these probabilities is challenging, but taking our best guess forces us to evaluate how much weight to place on the qualitative consideration that there are more alternative funders for bednet distribution than SMC.

Second, we don’t explicitly model the long-term financial sustainability of a program. One worldview we find plausible for the role of effective philanthropy is in demonstrating the effectiveness of novel projects that, in the long run, are taken up by governments. This is not captured within our current model, which only looks at the effects of leverage and funging in the short term. Due to the difficulty of explicitly modelling this consideration, we take it into account qualitatively.5For example, we allocated more discretionary funding than we would have on the basis of cost-effectiveness alone to No Lean Season in 2017 due to our view that it was demonstrating the effectiveness of a novel program, which may have long-run funding implications. jQuery("#footnote_plugin_tooltip_5").tooltip({ tip: "#footnote_plugin_tooltip_text_5", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

 

How much of a difference do leverage and funging make?

In the table below, we present how our new method of accounting for leverage and funging compares to (i) counting all costs equally and (ii) our previous method of accounting for leverage and funging.

Adjustments range between a modest penalty for AMF (because we expect AMF crowds out some funds from other sources) to a large boost to SCI (because the cost to pharmaceutical companies of manufacturing donated drugs comprises a substantial proportion of cost per treatment in SCI distributions, and we expect that without SCI, these resources would have been put to less valuable uses).

Note: 1.2x implies the adjustment makes the charity look 20% more cost-effective; 0.8x implies the adjustment makes the charity look 20% less cost-effective. All charities listed are GiveWell top charities as of November 2017.

Charity Versus counting all costs equally6Calculations here. “N/A” refers to charities for which we had not completed a cost-effectiveness analysis before October 2017. jQuery("#footnote_plugin_tooltip_6").tooltip({ tip: "#footnote_plugin_tooltip_text_6", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); Versus our 2014-16 methodology Commentary Against Malaria Foundation 0.8x 1.1x Government costs represent a small proportion of funding for AMF programs. Our analysis of distributions that AMF considered, but did not fund, suggests that some of these distributions are covered by alternative funders, who would otherwise have supported less valuable programs. Schistosomiasis Control Initiative 2x 1.2x We estimate ~60% of the costs of SCI-supported deworming programs are incurred by either governments or pharmaceutical companies. We expect that without SCI, most of these resources would have been used on less valuable programs. Evidence Action’s Deworm the World Initiative 1.4x 1.1x We estimate ~40% of the costs of Deworm the World-supported deworming programs are incurred by either governments or pharmaceutical companies. We expect that without Deworm the World, most of these resources would have been used on less valuable programs. Sightsavers’ deworming program 1.6x 1.3x We estimate ~50% of the costs of deworming in Sightsavers supported programs are from governments or donated drugs from pharmaceutical companies. We expect that without Sightsavers, most of these resources would have been used on less valuable programs. END Fund’s deworming program 1.3x N/A We estimate ~40% of the costs of END Fund-supported deworming programs are incurred by either governments or pharmaceutical companies. We expect that without the END Fund, most of these resources would have been used on less valuable programs. Helen Keller International (HKI)’s vitamin A supplementation (VAS) program 1.1x N/A We estimate ~25% of the costs of HKI-supported VAS programs are covered by governments. We expect that without HKI, most of these resources would have been used on less valuable programs. GiveDirectly 1x 1x Due to the scalability of GiveDirectly’s program, we believe it is unlikely that GiveDirectly crowds out funding from other sources. GiveDirectly does not leverage funds from other sources. Malaria Consortium’s seasonal malaria chemoprevention program .98x 1.04x Government costs represent a small proportion of funding for Malaria Consortium programs. We believe it is possible but unlikely that Malaria Consortium crowds out additional government funding. Evidence Action’s No Lean Season 1x N/A No Lean Season is a novel program, and we think it’s unlikely to be crowding out funding from other sources. No Lean Season does not leverage substantial funding from other sources.

Notes   [ + ]

1. ↑ For example, see row 3 of our 2013 cost-effectiveness analysis for Against Malaria Foundation. 2. ↑ See our May 2017 cost-effectiveness analysis. 3. ↑ Our current best guess of a reasonable benchmark for the counterfactual value of government funds is ~75% as cost-effective as GiveDirectly (discussed later in the post). We view this is a very rough guess. 4. ↑ In order to isolate the effect that leverage/funging has, we first calculate the impact of the program using the first method (including all costs equally), then apply a “leverage/funging” adjustment to transform the answer to the third method. 5. ↑ For example, we allocated more discretionary funding than we would have on the basis of cost-effectiveness alone to No Lean Season in 2017 due to our view that it was demonstrating the effectiveness of a novel program, which may have long-run funding implications. 6. ↑ Calculations here. “N/A” refers to charities for which we had not completed a cost-effectiveness analysis before October 2017. function footnote_expand_reference_container() { jQuery("#footnote_references_container").show(); jQuery("#footnote_reference_container_collapse_button").text("-"); } function footnote_collapse_reference_container() { jQuery("#footnote_references_container").hide(); jQuery("#footnote_reference_container_collapse_button").text("+"); } function footnote_expand_collapse_reference_container() { if (jQuery("#footnote_references_container").is(":hidden")) { footnote_expand_reference_container(); } else { footnote_collapse_reference_container(); } } function footnote_moveToAnchor(p_str_TargetID) { footnote_expand_reference_container(); var l_obj_Target = jQuery("#" + p_str_TargetID); if(l_obj_Target.length) { jQuery('html, body').animate({ scrollTop: l_obj_Target.offset().top - window.innerHeight/2 }, 1000); } }

The post Revisiting leverage appeared first on The GiveWell Blog.

We’re actively hiring for roles across GiveWell.

Operations

We’re hiring a Director of Operations. The job posting is here.

The Director of Operations is responsible for many domains and manages a team of eight people. A successful candidate will excel at prioritizing the most impactful work, shepherding improvements to completion, and managing the team.

This job is perfect for someone who wants to:

• be part of the leadership team at an organization that’s dedicated to making the world a better place.
• work with colleagues who are passionate about the problems they’re trying to solve.
• have significant personal ownership and responsibility.

We’re looking for someone based in the San Francisco Bay Area, where GiveWell’s office is located. This job has flexible hours and can partly be done remotely.

Outreach

We’re hiring a Head of Growth. The job posting is here.

The Head of Growth will be responsible for leading our efforts to increase the amount of money GiveWell’s recommended charities receive as a result of our recommendation. The Head of Growth will set a strategy to maximize our money moved by identifying, implementing, and testing a variety of growth strategies and will build a team to support these objectives.

We’re looking for a Head of Growth who is excited for the challenge of starting and building our Growth team and aligned with our commitment to honesty and transparency about our, and our recommended organizations’, shortcomings and strengths.

Research

We’re looking for talented people to add to our research team. Some of our most successful analysts are people who followed our work closely prior to joining GiveWell, so if you read our blog, please consider applying!

We’re hiring for three positions:

• Research Analyst and Senior Research Analyst.
• Summer Research Analyst.

Research Analysts and Senior Research Analysts are responsible for all of our research work: reviewing potential top charities and following up with current recommended charities, reviewing the evidence for charitable interventions, building cost-effectiveness models, and evaluating potential Incubation Grants.

Our Summer Research Analyst position is for rising college seniors or graduate students with one year left in their program, and offers the opportunity to work on a variety of research tasks at GiveWell over two to three months.

Research Analysts and Senior Research Analysts do not need to be based in the San Francisco Bay Area. Summer Research Analysts do need to be in the San Francisco Bay Area.

The post GiveWell is hiring! appeared first on The GiveWell Blog.

We’re actively hiring for roles across GiveWell.

Operations

We’re hiring a Director of Operations. The job posting is here.

The Director of Operations is responsible for many domains and manages a team of eight people. A successful candidate will excel at prioritizing the most impactful work, shepherding improvements to completion, and managing the team.

This job is perfect for someone who wants to:

• be part of the leadership team at an organization that’s dedicated to making the world a better place.
• work with colleagues who are passionate about the problems they’re trying to solve.
• have significant personal ownership and responsibility.

We’re looking for someone based in the San Francisco Bay Area, where GiveWell’s office is located. This job has flexible hours and can partly be done remotely.

Outreach

We’re hiring a Head of Growth. The job posting is here.

The Head of Growth will be responsible for leading our efforts to increase the amount of money GiveWell’s recommended charities receive as a result of our recommendation. The Head of Growth will set a strategy to maximize our money moved by identifying, implementing, and testing a variety of growth strategies and will build a team to support these objectives.

We’re looking for a Head of Growth who is excited for the challenge of starting and building our Growth team and aligned with our commitment to honesty and transparency about our, and our recommended organizations’, shortcomings and strengths.

Research

We’re looking for talented people to add to our research team. Some of our most successful analysts are people who followed our work closely prior to joining GiveWell, so if you read our blog, please consider applying!

We’re hiring for three positions:

• Research Analyst and Senior Research Analyst.
• Summer Research Analyst.

Research Analysts and Senior Research Analysts are responsible for all of our research work: reviewing potential top charities and following up with current recommended charities, reviewing the evidence for charitable interventions, building cost-effectiveness models, and evaluating potential Incubation Grants.

Our Summer Research Analyst position is for rising college seniors or graduate students with one year left in their program, and offers the opportunity to work on a variety of research tasks at GiveWell over two to three months.

Research Analysts and Senior Research Analysts do not need to be based in the San Francisco Bay Area. Summer Research Analysts do need to be in the San Francisco Bay Area.

The post GiveWell is hiring! appeared first on The GiveWell Blog.

Summary

• Two of GiveWell’s top charities fight malaria in sub-Saharan Africa.
• GiveWell’s valuations of these charities place some weight on research by Hoyt Bleakley on the impacts of malaria eradication efforts in the American South in the 1920s and in Brazil, Colombia, and Mexico in the 1950s.
• I reviewed the Bleakley study and mostly support its key findings: the campaigns to eradicate malaria from Brazil, Colombia, and Mexico, and perhaps the American South as well, were followed by accelerated income gains for people whose childhood exposure to the disease was reduced. The timing of these events is compatible with the theory that rolling back malaria increased prosperity. Full write-up here.

Introduction

I blogged three weeks ago about having reviewed and reanalyzed Hoyt Bleakley’s study of the effort in the 1910s to rid the American South of hookworm disease (not malaria). That study, published in 2007, seems to show that the children who benefited from the campaign attended school more and went on to earn more as adults.

For GiveWell, Bleakley’s 2010 study is to malaria parasites as his 2007 study is to intestinal worms. Like the 2007 paper, the 2010 one looks back at large-scale, 20th-century eradication campaigns in order to estimate impacts on schooling and adult income. It too produces encouraging results. And it has influenced GiveWell’s recommendations of certain charities—the Against Malaria Foundation and Malaria Consortium’s seasonal malaria chemoprevention program.

Because GiveWell had already invested in replicating and reanalyzing Bleakley (2007), and because the two papers overlap in data and method, I decided to do the same for Bleakley (2010). And here the parallel between the two papers breaks down: having run the evidence through my analytical sieve, my confidence that eradicating malaria boosted income is substantially higher than my confidence that eradicating hookworm did. I’m a bit less sure that it did so in the United States than in Brazil, Colombia, and Mexico; but the Latin American experience is probably more relevant for the places in which our recommended charities work.

This post will walk through the results. For details, see the new working paper. Because my malaria reanalysis shares so much with the hookworm one, I have written this post as if you read the last one. If you haven’t, please do that now.

How the malaria analysis differs from the hookworm one

Having just emphasized the commonality between Bleakley’s hookworm and malaria eradication studies—and my reanalyses thereof—in order to orient you, I should explain how the two differ:

• The hookworm study is set exclusively in the American South, while the malaria study looks at efforts in four countries. In the United States in the 1920s, no doubt inspired by the previous decade’s success against hookworm, the Rockefeller Foundation and the U.S. Public Health Service promoted a large-scale program to drain swamps and spray larvicides, which cut malaria mortality in the South by 60%. Then in the 1950s, with the discovery of DDT, the World Health Organization led a worldwide campaign against the disease. Partly because of data availability, Bleakley (2010) studies the consequences in Brazil, Colombia, and Mexico.1Bleakley (2010) also chose these countries because they had malarial and non-malarial regions, allowing comparisons. See Bleakley (2010), note 6. For sample maps see this. jQuery("#footnote_plugin_tooltip_1").tooltip({ tip: "#footnote_plugin_tooltip_text_1", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });
• Where the hookworm study groups data two ways—first by place of residence to study short-term effects, then by place of birth to study long-term effects—the malaria study does only the latter.
• I pre-registered my analysis plan for the malaria study with the Open Science Framework and hewed to it. While I did not allow the plan to bind my actions, it serves to disclose which analytical tactics I settled on before I touched the data and could know what results they would produce.2Actually we registered a plan for the hookworm study too, but the malaria plan was better informed—and better followed—precisely because it came on the heels of the similar hookworm reanalysis. For brevity, I skipped this theme in my blog post. I did write about it in the hookworm working paper. jQuery("#footnote_plugin_tooltip_2").tooltip({ tip: "#footnote_plugin_tooltip_text_2", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });
• The Bleakley malaria paper appeared in a journal published by the American Economic Association (AEA), which requires its authors to post data and computer code on the AEA website. This aided replication and reanalysis. Unfortunately, as appears to be the norm among AEA journals, the Bleakley (2010) data and code only reproduce the paper’s tables, not the graphs that in this case I see as central.
• For Brazil, Colombia, and Mexico, I mostly relied on that publicly posted data for the crucial information on which regions within a country had the most malaria, rather than trying to construct those variables from old maps and books in Spanish and Portuguese. I also relied on the public data for geographic control variables. I think it can be valuable to go back to primary sources, but for the time being at least, this step looked too time-consuming. I did update and expand the Latin outcome data, on such things as literacy and income, because it is already conveniently digitized in IPUMS International. And I reconstructed all the U.S. data from primary sources, simply by copying what we assembled for the hookworm reanalysis.

Results

In showing you what I found, I’ll follow nearly the same narrative as in my previous post’s section on the “long-term impact on earnings.” To start, here is a key graph from the Bleakley (2010) paper—or really four graphs. In each country’s graph, as with the hookworm graphs, each dot shows the association between historical disease burden in a state (or municipio) and the average income in adulthood of people born there in a given year. In all but Colombia, the leftmost dots line up with the negative range on the vertical axis, meaning that, initially, coming from a historically malarial area stunted one’s income. For example, some of the early U.S. dots are around –0.1 on the vertical axis, which means that being native to swampy Mississippi instead of arid Wyoming cut one’s adult earnings by about 10%.3For cross-country comparability, Bleakley (2010) normalizes the malaria mortality and ecology indexes so that the 5th- and 95th-percentile geographic units—Wyoming and Mississippi in the U.S. case—score 0 and 1. Income proxies are taken in logs. jQuery("#footnote_plugin_tooltip_3").tooltip({ tip: "#footnote_plugin_tooltip_text_3", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); The dots later rise, suggesting that the liability of coming from malarial areas faded, and even reversed. In Colombia, the dots start around zero but also then rise.

As in the hookworm study, here, Bleakley (2010) superimposes on the dots the step-like contour representing how malaria eradication is expected to play out in the data. The steps reach their full height when the campaigns are taken to have started—1920 in the United States and 1957 in the Latin nations. All babies born after these points were alike in that they grew up fully in the post–eradication campaign world. The step contours begin their rises 18 years earlier, when the first babies were born who would benefit from eradication at least a bit by their 18th birthdays.4These graphs incorporate all of Bleakley’s control variables. In my hookworm post, I began both results sections with “basic” graphs that did not include all the controls, imitating Bleakley (2007). In contrast, all the Bleakley (2010) graphs incorporate full controls. So I do the same. jQuery("#footnote_plugin_tooltip_4").tooltip({ tip: "#footnote_plugin_tooltip_text_4", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

Next is my closest replication of the key Bleakley (2010) graphs. These use Bleakley’s data, as posted, but not Bleakley’s computer code, since that was not posted:

The next version adds the latest rounds of census data from the Latin nations and the newer, larger samples from old census rounds for the United States. It also redefines childhood as lasting 21 instead of 18 years, because I discovered that the Bleakley (2010) code uses 18 but the text uses 21. That budges the first dashed lines back by three years:

I avoided superimposing step contours on these data points because I worried that it would trick the brain into thinking that the contours fit the data better than they do. But whether the step contour fits the plots above is exactly what you should ask yourself now. Does it seem as if the dots rise, or rise more, between each pair of vertical, dashed lines? I could see the answer being “yes” for all but Mexico. And that could be a fingerprint of malaria eradication.

I ask that question more formally in the next quartet, fitting line segments to successive ranges of the data. The dots in the four graphs are the same as above, but I’ve taken away the grey confidence intervals for readability. The p values in the lower-left of each pane speak to whether any upward or downward bends at the allowed kink points are statistically significant, i.e., hard to ascribe to chance alone. Where the p values are low—and they mostly are, even in Mexico—they favor the Bleakley (2010) reading that rolling back malaria raised incomes.

In Brazil, Colombia, and Mexico, this statistical test is fairly confident that red lines bend upward at the first kinks (p = 0.00 for Brazil and Colombia and 0.07 for Mexico). That is: in high-malaria areas, relative to low-malaria areas, as the first babies were born who could benefit in childhood from eradication, future incomes rose. The test is less confident for the United States, where the first allowed kink, in 1899, gets a high-ish p value of 0.39. However, the U.S. trend clearly bends upward—just earlier than predicted by the Bleakley (2010) theory. That might mean that the Bleakley (2010) theory is slightly wrong: maybe when it came to impacts on future earnings, malaria exposure continued to matter into one’s twenties, at least in the United States 100 years ago. Then, people born in the South even a bit before 1899 (the date of the first U.S. kink point) would have benefited from the eventual campaign against malaria; and that first kink should be moved to the left, where it would match the data better and produce a lower p value. Or perhaps that high p value of 0.39 signifies that the Bleakley (2010) model is completely wrong for the United States, and that forces other than malaria eradication drove the South’s catch-up on income.

Now, in addition to the four measures of income studied above–one for each country—the Bleakley (2010) paper looks at eight other outcomes. Six are literacy and years of schooling completed, tracked separately in Brazil, Colombia, and Mexico. In addition, there is, for Brazil, earned income—as distinct from total income (“earned” meaning earned through work). And there is, for the United States, Duncan’s Socioeconomic Index (SEI), which blends the occupational income score, explained in my last post, with information about a person’s education level. Your Duncan’s SEI is highest if you hold what is typically a high-paying job (as with the occupational income score) and you have a lot of education.

The first public version of the Bleakley study makes graphs for the additional eight outcomes too. But the final, journal-published version drops them, perhaps to save space. Since for me, the graphs are so central, I generated my own graphs for the other eight outcomes:

These figures hand us a mixed bag. In the United States, the trend on Duncan’s index appears to bend as predicted at the first allowed kink (p = 0.04) but not the second. Seemingly, relative income gains continued in the South well after malaria eradication could cause them. In Brazil, while relative progress on earned income slows when expected (second kink, p = 0.04), it does not appear to accelerate when expected (first kink), perhaps owing to small samples in the early years. In none of the Latin countries does relative progress on adult literacy or years of schooling slow with much statistical significance at the expected time (second kink points in bottom six graphs). The trend bends in all six at the first kink point, and with statistical significance—but the wrong way in Mexico.

In fact, the mixed bag partly corroborates Bleakley (2010), which also questions whether rolling back malaria increased schooling. The new results depart from Bleakley (2010) in also questioning the benefit for literacy. And they cast some doubt on the income impact in the United States. In both the U.S. plots—in the upper-left of the last two sets of graphs above—it’s clear that the income gap between the South and the rest narrowed over many decades. It’s less clear that it did so with a rhythm attributable to the malaria eradication effort of the 1920s.

Conclusion

For me, this reanalysis triggers a modest update to my understanding of the impacts of malaria prevention. With regard to adult income in Latin America, and perhaps the United States, the Bleakley (2010) theory withstands reexamination. It holds up less well for literacy, but this is not very surprising given that Bleakley (2010) also did not find clear impacts on schooling.

I wouldn’t say that my confirmation proves that malaria eradication campaigns in the Americas boosted income in the way that a large-scale randomized study might. But then neither, if you read him closely, does Bleakley. Rather, the evidence “indicates” impact. The theory that malaria eradication in the Americas increased earnings fits pretty well to the data we have. And that is probably about as much certainty as we can expect from this historical analysis.

Much of the data and code for this study are here (2 GB). Because of IPUMS licensing limitations, the download leaves out the census data for Brazil, Colombia, and Mexico. The included “read me” file explains how to obtain this data. The full write-up is here.

Notes   [ + ]

1. ↑ Bleakley (2010) also chose these countries because they had malarial and non-malarial regions, allowing comparisons. See Bleakley (2010), note 6. For sample maps see this. 2. ↑ Actually we registered a plan for the hookworm study too, but the malaria plan was better informed—and better followed—precisely because it came on the heels of the similar hookworm reanalysis. For brevity, I skipped this theme in my blog post. I did write about it in the hookworm working paper. 3. ↑ For cross-country comparability, Bleakley (2010) normalizes the malaria mortality and ecology indexes so that the 5th- and 95th-percentile geographic units—Wyoming and Mississippi in the U.S. case—score 0 and 1. Income proxies are taken in logs. 4. ↑ These graphs incorporate all of Bleakley’s control variables. In my hookworm post, I began both results sections with “basic” graphs that did not include all the controls, imitating Bleakley (2007). In contrast, all the Bleakley (2010) graphs incorporate full controls. So I do the same. function footnote_expand_reference_container() { jQuery("#footnote_references_container").show(); jQuery("#footnote_reference_container_collapse_button").text("-"); } function footnote_collapse_reference_container() { jQuery("#footnote_references_container").hide(); jQuery("#footnote_reference_container_collapse_button").text("+"); } function footnote_expand_collapse_reference_container() { if (jQuery("#footnote_references_container").is(":hidden")) { footnote_expand_reference_container(); } else { footnote_collapse_reference_container(); } } function footnote_moveToAnchor(p_str_TargetID) { footnote_expand_reference_container(); var l_obj_Target = jQuery("#" + p_str_TargetID); if(l_obj_Target.length) { jQuery('html, body').animate({ scrollTop: l_obj_Target.offset().top - window.innerHeight/2 }, 1000); } }

The post Revisiting the evidence on malaria eradication in the Americas appeared first on The GiveWell Blog.

Summary

• Two of GiveWell’s top charities fight malaria in sub-Saharan Africa.
• GiveWell’s valuations of these charities place some weight on research by Hoyt Bleakley on the impacts of malaria eradication efforts in the American South in the 1920s and in Brazil, Colombia, and Mexico in the 1950s.
• I reviewed the Bleakley study and mostly support its key findings: the campaigns to eradicate malaria from Brazil, Colombia, and Mexico, and perhaps the American South as well, were followed by accelerated income gains for people whose childhood exposure to the disease was reduced. The timing of these events is compatible with the theory that rolling back malaria increased prosperity. Full write-up here.

Introduction

I blogged three weeks ago about having reviewed and reanalyzed Hoyt Bleakley’s study of the effort in the 1910s to rid the American South of hookworm disease (not malaria). That study, published in 2007, seems to show that the children who benefited from the campaign attended school more and went on to earn more as adults.

For GiveWell, Bleakley’s 2010 study is to malaria parasites as his 2007 study is to intestinal worms. Like the 2007 paper, the 2010 one looks back at large-scale, 20th-century eradication campaigns in order to estimate impacts on schooling and adult income. It too produces encouraging results. And it has influenced GiveWell’s recommendations of certain charities—the Against Malaria Foundation and Malaria Consortium’s seasonal malaria chemoprevention program.

Because GiveWell had already invested in replicating and reanalyzing Bleakley (2007), and because the two papers overlap in data and method, I decided to do the same for Bleakley (2010). And here the parallel between the two papers breaks down: having run the evidence through my analytical sieve, my confidence that eradicating malaria boosted income is substantially higher than my confidence that eradicating hookworm did. I’m a bit less sure that it did so in the United States than in Brazil, Colombia, and Mexico; but the Latin American experience is probably more relevant for the places in which our recommended charities work.

This post will walk through the results. For details, see the new working paper. Because my malaria reanalysis shares so much with the hookworm one, I have written this post as if you read the last one. If you haven’t, please do that now.

How the malaria analysis differs from the hookworm one

Having just emphasized the commonality between Bleakley’s hookworm and malaria eradication studies—and my reanalyses thereof—in order to orient you, I should explain how the two differ:

• The hookworm study is set exclusively in the American South, while the malaria study looks at efforts in four countries. In the United States in the 1920s, no doubt inspired by the previous decade’s success against hookworm, the Rockefeller Foundation and the U.S. Public Health Service promoted a large-scale program to drain swamps and spray larvicides, which cut malaria mortality in the South by 60%. Then in the 1950s, with the discovery of DDT, the World Health Organization led a worldwide campaign against the disease. Partly because of data availability, Bleakley (2010) studies the consequences in Brazil, Colombia, and Mexico.1Bleakley (2010) also chose these countries because they had malarial and non-malarial regions, allowing comparisons. See Bleakley (2010), note 6. For sample maps see this. jQuery("#footnote_plugin_tooltip_1").tooltip({ tip: "#footnote_plugin_tooltip_text_1", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });
• Where the hookworm study groups data two ways—first by place of residence to study short-term effects, then by place of birth to study long-term effects—the malaria study does only the latter.
• I pre-registered my analysis plan for the malaria study with the Open Science Framework and hewed to it. While I did not allow the plan to bind my actions, it serves to disclose which analytical tactics I settled on before I touched the data and could know what results they would produce.2Actually we registered a plan for the hookworm study too, but the malaria plan was better informed—and better followed—precisely because it came on the heels of the similar hookworm reanalysis. For brevity, I skipped this theme in my blog post. I did write about it in the hookworm working paper. jQuery("#footnote_plugin_tooltip_2").tooltip({ tip: "#footnote_plugin_tooltip_text_2", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });
• The Bleakley malaria paper appeared in a journal published by the American Economic Association (AEA), which requires its authors to post data and computer code on the AEA website. This aided replication and reanalysis. Unfortunately, as appears to be the norm among AEA journals, the Bleakley (2010) data and code only reproduce the paper’s tables, not the graphs that in this case I see as central.
• For Brazil, Colombia, and Mexico, I mostly relied on that publicly posted data for the crucial information on which regions within a country had the most malaria, rather than trying to construct those variables from old maps and books in Spanish and Portuguese. I also relied on the public data for geographic control variables. I think it can be valuable to go back to primary sources, but for the time being at least, this step looked too time-consuming. I did update and expand the Latin outcome data, on such things as literacy and income, because it is already conveniently digitized in IPUMS International. And I reconstructed all the U.S. data from primary sources, simply by copying what we assembled for the hookworm reanalysis.

Results

In showing you what I found, I’ll follow nearly the same narrative as in my previous post’s section on the “long-term impact on earnings.” To start, here is a key graph from the Bleakley (2010) paper—or really four graphs. In each country’s graph, as with the hookworm graphs, each dot shows the association between historical disease burden in a state (or municipio) and the average income in adulthood of people born there in a given year. In all but Colombia, the leftmost dots line up with the negative range on the vertical axis, meaning that, initially, coming from a historically malarial area stunted one’s income. For example, some of the early U.S. dots are around –0.1 on the vertical axis, which means that being native to swampy Mississippi instead of arid Wyoming cut one’s adult earnings by about 10%.3For cross-country comparability, Bleakley (2010) normalizes the malaria mortality and ecology indexes so that the 5th- and 95th-percentile geographic units—Wyoming and Mississippi in the U.S. case—score 0 and 1. Income proxies are taken in logs. jQuery("#footnote_plugin_tooltip_3").tooltip({ tip: "#footnote_plugin_tooltip_text_3", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); The dots later rise, suggesting that the liability of coming from malarial areas faded, and even reversed. In Colombia, the dots start around zero but also then rise.

As in the hookworm study, here, Bleakley (2010) superimposes on the dots the step-like contour representing how malaria eradication is expected to play out in the data. The steps reach their full height when the campaigns are taken to have started—1920 in the United States and 1957 in the Latin nations. All babies born after these points were alike in that they grew up fully in the post–eradication campaign world. The step contours begin their rises 18 years earlier, when the first babies were born who would benefit from eradication at least a bit by their 18th birthdays.4These graphs incorporate all of Bleakley’s control variables. In my hookworm post, I began both results sections with “basic” graphs that did not include all the controls, imitating Bleakley (2007). In contrast, all the Bleakley (2010) graphs incorporate full controls. So I do the same. jQuery("#footnote_plugin_tooltip_4").tooltip({ tip: "#footnote_plugin_tooltip_text_4", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

Next is my closest replication of the key Bleakley (2010) graphs. These use Bleakley’s data, as posted, but not Bleakley’s computer code, since that was not posted:

The next version adds the latest rounds of census data from the Latin nations and the newer, larger samples from old census rounds for the United States. It also redefines childhood as lasting 21 instead of 18 years, because I discovered that the Bleakley (2010) code uses 18 but the text uses 21. That budges the first dashed lines back by three years:

I avoided superimposing step contours on these data points because I worried that it would trick the brain into thinking that the contours fit the data better than they do. But whether the step contour fits the plots above is exactly what you should ask yourself now. Does it seem as if the dots rise, or rise more, between each pair of vertical, dashed lines? I could see the answer being “yes” for all but Mexico. And that could be a fingerprint of malaria eradication.

I ask that question more formally in the next quartet, fitting line segments to successive ranges of the data. The dots in the four graphs are the same as above, but I’ve taken away the grey confidence intervals for readability. The p values in the lower-left of each pane speak to whether any upward or downward bends at the allowed kink points are statistically significant, i.e., hard to ascribe to chance alone. Where the p values are low—and they mostly are, even in Mexico—they favor the Bleakley (2010) reading that rolling back malaria raised incomes.

In Brazil, Colombia, and Mexico, this statistical test is fairly confident that red lines bend upward at the first kinks (p = 0.00 for Brazil and Colombia and 0.07 for Mexico). That is: in high-malaria areas, relative to low-malaria areas, as the first babies were born who could benefit in childhood from eradication, future incomes rose. The test is less confident for the United States, where the first allowed kink, in 1899, gets a high-ish p value of 0.39. However, the U.S. trend clearly bends upward—just earlier than predicted by the Bleakley (2010) theory. That might mean that the Bleakley (2010) theory is slightly wrong: maybe when it came to impacts on future earnings, malaria exposure continued to matter into one’s twenties, at least in the United States 100 years ago. Then, people born in the South even a bit before 1899 (the date of the first U.S. kink point) would have benefited from the eventual campaign against malaria; and that first kink should be moved to the left, where it would match the data better and produce a lower p value. Or perhaps that high p value of 0.39 signifies that the Bleakley (2010) model is completely wrong for the United States, and that forces other than malaria eradication drove the South’s catch-up on income.

Now, in addition to the four measures of income studied above–one for each country—the Bleakley (2010) paper looks at eight other outcomes. Six are literacy and years of schooling completed, tracked separately in Brazil, Colombia, and Mexico. In addition, there is, for Brazil, earned income—as distinct from total income (“earned” meaning earned through work). And there is, for the United States, Duncan’s Socioeconomic Index (SEI), which blends the occupational income score, explained in my last post, with information about a person’s education level. Your Duncan’s SEI is highest if you hold what is typically a high-paying job (as with the occupational income score) and you have a lot of education.

The first public version of the Bleakley study makes graphs for the additional eight outcomes too. But the final, journal-published version drops them, perhaps to save space. Since for me, the graphs are so central, I generated my own graphs for the other eight outcomes:

These figures hand us a mixed bag. In the United States, the trend on Duncan’s index appears to bend as predicted at the first allowed kink (p = 0.04) but not the second. Seemingly, relative income gains continued in the South well after malaria eradication could cause them. In Brazil, while relative progress on earned income slows when expected (second kink, p = 0.04), it does not appear to accelerate when expected (first kink), perhaps owing to small samples in the early years. In none of the Latin countries does relative progress on adult literacy or years of schooling slow with much statistical significance at the expected time (second kink points in bottom six graphs). The trend bends in all six at the first kink point, and with statistical significance—but the wrong way in Mexico.

In fact, the mixed bag partly corroborates Bleakley (2010), which also questions whether rolling back malaria increased schooling. The new results depart from Bleakley (2010) in also questioning the benefit for literacy. And they cast some doubt on the income impact in the United States. In both the U.S. plots—in the upper-left of the last two sets of graphs above—it’s clear that the income gap between the South and the rest narrowed over many decades. It’s less clear that it did so with a rhythm attributable to the malaria eradication effort of the 1920s.

Conclusion

For me, this reanalysis triggers a modest update to my understanding of the impacts of malaria prevention. With regard to adult income in Latin America, and perhaps the United States, the Bleakley (2010) theory withstands reexamination. It holds up less well for literacy, but this is not very surprising given that Bleakley (2010) also did not find clear impacts on schooling.

I wouldn’t say that my confirmation proves that malaria eradication campaigns in the Americas boosted income in the way that a large-scale randomized study might. But then neither, if you read him closely, does Bleakley. Rather, the evidence “indicates” impact. The theory that malaria eradication in the Americas increased earnings fits pretty well to the data we have. And that is probably about as much certainty as we can expect from this historical analysis.

Much of the data and code for this study are here (2 GB). Because of IPUMS licensing limitations, the download leaves out the census data for Brazil, Colombia, and Mexico. The included “read me” file explains how to obtain this data. The full write-up is here.

Notes   [ + ]

1. ↑ Bleakley (2010) also chose these countries because they had malarial and non-malarial regions, allowing comparisons. See Bleakley (2010), note 6. For sample maps see this. 2. ↑ Actually we registered a plan for the hookworm study too, but the malaria plan was better informed—and better followed—precisely because it came on the heels of the similar hookworm reanalysis. For brevity, I skipped this theme in my blog post. I did write about it in the hookworm working paper. 3. ↑ For cross-country comparability, Bleakley (2010) normalizes the malaria mortality and ecology indexes so that the 5th- and 95th-percentile geographic units—Wyoming and Mississippi in the U.S. case—score 0 and 1. Income proxies are taken in logs. 4. ↑ These graphs incorporate all of Bleakley’s control variables. In my hookworm post, I began both results sections with “basic” graphs that did not include all the controls, imitating Bleakley (2007). In contrast, all the Bleakley (2010) graphs incorporate full controls. So I do the same. function footnote_expand_reference_container() { jQuery("#footnote_references_container").show(); jQuery("#footnote_reference_container_collapse_button").text("-"); } function footnote_collapse_reference_container() { jQuery("#footnote_references_container").hide(); jQuery("#footnote_reference_container_collapse_button").text("+"); } function footnote_expand_collapse_reference_container() { if (jQuery("#footnote_references_container").is(":hidden")) { footnote_expand_reference_container(); } else { footnote_collapse_reference_container(); } } function footnote_moveToAnchor(p_str_TargetID) { footnote_expand_reference_container(); var l_obj_Target = jQuery("#" + p_str_TargetID); if(l_obj_Target.length) { jQuery('html, body').animate({ scrollTop: l_obj_Target.offset().top - window.innerHeight/2 }, 1000); } }

The post Revisiting the evidence on malaria eradication in the Americas appeared first on The GiveWell Blog.