Published: May 2016
[Added December 19, 2016: GiveWell's experimental work is now known as GiveWell Incubation Grants.]
Note: this page summarizes the rationale behind a grant to Evidence Action made by Good Ventures. Evidence Action staff reviewed this page prior to publication.
Summary
As part of GiveWell’s general effort to support the creation of future top charities, in March of 2016 Good Ventures granted $812,351 to Evidence Action to support the development of additional programs. Evidence Action intends to use these funds to develop and implement a scaled-up version of its No Lean Season program in Bangladesh.
Good Ventures made a grant in March 2014 in support of Evidence Action's No Lean Season program.
We spoke with Karen Levy and Guillaume Kroll about updates on this grant on August 2, 2016.
Table of Contents
The intervention
Evidence Action's No Lean Season offers subsidies (grants or loans) to low-income agricultural workers in Bangladesh as an incentive to migrate during the famine season to urban areas where higher wages can be earned.1 The Open Philanthropy Project previously completed a shallow investigation into seasonal migration within low-income countries.
Grant details
As part of GiveWell's experimental efforts to support the development of future top charities, Good Ventures granted $812,351 to Evidence Action to fully cover the projected No Lean Season costs in 2016.2 Good Ventures previously supported this project with grants in 2014 and 2015. Evidence Action expects to use this grant to:3
- Build and improve the No Lean Season program design.
- Create a model for implementing the program at scale.
- Develop a better estimate of delivery costs.
- Test the modified program by hiring staff and implementing it at a scale of 16,000 households.4
Track record
Evidence Action began supporting No Lean Season in 2013. We have previously evaluated Evidence Action for the Deworm the World Initiative and found it to be a strong organization.
No Lean Season builds upon a number of randomized controlled trials on incentives for seasonal migration in northern Bangladesh:
- An initial trial in 2008, published in Bryan, Chowdhury, and Mobarak 2014.
- A 2009 follow-up to the previous trial, also published in Bryan, Chowdhury, and Mobarak 2014.
- A 2011 trial conducted in the lesser lean season, also published in Bryan, Chowdhury, and Mobarak 2014.
- A 2013 trial, results from which are still being analyzed and have not been published.
- A 2014 trial, not yet published but with preliminary results discussed in No Lean Season - Preliminary 2014 Results.
Available evidence about the effect of subsidy offers on household income and expenditures in the lean season comes from the 2008 and 2014 trials. Both studies found significant effects of subsidy offers, but they measured different outcomes: the 2008 trial reported consumption and the 2014 trial reported income.5 This makes it difficult to directly compare effect sizes or to combine the results via meta-analysis.
Cost-effectiveness
Since our last grant to Evidence Action for No Lean Season, we have spent considerably more time on our cost-effectiveness analysis.6 Our current best guess is that No Lean Season is about three to seven times as cost-effective as unconditional cash transfers, though there is significant uncertainty about this figure. Our cost-effectiveness estimates have changed considerably as we have learned more about the program, and we think it is possible that our estimates could continue to change significantly in the next year.
Room for more funding
Based on conversations with Evidence Action, we believe its reserves of unrestricted funding are modest and that it would not be able to support No Lean Season itself, given its current priorities. In addition, Dr. Karen Levy, the Director of Global Innovation and Beta for Evidence Action, told us that she did not see any other opportunities to raise the 2016 funding by the end of March.7
Evidence Action is also seeking longer-term funding for No Lean Season. Dr. Levy told us that Evidence Action may be able to raise some of the 2017 funding from other funders, but it was not certain about this.8 We considered making this grant larger to fully fund the projected costs of No Lean Season through 2017; Evidence Action told us that the additional secured funds would enable it to make more attractive job offers when recruiting staff.9 We consider this a legitimate downside to our decision but not a major one.
Risks of the grant
We see a number of potential risks to the success of this grant:
- Cost-effectiveness: As stated above, we still maintain significant uncertainty about the cost-effectiveness of No Lean Season. Our estimate has shifted considerably as we have learned more about the program, and it is possible that we decide in the future that the program is not cost-effective enough to lead to a top charity recommendation.
- Implementation risk: Thus far, No Lean Season has only implemented this program at a scale of about 6,000 households,10 but it hopes to reach 16,000 households in 2016 and 70,000 the following year.11 It seems plausible that the program could encounter new difficulties as it tries to scale up. To reach its goals this year, Evidence Action and its local implementing partner will have to design the program to work at a larger scale and hire and train staff before the lean season. Dr. Levy estimated about a 2-in-3 chance that its partner is able to offer this program during the 2016 lean season, and about an 85% probability that it is able to in either 2016 or 2017.12
- External validity: While the No Lean Season program has appeared to work well in Bangladesh so far, it is possible that its success would not translate elsewhere. If the program failed to work outside of Bangladesh, its room for more funding could be relatively limited. Evidence Action has done some preliminary work that found that Malawi and Zambia were not promising locations for the program.13 Evidence Action is considering investigating whether Indonesia or eastern India are suitable contexts for No Lean Season.14
Plans for follow-up
We expect to receive updates from Evidence Action every 3-6 months for the next year about this grant; we plan to publish summaries of these updates when warranted. We expect to pay particular attention to these questions in following up:
- How cost-effective is the scaled-up program?
- Did implementation go as expected/hoped? Were there any major surprises?
- Are other funders interested in the program?
- How promising are other potential locations for No Lean Season?
- Should we fund No Lean Season’s activities in 2017?
In addition, we have some remaining questions about the 2014 study that we intend to do more work on after making this grant.
In the long term, we hope that No Lean Season will develop into a GiveWell top charity. We have been working on "seeding" potential top charities for more than two years, and opportunities as promising as this seem relatively rare; No Lean Season has the potential to be an evidence-backed, cost-effective program implemented by an outstanding organization that we have successfully evaluated in the past (for the Deworm the World Initiative).
We also hope to learn from the experience of seeding an effective charity in a way that helps us to repeat the process.
Internal forecasts
We’re experimenting with recording explicit numerical forecasts of events related to our decisionmaking (especially grantmaking). The idea behind this is to pull out the implicit predictions that are playing a role in our decisions, and make it possible for us to look back on how well-calibrated and accurate those are. For this grant, we are recording the following forecasts:
- Top charity predictions:
- No Lean Season (or a related organization) is a top charity in 2017: 15%.
- No Lean Season (or a related organization) is a top charity in 2018: 20%.
- No Lean Season (or a related organization) is a top charity in 2019: 25%.
- Cost-effectiveness predictions:
- Our 2016 cost-effectiveness estimate for No Lean Season is at least five times as good as cash transfers: 50%.
- Our 2016 cost-effectiveness estimate for No Lean Season is less than twice as good as cash transfers: 15%.
- Our 2016 cost-effectiveness estimate for No Lean Season is at least ten times as good as cash transfers: 15%.
- Implementation predictions:
- Evidence Action is running a No Lean Season program at the end of 2017: 80%.
Our process
Since our last grant in 2015, we have had multiple conversations with Evidence Action to understand the evidence for the program and its plans for future work. We also put significantly more work into our estimate of the cost-effectiveness of No Lean Season.
Sources
- 1
"Seasonal income insecurity is a problem in many regions of the world for the very poor during the period between planting food crops and harvest time. For example, northern Bangladesh is at risk of significant income insecurity during three months leading to the winter harvest that affects very poor people especially. Researchers working in this region identified a simple and effective solution that took advantage of the relative abundance of employment opportunities outside of the famine-prone north during this lean season.They provided households a travel subsidy for work-migration during the lean season, allowing them to send a member away to generate income that would otherwise not have been possible. This resulted in significant improvements in household welfare (including consumption and nutrition) during the lean season, an effect that held even in subsequent years. Based on this evidence, providing these travel subsidies is a promising way to avert seasonal insecurity." Evidence Action Beta Homepage.
- 2
See No Lean Season - January 2016 Update, Table 1, Pg 5.
- 3"We now propose to undertake the 'RCT at scale' in Year 2 (2017), and spend Year 1 (2016) focusing on building and improving program design, developing a solid implementation model, ground-truthing delivery cost estimates, and fine-tuning our projections through on-the-ground experimentation in 2016. Year 1 activities would involve a larger number of beneficiaries than previous research studies, but still far lower than the high-level target we hope to reach at scale. The idea here is to 'learn by doing' by co-developing the program components in fifteen branch offices, and to closely monitor the process to identify any potential challenges in delivery." No Lean Season - January 2016 Update, Pg 1.
- 4 See No Lean Season - January 2016 Update, Table 1, Pg 5.
- 5
- "In terms of magnitude of effects, monthly consumption among migrant families increases by about $5 per person, or $20 per household, due to induced migration. Our survey only asked about expenditures during the second month of monga, and the modal migrant in our sample had not yet returned home (which includes cases where they may have returned once, but left again). We therefore expect the effects to persist for at least another month, and the total expenditure increase therefore easily exceeds the amount of the treatment ($8.50). Furthermore, if households engage in consumption smoothing, then some benefits may persist even further in the future. In any case, the $8.50 is spent on transportation costs two months prior to the consumption survey." Bryan, Chowdhury, and Mobarak 2014, Pg 1689.
- No Lean Season - Preliminary 2014 Results, Pg 3.
- 6
Full Excel model available here: GiveWell's spring 2016 cost-effectiveness analysis of No Lean Season.
- 7
"By the end of March 2016, Evidence Action will need to have secured funding at least to cover the balance of year 1... Currently there are no funders other than Good Ventures that can make a decision on funding year 1 quickly enough for the program's planned schedule." GiveWell's non-verbatim summary of a conversation with Karen Levy on February 4, 2016.
- 8
"Other donors have expressed some interest in longer-term support. Evidence Action plans to approach other donors for funding years 2-4, with the latter two years contingent on the program's performance in year 2." GiveWell's non-verbatim summary of a conversation with Karen Levy on February 4, 2016.
- 9
"By the end of March 2016, Evidence Action will need to have secured funding at least to cover the balance of year 1. Ideally, it will have secured funding for year 2 as well, which will enable Evidence Action to plan and hire staff for two years at a time." GiveWell's non-verbatim summary of a conversation with Karen Levy on February 4, 2016.
- 10
See the 4th experiment in No Lean Season Timeline.
- 11
See No Lean Season - January 2016 Update, Table 1, Pg 5.
- 12Unpublished conversation with Karen Levy on February 23, 2016.
- 13
From Seasonal Migration Feasibility Analysis: Malawi and Zambia:
- "The main conclusion from fieldwork across Malawi is that there are insufficient potential migration destinations to absorb excess labor from rural areas." Pg 6.
- "The main conclusion from fieldwork across Zambia was that potential employment destinations are difficult to find but do exist, with the most promising area being rural-rural migration onto commercial farms." Pg 11.
- 14
"Professor Mushfiq Mobarak, a lead researcher on No Lean Season and a member of Evidence Action's advisory board, believes that Indonesia might be a good candidate, and Dr. Levy is investigating some of the states in India closest to northern Bangladesh (e.g. Bihar, Chhattisgarh, and Jharkhand)." GiveWell's non-verbatim summary of a conversation with Karen Levy on February 4, 2016.