We have published a more recent version of this page. See our most recent version of this page.

Published: November 2018

Background

GiveDirectly, one of our top charities, provides unconditional cash transfers to very poor households in Kenya, Uganda, and Rwanda.

In this May 2018 post, we explained that we were aware of recently released research which suggested that cash transfers may be less effective than we previously believed in two ways:

1. Cash may have negative spillovers on non-recipient households who live close to recipient households.
2. The benefits of cash for recipients may fade quickly over time.

We have now conducted a more thorough literature review on the spillover effects of cash transfers, the first of these two concerns. We plan to address the concern about the duration of benefits at a later date because we believe it has less potential to substantially affect our cost-effectiveness analysis for cash transfers, for reasons that are explained in this post.1

We have previously not included positive or negative spillover effects of cash transfers in our cost-effectiveness analysis model. We now build a small negative spillover effect into our model for the reasons that are explained in the rest of this report.2

Studies we considered

There are many studies and outcomes we could have considered in our spillovers analysis; at this stage, we have focused our analysis in two major ways:

1. We focus only on spillover effects on consumption because we believe effects on consumption are the most relevant to assess the welfare impact of cash transfers.3
2. We focus on the five RCTs on spillover effects we are aware of that a) study unconditional cash transfers that b) took place in contexts similar to where GiveDirectly operates (very poor settings in sub-Saharan Africa). For details of these RCTs, see below and the following footnote.4

For studies that we considered but deemed less relevant (either because they did not look at consumption, studied conditional cash transfers, or studied cash transfers in a context outside of sub-Saharan Africa), see the following footnote.5

Why might cash transfers have spillover effects on consumption?

In theory, there are several mechanisms by which cash transfers may have negative spillover effects on consumption, such as: price increases, harmful psychological effects on non-recipients, increased competition in the labor market or in business, inefficient sale of productive assets by non-recipients, and by discouraging wage-earning work amongst recipients. For more details on each of these, see the following footnote.6

It is also possible that cash transfers could have positive spillover effects on consumption, such as by providing economic stimulus in the relevant regions.7

What does the academic literature find about spillover effects on consumption?

The evidence base for spillover effects is mixed: of the five RCTs which look at the spillover effects of unconditional cash transfers on consumption in sub-Saharan African countries, three find substantial negative spillover effects, one finds no spillover effects, and the GE study finds no or even a small positive spillover effect. The GE study is much larger than the other studies and has fewer methodological limitations, but the large magnitude of the negative findings in some studies is concerning.

We summarise some of the key features of each study in the following table, but for more details and sources see this spreadsheet. Note that at least four out of five papers below estimate consumption by measuring household expenditure,8 which may be an imperfect proxy for true "consumption" (i.e., actual goods consumed) because prices may change.

Paper	Within or across-village spillover?	Effect on consumption for non-recipient households (as a % consumption)	Average size of cash transfer (nominal USD)	Proportion of households treated within treatment villages	Sample size for analysis (number of households)
GE study	Across	1%	1000	33%	8237
GE study	Within	7%*	1000	33%	2815
Haushofer and Shapiro 2018 (3 year follow-up)	Within	-16%***	490	9%	830
Haushofer and Shapiro 2016 (9 month follow-up)	Within	-4%	490	9%	901
McIntosh and Zeitlin 2018	Within	-12% (statistical significance unknown)	532	11%	966
Haushofer, Reisinger and Shapiro 2015	Within	-14%*	490	9%	939

(Note: stars denote statistical significance. * = 10% level, ** = 5% level, *** = 1% level. The Haushofer, Reisinger and Shapiro 2015 estimates are based on a very different identification strategy to the other within-village spillover estimates and so are less comparable. For more details, see this spreadsheet).

We provide a brief summary of the results in the academic literature in the following bullet points; for more detail see this spreadsheet. As mentioned previously, we focus on five RCTs which study spillover effects on consumption from unconditional cash transfers in sub-Saharan African countries. Though, note that three of these studies (Haushofer and Shapiro 2016, Haushofer and Shapiro 2018, and Haushofer, Reisinger and Shapiro 2015) were conducted on the same study population, so should not be thought of as fully independent data points.

• GiveDirectly's General Equilibrium study. This study estimates both across-village and within-village spillover effects using a large-scale RCT covering 653 villages.9 For methodological details of how spillover effects are estimated, see the following footnote.10 The study finds:
  • No evidence of negative spillover effects on household consumption across villages.
  • Evidence of a small positive spillover effect on household consumption within-villages, approximately 7 percent of the control group mean.
  • No evidence of across village spillover effects on household asset ownership or subjective well-being. They find a positive spillover effect on subjective well-being of ineligible households within treatment villages.
  • No evidence of effects on prices, and those null effects are precise.11

  The GE study was co-authored by Paul Niehaus, Chairman of GiveDirectly. However, given that a pre-analysis plan was submitted, and three of the four co-authors do not have an affiliation with GiveDirectly, we have no reason to believe that GiveDirectly's involvement altered the analysis undertaken. In addition, the GE study team informed us that Paul Niehaus recused himself from final decisions about what the team communicated to GiveWell.12

• Haushofer and Shapiro 2018. This study estimates within-village spillover effects using an RCT which covers 120 villages, and is a follow-up three years after the cash transfers began.13 Within-village spillover effects are identified by comparing the consumption of control households in treatment villages to the consumption of households in control villages.14 This study suffers from two important methodological limitations, which are described briefly in the following footnote and have been discussed at length in the blog posts linked to from this post.15 The study finds a negative spillover effect on household non-durable consumption equivalent to 16% of the mean for households in control villages, statistically significant at the 1% level. The effect on consumption is robust to the authors' attempts to deal with the two methodological limitations mentioned above.16
• Haushofer and Shapiro 2016. This study is part of the same RCT as Haushofer and Shapiro 2018, but is the first follow-up and took place nine-months after the cash transfers began.17 The authors find no evidence of spillover effects on consumption, though these null effects are not precisely estimated.18
• McIntosh and Zeitlin 2018. This study is an RCT which looks at the impact of small and large cash transfers as well as a bundled nutrition and sanitation program; methodological details in footnote.19 We focus on the effects of the large cash transfers. The authors do not directly estimate spillover effects, but the spillover effects can be backed out from the estimates in their paper, which economist Berk Ozler does in this blog post. He finds that ineligible households within treatment villages experience a 12% decrease in consumption, though the statistical significance of this effect is not tested and the study is underpowered to detect spillover effects because relatively few ineligible households are sampled.20
• We have previously reviewed Haushofer, Reisinger and Shapiro 2015 in detail here. In this RCT, which also relies on the same sample as the Haushofer and Shapiro studies discussed above, the authors find that an increase in mean village wealth by US$354 (the average change in the sample) is associated with a decrease in household non-durable consumption by approximately 14%, an effect that is statistically significant at the 10% level.21

Some of our takeaways are:

• The three negative studies find reductions in consumption of roughly 0-20 percentage points per non-recipient household, for non-recipient households who reside within the same village as the recipients during the first year or two after cash transfers are made.22 However, it is difficult to directly compare these results across papers due to major differences in the cash programs studied and the methodologies used.23 These studies suffer several methodological issues, and we are uncertain what mechanisms drive the negative spillover effects.
• The General Equilibrium study by contrast finds no evidence of negative spillover effects on consumption across or within villages. This study is much larger than the other studies and has fewer methodological limitations. However, the results are in private draft form and we do not have a strong understanding of the mechanisms explaining its findings.

Can we apply the results from the academic literature to estimate the spillover effects in GiveDirectly's program?

We attempted to combine the results from the academic literature and create a model of the magnitude of possible negative spillover effects (see footnote).24 However, we did not feel comfortable relying on this model because we lack basic information on a number of key parameters, such as how many non-recipient households may be affected by spillover effects for each treated household and how the magnitude of spillover effects changes with distance. We hope that future academic research might have more to say on these policy-relevant questions.

Example of the challenge we face

To illustrate the challenge of applying the academic results discussed above to GiveDirectly’s current program, consider the following example:

• There is an estimate from an RCT for the magnitude of spillover effects for the average non-recipient ineligible household that lives in a village where about 11% of all households receive a $500 transfer.
• We want to use that finding to estimate the spillover effects on non-recipient households (both eligible and ineligible) that live outside of a village where about 100% of households receive a $1,000 transfer.
• In order to make such an extrapolation, we would need an estimate for at least two quantities: 1) how the size of spillover effects change as one goes from within a village to outside of a village at various distances, and 2) how many non-recipient households live at various distances. So far, we have not seen this key information in the relevant literature.
• Note that this description leaves out several other factors that would also need to be estimated, discussed below. Also note that it provides an example of how to apply an estimate from one RCT, though we actually have estimates from multiple RCTs.

Policy-relevant questions that remain unanswered

In order to create an explicit model of spillover effects that we would feel comfortable relying on, we would need higher quality information on a variety of parameters, such as:

• How do spillover effects decline with distance? In the context of GiveDirectly's program, what we would ideally like to know is: for each of a given set of distance bands away from a treated village (e.g. 0-1km, 1-2km, 2-3km, and further), how many non-recipient households are there, and how does the size of spillover effects decline across those bands (see footnote for details).25
• What are the effects of inflation on overall social welfare in GiveDirectly’s contexts, and how can such effects be measured? Price increases may decrease consumption for some households while increasing it for others (see footnote for more).26 We have not yet tried to explicitly model the net effect on social welfare for a given price increase. Additionally, expenditure-based measures of consumption (as were used in at least four of the five studies cited above)27 may not capture the full impacts of price effects on true consumption (i.e., actual goods consumed), since people may simply spend the same amount to consume less if prices rise.28 This complicates empirical analysis of spillover effects via inflation.
• To what extent do spillover effects impact recipient households too? Whilst some mechanisms for spillover effects on consumption do not apply to recipient households, as explained here,29 the direct spillover effects caused by price increases, lowering consumption for a given level of expenditure, will impact recipient households as well as non-recipients. We need a better understanding of the mechanisms behind spillover effects to understand how much they apply to recipient households.

Smaller factors that would also affect our estimate include:

• How do spillover effects change over time? Academic papers measure spillover effects at a fixed point in time, often several months or longer after cash transfers have been made. We are worried that spillover effects might be more immediate than the time at which effects are estimated in academic papers.30
• How do spillover effects change with the size of cash transfers, the proportion of households receiving cash transfers and the spatial distribution of cash transfers? We would expect spillover effects to be larger as the cash transfer size increases and as a greater proportion of households in a village receive transfers. In addition, the spatial distribution of recipient villages will affect how many treated households a given household in a non-recipient village is surrounded by, and so the strength of spillover effects. We do not have a strong sense of how much larger spillover effects would be with cash transfer programs whose distribution style differs in these ways.31
• How do spillover effects differ among eligible and ineligible households? It is possible that spillover effects differ somewhat between ineligible households (who are often richer, at least in assets) and eligible households. Since studies may sometimes only measure spillover effects in one of these populations, we must extrapolate and make guesses to understand the spillover effects on the full population.32
• How do we adjust for the fact that estimates of within-village treatment effects are underestimates of the true effect in the presence of across-village spillovers? If across-village spillover effects exist, households in control villages will also experience spillovers, such that estimates in the RCTs will underestimate the absolute size of the true within-village spillover effect.33

What is our best guess for the spillover effect on consumption in GiveDirectly's program?

Given the limitations in the available information (described above), we must make a very uncertain and qualitative best guess about the magnitude of spillover effects for GiveDirectly's current program. Our best guess is that negative or positive spillovers of GiveDirectly's program are minimal on net.34

For more information on our reasoning for this best guess, see this document. Some of the major reasons underlying this guess include:

• Our theoretical prior is that potential negative effects via price increases and potential positive effects via economic stimulus would roughly offset each other, though we have not yet constructed models for these effects so this guess is very uncertain.
• Several papers have demonstrated evidence for moderate to large negative spillover effects on consumption, as described in this section. However, we have not seen strong evidence on the mechanisms that explain these effects, reducing our confidence in the results.
• We pre-registered our belief that the GE study would play a large role in our conclusions in this post, because it is the largest and highest quality study that we have seen on spillover effects. That study finds no evidence of negative spillover effects, no effect on prices, and some evidence for small positive spillover effects within villages.35 We therefore update our best guess for negative spillover effects towards zero.
• Our explicit model of spillover effects suggests that our current best guess is in a reasonable range. However, for the reasons outlined in this section, we are highly uncertain about several key parameters in that model, and so place little confidence in its conclusions.

It is our understanding that the design of GiveDirectly's program going forward is such that the spillover effects might be different to those observed in the academic literature for two major reasons. In each case, we believe there are valid arguments that the spillover effects may be larger or smaller than observed in the academic literature:

1. In Kenya and Uganda, almost all households in GiveDirectly's target villages receive cash transfers. (In Rwanda, GiveDirectly still only provides cash transfers to eligible households to comply with government requirements there.)36
   • Providing cash transfers to all households in a village may decrease the size of negative spillover effects because it is plausible that the within-village spillover effects experienced by recipient households are smaller than those experienced by non-recipient households for some of the mechanisms discussed in this section (e.g. psychological mechanisms), and there are no longer non-recipient households within treated villages.
   • Alternatively, the size of spillover effects may increase because the intensity of the treatment is larger when all households in target villages receive transfers.
2. GiveDirectly told us that it provides transfers to most villages in a relevant region.
   • This spatial pattern seems like it would reduce the number of households in non-recipient villages who live along a border with a recipient village and therefore have a high chance of being affected by negative spillovers.37
   • However, it also means that recipient villages are surrounded on all sides by other recipient villages, which increases the likely incidence of spillover effects on households in those recipient villages, for example through price effects.

We emphasize that our conclusion at this point is very tentative, and we hope to update our views next year if there is more public discussion or research on the areas of uncertainty highlighted in our analysis.

Sources

Document	Source
Angelucci and DiGiorgi 2009	Source
Baird, DeHoop & Ozler 2013	Source
Filmer, Friedman, Kandpal & Onishi 2018	Source
GE study: Note for GiveWell	Unpublished
GiveDirectly, Dashboard Metrics for GiveWell, April 2018	Source
GiveDirectly, Dashboard Metrics for GiveWell, May 2017	Source
Haushofer and Shapiro 2016	Source
Haushofer and Shapiro 2018	Source
Haushofer, Reisinger and Shapiro 2015	Source
McIntosh and Zeitlin 2018	Source
Registration of GiveDirectly's General Equilibrium study	Source

• 1
  • In this post we wrote: "Our current estimates are consistent with assuming little medium-term benefit of cash transfers. We estimate that about 60% of a typical transfer is spent on short-term goods such as eating more food, and count this as about 40-60% of the benefits of the program. If we were to instead assume that 100% of the transfer was spent on short-term consumption (i.e., none of it was invested), our estimate of the cost-effectiveness of cash would become about 10-30% worse. We think using the 100% short-term consumption estimate may be a reasonable and robust way to model the lower bound of effects of cash given various measurement challenges."
  • The version of our cost-effectiveness analysis which assumes that 100% of the cash transfer is immediately consumed is here.

• 2

  For more information, see our cost-effectiveness model changelog.

• 3
  • Whilst some papers directly look at subjective well-being as an outcome variable, it is typically measured as a combination of many underlying psychological components whose effects on welfare are not easy to understand.
  • In addition, we choose not to build effects on multiple outcomes into our cost-effectiveness model because we are worried that effects across outcomes within a given paper may not be independent. For example, if a paper were to find an effect on both subjective well-being and consumption, we wouldn't be sure how much of the effect on subjective well-being is driven by the effect on consumption. Attempts to build both effects into our cost-effectiveness model could therefore risk double counting a single effect.

• 4

  An important part of our literature review on spillover effects is the private draft results that we have seen from GiveDirectly's General Equilibrium study, which is a large-scale RCT that estimates the within-village and across-village spillover effects of GiveDirectly's program in Kenya. We have also studied Haushofer and Shapiro 2018, McIntosh and Zeitlin 2018 and Haushofer and Shapiro 2016 in greater detail, as well as the blog posts that discuss Haushofer and Shapiro 2018's methodology and results, which are linked to in this post.

• 5
  • Angelucci and DiGiorgi 2009 and Filmer, Friedman, Kandpal & Onishi 2018 study spillovers effects of conditional cash transfers in Mexico and the Philippines respectively, whilst Baird, DeHoop & Ozler 2013 study spillover effects on mental health amongst school girls in Malawi.
  • Some details on those papers can be found in this spreadsheet.
  • To identify relevant papers, we: 1) reviewed any papers linked from relevant blog posts about spillover effects of cash transfers (several such posts linked from here), 2) had a few conversations with researchers working on these topics and asked for relevant papers. We have not done a systematic literature search, so it is possible that there are papers that would have met our inclusion criteria that we did not review. If you are aware of such papers, please send them to us.

• 6
  • Price increases. Cash transfers may lead to local price increases. We previously reviewed four RCTs, which found mixed evidence of impacts on prices. We have not recently conducted a systematic review of the effect of cash transfers on prices, so we have likely not examined some studies with relevant evidence. Price increases affect both the recipients and non-recipients of cash transfers.

    We are uncertain about the best way to model the effect of price increases on social welfare. Theoretically, we expect that some goods in GiveDirectly's target villages would be non-tradable (e.g. local services such as haircuts) and that there would be some increase in the prices of such goods, as we believe is consistent with basic economic theory. Price increases may reduce net welfare by a) effectively redistributing gains from consumers to producers (producers may be richer and fewer in number, so may have lower marginal utility from consumption), and b) causing people to substitute away from non-tradable goods as prices for such goods increases. Note that in both of these cases value is transferred to some extent, and we are unsure about the magnitude of the final welfare impact. We have not yet attempted to explicitly model the magnitude of these competing factors or rigorously assessed all of the ways that inflation could affect social welfare. Nevertheless, we would roughly guess that overall effects of inflation are minorly negative.

  • Harmful psychological effects on non-recipients. Cash transfers may harm non-recipients psychologically, for example because they feel like they have lower social status, or are envious of recipients. If this causes non-recipients to become demotivated, their productivity and so income and consumption may decrease. Related to this, we previously reviewed two RCTs which look at the effect of cash transfers on social unrest.
  • Competition in the labor market or business. Non-recipients may compete with recipients in the labor market or in business, in which case they may be put at a disadvantage when their neighbour receives a cash transfer. As a simple hypothetical example, a recipient may purchase clothes for a job interview that increases their probability of becoming employed at the expense of a non-recipient. Though, note that this concern may apply to some extent to all interventions that increase income.
  • Sale of productive assets. If prices increase for local goods, non-recipients may sell productive assets to recipients in order to raise expenditure to maintain their initial level of consumption. In the medium-term, the loss of income from the productive asset may lead to a decrease in expenditure and consumption. Haushofer and Shapiro 2018 suggest that this may explain their results:
    • "We do not have conclusive evidence of the mechanism behind spillovers, but speculate it could be due to the sale of productive assets by spillover households to treatment households, which in turn reduces consumption among the spillover group", p.3. in Haushofer and Shapiro 2018.
    • We have not yet reviewed other cash literature with this mechanism in mind to determine if there is other relevant evidence.
  • Cash transfers could discourage wage-earning work amongst recipients. If recipients have control over their work and leisure time, they may substitute work for leisure after receiving a cash transfer. The welfare gain from an increase in leisure will come at the expense of lower consumption. We previously reviewed a World Bank review and three RCTs on this question, which overall found little evidence of decreases in work hours (two RCTs find an increase in work hours).

• 7

  Examples of potential economic stimulus include:

  • An increase in demand from cash transfers may lead to an increase in revenues for local producers (e.g. because prices of the goods they sell increase or because they are able to sell a larger quantity of goods), which in turn increases their consumption also. That increase in consumption can have the same knock on effect, and so on. The size of these multiplier effects will depend on the marginal propensity of households to consume, and the proportion of household expenditure that is spent on locally produced goods.
  • If local production increases in response to the increase in demand, there may be an increase in employment of local workers, which increases their consumption in turn. We are not sure whether or by how much local producers would increase supply in response to a temporary income shock.

• 8
  • For a detailed description of the expenditure-based consumption measures in GE study: Note for GiveWell, see Appendix A3 on p.29. (This reference is for internal use since the note for GiveWell is currently confidential.)
  • For a detailed description of the expenditure-based consumption measures in Haushofer and Shapiro 2016, see p.12 in the Online Appendix. The same outcome variables are used in the other two studies based on this RCT (Haushofer, Reisinger and Shapiro 2015 and Haushofer and Shapiro 2018).
  • We have not seen a detailed description of the consumption measure used in McIntosh and Zeitlin 2018. "The study focuses on five dimensions. Here we briefly summarize each; details of the construction of these outcomes are included in Appendix A", p.15. However, Appendix A does not seem to contain this information.

• 9

  "The intervention involves roughly USD 11 million in transfers across 653 villages (328 treatment and 325 control villages) in one Kenyan county", p.2 in GE study: Note for GiveWell.

• 10
  • Groups of on average ten villages, called sublocations, are randomly assigned to low or high saturation status. In low saturation sublocations, one third of villages are treated, whilst in high saturation sublocations, two-thirds of villages are treated. Within sublocations, villages are randomly assigned to treatment or control. Within treated villages, all eligible households (defined as having a thatched roof) receive the cash transfers.
    • "Treatment assignment is randomized at the village level, and within treatment villages, all households meeting GD’s eligibility requirement receive the unconditional cash transfer. A second level of randomization provides variation in treatment intensity: sublocations, an administrative unit directly above the village level comprising of an average of ten villages, were randomly assigned to high or low saturation status. In high saturation sublocations, two-thirds of villages were assigned to treatment, while in low saturation sublocations, only one-third of villages were assigned to treatment", p.2 in GE study: Note for GiveWell.
    • "GD provides unconditional cash transfers to poor households in rural Kenya, targeting (for villages in our study) households living in homes with thatched roofs, a basic means-test for poverty. In treatment villages, GD enrolls all households in treatment villages meeting its thatched-roof eligibility criteria (“eligible” households); approximately one-third of all households are eligible", p.6 in @GE study: Note for Givewell@.
  • Within-village spillover effects are identified by comparing the consumption of ineligible households in treatment villages with ineligible households in control villages. Across-village spillover effects are identified by comparing consumption across households in high and low saturation sublocations for a given household type (for each combination of eligible or ineligible, and treated or control).
    • "We then outline our strategy for measuring externality effects, both within-village externality effects for ineligible households in treatment versus control villages, and cross-village externalities, making use of variation in treatment saturation intensity at the sublocation level", p.14 in @GE study: Note for Givewell@.
    • "To estimate within-village externalities on ineligible households, we restrict our sample to ineligible households and use Equation (1)", p.15 in @GE study: Note for Givewell@. Equation (1) is on p.14.
    • "To measure cross-village externalities, we pool data from all households and include interaction terms between households’ eligibility status, village treatment status, and sublocation saturation status", p.15 in @GE study: Note for Givewell@. See p.15 and p.16 for a detailed description of the across-village spillover effects that the authors estimate, including the regression specification in Equation (2) on p.15.

• 11

  See this spreadsheet for further details and sources.

• 12
  We were told this during a discussion with the GE study team on October 17, 2018.
  
• 13
  • Sixty villages were randomly assigned to treatment, and within those villages half of the eligible households were treated and half were not.
    • "The results reported here capture the impacts of unconditional cash transfers approximately 3 years after the transfers were sent", p.3 in Haushofer and Shapiro 2018.
    • "This study is a two-level cluster-randomized controlled trial", p.5 in Haushofer and Shapiro 2018.
    • "The research team then identified the 120 villages with the highest proportion of thatched roofs within Rarieda. Sixty villages were randomly chosen to be treatment villages (first stage of randomization)", p.5 in Haushofer and Shapiro 2018.
    • "After baseline, the research team randomly chose half of the eligible households to be transfer recipients (second stage of randomization). This process resulted in 503 treatment households and 505 control households in treatment villages at baseline", p.6 in Haushofer and Shapiro 2018.

• 14

  See equation (12) on p.24 in Haushofer and Shapiro 2018. "Here, the sample includes only non-treatment households (in treatment and control villages). Thus, β1 identifies within-village spillover effects by comparing control households in treatment villages to control households in pure control villages", p.12.

• 15
  1. Households in control villages were selected into the sample at the first endline survey rather than at baseline. The eligibility criterion for selection (that households have a thatched roof) was therefore applied to households in control villages approximately one year after it was applied to spillover households (the control households in treatment villages). During that year, some households in control villages may have upgraded from a thatched to a metal roof, meaning they were ineligible for the survey. Households in control villages are therefore a selected sample who may not be comparable to spillover households.
     • "A potential weakness in this analysis is that the thatched-roof selection criterion for participation in the study was applied to households in control villages one year after it was applied to households in treatment villages. As a result, there is endogenous selection into the pure control condition, as some proportion of households in pure control villages are likely to have upgraded to a metal roof over this time period. These households are excluded from endline in the pure control villages", p.19 in Haushofer and Shapiro 2018.
  2. Attrition rates are significantly higher for control villages than treatment villages between the first and second endlines.
     • "...there is a statistically significant difference in attrition levels for households in control villages relative to households in treatment villages from endline 1 to endline 2: 6 percentage points more pure control households were not found at endline 2 relative to either group of households in treatment villages", p.9 in Haushofer and Shapiro 2018.

• 16

  For more detail and sources, see this spreadsheet.

• 17

  "Households received the first transfer an average of 4.8 months after baseline and an average of 9.3 months before endline", p.1984 in Haushofer and Shapiro 2016

• 18

  For more detail and sources, see this spreadsheet.

• 19
  • Villages were randomly assigned to treatment arms, with a total of 34 villages assigned to large cash transfers and 74 villages assigned to the control group.
    • "Randomization occurred at the village level across 248 villages, using a blocked randomization", p.13 in McIntosh and Zeitlin 2018.
    • "74 villages were assigned to the Gikuriro intervention, 74 were assigned to the control group (no intervention), and 100 were assigned to GiveDirectly household grants. The GiveDirectly villages were further split into four transfer amounts, randomized at the village level. Three treatment amount arms, with 22 villages in each, received transfer amounts in a range around the anticipated cost of Gikuriro. A final 34 villages were assigned to the ‘large’ GiveDirectly transfer amount which was selected by GiveDirectly as the amount anticipated to maximize the cost effectiveness of cash", p.13 in McIntosh and Zeitlin 2018.
  • All eligible households in treatment villages are treated, but the eligibility criterion differs from that used by GiveDirectly and specifically focuses on households with malnourished children, or pregnant or lactating mothers.
    • "We therefore used a definition of eligibility tailored to Gikuriro’s stated target population: namely, households that contained malnourished children, or pregnant and lactating mothers...CRS and USAID agreed that the following criteria represent the target population for Gikuriro:
      • Criteria 1. All households in a village with a malnourished child (defined by a threshold value of weight/age) were enrolled.
      • Criteria 2. All households in Ubudehe 1 or 2 with children under the age of 5 (Ubudehe is the Rwandan government household-level poverty classification, with 1 being the poorest, 3 being non-poor, and rural areas containing very few of the wealthiest Ubudehe 4 households).
      • Criteria 3. All households in Ubudehe 1 or 2 with a pregnant or lactating mother.

      Both implementers agreed to attempt to treat all eligible households that were identified as meeting any of these criteria", p.10 in McIntosh and Zeitlin 2018.

  • The authors estimate the direct effects of consumption on recipient households by comparing consumption for eligible households in treated villages to eligible households in control villages. They also estimate the "total causal effect" of the cash transfers by comparing the average consumption of eligible and ineligible households between treatment and control villages.
    • Equation (1) on p.17 in McIntosh and Zeitlin 2018 is the regression used to estimate the intent to treat effect amongst eligibles.
    • "The Total Causal Effect of the program on the average household in study villages can be
      estimated by running Equation (1) on the entire sample, ineligible and eligible alike", p.18 in McIntosh and Zeitlin 2018.
  • The authors do not directly estimate spillover effects, but the total causal effect of cash transfers is a weighted average of the direct effect for treated households and the spillover effect for ineligible households.

• 20
  • "The final column, which gives the impact sizes on ineligible households, implies that there may have been as much as a 50% decline in total HH wealth, 12% decline in consumption, and about a 7% decline in the dietary diversity score. That the SNT estimates are most likely to be statistically non-significant is a function of study design: despite the fact that ineligible population outnumbers the eligible one, there are almost twice as many eligible HHs in the sample, or four per village. The study is underpowered to detect the TCE and the SNT and that makes the findings suggestive. But, the implied negative spillover effect sizes are large", see this post.
  • For more detail and sources, see this spreadsheet.

• 21

  For more detail and sources, see this spreadsheet.

• 22

  See the "Results" columns in this spreadsheet for a detailed outline of the results in each paper.

• 23

  Differences in the size of cash transfers, the proportion of households in treatment villages who are treated, whether non-recipients are eligible or ineligible for the program, the identification strategy used, and the length of time between the cash transfer and the follow-up survey make it hard to directly compare effect sizes across papers. See this spreadsheet and descriptions of the studies in this report for further information.

• 24

  Our model is here, with explanatory document here.

  To arrive at a best guess of the true spillover effects of GiveDirectly's program, we must consider the reliability of relevant studies ("internal validity") as well as how applicable they are to GiveDirectly's program ("external validity").

  • On internal validity: major factors that affect our views include the sample size of the relevant studies, the study design and identification strategy used, and whether the study is able to assess the mechanisms for its findings.
  • On external validity: GiveDirectly's current program provides transfers of about $1,000 to all poor households in a village. All studies of spillover effects that we consider in this report have substantial differences from GiveDirectly's program, such as: differences in the size of cash transfers, the proportion of households in treatment villages who are treated, and how mechanisms of spillover effects may differ across contexts.

• 25
  • Spillover effect estimates in the academic literature tell us the average change in consumption for non-recipient households (where non-recipient households may be within the same village or in a different village). To apply this estimate to our cost-effectiveness analysis model, we need to know how many non-recipient households exist for each recipient household.
  • For example, in GiveDirectly's current program, in which all households are treated within treatment villages, we are interested in across-village spillover effects. Do those spillover effects apply only to the next village over, two villages over, or do they apply to villages further afield?
  • In the context of GiveDirectly's program, what we would ideally like to know is: for each of a given set of distance bands away from a treated village (e.g. 0-1km, 1-2km, 2-3km, and further), how many non-recipient households are there, and how does the size of spillover effects decline across those bands.
  • This is particularly relevant when we try to extrapolate within-village spillover effects estimated in several academic papers to make inferences about the across-village spillover effects of GiveDirectly's program. How large are across-village spillover effects relative to within-village? Again, this will depend on the relative distance between households across villages compared to within villages, as well as how spillovers decline with distance.
  • It is also worth noting that the concept of distance could be more complicated than just physical distance "as the crow flies", for example it may factor in geographical barriers like difficult terrain, or, say, language barriers.

• 26

  Cash transfers may lead to local price increases. We previously reviewed four RCTs, which found mixed evidence of impacts on prices. We have not recently conducted a systematic review of the effect of cash transfers on prices, so we have likely not examined some studies with relevant evidence. Price increases affect both the recipients and non-recipients of cash transfers.

  We are uncertain about the best way to model the effect of price increases on social welfare. Theoretically, we expect that some goods in GiveDirectly's target villages would be non-tradable (e.g. local services such as haircuts) and that there would be some increase in the prices of such goods, as we believe is consistent with basic economic theory. Price increases may reduce net welfare by a) effectively redistributing gains from consumers to producers (producers may be richer and fewer in number, so may have lower marginal utility from consumption), and b) causing people to substitute away from non-tradable goods as prices for such goods increases. Note that in both of these cases value is transferred to some extent, and we are unsure about the magnitude of the final welfare impact. We have not yet attempted to explicitly model the magnitude of these competing factors or rigorously assessed all of the ways that inflation could affect social welfare. Nevertheless, we would roughly guess that overall effects of inflation are minorly negative.

• 27
  • For a detailed description of the expenditure-based consumption measures in GE study: Note for GiveWell, see Appendix A3 on p.29. (This reference is for internal use since the note for GiveWell is currently confidential.)
  • For a detailed description of the expenditure-based consumption measures in Haushofer and Shapiro 2016, see p.12 in the Online Appendix. The same outcome variables are used in the other two studies based on this RCT (Haushofer, Reisinger and Shapiro 2015 and Haushofer and Shapiro 2018).
  • We have not seen a detailed description of the consumption measure used in McIntosh and Zeitlin 2018. "The study focuses on five dimensions. Here we briefly summarize each; details of the construction of these outcomes are included in Appendix A", p.15. However, Appendix A does not seem to contain this information.

• 28
  • Some indirect effects of price increases can be picked up in academic papers that measure consumption as expenditure. For example, as described in this section, price increases may cause non-recipients to sell productive assets so that they can increase expenditure to maintain their initial level of consumption. In the medium term, the loss of income from the sale of the productive asset may lead to a decrease in expenditure and a decrease in consumption. This effect can be picked up to at least some degree in expenditure-based measures.
  • This might be driving the negative spillover effects on consumption estimated in Haushofer and Shapiro 2018. The authors state that "We do not have conclusive evidence of the mechanism behind spillovers, but speculate it could be due to the sale of productive assets by spillover households to treatment households, which in turn reduces consumption among the spillover group. Though not always statistically different from zero, we do see suggestive evidence of negative spillover effects on the value of productive assets such as livestock, bicycles, motorbikes and appliances", p.3.

• 29

  In addition, a given decrease in consumption caused by an increase in prices will be more harmful for non-recipients if they consume less to begin with, due to the diminishing marginal utility of consumption. By virtue of the cash transfer itself, recipients are likely to start from a higher level of consumption.

• 30 For example, we are uncertain how long it takes for recipient households to spend cash transfers, how long it takes for this to filter through and increase prices, and at what rate local prices might then decrease through arbitrage over time. In turn, we are unsure how quickly non-recipients respond to price changes, for example through the sale of productive assets. We are also unsure how long it might take for psychological effects to impact non-recipients' productivity.
• 31
  • The size of cash transfers. Cash transfers in several RCTs in the academic literature are considerably smaller than the transfers in GiveDirectly's program, which are currently US$1,085 in Kenya, US$970 in Rwanda and US$963 in Uganda. For example, in McIntosh and Zeitlin 2018 the large cash transfers were only US$532.
    • "Transfer sizes for the standard lump sum projects: Kenya: $1,085, Rwanda: $970, Uganda: $963." Joe Huston, GiveDirectly CFO, email to GiveWell, February 20, 2018.
    • "The fourth and much larger transfer arm transferred $532", p.5 in McIntosh and Zeitlin 2018.
  • It seems intuitive that spillover effects will be larger as a result of a larger cash transfer, but how much larger is less certain. As a starting point in our model, we assume that spillover effects increase linearly with an increase in the size of the cash transfer.
  • The proportion of households who are treated. As a larger proportion of households are treated within treatment villages, we would expect the within-village spillover effects to be larger. In the General Equilibrium study, one third of households are treated, whilst in McIntosh and Zeitlin 2018 approximately 11% are treated, and in Haushofer and Shapiro 2018, Haushofer and Shapiro 2016 and Haushofer, Reisinger and Shapiro 2015, on average 9% of households are treated.
    • "With 11.4 percent of all households being defined as eligible", p.28 in McIntosh and Zeitlin 2018.
    • "An average of 19 percent of households per village were surveyed, and an average of 9 percent received transfers", p.5 in Haushofer and Shapiro 2018, which is based on the same RCT as Haushofer and Shapiro 2016 and Haushofer, Reisinger and Shapiro 2015.
    • "In treatment villages, GD enrolls all households in treatment villages meeting its thatched-roof eligibility criteria (“eligible” households); approximately one-third of all households are eligible", p.6 in GE study: Note for GiveWell.
  • In GiveDirectly's program in Kenya and Uganda, all households within treatment villages are treated. We are uncertain to what extent treated households are affected by within-village spillover effects as explained in this section, and so we have so far not made an adjustment for this consideration because we do not include within-village spillover effects on treated households in our model. This is one way that our model might underestimate the size of spillover effects, although the GE study finds that price effects (which is the most plausible mechanism behind a negative spillover effect on consumption for treated households) are small and statistically insignificant.
  • In the General Equilibrium study, across-village spillover effects are estimated by comparing sublocations where two-thirds of villages are treated to sublocations where one-third of villages are treated. If a different proportion of villages are treated in GD's program, the size of across-village spillovers may be different too.
    • GiveDirectly told us that they try to treat all villages in a given region as far as possible: "Generally, we try to go to all villages in the regions where we work, leaving out only urban or obviously substantially richer areas...The high-level answer...is that our enrollment is more "clumped" than "scattered"", Joe Huston, GiveDirectly CFO, email to GiveWell, November 1, 2018.
  • In that case, comparing sublocations that receive the transfers to sublocations that do not is more like an increase from zero to 100% of villages being treated within the sublocation. Combining this with the fact that in Kenya and Uganda the GD program treats all households within villages (compared to one third of households in the GE study), and the intensity of treatment may be much higher in GiveDirectly's program in a given region than in the GE study. As a result, across-village spillover effects from that region may be stronger too. In our current simple model, we base our assumption for the ratio of the size of across-village to within-village spillover effects on the size of the across-village compared to within-village spillover effects estimated in the GE study. For the reasons described here, this may underestimate the relative size of across-village spillover effects in GiveDirectly's program.
  • Spatial distribution of the program. As described in the email from Joe Huston above, in GiveDirectly's program, where villages in a single large block are treated, villages in non-recipient sublocations are more likely to be only surrounded by treated sublocations on one side. By contrast, in the GE study a low saturation sublocation may be surrounded by high saturation sublocations on multiple sides. This would suggest stronger across-village spillover effects in the GE study than in the program.
  • We do not currently have information on the spatial distribution of recipient and non-recipient villages in GiveDirectly's program in order to adjust for this.

• 32

  In some studies, cash transfers are randomly allocated amongst eligible households, such that spillover effects are estimated across both eligible and ineligibles. In other studies, all eligible households are assigned to treatment, meaning that spillover effects are estimated only for ineligible households. In GD's program, if non-recipient villages include households with both thatched and metal roofs, then spillovers affect both eligible and ineligible households.

• 33
  • Within-village treatment estimates are identified by comparing consumption between control households in treated villages and households in control villages.
  • What adjustment needs to be made for this depends on the relative size of across-village compared to within-village spillover effects, about which we are highly uncertain as previously explained.
    • The following simple model describes a possible relationship between these two parameters, X and Y:

      a) "True within-village spillover effect is a factor of Y times the within-village spillover effect estimated in the study"
      b) "Across-village spillover is X% as large as within-village spillover"

      We want to know the value of Y for some best guess for the value of X. Take a simple example:

      Suppose that $1,000 is given to 50% of households in village A, and no households are given the transfer in village B. The transfer has a within-village spillover effect which decreases consumption by W%, and an across-village spillover effect which decreases consumption by A%. Our study will compare the consumption of control households in village A to households in village B, and conclude that the spillover effect is (W% - A%).

      In that case:

      X = A/W
      Y = W/(W-A)

      Then:

      A = XW

      Therefore:

      Y = W/(W-A) = W/((1-X)W)
      = 1/(1-X)

• 34

  We currently include a 5% discount for this concern in our cost-effectiveness analysis. For more on the reasoning behind this adjustment, see this document.

• 35

  For more detail and sources, see this spreadsheet.

• 36
  • Our best guess is that Rwanda will comprise between 15-45% of GiveDirectly's overall program going forward. In the first quarter of 2017, approximately 16% of households enrolled in GiveDirectly cash transfer programs were in Rwanda. In the second half of 2017, Rwanda comprised approximately 45% of GiveDirectly's non-Universal Basic Income enrollees.
    • See rows 5 and 8 in GiveDirectly, Dashboard Metrics for GiveWell, May 2017. To estimate the number enrolled, we multiply the eligibility rate by the total number censused (we have not accounted for the refusals in this calculation). Doing so, approximately 404 were enrolled in Rwanda out of a total of 2,555 across all three countries, meaning that approximately 16% of enrollees in that quarter were from Rwanda.
    • See the right hand side figure on slide 14 in GiveDirectly, Dashboard Metrics for GiveWell, April 2018. From that figure, our best guess is that approximately 1,900 were enrolled in the Universal Basic Income (UBI) program in Kenya, and 3,200 were enrolled in the cash transfer program in Rwanda. Of 7,113 enrollees in the cash transfer program (9,013 total enrollees - 1,900 in the UBI program), this means that (3,200/7,113) = 45% were in Rwanda.
  • GiveDirectly told us via email that it only provides cash transfers to eligible households in Rwanda to comply with government requirements there. Joe Huston, GiveDirectly CFO, email to GiveWell, November 21, 2018.
  • For more information on GiveDirectly's targeting criteria in each of its country programs, see this section of our GiveDirectly review.

• 37
  • "Generally, we try to go to all villages in the regions where we work, leaving out only urban or obviously substantially richer areas...The high-level answer...is that our enrollment is more "clumped" than "scattered"", Joe Huston, GiveDirectly CFO, email to GiveWell, November 1, 2018.
  • Intuitively, the spatial pattern of GiveDirectly's program is likely to lead to a smaller number of non-recipient households experiencing across-village spillover effects, as the total length of the border between treated and non-recipient villages is smaller in a block pattern.