Available impact studies for economic empowerment programs

This page provides an overview of the impact studies we received as a part of our 2009 grant process for economic empowerment programs in sub-Saharan Africa.

Summary

• Only one organization submitted a compelling impact evaluation. By "compelling," we mean a study that not only observes a positive change in clients' lives, but carefully considers, discusses, and gives strong reasons to doubt "alternate hypotheses" for such a change. (Most randomized controlled trials would qualify for this subjective criterion, but there could also be studies meeting this criterion without being randomized controlled trials.)
• 17 of 49 organizations that applied for our grant submitted an impact study. Of these, we found 15 to have serious flaws that are noted below. One organization (Heifer International) submitted a study that remains confidential, so we cannot offer our thoughts on it. One organization, BRAC, submitted what we consider a compelling impact study.
• We would guess that many of the 108 organizations who did not apply did not do so because they had no impact study available. As we discuss at our process page for this grant, (a) we were straightforward about our criteria for this grant and (b) of the 40 organizations that gave a reason for declining the application, 10 specifically cited a lack of technical reports.

Common methodological flaws

Below, we list common methodological flaws in the studies we reviewed.

The table below notes the flaws we felt were present in each organization's submission.

• "Self-reported (retrospective)" refers to studies that rely on participants' reported well-being and reported improvement over time to measure a program's success. In some cases, participants were only asked about their attitude as opposed to objective measures of standard of living. We have a variety of concerns about these studies. Participants may feel pressure to tell interviewers what they want to hear; they may incorrectly recall their past conditions, changes over time, and the reasons for those changes. While we feel this sort of data could be part of a compelling study, none of the studies with this flaw listed spoke to these concerns.
• Lack of control group. Some studies point to observed improvements in the lives of program participants, but did not include a "control group" (non-participants) or otherwise address the question of whether these improvements were due to the program or due to other factors.
• High attrition. Some studies began with many participants but only ultimately examined the changes in outcomes for a subset of those that started in the program. In these cases, we are concerned that those individuals that completed the program and the evaluation might not be representative of all enrolled participants (and may systematically tend to have seen more favorable outcomes, thus creating a biased study result).
  
• Representativeness. Some studies were not clear as to how participants were selected. In these cases, we are concerned that the group evaluated may have been systematically different than the group surveyed, biasing the study's result.
• No baseline. Studies that reported conditions post-program but didn't offer a baseline (pre-program) assessment, and thus did not make it possible to assess what change had occurred in participants' lives.
• Measured non-impact outcome. Studies that pointed to an outcome that does not directly indicate a change in an individual's life. For example, if a study points to knowledge of health behaviors, we didn't consider this to demonstrate impact, unless the link between knowledge of health behaviors and improved outcomes (in the case of this cause, improved standards of living) is separately established.

None of these flaws need be fatal in itself, but in the table below they are used as shorthand for major concerns raised by the studies listed. We have made the full studies available as well (with the exception of Heifer International's, which is confidential).

Organization details

The table below provides the full list of organizations that applied, whether they submitted an impact evaluation or none at all, and what led us to conclude that a particular evaluation was not compelling.

(The charity names in the table below provide links to our review pages for each organization; the "Reason not compelling" column references terms explained above.)