Georgetown University Initiative on Innovation, Development, and Evaluation — Zusha! Road Safety Campaign

Published: June 2017

Note: This page summarizes the rationale behind a grant to the Georgetown University Initiative on Innovation, Development, and Evaluation (gui²de) made by Good Ventures. gui²de staff reviewed this page prior to publication.

The organization

The Zusha! Road Safety Campaign, managed by gui²de, targets unsafe drivers of public service vehicles. The campaign distributes stickers for buses with messages encouraging passengers to speak up and urge drivers to drive more safely. The program also includes a lottery that bus drivers with stickers displayed in their buses can win, to incentivize bus drivers to keep the stickers. The program’s goal is to reduce traffic accident deaths (which are more common in many developing countries than in most developed countries1 ).

There have been two randomized controlled trials (RCTs) conducted on Zusha!’s program in Kenya (where it has primarily operated so far) – see details below. gui²de has received $3 million in funding from Development Innovation Ventures to conduct three more RCTs in Tanzania, Rwanda, and Uganda.2

Note on terminology: In this writeup, we generally refer to the program we're interested in as Zusha! to distinguish it from gui²de's other programs (not related to road safety). However, Zusha! is only the name of the road safety campaign in Kenya; the road safety campaigns in Rwanda, Tanzania, and Uganda have other names.

Evidence of effectiveness

Two RCTs have been conducted on Zusha!’s program in Kenya: a small pilot study (which had a treatment group of roughly 1,000 vehicles)3 and a second, much larger study (involving roughly 12,000 vehicles, including treatment, control, and placebo groups).4 The studies' primary outcome was total number of claims made by insurance companies for traffic accidents.

The first study found a roughly 50% reduction in accidents due to the intervention;5 the second found a roughly 25% reduction.6 Both studies found the effects of the program to be statistically significant.7 The researchers extrapolate from these figures and estimates of accidents involving deaths to estimate the number of deaths averted by the intervention.8

Compliance with the intervention (i.e. buses retaining the stickers) appeared to decrease over time in the second study.9 (A time series analysis was not done for the first study.) The second study included a placebo sticker group and found a nearly statistically significant effect in the placebo group.10

Cost-effectiveness

One of our Research Analysts, Leon Zhang, examined the RCTs' cost-effectiveness analyses (CEAs) and found mathematical errors which, when corrected for, increased the published estimates of cost per life saved. (See our summary for details.) When we reached out to the papers’ authors to check our understanding, they immediately and graciously acknowledged the errors and prepared corrections. (For the authors' published corrections to the 2011 study, see Habyarimana and Jack 2016.)

Even taking these adjustments into account, the results of the two completed RCTs suggest that Zusha! may be roughly 3 to 4 times as cost-effective as cash transfers, which is roughly comparably cost-effective to some of our other top charities, such as the Against Malaria Foundation (AMF).11 However, we believe that our CEA could change substantially, either positively or negatively, based on further analysis and the results of the three upcoming RCTs.

Note that the intervention’s main impact comes from saving adult lives (whereas much of the impact of, e.g., distributing insecticide-treated bednets comes from saving the lives of children under five years old). We’ve used GiveWell staff's median ethical values from our November 2016 top charities CEA to estimate the relative value of saving adult lives.12 (For a more in-depth discussion of possible approaches to this type of valuation, see this December 2016 blog post.)

Grant details

This grant is intended to allow Zusha! to:

• Continue to operate at scale in Kenya for an additional six months to a year. Without this grant, our understanding is that Zusha! would be close to running out of funding for its operations in Kenya by September 2017.
• Improve monitoring of its program in Kenya (see below).
• Increase the sample size of its RCT in Uganda from about 4,000 to about 6,000 buses, improving the statistical power of the study.
• Collect higher-quality data in Tanzania through September 2017, which Zusha! has told us could improve precision of measurement in that RCT.
• Continue operating in Tanzania, Uganda, and Rwanda for an additional six months, so that it is able to scale up more quickly if those countries' RCT results are positive.

Improving monitoring

We hope that this grant will enable Zusha! to improve its monitoring in Kenya to accomplish two main goals:

1. Produce a robust measure of how many buses in the country are in fact using the stickers (Zusha! tries to reach every bus, but currently does not have a highly reliable system for monitoring whether stickers are used in the buses).
2. Track the long-term impact of the intervention, which could potentially be importantly different from the short-term impact (for example, it seems possible that people could adjust to the presence of the stickers, such that their effectiveness decreases over time).

Budget

Our grant breaks down roughly as follows:13

• $352,130 to fund monitoring and an additional six months of operations in Kenya.
• $189,000 to increase the sample size of the RCT in Uganda by about 50%.
• $97,500 for an additional six months of operations in Tanzania, Uganda, and Rwanda.
• $80,000 to improve the quality of data collection for the Tanzania RCT.
• $99,552 for miscellaneous administrative costs for the additional months of operations that our grant is funding, including staff salaries and travel costs.
• $81,818 to cover 10% overhead to Georgetown University.

Risks of the grant

We see a number of potential risks to the success of this grant:

• Negative RCT results: It may be that the new RCTs of this program either a) find that it is not effective, or b) are inconclusive due to lack of statistical power or methodological issues.
• Changes in evidence quality or estimated cost-effectiveness: We have not yet deeply vetted the two RCTs on Zusha!’s program in Kenya, and we have not analyzed our CEA as deeply as our top charities’ programs’ CEAs. Further analysis may lead to negative updates on evidence quality and/or estimated cost-effectiveness.
• Monitoring quality: As discussed above, we have not yet seen high-quality ongoing monitoring of Zusha!’s program in Kenya. It may be that Zusha! is unable to meet our criteria for becoming a top charity due to operational issues and/or lack of ongoing monitoring.

Follow-up expectations

Our goal is to have a charity review for Zusha! published by end of November 2017. We plan to check in with gui²de during the second half of 2017 about two main questions:

• What are the results of the new RCTs, and how does this affect the quality of evidence for the intervention and our CEA?
• Is Zusha! collecting high-quality monitoring data to achieve the two monitoring goals described above?

By around September 2017, we expect to see:

• Several months of 2017 monitoring data from Kenya.
• Full RCT results and data from Tanzania. gui²de thinks it is fairly likely it will also have a working paper by this point.
• Results from half of the Uganda RCT.

By February or March of 2018, we expect to see full results from the Uganda RCT.

We are not sure when results from the Rwanda RCT will be available; this RCT has been delayed indefinitely due to challenges getting government approval to run the program and RCT.

Internal forecasts

We are experimenting with recording explicit numerical forecasts of the probability of events related to our decision-making (especially grant-making). The idea behind this is to pull out the implicit predictions that are playing a role in our decisions, and to make it possible for us to look back on how well-calibrated and accurate those predictions were. For this grant, Josh Rosenberg, our senior research analyst who led GiveWell's investigation of Zusha!, records the following forecasts:

• Zusha! is recommended as a top charity by year-end 2017: 35%, broken down into:
  • Zusha! appears more cost-effective than AMF: 10%
  • Zusha! appears roughly as cost-effective as AMF: 15%
  • Zusha! appears less cost-effective than AMF (but is still a top charity recommendation): 10%

Our process

We had four conversations with James Habyarimana, Billy Jack, and Whitney Tate of gui²de over the course of the past year. We published notes from conversations we had in March 2016 and July 2016. We put significant time into producing our CEA of the Zusha! program (including Leon Zhang finding errors in the original studies’ CEAs). Recently, we have mainly focused on determining Zusha!'s room for more funding and our timeline for evaluating it as a potential top charity.

Subsequent to recommending this grant, we have published additional content on Zusha!:

• Notes from a site visit to Zusha!’s program in Nairobi, Kenya in February 2017
• A blog post on why we’re considering Zusha! as a potential top charity in 2017.

Sources

Document	Source
Georgetown University 2014, "East Africa Road Safety Project Gets $3 Million Grant"	Source (archive)
GiveWell, Zusha! CEA	Source
Habyarimana and Jack 2010, working paper	Source (archive)
Habyarimana and Jack 2011	Source (archive)
Habyarimana and Jack 2015	Source (archive)
Habyarimana and Jack 2016	Source (archive)
World Health Organization, Road Safety Interactive Map	Source
Zusha!, budget	Source (archive)
Zusha!, Power calculations - Tanzania	Source
Zusha!, Rwanda Study Protocol (Draft, preliminary)	Source
Zusha!, Tanzania Study Protocol (Draft, preliminary)	Source
Zusha!, Transition summary and narrative	Source
Zusha!, Uganda Study Protocol (Draft, preliminary)	Source

• 1See, for example, World Health Organization, Road Safety Interactive Map.
• 2

  "A $3 million grant…from USAID’s Development Innovation Ventures (DIV) supports the third phase of the Zusha! study in Kenya, and expands the project to three other countries – Tanzania, Uganda and Rwanda." Georgetown University 2014, "East Africa Road Safety Project Gets $3 Million Grant"

• 3

  "Evocative messages encouraging passengers to speak up were placed inside a random sample of over 1,000 long-distance Kenyan minibuses, or matatus" Habyarimana and Jack 2010, working paper, pg. 4. (Note: This source is a working version of Habyarimana and Jack 2011.)

• 4
  • Pure control: 2,093
  • Placebo: 1,759
  • Treatment stickers: 7,885
  • Total: 11,737

  Habyarimana and Jack 2015, pg. 4662, Table 1.

• 5

  "Our estimates consistently suggest that the intervention reduced the number of incidents leading to an insurance claim by about a half." Habyarimana and Jack 2010, working paper, pg. 28. (Note: This source is a working version of Habyarimana and Jack 2011.)

• 6

  "Among the roughly 8,000 vehicles in the treatment groups, the reduction [in insurance claims] was 25%." Habyarimana and Jack 2015, pg. 4661.

• 7
  • "We identify an impact on driver behavior that is both statistically significant and economically large." Habyarimana and Jack 2010, working paper, pg. 1439. (Note: This source is a working version of Habyarimana and Jack 2011.)
  • "All other message types [other than "text-only" stickers] reduce claims by statistically significant margins."
    Habyarimana and Jack 2015, pg. 4667.

• 8
  • "In our data, 11 percent of claims involved at least one death, although we do not know the actual number of deaths associated with each such accident. In our baseline case, the projected claims rate in the treatment group is about 10 percent, which the treatment reduces by five percentage points. Assuming the same rate of reduction in accidents involving a death as in accidents involving injuries or death (we are better able to estimate the impact on the latter), the intervention thus reduced the number of accidents including a death by about 6.0 per year per thousand vehicles. Conservatively we assume an average of two fatalities per accident including a death." Habyarimana and Jack 2010, working paper, pg. 20. (Note: This source is a working version of Habyarimana and Jack 2011.)
  • "Among the roughly 8,000 vehicles in the treatment groups, about 140 accidents were avoided over the course of a year…Associated with the accidents that were avoided, we estimate 55 lives were saved." Habyarimana and Jack 2015, pg. 4668.

• 9

  "Compliance with assignment to sticker groups, as measured by the number of [lottery] winners out of 10 drawn, was initially very high, but fell over the first 6 [months] of the intervention." Habyarimana and Jack 2015, pg. 4664.

• 10

  Habyarimana and Jack 2015:

  • "Although we do not detect a placebo effect directly, we cannot reject equality of the treatment and placebo effects…even the low dosage of the placebo may have been partially effective." Pg. 4666.
  • See Table 4, column "Claims", row "Placebo," pg. 4666.

• 11

  @GiveWell, Zusha! CEA@, "CEA" sheet, cell B37.

• 12@GiveWell, Zusha! CEA@, "CEA" sheet, cell B26.
• 13See @Zusha!, budget@.