Development Media International (DMI) - 2014 Interim Review

This page is an interim review. We have published a more recent version of this review here.

Development Media International (DMI) is applying to be a 2014 top-rated charity. Here we discuss what we have learned so far and our major outstanding questions.

More information: What is our evaluation process?

Published: October 2014

Summary

What do they do? Development Media International (DMI, developmentmedia.net) produces radio and television broadcasts in developing countries that encourage people to adopt improved health practices, such as exclusive breastfeeding of infants and seeking treatment for symptoms associated with fatal diseases. The program aims to reduce mortality among children less than five years old. (More)

Does it work? DMI is currently conducting a randomized controlled trial of its program in Burkina Faso that will measure impact on child mortality. At midline, DMI found a range of effects for different self-reported behaviors. If the survey results reflect real changes in behavior, DMI's program has likely caused a large reduction in child mortality, but we believe that the results should be interpreted with caution. We discuss questions about the effectiveness of DMI's program below. (More)

What do you get for your dollar? Our impression is that, if effective, DMI’s programs are in the range of, or possibly more cost-effective than, many of our other priority programs. (More)

Is there room for more funds? DMI believes it has the capacity to scale up to five total countries in 2015. This would cost a total of about $9 million in 2015, and DMI currently only expects to receive $2-5 million. (More)

What are GiveWell’s next steps? DMI has successfully completed the first phase of our investigation process and we view it as a contender for a 2014 recommendation. We now plan to (a) make a $100,000 grant to DMI (as part of our "top charity participation grants," funded by Good Ventures) and (b) continue our review process of DMI to try to answer our remaining questions.

Table of Contents

Why are we publishing this page?

As we discussed in our blog post on DMI’s midline results, GiveWell has been interested in mass media as a potential mechanism to save or improve lives at a low cost, and DMI’s ongoing study seems to provide significantly better evidence of this possibility than prior evidence. As such, we wanted to complete a full review of DMI in consideration of a 2014 GiveWell recommendation. This page is intended to update our followers on DMI's application and what we've learned so far.

Our investigation process

To date, our investigation process has consisted of:

What do they do?

DMI designs and delivers radio programming in Burkina Faso that encourages improved health practices to reduce under-5 mortality.2

In addition to DMI’s eight staff at its headquarters in London, there is also one staff member in the Democratic Republic of the Congo (DRC) and 35 in Burkina Faso.3 We have not yet seen a breakdown of DMI’s budget or spending by program category.

How is the broadcast material chosen?

DMI chooses which health topics to focus on based on its predictions of which behaviors are likely to save the most lives per broadcast in a given country.4 Once DMI selects health topics, it develops and monitors its messaging by:

  • Researching the most important factors affecting the targeted behaviors in the country before messaging and scriptwriting begins.5
  • Testing 60-second spots with groups of people in target areas to gauge initial reactions to different portrayals of the health messages and avoid unpopular or confusing spots.6
  • Soliciting feedback from listeners to gauge whether people are listening to and understanding the broadcasts, to monitor for misinterpreted or disliked broadcasts, to determine whether people are changing their behaviors, and to identify the remaining barriers to behavior change.7
  • Tracking self-reported behavior change over time and adjusting estimates of effectiveness of the messages relative to each other. In Burkina Faso this function is being served by the midline results of the RCT.8 In the future it would be accomplished with a similar survey as was used for the midline results, but without a randomized control to compare against.9

What health practices do they encourage?

In Burkina Faso, DMI’s messaging has focused on:10

  • Getting routine care during pregnancy and delivering in a health facility.
  • Seeking health care for young children for routine and emergency care (including fever, cough, difficulty breathing, and diarrhea with blood).
  • Treating diarrhea with oral rehydration salts (ORS) and increasing intake of liquids and foods.
  • Exclusive breastfeeding up to 6 months after birth.
  • Early breastfeeding.
  • Delaying the first bath and promoting skin-to-skin contact for low birth weight babies.
  • Complementary feeding.
  • Appropriate handwashing with soap.
  • Using bednets for children and pregnant women.
  • Using latrines and safely disposing of children's stools.
  • Participating in national vitamin A campaigns.

What content does DMI produce?

DMI produces two types of content in Burkina Faso: (a) one minute recorded commercial spots, and (b) two hour live broadcasts.11

The 60-second radio spots are drama-based and each focuses on a single health behavior message such as, "Start breastfeeding newborns within an hour after birth." Stations broadcast the same spot ten times per day for a week (see footnote for sample spots).12

During a two hour show each weekday evening, paid actors at each of the seven radio stations voice a live radio program including two fifteen minute stand-alone dramas and interactive call-in discussions. Programs are in the local language for each area (see footnote for sample clips).13 Scripts for the dramas are written in French in the capital and disseminated to the radio stations across the country. The actors then perform the dramas in their local languages during the live broadcasts. DMI told us that it takes this approach because it is cheaper than distributing recorded programs in multiple languages.14

Does it work?

We focus on several questions that most affect our understanding of DMI's impact on health and mortality:

  • Does DMI’s program change listeners’ behavior?
  • Do these changes in behavior result in saved lives?
  • What other information measures program success?

Does DMI’s program change listeners’ behavior?

DMI is currently conducting a randomized controlled trial of its intervention and has shared results from a midline survey.15 The midline survey measures uptake of various health practices and compares uptake in the treatment and control groups. Below, we discuss the RCT’s design, the midline results, and issues that influence our interpretation of these results. We have not used other studies to supplement the argument that mass media can successfully improve health behaviors.16

In short, DMI found a range of effects for different behaviors with more consistently positive improvements in the treatment group for "one-off" curative behaviors. If the survey results reflect real changes in behavior, DMI's program has likely caused a large reduction in child mortality. However, we believe that the issues noted below require interpreting these results with caution.

RCT design

From the midline results report:17

Background

[...] Fourteen local radio stations across the country, with high listenership, were selected for the evaluation, of which seven were randomly selected to broadcast short spots of 1 minute duration 10 times per day and long format programs for two hours per day, five days each week. Topics covered include antenatal consultations, health facility delivery, breastfeeding, nutrition, bed net use, sanitation and appropriate health care seeking in case of malaria, pneumonia and diarrhoea.

[...] This report presents the results of the midline survey which was conducted in November 2013, 20 months after the launch of the campaign. The endline survey will be undertaken within the last months of the intervention, after 30 plus months of campaigning.

The objective of the midline survey was to provide mid-term estimates of behaviour changes in order to adjust messages addressed by the campaign.

Methods, Sampling

Within each cluster, a random sample of 9 villages was drawn with probability proportional to size from villages surveyed during the baseline survey.

In each of these villages, a sample of women was selected by simple random sampling using the census data collected during the baseline survey. These women were randomly listed and fieldworkers were asked to interview the first forty available women who met the inclusion criteria in the survey: being aged from 15 to 49 years old and mother of a child less than five years old.

Results

The midline report shows self-reported health behaviors at baseline and at midline for the treatment and the control groups.18 The table below shows the average portion of the relevant population within the areas receiving broadcasts that reported using each behavior at midline minus the portion using it at baseline, minus the same difference within the control group (difference in difference) for 10 key behaviors selected by DMI.19

Target behaviors Difference in difference at midline p value Difference in difference, adjusted for distance to nearest health facility p value
Seeking professional treatment for childhood diarrhea 12.9% 0.037 16.0% 0.014
Using more liquids or rehydration supplements for childhood diarrhea 23.3% 0.012 - -
Using antibiotics for childhood pneumonia 11.2% 0.248 14.8% 0.079
Seeking professional treatment for childhood malaria 6.1% 0.249 9.1% 0.119
Women using bednets (ITNs) during pregnancy 3.4% 0.575 - -
Owning a latrine 2.3% 0.713 - -
Initiating breastfeeding within two hours of birth 10.7% 0.171 - -
Consuming nothing but breastmilk until 6 months of age -1.8% 0.813 - -
Giving birth with skilled attendant or in a health facility 1.0% 0.804 0.2% 0.962
Saving money during pregnancy 8.5% 0.033 - -

Issues to note

Several issues with the study lead us to interpret the results with caution.

  • It appears that the control group was noticeably better off than the treatment group. At baseline, there were noticeable differences between the treatment and control groups in terms of child mortality rates, distance and access to health facilities, remoteness from the capital city, and proportion of women giving birth in a health facility. These differences could conceivably make health behavior change easier in one of the two groups. For example, it may be easier to improve conditions from a worse baseline, which could cause the RCT to overstate impact. Alternatively, factors that have made the control group better off could also mean the secular trend in the control zones is steeper than that in the intervention zones, which could cause the RCT to understate the impact.
    • The control group appears to have had better access to health facilities than the treatment group. The study reports that "40% of women lived less than 2 km away from a health facility in the control arm versus 18% in the intervention arm. Among women sampled for the midline survey, these proportions were 47% versus 16% respectively."20
    • The treatment clusters had higher rates of child mortality in the baseline survey, perhaps due to the fact that they tended to be further from the capital of Burkina Faso and receive less investment than areas that are closer.21
    • There were differences in ethnic composition.22

    DMI told us that, having reviewed these differences between the control and treatment groups, the trial’s Independent Scientific Advisory Committee (ISAC) recommended a cluster-level adjustment for baseline levels in the analysis, including use of a composite confounder score to adjust for the key baseline imbalances, which has been followed by the LSHTM team in the midline results.23

  • The results available at this time are measures of self-reported behavior change. Health behaviors were assessed through self-report, which may be biased.24 For example, respondents in the treatment group who listened to the radio program may inflate the effect it had on their behavior if they believe the surveyor wants or approves of a certain answer.
  • Other health programs may have led to reductions in child mortality. There are two potential issues here: (a) if the programs affected treatment and control clusters differently, this could bias the results of the RCT, and (b) declines in mortality could lead to the study lacking sufficient statistical power to detect a program impact on mortality.
    • The Bill and Melinda Gates Foundation, among others, funded a large training program for health workers during the trial. The program is active in two clusters, one in the treatment group and one in the control group, and may affect one cluster more than the other.25
    • There was a large bednet distribution program in Burkina Faso during the trial as well as national immunization days which included vitamin A supplementation. The RCT researchers do not currently believe that the programs affected control and treatment clusters differently.26
    • Other health programs may have reduced child mortality.27
  • Blinding. Surveyors were aware of whether respondents were in the treatment or control groups. This could lead to bias in the results.
  • Contamination in the control group. Two-thirds of a cluster in the control group with 375 women was excluded from analysis because about half of the women surveyed in their villages reported listening to the radio campaign in the week before the midline survey.28 Including these women in the analysis might underestimate the impact of the campaign, while excluding them may introduce bias, though it would likely be small.

The above factors lead us to interpret DMI’s results with caution. It is possible that the factors above have little effect on the results.29 However, the facts that (a) the treatment and control groups had substantially different access to healthcare and levels of child mortality, and (b) the results are based on self-reported behavior change, seem to leave room for a plausible story where the midline results significantly overstate the impact that DMI’s program has had (and will have).

We plan to ask DMI and Prof. Cousens for additional technical data from the RCT to address some of the questions above, specifically:

  • To what extent were measured differences between the treatment and control groups driven by changes in (a) a small set of clusters or (b) a small set of villages? Were changes reasonably consistent across clusters?
  • What was the nature of the adjustment for distance from health centers for behaviors that depended on them?
  • What has Prof. Cousens been able to learn about the activities of other health programs in the area during the time of the trial?
  • How likely is the selection method for the midline survey, asking for women in a village in a random order and surveying the first 40 of those that are available, to lead to bias?

Do changed behaviors result in saved lives?

Are health supplies and services available when sought?

Some of the health activities that DMI promotes, such as seeking treatment for fever (primarily to ensure that cases of malaria are promptly treated), require that families are able to access health facilities or providers and that appropriate medicines are available.30

DMI receives reports from the Ministry of Health in Burkina Faso that document the availability of health supplies in the regional offices that distribute supplies to health centers throughout the country. In 2013, the government reported that 75.9% of health centers in Burkina Faso were never short on key health supplies and that the figure for the seven regions where DMI has been active was 88.4%.31 We have not seen the technical details of how these figures were compiled.

DMI told us that its research team visits regional health centers during field visits. So far, they have not noticed any major problems nor any inconsistencies with the government’s reports on availability.32 We have not seen any additional details from these visits.

Do the individual health practices save lives?

We have not completed a review of each of the practices DMI encourages through its broadcasts, but we believe that many of them – e.g., pneumonia treatment, malaria treatment, and oral rehydration therapy – represent well-established ways to prevent child mortality.33

What other information measures program success?

We have seen some additional information that helps us understand how well the program is working in Burkina Faso, and what limiting factors might affect its success in other countries. In particular, DMI has told us that it uses various mechanisms to determine whether (a) programs have aired at the agreed upon times, (b) people listen to DMI's programming, and (c) listeners have correctly understood the intended messages.34

The process DMI described to us seems reasonable, but we have seen limited data on whether broadcasts occur consistently and whether DMI's audience listens and understands the intended messages (beyond the evidence of behavior change from the midline results of the RCT).

Do the broadcasts happen?

DMI told us that it can use two tools to track whether broadcasts occur. The main method used in the RCT in Burkina Faso is broadcast monitors. In both cases, we have seen limited written reports and instead rely primarily on conversations with DMI staff members.

  • Broadcast monitors: DMI employs individuals to listen to the radio and record whether stations play DMI's programming at the agreed upon frequency. DMI told us that early in its work in Burkina Faso, broadcast monitors indicated that some radio stations were not playing the one-minute spots at the appropriate frequency. After DMI followed up with these radio stations, the broadcast monitors reported an increase in frequency.35
  • Automatic broadcast reports: DMI has received automatic reports on broadcasts from software used by two of the seven radio stations it works with. This method has only been tested in the current trial of Burkina Faso (alongside the main method of employing broadcast monitors) in two stations where it has been possible to install and train staff to use the necessary software. It is a method DMI is considering for future campaigns to automate and systemize the collection of broadcast monitoring data.36

DMI shared data from broadcast monitors for two of seven radio stations for September and October 2012. These two stations were also generating automatic broadcast reports. The data from the automatic broadcast reports and the two broadcast monitors seem to approximate each other in most cases.37

DMI has also shared three recent quarterly funder reports, which include discussions of challenges the radio stations face in maintaining consistent broadcasts. The primary issues discussed are loss of electrical power and personnel issues, and in some cases, these issues resulted in hours or days of missed broadcasts within a particular cluster.38 We have not yet asked DMI whether it plans to continue helping stations restore power after the RCT is completed in Burkina Faso and in other countries it works in in the future.

Do people listen to the radio?

DMI has conducted four media surveys in Burkina Faso during its RCT, all targeting women.39 The average household radio ownership rates reported in the four surveys ranged from 52% to 62%.40 Between 53% and 83% of the women in each survey reported having listened to the radio within the week prior to the survey; DMI believes the variation may be due to the seasonality of leisure time.41 The figures for women reporting having listened to the specific radio station over which DMI broadcasts were about 10 percentage points lower than the percentage of women listening to the radio.42

Do people understand the broadcasts?

DMI told us that it conducts periodic, informal focus groups in which its staff members ask participants to explain the messages they have heard. We have seen two qualitative summary reports from these feedback groups and a synthesis report summarizing this feedback research across all seven intervention zones.43

DMI reports that most individuals are able to recall at least one of DMI’s broadcast messages and are able to explain the recalled health messages accurately. However, some individuals make errors in recalling the messages.44

What do you get for your dollar?

We have not yet completed a full cost-effectiveness estimate for DMI’s work but plan to do so for our final review of DMI.

DMI estimates that the cost-effectiveness of its intervention is extremely strong relative to other cost-effective interventions (for example, more than 10x stronger than our estimate of our strongest top charities).45 We expect our final estimate of DMI’s "cost per life saved" to be substantially less optimistic (though still within the range of our current priority programs).46

Major factors of our initial cost-effectiveness estimate:

  • We plan to rely primarily on DMI’s midline results, rather than previous nonrandomized studies, to estimate the effect of DMI’s program on behaviors.47
  • We plan to discount the effect implied by DMI’s midline results because they come from a single study rather than multiple studies (as we have done with other top charities).48
  • We plan to discount the effect implied by DMI’s midline results to account for the possible bias associated with purely self-reported results.49
  • We plan to discount the expected benefits of treatment based on the possibility that the treatment received is of poor quality or is not given to the child on schedule, or at all.

For more on the role that cost-effectiveness estimates play in our reviews, see our page on cost-effectiveness analysis.

Is there room for more funds?

DMI has told us that it would like to scale up its program to five countries in 2015, for which they would need about $9 million. Existing and likely funders are currently expected to account for $2-5 million of that goal.50 These plans include scaling up the program being studied by the RCT in Burkina Faso to the 28 largest community radio stations (including the 7 currently serving as the control group for the RCT)51 for a total reach of about 15.4 million people, compared with the 3.5 million reached by the RCT.52

Details of DMI’s funding needs for 2015 are provided in this table:53

Country Estimated cost per year (millions) Optimistic 2015 funding scenario (millions) Minimum unmet 2015 funding need (millions)
Burkina Faso $2.1 $2.1 $0.0
Democratic Republic of the Congo $2.0 $1.0 $1.0
Mozambique $2.0 $2.0 $0.0
Cameroon $1.3 $0.0 $1.3
Cote d'Ivoire $1.8 $0.0 $1.8
5 countries in 2015 example scenario $9.2 $5.1 $4.1

DMI has said that it would likely use additional unrestricted funding to scale up 2015 campaigns in Mozambique, Cote d’Ivoire, and the DRC.54

Major questions for further investigation

How strong is the evidence from DMI’s midline results?

We hope to improve our understanding of the midline results with the questions listed above. Other relevant questions include:

  • Is there other external evidence about the effectiveness of promoting behavior change that dramatically strengthens the evidence for the effectiveness of DMI’s program?55
  • How strong is the evidence that the behaviors DMI promotes prevent child mortality? While we believe many of them represent well-established ways to prevent child mortality, we have not closely reviewed the relevant evidence.
  • How much health improvement that doesn’t result in saved lives should we expect to see from DMI’s program?

How representative is the evidence from DMI’s midline results?

The success of DMI’s program may vary significantly in different contexts. DMI attempts to account for many possible differences between countries in estimating the life-saving potential of its program, including the number of childhood deaths that could be prevented with relatively easy behavior changes, the demographics of the audience for various radio networks, and the costs of operating in the country.56

Other factors that may change the program’s effectiveness are more difficult to quantify or predict, including the aid activities of other non-profits, community and family decision-making practices, and the questions below. A better understanding of these issues, and how they vary among countries, would allow us to better estimate the cost-effectiveness (and associated level of confidence) in the countries to which DMI is considering expanding its program.

What portion of people have access to health supplies but are not already taking advantage of them?

Our understanding is that DMI has limited information on the availability of health products and services for the countries in which it works.57 If fewer people have such access in DRC than in Burkina Faso (among those not already receiving such treatments), we would expect DMI’s program in DRC to be less effective than its program in Burkina Faso.

How common are power outages, equipment failures, or personnel issues for radio stations?

Based on DMI’s experience with station failures in Burkina Faso (see above), these problems have the potential to increase DMI's costs and/or reduce the frequency of DMI's broadcasts. We are not sure how significant those costs will be in other countries compared with Burkina Faso, or how well DMI will be able to respond to them.58

Are there factors unique to Burkina Faso that make it a particularly fertile environment for mass media campaigns?

DMI told us that it chose Burkina Faso as the site for its RCT because it has an extremely decentralized media market, making it possibly the only country where 14 isolated media environments could be included in a randomized controlled trial. It is possible that a particularly decentralized or underdeveloped media market might make professional radio programming particularly attractive or compelling (when competing for attention with other radio stations or non-radio activities), which could make DMI’s program more effective in Burkina Faso than in other countries.

How much do relevant cultural attitudes vary among countries?

For example, could a dearth of independent media make individuals less likely to take the advice of health messages delivered through radio,59 or might some cultures have more resistance to changing specific health behaviors?

How will the effects of DMI’s program change over time?

To what extent will behavior changes persist after messages cease? Will listeners become less responsive to DMI’s program after many years of broadcasts?

How will we know if future programs are successful?

Since future programs will not have the benefit of a randomized control as is currently being used in Burkina Faso, we believe that it will be harder to know how much of an impact those programs are having. DMI’s ongoing monitoring efforts would provide some insight as to behavior changes over time, and how popular the broadcasts are in different communities. DMI has also expressed an intent to conduct time-series studies or non-randomized, quasi-experimental evaluations of future programs to get further understanding of impact; we are not aware of detailed plans.60

How does DMI’s program operate?

  • What has been learned from DMI’s monitoring efforts? DMI conducts several types of monitoring on their program (see above), but we do not yet understand to what extent those efforts help demonstrate program success, or the most important lessons that DMI has learned from them. For example, we would like to know how many broadcasts have been prevented due to technical and personnel issues at radio stations, and to what extent DMI expects those issues to be a problem in future programs (both for their effects on DMI's budget and impact).
  • How much of DMI’s budget goes to each aspect of its program?
  • How does DMI choose whether to include 2-hour programing in each country given the consideration that the shorter recorded messages alone may be more cost-effective?61
  • How will DMI decide when it is time to end a program in a country?
  • What are senior DMI staff members’ track record with running similar campaigns?

Sources

Document Source
DHS Survey DRC 2008 Source
DHS Survey DRC Preliminary results 2013 Source
DMI Broadcast Tracker Summary 2012 Source
DMI Burkina Proposal 2014 Unpublished
DMI cost-effectiveness Model 2014 Unpublished
DMI Country CEE 2014 Source
DMI DRC Proposal July 2014 Unpublished
DMI Email from Will Snell, July 17th, 2014 Source
DMI Email to GiveWell October 3rd, 2014 Unpublished
DMI Feedback Research Summary September 2013 Source
DMI Funder Report 2013 Q3 Source
DMI Funder Report 2013 Q4 Source
DMI Funder Report 2014 Q1 Source
DMI Health Supply Availability 2013 Source
DMI Message Brief on ARIs Source
DMI Message Brief on Breastfeeding Source
DMI Message Brief on Diarrhoea Source
DMI Message Brief on Family Planning Source
DMI Message Brief on Hygiene Source
DMI Message Brief on Low Birthweight Source
DMI Message Brief on Malaria Source
DMI Message Brief on Maternal Health Source
DMI Message Brief on Vitamin A Source
DMI Monitoring Report Bogande December 2013 Source
DMI Monitoring Report Solenzo July 2014 Source
DMI Pretesting Synthesis ARI 4 October 2013 Source
DMI Pretesting Synthesis Breastfeeding February 2013 Source
DMI Pretesting Synthesis Diarrhoea 2 July 2012 Source
DMI Pretesting Synthesis Hygiene 2 January 2013 Source
DMI Pretesting Synthesis Malaria 5 December 2013 Source
DMI Pretesting Synthesis Maternal 3 July 2013 Source
DMI Summary 2014 Source
DMI summary midline results 2014 Source (archive)
DMI website, cost-effectiveness 2014 Source (archive)
DMI website, partners page 2014 Source (archive)
DMI website, staff page 2014 Source (archive)
DMI website, what we do page 2014 Source (archive)
GiveWell interim cost-effectiveness model of DMI 2014 Source
GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014 Unpublished
GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 15th, 2014 Unpublished
GiveWell's non-verbatim summary of a conversation with Will Snell on July 31st, 2014 Unpublished
GiveWell’s non-verbatim summary of a conversation with DMI on April 24th, 2014 Source
GiveWell’s non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014 Source
Infosaid Media Landscape 2012 Source
LSHTM DMI RCT midline results 2014 Unpublished
LSHTM Protocol First Draft 2013 Source
UNICEF Facts for Life 2010 Source (archive)
WHO child mortality 2011 Source (archive)
  • 1

  • 2
    • "We design and deliver radio and TV campaigns that save lives in developing countries by encouraging people to adopt healthier behaviours for themselves and their families." DMI website, what we do page 2014, Pg 1.
    • "The specific messages that DMI broadcast[s] are weighted according to their predicted impact on child mortality. DMI is changing the weight of broadcasting across its range of messages midway through the program, based on effectiveness data from the first 20 months and on changes in health indicators in Burkina Faso. DMI is focusing more on messages that have been proven to have a greater impact and are predicted to save the most lives. It will also be adjusting its modeling at the end of the trial to reflect the new evidence the RCT will provide. DMI’s recommendations at the end of the trial will include information on which messages would be most effective for future child mortality programs." GiveWell’s non-verbatim summary of a conversation with DMI on April 24th, 2014, Pg 5.

  • 3
    • (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell on July 31st, 2014
    • "DMI has 35 staff members in Burkina Faso, 8 in London, and 1 in the Democratic Republic of the Congo (DRC)." GiveWell’s non-verbatim summary of a conversation with DMI on April 24th, 2014, Pg 1.
    • The titles of the staff listed on DMI’s website that appear to be based in Burkina Faso are:
      • Country Director
      • Chief Financial Officer
      • Head of Human Resources
      • Zones Liaison Manager
      • International Radio Producer (2)
      • Scriptwriter Manager
      • Radio Scriptwriter (15)
      • Research Manager
      • Senior Researcher (2)
      • Technical Manager
      • Zone Producer (2)
      • Logistician
      • Operations & Research Administrator
      • Finance Administrator
      • Driver & Mechanic
      • Driver (2)
    • DMI website, staff page 2014
    • DMI only expects about 20 staff in each country where they have a full program in the future, including only about 4-5 script writers. (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell on July 31st, 2014

  • 4
    • "The specific messages that DMI broadcast[s] are weighted according to their predicted impact on child mortality. DMI is changing the weight of broadcasting across its range of messages midway through the program, based on effectiveness data from the first 20 months and on changes in health indicators in Burkina Faso. DMI is focusing more on messages that have been proven to have a greater impact and are predicted to save the most lives. It will also be adjusting its modeling at the end of the trial to reflect the new evidence the RCT will provide. DMI’s recommendations at the end of the trial will include information on which messages would be most effective for future child mortality programs." GiveWell’s non-verbatim summary of a conversation with DMI on April 24th, 2014, Pg 5.
    • For more on the midline results and expectations of lives saved, see below.

  • 5

  • 6

  • 7

  • 8

    "Since March 2012, Development Media International (DMI) has been implementing a comprehensive mass media campaign, through local community radio stations, to address key behaviours for improving under-five child survival in Burkina Faso. The campaign is being evaluated using a cluster-randomised design, by LSHTM and Centre Muraz, to investigate whether the intervention can change behaviours on a scale large enough to result in measurable reductions in under-five child mortality... The objective of the midline survey was to provide mid-term estimates of behaviours changes in order to adjust messages addressed by the campaign." LSHTM DMI RCT midline results 2014, Pg 1.

  • 9
    • DMI reports that for campaigns that are only one year long, this type of mid-line adjustment would likely not be used, but that for longer campaigns it would be included, possibly using time series analysis (sampling part of the population regularly).
    • (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 15th, 2014
    • "For future child mortality programs, DMI plans to use a quasi-experimental design to measure a wide range of outcomes. This could involve administering a series of surveys to do a time series analysis, creating non-randomized controls, or using propensity score matching." GiveWell’s non-verbatim summary of a conversation with DMI on April 24th, 2014, Pg 6.

  • 10

    "DMI messages addressing health facility dependent behaviours and related to maternal and newborn health have focused on attending ANC and on health facility delivery... DMI messages addressing health facility dependent behaviours and related to child health have focused on growth monitoring/ attendance at well baby clinics, health care seeking in a health facility or with a Community Health Worker (CHW) in case of illness (fever, cough or fast/ difficult breathing, diarrhoea with blood in the stools) and ORS treatment in the case of diarrhoea (Tables 2a-2e)... DMI messages addressing home-based behaviours related to maternal and newborn health have focused on savings during pregnancy, early breastfeeding after birth and delay of the first bath for low birth weight babies... DMI messages addressing home-based behaviours related to child health have focused on compliance with antibiotic treatment in case of cough or fast/difficult breathing, increasing liquids and food during diarrhoea and nutrition (exclusive breastfeeding up to 6 months after birth, complementary food from the age of 6 months) (Tables 4a-4d). In addition, when national vitamin A distribution occurred, messages were broadcast to alert the population... DMI messages have recommended bed net use by children and pregnant women... DMI messages related to sanitation have promoted latrine ownership and safe disposal of children’s stools by throwing them into latrine, by making the child use a pot or by burying the stool outside the compound." LSHTM DMI RCT midline results 2014, Pg 3-5.

  • 11
    • "Fourteen local radio stations across the country, with high listenership, were selected for the evaluation, of which seven were randomly selected to broadcast short spots of 1 minute duration 10 times per day and long format programs for two hours per day, five days each week." LSHTM DMI RCT midline results 2014, Pg 1.
    • "For 30 months we are broadcasting 60-‐‐second advertisements at least 10 times per day on seven radio stations (one in each intervention zone), in six languages. In addition, we are broadcasting two hours per night, five nights per week on each station. This represents a total of 70 hours per week of live radio. This would be logistically almost impossible to do using a soap opera format, for example, given the six languages involved. We needed to devise a format that is cheap, that can be broadcast daily, that can be produced ‘live’ (which costs a fraction of pre-produced radio), and yet can be controlled centrally. We have created a system of self-contained drama modules that are written in French in the capital city, emailed to our partner radio stations, and improvised live by actors on location in their own language within their two-hour shows. This works well in a fragmented media environment, which is becoming the norm in most developing countries." DMI Summary 2014, Pg 3.
    • After the RCT ends DMI plans to stop producing the live broadcasts to improve the cost-effectiveness of the program. ("The campaign will broadcast one-minute radio spots. The advantages of using spots are that they can be very precisely scripted, they are drama-based and highly entertaining, and they can be broadcast up to 10 times per day. Previous campaigns have shown that the frequent broadcasting of spots can lead to large shifts in behaviour. During the RCT we have also been broadcasting nightly dramas (acted out live on each station in local languages). Our qualitative research suggests that, whilst these have been effective, the spots have greater impact, and are much more cost-effective as they require less manpower and so are cheaper to produce (especially on 28 stations)." DMI Burkina Proposal 2014, Pg 1.)

  • 12
    • (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell on July 31st, 2014
    • For a sample spot, see DMI’s website: http://developmentmedia.net/fr/audio-burkina-faso-breastfeeding-spot-20…
    • All participating stations in a country play the same spots during the same weeks, although the spots are translated to the local language for each station as necessary, and may be played at different times as the stations are managed independently. Each message ends with the same audio clip of a baby laughing, designed to brand the messages. (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell on July 31st, 2014

  • 13

  • 14
    • "We needed to devise a format that is cheap, that can be broadcast daily, that can be produced ‘live’ (which costs a fraction of pre-‐‐produced radio), and yet can be controlled centrally. We have created a system of self-‐‐contained drama modules that are written in French in the capital city, emailed to our partner radio stations, and improvised live by actors on location in their own language within their two-‐‐hour shows." DMI Summary 2014, Pg 3.
    • In Burkina Faso the community radio stations broadcasting these dramas are generally running on very low budgets and have limited professional programing. For example, DMI reports that many community radio stations are willing to play DMI’s broadcasts in exchange for the experience it will gain in live broadcasts from the dramas, rather than for cash payment. (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell on July 31st, 2014
    • For an example of DMI’s planned content schedule in DRC, see DMI DRC Proposal July 2014, Pg 8.

  • 15

    LSHTM DMI RCT midline results 2014

  • 16

    We conducted initial research on the evidence for health behavior change from mass media in 2012 and found it inconclusive, which led us to deprioritize further research at that time.

  • 17

    Note, footnotes have been omitted, and one typographical error was removed. LSHTM DMI RCT midline results 2014, Pg. 1

  • 18

    "Due to the relatively small number of clusters, Difference-in-Difference (DiD) analyses [Difference between arms in the percentage change in behaviours prevalence between surveys] were performed on cluster level summaries. The prevalence of key behaviours in both arms at baseline (BS) and midline (MD), as well as the 'crude' DiD (based on individual data) and results from cluster level analysis (DiD, 95% CI and p value) are presented." LSHTM DMI RCT midline results 2014, Pg. 2

  • 19
    • LSHTM DMI RCT midline results 2014
    • Selection of 10 behaviors from DMI summary midline results 2014
    • All values are based on cluster-level analysis, rather than portions of the total population.
    • The p-value for each estimate is the likelihood (as a fraction of 1) that we observe an effect at least as extreme as the estimated effect due to random chance alone if we assume that DMI’s broadcasts actually cause no change in behavior.

  • 20
    • "The results of the baseline survey indicated good balance between intervention and control clusters for home-based behaviours. While antibiotic and antimalarial treatments, as well as the delay in accessing these treatments, were balanced between arms, other health facility dependent behaviours - ANC, facility delivery, attendance at well baby clinics ('pese'e'), health care seeking, consultation and Oral Rehydration Solution (ORS) treatment - tended to be better in the control arm compared to the intervention arm." LSHTM DMI RCT midline results 2014, Pg. 2
    • "This imbalance probably reflects better access to health facilities in the control arm compared to the intervention arm. At baseline, 40% of women lived less than 2 km away from a health facility in the control arm versus 18% in the intervention arm. Among women sampled for the midline survey, these proportions were 47% versus 16% respectively (Figure 1)." LSHTM DMI RCT midline results 2014, Pg. 2

  • 21

    "On average though, the control group had a lower baseline child mortality rate. The average child mortality rate was 136 per 1000 in the intervention group and 102 per 1000 in the control group. Professor Cousens believes that one reason for this is that the intervention clusters were on average farther from the capital of Burkina Faso. There is a correlation between the distance of a cluster from the capital and the mortality rate in a cluster, partly because clusters that are farther from the capital receive less investment." GiveWell’s non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014.

  • 22

    "Demographic characteristics of interviewed women were similar between surveys. Women were 30 years old on average in both arms. About 40%, 20%, 13% and 12% of women overall belonged to the Mossi, Gourmantche, Gournoussi and Peulh ethnic groups respectively (Figures 2a and 2b). In the two arms, although Mossi remained the main ethnic group in both, Gourmantche and Peulh were more common in the intervention arm and Gournoussi were more common in the control arm. Gouin, Karaboro and Turka were specific to the Banfora cluster (intervention arm)...While about half of women were muslim and half christian in the control arm, 60% were muslim in the intervention arm. In both arms, nearly all women were married, of whom about 40% were in a polygamous union. Education was quite low with only around 15% of women in the control arm and 10% of women in the intervention having been to school." LSHTM DMI RCT midline results 2014, Pgs 2-3.

  • 23

    DMI Email to GiveWell October 3rd, 2014

  • 24

    "The first section of the interview collected information on basic demographic characteristics and radio listenership. Subsequent sections addressed preventive and curative behaviours of relevance to child health. Questions regarding maternal and newborn health, i.e. antenatal care (ANC) and delivery, referred to the woman’s last pregnancy of more than 6 months duration. Questions regarding health care seeking, bed net use, nutrition and sanitation applied to her youngest child less than five years old. Illnesses (fever, cough, fast or difficult breathing, diarrhoea, dysentery) were recorded using a recall period of two weeks preceding the interview." LSHTM DMI RCT midline results 2014, Pg. 2

  • 25

    "The research team is concerned about the impact of a large health worker training program funded by the Bill & Melinda Gates Foundation and other funders. The program is active in two clusters, one from the control group and one from the intervention group. The research team believes that the program is having a significant effect in the control cluster but is uncertain how this compares to its effect in the intervention cluster. The research team is reviewing the study’s analysis plan with the study’s advisory committee. If the research team decides that the training program is significantly affecting the results of its RCT, it may perform a sensitivity analysis in which it removes those two clusters from the data." GiveWell’s non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014.

  • 26

    "There was recently a large bed net distribution program. Nearly 100% of households interviewed in the midline survey had at least one bed net, which was a large increase from the baseline. There were also national immunization days during which polio vaccines and vitamin A supplements were administered. The research team is documenting smaller health programs in various clusters, but in general it does not believe that there is a major difference between the programs in the intervention clusters and the control clusters." GiveWell’s non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014.

  • 27

    "According to DHS, there was a large decline in mortality in Burkina Faso between 2003 and 2010. The mortality rate in Burkina Faso has been declining faster than that of most other countries. This complicates the results of the RCT, because the study may have less power than planned to detect a reduction in mortality. Some of the decline in mortality rates in Burkina Faso is due to large national health programs, which the research team documented in its analysis." GiveWell’s non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014.

  • 28

    "The survey detected 'contamination' in Gayeri control cluster with women in part of the cluster reporting listening to Djawoampo radio station broadcasting the campaign in Bogande intervention cluster. Among 375 women interviewed in Gayeri cluster, a third - mainly in villages located to the North and North West of Gayeri (towards Bogande) - reported having listened to Djawoampo radio station in the last seven days. All women interviewed in these villages (two-thirds of all women in the cluster) were therefore excluded from analysis. To account for the much smaller sample size of Gayeri cluster, analyses were weighted by the number of observations per cluster." LSHTM DMI RCT midline results 2014, Pg. 2

  • 29
    • For example, Prof. Cousens noted the following in our conversation: "It is unclear whether a higher baseline mortality would lead to a larger or smaller relative reduction in mortality. It may be that regions with high baseline mortality have more opportunity for improvement in mortality rates. On the other hand, it may be that regions with higher mortality rates are poorer, more remote, and have worse health services, which would make it more difficult to reduce mortality…. The fact that the midline survey did not demonstrate positive impacts on some behaviors suggests that there was not a strong tendency for subjects to overstate their behavioral changes." GiveWell’s non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014.
    • For more detail on common bias issues in formal studies, see this page.

  • 30

    For example, we believe the following issues might prevent a family from successfully receiving health supplies or services despite seeking them:

    • The distance to the nearest health center is too long to walk and any alternative transportation is too expensive.
    • The family doesn’t have a member that can make the trip, due to poor health or lack of free time.
    • The health center is open too few or unpredictable hours.
    • The health center has long wait times for service.
    • The health center is out of stock of the needed supplies.

  • 31

    DMI Health Supply Availability 2013

  • 32

    (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014

  • 33

    DMI told us that they follow the “Facts for Life” guidance developed by the United Nations Children’s Fund (UNICEF), the World Health Organization (WHO), and other organizations: UNICEF Facts for Life 2010. (DMI Email to GiveWell October 3rd, 2014)

  • 34

    (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014

  • 35

    (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014

  • 36

    (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014

  • 37
    • DMI Broadcast Tracker Summary 2012
    • We did notice that the numbers from one monitor are systematically about 40% higher than those from the other monitor and the software: the average broadcasts logged per day for the 2 observers and software at Radio Loudon in Sapouy, respectively, were 9.4, 13.3, and 9.6. We are not sure of the reason for the discrepancy or whether it is ongoing. (DMI Broadcast Tracker Summary 2012)

  • 38
    • General comments from the reports
      • "...with the exception of Radio Lotamu, our partner stations have all broadcast DMI spots 10-16 times per day throughout the entire quarter; and that Lotamu is now back up to the same level." DMI Funder Report 2013 Q3, Pg 2.
      • "As before, power supply has been a headache, but by loaning generators and subsidising solar installations we have kept all seven stations on air with only very minor interruptions… The most important insight—which seems obvious with hindsight—is that energy, perhaps even more than creativity or research, is likely to be the determining factor of a successful community radio campaign. This quarter’s energy problems, occurring during the midline field research, were particularly problematic… The second crucial lesson that our 22 months of broadcasting has confirmed is that in managing an output-intensive partnership with a community radio station there is no substitute for ‘ears on the ground’. In spite of two years of collaboration, of important investments made in energy security, studio equipment, training and programming, we still only learn of serious problems in our partner stations during our field visits. This is particularly true in regards to the many personnel issues that have an impact on our interactive programs. In Ouahigouya, La Voix du Paysan’s star program host, and the principal host of our interactive program, [name redacted] passed the civil service exams to be a teacher and enrolled in the inductory training program without informing the management of la Voix du Paysan, or its parent organization. As this training made her increasingly unavailable for her daily programming responsibilities and for ‘DMI’ rehearsals she was obliged to tell the station director that she was hoping to teach full-time while continuing to work for la Voix du Paysan. Bassirou learned of these developments during one of his scheduled visits only after expressing our frustrations at [her] having missed several rehearsals. In the end, the radio station’s management decided to fire [her]. Her departure has forced us to reorganize the team that broadcasts our interactive program. Cheikh has worked with the station’s management to redistribute programming responsibilities and convinced the station director to recruit a new staffer." DMI Funder Report 2013 Q4, Pg 1-2.
      • "The media project in Burkina Faso continues to run very much according to plan, notwithstanding the usual technical and logistical headaches." DMI Funder Report 2014 Q1, Pg 1.
    • Specific issues that caused missed broadcasts
      • Radio Tintaani in Kantchari failed to broadcast anything for 10 days in March and 2 days in November 2013 due to power outages. In both cases broadcasts only resumed when DMI bought and loaned a generator to the station.
        ("In early November we purchased a second 7kva backup generator and sent it to Radio Tintaani in Kantchari. As discussed in the Q1 2013 report, we purchased a first 7kva generator in March 2013. Our intention was that this generator be moved from station to station as our different partner stations experienced energy outages. This generator was first sent to Kantchari where Radio Tantaani had lost ten days of broadcasting to power outages. Once Radio Tintaani’s energy supply stabilized in June—the local cooperative repaired their generator—we sent DMI’s generator to Radio Djawoampo in Bogande who were experiencing seasonal outages caused by heavy rains. The second week of November, during the midline field research, both Radio Djawoampo and Tintaani were experiencing power outages. Anxious to get Radio Tintaani back on air during the midline research we purchased (€5,393) a second generator and installed it in Kantchari, limiting the power outages to two days." DMI Funder Report 2013 Q4, Pg 1.)
      • Radio Lotamu in Solenzo had power outages for much of 2013 which reduced their broadcasting from 16 hours per day to four, preventing some of DMI’s programming from being broadcast. DMI bought the station a solar generator in October and rented them a fuel generator to last until the solar system was installed, which ultimately happened in February 2014. We do not know how many broadcasts were skipped during that time.
        • "...as feared/anticipated Radio Lotamu in Solenzo has started to suffer serious energy shortages. From sixteen hours of broadcasting a day Radio Lotamu has been reduced to 4 hours on most days. The power shortages have started to reduce the number of spots they broadcast per day and have disrupted the interactive program. We have rented a backup generator in Solenzo—the station pays the fuel—to get the station back on-air immediately and have contracted with a supplier to install a solar energy system at Radio Lotamu (the station is the only one of our partner stations who has saved some of DMI’s monthly production stipend in order to make capital investments)." DMI Funder Report 2013 Q3, Pg 2.
        • "Radio Lotamu in Solenzo also experienced recurring power outages throughout November and December. As mentioned in the Q3 report, worried that Solenzo’s power supply problems would only get worse—the town depends on a local energy cooperative that is keeping two very old generators on life-support—we decided to install a solar energy system at Radio Lotamu. We signed a contract with a supplier in early October and rented a local generator to keep the station on air through the midline. Our pessimistic analysis of Solenzo’s energy supply has been confirmed by recent events; both of the Solenzo energy cooperatives’ generators caught fire in early January and the town of Solenzo no longer has electricity. Frustratingly, our solar supplier has experienced delays with his Spanish supplier of rechargeable batteries. We have prolonged the rental of the generator for Radio Lotamu through December and our solar supplier has agreed to cover a percentage of the additional rental costs." DMI Funder Report 2013 Q4, Pg 2.
        • "In February, the solar energy system contracted in October 2013 was installed at Radio Lotamu, Solenzo. The station now has energy independence 8 hours a day, allowing them to broadcast during both the morning and evening primetime hours. Frustratingly, only weeks after the solar energy system was brought online the radio station’s amplifier overheated and was seriously damaged. The station is currently broadcasting without the amplifier, with a reduced range of 25 km. We are working with the station to replace their amplifier as quickly as possible—radio amplifiers, and their replacement components, are not available in Burkina Faso." DMI Funder Report 2014 Q1, Pg 3.
      • Radio Lotamu in Solenzo's amplifier was seriously damaged a couple weeks after the solar system was installed, reducing their broadcast range from approximately 50km to 25km. We do not know what percent of the target population previously reached by Radio Lotamu is not reached with the reduced range (three fourths of the broadcast area would be lost under simple assumptions), or whether the amplifier has since been replaced.
        • "Frustratingly, only weeks after the solar energy system was brought online the radio station’s amplifier overheated and was seriously damaged. The station is currently broadcasting without the amplifier, with a reduced range of 25 km. We are working with the station to replace their amplifier as quickly as possible—radio amplifiers, and their replacement components, are not available in Burkina Faso." DMI Funder Report 2014 Q1, Pg 3.
        • "Burkina Faso has a system of localised media whereby FM radio stations typically have a range of approximately 50km, which permit a cluster randomised trial." LSHTM Protocol First Draft 2013, Pg 4.
        • A perfect circle with a radius twice as large as another will have four times the area (7854 square kilometers versus 1963 square kilometers in the case of a radius of 50km and 25km, respectively). DMI doesn’t use population less than 5km away from radio towers for its surveys, but this calculation assumes those people still receive the broadcasts. In practice, changes in population density and topography complicate the calculation of lost audience; we have not seen an estimate of the lost audience from DMI.
      • Radio Djawoampo in Bogande experienced seasonal outages caused by heavy rains in the first half of 2013, and separately in November. DMI arranged for a generator to supply the station power starting in June 2013. We do not know how many broadcasts were skipped as a result of power outages.
        • "Radio Djawoampo in Bogandé continues to rely on DMI’s backup generator to get their program on the air" DMI Funder Report 2013 Q3, Pg 2.
        • "Once Radio Tintaani’s energy supply stabilized in June—the local cooperative repaired their generator—we sent DMI’s generator to Radio Djawoampo in Bogande who were experiencing seasonal outages caused by heavy rains. The second week of November, during the midline field research, both Radio Djawoampo and Tintaani were experiencing power outages. Anxious to get Radio Tintaani back on air during the midline research we purchased (€5,393) a second generator and installed it in Kantchari, limiting the power outages to two days" DMI Funder Report 2013 Q4, Pg 1.

  • 39
    • "In addition to the baseline and midline surveys, two media surveys recorded, in March 2011 and in June 2012, household ownership of a radio and women’s listenership... The second media survey, in June 2012, was prompted by the low radio listenership reported by women at baseline. At the time, we believed that a plausible explanation for the low figures recorded at baseline was a result of the questions on radio listenership coming at the end of a long questionnaire (the first media survey had a very short questionnaire). The second media survey again used a very short questionnaire and recorded higher levels of listenership which seemed to support the hypothesis relating to questionnaire length. However, at midline, the radio listenership questions came at the beginning of the questionnaire but still resulted in lower estimates of radio listenership. Ordering estimates by month of the year reveals a similar pattern across clusters suggesting possible seasonality in listenership. This seasonality may be explained by women’s activities in the fields. From October to December women spent most of their time in the fields harvesting cereals. From January to March, market gardening and minor season farming are their main activities with collection of firewood, local wild fruits and seeds ("néré", "zamanè", "pain de singe", "tamarin"). Although there may be variation between years depending on the rainy season and across regions, March to June are generally a less busy time for women with no farming activities typically taking place. During the second media survey, nearly 60% of regular radio listeners mentioned that farm work prevented them from listening to the radio. The low figures for listenership recorded at baseline may therefore have been a reflection of women’s farming activities during that period, perhaps exacerbated by the placing of the questions at the end of a long questionnaire." LSHTM DMI RCT midline results 2014, Pg 5-8.
    • Women were the focus of the surveys because in Burkina Faso they are most often the ones that conduct behaviors that affect children’s health. (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014

  • 40

    "Household radio ownership in the intervention arm was consistent across the four surveys: ranging from 52% to 62% (Figure 5)." LSHTM DMI RCT midline results 2014, Pg 5.

  • 41
    • "The proportions of women who reported listening to the radio in the seven days prior to the interview varied from 53% at baseline to 83% in the second[sic] media survey" LSHTM DMI RCT midline results 2014, Pg 5. Note that the graph in Figure 5 appears to indicate that the 83% listenership figure was from the first media survey, not the second.
    • "The second media survey, in June 2012, was prompted by the low radio listenership reported by women at baseline. At the time, we believed that a plausible explanation for the low figures recorded at baseline was a result of the questions on radio listenership coming at the end of a long questionnaire (the first media survey had a very short questionnaire). The second media survey again used a very short questionnaire and recorded higher levels of listenership which seemed to support the hypothesis relating to questionnaire length. However, at midline, the radio listenership questions came at the beginning of the questionnaire but still resulted in lower estimates of radio listenership. Ordering estimates by month of the year reveals a similar pattern across clusters suggesting possible seasonality in listenership. This seasonality may be explained by women’s activities in the fields. From October to December women spent most of their time in the fields harvesting cereals. From January to March, market gardening and minor season farming are their main activities with collection of firewood, local wild fruits and seeds ("néré", "zamanè", "pain de singe", "tamarin"). Although there may be variation between years depending on the rainy season and across regions, March to June are generally a less busy time for women with no farming activities typically taking place. During the second media survey, nearly 60% of regular radio listeners mentioned that farm work prevented them from listening to the radio. The low figures for listenership recorded at baseline may therefore have been a reflection of women’s farming activities during that period, perhaps exacerbated by the placing of the questions at the end of a long questionnaire." LSHTM DMI RCT midline results 2014, Pg 6.

  • 42

    10 percentage point estimate is based on eyeballing Figure 5. LSHTM DMI RCT midline results 2014, Pg 10.

  • 43

  • 44
    • "La plupart des répondants restituent les messages correctement, mais certains répondants reprennent les messages avec des erreurs, par exemple sur : le nombre de CPN recommandé, l’âge d’introduction de l’eau et de la bouillie enrichie dans l’alimentation du bébé et l’âge limite de la consultation du nourrisson sain (pesée)." DMI Feedback Research Summary September 2013, Pg 5.
    • Google’s Translation tool suggests this translation for the above passage: "Most respondents restore the messages correctly, but some respondents show messages with errors, for example: the number of CPN recommended age introduction of the water and porridge enriched in the baby's diet and the age limit of the consultation healthy infant (weighing)."
    • "Some hear our messages but misunderstand or reinterpret parts of them, e.g., claiming herbal concoctions are okay after a baby is 6 months or thinking exclusive breastfeeding should only be done until 3 months." DMI Funder Report 2014 Q1, Pg 6.
    • DMI also reports that it has adjusted its message several times based on misinterpretations and errors: (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014

  • 45
    • "According to our model, the cost per DALY of a DMI mass media campaign in most countries is in the range of $2-$10. This would make mass media behaviour change campaigns in most countries as cost-effective as any other interventions currently used in public health. We are now carrying out a scientific trial in Burkina Faso to prove this prediction. (As a rule of thumb, there are around 30 DALYS per individual 'life', so the cost per life saved of our campaigns is $60-$300, depending on the country.)" DMI website, cost-effectiveness 2014
    • See the 2013 GiveWell cost-effectiveness analysis (xlsx) for details on our estimates of the cost-effectiveness of our top charities.

  • 46

    See GiveWell interim cost-effectiveness model of DMI 2014.

  • 47

    As a result, the benefits we expect to see will be smaller, and we plan to rely on the few behaviors that DMI encouraged during the trial that have already shown results consistent with large life savings. In particular, we only plan to include the effects of encouraging families to seek treatment for young children that show specific symptoms of three commonly fatal illnesses: diarrhea, pneumonia, and malaria.

  • 48

    It is difficult to know how strong a discount to apply for this consideration. We have seen some potentially relevant figures in John Ioannidis’s analysis of biomedical literature (here and here), though the analogy between biomedical studies and DMI’s midline results has substantial limitations.

  • 49

    Note that the final results from DMI’s trial, expected in late 2015, will include parent-reported deaths, which we believe to be significantly less prone to bias than reporting on events or behaviors that are not as well defined, memorable, and publicly known.

  • 50
    • DMI currently has funding for about $750k of global costs, is expecting $1 million for its DRC program, and is hoping to reach an agreement to fully fund Mozambique and Burkina Faso for campaigns in 2015. DMI is seeking additional funding from other sources as well.
    • (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell on July 31st, 2014

  • 51
    • "Our analysis indicates that partnering with the biggest 28 stations (using 12 languages) will reach 78% of the population of Burkina Faso (15.4m of a total projected population of 19.7m in 2016), after which we reach rapidly diminishing returns." DMI Burkina Proposal 2014, Pg 1.
    • Notably, DMI does not plan to include the live programs in the scaled version of the Burkina program because it believes the program will be more cost-effective without them. We are not yet sure why the same is not true in DRC. ("The campaign will broadcast one-minute radio spots. The advantages of using spots are that they can be very precisely scripted, they are drama-based and highly entertaining, and they can be broadcast up to 10 times per day. Previous campaigns have shown that the frequent broadcasting of spots can lead to large shifts in behaviour. During the RCT we have also been broadcasting nightly dramas (acted out live on each station in local languages). Our qualitative research suggests that, whilst these have been effective, the spots have greater impact, and are much more cost-effective as they require less manpower and so are cheaper to produce (especially on 28 stations)." DMI Burkina Proposal 2014, Pg 1.)
    • DMI is expecting to improve the cost-effectiveness of working in Burkina Faso by more than a factor of three by scaling up to 28 stations from 7. ($27.86 versus $8.04 per DALY DMI Burkina Proposal 2014, Pg 2.) Thus DMI expects that the fully scaled program will cost about as much as the RCT per year while reducing the under-five mortality rate in the country by 15%. ("We estimate (conservatively) that a national campaign will reduce under-five mortality in Burkina Faso by 15% at a cost of $8.04 per DALY averted. The cost-effectiveness of this intervention is therefore comparable to the cost of the cheapest health intervention: childhood immunisation (at $1-$8 per DALY averted)." DMI Burkina Proposal 2014, Pg 2.)

  • 52

    For the "RCT" and "Nationwide campaign", respectively, "Population reached: 3,502,354/15,837,253", "Cost per person reached: $0.65/$0.14", which implies costs of $2.3M/$2.2M DMI Burkina Proposal 2014, Pg 2.) GiveWell’s estimate of lives saved per dollar is more conservative (see above).

  • 53
    • All values in the table below are from GiveWell's non-verbatim summary of a conversation with Will Snell on July 31st, 2014.
    • The "optimistic scenario" assumes that all funders that have expressed significant interest in funding DMI's work decide to do so in 2015. (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell on July 31st, 2014

  • 54
    • (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell on July 31st, 2014
    • DMI shared the following preliminary cost-effectiveness estimates (DMI Country CEE 2014, except where specified):
      • DRC: $6.41 per disability adjusted life-year (DALY) (DMI Email to GiveWell October 3rd, 2014)
      • Mozambique: $5.12 per DALY
      • Cameroon: $5.36 per DALY
      • Cote d’Ivoire: $6.80 per DALY
      • Burkina Faso: $5 - $6 per DALY ((Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 15th, 2014)

  • 55

    We did not find any such evidence in our initial search for in 2012.

  • 56

    @DMI Cost Effectiveness Model 2014@

  • 57
    • We believe DMI is using metrics from DHS and MICS to try to answer one part of this question. In particular, those two surveys attempt to measure, among families with children that suffered from specific symptoms in the two weeks before the survey (such as diarrhea or those associated with acute respiratory infections or with malaria), (a) how many families report seeking professional treatment for that child, and (b) how many families report receiving treatment for that child. (For example, "Pourcentage d’enfants de moins de cinq ans ayant présenté des symptômes d’Infection Respiratoire Aiguë au cours des deux semaines ayant précédé l’enquête et, parmi ces enfants, pourcentage pour lesquels on a recherché un traitement auprès d’un établissement ou d’un prestataire de santé et pourcentage à qui on a administré des antibiotiques comme traitement" DHS Survey DRC 2008, Pg 128. Google Translate suggests this translation: "Percentage of children under five years who had symptoms of acute respiratory infection in the two weeks preceding the survey, and of these children, the percentage for which we have sought treatment to an institution or a health care provider and percentage who were given antibiotics as treatment" translate.google.com, translated September 16th, 2014.)
    • Our understanding is that DMI uses (b) divided by (a) (as defined in the prior bullet point) as a proxy for "the portion of the time that treatment for diarrhea is available", since that is the portion of the time which it was received, of the total times it was sought. ((Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 15th, 2014) We see two possible problems with using this approximation:
      • It assumes that people that are not seeking treatment would seek treatment if they were convinced they should, whereas some might not be able to seek treatment at all. For example, if there is no health facility within walking distance, or if no one in the family has the time and ability to seek treatment, then being convinced to seek treatment may not help.
      • It assumes that all received treatment is caused by seeking it, whereas at least some appears to be received without being sought. In the summary results from the most recent DHS survey in DRC, for example, more people receive treatment for several diseases than seek it in several subgroups, so there must be ways to receive treatment without seeking it. (For example, families of boys with diarrhea sought treatment 37.7% of the time, and used treatment 38.9% of the time. DHS Survey DRC Preliminary results 2013, Pg 23.) We are not sure how to account for treatment that would be received without being sought.

  • 58

    We are aware of some indications that infrastructure is particularly problematic in DRC, which may affect both the likelihood of issues, and the difficulty of fixing them. For example, "There is no national network of all weather roads to unite this sprawling expanse of tropical rain forest and savannah grasslands." Infosaid Media Landscape 2012, Pg 3.

  • 59

    This might be a particular concern in DRC: "The political allegiance of most media outlets in DRC normally reflects that of their owner. Even media outlets which attempt to remain politically neutral often run stories which have quite clearly been sponsored by an interested party. The fact that they agree to do so reflects the chronic shortage of cash in most Congolese media organisations. Certain newspapers do run critical pieces and some TV and radio stations have openly criticised government actions or the behaviour of individual ministers. However, independent media organisations increasingly practice self-censorship. Sometimes they couch criticism of the government in obtuse editorials. Another trick is to balance criticism with laudatory coverage of other aspects of government action. The government does not react to every critical report in the local media, but crackdowns are very frequent. Critical journalists and those working for opposition media outlets are often harassed, intimidated and arrested. Several journalists have have been killed in mysterious circumstances after publishing reports critical of the government. Their murderers are almost never tried and convicted. TV and Radio stations which incur the government’s wrath are often forced off air for a period. The DRC ranked 145th out of the 179 countries listed in the Reporters Sans Frontieres www.rsf.org 2011-2012 World Press Freedom Index." Infosaid Media Landscape 2012, Pg 20.

  • 60

    "DMI does not plan to do further child mortality RCTs after the Burkina Faso trial because they would be expensive and would mean that half of the population would not receive the intervention. Additionally, other countries would be less suited than Burkina Faso for a cluster RCT testing a mass media campaign because other countries have less localized media, so it is more likely that the control group would receive the messages broadcast to the intervention group. For future child mortality programs, DMI plans to use a quasi-experimental design to measure a wide range of outcomes. This could involve administering a series of surveys to do a time series analysis, creating non-randomized controls, or using propensity score matching. DMI may compare the effectiveness of various messages within a program." GiveWell’s non-verbatim summary of a conversation with DMI on April 24th, 2014, Pg 6.

  • 61

    "The campaign will broadcast one-minute radio spots. The advantages of using spots are that they can be very precisely scripted, they are drama-based and highly entertaining, and they can be broadcast up to 10 times per day. Previous campaigns have shown that the frequent broadcasting of spots can lead to large shifts in behaviour. During the RCT we have also been broadcasting nightly dramas (acted out live on each station in local languages). Our qualitative research suggests that, whilst these have been effective, the spots have greater impact, and are much more cost-effective as they require less manpower and so are cheaper to produce (especially on 28 stations)." DMI Burkina Proposal 2014, Pg 1.