Development Media International - 2014 Review

We have published a more recent review of this organization. See our most recent report on DMI.

Published: December 2014

Summary

What do they do? Development Media International (DMI, developmentmedia.net) produces radio and television broadcasts in developing countries that encourage people to adopt improved health practices, such as exclusive breastfeeding of infants and seeking treatment for symptoms associated with fatal diseases. The program aims to reduce mortality among children less than five years old. (More)

Does it work? DMI is currently conducting a randomized controlled trial of its program in Burkina Faso that will measure impact on child mortality. At midline, DMI found a range of effects for different self-reported behaviors. If the survey results reflect real changes in behavior, DMI's program has likely caused a substantial reduction in child mortality, but we believe that the results should be interpreted with caution. We discuss questions about the effectiveness of DMI's program below. (More)

What do you get for your dollar? Our impression is that, if effective, DMI’s programs are in the range of many of our other priority programs. (More)

Is there room for more funds? DMI believes it has the capacity to scale up to five total countries over the course of 2015. This would cost up to $10 million in 2015, and DMI currently only expects to receive $2.5 - 4 million. (More)

DMI is a standout because of its:

• Unusually strong self-analysis - particularly in supporting a randomized-controlled trial (RCT) on its program.
• Midline results from the RCT- which may be consistent with significant life-saving effects in the cost-effectiveness range of many of our other priority programs (and may also be consistent with weaker results).
• Standout transparency - it has shared significant, detailed information about its program with us.

Major unresolved issues include:

• The main evidence for DMI’s program is from a single RCT’s midline results, whereas other programs we recommend are generally supported by multiple strong sources of evidence. We have concerns about possible bias in DMI’s RCT, particularly resulting from differences between the control group and treatment group at baseline, biases associated with purely self-reported results, and the possible impact of other health organizations that were active in the same areas.
• Even taken at face value, we are unsure what DMI’s midline results imply about the number of lives being saved by their program. We expect the RCT’s final results, expected around the end of 2015, will help inform the answer to this question.
• Even if we assume DMI’s program is saving a large number of lives, we are unsure how the efficacy and costs of the program will compare in new countries where the cultural barriers to behavior change may be very different, and where DMI will not be spending as much time and money on each radio station from which it broadcasts.
• Details of what we know below and of our remaining questions further below.

Our review process

To date, our review process has consisted of:

• Site visit to DMI’s operations in Burkina Faso (see notes and photos), including:
  • Extensive conversations with Country Director Matthew Lavoie, Radio Executive Producer and Trainer Pieter Remes, and other staff.1
  • A visit to one of DMI’s seven partner radio stations to watch preparation and live broadcast of DMI programming and speak with station staff.2
  • Meetings with national government officials that have supported DMI’s campaign in the Ministry of Health and the Ministry of Communication.3
  • Meetings with representatives from Micronutrient Initiative and the Initiaties Conseil International.4
  • Interviews with professionals in the rural health system.5
• Conversations with DMI CEO Roy Head, Director of Development Will Snell, Research Manager Jo Murray, and Public Engagement and Innovation Manager Cathryn Wood.6
• Two conversations with the lead researcher conducting the trial to independently evaluate DMI’s program, Professor Simon Cousens, Epidemiology and Medical Statistics, London School of Hygiene & Tropical Medicine (LSHTM).7
• Conversations with independent actors with perspective on DMI or possible challenges to launching campaigns in new countries.8
• Reviewing documents DMI sent in response to our queries, including details of the midline results from its ongoing randomized controlled trial.

All content on DMI, including past reviews, updates, blog posts, and conversation notes, is available here.

What do they do?

DMI designs and delivers radio programming (currently in Burkina Faso, though DMI hopes to expand to other countries) that encourages improved health practices to reduce child mortality.9

In addition to DMI’s eight staff at its headquarters in London, there is also one staff member in the Democratic Republic of the Congo (DRC) and 35 in Burkina Faso.10

Below is a breakdown of DMI’s costs in Burkina Faso and its headquarters in London since the beginning of the trial (category explanations in this footnote).11

What health practices do they encourage?

DMI says it chooses which health topics to focus on based on its predictions of which behaviors are likely to save the most lives per broadcast in a given country.12 In Burkina Faso, DMI’s messaging has focused on:13

• Getting routine care during pregnancy and delivering in a health facility.
• Seeking health care for young children who manifest symptoms including fever, cough, difficulty breathing, and diarrhea with blood.
• Treating diarrhea with oral rehydration salts (ORS) and increasing intake of liquids and foods.
• Initiating breastfeeding within an hour after birth.
• Exclusive breastfeeding up to 6 months after birth.
• Delaying the first bath and promoting skin-to-skin contact for low birthweight babies.
• Complementary feeding.
• Appropriate handwashing with soap.
• Using bednets for children and pregnant women.
• Using latrines and safely disposing of children's stools.
• Participating in national vitamin A campaigns.

What is the format of DMI’s broadcasts?

DMI produces two types of content in Burkina Faso: (a) one minute recorded commercial spots, and (b) 10-15 minute live drama modules.14 DMI believes the former is much more important for affecting the behavior of listeners, and plans to rely less on the latter for future campaigns.15 It may also use video versions of the spots for TV and mobile phones in certain countries.16

Spots (60 seconds)

The 60-second radio spots are acted scenarios advertising a specific health behavior message such as, "Start breastfeeding newborns within an hour after birth." Stations broadcast the same spot at least ten times per day for a week before DMI switches to a new spot.17

Here is DMI’s English translation of a spot encouraging mothers to feed their babies their colostrum (first breastmilk):18

Modules (10 - 15 minutes)

When DMI signed contracts with the seven radio stations in Burkina Faso, it got their agreement to set aside two hours of their prime time each weekday evening for tightly formatted programming.19 The content is narrated by DJs and includes two 10-15 minute modules written by DMI and performed live by station actors, with the rest of the content chosen by the radio station, such as listener call-in discussions, music clips, and news.20

Scripts for the modules are written at the country headquarters in French and disseminated to the radio stations across the country. The actors then perform the dramas in their local languages during the 2 hour prime time program. DMI told us that distributing storylines in french and supporting actors to improvise them live in each station is cheaper than distributing recorded programs in multiple languages.21 The health message focus for the storylines changes from module to module (see footnote for sample clips).22

How is the broadcast material produced?

Once a health topic is chosen, there are several stages of qualitative research and production involved in producing spots.23

• Discerning specific obstacles to behavior change. DMI has a qualitative research team with three full time employees.24 Before DMI broadcasts on a specific health topic, the research team seeks to understand the biggest limiting factors that prevent people in the broadcast areas from adopting the behaviors DMI plans to encourage. They do so by visiting rural villages within range of broadcasting radio stations and interviewing caregivers of children under five, chiefs, religious leaders, midwives, community health workers, and others. They combine what they learn with WHO recommendations to create 2-3 recommended messages for each health topic. The team writes a one-page message brief for each recommended message to give context to it (see footnote for examples).25

  
  

  Here is one of the two behavior messages in DMI’s message brief on pneumonia, and the three barriers to that behavior it hopes to overcome:26

  Behaviour to promote
  If your child has a cough and rapid breathing or difficult breathing, they may be suffering from pneumonia, a severe lung disease. Bring the child to a health centre for treatment immediately as the disease can be fatal.
  ...
  Barriers to behaviour change

  • Many children die of pneumonia at home because their parents do not understand the seriousness of this disease and do not know that it requires immediate professional medical help.
  • Many people do not recognize the signs of pneumonia. They often confuse the symptoms of pneumonia with various forms of cough, and also with those of other diseases such as malaria and meningitis. It is necessary to emphasize the importance of recognizing the key signs of pneumonia: fast or difficult breathing.
  • Many people will first self-medicate. They use traditional herbal “tisanes” or various concoctions. Only after the failure of various attempts at home or when the child develops more serious symptoms, such as fever, loss of appetite or sleep, will they make use of the skilled care available at the health centre.
• Writing and producing spots. Each scriptwriter in DMI’s Burkina Faso headquarters in Ouagadougou comes up with two written scripts for a particular message that will be the focus of an upcoming week’s broadcast.27 (see above for sample script.) The scriptwriting team then narrows down the scripts to their 12 favorite, from which the creative director and producers choose the final six that will be produced.28 The scriptwriting team then produces those six spots in the studio at the country headquarters in 2 languages for pretesting.29
• Pretesting sample spots with locals. The qualitative research team
  brings recordings of the six sample spots to two villages that they haven’t visited for this purpose before within one of the seven broadcasting zones. Each of the three researchers (two female, one male) interviews one focus group in each village composed of 12-13 adults of the same gender as the researcher. The researchers play all six spots and lead discussions to determine whether people understand and appreciate the spots, and to learn if the spots match local reality. Using this information, DMI selects four of the six spots to air in future weeks. (For examples of the “pretesting synthesis” reports that result from the spot testing research, see this footnote.)30
• Sending spots to each station in the local language. The Scriptwriting team finishes producing the four selected spots in all six languages and then DMI distributes the recordings to each of the radio stations for broadcast.31
• Soliciting feedback from locals after broadcasts. The qualitative research team visits two villages in one of the broadcast zones about once a month to see how broadcasts on recent messages have affected behavior in those areas. As with the visits for testing sample spots, the researchers each meet with one focus group of 12-13 people of their own gender in each village. To solicit feedback the researchers ask questions to understand whether people have heard the spots, understood the messages, and changed their behaviors, and to understand what other influences are affecting their health behaviors. In some cases that feedback has affected the messaging of future spots, such as when DMI realized that some people were interpreting DMI’s message on exclusive breastfeeding to prescribe excluding drinks and liquids but not necessarily herbal concoctions used for traditional medicine (which risk causing diarrhea). (For examples of the “monitoring reports” from this feedback research see this footnote.)32
• Measuring self-reported behavior change over time. In addition to the qualitative research in the above steps, DMI conducts quantitative surveys to measure behavior change from its programs to help improve its messaging. Primarily, such information allows DMI to adjust the number of weeks of broadcasts dedicated to each health message. In Burkina Faso this role was filled by the midline survey, while in future countries the survey will likely not use a randomized control, which are more expensive and leave more people without broadcasts.33

How does DMI choose and partner with radio stations?

In Burkina Faso, DMI works with seven unaffiliated community radio stations, while in other countries it is also considering working with networks of affiliated community radio stations, national radio, or some combination.34

When selecting which radio stations or networks to partner with, DMI considers the station’s management, staff, listenership, and cost.35 When DMI was choosing which stations to partner with for its trial in Burkina Faso, it met with over 50 community radio stations and decided on 14 with non-overlapping range that it thought would be the best to work with, from which the seven broadcast stations were randomly selected.36

DMI’s involvement and time or financial investment in each of the seven community radio stations has varied significantly based on needs of the stations.37 Each station receives a monthly stipend from DMI (between $663 and $1421/month in the 3 months for which we’ve seen data).38 The stations use the stipend to hire actors to perform DMI’s long format modules, pay other station staff, buy equipment, and in one case, build station financial reserves.39 One station had major problems mid-trial and DMI ultimately hired a person to work at the station full time to make sure it continued to remain on-air and broadcast DMI’s material throughout the trial.40

Does it work?

We focus on several questions that most affect our understanding of DMI's impact on health and mortality:

• Does DMI’s program change listeners’ behavior?
• Do these changes in behavior result in saved lives?
• What other information measures program success?

DMI’s study measured moderate increases in self-reported behavior on several important health outcomes. However, we are unsure to what extent (a) these changes reflect actual (rather than merely self-reported) behavior change, (b) they are due to external, non-DMI factors (rather than DMI’s program), and (c) they are dependent upon conditions specific to the DMI’s work in Burkina Faso such that the results might not be applicable to its work in other countries in the future.

Note that the results released to date represent interim results. DMI plans to collect mortality (as opposed to self-reported) data as part its final survey, which should help answer some of the questions above.41

Does DMI’s program change listeners’ behavior?

The strongest evidence of DMI’s impact on listener behavior comes from the midline survey of a randomized controlled trial of DMI’s intervention.42 The midline survey measures uptake of various health practices and compares uptake in the treatment and control groups. Below, we discuss the RCT’s design, the midline results, and issues that influence our interpretation of these results. We have not used other studies to supplement the argument that mass media can successfully improve health behaviors.43

DMI’s study measured moderate increases in self-reported behavior on several important health outcomes. However, we are unsure of how to interpret these results and believe that the issues noted below require interpreting these results with caution.

RCT design

From the midline results report:44

Results

DMI’s draft midline results report increases in some behaviors, particularly curative behaviors such as seeking treatment for a child who has malaria, diarrhea or pneumonia, but limited evidence of an effect on other behaviors, particularly ongoing preventative behaviors such as washing hands with soap after cleaning a child who has defecated.45

The researchers who conducted the trial are seeking to publish a paper in an academic journal. We have seen a draft of the paper (which we relied on in writing this review), but we do not yet have permission to share all the results data we have seen. We summarize the results we have seen below to give the reader a sense of some of the results in various categories of behaviors.

DMI reported 54 metrics in its midline report, many of which relate to the same behavior (such as, “things you may do in response to finding that your child has diarrhea”), and a few of which measure behaviors not addressed by DMI’s campaign at all. Without an easy way to choose the most relevant metrics, we removed behaviors that were strict subsets of other measured behaviors,46 and grouped the remaining 32 into the eight categories below), then calculated the average percentage point change for each column, for each category (category definitions in this footnote).47

Note that of the 54 variables LSHTM/DMI measured, only 6 of the unadjusted difference in difference measures were statistically significant at the p < .05 level and only 3 of the adjusted difference in difference measures were statistically significant at the p < .05 level. These figures may change in the final midline results.48 Nevertheless, the consistency with which the treatment group improved relative to the control group leads us to believe that there was a real effect on self-reported behavior and that the measured results are not merely due to random chance. We have not conducted a statistical test to verify this.

Issues to note

Several issues with the study lead us to interpret the results with caution.

• It appears that the control group was noticeably better off than the treatment group at baseline. At baseline, there were noticeable differences between the treatment and control groups in terms of child mortality rates, distance and access to health facilities, remoteness from the capital city, and proportion of women giving birth in a health facility. These differences could conceivably make health behavior change easier in one of the two groups. For example, it may be easier to improve conditions from a worse baseline, which could cause the RCT to overstate impact. Alternatively, factors that have made the control group better off could also mean the secular trend in the control zones is steeper than that in the intervention zones, which could cause the RCT to understate the impact.
  • The control group appears to have had better access to health facilities than the treatment group. The study reports that "40% of women lived less than 2 km away from a health facility in the control arm versus 18% in the intervention arm. Among women sampled for the midline survey, these proportions were 47% versus 16% respectively."49
  • The treatment clusters had higher rates of child mortality in the baseline survey, perhaps due to the fact that they tended to be further from the capital of Burkina Faso and receive less investment than areas that are closer.50
  • There were differences in ethnic composition.51

  DMI told us that, having reviewed these differences between the control and treatment groups, the trial’s Independent Scientific Advisory Committee (ISAC) recommended a cluster-level adjustment for baseline levels in the analysis, including use of a composite confounder score to adjust for the key baseline imbalances, which has been followed by the LSHTM team in the midline results.52

• The results available at this time are measures of self-reported behavior change. Health behaviors were assessed through self-report, which may be biased.53 For example, respondents in the treatment group who listened to the radio program may inflate the effect it had on their behavior if they believe the surveyor wants or approves of a certain answer.
• Other health programs may have led to reductions in child mortality, as hinted at by the fact that the mortality rate in Burkina Faso as a whole has recently declined faster than that of most other countries. There are two potential issues here: (a) if the programs affected treatment and control clusters differently, this could bias the results of the RCT, and (b) declines in mortality could lead to the study lacking sufficient statistical power to detect a program impact on mortality.
  • Population Media Center (PMC) conducted a nationwide mass media campaign to attempt to increase family planning behaviors in Burkina using two 156 episode serial dramas while DMI’s trial was taking place. Their programs were broadcast on the national radio stations across the country and in 22 community stations that didn’t include DMI’s control stations (at DMI’s request) but did include 6 of DMI’s 7 intervention stations. PMC believes it was likely to have had some impact on childhood mortality, although we have not seen strong evidence to support that conclusion.54 (more)
  • The Bill and Melinda Gates Foundation, among others, funded a large training program for health workers during the trial. The program is active in two clusters, one in the treatment group and one in the control group, and may affect one cluster more than the other.55
  • There was a large national bednet distribution program in Burkina Faso during the trial as well as national immunization days which included vitamin A supplementation. The RCT researchers do not currently believe that the programs affected control and treatment clusters differently.56
  • Other health programs may have reduced child mortality.57
• Blinding. Surveyors were aware of whether respondents were in the treatment or control groups, which could lead to bias in the results. Since some of the questions were about whether the participants had heard DMI’s spots, blinding the surveyors would have been difficult.
• Contamination in the control group. Two-thirds of a cluster in the control group with 375 women was excluded from analysis because about half of the women surveyed in their villages reported listening to the radio campaign in the week before the midline survey.58 Including these women in the analysis might underestimate the impact of the campaign, while excluding them may introduce bias, though it would likely be small.
• Unusually aggressive intervention. DMI has told us its first programmatic priority during the trial was to make sure there would be a measurable decrease in lives saved in the RCT through mass media activities. While the primary mechanism for achieving the reduction was broadcasting the 60 second spots, DMI went above and beyond what it would typically include in a mass media campaign, making it less clear how much impact can be expected by future (slimmed) campaigns. Some examples of DMI activities that we don’t believe will be typical in future campaigns are listed here. Note that the reason DMI plans to have a less intense campaign in the future is because it believes many of the elements of its current campaign to be less cost-effective than the campaign as a whole; if they are correct then each of these things will improve the cost-effectiveness of future campaigns despite possibly decreasing impact. Still, the changes add uncertainty to the expected impacts.
  • Negotiating with PMC for PMC to avoid broadcasting on community radio stations in control zones, while PMC did broadcast its own programming in intervention zones. (more)
  • Producing 10 long format module scripts per week that dramatize health messages. Producing this content was originally planned as part of the RCT as a way to help radio stations build quality content, thus developing a better listenership and improving DMI’s relationship with the radio stations. DMI ended up deciding early in the RCT to produce more long format modules than originally intended and to have them focus on the same health messages as the spots instead of merely providing entertainment content.59
  • Renting and buying generators, solar panels, and broadcast equipment for radio stations. DMI plans to continue to help radio stations avoid power outages, such as possibly loaning or giving solar panels to each station in lieu of cash payments, but does not plan to react as aggressively in response to outages or equipment failures that do happen in the future.60
  • Intervening heavily in management of struggling radio stations. DMI has spent significant staff time and energy assisting station management when necessary, including employing two individuals to work full time at problematic radio stations. DMI has said that when it’s not running an RCT in future programs, it will have more flexibility to stop working with stations that collapse or don’t broadcast their programs regularly. That flexibility may significantly reduce costs to run the program, although doing less to assist radio stations may have some negative effects on program impact, such as resulting in partner radio stations with lower quality programming and less engaged listeners.61
• Unreported results. One complicating factor in interpreting the midline results is that we are not sure which results speak most directly to the success of DMI’s program so far. In addition to the 54 metrics we have seen, the midline survey, on which the midline results are based, contains some potentially relevant questions for which we have not yet seen the results, including: (conceptual translations)62
  • For your most recent birth, is your child still alive, or at what age did they die?
  • When your child has diarrhea, what should you do?
  • If your child received treatment for diarrhea/malaria/pneumonia, may I see the marked health card/prescription/medicine packaging?
  • How many days were there between your child’s first symptoms and first receiving treatment?
  • For how many days was your child sick?
• Better partner station opportunities in treatment than in control group. DMI has said that the seven radio stations randomly selected by the randomization process represented more promising “opportunities to establish effective working partnerships” than those selected as control, a difference which may have made it easier to create a successful campaign.63

The above factors lead us to interpret DMI’s results with caution. It is possible that the factors above have little effect on the results.64 However, the issues above seem to leave room for a plausible story where the midline results significantly overstate the impact that DMI’s program has had (and will have).

See our outstanding questions about the reliability of DMI’s midline results below.

Do changed behaviors result in saved lives?

Are health supplies and services available when sought?

Some of the health activities that DMI promotes, including the three that DMI believes are saving the most lives in its trial in Burkina Faso (quickly treating diarrhea and symptoms of malaria and pneumonia in children), require that families are able to access health facilities or providers and that appropriate medicines are available.65

We are not confident in our understanding of how accessible these treatments are to those that seek them, but DMI’s midline results measured self-reported treatments received, in addition to treatment sought, so we rely on the former for our estimates of DMI’s effectiveness.66 (more)

Do the individual health practices save lives?

We have not completed a review of each of the practices DMI encourages through its broadcasts, but we believe that many of them – e.g., pneumonia treatment, malaria treatment, and oral rehydration therapy – represent well-established ways to prevent child mortality.67

What other information measures program success?

DMI has told us that it uses various mechanisms to determine whether (a) programs have aired at the agreed upon times, (b) people listen to DMI's programming, and (c) listeners have correctly understood the intended messages.68

Below we discuss the evidence we have seen. Broadly, we have evidence that broadcasts occur consistently and are heard by DMI’s audience, although the evidence that listeners understand the intended messages is weaker (outside of the evidence of behavior change from the midline results of the RCT above).

How reliably do the broadcasts happen?

DMI told us that it can use two tools to track whether broadcasts occur. The main method used in the RCT in Burkina Faso is broadcast monitors.

• Broadcast monitors: DMI employs individuals to listen to the radio and record whether stations play DMI's programming at the agreed upon frequency. DMI told us that early in its work in Burkina Faso, broadcast monitors indicated that some radio stations were not playing the one-minute spots at the appropriate frequency. After DMI followed up with these radio stations, the broadcast monitors reported an increase in frequency.69
• Automatic broadcast reports: DMI has received automatic reports on broadcasts from software used by two of the seven radio stations it works with. This method has only been tested in the current trial of Burkina Faso (alongside the main method of employing broadcast monitors) in two stations where it has been possible to install and train staff to use the necessary software. It is a method DMI is considering for future campaigns to automate and systemize the collection of broadcast monitoring data.70

DMI shared data from broadcast monitors for three sample months that we requested, as well as for two stations were also generating automatic broadcast reports for two months.71 The data from the automatic broadcast reports and the two broadcast monitors seem to approximate each other in most cases.72

DMI has also shared 14 quarterly funder reports, which include discussions of challenges the radio stations face in maintaining consistent broadcasts. The primary issues discussed are loss of electrical power and personnel issues, and in some cases, these issues resulted in hours or days of missed broadcasts within a particular cluster.73 In total we counted about 30 station-days of no broadcasts in the 2.5 years covered by the reports, representing about 0.4% of all broadcasts in that time. We also counted about 150 station-days of reduced broadcasts (either due to shortened hours or restricted range of signal), representing about 2% of all broadcast-days.74 DMI has told us it is interested in continuing to find cost-effective ways to reduce power outages at partner stations, such as the possibility of buying partner stations solar power systems rather than paying for airtime or station actors.

Do people hear DMI’s broadcasts?

DMI’s midline survey found that 75% of women in DMI’s broadcast areas reported recognizing at least one of two recorded spots (that had been broadcast recently) when it was played for them during the survey, compared with 20% of women in the control zone.75 This result was statistically significant.76

Do people understand the broadcasts?

Overall, DMI’s qualitative research adds some modest evidence that people in DMI’s broadcast zones understand DMI’s health messages, helping confirm the midline survey’s finding that one-off behaviors targeted by DMI were increasing from DMI’s broadcasts.

DMI’s qualitative research team asks participants to explain the messages they have heard on the radio during the feedback focus groups they conduct (described above). We have seen qualitative summary reports from these feedback groups and a synthesis report summarizing this feedback research across all seven intervention zones.77

DMI reports that most individuals are able to recall at least one of DMI’s broadcast messages and are able to explain the recalled health messages accurately. However, some individuals make errors in recalling the messages.78 The messages that were voluntarily recalled most frequently were taking children to a health center immediately when they become sick, and messages relating to malaria, diarrhea, and pre-natal care.79 Since the qualitative research team did not conduct similar interviews in the control zones it is more difficult to tell how much of participants’ message recall is due to DMI’s program alone.

What do you get for your dollar?

We estimate the cost per child life saved through DMI’s program to be about $5,236.80 This does not include other potential benefits of DMI’s program, such as preventing non-fatal cases of illnesses or other damages. DMI’s estimate of the cost per life saved is about 17 to 87 times stronger than our best guess.81

Major factors driving our cost-effectiveness estimate: [xlsx]

• We rely primarily on DMI’s midline results, rather than previous nonrandomized studies, to estimate the effect of DMI’s program on behaviors.82 As a result, we have not predicted any lives saved from higher treatment rates of malaria, a major focus of DMI’s program, since the midline results didn’t find an increase in the use of antimalarials (despite a modest increase in families seeking treatment for malaria symptoms).83
• We discount the effect implied by DMI’s midline results because they come from a single study rather than multiple studies (as we have done with other top charities).84
• We discount the effect implied by DMI’s midline results because they come exclusively from Burkina Faso, whereas additional unrestricted funds are likely to cause new programs in other countries (as we have done with other top charities). (See our Outstanding questions for a discussion of possible differences between Burkina Faso and other countries DMI may operate.)
• We discount the effect implied by DMI’s midline results to account for the possible bias associated with self-reported results.85
• We discount the expected benefits of treatment based on the possibility that the treatment received is of poor quality or is not given to the child on schedule.

For more on the role that cost-effectiveness estimates play in our reviews, see our page on cost-effectiveness analysis.

Is there room for more funds?

DMI has told us that it would like to scale up its program to five countries over the course of 2015, for which it would need about $10 million. Existing and likely funders are currently expected to account for $2.5 - $4 million of that goal.87

DMI has said that it would likely use additional unrestricted funding in this order:

• If necessary, up to the first $600,000 DMI receives would go to Burkina Faso; DMI believes it will obtain this funding elsewhere and it is therefore unlikely that additional funds effectively support Burkina Faso. DMI’s RCT program is ending in January 2015, but it has secured funding for some continued operations in Burkina Faso for the next two years (possibly only broadcasting spots four months a year, and few if any longer modules, across 28 community radio stations). DMI is also in conversations with two funders to increase funding in Burkina Faso, which would enable them to broadcast on more health messages. If those two funders decided not to make grants but DMI got unrestricted funding from a GiveWell recommendation, it would likely allocate money there first; DMI does not believe this scenario is very likely.
• If DMI receives less than $800,000, it would likely use the bulk of the funds to make a new program in Mozambique larger than it otherwise would be. DMI is actively seeking funding from bilateral agencies and others to launch a campaign in Mozambique, and expects to know what funding it will receive by mid 2015. Unrestricted funding could be used to supplement that work for a larger campaign (more stations) or to include messages that are efficient at saving lives but not favored by the groups that decide to fund Mozambique.
• If DMI receives at least $800,000 in additional unrestricted funds, DMI would likely launch a campaign in Cameroon. DMI would like to launch programming in one or two new countries (beyond Burkina Faso, DRC, and Mozambique), particularly one where bilateral agencies are less likely to provide funding, such as Cameroon, Cote d'Ivoire, Niger, Chad, or Mali. It takes about $800k to launch an office in a new country and broadcast 4 months a year for one year. Broadcasting for 12 months a year is somewhat more efficient. While some activity can be conducted for less than $800k in a new country, such as hiring someone to attract more funding for a project (as DMI successfully did in DRC with $150k of unrestricted funding from Mulago Foundation in 2014), DMI believes it is not particularly likely to take action in Cameroon with unrestricted funding less than $800k.
• If DMI receives more than $1.6 million in unrestricted funds, it would likely launch a campaign in a second new country such as Cote d’Ivoire in addition to Cameroon.
• Depending on the amount it receives, DMI would likely use some unrestricted funds to expand campaigns to reach more people or broadcast on more health topics. For example, DMI has also secured funding to launch programming in DRC in early 2015, and is continuing to look for more funders to scale up its program there, particularly to more stations. There are many bilateral aid agencies that are interested in funding health work in DRC so DMI is optimistic about its ability to find funding over the next couple years. Unrestricted funding might be used to ‘fill gaps’ in the short term or take advantage of particularly compelling opportunities that bilateral agencies aren’t interested in funding in DRC or other countries such as Mozambique and Burkina Faso.

For any campaign in a new country, DMI estimates it may take about 11 months from the time that DMI has committed funding to when the first broadcasts happen (4 months of recruiting project leads, 5-6 months to set up the project in-country, including recruiting staff, setting up an office, conducting qualitative research on obstacles to behavior change, and producing spots). Money used to supplement existing campaigns can have a quicker impact on broadcasts.88

Major outstanding questions

What biases affected DMI’s midline results?

• How well did self-reported behavior match actual behavior in the midline results?
• How much did differences between the control and intervention groups at baseline affect the midline results? Will LSHTM’s adjustment in the published midline results successfully reduce this issue or introduce another bias?
• To what extent were measured differences between the treatment and control groups driven by changes in (a) a small set of clusters or (b) a small set of villages?
• How likely is the selection method for participating villages and individuals in the midline survey to lead to bias?

What do DMI’s midline results imply about lives saved during the trial?

We expect to have a better answer to this general question once the endline results of DMI’s trial are published (expected late 2015 or early 2016). Below are narrower questions that affect our current understanding, many of which we have included in our initial cost-effectiveness estimate89 with guessed answers.

• Which metrics are most meaningful for estimating lives saved? For example, is it more appropriate to base estimates of lives saved from a specific illness on increases in treatment seeking or treatments received?
• How often does a family seek or receive treatment for a child’s illness and fail to save the life of that child anyway (for example, by failing to receive the drug on time, receiving a low quality drug, or not using the entire recommended treatment)?
• Are families that seek treatment in response to DMI’s messages more or less likely to actually need the treatment than those that seek such treatments regardless of DMI’s messages? Do DMI’s messages increase mis- or over-prescribing treatments such as anti-malarials and antibiotics? How costly are such errors?
• How strong is the evidence that the behaviors DMI promotes prevent child mortality? While we believe many of them represent well-established ways to prevent child mortality, we have not closely reviewed the relevant evidence for all of them.90
• How much health improvement that doesn’t result in saved lives should we expect to see from DMI’s program?
• Is there other external evidence about the effectiveness of promoting behavior change through mass media that dramatically strengthens the evidence for the effectiveness of DMI’s program? (Note that we did not find any such evidence in our initial search in 2012).91

How representative is the evidence from DMI’s midline results?

How much of DMI’s results rely on long format broadcasts and station development?

DMI invested a lot of time and money in helping radio stations remain on the air with quality programming during prime listening hours; it does not plan to invest as heavily in each radio station in future campaigns. The investments during the trial may have been important to DMI’s midline results in several ways:

• Enabling the stations to broadcast more of DMI’s spots than they otherwise would have by remaining on the air
• Reinforcing the health messages in the minute-long spots with the 10-15 minute modules which also included the same messages in longer narratives.
• Improving listenership of partner radio stations by increased air time and higher quality programming
• Incentivizing partner stations to prioritize airing DMI’s spots or using DMI’s suggestions for prime-time programming due to the perceived benefits of the relationship with DMI.

How much of DMI’s results rely on PMC’s concurrent broadcasts?

PMC conducted a nationwide mass media campaign to attempt to increase family planning behaviors in Burkina Faso using two 156 episode serial dramas while DMI’s trial was taking place. Their programs were broadcast on national radio stations across the country and in 22 community stations that didn’t include DMI’s control stations (at DMI’s request) but did include 6 of DMI’s 7 intervention stations. Our impression is that DMI’s campaign was significantly more intensive than PMC’s campaign.

PMC believes it was likely to have had some impact on childhood mortality because:

• PMC believes its program was successful at affecting family planning and other health behaviors, based on its endline survey (which did not have a randomized control).92
• PMC expects successful family planning, which reduces the rate of child bearing in families, increases maternal and child health as well as family resources during childhood, thus decreasing childhood mortality.93
• PMC’s dramas touched on behaviors that affect childhood mortality along with other health behaviors in addition to family planning.94

From limited information, we would guess that PMC’s broadcasts did not have a significant effect on the key behavior changes measured in DMI’s midline results (seeking and receiving treatment for children with diarrhea or symptoms of malaria or pneumonia) and so they have not affected our current estimates of DMI’s impact. However, if PMC’s broadcasts in community radio stations did have an effect on the mortality rate in some of DMI’s intervention zones, it would inflate DMI’s endline mortality results. DMI has noted that even in that scenario, the results would still represent evidence of reduced childhood mortality from mass media campaigns.

How much of DMI’s results rely on factors specific to Burkina Faso?

The success of DMI’s program may vary significantly in different contexts. DMI attempts to account for many possible differences between countries in estimating the life-saving potential of its program, including the number of childhood deaths that could be prevented with relatively easy behavior changes and the demographics of the audience for various radio networks.95

Other factors that may change the program’s effectiveness are more difficult to quantify or predict. A better understanding of these issues, and how they vary among countries, would allow us to better estimate the cost-effectiveness (and associated level of confidence) in the countries to which DMI is considering expanding its program.

What portion of people have access to antimalarials, antibiotics, and ORS but are not already taking advantage of them?

Our understanding is that DMI has limited information on the availability of health products and services for the countries in which it works.96 If fewer people have such access in Cameroon or other countries than in Burkina Faso (among those not already receiving such treatments), we would expect DMI’s program in those countries to be less effective than its program in Burkina Faso.

How much do the obstacles to improved behavior vary between and within countries?

We expect that some obstacles to behavior change, such as lack of awareness, are more easily overcome by DMI’s program than others, such as lack of availability of health supplies. For many obstacles to behavior change such as time or financial costs, it is not clear how much DMI’s program will help.

How much do relevant cultural attitudes vary among countries?

For example, could a dearth of independent media make individuals less likely to take the advice of health messages delivered through radio,97 or might some cultures have more resistance to changing specific health behaviors?

Relatedly, DMI told us that it chose Burkina Faso as the site for its RCT because it has an extremely decentralized media market, making it possibly the only country where 14 isolated media environments could be included in a randomized controlled trial. It is possible that a particularly decentralized or underdeveloped media market might make professional radio programming particularly attractive or compelling (when competing for attention with other radio stations or non-radio activities), which could make DMI’s program more effective in Burkina Faso than in other countries. The same difference might also make the program more expensive in Burkina Faso than elsewhere.

How much will related activities by governments or other NGOs affect DMI’s impact?

For example, a program that increases the availability of health supplies may significantly improve the effectiveness of DMI’s program, while a program that already uses mass media to promote similar behavior change may reduce DMI’s impact.

How much will the cost per listener vary among countries?

For example, frequency of power outages, equipment failures, or personnel issues for radio stations may vary significantly, as might labor costs, airtime costs, and the difficulty of enforcing broadcast agreements.

Based on DMI’s experience with station failures in Burkina Faso (see above), these problems have the potential to increase DMI's costs and/or reduce the frequency of DMI's broadcasts.98

How will the effects of DMI’s program change over time?

To what extent will behavior changes persist after messages cease? Will listeners become less responsive to DMI’s program after many years of broadcasts?

How will we know if future programs are successful?

Once the final results from the RCT are available, we expect to have significantly more confidence about the success of DMI’s current program, and by extension, its future programs.

However, as noted above, there may be differences between the effectiveness of DMI’s program in Burkina Faso and the effectiveness of programs in other countries. Since future programs will not have the benefit of a randomized control as is currently being used in Burkina Faso, we believe that it will be harder to know how much of an impact those programs are having. DMI’s ongoing monitoring efforts would provide some insight into how behavior changes over time, and how popular the broadcasts are in different communities. DMI has also expressed an intent to conduct time-series studies or non-randomized, quasi-experimental evaluations of future programs to get further understanding of impact; we are not aware of detailed plans.99

DMI as an organization

We have limited observations on which to base an assessment of Development Media International as an organization. However, our sense is that DMI is a strong organization:

• Track record: DMI has run four other mass media campaigns since 2006, although each with budgets less than a tenth the size of its current program in Burkina Faso. We have not investigated the earlier campaigns.
• Self-evaluation: DMI has invested unusually heavily in understanding the impact of its program by conducting an RCT of its program.
• Communication: DMI has communicated clearly and directly with us and given thoughtful answers to our critical questions.
• Transparency: DMI has been very transparent; we have not seen it hesitate to share information with us.

More on how we think about evaluating organizations at our 2012 blog post.

Sources

• 1
  • GiveWell non-verbatim summary of a conversation with Matthew Lavoie on October 14th, 2014
  • GiveWell non-verbatim summary of a conversation with Scriptwriter Guikierba Nanoano on October 14th, 2014 (Unpublished conversation)
  • GiveWell non-verbatim summary of a conversation with Matthew Lavoie on October 15th, 2014
  • GiveWell non-verbatim summary of a conversation with DMI Scriptwriters on October 16th and 17th, 2014 (Unpublished conversation)
  • GiveWell non-verbatim summary of a conversation with DMI’s qualitative research team on October 17th, 2014
  • GiveWell non-verbatim summary of a conversation with Matthew Lavoie and Pieter Remes on October 17th, 2014 (Unpublished conversation)

• 2

  GiveWell non-verbatim summary of a conversation with Radio Djawoampo staff on October 15th, 2014 (Unpublished conversation)

• 3
  • GiveWell non-verbatim summary of a conversation with Technical Advisor to the Ministry of Health, Bogard Create, on October 16th, 2014 (Unpublished conversation)
  • GiveWell non-verbatim summary of a conversation with Secretary General of the Ministry of Communication, Adama Barro, on October 17th, 2014 (Unpublished conversation)

• 4
  • GiveWell non-verbatim summary of a conversation with Basilia Coefe on October 16th, 2014 (Unpublished conversation)
  • GiveWell non-verbatim summary of a conversation with Jean-Baptiste Guidard-Schmid on October 17th, 2014 (Unpublished conversation)

• 5
  • GiveWell non-verbatim summary of a conversation with Anonymous District Medical Officer on October 14th, 2014 (Unpublished conversation)
  • GiveWell non-verbatim summary of a conversation with Midwife Thiombano Youmanli on October 14th, 2014 (Unpublished conversation)

• 6
  • GiveWell’s non-verbatim summary of a conversation with DMI on April 24th, 2014
  • GiveWell's non-verbatim summary of a conversation with Will Snell on July 31st, 2014
  • GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014
  • GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 15th, 2014
  • GiveWell's non-verbatim summary of a conversation with Roy Head and Will Snell on October 29th, 2014
  • GiveWell's non-verbatim summary of a conversation with Joanna Murray on October 30th, 2014

• 7
  • GiveWell’s non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014
  • GiveWell’s non-verbatim summary of a conversation with Prof. Simon Cousens on October 28th, 2014

• 8
  • GiveWell’s non-verbatim summary of a conversation with Kriss Barker of PMC on October 31st, 2014
  • GiveWell’s non-verbatim summary of a conversation with Julie Archer on October 27th, 2014

• 9
  • "We design and deliver radio and TV campaigns that save lives in developing countries by encouraging people to adopt healthier behaviours for themselves and their families." DMI website, what we do page 2014, Pg 1.
  • "The specific messages that DMI broadcast[s] are weighted according to their predicted impact on child mortality. DMI is changing the weight of broadcasting across its range of messages midway through the program, based on effectiveness data from the first 20 months and on changes in health indicators in Burkina Faso. DMI is focusing more on messages that have been proven to have a greater impact and are predicted to save the most lives. It will also be adjusting its modeling at the end of the trial to reflect the new evidence the RCT will provide. DMI’s recommendations at the end of the trial will include information on which messages would be most effective for future child mortality programs." GiveWell’s non-verbatim summary of a conversation with DMI on April 24th, 2014, Pg 5.

• 10
  • (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell on July 31st, 2014
  • "DMI has 35 staff members in Burkina Faso, 8 in London, and 1 in the Democratic Republic of the Congo (DRC)." GiveWell’s non-verbatim summary of a conversation with DMI on April 24th, 2014, Pg 1.
  • The titles of the staff listed on DMI’s website that appear to be based in Burkina Faso are:
    • Country Director
    • Chief Financial Officer
    • Head of Human Resources
    • Zones Liaison Manager
    • International Radio Producer (2)
    • Scriptwriter Manager
    • Radio Scriptwriter (15)
    • Research Manager
    • Senior Researcher (2)
    • Technical Manager
    • Zone Producer (2)
    • Logistician
    • Operations & Research Administrator
    • Finance Administrator
    • Driver & Mechanic
    • Driver (2)
  • DMI website, staff page 2014
  • DMI only expects about 20 staff in each country where they have a full program in the future, including only about 4-5 script writers. (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell on July 31st, 2014

• 11
  • DMI Budget 2011-2015
  • Original figures were all in British pounds but have been converted to U.S. dollars at a 1:1.6 rate.
  • The categories in the first column were created by GiveWell (we do not have high confidence in our success at assigning costs as intended):
    • Local and global operations: Administrative, financial, and global headquarters costs, as well as managerial, transportation, and other costs not easily assigned to one of the other categories.
    • Radio content production: Costs directly related to production of spots. Production of long format modules, where distinguishable, were assigned to Partner Stations since DMI believes their primary role was to build the relationship with and quality of the partner stations, rather than have a large direct impact on health behaviors.
    • Partner station support: Costs directly related to: production of long format modules (where discernible from spots), meeting with partner stations, and equipment and stipends for partner stations.
    • Qualitative research and monitoring: Research not conducted by LSHTM as a tool of the RCT, but qualitative research and media survey which are part of DMI’s program. (more)
    • External communications and fundraising: Costs associated with attracting and hosting funders, as well as all external communications, such as with the Burkinabe government.
  • DMI has told us that the only costs not included in this budget is the roughly $150k donated by Mulago Foundation, which DMI used in 2014 to hire a staffer in DRC in order to attract bilateral funding for a program in 2015. DMI believes that this staffer made the critical difference receiving about $1M for launching a program there. (GiveWell's non-verbatim summary of a conversation with Roy Head and Will Snell on October 29th, 2014)

• 12
  • "The specific messages that DMI broadcast[s] are weighted according to their predicted impact on child mortality. DMI is changing the weight of broadcasting across its range of messages midway through the program, based on effectiveness data from the first 20 months and on changes in health indicators in Burkina Faso. DMI is focusing more on messages that have been proven to have a greater impact and are predicted to save the most lives. It will also be adjusting its modeling at the end of the trial to reflect the new evidence the RCT will provide. DMI’s recommendations at the end of the trial will include information on which messages would be most effective for future child mortality programs." GiveWell’s non-verbatim summary of a conversation with DMI on April 24th, 2014, Pg 5.
  • DMI has received funds that are restricted to broadcasting on a particular set of health messages, which limits DMI’s ability to focus on health topics it believes saves the most lives. (GiveWell non-verbatim summary of a conversation with Matthew Lavoie on October 15th, 2014)
  • For more on the midline results and expectations of lives saved, see below.

• 13
  • "DMI messages addressing health facility-dependent behaviours have focused on attending ANC, health facility delivery, child’s growth monitoring, health care seeking in a health facility or with a Community Health Worker (CHW) for childhood illnesses (fever, cough or fast/ difficult breathing, diarrhoea with blood in the stools) and ORS for treating diarrhoea... DMI messages addressing home-based behaviours and related to maternal and newborn health have focused on savings during pregnancy, early breastfeeding after birth and delay of the first bath for low birth weight babies. Those messages related to child health have addressed exclusive breastfeeding, complementary food, increasing liquids and foods during diarrhoea and compliance with antibiotic for treating fast/difficult breathing. When national vitamin A distribution occurred, messages were broadcast to alert the population. Bed nets for pregnant women and under five children as well as sanitation through latrine ownership, safe disposal of children’s stools and hand washing with soap have also been promoted." LSHTM DMI RCT draft midline results June 2014, Pgs 5-6.
  • For an example of DMI’s planned content schedule in DRC, see DMI DRC Proposal July 2014, Pg 8.

• 14
  • "Fourteen local radio stations across the country, with high listenership, were selected for the evaluation, of which seven were randomly selected to broadcast short spots of 1 minute duration 10 times per day and long format programs for two hours per day, five days each week." LSHTM DMI RCT draft midline results June 2014, Pg 3.
  • "For 30 months we are broadcasting 60-second advertisements at least 10 times per day on seven radio stations (one in each intervention zone), in six languages. In addition, we are broadcasting two hours per night, five nights per week on each station. This represents a total of 70 hours per week of live radio. This would be logistically almost impossible to do using a soap opera format, for example, given the six languages involved. We needed to devise a format that is cheap, that can be broadcast daily, that can be produced ‘live’ (which costs a fraction of pre-produced radio), and yet can be controlled centrally. We have created a system of self-contained drama modules that are written in French in the capital city, emailed to our partner radio stations, and improvised live by actors on location in their own language within their two-hour shows. This works well in a fragmented media environment, which is becoming the norm in most developing countries." DMI Summary 2014, Pg 3.
  • In future campaigns DMI plans to significantly reduce or stop producing the live broadcasts to improve the cost-effectiveness of the program. ("The campaign will broadcast one-minute radio spots. The advantages of using spots are that they can be very precisely scripted, they are drama-based and highly entertaining, and they can be broadcast up to 10 times per day. Previous campaigns have shown that the frequent broadcasting of spots can lead to large shifts in behaviour. During the RCT we have also been broadcasting nightly dramas (acted out live on each station in local languages). Our qualitative research suggests that, whilst these have been effective, the spots have greater impact, and are much more cost-effective as they require less manpower and so are cheaper to produce (especially on 28 stations)." DMI Burkina Proposal 2014, Pg 1.)

• 15

  DMI believed spots would be a more effective format before the trial started because of the repetition and clarity of health messages in spots, and the lack of attention span required. Feedback from listeners during the campaigns has consistently indicated that people are better able to recall the spots than descriptions of the long format modules. Finally, regressing behavior changes measured at midline on the frequency of a message being covered by spots, and separately by long format modules, found a stronger correlation with the former, suggesting that frequency of spots on a given health behavior explain variance in self-reported behavior change more than frequency of long format modules on a given topic.
  (GiveWell's non-verbatim summary of a conversation with Roy Head and Will Snell on October 29th, 2014, LSHTM DMI RCT draft midline results June 2014)

• 16

  GiveWell's non-verbatim summary of a conversation with Roy Head and Will Snell on October 29th, 2014

• 17
  • (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell on July 31st, 2014
  • For a sample spot, see DMI’s website: http://developmentmedia.net/fr/audio-burkina-faso-breastfeeding-spot-20…
  • All participating stations in a country play the same spots during the same weeks, although the spots are translated to the local language for each station as necessary, and may be played at different times as the stations are managed independently. Each message ends with the same audio clip of a baby laughing, designed to brand the messages. (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell on July 31st, 2014

• 18
  • DMI Website Spot A 2014
  • DMI Website Spot A Audio 2014

• 19

  DMI has told us that in many community radio stations, the tendency is to have 1 or 2 hour programs that largely consist of one person speaking in an unstructured format. DMI believes the format it encourages radio stations to use is more interesting for people to listen to. It breaks the two hours into pieces to keep people’s attention and interest. In the show we saw, most of the pieces lasted 4-5 minutes, other than shorter musical pieces and the longer format modules DMI writes. (GiveWell non-verbatim summary of a conversation with Matthew Lavoie on October 14th, 2014)

• 20
  • GiveWell non-verbatim summary of a conversation with Matthew Lavoie on October 14th, 2014
  • In addition to the health messages contained in DMI’s two modules, the two-hour program also may contain other health and behavior-related topics from the content chosen by the radio station. When GiveWell’s team watched a program live in Bogande (with DMI staff serving as translators) we noted 6 behavior messages on 4 topics promoted directly or indirectly, in addition to DMI’s two modules (which were on handwashing and seeking treatment for children’s pneumonia symptoms):
    • Sleep under a bednet (subject of a listener call-in session).
    • End female circumcision (announcement of a public event promoting ending female circumcision).
    • Send children to school rather than just using them for labor.
    • Send girls to school rather than just boys.
    • Don’t eat bush meat to avoid catching Ebola.
    • Wash your hands appropriately to avoid spreading Ebola.

    (GiveWell’s observations while visiting Radio Djawoampo, Bogande, Burkina Faso on October 14th and 15th, 2014)

• 21
  • "We needed to devise a format that is cheap, that can be broadcast daily, that can be produced ‘live’ (which costs a fraction of pre-produced radio), and yet can be controlled centrally. We have created a system of self-contained drama modules that are written in French in the capital city, emailed to our partner radio stations, and improvised live by actors on location in their own language within their two-hour shows." DMI Summary 2014, Pg 3.
  • In Burkina Faso the community radio stations broadcasting these dramas are generally running on very low budgets and have limited professional programing. For example, DMI reports that many community radio stations are willing to play DMI’s broadcasts in exchange for the experience it will gain in live broadcasts from the dramas, rather than for cash payment. (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell on July 31st, 2014

• 22
  • (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell on July 31st, 2014
  • For sample clips, see DMI’s website: http://developmentmedia.net/fr/video-burkina-faso-hygiene-drama-2013

• 23

  The longer format modules are guided by the same health and messaging research described below, but require significantly less production work per script since they are not recorded at DMI headquarters and do not receive the same level of oversight. (GiveWell non-verbatim summary of a conversation with DMI Scriptwriters on October 16th and 17th, 2014 (Unpublished conversation))

• 24

  Among the three researchers there are at least two advanced degrees in social psychology and sociology.

  • DMI profile of Research Manager Rita Lamoukri 2014
  • ”He holds a Master’s degree in social psychology.” DMI profile of Senior Researcher Souleymane Salouka 2014
  • ”She holds a degree in environmental sciences and a Master's Degree in sociology.” DMI profile of Senior Researcher Mireille Belem 2014

• 25
  • (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014
  • DMI sent us message briefs it created in Burkina Faso:
    • DMI Message Brief on ARIs
    • DMI Message Brief on Breastfeeding
    • DMI Message Brief on Vitamin A
    • DMI Message Brief on Maternal Health
    • DMI Message Brief on Malaria
    • DMI Message Brief on Low Birthweight
    • DMI Message Brief on Hygiene
    • DMI Message Brief on Family Planning
    • DMI Message Brief on Diarrhoea

• 26

  DMI Message Brief on ARIs, Pg 1.

• 27

  GiveWell non-verbatim summary of a conversation with DMI Scriptwriters on October 16th and 17th, 2014 (Unpublished conversation)

• 28

  GiveWell non-verbatim summary of a conversation with DMI Scriptwriters on October 16th and 17th, 2014 (Unpublished conversation)

• 29

  GiveWell non-verbatim summary of a conversation with DMI Scriptwriters on October 16th and 17th, 2014 (Unpublished conversation)

• 30
  • (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014
  • DMI researchers create a synthesis from each "pretesting research" session; we have seen several examples from Burkina Faso:
    • DMI Pretesting Synthesis Diarrhoea 2 July 2012
    • DMI Pretesting Synthesis Hygiene 2 January 2013
    • DMI Pretesting Synthesis Breastfeeding February 2013
    • DMI Pretesting Synthesis Maternal 3 July 2013
    • DMI Pretesting Synthesis ARI 4 October 2013
    • DMI Pretesting Synthesis Malaria 5 December 2013

• 31

  GiveWell non-verbatim summary of a conversation with Matthew Lavoie on October 14th, 2014

• 32
  • (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014
  • DMI researchers create a report from each "feedback research" session and has showed us these examples and summary report:
    • DMI Monitoring Report Solenzo July 2014
    • DMI Monitoring Report Bogande December 2013
    • DMI Feedback Research Summary September 2013

• 33
  • ”As a result of the midline results, we have revised the message calendar to ensure that we maximise the potential impact for the remainder of the intervention. The process used is explained below. Stage 1 For each message, the following information was compiled: Impact (baseline to midline): (1) The absolute percentage increase in coverage of the intervention between the baseline and midline and (2) the Odds Ratio Predictions (midline to endline): (1) Total number of deaths prevented by message and (2) deaths prevented as a percentage of total number of deaths prevented. These predictions were generated using the Lives Saved Tool (LiST) and were based on a national projection in which coverage for all key behaviours increased by 5% between midline and endline. Dose (baseline to midline): Percentage of time that the message has been broadcast as a proportion of total broadcasting time at midline. Based on: the above information; the midline results impact graphs (which illustrate baseline levels); factors regarding seasonality and campaign timing (e.g. likelihood of impact within final 43 weeks of campaign), a revision to the broadcast dose in the remainder of the campaign was proposed. A meeting was held to review the proposed revised ‘doses’ and some changes were made. Specifically: continued promotion of colostrum and ANC promotion (at a lower dose) during the remainder of the campaign. Stage 2A table was compiled that predicted the percentage number of lives that would be saved by each message (as a proportion of the total number of lives saved). These predictions were generated using LiST and were based on the assumption that each message would have a similar impact per ‘broadcast week’ in the final 43 weeks of the campaign as it did in the first 87 weeks of broadcasting (BL to MD). The prediction was based on the revised dose for each message that had been decided in stage 1… [see chart in original] Given the emphasis and concentration of spots promoting treatment-seeking in the remainder of the campaign and in order to increase the variety of messaging from both a creative and listener-experience perspective, it was decided that we should maximise the variety of messages contained within the modules each week. It was decided that the 10 modules broadcast each week would include 1 module on each of the 8 main themes selected for spots, plus one module promoting bednet use and an additional module on seeking treatment for malaria.” DMI Funder Report 2014 Q1, Pg 1.
  • "The specific messages that DMI broadcast[s] are weighted according to their predicted impact on child mortality. DMI is changing the weight of broadcasting across its range of messages midway through the program, based on effectiveness data from the first 20 months and on changes in health indicators in Burkina Faso. DMI is focusing more on messages that have been proven to have a greater impact and are predicted to save the most lives. It will also be adjusting its modeling at the end of the trial to reflect the new evidence the RCT will provide." GiveWell’s non-verbatim summary of a conversation with DMI on April 24th, 2014, Pg 5.
  • Our understanding is that the estimates of lives saved was largely based on the increase in specific behaviors between baseline and midline in the intervention zones, per week of broadcasts on that message. (DMI midline results presentation 2014, slide 21.)
  • If true, we are unsure why DMI used gross increases in the intervention zones for this estimate rather than the difference in difference estimates (which subtract out the parallel changes in behavior in the control zone to account for background trends). For example, DMI notes that use of antimalarials increased by 15.9 percentage points in the intervention zones, or 1.59 percentage points per week of broadcasts, thus predicting 5042 lives saved in the final 43 weeks of broadcasts. However, the difference in difference estimate for DMI’s impact on antimalarial treatment received was about zero, implying DMI’s program is saving no lives from increased antimalarial treatments.
  • Compared with the frequency of messages in the first 1.5 years of the trial, the updated message schedule increases frequency of seeking treatment for malaria symptoms, pneumonia symptoms, and diarrhea; exclusive breastfeeding; and giving birth in a health center. (DMI midline results presentation 2014, slide 21)
  • DMI reports that for campaigns that are only one year long, this type of mid-line adjustment would likely not be used, but that for longer campaigns it would be included, possibly using time series analysis (sampling part of the population regularly).
  • (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 15th, 2014
  • "For future child mortality programs, DMI plans to use a quasi-experimental design to measure a wide range of outcomes. This could involve administering a series of surveys to do a time series analysis, creating non-randomized controls, or using propensity score matching." GiveWell’s non-verbatim summary of a conversation with DMI on April 24th, 2014, Pg 6.

• 34
  • DMI DRC Proposal July 2014 (unpublished document)
  • @DMI Cameroon Proposal 2014@ (unpublished document)
  • DMI says that in many of the countries it has considered working, the national stations tend to be less expensive per listener, but the listeners tend to already have a lower childhood mortality rate as the national station broadcasts reach a higher ratio of urban listeners. Working with community radio stations makes it possible to focus more directly on the poorest communities with the highest mortality rates, and to tailor messages to those communities by broadcasting in local languages and accounting for local customs where possible. DMI has also considered using a combination of national and community radio broadcasts in some countries. (GiveWell non-verbatim summary of a conversation with Matthew Lavoie on October 14th, 2014)

• 35

  DMI believes the quality and motivation of a station’s management is important for expecting that the station will continue to operate for the duration of the program and will be resilient to challenges such as equipment failures or personnel issues. High quality staff make it much easier for DMI to be confident that the station will be able to play the broadcasts on schedule. The listenership’s size, language mix, and starting childhood mortality rate all effect estimated impact. Contracts with radio stations formalize compensation for airtime and can include in-kind contributions such as training station personnel or providing solar power systems. The biggest differences in costs per person reached are between using national networks versus working with many unaffiliated community radio stations. (GiveWell non-verbatim summary of a conversation with Matthew Lavoie on October 14th, 2014)

• 36
  • GiveWell non-verbatim summary of a conversation with Matthew Lavoie on October 14th, 2014
  • After the randomization process, DMI reported, “...The good news is that we have by chance selected the radio stations with which we are most positive about the opportunities to establish effective working partnerships.” (DMI Funder Report 2011 Q3) That comment suggests a possible source of bias that might inflate the midline results.

• 37

  GiveWell non-verbatim summary of a conversation with Matthew Lavoie on October 14th, 2014

• 38
  • DMI partner station stipend expenses June 2012 (unpublished document)
  • DMI partner station stipend expenses March 2013 (unpublished document)
  • DMI partner station stipend expenses December 2013 (unpublished document)
  • The actual amounts ranged from 350k - 750k West African CFA francs/month.

• 39
  • GiveWell non-verbatim summary of a conversation with Matthew Lavoie on October 14th, 2014
  • DMI partner station stipend expenses June 2012 (unpublished document)
  • DMI partner station stipend expenses March 2013 (unpublished document)
  • DMI partner station stipend expenses December 2013 (unpublished document)

• 40

  GiveWell non-verbatim summary of a conversation with Matthew Lavoie on October 14th, 2014

• 41

  Technically the method of measuring child mortality will also be through self-report of the mother, but we believe that reporting on extremely discreet, memorable, and publicly-known events very likely less prone to the biases associated with other self-reported data.

• 42

  LSHTM DMI RCT draft midline results June 2014

• 43

  We conducted initial research on the evidence for health behavior change from mass media in 2012 and found it inconclusive, which led us to deprioritize further research at that time.

• 44

  Note, footnotes have been omitted, and one typographical error was removed. LSHTM DMI RCT draft midline results June 2014, Pg. 3

• 45
  • Dr. Simon Cousens, the lead researcher on DMI’s trial, made a similar point to us, saying, “DMI’s campaigns had a large effect on some behaviors but little or no effect on others. They
    were more effective in changing infrequent behaviors, such as treatment of diarrhea using
    oral rehydration therapy (ORT), than in changing daily habits. Professor Cousens suggested
    that this might be because a woman with an ill child has a clear incentive to try a new
    treatment, but people experience fewer obvious concrete benefits to changing daily habits.” GiveWell’s non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014
  • As the version of the midline results we have seen is a draft, further changes to the results may happen before the midline results are published. GiveWell’s non-verbatim summary of a conversation with Prof. Simon Cousens on October 28th, 2014

• 46

  For example, “sought treatment for a child’s diarrhea within 24 hours” is a strict subset of “sought treatment for a child’s diarrhea”, since all who did the former did the latter as well, by definition. We made an exception for “received oral rehydration salts (ORS) for a child’s diarrhea,” which is technically a subset of “received ORS or more liquids or home made solution (HMS) for a child’s diarrhea”, but is an important metric we didn’t want to exclude.

• 47

  Row definitions

  • ”Sought medical assistance for symptoms” includes seeking treatment or receiving a consultation in response to a child’s symptoms.
  • ”Received specific medicine after symptoms” for diarrhea (ORS), malaria (antimalarials), and pneumonia (antibiotics).
  • ”Home-based diarrhea responses” includes home made solutions (HMS), more liquids, or similar or more food.
  • ”Used full antibiotic/antimalarial course” includes a five day treatment for antibiotics and a three day treatment for antimalarials.
  • ”Used a health facility during pregnancy and birth” includes prenatal checkups and delivery.
  • ”Sanitation and hygiene” includes washing hands with soap before and after specific activities and having facilities such as latrine, pot, or soap available.
  • ”Food and nutrition” is about diet and frequent supplements for children.
  • ”Other health behaviors” includes using bednets, taking vitamin A supplements, and others.

  Column definitions

  • ”Improvement in control” refers to the percentage of women in the control group that reported this behavior when relevant at midline minus the percentage at baseline. For example, in the control group, the portion of women that gave birth in a health facility was 81.8% at baseline and 93.3% at midline, so for this measure, the control improved 11.5 percentage points. The values in this column are unweighted pooled averages of all the values in each category (this example is averaged into “Used a health facility during pregnancy and birth”).
  • ”Improvement in intervention” refers to the percentage of women in the intervention group that reported this behavior when relevant at midline minus the percentage at baseline. To continue the example above, the portion of women in the treatment group who gave birth in a health facility was 56.0% at baseline and 65.3% at midline, for a 9.3 percentage point increase. Again, the values shown are unweighted pooled averages of the values in each category.
  • ”Individual-based improvement in intervention minus control” is the difference between the prior two columns. In the example above, it is (9.3 - 11.5) = -2.2 percentage points. Again, the values shown are unweighted pooled averages of the values in each category.

• 48

  GiveWell’s non-verbatim summary of a conversation with Prof. Simon Cousens on October 28th, 2014

• 49
  • "The results of the baseline survey indicated good balance between intervention and control clusters for home-based behaviours. While antibiotic and antimalarial treatments, as well as the delay in accessing these treatments, were balanced between arms, other health facility-dependent behaviours - ANC, facility delivery, attendance at well baby clinics, health care seeking and Oral Rehydration Solution (ORS) treatment - tended to be better in the control arm compared to the intervention arm." LSHTM DMI RCT draft midline results June 2014, Pg 4.
  • "This imbalance probably reflects better access to health facilities in the control arm compared to the intervention arm. At baseline, GPS coordinates were recorded for each compound and distance to the closest health facility was calculated as the crow flies. 40% of women lived less than 2 km away from a health facility in the control arm versus 18% in the intervention arm. Among women sampled for the midline survey, these proportions were 47% versus 16% respectively (Figure 1)." LSHTM DMI RCT draft midline results June 2014, Pg 4.

• 50

  "On average though, the control group had a lower baseline child mortality rate. The average child mortality rate was 136 per 1000 in the intervention group and 102 per 1000 in the control group. Professor Cousens believes that one reason for this is that the intervention clusters were on average farther from the capital of Burkina Faso. There is a correlation between the distance of a cluster from the capital and the mortality rate in a cluster, partly because clusters that are farther from the capital receive less investment." GiveWell’s non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014.

• 51

  "Demographic characteristics of interviewed women were similar between surveys. Women were 30 years old on average in both arms. About 40%, 20%, 13% and 12% of women overall belonged to the Mossi, Gourmantche, Gournoussi and Peulh ethnic groups respectively (Figures 2a and 2b). In the two arms, although Mossi remained the main ethnic group, Gourmantche and Peulh were more common in the intervention arm and Gournoussi were more common in the control arm. Gouin, Karaboro and Turka were specific to the [town #2] cluster (intervention arm). While about half of women were muslim and half christian in the control arm, 60% were muslim in the intervention arm. In both arms, nearly all women were married, of whom about 40% were in a polygamous union. Education was quite low with only around 15% of women in the control arm and 10% of women in the intervention having been to school." LSHTM DMI RCT draft midline results June 2014, Pg 4.

• 52

  DMI Email to GiveWell October 3rd, 2014

• 53

  "The first section of the interview collected information on basic demographic characteristics and radio listenership. Subsequent sections addressed preventive and curative behaviours of relevance to child health. Questions regarding maternal and newborn health, i.e. antenatal care (ANC) and delivery, referred to the woman’s last pregnancy of more than 6 months duration. Questions regarding health care seeking, bed net use, nutrition and sanitation applied to her youngest child less than five years old. Illnesses (fever, cough, fast or difficult breathing, diarrhoea, dysentery) were recorded using a recall period of two weeks preceding the interview." LSHTM DMI RCT draft midline results June 2014, Pg 3.

• 54
  • GiveWell’s non-verbatim summary of a conversation with Kriss Barker of PMC on October 31st, 2014
  • PMC summary results September 2014

• 55

  "The research team is concerned about the impact of a large health worker training program funded by the Bill & Melinda Gates Foundation and other funders. The program is active in two clusters, one from the control group and one from the intervention group. The research team believes that the program is having a significant effect in the control cluster but is uncertain how this compares to its effect in the intervention cluster. The research team is reviewing the study’s analysis plan with the study’s advisory committee. If the research team decides that the training program is significantly affecting the results of its RCT, it may perform a sensitivity analysis in which it removes those two clusters from the data." GiveWell’s non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014.

• 56

  "There was recently a large bed net distribution program. Nearly 100% of households interviewed in the midline survey had at least one bed net, which was a large increase from the baseline. There were also national immunization days during which polio vaccines and vitamin A supplements were administered. The research team is documenting smaller health programs in various clusters, but in general it does not believe that there is a major difference between the programs in the intervention clusters and the control clusters." GiveWell’s non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014.

• 57

  "According to DHS, there was a large decline in mortality in Burkina Faso between 2003 and 2010. The mortality rate in Burkina Faso has been declining faster than that of most other countries. This complicates the results of the RCT, because the study may have less power than planned to detect a reduction in mortality. Some of the decline in mortality rates in Burkina Faso is due to large national health programs, which the research team documented in its analysis." GiveWell’s non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014.

• 58

  "The survey detected “contamination” in Gayeri control cluster with women in part of the cluster reporting listening to Djawoampo radio station broadcasting the campaign in Bogande intervention cluster. Among 375 women interviewed in Gayeri cluster, a third - mainly in villages located to the North and North West of Gayeri (towards Bogande) - reported having listened to Djawoampo radio station in the last seven days. All women interviewed in these villages (two-thirds of all women in the cluster) were therefore excluded from analysis. To account for the much smaller sample size of Gayeri cluster, analyses were weighted by the number of observations per cluster." LSHTM DMI RCT draft midline results June 2014, Pg 4.

• 59

  GiveWell non-verbatim summary of a conversation with Matthew Lavoie on October 14th, 2014

• 60

  GiveWell non-verbatim summary of a conversation with Matthew Lavoie on October 15th, 2014

• 61

  GiveWell non-verbatim summary of a conversation with Matthew Lavoie on October 14th, 2014

• 62
  • The survey instrument is written in French and is translated by the surveyors into the local languages of the interviewees. Where necessary, a translator was also used. We have used Google Translate to convert the French instrument into English, and interpreted, combined, or added context to some of the questions to make them more clear to the reader.
  • LSHTM midline survey instrument 2014

• 63

  After the randomization process, DMI reported, “...The good news is that we have by chance selected the radio stations with which we are most positive about the opportunities to establish effective working partnerships.” (DMI Funder Report 2011 Q3) That comment suggests a possible source of bias that would inflate the midline results.

• 64
  • For example, Prof. Cousens noted the following in our conversation: "It is unclear whether a higher baseline mortality would lead to a larger or smaller relative reduction in mortality. It may be that regions with high baseline mortality have more opportunity for improvement in mortality rates. On the other hand, it may be that regions with higher mortality rates are poorer, more remote, and have worse health services, which would make it more difficult to reduce mortality…. The fact that the midline survey did not demonstrate positive impacts on some behaviors suggests that there was not a strong tendency for subjects to overstate their behavioral changes." GiveWell’s non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014.
  • For more detail on common bias issues in formal studies, see this page.

• 65

  For example, we believe the following issues might prevent a family from successfully receiving health supplies or services despite seeking them:

  • The distance to the nearest health center is too long to walk and any alternative transportation is too expensive.
  • The family doesn’t have a member that can make the trip, due to poor health or lack of free time.
  • The health center is open too few or unpredictable hours.
  • The health center has long wait times for service.
  • The health center is out of stock of the needed supplies.

• 66
  • DMI receives reports from the Ministry of Health in Burkina Faso that document the availability of health supplies in the regional offices that distribute supplies to health centers throughout the country. In 2013, the government reported that 75.9% of health centers in Burkina Faso were never short on key health supplies and that the figure for the seven regions where DMI has been active was 88.4%. (DMI Health Supply Availability 2013)
  • We have not seen the technical details of how these figures were compiled and are not aware of any monitoring or auditing process that is used to verify the accuracy of these numbers. Furthermore, the existence of key health supplies at regional distribution centers does not guarantee their availability at health centers or community health workers (which are supposed to make antimalarials and ORS available for sale in each village).
    DMI told us that its research team visits community health workers and health centers during field visits and ask about and verify the availability of antimalarials, ORS, and antibiotics for pneumonia.
    • GiveWell non-verbatim summary of a conversation with DMI’s qualitative research team on October 17th, 2014
    • DMI Updated Monitoring Guide October 2013
  • DMI does not systematically record the health supply information they discover; instead their monitoring reports include discussions of the major limiting factors to behavior change they perceive in each village (including the availability of health supplies where applicable).
    • DMI email from Joanna Murray on November 3rd, 2014
    • See the “Monitoring Reports” in the sources table or the DMI Feedback Research Summary September 2013.
  • Overall, DMI does not believe that the availability of health supplies is a major limiting factor in the effectiveness of DMI’s program in Burkina, although antibiotics for pneumonia are harder to access than antimalarials and ORS for diarrhea.
    • DMI email from Joanna Murray on November 3rd, 2014
    • DMI has not noticed any inconsistencies with the government’s reports of stockouts. (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014
    • Antibiotics for pneumonia are generally not available from community health workers. (GiveWell non-verbatim summary of a conversation with DMI’s qualitative research team on October 17th, 2014)
    • DMI reports that not having health centers (CSPS) nearby is a limiting factor in treatment for pneumonia.
      “Les diarrhées: les messages DMI ont apporté aux parents des connaissances supplémentaires pour le traitement de leurs enfants, mais il y a encore quelques parents qui affirment que le "toupai" est préconisé comme traitement de la diarrhée de l’enfant ainsi que les décoctions de plante.
      Les IRAs/pneumonie: l’absence de CSPS dans le village, la méconnaissance même des symptômes de la maladie et de son processus d’évolution (certains parents trouvent que la toux et le rhume ne sont pas graves), la valorisation des soins traditionnels qui favorise le recours aux plantes traditionnelles pour soigner la toux avant de recourir au CSPS en cas d’échec (surtout à Bogandé).” DMI Feedback Research Summary September 2013, Pg 14.
      Google Translate suggests this translation: “Diarrhoea: DMI messages provided parents with additional knowledge the treatment of their children, but there are still some parents who say the ‘Toupai’ is recommended as a treatment for childhood diarrhea and decoctions plant.
      IRAs / pneumonia: the absence of CSPS in the village, the same lack of knowledge symptoms of the disease and its process of evolution (some parents find that cough and colds are not serious), the development of traditional care that promotes the use of plants traditional to treat cough before resorting to CSPS in case of failure (especially Bogandé).”
  • In assessing the extent to which access to health supplies is a limiting factor in health behaviors of other countries in which DMI is considering launching a program, DMI’s primary quantitative metric has been looking at the portion of those who say they sought treatment for a child’s symptoms (such as a fever) that also report actually receiving treatment (such as antimalarials). (GiveWell's non-verbatim summary of a conversation with Joanna Murray on October 30th, 2014)
  • The metric has flaws, such as “over-counting” people that report their child receiving treatment despite not having sought such treatment (such as when a family keeps an extra supply of ORS for diarrhea in the home, a practice many health NGOs encourage) and the possibility that some people who don’t believe they have access to treatment may not report seeking it even if they would have if it were available. This is a concern for DMI’s ability to correctly predict the availability of health supplies in other countries (see our Outstanding questions) and if the ratio is a reasonable metric for the availability of health supplies, it seems to imply that the availability in Burkina Faso is a major limiting factor. Looking at the “received/sought” ratio for DMI’s treatment and control groups at baseline and at midline, the samples range from 27% - 50% for antimalarials, 30% - 46% for ORS, and 51% - 74% for antibiotics to treat pneumonia.
  • In other words, if less than half of the families seeking treatment for childhood malaria and diarrhea actually receive that treatment, that seems to imply a major availability problem. Additionally, the higher rates for pneumonia antibiotics is counter-intuitive given that it is not stocked by community health workers like the other two drugs are, possibly implying that people are not claiming to have sought treatment for pneumonia as often because they know there isn’t a source nearby.
  • For our cost effectiveness analysis we rely on the measures of increases in treatment received, rather than treatment sought, as a more direct measure of expected lives saved during DMI’s trial.

• 67

  DMI suggests the following sources for evidence of the life saving effects of these treatments:

  • UNICEF Facts for Life 2010
  • Lancet Child Survival Series 2003
  • WHO/PMNCH (2011) essential interventions, commodities and guidelines for RMNCH. http://www.who.int/pmnch/topics/part_publications/essential_interventio…
  • Development and use of the Lives Saved Tool (LiST): A model to estimate the impact of scaling up proven interventions on maternal, neonatal and child mortality (2010) International Journal of Epidemiology; 39, Supplement 1
  • Technical inputs, enhancements and applications of the LiST (2011) BMC Public Health, 11, Supplement 3
  • Lassi et al (2014) Essential interventions for maternal, newborn and child health: background and methodology, Reproductive Health, 11, supplement 1.

  DMI Email to GiveWell October 3rd, 2014

• 68

  (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014

• 69

  (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014

• 70

  (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014

• 71
  • DMI broadcast monitor guide 2014
  • DMI broadcast monitoring results June 2012
  • DMI broadcast monitoring results March 2013
  • DMI broadcast monitoring results December 2013

• 72
  • DMI Broadcast Tracker Summary 2012
  • We did notice that the numbers from one monitor are systematically about 40% higher than those from the other monitor and the software: the average broadcasts logged per day for the 2 observers and software at Radio Loudon in Sapouy, respectively, were 9.4, 13.3, and 9.6. We are not sure of the reason for the discrepancy or whether it is ongoing. (DMI Broadcast Tracker Summary 2012)

• 73
  • General comments from the reports
    • "...with the exception of [radio #1], our partner stations have all broadcast DMI spots 10-16 times per day throughout the entire quarter; and that [radio #1] is now back up to the same level." DMI Funder Report 2013 Q3, Pg 2.
    • "As before, power supply has been a headache, but by loaning generators and subsidising solar installations we have kept all seven stations on air with only very minor interruptions… The most important insight—which seems obvious with hindsight—is that energy, perhaps even more than creativity or research, is likely to be the determining factor of a successful community radio campaign. This quarter’s energy problems, occurring during the midline field research, were particularly problematic… The second crucial lesson that our 22 months of broadcasting has confirmed is that in managing an output-intensive partnership with a community radio station there is no substitute for ‘ears on the ground’. In spite of two years of collaboration, of important investments made in energy security, studio equipment, training and programming, we still only learn of serious problems in our partner stations during our field visits. This is particularly true in regards to the many personnel issues that have an impact on our interactive programs. In [town #6], [radio #6]’s star program host, and the principal host of our interactive program, [name redacted] passed the civil service exams to be a teacher and enrolled in the inductory training program without informing the management of [radio #6], or its parent organization. As this training made her increasingly unavailable for her daily programming responsibilities and for ‘DMI’ rehearsals she was obliged to tell the station director that she was hoping to teach full-time while continuing to work for [radio #6]. Bassirou learned of these developments during one of his scheduled visits only after expressing our frustrations at [her] having missed several rehearsals. In the end, the radio station’s management decided to fire [her]. Her departure has forced us to reorganize the team that broadcasts our interactive program. Cheikh has worked with the station’s management to redistribute programming responsibilities and convinced the station director to recruit a new staffer." DMI Funder Report 2013 Q4, Pg 1-2.
    • "The media project in Burkina Faso continues to run very much according to plan, notwithstanding the usual technical and logistical headaches." DMI Funder Report 2014 Q1, Pg 1.
  • Specific issues that caused missed broadcasts
    • [Radio #5] in [town #5] failed to broadcast anything for 10 days in March and 2 days in November 2013 due to power outages. In both cases broadcasts only resumed when DMI bought and loaned a generator to the station.
      ("In early November we purchased a second 7kva backup generator and sent it to [radio #5] in [town #5]. As discussed in the Q1 2013 report, we purchased a first 7kva generator in March 2013. Our intention was that this generator be moved from station to station as our different partner stations experienced energy outages. This generator was first sent to [town #5] where [radio #5] had lost ten days of broadcasting to power outages. Once [radio #5]’s energy supply stabilized in June—the local cooperative repaired their generator—we sent DMI’s generator to [radio #4] in [town #4] who were experiencing seasonal outages caused by heavy rains. The second week of November, during the midline field research, both [radio #4] and [radio #5] were experiencing power outages. Anxious to get [radio #5] back on air during the midline research we purchased (€5,393) a second generator and installed it in [town #5], limiting the power outages to two days." DMI Funder Report 2013 Q4, Pg 1.)
    • [Radio #1] in [town #1] had power outages for much of 2013 which reduced their broadcasting from 16 hours per day to four, preventing some of DMI’s programming from being broadcast. DMI bought the station a solar generator in October and rented them a fuel generator to last until the solar system was installed, which ultimately happened in February 2014. We do not know how many broadcasts were skipped during that time.
      • "...as feared/anticipated [radio #1] in [town #1] has started to suffer serious energy shortages. From sixteen hours of broadcasting a day [radio #1] has been reduced to 4 hours on most days. The power shortages have started to reduce the number of spots they broadcast per day and have disrupted the interactive program. We have rented a backup generator in [town #1]—the station pays the fuel—to get the station back on-air immediately and have contracted with a supplier to install a solar energy system at [radio #1] (the station is the only one of our partner stations who has saved some of DMI’s monthly production stipend in order to make capital investments)." DMI Funder Report 2013 Q3, Pg 2.
      • "[Radio #1] in [town #1] also experienced recurring power outages throughout November and December. As mentioned in the Q3 report, worried that [town #1]’s power supply problems would only get worse—the town depends on a local energy cooperative that is keeping two very old generators on life-support—we decided to install a solar energy system at [radio #1]. We signed a contract with a supplier in early October and rented a local generator to keep the station on air through the midline. Our pessimistic analysis of [town #1]’s energy supply has been confirmed by recent events; both of the [town #1] energy cooperatives’ generators caught fire in early January and the town of [town #1] no longer has electricity. Frustratingly, our solar supplier has experienced delays with his Spanish supplier of rechargeable batteries. We have prolonged the rental of the generator for [radio #1] through December and our solar supplier has agreed to cover a percentage of the additional rental costs." DMI Funder Report 2013 Q4, Pg 2.
      • "In February, the solar energy system contracted in October 2013 was installed at [radio #1], [town #1]. The station now has energy independence 8 hours a day, allowing them to broadcast during both the morning and evening primetime hours. Frustratingly, only weeks after the solar energy system was brought online the radio station’s amplifier overheated and was seriously damaged. The station is currently broadcasting without the amplifier, with a reduced range of 25 km. We are working with the station to replace their amplifier as quickly as possible—radio amplifiers, and their replacement components, are not available in Burkina Faso." DMI Funder Report 2014 Q1, Pg 3.
    • [Radio #1] in [town #1]'s amplifier was seriously damaged a couple weeks after the solar system was installed, reducing their broadcast range from approximately 50km to 25km. We do not know what percent of the target population previously reached by [radio #1] is not reached with the reduced range (three fourths of the broadcast area would be lost under simple assumptions), or whether the amplifier has since been replaced.
      • "Frustratingly, only weeks after the solar energy system was brought online the radio station’s amplifier overheated and was seriously damaged. The station is currently broadcasting without the amplifier, with a reduced range of 25 km. We are working with the station to replace their amplifier as quickly as possible—radio amplifiers, and their replacement components, are not available in Burkina Faso." DMI Funder Report 2014 Q1, Pg 3.
      • "Burkina Faso has a system of localised media whereby FM radio stations typically have a range of approximately 50km, which permit a cluster randomised trial." LSHTM Protocol First Draft 2013, Pg 4.
      • A perfect circle with a radius twice as large as another will have four times the area (7854 square kilometers versus 1963 square kilometers in the case of a radius of 50km and 25km, respectively). DMI doesn’t use population less than 5km away from radio towers for its surveys, but this calculation assumes those people still receive the broadcasts. In practice, changes in population density and topography complicate the calculation of lost audience; we have not seen an estimate of the lost audience from DMI.
    • [Radio #4] in [town #4] experienced seasonal outages caused by heavy rains in the first half of 2013, and separately in November. DMI arranged for a generator to supply the station power starting in June 2013. We do not know how many broadcasts were skipped as a result of power outages.
      • "[Radio #4] in [town #4] continues to rely on DMI’s backup generator to get their program on the air" DMI Funder Report 2013 Q3, Pg 2.
      • "Once [radio #5]’s energy supply stabilized in June—the local cooperative repaired their generator—we sent DMI’s generator to [radio #4] in [town #4] who were experiencing seasonal outages caused by heavy rains. The second week of November, during the midline field research, both [radio #4] and [radio #5] were experiencing power outages. Anxious to get [radio #5] back on air during the midline research we purchased (€5,393) a second generator and installed it in [town #5], limiting the power outages to two days" DMI Funder Report 2013 Q4, Pg 1.

• 74
  • We are not sure how complete the funder reports accounting of missed broadcasts are, and they are not precisely documented.
  • DMI Funder Report 2011 Q1
  • DMI Funder Report 2011 Q2
  • DMI Funder Report 2011 Q3
  • DMI Funder Report 2011 Q4
  • DMI Funder Report 2012 Q1
  • DMI Funder Report 2012 Q2
  • DMI Funder Report 2012 Q3
  • DMI Funder Report 2012 Q4
  • DMI Funder Report 2013 Q1
  • DMI Funder Report 2013 Q2
  • DMI Funder Report 2013 Q3
  • DMI Funder Report 2013 Q4
  • DMI Funder Report 2014 Q1
  • DMI Funder Report 2014 Q2

• 75

  “In the intervention arm, 75% of women reported recognising at least one of the two spots played at the end of the interview. Among regular radio listeners, this proportion increased at 88%. Reported recognition of long format program was lower, at 54% and 67% of all women and regular radio listeners respectively. In contrast, in the control arm, 20% of women reported recognising at least one of the two spots and 12% reported listening to the long format program.” LSHTM DMI RCT draft midline results June 2014, pg 7.

• 76
  • Based on the non-overlapping confidence interval bars in figure 7a
  • LSHTM DMI RCT draft midline results June 2014, Pg 13.

• 77
  • DMI Monitoring Report Solenzo July 2014
  • DMI Monitoring Report Bogande December 2013
  • DMI monitoring report Banfora July 2013
  • DMI monitoring report Bogande April 2013
  • DMI monitoring report Djibo March 2013
  • DMI monitoring report Kantchari May 2013
  • DMI monitoring report Ohya July 2013
  • DMI monitoring report Sapouy March 2013
  • DMI monitoring report Solenzo June 2013
  • DMI monitoring report Banfora January 2014
  • DMI monitoring report Djibo February 2014
  • DMI monitoring report Kantchari March 2014
  • DMI monitoring report Ohya June 2014
  • DMI monitoring report Sapouy December 2013
  • DMI partner station stipend expenses December 2013
  • DMI partner station stipend expenses June 2012
  • DMI partner station stipend expenses March 2013
  • DMI Feedback Research Summary September 2013

• 78
  • "La plupart des répondants restituent les messages correctement, mais certains répondants reprennent les messages avec des erreurs, par exemple sur : le nombre de CPN recommandé, l’âge d’introduction de l’eau et de la bouillie enrichie dans l’alimentation du bébé et l’âge limite de la consultation du nourrisson sain (pesée)." DMI Feedback Research Summary September 2013, Pg 5.
  • Google’s Translation tool suggests this translation for the above passage: "Most respondents restore the messages correctly, but some respondents show messages with errors, for example: the number of CPN recommended age introduction of the water and porridge enriched in the baby's diet and the age limit of the consultation healthy infant (weighing)."
  • "Some hear our messages but misunderstand or reinterpret parts of them, e.g., claiming herbal concoctions are okay after a baby is 6 months or thinking exclusive breastfeeding should only be done until 3 months." DMI Funder Report 2014 Q1, Pg 6.
  • DMI also reports that it has adjusted its message several times based on misinterpretations and errors: (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014

• 79
  • “On constate que les hommes autant que les femmes retiennent bien les messages clés des spots et des modules notamment sur l’aspect du recours précoce au CSPS en cas de maladie de l’enfant. Généralement, parmi les thèmes rappelés, les mieux retenus se rapportent à la CPN, au paludisme et aux diarrhées. Les moins rappelés concerne les IRAs, la pesée des enfants et le colostrum. En outre, les thèmes diffusés la veille des entretiens individuels ou des focus group sont facilement rappelés par les participants.
    De nombreux répondants ont retenu un message transversal aux thèmes de santé dont traite DMI et qui se résume ainsi : "en cas de maladie de l’enfant il faut le conduire rapidement au centre de santé".
    La plupart des répondants restituent les messages correctement, mais certains répondants reprennent les messages avec des erreurs, par exemple sur : le nombre de CPN recommandé, l’âge d’introduction de l’eau et de la bouillie enrichie dans l’alimentation du bébé et l’âge limite de la consultation du nourrisson sain (pesée).” DMI Feedback Research Summary September 2013, Pg 5.
  • Google translate suggests this translation: “It is found that both men and women hold the key messages of good spots and modules including the appearance of the early use of the CSPS in case of illness of the child. Generally, among the subjects recalled the best selected relate to the ANC, malaria and diarrhea. Recalled less respect IRAs, weighing children and colostrum. In addition, themes released on the eve of individual interviews and focus groups are easily recalled by the participants.
    Many respondents identified a transverse message to health topics including trafficking and DMI which is as follows: "in case of illness of the child must quickly lead to the center health. "
    Most respondents correctly restore the messages, but some respondents show messages with errors, for example: the number of CPN recommended age introduction of the water and the slurry enriched in the baby's diet and the age limit of the consultation healthy infant (weighing).” DMI Feedback Research Summary September 2013, Pg 5.

• 80
  • See GiveWell cost-effectiveness model of DMI 2014.
  • GiveWell’s estimate is based in part on private information, and does not match precisely with the default values in the cost-effectiveness model above.
  • Cost per child life (under five years old) is estimated, rather than other ages, only because that’s the age group in which the vast majority of lives saved are expected to be, and on which DMI’s program focuses.
  • Our estimate uses Cameroon as an example country for our estimate as it is a country that is likely to be the destination for unrestricted funds to DMI. (more)
  • Our estimate uses the radio-listening behavior of women (rather than men or all adults) because in Burkina Faso and many of the places to which DMI expects to expand, women are more often responsible for the decisions and activities targeted by DMI. DMI believes that it is important for both men and women to hear DMI’s messages, but especially for women. (GiveWell non-verbatim summary of a conversation with DMI’s qualitative research team on October 17th, 2014)

• 81
  • "According to our model, the cost per DALY of a DMI mass media campaign in most countries is in the range of $2-$10. This would make mass media behaviour change campaigns in most countries as cost-effective as any other interventions currently used in public health. We are now carrying out a scientific trial in Burkina Faso to prove this prediction. (As a rule of thumb, there are around 30 DALYS per individual 'life', so the cost per life saved of our campaigns is $60-$300, depending on the country.)" DMI website, cost-effectiveness 2014
  • See the GiveWell cost-effectiveness model 2014 for details on our estimates of the cost-effectiveness of our top charities.

• 82

  Consequently, the benefits we predict are smaller, and we only rely on the few behaviors that DMI encouraged during the trial that have already shown results consistent with large life savings, such as increased treatment for childhood diarrhea and symptoms of pneumonia.

• 83
  • The cluster-based difference in difference (DiD) analysis found a 6.6% point increase in the families that sought treatment for their child’s malaria symptoms in areas with DMI broadcasts, relative to the control areas (p=0.249). After adjusting for the cluster’s median distance to a health center, the figure was 9.1% points (p=0.119).
  • Similarly, the cluster based DiD also found a -2.6% point increase in the families whose children received a recommended antimalarial after having malaria symptoms (p=0.577). After adjusting for the cluster’s median distance to a health center, the figure was 0.1% (p=0.976).
  • LSHTM DMI RCT draft midline results June 2014

• 84

  It is difficult to know how strong a discount to apply for this consideration. We have seen some potentially relevant figures in John Ioannidis’s analysis of biomedical literature (here and here), though the analogy between biomedical studies and DMI’s midline results has substantial limitations.

• 85

  Note that the final results from DMI’s trial, expected in late 2015, will include parent-reported deaths, which we believe to be significantly less prone to self-report bias than reporting on events or behaviors that are not as well defined, memorable, and publicly known.

• 86

  DMI has seen a draft of this review and we offered them a chance to respond to any part of it. They suggested this comment be added to this section.

• 87

  (Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Roy Head and Will Snell on October 29th, 2014

• 88

  GiveWell's non-verbatim summary of a conversation with Roy Head and Will Snell on October 29th, 2014

• 89

  GiveWell cost-effectiveness model of DMI 2014

• 90

  DMI suggests the following sources for evidence of the life saving effects of these treatments:

  • UNICEF Facts for Life 2010
  • Lancet Child Survival Series 2003
  • WHO/PMNCH (2011) essential interventions, commodities and guidelines for RMNCH. http://www.who.int/pmnch/topics/part_publications/essential_interventio…
  • Development and use of the Lives Saved Tool (LiST): A model to estimate the impact of scaling up proven interventions on maternal, neonatal and child mortality (2010) International Journal of Epidemiology; 39, Supplement 1
  • Technical inputs, enhancements and applications of the LiST (2011) BMC Public Health, 11, Supplement 3
  • Lassi et al (2014) Essential interventions for maternal, newborn and child health: background and methodology, Reproductive Health, 11, supplement 1.

  DMI Email to GiveWell October 3rd, 2014

• 91

  DMI points to this review of the evidence, which we have not looked at closely: http://www.tandfonline.com/doi/abs/10.1080/10810730.2014.918217?url_ver….

• 92
  • GiveWell’s non-verbatim summary of a conversation with Kriss Barker of PMC on October 31st, 2014
  • PMC summary results September 2014

• 93

  GiveWell’s non-verbatim summary of a conversation with Kriss Barker of PMC on October 31st, 2014

• 94

  GiveWell’s non-verbatim summary of a conversation with Kriss Barker of PMC on October 31st, 2014

• 95

  DMI cost-effectiveness Model 2014

• 96
  • We believe DMI is using metrics from DHS and MICS to try to answer one part of this question. In particular, those two surveys attempt to measure, among families with children that suffered from specific symptoms in the two weeks before the survey (such as diarrhea or those associated with acute respiratory infections or with malaria), (a) how many families report seeking professional treatment for that child, and (b) how many families report receiving treatment for that child. (For example, "Pourcentage d’enfants de moins de cinq ans ayant présenté des symptômes d’Infection Respiratoire Aiguë au cours des deux semaines ayant précédé l’enquête et, parmi ces enfants, pourcentage pour lesquels on a recherché un traitement auprès d’un établissement ou d’un prestataire de santé et pourcentage à qui on a administré des antibiotiques comme traitement" DHS Survey DRC 2008, Pg 128. Google Translate suggests this translation: "Percentage of children under five years who had symptoms of acute respiratory infection in the two weeks preceding the survey, and of these children, the percentage for which we have sought treatment to an institution or a health care provider and percentage who were given antibiotics as treatment" translate.google.com, translated September 16th, 2014.)
  • Our understanding is that DMI uses (b) divided by (a) (as defined in the prior bullet point) as a proxy for "the portion of the time that treatment for diarrhea is available", since that is the portion of the time which it was received, of the total times it was sought. ((Notes from this conversation are not yet published.) GiveWell's non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 15th, 2014) We see two possible problems with using this approximation:
    • It assumes that people that are not seeking treatment would seek treatment if they were convinced they should, whereas some might not be able to seek treatment at all. For example, if there is no health facility within walking distance, or if no one in the family has the time and ability to seek treatment, then being convinced to seek treatment may not help.
    • It assumes that all received treatment is caused by seeking it, whereas at least some appears to be received without being sought. In the summary results from the most recent DHS survey in DRC, for example, more people receive treatment for several diseases than seek it in several subgroups, so there must be ways to receive treatment without seeking it. (For example, families of boys with diarrhea sought treatment 37.7% of the time, and used treatment 38.9% of the time. DHS Survey DRC Preliminary results 2013, Pg 23.) We are not sure how to account for treatment that would be received without being sought.

• 97

  This might be a particular concern in DRC: "The political allegiance of most media outlets in DRC normally reflects that of their owner. Even media outlets which attempt to remain politically neutral often run stories which have quite clearly been sponsored by an interested party. The fact that they agree to do so reflects the chronic shortage of cash in most Congolese media organisations. Certain newspapers do run critical pieces and some TV and radio stations have openly criticised government actions or the behaviour of individual ministers. However, independent media organisations increasingly practice self-censorship. Sometimes they couch criticism of the government in obtuse editorials. Another trick is to balance criticism with laudatory coverage of other aspects of government action. The government does not react to every critical report in the local media, but crackdowns are very frequent. Critical journalists and those working for opposition media outlets are often harassed, intimidated and arrested. Several journalists have have been killed in mysterious circumstances after publishing reports critical of the government. Their murderers are almost never tried and convicted. TV and Radio stations which incur the government’s wrath are often forced off air for a period. The DRC ranked 145th out of the 179 countries listed in the Reporters Sans Frontieres www.rsf.org 2011-2012 World Press Freedom Index." Infosaid Media Landscape 2012, Pg 20.

• 98

  We are aware of some indications that infrastructure is particularly problematic in DRC, which may affect both the likelihood of issues, and the difficulty of fixing them. For example, "There is no national network of all weather roads to unite this sprawling expanse of tropical rain forest and savannah grasslands." Infosaid Media Landscape 2012, Pg 3.

• 99

  "DMI does not plan to do further child mortality RCTs after the Burkina Faso trial because they would be expensive and would mean that half of the population would not receive the intervention. Additionally, other countries would be less suited than Burkina Faso for a cluster RCT testing a mass media campaign because other countries have less localized media, so it is more likely that the control group would receive the messages broadcast to the intervention group. For future child mortality programs, DMI plans to use a quasi-experimental design to measure a wide range of outcomes. This could involve administering a series of surveys to do a time series analysis, creating non-randomized controls, or using propensity score matching. DMI may compare the effectiveness of various messages within a program." GiveWell’s non-verbatim summary of a conversation with DMI on April 24th, 2014, Pg 6.