Development Media International - November 2015 Version

We have published a more recent review of this organization. See our most recent report on Development Media International.

Donate

Published: November 2015

Since publishing this review, we have published notes from conversations with DMI in October 2020, July 2020, April 2020, March 2020 (here and here), January 2020, May 2019, June 2018, October 2017, October 2016, and March 2016. In addition, DMI shared the following documents with us in 2016:

• RCT Analysis Plan (July 2014)
• Track Record: Completed Projects
• Time Series Methodology

Summary

What do they do? Development Media International (DMI, developmentmedia.net) produces radio and television programming in developing countries that encourage people to adopt improved health practices, such as exclusive breastfeeding of infants and seeking treatment for symptoms associated with fatal diseases. The program aims to reduce mortality among children less than five years old. (More)

Does it work? From 2011 to 2015, DMI conducted a randomized controlled trial (RCT) of its program in Burkina Faso to measure impact on behavior change and child mortality. At midline, DMI found some positive effects on key self-reported health behaviors that suggested DMI’s program may cause a substantial reduction in child mortality.

In June 2015, DMI told us that it no longer expected to find a statistically significant effect on child mortality at endline because overall child mortality declined extremely rapidly in Burkina Faso over the course of the RCT, making it more difficult to distinguish the effect of DMI’s program.

In November 2015, DMI privately shared preliminary RCT endline results with us. The preliminary endline results did not find any effect of DMI's program on child mortality, and it found substantially less effect on behavior change than was found at midline. DMI believes that there were serious problems with endline data collection (note that we have not yet tried to independently assess this claim), and with the support of the trial’s Independent Scientific Advisory Committee, is planning to conduct another endline survey in late 2016 (with results available in 2017). We cannot publicly discuss the details of the endline results we have seen, because they are not yet finalised and because the finalised results will be embargoed prior to publication, but we have informally incorporated the results into our view of DMI's program effectiveness. We expect some further information about the RCT endline results to be available in 2016. We discuss the effectiveness of DMI's program in greater detail below. (More)

What do you get for your dollar? Taking into account the preliminary endline results we have seen, we do not have strong evidence to suggest that DMI's program cost-effectiveness is within the range of our other priority programs. (More)

Is there room for more funding? In September 2015, DMI estimated that over the next two years, it could productively use approximately $12 million in unrestricted funds to scale up and launch more campaigns to reduce under-5 mortality. In November 2015, DMI provided an update that it plans to use some of its existing funding to conduct a second endline survey for its RCT because it believes that serious data collection issues occurred during the first endline survey. DMI estimates that this survey will cost in the region of $200,000 to $300,000. (More)

DMI is one of our standout charities because of its:

• Unusually strong self-analysis – particularly in supporting a randomized-controlled trial (RCT) on its program.
• Standout transparency – it has shared significant, detailed information about its program with us.
• Work on a potentially cost-effective program – conceptually, mass media interventions have the potential to be highly cost-effective, though we have not seen strong evidence that they are cost-effective in practice.

Major unresolved issues include:

• The main evidence underlying DMI’s program is from a single RCT’s results that found some positive effects on key self-reported health behaviors at midline but that preliminarily find no effect of DMI's program on child mortality and substantially less effect on behavior change at endline. We cannot publicly discuss the details of the endline results we have seen, because they are not yet finalised and because the finalised results will be embargoed prior to publication, but we have informally incorporated the results into our view of DMI's program effectiveness. Other programs we recommend are generally supported by multiple strong sources of evidence.
• Even if we assume DMI’s program is saving a large number of lives, we are unsure how the efficacy and costs of the program will compare in new countries where the cultural barriers to behavior change may be very different, and where DMI will not be spending as much time and money on each radio station from which it broadcasts as it did in its RCT.

Our review process

To date, our review process has consisted of:

• Conversations with DMI CEO Roy Head, Director of Development Will Snell, Research Manager Jo Murray, and Public Engagement and Innovation Manager Cathryn Wood.1
• A site visit (photos) to DMI’s operations in Burkina Faso, including:
  • Extensive conversations with Country Director Matthew Lavoie, Radio Executive Producer and Trainer Pieter Remes, and other staff.2
  • A visit to one of DMI’s seven partner radio stations to watch preparation and live broadcast of DMI programming and speak with station staff.3
  • Meetings with national government officials that have supported DMI’s campaign in the Ministry of Health and the Ministry of Communication.4
  • Meetings with representatives from Micronutrient Initiative and the Initiatives Conseil International.5
  • Interviews with professionals in the rural health system.6
• Two conversations with the lead researcher conducting the trial to independently evaluate DMI’s program, Professor Simon Cousens, Epidemiology and Medical Statistics, London School of Hygiene & Tropical Medicine (LSHTM).7
• Conversations with independent actors with perspective on DMI or possible challenges to launching campaigns in new countries.8
• Reviewing documents DMI sent in response to our queries, including details of the midline results from its ongoing randomized controlled trial.

Previous versions of this page available here.

What do they do?

DMI produces and broadcasts radio and television programming that encourages improved health practices in order to reduce maternal and child mortality.9

DMI currently works in Burkina Faso, the Democratic Republic of the Congo (DRC), and Mozambique. Our review focuses on DMI's “child survival” campaigns (campaigns primarily aimed at reducing child mortality) because (a) we believe that any GiveWell-influenced donations to DMI are most likely to lead to scaling up child survival campaigns (more in the “Is there room for more funding?” section below), and (b) these are the campaigns for which we have seen the most evidence of effectiveness.

Table 1a summarizes the campaigns we focus on in this review. Table 1b summarizes most of DMI’s other recent and planned activities, which we do not focus on in this review; these activities have been supported by restricted funds in the past, and we expect them to only be supported by restricted funds in the future, so they are substantially less relevant to GiveWell-influenced donors (more below).10

DMI estimates its total expenditure for its 2015-2016 fiscal year to be at least $3,932,704.33 This figure does not include funds allocated to the upcoming family planning RCT in Burkina Faso or the campaigns in Tanzania. In the 2014 calendar year, DMI's total expenditure was approximately $2,328,674.34

Where do they work?

Burkina Faso

From March 2012 to January 2015, DMI broadcast messages from seven radio stations in Burkina Faso with the aim of reducing the child mortality rate in the broadcast area. The project was evaluated by an RCT (more below).35

Starting in May 2015, DMI scaled up its child survival campaign to 29 stations in Burkina Faso. DMI plans to continue broadcasting through at least 2017.36

Concurrent with the scale-up of its child survival campaign, DMI is conducting a nutrition campaign in the Sahel region of Burkina Faso. Additionally, DMI plans to begin a family planning campaign in 2016, which will be evaluated by an RCT.37 We have not evaluated either of these programs.

Democratic Republic of the Congo

In 2015, DMI conducted a child survival radio campaign in the DRC, broadcasting on 35 stations in eight provinces. DMI plans to expand this campaign in 2016 if it is able to raise funding for the project.38

DMI also conducted a radio and television family planning campaign in Kinshasa in 2015.39 We have not evaluated this campaign.

Mozambique

DMI plans to launch a child survival, family planning, and maternal health radio and television campaign in Mozambique in 2016.40 DMI's Mozambique country representative is currently doing preparatory work for this campaign.41

What health practices do they encourage?

DMI says that it chooses which health topics to focus on based on its predictions of which behaviors are likely to save the most lives per broadcast in a given country.42

DMI's messaging during its child survival campaign in Burkina Faso has focused on:43

• Getting routine care during pregnancy and delivering in a health facility.
• Seeking health care for young children who manifest symptoms including fever, cough, difficulty breathing, and diarrhea with blood.
• Treating diarrhea with oral rehydration salts (ORS) and increasing intake of liquids and foods.
• Savings during pregnancy.
• Initiating breastfeeding within an hour after birth.
• Exclusive breastfeeding up to 6 months after birth.
• Delaying the first bath and promoting skin-to-skin contact for low birthweight babies.
• Complementary feeding during the transition from exclusive breastfeeding to family foods.
• Appropriate handwashing with soap.
• Using bednets for children and pregnant women.
• Using latrines and safely disposing of children's stools.
• Participating in national vitamin A campaigns.

In the DRC, DMI's messaging has focused on child survival topics similar to those of the Burkina Faso campaign. The DRC campaign also contains some messaging on family planning topics, as well as some messaging on health topics like registering births and drinking water purification.44

The upcoming child survival campaign in Mozambique is focusing on messages similar to those broadcast during the child survival RCT. The Mozambique campaign will also include messaging on family planning and maternal health.45

What is the format of DMI’s broadcasts?

In Burkina Faso and the DRC, DMI uses 60-second recorded spots for its radio broadcasts. In Mozambique, DMI plans to produce radio spots as well as content for television.46

The 60-second radio spots are acted scenarios advertising a specific health behavior message such as, "Start breastfeeding newborns within an hour after birth." Stations broadcast the same spot at least ten times per day for a week before DMI switches to a new spot.47

Here is DMI’s English translation of a spot encouraging mothers to feed their babies colostrum (first breastmilk):48

For its national child survival campaign in Burkina Faso, DMI has phased out the 10- to 15-minute-long live drama modules that it produced during the child survival RCT. DMI believes the live drama modules to be less cost-effective than the spots, and phased them out of child survival campaigns to reduce campaign costs.49

In Burkina Faso, DMI is also piloting a viral video project. Short videos, produced in the local language and containing a health message relevant to child health, are distributed and shared via mobile phones.50

How is the broadcast material produced?

Once a health topic is chosen, there are several stages of qualitative research and production involved in creating spots. Here, we outline the standard production process for DMI's radio campaigns. DMI followed this process for its RCT campaign in Burkina Faso and plans to follow a similar production process for ongoing and future campaigns.51

Discerning specific obstacles to behavior change

DMI has a qualitative research team with three full-time employees based in Burkina Faso, and employs one qualitative researcher in the DRC.52 Before DMI broadcasts on a specific health topic, the research team seeks to understand the biggest limiting factors that prevent people in the broadcast areas from adopting the behaviors DMI plans to encourage. They do so by visiting rural villages within range of broadcasting radio stations and interviewing caregivers of children under five, chiefs, religious leaders, midwives, community health workers, and others. They combine what they learn in interviews with World Health Organization (WHO) recommendations to create 2-3 recommended messages for each health topic. The team writes a one-page message brief for each recommended message to give context for it (see footnote for examples).53

Here is one of the two behavior messages in DMI’s message brief on pneumonia, and the three barriers to that behavior it hopes to overcome:54

Writing and producing spots

Burkina Faso

In Burkina Faso, scriptwriters at DMI’s office in Ouagadougou each write two scripts for the particular message that will be the focus of an upcoming week’s broadcast (see above for a sample script).55 The scriptwriting team then narrows down the scripts to their approximately 10 favorite scripts, which are sent to London for DMI's creative director to review. The creative director makes sure that the scripts are clear and adhere to the message brief. The top four or five scripts of the batch are selected by the London office and returned to the country office. The scriptwriting team then produces these spots for pretesting.56

Democratic Republic of the Congo

For its 2015 child survival campaign in the DRC, DMI made alterations to its message creation process – in addition to producing original scripts for the campaign, scripts from Burkina Faso were repurposed and reviewed by a researcher familiar with the DRC. To assist with the spot production process, two experienced DMI scriptwriters from Burkina Faso joined the DRC team.57

Spots were selected for broadcast using a procedure similar to that used in Burkina Faso.58

For its 2016 campaign in the DRC, DMI plans to reuse the spots it used in 2015.59

Pretesting sample spots in the field

The qualitative research team brings recordings of the sample spots to a sample of villages (e.g., in the case of Burkina Faso, the sample spots were brought to two villages within one of the seven broadcasting zones). Each of the three researchers (two female, one male) interviews one focus group in each village composed of 12-13 adults of the same gender as the researcher. The researchers play all the spots and lead discussions to determine whether people understand and appreciate them, and to learn if the content matches local reality. Using this information, DMI selects four spots to air in future weeks (for examples of the “pretesting synthesis” reports that result from the spot-testing research, see this footnote).60

Sending spots to each station in the local language

The country office finishes producing the selected spots in the languages spoken in the broadcast zones. DMI then distributes the recordings to each partner radio station for broadcast.61

Soliciting feedback from listeners after broadcasts

During the RCT in Burkina Faso, the qualitative research team visited villages in one of the broadcast zones about once a month to see how broadcasts on recent messages had affected behavior in those areas. As with the visits for pretesting sample spots, the researchers each met with one focus group of 12-13 people of their own gender in each village. To solicit feedback, the researchers asked questions to understand whether people had heard the spots, understood the messages, and changed their behaviors. They also asked questions to understand what other influences were affecting their health behaviors. In some cases, this feedback affected the messaging of future spots, such as when DMI realized that some people were interpreting DMI’s message on exclusive breastfeeding to prescribe excluding drinks and liquids but not traditional herbal concoctions, which may cause diarrhea (for examples of reports produced by this feedback research, see this footnote).62

We are not sure if these feedback research visits are being conducted during the national campaign in Burkina Faso, though DMI did tell us that its qualitative research team is continuing to operate during the national campaign.63

For discussion of the qualitative research from the DRC campaign we have seen, see below.

How does DMI choose and partner with radio stations?

Burkina Faso child survival campaign

For its national child survival campaign in Burkina Faso, DMI is working with 29 radio stations.64

Some stations broadcast spots for DMI's nutrition campaign, and some stations will broadcast family planning messaging in addition to child survival messages during the family planning RCT (which will run from 2016 to 2019). We have not confirmed whether these stations are the same stations as those that are broadcasting DMI's child survival messaging.65

When selecting which radio stations or networks to partner with, DMI considers the station’s management, staff, listenership, and cost.66

DMI believes that it has developed a good reputation among Burkina Faso radio stations, which has made it easy to find stations to partner with for the national campaign.67 DMI is currently making small airtime payments to its national campaign partner stations. If more funding became available, DMI would increase these payments. DMI is also considering purchasing solar panels for partner stations as an alternative to airtime payments, but does not yet have the funding to do this.68

Democratic Republic of the Congo child survival campaign

In the DRC, DMI is working with a partner organization that handles distribution of broadcast spots and relationships with community radio stations. DMI has chosen this arrangement in part due to timing constraints (the 2015 DRC campaign was funded for one year – 4 months of setup and 8 months of broadcasting), in part due to the difficulty of traveling in the DRC, and in part because setting up a ‘new’ network of community radio stations in the DRC would unnecessarily duplicate work already done by other organizations.69

DMI told us that it had difficulty working with its DRC distribution partner this year, and plans to work with a different distribution partner for its 2016 scale-up of the DRC child survival campaign.70

Mozambique child survival, family planning, and maternal health campaign

In Mozambique, DMI is in broadcast negotiations with the national radio network and television network. DMI is optimistic about developing a positive relationship with the Mozambican national networks. DMI also plans to broadcast on community radio stations in areas where the national networks have weak coverage and/or audience share, in order to maximize the geographic coverage and audience reach of the campaign.71

Does it work?

The main evidence for the effectiveness of DMI's program comes from the midline results of a randomized controlled trial (RCT) on its child survival campaign in Burkina Faso. We discuss the results of the trial in detail below.

We focus on four questions that most affect our understanding of DMI's impact on health and mortality:

• Does DMI’s program change listeners’ behavior? (More)
• Do these behavior changes result in saved lives? (More)
• How does DMI monitor and evaluate its ongoing programs? (More)
• What monitoring and evaluation does DMI plan to implement for future campaigns? (More)

Does DMI’s program change listeners’ behavior?

The strongest evidence of DMI’s impact on listener behavior comes from the midline survey of a randomized controlled trial of DMI’s intervention.72 The midline survey measures uptake of various health practices and compares uptake between the treatment and control groups. Below, we discuss the RCT’s design, the midline results, and issues that influence our interpretation of these results. We have not used other studies to supplement the argument that mass media can successfully improve health behaviors.73

DMI’s study measured moderate increases in self-reported behavior on several important health outcomes. However, we are unsure of how to interpret these results generally and believe that the issues noted below require that any interpretation be made with caution.

Note that the results released to date are interim results. DMI plans to collect mortality data as part of its final survey, which may help answer some of our outstanding concerns about the reliability of the results.74 Update: see note above about preliminary endline results.

RCT design

From the midline results report:75

Results

DMI’s draft midline results report increases in some behaviors, particularly curative behaviors such as seeking treatment for a child who has malaria, diarrhea or pneumonia, but limited evidence of an effect on other behaviors, particularly ongoing preventative behaviors such as washing hands with soap after cleaning a child who has defecated.76

Two papers and an editorial discussing the midline results have been published in Global Health: Science and Practice, an open-access journal.77

DMI reported 54 metrics in its midline report, many of which relate to the same behavior (such as, “things you may do in response to finding that your child has diarrhea”), and a few of which measure behaviors not addressed by DMI’s campaign at all. Lacking a clear way to choose the most relevant metrics, we excluded behaviors that were strict subsets of other measured behaviors,78 and grouped the remaining 32 into the eight categories below, then calculated the average difference in difference for each category (category definitions in this footnote).79

Note that Tables 2a and 2b report summary data from a draft version of the midline results paper. A more recent version of these results is available in Sarrassat et al. 2015.

Our cost-effectiveness analysis of DMI's program is driven by changes to three curative behaviors: treatment of pneumonia, malaria, and diarrhea. The midline results measured treatment seeking and treatment received for each of these behaviors. Of these six measures, only diarrhea treatment seeking (adjusted), diarrhea treatment received (unadjusted), and pneumonia treatment received (adjusted) were statistically significant at the p < .05 level.80

Issues related to the DMI RCT

The following factors lead us to interpret DMI’s results with caution. Due to the below, there is room for a plausible story in which the midline results significantly overstate the impact that DMI’s program has had. However, it is possible that these factors have little effect on the results.81

The control group was noticeably better off than the treatment group at baseline

At baseline, there were noticeable differences between the treatment and control groups in terms of child mortality rates, distance and access to health facilities, remoteness from the capital city, and proportion of women giving birth in a health facility. These differences could conceivably make health behavior change easier for one of the two groups. For example, it may be easier to improve conditions from a worse baseline, which could cause the RCT to overstate impact. Alternatively, factors that have made the control group better off could also mean the secular trend in the control zones is steeper than that in the intervention zones, which could cause the RCT to understate the impact.82

• The control group appears to have had better access to health facilities than the treatment group. The study reports that "40% of women lived less than 2 km away from a health facility in the control arm versus 18% in the intervention arm. Among women sampled for the midline survey, these proportions were 47% versus 16% respectively."83
• The treatment clusters had higher rates of child mortality in the baseline survey, perhaps due to the fact that they tended to be further from the capital of Burkina Faso and receive less investment than areas that are closer.84
• There were differences in ethnic and cultural composition between the treatment and control arms.85

DMI told us that, having reviewed these differences between the control and treatment groups, the trial’s Independent Scientific Advisory Committee recommended a cluster-level adjustment for baseline levels in the analysis, including use of a composite confounder score to adjust for the key baseline imbalances, which has been followed by the LSHTM team in the midline results.86

Better partner station opportunities in the treatment group than in control group

The seven radio stations selected for the intervention group by the randomization process represented more promising “opportunities to establish effective working partnerships” than those selected for the control, a difference which may have made it easier to create a successful campaign.87

The midline results are measures of self-reported behavior change

Health behaviors were assessed through self-report, which may be biased.88 For example, respondents in the treatment group who listened to the radio program may inflate the effect it had on their behavior if they believe the surveyor wants or approves of a certain answer.

Declining overall child mortality

The mortality rate in Burkina Faso as a whole has recently declined faster than that of most comparable countries.89 There are two potential issues here: (a) if the programs that caused child mortality to fall affected treatment and control clusters differently, this could bias the results of the RCT, and (b) declines in mortality could lead to the study lacking sufficient statistical power to detect a program impact on mortality.

Other interventions focused on child mortality in Burkina Faso

• Population Media Center (PMC) conducted a nationwide mass media campaign to attempt to increase family planning behaviors in Burkina using two 156-episode serial dramas while DMI’s trial was taking place. PMC programs were broadcast on the national radio stations across the country and in 22 community stations that did not include DMI’s control stations (at DMI’s request) but did include six of DMI’s seven intervention stations. PMC believes it was likely to have had some impact on childhood mortality because:
  • PMC believes its program was successful at affecting family planning and other health behaviors, based on its endline survey (which did not have a randomized control).90
  • PMC expects that successful family planning, which reduces the rate of child bearing in families, increases maternal and child health as well as family resources during childhood, thus decreasing childhood mortality.91
  • PMC’s dramas touched on behaviors that affect childhood mortality along with other health behaviors in addition to family planning.92

  Our impression is that DMI’s campaign was significantly more intensive than the PMC campaign. From limited information, we would guess that PMC’s broadcasts did not have a significant effect on the key behavior changes measured in DMI’s midline results (seeking and receiving treatment for children with diarrhea or symptoms of malaria or pneumonia) and so have not affected our current estimates of DMI’s impact.

  However, if PMC’s broadcasts in community radio stations did have an effect on the mortality rate in some of DMI’s intervention zones, it would inflate DMI’s endline mortality results. Prof. Cousens has noted that even in that scenario, the results would still represent evidence of reduced childhood mortality from mass media campaigns.93

• The Bill & Melinda Gates Foundation and other funders supported a large training program for Burkinabé health workers during the trial. The program is active in two clusters, one in the treatment group and one in the control group, and may have affected one cluster more than the other.94
• There was a large national bednet distribution program in Burkina Faso during the trial as well as national immunization days that included vitamin A supplementation. The RCT researchers do not believe that the programs affected control and treatment clusters differently.95
• Other health programs may have reduced child mortality during the trial period.96

Decline in statistical power

DMI's RCT was designed to have an 80% chance of rejecting the hypothesis of no difference between the mortality rates in the treatment and control groups (at the 95% confidence level), assuming that the campaign reduced mortality by at least 19.9%. Assuming that the campaign reduced mortality by at least 15%, the RCT was designed to have a 50% chance of rejecting the hypothesis of no difference between groups. This study design was based on an assumed child mortality rate of 168 deaths per 1,000.97

However, DMI has told us that child mortality in Burkina Faso has declined to 96 deaths per 1,000 in the five years after the study was designed. Due to this decline, DMI no longer expects its RCT to demonstrate a statistically significant effect on child mortality at endline.98

Lack of blinding

Surveyors were aware of whether respondents were in the treatment or control groups, which could lead to bias in the results.99 Since some of the questions were about whether the participants had heard DMI’s spots, blinding the surveyors would have been difficult.

Contamination in the control group

Two-thirds of a control group cluster with 375 women was excluded from analysis because about half of the women surveyed in their villages reported listening to the radio campaign in the week before the midline survey.100 Including these women in the analysis might underestimate the impact of the campaign, while excluding them may introduce bias, though it would likely be small.

Unusually intensive intervention

DMI has told us its first programmatic priority during the trial was to make sure there would be a measurable decrease in lives saved in the RCT through mass media activities.101 While the primary mechanism for achieving the reduction was broadcasting the 60-second spots, DMI went above and beyond what it would typically include in a mass media campaign. This increased effort makes it difficult to estimate the impact of future, less-intensive campaigns.

Note that DMI plans to conduct less-intensive campaigns in the future because it believes that many of the elements of its current campaign to be less cost-effective than the campaign as a whole; if this is correct, then removing each of these elements will improve the cost-effectiveness of future campaigns despite a possible decrease in impact. Still, the changes add uncertainty to the expected impacts. Some examples of DMI activities not to be included in future campaigns are listed below.

• Negotiating with PMC to keep PMC broadcasts out of community radio stations in DMI control zones (PMC did broadcast programming in DMI intervention zones and on national radio).
• Producing 10 long-format module scripts per week that dramatize health messages. Producing this content was originally planned as part of the RCT as a way to help radio stations build quality content, thus developing a better listenership and improving DMI’s relationship with the radio stations. Early in the RCT, DMI decided to produce more long-format modules than originally intended and to have them focus on the same health messages as the spots instead of merely providing entertainment content.102
• Renting and buying generators, solar panels, and broadcast equipment for radio stations. DMI plans to continue to help radio stations avoid power outages, such as possibly loaning or giving solar panels to each station in lieu of cash payments, but does not plan to respond as aggressively to future outages and equipment failures.103
• Intervening heavily to improve management of struggling radio stations. DMI has spent significant staff time and energy assisting station management when necessary, including employing two individuals to work full-time at problematic partner stations. Because DMI will not be running an RCT during future campaigns, it will have more flexibility to stop working with stations that collapse or do not broadcast DMI content regularly. This flexibility may significantly reduce program costs, although doing less to assist radio stations may have negative effects on program impact, such as lower quality programming and less engaged listeners.104

The investments made during the trial may have affected the midline results in several ways:

• Enabling the stations to broadcast more of DMI’s spots than they otherwise would have by remaining on the air.
• Reinforcing the health messages in the minute-long spots with the 10-15 minute modules which also included similar messages in longer narratives.
• Improving listenership of partner radio stations through increased airtime and higher quality programming.
• Incentivizing partner stations to prioritize airing DMI’s spots or using DMI’s suggestions for prime-time programming due to the perceived benefits of the relationship with DMI.

Unreported results

One complicating factor in interpreting the midline results is that we are not sure which results speak most directly to the success of DMI’s program so far. The midline survey, on which the midline results are based, contains some potentially relevant questions for which we have not yet seen the results, including:105

• For your most recent birth, is your child still alive, or at what age did they die?
• When your child has diarrhea, what should you do?
• If your child received treatment for diarrhea/malaria/pneumonia, may I see the marked health card/prescription/medicine packaging?
• How many days were there between your child’s first symptoms and first receiving treatment?
• For how many days was your child sick?

Do changed behaviors result in saved lives?

Are health supplies and services available when sought?

Some of the health activities that DMI promotes, including the three that DMI believes saved the most lives during the RCT (quickly treating diarrhea, malaria, and pneumonia in children), require that families are able to access health facilities or providers and that appropriate medicines are available.106

We are not confident in our understanding of how accessible these treatments are to those that seek them. DMI’s midline results measured self-reported treatments received in addition to treatment sought; we use the former in our estimate of DMI’s effectiveness.107

To determine whether targeted populations have access to key health supplies in countries in which DMI is considering launching a program, DMI uses Demographic and Health Survey (DHS) data to determine the portion of those who say they sought treatment for a child’s symptoms (such as a fever) that also report receiving treatment (such as antimalarials).108

DMI notes that the DHS program has the best data on health supply availability and health center access, and that DHS data suggests that availability and health center access are similar in Burkina Faso and the DRC.109 We have not verified this claim.

Do the individual health practices save lives?

We have not completed a review of each of the practices DMI encourages through its broadcasts, but we believe that many of them – e.g., pneumonia treatment, malaria treatment, and oral rehydration therapy – represent well-established ways to prevent child mortality.110

How does DMI monitor and evaluate its ongoing programs?

DMI has told us that it uses various mechanisms to monitor its ongoing campaigns to determine whether (a) programs have aired at the scheduled times, (b) people have listened to DMI's programming, and (c) listeners have correctly understood the intended messages.111 We have seen some data from DMI’s monitoring of its Burkina Faso RCT campaign, its national campaign in Burkina Faso, and its campaign in DRC. When possible, we compare data from the RCT to the two post-RCT campaigns to try to understand whether the quality of DMI’s RCT campaign is being maintained as it scales up.

Below we discuss the evidence we have seen. Broadly, in DMI’s post-RCT campaigns (which have less intensive monitoring and data collection than the RCT), we have some evidence that some broadcasts are occuring and are heard by some portion of DMI’s audience, but we are very uncertain about how frequently broadcasts are occurring, what portion of DMI’s target audience is hearing DMI’s spots, and what portion of people who hear the spots understand the intended messages. Overall, we have limited evidence on the quality of DMI’s national campaign in Burkina Faso and its campaign in the DRC.

How reliably do the broadcasts happen?

Burkina Faso RCT

We would guess that broadcast frequency (i.e., how often programs were aired) was fairly high in the RCT largely because midline results from the RCT suggest that a large portion of women in the broadcast zones heard one of DMI’s spots and may have changed their health behavior.112 DMI also says it had frequent direct contact with each of its partner stations,113 and we have seen a large number of anecdotes about DMI’s efforts to keep its partners broadcasting frequently.114

Burkina Faso national campaign

We have limited information about broadcast frequency in DMI’s national campaign in Burkina Faso:

• As far as we know, DMI has not yet conducted qualitative research on whether people are listening to its programs or changing their health behaviors as a result of its national campaign.
• We do not know if DMI will be able to maintain the intensity of its contact with partners that it had during its RCT now that it is working with significantly more partners and generally scaling up its operations,115 but we would guess that its contact with partners will lessen somewhat.
• Taken at face value, the broadcast monitoring data (more details below) suggest that DMI may be falling short of its broadcasting target in this campaign, but we do not put much weight on this evidence.

DRC

We have limited information about broadcast frequency in DMI’s campaign in the DRC:

• As far as we know, DMI has not had much direct contact with its partners in the DRC. Instead, it has contracted its distribution partner to monitor local stations,116 but we have not seen monitoring data from its partner.
• We have seen limited data from only 3 of DMI’s 35 DRC broadcast zones on whether people are hearing DMI’s spots in DRC and whether they’re changing their behaviors. We discuss this data in the next section.
• Taken at face value, the broadcast monitoring data suggest that DMI may be falling short of its broadcasting target in this campaign, but we do not put much weight on this evidence.
• DMI may expect its broadcasting frequency to be lower than usual in this campaign; it has encountered some issues with this campaign.117

Details on how we reached the above conclusions follow.

Monitoring processes

Though details vary by location, DMI generally uses a variety of methods to determine whether its spots are being broadcast and how frequently they are aired:

• Hiring “broadcast monitors.” For its campaigns in Burkina Faso and the DRC, DMI employs individuals in its broadcast zones to listen to the radio and track the frequency of DMI broadcasts.118 DMI tries to hire two to three broadcast monitors for each broadcast zone. The monitors are instructed to record each time they hear a DMI spot (which are tagged with a "laughing baby" identifier), and to report when the station is off the air or has technical difficulties.119 The monitors are independent, do not know that the other is employed by DMI, and do not know how frequently DMI spots are supposed to be played. Partner stations do not know the identity of the monitors, and the monitors are instructed to not disclose their identity to the radio station.120 Discussion of results from this monitoring are below.
• Direct contact with stations. DMI uses a combination of communication with radio stations (e.g., visits, phone calls), requesting and reviewing broadcast reports from stations, and, in some cases, employing staff at radio stations to determine whether the stations are broadcasting.121 With the exception of anecdotes about issues that stations encountered during DMI’s RCT,122 we have not seen documentation from these activities.
• Surveying people in target areas about whether they have heard DMI spots and whether they have changed health behaviors. DMI employs qualitative researchers to survey people in target areas about whether they have heard DMI’s spots and whether they have changed their health behaviors. These surveys may provide some evidence about whether and how often DMI’s spots are being broadcast. We discuss this evidence in the sections below.

Broadcast monitoring results

Overall, we feel that the broadcast monitoring data does not provide a strong indication of how frequently DMI spots are being broadcast; we could easily imagine these figures overestimating or underestimating the true broadcast frequency. The usefulness of this data seems limited by the fact that:

• We are not aware of an auditing process for this data, though we have not asked DMI about this explicitly.
• Monitors of the same station often report sizable discrepancies in the station's broadcast frequency. We are unsure how to explain these discrepancies. There also seems to be a significant amount of missing data that we are unsure how to interpret.

However, we believe this data provides weak evidence that broadcasts are occurring, partly because monitors sometimes seem to simultaneously report when a station is off the air, which provides some support to the idea that it is meaningful when monitors report that spots are being played.123

Table 3 summarizes our analysis of a sample of broadcast monitoring data we have seen from 2015.124

We believe that DMI generally intended to broadcast 6-10 spots per day to achieve “saturation,” which it believes is best for the effect of its program.133 We do not know how important it is for DMI to meet its threshold for saturation. Our best guess is that lower broadcast intensity implies less behavior change, though we are uncertain about this.

Methodological details by campaign:

• Burkina Faso RCT campaign: In the above table, we analyze one of the four months of data that DMI shared with us (January 2015); we analyzed only one month due to time constraints (we chose to analyze January 2015 because it was the most recent of the four) and the four months of data that DMI provided were randomly chosen by us.134 The January 2015 monitoring data contains data for each of the seven intervention zones in the RCT.135

  We also looked for instances of stations being off the air in the funder reports that DMI shared with us; details in this footnote.136

• Burkina Faso national campaign: DMI shared broadcast monitoring data from the Burkina Faso national child survival campaign that started in May 2015; it shared data from late June to the beginning of October for what appears to be all 29 stations it is working with, from which we analyzed a random sample of 8 stations due to capacity constraints.137 Many more details about this data are in this footnote.138
• Democratic Republic of the Congo: DMI has told us that establishing good campaign monitoring is more difficult in the DRC than in Burkina Faso; it told us that it must recruit and manage monitors by phone.139 As of July 2015, DMI had recruited 61 broadcast monitors in the DRC.140 Its campaign in DRC began in May 2015.141 DMI shared broadcast monitor logs from mid-May to the end of September for what appears to be at least all 35 stations that it partnered with in its campaign; we randomly selected a sample of 11 stations for more thorough analysis.142 Many more details about this data are in this footnote.143

Do people hear DMI’s broadcasts?

Burkina Faso

There is fairly strong evidence that a large proportion of people in DMI’s RCT broadcast zones in Burkina Faso heard DMI’s spots. However, we have not seen qualitative research or any other evidence on whether people are listening to DMI’s broadcasts for DMI's 2015 post-RCT, national campaign in Burkina Faso.

DMI’s midline survey found that 75% of women in DMI’s RCT broadcast zones in Burkina Faso reported recognizing at least one of two recorded spots (that had been broadcast soon before the women were surveyed) when it was played for them during the survey, compared with 20% of women in the control zone.144 This result was statistically significant.145

Democratic Republic of the Congo

There is some evidence from DMI’s qualitative surveys that some people are hearing DMI’s spots in 3 of the 35 broadcast zones where DMI is working in the DRC. We are uncertain what to conclude about what portion of people are hearing DMI’s spots in these zones, and are highly uncertain about how representative we should expect these results to be of DMI’s work in the DRC generally.

In the DRC, DMI conducted qualitative research in August 2015 in villages that were in the broadcast zones of three of DMI’s 35 partner stations: Nsemo, Tomisa, and Kimvuka na Lutondo.146 Because the research was conducted in just three of the campaign's 35 broadcast zones (in areas close to Kinshasa, which were easier to reach and may be better developed than more remote provinces), we are unsure of how representative this evidence is. The research consisted of interviews with individuals and focus groups in eight villages in these provinces. Villages were randomly selected but the provinces were not. Focus group participants were not randomly selected; community leaders assisted in assembling a group by making an announcement. The first twelve people that responded and who matched the participant profile formed the group.147 Because of potential bias introduced by this survey methodology, we are uncertain what to conclude about the representativeness of survey responses within these zones.

In total, 179 mothers and fathers of children under 5 were interviewed in single-gender focus groups of about 12 participants each. Additionally, in Idiofa 15 boys and girls under 8 were interviewed in a focus group. 11 community leaders were interviewed individually.148

Table 4 displays the number of interviewees who said they listened to DMI's partner station.149

Overall, approximately 54% of focus group participants reported listening to the DMI partner station that was broadcasting in their area.154 Of the people who reported listening to the station that broadcasts DMI’s messages, all of them reported hearing spots tagged with a laughing baby (the identifier DMI attaches to its spots).155 The research report states that in all three broadcast zones, almost all participants who reported listening to the partner station also reported listening to subject matter associated with child and maternal health, such as messages on bednet use and the treatment of fever, diarrhea, and cough.156 We do not know the details of how people were asked about whether they had heard DMI’s messages, though we have seen the question guide used by the qualitative researcher.157 We have not seen granular data on interviewee’s responses.

Do people understand the broadcasts?

DMI’s qualitative research adds some weak, anecdotal evidence that people in DMI’s broadcast zones understand DMI’s health messages. We have a fairly large amount of qualitative research from DMI’s Burkina Faso RCT campaign, have not yet seen any qualitative research from DMI’s 2015 national child survival campaign in Burkina Faso, and have only seen a small sample of such research from DMI’s work in the DRC.

Burkina Faso

During the child survival RCT in Burkina Faso, DMI’s qualitative research team asked participants to explain the messages they heard on the radio during feedback focus groups. We have seen summary reports from these feedback groups and a synthesis report summarizing this feedback research across all seven intervention zones.158 Because the qualitative research team did not conduct similar interviews in the control zones, it is difficult to tell how much of participants’ message recall is due to DMI’s program alone.

DMI reported that most individuals are able to recall at least one of DMI’s broadcast messages and are able to explain the recalled health messages accurately. However, some individuals make errors in recalling the messages.159 The messages that were voluntarily recalled most frequently were taking children to a health center immediately when they become sick, and messages relating to malaria, diarrhea, and prenatal care.160 We have also reviewed several regional research syntheses from 2014 on breastfeeding topics.161 These syntheses seem to provide some anecdotal evidence that listeners have understood DMI's breastfeeding messaging (see following footnote for details).162

We have not seen qualitative research on child survival topics such as the treatment of malaria, diarrhea, and pneumonia for DMI's 2015 national campaign in Burkina Faso, so we do not know if listener understanding of these topics has changed during the transition from the RCT campaign to the national campaign.

Going forward, we expect future qualitative research and time series data from DMI's planned evaluation system (see below) to provide information on listener understanding of DMI campaign topics in Burkina Faso.

Democratic Republic of the Congo

DMI's qualitative research in Bandundu province (see above for further discussion of methodology and results) provides some anecdotal evidence that listeners have understood the content of DMI's messaging. Specifically, the research reports that interviewees were able to repeat the content of DMI messages on bednets, prevention of diarrhea, treatment of malaria in children, exclusive breastfeeding, and vaccination.163

The research report also noted that traditional healthcare practices remain prevalent and sometimes take precendence over Western treatments.164

We have not reviewed granular data related to this research. Because interviews were conducted in just three of the campaign's 35 broadcast zones (in areas close to Kinshasa, which may be more developed than more remote provinces), we are unsure of how representative this evidence is.

What monitoring and evaluation does DMI plan to implement for future campaigns? Burkina Faso For its national child survival campaign in Burkina Faso, DMI plans to deploy a new evaluation system. DMI has shared its plans for this evaluation system with us (see footnote for details).165 Because we have not seen data from this system, we do not have an opinion about the quality of the information it provides. We expect to see data from this system in 2016. In addition to the new monitoring system, DMI will retain its qualitative research team, and will continue to employ monitors in each of its broadcast zones.166 DMI has told us it is interested in continuing to find cost-effective ways to reduce power outages at partner stations, such as the possibility of buying solar power systems for partner stations rather than making payments to them.167 Democratic Republic of the Congo For its child survival campaign in the DRC, DMI conducted a baseline survey in an intervention zone and a control zone in Bandundu Province in May 2015.168 In addition to demographic information, the survey measured radio ownership, radio penetration, listening habits, and maternal and child health behaviors and knowledge.169 We have not yet seen the results of the baseline survey. DMI plans to conduct an endline survey in these zones in December 2015, which will measure maternal and child health behaviors and knowledge as well as the mothers' self-reported exposure to the DMI campaign. DMI has partnered with the Kinshasa School of Public Health (KPSH) to conduct these surveys.170 Results from the endline survey are expected in the first half of 2016.171 In addition to the endline surveying, DMI plans to conduct three more qualitative research trips in the DRC in the second half of 2015, and will continue to employ monitors in each of its broadcast zones.172 Mozambique Because DMI's Mozambique operation is still in the planning and fundraising phase, it has not yet developed a country-specific monitoring and evaluation protocol for Mozambique.173 What do you get for your dollar? Note: The following section was written before DMI sent us preliminary endline results for its RCT in Burkina Faso; see the details on the preliminary results at the top of this page. As such, the below discussion does not reflect our current view of DMI's cost-effectiveness. We do not have strong evidence to suggest that DMI's program cost-effectiveness is within the range of our other priority programs. We estimate the cost per child life saved through DMI’s program to be approximately $5,900.174 This estimate does not include other potential benefits of DMI’s program, such as preventing non-fatal cases of illnesses or other issues. DMI’s estimate of its cost per life saved is about 12 to 87 times stronger than our best guess.175 Major judgment calls underlying our cost-effectiveness estimate include: We rely primarily on DMI’s midline results, rather than previous nonrandomized studies, to estimate the effect of DMI’s program on behaviors.176 As a result, we have not predicted any lives saved from higher treatment rates of malaria, a major focus of DMI’s program, since the midline results did not find an increase in the use of antimalarials.177 We draw country-specific data from Demographic and Health Survey (DHS) reports and the Institute for Health Metrics and Evaluation's Global Burden of Disease Comparative Tool.178 We have not closely examined the methodology behind these figures; our general impression is that the quality of survey data from the developing world is quite poor. Data quality issues might mean that our estimate does not accurately reflect the true cost per life saved for DMI's program. We discount the effect implied by DMI’s midline results because they come exclusively from Burkina Faso, whereas additional unrestricted funding will likely support new programs in other countries (see below). We have made this adjustment in other charity cost-effectiveness estimates as well.179 See our outstanding questions for a discussion of possible differences between Burkina Faso and other countries in which DMI may operate. We discount the effect implied by DMI’s midline results because they come from a single study rather than multiple studies (as we have done in other cost-effectiveness estimates).180 We discount the effect implied by DMI’s midline results to account for the possible bias associated with self-reported results.181 We discount the expected benefits of treatment based on the possibility that the treatment received is of poor quality or does not occur on schedule. For its upcoming campaign in Mozambique, DMI plans to message on maternal health and family planning topics as well as child survival topics.182 Our model assumes that this campaign will result in the same behavior change percentages as the Burkina Faso RCT campaign. Plausibly behavior changes resulting from the Mozambique campaign will differ due to a different allocation of messaging topics. DMI's response183 “GiveWell's cost-effectiveness estimate for our media campaigns differs significantly from our own internal estimates. Whilst we respect GiveWell’s analysis, we stand by our own figures. We also note that GiveWell’s estimate is significantly affected by four discounts that they have applied to take into account the strength of the existing evidence, and that without these adjustments, the estimate would be $1,520 per life saved [$1,400 per life saved using the 2015 model], rather than $7,264 [$5,900 per life saved using the 2015 model]. We expect that the endline results of our RCT in Burkina Faso, which will be available in late 2015, will address most if not all of these remaining questions about the evidence base.” For more on the role that cost-effectiveness estimates play in our reviews, see our page on cost-effectiveness analysis. Is there room for more funding? Note: The following section was written before DMI sent us preliminary endline results for its RCT in Burkina Faso in November 2015 (see the details on the preliminary results at the top of this page). In November 2015, DMI provided an update that it plans to use some of its existing funding to conduct a second endline survey in 2016 for its RCT because it believes that serious data collection issues occurred during the first endline survey. DMI estimates that this survey will cost in the region of $200,000 to $300,000. In short: Estimated needs: DMI estimates that over the next two years, it could productively use approximately $12 million in unrestricted funds (excluding reserves).184 (More) Cash on hand: DMI currently holds about $1.25 million in unrestricted funding. This figure includes (a) some funding that DMI has already allocated for spending in 2016, and (b) reserves; DMI expects to hold approximately $765,000 in reserves going forward.185 We do not know how DMI plans to use the unrestricted funds that it has already allocated or how much of its unrestricted funding has been allocated. We plan to follow up with DMI about its use of unrestricted funding as part of our process of staying up to date on the organization in 2016. Other sources of funding: DMI currently has other funding prospects that may provide up to about $3.6 million to help fill its $12 million funding gap, but the large majority of this potential funding is fairly uncertain. DMI also receives a large amount of restricted funding from other funders, but we do not focus on DMI’s restricted funding in our room for more funding analysis because we believe that it is unlikely to affect how DMI uses unrestricted funds. (More) Past spending: We have limited information about how DMI used the unrestricted funding it received from GiveWell in 2014 (approximately $510,000). (More) Additional considerations: Below, we discuss issues material to our assessment of DMI's room for more funding. Specifically, (a) how DMI prioritizes its fundraising activities, (b) other potential uses of unrestricted funding, and (c) whether money is the limiting factor for the organization. (More) Uses of additional funding Table 5 summarizes the ways in which DMI plans to use additional unrestricted funding.186 Table 5: DMI's prioritization of incoming unrestricted funds Priority Amount Country Purpose 1 $2 million Mozambique To cover the 2016-2017 funding gap of its child survival, family planning, and maternal health radio and TV campaign.187 2 $0.85 million DRC To cover the 2016 funding gap for its child survival campaign.188 3 Approximately$1.5 million Burkina Faso To cover the 2016-2017 funding gap for its national child survival campaign.189 4 $4.5 million Tanzania To open a two-year national child survival campaign.190 5 $3 million Mali and Niger Either to open a two-year child survival campaign in Mali or Niger, or to open a one-year child survival campaign in both countries.191 A more detailed breakdown of DMI’s funding needs for 2016 versus 2017 is in this footnote.192 DMI has told us that, for a new campaign, it takes approximately 9-10 months from obtaining funding to beginning broadcasts.193 However, because DMI has already been working in the DRC and Burkina Faso, it may be that funding in those countries would prevent gaps in broadcasts from occurring or enable DMI to restart its broadcasts very quickly.194 We would guess that DMI would also estimate that the gap between receiving funding and beginning to broadcast would be less than 9-10 months in Mozambique since it has already established some operations there.195 Other sources of funding DMI has other potential funders that may provide up to about $3.6 million for its child survival campaigns.196 Its prospects include:197 A roughly 50% chance to fill its $2 million funding gap in Mozambique.198 A roughly 50% chance to fill its $850K funding gap in the DRC.199 ”Likely” funding for its $500K 2016 funding gap in Burkina Faso.200 We do not know if DMI expects other funders to fill the roughly $1 million gap in Burkina Faso in 2017.201 About $240K in unrestricted funds that it expects to receive from the Mulago Foundation. It has not yet allocated this funding.202 We are not aware of any funding prospects for DMI’s potential campaigns in Tanzania, Mali, and Niger. DMI has a for-profit arm through which it has run some campaigns.203 We have little information about DMI’s plans for its for-profit arm, but our impression is that DMI is unlikely to receive a substantial amount of funding through this arm in the next year.204 DMI also receives a large amount of restricted funding from other funders,205 but we do not focus on restricted funding in our room for more funding analysis unless it could potentially affect how DMI uses unrestricted funds. For reasons discussed below, we believe that DMI’s restricted funding typically does not affect its use of unrestricted funds. How did DMI spend money moved by GiveWell in 2014? GiveWell moved approximately $510,000 in unrestricted funding to DMI in the 2014 giving season.206 DMI has not yet committed most of the GiveWell-directed funding it received in 2014.207 It allocated approximately $100,000 of GiveWell-directed funding to implement a regional data collection system in Burkina Faso (see above for details).208 We believe that DMI has continued to hold the remaining $410,000 of GiveWell-directed funding because it does not yet know where the funding would be best used. Because DMI has not yet committed 2014 GiveWell-directed funding, we view DMI's use of GiveWell-directed funding as potentially consistent with the plans it outlined for these funds in 2014 (see this section of last year's review or the following footnote for a summary of DMI’s previous plans).209 DMI has told us it plans to spend most of 2014 GiveWell-directed funding in 2016.210 Our best guess is that DMI will allocate 2014 GiveWell-directed funding to one of the funding gaps outlined above. We will follow up with DMI about its use of GiveWell-directed funding as part of our process of staying up to date with the organization in 2016. Additional considerations relevant to assessing DMI's room for more funding DMI's fundraising priorities Our understanding is that DMI’s current top fundraising priority is raising funds for its child survival campaigns.211 However, DMI also actively raises a substantial amount of restricted funding for campaigns that do not focus on child survival because it wants to take opportunities to grow its organization and carry out beneficial activities, even if those activities may be less evidence-backed and cost-effective than child survival campaigns.212 There is a potential concern that unrestricted and restricted funds may be partially fungible with each other even if DMI’s top priority is to conduct child survival campaigns. For example, if other DMI funders are interested in having DMI carry out both child survival campaigns and other types of campaigns, it is possible that GiveWell-directed funds (expected to be spent on child survival campaigns) would make other funders more likely to give restricted funds for other types of campaigns. Scenarios like these are difficult to avoid, and do not seem especially problematic in DMI’s case relative to other charities. DMI’s stated priorities suggest that receiving unrestricted funding for its child survival campaigns would not cause it to make any less effort to fundraise for such campaigns from other funders,213 so it seems reasonable to expect that additional unrestricted funding to DMI is not fungible with restricted funds. Other potential uses of funding Because DMI carries out a variety of activities, there seem to be many other potential ways that it could use its unrestricted funds, such as filling small funding gaps in its restricted campaigns, funding research, maintaining reserves, or funding new program components.214 Though it seems like there are other possibilities for how DMI could use unrestricted funding, our expectation is that DMI will spend additional unrestricted funding according to the prioritization outlined above. Is money the limiting factor to DMI's organizational growth? DMI has a long-term initiative called Media Million Lives, which is a plan to save one million lives via child survival mass media campaigns by the end of 2024.215 To reach the scale needed to potentially achieve such a goal, DMI would need to have an annual budget of roughly $24 million, which is substantially larger than its current budget of approximately $5 million/year.216 DMI has told us that lack of funding is the primary factor limiting its growth.217 DMI already has core staff such as country directors in place in Burkina Faso, the DRC, Mozambique, and Tanzania,218 so it seems plausible that funding (rather than, e.g., staff capacity) is the primary limiting factor for DMI in several of the countries where it is already working.219 Once DMI fully scales up its activities in countries where it already has staff capacity, additional funding may help DMI to build capacity in new countries. Major outstanding questions What do the preliminary endline RCT results imply about DMI's program effectiveness? In November 2015, DMI privately shared preliminary RCT endline results with us. The preliminary endline results did not find any effect of DMI's program on child mortality, and it found substantially less effect on behavior change than was found at midline. DMI believes that there were serious problems with endline data collection (note that we have not yet tried to independently assess this claim), and with the support of the trial’s Independent Scientific Advisory Committee, is planning to conduct another endline survey in late 2016 (with results available in 2017). We understandably cannot publicly discuss the details of the endline results we have seen, because they are not yet finalized and because the finalized results will be embargoed prior to publication. We have not closely analyzed the preliminary results we have seen. We expect some further information about the RCT endline results to be available in 2016.  Issues related to DMI's child survival RCT midline results See above for discussion of our outstanding questions about DMI’s midline results. What do DMI’s midline results imply about lives saved during the trial? We expect to have a better answer to this general question once the endline results of DMI’s trial are published (expected by mid-2016). Below are narrower questions that affect our current understanding, many of which we have included in our cost-effectiveness estimate with guessed answers. Which metrics are most meaningful for estimating lives saved? For example, is it more appropriate to base estimates of lives saved from a specific illness on increases in treatment seeking or increases in treatments received? How often does a family seek or receive treatment for a child’s illness and fail to save the life of that child anyway (for example, by failing to receive the drug on time, by receiving a low-quality drug, or by not using the entire recommended treatment)? Are families that seek treatment in response to DMI’s messages more or less likely to actually need the treatment than those that seek such treatments regardless of DMI’s messages? Do DMI’s messages increase mis- or over-prescription of treatments such as antimalarials and antibiotics? How costly are such errors? How strong is the evidence that the behaviors DMI promotes prevent child mortality? While we believe many of the behaviors are well-established ways to prevent child mortality, we have not closely reviewed the relevant evidence for all of them.220 How much health improvement that doesn’t result in saved lives should we expect to see from DMI’s program? Is there other external evidence about the effectiveness of promoting behavior change through mass media that dramatically strengthens the evidence for the effectiveness of DMI’s program? (Note that we did not find any such evidence in our initial search in 2012).221 How representative is the evidence from DMI’s midline results? In addition to issues noted above (such as PMC's concurrent mass media campaign and the unusually intensive nature of DMI's campaign during the RCT), the following factors might reduce the representativeness of the midline results. How does DMI's post-RCT programming in Burkina Faso differ from its programming during the RCT? In Burkina Faso, DMI is currently broadcasting on a variety of child survival topics (see above). DMI is also running a campaign focused on nutrition messaging in seven zones.222 We do not know how airtime is being allocated between child survival topics and nutrition topics at these stations, and we do not know if child survival topics are being broadcast at a "saturation" intensity of 6-10 times/day at these stations. When its family planning RCT begins, some DMI partner stations will be broadcasting family planning messaging in addition to child survival messaging. We do not know how airtime will be allocated between the two campaigns, and we do not know if child survival topics will continue to be broadcast at a "saturation" intensity of 6-10 times/day at partner stations that are part of the family planning RCT. We do not know how important it is for DMI's broadcast intensity to be meeting DMI's threshold for saturation. Our best guess is that lower broadcast intensity implies less behavior change, though we are uncertain about the exact relationship of these variables. Finally, as mentioned above, DMI’s post-RCT child survival campaign will not use the 10- to 15-minute-long live drama modules that were used during the RCT because DMI believes these modules to be less cost-effective than its 60-second spots.223 This change represents a possible external validity concern. How much of DMI’s results rely on factors specific to Burkina Faso? The success of DMI’s program may vary significantly in different contexts. DMI attempts to account for many possible differences between countries in estimating the life-saving potential of its program, including the number of childhood deaths that could be prevented with relatively easy behavior changes and the demographics of the audience of various radio networks.224 Other factors that may affect the program’s effectiveness are more difficult to quantify and predict. A better understanding of these issues, and how they vary among countries, would allow us to better estimate the cost-effectiveness (and associated level of confidence) in the countries to which DMI is considering expanding its program. How does health supply availability in Burkina Faso compare to availability in other countries in which DMI might operate? Our understanding is that DMI has limited information on the availability of health products and services for the countries in which it works.225 If access to health supplies in other countries is worse than in Burkina Faso (among those not already receiving such treatments), we would expect DMI’s program in those countries to be less effective than its program in Burkina Faso. How much do the obstacles to behavior change vary between and within countries? We expect that some obstacles to behavior change, such as lack of awareness, are more easily overcome by DMI’s program than others, such as lack of availability of health supplies. For other obstacles to behavior change such as time or financial costs, it is not clear how much DMI’s program would help. How much do relevant cultural attitudes vary among countries? For example, could a dearth of independent media make individuals less likely to take the advice of health messages delivered through radio, or might some cultures have more resistance to changing specific health behaviors?226 Relatedly, DMI told us that it chose Burkina Faso as the site for its RCT because it has an extremely decentralized media market, making it possibly the only country where 14 isolated media environments could be included in a randomized controlled trial. It is possible that a particularly decentralized or underdeveloped media market might make professional radio programming particularly attractive or compelling to listeners, which could make DMI’s program more effective in Burkina Faso than in other countries. This difference might also make program production more expensive in Burkina Faso than elsewhere. How much will related activities by governments or other non-governmental organizations affect DMI’s impact? For example, a program that increases the availability of health supplies may significantly improve the effectiveness of DMI’s program, while a program that already uses mass media to promote similar behavior change may reduce DMI’s impact. How has the recent unrest in Burkina Faso affected DMI's operations there? In October 2014, long-term Burkinabé President Blaise Compaoré fled the country amid civil unrest. In September 2015, a reactionary coup was carried out against the interim government, though the interim government was later restored.227 The current political situation is unstable. The effect of this instability on DMI's impact in Burkina Faso is unclear. One of DMI's partner stations suspended live, long-format broadcasts during October 2014, and DMI briefly shut down its office in Ouagadougou during the 2014 unrest as a security precaution.228 In a recent report to funders, DMI noted that the security situation in rural Burkina has been deteriorating.229 We do not know how the worsening rural security situation will affect the impact of DMI's campaigns. How much will the cost per listener vary among countries? For example, frequency of power outages, equipment failures, or personnel issues for radio stations may vary significantly, as might labor costs, airtime costs, and the difficulty of enforcing broadcast agreements. Based on DMI’s experience with station failures in Burkina Faso (see above), these problems have the potential to increase DMI's costs and/or reduce the frequency of DMI's broadcasts.230 How will the effects of DMI’s program change over time? To what extent will behavior changes persist after messaging campaigns cease? Will listeners become less responsive to DMI’s program after many years of broadcasts? How will we know if future programs are successful? Once the final results from the RCT are available (expected in mid-2016), we expect to have significantly more confidence about the success of DMI’s current program, and by extension, its future programs. However, as noted above, there may be differences between the effectiveness of DMI’s program in Burkina Faso and the effectiveness of programs in other countries. Since future programs will not be studied by a randomized control trial as is the case in Burkina Faso, we believe that it will be harder to know how much of an impact those programs are having. DMI’s ongoing monitoring and qualitative research may provide information about how behavior changes over time, and how popular the broadcasts are in different communities. Additionally, DMI's planned time-series evaluation of its future campaigns should provide some evidence of future impacts.231 DMI as an organization Our sense is that DMI is a strong organization: Track record: DMI’s track record is fairly limited; it has been operating since 2006 but has only run relatively large campaigns similar to its current child survival campaigns since 2011, when it began its child survival campaign and RCT in Burkina Faso. Recently, it has begun to implement its model on a larger scale and in multiple countries. We have limited information about the quality of its scaled-up programs. Self-evaluation: DMI has invested unusually heavily in understanding the impact of its program by conducting an RCT of its program in Burkina Faso, and by adding monitoring and evaluation components to its other campaigns. Communication: DMI has communicated clearly and directly with us, and has given thoughtful answers to our critical questions. Transparency: DMI has been very transparent; we have not seen it hesitate to share information with us. More on how we think about evaluating organizations in our 2012 blog post. Sources Document Source 2014 Burkinabé Uprising Wikipedia page Source (archive) 2015 Burkina Coup D'Etat Wikipedia page Source (archive) Blaise Compaoré Wikipedia page Source (archive) Demographic and Health Surveys website Source (archive) Development and use of the Lives Saved Tool (LiST) Source (archive) DHS Survey DRC 2008 Source DHS Survey DRC preliminary results 2013 Source DMI 2015-2016 financial projection Unpublished DMI 2015-2016 fiscal year financials Unpublished DMI Alive & Thrive progress report August 2015 Source DMI breastfeeding research regional synthesis – Boucle du Mouhoun Source DMI breastfeeding research regional synthesis – Cascades Source DMI breastfeeding research regional synthesis – Centre Sud et Centre Est Source DMI breastfeeding research regional synthesis – Centre West Source DMI breastfeeding research regional synthesis – East Source DMI breastfeeding research regional synthesis – North Source DMI breastfeeding research regional synthesis – Sahel Source DMI breastfeeding research summary March 2014 Source DMI Burkina Faso automated monitoring summary 2012 Source DMI Burkina Faso broadcast monitor guide 2014 Unpublished DMI Burkina Faso broadcast monitoring data December 2013 Unpublished DMI Burkina Faso broadcast monitoring data January 2015 Unpublished DMI Burkina Faso broadcast monitoring data June 2012 Unpublished DMI Burkina Faso broadcast monitoring data June to September 2015 Unpublished DMI Burkina Faso broadcast monitoring data March 2013 Unpublished DMI Burkina Faso funder report 2011 Q1 Unpublished DMI Burkina Faso funder report 2011 Q2 Unpublished DMI Burkina Faso funder report 2011 Q3 Unpublished DMI Burkina Faso funder report 2011 Q4 Unpublished DMI Burkina Faso funder report 2012 Q1 Unpublished DMI Burkina Faso funder report 2012 Q2 Unpublished DMI Burkina Faso funder report 2012 Q3 Unpublished DMI Burkina Faso funder report 2012 Q4 Unpublished DMI Burkina Faso funder report 2013 Q1 Unpublished DMI Burkina Faso funder report 2013 Q2 Unpublished DMI Burkina Faso funder report 2013 Q3 Source DMI Burkina Faso funder report 2013 Q4 Source DMI Burkina Faso funder report 2014 Q1 Source DMI Burkina Faso funder report 2014 Q2 Unpublished DMI Burkina Faso funder report 2014 Q3 Unpublished DMI Burkina Faso funder report 2014 Q4 Source DMI Burkina Faso funder report 2015 Q1 Source DMI Burkina Faso funder report 2015 Q2 Source DMI Burkina Faso message brief on 1000 days Source DMI Burkina Faso message brief on ARIs Source DMI Burkina Faso message brief on breastfeeding Source DMI Burkina Faso message brief on complementary feeding Source DMI Burkina Faso message brief on diarrhoea Source DMI Burkina Faso message brief on exclusive breastfeeding Source DMI Burkina Faso message brief on family planning Source DMI Burkina Faso message brief on hygiene Source DMI Burkina Faso message brief on hygiene 2 Source DMI Burkina Faso message brief on low birthweight Source DMI Burkina Faso message brief on malaria Source DMI Burkina Faso message brief on maternal health Source DMI Burkina Faso message brief on maternal nutrition Source DMI Burkina Faso message brief on vitamin A Source DMI Burkina Faso proposal 2014 Unpublished DMI DRC broadcast monitoring data, May to September 2015 Unpublished DMI DRC funder report 2015 Q1 Source DMI DRC funder report 2015 Q2 Source DMI DRC message brief on birth registration Source DMI DRC message brief on care of low birthweight babies Source DMI DRC message brief on complementary feeding Source DMI DRC message brief on diarrhea treatment Source DMI DRC message brief on drinking water purification and storage Source DMI DRC message brief on exclusive breastfeeding and colostrum Source DMI DRC message brief on family planning (in Kinshasa) Source DMI DRC message brief on handwashing with soap Source DMI DRC message brief on immunizations Source DMI DRC message brief on malaria Source DMI DRC message brief on maternal health Source DMI DRC message brief on pneumonia treatment-seeking Source DMI DRC qualitative research final report August 2015 Source DMI DRC qualitative research protocol June 2015 Source DMI email from Dr. Joanna Murray on October 14, 2015 Unpublished DMI email from Dr. Joanna Murray on September 21st, 2015 Unpublished DMI email from Joanna Murray on November 3rd, 2014 Unpublished DMI email from Will Snell on November 6, 2015 Unpublished DMI email from Will Snell on May 15, 2015 Unpublished DMI email from Will Snell on October 2, 2015 Unpublished DMI email from Will Snell on October 30, 2015 Unpublished DMI email to GiveWell October 3rd, 2014 Unpublished DMI feedback research summary September 2013 Source DMI health supply availability 2013 Source DMI pretesting synthesis ARI October 2013 Source DMI pretesting synthesis breastfeeding February 2013 Source DMI pretesting synthesis diarrhoea July 2012 Source DMI pretesting synthesis hygiene January 2013 Source DMI pretesting synthesis malaria December 2013 Source DMI pretesting synthesis maternal July 2013 Source DMI qualitative monitoring report Banfora January 2014 Unpublished DMI qualitative monitoring report Banfora July 2013 Unpublished DMI qualitative monitoring report Bogandé April 2013 Unpublished DMI qualitative monitoring report Bogandé December 2013 Source DMI qualitative monitoring report Djibo February 2014 Unpublished DMI qualitative monitoring report Djibo March 2013 Unpublished DMI qualitative monitoring report Kantchari March 2014 Unpublished DMI qualitative monitoring report Kantchari May 2013 Unpublished DMI qualitative monitoring report Ohya July 2013 Unpublished DMI qualitative monitoring report Ohya June 2014 Unpublished DMI qualitative monitoring report Sapouy December 2013 Unpublished DMI qualitative monitoring report Sapouy March 2013 Unpublished DMI qualitative monitoring report Solenzo July 2014 Source DMI qualitative monitoring report Solenzo June 2013 Unpublished DMI SPRING quarterly report July 2015 Source DMI summary 2014 Source DMI updated monitoring guide October 2013 Unpublished DMI website, "Burkina Faso RCT: radio spots" spot audio Source DMI website, "Burkina Faso RCT: radio spots" page Source (archive) DMI website, "Burkina Faso" page Source (archive) DMI website, "Costing Impact" page Source (archive) DMI website, "DR Congo" page Source (archive) DMI website, "Future Campaigns" page Source (archive) DMI website, "Mozambique" page Source (archive) DMI website, "Issues" page Source (archive) DMI website, staff page Source (archive) DRC learning and feedback session summary Unpublished Gates Global Grand Challenges DMI page Source (archive) GiveWell 2014 DMI review Source GiveWell analysis of DMI broadcast monitor data Unpublished GiveWell analysis of DMI DRC qualitative research final report Source GiveWell non-verbatim summary of a conversation with Anonymous District Medical Officer on October 14th, 2014 Unpublished GiveWell non-verbatim summary of a conversation with Basilia Coefe on October 16th, 2014 Unpublished GiveWell non-verbatim summary of a conversation with DMI on April 24th, 2014 Source GiveWell non-verbatim summary of a conversation with DMI scriptwriters on October 16th and 17th, 2014 Unpublished GiveWell non-verbatim summary of a conversation with DMI’s qualitative research team on October 17th, 2014 Source GiveWell non-verbatim summary of a conversation with Jean-Baptiste Guidard-Schmid on October 17th, 2014 Unpublished GiveWell non-verbatim summary of a conversation with Joanna Murray on October 30th, 2014 Unpublished GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015 Source GiveWell non-verbatim summary of a conversation with Julie Archer on October 27th, 2014 Unpublished GiveWell non-verbatim summary of a conversation with Kriss Barker of PMC on October 31st, 2014 Source GiveWell non-verbatim summary of a conversation with Matthew Lavoie and Bassirou Kagone on October 14th, 2014 Source GiveWell non-verbatim summary of a conversation with Matthew Lavoie and Pieter Remes on October 17th, 2014 Unpublished GiveWell non-verbatim summary of a conversation with Matthew Lavoie on October 15th, 2014 Source GiveWell non-verbatim summary of a conversation with Midwife Thiombano Youmanli on October 14th, 2014 Unpublished GiveWell non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014 Source GiveWell non-verbatim summary of a conversation with Prof. Simon Cousens on October 28th, 2014 Source GiveWell non-verbatim summary of a conversation with Radio Djawoampo staff on October 15th, 2014 Unpublished GiveWell non-verbatim summary of a conversation with Roy Head and Will Snell on October 29th, 2014 Unpublished GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015 Source GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on May 13, 2015 Source GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015 Source GiveWell non-verbatim summary of a conversation with Roy Head, CEO; Jo Murray, Research Manager; and Will Snell, Director of Development on February 12, 2015 Source GiveWell non-verbatim summary of a conversation with Roy Head, CEO; Jo Murray, Research Manager; and Will Snell, Director of Development on November 5, 2015 Unpublished GiveWell non-verbatim summary of a conversation with Scriptwriter Guikierba Nanoano on October 14th, 2014 Unpublished GiveWell non-verbatim summary of a conversation with Secretary General of the Ministry of Communication, Adama Barro, on October 17th, 2014 Unpublished GiveWell non-verbatim summary of a conversation with Technical Advisor to the Ministry of Health, Bogard Creaté, on October 16th, 2014 Unpublished GiveWell non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014 Source GiveWell non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 15th, 2014 Source GiveWell non-verbatim summary of a conversation with Will Snell on July 31st, 2014 Source Global Burden of Disease Comparative Tool Source (archive) Global Health: Science and Practice DMI Editorial November 2015 Source Infosaid Media Landscape 2012 Source Ioannidis 2005 1 Source (archive) Ioannidis 2005 2 Source (archive) Lancet Child Survival Series 2003 Source (archive) Lassi et al (2014) Source (archive) LSHTM DMI RCT draft midline results June 2014 Unpublished LSHTM midline survey instrument 2014 Unpublished LSHTM protocol first draft 2013 Source Media Million Lives website Source (archive) Mulago Foundation website, "DMI" page Source (archive) Murray et al. 2015 Source Naugle and Hornik 2014 Source PMC summary results September 2014 Source Sarrassat et al. 2015 Source Technical inputs, enhancements and applications of the LiST (2011) Source (archive) UNICEF Facts for Life 2010 Source (archive) Unorthodox Philanthropy's 2015 Awards page Source (archive) WHO RMNCH guidelines Source 1 GiveWell non-verbatim summary of a conversation with DMI on April 24th, 2014 GiveWell non-verbatim summary of a conversation with Will Snell on July 31st, 2014 GiveWell non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014 GiveWell non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 15th, 2014 GiveWell non-verbatim summary of a conversation with Roy Head and Will Snell on October 29th, 2014 (unpublished conversation) GiveWell non-verbatim summary of a conversation with Joanna Murray on October 30th, 2014 (unpublished conversation) GiveWell non-verbatim summary of a conversation with Roy Head, CEO; Jo Murray, Research Manager; and Will Snell, Director of Development on February 12, 2015 GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on May 13, 2015 GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015 GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015 GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015 2 GiveWell non-verbatim summary of a conversation with Matthew Lavoie and Bassirou Kagone on October 14th, 2014 GiveWell non-verbatim summary of a conversation with Scriptwriter Guikierba Nanoano on October 14th, 2014 (unpublished conversation) GiveWell non-verbatim summary of a conversation with Matthew Lavoie on October 15th, 2014 GiveWell non-verbatim summary of a conversation with DMI scriptwriters on October 16th and 17th, 2014 (unpublished conversation) GiveWell non-verbatim summary of a conversation with DMI’s qualitative research team on October 17th, 2014 GiveWell non-verbatim summary of a conversation with Matthew Lavoie and Pieter Remes on October 17th, 2014 (unpublished conversation) 3 GiveWell non-verbatim summary of a conversation with Radio Djawoampo staff on October 15th, 2014 (unpublished conversation) 4 GiveWell non-verbatim summary of a conversation with Technical Advisor to the Ministry of Health, Bogard Creaté, on October 16th, 2014 (unpublished conversation) GiveWell non-verbatim summary of a conversation with Secretary General of the Ministry of Communication, Adama Barro, on October 17th, 2014 (unpublished conversation) 5 GiveWell non-verbatim summary of a conversation with Basilia Coefe on October 16th, 2014 (unpublished conversation) GiveWell non-verbatim summary of a conversation with Jean-Baptiste Guidard-Schmid on October 17th, 2014 (unpublished conversation) 6 GiveWell non-verbatim summary of a conversation with Anonymous District Medical Officer on October 14th, 2014 (unpublished conversation) GiveWell non-verbatim summary of a conversation with Midwife Thiombano Youmanli on October 14th, 2014 (unpublished conversation) 7 GiveWell non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014 GiveWell non-verbatim summary of a conversation with Prof. Simon Cousens on October 28th, 2014 8 GiveWell non-verbatim summary of a conversation with Kriss Barker of PMC on October 31st, 2014 GiveWell non-verbatim summary of a conversation with Julie Archer on October 27th, 2014 (unpublished conversation) 9 "We improve health outcomes by creating demand for healthy behaviours through radio and TV campaigns ... 5.9 million children worldwide die under the age of five every year ... Meanwhile, maternal mortality rates remain stubbornly high in many countries ... Many of the health interventions that can reduce maternal and child mortality (such as exclusive breastfeeding, treatment of diarrhoea, malaria and pneumonia, and giving birth in a health facility) are highly dependent on the actions of parents and other members of the community. As such, mass media campaigns that can change behaviours can help millions of people to provide or access these interventions, thus saving many thousands of lives." DMI website, "Issues" page, accessed September 2015 "The specific messages that DMI broadcast[s] are weighted according to their predicted impact on child mortality. DMI is changing the weight of broadcasting across its range of messages midway through the program, based on effectiveness data from the first 20 months and on changes in health indicators in Burkina Faso. DMI is focusing more on messages that have been proven to have a greater impact and are predicted to save the most lives. It will also be adjusting its modeling at the end of the trial to reflect the new evidence the RCT will provide. DMI’s recommendations at the end of the trial will include information on which messages would be most effective for future child mortality programs." GiveWell non-verbatim summary of a conversation with DMI on April 24th, 2014, Pg. 5 10 Some figures were originally in British pounds but have been converted to US dollars at a £1:$1.53 rate. Note that in addition to the projects summarized in Tables 1a and 1b, DMI has a few other activities: It is fundraising for a mass media tuberculosis study in Mozambique and for campaigns on gender equality and child survival topics in countries in the Sahel region. See DMI website, "Future Campaigns" page for details. We have not included these projects in Table 1b because DMI has not raised any funding for them and is unlikely to do so in the immediate future. DMI also has a for-profit arm through which it has run some campaigns. We did not include these campaigns in Table 1b because we have little information about them and have the impression that they do not represent a meaningful portion of DMI’s activities. A description of DMI’s goals for its for-profit arm is available in GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on May 13, 2015: “DMI has a for-profit arm (titled DMI, Ltd.). This for-profit organization functions as an alternate funding channel. For some funders, it is easier to give money to a for-profit organization. For example, there are types of USAID funding that can be more easily directed to for-profit organizations. DMI is also exploring using the for-profit arm to generate commercial earned income to cover DMI core costs. The for-profit arm has not been used for earned income activities very frequently; in late 2014, an early childhood development campaign was run through the arm, on behalf of the University of the West Indies. Generating income could include creating standard DMI campaigns for governments, or producing training or educational films on contract. The income-generating activities would not necessarily have to align with DMI's social mission, as long as they do not conflict with its ethical principles. Earned-income activities would have to generate a significant profit to justify the capacity they require. Roy is interested in pursuing earned-income activities. DMI has recruited a team of six MBA students from the London Business School to do scoping work on possible earned-income activities. DMI feels that its core activities have been picking up momentum. Its first priority is to maintain this momentum.” 11 Note that at least some of DMI's central office costs are distributed across these figures. DMI at least partially funds its central office via a management fee attached to grants it receives. In its 2015-2016 fiscal year, DMI collected $273,131 in management fees (sourced from DMI 2015-2016 fiscal year financials (unpublished document)). We do not believe that this figure accounts for all of DMI's central office costs, and we are unsure what other funding DMI's central office receives. We have not calculated DMI's total central office cost, and we have not separated out management fees from the figures in this table. 12 In 2014. This figure includes costs associated with DMI's central office in London. GiveWell 2014 DMI review 13 DMI is also broadcasting nutrition messaging in the Sahel region of Burkina Faso. DMI incorporated financial information for this nutrition campaign with financials from the child survival campaign. We have not separated out these costs. "DMI has partnered with SPRING (Strengthening Partnerships, Results, and Innovations in Nutrition Globally) to run a maternal, infant, and young child nutrition campaign in seven clusters concentrated around the Sahel Region." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 2 14With the potential to be extended further. 15 DMI 2015-2016 financial projection (unpublished document) 16DMI 2015-2016 financial projection (unpublished document) 17With the potential to be extended further. 18GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 6 19 DMI told us that messaging in this campaign will focus on “Mostly child survival and family planning, plus maternal health (antenatal care, giving birth in a health facility, etc).” DMI email from Will Snell on October 30, 2015 (unpublished document) 20With the potential to be extended further. 21 "DMI has secured an initial grant of $2.5m from Unorthodox Philanthropy, a US foundation. We need to raise a further $2m to cover the remaining costs of the first two years of the campaign." DMI website, "Mozambique" page, accessed October 2015 DMI has also used at least some of its funding from the Mulago Foundation (from which it received about $177K in 2015, according to DMI 2015-2016 financial projection) to start its campaigns in Mozambique. “We visited Mozambique in July 2014, and generated strong levels of interest from the government, broadcasters and funders in supporting mass media health behaviour change campaigns. In February 2015, with the support of the Mulago Foundation, we appointed a Country Representative, based in Maputo, to build on this interest by working alongside the Ministry of Health and other partners to design these campaigns, negotiate media partnerships, and secure funding. DMI is now officially registered in Mozambique as an NGO working on health promotion media campaigns, and is signing agreements with the Ministry of Health and broadcasters.” DMI website, "Mozambique" page 22 Note that DMI's central office costs are distributed across these figures. DMI funds its central office via a management fee attached to grants it receives. In its 2015-2016 fiscal year, DMI collected $273,131 in management fees (sourced from DMI 2015-2016 fiscal year financials (unpublished document)). We do not believe that this figure accounts for all of DMI's central office costs, and we are unsure what other funding DMI's central office receives. We have not calculated DMI's total central office cost, and we have not separated out management fees from the figures in this table. 23 "'Viral videos' mHealth pilot (2014-15) Since late 2014 we have been running a pilot project in south-western Burkina Faso: ‘viral videos’. Funded by a Grand Challenges Exploration grant from the Bill & Melinda Gates Foundation, the project promotes maternal and child health using short entertaining films that can be watched on mobile phones. People in Burkina Faso are increasingly watching and sharing short videos on their mobile phones. The project taps into this trend by creating ‘viral videos’ in local languages that promote positive maternal and child health behaviours. Each of the films delivers a specific message, such as the benefits of hand washing, and the promotion of treatment-seeking for diseases such as malaria, pneumonia and diarrhoea. We are currently analysing the initial evaluation results from this pilot and will publish more information soon." DMI website, "Burkina Faso" page, accessed October 2015 See also Gates Global Grand Challenges DMI page, accessed October 2015 24DMI 2015-2016 financial projection (unpublished document) 25 "We are currently broadcasting a family planning campaign on commercial radio and television stations in Kinshasa, the capital of DRC, with a population of 11 million people. Radio and television have the highest penetration of any media for both genders. We are working with commercial radio and TV networks in Kinshasa that have strong audience share. We are broadcasting short adverts in French and Lingala, which can be repeated several times per day and are effective at getting basic messages across." DMI website, "DR Congo" page, accessed October 2015 DMI told us that the campaign would be extended into 2016 in DMI email from Will Snell on October 30, 2015 (unpublished document). 26DMI 2015-2016 financial projection (unpublished document) 27 "Family planning RCT (2015-2019) We have recently secured $4.5m (from a total budget of $4.9m) to run a second RCT in Burkina Faso. This will test the impact of a community radio campaign on family planning (the primary outcome will be modern contraceptive prevalence). The campaign will broadcast a mixture of one-minute radio spots and weekly phone-in programmes on eight community radio stations, with eight control zones, over a three year period. The Principal Investigator is Rachel Glennerster, Executive Director of J-PAL. We are planning to run a baseline survey in late 2015 and to begin production and broadcasting in early 2016. We are currently seeking to identify a third funder to provide the remaining $400,000 needed for this project." DMI website, "Burkina Faso" page, accessed October 2015 28 "We have recently secured $4.5m (from a total budget of $4.9m) to run a second RCT in Burkina Faso." DMI website, "Burkina Faso" page, accessed October 2015 $4,900,000 / 4 years = roughly $1,225,000/year 29 GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015 (unpublished section) 30 DMI 2015-2016 financial projection (unpublished document) 31 "DMI is now partnering with Zungumza na Mtoto Mchanga (Talk to Your Baby, ZUMM) to develop and secure funds for an 18-month nationwide mass media campaign in Tanzania, to promote enhanced early childhood development. The first year of life presents a crucial opportunity for parents to develop the cognitive abilities of their children, simply by talking to them, a great deal, and in the right way. Children who have good spoken language skills (understanding, vocabulary and beginning to put words together) at the age of two enjoy much greater success at school (and in later life). However, few parents in Tanzania talk very much to their children before they themselves start talking. This is a huge missed opportunity, yet it is straightforward to persuade parents to talk to their babies by educating them about the benefits (an immediate reward of joyful interaction, and a long-term increase in their child’s cognitive capabilities and success). ZUMM has demonstrated that this works well at a small scale in Tanzania. DMI and ZUMM are partnering to bring this proven and cost-effective intervention to national scale in Tanzania through a mass media campaign." DMI website, "Future Campaigns" page, accessed October 2015 32 DMI 2015-2016 financial projection (unpublished document) 33 DMI 2015-2016 fiscal year financials (unpublished document) DMI's fiscal year runs from April 1 to March 31. 34 GiveWell 2014 DMI review Note that the calendar year and fiscal year figures are over different time periods, and thus cannot be compared directly. We have not attempted to put these figures in directly comparable terms. 35 For broadcasts beginning in March 2012, see DMI Burkina Faso funder report 2012 Q1 (unpublished document). For broadcasts ending in January 2015, see DMI Burkina Faso funder report 2015 Q2 (partially redacted). The baseline surveys for the RCT were conducted from December 2011 to February 2012. A midline survey was conducted in November 2013. "Cross-sectional surveys were performed in all clusters at 3 time points: At baseline, from December 2011 to February 2012, before the launch of the campaign; at midline, in November 2013, after 20 months of campaigning; and at endline, between November 2014 and April 2015, at the end of the campaign. (Endline results will be reported separately.)" Sarrassat et al. 2015, Pg. 4 The broadcast portion of the RCT was conducted from March 2012 to January 2015. "The DMI’s radio campaign was launched in March 2012 and ended in January 2015." Sarrassat et al. 2015, Pg. 4 The endline survey was conducted during the first quarter of 2015. "The endline results of the RCT are currently being collected; collection will be finished by mid-April." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; Jo Murray, Research Manager; and Will Snell, Director of Development on February 12, 2015, Pg. 1 "DMI expects to see the endline data by September. The endline results will likely be submitted to the Lancet for publication in December, but may be delayed. The results will probably be published in the Lancet in March or April 2016." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; Jo Murray, Research Manager; and Will Snell, Director of Development on February 12, 2015, Pg. 1 "Timeline for endline results ... DMI expects to receive and share high-level figures with partners in September or October 2015." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015, Pg. 3 "DMI does not have endline results yet from its child survival RCT in Burkina Faso, but expects to have them by the end of November [2015] (or by early November at the earliest)." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 1 Preliminary endline results were shared with us during GiveWell non-verbatim summary of a conversation with Roy Head, CEO; Jo Murray, Research Manager; and Will Snell, Director of Development on November 5, 2015 (unpublished document) 36 "The scale-up of DMI’s child health campaign in Burkina Faso is proceeding as planned. Broadcasting is under way, and will continue until the end of 2017." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015, Pg. 4 "The child survival national campaign will be aired on 29 stations (including one in Ouagadougou), which covers about 75% of the population." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 1 "DMI will try to extend this campaign through the end of 2018." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015, Pg. 5 37 "DMI has partnered with SPRING (Strengthening Partnerships, Results, and Innovations in Nutrition Globally) to run a maternal, infant, and young child nutrition campaign in seven clusters concentrated around the Sahel Region." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 2 "Family planning RCT (2015-2019) – We have recently secured $4.5m (from a total budget of $4.9m) to run a second RCT in Burkina Faso. This will test the impact of a community radio campaign on family planning (the primary outcome will be modern contraceptive prevalence). The campaign will broadcast a mixture of one-minute radio spots and weekly phone-in programmes on eight community radio stations, with eight control zones, over a three year period. The Principal Investigator is Rachel Glennerster, Executive Director of J-PAL. We are planning to run a baseline survey in late 2015 and to begin production and broadcasting in early 2016." DMI website, "Burkina Faso" page, accessed September 2015 38 "DMI designed and recruited funders for a one-year child survival campaign in DRC. It is currently broadcasting in 4 different languages on 35 stations, in eight of the country’s 11 provinces." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015, Pg. 6 "DMI plans to rebroadcast the radio spots it ran this year on a larger number of community radio stations for a full twelve months [in 2016] (up from eight months in the past year). … DMI’s next steps are to: 1. Negotiate costs for broadcasting spots 2. Finalize [monitoring and evaluation] arrangements and costs 3. Present its plan to potential funders If there is a small funding gap for the program, DMI may choose either to fill it using unrestricted funding or to go off the air for a few months. DMI is considering using GiveWell-directed funds for this program in the absence of funding from an external donor." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 5 39 "We are currently broadcasting a family planning campaign on commercial radio and television stations in Kinshasa, the capital of DRC…" DMI website, "DR Congo" page, accessed September 2015 40 DMI told us that messaging in this campaign will focus on “Mostly child survival and family planning, plus maternal health (antenatal care, giving birth in a health facility, etc).” DMI email from Will Snell on October 30, 2015 (unpublished document) 41 "DMI is aiming to launch its Mozambique child survival campaign next year." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 5 "DMI currently has one representative in Mozambique doing general planning and producing job descriptions for future hires." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 6 42 "The specific messages that DMI broadcast[s] are weighted according to their predicted impact on child mortality. DMI is changing the weight of broadcasting across its range of messages midway through the program, based on effectiveness data from the first 20 months and on changes in health indicators in Burkina Faso. DMI is focusing more on messages that have been proven to have a greater impact and are predicted to save the most lives. It will also be adjusting its modeling at the end of the trial to reflect the new evidence the RCT will provide. DMI’s recommendations at the end of the trial will include information on which messages would be most effective for future child mortality programs." GiveWell non-verbatim summary of a conversation with DMI on April 24th, 2014, Pg. 5 DMI has received funds that are restricted to broadcasting on a particular set of health messages, which limits DMI’s ability to focus on health topics it believes saves the most lives. GiveWell non-verbatim summary of a conversation with Matthew Lavoie on October 15th, 2014 For more on the midline results, see below. 43 "DMI messages addressing health facility-dependent behaviours have focused on attending ANC, health facility delivery, child’s growth monitoring, health care seeking in a health facility or with a Community Health Worker (CHW) for childhood illnesses (fever, cough or fast/ difficult breathing, diarrhoea with blood in the stools) and ORS for treating diarrhoea... DMI messages addressing home-based behaviours and related to maternal and newborn health have focused on savings during pregnancy, early breastfeeding after birth and delay of the first bath for low birth weight babies. Those messages related to child health have addressed exclusive breastfeeding, complementary food, increasing liquids and foods during diarrhoea and compliance with antibiotic for treating fast/difficult breathing. When national vitamin A distribution occurred, messages were broadcast to alert the population. Bed nets for pregnant women and under five children as well as sanitation through latrine ownership, safe disposal of children’s stools and hand washing with soap have also been promoted." LSHTM DMI RCT draft midline results June 2014, Pg. 5-6 (unpublished document) "DMI will continue to use essentially the same messaging and allocation of airtime for its child survival national campaign (which includes messaging on diarrhea, pneumonia, and malaria) as during the RCT." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 1 44 "DMI follows the same message selection process in the DRC as in Burkina Faso, though with more severe limitations on its ability to do on-the-ground research. Staff mobility is much more limited, as DMI does not allow staff to take unsafe flights, roads are often bad or nonexistent, and the DRC is about ten times larger than Burkina Faso." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 4 DMI has shared the following message briefs with us from its DRC campaign: DMI DRC message brief on birth registration DMI DRC message brief on exclusive breastfeeding and colostrum DMI DRC message brief on complementary feeding DMI DRC message brief on diarrhea treatment DMI DRC message brief on family planning (in Kinshasa) DMI DRC message brief on handwashing with soap DMI DRC message brief on care of low birthweight babies DMI DRC message brief on malaria DMI DRC message brief on maternal health DMI DRC message brief on pneumonia treatment-seeking DMI DRC message brief on immunizations DMI DRC message brief on drinking water purification and storage GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, unpublished section 45 "The campaign in Mozambique will include essentially the same topics as the Burkina Faso campaign, though with slightly different emphases. DMI is also including messaging on family planning and reproductive health, based on conversations with the Ministry of Health (MoH) and funders in the country." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 6 DMI told us that messaging in this campaign will focus on “Mostly child survival and family planning, plus maternal health (antenatal care, giving birth in a health facility, etc).” DMI email from Will Snell on October 30, 2015 (unpublished document) 46 "The campaign will broadcast one-minute radio spots. The advantages of using spots are that they can be very precisely scripted, they are drama-based and highly entertaining, and they can be broadcast up to 10 times per day. Previous campaigns have shown that the frequent broadcasting of spots can lead to large shifts in behaviour. During the RCT we have also been broadcasting nightly dramas (acted out live on each station in local languages). Our qualitative research suggests that, whilst these have been effective, the spots have greater impact, and are much more cost-effective as they require less manpower and so are cheaper to produce (especially on 28 stations)." DMI Burkina Faso proposal 2014 (unpublished document) DMI believed spots would be a more effective format before the trial started because of the repetition and clarity of health messages in spots, and the lack of attention span required. Feedback from listeners during the campaigns has consistently indicated that people are better able to recall the spots than descriptions of the long-format modules. Finally, regressing behavior changes measured at midline on the frequency of a message being covered by spots, and separately by long-format modules, found a stronger correlation with the former, suggesting that frequency of spots on a given health behavior explain variance in self-reported behavior change more than frequency of long-format modules on a given topic. GiveWell non-verbatim summary of a conversation with Roy Head and Will Snell on October 29th, 2014 (unpublished conversation), LSHTM DMI RCT draft midline results June 2014 (unpublished document) "DMI has received a letter of support from Mozambique’s MoH, has signed a memorandum of understanding (MoU) with the country’s main commercial television network, and is currently pursuing MoUs with the state-run television and radio networks." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 6 47 For a sample spot, see DMI website, "Burkina Faso RCT: radio spots" page, accessed October 2015 All participating stations in a country play the same spots during the same weeks, although the spots are translated to the local language for each station as necessary, and may be played at different times as the stations are managed independently. Each message ends with the same audio clip of a baby laughing, designed to brand the messages. GiveWell non-verbatim summary of a conversation with Will Snell on July 31st, 2014 (unpublished section); see also GiveWell non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014, Pg. 6 48 DMI website, "Burkina Faso RCT: radio spots" page DMI website, "Burkina Faso RCT: radio spots" spot audio 49 "DMI will not use long-format modules for its nationwide child survival campaign, mainly because they are expensive to produce, particularly in fragmented markets." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 3 DMI also noted that the midline results showed a positive dose-response correlation between exposure to spots and behavior change (i.e. more exposure to spots correlated with more behavior change), but did not show a dose-response correlation between long-format modules and behavior change. We have not independently analyzed this information. See Sarrassat et al. 2015, Pg. 15 DMI has not entirely abandoned the long module format, and plans to use it in some upcoming non-child survival campaigns. "DMI does not plan to eliminate long-format modules entirely (e.g., DMI’s eight-region family planning RCT in Burkina Faso will use long-formats, and DMI is using long-formats for its nutrition campaign at the request of the campaign’s funders)." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 3 50 "Since late 2014 we have been running a pilot project in south-western Burkina Faso: ‘viral videos’. Funded by a Grand Challenges Exploration grant from the Bill & Melinda Gates Foundation, the project promotes maternal and child health using short entertaining films that can be watched on mobile phones." DMI website, "Burkina Faso" page, accessed October 2015 See also Gates Global Grand Challenges DMI page, accessed October 2015 Because the structure of this project is very different from the structure of DMI's child survival campaigns, we do not focus on it in this review. 51 "To create DMI’s materials: DMI’s research team in the field produces message briefs and sends them to the research manager in DMI’s London office, who runs the briefs by Mr. Head and gives final approval. Approved message briefs are sent to the team of scriptwriters in Burkina Faso to be developed into scripts. The best ten or so scripts (out of about 20 or 30) are sent to the London office for DMI’s creative director to make sure that the message is clear and in line with the message brief. DMI’s London office selects the top four or five scripts to be pretested using focus groups in the field. This process will be standard procedure for DMI campaigns whenever possible going forward." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 2 DMI did not fully implement its standard production process for its 2015 campaigns in the DRC, and instead relied partly on previously produced material (more). DMI used some previously written scripts from its Burkina Faso script bank, and sent two of its Burkinabé scriptwriters to Burkina Faso to write new scripts as well: "For topics DMI had previously written on in Burkina the Burkina script bank was used to find relatively generic scripts. These scripts were given to a DRC researcher who selected the most applicable scripts and suggested edits that would make the scripts suitable for a Congolese audience. This worked very well, without this we would have had to reuse spots throughout the campaign." DRC learning and feedback session summary, Pg. 4 (unpublished document) "In order to train the local scriptwriters we reduced the number of local scriptwriters hired and seconded 2 Burkinabe scriptwriters. This was very successful and enabled output production to start quickly." DRC learning and feedback session summary, Pg. 3 (unpublished document) 52 "DMI has also done some qualitative research and hopes to do three or four more qualitative research trips. DMI uses one qualitative researcher, who is based in Kinshasa but travels extensively for fieldwork." GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 1 53 GiveWell non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014, Pg. 2-3 DMI sent us message briefs it created in Burkina Faso and has translated into English: DMI Burkina Faso message brief on ARIs DMI Burkina Faso message brief on breastfeeding DMI Burkina Faso message brief on vitamin A DMI Burkina Faso message brief on maternal health DMI Burkina Faso message brief on malaria DMI Burkina Faso message brief on low birthweight DMI Burkina Faso message brief on hygiene DMI Burkina Faso message brief on family planning DMI Burkina Faso message brief on diarrhoea DMI Burkina Faso message brief on 1000 days (French) DMI Burkina Faso message brief on hygiene 2 (French) DMI Burkina Faso message brief on exclusive breastfeeding (French) DMI Burkina Faso message brief on maternal nutrition (French) DMI Burkina Faso message brief on complementary feeding (French) It also sent us message briefs from the DRC campaign: DMI DRC message brief on birth registration DMI DRC message brief on exclusive breastfeeding and colostrum DMI DRC message brief on complementary feeding DMI DRC message brief on diarrhea treatment DMI DRC message brief on family planning (in Kinshasa) DMI DRC message brief on handwashing with soap DMI DRC message brief on care of low birthweight babies DMI DRC message brief on malaria DMI DRC message brief on maternal health DMI DRC message brief on pneumonia treatment-seeking DMI DRC message brief on immunizations DMI DRC message brief on drinking water purification and storage 54 DMI Burkina Faso message brief on ARIs, Pg. 1 55 GiveWell non-verbatim summary of a conversation with DMI scriptwriters on October 16th and 17th, 2014 (unpublished conversation) 56 "The best ten or so scripts (out of about 20 or 30) are sent to the London office for DMI’s creative director to make sure that the message is clear and in line with the message brief." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 2 GiveWell non-verbatim summary of a conversation with DMI scriptwriters on October 16th and 17th, 2014 (unpublished conversation) "DMI’s London office selects the top four or five scripts to be pretested using focus groups in the field." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 2 See also GiveWell non-verbatim summary of a conversation with DMI scriptwriters on October 16th and 17th, 2014 (unpublished conversation) 57 "During April and May our local scriptwriters were given intensive training on the writing and production of spots. The training was given by two experienced scriptwriters from our Burkina Faso project (appendix 3 contains the training schedule). Initially we recruited two local scriptwriters to work on the campaign, however midway through May one of these scriptwriters left DMI to join a TV show created by the Ministry of Culture. The campaign now had three scriptwriters producing the scripts; two experienced writers plus one DRC scriptwriter to provide local insight and knowledge." DMI DRC funder report 2015 Q2, Pg. 16 "There was no time or budget for training of creative staff. As national recruits are likely to be selected on basis of raw talent we must ensure there is time to train them. Overall the creative outputs required were too many, there were many subjects and no time to create a bank of recorded material. In order to produce the outputs required we used the Burkina script bank and made minor edits to scripts, this worked very well." DRC learning and feedback session summary, Pg. 3 (unpublished document) "The creative outputs were too great for the resource provided. Each new topic is a challenge for the writing team so number of topics should be limited or more lead in time provided. ... For topics DMI had previously written on in Burkina the Burkina script bank was used to find relatively generic scripts. These scripts were given to a DRC researcher who selected the most applicable scripts and suggested edits that would make the scripts suitable for a Congolese audience. This worked very well, without this we would have had to reuse spots throughout the campaign." DRC learning and feedback session summary, Pg. 4 (unpublished document) We do not know the specific reasons why the DRC campaign production process differed from DMI's process in Burkina Faso. Our understanding (largely based on the DRC learning and feedback session summary) is that the project's tight timeline and difficult operating conditions caused DMI to alter their standard practices. 58 "The spot production process is as follows. The Research Consultant and Research Manager in London have created detailed message briefs outlining each health message, barriers and contributing factors to behaviour change (see 4.1 Formative Research). At the start of a scriptwriting cycle the scriptwriters are briefed by the Country Director and the Research Consultant on the health message to be covered using the message briefs. Once the scriptwriters fully understand the health message they will begin to brainstorm script ideas. The best of these ideas are developed into written scripts (in French) and reviewed by the DRC team. A selection of the best scripts is made and these are then sent to the Creative Director in London. The Creative Director will then make comments on the scripts and select those scripts to be developed further. The local team will then make the revisions required and send the scripts back to the Creative Director for final approval. Once the scripts have been approved for use they are then sent to translators to translate scripts into the 4 broadcast languages. The translated scripts are then reviewed by the DMI team. Whilst the translations are taking place actors are selected for each script to be recorded (from a pre- selected pool of actors recruited by DMI in quarters 1 and 2). On the recording day the actors will rehearse the spots and then record them in the studio. An edit of the spots then takes place, adding sound effects and the DMI ident. The DMI ident is the sound of a baby laughing, and is played at the start and end of spots in order to help listeners identify them. The recorded spots are then reviewed by the DMI team in country and sent to the Creative Director for approval to broadcast. Once they are approved they can then be distributed to the radio stations (in accordance with the broadcast schedule in section 3)." DMI DRC funder report 2015 Q2, Pg. 16-17 59 The campaign will be expanding into areas where DMI content has not been broadcast before, so the reused spots will be fresh for the target audience. "DMI has already produced spots in all the four major languages in the DRC, and will not need to produce any new spots for the DRC next year." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 5, also discussed in an unpublished section 60 GiveWell non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014, Pg. 3-4 DMI researchers create a synthesis from each "pretesting research" session; we have seen several examples from Burkina Faso: DMI pretesting synthesis diarrhoea July 2012 (French) DMI pretesting synthesis hygiene January 2013 (French) DMI pretesting synthesis breastfeeding February 2013 (French) DMI pretesting synthesis maternal July 2013 (French) DMI pretesting synthesis ARI October 2013 (French) DMI pretesting synthesis malaria December 2013 (French) 61 GiveWell non-verbatim summary of a conversation with Matthew Lavoie and Bassirou Kagone on October 14th, 2014, unpublished section 62 GiveWell non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014, Pg. 4, also unpublished section DMI researchers create a report from each "feedback research" session. DMI has given us permission to publish the following feedback research reports: DMI qualitative monitoring report Solenzo July 2014 DMI qualitative monitoring report Bogandé December 2013 DMI feedback research summary September 2013 63 "DMI currently has three expatriate staff members in its Burkina Faso office and has retained its full team of 30 Burkinabe staff. DMI’s qualitative research team is also still operating." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 1 64 "The child survival national campaign will be aired on 29 stations (including one in Ouagadougou), which covers about 75% of the population." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 1 65 GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 1-2 "Family planning RCT (2015-2019) ... We are planning to run a baseline survey in late 2015 and to begin production and broadcasting in early 2016." DMI website, "Burkina Faso" page, accessed September 2015 66 DMI believes the quality and motivation of a station’s management is an important factor for determining if a station will continue to operate for the duration of the campaign, and if it will be resilient to challenges such as equipment failures or personnel issues. High-quality station staff make DMI much more confident that the station will be able to broadcast on schedule. The listenership’s size, language mix, and starting childhood mortality rate all affect estimated impact. Contracts with radio stations formalize compensation for airtime and can include in-kind contributions such as training station personnel or providing solar power systems. The biggest differences in costs per person reached come from working with national networks versus working with many unaffiliated community radio stations. GiveWell non-verbatim summary of a conversation with Matthew Lavoie and Bassirou Kagone on October 14th, 2014, Pg. 2-5 67 "DMI has good relationships with its 29 partner radio stations in Burkina Faso. These relationships have been easy to establish, especially as DMI’s positive reputation has spread. DMI has almost no competition from other advertisers in Burkina Faso, so there is a lot of available airtime and a significant opportunity to continue scaling up its messaging." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 3 68 "DMI has been making small airtime payments to its partner stations, and intends to increase the payments if more funding becomes available ... For now, DMI is continuing to make small airtime payments to the stations doing scale-up messaging for its child survival campaign (which will not use long-formats)." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 3-4 "DMI is also considering providing solar panel arrays to partner stations. This would give the stations access to reliable electricity, and would help DMI build its relationships with its partners. Partner stations are not able to afford the upfront costs of solar panels, but the approximately $500,000 project is more affordable for DMI (relatively)." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015, Pg. 7 "DMI hopes to provide solar panels to partner stations as a form of compensation in the future, but does not yet have the budget." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 3 DMI is not making airtime payments to partner stations that will be part of the upcoming family planning RCT. These stations will be broadcasting long-format modules as part of the RCT. DMI plans to pay production costs and the salaries of long-format presenters at these stations. "For stations that broadcast under the family planning RCT, DMI plans to pay production costs and the salaries of the presenters for long-format modules in lieu of airtime payments. Some radio stations prefer DMI program support and production assistance instead of cash payments." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 3 69 "DMI designed and recruited funders for a one-year child survival campaign in DRC. ... The DRC radio campaign is running for one year on a budget of approximately $1 million. The set-up period lasted four months, and broadcasting runs for eight months ending December 2015. " GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015, Pg. 6 "DMI has subcontracted the distribution work to an NGO that was already working with a network of community radio stations across the country. This may not be a long-term solution, but it has allowed DMI to initiate the program in a timely manner." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015, Pg. 6 "Because the DRC’s size and the difficulty of travel make building individual relationships with local radio stations difficult, DMI currently partners with Search for Common Ground (SFCG), a media network with existing relationships with local radio stations, as its distributor. DMI considered building its own relationships with local radio stations, working with national radio or television, or setting up its own radio station, but none of these options proved viable." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 4 70 DRC learning and feedback session summary (unpublished document) GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, unpublished section 71 "DMI is currently negotiating with Mozambique’s national radio network for a certain number of free spots per day. The national network is comprised of ten regional stations with additional stations in Maputo. The network broadcasts local content in local languages at certain times of day and national content from the capitol in Portuguese at other times. The national network is complicated but offers the advantage of being a single organization (as opposed to, e.g., DMI’s 29 partner stations in Burkina Faso). DMI will only partner directly with community stations in areas where there are coverage gaps in the national network. Mozambique’s national radio network has been willing to work with DMI, and Mr. Head believes it has a genuine desire to engage with its audience in a positive way." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 6 DMI also recently signed a memorandum of understanding with a major commercial television network in Mozambique. "DMI has received a letter of support from Mozambique’s [Ministry of Health], has signed a memorandum of understanding (MoU) with the country’s main commercial television network, and is currently pursuing MoUs with the state-run television and radio networks." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 6 We gained information about working with community radio stations in Mozambique from DMI email from Will Snell on November 6, 2015 (unpublished document) 72 LSHTM DMI RCT draft midline results June 2014 (unpublished document) 73 In 2012, we conducted initial research on the evidence for health behavior change from mass media and found it inconclusive, which led us to deprioritize further research. We are aware of Naugle and Hornik 2014, a recently published systematic review of mass media intervention effectiveness in the developing world, though we have not looked closely at the review due to our constrained capacity. 74 Technically the method of measuring child mortality will also be through self-report of the mother, but we believe that reporting on extremely discreet, memorable, and publicly-known events is much less likely prone to the biases associated with other self-reported data. "DMI expects to see the endline data by September. The endline results will likely be submitted to the Lancet for publication in December, but may be delayed. The results will probably be published in the Lancet in March or April 2016." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; Jo Murray, Research Manager; and Will Snell, Director of Development on February 12, 2015, Pg. 1 "Timeline for endline results ... DMI expects to receive and share high-level figures with partners in September or October 2015." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015, Pg. 3 "DMI does not have endline results yet from its child survival RCT in Burkina Faso, but expects to have them by the end of November [2015] (or by early November at the earliest)." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 1 75 LSHTM DMI RCT draft midline results June 2014, Pg. 3 (unpublished document) 76 “DMI’s campaigns had a large effect on some behaviors but little or no effect on others. They were more effective in changing infrequent behaviors, such as treatment of diarrhea using oral rehydration therapy (ORT), than in changing daily habits. Professor Cousens suggested that this might be because a woman with an ill child has a clear incentive to try a new treatment, but people experience fewer obvious concrete benefits to changing daily habits.” GiveWell non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014, Pg. 3 The version of the midline results we have seen is a draft; further changes to the results may occur before the midline results are published. GiveWell non-verbatim summary of a conversation with Prof. Simon Cousens on October 28th, 2014, unpublished section 77 Midline results paper: Sarrassat et al. 2015 Methodology paper: Murray et al. 2015 Editorial: Global Health: Science and Practice DMI Editorial November 2015 78 For example, “sought treatment for a child’s diarrhea within 24 hours” is a strict subset of “sought treatment for a child’s diarrhea”, since all who did the former did the latter as well, by definition. We made an exception for “received oral rehydration salts (ORS) for a child’s diarrhea,” which is technically a subset of “received ORS or more liquids or home made solution (HMS) for a child’s diarrhea”, but is an important metric we did not want to exclude. 79 Note that the values displayed in Tables 2a and 2b are crude difference-in-differences, which do not incorporate adjustments made by the RCT researchers to adjust for potential confounding. Also note that a more recent version of these results is available in Sarrassat et al. 2015. The headline results of the more recent version did not differ substantially from the results reported here. Row definitions "Sought medical assistance for symptoms" includes seeking treatment or receiving a consultation in response to a child’s symptoms. "Received specific medicine after symptoms" for diarrhea (ORS), malaria (antimalarials), and pneumonia (antibiotics). "Home-based diarrhea responses" includes homemade solutions, more liquids, an increased food intake, or an unchanged food intake. "Used full antibiotic/antimalarial course" includes a five-day treatment for antibiotics and a three-day treatment for antimalarials. "Used a health facility during pregnancy and birth" includes prenatal checkups and delivery. "Sanitation and hygiene" includes washing hands with soap before and after specific activities and having facilities such as latrine, pot, or soap available. "Food and nutrition" includes dietary behaviors and supplementation for children. "Other health behaviors" includes using bednets, taking vitamin A supplements, and others. Column definitions "Average % improvement in control group" refers to the percentage of women in the control group that reported this behavior when relevant at midline minus the percentage at baseline. For example, in the control group, the portion of women that gave birth in a health facility was 81.8% at baseline and 93.3% at midline, so for this measure, the control improved 11.5 percentage points. The values in this column are unweighted pooled averages of all the values in each category (this example is averaged into “Used a health facility during pregnancy and birth”). "Average % improvement in intervention group" refers to the percentage of women in the intervention group that reported this behavior when relevant at midline minus the percentage at baseline. To continue the example above, the portion of women in the treatment group who gave birth in a health facility was 56.0% at baseline and 65.3% at midline, for a 9.3 percentage point increase. Again, the values shown are unweighted pooled averages of the values in each category. "Average % intervention improvement minus % control improvement" is the difference between the treatment group improvement and the control group improvement. In the example above, it is (9.3 - 11.5) = -2.2 percentage points. Again, the values shown are unweighted pooled averages of the values in each category. 80 LSHTM DMI RCT draft midline results June 2014 (unpublished document) 81 Prof. Cousens, the lead researcher of the RCT, made the following points: Regarding an imbalance between treatment and control groups at baseline: "It is unclear whether a higher baseline mortality would lead to a larger or smaller relative reduction in mortality. It may be that regions with high baseline mortality have more opportunity for improvement in mortality rates. On the other hand, it may be that regions with higher mortality rates are poorer, more remote, and have worse health services, which would make it more difficult to reduce mortality...", GiveWell non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014, Pg. 4 Regarding biases associated with self-reported results: "The fact that the midline survey did not demonstrate positive impacts on some behaviors suggests that there was not a strong tendency for subjects to overstate their behavioral changes." GiveWell non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014, Pg. 4 For more detail on common bias issues in formal studies, see this page. 82 Regarding the issue of baseline imbalance, Prof. Cousens noted the following: "It is unclear whether a higher baseline mortality would lead to a larger or smaller relative reduction in mortality. It may be that regions with high baseline mortality have more opportunity for improvement in mortality rates. On the other hand, it may be that regions with higher mortality rates are poorer, more remote, and have worse health services, which would make it more difficult to reduce mortality..." GiveWell non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014, Pg. 4 83 "The results of the baseline survey indicated good balance between intervention and control clusters for home-based behaviours. While antibiotic and antimalarial treatments, as well as the delay in accessing these treatments, were balanced between arms, other health facility-dependent behaviours – ANC [antenatal care], facility delivery, attendance at well baby clinics, health care seeking and Oral Rehydration Solution (ORS) treatment - tended to be better in the control arm compared to the intervention arm." LSHTM DMI RCT draft midline results June 2014, Pg. 4 (unpublished document) "This imbalance probably reflects better access to health facilities in the control arm compared to the intervention arm. At baseline, GPS coordinates were recorded for each compound and distance to the closest health facility was calculated as the crow flies. 40% of women lived less than 2 km away from a health facility in the control arm versus 18% in the intervention arm. Among women sampled for the midline survey, these proportions were 47% versus 16% respectively (Figure 1)." LSHTM DMI RCT draft midline results June 2014, Pg. 4 (unpublished document) 84 "On average though, the control group had a lower baseline child mortality rate. The average child mortality rate was 136 per 1000 in the intervention group and 102 per 1000 in the control group. Professor Cousens believes that one reason for this is that the intervention clusters were on average farther from the capital of Burkina Faso. There is a correlation between the distance of a cluster from the capital and the mortality rate in a cluster, partly because clusters that are farther from the capital receive less investment." GiveWell non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014, Pg. 4 85 "Demographic characteristics of interviewed women were similar between surveys. Women were 30 years old on average in both arms. About 40%, 20%, 13% and 12% of women overall belonged to the Mossi, Gourmantche, Gournoussi and Peulh ethnic groups respectively (Figures 2a and 2b). In the two arms, although Mossi remained the main ethnic group, Gourmantche and Peulh were more common in the intervention arm and Gournoussi were more common in the control arm. Gouin, Karaboro and Turka were specific to the [town #2] cluster (intervention arm). While about half of women were muslim [sic] and half christian [sic] in the control arm, 60% were muslim [sic] in the intervention arm. In both arms, nearly all women were married, of whom about 40% were in a polygamous union. Education was quite low with only around 15% of women in the control arm and 10% of women in the intervention having been to school." LSHTM DMI RCT draft midline results June 2014, Pg. 4 (unpublished document) 86 DMI email to GiveWell October 3rd, 2014 (unpublished document) 87 After the randomization process, DMI reported, “...The good news is that we have by chance selected the radio stations with which we are most positive about the opportunities to establish effective working partnerships.” DMI Burkina Faso funder report 2011 Q3 (unpublished document). This comment suggests a possible source of bias that would inflate the midline results. 88 "The first section of the interview collected information on basic demographic characteristics and radio listenership. Subsequent sections addressed preventive and curative behaviours of relevance to child health. Questions regarding maternal and newborn health, i.e. antenatal care (ANC) and delivery, referred to the woman’s last pregnancy of more than 6 months duration. Questions regarding health care seeking, bed net use, nutrition and sanitation applied to her youngest child less than five years old. Illnesses (fever, cough, fast or difficult breathing, diarrhoea, dysentery) were recorded using a recall period of two weeks preceding the interview." LSHTM DMI RCT draft midline results June 2014, Pg. 3 (unpublished document) 89 To illustrate this point, the Global Burden of Disease Comparative Tool shows under-5 child mortality in Burkina Faso moving from 5,060 deaths per 100,000 in 1990 to 2,317 deaths per 100,000 in 2013 (a reduction of 2,743 deaths per 100,000). By comparison, under-5 child mortality in Western Sub-Saharan Africa was 4,731 deaths per 100,000 in 1990 and moved to 2,601 deaths per 100,000 in 2013 (a reduction of 2,130 deaths per 100,000). Under-5 child mortality in Sub-Saharan Africa overall declined from 4,138 deaths per 100,000 to 2,129 per 100,000 over this timeframe (a reduction of 2,009 deaths per 100,000). 90 GiveWell non-verbatim summary of a conversation with Kriss Barker of PMC on October 31st, 2014 PMC summary results September 2014 91 GiveWell non-verbatim summary of a conversation with Kriss Barker of PMC on October 31st, 2014 92 GiveWell non-verbatim summary of a conversation with Kriss Barker of PMC on October 31st, 2014 93 GiveWell non-verbatim summary of a conversation with Prof. Simon Cousens on October 28th, 2014, unpublished section 94 "The research team is concerned about the impact of a large health worker training program funded by the Bill & Melinda Gates Foundation and other funders. The program is active in two clusters, one from the control group and one from the intervention group. The research team believes that the program is having a significant effect in the control cluster but is uncertain how this compares to its effect in the intervention cluster. The research team is reviewing the study’s analysis plan with the study’s advisory committee. If the research team decides that the training program is significantly affecting the results of its RCT, it may perform a sensitivity analysis in which it removes those two clusters from the data." GiveWell non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014, Pg. 5 95 "There was recently a large bed net distribution program. Nearly 100% of households interviewed in the midline survey had at least one bed net, which was a large increase from the baseline. There were also national immunization days during which polio vaccines and vitamin A supplements were administered. The research team is documenting smaller health programs in various clusters, but in general it does not believe that there is a major difference between the programs in the intervention clusters and the control clusters." GiveWell non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014, Pg. 4 96 "According to DHS, there was a large decline in mortality in Burkina Faso between 2003 and 2010. The mortality rate in Burkina Faso has been declining faster than that of most other countries. This complicates the results of the RCT, because the study may have less power than planned to detect a reduction in mortality. Some of the decline in mortality rates in Burkina Faso is due to large national health programs, which the research team documented in its analysis." GiveWell non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014, Pg. 3-4 97 "The child survival RCT in Burkina Faso was designed to have 80% statistical power to detect a 19.9% reduction in mortality, and 50% statistical power to detect a 15% reduction in mortality. This initial modeling was based on a mortality rate of 168 deaths per 1,000 with a sample of 100,000 in 14 clusters. The small number of clusters limited the power of the study design." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015, Pg. 1 We are assuming that statistical tests will be conducted at a 95% confidence level. We have not vetted DMI's statistical power calculations. 98 "The mortality rate in Burkina Faso has fallen to around 96/1,000 in the five years that have passed since the original modeling. The declining mortality rate has led to fewer deaths being included in the data, which has reduced the study’s power. Due to this reduction, DMI does not expect that endline RCT results will demonstrate a statistically significant impact on child mortality. It is not sure what the precise magnitude of the reduction will be, but estimates that statistical power might have been reduced to approximately 20%." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015, Pg. 1 99 “The research team tried to avoid biasing interviewers. For example, it did not tell them which clusters were part of the intervention group and which were part of the control group. However, it was difficult to prevent the interviewers from learning this information, because some of the interview questions asked subjects about whether they had heard DMI’s radio messages.” GiveWell non-verbatim summary of a conversation with Prof. Simon Cousens on June 2, 2014, Pg. 2 100 "The survey detected 'contamination' in Gayéri control cluster with women in part of the cluster reporting listening to Djawoampo radio station broadcasting the campaign in Bogandé intervention cluster. Among 375 women interviewed in Gayéri cluster, a third - mainly in villages located to the North and North West of Gayéri (towards Bogandé) - reported having listened to Djawoampo radio station in the last seven days. All women interviewed in these villages (two-thirds of all women in the cluster) were therefore excluded from analysis. To account for the much smaller sample size of Gayéri cluster, analyses were weighted by the number of observations per cluster." LSHTM DMI RCT draft midline results June 2014, Pg. 4 (unpublished document) 101 DMI email to GiveWell October 3rd, 2014 (unpublished document) 102 GiveWell non-verbatim summary of a conversation with Matthew Lavoie and Bassirou Kagone on October 14th, 2014 103 GiveWell non-verbatim summary of a conversation with Matthew Lavoie on October 15th, 2014 104 GiveWell non-verbatim summary of a conversation with Matthew Lavoie and Bassirou Kagone on October 14th, 2014 105 The survey instrument is written in French and is translated by the surveyors into the local languages of the interviewees. Where necessary, a translator was used. We have used Google Translate to convert the French into English, and added context to some of the questions to make them clearer. LSHTM midline survey instrument 2014 (unpublished document) 106 We believe the following issues might prevent a family from successfully receiving health supplies or services despite seeking them: The distance to the nearest health center is too long to walk and alternative transportation is too expensive. The family does not have a member that can make the trip, due to poor health or lack of free time. The health center is open too few (or unpredictable) hours. The health center has long wait times for service. The health center is out of stock of the needed supplies. 107 DMI receives reports from the Ministry of Health in Burkina Faso that document the availability of health supplies in the regional offices that distribute supplies to health centers throughout the country. In 2013, the government reported that 75.9% of health centers in Burkina Faso were never short on key health supplies and that the figure for the seven regions where DMI has been active was 88.4%. DMI health supply availability 2013 We have not seen the technical details of how these figures were compiled and are not aware of any monitoring or auditing process that is used to verify the accuracy of these numbers. Furthermore, the existence of key health supplies at regional distribution centers does not guarantee their availability at health centers or with community health workers (which are supposed to sell antimalarials and ORS available in each village). DMI told us that its research team visits community health workers and health centers during field visits and verifies the availability of antimalarials, ORS, and antibiotics for pneumonia. “The QRT always checks a CHW’s stock of treatments when the QRT conducts field visits to villages.” GiveWell non-verbatim summary of a conversation with DMI’s qualitative research team on October 17th, 2014, Pg. 3 @DMI Updated Monitoring Guide October 2013@ (unpublished document) DMI does not systematically record the health supply information it discovers; instead its monitoring reports include discussions of the major limiting factors to behavior change that the research team perceives in each village (including the availability of health supplies where applicable). DMI email from Joanna Murray on November 3rd, 2014 (unpublished document) See DMI qualitative monitoring reports in the sources table or the DMI feedback research summary September 2013. Overall, DMI does not believe that the availability of health supplies is a major limiting factor in the effectiveness of DMI’s program in Burkina Faso, although antibiotics for pneumonia are harder to access than antimalarials and ORS for diarrhea. DMI email from Joanna Murray on November 3rd, 2014 (unpublished document) Antibiotics for pneumonia are generally not available from community health workers. GiveWell non-verbatim summary of a conversation with DMI’s qualitative research team on October 17th, 2014, Pg. 5 DMI reports that not having health centers (CSPS) nearby is a limiting factor in treatment for pneumonia. "Les IRAs/pneumonie: l’absence de CSPS dans le village, la méconnaissance même des symptômes de la maladie et de son processus d’évolution (certains parents trouvent que la toux et le rhume ne sont pas graves), la valorisation des soins traditionnels qui favorise le recours aux plantes traditionnelles pour soigner la toux avant de recourir au CSPS en cas d’échec (surtout à Bogandé).” DMI feedback research summary September 2013, Pg. 14 Google Translate suggests this translation: “IRAs [acute respiratory infections]/ pneumonia: the absence of CSPS in the village, the same lack of knowledge symptoms of the disease and its process of evolution (some parents find that cough and colds are not serious), the development of traditional care that promotes the use of plants traditional to treat cough before resorting to CSPS in case of failure (especially Bogandé).” For further discussion of how DMI thinks about health supply monitoring in Burkina Faso, see GiveWell non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 15th, 2014, Pg. 1-3 For our cost-effectiveness analysis we rely on the measures of increases in treatment received, rather than treatment sought, as a more direct measure of expected lives saved during DMI’s trial. 108 GiveWell non-verbatim summary of a conversation with Joanna Murray on October 30th, 2014 (unpublished conversation) The DHS program conducts standardized, nationally representative surveys on health and population topics in many countries. Surveys are conducted approximately every five years in each country. For more on the DHS, see Demographic and Health Surveys website This metric has flaws, such as over-counting people that report their child receiving treatment despite not having sought such treatment (such as when a family keeps an extra supply of ORS for diarrhea in the home, a practice many health NGOs encourage) and the possibility that some people who do not believe they have access to treatment may not report seeking it even if they would have if it were available. This is a concern for DMI’s ability to correctly predict the availability of health supplies in other countries (see our outstanding questions). If the ratio is a reasonable metric for the availability of health supplies, it seems to imply that health supply availability is a major limiting factor in Burkina Faso. Looking at the “received/sought” ratio for DMI’s treatment and control groups at baseline and at midline, the samples range from 27%-50% for antimalarials, 30%-46% for ORS, and 51%-74% for antibiotics to treat pneumonia. LSHTM DMI RCT draft midline results June 2014 (unpublished document) We note that the higher availability rates for pneumonia antibiotics than for malaria or diarrhea treatment is counter-intuitive given that it is not stocked by community health workers like the other two drugs are, possibly implying that people are not claiming to have sought treatment for pneumonia as often because they know there is not a source nearby. 109 "Dr. Murray believes that the Demographic and Health Surveys (DHS) Program has the best available data on supply-side availability of health supplies in the DRC. DHS data suggests that DRC health supply availability is similar to Burkina Faso." GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 3 110 DMI suggests the following sources for evidence of the life-saving effects of these treatments: UNICEF Facts for Life 2010 Lancet Child Survival Series 2003 WHO RMNCH guidelines: Essential Interventions, Commodities and Guidelines for Reproductive, Maternal, Newborn and Child Health Development and use of the Lives Saved Tool (LiST): A model to estimate the impact of scaling up proven interventions on maternal, neonatal and child mortality (2010); International Journal of Epidemiology; 39, Supplement 1 Technical inputs, enhancements and applications of the LiST (2011); BMC Public Health, 11, Supplement 3 Lassi et al (2014): Essential interventions for maternal, newborn and child health: background and methodology; Reproductive Health, 11, Supplement 1. DMI email to GiveWell October 3rd, 2014 (unpublished document) 111 "DMI hires two independent 'trackers' at each station to record whether the DMI spots are broadcast, and the broadcast times. The trackers are anonymous – the radio stations do not know their identity, and they do not know each other. This method has found that DMI stations have been broadcasting at least 10 times a day with good consistency between the trackers." GiveWell non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014, Pg. 1 "The third stage of the research process is feedback research, wherein listeners are surveyed after a broadcast. Feedback research aims to discover if DMI messages were heard, if they were understood, and if they lead to behavior change." GiveWell non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014, Pg. 4 112 See section on whether people hear DMI’s broadcasts and on the RCT results. 113 During its RCT in Burkina Faso, it appears that DMI had frequent contact with each of its radio broadcast partners through visits to radio stations, reports that it received from stations, and in some cases having DMI employees work at stations. ”Every radio station has one designated producer. In 2 of the 7 intervention zones, that producer is a DMI employee because the radio stations needed DMI’s assistance to consistently play DMI’s spots. All seven radio stations send DMI reports on a daily or weekly basis.  One of DMI’s two radio producers (Cheick Tall and Salim Salam) visits each radio station approximately once a month. A scriptwriter accompanies the producer. They speak with the radio producers, who usually speak French.  Mr. Kagone visits all 7 stations, but at a slightly slower pace. He visits the management team at the radio station. Generally, two DMI teams visit each radio station within every six-week period.” GiveWell non-verbatim summary of a conversation with Matthew Lavoie and Bassirou Kagone on October 14th, 2014, Pg. 5 ”As Matthew emphasises below, the importance of frequent visits to partner stations (who are sometimes slow to flag up potential problems) cannot be overstated. Regular visits also help to raise the quality of our partners’ broadcasting, and hence their market share… …The second crucial lesson that our 22 months of broadcasting has confirmed is that in managing an output-intensive partnership with a community radio station there is no substitute for ‘ears on the ground’. In spite of two years of collaboration, of important investments made in energy security, studio equipment, training and programming, we still only learn of serious problems in our partner stations during our field visits. ... Bassirou learned of these developments during one of his scheduled visits only after expressing our frustrations at Soro having missed several rehearsals.” DMI Burkina Faso funder report 2013 Q4, Pgs. 1-2 114 DMI shared 22 quarterly funder reports, which include discussions of challenges the radio stations face in maintaining consistent broadcasts. The primary issues discussed are loss of electrical power and personnel issues, and in some cases, these issues resulted in hours or days of missed broadcasts within a particular cluster. General comments from the reports "...with the exception of [radio #1], our partner stations have all broadcast DMI spots 10-16 times per day throughout the entire quarter; and that [radio #1] is now back up to the same level." DMI Burkina Faso funder report 2013 Q3, Pg. 2 "As before, power supply has been a headache, but by loaning generators and subsidising solar installations we have kept all seven stations on air with only very minor interruptions… The most important insight—which seems obvious with hindsight—is that energy, perhaps even more than creativity or research, is likely to be the determining factor of a successful community radio campaign. This quarter’s energy problems, occurring during the midline field research, were particularly problematic… The second crucial lesson that our 22 months of broadcasting has confirmed is that in managing an output-intensive partnership with a community radio station there is no substitute for ‘ears on the ground’. In spite of two years of collaboration, of important investments made in energy security, studio equipment, training and programming, we still only learn of serious problems in our partner stations during our field visits. This is particularly true in regards to the many personnel issues that have an impact on our interactive programs. In [town #6], [radio #6]’s star program host, and the principal host of our interactive program, [name redacted] passed the civil service exams to be a teacher and enrolled in the inductory training program without informing the management of [radio #6], or its parent organization. As this training made her increasingly unavailable for her daily programming responsibilities and for ‘DMI’ rehearsals she was obliged to tell the station director that she was hoping to teach full-time while continuing to work for [radio #6]. Bassirou learned of these developments during one of his scheduled visits only after expressing our frustrations at [her] having missed several rehearsals. In the end, the radio station’s management decided to fire [her]. Her departure has forced us to reorganize the team that broadcasts our interactive program. Cheikh has worked with the station’s management to redistribute programming responsibilities and convinced the station director to recruit a new staffer." DMI Burkina Faso funder report 2013 Q4, Pg. 1-2 "The media project in Burkina Faso continues to run very much according to plan, notwithstanding the usual technical and logistical headaches." DMI Burkina Faso funder report 2014 Q1, Pg. 1 Specific issues that caused missed broadcasts [Radio #5] in [town #5] failed to broadcast anything for 10 days in March and 2 days in November 2013 due to power outages. In both cases broadcasts only resumed when DMI bought and loaned a generator to the station. ("In early November we purchased a second 7kva backup generator and sent it to [radio #5] in [town #5]. As discussed in the Q1 2013 report, we purchased a first 7kva generator in March 2013. Our intention was that this generator be moved from station to station as our different partner stations experienced energy outages. This generator was first sent to [town #5] where [radio #5] had lost ten days of broadcasting to power outages. Once [radio #5]’s energy supply stabilized in June—the local cooperative repaired their generator—we sent DMI’s generator to [radio #4] in [town #4] who were experiencing seasonal outages caused by heavy rains. The second week of November, during the midline field research, both [radio #4] and [radio #5] were experiencing power outages. Anxious to get [radio #5] back on air during the midline research we purchased (€5,393) a second generator and installed it in [town #5], limiting the power outages to two days." DMI Burkina Faso funder report 2013 Q4, Pg. 1 [Radio #1] in [town #1] had power outages for much of 2013 which reduced their broadcasting from 16 hours per day to 4, preventing some of DMI’s programming from being broadcast. DMI bought the station a solar generator in October and rented them a fuel generator to last until the solar energy system was installed, which ultimately happened in February 2014. We do not know how many broadcasts were skipped during that time. "... as feared/anticipated [radio #1] in [town #1] has started to suffer serious energy shortages. From sixteen hours of broadcasting a day [radio #1] has been reduced to 4 hours on most days. The power shortages have started to reduce the number of spots they broadcast per day and have disrupted the interactive program. We have rented a backup generator in [town #1]—the station pays the fuel—to get the station back on-air immediately and have contracted with a supplier to install a solar energy system at [radio #1] (the station is the only one of our partner stations who has saved some of DMI’s monthly production stipend in order to make capital investments)." DMI Burkina Faso funder report 2013 Q3, Pg. 2 "[Radio #1] in [town #1] also experienced recurring power outages throughout November and December. As mentioned in the Q3 report, worried that [town #1]’s power supply problems would only get worse—the town depends on a local energy cooperative that is keeping two very old generators on life-support—we decided to install a solar energy system at [radio #1]. We signed a contract with a supplier in early October and rented a local generator to keep the station on air through the midline. Our pessimistic analysis of [town #1]’s energy supply has been confirmed by recent events; both of the [town #1] energy cooperatives’ generators caught fire in early January and the town of [town #1] no longer has electricity. Frustratingly, our solar supplier has experienced delays with his Spanish supplier of rechargeable batteries. We have prolonged the rental of the generator for [radio #1] through December and our solar supplier has agreed to cover a percentage of the additional rental costs." DMI Burkina Faso funder report 2013 Q4, Pg. 2 "In February, the solar energy system contracted in October 2013 was installed at [radio #1], [town #1]. The station now has energy independence 8 hours a day, allowing them to broadcast during both the morning and evening primetime hours. Frustratingly, only weeks after the solar energy system was brought online the radio station’s amplifier overheated and was seriously damaged. The station is currently broadcasting without the amplifier, with a reduced range of 25 km. We are working with the station to replace their amplifier as quickly as possible—radio amplifiers, and their replacement components, are not available in Burkina Faso." DMI Burkina Faso funder report 2014 Q1, Pg. 3 [Radio #1] in [town #1]'s amplifier was seriously damaged a couple weeks after the solar energy system was installed, reducing their broadcast range from approximately 50km to 25km. We do not know what percent of the target population previously reached by [radio #1] is not reached with the reduced range (three fourths of the broadcast area would be lost under simple assumptions), or whether the amplifier has since been replaced. "Frustratingly, only weeks after the solar energy system was brought online the radio station’s amplifier overheated and was seriously damaged. The station is currently broadcasting without the amplifier, with a reduced range of 25 km. We are working with the station to replace their amplifier as quickly as possible—radio amplifiers, and their replacement components, are not available in Burkina Faso." DMI Burkina Faso funder report 2014 Q1, Pg. 3 "Burkina Faso has a system of localised media whereby FM radio stations typically have a range of approximately 50km, which permit a cluster randomised trial." LSHTM protocol first draft 2013, Pg. 4 A perfect circle with a radius twice as large as another will have four times the area (7854 square kilometers versus 1963 square kilometers in the case of a radius of 50km and 25km, respectively). DMI does not use population less than 5km away from radio towers for its survey; the above calculation assumes those people still receive the broadcasts. In practice, changes in population density and topography complicate the calculation of lost audience; we have not seen an estimate of the lost audience from DMI. [Radio #4] in [town #4] experienced seasonal outages caused by heavy rains in the first half of 2013, and separately in November. DMI arranged for a generator to supply the station power starting in June 2013. We do not know how many broadcasts were skipped as a result of power outages. "[Radio #4] in [town #4] continues to rely on DMI’s backup generator to get their program on the air" DMI Burkina Faso funder report 2013 Q3, Pg. 2 "Once [radio #5]’s energy supply stabilized in June—the local cooperative repaired their generator—we sent DMI’s generator to [radio #4] in [town #4] who were experiencing seasonal outages caused by heavy rains. The second week of November, during the midline field research, both [radio #4] and [radio #5] were experiencing power outages. Anxious to get [radio #5] back on air during the midline research we purchased (€5,393) a second generator and installed it in [town #5], limiting the power outages to two days" DMI Burkina Faso funder report 2013 Q4, Pg. 1 DMI Burkina Faso funder report 2011 Q1 (unpublished document) DMI Burkina Faso funder report 2011 Q2 (unpublished document) DMI Burkina Faso funder report 2011 Q3 (unpublished document) DMI Burkina Faso funder report 2011 Q4 (unpublished document) DMI Burkina Faso funder report 2012 Q1 (unpublished document) DMI Burkina Faso funder report 2012 Q2 (unpublished document) DMI Burkina Faso funder report 2012 Q3 (unpublished document) DMI Burkina Faso funder report 2012 Q4 (unpublished document) DMI Burkina Faso funder report 2013 Q1 (unpublished document) DMI Burkina Faso funder report 2013 Q2 (unpublished document) DMI Burkina Faso funder report 2013 Q3 DMI Burkina Faso funder report 2013 Q4 DMI Burkina Faso funder report 2014 Q1 DMI Burkina Faso funder report 2014 Q2 (unpublished document) DMI Burkina Faso funder report 2014 Q3 (unpublished document) DMI Burkina Faso funder report 2014 Q4 (partially redacted) DMI Burkina Faso funder report 2015 Q1 (partially redacted) DMI Burkina Faso funder report 2015 Q2 (partially redacted) DMI Alive & Thrive progress report August 2015 (unpublished document) DMI SPRING quarterly report July 2015 DMI DRC funder report 2015 Q1 DMI DRC funder report 2015 Q2 115 See the above section on the unusual intensity of DMI’s RCT program for some details on how it interacted with partners during the RCT. 116 In addition to employing broadcast monitors, DMI contracted its DRC distribution partner to do broadcast monitoring. The distribution partner was contracted to provide data logs with spot broadcast times from 8 radio stations, as well as feedback from listener groups related to DMI broadcasts. "DMI plans to pay its distribution network a small amount to do rudimentary monitoring, in addition to setting up an external monitoring system (which would include spot checks of the distributor’s monitoring)." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 4 "In order to ensure that broadcast outputs are as programmed DMI have taken a number of steps. SFCG are contractually obliged to provide various pieces of data to show that broadcasting is taking place. This includes: data logs from 8 radio stations showing broadcast time of spots; feedback from radio listening groups confirming spots are being broadcast and; any feedback from listener groups or frontline SMS system relating to DMI broadcast. Due to a fire at SFCG during May this data was not available for the first month of broadcast, June data will be received by DMI during July." DMI DRC funder report 2015 Q2, Pg. 19 117 DMI told us that it had difficulty working with its DRC distribution partner this year, and plans to work with a different distribution partner for its 2016 scale-up of the DRC child survival campaign. DRC learning and feedback session summary (unpublished document) GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, unpublished section 118 GiveWell non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014, Pg. 1 119 "We have aimed to recruit an average of two independent trackers (i.e. not linked to one another or aware of their fellow tracker’s identity) to monitor each radio station. In the occasional area where we encountered problems with the radio we have three trackers." DMI DRC funder report 2015 Q2, Pg. 19-20 "They have been instructed to record every day the number of times they hear a DMI spot played" DMI DRC funder report 2015 Q2, Pg. 20 "The trackers have been taught to identify DMI spots by the sound of the laughing baby at the beginning and the end of each spot. They have had this sound played down the phone to them." DMI DRC funder report 2015 Q2, Pg. 20 "Trackers are also asked to report if a station is off air or if its programming has been affected by technical difficulties or power outages." DMI DRC funder report 2015 Q2, Pg. 20 120 "We have aimed to recruit an average of two independent trackers (i.e. not linked to one another or aware of their fellow tracker’s identity) to monitor each radio station." DMI DRC funder report 2015 Q2, Pg. 19 "Trackers are not told how many times a day the radio is supposed to play the spot, what the subject of the spot is or the times of day at which it is supposed to be played. Trackers are also asked to report if a station is off air or if its programming has been affected by technical difficulties or power outages. We have impressed upon the trackers that they must not tell the radio that they are working as a tracker. Likewise radio stations are not informed of the identity of trackers." DMI DRC funder report 2015 Q2, Pg. 20 121 For example, during its RCT in Burkina Faso, it appears that DMI had frequent contact with each of its radio broadcast partners through visits to radio stations, reports that it received from stations, and in some cases having DMI employees work at stations. ”Every radio station has one designated producer. In 2 of the 7 intervention zones, that producer is a DMI employee because the radio stations needed DMI’s assistance to consistently play DMI’s spots. All seven radio stations send DMI reports on a daily or weekly basis.  One of DMI’s two radio producers (Cheick Tall and Salim Salam) visits each radio station approximately once a month. A scriptwriter accompanies the producer. They speak with the radio producers, who usually speak French.  Mr. Kagone visits all 7 stations, but at a slightly slower pace. He visits the management team at the radio station. Generally, two DMI teams visit each radio station within every six-week period.” GiveWell non-verbatim summary of a conversation with Matthew Lavoie and Bassirou Kagone on October 14th, 2014, Pg. 5. ”As Matthew emphasises below, the importance of frequent visits to partner stations (who are sometimes slow to flag up potential problems) cannot be overstated. Regular visits also help to raise the quality of our partners’ broadcasting, and hence their market share… …The second crucial lesson that our 22 months of broadcasting has confirmed is that in managing an output-intensive partnership with a community radio station there is no substitute for ‘ears on the ground’. In spite of two years of collaboration, of important investments made in energy security, studio equipment, training and programming, we still only learn of serious problems in our partner stations during our field visits. ... Bassirou learned of these developments during one of his scheduled visits only after expressing our frustrations at Soro having missed several rehearsals.” DMI Burkina Faso funder report 2013 Q4, Pgs. 1-2. 122 See this footnote for details. 123 GiveWell analysis of DMI broadcast monitor data (unpublished document) 124 DMI Burkina Faso broadcast monitoring data January 2015 (unpublished documents) DMI Burkina Faso broadcast monitoring data June to September 2015 (unpublished document) DMI DRC broadcast monitoring data, May to September 2015 (unpublished document) GiveWell analysis of DMI broadcast monitor data (unpublished document) 125 A station-week with no broadcast monitor data could be attributed to either both monitors failing to report, or to the station not broadcasting DMI spots. We are unsure which attribution is more likely. We did not include partial weeks (weeks where data was only recorded for some of the days) in this count. Thus, this measure somewhat underestimates the total amount of station-time where no broadcasts were logged by both monitors. 1261 out of 21 station-weeks total. 127For a random sample of eight stations. 128For a random sample of eight stations. 12960 out of 448 station-weeks total. 130For a random sample of 11 stations. 131Actually an average standard deviation, because DMI hired more than 2 monitors at some stations. For a random sample of 11 stations. 132 281 station-weeks out of 735 station-weeks. The DRC dataset contained a variety of record-keeping methods. A day's entry could either: Contain no data Contain a note that the station was off the air Contain a note that the broadcast monitor failed to report Contain the number of spots that the monitor tracked that day To calculate the "% of station-weeks when both monitors did not record DMI spots", we counted weeks where one of the first three possibilities occurred for all monitors of a station. We also included instances of all monitors recording 0 broadcasts/day for an entire week in this count. We do not know if lack of data implies that the station was not broadcasting DMI spots, that DMI had not recruited monitors for that station yet, or that DMI monitors failed to report. In some cases, the dataset noted that the station was off the air, or that there were no trackers, or that the cause of the missing data was unknown. DMI DRC broadcast monitoring data, May to September 2015 (unpublished document) 133 "We use a ‘saturation’ approach that involves broadcasting ‘spots’ 6-10 times per day, over a prolonged period, and daily dramas if possible, to maximise audience exposure to our campaign." DMI summary 2014, Pg. 4 134 DMI shared four months of broadcast monitor reports from the RCT, as well as two months of automated reports from two stations. We asked for four months of randomly selected data because asking for more data seemed like it may have been burdensome. DMI Burkina Faso broadcast monitor guide 2014 (unpublished document) DMI Burkina Faso broadcast monitoring data June 2012 (unpublished document) DMI Burkina Faso broadcast monitoring data March 2013 (unpublished document) DMI Burkina Faso broadcast monitoring data December 2013 (unpublished document) DMI Burkina Faso broadcast monitoring data January 2015 (unpublished documents) DMI Burkina Faso automated monitoring summary 2012 Note that the some of the broadcast monitor data (DMI Burkina Faso broadcast monitoring data June 2012) and the automated reports are from 2012, and may not be representative of DMI's current national campaign in Burkina Faso. DMI has trialed automatic broadcast monitoring software at two Burkinabé stations. GiveWell non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014, Pg. 1. The data from the automated broadcast reports and the corresponding broadcast monitors seem to roughly approximate each other. DMI Burkina Faso automated monitoring summary 2012 The numbers from one monitor are systematically about 40% higher than those from the other monitor and the software: the average broadcasts logged per day for the two observers and software at Radio Loudon in Sapouy, respectively, were 9.4, 13.3, and 9.6. We are not sure of the reason for this discrepancy or whether it is ongoing. Note that this data is from 2012 and may not be representative of DMI's current national campaign in Burkina Faso. However, not all of DMI's partner stations have the broadcast software DMI uses (RadioBoss), and those stations that have the software do not use it consistently. "Although we piloted the use of RadioBoss software to monitor radio station outputs in Burkina Faso (as per data shared with GiveWell last year) we are not routinely using this software for monitoring since not all our radio stations have or use RadioBoss software. Even among those that do, their use of the software is not consistent enough for us to rely on this for monitoring purposes." DMI email from Dr. Joanna Murray on October 14, 2015 (unpublished document) Our impression is that DMI does not plan to do a significant amount of automatic broadcast monitoring in the near future. 135 DMI Burkina Faso broadcast monitoring data January 2015 (unpublished documents) 136 We counted about 30 station-days of no broadcasts from 2011 to the third quarter of 2014, about 0.4% of all broadcasts in that time. We also counted about 150 station-days of reduced broadcasts over this period (either due to shortened hours or restricted range of signal), representing about 2% of all broadcast-days. We counted no-broadcast days over the period of Q1 2011 to Q3 2014. DMI does not precisely account for missed broadcasting time. We are not sure how complete the accounting in the funder reports is. DMI Burkina Faso funder report 2011 Q1 (unpublished document) DMI Burkina Faso funder report 2011 Q2 (unpublished document) DMI Burkina Faso funder report 2011 Q3 (unpublished document) DMI Burkina Faso funder report 2011 Q4 (unpublished document) DMI Burkina Faso funder report 2012 Q1 (unpublished document) DMI Burkina Faso funder report 2012 Q2 (unpublished document) DMI Burkina Faso funder report 2012 Q3 (unpublished document) DMI Burkina Faso funder report 2012 Q4 (unpublished document) DMI Burkina Faso funder report 2013 Q1 (unpublished document) DMI Burkina Faso funder report 2013 Q2 (unpublished document) DMI Burkina Faso funder report 2013 Q3 DMI Burkina Faso funder report 2013 Q4 DMI Burkina Faso funder report 2014 Q1 DMI Burkina Faso funder report 2014 Q2 (unpublished document) DMI Burkina Faso funder report 2014 Q3 (unpublished document) 137 "DMI has scaled up monitoring and now has broadcast monitors for all 29 of its partner radio stations. DMI is transitioning to a more automated system in which its monitors will submit information via text message, which will be collected in a single central database." GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 3 We actually received data for what appears to be 32 stations, but DMI has told us it works with 29 stations. We assume we are minorly misinterpreting something or that there is a small discrepancy. The national campaign monitoring dataset is much larger than the RCT monitoring dataset we received; due to our constrained capacity we analyzed a random sample of 8 stations (out of 32 stations in all). The dataset contains data from the week of 6/26/2015 to the week of 9/28/2015. Not all broadcast monitors reported during all weeks in that range; many broadcast monitors were not reporting in the first weeks. We do not know if the lack of data in the early weeks of the campaign is due to the monitors failing to report, to monitors not having been recruited by that time, or to the stations not broadcasting DMI messaging at that time. There are many instances of one broadcast monitor not logging data during a week when their counterpart did; we attribute these instances to broadcast monitors failing to report. DMI Burkina Faso broadcast monitoring data June to September 2015 (unpublished document) GiveWell analysis of DMI broadcast monitor data (unpublished document) DMI recruits Burkinabé broadcast monitors via its staff's local connections. "In Burkina Faso, DMI uses its staffers’ strong local connections to find broadcast monitors." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 5 138 During the week of July 27th, 2015, no DMI broadcasts were logged by any monitor, across all stations. DMI Burkina Faso broadcast monitoring data June to September 2015 (unpublished document) Due to the consistency of this result across stations, we attribute this gap to DMI broadcasts being off the air (rather than to monitors failing to report). We do not know why this gap in broadcasting occurred. It is possible that it was related to the recent unrest in Burkina Faso, though this is speculation on our part. For more on unrest in Burkina Faso, see: DMI Burkina Faso funder report 2015 Q2, Pg. 1-3 2014 Burkinabé Uprising Wikipedia page 2015 Burkina Coup D'Etat Wikipedia page DMI notes that there are several reasons why the broadcast frequency logged by its monitors should be expected to be low during the initiation of its national child survival campaign. Specifically: "It is important to keep in mind that the tracker data from May to date is a little more erratic than the data collected during the RCT, which is to be expected for several reasons: These past six months we have been going through a transition phase as we scale up in Burkina. To prepare for the A&T campaign, we have been putting in place both the national broadcast network and the expanded tracker network. We have quadrupled the number of radio stations we partner with and have increased our number of trackers from 14 to 60. We know that there is always a learning curve with the radio stations, so it is going to take us a few months to get them to fully comply with our requested number of spots broadcasts. We also need to verify the effectiveness of our tracker network. We expect that not all of our new trackers will be able to accurately and effectively do the job (for example we have already weeded out two in Kaya). This also, unfortunately, takes a little time and some effort. … We have also changed the schedule of data collection. We now call the trackers at the beginning of the week for their data from the previous week as opposed to calling them on Fridays. This change created a bit of confusion among trackers for a few weeks." DMI email from Dr. Joanna Murray on October 14, 2015 (unpublished document) 139 Recruiting broadcast monitors is difficult due to poor travel conditions and the lack of strong local contacts. "In the DRC, DMI identifies, trains, and receives reports from its broadcast monitors by phone (in contrast to Burkina Faso, where DMI is able to actively meet and recruit monitors in the field)." GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 2 "In Burkina Faso, DMI uses its staffers’ strong local connections to find broadcast monitors. DMI did not enter into the DRC with a strong network like this. This, coupled with the difficulty of travel, makes it challenging to recruit monitors." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 5 In the DRC, DMI recruits and manages its broadcast monitors by phone. "DMI also uses local individuals as broadcast monitors who report how often they hear the spots. Finding these monitors via recommendation without strong local contacts is very difficult. DMI has considered partnering with other organizations (e.g., Caritas) for help finding monitors. DMI reaches potential monitors by phone. In Burkina Faso, DMI uses its staffers’ strong local connections to find broadcast monitors. DMI did not enter into the DRC with a strong network like this. This, coupled with the difficulty of travel, makes it challenging to recruit monitors." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 5 DMI notes that poorer-quality data from DRC broadcast monitors is to be expected due to the less robust recruitment and management DMI is able to implement in the DRC. "As discussed during our recent call (15th Sept) and detailed in the corresponding call notes, we have some concerns around our tracker data from DRC. We have identified, trained and received data from our trackers in DRC all by phone (in contrast to Burkina where we can actively meet, recruit and monitor our trackers in the field). Despite diligent monitoring by our team in Kinshasa, some uncertainty over the reliability of our tracker data in DRC exists. As you will have read in report from our August field research trip, recall of our spots in the areas visited was high after only a couple of months on air. This suggests frequency of spot broadcasts (in the Bandundu region at least) may well be higher than our trackers in this region have been reporting." DMI email from Dr. Joanna Murray on September 21st, 2015 (unpublished document) DMI is not using broadcast monitoring software for its 2015 DRC child survival campaign. DMI may use monitoring software in the DRC in future campaigns. "DMI is not currently using monitoring software for its DRC campaigns. DMI may potentially partner with a different network of stations next year, and may consider using that network’s software for monitoring." GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 2-3. 140 "As at the time of reporting DMI has 61 trackers in place." DMI DRC funder report 2015 Q2, Pg. 20 141 “Since May 2015 we have been broadcasting a child health campaign on 35 community radio stations in selected districts in eight of the 11 provinces of DRC (Bandundu, Bas Congo, Equateur, Kasai Occidental, Kasai Oriental, Maniema, North Kivu and Orientale).” DMI website, "DR Congo" page, accessed October 2015 142 “Since May 2015 we have been broadcasting a child health campaign on 35 community radio stations in selected districts in eight of the 11 provinces of DRC (Bandundu, Bas Congo, Equateur, Kasai Occidental, Kasai Oriental, Maniema, North Kivu and Orientale).” DMI website, "DR Congo" page, accessed October 2015 The broadcasting monitoring data appears to contain data from 37 stations, but DMI has written that it is working with 35 stations. We are unsure what explains this discrepancy; we would not be surprised if we were slightly misinterpreting the data in some way or if the information on DMI’s site is slightly outdated. The DRC campaign monitoring dataset we received was large; due to our constrained capacity we analyzed a random sample of 11 stations (out of 35 stations in all). The dataset contains data from the week of 5/11/2015 to the week of 9/28/2015. Not all broadcast monitors reported during all weeks in that range; many broadcast monitors were not reporting in the first weeks. We do not know if the lack of data in the early weeks of the campaign is due to the monitors failing to report, to monitors not having been recruited by that time, or to the stations not broadcasting DMI messaging at that time. DMI DRC broadcast monitoring data, May to September 2015 (unpublished document) GiveWell analysis of DMI broadcast monitor data (unpublished document) 143 The DRC campaign monitoring dataset we received was large; due to our constrained capacity we analyzed a random sample of 11 stations (out of 35 stations in all). In this sample, one station was off the air for the entire period (a DMI monitor reported that the station was not broadcasting), and two others either were off the air or were not being tracked by broadcast monitors. For another two stations in the sample, no monitoring data was reported for approximately half of the period. The dataset contains data from the week of 5/11/2015 to the week of 9/28/2015. Not all broadcast monitors reported during all weeks in that range; many broadcast monitors were not reporting in the first weeks. We do not know if the lack of data in the early weeks of the campaign is due to the monitors failing to report, to monitors not having been recruited by that time, or to the stations not broadcasting DMI messaging at that time. DMI DRC broadcast monitoring data, May to September 2015 (unpublished document) GiveWell analysis of DMI broadcast monitor data (unpublished document) For the eight stations in the sample for which data was reported, the broadcast monitors logged 4.75 DMI spots per day, on average. The average standard deviation between the reporting of the broadcast monitors in each zone was 1.93 spots. Note that the average standard deviation is a different measure of variation than the average discrepancy we reported for the Burkina Faso campaigns. We calculated standard deviations for the DRC campaign because DMI recruited 3 or more broadcast monitors at some stations. Average discrepancy can be divided by 2 to approximate average standard deviation. Also note that there was substantial variation in broadcast frequency across stations. Station averages ranged from 0.84 spots/day to 11.71 spots/day. DMI DRC broadcast monitoring data, May to September 2015 (unpublished document) GiveWell analysis of DMI broadcast monitor data (unpublished document) 144 “In the intervention arm, 75% of women reported recognising at least one of the two spots played at the end of the interview. Among regular radio listeners, this proportion increased at 88%. Reported recognition of long format program was lower, at 54% and 67% of all women and regular radio listeners respectively. In contrast, in the control arm, 20% of women reported recognising at least one of the two spots and 12% reported listening to the long format program.” LSHTM DMI RCT draft midline results June 2014, Pg. 7 (unpublished document) 145 Based on the non-overlapping confidence interval bars in figure 7a. LSHTM DMI RCT draft midline results June 2014, Pg. 13 (unpublished document) 146 A fourth radio station, Lumière, was mentioned in the qualitative research report, which noted that Lumière does not broadcast in months when school was not in session (see @DMI DRC Qualitative Research Final Report August 2015@, Pg. 6). Lumière was not listed in the DRC broadcast monitor dataset DMI shared with us, and we do not know if it was intended to be a DMI partner station. The villages were in Kwilu Province and Kwango Province. 147 "Nous proposons de mener des recherches qualitatives dans les provinces où les messages DMI sont diffusés. Ce sera surtout limité à Bandundi, Bas-Congo et de Kinshasa, en raison de contraintes logistiques et il l'accessibilité par la route. Lorsque cela est possible si, nous visons également à mener des recherches qualitatives dans d'autres régions qui reçoivent DMI émissions de la campagne." DMI DRC qualitative research protocol June 2015, Pg. 10 Google Translate suggests the following translation: "We propose to conduct qualitative research in provinces where DMI messages are broadcast. This will be mostly limited to Bandundi, Bas-Congo and Kinshasa, due to logistical constraints and accessibility by road. When possible if we also aim to conduct qualitative research in other areas that receive DMI emissions of the campaign." "Nous allons choisir au hasard des villages qui sont situés dans la plage de diffusion de nos stations de radio partenaires. Ceux-ci devront être villages qui sont accessibles par la route. Nous prendrons contact avec nos stations de radio partenaires pour nous aider à assurer la liaison avec les chefs de village locaux et d'obtenir l'approbation et le soutien requis pour mener à bien des groupes de discussion et des entrevues dans leurs communautés locales." DMI DRC qualitative research protocol June 2015, Pg. 13 Google Translate suggests the following translation: "We will randomly select villages that are located within the broadcast range of our radio station partners. These should be villages that are accessible by road. We will contact our partner radio stations to help us liaise with local village leaders and get the approval and support required to conduct focus groups and interviews in their local communities." "Pour la collecte des données proprement dite, la démarche a consisté à identifier au préalable les personnes à interroger en respectant les quotas prévus pour chaque focus group (sexe et nombre de participant(e)s), à négocier les rendez-vous avec les personnes identifiées, à organiser et planifier la réalisation des entretiens et enfin, à les réaliser. En effet, en ce qui concerne les entretiens de groupe en milieu rural, une personne désignée par le chef de village avait pour rôle d’aider à lancer un appel destiné aux femmes ayant des enfants de moins de cinq ans et aux hommes présentant les mêmes caractéristiques. Les 12 femmes et les 12 hommes (arrivés les premiers sur les lieux) s’ils (elles) correspondaient au profil exigé, étaient sélectionné(e)s, tout en veillant à maximiser la diversité sociodémographique des postulant(e)s, autant que faire se peut." @DMI DRC Qualitative Research Final Report August 2015@, Pg. 6 Google Translate suggests the following translation: "For the collection of actual data, the approach has been to identify the advance the interviewees respecting quotas provided for each focus group (gender and number of participant(s)), to negotiate the appointment with the identified persons, organize and plan the realization of interviews and finally to the achieve. Indeed, as regards group interviews in rural areas, a person designated by the village chief 's role was to help launch a call for women with children under five and men having the same characteristics. The 12 women and 12 men (who arrived first on the scene) if they matched the required profile, were selected, while ensuring to maximize the social and demographic diversity of the applicants, provided that possible." We do not know exactly what participant profile was selected for. Being the parent of a child under the age of 5 was a primary criterion. 148 @DMI DRC Qualitative Research Final Report August 2015@, Pg. 5-6 GiveWell analysis of DMI DRC qualitative research final report 149 @DMI DRC Qualitative Research Final Report August 2015@, Pg. 5-8 GiveWell analysis of DMI DRC qualitative research final report Average broadcast monitor rates are given for comparison. Rates are from DMI DRC broadcast monitoring data, May to September 2015, "Averages" sheet (unpublished document); averaged over a period of July 6, 2015 to September 28, 2015 Note that, on average, monitors did not record broadcast frequencies reaching DMI's "saturation" intensity (6-10 times/day) at these stations. Nsemo broadcast at saturation intensity in late August through September. 150 Nsemo: monitors logged 5.03 broadcasts/day on average, standard deviation (by week): 2.73 broadcasts/day Tomisa: monitors logged 3.02 broadcasts/day on average, standard deviation (by week): 1.14 broadcasts/day Kimvuka na Lutondo: monitors logged 4.52 broadcasts/day on average, standard deviation (by week): 0.83 broadcasts/day DMI DRC broadcast monitoring data, May to September 2015, "Averages" sheet (unpublished document) 151Two focus groups were conducted in Langa, but the qualitative research report did not record how many participants listened to the partner station for either group. See DMI DRC qualitative research final report August 2015, Pg. 6, 8 152Two focus groups were conducted in Makiala, but the qualitative research report only recorded the number of participants who listen to the partner station for one group. See DMI DRC qualitative research final report August 2015, Pg. 8 153Only one focus group reported on both measures, thus 9/12 = 75% 154 A weighted average of all groups that reported on both measures. Both Langa groups and one of the Makiala groups were excluded from this average because the number of participants who reported listening to the partner station was not recorded. GiveWell analysis of DMI DRC qualitative research final report 155 DMI spots are tagged with a laughing baby at the beginning and end. This laughing baby tag is used to determine whether listeners heard the spots during research interviews. GiveWell non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014, Pg. 4 "Tous les auditeurs interviewé(e)s de ces radios respectives, partenaires locales de DMI dans le Kwilu et le Kwango, déclarent qu’ils entendent et reconnaissent les spots qui commencent et se terminent par un rire de bébé." @DMI DRC Qualitative Research Final Report August 2015@, Pg. 8 Google Translate suggests this translation: "All interviewed auditors of the respective radios, local partner of DMI in the Kwilu and Kwango, say they hear and recognize the spots that start and end with a baby laughing." 156 We do not know the precise proportion of participants who reported listening to the radio that also reported listening to child and maternal health topics. "De manière globale, que ce soit dans le Kwilu, à Idiofa Cendre et à Itshuam (où la Radio ciblée est Nsemo) ; que ce soit dans la Ville de Kikwit, au Village Kongila ou au Village Langa (où la Radio ciblée est Tomisa) ou dans le Kwango, à Kenge Centre, au Village Makiala ou au Village Kongomitela, (où la Radio ciblée est Kinvuka na lutondo); au cours des entretiens de groupe ou individuels, les répondant(e)s qui suivent une radio partenaire locale de DMI ont soutenu, quasi à l’unanimité, suivre aussi des messages sur la santé des enfants et des femmes enceintes. Notamment, l’usage de moustiquaire imprégnée, l’utilisation de préservatif, la prise en charge médicale d’un enfant qui fait de la fièvre, de la diarrhée et/ou de la toux ; la mise en pratique des mesures préventives contre le paludisme ou les IRA, etc." @DMI DRC Qualitative Research Final Report August 2015@, Pg. 8 Google Translate suggests this translation: "Overall, whether in Kwilu in Idiofa Ash and Itshuam (where Radio Nsemo is targeted); whether in the city of Kikwit, the Village Kongila or Langa Village (where the target Radio is Tomisa) or in the Kwango at Kenge Centre at Makiala Village or Kongomitela Village (where the target Radio is Kinvuka na lutondo); in the individual or group interviews, the respondent(s) that follow a local partner radio DMI supported, almost unanimously, also follow messages on the health of children and pregnant women. In particular, the use of treated nets, use of condoms, the medical care of a child who has a fever, diarrhea and / or cough; the practical application of preventive measures against malaria, ARI, etc." 157 DMI DRC qualitative research protocol June 2015 The question guide outlined the questions that the researcher should ask the interviewee. It did not give information about other potentially relevant factors, such as whether the researcher mentioned that they were affiliated with DMI and the partner station, how much information the researcher revealed about health behaviors during the interview, or how the interviewee responses were recorded. 158 DMI qualitative monitoring report Solenzo July 2014 DMI qualitative monitoring report Bogandé December 2013 DMI qualitative monitoring report Banfora July 2013 (unpublished document) DMI qualitative monitoring report Bogandé April 2013 (unpublished document) DMI qualitative monitoring report Djibo March 2013 (unpublished document) DMI qualitative monitoring report Kantchari May 2013 (unpublished document) DMI qualitative monitoring report Ohya July 2013 (unpublished document) DMI qualitative monitoring report Sapouy March 2013 (unpublished document) DMI qualitative monitoring report Solenzo June 2013 (unpublished document) DMI qualitative monitoring report Banfora January 2014 (unpublished document) DMI qualitative monitoring report Djibo February 2014 (unpublished document) DMI qualitative monitoring report Kantchari March 2014 (unpublished document) DMI qualitative monitoring report Ohya June 2014 (unpublished document) DMI qualitative monitoring report Sapouy December 2013 (unpublished document) DMI feedback research summary September 2013 159 "La plupart des répondants restituent les messages correctement, mais certains répondants reprennent les messages avec des erreurs, par exemple sur : le nombre de CPN recommandé, l’âge d’introduction de l’eau et de la bouillie enrichie dans l’alimentation du bébé et l’âge limite de la consultation du nourrisson sain (pesée). DMI feedback research summary September 2013, Pg. 5 Google Translate suggests this translation: "Most respondents recall the messages correctly, but some respondents show messages with errors, for example: the number of CPN recommended age introduction of the water and porridge enriched in the baby's diet and the age limit of the consultation healthy infant (weighing)." "Some hear our messages but misunderstand or reinterpret parts of them, e.g., claiming herbal concoctions are okay after a baby is 6 months or thinking exclusive breastfeeding should only be done until 3 months." DMI Burkina Faso funder report 2014 Q1, Pg. 6 DMI also reports that it has adjusted its message several times based on misinterpretations and errors. GiveWell non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 11th, 2014, Pg. 4 160 "On constate que les hommes autant que les femmes retiennent bien les messages clés des spots et des modules notamment sur l’aspect du recours précoce au CSPS en cas de maladie de l’enfant. Généralement, parmi les thèmes rappelés, les mieux retenus se rapportent à la CPN, au paludisme et aux diarrhées. Les moins rappelés concerne les IRAs, la pesée des enfants et le colostrum. En outre, les thèmes diffusés la veille des entretiens individuels ou des focus group sont facilement rappelés par les participants. De nombreux répondants ont retenu un message transversal aux thèmes de santé dont traite DMI et qui se résume ainsi : 'en cas de maladie de l’enfant il faut le conduire rapidement au centre de santé.' La plupart des répondants restituent les messages correctement, mais certains répondants reprennent les messages avec des erreurs, par exemple sur : le nombre de CPN recommandé, l’âge d’introduction de l’eau et de la bouillie enrichie dans l’alimentation du bébé et l’âge limite de la consultation du nourrisson sain (pesée)." DMI feedback research summary September 2013, Pg. 5 Google Translate suggests this translation: “It is found that both men and women hold the key messages of good spots and modules including the appearance of the early use of the CSPS in case of illness of the child. Generally, among the subjects recalled the best selected relate to the ANC [antenatal care], malaria and diarrhea. Recalled less respect IRAs [acute respiratory infections], weighing children and colostrum. In addition, themes released on the eve of individual interviews and focus groups are easily recalled by the participants. Many respondents identified a transverse message to health topics including trafficking and DMI which is as follows: 'in case of illness of the child must quickly lead to the center health.' Most respondents correctly restore the messages, but some respondents show messages with errors, for example: the number of CPN recommended age introduction of the water and the slurry enriched in the baby's diet and the age limit of the consultation healthy infant (weighing).” 161 DMI breastfeeding research summary March 2014 DMI breastfeeding research regional synthesis – Sahel DMI breastfeeding research regional synthesis – Boucle du Mouhoun DMI breastfeeding research regional synthesis – Centre Sud et Centre Est DMI breastfeeding research regional synthesis – Cascades DMI breastfeeding research regional synthesis – Centre West DMI breastfeeding research regional synthesis – East DMI breastfeeding research regional synthesis – North 162 DMI's messaging on colostrum (first milk) feeding have generally achieved wider acceptances than its messaging on exclusive breastfeeding (i.e. only giving infants breastmilk during their first six months of life). For anecdotal examples of listener understanding, see these excerpts: On colostrum feeding in the Boucle du Mouhoun region: "The advent of health centre deliveries and sensitisation by health workers, reinforced by the DMI radio campaign, has led to wide-spread acceptance of exclusive breastfeeding among the current generation of new-borns: 'The radio and health agents say that it is the best milk that protects infants against illnesses' (focus group, Dira). Radio has played an important role: 'It’s the first time that we learn that the first milk is good and that we should not discard it; we used to throw it away because they said it was dirty; we gave the milk that comes afterwards' (pretest focus group, Koma)." DMI breastfeeding research regional synthesis – Boucle du Mouhoun, Pg. 3 On exclusive breastfeeding in the Boucle du Mouhoun region: "The number of women who exclusively breastfeed is still low in Solenzo. This is because elder women in a household are the key decision-makers when it comes to infant feeding practices, and because they strongly believe in traditional feeding ways – which includes giving water, herbals brews and solid foods before the age of 6 months, young mothers feel compelled to follow their advice, even if they may be inclined to try exclusive breastfeeding. The pressure of elder women is strong: 'Once a mother returns from the health centre after delivering her baby, they will give brews to the child. At 4 months, they’ll introduce porridge in the diet' (pretest focus group, Solenzo) or 'Here, mothers may refuse brews, but the grandmothers, when they are in the house, will insist and often, their ideas rule' (41-year-old father, Solenzo) … Many mothers and their entourage lack a correct understanding of the true nutritional value of breast milk. There is a persistent belief that breast milk alone will not suffice to help a baby grow and that is why people will insist on introducing solid food even in the first months of life. It is a tradition in Solenzo to do so, and there is plenty of resistance to change this: 'What I don’t like, is telling people to only give breast milk during the first 6 months of life, because will that not weaken the baby? Here, we see babies at 4 months drinking porridge and eating other foods. It frequently happens' (father, Koma)." DMI breastfeeding research regional synthesis – Boucle du Mouhoun, Pg. 3-4 On colostrum feeding in the Sahel region: " Some women say that other ethnic groups let the new-born breastfeed with another lactating woman, until the new-born mother’s ‘real’ and ‘good’ milk comes through (Tondiata village). Among those who did not give the first milk in the past, there has also been a change, according to our last feedback research trip. A 39-year-old father in Maty said his wife used to discard the colostrum but now, 'since we heard on the radio that one should not discard this milk, we give the baby the first milk. We’ve done that the past year, because it gives the baby health and makes it develop well.'" DMI breastfeeding research regional synthesis – Sahel, Pg. 3 On exclusive breastfeeding in the Sahel region: "A 30-year-old mother of two says she believes in giving only breast milk and even admitted to giving up herbal brews but 'You have to give a baby plain water because his throat needs it. Even if other mothers do not want to give water, I cannot accept that because without water the baby will die' (Maty village) … Another important reason is a belief that women may not have sufficient breast milk and that babies need complementary food sooner than at 6 months … 'A grandmother said her daughter could not breastfeed her baby, so “from the 5th day, I gave the baby an enriched porridge' (Tondiata village). Another 50-year-old mother of 7 said that it is difficult to avoid introducing solid foods before 6 months 'because mothers don’t have enough nutritious food to produce enough milk' (Maty village). … Brews are thought to be curative and give strength but they are also given to pacify a baby. A 40-year-old mother of 6 said, 'The single advantage of brews (tisanes) is that they help the baby fall asleep; when you force-feed a baby with a brew and shea butter, he sleeps well, all day long. You are not disturbed and you can work well' (Maty village)." DMI breastfeeding research regional synthesis – Sahel, Pg. 3-4 163 "Comme illustration, les répondant(e)s ont identifié clairement, en les rapportant les conseils tels qu’ils passent sur les chaines partenaires, plusieurs spots DMI, notamment ceux relatifs au lavage des mains suite aux recommandations du nouveau chef de village et celui qui se réfère à l’économie du savon ; celui relatif à de moustiquaire imprégné d’insecticide, à la prévention de la diarrhée ou du paludisme/malaria et à la prise en charge de l’enfant qui en est affecté, à l’allaitement exclusif du bébé jusqu’à 6 mois et à enrichir son alimentation à partir de cet âge, à bouillir l’eau douteuse et bien la conserver pour en consommer, la vaccination, etc." DMI DRC qualitative research final report August 2015, Pg. 8 Google Translate suggests this translation: "As illustration, the respondent(s) identified clearly by relating the advice as they spend on partner channels, DMI several spots, including those related to hand washing following the recommendations of the new village chief and one that refers to the soap economy; that relating to mosquito nets impregnated with insecticide, prevention of diarrhea or malaria / malaria and care of the child who is affected, the baby exclusive breastfeeding to 6 months and enrich its supply from this age to boil questionable water and keep it for good to consume, vaccination, etc." "D’une manière générale, à Idiofa Cendre, à Itshuam (Idiofa rural où la Radio ciblée est Nsemo), dans la Ville de Kikwit, au village Kongila et au village Langa, dans le Kwilu comme à Kenge Centre, au village Makiala, au village Kongomitela, dans le Kwango, plusieurs histoires de vie manifestent l’adoption de comportement positif en matière de santé maternelle et infantile." DMI DRC qualitative research final report August 2015, Pg. 10 Google Translate suggests this translation: "Generally, in Idiofa Ash at Itshuam (which the targeted rural Idiofa Radio is Nsemo) in the city of Kikwit, the village and the village Kongila Langa, in Kwilu as Kenge Centre Makiala the village, the village Kongomitela in the Kwango, many life stories show the adoption of positive behavior maternal and child health." 164 "Mais à ce propos, le bémol c’est que, parallèlement, à entendre les répondant(e)s, lors des entretiens de groupe comme individuels, en ce qui concerne la prise en charge de l’enfant qui manifeste un début de fièvre ou de toux, les vielles recettes des ancêtres telles que l’enduire et/ou le masser avec du nsaku nsaku pillé et dilué dans de l’eau versé sur sa tête, de l’encens mélangé à de l’huile de palme; des tisanes ou décoctions ancestrales à base de feuille de nyoka nyoka pressée, lumbalumba, mulolo, et tant d’autres, la liste étant loin, très loin d’être exhaustive, font encore autorité." DMI DRC qualitative research final report August 2015, Pg. 8-9 Google Translate suggests this translation: "But in this regard, the downside is that, in parallel , to hear the respondent (s) , during group discussions as individual , regarding the care of the child who shows the beginnings of fever or cough , the old recipes of ancestors such as coated and / or massaging it with Nsaku Nsaku looted and diluted with water poured on his head , incense mixed with palm oil ; herbal decoctions or ancestral base sheet nyoka nyoka pressed Lumbalumba , mulolo , and many others , the list is far, very far from being exhaustive, are still authority." "Dans tous ces cas, toux comme fièvres, les recettes traditionnels d’abord et l’automédication ensuite, priment encore sur l’acheminement de l’enfant au centre médical, auquel on ne se tourne qu’en cas d’urgence, comme dernier recours, après que toutes les autres tentatives de soigner se soient révélées vaines." DMI DRC qualitative research final report August 2015, Pg. 9 Google Translate suggests this translation: "In all these cases, coughing as fevers, traditional recipes first and then self-medication, still take precedence over the child's transportation to the medical center, which we will turn in an emergency, as last resort after all other attempts to heal themselves be proved fruitless." 165 DMI will recruit seven or eight data collectors, each of whom will gather survey data from a region of Burkina Faso where DMI's campaign is being broadcast. The collectors will record data on several indicators, including demographic information, media exposure, mobile phone use, and maternal and child health knowledge and behaviors. This data will be collated by a data manager at DMI's office in Ouagadougou, then sent to DMI's London office for time series analysis. DMI plans to roll out this system in late 2015; data collected via the system will replace data collected for the RCT. GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 3-4 "Using GiveWell funds, we have also been developing plans to build our own in-house research capacity to conduct times series evaluations. Starting from September, we will be recruiting a team of seven regional data collectors and a data manager based in our Ouagadougou office." DMI Burkina Faso funder report 2015 Q2 (partially redacted), Pg. 3 "This team will collect data on a monthly basis, on a wide range of indicators including demographic information, media exposure, mobile phone use, as well as knowledge, attitudes and behaviours related to reproductive, maternal, neonatal and child health issues. This will enable DMI to perform time series analyses that can be used to monitor and inform program activities as well as to evaluate the impact of our campaigns." DMI Burkina Faso funder report 2015 Q2 (partially redacted), Pg. 3 "The M&E researchers will replace the evaluation work that Centre Muraz and [the London School of Hygiene and Tropical Medicine] are currently doing as part of the RCT." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; Jo Murray, Research Manager; and Will Snell, Director of Development on February 12, 2015, Pg. 2-3 166 "DMI also plans to maintain its current qualitative research team." GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 4 "During these trips we also started putting in place the network of radio trackers in each broadcast zone. We recruited two people in each zone to monitor the radio broadcasts and track the number of spots broadcast per day; contracts have been signed with 50 monitors for the Alive & Thrive campaign." DMI Alive & Thrive progress report August 2015, Pg. 1 167 “Were DMI to scale up in Burkina Faso, Mr. Lavoie believes he could create a simpler, cleaner proposal to pitch to new partner stations. For example, he is interested in an agreement that would have DMI install solar panels for a radio station in return for that station broadcasting DMI’s spots. DMI would own the solar panels for the three years the spots were playing and could remove the panels if the station reneged on its end of the agreement. After the three years, ownership of the solar panels would be transferred to the radio station. DMI would still pay subsidies to the radio stations, but these subsidies would be reduced. Mr. Lavoie notes that the cost of installing solar panels is cheaper than the cost of paying for airtime for three years. This agreement would appeal to radio stations because: Radio stations frequently have energy issues. Solar panels provide a consistent energy source. A radio station’s existence is threatened when it loses energy; if it cannot stay on air, it will go out of business as listeners stop tuning in. Repairing broken generators is expensive. Solar panels would save the station money by avoiding these repairs. There would be a guard for the solar panels (most radio stations already have a guard for their equipment). Mr. Lavoie believes a proposal like this could work in other countries as well. He also believes it would be more cost-effective than DMI’s current model. An additional benefit of using solar panels is that there would be less need for the radio’s financial office to send as many reports to DMI about repair expenses.” GiveWell non-verbatim summary of a conversation with Matthew Lavoie and Bassirou Kagone on October 14th, 2014, Pg. 7 168 "DMI performed a baseline survey in May and will perform an endline in December in two regions (an intervention area and a control area) in Bandundu province." GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 1 169 "In total, 395 mothers of a child aged under five years were interviewed in Kenge (the intervention area) and 450 mothers were interviewed in Popokabaka (the control area). The survey captured basic demographic information (mother’s age, ethnicity, education, religion, occupation) as well as data on radio ownership and penetration, listening habits, maternal and child health behaviours and knowledge (questionnaire, research protocol and ethical approval attached in appendix 6). The majority of women interviewed (73%) had lived in their village for 2 or more years, and most were of Lunda ethnicity (71%)." DMI DRC funder report 2015 Q2, Pg. 16 170 "The endline survey is scheduled to take place in early December 2015. In addition to measuring maternal and child health knowledge, attitudes and behaviours, we will capture data on mothers’ exposure to the DMI campaign. This will allow us to compare effects among different levels of campaign exposure, as well as comparing to the control areas, helping us to attribute any measured changes to the intervention." DMI DRC funder report 2015 Q2, Pg. 16 DMI has conducted some informal quality control checks on KPSH's surveying work. Overall, DMI has found that KSPH is carrying out the surveying appropriately, though there have been some communication issues between the organizations. "During surveying, DMI’s team performed some informal quality control checks in the field by, e.g., speaking with locals to ensure that KSPH had surveyed the correct villages. DMI did not identify any villages that had not been surveyed that were supposed to be. DMI did find that transportation difficulties had delayed some surveying (and that there had been some miscommunication with KSPH about this)." GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 2 171 "DMI performed a baseline survey in May and will perform an endline in December in two regions (an intervention area and a control area) in Bandundu province. DMI expects to receive data from its child survival campaign survey by February and to have results a couple of months after that (DMI is due to report on these results to its funders in June)." GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 2 172 Note that due to the difficulty of travel in the DRC, qualitative research trips will only occur in two provinces close to Kinshasa, the capital of the DRC where DMI's country office is located. "We are planning to conduct four qualitative field research trips during the third and fourth quarters of 2015. These will allow us to conduct formative research as well as post-broadcasting feedback research in rural areas of Bandundu and Bas Congo provinces (since these are accessible by car from our Kinshasa office)." DMI DRC funder report 2015 Q2, Pg. 15 We have reviewed DMI's broadcast monitor dataset from its 2015 DRC child survival campaign, from which we infer that DMI is currently employing broadcast monitors for the campaign and plans to continue to do so. 173 GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, unpublished section 174 GiveWell’s bottom-line cost per child life saved estimate is based in part on privately shared information, and does not match precisely with the default outputs of the cost-effectiveness model available above. We estimate cost per child life saved for children under five, because that is the age group in which the vast majority of lives saved are expected, and on which DMI’s program focuses. Our model defaults to incorporating effects and costs from Burkina Faso, DRC, and Mozambique, because that is our best guess for where DMI will run child survival campaigns in 2016 (see below for more on DMI's 2016 plans). The countries that are incorporated in the overall cost per life saved estimate can be adjusted in the "Overall CEA" worksheet. Our estimate uses the radio-listening behavior of women (rather than men or all adults) because in Burkina Faso and many of the places to which DMI expects to expand, women are more often responsible for the decisions and activities targeted by DMI. DMI believes that it is important for both men and women to hear DMI’s messages, but especially women. See GiveWell non-verbatim summary of a conversation with DMI’s qualitative research team on October 17th, 2014 Our current estimate of DMI's cost per life saved is approximately the same as the estimate we published in December 2014. However, our model has been substantially altered since December 2014, and the similarity in cost per life saved is coincidental. In the first half of 2015, we noticed some errors in our analysis of DMI's cost-effectiveness, and in June 2015 we published a corrected version of the analysis (see this post for details). Both of these versions modelled DMI's cost-effectiveness for a hypothetical campaign in Cameroon, where DMI was considering opening a child survival campaign. DMI has since deprioritized opening a campaign in Cameroon, so we have adjusted our analysis for this year to model campaigns in Burkina Faso, the DRC, and Mozambique, with the option to incorporate estimates from additional countries in which DMI may open child survival campaigns. For DMI deprioritizing a campaign in Cameroon, see GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 4 175 "According to our model, the cost per DALY of a DMI mass media campaign in most countries is in the range of $2-$15. This would make mass media behaviour change campaigns in most countries as cost-effective as any other interventions currently used in public health. We are now carrying out a scientific trial in Burkina Faso to prove this prediction. (As a rule of thumb, there are around 30 DALYS per individual 'life', so the cost per life saved of our campaigns is $60-$450, depending on the country.)" DMI website, "Costing Impact" page, accessed October 2015 176 Consequently, the benefits we predict are smaller, and we only rely on the few behaviors that DMI encouraged during the trial that have already shown results consistent with large life savings, such as increased treatment for childhood diarrhea and symptoms of pneumonia. 177 LSHTM DMI RCT draft midline results June 2014 (unpublished document) 178 Demographic and Health Surveys website Global Burden of Disease Comparative Tool 179 For another example of external validity discounts in our analyses, see row 11 of the "Parameters" sheet of our 2014 analysis of deworming, cash transfers, and iodine fortification (.xlsx). 180 It is difficult to know how strong a discount to apply for this consideration. We have seen some potentially relevant figures in John Ioannidis’s analysis of biomedical literature (see Ioannidis 2005 1 and Ioannidis 2005 2), though there are substantial differences between biomedical studies and DMI’s midline results. For another example of replicability discounts in our analyses, see row 13 of the "Parameters" sheet of our 2014 analysis of deworming, cash transfers, and iodine fortification (.xlsx). 181 Note that the final results from DMI’s trial, expected in late 2015, will include child deaths reported by parents, which we believe to be significantly less prone to self-report bias than self-reporting on events or behaviors that are not as well-defined, memorable, and publicly known. 182 DMI website, "Mozambique" page, accessed November 2015 183 DMI has seen a draft of this review and we offered them a chance to respond to any part of it. They suggested this comment be added to this section. 184 GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 5-6 185 Figures originally in British pounds, converted to US dollars at a £1:$1.53 rate. "DMI currently holds £817,202 of unrestricted funding. Some of these funds have been allocated to be used on costs in future years and costs such as tax have not yet been deducted." DMI email from Will Snell on October 30, 2015 (unpublished document) (£817,202)*1.53 = $1,250,319.06 DMI 2015-2016 financial projection (unpublished document) "We therefore need to have a stable £500k of reserves to cover six month’s running costs. ... The money from / due to GiveWell’s recommendation is shown as in our reserves for 2015, even though most of it is scheduled to be spent in 2016" DMI email from Will Snell on October 2, 2015 (unpublished document) 186 "If DMI were to receive $12 million to use over two years, Mr. Snell would prioritize spending in the following order: $2 million: to fill the funding gap for the Mozambique campaign for 2016 and 2017. $850,000: one year of the DRC program. $1.5 million: national scale-up in Burkina Faso over two years. (This is an estimate because DMI will not know the exact funding gap in Burkina Faso until it finalizes the budget for its family planning RCT.) $4.5 million: two-year national child survival campaign in Tanzania $3.5 million: maternal and neonatal child health campaign in either Mali or Niger over two years, or in each country for one year (the latter could be beneficial because one year’s worth of funding might enable DMI to leverage follow-up funding in-country)." GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 6 Note that we estimate a cost of $3 million for the potential Mali and Niger campaigns instead of $3.5 million because of an update we received by email. “Campaigns in either country would cost in the region of $1.5m per year.“ DMI email from Will Snell on October 2, 2015 (unpublished document) 187 If DMI received $1 million unrestricted and there was still a funding gap for its Mozambique campaign, DMI would put the funds towards closing the Mozambique campaign funding gap. However, if DMI were to receive $1 million after the Mozambique campaign had been funded, it would use the money differently. We do not know the details of what DMI would do in this situation; we assume that DMI would apply the funding to the next highest-priority use (i.e. its child survival campaigns in the DRC and Burkina Faso). "If DMI received $1 million now, it would likely use it to close half of the funding gap in Mozambique. However, if $1 million comes in later (e.g., three or four months from now), that funding gap may already be closed." GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 6 If DMI received GiveWell-directed funding in December 2015/January 2016, and the Mozambique funding gap still existed, we believe that GiveWell-directed funding would be committed to the Mozambique gap. 188 The cost of this campaign is lower than average because DMI plans to reuse content produced in 2015 for the 2016 scale-up, and will not need to produce additional content in 2016. "DMI needs $850,000 of funding for the DRC, mostly for scaled-up broadcasting and M&E." GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 5 "DMI has already produced spots in all the four major languages in the DRC, and will not need to produce any new spots for the DRC next year." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 5 If DMI does not raise sufficient funding for the campaign by the time the current DRC campaign goes off the air (at the end of December 2015), DMI may cease broadcasting in the DRC for a time as it continues to fundraise. "We are currently fundraising for the second phase of our child survival radio campaign in DRC (from January to December 2016). Our ambition is that this will operate across all 11 provinces, reaching at least 10 million people and saving 13,700 lives." DMI website, "DR Congo" page, accessed September 2015 GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 4-5 For 50% chance: @DMI 2015-2016 Financial Projection@ (unpublished document) DMI may use GiveWell-directed funds (from the 2014 giving season) to fill this funding gap if it occurs. "If there is a small funding gap for the program, DMI may choose either to fill it using unrestricted funding or to go off the air for a few months. DMI is considering using GiveWell-directed funds for this program in the absence of funding from an external donor." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 5 189 DMI's funding situation in Burkina Faso is also influenced by funding it has received for its family planning RCT, which is projected to run until 2019. DMI has received $4.5 million for this RCT (out of a total budget of $4.9 million). "Family planning RCT (2015-2019) ... We have recently secured $4.5m (from a total budget of $4.9m) to run a second RCT in Burkina Faso. This will test the impact of a community radio campaign on family planning (the primary outcome will be modern contraceptive prevalence). ... We are currently seeking to identify a third funder to provide the remaining $400,000 needed for this project." DMI website, "Burkina Faso" page, accessed September 2015 DMI's Burkina Faso funding gap is currently uncertain because the family planning RCT funding has not been finalized, and some cost-sharing will occur between the family planning and child survival campaigns (such as both campaigns being run out of one office and overseen by a single country director. "$1.5 million: national scale-up in Burkina Faso over two years. (This is an estimate because DMI will not know the exact funding gap in Burkina Faso until it finalizes the budget for its family planning RCT.)" GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 6 There is also a $400,000 funding gap in Burkina Faso for DMI's family planning RCT. DMI is not prioritizing this gap because it does not need to be filled until 2017. "The $400k FP RCT gap is not a priority for immediate funding since it doesn’t need to be filled until 2017." DMI email from Will Snell on October 30, 2015 (unpublished document) We believe that DMI would use GiveWell-directed funding allocated to the Burkina Faso campaign to cover its full range of messaging topics (see above) for 2016 and 2017. "DMI may use unrestricted funding to fill funding gaps at the conclusion of grant periods, and hopes to maintain the full range of messaging topics throughout the campaign’s duration. Funding received for a family planning RCT should cover the costs of key staff, so DMI expects it will be able to fill the remaining funding gaps. DMI will try to extend this campaign through the end of 2018." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015, Pg. 4-5 190 DMI plans to run two non-child survival campaigns in Tanzania in the coming years. We believe that DMI has prioritized opening a child survival campaign in Tanzania over a campaign in Mali or Niger due to the cost savings from already having a Tanzania country office established, though we have not confirmed this with DMI. GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 6, also unpublished section 191 Last year, DMI prioritized opening a campaign in Cameroon (see the room for more funding section of last year's review). DMI had achieved a high level of government buy-in in Cameroon, which contributed to the country's high priority. However, DMI has found that it is difficult to raise funding for a campaign in Cameroon. In addition, DMI believes that the Sahel region of Africa is growing in strategic importance, which may imply an increase in aid in the region. GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 4 192 "DMI's funding gaps for next year include: $1 million in Mozambique $500,000 in Burkina Faso $850,000 in the DRC Funding for a scale-up in Tanzania from a regional to a national campaign " GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 5 It is our understanding that DMI may be able to use about $1.5-3 million in 2016 for new campaign(s) in Mali and/or Niger even though it is not mentioned explicitly here. It also seems that DMI may be able to use about $2.25 million in Tanzania in 2016 (assuming that its $4.5 million budget for 2016-2017 in Tanzania is split evenly across the two years). However, we believe that DMI would prioritize funding its campaigns in Mozambique and Burkina Faso through the end of 2017 before allocating funding to new programs in Tanzania, Mali, and Niger; our understanding of its priorities is summarized in the table above. 193 For any campaign in a new country, DMI estimates that it may take about 9-10 months from the time that DMI has committed funding to when the first broadcasts happen (four months of recruiting project leads, five to six months to set up the project in-country, including recruiting staff, setting up an office, conducting qualitative research on obstacles to behavior change, and producing spots). Money used to supplement existing campaigns can have a quicker impact on broadcasts. GiveWell non-verbatim summary of a conversation with Roy Head and Will Snell on October 29th, 2014 (unpublished conversation) 194 “For this campaign [DRC], the delay between receiving funding and going on-air could be as short as a couple of weeks.” GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 5. 195 "DMI currently has one representative in Mozambique doing general planning and producing job descriptions for future hires" GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 6 "DMI is currently negotiating with Mozambique’s national radio network for a certain number of free spots per day. The national network is comprised of ten regional stations with additional stations in Maputo. The network broadcasts local content in local languages at certain times of day and national content from the capitol in Portuguese at other times. The national network is complicated but offers the advantage of being a single organization (as opposed to, e.g., DMI’s 29 partner stations in Burkina Faso). DMI will only partner directly with community stations in areas where there are coverage gaps in the national network. Mozambique’s national radio network has been willing to work with DMI, and Mr. Head believes it has a genuine desire to engage with its audience in a positive way." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 6 DMI also recently signed a memorandum of understanding with a major commercial television network in Mozambique. "DMI has received a letter of support from Mozambique’s [Ministry of Health], has signed a memorandum of understanding (MoU) with the country’s main commercial television network, and is currently pursuing MoUs with the state-run television and radio networks." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 6 196 The potential funding opportunities include: $2M+$850K+$500K+$240K = $3.59 million. 197DMI has privately shared its estimated probabilities of receiving funding for various campaigns from other sources it has pursued. In addition to the probabilities given here, DMI estimates it has a high chance of receiving funding for a non-child-survival campaign in Tanzania. DMI 2015-2016 financial projection (unpublished document) 198 DMI raised $2.5 million from Unorthodox Philanthropy for a campaign in Mozambique that includes child survival messaging; DMI still needs to raise $2 million to fully fund the first two years of this campaign. DMI website, "Mozambique" page, accessed October 2015 DMI has privately shared its estimated probabilities of receiving funding for various campaigns from other sources it has pursued. DMI 2015-2016 financial projection (unpublished document) 199 "DRC DMI needs $850,000 of funding for the DRC, mostly for scaled-up broadcasting and M&E. DMI will try to raise this funding in-country first. Mr. Snell believes DMI is likely to find interested funders because the amount is relatively small and the potential return-on-investment is six times greater (now about $2.30 per DALY) than during the first campaign, due to DMI's expanded reach (by broadcasting on more stations, including stations in more populous areas) and lower costs (because DMI is reusing the spots already produced for the first campaign. The timeline for raising this funding is not yet set. Ideally, DMI would like to raise it by the end of the year, but this is not an essential deadline. For this campaign, the delay between receiving funding and going on-air could be as short as a couple of weeks." GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 5. DMI has privately shared its estimated probabilities of receiving funding for various campaigns from other sources it has pursued. DMI 2015-2016 financial projection (unpublished document) 200 "DMI's funding gaps for next year include...$500,000 in Burkina Faso...DMI is likely to fill this gap because of its existing relationships and track record in the country." GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pgs. 5-6. 201 "$1.5 million: national scale-up in Burkina Faso over two years. (This is an estimate because DMI will not know the exact funding gap in Burkina Faso until it finalizes the budget for its family planning RCT.)" GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 6. In Burkina Faso, DMI's national child survival campaign is currently funded by three funders (Comic Relief, the Vitol Foundation, and Alive & Thrive), and two of these funding sources stop by mid-2016. "The campaign is supported by four funders: SPRING (Strengthening Partnerships, Results and Innovations in Nutrition Globally) provided one year of funding, ending in September 2015. Comic Relief provided funding for the longest period, 2015-2017. Alive & Thrive provided funding through August 2016. Vitol Foundation provided funding for the period January-December 2015. Different funders prioritize different issue areas (e.g. Comic Relief’s priority areas are malaria, diarrhea, and pneumonia). DMI may use unrestricted funding to fill funding gaps at the conclusion of grant periods, and hopes to maintain the full range of messaging topics throughout the campaign’s duration. Funding received for a family planning RCT should cover the costs of key staff, so DMI expects it will be able to fill the remaining funding gaps. DMI will try to extend this campaign through the end of 2018." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015, Pg. 4-5 202 Figures originally in British pounds, converted to US dollars at a £1:$1.53 rate. @DMI 2015-2016 Financial Projection@ (unpublished document) "[2016 Mulago Foundation funding] will be unrestricted and is currently unallocated." DMI email from Will Snell on October 30, 2015 (unpublished document) DMI has received $350,000 in unrestricted funding from the Mulago Foundation since 2013; DMI used some of this funding to hire a country representative to do preparatory work for a campaign in Mozambique. Mulago Foundation website, "DMI" page, accessed October 2015 DMI website, "Mozambique" page, accessed October 2015 We do not know precisely how much of this funding was allocated toward the Mozambique country representative, or what other activities were supported by unrestricted funding from the Mulago Foundation. This passage from an email from DMI leads us to believe that the main use of the Mulago funding was support for DMI's Mozambique country representative: "You also asked about whether and how we monitor unrestricted funds. We don’t yet have a process for monitoring how unrestricted funds are either earmarked or spent, because up until the start of 2015 we hadn't really received any unrestricted funds (other than Mulago funding, which was earmarked and so treated as restricted by our accounts), so we are still setting up our processes for managing it." DMI email from Will Snell on May 15, 2015 (unpublished document) 203 “DMI has a for-profit arm (titled DMI, Ltd.). This for-profit organization functions as an alternate funding channel. For some funders, it is easier to give money to a for-profit organization. For example, there are types of USAID funding that can be more easily directed to for-profit organizations. DMI is also exploring using the for-profit arm to generate commercial earned income to cover DMI core costs. The for-profit arm has not been used for earned income activities very frequently; in late 2014, an early childhood development campaign was run through the arm, on behalf of the University of the West Indies. Generating income could include creating standard DMI campaigns for governments, or producing training or educational films on contract. The income-generating activities would not necessarily have to align with DMI's social mission, as long as they do not conflict with its ethical principles. Earned-income activities would have to generate a significant profit to justify the capacity they require. Roy is interested in pursuing earned-income activities. DMI has recruited a team of six MBA students from the London Business School to do scoping work on possible earned-income activities. DMI feels that its core activities have been picking up momentum. Its first priority is to maintain this momentum.” GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on May 13, 2015 204 “Earned-income activities would have to generate a significant profit to justify the capacity they require. Roy is interested in pursuing earned-income activities. DMI has recruited a team of six MBA students from the London Business School to do scoping work on possible earned-income activities. DMI feels that its core activities have been picking up momentum. Its first priority is to maintain this momentum.” GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on May 13, 2015 205 "DMI's projected budget for the fiscal year of March 2015-2016 is $3.9 million. This does not include its Burkina Faso family planning RCT or its Tanzania child survival campaign (which have not yet been officially signed off on, but which DMI expects to do). With these projects included, DMI's budget will be $4.9 million." GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 5. The large majority of DMI’s budget is funded by restricted funding. As mentioned above, DMI only has about $1.25 million in unrestricted funding. 206 See our impact page for details on GiveWell money moved. Note that this figure does not include other funding DMI received that we may have influenced, such as the funding for DMI's upcoming family planning RCT in Burkina Faso. The funding that GiveWell moved to DMI in 2014 is a substantial portion of all the unrestricted funding that DMI has ever received. "You also asked about whether and how we monitor unrestricted funds. We don’t yet have a process for monitoring how unrestricted funds are either earmarked or spent, because up until the start of 2015 we hadn't really received any unrestricted funds (other than Mulago funding, which was earmarked and so treated as restricted by our accounts), so we are still setting up our processes for managing it." DMI email from Will Snell on May 15, 2015 (unpublished document) 207 Early in 2015, DMI told us that it planned to hold GiveWell-directed funding for a few months in order to increase the chance of raising match funding from other donors. We do not believe that DMI raised match funding in 2015, though we have not confirmed this with DMI. "DMI is planning to hold some of the GiveWell-directed money for a few months to take advantage of potential matching opportunities with other donors (e.g. a donor who offers to provide $1 million for a campaign that costs $1.5 million). Finding a matching opportunity is quite likely; there is perhaps a 50% chance of this happening." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; Jo Murray, Research Manager; and Will Snell, Director of Development on February 12, 2015, Pg. 2 208 "Using GiveWell funds, we have also been developing plans to build our own in-house research capacity to conduct times series evaluations. Starting from September, we will be recruiting a team of seven regional data collectors and a data manager based in our Ouagadougou office." DMI Burkina Faso funder report 2015 Q2, Pg. 3 (partially redacted) GiveWell non-verbatim summary of a conversation with Roy Head, CEO; Jo Murray, Research Manager; and Will Snell, Director of Development on February 12, 2015, Pg. 2 209 In 2014, DMI told us its plan for using various levels of GiveWell-directed funding. Specifically, DMI told us that: The first $600,000 it received would go toward its child survival campaign in Burkina Faso, if this gap were not filled by other funders. If it received less than $800,000 in GiveWell-directed funds and its Burkina Faso gap were filled, GiveWell-directed funds would be committed to its upcoming campaign in Mozambique. If it received more than $800,000 in GiveWell-directed funds, those funds would be used to open a new campaign in Cameroon. If it received more than $1.6 million in GiveWell-directed funds, it would open campaigns in Cameroon and another country, such as Cote d’Ivoire. DMI did not raise $600,000 for the Burkina Faso child survival campaign from other funders, and there remains a funding gap for the Burkina Faso campaign beginning in mid-2016, when two current funding sources end (discussed above). "… we have not raised [Burkina Faso child survival campaign] money from funders, so we still have a funding gap for continuing the BF child survival campaign from mid 2016 onwards; although some overheads in Burkina are covered by the family planning RCT." DMI email from Will Snell on October 30, 2015 (unpublished document) Note that DMI is no longer prioritizing a campaign in Cameroon as highly as it did last year. DMI is now placing a higher priority on countries in the Sahel region. "Cameroon was initially attractive to DMI as a program area because of a high level of government buy-in, but it has been difficult for DMI to raise funds for this program. DMI has also decided to prioritize Mali and Niger over Cameroon because DMI is aiming to build a regional presence around Burkina Faso. DMI believes it is more likely to successfully raise long-term funding for Mali and Niger because of the Sahel Region’s increasing strategic importance. The Sahel Region’s geopolitical impact, especially on European nations, has increased as instability due to poverty and population growth in the region contributes to terrorism and migration issues. For these reasons, European nations might be incentivized to increase aid to the region." GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 4 210 "The money from / due to GiveWell’s recommendation is shown as in our reserves for 2015, even though most of it is scheduled to be spent in 2016" DMI email from Will Snell on October 2, 2015 (unpublished document) 211 GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 6 212 For example, DMI is planning to open a campaign in Tanzania on early childhood development. We believe it is opening this campaign because it secured funding to do so. See DMI website, "Future Campaigns" page, accessed October 2015. DMI has told us that it operates in a competitive funding environment, and that it must operate in countries with higher levels of competition in order to secure funding (because less funding is available in countries with little competition). "However, the [development communications] sector operates in a competitive funding environment. Organizations must compete against each other for unrestricted funding. In addition, organizations often compete via tenders (a funding mechanism similar to a request-for-proposals). DMI does not often seek out tender opportunities, but it does engage in them when they align with its mission ... If DMI had entirely unrestricted funding, it would launch campaigns in areas with little or no competition. However, given that much of development funding is focused on certain countries, DMI must seek some of its work in more competitive environments." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on May 13, 2015, Pg. 3 Due to this funding dynamic, DMI may make strategic decisions aimed at maximizing its chance of securing funding. These decisions may mean opening campaigns that are not in the most cost-effective areas, or that don't focus on topics with the most evidential support. "DMI wants to continue to do both research and scale-up campaigns. DMI's strategy is to carry out research projects to figure out which interventions are effective, then to scale-up the interventions that work. ... At any given time, the allocation of resources between research and scale-up will depend on what stage projects are in, as well as on the amount of funding DMI receives for each. However, DMI's first priority is to scale up proven interventions." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on May 13, 2015, Pg. 3 213 DMI’s long term goal is to save one million lives via child survival mass media campaigns by the end of 2024. It seems that it will need to make raising funds for child survival campaigns its top priority in order to have a chance of achieving this goal. "DMI is looking to implement its Media Million Lives initiative, which aims to save one million lives through DMI’s child survival campaigns." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015, Pg. 8 214 In particular, it seems possible that DMI could use unrestricted funding to: Fill small gaps of partially-funded campaigns or research projects that were launched with restricted funding. DMI might raise some restricted funding for a non-child survival campaign or research, though not enough to fully fund the activity. For example, DMI has a $400,000 funding gap for its family planning RCT. We are unsure how DMI plans to fill this gap, though it has told us that the gap does not need to be filled until 2017. "The $400k FP RCT gap is not a priority for immediate funding since it doesn’t need to be filled until 2017." DMI email from Will Snell on October 30, 2015 (unpublished document) Fund additional research on the efficacy of mass media interventions. "DMI wants to continue to do both research and scale-up campaigns. DMI's strategy is to carry out research projects to figure out which interventions are effective, then to scale-up the interventions that work. ... At any given time, the allocation of resources between research and scale-up will depend on what stage projects are in, as well as on the amount of funding DMI receives for each. However, DMI's first priority is to scale up proven interventions." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on May 13, 2015, Pg. 3 For more on how DMI is thinking about prioritizing research and scale-up efforts, see GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on May 13, 2015, Pg. 3-5 DMI's family planning RCT begins in Burkina Faso at the end of 2015, and DMI intends to run more RCTs in Burkina Faso in the future. "Burkina Faso has a fragmented mass media landscape that makes it an excellent setting for mass media RCTs; DMI wants to continue to take advantage of this landscape to generate RCTs studying the effect of media mass on various behaviors." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on May 13, 2015, Pg. 3 "Family planning RCT (2015-2019) ... We are planning to run a baseline survey in late 2015 and to begin production and broadcasting in early 2016." DMI website, "Burkina Faso" page, accessed September 2015 "Early childhood development campaign and RCT – DMI would like to focus its next campaign on early childhood development. The main objective would be to encourage mothers to mentally stimulate their children by speaking with them. ... DMI wants to move forward with both the early childhood development campaign and RCT. It must reflect on whether funds would be better spent on this RCT, or on scaling up other campaigns." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015, Pg. 4 Maintain reserves or fund the costs of its central office in London. We are somewhat uncertain about how DMI’s central office in London is funded and the size of its budget, so we do not know how raising funding for the central office compares in priority to raising funding for child survival campaigns, or how else central office budgeting might interact with the financial planning of upcoming campaigns. From what we understand, DMI's central office in London is funded in part by a "management fee" attached to some of its funder contracts. In an expenditure file DMI shared with us, some funder contracts also included line items for expenditures associated with the central office, separate from this management fee. @DMI 2015-2016 Financial Projection@ (unpublished document) @DMI 2015-2016 Fiscal Year Financials@ (unpublished document) We do not know if these two funding sources are the primary means of funding DMI's central office, and we do not know the total budget for the central office. DMI has expressed interest in a variety of activities that differ from the structure of the child survival campaign that was evaluated by the RCT in Burkina Faso, such as funding television campaigns in Burkina Faso, or purchasing solar panels for its partner stations in lieu of making airtime payments. For a possible television campaign in Burkina Faso, see GiveWell non-verbatim summary of a conversation with Roy Head, CEO; Jo Murray, Research Manager; and Will Snell, Director of Development on February 12, 2015, Pg. 3 "DMI is also considering providing solar panel arrays to partner stations. This would give the stations access to reliable electricity, and would help DMI build its relationships with its partners. Partner stations are not able to afford the upfront costs of solar panels, but the approximately $500,000 project is more affordable for DMI (relatively)." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015, Pg. 7 "DMI hopes to provide solar panels to partner stations as a form of compensation in the future, but does not yet have the budget." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 3 215 "Media Million Lives is a new initiative from Development Media International. It is raising funds to create integrated radio, TV and mobile phone campaigns to promote key maternal and child health behaviours in ten African countries, with the objective of saving a million lives." Media Million Lives website, accessed September 2015 "DMI is looking to implement its Media Million Lives initiative, which aims to save one million lives through DMI’s child survival campaigns." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015, Pg. 8 Here is a schema of DMI's Media Million Lives (MML) scale-up proposal: DMI’s programs would reach national scale in Burkina Faso and DRC in 2016, and in Mozambique and Cameroon in 2017 DMI would initiate campaigns in Chad, Ethiopia, Mali, and Niger in 2017 (reaching national scale in 2018) DMI would initiate campaigns in Cote d’Ivoire and Tanzania in 2018 (reaching national scale in 2019) DMI would initiate a campaign in some parts of Nigeria in 2019 (reaching national scale in 2020) GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015, Pg. 8 Note that DMI's MML proposal prioritization is different than the prioritization given above (e.g. in the MML proposal below, a Cameroon campaign reaches national scale in 2017, whereas Tanzania does not reach national scale until 2019). DMI has not solidified its priorities regarding which countries to enter next, and we believe that it will open campaigns where it is able to raise funding for campaigns. The categorization given above is more recent. 216 "The proposed budget for Media Million Lives is approximately $185 million, or around $24 million per year. This is a significant increase from DMI’s current $5 million per year budget, which encompasses all of its current campaigns (including campaigns not focused on child survival). DMI must also consider the fact that some of the funding it receives will be for campaigns that aren’t focused on child survival. DMI expects that approximately 50-65% of the funding it receives will be for child survival campaigns." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015, Pgs. 8-9 217 "Securing funding is the main obstacle towards scaling up DMI’s operations. Hiring high-quality staff is also difficult, but Mr. Head is confident that DMI will be able to find good people for future campaigns." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015, Pg. 9 218 For Tanzania country director: GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on June 16, 2015, unpublished section For DRC and Burkina Faso country directors, as well as Burkina Faso staff: DMI website, staff page, accessed October 2015 For DRC staff: DMI DRC funder report 2015 Q1, Pg. 2 Details on Mozambique: "DMI currently has one representative in Mozambique doing general planning and producing job descriptions for future hires" GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 6 "DMI is currently negotiating with Mozambique’s national radio network for a certain number of free spots per day. The national network is comprised of ten regional stations with additional stations in Maputo. The network broadcasts local content in local languages at certain times of day and national content from the capitol in Portuguese at other times. The national network is complicated but offers the advantage of being a single organization (as opposed to, e.g., DMI’s 29 partner stations in Burkina Faso). DMI will only partner directly with community stations in areas where there are coverage gaps in the national network. Mozambique’s national radio network has been willing to work with DMI, and Mr. Head believes it has a genuine desire to engage with its audience in a positive way." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 6 DMI also recently signed a memorandum of understanding with a major commercial television network in Mozambique. "DMI has received a letter of support from Mozambique’s [Ministry of Health], has signed a memorandum of understanding (MoU) with the country’s main commercial television network, and is currently pursuing MoUs with the state-run television and radio networks." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 6 219 Given DMI's funding priorities, it seems that it would spend its next roughly $9 million in countries in which it already has country directors (see above): The first $2 million of unrestricted funding DMI receives would go to Mozambique, where a DMI country director is established, though DMI staff would have to be hired and trained. The next approximately $2.35 million would go to the DRC and Burkina Faso, where DMI staff and country directors are already in place. The following $4.5 million would go to Tanzania, where a country director is in place, though staff would have to be hired and trained. Funding past the $8.85 million delineated in the above bullets would go towards countries where DMI employs neither a country director nor staff. 220 DMI suggests the following sources for evidence of the life-saving effects of these treatments: UNICEF Facts for Life 2010 Lancet Child Survival Series 2003 WHO RMNCH guidelines: Essential Interventions, Commodities and Guidelines for Reproductive, Maternal, Newborn and Child Development and use of the Lives Saved Tool (LiST): A model to estimate the impact of scaling up proven interventions on maternal, neonatal and child mortality (2010); International Journal of Epidemiology; 39, Supplement 1 Technical inputs, enhancements and applications of the LiST (2011); BMC Public Health, 11, Supplement 3 Lassi et al (2014): Essential interventions for maternal, newborn and child health: background and methodology; Reproductive Health, 11, Supplement 1. DMI email to GiveWell October 3rd, 2014 (unpublished document) 221 DMI suggests Naugle and Hornik 2014, a systematic review of the evidence for mass media child survival interventions in developing contexts. We have not looked at this review closely due to our constrained capacity. 222 "DMI has partnered with SPRING (Strengthening Partnerships, Results, and Innovations in Nutrition Globally) to run a maternal, infant, and young child nutrition campaign in seven clusters concentrated around the Sahel Region." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 2 223 "DMI will not use long-format modules for its nationwide child survival campaign, mainly because they are expensive to produce, particularly in fragmented markets." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 3 DMI has not entirely abandoned the long module format, and plans to use it in some upcoming non-child survival campaigns. "DMI does not plan to eliminate long-format modules entirely (e.g., DMI’s eight-region family planning RCT in Burkina Faso will use long-formats, and DMI is using long-formats for its nutrition campaign at the request of the campaign’s funders)." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 3 224 See our cost-effectiveness model for details on how we model these factors, though note that this model does not reflect our current understanding of DMI’s cost-effectiveness given the preliminary endline RCT results. 225 We believe DMI is using metrics from DHS and MICS to try to answer one part of this question. In particular, those two surveys attempt to measure, among families with children who suffered from specific symptoms in the two weeks before the survey (such as diarrhea or those associated with acute respiratory infections or with malaria), (a) how many families reported seeking professional treatment for that child, and (b) how many families reported receiving treatment for that child. (For example, "Pourcentage d’enfants de moins de cinq ans ayant présenté des symptômes d’Infection Respiratoire Aiguë au cours des deux semaines ayant précédé l’enquête et, parmi ces enfants, pourcentage pour lesquels on a recherché un traitement auprès d’un établissement ou d’un prestataire de santé et pourcentage à qui on a administré des antibiotiques comme traitement" DHS Survey DRC 2008, Pg. 128. Google Translate suggests this translation: "Percentage of children under five years who had symptoms of acute respiratory infection in the two weeks preceding the survey, and of these children, the percentage for which we have sought treatment to an institution or a health care provider and percentage who were given antibiotics as treatment" Our understanding is that DMI uses the number of families who report receiving treatment divided by the number who report seeking treatment as a proxy for "the portion of the time that treatment for diarrhea is available", since that is the portion of the time which it was received, of the total times it was sought. GiveWell non-verbatim summary of a conversation with Will Snell and Joanna Murray on August 15th, 2014, Pg. 4 We see two possible problems with using this approximation: It assumes that people who are not seeking treatment would seek treatment if they were convinced they should, whereas some might not be able to seek treatment at all. For example, if no one in the family has the time and ability to seek treatment, then being convinced to seek treatment may not help. It assumes that all received treatment is caused by seeking it, whereas at least some appears to be received without being sought. In the summary results from the most recent DHS survey in DRC, for example, more people receive treatment for several diseases than seek it in several subgroups, so there must be ways to receive treatment without seeking it. (For example, families of boys with diarrhea sought treatment 37.7% of the time, and used treatment 38.9% of the time. DHS Survey DRC preliminary results 2013, Pg. 23) We are not sure how to account for treatment that would be received without being sought. 226 This might be a particular concern in the DRC: "The political allegiance of most media outlets in DRC normally reflects that of their owner. Even media outlets which attempt to remain politically neutral often run stories which have quite clearly been sponsored by an interested party. The fact that they agree to do so reflects the chronic shortage of cash in most Congolese media organisations. Certain newspapers do run critical pieces and some TV and radio stations have openly criticised government actions or the behaviour of individual ministers. However, independent media organisations increasingly practice self-censorship. Sometimes they couch criticism of the government in obtuse editorials. Another trick is to balance criticism with laudatory coverage of other aspects of government action. The government does not react to every critical report in the local media, but crackdowns are very frequent. Critical journalists and those working for opposition media outlets are often harassed, intimidated and arrested. Several journalists have have been killed in mysterious circumstances after publishing reports critical of the government. Their murderers are almost never tried and convicted. TV and Radio stations which incur the government’s wrath are often forced off air for a period. The DRC ranked 145th out of the 179 countries listed in the Reporters Sans Frontieres www.rsf.org 2011-2012 World Press Freedom Index." Infosaid Media Landscape 2012, Pg. 20 227 For more on recent unrest in Burkina Faso, see 2014 Burkinabé Uprising Wikipedia page, 2015 Burkina Coup D'Etat Wikipedia page, and Blaise Compaoré Wikipedia page 228 DMI Burkina Faso funder report 2014 Q4, Pg. 2 (partially redacted) 229 "Of more concern to DMI is the security situation in rural Burkina. Over the first six months of this year there seems to have been an increase in rural insecurity, with an increase in roadside banditry, more local political disputes erupting into armed conflict, and mob justice deciding more and more village conflicts. The area of most concern to DMI is Eastern Burkina. With the increase in artisanal gold mines in the region there has been an increase in armed robbery on rural roads." DMI Burkina Faso funder report 2015 Q2, Pg. 3 (partially redacted) 230 Infrastructure appears to be particularly problematic in the DRC, which may affect both the likelihood of issues, and the difficulty of fixing them. For example, "There is no national network of all weather roads to unite this sprawling expanse of tropical rain forest and savannah grasslands." Infosaid Media Landscape 2012, Pg. 3 "Staff mobility [in the DRC] is much more limited, as DMI does not allow staff to take unsafe flights, roads are often bad or nonexistent, and the DRC is about ten times larger than Burkina Faso." GiveWell non-verbatim summary of a conversation with Roy Head, CEO; and Will Snell, Director of Development on September 8, 2015, Pg. 4 231 "DMI does not plan to do further child mortality RCTs after the Burkina Faso trial because they would be expensive and would mean that half of the population would not receive the intervention. Additionally, other countries would be less suited than Burkina Faso for a cluster RCT testing a mass media campaign because other countries have less localized media, so it is more likely that the control group would receive the messages broadcast to the intervention group. For future child mortality programs, DMI plans to use a quasi-experimental design to measure a wide range of outcomes. This could involve administering a series of surveys to do a time series analysis, creating non-randomized controls, or using propensity score matching. DMI may compare the effectiveness of various messages within a program." GiveWell non-verbatim summary of a conversation with DMI on April 24th, 2014, Pg. 6 "DMI is developing internal capacity to collect time series data on a monthly basis. This will include data on maternal and neonatal child health through surveying knowledge and reported behavior, as well as some observed behaviors where possible (e.g., the availability of clean water). An important benefit of this system will be DMI’s ability to use this information to improve its campaign design and implementation on an ongoing basis. The time series system will employ seven or eight regional, locally-recruited data collectors equipped with a smartphone and a motorcycle to continuously collect data and submit it to a data manager based in Ouagadougou each month." GiveWell non-verbatim summary of a conversation with Joanna Murray, Research Manager; and Will Snell, Director of Development on September 15, 2015, Pg. 3 For more detail on DMI's regional evaluation research proposal, see GiveWell non-verbatim summary of a conversation with Roy Head, CEO; Jo Murray, Research Manager; and Will Snell, Director of Development on February 12, 2015, Pg. 2-3