Against Malaria Foundation – November 2016 version

Donate

Published: November 2016

Summary

What do they do? AMF (againstmalaria.org) provides funding for long-lasting insecticide-treated net (LLIN) distributions (for protection against malaria) in developing countries.

Does it work? There is strong evidence that distributing nets reduces child mortality and malaria cases. AMF has relatively strong reporting requirements for its distribution partners and provides a level of public disclosure and tracking of distributions that we have not seen from any other net distribution charity. AMF's post-distribution surveys have generally found positive results, but have some methodological limitations.

What do you get for your dollar? We estimate that the cost to purchase and distribute an AMF-funded net is $4.35 in Malawi, $5.92 in the Democratic Republic of the Congo (DRC), and $5.14 in Ghana (the three countries that AMF has completed large-distributions in). We also very roughly estimate, based on past and planned distributions, that the cost per net in distributions AMF may fund with additional donations is $4.85. The numbers of malaria cases prevented and lives saved are a function of a number of difficult to estimate factors, which we discuss in detail below.

Is there room for more funding? We estimate that AMF could productively use or commit a maximum of between $78 million (50% confidence) and $191 million (5% confidence) in additional unrestricted funding in its next budget year. We are most confident in the value of the funds for the first $11 million it receives, but we expect funds to continue to be valuable, though possibly somewhat less cost-effective, above that level.

AMF is recommended because of its:

• Focus on a program with excellent evidence of effectiveness and cost-effectiveness.
• Processes for ensuring that nets reach their intended recipients and monitoring whether they are used over the long-term.
• Room for more funding – we believe AMF will be able to use additional funds to deliver additional nets.
• Transparency – AMF shares significant information about its work with us and we are able to closely follow and understand its work.

Major open questions:

• We have seen detailed data from before and during distributions. AMF also seeks to collect follow-up data after distributions, but its track record of collecting the data it seeks from distributions is limited outside of Malawi. In particular, we have seen only two follow-up surveys from distributions outside of Malawi, and these surveys are of lower quality than previous surveys from Malawi. The surveys we have seen from Malawi also have methodological limitations.
• The best evidence for nets was collected before they were widely used and there is some evidence that mosquitoes have since adapted to the insecticide used in nets, possibly making them less effective. It seems that insecticide resistance is a growing concern, but it remains difficult to quantify the impact of resistance. There are several studies in progress that may help shed light on this within the next year. We discuss this issue in more detail in our page on this topic.
• We are considerably uncertain about the size of AMF's room for more funding. There are several major funders of nets, which provide most of the funding needed to distribute nets to all at risk populations. The Global Fund to fight AIDS, Tuberculosis, and Malaria is by far the largest and has not yet determined how much it will allocate to nets over the next few years. We believe there is a significant remaining gap (one rough estimate was $125 million per year) but do not have high confidence in the size of the gap.

Our review process

We began reviewing AMF in 2009. Our review process has consisted of:

• Reviewing AMF's public records for each of its net distributions and other documents AMF has shared with us.
• Extensive communication, including several meetings at AMF's London headquarters, with AMF Founder Rob Mather to discuss AMF's methods and funding needs.
• A visit to AMF's distribution partner organization, Concern Universal, in Malawi in October 2011 (notes and photos from this visit). We spoke with Concern Universal by phone in April 2016.
• Reviewing materials from AMF’s distributions in Malawi, the Democratic Republic of the Congo (DRC), and Ghana, including pre-distribution registration data, distribution reports, and data from post-distribution follow-up surveys.
• Reviewing distribution agreements AMF has signed with National Malaria Control Programs and NGO partners for future distributions.
• Conversations with Peter Sherratt, AMF's Executive Chairman; Don de Savigny, a member of AMF's Malaria Advisory Group; and other individuals (who requested to remain anonymous) familiar with AMF's work and its attempts to finalize distributions.
• Conversations with Melanie Renshaw of the African Leaders Malaria Alliance, Marcy Erskine of the International Federation of the Red Cross, and Scott Filler of the Global Fund to Fight AIDS, Tuberculosis, and Malaria about funding needs for nets.
• A visit to Greater Accra, Ghana in August 2016 to meet with representatives of AMF, AMF's distribution partners Episcopal Relief & Development and Anglican Diocesan Development and Relief Organization (ADDRO), Ghana's National Malaria Control Program, and other non-profit and government organizations involved in the AMF-funded LLIN distributions in Ghana in 2016. Notes and photos from our site visit are available here.

All content on AMF, including past reviews, updates, blog posts and conversation notes, is available here.

A few key terms for this review

• A net refers to any bed net intended to block mosquitoes, whether treated with insecticide or not.
• An ITN is an insecticide-treated net. As discussed at our write-up on ITNs, distribution of ITNs has been rigorously studied and associated with declines in under-5 mortality and overall malaria cases.
• A LLIN is a long-lasting insecticide-treated net, intended to serve as an effective ITN for several years (details at our write-up on ITNs). The nets distributed by AMF are LLINs,1 and (as noted in our write-up on large-scale ITN distributions) most large-scale donor-financed malaria control today utilizes LLINs.
• A PBO LLIN is a newer type of net incorporating piperonyl butoxide (PBO) alongside pyrethroid insecticide used in other LLINs. These nets may be more effective than other LLINs in areas where mosquitoes have developed insecticide resistance.

What do they do?

AMF provides long-lasting insecticide-treated nets (for protection against malaria) in bulk to other non-profit organizations or government agencies, which then distribute the nets in developing countries.

As of November 2016, AMF has completed eleven large-scale distributions in three countries (Malawi, DRC, and Ghana) for a total of 5.79 million LLINs distributed.2

AMF has signed agreements to fund distributions of 10.7 million LLINs in Uganda, 1.2 million LLINs in Papua New Guinea, and 2.4 million LLINs in Togo in 2017; and 1.5 million LLINs in Malawi in 2018.3 AMF also expects to fund an additional distribution of 1.6 million LLINs in Papua New Guinea in 2018, but has not yet signed a distribution agreement.4 AMF's agreements with distribution partners include requirements that distribution partners collect and share data from the household registration process and follow-up surveys, as well as other accountability and transparency requirements.

A summary of AMF's distributions can be found in this spreadsheet.

The role of AMF and its partners in LLIN distributions

AMF's role in LLIN distributions is to:5

1. Identify countries with funding gaps for LLINs.
2. Find distribution partners (in-country non-profit organizations or government agencies) to carry out LLIN distributions. AMF and its partners agree on expectations for the distribution, including who pays for costs other than the purchase price of LLINs (which are always covered by AMF), the process that will be used to carry out the distribution, and what information will be collected and shared with AMF.
3. Purchase LLINs and have them shipped to the distribution partners.
4. Work with distribution partners to collect reports on the distribution and follow-up surveys. AMF posts these reports on its website.

Distribution partners implement on-the-ground activities, including registering residents in targeted areas, distributing LLINs, monitoring the registration and distribution processes, and conducting follow-up surveys.6

Details follow.

Selecting locations for distributions and finding distribution partners

When selecting locations for future distributions, AMF told us it consults a series of sources, as it believes there is no single reliable resource with up-to-date information to determine where there are funding gaps for LLINs. Sources it consults include the Alliance for Malaria Prevention's (AMP's) list of countries with significant net gaps, other malaria control funders, in-country technical advisors, the relevant National Malaria Control Program (NMCP), implementing organizations, and the African Leaders Malaria Alliance.7 AMF told us that it has been receiving more funding requests since it started funding larger distributions,8 and notes that its largest commitment so far—10.6 million LLINs in Uganda in 2017—was made in response to an in-bound request.9

As AMF investigates countries with existing net gaps, it also looks into organizations working within those countries that could serve as distribution partners.10 AMF looks for distribution partners that have the capacity and willingness to implement registration, distribution, and monitoring processes that meet agreed-upon requirements.11 So far, AMF has worked with Concern Universal in Malawi, IMA World Health in the Democratic Republic of the Congo, and Episcopal Relief & Development and Ghana's National Malaria Control Program (NMCP) in Ghana on large-scale LLIN distributions.12

Registration and distribution

• Registration: During the registration process, AMF's distribution partners' staff or volunteers travel door-to-door in targeted areas to collect the information used to determine the number of LLINs to allocate to each household (e.g., the number of sleeping spaces, the number of previously-owned LLINs in usable condition, and/or the number of household members), as well as the information used to identify the household for the distribution and post-distribution surveys (e.g., the name of the head of the household, and/or household location). AMF has shared full or sample registration data from each completed large-scale distribution with us (with the exception of the recent distribution in the Upper West Region, Ghana in 2016, for which we have only seen a summary of registration data so far; full data is expected later). The specifics of the registration process and LLIN allocation strategy have differed by country (process details and registration data sources in footnote).13
• Distribution: To distribute LLINs to recipients, AMF and its distribution partners have primarily used "point distribution" (LLIN recipients pick up their nets from a specified point in or near their community), but have also used "hang-up distribution" (staff or volunteers travel door-to-door to deliver and hang up LLINs) in one distribution in Kasaï-Occidental, DRC.14 Distribution partners also manage the logistics of in-country shipping and storage of LLINs prior to the distribution. The specifics of distribution processes have varied in the different countries AMF has worked in (details in footnote).15

Monitoring

AMF's distribution partners also implement a set of monitoring activities to produce evidence on whether the registration and distribution processes operated as intended and on the long-term impact of the LLIN distribution. AMF and its distribution partners agree on expectations for monitoring activities for each distribution agreement, so somewhat different monitoring activities have been implemented for AMF's different distributions.

• Process monitoring (i.e., the activities used to assess whether the registration and distribution processes operated as intended):
  • Data validation: AMF's distribution partners in Malawi, DRC, and Ghana have used different processes to verify the accuracy of data collected during household registration and distribution of LLINs. In Malawi, Concern Universal sends registration lists to community leaders; these leaders read the lists out loud at a community meeting, supervised by Concern Universal, so community members can request edits.16 In Kasaï-Occidental, DRC, AMF told us that IMA World Health employed supervisors who checked data for obvious errors and compared electronic and paper records. AMF has also checked the registration and distribution data for missing or inconsistent household-level records.17 We are uncertain about what data validation processes may have been used in Nord Ubangi, DRC, since we have not yet seen a report from that distribution. In Ghana, several different processes were used to validate data collected during registration and distribution: "pre-distribution data validation" (comparing the number of LLINs allocated to each community based on registration data to what would be expected based on the most recent census data, and double-checking and correcting—by phone or a follow-up household visit—the source of implausible registration data, e.g., households with more than 20 members), "post-distribution data validation" (checking whether data collected during the distribution indicates that households received appropriate numbers of LLINs), and, to check the accuracy of the electronic input of registration and distribution data, "6% data entry validation" (independent re-entering of 6% of the data collected during registration and distribution and checking it against the original electronic records). Details in footnote.18
  • "Embedded" monitoring: In the 2016 LLIN distributions in Ghana, AMF's non-profit distribution partners, Episcopal Relief & Development and the Anglican Diocesan Development and Relief Organization (ADDRO), monitored the implementation of the registration and distribution processes by AMF's government distribution partner, Ghana Health Services. Episcopal Relief & Development and ADDRO were "embedded" in the registration and distribution processes; they attended district-level planning meetings to ensure that they were operating as intended, observed the registration of households by volunteers organized through Ghana Health Services, and observed selected distribution points (details in footnote).19 There was no "embedded" monitoring in AMF's distributions in Malawi and DRC since AMF worked with a single non-profit partner for those distributions.
  • Writing distribution reports: Distribution reports provide narrative summaries of activities implemented and challenges encountered by distribution partners. Concern Universal has provided distribution reports for four of the six large-scale distributions it has completed in Malawi.20 We have not yet seen distribution reports for the Ntcheu 2015 or Balaka 2015 distributions in Malawi. IMA World Health has provided a distribution report for the Kasaï-Occidental, DRC 2014 distribution.21 We have not yet seen a distribution report from the Nord Ubangi, DRC distribution, which was completed in May 2016. From the three distributions in Ghana in 2016, we have seen a full distribution report from Episcopal Relief & Development from the Northern Region distribution, and short-term "Activity Reports" covering some periods of the pre-distribution and distribution activities in Greater Accra and the Upper West Region.22 It is our understanding that full distribution reports will be available for the Greater Accra and Upper West distributions later.
  • Post-distribution validation tracing: After distributions were complete, AMF's distribution partners in Ghana checked that a random sample of households (100 households per district) had actually received the number of LLINs they were allocated by calling or visiting the households.23 Episcopal Relief & Development noted that, through this process, it discovered that a significant number of residents of two districts in Greater Accra, Ghana had not picked up the LLINs they had been allocated; we have not seen results beyond that.24 It is our understanding that similar processes have not been used in AMF's distributions in Malawi and DRC.
• Impact monitoring (i.e., the activities used to assess the long-term impact of the distribution):
  • Post-distribution check ups (PDCUs): AMF has required all distribution partners to conduct follow-up surveys (called post-distribution check-ups, or PDCUs) in 5% of households at 6-month intervals for 2.5 years after a distribution. PDCUs estimate the presence, usage, and condition of LLINs from the AMF-funded distribution (details in footnote).25 We summarize which PDCUs have been completed and whether they were completed on time in this spreadsheet (see "PDCUs" sheet), and summarize the results of PDCUs we have seen in this spreadsheet.

    In 2016, AMF published data and reports from its first two PDCUs from Kasaï-Occidental, DRC, its first PDCUs completed outside of Malawi.26 We believe these PDCUs were poorly implemented and do not provide an accurate estimate of the proportion of AMF's LLINs that reached their intended destinations or the impact of AMF's distribution on LLIN usage over time in Kasaï-Occidental (see this blog post for details).

  • Malaria case rate data: Previously, AMF expected to collect data on malaria case rates from the regions in which it funded LLIN distributions: in the distribution agreements for the Ntcheu 2011-2012, Balaka 2013, and Dedza 2014 distributions, AMF asked Concern Universal to provide monthly malaria case rate data from before and after the distributions from all health centers in the three districts.27 In 2016, AMF shared malaria case rate data from Malawi covering January 2010 through March 2015 for Balaka, Ntcheu, and Dedza, and for Dowa from July 2012 to March 2015, but we have not prioritized analyzing it closely.28 AMF believes that this data is not high quality enough to reliably indicate actual trends in malaria case rates, so we do not believe that the fact that AMF collects malaria case rate data is a consideration in AMF’s favor, and do not plan to continue to track AMF's progress in collecting malaria case rate data.29

In previous versions of this review, we noted that AMF used "105%" registration data collection in some distributions. For distributions with "105%" registration data collection, after staff or volunteers have collected registration data from households, a different group of staff or volunteers revisit 5% of households and independently collect registration data.30 We do not believe this process, as it has been carried out, has been a useful form of monitoring. AMF and its partner in Malawi, Concern Universal, have not matched the two sets of registration data in time to correct potential errors in the distribution process (e.g., distributing the incorrect number of LLINs to a particular household). We have seen the 5% registration check data from one distribution in Malawi which AMF partially matched to the original data several months after the distribution was completed,31 and AMF has told us that 105% registration data was collected for other Malawi distributions but that it has not prioritized analyzing it.32 AMF has told us it sees the primary purpose of 105% data collection as being a tool to motivate data collectors, and that it does not need to check the data to achieve that goal.33

In previous versions of this review, we also discussed AMF's use of photos and videos as evidence that nets had reached their intended destinations. Because AMF has told us that it does not now consider photo and video footage part of its monitoring and evaluation activities, and because photos are missing for many recent distributions and there are fairly small numbers of photos available for other distributions, we no are longer tracking whether AMF is collecting photos and videos from distributions.

Other activities

AMF occasionally supports malaria control activities beyond the direct distribution of LLINs. For example:

• AMF plans to fund research on insecticide resistance that will be carried out in conjunction with AMF distributions. AMF plans to fund research on the effectiveness of PBO LLINs in conjunction with its Uganda 2017 distribution. The budget for this research is not finalized; the work is currently projected to cost $2.7 million.34 AMF also funded the first phase of a study (which cost around $100,000) on insecticide resistance in Nord Ubangi, DRC; in November 2016, AMF told us that it no longer planned to fund the completion of this study (which would have cost around an additional $700,000) because it was funding the PBO study in Uganda.35

• AMF is in the process of creating and co-funding a "Malaria Control Unit" (MCU) in Malawi with Concern Universal.36 The MCU will consist of up to 14 permanent staff members who will work on a variety of malaria control projects: conducting post-distribution surveys, improving malaria case rate data collection practices, monitoring the levels of malaria prevention and treatment supplies at local health centers, developing efficient methods to keep net coverage rates high in between mass distribution campaigns, and more.37 The MCU is also intended to assist with AMF and Concern Universal's distributions in Malawi.38 AMF has committed approximately $636,000 over three years to this project.39
• AMF has encouraged Concern Universal staff to attend national malaria control strategy meetings in Malawi to share AMF's processes and results in other districts. As a result, AMF has told us that some of its monitoring practices were being adopted for a large national distribution (approximately 7 million nets) in 18 of Malawi's 28 districts in March to May 2016.40 The practices being implemented include (a) using sleeping spaces data (as opposed to population data) to calculate the number of nets needed, (b) "105%" registration data collection (described above), and (c) putting summary data (at the village or health center level) into electronic form.41 AMF told us in February 2015 that it expects to be able to see the data collected during this distribution.42 We have not yet followed up with AMF on whether this happened as expected.

Spending breakdown

The following table shows AMF's total expenditure, categorized into purchases of bed nets, spending on running the organization, and spending on other non-net costs (such as providing funding to other organizations to conduct post-distribution check-ups, which are described above). We include spending since FY 2012 (July 2011 to June 2012) because this is when AMF shifted to its current model of larger-scale distributions.

Prior to the Balaka 2013 distribution, AMF asked all distribution partners to use their own funds or to find another funder for all non-net costs of the distribution.44 More recently, it has paid for some of these costs in certain distributions. AMF told us that it considers funding non-net costs in cases where (a) non-net costs are not covered by other partners, and (b) AMF feels confident that its distribution partners will manage and report on spending well (details in footnote).45

It is our understanding that AMF aims to primarily fund distributions where other partners cover non-net costs, which could limit its capacity to distribute nets, though AMF has shown flexibility on this issue.46

Does it work?

On a separate page, we discuss the general evidence behind distributions of LLINs. We conclude that there is strong evidence that these distributions can be expected to reduce child mortality and malaria cases.

When evaluating the effectiveness of an LLIN distribution organization, we seek to answer the following questions:

• Are LLINs targeted at people who do not already have them? When determining how many LLINs to allocate to each household, AMF's distribution partners in Malawi and DRC checked households for previously-owned LLINs in good condition, but AMF's partners in Ghana did not. In Ghana, we don't know what the LLIN coverage rate was before the distribution, though we note that the previous mass LLIN distribution was four years prior and most nets wear out in less than four years.
• Do the LLINs reach the intended destinations? The main evidence that LLINs reached their intended destinations at a high rate are (a) post-distribution surveys, where surveyors visit a sample of households that registered for nets six months after the distribution, and (b) data from registering households and, for some distributions, which households are reported to have received their nets. We have seen (a) and/or (b) from all of AMF's distributions; we discuss some limitations to the post-distribution surveys below.
• Are LLINs targeted at areas with high rates of malaria? AMF seeks out distribution partners in countries that are known to have high rates of malaria, or where malaria rates are likely to increase significantly if LLIN distribution programs are not sustained. We note that Papua New Guinea (where AMF plans to fund LLIN distributions in 2017 and 2018) has lower (but still significant) rates of malaria than other countries where AMF has worked or plans to work.
• Do those who receive the LLINs install them in their homes properly? Do they utilize them consistently over the long term? AMF requires partners to conduct follow-up surveys at 6-month intervals for a period of 2.5 years. We discuss some methodological limitations to the surveys from Malawi below. We have little data to compare these results to, but in most cases they seem consistent with high, proper usage of nets for an extended period after distribution.47 We believe that the two surveys from Kasaï-Occidental, DRC were poorly implemented, and do not provide a reliable estimate of long-term LLIN usage;48 we have not yet seen (or expected to see) any results from post-distribution surveys in Ghana.
• Do AMF's LLINs increase the total number of LLINs distributed, or would the recipients have received LLINs from other sources if not for AMF? The evidence we have seen suggests that donations to AMF increase the total number of LLINs distributed, but that a portion of the impact is offset by displacing funding from other sources.

Details follow.

Are LLINs targeted at people who do not already have them?

Malawi and the Democratic Republic of the Congo

We believe that the LLIN allocation strategy used in Malawi and DRC is reasonably likely to avoid supplying more LLINs than needed to individual households.

For AMF's past large-scale distributions in Malawi, a household's allocation of LLINs was determined by subtracting the number of LLINs in usable condition from the total number of sleeping spaces in the household.49 AMF's distribution partner in Malawi, Concern Universal, told us that during registration health workers enter each household, ask about who is living in the household, and visually inspect sleeping spaces and existing nets.50 Concern Universal also has conducted village meetings in which its staff read off household names and the number of LLINs that have been allocated to each household; households then indicate if any errors have been made.51 We observed one such village verification meeting during our visit to Malawi in October 2011.52 See footnote for more detail on the registration process used in Malawi.53

We have seen registration data (which includes records of numbers of sleeping spaces and usable LLINs per household) from all six of AMF's completed large-scale distributions in Malawi: full data from its Ntcheu 2012, Balaka 2015, and Ntcheu 2015 distributions, and sample results from its Balaka 2013, Dedza 2014, and Dowa 2015 distributions.54

For IMA's Kasaï-Occidental, DRC distribution, we discuss what we know about the process from reports IMA has shared here. The process appears to be largely similar to Concern Universal's process for going house-to-house to determine net need and allocating LLINs by subtracting a household's usable LLINs from its number of sleeping spaces, though it did not include village verification meetings (in part because nets were distributed during the same visit in which net need was determined). We have seen full distribution data from Kasaï-Occidental, which includes data on numbers of usable LLINs and sleeping spaces per household.55 Due to logistical issues encountered with registering households and distributing nets at the same visit in the Kasaï-Occidental distribution,56 IMA and AMF implemented separate registration and distribution processes for the distribution in Nord Ubangi, DRC.57 We have seen a sample of household-level registration and distribution data from Nord Ubangi (as well as summaries of data for the entire distribution), which includes numbers of sleeping spaces per household, numbers of previously-owned LLINs, and numbers of LLINs distributed.58 The summary registration data indicates that ownership of LLINs in good condition before AMF's distribution was low, with around 18,000 LLINs in good condition owned for a population of around 1.3 million.59 We have not yet seen a distribution report from Nord Ubangi, so we do not have full details on how this data was collected.

It is possible that some households may hide previously-owned LLINs during registration; cases of households hiding nets in order to receive extra LLINs have been observed in campaigns in Senegal and Nigeria.60 We have not seen any evidence that this has occurred in AMF's distributions.

Ghana

We believe that the LLIN allocation strategy used in AMF's distributions in Ghana in 2016 may have oversupplied some households with LLINs.

Prior to these distributions, volunteers organized by the government collected information on the number of people in each household in targeted areas.61 Registered households were allocated one net per two people (rounding up for households with an odd number of people). Previously-owned nets in usable condition were not subtracted from households' LLIN allocations.62 It seems likely to us that some households in areas targeted by these distributions already owned some LLINs in usable condition, which may have been obtained through "continuous distributions" or by purchase, so these households may have been allocated more LLINs than necessary in AMF's distribution.63 However, since the most recent mass distribution of LLINs in the Northern, Greater Accra, and Upper West Regions was in 2012 (four years before AMF's distributions), we would guess that a large proportion of targeted households were in need of new LLINs.64

We have seen complete registration data from the Northern Region and Greater Accra distributions (which includes information on the number of individuals in each household and the number of LLINs allocated to each household). AMF and the government decided not to collect information on previously owned LLINs for this distribution because they did not expect to find many.65

Future distributions

AMF has told us its distribution partners will not subtract out previously-owned usable LLINs from households' LLIN allocations during its 2017 distributions in Togo and Uganda.66 AMF told us that this decision was made because, in regions where the previous mass distribution was more than three years earlier, it is likely that the number of previously-owned nets in good condition will be low.67 AMF also told us that it has not yet determined whether usable LLINs will be subtracted from households' allocations during distributions in Malawi in 2018.68 We are uncertain what type of LLIN allocation strategy will be used in the distributions in Papua New Guinea in 2017 and 2018.

Do LLINs reach intended destinations?

We have seen reasonable evidence that a large proportion of AMF's LLINs have reached intended destinations in AMF's past distributions. The strongest evidence we have seen on this question in Malawi is from post-distribution surveys, and the strongest evidence from DRC and Ghana is from household-level data on nets received.

The main sources of evidence that LLINs reached their intended destinations at a high rate are (a) post-distribution surveys, where surveyors visit a sample of households that registered for nets six months after the distribution, and (b) data on which households are reported to have received their nets. We have also seen narrative distribution reports for most of AMF's distributions, and a few other minor sources of evidence that LLINs reached their intended destinations.

Post-distribution surveys

AMF's distribution partners conduct post-distribution surveys, discussed in detail below, about six months after each distribution (and every six months after that up to 2.5 years after the distribution) to determine whether nets are in place, being used, and in good condition.69 The first post-distribution surveys that have been completed to date for each large-scale distribution in Malawi have generally found relatively high rates of LLINs from the recent AMF distribution hung over sleeping spaces: 90% of sampled AMF LLINs were found hanging at the 6-month post-distribution check-up (PDCU) following the Ntcheu 2012 distribution, 87% at the 6-month PDCU for Balaka 2013, 93% at the 8-month PDCU for Dedza 2014, 81% at the 6-month PDCU for Dowa 2015, and 69% at the 6-month PDCU for the Ntcheu 2015 distribution (see this summary of results for details).70 We have not yet seen results from the first post-distribution survey for the Balaka 2015 distribution.

We believe that the results of the first post-distribution surveys from distributions in Malawi provide reasonable evidence that nets reached their intended locations at a high rate, though we note that the surveys have some methodological limitations, including that they are conducted by the same organization that carried out the distribution, and have not used fully random sampling (more detail below).

We believe that the 8- and 12-month surveys in Kasaï-Occidental were poorly implemented, and that the results of the surveys are not a reliable estimate of the proportion of AMF's LLINs that reached intended destinations (details in this blog post). We have not yet seen (or expected to see) results from any post-distribution surveys from Ghana.

Household-level data on nets received

In addition to recording data on the number of LLINs allocated to each household during the registration process, AMF's distribution partners also record data on the number of LLINs actually distributed to each household during the distribution process.

For Malawi, we have not seen household-level data on the number of nets received. Our understanding is that this data exists in paper form only.

For the Kasaï-Occidental distribution in DRC, households received nets at the same time they were registered, so the registration list provides some evidence that nets reached their intended destinations.71 We have seen registration and distribution data from Nord Ubangi, DRC that includes numbers of LLINs actually received and hung up for each household.72 We have not yet seen a distribution report from Nord Ubangi, so we are largely uncertain about how data on the number of LLINs households actually received was collected.

In Ghana, data on the number of LLINs actually distributed to each household was recorded by LLIN distributors, and later entered electronically into AMF's Data Entry System; we have seen this data for the Greater Accra and Northern Region distributions, but not yet for the Upper West distribution (this data is planned to be entered into AMF's Data Entry System in November 2016).73

Distribution reports

Distribution reports provide narrative summaries of distribution activities and discuss challenges encountered. These reports provide some evidence that distributions generally operated as intended (or that distribution partners are aware of specific challenges and have plans to address them) and that households actually received LLINs; however, we do not think this type of evidence is as strong as the quantitative evidence discussed above.

Concern Universal has provided distribution reports for four of the six large-scale distributions it has completed in Malawi.74 As we report in our March 2012 update, reports from the Ntcheu 2012 distribution note challenges including attempted thefts, double registrations, and logistical problems.75 Concern Universal provided a similar level of detail on challenges encountered in the Balaka 201376 and Dedza 2014 distributions,77 and the first half of the Dowa 2015 distribution.78 These reports increase our confidence that Concern Universal is aware of potential problems and has a system in place to address them.79 We have not yet seen reports for the Ntcheu 2015 or Balaka 2015 distributions in Malawi.

IMA World Health has provided a distribution report for the Kasaï-Occidental, DRC 2014 distribution (but we have not yet seen any distribution reports from the Nord Ubangi, DRC distribution).80

We have also seen progress reports covering periods of pre-distribution, distribution, and post-distribution activities from Episcopal Relief & Development, AMF's distribution partner in Ghana, for the 2016 distributions in Greater Accra, Northern Region, and Upper West Region, and a full report on the distribution in the Northern Region.81 These reports discuss problems encountered in each of these stages.82

Other evidence

For the Kasaï-Occidental 2014 distribution, AMF's distribution partner, IMA World Health, piloted the use of smartphones to record household data, including GPS coordinates, for registration and LLIN distribution.83 AMF has sent us detailed GPS data that shows the GPS coordinates for each household visited.84 The registration and distribution data from Nord Ubangi, DRC, also included GPS coordinates for each household.85

In Ghana, it is our understanding that "post-distribution validation tracing" (checking, immediately after the distribution, by phone or in-person, that a randomly selected sample of households actually received the correct number of LLINs) was used for all three AMF-funded distributions in 2016, but we have not seen comprehensive results from this process (e.g., the proportion of selected households that received the appropriate number of LLINs).86 AMF told us that it expects to see data from post-distribution validation tracing from Episcopal Relief & Development in January 2017.87

Are LLINs targeted at areas with high rates of malaria?

At the highest level, AMF appears to exclusively target countries with known malaria risk.88 Since 2012, AMF’s large-scale distributions have occurred in Malawi, DRC, and Ghana; AMF has signed agreements for distributions in 2017 and 2018 in Uganda, Togo, Papua New Guinea, and Malawi.89 Based on 2013 data, the World Health Organization estimated that Malawi, DRC, Ghana, Uganda, and Togo had malaria death rates of between 50 and 99 deaths per 100,000 people, and that Papua New Guinea's malaria death rate was between 10 and 49 deaths per 100,000 people.90

Do those who receive the LLINs install them in their homes properly? Do those who receive the LLINs utilize them consistently over the long term?

The data we have seen from post-distribution check-ups (PDCUs) from Malawi is generally consistent with what we expect based on the "decay model" of LLIN usage (high initial usage followed by declines in usage in the second and third years following the distribution). We believe the data we have seen from AMF's distributions in DRC so far is unreliable, and we have not yet seen PDCU results from Ghana since the distributions occurred in mid-to-late 2016.

AMF requires partners to conduct PDCUs at 6-month intervals (plus or minus a month) for a period of 2.5 years, or until the next community-wide net distribution in the same area, to determine whether LLINs are present, are hung, and what condition they are in.91 For AMF's large-scale distributions in Malawi, results are available from:92

• 6-, 15-, 24-, and 33-month surveys following the Ntcheu 2012 distribution.
• 6-, 14-, and 19-month surveys following the Balaka 2013 distribution.93
• 8-, 12-, and 18-month surveys following the Dedza 2014 distribution.94
• 6- and 12-month surveys from the Dowa 2015 distribution.
• 6-month survey from the Ntcheu 2015 distribution

We have not yet seen results from a PDCU following a small-scale emergency distribution following flooding in Malawi or from the first PDCU following the Balaka 2015 distribution.95

For distributions in the Democratic Republic of the Congo, results are available from 8- and 12-month surveys from Kasaï-Occidental; we have not yet expected to see any PDCU results from distributions in Ghana.96

Details of the methodology used in surveys from Malawi are available in our August 2014 update (see the description of the Ntcheu 15- and 24-month surveys) and August 2015 update (see the descriptions of the Ntcheu 33-, Balaka 14-, and Dedza 8-month surveys) on AMF. We discuss the methodology of the Balaka 6-month survey in this footnote.97 We also spoke with Concern Universal about the process used for PDCUs in Malawi and summarize the process here. We have not yet reviewed the methodologies for the 12-month survey from the Dedza 2014 distribution, the 6- and 12-month surveys from the Dowa 2015 distribution, or the 6-month survey from the Ntcheu 2015 distribution in depth, but believe they likely use similar methodologies to the other surveys we have seen from Malawi implemented by Concern Universal.

We believe the methodology used by Concern Universal in Malawi has several limitations. The surveys were conducted by the same organization that conducted the distribution and which may have an incentive to bias the results. Villages were not selected completely randomly for surveys, which may have led to under-sampling more difficult to reach areas. See this blog post for more details on non-random sampling in past post-distribution surveys in Malawi. In October 2016, AMF told us that it plans to implement fully-random sampling of households for future PDCUs for distributions that use AMF's Data Entry System.98

Generally, we note that we have not seen technical details at the level of a high-quality academic study on how the surveys in Malawi were carried out,99 and we therefore have not been able to fully vet the results.

We believe that the 8- and 12-month surveys in Kasaï-Occidental were poorly implemented, and that the results of the surveys are not a reliable estimate of the proportion of AMF's LLINs used effectively over the long term (details in this blog post).

Results from follow-up surveys

We have summarized the results from all available PDCUs in this spreadsheet (data sources in footnote).100 As an example, the following table gives results from the Ntcheu 2012 distribution PDCUs.

The following definitions are based on our understanding from multiple sources and may not have been used fully consistently for each PDCU. Definitions:

• Nets hung: AMF told us that interviewers are asked to observe whether nets are hung by entering interviewees' houses (rather than simply asking interviewees if they are hung). Concern Universal told us that PDCU surveyors use their judgment to assess whether nets are being hung properly when they are interviewing recipients and inspecting their nets and sleeping spaces.101 It is our understanding that none of the nets included in the "nets hung" category are "worn out" (i.e., all are in "viable", "good", or "very good" condition).102
• "Very good" condition: LLIN has fewer than 2 holes of less than 2 cm in size.103
• "Good" condition: LLIN has fewer than 10 small holes.104
• "Viable" condition: LLIN has more than 10 small holes or has one large hole.105
• "Worn out": Survey reports from Concern Universal did not provide a definition of this term. AMF has told us in the past that an LLIN is considered worn out if it has multiple large holes and the LLIN is unrepairable, such that it would not provide protection against mosquitoes.106
• Used correctly: According to a survey form from 2012, interviewers are asked to "ask the householder to demonstrate how the nets are used at night" and to then select yes or no to the question "are the nets being used correctly."107 In 2014, AMF told us that the procedure had not changed in more recent surveys.108 Note that some of the nets included in this category may not be AMF nets and that it is possible that some "used correctly" nets are also "worn out" nets.109 In this blog post, we discuss the possibility of bed nets being used for other purposes than covering sleeping spaces (e.g., being used as fishing nets). We have not seen evidence that AMF-distributed nets are being misused at a meaningful scale, and the PDCU results we have seen to date indicate that, at least in Malawi, a large majority of AMF-distributed nets are being used appropriately.
• People covered by nets: We are unsure of how to interpret this metric. We previously believed that this included all people who sleep under a properly hung and not worn out net. However, the data for some post-distribution check-ups is not consistent with this interpretation.110

The percentage of people covered by nets in AMF's post-distribution check-ups from Malawi is roughly in line with the net usage rates of the trials of bed net efficacy documented in our page on Long Lasting Insecticide Treated Nets, in which small-scale studies of bed net efficacy had net usage rates generally in the 60%-80% range.111

The "decay model" we use to estimate the lifespan of LLINs assumes that 92% of LLINs are functional and in use for the first year after a distribution, 80% of nets are functional and in use for the second year, and 50% of nets are functional and in use for the third year.112 AMF's post-distribution check-ups for the Ntcheu 2012 distribution generally seem to match the decay model:113

• 6 months after the distribution, 90% of nets were hung and 1% were worn out or missing (the model predicts 92% of nets should be functional and in use).
• 15 and 24 months after the distribution, 85% and 81% of nets were hung and 6% and 12% were worn out or missing, respectively (the model predicts 80% of nets should be functional and in use).
• 33 months after the distribution, 52% of nets were hung and 48% were worn out or missing (the model predicts 50% should be functional and in use).

PDCUs from the Balaka 2013 and Dedza 2014 distributions in Malawi have also found rates of LLINs hanging roughly in line with the expectations of the decay model (see this spreadsheet, "Decay rate comparison" sheet). Rates of LLINs hanging from the recent distribution for the PDCUs following the Dowa 2015 and Ntcheu 2015 distributions are somewhat lower than predicted by the decay model; AMF believes that the low hang-up rates for the 6-month PDCU following the Ntcheu 2015 distribution and the 6-month and 12-month PDCUs following the Dowa 2015 distribution may be due to the continued use of older LLINs from the 2012 distribution in Ntcheu and use of LLINs from other sources in Dowa (or other possible explanations, see footnote).114

It is not fully clear to us how to compare the net quality and net usage rates found in AMF's post-distribution check-ups to the assumptions in the decay model, in part because it is not clear whether the definition of a "functional and in use" net in the decay model depends on the type of metrics that AMF's data provides. We may do further analysis to more directly compare the model and AMF's data in the future.

As noted above, we have not yet seen (or expected to see) results from PDCUs from Ghana, and do not think that the results from PDCUs we have seen from DRC are reliable.

Do AMF's LLINs increase the total number of LLINs distributed, or would the recipients have received LLINs from other sources if not for AMF?

On a separate page, we discuss some cases where AMF was in discussions to fund a distribution, but ultimately did not. In most of these cases, the net gap AMF was in discussions to fill persisted for six or more months after AMF's discussions closed, and, in two out of the five cases we looked at, gaps persisted for long periods (18 months and ~3 years, respectively). In most cases, the gap was eventually filled by another funder. As far as we can tell, during the time between AMF withdrawing from discussions and another funder stepping in, the populations targeted for distributions did not receive nets and likely were inadequately protected from malaria.

Additionally, Ghana's National Malaria Control Program told us in August 2016 that, without AMF providing funding for the distributions in the Upper West, Greater Accra, and Northern Regions in 2016:115

• No LLINs may have been available to deliver in the Upper West Region in 2016. The Upper West Region had been deprioritized relative to other regions for mass LLIN distribution, since other malaria control programs (Seasonal Malaria Chemoprevention and Indoor Residual Spraying) were ongoing.
• Distributions in Greater Accra may have been targeted to slums and rural areas only. Note that we have seen some limited evidence that LLIN usage is generally higher in rural areas of Greater Accra than in semi-urban areas.116
• LLINs originally intended for continuous distributions (e.g., LLINs distributed through antenatal clinics) may have been re-directed to mass campaigns, which may have later led to shortages in continuous distribution channels.117

Overall, our understanding from our site visit and our discussions with Ghana's National Malaria Control Program is that the counterfactual impact of AMF funding LLINs in Ghana in 2016 is complex, but that Ghana's National Malaria Program may have been appropriately triaging its malaria control needs prior to AMF's involvement, making the remaining opportunities (e.g., distributing LLINs in regions with other ongoing malaria control strategies, distributing LLINs in semi-urban areas) somewhat less impactful in expectation.

Across Africa, there are substantial funding gaps for LLINs (more below) and because our impression from following AMF's progress over time is that, due to AMF's more limited funding and, perhaps, greater data requirements, governments often seek funding first from larger funders (particularly the Global Fund to fight AIDS, TB, and Malaria) and then may ask AMF to fill gaps. However, we note that this dynamic may change if AMF has significant resources in the future (more below) and that countries are sometimes able to choose how they allocate Global Fund grants among malaria interventions (including LLINs, treatment, and diagnosis), so the availability of funding for LLINs from AMF could cause countries to allocate less Global Fund resources to LLINs.118

Are there any negative or offsetting impacts?

• Do donated nets displace government health funding? This could be a concern if government funding that otherwise would have been spent on LLINs is spent on other, less worthwhile budget items. We have little sense of how important a concern this is in AMF's case. Our rough understanding is that AMF is operating in countries with substantial net gaps that the country government would not have been able to fill with its own funding.119
• Will insecticide-treated nets continue to be effective? As discussed in our report on insecticide-treated nets, there is strong evidence for the effectiveness of this intervention; however, the best evidence for the intervention was collected before LLINs were widely used and there is some evidence that mosquitoes have since adapted to the insecticide used in LLINs, possibly making them less effective. We have reviewed the evidence on the state of insecticide resistance. We concluded, "Broadly, it seems that insecticide resistance is a larger concern now than it was when we last thoroughly evaluated the evidence in 2012, but it remains difficult to quantify the impact of resistance. Our very rough best guess (methodology described in more detail below) is that ITNs are roughly one-third less effective on average across sub-Saharan Africa than they would be in the absence of insecticide resistance. ITNs remain a highly cost-effective intervention after incorporating this discount."
• Do free LLIN distributions distort incentives for recipients or distort local markets for nets? As discussed in our report on insecticide-treated nets, we feel that there is a reasonably strong case for distributing LLINs freely rather than selling them at market (or even below-market) prices. We also think that the benefits of distributing LLINs freely to a population likely outweigh the negative consequences of distortion in local net markets, though we have not factored these potentially negative consequences into our cost-effectiveness analysis of AMF.
• Could distribution of LLINs be inequitable and unfair, causing problems in the targeted communities? We feel that AMF's processes for determining needs for LLINs at the household level are fairly well-suited to ensuring that LLINs are distributed equitably. We have some concerns about whether AMF's process succeeds at identifying all villages or households located outside of villages.
• Does AMF divert skilled labor from other areas? In Malawi, net distributions have been conducted by low-level government health staff in partnership with the staff of AMF's partner NGO.120 AMF's partner in Malawi told us in 2012 that government health staff are normally involved in activities such as disseminating health-related information, reporting on levels of stunting and disease, carrying out immunization campaigns, and providing nutrition support.121 We do not know the extent to which net distribution reduces their ability to complete other duties, though we note that net distributions are generally completed within a few days in each local area, and we would guess that LLIN distributions are likely among the most cost-effective work they engage in.122 Diversion of skilled labor may be more of a concern in the DRC where 22 senior district health staff were employed as Field Supervisors for the Kasaï-Occidental distribution.123 AMF's distributions in Ghana in 2016 were planned by high-level staff from a government health agency and implemented by local government staff and volunteers; monitoring of the distribution was largely implemented by a Ghanaian non-profit organization.124 We have not investigated what other activities the government and non-profit staff and volunteers engage in, or whether the LLIN distribution interferes with their ability to perform other duties.

What do you get for your dollar?

Cost per LLIN distributed

AMF-specific estimates

• Estimates based on AMF’s completed distributions: Using budgets and actual cost data provided by AMF, we estimate that the total cost to purchase, distribute and follow up on the distribution of an AMF-funded LLIN is about $4.35 in Malawi, roughly $5.92 in DRC, and roughly $5.14 in Ghana.125
• Estimates based on AMF's planned distributions: The very roughly-estimated average costs (including LLIN purchase, distribution, and follow-up) for future distributions for which AMF has signed distribution agreements or expects to signs agreements soon are $4.52 per LLIN for the distribution in Togo in 2017, $5.14 for Uganda 2017, $4.52 for Papua New Guinea 2017 and 2018, and $3.87 for Malawi 2018.126 AMF also told us that it may sign distributions agreements for two other countries soon (country names not yet public); our weighted average for the cost per LLIN for these distributions is $4.69.127
• Estimate for distributions funded with additional donations: We also very roughly estimate, based on past and planned distributions, that the cost per net in distributions AMF may fund with additional donations is $4.85.128

Note that we prefer to include all costs incurred to carry out a project, not just those that the charity in question pays for itself. We believe that this gives the best view of what it costs to achieve a particular impact (such as saving a life) and also avoids the lack of clarity and complications of leverage in charity. Our estimate of the cost to distribute an AMF-funded LLIN aims to include both costs AMF pays and costs paid by others. However, note that in our cost per net estimates we have excluded the costs of research on insecticide resistance that AMF plans to fund in Uganda in 2017 (which is currently budgeted at $2.7 million), and the first phase of a study on insecticide resistance AMF funded in DRC (which cost around $0.1 million).129 We exclude these costs because we expect that insecticide resistance research may have additional benefits that are not accounted for in our cost-effectiveness analysis.

Full details in our AMF cost per net spreadsheet.

The estimate for completed distributions in Malawi uses:130

• The purchase price per LLIN, calculated from AMF's total LLIN budget and the number of LLINs distributed.
• Actual costs for five completed large-scale distributions in Malawi as reported by Concern Universal131 and a combination of budgeted and actual costs for post-distribution check-up surveys following these distributions.132 AMF told us that the non-net budgets include all monetary costs incurred by Concern Universal, as well as the use of vehicles and managerial staff time;133 they may exclude non-monetary costs such as the use of office space.
• An estimate of non-monetary costs incurred by Concern Universal and local governments, based on estimates provided in 2012 by Concern Universal for its distribution in Ntcheu province.134
• A calculated estimate of the non-monetary costs incurred by Concern Universal and local governments due to post-distribution check-ups (calculated based on the Ntcheu 2012 estimate discussed above).135
• An estimate of AMF's organizational costs per net, including an estimate of the value of donated services and volunteer time. Because these costs generally do not vary with the number of LLINs distributed and remain roughly constant on a per year basis, the per LLIN estimate is highly dependent on the time period used and the number of LLINs distributed in that period. We estimate AMF's organizational costs as $0.29 per LLIN for July 2011 through the beginning of October 2016.136

For DRC, the estimate is rougher. We constructed the estimate similarly, although the information we have for DRC is likely of lower quality.137 The estimate for the DRC is based on:138

• The purchase price per LLIN, calculated from AMF's total LLIN budget and the number of LLINs distributed.
• Actual costs for the Kasaï-Occidental 2014 and Nord Ubangi 2015 distributions.
• Actual costs for the first three post-distribution check-ups following the Kasaï-Occidental 2014 distribution, an estimate of the costs of the remaining Kasaï-Occidental post-distribution check-ups based on the costs of the completed post-distribution check-ups, and budgeted costs for the post-distribution check-ups for the Nord Ubangi 2015 distribution.
• The same estimate of AMF organizational costs as we used for Malawi.
• Calculated estimates of non-monetary costs incurred by IMA World Health and local governments, based on the estimate from the 2012 distribution in Ntcheu, Malawi.139

The higher cost per net in the DRC may be explained by a considerably more difficult operating environment in DRC,140 or our data for DRC may simply be of lower quality.

The estimate for Ghana is also rough; it is based on:141

• The purchase price per LLIN, calculated from AMF's total LLIN budget and the number of LLINs distributed.
• Budgeted costs for pre-distribution, distribution, and post-distribution activities (primarily monitoring activities, including post-distribution check-ups) from AMF's non-profit distribution partner in Ghana, Episcopal Relief & Development.142
• A rough estimate of costs incurred by Ghana Health Services for its implementation of the distributions.143
• The same estimate of AMF organizational costs as we used for Malawi and DRC.
• Calculated estimates of non-monetary costs incurred by Episcopal Relief & Development, ADDRO, and local governments, based on the estimate from the 2012 distribution in Ntcheu, Malawi.144

Our estimates of the cost per net for planned distributions that AMF has signed or is considering are also rough; they are based on:

• The purchase price per LLIN, based on what AMF told us it expects the cost to be.
• AMF's estimate of the non-net costs associated with pre-distribution surveys, distribution days, and post-distribution check-ups for each upcoming distribution.145 These estimates are consistent with what we would expect based on past distributions in Malawi, DRC, and Ghana.146
• An estimate of AMF's organizational costs per net in the future, including increases in AMF's salary costs in FY 2017 and FY 2018 due to planned hiring of additional staff.147 We estimate AMF's organizational costs will be about $0.08 per net in the future, compared with $0.29 in our estimates for AMF's past distributions in Malawi, DRC, and Ghana.148

We are uncertain about the use of marginal funding to AMF this year. Our best guess at the cost per LLIN in distributions AMF may fund with additional funding this year is an average of the cost per LLIN for all of AMF's past large-scale distributions and its planned distributions discussed above (weighted by AMF's spending in each country). We calculate that the weighted average is $4.85 per LLIN.149

Global estimate

The Roll Back Malaria Partnership, which estimates funding gaps for LLINs across all African countries,150 uses an estimate of $2.70 to purchase an LLIN and $2.50 for all non-net costs of a distribution, for a total of $5.20 per LLIN.151 This assumption is used in cases where country-specific data is not available. Note that we have not seen details about where this estimate comes from, so we put little weight on it. The Co-Chair of the Roll Back Malaria Harmonization Working Group told us that non-net costs can vary significantly by country and that non-net costs in DRC may be more than $5 per LLIN.152 We have little information on how reliable the Roll Back Malaria Partnership's estimate is likely to be. Broadly, this estimate is consistent with the data we have seen from AMF. We have used estimates based on data from AMF in our cost per life saved calculations, because we have more information on their reliability. We do not believe our estimate of AMF's cost-per-net ought to be compared directly to the Roll Back Malaria Partnership's estimate, given how uncertain we are about the latter.

Cost per life saved

See our most recent cost-effectiveness model for estimates of the cost per life saved through AMF-funded LLIN distributions.

Note that our cost-effectiveness analyses are simplified models that do not take into account a number of factors. For example, our model does not include the short-term impact of non-fatal cases of malaria prevented or prevention of other mosquito-borne diseases. It also does not include possible offsetting impacts or other harms. We do include possible developmental impacts on children who sleep under an LLIN.153

There are limitations to this kind of cost-effectiveness analysis, and we believe that cost-effectiveness estimates such as these should not be taken literally, due to the significant uncertainty around them. We provide these estimates (a) for comparative purposes and (b) because working on them helps us ensure that we are thinking through as many of the relevant issues as possible.

The full details of our cost-effectiveness analysis are at our report on mass distribution of LLINs.

Is there room for more funding?

We estimate that AMF could productively use or commit a maximum of between $78 million (50% confidence) and $191 million (5% confidence) in additional unrestricted funding in its next budget year.

In short, we calculate this from (more details in the sections below):

• Total opportunities to spend funds productively: We estimate that AMF could use between $87 million (50% confidence) and $200 million (5% confidence) in the next year. We are most confident in the value of the funds for the first $11 million it receives, but we expect funds to continue to be valuable, though possibly somewhat less cost-effective, above that level.
• Cash on hand: As of October 2016, AMF held $17.8 million that it could allocate to additional distributions and expected to soon sign agreements for an additional $12.9 million, leaving $4.9 million available.
• Expected additional funding: We estimate that AMF will receive an additional $4 million in unrestricted funding for its work in 2017.

Below, we also discuss:

• Past spending: AMF's progress in 2016 on signing agreements in several countries it has not previously worked in has considerably increased our confidence in AMF's ability to productively use large amounts of money.
• Risks to room for more funding: The main risks we see to AMF's ability to spend funds productively in 2017 are sufficient funding being available from other funders, NMCPs capacity to sign agreements with AMF while applying for Global Fund funding, AMF staff capacity, and the specifics of where funding gaps exist.

Available and expected funds

As of November 2, 2016, AMF held $56.6 million. Of this, $38.8 million was committed to five future distributions and two other projects,154 leaving $17.8 million in funds it could allocate to future distributions.155 It expected to commit $12.9 million to two additional 2017 distributions and a research project soon, leaving $4.9 million available to commit in 2017 (primarily to distributions taking place in 2018).156

We expect that AMF will receive additional donations over the remainder of 2016 and in 2017 from:

• Donors who are not influenced by GiveWell's research: Historically, the vast majority of AMF's funding has been due to GiveWell's recommendation.157 In its FY 2016 (July 2015 to June 2016), AMF received approximately $10 million that it did not directly attribute to GiveWell.158 This is significantly more than we expected and we believe GiveWell may have influenced a large portion (perhaps 80%) of these donations. As a rough guess, we expect AMF to receive $2 million from non-GiveWell sources in the next year.
• Donors who give based on GiveWell's top charity list, but do not follow our recommendation for marginal funding: GiveWell maintains both a list of all top charities that meet our criteria and a recommendation for which charity or charities to give to in order to maximize the impact of additional donations, given cost-effectiveness of remaining funding gaps. We estimate that AMF will receive about $2 million from donors who use our top charity list but don't follow our recommendation for marginal donations.159
• Donors who follow GiveWell's recommendation for marginal donations: Our estimate of room for more funding is used to make a recommendation to these donors.

With $4.9 million in currently available funding and $4 million expected in additional funding ($2 million from each of the first two groups), we estimate that AMF will have about $9 million available in 2017.

Expected maximum

We believe that there will be a large funding gap for mass LLIN campaigns in the next year, but we are quite uncertain about the size of the gap.

Rough ALMA estimate

The African Leaders Malaria Alliance (ALMA) estimates funding gaps for LLINs across all African countries. In 2014, 2015, March 2016, and June 2016 we asked Melanie Renshaw of ALMA for updates on the size of the funding gap. On all occasions she told us that there remained significant gaps in funding for LLINs.160 We spoke again with Melanie Renshaw in October and November 2016 about funding gaps for 2018 (when distributions AMF signed in 2017 would likely take place).161 At that time, it was difficult for her to estimate the size of the funding gap for 2018 because the Global Fund to fight AIDS, Tuberculosis and Malaria, the largest funder of LLINs, works on three-year cycles and had not yet determined how much funding it would allocate for LLINs for 2018-2020. In September 2016, it raised $12.9 billion of its $13 billion goal for the 2018-2020 period and expects to announce how much it will allocate to each country for work on each disease in mid-December 2016. In early 2017, countries will begin applying for funding, which will include specifying how much they would like to spend on LLINs.162

Dr. Renshaw roughly estimated that there will be a funding gap for 100 million nets in 2018-2020. She estimated that the gap in Nigeria would account for a quarter of the total gap, or about 25 million nets. This assumes that funders other than the Global Fund (including AMF) will maintain their current level of support for LLINs in this period. Dr. Renshaw believed that less funding from the Global Fund would be available for LLINs because of changes in the way it is structuring its funding.163

We exclude the gap in Nigeria from our estimates of AMF's room for more funding because AMF has told us that it believes that funding distributions in Nigeria carries more risks than other countries.164

Because Dr. Renshaw's estimate assumes that AMF will replace nets it funds in the current three-year period, 2015-2017, we calculate the number of nets AMF has funded, has committed funding for, or is in late stages of discussions to provide funding for over that period and add it to the total gap that is relevant to AMF. AMF has funded or expects to fund the equivalent of the full costs of 14.7 million nets in 2015-2017, or an average of about 5 million nets per year.165 Therefore, we estimate the gap outside of Nigeria at roughly 30 million nets per year.166 At $4.16 per net (this is in variable monetary costs only—it excludes in-kind costs and AMF organizational costs), this is a funding gap of about $125 million per year. The gap may be larger if countries begin using PBO or other "next generation" nets.

This is in-line with what the end of year gaps have been in recent years. Dr. Renshaw estimated that the unfilled gaps in 2013-2015 were 22 million (13% of the total need), 38 million (15% of the need), and 39 million (18% of the need), respectively.167 That is a total gap of 99 million nets over three years, essentially the same as Dr. Renshaw's rough estimate of 100 million over the three year period (2018-2020).

Specific country gaps

We spoke to individuals who are involved in national planning for mass LLIN campaigns in two countries AMF is interested in working in in the future. They provided details on why other funding sources are unlikely to be sufficient to replace LLINs when the next campaign is due.

In one case, the next LLIN campaign will be due before it is likely feasible for Global Fund funding to be secured and used to order and ship LLINs. AMF may be able to move up the distribution by a year or more. There are also expected to be funding gaps for the next campaign, regardless of when it occurs, because, due to government regulations and systems, the country has been slow to spend Global Fund resources and, as a result, the Global Fund has reduced how much funding it allocates to the country. Note that the Global Fund has not yet announced its allocation to the country for the next round, so it is possible that the allocation will be higher than expected.

In another case, past campaigns have had major funding gaps, and recently two major funders of previous mass campaigns, the President's Malaria Initiative and the UK Department for International Development, have shifted their focus to other malaria interventions, such as indoor residual spraying, treatment for pregnant women, strengthening health systems, and private sector case management. These two funders are not expected to provide as much funding for mass LLIN campaigns in the future as they have in the past.

GiveWell's prioritization of AMF's funding gaps

We have broken down our top charities' funding gaps and ranked them based on:

• Capacity relevance: how important the funding is for the charity's development and future success.
• Execution relevance: how likely it is that the charity's activities will be constrained if it does not receive the funding.

We believe that "capacity-relevant" gaps are the most important to fill, and "execution"-related gaps vary in importance. More explanation of this model is in this blog post.

We don't consider any of AMF's current funding gaps to be capacity-relevant.

We consider the funding gaps to be "execution" gaps and assign them a level (1, 2 or 3) based on how likely we believe it is that AMF would be constrained by funding (rather than other factors, such as inability to find additional distribution partners) if it is unable to fill the funding gap. Level 1 is a 50% chance of funding being the constraint, level 2 is a 20% chance, and level 3 is a 5% chance. These judgments are rough and largely based on intuitions formed from following AMF over several years.168

We are largely in agreement with AMF about the size of its execution levels.169

We believe there are differences in cost-effectiveness within execution level 1 and believe the value of filling the first part of AMF's gap may be higher than additional funding at higher levels. AMF told us its priorities are first to ensure that countries it has worked in before have funding for their next mass LLIN campaigns, and then to seek out gaps it could fill in other countries (more on this below). Due to AMF's large scale up in 2016, which was in a large part for distributions taking place in 2017, and a standard length of time between distributions of three years, many of the nets AMF would be replacing would be distributed in late 2019 or 2020. We understand that more lead time is beneficial for countries' planning processes,170 but are concerned that committing to distributions far in advance increases the uncertainty about whether funding would have been available from another source. By our estimates, the first $11 million in additional funding AMF receives would be used to fund 2018 distributions, and funding beyond that, up to about $61 million in additional funding, may be used to fund 2019-2020 distributions. Funding beyond that might be used to help fill gaps in countries AMF has not worked in before.171

Rate of funds moved

AMF expects to spend $47 million on nets and non-net distribution costs in 2017, of which $37 million has been committed.172 In 2016, AMF signed agreements to fund its largest distributions to date, in Ghana, Uganda, Togo, and Papua New Guinea, totaling 16.9 million LLINs. AMF also told us that it expects to sign two additional agreements for distributions taking place in 2017, totaling 4.0 million LLINs. We believe there is significantly less uncertainty about AMF's ability to find new partners and countries to work with than there was in the past.

In November 2013, we wrote in this blog post that we believed AMF did not have room for more funding because it had raised over $10.6 million due to GiveWell's recommendation since 2011 and had not spent the bulk of these funds. In 2014, AMF completed and signed agreements for several large distributions, increasing our confidence that it could productively use the donated funds it held.173 In 2015, AMF did not sign any new distribution agreements for the near future, although it signed agreements to replace nets it had previously funded in Malawi in 2018.174

Risks to room for more funding

The main risks we see to AMF's ability to spend funds productively in 2017 are:

• Sufficient funding available for nets from other funders. There seems to be a broad consensus that there will be a funding gap for mass LLIN distributions in the next few years—we have spoken with ALMA and people involved in planning distributions at the national level (see above), as well as the Global Fund, and all strongly believed that there would be large gaps. That said, the estimate we have for the funding gap for nets is quite rough. The gap as a portion of the total need is fairly small and it is possible, though we believe unlikely, that the true gap is very small or zero. If so, there would be a high risk of AMF's funding largely being used to displace funding from other funders (more on this in the next section). It's not clear to us that we would be able to understand, after the fact, that this has occurred, because we don't have good information on how funding would be allocated in AMF's absence. It is also possible that there will be sizable gaps in 2018-2020 but that countries will choose to front-load funding from the Global Fund so that AMF funding is not needed until later years. AMF notes that it believes that this outcome is highly unlikely (details in footnote).175
• Planning horizons and capacity at NMCPs. The Global Fund has invited countries to apply for funding over the course of next year, with the first round of applications due in March and with decisions made on the first round in September.176 If NMCPs are uncertain about the funding they will receive from the Global Fund and/or are devoting much of their time to the Global Fund application process, this may make it difficult for them to come to agreements with AMF. AMF notes that it believes that it is unlikely that the Global Fund application process would interfere with signing agreements with AMF (details in footnote).177
• Staff capacity. AMF has been constrained by limited staff capacity in 2016.178 If AMF remains capacity constrained going forward, capacity, not funding, may be the limiting factor on AMF's ability to fund more distributions. AMF has told us that it plans to hire more staff,179 but has made only limited progress on this in 2016, hiring one Program Director.180 In the short term, we'd expect hiring staff to reduce AMF's staff capacity as it focuses on recruiting and training. AMF notes that it believes that staff capacity has not constrained its ability to sign agreements or manage distributions.181
• Specifics of where gaps exist. AMF will likely have difficulty filling gaps if they are in countries with security concerns or other difficult operating environments, or if there are small gaps across many countries.

Considerations around the size of AMF distributions

Efficiency and leverage

AMF has told us that focusing on large distributions allows it to:182

• Be more efficient, since its staff capacity is largely limited by the number of distributions it is in discussions about and following up on.
• Focus on the countries that it has experience working with and where it has relationships with partners.
• Have more leverage to ask countries to carry out distributions and monitoring according to AMF's preferred processes. Funding a distribution requires negotiating with NMCPs, which we perceive to have some discretion in which funders they work with, and which we perceive to be choosing funders based on a variety of factors, including size and reporting requirements.183 In the past, AMF has been able to fund only a relatively small piece of countries' distributions (Uganda is an exception), but has maintained substantial reporting requirements. This dynamic may create fundamental reasons for governments to prefer partnerships with other funders.
• Give countries more confidence that their distributions will be fully funded and allow for easier planning and more timely distributions.

For these reasons, AMF's preferred approach for 2017 is to offer to purchase all the nets needed for one or more countries' distributions, and, if additional funds are available, fill gaps in LLIN funding for other countries; it would not fund the non-net costs, so this would mean funding about half of the full cost of the distribution.

Fungibility

AMF's approach in the past has been to look for funding gaps—countries that do not have sufficient funding from the Global Fund and other funders for nets—and offer to fill or partially fill those gaps. Given that countries and other funders have some discretion over how funds will be used, it is likely that some portion of AMF's funding has displaced other funding into other malaria interventions and into other uses.

We would guess that this effect would be significantly greater (i.e., there would be more displacement of other funding) if AMF were to pursue the strategy of offering to purchase all of the nets needed for one or more countries, rather than filling in gaps once the Global Fund has made its allocations. It is our understanding that once the Global Fund tells countries how much funding they have been allocated for each disease, it would be difficult to change those allocations and to shift funding to countries that have funding gaps for nets. Therefore, the result of AMF providing a large amount of funding for nets to one country would be for that country to spend more Global Fund resources on other malaria interventions (including general health systems strengthening) or (less likely) to reallocate funds from malaria to AIDS and/or tuberculosis work.

It is possible that AMF could coordinate with the Global Fund before it announces country allocations in December 2016, which might allow the Global Fund to shift resources to countries that would use more of the funding for nets. We feel this is a risky strategy because the Global Fund's allocation process is complex and we don't expect to get good information on how or whether the allocation was adjusted due to AMF's contributions. There would be only a few weeks, at most, for AMF to coordinate with the Global Fund and with countries, which increases the chances of miscommunication. There may also be some risk for the countries if they use the additional resources to scale-up other interventions and cannot maintain the work if AMF is unable to renew the funding for subsequent distributions.

Our conclusion on distribution size

We feel that the risk of displacing a large amount of funding using the approach where AMF purchases all of the nets for one or more countries outweighs the benefits. We have requested that AMF use GiveWell-influenced funding to seek out gaps that other funders are unlikely to fill.

What portion of all net distributions are funded by AMF?

In 2015 and the first three quarters of 2016, approximately 274 million nets were distributed in Sub-Saharan Africa. 56% of these nets were funded by the Global Fund, 25% of these nets were funded by the President's Malaria Initiative, and 6% were funded by UNICEF.184

At 5 million nets per year (see above), AMF's contribution is about 2.5% of all nets distributed.185 In 2017, AMF has committed to fund 14.2 million LLINs (and may fund an additional 3.5 million), which would be about 10% of the annual total, projecting forward from 2015-2016.

AMF as an organization

We believe the Against Malaria Foundation to be an exceptionally effective organization:

• Track record: We feel AMF has built up a strong track record of raising money, finding distribution partners, and getting partners to report information publicly at an unusual level. In 2015-16, AMF has begun to demonstrate its ability to implement its model at significantly larger scale.
• Communication: AMF has communicated clearly and directly with us and given thoughtful answers to our critical questions.
• Self-evaluation: AMF has invested heavily in self-evaluation, going above and beyond what’s usual for data collection on bed net distributions: it has demonstrated a commitment to collecting long-term net usage data.
  
• Transparency: AMF has generally appeared to value transparency as much as any organization we’ve encountered. It hasn’t just shared information with us; it publishes significant amounts of useful information publicly on its own, far more than the norm. This year, we felt that AMF had not been as transparent as it could be; we wrote about this issue in this blog post.
  Response from AMF

  In this [June 2016] update review GiveWell has said AMF has recently been less transparent in some areas. They are right and we accept this criticism. It is not, however, a change in our policy and we remain committed to being a highly transparent organisation. We have not sent GiveWell some net monitoring information on DRC in a timely fashion due to it taking longer for us to receive the required data, it requiring more validating than we expected and this coinciding with a busy time at AMF in which we were finalising several agreements. All of the information critiqued here will be made public, as normal, and we are in the process of completing that now. This is an explanation not an excuse and GiveWell’s criticism is fair and we will seek to avoid this happening again as timely delivery of information is as important as its delivery.

However, we see some potential room for improvement:

• AMF has not been as aggressive and ambitious about growth and capacity as it could have been. It operated for most of its 11 years with only two full-time staff members; it does not appear to us to have built as robust a fundraising operation as it could. More recently, AMF has taken modest steps to increase its capacity. Since 2014, AMF's executive chairman has dedicated half his time to AMF, and in February 2016, AMF hired a full-time program director.186 In November 2016, AMF told us that it expects to hire for a few positions in the near future.187

More on how we think about evaluating organizations at our 2012 blog post.

Sources

• 1

  "How will the money raised be spent? On bed nets (mosquito nets). Specifically, long-lasting insecticidal nets (LLINs)." AMF Frequently Asked Questions.

• 2

  See our Summary of AMF Distributions spreadsheet. Adding up the "# LLINs" column for the Ntcheu District 2012, Balaka District 2013, Dedza District 2014, Kasaï-Occidental 2014, Dowa District 2015, Balaka District 2015, Ntcheu District 2015, Nord Ubangi 2015-16, Greater Accra 2016, Upper West Region 2016, and Northern Region 2016 distributions yields 5,785,420 nets. Spreadsheet sourced from AMF Distributions.

• 3
  • See our Summary of AMF Distributions spreadsheet. Spreadsheet sourced from AMF Distributions
  • The sum of "# of LLINs" for 2018 distributions in Balaka, Ntcheu, Dowa, and Dedza, Malawi is 1,480,000.

• 4

  AMF Papua New Guinea blog post 2016:

  • "AMF has signed an agreement with partners in Papua New Guinea to fund 1.2 million nets for distribution in 2017 and has committed to fund 1.6 million nets to be distributed in 2018."
  • "Note: We have signed the agreement for the nets to be distributed in 2017. We have not signed the agreement for the 2018 nets as this will take place in the first part of next year when other partners receive formal notification of the relevant funds allocation."

• 5

  This understanding is based on many conversations with AMF, and from following AMF's progress over time.

• 6

  This understanding is based on many conversations with AMF and its distribution partners, and from following AMF's progress over time.

• 7
  • "This is a list of the countries with known gaps and where there are significant contiguous areas without nets, or a significant percentage required, and for which the estimate of need is believed to be reasonably accurate. It does not include countries where there are gaps, typically up to 40% of what the nation needs, but they are spread more uniformly across the country and would therefore require an ‘in-fill campaign’. An in-fill campaign is different from a so-called ‘universal coverage campaign’ because the percentage installed base of nets is higher in the former case and so a pre-distribution registration survey (PDRS) is an absolute requirement to ensure an efficient allocation of nets. Our methodology would lend itself to these campaigns if the relevant National Malaria Control Programme (NMCP) were to embrace a detailed PDRS. The list does not include, in our view, other countries where the need has not yet been quantified. Given there are many countries with needs estimated, we have not chosen to seek out other countries in need of nets. Our assumption is groups like AMP will be a source of reporting on additional countries as quantified needs emerge." Rob Mather, AMF Founder, email to GiveWell, August 8, 2012.
  • In September 2015, we checked in with AMF about its process for determining in which countries it works. Rob Mather noted that AMF continues to review malaria prevalence data (where that data exists), although maintains a level of skepticism about that data given that it can be unreliable. Even so, AMF feels comfortable drawing conclusions about which countries have high malaria mortality burdens based on the data it sees and the conversations that it engages in. AMF checks the following sources to keep up-to-date on which countries have a significant malaria burden:
    • The Alliance for Malaria Prevention, which sends out a weekly email with malaria-related data and information.
    • The African Leaders Malaria Alliance (ALMA), which AMF has conversations with on a quarterly basis.
    • In-country partners, who frequently attend malaria task force meetings and have recent news
    • Members of malaria advisory groups
    • Other connections

    Before AMF decides to approach a country to offer funding for nets, it has many conversations to confirm the level of need that country has with the other actors that are working on malaria in the country. Rob Mather and Peter Sherratt, conversation with GiveWell, September 9, 2015

  • In early 2016, AMF described a location-selection process similar to that described previously: "AMF learns about net gaps and receives funding requests through its network in the malaria control community, particularly through the Alliance for Malaria Prevention and the African Leaders Malaria Alliance." Rob Mather and Peter Sherratt, conversation with GiveWell, February 28, 2016

• 8

  "As it becomes involved in larger distributions, AMF is receiving a growing number of funding requests. As its funding increases, AMF aims to make more strategic investments by engaging in the planning cycles of countries where it has strong connections and experience." Rob Mather and Peter Sherratt, conversation with GiveWell, February 28, 2016

• 9

  Comment provided in response to a draft version of this review in November 2016.

• 10

  Rob Mather, AMF Founder, conversation with GiveWell, July 19, 2012

• 11

  Rob Mather, AMF Founder, email to GiveWell, August 8, 2012

• Ghana's National Malaria Control Program, one of AMF's distribution partners for the June-July 2016 distribution campaign in Ghana, described negotiating and compromising with AMF on requirements for the distribution:
  • "The NMCP has been working with AMF for a relatively short period of time. Their working relationship has proceeded relatively smoothly thus far, especially since AMF has shown willingness to negotiate and compromise on some areas to conform with the country's specific scenario." GiveWell's non-verbatim summary of a conversation with Ghana's National Malaria Control Program, August 16-18, 2016

• 12

  See our Summary of AMF Distributions spreadsheet. Spreadsheet sourced from AMF Distributions

• 13
  • Concern Universal, AMF's partner in Malawi, has collected data from each household on the number of sleeping spaces and LLINs in a condition such that they have at least one year of remaining use. Households were allocated enough LLINs to cover all sleeping spaces, minus the number of existing, usable nets. Concern Universal also collects the name of the head of the household.
    • Rob Mather, AMF Founder, conversation with GiveWell, July 19, 2012.
    • "The following information will be collected from each household:
      R Name of head of household
      R Number of sleeping spaces
      R Number of perfectly usable nets (with at least two years life remaining)
      O Names (or first names) of people in each sleeping space (‘for the purpose of nets’)
      O Age, sex of each person
      O Number of pregnant women
      O Number of children under 5
      Information that will not be collected (from list above):
      None
      R = Required, O = Optional" Concern Universal, Dowa 2015 planning document, pgs. 3-4.
  • We have seen pre-distribution survey results from all of AMF's completed large-scale distributions in Malawi: full data from its Ntcheu 2012, Balaka 2015, and Ntcheu 2015 distributions; and sample results from its Balaka 2013, Dedza 2014, Dowa 2015 distributions:
    • Concern Universal Ntcheu 2012 pre-distribution registration survey data
    • Concern Universal Balaka 2013 pre-distribution registration survey data
    • Concern Universal Dedza 2014 pre-distribution registration survey data
    • Concern Universal Dowa 2015 pre-distribution registration survey data
    • Concern Universal Balaka 2015 registration data 100% vs 5%
    • Concern Universal Ntcheu 2015 pre-distribution registration survey data
  • IMA World Health, AMF's partner in the Democratic Republic of the Congo (DRC) used different registration processes for the two distributions it has completed (in Kasaï-Occidental and Nord Ubangi).

    For the Kasaï-Occidental distribution, IMA World Health concurrently collected registration data on smartphones, distributed LLINs, and hung LLINs in recipients' homes. IMA World Health collected registration data on the number of sleeping spaces and previously-owned nets in usable condition in each household, and allocated enough LLINs to cover all sleeping spaces, minus the number of existing, usable nets. GPS coordinates identifying the location of the household and data on the number of AMF's LLINs hung up during the distribution process were also collected. We have seen full registration data from Kasaï-Occidental:

    • AMF: "DRC, West Kasaï Province: Distribution Report and separate Technology Report":
      • "In West Kasaï the fight against malaria involves the distribution and hang‐up of LLINs, a so‐called ‘Hang‐Up’ strategy, in which volunteer Community Health Workers (CHWs) distribute the nets and hang them at the same time. This is in contrast to strategies that simply hand out the nets. A modified version of the Hang‐Up strategy, called a ‘Hang‐Up and Track’ (HUT) strategy, was used on this distribution. With the HUT approach, in addition to distribution and hang‐up, information on each net hung is captured using smartphones equipped with a data collection program." Distribution Report, Pg 4.
      • "Teams of three or four CHWs [Community Health Workers] visited households sequentially.

        This was a crucial stage of the net distribution that consisted of collecting relevant household data for every single household in the distribution catchment area including, but not limited to, the following ‘must have’ information:

        • Name of household head
        • Number of people in the household
        • Number of sleeping spaces (‘for the purpose of nets’*)
        • Number of perfectly good nets with at least two years of life left in them
        • Location of household (GPS coordinates)
        • Household identifier**

        The last two pieces of information were to facilitate later identification of the individual household for post‐distribution check‐up work.
        * When assessing the number of sleeping spaces it is important to avoid confusion between the number of separate areas in which individuals might sleep and, given nets distributed are ‘double’ nets that will accommodate two people, the number of sleeping spaces that is equal to the number of nets that are needed. For example, in a household with mum plus dad, four children aged eight, four, six and two and a grandfather there may be six separate sleeping areas: mum plus dad, one each for the four children and one for the grandfather. However, for the purposes of nets, there would be four sleeping spaces: one for mum and dad, one for the two older children, one for the two younger children and one for the grandfather.
        **Creating a household identifier is necessary as many households do not have clear addresses. For example, it is frequently the case in rural and semi‐rural locations that street names and household numbers do not exist." Distribution Report, Pg 16.

    • IMA World Health, Kasaï-Occidental 2014 distribution data
  • In Nord Ubangi, DRC, registration data was also collected on smartphones, but it was collected prior to the distribution rather than during it. IMA World Health collected registration data on the number of sleeping spaces and previously-owned nets in usable condition in each household, and allocated enough LLINs to cover all sleeping spaces, minus the number of existing, usable nets. GPS coordinates identifying the location of the household and data on the number of AMF's LLINs hung up were also collected. We have seen sample registration data from Nord Ubangi:
    • "Differences [between the Nord Ubangi distribution and the Kasaï-Occidental 2014 distribution]
      1. The North Ubangi distribution will be a two-phase distribution – a distinct pre-distribution phase and a subsequent distribution phase. In West Kasaï establishing net need per household and distributing of nets was contemporaneous.
      2. Distribution of nets will be done from distribution points rather than at the household
      3. The ODK registration form has been re-designed to better capture the required data, to eliminate some types of error and to reduce the chance of others.
      4. Operational lessons learned from the West Kasaï distribution will be implemented.
      5. An experienced, full time technology manager has been employed by IMA meaning there will be more extensive and easier liaising with AMF on data elements of the distribution. We have met the new technology manger and have spent a week working with him and are impressed.

      Similarities [between the Nord Ubangi distribution and the Kasaï-Occidental 2014 distribution]
      1. The same smartphone-based data collection technology will be used as in West Kasaï.

      Comment
      IMA has agreed there are benefits to conducting a separate, earlier pre-distribution phase during which accurate household-level net need is established. IMA are less able to estimate accurately population and net need in North Ubangi compared to what they felt was possible in West Kasaï.

      IMA believes point distributions rather than at-household distribution will be more cost-effective." Rob Mather, AMF Founder, email to GiveWell, September 9, 2015

    • "IMA will ensure the collection and availability, for independent inspection, of household level data (sleeping spaces and number of perfectly usable nets in place) for all households in the distribution zones identified in section 4." IMA World Health and AMF distribution agreement for Nord Ubangi 2015
    • IMA World Health Nord Ubangi 2015-16 registration data
  • In Ghana, volunteers organized by one of AMF's distribution partners, Ghana Health Services (the National Malaria Control Program is a part of Ghana Health Services), collected information on the number of people in each household prior to the distribution. Households were allocated one net per two people (rounding up for households with an odd number of people). Previously-owned nets in usable condition were not taken into account when allocating LLINs.
    • GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016:
      • "GHS [Ghana Health Services] volunteers carry out the registration process, which takes place more than one month before the distribution. Volunteers are organized by the Ministry of Health (MoH) and participate in a number of government health programs, such as vaccination campaigns (immunization days). Many of them are long-time volunteers with considerable experience." Pg. 5.
      • " Volunteers visit each house in their zone. The head of household, or someone else who is present, is asked how many people are in the household based on the 'people who eat from the same pot' definition. Ghana's policy is to provide one net for every two individuals in a household; as a result, in contrast to previous AMF distributions, volunteers were not required to check households' existing nets or number of sleeping spaces. The only exception to this policy is that household members aged 70 or older are not counted in the household total, and each receive their own net." Pg. 5.
  • AMF has shared all registration data collected from its Northern Region and Greater Accra distributions, and a summary of registration data from its Upper West distribution (which has not yet been validated, see here) with us.
    • AMF Ghana Data Entry System 2016 (for the Northern and Greater Accra regions).
    • AMF Upper West Region Ghana pre-validation registration data 2016
    • The full registration data we have seen in AMF Ghana Data Entry System 2016 includes names of heads of households and either an address or a written description of the location of a household (e.g., "Next to the church").

• 14

  Point distributions have been implemented in Malawi, Ghana, and Nord Ubangi, DRC, and a hang-up distribution was implemented in Kasaï-Occidental, DRC. See our Summary of AMF Distributions spreadsheet. Spreadsheet sourced from AMF Distributions.

• 15
  • In Malawi, Concern Universal uses the following process for its distributions:
    • "After information is collected and verified, Concern Universal creates a distribution planning document which:
      • Assigns each village to a distribution site, with a goal of minimizing the distance that beneficiaries need to travel (most beneficiaries travel 4-5 miles to their distribution site). Each catchment area is divided into several distribution sites, and each site serves 2-12 villages, depending on the geographic location, number of nets to be distributed, etc.
      • Allocates a number of nets to be delivered to each location.

      Concern Universal staff then meet with traditional authorities and district health officials to explain the distribution plan and set a date for the distribution, avoiding days that conflict with other community events. Concern Universal communicates the date, time, and place of distributions verbally to village leaders, who relay the information verbally to village members.

      On distribution day, Concern Universal staff bring bed nets to the distribution sites using Concern Universal's vehicles or vehicles rented for that purpose. Village members typically walk to the distribution site; some travel by bike or organize transit in a motor vehicle. At the site, village leaders and HSAs group the beneficiaries by village. A roll call is taken, and as household names are called, members approach to receive nets. Household heads sign or fingerprint two copies of the distribution register to confirm receipt of their nets. If the household head is not present, a representative may receive nets on his/her behalf, as long as that person is recognized by the village head as being part of the household. When distribution is complete, each village leader receives a copy of the distribution register for their village, and Concern Universal retains the other copy." GiveWell's non-verbatim summary of a conversation with Nelson Coelho, April 15, 2016, pg. 4.

    • An example of managing in-country shipping and storage logistics from Concern Universal Dedza 2014 distribution report (our understanding is that Concern Universal has used similar processes for its other distributions in Malawi):
      • "7. Logistics and security of LLINs
        1. Storage and movement of nets
        Upon arrival in-country and after completing all custom clearance formalities, the nets were stored in the main warehouse located within the premises of the District Hospital in Dedza Town pending relocation to the respective Health Centres.

        ii) Security of nets
        In order to ensure the security of nets, new padlocks were installed on all warehouse doors and the keys kept by CU’s storekeeper. Comprehensive insurance was contracted to cover potential losses caused by theft, fire or water damage and security guards were deployed throughout the distribution period, until all nets were distributed. All movements of nets were supervised by at least two CU staff members and the security guard." Pg 6.

      • "3. Distribution activities
        i) Movement of nets
        One or two days prior to distribution, the required quantity of nets was ferried to the health center for temporary safe keeping pending relocation to the distribution points on the day of distribution. Two vehicles were used to carry the distribution personnel and nets." Pg 7.
  • In Kasaï-Occidental, DRC, IMA World Health implemented registration and hang-up distribution at the same time. The following bullet points are a summary of the distribution process (more details in full distribution report). IMA World Health, Kasaï-Occidental 2014 distribution report:
    • "Nets arrived in Kinshasa on 22 Jul 2014 in 21 x 40HC shipping containers each containing approximately 32,000 nets." Pg 5.
    • "The nets travelled 800 kms by road over the next three days and arrived in Tshikapa District with individual trucks driving directly to eight individual locations in each of the Health Zones where the relevant quantity of nets needed for the distribution in that district were pre‐positioned." Pg 6.
    • "Meetings were held with local leaders to seek permission and support for the distribution." Pg 12.
    • "Two categories of staff were required for the distribution. Field Supervisors (FSs), who had a management responsibility in each HA, and Community Health Workers (CHWs) who were responsible for visiting beneficiary households to collection registration information, hand out and hang the nets." Pg 12.
    • "The CHW responsible for the hanging of the nets does so with the help of the head of household and/or other household members, using the string and nails brought along if necessary. A net is hung over each sleeping space (two people per sleeping space) and then rolled up out of the way (for example off the ground) during the day to help avoid unnecessary wear and tear or damage. It is then rolled down at night." Pg 17.
  • In Nord Ubangi, DRC, IMA World Health's distribution process was more similar to distributions in Malawi, with the separation of registration and distribution processes and the use of point distribution:
    • "Differences [between the Nord Ubangi distribution and the Kasaï-Occidental 2014 distribution]
      1. The North Ubangi distribution will be a two-phase distribution – a distinct pre-distribution phase and a subsequent distribution phase. In West Kasaï establishing net need per household and distributing of nets was contemporaneous.
      2. Distribution of nets will be done from distribution points rather than at the household
      3. The ODK registration form has been re-designed to better capture the required data, to eliminate some types of error and to reduce the chance of others.
      4. Operational lessons learned from the West Kasaï distribution will be implemented.
      5. An experienced, full time technology manager has been employed by IMA meaning there will be more extensive and easier liaising with AMF on data elements of the distribution. We have met the new technology manger and have spent a week working with him and are impressed.

      Similarities [between the Nord Ubangi distribution and the Kasaï-Occidental 2014 distribution]
      1. The same smartphone-based data collection technology will be used as in West Kasaï. Comment
      IMA has agreed there are benefits to conducting a separate, earlier pre-distribution phase during which accurate household-level net need is established. IMA are less able to estimate accurately population and net need in North Ubangi compared to what they felt was possible in West Kasaï. IMA believes point distributions rather than at-household distribution will be more cost-effective." Rob Mather, AMF Founder, email to GiveWell, September 9, 2015

  • We have learned about the distribution processes for AMF's three distributions in 2016 in Ghana through conversations with AMF's distribution partners in Ghana (including the Greater Accra Regional Health Directorate of the Ghana Health Service, which managed in-country shipping and storage logistics for LLINs). From GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016:
    • "Steps in the distribution process
      1. Beneficiaries walk a short distance to their distribution point. One Community Health Officer (CHO) employed by GHS and one GHS volunteer are stationed at each distribution point; there are never just two GHS volunteers.
      2. Before giving out each net, the distributors rip the packaging to help prevent reselling.
      3. Distributors gather together groups of recipients in order to transmit educational messages.
      4. The nets are distributed. Distributors make sure the recipients' coupons (brought by recipients) match the registration copies (brought by distributors) and make a mark on each side. The recipient's coupon is returned to them. Distributors also have a tally sheet with rows of 5 bubbles, and mark off a bubble for every net they give out. Distributors do not have an opportunity to sit or take a lunch break during distribution days." Pgs. 8-9.
    • "When deciding where to locate distribution points, the Greater Accra GHS considered safety, costs (for example, the cost of renting a warehouse) and proximity to other distribution points. They also selected 12-13 pre-positioning points in each distribution area." Pg. 18.

• 16
  • "A copy of the beneficiary list for their village or community will be passed to the relevant leaders to check for duplications, omissions, or inaccuracies. This will include reading out the list publically so the community is involved and engaged in the verification and the net distribution preparations more generally. CU field staff supervise this process." Concern Universal, Dowa 2015 planning document, pg. 7.
  • Based on a conversation with AMF, it is our understanding that this same process is used for all of AMF's distributions in Malawi with Concern Universal. Rob Mather, AMF Founder, Conversation with GiveWell, September 28, 2016
  • The results of the registration data verification process in the Dedza 2014 distribution:
    • "During the verification process, CU staff roll-called the beneficiaries from the printed registers whilst the HSAs cross-checked with their list compiled from the registration forms in order to detect any mismatch. This verification exercise revealed that a total of 32,183 households had been skipped during the registration.
      […]
      "After the village household verifications, the corrected and edited registers were referred back to the data entry team and the corrections made to the database." Concern Universal Dedza 2014 distribution report Pg 5.
  • Results of verification process for other large-scale AMF distributions in Malawi:
    • Concern Universal Ntcheu 2012 distribution report, Pgs 5-7.
    • Concern Universal Balaka 2013 distribution report, Pg 6.
    • We have not yet seen the results of the registration data verification process for AMF's other large-scale distributions in Malawi (Dowa 2015, Balaka 2015, and Ntcheu 2015). We expect that the results of the registration data verification process will be reported in the full distribution reports for these distributions.

• 17
  • IMA World Health, Kasaï-Occidental 2014 distribution report:
    • "Household data went through the following steps:
      1. Data collected at the household by a CHW
      2. Independent supervisor uploaded the data from the smartphone to a laptop.
      3. The supervisor reviewed the data and corrected obvious mistake or inconsistencies."

      Pg 18.

    • "After the distribution had been completed in a Health Area and all data uploaded from the smartphones, the data team compared the electronic data with the paper data collected by each distribution site in each health area. At the end of the day each CHW returned undistributed nets to the supervisor and depot manager and reported the number of nets they distributed. These numbers were recorded on paper." Pg 18.
  • We have not seen the results of the supervisors' validation activities (e.g., the number of household records corrected for obvious errors and for discrepancies between electronic and paper records).
  • We have seen a high level analysis of the quality of the registration and distribution data, which finds that some data is available for 91% of the LLINs distributed (data is missing for the other 9% of LLINs). For households for which data is available, 96% had internally consistent data and 4% had apparent errors or anomalies. AMF distribution verification Kasaï-Occidental 2014, pg. 3.
  • We are not certain when AMF completed this analysis, or if this analysis was used in-country (e.g., to check whether households with inconsistent data had received the appropriate number of LLINs).

• 18
  • "Step 7 - Pre-distribution validation
    In all three regions with AMF-supported distributions, all coupons underwent a pre-distribution validation process by the NMCP and non-profit partners.
    At the community level, total estimated nets (based on 2010 census data) are compared to the total number of nets allocated during registration. This gives a sense of the variance between these figures; communities with more than 2% variance might be spot-checked.
    The validators also look for other potential problems with the coupons, including:
    • illegible entries
    • households with over 20 members
    • mismatches between the numeral and text versions (both are required) of figures in the 'nets allocated' field
    • miscalculation of figure in 'nets allocated' field

    If there are problems, the representative follows up with the volunteer who collected the data. The volunteer might return to the household to correct mistaken entries; if not, the representative will do so in person, or if that is too costly, by phone. If the information cannot be verified or corrected, the coupon is discarded." GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, pg. 6.

  • "This activity [pre-distribution validation] has been completed in the Northern and Greater Accra regions. In both regions, ADDRO observed a few mismatches in terms of number of members registered as a HH and the number of LLINs allocated. Following discovery of this misinformation, validation teams returned to the communities to verify valid information." Episcopal Relief & Development Ghana Activity Report 1 2016, pg. 6.
  • We have not seen comprehensive data on the results of the pre-distribution data validation process (e.g., exact numbers of household records corrected by this process).
  • "Post-distribution validation
    "After the grace period, coupon books are returned to the district level for the post-distribution validation process. This is carried out by the NMCP and non-profit partners, and involves analyzing the coupon books in order to:
    • Check for coupons that were not crossed out; for example, this scenario might arise if a household did not pick up their nets.
    • Check that the number of nets allocated matches the number of nets given.
    • Check that the number of nets allocated is accurate based on household size.
    • Add total values for household size, nets allocated, and nets given, and record them on the inside cover of each coupon book."

    GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, pg. 10.

  • We have not seen comprehensive data on the results of the post-distribution data validation process (e.g., exact numbers of household records corrected or households re-visited during this process).
  • "As it might be confusing to use a 5% figure in too many contexts, AMF and Episcopal Relief & Development decided to run verifications on 6% of the data entered by clerks. This data is reentered by another individual and checked against the original data. If there are discrepancies, the coupon book is verified." GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, pg. 13.
  • We have seen results from the 6% data entry checks for the Greater Accra distribution, but (as of November 2016), not from the Northern Region or Upper West Region distributions. As of November 6, 2016 33,269 "6% check" records had been entered (5.4% of the 566,947 total records). 2,586 of these 6% check records had not yet been matched to the original electronic records. Of the 30,683 matching pairs of records, 10,632 (35%) were an exact match without correction from a supervisor, an additional 19,598 (64%) had "key data" (number of people in a household, LLINs allocated, and LLINs give) match without correction from a supervisor, and 247 (0.8%) had mismatches of key data. AMF Ghana Data Entry System 2016

• 19
  GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016:
  • "Instead, Episcopal Relief & Development and ADDRO carried out 'light monitoring,' and considered themselves 'embedded' in the pre-distribution and distribution work. Monitoring activities included:
    • Attending meetings, including planning meetings at the national level, as well as a sample of district meetings and training sessions. This was done to observe whether activities were proceeding according to the NMCP's checklist and, if there were issues, to follow up with the NMCP.
    • ADDRO monitors observed the registration process in a randomly selected sample of 50% of the 20 districts in the Northern Region and all districts in the other two regions. Names of all of the sub-districts in these districts were written on slips of paper and randomly selected. Episcopal Relief & Development reported that it did not skip any of the randomly selected districts, even those that were far away. The order of sub-district visits was determined by what made sense geographically.
      • Monitors called the supervisors before arrival and visited some households to see whether they were aware that registration was happening in their area that day; almost everyone was aware.
      • Monitors also observed the work of multiple community-based agents (CBAs); sometimes, they would do a cross-check by performing this task without the supervisor. They listened to the CBA's conversations with households, verified that the correct information had been recorded in the book, and addressed any issues with the CBA. They then went back through the route traveled by the CBA to ensure that households that should have been registered were registered, and that the coupons had been correctly filled out. This process is facilitated by the practice of marking registered houses. Monitors also asked households if they were aware of the date and location of the distribution.
      • Monitors met with supervisors, analyzed completed coupon books to identify any problems, and filled out forms summarizing their daily observations.
      • Monitors might address issues by:
        • Debriefing with supervisors at the end of the day, so that learnings could be applied to the next day's work;
        • Communicating directly with CBAs if they encounter the CBA before meeting with the supervisor;
        • Observing CBAs without informing the supervisor, if they observed a number of issues the previous day." Pg. 7.
  • "Monitoring of the distribution process by ADDRO
    ADDRO has two or more monitoring teams per region; each of these has a vehicle. The NMCP has 2-3 evaluators per district; each has a vehicle. ADDRO representatives attend the NMCP planning meeting. ADDRO monitors introduce themselves to the municipal health directorate, and are taken by a representative to each distribution point in a sub-district to ensure that their presence will be accepted by those whose work they will be monitoring.

    In the Greater Accra Region, the malaria focal person for the district took monitors to a distribution point. From there, they went to other distribution points in the sub-district. They asked the focal person which areas were the least accessible and attempted to focus on those; in total, they visited 60% of the distribution points. They had the distributors' phone numbers so they could ask for directions if necessary.

    Monitors verified that:

    • nets were being distributed
    • distributors were using the tally sheet
    • nets were kept in the shade
    • coupons were marked when nets were distributed
    • a Ghana Health Services staff member was present
    • distributors were giving helpful messages to the recipients

    NMCP monitors were also present during the distribution." Pgs. 10-11.

  • We do not have a clear understanding of exactly how ADDRO monitors choose which distribution points to observe, but note that the selection process described above does not appear to be random.
  • Examples of challenges encountered and actions taken during distribution by ADDRO monitors in Zabzugu district, Northern Region:
    • "A lot of household members did not understand the distribution method of giving one LLIN to two persons per household according to the universal coverage strategy. Although this was explained to them and most of them understood, there were complaints by a few.
    • On the first day of the distribution, the DP attendants could not locate some few booklets. Apparently, these had been mistakenly taken to the district, making it difficult for some households to redeem their nets. Exercise books were procured to capture the information of those households as they presented their coupons to enable them redeem their nets. Arrangements were made to get the booklets from the region to the distribution point.
    • Some households who were not registered turned up at the distribution sites. For example, all the households in the Zabzugu police barracks were not registered due to misinformation from their command about the registration procedure (i.e., that they don’t need to register as the nets will be given to them without prior registration). All these households did not benefit from the nets because of the no coupon, no net policy.
    • At a distribution point in Zabzugu sub-district, DP attendants detected that some households altered the household numbers on their coupons, ostensibly to receive more nets. The DP attendants noted the right numbers on the coupon counterfoil (validated data) and gave out the correct number of nets to the households.
    • Some beneficiaries waited all day to receive their nets because the booklets from which they were issued coupons could not be found on the day of the distribution. They booklets were mistakenly sent to different distribution point but were eventually brought to the DP and the people received their nets."
      Episcopal Relief & Development Ghana Northern Region distribution report 2016, Pg 6.
  • Challenges encountered and actions taken by ADDRO monitors for other districts in the Northern Region reported in Episcopal Relief & Development Ghana Northern Region distribution report 2016, Pgs 6-10.
  • Challenges encountered and actions taken during distribution point monitoring for the distribution in Greater Accra reported in Episcopal Relief & Development Ghana Activity Report 2 2016, Pg 6.
  • The distribution reports we have seen so far from Episcopal Relief & Development do not cover challenges encountered and actions taken during distribution point monitoring for the Upper West Region distribution.

• 20
  • AMF page on Ntcheu 2012 distribution
  • AMF page on Balaka 2013 distribution
  • AMF page on Dedza 2014 distribution
  • AMF page on Dowa 2015 distribution

• 21
  • IMA World Health, Kasaï-Occidental 2014 distribution report
  • IMA World Health, Kasaï-Occidental 2014 technology report

• 22
  • Episcopal Relief & Development Ghana Northern Region distribution report 2016
  • Episcopal Relief & Development Ghana Activity Report 1 2016
  • Episcopal Relief & Development Ghana Activity Report 2 2016
  • Episcopal Relief & Development Ghana Activity Report 3 2016

• 23
  • Episcopal Relief & Development Ghana Activity Report 3 2016:
    • "The validation was to establish precisely how many LLINs were distributed during the distribution campaign in all 12 districts. Apart from validating counterfoils, the validators undertook 'End-User Verification'. The end-user verification (EUV) is a rapid check-up to determine whether the beneficiaries really received the number of LLINs allocated for the households and are using the LLINs for the intended purpose. This involved randomly selecting 100 households in each district to verify LLINs received, LLIN use by household members, etc. Validators randomly sampled 100 booklets and from each booklet, randomly sampled one coupon counterfoil for the EUV visit. Validators then called the beneficiaries of the sampled coupon counterfoils and followed up to the households for the end-user verification exercise." Pgs 4-5.
    • "24 NMCP validators were assigned to work in the 12 AMF supported districts (2 validators per district). Two teams from ADDRO (a team from the ADDRO HQ and a team from the ADDRO Greater Accra office) visited all the 12 AMF districts to monitor the validation process and to provide support for the packaging of coupon counterfoils for transportation to ADDRO headquarters in Bolgatanga. Key findings were as follows:
      • It was comparatively easier for the validators/monitoring team to enter into bedrooms/sleeping places of beneficiaries in the rural areas to inspect or observe net usage (LLINs hanging and being used) than it was in the urban areas. Residents in the urban areas felt very reluctant to allow ‘strangers’ to observe their sleeping places.
      • The Ashaiman and Ningo Prampram districts had significant numbers of nets not redeemed. For example, Ashaiman had 127 bales (12,700 pieces) at the District Health Directorate not distributed. The reasons given were that during the distribution in July, there was a LLIN shortage (because the districts had not received all their nets) so later when the districts finally had their nets, beneficiaries did not turn up to claim them. The GHS staff had called most of the beneficiaries per the contact numbers in the coupon counterfoils but only few came for the LLINs. This issue was reported to NMCP to take action."

      Pgs 4-5.

    • We have not seen comprehensive results from the post-distribution tracing exercises in Ghana (e.g., the proportion of selected households that had originally received the appropriate number of LLINs).
  • "In Greater Accra, post-distribution validation tracing was implemented for a random sample of households. The same process is planned for the Upper West Region. The Global Fund imposed this requirement." GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, pg. 10.

• 24
  • Episcopal Relief & Development Ghana Activity Report 3 2016:
    • "The validation was to establish precisely how many LLINs were distributed during the distribution campaign in all 12 districts. Apart from validating counterfoils, the validators undertook 'End-User Verification'. The end-user verification (EUV) is a rapid check-up to determine whether the beneficiaries really received the number of LLINs allocated for the households and are using the LLINs for the intended purpose. This involved randomly selecting 100 households in each district to verify LLINs received, LLIN use by household members, etc. Validators randomly sampled 100 booklets and from each booklet, randomly sampled one coupon counterfoil for the EUV visit. Validators then called the beneficiaries of the sampled coupon counterfoils and followed up to the households for the end-user verification exercise." Pgs 4-5.
    • "24 NMCP validators were assigned to work in the 12 AMF supported districts (2 validators per district). Two teams from ADDRO (a team from the ADDRO HQ and a team from the ADDRO Greater Accra office) visited all the 12 AMF districts to monitor the validation process and to provide support for the packaging of coupon counterfoils for transportation to ADDRO headquarters in Bolgatanga. Key findings were as follows:
      • It was comparatively easier for the validators/monitoring team to enter into bedrooms/sleeping places of beneficiaries in the rural areas to inspect or observe net usage (LLINs hanging and being used) than it was in the urban areas. Residents in the urban areas felt very reluctant to allow ‘strangers’ to observe their sleeping places.
      • The Ashaiman and Ningo Prampram districts had significant numbers of nets not redeemed. For example, Ashaiman had 127 bales (12,700 pieces) at the District Health Directorate not distributed. The reasons given were that during the distribution in July, there was a LLIN shortage (because the districts had not received all their nets) so later when the districts finally had their nets, beneficiaries did not turn up to claim them. The GHS staff had called most of the beneficiaries per the contact numbers in the coupon counterfoils but only few came for the LLINs. This issue was reported to NMCP to take action."

      Pgs 4-5.

  • "In Greater Accra, post-distribution validation tracing was implemented for a random sample of households. The same process is planned for the Upper West Region. The Global Fund imposed this requirement." GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, pg. 10.
  • AMF notes that: "Our understanding is this was a temporary logistical issue and we asked for this to be addressed in the next ERD report. We have now written to ERD and asked for a quicker, specific update on this issue." Comment provided in response to a draft version of this review in November 2016.

• 25
  • "Post-distribution Check-Ups (PDCUs) are carried out every 6 months for 2.5 years (so 5 checks) with 5% of households in each Health Area selected at random and visited unannounced. These data are put in electronic form." AMF Summary features of an AMF distribution
  • PDCUs every six months for 2.5 years are included in the distribution agreements AMF signed for its distributions in Ghana and Uganda. Ghana: AMF Ghana 2016 distribution agreement; Uganda: AMF Uganda 2016 distribution agreement
  • AMF used to require that distribution partners conduct follow-ups every 6 months for up to 3 or 4 years.
    • "[AMF:] Post-Distribution Surveys (see Note 3). Please confirm you will carry out Post-Distribution Surveys (PDSs) every 6 months post-distribution for a period of up to four years to assess the level of net usage (hang-up %), correct usage and condition of the nets and you will provide us with the findings. Each survey would cover approximately 5% of households.
      [Concern Universal:] I can confirm Concern Universal and Dedza District Council will conduct a Post-Distribution Survey every six months following the distribution." Concern Universal Balaka 2013 distribution proposal, pg. 2
    • Concern Universal Dedza 2014 distribution proposal, pg. 2 contains the same language.
    • IMA World Health agreed to conduct surveys every six months for 3 year post-distribution. IMA World Health and AMF distribution agreement for Kasaï-Occidental 2014, pgs. 3-4 and IMA World Health and AMF distribution agreement for Nord Ubangi 2015, pgs. 3-4.
  • Concern Universal uses the following procedure for its PDCUs:
    • From a list of all villages that received nets in a distribution, Concern Universal's project manager arbitrarily picks 5% of the villages to carry out PDCUs in. For each PDCU, the project manager tries to not select villages that are geographically close to each other or villages that have been visited in the previous PDCU. (Note that this selection procedure is not random and may be subject to bias.)
    • For each selected village, the project manager arbitrarily selects 25 households to be surveyed during the PDCU. (Note that this selection procedure does not appear to be random and may be subject to bias.)
    • On the dates of the PDCU, a supervisor or HSA from the local health facility accompanies the enumerator (a contracted employee of Concern Universal) to the village being surveyed. The health facility supervisor (or HSA) and enumerator meet with the village leader to discuss the purpose of their visit. The health facility staffer and enumerator arrive at the village unannounced to avoid knowledge of the PDCU influencing beneficiary behavior.
    • The village leader (or a representative of the village leader) then accompanies the enumerator and health facility staffer to each household selected for the PDCU. The village leader knows the location of each selected household, and introduces the household head to the PDCU team.
    • The household head gives consent for the PDCU team to enter the household and perform its check. If a household head refuses, the enumerator informs the project manager and the project manager selects another household. If the household head is away when the PDCU team arrives, the enumerator will return later in the day to perform the check. Household heads are frequently away when the PDCU team arrives, often because they are doing fieldwork far from their home.
    • During the PDCU check, the enumerator:
      • assesses the number of regularly used sleeping spaces.
      • asks how many people are in the household, and how many have had blood-diagnosed malaria in the past month.
      • checks the brand of the bed nets. Nets are recorded either as 'Permanet,' 'Olyset' (the two brands AMF distributes), or 'other.'
      • records whether the net was distributed by an AMF-funded distribution. AMF-funded nets have a tag with the AMF logo and the date of the distribution.
      • assesses the net condition. Nets are assessed as 'very good,' 'okay,' or 'poor.'
        • A net in very good condition has two or fewer holes of less than 2 cm.
        • A net in okay condition has fewer than 10 small holes (less than 2 cm).
        • A net in poor condition has more than 10 small holes, or one large hole (greater than 10 cm).

        Net quality is assessed by physical examination by the enumerator. If a net is not hung, the net is unfolded and physically assessed.

      • asks who sleeps under each net, categorized by children under 5, children ages 6-18, pregnant women, or other people.
      • asks how many nets were originally distributed to the household during the AMF distribution, and also notes how many of these nets are not present, or worn out.
      • records how many nets in the household are hung, and how many are not hung.
      • assesses whether nets are being used correctly. For this assessment, the enumerator makes a judgment call about whether the nets being are used properly. Often, nets will not be hung during the day due to limited space, but still may be used properly at night.

    Nelson Coelho, conversation with GiveWell, April 15, 2016

  • Episcopal Relief & Development agreed to carry out post-distribution check-up surveys for the three distributions in Ghana in 2016:
    • "Post-distribution Check-Ups (PDCUs) will be carried out every six months for 2.5 years with five PDCU to be conducted in total. With the PDCUs, all households will be visited by trained volunteers who will collect household data. A different set of volunteers from the ones who will collect data on all households will be trained to visit unannounced, 5% of households in each community and collect data." Episcopal Relief & Development Planning Document 2016, Pg 23.
  • AMF has told us that, for the first PDCU for the Northern Region in Ghana in November 2016, that there will be "5% of the 5%" data collection—a separate set of data collectors re-visiting a random selection of 5% of the households selected for the PDCU to re-collect data. AMF also told us that it expects to be able to quickly match the "5% of the 5%" data to the original PDCU data, and intends to use this check to assess the accuracy of PDCU data collection. Rob Mather, AMF Founder, email to GiveWell, October 31, 2016

• 26

  AMF page on Kasaï-Occidental 2014 distribution

• 27
  • We discuss case rate data from Ntcheu in our February 2013 update on AMF.
  • "[AMF:] Please confirm you are able to provide monthly malaria case rate data going back at least 12 months for each health centre/clinic in the distribution area and will continue to provide monthly data for a period of four years post‐distribution. This ensures we understand pre‐distribution malaria levels and can monitor them post‐distribution.
    [Concern Universal:] Malaria case rate information for each of the 16 Health Centres is available from the Balaka District Health Office using the Malawi Government’s national Health Management Information System (HMIS). This information will be sent to AMF separately." Concern Universal Balaka 2013 distribution proposal, pg. 2
  • Concern Universal Dedza 2014 distribution proposal, pg. 2, contains the same language as above

• 28
  • Balaka 2010-2015 MCRD
  • Dowa 2012-2015 MCRD
  • Ntcheu 2010-2015 MCRD
  • Dedza 2010-2015 MCRD

• 29

  AMF told us that the data is likely to be poor quality, and that it will put little to no weight on the results. Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015

• 30

  "After the HSAs have collected data from all villages, spot-checkers independently collect the same data from 5% of households in each village. The spot-checkers are employed by the government and receive lunch allowances paid by Concern Universal during the time they are engaged in the exercise. This data is later compared to the HSA data to check for consistency. The two sets of data are entered into a Microsoft Access database separately. Reconciliation of the data is done after data entry. The data is used to calculate the number of nets required in each village, health center catchment area, and district. A printed register including all information recorded during registration is created for each village." GiveWell's non-verbatim summary of a conversation with Nelson Coelho, April 15, 2016, pg. 3.

• 31

  Concern Universal Balaka 2015 registration data 100% vs 5%

• 32

  AMF told us that 105% registration data was collected for the Dowa 2015 and Ntcheu 2015 distributions. @Rob Mather, AMF Founder, conversation with GiveWell, September 28, 2016@

• 33

  @Rob Mather, AMF Founder, conversation with GiveWell, September 28, 2016@

• 34

  AMF funds status (November 2016) Anonymized

• 35
  • The original budget for this study was $800,000. Rob Mather, AMF Founder, conversation with GiveWell, February 24, 2015
  • AMF sent us a draft research proposal for the study in early 2015; some details are in our May 2015 update. AMF sent us what we believe was the final version of the research proposal in June 2015; we have not yet reviewed the final version. AMF insecticide research proposal from the London School of Tropical Medicine
  • "Given the nature of the Uganda PBO study we have just last week decided to not proceed with Phase 2 of the DRC study. The total cost of Phase 1 was £75,667. Phase 1 looked at establishing where resistant mosquitoes were present in Nord Ubangi (they are) and Phase 2 was to look at the effectiveness of the PBO nets." Comment provided in response to a draft version of this review in November 2016.
  • 1 GBP equals 1.25 USD. Google, November 21, 2016. 75,667 * 1.25 = $94,584.

• 36

  "We are close to concluding discussion with Concern Universal for AMF and CU to co-fund a Malaria Unit (a larger team of people than currently in place dedicated to malaria control related work) in Malawi led by a senior CU Manager and employing 12 Malawians (10 additional hires)." Rob Mather, AMF Founder, email to GiveWell, September 9, 2015

• 37

  "The aim of this unit will be to provide more resource to allow 1. improved efficiency in managing the four contracted three-yearly distributions and associated work and 2. allow us to together innovate and develop additional malaria control support for the four districts and the NMCP, specifically: i) intended close liaising with Health Centre re malaria case data and elements related to the monitoring/recording of malaria data i.e. stock levels of RDTKs, rubber gloves and other diagnosis equipment; qualified staff able to test for malaria, presence at the clinic so diagnosis can happen; systems, capacities etc) ii) research (data), discussion and involvement in ways of ensuring 80% sleeping space coverage throughout the three year net-distribution cycle, including investigation of ‘injection strategy’ net distributions involving mini-mass distributions at two years post-distribution and in the subsequent distribution cycle at one-year post distribution." Rob Mather, AMF Founder, email to GiveWell, September 9, 2015

• 38

  The distributions and post-distribution check-ups occur frequently enough that having a trained, consistent staff to manage them is worthwhile. Rob Mather and Peter Sherratt, conversation with GiveWell, September 9, 2015

• 39
  • $636,000 was allocated to the Malaria [Control] Unit in AMF funds status (June 2016), of which $100,000 was already paid. This allocation is about $200,000 higher than the amount AMF mentioned in our September 2015 conversation.
  • "This would be an initial three year commitment from AMF of US$413k…Savings made on non-net costs (with the Malaria Unit funded, some costs included in non-net cost budgets would already be covered) we estimate to be of the order of US$70k, and perhaps as high as US$95k per distribution of which there are four in a three year cycle i.e. US$280-380k in total... If all progresses well we may be able to publish plans in the next six weeks once budgets have been agreed and assuming approval from the Malawi NMCP. We do not anticipate problems, although there is a work visa issue to resolve." Rob Mather, AMF Founder, email to GiveWell, September 9, 2015
  • AMF Malaria Unit draft budget

• 40

  Over the last several years, AMF believes that its high net hang-up rates have begun to catch the attention of a number of groups in Malawi, including the NMCP, local organizations that distribute nets, and large international NGOs. AMF also told us that it encouraged Concern Universal to take a more active role in a task force on malaria control in Malawi. This meant that as interest in learning from AMF and Concern Universal grew, Concern Universal was available to explain AMF’s methodologies at the malaria task force meetings, and it was eventually decided to implement some of AMF’s practices for the upcoming distribution. Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015

• 41

  Rob Mather, AMF Founder, email to GiveWell, October 12, 2015
  

• 42

  "Although AMF won’t be managing this process, it is interested to observe how multiple districts will carry out distribution according to this particular data-driven approach. AMF believes that it will be allowed access to that data, either directly or via its in-country partners." Rob Mather and Peter Sherratt, conversation with GiveWell, February 6, 2015, pg. 2.

• 43

  For underlying calculations, see this spreadsheet. Sourced from AMF Financial information.

• 44

  Rob Mather, AMF Founder, conversation with GiveWell, July 19, 2012.

• 45
  • Rob Mather, AMF Founder, email to GiveWell, November 26, 2014.
  • AMF has paid for all non-net costs for the Balaka 2013, Dedza 2014, and Dowa 2015 distributions and has agreed to pay for all non-net costs for the upcoming Malawi distributions in 2015, 2016, and 2018. GiveWell summary of AMF large-scale distributions
  • It has also paid for the costs of the 24- and 33-month post-distribution check-ups for the Ntcheu 2012 distribution, and the shipping costs for the Nord Ubangi 2015 and flood-affected districts distributions.
    • AMF page on non-net costs
    • Rob Mather and Peter Sherratt, conversation with GiveWell, September 9, 2015
    • GiveWell summary of AMF large-scale distributions
  • For its 2016 Ghana distributions and its upcoming distributions in Uganda, Togo, and Papua New Guinea, AMF has agreed to pay for "non-standard" non-net costs, including PDCUs and, in Uganda, the cost of an "independent assessor [who will] determine whether nets are distributed to beneficiaries in the quantities listed." The Global Fund paid for the "standard" non-net costs for the Ghana distributions and the government is paying for these costs in Uganda. "Standard" costs include shipping to the country, clearance charges, in-country transportation, pre-distribution planning and registration, and distribution costs.
    • Rob Mather, AMF Founder, email to GiveWell, January 28, 2016
    • "The Ghana distribution is now going ahead: AMF is funding the costs of the nets and AMF’s additional monitoring costs, and the Global Fund will fund other non-net costs." Rob Mather and Peter Sherratt, conversation with GiveWell, February 11, 2016, pg. 2.
    • "The non-net costs will be funded by the Ghana Malaria Global Fund Grant. These costs include those for shipping to Ghana, clearance, in-country transport, pre-distribution, distribution." AMF Ghana 2016 distribution agreement, pg. 1.
    • "The non-net costs will be funded by the Uganda Ministry of Health which may use funding from its Roll Back Malaria Partners. These costs include those for shipping to Uganda, clearance, in-country transport, pre-distribution, and distribution." AMF Uganda 2016 distribution agreement, pg 2.
    • Discussion of independent assessor is from AMF Uganda 2016 distribution agreement, pg 4.
    • "Post-Distribution Check-Ups… AMF will fund an NGO to run the process." AMF Uganda 2016 distribution agreement, pg 5.
    • AMF Togo 2017 distribution agreement Redacted:
      • "1) AMF will fund 2,413,250 LLINs.
        2) The non-net costs will be funded by the MSPS which may use funding from The Global Fund or other sources." Pg 7.
      • "Post-Distribution Check-Ups (PDCUs) will take place across all of the districts to monitor net use and condition. The results will be owned by Togo and shared with AMF. AMF will fund an NGO to run the process in full consultation with the MSPS. The MSPS will facilitate the check-ups." Pg 10.
    • AMF Papua New Guinea 2017 distribution agreement Redacted:
      • "1) AMF will fund 1,159,400 extra-large LLINs for distribution in 2017.
        2) The costs, with the exception of the purchase of the LLINs, will be borne by RCPM which may use dedicated funding from The Global Fund and other sources." Pg 1.
      • "Post-Distribution Check-Ups (PDCUs) to assess correct net distribution, net use and condition of LLINs will take place across all of the districts to monitor net use and condition. The results will be owned by PNG but will be shared with AMF. AMF will fund a yet-to-be-decided NGO to run the process in full consultation with the NDoH and RCPM. RCPM will facilitate the check-ups." Pg 4.

• 46

  "AMF funding non-net costs can remove the final barrier to a distribution proceeding and avoid potential lengthy delays, of up to a year, given the ideal timing of a distribution relative to other events, for example a rainy season." AMF page on non-net costs

• 47

  See our summary of post-distribution check-up (PDCU) results (the follow-up surveys) here.

• 48

  See this blog post for details.

• 49

  See our summary of AMF distributions spreadsheet, "Overview" sheet, "Allocation strategy" column.

• 50

  "HSAs introduce themselves to the head of household (or a representative thereof) and explain that they are there to assess the number of nets required in that household. After obtaining permission to enter the household, the HSA gathers the following information:
  Number of people over and under 5 years old in the household
  […]
  "The HSA determines the number of sleeping spaces by talking to household members and personally looking at sleeping spaces. The village head sometimes verifies whether the number of household members reported by the beneficiary is correct.
  […]
  "If a household owns nets, the HSA inspects them to determine whether they are in viable condition." GiveWell's non-verbatim summary of a conversation with Nelson Coelho, April 15, 2016, pg. 2.

• 51

  "Verification
  Concern Universal staff gather 4-5 villages in each catchment area at a central location, and use the registers to take roll call of the household names and the number of nets each household is registered to receive. People, households, or villages who claim to have been skipped during registration are added to the register if local leaders confirm these claims. Verification is conducted for all villages." GiveWell's non-verbatim summary of a conversation with Nelson Coelho, April 15, 2016, pg. 3.

• 52
  • A representative of Concern Universal told us that Concern Universal performs this data verification for its distributions in Malawi. See this footnote for detail. Nelson Coelho, conversation with GiveWell, April 15, 2016
  • @GiveWell Notes from Site Visit with Concern Universal in Malawi (October 2011)@

• 53

  
  For Malawi, Concern Universal have described the following pre-distribution registration survey (PDRS) process to us:

  • Concern Universal collects demographic data from the district health office, including the number of households in the region and the number of people per household.
  • Concern Universal briefs the HSAs, who will do the door-to-door surveying, about the PDRS.
  • There are no detailed maps or geographical information system data for rural Malawi, so Concern Universal and HSAs rely on traditional authorities and community leaders to locate all the households in the targeted villages.
  • HSAs go door-to-door to register all households in the targeted villages. A village leader accompanies the HSAs during this process. During household registration:
    • The village leader and HSAs introduce themselves to the head of the household and explain why they are there.
    • The has obtains permission to enter the household from the household head. (The difficulty of obtaining this permission varies based on the local culture. Local leaders can lend credibility to thasHSA when asking for permission to enter a household.)
    • Once in the householdhashe HSA collects information including the name of the head of the household, the number of over-5 and under-5 people in the household, the number of sleeping spaces, and the number of viable nets owned by the househohas (The HSA asks household members for demographic information, and assesses the number of sleeping spaces and number of nets via visual inspection; 'viable' nets are nets that are less than 18 months old with fewer than 10 small holes and no large holes.)
    • Concern Universal conducts "105%" data verification during the PDRS. HSAs are responsible for registering 100% of households in a village. In addition, supervisors independently register 5% of households in each village during the PDRS. This data is later compared to the HSA-collected data to check for consistency. Note that in September 2016, AMF told us that this piece of the process has not been emphasized and may not be happening consistently. We have seen only limited data from the data verification step from the Balaka 2015 distribution.
      • @Rob Mather, AMF Founder, conversation with GiveWell, September 28, 2016@
      • Concern Universal Balaka 2015 registration data 100% vs 5%

    Concern Universal staff and government health officials supervise the registration.

  • The data from the registration is entered into a Microsoft Access database, then printed out into registration books.
  • After the registration, Concern Universal performs a data verification by assembling a few villages in each catchment area and taking a roll call of the household names in the registration book, and the number of nets that household is scheduled to receive. Verification is conducted for all villages. If someone claims to have been missed during this process, or claims that their number of scheduled nets is incorrect, then this is confirmed by a village leader, and, if confirmed, the correction is made.

  Nelson Coelho, conversation with GiveWell, April 15, 2016

• 54
  • Concern Universal Ntcheu 2012 pre-distribution registration survey data
  • Concern Universal Balaka 2013 pre-distribution registration survey data
  • Concern Universal Dedza 2014 pre-distribution registration survey data
  • Concern Universal Dowa 2015 pre-distribution registration survey data
  • Concern Universal Balaka 2015 registration data 100% vs 5%
  • Concern Universal Ntcheu 2015 pre-distribution registration survey data

• 55

  IMA World Health, Kasaï-Occidental 2014 distribution data

• 56
  • "IMA obtained population data for the year 2012 from the Ministry of Health and applied the MOH’s estimate for population growth of 3% per annum." IMA World Health, Kasaï-Occidental 2014 distribution report, pg. 6.
  • "There were many occasions where the number of nets needed for a community was overestimated. This was discovered in the registration and hang up because of diligent work done. Lesson learned[:] Better population estimates are needed and 1.8 people per nets should be the ratio always used." IMA World Health, Kasaï-Occidental 2014 distribution report, pg. 46.
  • In cases where too few nets are sent to a location, AMF's distribution partner typically performs a "mop up" exercise in which it orders extra nets and distributes them several months later to the households that didn't receive the nets the first time around. Rob Mather, AMF Founder, conversation with GiveWell, February 24, 2015
  • "If there are a significant number of nets left over – more than a few hundred for example, the nets are moved to the next distribution. An example would be the 5,770 so nets left over in Balaka (the need was assessed as 149,500 and we had moved in 160,000 nets to ensure we had enough nets to cover any additional need and the final number of nets distributed after mop-up was 154,230) which were moved up to Dedza and distributed as part of that distribution. The number of nets ordered and moved in to Dedza reflected that 5,700. If the number of nets left over is in the hundreds, they are passed to the District Health Officer (DHO) to be included a nets for routine distribution. Sometimes there has been a small, several hundred net shortfall and that gap has in the past been filled by nets in zone from other sources." Rob Mather, AMF Founder, email to GiveWell, September 16, 2015

• 57

  "Differences [between the Nord Ubangi distribution and the Kasaï-Occidental 2014 distribution]
  1. The North Ubangi distribution will be a two-phase distribution – a distinct pre-distribution phase and a subsequent distribution phase. In West Kasaï establishing net need per household and distributing of nets was contemporaneous.
  2. Distribution of nets will be done from distribution points rather than at the household
  3. The ODK registration form has been re-designed to better capture the required data, to eliminate some types of error and to reduce the chance of others.
  4. Operational lessons learned from the West Kasaï distribution will be implemented.
  5. An experienced, full time technology manager has been employed by IMA meaning there will be more extensive and easier liaising with AMF on data elements of the distribution. We have met the new technology manger and have spent a week working with him and are impressed.

  Similarities [between the Nord Ubangi distribution and the Kasaï-Occidental 2014 distribution]
  1. The same smartphone-based data collection technology will be used as in West Kasaï.

  Comment
  IMA has agreed there are benefits to conducting a separate, earlier pre-distribution phase during which accurate household-level net need is established. IMA are less able to estimate accurately population and net need in North Ubangi compared to what they felt was possible in West Kasaï.

  IMA believes point distributions rather than at-household distribution will be more cost-effective." Rob Mather, AMF Founder, email to GiveWell, September 9, 2015

• 58
  • IMA World Health Nord Ubangi 2015-16 registration data:
    • Summary data reported for "Nbre de menages," "Population dans menage," "Nbre places a dormir," "Nbre bonne MILD," "Nbre MILD installees," and "Couverture."
    • Translated to English, from Google Translate: "No. of households," "Population in household," "No. of places to sleep," "No. good MILD [LLIN]," "No. MILD [LLIN] be built [installed, or hung]," and "Coverage."

• 59

  IMA World Health Nord Ubangi 2015-16 registration data

• 60

  "In 2010, Senegal and Cross River State in Nigeria worked on mop-up campaigns following earlier integrated campaigns targeting households with children under five years of age. In both countries, trained health workers or volunteers undertook a household registration to determine:

  • the total net need for each household (this was based on one LLIN for two people, rounding up in the case of odd numbers of people in the household, but Senegal also looked at the number of habitual sleeping spaces in each household)
  • how many nets each household already had
  • how many nets in each household were still viable (in Senegal this number was based on net condition, while in Cross River State, it was based on how long the net had been hanging)
  • how many new nets each household would need for full coverage

  Both countries had previously undertaken post-distribution surveys which showed high household coverage with LLINs, but during the mop-up exercise they experienced challenges with finding nets in households. Significantly lower numbers of nets were found (50-60 per cent) than would have been expected based on the surveys…
  In both countries, it seemed that families often hid nets once word spread that ownership of nets meant no new nets would be received. Despite efforts to encourage families to hang pre-existing nets prior to the household registration in Senegal, people hid nets in order to receive more." Alliance for Malaria Prevention Toolkit (version 2.0) - Chapter 3, pgs. 5-6.

• 61

  GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016:

  • "GHS [Ghana Health Services] volunteers carry out the registration process, which takes place more than one month before the distribution. Volunteers are organized by the Ministry of Health (MoH) and participate in a number of government health programs, such as vaccination campaigns (immunization days). Many of them are long-time volunteers with considerable experience." Pg. 5.
  • "Volunteers visit each house in their zone. The head of household, or someone else who is present, is asked how many people are in the household based on the 'people who eat from the same pot' definition. Ghana's policy is to provide one net for every two individuals in a household; as a result, in contrast to previous AMF distributions, volunteers were not required to check households' existing nets or number of sleeping spaces. The only exception to this policy is that household members aged 70 or older are not counted in the household total, and each receive their own net." Pg. 5.

• 62
  • See our summary of AMF distributions spreadsheet, "Overview" sheet, "Allocation strategy" column.
  • GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016:
    • "GHS [Ghana Health Services] volunteers carry out the registration process, which takes place more than one month before the distribution. Volunteers are organized by the Ministry of Health (MoH) and participate in a number of government health programs, such as vaccination campaigns (immunization days). Many of them are long-time volunteers with considerable experience." Pg. 5.
    • "Volunteers visit each house in their zone. The head of household, or someone else who is present, is asked how many people are in the household based on the 'people who eat from the same pot' definition. Ghana's policy is to provide one net for every two individuals in a household; as a result, in contrast to previous AMF distributions, volunteers were not required to check households' existing nets or number of sleeping spaces. The only exception to this policy is that household members aged 70 or older are not counted in the household total, and each receive their own net." Pg. 5.

• 63
  • "In continuous distributions, nets are delivered through antenatal clinics, child welfare clinics, or schools." GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, pg. 3.
  • During GiveWell's site visit to Greater Accra, Ghana in August 2016, some residents of areas targeted for LLIN distribution that we met with told us that they had owned some LLINs prior to the recent AMF distribution, including LLINs obtained by purchasing them from a market and LLINs obtained through antenatal clinics. See our rough notes from household visits in this spreadsheet, column Q, and a description of our selection process for household visits in Greater Accra in GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, Pgs 15-18.

• 64
  • "The last national distribution in Ghana occurred between 2010 and 2013." GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, pg. 4.
  • During GiveWell's site visit to Greater Accra, Ghana in August 2016, some residents of areas targeted for LLIN distribution that we met with told us that they last received LLINs from a mass distribution four years ago. See our rough notes from household visits in this spreadsheet, column Q, and a description of our selection process for household visits in Greater Accra in GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, Pgs 15-18.
  • LLINs are distributed to different regions of Ghana on a rolling basis (i.e., distributions occur in one region one month and in another region the next month). Ghana net gap and schedule 2016
  • We have not seen the schedule for the 2010 to 2013 distributions in Ghana, but we would guess that the last LLIN distribution in the Upper West Region and the Northern Region was also in 2012, since these distributions were scheduled around the same time in 2016.
  • The most common "decay model" we have seen estimates that around 50% of LLINs are still in use in the third year after distribution.

• 65
  • AMF Ghana Data Entry System 2016 (for the Northern and Greater Accra regions).
  • We have not yet seen complete registration data from the Upper West Region distribution, but have seen a summary of this data AMF Upper West Region Ghana pre-validation registration data 2016
  • AMF told us that it and the government decided that the collection of information on previously-owned LLINs was not necessary for this distribution. Comment provided in response to a draft version of this review in November 2016.

• 66
  • Rob Mather, AMF Founder, email to GiveWell, October 10, 2016
  • See our summary of AMF distributions spreadsheet, "Overview" sheet, column M.

• 67

  "[…} there are two reasons for this: where a previous distribution was at least three years earlier, the number of ‘perfectly usable nets’ in situ is likely to be low, given the lifetime of the nets, making the question of limited use so ‘not critical’; nets with some useful life left in them are likely to need replacing before the next distribution campaign so the additional nets will be of protective value before then." Comment provided in response to a draft version of this review in November 2016.

• 68
  • Rob Mather, AMF Founder, email to GiveWell, October 10, 2016
  • See our summary of AMF distributions spreadsheet, "Overview" sheet, column M.

• 69
  • "Continued use of nets is very important. Every six months, a post-distribution survey is carried out to assess net usage and net condition. Approximately 5% of the nets distributed are assessed through visits to randomly selected households. The data collected are used to determine if additional community-level malaria education activities are required. All data are published." AMF information we publish
  • From a recent distribution agreement:
    • "Post-Distribution Check-Ups (PDCUs) will take place across all of the districts to monitor net use and condition.
      […]
      A PDCU is carried out every 6-months for two and a half years' post-distribution therefore at 6, 12, 18, 24 and 30-months post-distribution." AMF Togo 2017 distribution agreement Redacted, Pg 4, English version (Pg 10 in PDF).
  • In older distribution agreements it appears that AMF planned to require PDCUs for longer than 2.5 years following a distribution:
    • "Please confirm you will carry out Post‐Distribution Surveys (PDSs) every 6 months post‐distribution for a period of up to four years to assess the level of net usage (hang‐up %), correct usage and condition of the nets and you will provide us with the findings. Each survey would cover approximately 5% of households." Concern Universal Dedza 2014 distribution proposal, pg. 2.

• 70
  • LLINs are categorized as "Hung," "Present but not hung," "Missing," or "Worn out / not usable." See, for example, Ntcheu 2016 6-month post-distribution check-up data
  • A Concern Universal representative told us that PDCU enumerators assess whether the net is an AMF net by looking at the net's label. Nets sourced from AMF distributions have AMF's logo and the month and year of the intended distribution date on the label. Nelson Coelho, conversation with GiveWell, April 15, 2016
  • In its report on the 6-month PDCU for the Ntcheu 2015 distribution, AMF notes that the rate of LLINs from the recent distribution hung is lower than for other distributions, and discusses a few possible explanations for the low rate:
    • "This is the second universal coverage distribution in Ntcheu, with the first taking place in December 2012, a little over four years before."
      […]
      Initial hypotheses for data from this PDCU-06:
      Hypothesis 1: Timing differences between the 2012 PDCU-06 and the 2016 PDCU-06 mean there is a seasonal variation (e.g. linked to mosquito levels and average temperatures)
      Hypothesis 2: The previously distributed nets have lasted beyond the normal three years life and some of the new nets are being held in reserve to replace them when worn out.
      Hypothesis 3: Our criteria for assessing, prior to a mass distribution, which nets are ‘perfectly usable’ (have at least 18 months of life left) is too strict and materially more nets than we are judging to be so have extended life in them." Ntcheu 2015 6-month post-distribution check-up report, Pg 4.

• 71

  IMA World Health, Kasaï-Occidental 2014 distribution report:

  • "1. Household registration, handing out and hanging nets (3 days)" Pg 16.
  • "CHWs take the time to introduce themselves to the head of household and explain the reason for their visit in order to gain the householders’ permission to collect the required household information, distribute and hang the nets." Pg 17.

• 72
  • IMA World Health Nord Ubangi 2015-16 registration data:
    • Summary data reported for "Nbre de menages," "Population dans menage," "Nbre places a dormir," "Nbre bonne MILD," "Nbre MILD installees," and "Couverture."
    • Translated to English, from Google Translate: "No. of households," "Population in household," "No. of places to sleep," "No. good MILD [LLIN]," "No. MILD [LLIN] be built [installed, or hung]," and "Coverage."
    • Household-level data for these categories also included.

• 73
  • "Distributors make sure the recipients' coupons (brought by recipients) match the registration copies (brought by distributors) and make a mark on each side. The recipient's coupon is returned to them. Distributors also have a tally sheet with rows of 5 bubbles, and mark off a bubble for every net they give out." GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, pg. 9.
  • AMF has shared all registration data collected from its Northern Region and Greater Accra distributions, and a summary of registration data from its Upper West distribution (which has not yet been validated, see here) with us.
    • AMF Ghana Data Entry System 2016 (for the Northern and Greater Accra regions).
    • AMF Upper West Region Ghana pre-validation registration data 2016
    • The full registration data we have seen in AMF Ghana Data Entry System 2016 includes data on "# LLINs allocated" and "# LLINs given."

• 74
  • AMF page on Ntcheu 2012 distribution
  • AMF page on Balaka 2013 distribution
  • AMF page on Dedza 2014 distribution
  • AMF page on Dowa 2015 distribution

• 75

  See our March 2012 update on AMF.

• 76

  All weekly reports at AMF page on Balaka 2013 distribution. Examples of problems reported:

  • "The exercise despite the verification process and data cleaning faced some duplicates that were discovered during the distributions. The duplicate situation had not been dealt with during data cleaning as it was deemed only when both villages were asked to collect nets from the same distribution point that it became clear that there had been duplicate entries. However, different distribution points at the same time though at the same cluster makes it practically impossible for villagers to collect two nets from different sites by double registering." Concern Universal Balaka 2013 week 1 report, pg. 1.
  • "There was a high number of absenteeism during the urban distribution, which resulted in some members receiving the nets on behalf of others. This absenteeism was explained by the unavailability of the beneficiaries due to professional reasons. There were some complaints from beneficiaries whose nets were received by representatives claiming that the nets weren’t handed over. Handing over of nets to representatives was cancelled since it was clear that, unlike in rural areas where all community members know each other and certify the representative’s identification and the nets were handed over to the legitimate beneficiaries, nets were being misappropriated... For the urban distributions we anticipate to conduct them during the weekend to assure that most of the household owners are free from their daily work related activities." Concern Universal Balaka 2013 week 5 report, pgs. 1-2.
  • "One health worker assigned to facilitate the distribution process, in the community under his supervision, was caught by the beneficiaries trying to steal about 25 nets. He was reported to the authorities and discharged of his duties. This episode disrupted the distribution process and CU staff had to intervene to keep the population from beating the health worker." Concern Universal Balaka 2013 week 5 report, pg. 2.

• 77

  All weekly reports at AMF page on Dedza 2014 distribution. Examples of problems reported:

  • "The major challenge encountered during this week’s distributions was the misplacing of villages in clusters, which required us to the transfer the nets and distribution registers to the clusters where the villages have presented themselves. This delayed our distribution process but we still managed to reach and carry out the distributions to the affected villages." Concern Universal Dedza 2014 week 1 report, pg. 2.
  • "There were 20 villages under Kaphuka that did not receive the nets because of poor communication as their HSAs were attending performance appraisals and failed to communicate the distribution dates to their respective villagers." Concern Universal Dedza 2014 week 3 report, pg. 1.
  • "The major challenge during the week was duplication of registration in a way that some beneficiaries seem to have been deliberately registered in more than one village. The registration data was corrected and the affected beneficiaries only received the nets they were entitled to according their respective village data." Concern Universal Dedza 2014 week 3 report, pg. 2.

  AMF notes these issue and an additional issue on its blog, "An isolated incident of 300 nets missing from one storage location. This is being investigated and pursued with the police as any nets missing is taken very seriously. 300 nets represents 0.12% of the total nets being distributed." AMF: "Mid-distribution weekly reports for Dedza distribution, Malawi"

• 78

  "Mid-distribution Weekly Reports" at AMF page on Dowa 2015 distribution cover the first 184,554 nets distributed out of a total of 396,900 nets. Examples of problems reported:

  • "The major challenge we encountered during the distribution was the breaking down of our distribution vehicles, which forced us to interrupt the distribution for almost one month to have them fixed, as some spare parts could not be sourced locally." Concern Universal Dowa 2015 weeks 1-3 report, pg. 2.
  • "Some beneficiaries didn’t show up as they were attending a clothing items distribution on the same day. The fact that most inhabitants of the said villages are refugees from Rwanda, Burundi or Somalia caused identification challenges, preventing individuals claiming nets on behalf of the beneficiaries from receiving the nets. These beneficiaries will be considered during the mop up exercise." Concern Universal Dowa 2015 weeks 1-3 report, pg. 3.
  • "Distributions were not conducted in ten of the planned villages under the three health facilities due to funerals hence we deferred distribution and managed to reach ... 493 of the 503 planned villages. However, arrangements will be made at a later date when we will reach them and conclude the distributions in the deferred villages hence their nets have been currently taken back to the warehouse for safe keeping. The above mentioned villages are: Msaka, Kancheri, Chimbalanga Mononga and Nkhota villages from Mtengowanthenga health facility with, respectively, 20, 100, 201, 91 and 49 nets returned; Masiya, Sintala 2, Mulode 2 and Mgoli from Dzoole health facility with a nets requirement of 42, 274, 26 and 84 nets respectively and Mphinda village under Kayembe health facility with a requirement of 140 nets." Concern Universal Dowa 2015 weeks 1-3 report, pg. 5.

• 79

  “AMF has provided regular, public updates on the large, ongoing net distribution in the Ntcheu district of Malawi. Expected data collection has occurred and the distribution has proceeded close to schedule. AMF's distribution partner, Concern Universal, has been transparent about problems it has encountered, and seems to have a robust process to catch problems (such as attempts to steal nets) when they arise.” See our March 2012 update on AMF.

• 80
  • IMA World Health, Kasaï-Occidental 2014 distribution report
  • IMA World Health, Kasaï-Occidental 2014 technology report

• 81
  • Episcopal Relief & Development Pre-distribution Report Ghana Northern Region June 2016
  • Episcopal Relief & Development Ghana Activity Report 1 2016
  • Episcopal Relief & Development Ghana Activity Report 2 2016
  • Episcopal Relief & Development Ghana Activity Report 3 2016
  • Episcopal Relief & Development Ghana Northern Region distribution report 2016

• 82

  Examples of problems encountered:

  • Episcopal Relief & Development Pre-distribution Report Ghana Northern Region June 2016:
    • "Some challenges noted [in pre-distribution activities in the Northern Region] include:
      • Late submission of summary of sub-districts data to district for collation
      • Large numbers of booklets to validate
      • Poor telephone network making it difficult to reach some of the volunteers who had issues with some coupons and needed to get to the sub-district to clarify and if necessary go back to the households to make corrections to the registration or to re-register.
      • Difficulty in reaching Mankarigu, a hard-to-reach sub-district across the White Volta River. The team had to travel across five districts to get to Mankarigu.

      Some actions taken to resolve some of the above challenges included:

      • With the delay in the submission of sub-districts data, the team continued to move to the sub-district and start the validation of the coupons whilst still waiting for the summary of sub-districts data.
      • The validation team (including ADDRO staff) visited the communities to ascertain the veracity of the information captured on the coupons especially household sizes ranging from 15 to 20
      • The households with large household sizes were re-registered to reduce the sizes of the households and the earlier coupons issued were retrieved and replaced with new coupons." Pgs 14-15.
  • Episcopal Relief & Development Ghana Activity Report 1 2016:
    • "The distribution activity itself has taken place in the Northern region and reportedly went well. While complete details will be presented in the September Distribution Report, some initial, key observations made during the monitoring of the LLIN point distribution exercise included the following:
      • Some distribution points had issues with crowd control but were able to resolve them by forming and maintaining queues
      • Some registered beneficiaries rejected nets given to them on the basis that each household member should be given one net. Although the universal coverage strategy was explained to the crowds some still rejected the LLINs given to them.
      • Other households who missed out on the registration exercise turned up at some distribution points demanding nets but were not able to be served due to the protocols of pre-registration.
      • A few household members discovered the distribution strategy (universal coverage formula) and changed their household sizes/numbers on the coupon ostensibly to receive more nets than originally allocated so each HH member gets a net. However, this was easily found out and corrected as the distribution point attendants checked the counterfoils with the coupons submitted by the households." Pg 7.
  • Episcopal Relief & Development Ghana Activity Report 3 2016:
    • "The Ashaiman and Ningo Prampram districts had significant numbers of nets not redeemed. For example, Ashaiman had 127 bales (12,700 pieces) at the District Health Directorate not distributed. The reasons given were that during the distribution in July, there was a LLIN shortage (because the districts had not received all their nets) so later when the districts finally had their nets, beneficiaries did not turn up to claim them. The GHS staff had called most of the beneficiaries per the contact numbers in the coupon counterfoils but only few came for the LLINs. This issue was reported to NMCP to take action." Pg 5.

• 83

  "Our distribution of 676,000 nets in Kasaï-Occidental in partnership with IMA World Health (IMA) is our first one using smartphone technology for data collection. We see this as an exciting development with significant potential benefits including:

  • Acts against potential theft
  • Improved accountability
  • Greater transparency
  • Greater data accuracy
  • Improved cost effectiveness
  • Additional data can be collected
  • Reduced operational risk

  The use of this technology may become a significant determinant of future net distributions that we fund. We will report publicly on our experience with the Kasaï-Occidental 2014 distribution and the data gathered…
  GPS information can also be gathered helping to locate households and tie the number of nets delivered to each." AMF: "Introduction of smartphone technology to collect distribution data"

• 84

  IMA World Health, Kasaï-Occidental 2014 distribution data

• 85

  IMA World Health Nord Ubangi 2015-16 registration data

• 86
  • Episcopal Relief & Development Ghana Activity Report 3 2016:
    • "The validation was to establish precisely how many LLINs were distributed during the distribution campaign in all 12 districts. Apart from validating counterfoils, the validators undertook 'End-User Verification'. The end-user verification (EUV) is a rapid check-up to determine whether the beneficiaries really received the number of LLINs allocated for the households and are using the LLINs for the intended purpose. This involved randomly selecting 100 households in each district to verify LLINs received, LLIN use by household members, etc. Validators randomly sampled 100 booklets and from each booklet, randomly sampled one coupon counterfoil for the EUV visit. Validators then called the beneficiaries of the sampled coupon counterfoils and followed up to the households for the end-user verification exercise." Pg 4-5.
    • "24 NMCP validators were assigned to work in the 12 AMF supported districts (2 validators per district). Two teams from ADDRO (a team from the ADDRO HQ and a team from the ADDRO Greater Accra office) visited all the 12 AMF districts to monitor the validation process and to provide support for the packaging of coupon counterfoils for transportation to ADDRO headquarters in Bolgatanga. Key findings were as follows:
      • It was comparatively easier for the validators/monitoring team to enter into bedrooms/sleeping places of beneficiaries in the rural areas to inspect or observe net usage (LLINs hanging and being used) than it was in the urban areas. Residents in the urban areas felt very reluctant to allow ‘strangers’ to observe their sleeping places.
      • The Ashaiman and Ningo Prampram districts had significant numbers of nets not redeemed. For example, Ashaiman had 127 bales (12,700 pieces) at the District Health Directorate not distributed. The reasons given were that during the distribution in July, there was a LLIN shortage (because the districts had not received all their nets) so later when the districts finally had their nets, beneficiaries did not turn up to claim them. The GHS staff had called most of the beneficiaries per the contact numbers in the coupon counterfoils but only few came for the LLINs. This issue was reported to NMCP to take action."

      Pgs 4-5.

  • "In the Greater Accra, post-distribution validation tracing was implemented for a random sample of households. The same process is planned for the Upper West Region. The Global Fund imposed this requirement." GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, pg. 10.

• 87

  Comment provided in response to a draft version of this review in November 2016.

• 88

  AMF lists the countries it has provided nets to at @AMF Countries Involved@. The Malaria Atlas Project has compiled data on malaria risk by location at Malaria Atlas Project Endemic countries.

• 89

  AMF Distributions

• 90
  • WHO 2014 Malaria World Report, pg. 37, Figures 8.6 and 8.7.
  • An additional source of data on malaria deaths in these countries is the Institute for Health Metrics and Evaluation's (IHME) Global Burden of Disease tool. IHME Global Burden of Disease tool:
    • 95.3 malaria deaths per 100,000 people in DRC in 2015 (95% confidence interval: 62.42 to 135.06).
    • 68.61 malaria deaths per 100,000 people in Malawi in 2015 (95% confidence interval: 41.28 to 103.36).
    • 52.81 malaria deaths per 100,000 people in Ghana in 2015 (95% confidence interval: 28.6 to 81.65)
    • 41.5 malaria deaths per 100,000 people in Uganda in 2015 (95% confidence interval: 18.99 to 68.58).
    • 99.27 malaria deaths per 100,000 people in Togo in 2015 (95% confidence interval: 70.93 to 132.55).
    • 15.43 malaria deaths per 100,000 people in Papua New Guinea in 2015 (95% confidence interval: 9.44 to 24.59).

• 91
  • "Continued use of nets is very important. Every six months, a post-distribution survey is carried out to assess net usage and net condition. Approximately 5% of the nets distributed are assessed through visits to randomly selected households. The data collected are used to determine if additional community-level malaria education activities are required. All data are published." AMF information we publish
  • Example from a distribution proposal: "Please confirm you will carry out Post‐Distribution Surveys (PDSs) every 6 months post‐distribution for a period of up to four years to assess the level of net usage (hang‐up %), correct usage and condition of the nets and you will provide us with the findings. Each survey would cover approximately 5% of households." Concern Universal Dedza 2014 distribution proposal, pg. 2.
  • AMF has to be flexible about the timing of PDCUs because often there are delays that cannot be prevented. If a PDCU is delayed, the next PDCU will occur on schedule (e.g. even though the Dedza 2014 distribution's first PDCU was done at 8 months, the next distribution will occur at 12 months), unless the delay was severe enough that sticking to the schedule would not make sense (as happened with the Ntcheu 2012 PDCUs). Although AMF originally planned to request PDCUs for four years after a distribution, this doesn't make sense in some contexts where it is required that a mass distribution of LLINs occur at least every 3 years (such as Malawi). Rob Mather and Peter Sherratt, conversation with GiveWell, September 9, 2015

• 92

  See our summary of AMF distributions spreadhseet, "PDCUs" sheet.

• 93

  No report, only data, is available for the 19-month Balaka 2013 PDCU, and we do not know if this data is complete.

• 94

  No report, only data, is available for the 18-month Dedza 2014 PDCU, and we do not know if this data is complete.

• 95
  • AMF told us that it would send reports on the PDCU following the emergency distribution in Malawi in October 2016, but, as of November 2016, we have not yet received any reports from AMF. Rob Mather, email to GiveWell, October 5, 2016
  • We are uncertain whether we should have expected to see results from the first PDCU following to Balaka 2015 distribution by now (November 2016), since we are not sure whether the PDCU was intended to be scheduled six months after the original distribution or six months after the completion of the mop-up distribution:
    • "The Balaka 2015 distribution was conducted from January to March 2016. A mop up distribution was carried out in June 2016." Rob Mather, AMF Founder, email to GiveWell, October 10, 2016

• 96
  See our summary of AMF distributions spreadhseet, "PDCUs" sheet

• 97

  The methodology used in the Balaka 2013 6-month follow up survey seems similar to that used in the Ntcheu surveys discussed in our August 2014 update on AMF. Specifics from Concern Universal Balaka 2013 6-month post-distribution check-up report:

  • Sample selection: "Balaka district has 14 health facilities. It was decided to collect data from 5% of households in each HCCA [health center catchment area], which meant a different number of households in each HCCA as per individual health facility populations. Between 25 and 90 households were randomly selected from each of the selected seven to ten villages, depending on the HCCA, with the villages also selected at random. Villages were randomly selected using the village lists generated from the pre-distribution and distribution work for the October 2013 AMF-funded universal coverage LLIN distribution. A random number table was used to select the villages. Households were randomly selected using the household lists produced during the same campaign. A random number table was used to select the households. Five more households were put on reserve in case no one was at home in the selected households." Pg. 5.
  • Who carried out the survey: "10 data collectors and 2 supervisors from the District Health Office were involved in the PDCU. The supervisors were responsible for checking the data collection exercise at the same time monitoring how the data was being collected as per requirement." Pg. 5. Previous reports were more specific about who the data collectors were (for example, in the Ntcheu 24-month survey they were Health Surveillance Assistants, Malawi's version of community health workers).
  • Data quality audits: "Supervisors were required to visit 5% of the households in their area to check the accuracy of the data collectors’ work and had to check all the completed forms submitted to them before submitting them to the Project Manager. The sampled visited households were also chosen at random so the work of all data collectors was checked." Pg. 6.
  • Cost: "The PDCU cost was US$8,060 equal to US$1.78 per household visited (or $0.052 per net originally distributed which numbered 154,230)." Pg. 3.
  • Data entry/cleaning: "Improvements in the data entry interface since the last PDCU carried out in Ntcheu (Ntcheu PDCU-24) by AMF meant the data entry proceeded with almost no errors. This reduced the error-checking phase to almost nothing." Pg. 6.

• 98
  • "For those distributions for which registration data is entered into AMF’s Data Entry System (DES), we are now able to generate the random selection of households from the UK, which eliminates some of the potential issues that can arise in-country. Natalie and Andrew [of GiveWell] questioned the process in Malawi, and we agree it is important to make the process as truly random as possible. Randomising the selection ourselves from the database also saves expense in country, as well as making it easier to audit."
    […]
    "Togo and Papua New Guinea will be using AMF’s DES for all household pre-distribution registration data, country-wide, so we expect to replicate this approach across the PDCUs associated with these distributions."
    Rob Mather, AMF founder, email to GiveWell, October 24, 2016
  • AMF has used its Data Entry System for the 2016 distributions in Ghana. AMF Ghana Data Entry System 2016

• 99

  The type of technical information we would ideally like to see includes:

  • Results of audits on data quality.
  • Analysis of the extent to which respondent selection was carried out according to the protocol.
  • Analysis of the extent to which selected households were replaced with pre-selected alternate households (i.e., attrition from the sample).
  • Survey text used in each survey round and detailed protocols for responses that are directly observed by surveyors.

• 100
  • Ntcheu 6-month post-distribution check-up data
  • Ntcheu 15-month post-distribution check-up data
  • Ntcheu 24-month post-distribution check-up data
  • Ntcheu 33-month post-distribution check-up data
  • Balaka 2013 6-month post-distribution check-up data
  • Balaka 2013 14-month post-distribution check-up data
  • Balaka 2013 19-month post-distribution check-up data
  • Dedza 2014 8-month post-distribution check-up data
  • Dedza 2014 12-month post-distribution check-up data
  • Dedza 2014 18-month post-distribution check-up data
  • Kasaï-Occidental 2014 8-month post-distribution check-up data
  • Kasaï-Occidental 2014 12-month post-distribution check-up data
  • Dowa 2015 6-month post-distribution check-up data
  • Dowa 2015 12-month post-distribution check-up data
  • Ntcheu 2016 6-month post-distribution check-up data

• 101
  • Rob Mather, AMF Founder, email to GiveWell, June 30, 2014
  • A representative of Concern Universal told us that, due to limited indoor space, some recipients take down their nets during the day and hang them up each night. When assessing whether a net is being used, enumerators observe the net and the sleeping space, ask the recipient questions, then make a judgment call. A net that is being used regularly will likely show signs of wear that an unused net would not; and a sleeping space over which a net is hung nightly may show evidence of the hanging even when the net is not present. Nelson Coelho, conversation with GiveWell, April 15, 2016

• 102

  In the data for AMF's post-distribution check ups, it looks to us like "Nets Received" = "Hung" + "Present Not Hung" + "Missing" +" Worn out/not usable", implying that "nets hung" and "nets that are worn out" are exclusive categories. See, for example, Concern Universal Ntcheu 2012 33-month post-distribution check-up data

• 103

  "'very good condition' (fewer than 2 holes of up to 2cms in size)." Concern Universal Ntcheu 2012 33-month post-distribution check-up report, pg. 4

• 104
  • "‘good condition’ (fewer than 10 small holes on them)" Concern Universal Ntcheu 2012 33-month post-distribution check-up report, pg. 3.
  • Note that nets with fewer than 10 small holes used to be considered in "OK" condition.
    "'OK' (fewer than 10 small holes on them)" Concern Universal Ntcheu 2012 24-month post-distribution check-up report, pg. 4.

• 105
  • "‘viable condition’, with more than 10 holes or 1 hole larger than 10 cm," Concern Universal Ntcheu 2012 33-month post-distribution check-up report, pg. 3.
  • Note that nets with more than 10 small holes used to considered in "poor" condition. "'Poor' condition with more than 10 holes or 1 hole larger than 10 cm." Concern Universal Ntcheu 2012 24-month post-distribution check-up report, pg. 4.

• 106

  "Worn out means not usable. Typically, holes will be 10 cms of cms in size and numerous and the net considered effectively unrepairable and/or structural integrity will have gone. If hung the net would not provide protection against mosquitoes given the extent of holes or tears or other loss of structural integrity." Rob Mather, AMF Founder, email to GiveWell, June 30, 2014.

• 107
  • @AMF how we work with distribution partners@, pg. 4.
  • According to a representative of Concern Universal, when assessing whether a net is being used, enumerators make a judgment call. Nelson Coelho, conversation with GiveWell, April 15, 2016

• 108

  Rob Mather, AMF Founder, email to GiveWell, June 30, 2014

• 109

  For example, consider data from the Ntcheu 2012 33-month PDCU:

  • [GiveWell]: "Second, I'm trying to better understand the outcomes measured for post-distribution check-ups. In particular, I'm curious what the "People - Covered", "People sleeping under nets", and "Nets Used - Correctly/Not Correctly" metrics mean (under the "Usage of nets" tab). In the Ntcheu 33-month PDCU results , the "Presence of AMF Nets" tab lets me see how many nets are "hung", "present but not hung, "missing", or "worn out": the numbers of "hung" and "present but not hung" nets are 10,667 and 630, respectively, for a total of 11,297 nets that aren't missing or worn out. However, under the "Usage of nets" tab, I see that 11,498 nets are used "Correctly." Since 11,498 > 11,297, I believe that some of the "worn out" nets must be included in the "used correctly" metric. If anyone sleeping under a net that is "used correctly" is considered to be "covered", then some of the people counted in the "covered" category are actually sleeping under worn out nets. Is my understanding correct?"
    [AMF]: "This is because the latter figure includes non-AMF nets i.e. there are 462 non-AMF nets included in the ‘Usage of the nets’ tab as we wished to have an overall figure for sleeping space coverage (52%). We may add a note to this tab to indicate there are some non-AMF nets included in the data on this tab." Rob Mather, AMF Founder, email to GiveWell, October 13, 2015. Note that we did not ask AMF where the 462 figure comes from.
  • 10,664 AMF nets were found to be "very good", "good", or "viable" condition. 10,664 + 462 non-AMF nets = 11,126 nets, which is less than the 11,497 nets "used correctly", implying that some AMF nets that are "worn out" are included in the "used correctly" category. Concern Universal Ntcheu 2012 33-month post-distribution check-up data

• 110

  For example, in the Concern Universal Ntcheu 2012 33-month post-distribution check-up data, we see that:

  • 10,664 AMF nets are hung ("Presence of AMF Nets" tab)
  • 10,664 AMF nets are also in "viable", "good", or "very good" condition ("Condition of AMF Nets tab"). It seems odd to us that these numbers match.
  • 11,497 nets are "used correctly." ("Usage of the nets" tab) AMF told us 462 non-AMF nets were included in this tab, meaning that 11,035 AMF nets are included.
    • [GiveWell]: "Second, I'm trying to better understand the outcomes measured for post-distribution check-ups. In particular, I'm curious what the "People - Covered", "People sleeping under nets", and "Nets Used - Correctly/Not Correctly" metrics mean (under the "Usage of nets" tab). In the Ntcheu 33-month PDCU results , the "Presence of AMF Nets" tab lets me see how many nets are "hung", "present but not hung, "missing", or "worn out": the numbers of "hung" and "present but not hung" nets are 10,667 and 630, respectively, for a total of 11,297 nets that aren't missing or worn out. However, under the "Usage of nets" tab, I see that 11,498 nets are used "Correctly." Since 11,498 > 11,297, I believe that some of the "worn out" nets must be included in the "used correctly" metric. If anyone sleeping under a net that is "used correctly" is considered to be "covered", then some of the people counted in the "covered" category are actually sleeping under worn out nets. Is my understanding correct?"
      [AMF]: "This is because the latter figure includes non-AMF nets i.e. there are 462 non-AMF nets included in the ‘Usage of the nets’ tab as we wished to have an overall figure for sleeping space coverage (52%). We may add a note to this tab to indicate there are some non-AMF nets included in the data on this tab." Rob Mather, AMF Founder, email to GiveWell, October 13, 2015. Note that we did not ask AMF where the 462 figure comes from.
  • 11,035 nets "used correctly" is more than the 10,664 nets "hung", implying that nets that were "present but not hung" or "worn out/not usable" could be included. ("Presence of AMF Nets" tab).
  • 11,035 nets "used correctly" is also more than the 10,664 nets found in "viable", "good", or "very good" condition, implying that some "worn out" nets are "used correctly."
  • 24,980 people were found to be "sleeping under nets" ("Usage of the nets" tab). These are the people who are also labelled as "Covered" by nets.
  • 24,980 people / 11,035 nets "used correctly" = 2.26 people per net, which seems reasonable, though higher than we would have expected. 24,980 people / 10,664 nets "hung" = 2.34 people per net, which also seems reasonable but higher than we would have expected. We are uncertain about whether "sleeping under a net" means sleeping under a "used correctly" net, a "hung" net, or a non-"worn out" net.

  Concern Universal Ntcheu 2012 33-month post-distribution check-up data
  

• 111
  • "Usage does not appear to have been near-universal. Most studies report usage rates in the range of 60-80%, though some report 90%+ usage." From our page on Long Lasting Insecticide Treated Nets.
  • Note that the World Malaria Reports that we looked at showed similar usage rates.

• 112
  • "Alliance for Malaria Prevention 2011 lays out a model for estimating the number of LLINs still in use after distributions: 'the number of LLINs already distributed over the last three years and considered to be available in households should be calculated and subtracted from the total need, working with a decay rate of 8 per cent at one year (0-12 months), 20 per cent at two years (13-24 months) and 50 per cent at three years (25-36 months).'…
    Bottom line: We believe that the "8%-20%-50%" model is the most widely used and most reasonable approximation available at the moment for capturing the extent to which LLINs remain in use in the years following distribution, accounting for any factors that might cause LLINs to be discarded or additional LLINs to be purchased. It implies an average of 2.22 years of use for each LLIN distributed. Data and analysis on this topic appears extremely thin; we have little sense for how long LLINs last in practice."
  • "If we assume that an LLIN has a 92% chance of being in use at a given point in the first year after distribution, this implies that for each LLIN delivered, an average of 0.92 LLIN-years of use are obtained in the first year. Assuming 0.92 LLIN-years of use in the first year, 0.8 in the second year and 0.5 in the third year would yield an overall average of 2.22 years of use per LLIN. This is substantially less than the "official life" of an LLIN. As discussed below, we believe this makes sense because the decay function is intended to account for wastage of all kinds, including loss/failed delivery of LLINs, improper use resulting in disrepair, etc."

  GiveWell report on the "decay model" for LLINs.

• 113

  See this spreadsheet, "Decay rate comparison" sheet for the data referenced in the following discussion.

• 114
  • See all results compared to the predictions of the decay rate model in (see this spreadsheet, "Decay rate comparison" sheet.)
  • "This is the second universal coverage distribution in Ntcheu, with the first taking place in December 2012, a little over four years before.
    […]
    Initial hypotheses for data from this PDCU-06:
    Hypothesis 1: Timing differences between the 2012 PDCU-06 and the 2016 PDCU-06 mean there is a seasonal variation (e.g. linked to mosquito levels and average temperatures)
    Hypothesis 2: The previously distributed nets have lasted beyond the normal three years life and some of the new nets are being held in reserve to replace them when worn out.
    Hypothesis 3: Our criteria for assessing, prior to a mass distribution, which nets are ‘perfectly usable’ (have at least 18 months of life left) is too strict and materially more nets than we are judging to be so have extended life in them." Ntcheu 2015 6-month post-distribution check-up report, Pg 4.
  • "The data collected show the level of sleeping space coverage with nets that were distributed during April 2015 was 81%.
    We expected this figure to be about 5 to 10 percentage points higher.
    Data for the proportion of all sleeping spaces covered shows that 85% are covered. This suggests that some sleeping spaces may be covered with nets not distributed during the mass campaign. If so, these are likely to be nets distributed in the prior campaign (few, we estimate) and some nets distributed via routine mechanisms e.g. ante-natal clinics (most, we estimate).
    We do not have further information or data on a likely split. The level of nets present but not hung is 15%. Normally we see levels around 4-8%. This suggests householders may not be using new nets as they still have acceptable older nets. We will consider what further information we could gather to understand if a) newer nets are being held back due to being not needed (and what the implication, if any, that has for the assessment at the time of distribution of household net need and the presence of ‘perfectly usable nets’.); and/or b) whether all sleeping spaces that should be covered (ones being slept in) are not being covered and there is a need to encourage greater hang-up." Dowa 2015 6-month post-distribution check-up report, Pg 4.

• 115

  "In November or December 2015, the NMCP learned that the Against Malaria Foundation (AMF) would be providing funding for Ghana's 2016 net distribution. In the absence of this funding:

  1. There might not have been any nets available for mass distribution in the Upper West Region this year, where the NMCP carries out both IRS and SMC activities. The NMCP believes nets should still be used in regions with IRS and SMC, but prioritizes distribution in regions without them.
  2. There might not have been enough nets to cover the Greater Accra Region. As a result, the NMCP might have only pursued a targeted mass campaign in the region, and focused on slums and rural areas.
  3. Continuous distribution of nets, which are carried out in health clinics and schools, might have faced net shortages due to continuous distribution nets being redirected to the mass campaign."

  GiveWell's non-verbatim summary of a conversation with Ghana's National Malaria Control Program, August 16-18, 2016, pg. 2.

• 116
  • "The Greater Accra GHS told us that they expected that rural areas were likely to
    have a higher uptake level of nets than semi-urban areas." GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, pg. 19.
  • In our visits in August 2016 to a few households in semi-urban and rural communities targeted by the July 2016 AMF-funded distribution, we found several households in rural communities with LLINs from the recent AMF distribution hung over sleeping spaces, but none in semi-urban areas. GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016:
    • "Semi-urban community visits
      GiveWell staff visited 13 households in Ga South and La Dadekotopon districts. The Greater Accra distribution began on July 5th and lasted approximately one week; we are not sure exactly when the distributions in these two communities took place.
      Main observations
      None of the households we visited were sleeping under an AMF net: some were sleeping under an older net, and some had received AMF nets but were not yet sleeping under them." Pg. 15.
    • "Our main observations from visiting two rural communities included:
      • All eight households we talked to registered for and received nets from the 2016 AMF distribution.
        […]
      • Four households told us that everyone in their household had slept under 2016 AMF nets the previous night." Pg. 17.
    • Our selection process for choosing households to visit included in GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, Pg 15.

• 117

  "Continuous Distribution of Nets
  This involves nets given for free at health facilities (antenatal clinics for pregnant women and child welfare clinics i.e. during vaccinations) and in schools.
  Pregnant women receive a free net when they register their pregnancy at a health center; most women are aware, and take advantage, of this opportunity. Nets are given to children between 18 and 36 months who visit clinics for the measels. Two booster doses are used as an incentive to improve vaccination coverage.
  Nets are also distributed to children in selected classes in primary schools in most regions of the country. In 2016, nets were distributed in six regions." GiveWell's non-verbatim summary of a conversation with Ghana's National Malaria Control Program, August 16-18, 2016, pg. 4.

• 118

  This is an understanding formed over many conversations.

• 119
  • ALMA LLIN gap analysis (June 2016)
  • Melanie Renshaw, conversation with GiveWell, March 16, 2016
  • Marcy Erskine, conversation with GiveWell, March 29, 2016
  • Rob Mather and Peter Sherratt, conversation with GiveWell, April 25, 2016

• 120

  "Health Surveillance Assistants (HSAs) are Government extension workers- they are the lowest tier of government presence in the decentralized health system." Robin Todd, Concern Universal Malawi Director, email to GiveWell, April 27, 2012.

• 121

  "As such they are the first line of response to any public health issues in communities. Their job involves disseminating health related information (such as encouraging people to make use of sanitary facilities, go for immunizations, sleep under mosquito nets etc.), carrying out sanitation and hygiene campaigns and sending data on take-up of facilities to the District Council, conducting basic nutrition support, weighing children and reporting levels of stunting and wasting, detecting common communicable diseases and reporting these to clinicians and other health providers, implementing immunization campaigns etc. As you can see being involved in universal net distribution fits very well with their core public health responsibilities. HSAs need to have a primary school completion certificate as a minimum but the majority of them will have O-Levels (exams sat by pupils aged 16 if they have completed the school system at the recommended pace). Once they have been selected as HSAs they are sent on an initial 9 months intensive training course where they will be trained in many aspects of public health including how to recognize common diseases, how to administer immunizations etc." Robin Todd, Concern Universal Malawi Director, email to GiveWell, April 27, 2012.

• 122
  • Pre-distribution registration surveys appear to be completed relatively quickly: "Approximately 480 personnel will be involved in the PDRS, with the majority, some 460, involved for 5-7 days over the data collection period." Concern Universal, Dowa 2015 planning document, pg. 4.
  • In the Ntcheu 2012 distribution, the verification of PDRS data took several weeks: "The verification process took the verification team of 10 members 18 days to complete and in a day 20 clusters were verified with 10 verification sites in the morning and 10 verification sites in the afternoon." Concern Universal Ntcheu 2012 distribution report, pg. 7.
  • In the Ntcheu 2012 distribution, distributions were scheduled for several weeks, but covered approximately 10 "clusters" per day: "The distributions were scheduled to have been concluded within 28 days with the team distributing at 10 clusters per day covering five weeks." Concern Universal Ntcheu 2012 distribution report, pgs. 7-8.
  • Concern Universal has recently started to use fewer staff for post-distribution check ups, which causes them to take somewhat longer: "However, in collaboration with the District Environmental Health Office (DEHO) and Malaria Coordinator (MC) and lessons lea[r]nt from 24 month Ntcheu PDCU, it was recommended to have a focused team of 10 data collectors rather than have the HSAs as data collectors from each HCA. This was based on the following reasons. First, this would reduce the number of data collectors that would need to be monitored and trained. Second, we would be able to select reliable individuals whom we could trust to do a diligent and accurate job of collecting the data. Third, it would leave the majority of HSAs to carry on with the normal health tasks and duties. Fourth, by having the same people covering the whole exercise they will get acquainted to the task and reduce errors on data collection. This meant the data collectors would spend thirty seven days collecting data rather than the one or several days if many more data collectors were to be used. This was judged the preferable way of organising and managing the data collection phase." Concern Universal Ntcheu 2012 33-month post-distribution check-up report, pg. 5. Other recent post-distribution check-up reports have similar language.

• 123

  "i) Field Supervisors (FSs)
  22 FSs were selected from permanent and senior health staff in Tshikapa Health District.
  ii) Community Health Workers (CHWs) – data collectors
  Each of the 22 FSs had the responsibility of recruiting, in each HA they were designated, enough CHWs to gather household data and hang nets. Two primary recruitment criteria were literacy and familiarity with using a mobile phone. The number of CHWs recruited depended on the size of the HA and the number of households to be visited. The aim was to recruit enough CHWs to carry out the entire registration and hang‐up, once it commenced, in a five day period. Between 20 and 40 CHWs were recruited by each FS for a total of 4,000 CHWs across the 8 HZs (8 HZs x 20 HAs x 25 CHWs per HA = 4,000 CHWs)." IMA World Health, Kasaï-Occidental 2014 distribution report, pg. 13.

• 124

  GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016:

  • "In the present distribution, core responsibilities of Episcopal Relief & Development and ADDRO include:
    • Monitoring the NMCP-led pre-distribution and distribution activities, and providing feedback to the NMCP. NMCP also does its own monitoring.
    • Post-distribution monitoring." Pg. 4.
  • "Phase 1 – Planning and registration
    The registration phase was implemented by the government and is now complete in all three regions.
    Step 1 – Informative Meetings with Ghana Health Service Regional Health Directorates
    At these meetings, stakeholders discuss the LLIN distribution implementation model, the schedule of activities, and budgets. Stakeholders also agree on a date for the regional planning workshop.
    Step 2 – Regional planning workshops
    At these workshops, stakeholders discuss the registration and distribution processes, budgets, rules, and responsibilities for different groups. In the Greater Accra regional planning workshop, the stakeholders also discussed which households to target (in other regions, all households were targeted)." Pg. 4.
  • "GHS volunteers carry out the registration process, which takes place more than one month before the distribution. Volunteers are organized by the Ministry of Health (MoH) and participate in a number of government health programs, such as vaccination campaigns (immunization days)." Pg. 5.
  • " Steps in the distribution process
    1. Beneficiaries walk a short distance to their distribution point. One staff member employed by GHS and at least one GHS volunteer are stationed at each distribution point; there are never just two GHS volunteers." Pg. 8.

• 125

  Full details in our AMF cost per net spreadsheet.

• 126

  Full details in our AMF cost per net spreadsheet.

• 127

  Full details in our AMF cost per net spreadsheet.

• 128

  This estimate is an average (weighted by AMF's spending) of the cost per LLIN in AMF's past and planned distributions. See our most recent cost-effectiveness analysis, "Bednets" sheet, cell J71.
  

• 129
  • The current budget for the PBO RCT in Uganda is $2.7 million. AMF funds status (November 2016) Anonymized
  • The original budget for the insecticide resistance study in DRC was $800,000. Rob Mather, AMF Founder, conversation with GiveWell, February 24, 2015
  • AMF sent us a draft research proposal for the DRC study in early 2015; some details are in our May 2015 update. AMF sent us what we believe was the final version of the research proposal in June 2015; we have not yet reviewed the final version. AMF insecticide research proposal from the London School of Tropical Medicine
  • "Given the nature of the Uganda PBO study we have just last week decided to not proceed with Phase 2 of the DRC study. The total cost of Phase 1 was £75,667. Phase 1 looked at establishing where resistant mosquitoes were present in Nord Ubangi (they are) and Phase 2 was to look at the effectiveness of the PBO nets." Comment provided in response to a draft version of this review in November 2016.
  • 1 GBP equals 1.25 USD. Google, November 21, 2016. 75,667 * 1.25 = $94,412.

• 130

  Full details in our AMF cost per net spreadsheet.

• 131
  • The estimate includes actual costs for the Balaka 2013, Dedza 2014, Dowa 2015, Ntcheu 2015, and Balaka 2015 distributions. The only completed large-scale AMF distribution in Malawi not included is Dowa 2012. Concern Universal paid for the non-net costs of the Dowa 2012 distribution; we have not asked for CU's actual or budgeted costs.
  • Full details in our AMF cost per net spreadsheet.

• 132

  Full details in our AMF cost per net spreadsheet.
  In estimating the overall cost per LLIN in Malawi, we have used an average that weights distribution costs by distribution size (i.e. costs associated with larger distributions are given proportionally more weight). Another option would be to put more weight on the actual costs to account for the fact that actual costs are often more reliable than budgeted costs.

• 133

  Rob Mather, AMF Founder, conversation with GiveWell, May 23, 2014. Managerial staff time is listed in the budget.

• 134

  See the “Ntcheu 2012 other costs” sheet in in our AMF cost per net spreadsheet.

• 135

  Full details in our AMF cost per net spreadsheet.

• 136
  • See our AMF cost per net spreadsheet, "AMF org" sheet for details.
  • AMF's fiscal year runs from Jul 1 - Jun 30 (e.g. FY 2012 is Jul 1 2011 - Jun 30 2012).

• 137

  For example:

  • The spreadsheet IMA shared of actual costs for the Kasaï-Occidental distribution is hard to make sense of and appears to contain several errors. Also, about 20% of the costs are "unconfirmed charges," which are not listed individually.
  • For Malawi, Concern Universal shared, in 2012, information on non-monetary costs (such as office space) and the value of time provided by government employees. IMA's actual cost data includes an estimate of indirect costs (as a percentage of total direct costs), but it's unclear if it includes costs paid for by the government or other actors. As a rough approximation, we have added $0.13 per net, which was the estimate of non-monetary partner and local government costs from the 2012 Ntcheu, Malawi distribution.

• 138

  Full details in our AMF cost per net spreadsheet.

• 139

  Full details in our AMF cost per net spreadsheet.

• 140
  • AMF told us that in DRC net costs and non-net costs are very roughly the same. Rob Mather, AMF Founder, conversation with GiveWell, May 23, 2014
  • "In other cases, non-net costs are assumed to $2.50 per LLIN, which is the historical average. Non-net costs can vary significantly across countries. For example, in Angola and DRC, non–net costs may be more than $5." Melanie Renshaw, African Leaders Malaria Alliance Chief Technical Advisor, phone conversation with GiveWell, May 23, 2014, pg. 3.

• 141

  Full details in our AMF cost per net spreadsheet.

• 142
  • Major categories of activities included in Episcopal Relief & Development's budget: attending planning meetings and workshops, monitoring the household registration process, pre-distribution and post-distribution data validation, distribution exercises, post-distribution check-up surveys, data entry, and community sensitization activities. A budget for equipment, office costs, and staff time is also included. Episcopal Relief & Development Ghana non-net costs budget 2016
  • See our AMF cost per net spreadsheet for our calculations.

• 143
  • We have not yet seen budgets or actual costs from Ghana Health Services from these distributions.
  • In June 2016, AMF estimated that standard non-net costs (other than PDCUs) would cost around $1.50 or $2.00 per net for AMF's future distributions (listed as "Addntl non-net costs (per net)" in source). AMF funds status (June 2016)
  • We have chosen to use $1.75 per LLIN as an estimate of the costs incurred by Ghana Health Services for its pre-distribution planning activities and distribution implementation activities, though we are highly uncertain of this estimate.

• 144

  Full details in our AMF cost per net spreadsheet.

• 145
  • AMF funds status (November 2016) Anonymized
  • See our AMF cost per net spreadsheet.

• 146

  Full details in our AMF cost per net spreadsheet
  

• 147
  • "AMF Medium Term Strategy… Increase AMF central staffing by four people before the end of 2016: two program directors, one operations assistant and a technology assistant. Keep under review to assess hiring ahead of need. Hire one to two permanent staff in each country in which large scale distributions are funded." AMF medium term strategy discussion document (May 2016).
  • AMF provided us with its estimates for increases in salary costs due to additional hiring in FY 2017 and FY 2018. Rob Mather, AMF Founder, email to GiveWell, November 14, 2016

• 148

  For our estimate of AMF's organizational costs for future distributions, we use a weighted average of our annual estimates of AMF's organizational costs per net beginning in AMF's FY 2015, through our projected organizational cost per net for FY 2018. We exclude earlier years, when AMF was operating at a smaller scale and its organizational cost per net was higher than we expect it to be in future years. See our cost per net spreadsheet, "AMF org" sheet for calculations.

• 149

  See our most recent cost-effectiveness analysis, "Bednets" sheet, cell J71.

• 150

  See for example Roll Back Malaria Partnership gap analysis (September 2013).

• 151

  Melanie Renshaw, African Leaders Malaria Alliance Chief Technical Advisor, phone conversation with GiveWell, October 20, 2015

• 152

  "The gap analysis includes both the cost to purchase LLINs and non-net costs, such as shipping and distribution. In cases where national governments provided data on non-net costs, this data was used in the analysis. In other cases, non-net costs are assumed to $2.50 per LLIN, which is the historical average. Non-net costs can vary significantly across countries. For example, in Angola and DRC, non–net costs may be more than $5." Melanie Renshaw, African Leaders Malaria Alliance Chief Technical Advisor, phone conversation with GiveWell, May 23, 2014, pgs. 2-3.

• 153

  See our most recent model, "Bednets" and "Results" sheets.

• 154

  Funding commitments as of October 2016 (AMF funds status (November 2016) Anonymized):

  • Malawi 2018 distribution (1.48 million nets and all non-net costs): $5.5 million
  • Insecticide resistance study in Nord Ubangi: $0.7 million
  • Malaria Unit, Malawi: $0.54 million
  • Ghana 2016 distribution (non-standard non-net costs): $1.2 million
  • Uganda 2017 distribution (5.54 million nets and non-standard non-net costs): $18.4 million
  • Togo 2017 distribution (2.41 million nets and non-standard non-net costs): $5.8 million
  • Papua New Guinea 2017 and 2018 distribution (2.8 million nets and non-standard non-net costs): $6.7 million

• 155

  AMF funds status (November 2016) Anonymized

• 156

  AMF funds status (November 2016) Anonymized

• 157
  • In GiveWell's 2011 metrics year (February 1, 2011 to January 31, 2012), GiveWell-influenced donors gave $2,720,750 to AMF. In AMF's 2011 fiscal year (July 1, 2010 to June 30, 2011), AMF took in $2,092,594 in revenue. Note that our metrics year does not align with AMF's fiscal year, so these figures cannot be directly compared. We have not tried to match our AMF money moved to AMF's revenue at a more granular level.
  • In our 2012 metrics year, GiveWell-influenced donors gave $5,895,544 to AMF. In AMF's FY 2012, it took in $3,921,079 in revenue.
  • In our 2013 metrics year, GiveWell-influenced donors gave $2,502,125 to AMF. In AMF's FY 2013, it took in $7,956,078 in revenue.
  • In our 2014 metrics year, GiveWell-influenced donors gave $9,797,274
    to AMF. In AMF's FY 2014, it took in $3,972,611 in revenue.
  • In total, over GiveWell's 2011 to 2014 metrics years, GiveWell-influenced donors gave $20,915,693 to AMF. Over AMF's FY 2011 to 2014, AMF took in $17,942,362. Note that our metrics year does not align with AMF's fiscal year, so these figures cannot be directly compared.
  • For GiveWell money moved figures, see our impact page.
  • AMF Financial information

• 158

  See calculations in this spreadsheet, "Room for more funding" sheet.

• 159

  This is based on internal records of how much GiveWell-influenced donors gave to AMF in 2013 when we listed AMF as a "top charity with limited room for more funding," inflated for growth in GiveWell's annual money moved since 2013.

• 160
  • Melanie Renshaw, African Leaders Malaria Alliance Chief Technical Advisor, phone conversation with GiveWell, May 23, 2014
  • Melanie Renshaw, African Leaders Malaria Alliance Chief Technical Advisor, phone conversation with GiveWell, October 20, 2015
  • Melanie Renshaw, conversation with GiveWell, March 16, 2016
  • Summary of year-end gaps in Rob Mather, email to GiveWell, May 9, 2016. 2013-2015 year-end gaps were 22m, 38m, and 39m, respectively, and "does not include distributions delayed and pushed into the next year with the hope of finding funding for the subsequent year."
  • ALMA LLIN gap analysis (June 2016)

• 161
  • GiveWell's non-verbatim summary of a conversation with Melanie Renshaw and Marcy Erskine, October 11, 2016
  • GiveWell's non-verbatim summary of a conversation with Melanie Renshaw, November 2, 2016

• 162

  GiveWell's non-verbatim summary of a conversation with Melanie Renshaw and Marcy Erskine, October 11, 2016:

  • "The target for the Global Fund replenishment for 2018-2020 was $13 billion, and $12.9 billion has been pledged." Pg. 1.
  • "Funding for specific interventions will be determined through the following process:
    1. Allocations per country – The amount allocated to each country for 2018-2020 will be announced on December 15, 2016. Per-country allocations are unlikely to change after this time.
    2. Planning and proposals for diseases and interventions – After per-country allocations are announced, countries will begin putting together proposals for the specific diseases and interventions they plan to target with their allocations. The majority of countries will need to submit their proposals for malaria-related spending – including proposed LLIN allocations – by March 20th, 2017. The technical review panel will review the proposals prior to approval.

    Until this process is complete, it is difficult to estimate the size of the investment that countries will make in LLIN programs for 2018-2020. It is therefore difficult to predict what the global gap in funding for LLIN programs will be during this funding cycle." Pgs. 1-2.

• 163

  GiveWell's non-verbatim summary of a conversation with Melanie Renshaw, November 2, 2016:

  • "The overall resources spent on malaria programs are likely to remain largely the same in 2018-2020 as in the past few years. The Global Fund to Fight AIDS, Tuberculosis and Malaria (the Global Fund) and The U.S. President’s Malaria Initiative (PMI) are expected to make contributions at roughly the same scale as during the previous Global Fund funding cycle. However, even if AMF and other funders also continue to provide the same level of support for LLINs as they did in the past few years, there is still likely to be a significant gap in LLIN funding in 2018–2020. Dr. Renshaw estimates that this could translate to a global gap of approximately 100 million LLINs over the three-year period, including a 25 million LLIN gap in Nigeria alone." Pg. 1.
  • "The Global Fund is the main factor in the size and scope of LLIN funding gaps. Country allocations have not yet been finalized for the upcoming funding cycle. They will be announced on December 15. However, there may be several key changes to how funding is allocated:
    • Smaller allocation for countries – The total amount of funding for all countries during the upcoming Global Fund cycle is less than during the previous cycle. This alone would not necessarily be a significant enough reduction to impact LLIN funding. However, an additional $300 million in catalytic funding – which was previously used to fill funding gaps for LLINs – will not go toward LLIN programs in the upcoming funding cycle. These two factors combined are likely to decrease per-country LLIN funding.
    • Allocations more evenly divided among countries – by reducing funding for large, high-burden countries. In the previous funding cycle, some large, high-burden countries were "over-allocated" – i.e., received substantially more funding than others. In the upcoming funding cycle, the Global Fund will work to distribute funds to countries more evenly, allocating less to some “over-allocated” countries and more to "under-allocated" countries. Most African countries have a high malaria burden, and “under-allocated” countries are not low burden countries like Swaziland, South Africa, etc. Rather, they are countries that did not receive sufficient funding to implement a wide variety of malaria control programs, e.g., expanding case management to community case management, rolling out seasonal malaria chemoprevention programs, or enhancing intermittent preventative treatment in pregnancy. They were generally only able to fund LLIN universal coverage campaigns and public sector case management in the past. Given additional funding, they would likely invest in enhancing case management and perhaps routine LLIN distribution.
    • Less shock buffer – In the past, in the Global Fund rounds system, countries proposed the amount of funding needed. The Global Fund reviewed their proposals and either provided the funding requested or did not. With the move to the New Funding Model, the amount allocated to a country was required to be at least 75% of its previous allocation, to prevent shock to the programs. In the upcoming cycle, the Global Fund may reduce this shock buffer to 50%. This, combined with "over–allocated" countries possibly receiving less, could mean that countries will receive substantially less than needed to sustain coverage. It would be difficult for them to fully support their LLIN programs as a result." Pgs. 1-2.

• 164
  • Rob Mather and Peter Sherratt, conversation with GiveWell, October 13, 2016
  • Comment provided by AMF in response to a draft of this review in November 2016

• 165

  Here we've used $4.16 per net, which excludes AMF's organizational expenses ($0.16 per net) and in-kind contributions from partners and local governments ($0.13 per net). See this spreadsheet.

  • Effective number of nets fully funded or committed for 2015-2017: 12.29 million. See data and calculations in this spreadsheet, sheet 'Overview,' column 'effective number of nets fully funded (ex. In-kind contributions).'
  • Agreements imminent but not yet signed for 2017 – two distributions (AMF funds status (November 2016) Anonymized): $10.17 million / $4.16 per net total = 2.44 million nets
  • Total over 3 years: 14.73 million. Average per year: 4.91 million

• 166

  75 million nets over 3 years, or 25 million nets per year, plus 5 million nets per year to replace the nets AMF has funded in the past.

• 167

  As reported to us in Rob Mather, email to GiveWell, May 9, 2016.

• 168

  See this spreadsheet, "Room for more funding" sheet.

• 169

  AMF estimated these levels as:
  Execution level 1: $75-100m
  Execution level 2: $70-45m
  Execution level 3: $54.7m
  Rob Mather, email to GiveWell, November 22, 2016

• 170
  • Rob Mather and Peter Sherratt, conversation with GiveWell, April 25, 2016
  • "The operational plan for a distribution typically covers 12 months. The planning horizon for a specific distribution is, however, typically 18 to 24 months as discussions to assess and approve a distribution start six months earlier and the funds required (we need to have funds in hand in order to have serious discussions) are aggregated in the preceding months." AMF: "Operational planning (12 months) and planning horizon (18 to 24 months)"

• 171

  See this spreadsheet, sheet "Room for more funding."

• 172
  • $36.8 million committed to distributions in Togo, Papua New Guinea, and Uganda in 2017. Sum of cells D7-D9, "Overview" sheet, in this spreadsheet.
  • AMF may commit an additional $10.2 million to "Country C" and "Country R" for distributions in 2017. AMF funds status (November 2016) Anonymized.

• 173

  In 2014, we believed that AMF had additional room for more funding because:

  • In 2013, AMF had completed two large-scale distributions, both of which were with a single distribution partner, Concern Universal in Malawi. It had not signed agreements with other partners. As of November 2014, AMF had signed agreements for two large distributions with IMA World Health in the DRC, helping to demonstrate its ability to finalize distributions with other partners in other locations.
  • By the end of 2014, AMF had spent or committed a large portion of the funds it raised prior to 2014. As of November 2014, AMF held $4.9 million in uncommitted funds, of which it raised $2.65 million in 2014. Of committed funds, $1.9 million were for expenses that AMF would incur more than a year later, and AMF was considering reallocating these funds to the nearer-term with the expectation of raising enough to cover its later commitments by the time they were due. This reallocation brought AMF's total available funds to $6.8 million.
    • "Funds in hand as at Nov 2013: $12.5 m
      Committed since Nov13: $10.30m (10.3/12.5 = 82%)
      - DRC, West Kasaï 2014 $2.0m (NETS only)
      - DRC, Idjwi 2015 $0.37m (NETS + all non-net costs)
      - Malawi, Dowa, 2015 $1.49m (NETS + all non-net costs)
      - Malawi, Ntcheu 2015 $1.34m (NETS + all non-net costs)
      - Malawi, Balaka 2015 $0.94m (NETS + all non-net costs)
      - Malawi, Dedza 2017 $1.72m (NETS + all non-net costs)
      - DRC, North Ubangi 2015 $2.34m (NETS + shipping)
      - Malawi, Dedza, 2014 $0.10m (additional nets)

      Funds uncommitted $2.2m
      New funds raised $2.65m
      Total funds uncommitted $4.85m"
      Rob Mather, AMF Founder, email to GiveWell, November 26, 2014

    • "If ‘make available’ all Dedza17: $1.720m [i.e. if funds committed to the Dedza 2017 distribution were made available for an earlier distribution]
      Ditto, Yr2+ PDCU Do/Nt/Ba: $0.2 m [refers to future post-distribution check ups in Dowa, Ntcheu, and Balaka]"
      Rob Mather, AMF Founder, email to GiveWell, November 8, 2014
  • AMF's pipeline for potential distributions included several distributions with established partners. Given AMF's history with these partners, we guessed that there were fewer barriers to signing agreements for additional distributions. Funding all such distributions would have required AMF to raise significantly more funds. AMF may also have had opportunities to fund distributions with new partners.

• 174
  • Rob Mather, AMF Founder, email to GiveWell, September 9, 2015
  • "AMF and Concern Universal, Malawi (CU) have accepted the Malawi National Malaria Control Programme’s (NMCP) invitation to assume responsibility for the district-wide net distribution re-coverage campaigns in Ntcheu, Balaka, Dedza and Dowa districts in 2018[…] This timing is consistent with the three year re-coverage cycle needed to keep affected populations protected from malaria-carrying mosquitoes when they sleep at night." AMF: "US$6m commitment to malaria control support in Malawi in 2018".

• 175

  "Whilst this statement of the possible cannot be disagreed with, the likelihood of this outcome – AMF not being able to allocate sizeable funds – is not believed at all likely by us or anyone with whom we have spoken in the malaria community and we feel that to place weight on this view [is not reasonable] would be a mis-assessment of the situation. Direct approaches to AMF by both NMCPs and the [Global Fund] give a strong indication of that." Comment provided in response to a draft of this review in November 2016

• 176

  Global Fund introduction to the 2017-2019 funding cycle, Pgs 23-24.

• 177

  "In our view, this mis-understands the knowledge the countries have at an early stage – from 15Dec. It is at that point – through January certainly – they know the level of funding the Country Coordinating Mechanism (CCM) has to cover the three diseases. Given many countries are then very keen to secure additional funding – as many if not all will have gaps – our and others’ experience is countries are keen to sign agreements with other funding partners as they know the limiting resources is those other funders. We are confident of being able to sign agreements early in 2017 if we in a position to do so." Comment provided in response to a draft of this review in November 2016.
  

• 178
  • Rob Mather, AMF Founder, conversation with GiveWell, April 13, 2016
  • Rob Mather and Peter Sherratt, conversation with GiveWell, April 25, 2016

• 179

  "AMF Medium Term Strategy… Increase AMF central staffing by four people before the end of 2016: two program directors, one operations assistant and a technology assistant." AMF medium term strategy discussion document (May 2016)

• 180

  Rob Mather, email to GiveWell, May 13, 2016

• 181

  "Staff capacity has not hindered in any way our ability to sign agreements or manage distributions. It has slowed aspects of IT development (website redesign) and reporting." Comment provided in response to a draft of this review in November 2016
  

• 182

  This understanding comes from many conversations with AMF, most recently in November 2016.

• 183

  "With or without a co-funding partner, our sense, is NMCPs will first try and achieve funding from:
  a) Organisations from whom they have received funding before and with whom they have established relationships (know how the relationship operates, reporting requirements etc)
  b) Organisations with high levels of funding to minimise the number of agreements reached (often 2)
  c) Organisations who have the least accountability requirements. We do not have hard information to support this but is a view shared by others within the malaria and wider aid community." Rob Mather, AMF Founder, email to GiveWell, September 9, 2015

• 184

  Alliance for Malaria Prevention 2016 Q3 Net Mapping Project

• 185

  In 2014-2016 (assuming 75% of the nets that will be delivered in 2016 were delivered in the first 3 quarters), 191 million nets per year were delivered. 5 million nets per year is 2.6% of this. Alliance for Malaria Prevention 2016 Q3 Net Mapping Project

• 186

  AMF provided this information as a comment on a draft of this page.

• 187

  Rob Mather, AMF Founder, email to GiveWell, November 8, 2016