We have published a more recent supplementary information page. See our most recent supplementary information page for AMF.
This page contains further discussion and information for our review of Against Malaria Foundation. This page is intended to provide supplementary information on topics covered in the main review and is not intended to be read independently of the main review. The information on this page is less frequently updated than our main review; there is a note at the top of each section indicating when it was last updated.
Table of Contents
Registration process
Section last updated: November 2016
Malawi
Concern Universal, AMF's partner in Malawi, has collected data from each household on the number of sleeping spaces and long-lasting insecticide-treated nets (LLINs) on the condition that they have at least one year of remaining use. Households were allocated enough LLINs to cover all sleeping spaces, minus the number of existing, usable nets. Concern Universal have described the following pre-distribution registration survey (PDRS) process to us:1
- Concern Universal collects demographic data from the district health office, including the number of households in the region and the number of people per household.
- Concern Universal briefs the health facility supervisor (HSA), who will do the door-to-door surveying, about the PDRS.
- There are no detailed maps or geographical information system data for rural Malawi, so Concern Universal and HSAs rely on traditional authorities and community leaders to locate all the households in the targeted villages.
- HSAs go door-to-door to register all households in the targeted villages. A village leader accompanies the HSAs during this process. During household registration:
- The village leader and HSAs introduce themselves to the head of the household and explain why they are there.
- The HSA obtains permission to enter the household from the household head. (The difficulty of obtaining this permission varies based on the local culture. Local leaders can lend credibility to the HSA when asking for permission to enter a household.)
- Once in the household the HSA collects information including the name of the head of the household, the number of over-5 and under-5 people in the household, the number of sleeping spaces, and the number of viable nets owned by the household (The HSA asks household members for demographic information, and assesses the number of sleeping spaces and number of nets via visual inspection; 'viable' nets are nets that are less than 18 months old with fewer than 10 small holes and no large holes.)
- Concern Universal conducts "105%" data verification during the PDRS. HSAs are responsible for registering 100% of households in a village. In addition, supervisors independently register 5% of households in each village during the PDRS. This data is later compared to the HSA-collected data to check for consistency. Note that in September 2016, AMF told us that this piece of the process has not been emphasized and may not be happening consistently. We have seen only limited data from the data verification step from the Balaka 2015 distribution.2
- Concern Universal staff and government health officials supervise the registration.
- The data from the registration is entered into a Microsoft Access database, then printed out into registration books.
- After the registration, Concern Universal performs a data verification by assembling a few villages in each catchment area and taking a roll call of the household names in the registration book, and the number of nets that household is scheduled to receive. Verification is conducted for all villages. If someone claims to have been missed during this process, or claims that their number of scheduled nets is incorrect, then this is confirmed by a village leader, and, if confirmed, the correction is made.
We have seen pre-distribution survey results from all of AMF's completed large-scale distributions in Malawi: full data from its Ntcheu 2012, Balaka 2015, and Ntcheu 2015 distributions; and sample results from its Balaka 2013, Dedza 2014, Dowa 2015 distributions.3
DRC
IMA World Health, AMF's partner in the Democratic Republic of the Congo (DRC) used different registration processes for the two distributions it has completed (in Kasaï-Occidental and Nord Ubangi).
For the Kasaï-Occidental distribution, IMA World Health concurrently collected registration data on smartphones, distributed LLINs, and hung LLINs in recipients' homes. IMA World Health collected registration data on the number of sleeping spaces and previously-owned nets in usable condition in each household, and allocated enough LLINs to cover all sleeping spaces, minus the number of existing, usable nets. GPS coordinates identifying the location of the household and data on the number of AMF's LLINs hung up during the distribution process were also collected. We have seen full registration data from Kasaï-Occidental.4
In Nord Ubangi, DRC, registration data was also collected on smartphones, but it was collected prior to the distribution rather than during it. IMA World Health collected registration data on the number of sleeping spaces and previously-owned nets in usable condition in each household, and allocated enough LLINs to cover all sleeping spaces, minus the number of existing, usable nets. GPS coordinates identifying the location of the household and data on the number of AMF's LLINs hung up were also collected. We have seen sample registration data from Nord Ubangi.5
Ghana
In Ghana, volunteers organized by one of AMF's distribution partners, Ghana Health Services (the National Malaria Control Program is a part of Ghana Health Services), collected information on the number of people in each household prior to the distribution. Households were allocated one net per two people (rounding up for households with an odd number of people). Previously-owned nets in usable condition were not taken into account when allocating LLINs.6
AMF has shared all registration data collected from its Northern Region and Greater Accra distributions, and a summary of registration data from its Upper West distribution (which has not yet been validated, see here) with us.7
Distribution process
Section last updated: November 2016
Malawi
In Malawi, Concern Universal uses the following process for its distributions:8
After information is collected and verified, Concern Universal creates a distribution planning document which:
- Assigns each village to a distribution site, with a goal of minimizing the distance that beneficiaries need to travel (most beneficiaries travel 4-5 miles to their distribution site). Each catchment area is divided into several distribution sites, and each site serves 2-12 villages, depending on the geographic location, number of nets to be distributed, etc.
- Allocates a number of nets to be delivered to each location.
Concern Universal staff then meet with traditional authorities and district health officials to explain the distribution plan and set a date for the distribution, avoiding days that conflict with other community events. Concern Universal communicates the date, time, and place of distributions verbally to village leaders, who relay the information verbally to village members.
On distribution day, Concern Universal staff bring bed nets to the distribution sites using Concern Universal's vehicles or vehicles rented for that purpose. Village members typically walk to the distribution site; some travel by bike or organize transit in a motor vehicle. At the site, village leaders and HSAs group the beneficiaries by village. A roll call is taken, and as household names are called, members approach to receive nets. Household heads sign or fingerprint two copies of the distribution register to confirm receipt of their nets. If the household head is not present, a representative may receive nets on his/her behalf, as long as that person is recognized by the village head as being part of the household. When distribution is complete, each village leader receives a copy of the distribution register for their village, and Concern Universal retains the other copy.
An example of managing in-country shipping and storage logistics from the 2014 Dedza distribution (our understanding is that Concern Universal has used similar processes for its other distributions in Malawi):9
Upon arrival in-country and after completing all custom clearance formalities, the nets were stored in the main warehouse located within the premises of the District Hospital in Dedza Town pending relocation to the respective Health Centres[...] In order to ensure the security of nets, new padlocks were installed on all warehouse doors and the keys kept by CU’s storekeeper. Comprehensive insurance was contracted to cover potential losses caused by theft, fire or water damage and security guards were deployed throughout the distribution period, until all nets were distributed. All movements of nets were supervised by at least two CU staff members and the security guard[...] One or two days prior to distribution, the required quantity of nets was ferried to the health center for temporary safe keeping pending relocation to the distribution points on the day of distribution. Two vehicles were used to carry the distribution personnel and nets.
DRC
In Kasaï-Occidental, DRC, IMA World Health implemented registration and hang-up distribution at the same time. The following bullet points are a summary of the distribution process (more details in full distribution report).10
- "Nets arrived in Kinshasa on 22 Jul 2014 in 21 x 40HC shipping containers each containing approximately 32,000 nets." Pg 5.
- "The nets travelled 800 kms by road over the next three days and arrived in Tshikapa District with individual trucks driving directly to eight individual locations in each of the Health Zones where the relevant quantity of nets needed for the distribution in that district were pre‐positioned." Pg 6.
- "Meetings were held with local leaders to seek permission and support for the distribution." Pg 12.
- "Two categories of staff were required for the distribution. Field Supervisors (FSs), who had a management responsibility in each HA, and Community Health Workers (CHWs) who were responsible for visiting beneficiary households to collection registration information, hand out and hang the nets." Pg 12.
- "The CHW responsible for the hanging of the nets does so with the help of the head of household and/or other household members, using the string and nails brought along if necessary. A net is hung over each sleeping space (two people per sleeping space) and then rolled up out of the way (for example off the ground) during the day to help avoid unnecessary wear and tear or damage. It is then rolled down at night." Pg 17.
In Nord Ubangi, DRC, IMA World Health's distribution process was more similar to distributions in Malawi, with the separation of registration and distribution processes and the use of point distribution.11
Differences [between the Nord Ubangi distribution and the Kasaï-Occidental 2014 distribution]
- The North Ubangi distribution will be a two-phase distribution – a distinct pre-distribution phase and a subsequent distribution phase. In West Kasaï establishing net need per household and distributing of nets was contemporaneous.
- Distribution of nets will be done from distribution points rather than at the household
- The ODK registration form has been re-designed to better capture the required data, to eliminate some types of error and to reduce the chance of others.
- Operational lessons learned from the West Kasaï distribution will be implemented.
- An experienced, full time technology manager has been employed by IMA meaning there will be more extensive and easier liaising with AMF on data elements of the distribution. We have met the new technology manger and have spent a week working with him and are impressed.
Similarities [between the Nord Ubangi distribution and the Kasaï-Occidental 2014 distribution] [...] The same smartphone-based data collection technology will be used as in West Kasaï.
[...]
IMA has agreed there are benefits to conducting a separate, earlier pre-distribution phase during which accurate household-level net need is established. IMA are less able to estimate accurately population and net need in North Ubangi compared to what they felt was possible in West Kasaï. IMA believes point distributions rather than at-household distribution will be more cost-effective.Ghana
We have learned about the distribution processes for AMF's three distributions in 2016 in Ghana through conversations with AMF's distribution partners in Ghana (including the Greater Accra Regional Health Directorate of the Ghana Health Service, which managed in-country shipping and storage logistics for LLINs). From GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016:
Steps in the distribution process
- Beneficiaries walk a short distance to their distribution point. One Community Health Officer (CHO) employed by GHS and one GHS volunteer are stationed at each distribution point; there are never just two GHS volunteers.
- Before giving out each net, the distributors rip the packaging to help prevent reselling.
- Distributors gather together groups of recipients in order to transmit educational messages.
- The nets are distributed. Distributors make sure the recipients' coupons (brought by recipients) match the registration copies (brought by distributors) and make a mark on each side. The recipient's coupon is returned to them. Distributors also have a tally sheet with rows of 5 bubbles, and mark off a bubble for every net they give out. Distributors do not have an opportunity to sit or take a lunch break during distribution days.
[...] When deciding where to locate distribution points, the Greater Accra GHS considered safety, costs (for example, the cost of renting a warehouse) and proximity to other distribution points. They also selected 12-13 pre-positioning points in each distribution area.
Monitoring overview
Section last updated: November 2016
AMF's distribution partners also implement a set of monitoring activities to produce evidence on whether the registration and distribution processes operated as intended and on the long-term impact of the LLIN distribution. Monitoring activities have varied somewhat for different distributions.
- Process monitoring (i.e., the activities used to assess whether the registration and distribution processes operated as intended):
- Data validation: In Malawi, AMF's distribution partner Concern Universal provides registration lists to community leaders who read the lists out loud at a community meeting, supervised by Concern Universal, so community members can request edits.12 AMF told us that in Kasaï-Occidental, DRC, its partner IMA World Health employed supervisors who checked data for obvious errors and compared electronic and paper records. AMF has also checked the registration and distribution data for missing or inconsistent household-level records.13 We have not yet seen details of what data validation processes were used in Nord Ubangi, DRC. In Ghana, several different processes were used to validate data collected during registration and distribution: "pre-distribution data validation" (comparing the number of LLINs allocated to each community based on registration data to what would be expected based on the most recent census data, and double-checking and correcting—by phone or a follow-up household visit—the source of implausible registration data, e.g., households with more than 20 members), "post-distribution data validation" (checking whether data collected during the distribution indicates that households received appropriate numbers of LLINs), and, to check the accuracy of the electronic input of registration and distribution data, "6% data entry validation" (independent re-entering of 6% of the data collected during registration and distribution and checking it against the original electronic records). Details in footnote.14
- "Embedded" monitoring: In the 2016 LLIN distributions in Ghana, AMF's non-profit distribution partners, Episcopal Relief & Development and the Anglican Diocesan Development and Relief Organization (ADDRO), monitored the implementation of the registration and distribution processes by AMF's government distribution partner, Ghana Health Services. Episcopal Relief & Development and ADDRO were "embedded" in the registration and distribution processes; they attended district-level planning meetings to ensure that they were operating as intended, observed the registration of households by volunteers organized through Ghana Health Services, and observed selected distribution points (details in footnote).15 There was no "embedded" monitoring in AMF's distributions in Malawi and DRC since AMF worked with a single non-profit partner for those distributions.
- Distribution reports: Distribution reports provide narrative summaries of activities implemented and challenges encountered by distribution partners. Concern Universal has provided distribution reports for four of the six large-scale distributions it has completed in Malawi.16 IMA World Health has provided a distribution report for the Kasaï-Occidental, DRC 2014 distribution.17 We have not yet seen a distribution report from the Nord Ubangi, DRC distribution, which was completed in May 2016. From the three distributions in Ghana in 2016, we have seen a full distribution report from Episcopal Relief & Development from the Northern Region distribution, and short-term "Activity Reports" covering some periods of the pre-distribution and distribution activities in Greater Accra and the Upper West Region.18 It is our understanding that full distribution reports will be available for the Greater Accra and Upper West distributions later.
- Post-distribution validation tracing: After distributions were complete, AMF's distribution partners in Ghana checked that a random sample of households (100 households per district) had actually received the number of LLINs they were allocated by calling or visiting the households.19 Episcopal Relief & Development noted that, through this process, it discovered that a significant number of residents of two districts in Greater Accra, Ghana had not picked up the LLINs they had been allocated; we have not seen results beyond that.20 It is our understanding that similar processes have not been used in AMF's distributions in Malawi and DRC.
- Impact monitoring (i.e., the activities used to assess the long-term impact of the distribution):
- Post-distribution check ups (PDCUs): Distribution partners conduct follow-up surveys (called post-distribution check-ups, or PDCUs) in 5% of households at 6-month intervals for 2.5 years after a distribution. PDCUs estimate the presence, usage, and condition of LLINs from the AMF-funded distribution (details below).
We summarize which PDCUs have been completed and whether they were completed on time in this spreadsheet (see "PDCUs" sheet), and summarize the results of PDCUs we have seen in this spreadsheet.
Most scheduled PDCUs have been completed in Malawi. In 2016, AMF published data and reports from its first two PDCUs from Kasaï-Occidental, DRC, its first PDCUs completed outside of Malawi.21 We believe these PDCUs were poorly implemented and do not provide an accurate estimate of the proportion of AMF's LLINs that reached their intended destinations or the impact of AMF's distribution on LLIN usage over time in Kasaï-Occidental (see this blog post for details). PDCUs from Ghana are expected later.
- Malaria case rate data: Previously, AMF expected to collect data on malaria case rates from the regions in which it funded LLIN distributions: in the distribution agreements for several Malawi distributions, AMF asked Concern Universal to provide monthly malaria case rate data from before and after the distributions from all health centers in the three districts.22 In 2016, AMF shared malaria case rate data from Malawi covering January 2010 through March 2015 for Balaka, Ntcheu, and Dedza, and for Dowa from July 2012 to March 2015, but we have not prioritized analyzing it closely.23 AMF believes that this data is not high quality enough to reliably indicate actual trends in malaria case rates, so we do not believe that the fact that AMF collects malaria case rate data is a consideration in AMF’s favor, and do not plan to continue to track AMF's progress in collecting malaria case rate data.24
- Post-distribution check ups (PDCUs): Distribution partners conduct follow-up surveys (called post-distribution check-ups, or PDCUs) in 5% of households at 6-month intervals for 2.5 years after a distribution. PDCUs estimate the presence, usage, and condition of LLINs from the AMF-funded distribution (details below).
In previous versions of our review of AMF, we noted that AMF used "105%" registration data collection in some distributions. For distributions with "105%" registration data collection, after staff or volunteers have collected registration data from households, a different group of staff or volunteers revisit 5% of households and independently collect registration data.25 We do not believe this process, as it has been carried out, has been a useful form of monitoring. AMF and its partner in Malawi, Concern Universal, have not matched the two sets of registration data in time to correct potential errors in the distribution process (e.g., distributing the incorrect number of LLINs to a particular household). We have seen the 5% registration check data from one distribution in Malawi which AMF partially matched to the original data several months after the distribution was completed,26 and AMF has told us that 105% registration data was collected for other Malawi distributions but that it has not prioritized analyzing it.27 AMF has told us it sees the primary purpose of 105% data collection as being a tool to motivate data collectors, and that it does not need to check the data to achieve that goal.28
In previous versions of our review of AMF, we also discussed AMF's use of photos and videos as evidence that nets had reached their intended destinations. Because AMF has told us that it does not now consider photo and video footage part of its monitoring and evaluation activities, and because photos are missing for many recent distributions and there are fairly small numbers of photos available for other distributions, we no are longer tracking whether AMF is collecting photos and videos from distributions.
PDCU process
Section last updated: November 2016
"Post-distribution Check-Ups (PDCUs) are carried out every 6 months for 2.5 years (so 5 checks) with 5% of households in each Health Area selected at random and visited unannounced. These data are put in electronic form."29
Malawi
AMF and Concern Universal are in the process of revising the methodology used for PDCUs in Malawi. Concern Universal previously used the following procedure for its PDCUs:30
- From a list of all villages that received nets in a distribution, Concern Universal's project manager arbitrarily picks 5% of the villages to carry out PDCUs in. For each PDCU, the project manager tries to not select villages that are geographically close to each other or villages that have been visited in the previous PDCU. (Note that this selection procedure is not random and may be subject to bias.)
- For each selected village, the project manager arbitrarily selects 25 households to be surveyed during the PDCU. (Note that this selection procedure does not appear to be random and may be subject to bias.)
- On the dates of the PDCU, a supervisor or HSA from the local health facility accompanies the enumerator (a contracted employee of Concern Universal) to the village being surveyed. The HSA and enumerator meet with the village leader to discuss the purpose of their visit. The health facility staffer and enumerator arrive at the village unannounced to avoid knowledge of the PDCU influencing beneficiary behavior.
- The village leader (or a representative of the village leader) then accompanies the enumerator and health facility staffer to each household selected for the PDCU. The village leader knows the location of each selected household, and introduces the household head to the PDCU team.
- The household head gives consent for the PDCU team to enter the household and perform its check. If a household head refuses, the enumerator informs the project manager and the project manager selects another household. If the household head is away when the PDCU team arrives, the enumerator will return later in the day to perform the check. Household heads are frequently away when the PDCU team arrives, often because they are doing fieldwork far from their home.
- During the PDCU check, the enumerator:
- assesses the number of regularly used sleeping spaces.
- asks how many people are in the household, and how many have had blood-diagnosed malaria in the past month.
- checks the brand of the bed nets. Nets are recorded either as 'Permanet,' 'Olyset' (the two brands AMF distributes), or 'other.'
- records whether the net was distributed by an AMF-funded distribution. AMF-funded nets have a tag with the AMF logo and the date of the distribution.
- assesses the net condition. Nets are assessed as 'very good,' 'okay,' or 'poor.'
- A net in very good condition has two or fewer holes of less than 2 cm.
- A net in okay condition has fewer than 10 small holes (less than 2 cm).
- A net in poor condition has more than 10 small holes, or one large hole (greater than 10 cm).
Net quality is assessed by physical examination by the enumerator. If a net is not hung, the net is unfolded and physically assessed.
- asks who sleeps under each net, categorized by children under 5, children ages 6-18, pregnant women, or other people.
- asks how many nets were originally distributed to the household during the AMF distribution, and also notes how many of these nets are not present, or worn out.
- records how many nets in the household are hung, and how many are not hung.
- assesses whether nets are being used correctly. For this assessment, the enumerator makes a judgment call about whether the nets being are used properly. Often, nets will not be hung during the day due to limited space, but still may be used properly at night.
Ghana
Episcopal Relief & Development agreed to carry out post-distribution check-up surveys for the three distributions in Ghana in 2016.31
AMF has told us that, for the first PDCU for the Northern Region in Ghana in November 2016, there will be "5% of the 5%" data collection—a separate set of data collectors re-visiting a random selection of 5% of the households selected for the PDCU to re-collect data. AMF also told us that it expects to be able to quickly match the "5% of the 5%" data to the original PDCU data, and intends to use this check to assess the accuracy of PDCU data collection.32
PDCU definitions
Section last updated: November 2016
The following definitions are based on our understanding from multiple sources and may not have been used fully consistently for each post-distribution survey (PDCU). Definitions:
- Nets hung: AMF told us that interviewers are asked to observe whether nets are hung by entering interviewees' houses (rather than simply asking interviewees if they are hung). Concern Universal told us that PDCU surveyors use their judgment to assess whether nets are being hung properly when they are interviewing recipients and inspecting their nets and sleeping spaces.33 It is our understanding that none of the nets included in the "nets hung" category are "worn out" (i.e., all are in "viable", "good", or "very good" condition).34
- "Very good" condition: LLIN has fewer than 2 holes of less than 2 cm in size.35
- "Good" condition: LLIN has fewer than 10 small holes.36
- "Viable" condition: LLIN has more than 10 small holes or has one large hole.37
- "Worn out": Survey reports from Concern Universal did not provide a definition of this term. AMF has told us in the past that an LLIN is considered worn out if it has multiple large holes and the LLIN is unrepairable, such that it would not provide protection against mosquitoes.38
- Used correctly: According to a survey form from 2012, interviewers are asked to "ask the householder to demonstrate how the nets are used at night" and to then select yes or no to the question "are the nets being used correctly."39 In 2014, AMF told us that the procedure had not changed in more recent surveys.40 Note that some of the nets included in this category may not be AMF nets and that it is possible that some "used correctly" nets are also "worn out" nets.41 In this blog post, we discuss the possibility of bed nets being used for other purposes than covering sleeping spaces (e.g., being used as fishing nets). We have not seen evidence that AMF-distributed nets are being misused at a meaningful scale, and the PDCU results we have seen to date indicate that, at least in Malawi, a large majority of AMF-distributed nets are being used appropriately.
- People covered by nets: We are unsure of how to interpret this metric. We previously believed that this included all people who sleep under a properly hung and not worn out net. However, the data for some post-distribution check-ups is not consistent with this interpretation.42
What's included in our cost per LLIN estimates
The estimate for completed distributions in Malawi uses:43
- The purchase price per LLIN, calculated from AMF's total LLIN budget and the number of LLINs distributed.
- Actual costs for five completed large-scale distributions in Malawi as reported by Concern Universal44 and a combination of budgeted and actual costs for post-distribution check-up surveys following these distributions.45 AMF told us that the non-net budgets include all monetary costs incurred by Concern Universal, as well as the use of vehicles and managerial staff time;46 they may exclude non-monetary costs such as the use of office space.
- An estimate of non-monetary costs incurred by Concern Universal and local governments, based on estimates provided in 2012 by Concern Universal for its distribution in Ntcheu province.47
- A calculated estimate of the non-monetary costs incurred by Concern Universal and local governments due to post-distribution check-ups (calculated based on the Ntcheu 2012 estimate discussed above).48
- An estimate of AMF's organizational costs per net, including an estimate of the value of donated services and volunteer time. Because these costs generally do not vary with the number of LLINs distributed and remain roughly constant on a per year basis, the per LLIN estimate is highly dependent on the time period used and the number of LLINs distributed in that period. We estimate AMF's organizational costs as $0.29 per LLIN for July 2011 through the beginning of October 2016.49
For DRC, the estimate is rougher. We constructed the estimate similarly, although the information we have for DRC is likely of lower quality.50 The estimate for the DRC is based on:51
- The purchase price per LLIN, calculated from AMF's total LLIN budget and the number of LLINs distributed.
- Actual costs for the Kasaï-Occidental 2014 and Nord Ubangi 2015 distributions.
- Actual costs for the first three post-distribution check-ups following the Kasaï-Occidental 2014 distribution, an estimate of the costs of the remaining Kasaï-Occidental post-distribution check-ups based on the costs of the completed post-distribution check-ups, and budgeted costs for the post-distribution check-ups for the Nord Ubangi 2015 distribution.
- The same estimate of AMF organizational costs as we used for Malawi.
- Calculated estimates of non-monetary costs incurred by IMA World Health and local governments, based on the estimate from the 2012 distribution in Ntcheu, Malawi.52
The higher cost per net in the DRC may be explained by a considerably more difficult operating environment in DRC,53 or our data for DRC may simply be of lower quality.
The estimate for Ghana is also rough; it is based on:54
- The purchase price per LLIN, calculated from AMF's total LLIN budget and the number of LLINs distributed.
- Budgeted costs for pre-distribution, distribution, and post-distribution activities (primarily monitoring activities, including post-distribution check-ups) from AMF's non-profit distribution partner in Ghana, Episcopal Relief & Development.55
- A rough estimate of costs incurred by Ghana Health Services for its implementation of the distributions.56
- The same estimate of AMF organizational costs as we used for Malawi and DRC.
- Calculated estimates of non-monetary costs incurred by Episcopal Relief & Development, ADDRO, and local governments, based on the estimate from the 2012 distribution in Ntcheu, Malawi.57
Our estimates of the cost per net for planned distributions that AMF has signed or is considering are also rough; they are based on:
- The purchase price per LLIN, based on what AMF told us it expects the cost to be.
- AMF's estimate of the non-net costs associated with pre-distribution surveys, distribution days, and post-distribution check-ups for each upcoming distribution.58 These estimates are consistent with what we would expect based on past distributions in Malawi, DRC, and Ghana.59
- An estimate of AMF's organizational costs per net in the future, including increases in AMF's salary costs in FY 2017 and FY 2018 due to planned hiring of additional staff.60 We estimate AMF's organizational costs will be about $0.08 per net in the future, compared with $0.29 in our estimates for AMF's past distributions in Malawi, DRC, and Ghana.61
We are uncertain about the use of marginal funding to AMF this year. Our best guess at the cost per LLIN in distributions AMF may fund with additional funding this year is an average of the cost per LLIN for all of AMF's past large-scale distributions and its planned distributions discussed above (weighted by AMF's spending in each country). We calculate that the weighted average is $4.85 per LLIN.62
Global estimate of cost per LLIN
The Roll Back Malaria Partnership, which estimates funding gaps for LLINs across all African countries,63 uses an estimate of $2.70 to purchase an LLIN and $2.50 for all non-net costs of a distribution, for a total of $5.20 per LLIN.64 This assumption is used in cases where country-specific data is not available. Note that we have not seen details about where this estimate comes from, so we put little weight on it. The Co-Chair of the Roll Back Malaria Harmonization Working Group told us that non-net costs can vary significantly by country and that non-net costs in DRC may be more than $5 per LLIN.65 We have little information on how reliable the Roll Back Malaria Partnership's estimate is likely to be. Broadly, this estimate is consistent with the data we have seen from AMF. We have used estimates based on data from AMF in our cost per life saved calculations, because we have more information on their reliability. We do not believe our estimate of AMF's cost-per-net ought to be compared directly to the Roll Back Malaria Partnership's estimate, given how uncertain we are about the latter.
Specific country funding gaps
Section last updated: November 2016
We spoke to individuals who are involved in national planning for mass LLIN campaigns in two countries where AMF is interested in working in the future. They provided details on why other funding sources are unlikely to be sufficient to replace LLINs when the next campaign is due.
In one case, the next LLIN campaign will be due before it is likely feasible for Global Fund funding to be secured and used to order and ship LLINs. AMF may be able to move up the distribution by a year or more. There are also expected to be funding gaps for the next campaign, regardless of when it occurs, because, due to government regulations and systems, the country has been slow to spend Global Fund resources and, as a result, the Global Fund has reduced how much funding it allocates to the country. Note that the Global Fund has not yet announced its allocation to the country for the next round, so it is possible that the allocation will be higher than expected.
In another case, past campaigns have had major funding gaps, and recently two major funders of previous mass campaigns, the President's Malaria Initiative and the UK Department for International Development, have shifted their focus to other malaria interventions, such as indoor residual spraying, treatment for pregnant women, strengthening health systems, and private sector case management. These two funders are not expected to provide as much funding for mass LLIN campaigns in the future as they have in the past.
All sources for AMF review
Section last updated: November 2016
- 1
Nelson Coelho, conversation with GiveWell, April 15, 2016
- 2
- @Rob Mather, AMF Founder, conversation with GiveWell, September 28, 2016@
- Concern Universal Balaka 2015 registration data 100% vs 5%
- 3
- Concern Universal Ntcheu 2012 pre-distribution registration survey data
- Concern Universal Balaka 2013 pre-distribution registration survey data
- Concern Universal Dedza 2014 pre-distribution registration survey data
- Concern Universal Dowa 2015 pre-distribution registration survey data
- Concern Universal Balaka 2015 registration data 100% vs 5%
- Concern Universal Ntcheu 2015 pre-distribution registration survey data
- 4
- AMF: "DRC, West Kasaï Province: Distribution Report and separate Technology Report":
- "In West Kasaï the fight against malaria involves the distribution and hang‐up of LLINs, a so‐called ‘Hang‐Up’ strategy, in which volunteer Community Health Workers (CHWs) distribute the nets and hang them at the same time. This is in contrast to strategies that simply hand out the nets. A modified version of the Hang‐Up strategy, called a ‘Hang‐Up and Track’ (HUT) strategy, was used on this distribution. With the HUT approach, in addition to distribution and hang‐up, information on each net hung is captured using smartphones equipped with a data collection program." Distribution Report, Pg 4.
- "Teams of three or four CHWs [Community Health Workers] visited households sequentially.
This was a crucial stage of the net distribution that consisted of collecting relevant household data for every single household in the distribution catchment area including, but not limited to, the following ‘must have’ information:
- Name of household head
- Number of people in the household
- Number of sleeping spaces (‘for the purpose of nets’*)
- Number of perfectly good nets with at least two years of life left in them
- Location of household (GPS coordinates)
- Household identifier**
The last two pieces of information were to facilitate later identification of the individual household for post‐distribution check‐up work.
* When assessing the number of sleeping spaces it is important to avoid confusion between the number of separate areas in which individuals might sleep and, given nets distributed are ‘double’ nets that will accommodate two people, the number of sleeping spaces that is equal to the number of nets that are needed. For example, in a household with mum plus dad, four children aged eight, four, six and two and a grandfather there may be six separate sleeping areas: mum plus dad, one each for the four children and one for the grandfather. However, for the purposes of nets, there would be four sleeping spaces: one for mum and dad, one for the two older children, one for the two younger children and one for the grandfather.
**Creating a household identifier is necessary as many households do not have clear addresses. For example, it is frequently the case in rural and semi‐rural locations that street names and household numbers do not exist." Pg 16.
- IMA World Health, Kasaï-Occidental 2014 distribution data
- AMF: "DRC, West Kasaï Province: Distribution Report and separate Technology Report":
- 5
- "Differences [between the Nord Ubangi distribution and the Kasaï-Occidental 2014 distribution]
1. The North Ubangi distribution will be a two-phase distribution – a distinct pre-distribution phase and a subsequent distribution phase. In West Kasaï establishing net need per household and distributing of nets was contemporaneous.
2. Distribution of nets will be done from distribution points rather than at the household
3. The ODK registration form has been re-designed to better capture the required data, to eliminate some types of error and to reduce the chance of others.
4. Operational lessons learned from the West Kasaï distribution will be implemented.
5. An experienced, full time technology manager has been employed by IMA meaning there will be more extensive and easier liaising with AMF on data elements of the distribution. We have met the new technology manger and have spent a week working with him and are impressed.Similarities [between the Nord Ubangi distribution and the Kasaï-Occidental 2014 distribution]
1. The same smartphone-based data collection technology will be used as in West Kasaï.Comment
IMA has agreed there are benefits to conducting a separate, earlier pre-distribution phase during which accurate household-level net need is established. IMA are less able to estimate accurately population and net need in North Ubangi compared to what they felt was possible in West Kasaï.IMA believes point distributions rather than at-household distribution will be more cost-effective." Rob Mather, AMF Founder, email to GiveWell, September 9, 2015
- "IMA will ensure the collection and availability, for independent inspection, of household level data (sleeping spaces and number of perfectly usable nets in place) for all households in the distribution zones identified in section 4." IMA World Health and AMF distribution agreement for Nord Ubangi 2015
- IMA World Health Nord Ubangi 2015-16 registration data
- "Differences [between the Nord Ubangi distribution and the Kasaï-Occidental 2014 distribution]
- 6
- GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016:
- "GHS [Ghana Health Services] volunteers carry out the registration process, which takes place more than one month before the distribution. Volunteers are organized by the Ministry of Health (MoH) and participate in a number of government health programs, such as vaccination campaigns (immunization days). Many of them are long-time volunteers with considerable experience." Pg. 5.
- " Volunteers visit each house in their zone. The head of household, or someone else who is present, is asked how many people are in the household based on the 'people who eat from the same pot' definition. Ghana's policy is to provide one net for every two individuals in a household; as a result, in contrast to previous AMF distributions, volunteers were not required to check households' existing nets or number of sleeping spaces. The only exception to this policy is that household members aged 70 or older are not counted in the household total, and each receive their own net." Pg. 5.
- GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016:
- 7
- AMF Ghana Data Entry System 2016 (for the Northern and Greater Accra regions).
- AMF Upper West Region Ghana pre-validation registration data 2016
- The full registration data we have seen in AMF Ghana Data Entry System 2016 includes names of heads of households and either an address or a written description of the location of a household (e.g., "Next to the church").
- 8
GiveWell's non-verbatim summary of a conversation with Nelson Coelho, April 15, 2016, pg. 4.
- 9
- 10
- 11
Rob Mather, AMF Founder, email to GiveWell, September 9, 2015
- 12
- "A copy of the beneficiary list for their village or community will be passed to the relevant leaders to check for duplications, omissions, or inaccuracies. This will include reading out the list publically so the community is involved and engaged in the verification and the net distribution preparations more generally. CU field staff supervise this process." Concern Universal, Dowa 2015 planning document, pg. 7.
- Based on a conversation with AMF, it is our understanding that this same process is used for all of AMF's distributions in Malawi with Concern Universal. Rob Mather, AMF Founder, Conversation with GiveWell, September 28, 2016
- The results of the registration data verification process in the Dedza 2014 distribution:
- "During the verification process, CU staff roll-called the beneficiaries from the printed registers whilst the HSAs cross-checked with their list compiled from the registration forms in order to detect any mismatch. This verification exercise revealed that a total of 32,183 households had been skipped during the registration.
[…]
"After the village household verifications, the corrected and edited registers were referred back to the data entry team and the corrections made to the database." Concern Universal Dedza 2014 distribution report Pg 5.
- "During the verification process, CU staff roll-called the beneficiaries from the printed registers whilst the HSAs cross-checked with their list compiled from the registration forms in order to detect any mismatch. This verification exercise revealed that a total of 32,183 households had been skipped during the registration.
- Results of verification process for other large-scale AMF distributions in Malawi:
- Concern Universal Ntcheu 2012 distribution report, Pgs 5-7.
- Concern Universal Balaka 2013 distribution report, Pg 6.
- We have not yet seen the results of the registration data verification process for AMF's other large-scale distributions in Malawi (Dowa 2015, Balaka 2015, and Ntcheu 2015). We expect that the results of the registration data verification process will be reported in the full distribution reports for these distributions.
- 13
- IMA World Health, Kasaï-Occidental 2014 distribution report:
- "Household data went through the following steps:
- Data collected at the household by a CHW
- Independent supervisor uploaded the data from the smartphone to a laptop.
- The supervisor reviewed the data and corrected obvious mistake or inconsistencies."
Pg 18.
- "After the distribution had been completed in a Health Area and all data uploaded from the smartphones, the data team compared the electronic data with the paper data collected by each distribution site in each health area. At the end of the day each CHW returned undistributed nets to the supervisor and depot manager and reported the number of nets they distributed. These numbers were recorded on paper." Pg 18.
- "Household data went through the following steps:
- We have not seen the results of the supervisors' validation activities (e.g., the number of household records corrected for obvious errors and for discrepancies between electronic and paper records).
- We have seen a high level analysis of the quality of the registration and distribution data, which finds that some data is available for 91% of the LLINs distributed (data is missing for the other 9% of LLINs). For households for which data is available, 96% had internally consistent data and 4% had apparent errors or anomalies. AMF distribution verification Kasaï-Occidental 2014, pg. 3.
- We are not certain when AMF completed this analysis, or if this analysis was used in-country (e.g., to check whether households with inconsistent data had received the appropriate number of LLINs).
- IMA World Health, Kasaï-Occidental 2014 distribution report:
- 14
- "Step 7 - Pre-distribution validation
In all three regions with AMF-supported distributions, all coupons underwent a pre-distribution validation process by the NMCP and non-profit partners.
At the community level, total estimated nets (based on 2010 census data) are compared to the total number of nets allocated during registration. This gives a sense of the variance between these figures; communities with more than 2% variance might be spot-checked.
The validators also look for other potential problems with the coupons, including:- illegible entries
- households with over 20 members
- mismatches between the numeral and text versions (both are required) of figures in the 'nets allocated' field
- miscalculation of figure in 'nets allocated' field
If there are problems, the representative follows up with the volunteer who collected the data. The volunteer might return to the household to correct mistaken entries; if not, the representative will do so in person, or if that is too costly, by phone. If the information cannot be verified or corrected, the coupon is discarded." GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, pg. 6.
- "This activity [pre-distribution validation] has been completed in the Northern and Greater Accra regions. In both regions, ADDRO observed a few mismatches in terms of number of members registered as a HH and the number of LLINs allocated. Following discovery of this misinformation, validation teams returned to the communities to verify valid information." Episcopal Relief & Development Ghana Activity Report 1 2016, pg. 6.
- We have not seen comprehensive data on the results of the pre-distribution data validation process (e.g., exact numbers of household records corrected by this process).
- "Post-distribution validation
"After the grace period, coupon books are returned to the district level for the post-distribution validation process. This is carried out by the NMCP and non-profit partners, and involves analyzing the coupon books in order to:- Check for coupons that were not crossed out; for example, this scenario might arise if a household did not pick up their nets.
- Check that the number of nets allocated matches the number of nets given.
- Check that the number of nets allocated is accurate based on household size.
- Add total values for household size, nets allocated, and nets given, and record them on the inside cover of each coupon book."
- We have not seen comprehensive data on the results of the post-distribution data validation process (e.g., exact numbers of household records corrected or households re-visited during this process).
- "As it might be confusing to use a 5% figure in too many contexts, AMF and Episcopal Relief & Development decided to run verifications on 6% of the data entered by clerks. This data is reentered by another individual and checked against the original data. If there are discrepancies, the coupon book is verified." GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, pg. 13.
- We have seen results from the 6% data entry checks for the Greater Accra distribution, but (as of November 2016), not from the Northern Region or Upper West Region distributions. As of November 6, 2016 33,269 "6% check" records had been entered (5.4% of the 566,947 total records). 2,586 of these 6% check records had not yet been matched to the original electronic records. Of the 30,683 matching pairs of records, 10,632 (35%) were an exact match without correction from a supervisor, an additional 19,598 (64%) had "key data" (number of people in a household, LLINs allocated, and LLINs give) match without correction from a supervisor, and 247 (0.8%) had mismatches of key data. AMF Ghana Data Entry System 2016
- "Step 7 - Pre-distribution validation
- 15
- "Instead, Episcopal Relief & Development and ADDRO carried out 'light monitoring,' and considered themselves 'embedded' in the pre-distribution and distribution work. Monitoring activities included:
- Attending meetings, including planning meetings at the national level, as well as a sample of district meetings and training sessions. This was done to observe whether activities were proceeding according to the NMCP's checklist and, if there were issues, to follow up with the NMCP.
- ADDRO monitors observed the registration process in a randomly selected sample of 50% of the 20 districts in the Northern Region and all districts in the other two regions. Names of all of the sub-districts in these districts were written on slips of paper and randomly selected. Episcopal Relief & Development reported that it did not skip any of the randomly selected districts, even those that were far away. The order of sub-district visits was determined by what made sense geographically.
- Monitors called the supervisors before arrival and visited some households to see whether they were aware that registration was happening in their area that day; almost everyone was aware.
- Monitors also observed the work of multiple community-based agents (CBAs); sometimes, they would do a cross-check by performing this task without the supervisor. They listened to the CBA's conversations with households, verified that the correct information had been recorded in the book, and addressed any issues with the CBA. They then went back through the route traveled by the CBA to ensure that households that should have been registered were registered, and that the coupons had been correctly filled out. This process is facilitated by the practice of marking registered houses. Monitors also asked households if they were aware of the date and location of the distribution.
- Monitors met with supervisors, analyzed completed coupon books to identify any problems, and filled out forms summarizing their daily observations.
- Monitors might address issues by:
- Debriefing with supervisors at the end of the day, so that learnings could be applied to the next day's work;
- Communicating directly with CBAs if they encounter the CBA before meeting with the supervisor;
- Observing CBAs without informing the supervisor, if they observed a number of issues the previous day." Pg. 7.
- "Monitoring of the distribution process by ADDRO
ADDRO has two or more monitoring teams per region; each of these has a vehicle. The NMCP has 2-3 evaluators per district; each has a vehicle. ADDRO representatives attend the NMCP planning meeting. ADDRO monitors introduce themselves to the municipal health directorate, and are taken by a representative to each distribution point in a sub-district to ensure that their presence will be accepted by those whose work they will be monitoring.In the Greater Accra Region, the malaria focal person for the district took monitors to a distribution point. From there, they went to other distribution points in the sub-district. They asked the focal person which areas were the least accessible and attempted to focus on those; in total, they visited 60% of the distribution points. They had the distributors' phone numbers so they could ask for directions if necessary.
Monitors verified that:
- nets were being distributed
- distributors were using the tally sheet
- nets were kept in the shade
- coupons were marked when nets were distributed
- a Ghana Health Services staff member was present
- distributors were giving helpful messages to the recipients
NMCP monitors were also present during the distribution." Pgs. 10-11.
- We do not have a clear understanding of exactly how ADDRO monitors choose which distribution points to observe, but note that the selection process described above does not appear to be random.
- Examples of challenges encountered and actions taken during distribution by ADDRO monitors in Zabzugu district, Northern Region:
- "A lot of household members did not understand the distribution method of giving one LLIN to two persons per household according to the universal coverage strategy. Although this was explained to them and most of them understood, there were complaints by a few.
- On the first day of the distribution, the DP attendants could not locate some few booklets. Apparently, these had been mistakenly taken to the district, making it difficult for some households to redeem their nets. Exercise books were procured to capture the information of those households as they presented their coupons to enable them redeem their nets. Arrangements were made to get the booklets from the region to the distribution point.
- Some households who were not registered turned up at the distribution sites. For example, all the households in the Zabzugu police barracks were not registered due to misinformation from their command about the registration procedure (i.e., that they don’t need to register as the nets will be given to them without prior registration). All these households did not benefit from the nets because of the no coupon, no net policy.
- At a distribution point in Zabzugu sub-district, DP attendants detected that some households altered the household numbers on their coupons, ostensibly to receive more nets. The DP attendants noted the right numbers on the coupon counterfoil (validated data) and gave out the correct number of nets to the households.
- Some beneficiaries waited all day to receive their nets because the booklets from which they were issued coupons could not be found on the day of the distribution. They booklets were mistakenly sent to different distribution point but were eventually brought to the DP and the people received their nets."
Episcopal Relief & Development Ghana Northern Region distribution report 2016, Pg 6.
- Challenges encountered and actions taken by ADDRO monitors for other districts in the Northern Region reported in Episcopal Relief & Development Ghana Northern Region distribution report 2016, Pgs 6-10.
- Challenges encountered and actions taken during distribution point monitoring for the distribution in Greater Accra reported in Episcopal Relief & Development Ghana Activity Report 2 2016, Pg 6.
- The distribution reports we have seen so far from Episcopal Relief & Development do not cover challenges encountered and actions taken during distribution point monitoring for the Upper West Region distribution.
-
GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016:
- "Instead, Episcopal Relief & Development and ADDRO carried out 'light monitoring,' and considered themselves 'embedded' in the pre-distribution and distribution work. Monitoring activities included:
- 16
- AMF page on Ntcheu 2012 distribution
- AMF page on Balaka 2013 distribution
- AMF page on Dedza 2014 distribution
- AMF page on Dowa 2015 distribution
We have not yet seen distribution reports for the Ntcheu 2015 or Balaka 2015 distributions in Malawi.
- 17
- 18
- 19
- Episcopal Relief & Development Ghana Activity Report 3 2016:
- "The validation was to establish precisely how many LLINs were distributed during the distribution campaign in all 12 districts. Apart from validating counterfoils, the validators undertook 'End-User Verification'. The end-user verification (EUV) is a rapid check-up to determine whether the beneficiaries really received the number of LLINs allocated for the households and are using the LLINs for the intended purpose. This involved randomly selecting 100 households in each district to verify LLINs received, LLIN use by household members, etc. Validators randomly sampled 100 booklets and from each booklet, randomly sampled one coupon counterfoil for the EUV visit. Validators then called the beneficiaries of the sampled coupon counterfoils and followed up to the households for the end-user verification exercise." Pgs 4-5.
- "24 NMCP validators were assigned to work in the 12 AMF supported districts (2 validators per district). Two teams from ADDRO (a team from the ADDRO HQ and a team from the ADDRO Greater Accra office) visited all the 12 AMF districts to monitor the validation process and to provide support for the packaging of coupon counterfoils for transportation to ADDRO headquarters in Bolgatanga. Key findings were as follows:
- It was comparatively easier for the validators/monitoring team to enter into bedrooms/sleeping places of beneficiaries in the rural areas to inspect or observe net usage (LLINs hanging and being used) than it was in the urban areas. Residents in the urban areas felt very reluctant to allow ‘strangers’ to observe their sleeping places.
- The Ashaiman and Ningo Prampram districts had significant numbers of nets not redeemed. For example, Ashaiman had 127 bales (12,700 pieces) at the District Health Directorate not distributed. The reasons given were that during the distribution in July, there was a LLIN shortage (because the districts had not received all their nets) so later when the districts finally had their nets, beneficiaries did not turn up to claim them. The GHS staff had called most of the beneficiaries per the contact numbers in the coupon counterfoils but only few came for the LLINs. This issue was reported to NMCP to take action."
Pgs 4-5.
- We have not seen comprehensive results from the post-distribution tracing exercises in Ghana (e.g., the proportion of selected households that had originally received the appropriate number of LLINs).
- "In Greater Accra, post-distribution validation tracing was implemented for a random sample of households. The same process is planned for the Upper West Region. The Global Fund imposed this requirement." GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, pg. 10.
- Episcopal Relief & Development Ghana Activity Report 3 2016:
- 20
- Episcopal Relief & Development Ghana Activity Report 3 2016:
- "The validation was to establish precisely how many LLINs were distributed during the distribution campaign in all 12 districts. Apart from validating counterfoils, the validators undertook 'End-User Verification'. The end-user verification (EUV) is a rapid check-up to determine whether the beneficiaries really received the number of LLINs allocated for the households and are using the LLINs for the intended purpose. This involved randomly selecting 100 households in each district to verify LLINs received, LLIN use by household members, etc. Validators randomly sampled 100 booklets and from each booklet, randomly sampled one coupon counterfoil for the EUV visit. Validators then called the beneficiaries of the sampled coupon counterfoils and followed up to the households for the end-user verification exercise." Pgs 4-5.
- "24 NMCP validators were assigned to work in the 12 AMF supported districts (2 validators per district). Two teams from ADDRO (a team from the ADDRO HQ and a team from the ADDRO Greater Accra office) visited all the 12 AMF districts to monitor the validation process and to provide support for the packaging of coupon counterfoils for transportation to ADDRO headquarters in Bolgatanga. Key findings were as follows:
- It was comparatively easier for the validators/monitoring team to enter into bedrooms/sleeping places of beneficiaries in the rural areas to inspect or observe net usage (LLINs hanging and being used) than it was in the urban areas. Residents in the urban areas felt very reluctant to allow ‘strangers’ to observe their sleeping places.
- The Ashaiman and Ningo Prampram districts had significant numbers of nets not redeemed. For example, Ashaiman had 127 bales (12,700 pieces) at the District Health Directorate not distributed. The reasons given were that during the distribution in July, there was a LLIN shortage (because the districts had not received all their nets) so later when the districts finally had their nets, beneficiaries did not turn up to claim them. The GHS staff had called most of the beneficiaries per the contact numbers in the coupon counterfoils but only few came for the LLINs. This issue was reported to NMCP to take action."
Pgs 4-5.
- "In Greater Accra, post-distribution validation tracing was implemented for a random sample of households. The same process is planned for the Upper West Region. The Global Fund imposed this requirement." GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, pg. 10.
- AMF notes that: "Our understanding is this was a temporary logistical issue and we asked for this to be addressed in the next ERD report. We have now written to ERD and asked for a quicker, specific update on this issue." Comment provided in response to a draft version of this review in November 2016.
- Episcopal Relief & Development Ghana Activity Report 3 2016:
- 21
- 22
- We discuss case rate data from Ntcheu in our February 2013 update on AMF.
- "[AMF:] Please confirm you are able to provide monthly malaria case rate data going back at least 12 months for each health centre/clinic in the distribution area and will continue to provide monthly data for a period of four years post‐distribution. This ensures we understand pre‐distribution malaria levels and can monitor them post‐distribution.
[Concern Universal:] Malaria case rate information for each of the 16 Health Centres is available from the Balaka District Health Office using the Malawi Government’s national Health Management Information System (HMIS). This information will be sent to AMF separately." Concern Universal Balaka 2013 distribution proposal, pg. 2 - Concern Universal Dedza 2014 distribution proposal, pg. 2, contains the same language as above
- 23
- Balaka 2010-2015 MCRD
- Dowa 2012-2015 MCRD
- Ntcheu 2010-2015 MCRD
- Dedza 2010-2015 MCRD
- 24
AMF told us that the data is likely to be poor quality, and that it will put little to no weight on the results. Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015
- 25
"After the HSAs have collected data from all villages, spot-checkers independently collect the same data from 5% of households in each village. The spot-checkers are employed by the government and receive lunch allowances paid by Concern Universal during the time they are engaged in the exercise. This data is later compared to the HSA data to check for consistency. The two sets of data are entered into a Microsoft Access database separately. Reconciliation of the data is done after data entry. The data is used to calculate the number of nets required in each village, health center catchment area, and district. A printed register including all information recorded during registration is created for each village." GiveWell's non-verbatim summary of a conversation with Nelson Coelho, April 15, 2016, pg. 3.
- 26
Concern Universal Balaka 2015 registration data 100% vs 5%
- 27
AMF told us that 105% registration data was collected for the Dowa 2015 and Ntcheu 2015 distributions. @Rob Mather, AMF Founder, conversation with GiveWell, September 28, 2016@
- 28
@Rob Mather, AMF Founder, conversation with GiveWell, September 28, 2016@
- 29
- AMF Summary features of an AMF distribution
- PDCUs every six months for 2.5 years are included in the distribution agreements AMF signed for its distributions in Ghana and Uganda. Ghana: AMF Ghana 2016 distribution agreement; Uganda: AMF Uganda 2016 distribution agreement
- AMF used to require that distribution partners conduct follow-ups every 6 months for up to 3 or 4 years.
- "[AMF:] Post-Distribution Surveys (see Note 3). Please confirm you will carry out Post-Distribution Surveys (PDSs) every 6 months post-distribution for a period of up to four years to assess the level of net usage (hang-up %), correct usage and condition of the nets and you will provide us with the findings. Each survey would cover approximately 5% of households.
[Concern Universal:] I can confirm Concern Universal and Dedza District Council will conduct a Post-Distribution Survey every six months following the distribution." Concern Universal Balaka 2013 distribution proposal, pg. 2 - Concern Universal Dedza 2014 distribution proposal, pg. 2 contains the same language.
- IMA World Health agreed to conduct surveys every six months for 3 year post-distribution. IMA World Health and AMF distribution agreement for Kasaï-Occidental 2014, pgs. 3-4 and IMA World Health and AMF distribution agreement for Nord Ubangi 2015, pgs. 3-4.
- "[AMF:] Post-Distribution Surveys (see Note 3). Please confirm you will carry out Post-Distribution Surveys (PDSs) every 6 months post-distribution for a period of up to four years to assess the level of net usage (hang-up %), correct usage and condition of the nets and you will provide us with the findings. Each survey would cover approximately 5% of households.
- 30
Nelson Coelho, conversation with GiveWell, April 15, 2016
- 31
"Post-distribution Check-Ups (PDCUs) will be carried out every six months for 2.5 years with five PDCU to be conducted in total. With the PDCUs, all households will be visited by trained volunteers who will collect household data. A different set of volunteers from the ones who will collect data on all households will be trained to visit unannounced, 5% of households in each community and collect data." Episcopal Relief & Development Planning Document 2016, Pg 23.
- 32
Rob Mather, AMF Founder, email to GiveWell, October 31, 2016
- 33
- Rob Mather, AMF Founder, email to GiveWell, June 30, 2014
- A representative of Concern Universal told us that, due to limited indoor space, some recipients take down their nets during the day and hang them up each night. When assessing whether a net is being used, enumerators observe the net and the sleeping space, ask the recipient questions, then make a judgment call. A net that is being used regularly will likely show signs of wear that an unused net would not; and a sleeping space over which a net is hung nightly may show evidence of the hanging even when the net is not present. Nelson Coelho, conversation with GiveWell, April 15, 2016
- 34
In the data for AMF's post-distribution check ups, it looks to us like "Nets Received" = "Hung" + "Present Not Hung" + "Missing" +" Worn out/not usable", implying that "nets hung" and "nets that are worn out" are exclusive categories. See, for example, Concern Universal Ntcheu 2012 33-month post-distribution check-up data
- 35
"'very good condition' (fewer than 2 holes of up to 2cms in size)." Concern Universal Ntcheu 2012 33-month post-distribution check-up report, pg. 4
- 36
- "‘good condition’ (fewer than 10 small holes on them)" Concern Universal Ntcheu 2012 33-month post-distribution check-up report, pg. 3.
- Note that nets with fewer than 10 small holes used to be considered in "OK" condition.
"'OK' (fewer than 10 small holes on them)" Concern Universal Ntcheu 2012 24-month post-distribution check-up report, pg. 4.
- 37
- "‘viable condition’, with more than 10 holes or 1 hole larger than 10 cm," Concern Universal Ntcheu 2012 33-month post-distribution check-up report, pg. 3.
- Note that nets with more than 10 small holes used to considered in "poor" condition. "'Poor' condition with more than 10 holes or 1 hole larger than 10 cm." Concern Universal Ntcheu 2012 24-month post-distribution check-up report, pg. 4.
- 38
"Worn out means not usable. Typically, holes will be 10 cms of cms in size and numerous and the net considered effectively unrepairable and/or structural integrity will have gone. If hung the net would not provide protection against mosquitoes given the extent of holes or tears or other loss of structural integrity." Rob Mather, AMF Founder, email to GiveWell, June 30, 2014.
- 39
- @AMF how we work with distribution partners@, pg. 4.
- According to a representative of Concern Universal, when assessing whether a net is being used, enumerators make a judgment call. Nelson Coelho, conversation with GiveWell, April 15, 2016
- 40
Rob Mather, AMF Founder, email to GiveWell, June 30, 2014
- 41
For example, consider data from the Ntcheu 2012 33-month PDCU:
- [GiveWell]: "Second, I'm trying to better understand the outcomes measured for post-distribution check-ups. In particular, I'm curious what the "People - Covered", "People sleeping under nets", and "Nets Used - Correctly/Not Correctly" metrics mean (under the "Usage of nets" tab). In the Ntcheu 33-month PDCU results , the "Presence of AMF Nets" tab lets me see how many nets are "hung", "present but not hung, "missing", or "worn out": the numbers of "hung" and "present but not hung" nets are 10,667 and 630, respectively, for a total of 11,297 nets that aren't missing or worn out. However, under the "Usage of nets" tab, I see that 11,498 nets are used "Correctly." Since 11,498 > 11,297, I believe that some of the "worn out" nets must be included in the "used correctly" metric. If anyone sleeping under a net that is "used correctly" is considered to be "covered", then some of the people counted in the "covered" category are actually sleeping under worn out nets. Is my understanding correct?"
[AMF]: "This is because the latter figure includes non-AMF nets i.e. there are 462 non-AMF nets included in the ‘Usage of the nets’ tab as we wished to have an overall figure for sleeping space coverage (52%). We may add a note to this tab to indicate there are some non-AMF nets included in the data on this tab." Rob Mather, AMF Founder, email to GiveWell, October 13, 2015. Note that we did not ask AMF where the 462 figure comes from. - 10,664 AMF nets were found to be "very good", "good", or "viable" condition. 10,664 + 462 non-AMF nets = 11,126 nets, which is less than the 11,497 nets "used correctly", implying that some AMF nets that are "worn out" are included in the "used correctly" category. Concern Universal Ntcheu 2012 33-month post-distribution check-up data
- [GiveWell]: "Second, I'm trying to better understand the outcomes measured for post-distribution check-ups. In particular, I'm curious what the "People - Covered", "People sleeping under nets", and "Nets Used - Correctly/Not Correctly" metrics mean (under the "Usage of nets" tab). In the Ntcheu 33-month PDCU results , the "Presence of AMF Nets" tab lets me see how many nets are "hung", "present but not hung, "missing", or "worn out": the numbers of "hung" and "present but not hung" nets are 10,667 and 630, respectively, for a total of 11,297 nets that aren't missing or worn out. However, under the "Usage of nets" tab, I see that 11,498 nets are used "Correctly." Since 11,498 > 11,297, I believe that some of the "worn out" nets must be included in the "used correctly" metric. If anyone sleeping under a net that is "used correctly" is considered to be "covered", then some of the people counted in the "covered" category are actually sleeping under worn out nets. Is my understanding correct?"
- 42
For example, in the Concern Universal Ntcheu 2012 33-month post-distribution check-up data, we see that:
- 10,664 AMF nets are hung ("Presence of AMF Nets" tab)
- 10,664 AMF nets are also in "viable", "good", or "very good" condition ("Condition of AMF Nets tab"). It seems odd to us that these numbers match.
- 11,497 nets are "used correctly." ("Usage of the nets" tab) AMF told us 462 non-AMF nets were included in this tab, meaning that 11,035 AMF nets are included.
- [GiveWell]: "Second, I'm trying to better understand the outcomes measured for post-distribution check-ups. In particular, I'm curious what the "People - Covered", "People sleeping under nets", and "Nets Used - Correctly/Not Correctly" metrics mean (under the "Usage of nets" tab). In the Ntcheu 33-month PDCU results , the "Presence of AMF Nets" tab lets me see how many nets are "hung", "present but not hung, "missing", or "worn out": the numbers of "hung" and "present but not hung" nets are 10,667 and 630, respectively, for a total of 11,297 nets that aren't missing or worn out. However, under the "Usage of nets" tab, I see that 11,498 nets are used "Correctly." Since 11,498 > 11,297, I believe that some of the "worn out" nets must be included in the "used correctly" metric. If anyone sleeping under a net that is "used correctly" is considered to be "covered", then some of the people counted in the "covered" category are actually sleeping under worn out nets. Is my understanding correct?"
[AMF]: "This is because the latter figure includes non-AMF nets i.e. there are 462 non-AMF nets included in the ‘Usage of the nets’ tab as we wished to have an overall figure for sleeping space coverage (52%). We may add a note to this tab to indicate there are some non-AMF nets included in the data on this tab." Rob Mather, AMF Founder, email to GiveWell, October 13, 2015. Note that we did not ask AMF where the 462 figure comes from.
- [GiveWell]: "Second, I'm trying to better understand the outcomes measured for post-distribution check-ups. In particular, I'm curious what the "People - Covered", "People sleeping under nets", and "Nets Used - Correctly/Not Correctly" metrics mean (under the "Usage of nets" tab). In the Ntcheu 33-month PDCU results , the "Presence of AMF Nets" tab lets me see how many nets are "hung", "present but not hung, "missing", or "worn out": the numbers of "hung" and "present but not hung" nets are 10,667 and 630, respectively, for a total of 11,297 nets that aren't missing or worn out. However, under the "Usage of nets" tab, I see that 11,498 nets are used "Correctly." Since 11,498 > 11,297, I believe that some of the "worn out" nets must be included in the "used correctly" metric. If anyone sleeping under a net that is "used correctly" is considered to be "covered", then some of the people counted in the "covered" category are actually sleeping under worn out nets. Is my understanding correct?"
- 11,035 nets "used correctly" is more than the 10,664 nets "hung", implying that nets that were "present but not hung" or "worn out/not usable" could be included. ("Presence of AMF Nets" tab).
- 11,035 nets "used correctly" is also more than the 10,664 nets found in "viable", "good", or "very good" condition, implying that some "worn out" nets are "used correctly."
- 24,980 people were found to be "sleeping under nets" ("Usage of the nets" tab). These are the people who are also labelled as "Covered" by nets.
- 24,980 people / 11,035 nets "used correctly" = 2.26 people per net, which seems reasonable, though higher than we would have expected. 24,980 people / 10,664 nets "hung" = 2.34 people per net, which also seems reasonable but higher than we would have expected. We are uncertain about whether "sleeping under a net" means sleeping under a "used correctly" net, a "hung" net, or a non-"worn out" net.
Concern Universal Ntcheu 2012 33-month post-distribution check-up data
- 43
Full details in our AMF cost per net spreadsheet.
- 44
- The estimate includes actual costs for the Balaka 2013, Dedza 2014, Dowa 2015, Ntcheu 2015, and Balaka 2015 distributions. The only completed large-scale AMF distribution in Malawi not included is Dowa 2012. Concern Universal paid for the non-net costs of the Dowa 2012 distribution; we have not asked for CU's actual or budgeted costs.
- Full details in our AMF cost per net spreadsheet.
- 45
Full details in our AMF cost per net spreadsheet.
In estimating the overall cost per LLIN in Malawi, we have used an average that weights distribution costs by distribution size (i.e. costs associated with larger distributions are given proportionally more weight). Another option would be to put more weight on the actual costs to account for the fact that actual costs are often more reliable than budgeted costs. - 46
Rob Mather, AMF Founder, conversation with GiveWell, May 23, 2014. Managerial staff time is listed in the budget.
- 47
See the “Ntcheu 2012 other costs” sheet in in our AMF cost per net spreadsheet.
- 48
Full details in our AMF cost per net spreadsheet.
- 49
- See our AMF cost per net spreadsheet, "AMF org" sheet for details.
- AMF's fiscal year runs from Jul 1 - Jun 30 (e.g. FY 2012 is Jul 1 2011 - Jun 30 2012).
- 50
For example:
- The spreadsheet IMA shared of actual costs for the Kasaï-Occidental distribution is hard to make sense of and appears to contain several errors. Also, about 20% of the costs are "unconfirmed charges," which are not listed individually.
- For Malawi, Concern Universal shared, in 2012, information on non-monetary costs (such as office space) and the value of time provided by government employees. IMA's actual cost data includes an estimate of indirect costs (as a percentage of total direct costs), but it's unclear if it includes costs paid for by the government or other actors. As a rough approximation, we have added $0.13 per net, which was the estimate of non-monetary partner and local government costs from the 2012 Ntcheu, Malawi distribution.
- 51
Full details in our AMF cost per net spreadsheet.
- 52
Full details in our AMF cost per net spreadsheet.
- 53
- AMF told us that in DRC net costs and non-net costs are very roughly the same. Rob Mather, AMF Founder, conversation with GiveWell, May 23, 2014
- "In other cases, non-net costs are assumed to $2.50 per LLIN, which is the historical average. Non-net costs can vary significantly across countries. For example, in Angola and DRC, non–net costs may be more than $5." Melanie Renshaw, African Leaders Malaria Alliance Chief Technical Advisor, phone conversation with GiveWell, May 23, 2014, pg. 3.
- 54
Full details in our AMF cost per net spreadsheet.
- 55
- Major categories of activities included in Episcopal Relief & Development's budget: attending planning meetings and workshops, monitoring the household registration process, pre-distribution and post-distribution data validation, distribution exercises, post-distribution check-up surveys, data entry, and community sensitization activities. A budget for equipment, office costs, and staff time is also included. Episcopal Relief & Development Ghana non-net costs budget 2016
- See our AMF cost per net spreadsheet for our calculations.
- 56
- We have not yet seen budgets or actual costs from Ghana Health Services from these distributions.
- In June 2016, AMF estimated that standard non-net costs (other than PDCUs) would cost around $1.50 or $2.00 per net for AMF's future distributions (listed as "Addntl non-net costs (per net)" in source). AMF funds status (June 2016)
- We have chosen to use $1.75 per LLIN as an estimate of the costs incurred by Ghana Health Services for its pre-distribution planning activities and distribution implementation activities, though we are highly uncertain of this estimate.
- 57
Full details in our AMF cost per net spreadsheet.
- 58
- 59
Full details in our AMF cost per net spreadsheet
- 60
- "AMF Medium Term Strategy… Increase AMF central staffing by four people before the end of 2016: two program directors, one operations assistant and a technology assistant. Keep under review to assess hiring ahead of need. Hire one to two permanent staff in each country in which large scale distributions are funded." AMF medium term strategy discussion document (May 2016).
- AMF provided us with its estimates for increases in salary costs due to additional hiring in FY 2017 and FY 2018. Rob Mather, AMF Founder, email to GiveWell, November 14, 2016
- 61
For our estimate of AMF's organizational costs for future distributions, we use a weighted average of our annual estimates of AMF's organizational costs per net beginning in AMF's FY 2015, through our projected organizational cost per net for FY 2018. We exclude earlier years, when AMF was operating at a smaller scale and its organizational cost per net was higher than we expect it to be in future years. See our cost per net spreadsheet, "AMF org" sheet for calculations.
- 62
See our most recent cost-effectiveness analysis, "Bednets" sheet, cell J71.
- 63
See for example Roll Back Malaria Partnership gap analysis (September 2013).
- 64
- 65
"The gap analysis includes both the cost to purchase LLINs and non-net costs, such as shipping and distribution. In cases where national governments provided data on non-net costs, this data was used in the analysis. In other cases, non-net costs are assumed to $2.50 per LLIN, which is the historical average. Non-net costs can vary significantly across countries. For example, in Angola and DRC, non–net costs may be more than $5." Melanie Renshaw, African Leaders Malaria Alliance Chief Technical Advisor, phone conversation with GiveWell, May 23, 2014, pgs. 2-3.