This page contains further discussion and information for our review of Against Malaria Foundation. This page is intended to provide supplementary information on topics covered in the main review and is not intended to be read independently of the main review. The information on this page is less frequently updated than our main review; there is a note at the top of each section indicating when it was last updated.
Table of Contents
Registration process
Section last updated: November 2018, except where otherwise noted
Malawi
In November 2018, AMF sent us a document explaining its current process in Malawi, which is available as AMF Malawi Registration Process 2018. The rest of this section details our understanding of registrations AMF has conducted in Malawi in recent years.
United Purpose (formerly Concern Universal), AMF's partner in Malawi, collects data from each household on the number of sleeping spaces. In the past, it also collected data on the number of long-lasting insecticide-treated nets (LLINs) that have at least one year of remaining use, but going forward it will not be taking existing LLINs into account.1 Our understanding of the pre-distribution registration survey (PDRS) process is as follows:2
- United Purpose collects demographic data from the district health office, including the number of households in the region and the number of people per household.
- United Purpose briefs the health facility supervisor (HSA), who will do the door-to-door surveying, about the PDRS.
- There are no detailed maps or geographical information system data for rural Malawi, so United Purpose and HSAs rely on traditional authorities and community leaders to locate all the households in the targeted villages.
- HSAs go door-to-door to register all households in the targeted villages. A village leader or other volunteer accompanies the HSA during this process. During household registration:
- The volunteer and HSA introduce themselves to the head of the household and explain why they are there.
- The HSA obtains permission to enter the household from the household head. (The difficulty of obtaining this permission varies based on the local culture. Local leaders can lend credibility to the HSA when asking for permission to enter a household.)
- Once in the household the HSA collects information including the name of the head of the household and the number of sleeping spaces. (The HSA asks household members for demographic information, and assesses the number of sleeping spaces via visual inspection.
- United Purpose conducts "105%" data verification during the PDRS. HSAs are responsible for registering 100% of households in a village. In addition, supervisors independently register 5% of households in each village during the PDRS. This data is later compared to the HSA-collected data to check for consistency. Note that in September 2016, AMF told us that this piece of the process has not been emphasized and may not be happening consistently. The only data we have seen from the 5% checks from any AMF distributions is partial data from from the Balaka 2015 distribution.3
- United Purpose staff and government health officials supervise the registration. We are not sure what this involves in practice.
We have seen pre-distribution survey results from all of AMF's completed large-scale distributions in Malawi: full data from its Ntcheu 2012, Balaka 2015, and Ntcheu 2015 distributions; and sample results from its Balaka 2013, Dedza 2014, Dowa 2015 distributions.4
DRC
IMA World Health, AMF's partner in the Democratic Republic of the Congo (DRC), used different registration processes for the two distributions it has completed (in Kasaï-Occidental and Nord Ubangi).
For the Kasaï-Occidental distribution, IMA World Health concurrently collected registration data on smartphones, distributed LLINs, and hung LLINs in recipients' homes. IMA World Health collected registration data on the number of sleeping spaces and previously-owned nets in usable condition in each household, and allocated enough LLINs to cover all sleeping spaces, minus the number of existing, usable nets. GPS coordinates identifying the location of the household and data on the number of AMF's LLINs hung up during the distribution process were also collected. We have seen full registration data from Kasaï-Occidental.5
In Nord Ubangi, DRC, registration data was also collected on smartphones, but it was collected prior to the distribution rather than during it. IMA World Health collected registration data on the number of sleeping spaces and previously-owned nets in usable condition in each household, and allocated enough LLINs to cover all sleeping spaces, minus the number of existing, usable nets. GPS coordinates identifying the location of the household and data on the number of AMF's LLINs hung up were also collected. We have seen sample registration data from Nord Ubangi.6
Ghana
Sub-section last updated: November 2017
In Ghana, volunteers organized by one of AMF's distribution partners, Ghana Health Services (the National Malaria Control Program is a part of Ghana Health Services), collected information on the number of people in each household prior to the distribution. Households were allocated one net per two people (rounding up for households with an odd number of people). Previously-owned nets in usable condition were not taken into account when allocating LLINs.7
AMF has shared all registration data collected from each of the three regions it has funded distributions in with us.8
Uganda
AMF's distribution partners in Uganda collected information on the number of people in each household prior to the distribution. Data on the number of existing usable nets in each household were not recorded in AMF's Data Entry System, so we assume these data were not collected.9
We have seen data down to the household level on the distribution in Uganda. AMF has also shared distribution reports from three of the four waves of the distribution that have occurred in Uganda.10 Based on our review of these distribution reports, the registration process used in Uganda appears to be broadly similar to that used in Malawi and Ghana. One report indicates that registration did not occur in at least some areas; we are uncertain how widespread this issue was.11
AMF told us that verification data (i.e. data from different surveyors re-surveying of 5% of households to check the quality of the initial round of surveys) were collected;12 we have not yet seen them. One of the distribution reports we have seen suggests that verification may have largely taken place in convenient-to-reach villages.13
Papua New Guinea
We have seen several distribution reports from Papua New Guinea. Based on our review of these distribution reports, the registration process used in Papua New Guinea appears to be broadly similar to that used in Malawi and Ghana.
Rotary Club of Port Moresby (RCPM)/Rotarians Against Malaria, AMF's distribution partner in Papua New Guinea, collected data on the number of people in each household prior to the distribution. AMF told us that data were not collected on the number of existing LLINs in each household;14 we have seen these data for individual households.15
According to the reports we received, as a means of ensuring all households had been registered, a summary of every village surveyed is posted for 24 hours in a public place so that people can ensure that all their family members have been included in the survey. Prior to AMF's involvement, registered household information was read aloud at a village meeting. Additionally, AMF has told us that different surveyors re-survey a randomly-selected 6% of households in every village to verify the collected information;16 we have not seen data from this process.
Togo
The registration process used in Togo appears to be broadly similar to that used in Malawi and Ghana. More information can be found in the distribution report we have received from Togo; see footnote.17 Upon registering, households were issued coupons redeemable for LLINs.18 Supervisors revisited a subset of households to verify the data collected.19
After registration, a "rapid evaluation" was conducted, revisiting 800 households and asking whether they were visited by a surveyor and whether they received a coupon for LLINs. We do not have much information about the robustness of this process, which found generally very positive results.20
Zambia
The registration process used in Zambia appears to be broadly similar to that used in Malawi and Ghana, based on the information in the distribution reports we have received from Zambia (see footnote).21
Distribution process
Section last updated: November 2016, except where otherwise noted
Malawi
In Malawi, Concern Universal uses the following process for its distributions:22
After information is collected and verified, Concern Universal creates a distribution planning document which:
- Assigns each village to a distribution site, with a goal of minimizing the distance that beneficiaries need to travel (most beneficiaries travel 4-5 miles to their distribution site). Each catchment area is divided into several distribution sites, and each site serves 2-12 villages, depending on the geographic location, number of nets to be distributed, etc.
- Allocates a number of nets to be delivered to each location.
Concern Universal staff then meet with traditional authorities and district health officials to explain the distribution plan and set a date for the distribution, avoiding days that conflict with other community events. Concern Universal communicates the date, time, and place of distributions verbally to village leaders, who relay the information verbally to village members.
On distribution day, Concern Universal staff bring bed nets to the distribution sites using Concern Universal's vehicles or vehicles rented for that purpose. Village members typically walk to the distribution site; some travel by bike or organize transit in a motor vehicle. At the site, village leaders and HSAs group the beneficiaries by village. A roll call is taken, and as household names are called, members approach to receive nets. Household heads sign or fingerprint two copies of the distribution register to confirm receipt of their nets. If the household head is not present, a representative may receive nets on his/her behalf, as long as that person is recognized by the village head as being part of the household. When distribution is complete, each village leader receives a copy of the distribution register for their village, and Concern Universal retains the other copy.
An example of managing in-country shipping and storage logistics from the 2014 Dedza distribution (our understanding is that Concern Universal has used similar processes for its other distributions in Malawi):23
Upon arrival in-country and after completing all custom clearance formalities, the nets were stored in the main warehouse located within the premises of the District Hospital in Dedza Town pending relocation to the respective Health Centres[...] In order to ensure the security of nets, new padlocks were installed on all warehouse doors and the keys kept by CU’s storekeeper. Comprehensive insurance was contracted to cover potential losses caused by theft, fire or water damage and security guards were deployed throughout the distribution period, until all nets were distributed. All movements of nets were supervised by at least two CU staff members and the security guard[...] One or two days prior to distribution, the required quantity of nets was ferried to the health center for temporary safe keeping pending relocation to the distribution points on the day of distribution. Two vehicles were used to carry the distribution personnel and nets.
DRC
In Kasaï-Occidental, DRC, IMA World Health implemented registration and hang-up distribution at the same time. The following bullet points are a summary of the distribution process (more details in full distribution report).24
- "Nets arrived in Kinshasa on 22 Jul 2014 in 21 x 40HC shipping containers each containing approximately 32,000 nets." Pg 5.
- "The nets travelled 800 kms by road over the next three days and arrived in Tshikapa District with individual trucks driving directly to eight individual locations in each of the Health Zones where the relevant quantity of nets needed for the distribution in that district were pre‐positioned." Pg 6.
- "Meetings were held with local leaders to seek permission and support for the distribution." Pg 12.
- "Two categories of staff were required for the distribution. Field Supervisors (FSs), who had a management responsibility in each HA, and Community Health Workers (CHWs) who were responsible for visiting beneficiary households to collection registration information, hand out and hang the nets." Pg 12.
- "The CHW responsible for the hanging of the nets does so with the help of the head of household and/or other household members, using the string and nails brought along if necessary. A net is hung over each sleeping space (two people per sleeping space) and then rolled up out of the way (for example off the ground) during the day to help avoid unnecessary wear and tear or damage. It is then rolled down at night." Pg 17.
In Nord Ubangi, DRC, IMA World Health's distribution process was more similar to distributions in Malawi, with the separation of registration and distribution processes and the use of point distribution.25
Differences [between the Nord Ubangi distribution and the Kasaï-Occidental 2014 distribution]
- The North Ubangi distribution will be a two-phase distribution – a distinct pre-distribution phase and a subsequent distribution phase. In West Kasaï establishing net need per household and distributing of nets was contemporaneous.
- Distribution of nets will be done from distribution points rather than at the household
- The ODK registration form has been re-designed to better capture the required data, to eliminate some types of error and to reduce the chance of others.
- Operational lessons learned from the West Kasaï distribution will be implemented.
- An experienced, full time technology manager has been employed by IMA meaning there will be more extensive and easier liaising with AMF on data elements of the distribution. We have met the new technology manger and have spent a week working with him and are impressed.
Similarities [between the Nord Ubangi distribution and the Kasaï-Occidental 2014 distribution] [...] The same smartphone-based data collection technology will be used as in West Kasaï.
[...]
IMA has agreed there are benefits to conducting a separate, earlier pre-distribution phase during which accurate household-level net need is established. IMA are less able to estimate accurately population and net need in North Ubangi compared to what they felt was possible in West Kasaï. IMA believes point distributions rather than at-household distribution will be more cost-effective.Ghana
We have learned about the distribution processes for AMF's three distributions in 2016 in Ghana through conversations with AMF's distribution partners in Ghana (including the Greater Accra Regional Health Directorate of the Ghana Health Service, which managed in-country shipping and storage logistics for LLINs). From GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016:
Steps in the distribution process
- Beneficiaries walk a short distance to their distribution point. One Community Health Officer (CHO) employed by GHS and one GHS volunteer are stationed at each distribution point; there are never just two GHS volunteers.
- Before giving out each net, the distributors rip the packaging to help prevent reselling.
- Distributors gather together groups of recipients in order to transmit educational messages.
- The nets are distributed. Distributors make sure the recipients' coupons (brought by recipients) match the registration copies (brought by distributors) and make a mark on each side. The recipient's coupon is returned to them. Distributors also have a tally sheet with rows of 5 bubbles, and mark off a bubble for every net they give out. Distributors do not have an opportunity to sit or take a lunch break during distribution days.
[...] When deciding where to locate distribution points, the Greater Accra GHS considered safety, costs (for example, the cost of renting a warehouse) and proximity to other distribution points. They also selected 12-13 pre-positioning points in each distribution area.
Papua New Guinea
Sub-section last updated: November 2017
In the 2017 distribution in Papua New Guinea, Rotarians Against Malaria distributed nets using the following process (more details in footnote):26
- Nets were allocated to each family based on household survey data:
Following the survey, all data was entered into a computer where possible or entered into a master sheet of the area concerned. For each village, nets were allocated to each family on a needs basis and this is also entered into the computer or master sheet. This process ensured that all villages received nets and that there would be no shortfalls.
- Communities were informed that the distribution teams would be arriving:
There was no official launch for the LLINs distribution. However, prior to distribution there was a social awareness phase where the project teams informed communities about the arrival of the distribution teams, especially those accessible by road.
- Nets were distributed from a central location in a village or ward:
The LLINs were then distributed to teams depending on the needs of each village. The LLINs were then distributed by village or wards depending on whether villages were close together. In this latter case, a central location was identified in the ward and distribution of nets was carried out. In terms of distribution sites, villages were organised in different ways to make the flow of nets easy and effective. However, the actual organisation depended on each team and the structure of the various groups receiving nets.
- Rotarians Against Malaria's description of the day of the distribution:
- "Wait for everyone to come to the distribution site, collect all the white copies [of the vouchers given during the household survey] and sort the copies with the pink copies. Call the names on the white copies and give the nets. [...]
- Everyone who presented with a white copy was given LLINs at the distribution site with LLINs only given to the owner of the house or a member of the house whose name is written on the back of the survey form.
- Those people who lost the white receipts were left to the end of each distribution, with nets only being issued if the name of the person collecting the nets was on the back of the pink copies.
- For those families who were absent at the time of distribution, their names were written in a form – MOSQUITO NETS HELD IN TRUST BY VILLAGE CHIEFS FORM and the nets allocated to missing families were given to chiefs or any person entrusted by the community with the forms for these families to collect their LLINs later.
- Householders are now being asked to make a thumb print as proof of delivery of nets to a household. The Global Fund had formally asked that householders should sign for delivery of nets. This was considered problematic as many householders are illiterate and even where householders can write, this process takes a long time in what is often a very chaotic and charged environment where speed is of the essence."
Surveys and distributions took place at the same time in some villages that are particularly difficult to reach:27
It should also be noted, that due to the very remote locations found in some LLGs of the province, surveys and distributions took place at more or less the same time in some places. This is particularly true for mountainous villages or small islands which might take several hours or days to reach. In these instances, estimations based on the 2000 Census data and local knowledge to pre-allocate nets to these areas. Teams were then dispatched to these areas with nets. On arrival in these remote places, surveys took place and were immediately followed by a distribution based on the survey results.
Monitoring overview
Section last updated: November 2016
AMF's distribution partners also implement a set of monitoring activities to produce evidence on whether the registration and distribution processes operated as intended and on the long-term impact of the LLIN distribution. Monitoring activities have varied somewhat for different distributions.
- Process monitoring (i.e., the activities used to assess whether the registration and distribution processes operated as intended):
- Data validation: In Malawi, AMF's distribution partner Concern Universal provides registration lists to community leaders who read the lists out loud at a community meeting, supervised by Concern Universal, so community members can request edits.28 AMF told us that in Kasaï-Occidental, DRC, its partner IMA World Health employed supervisors who checked data for obvious errors and compared electronic and paper records. AMF has also checked the registration and distribution data for missing or inconsistent household-level records.29 We have not yet seen details of what data validation processes were used in Nord Ubangi, DRC. In Ghana, several different processes were used to validate data collected during registration and distribution: "pre-distribution data validation" (comparing the number of LLINs allocated to each community based on registration data to what would be expected based on the most recent census data, and double-checking and correcting—by phone or a follow-up household visit—the source of implausible registration data, e.g., households with more than 20 members), "post-distribution data validation" (checking whether data collected during the distribution indicates that households received appropriate numbers of LLINs), and, to check the accuracy of the electronic input of registration and distribution data, "6% data entry validation" (independent re-entering of 6% of the data collected during registration and distribution and checking it against the original electronic records). Details in footnote.30
- "Embedded" monitoring: In the 2016 LLIN distributions in Ghana, AMF's non-profit distribution partners, Episcopal Relief & Development and the Anglican Diocesan Development and Relief Organization (ADDRO), monitored the implementation of the registration and distribution processes by AMF's government distribution partner, Ghana Health Services. Episcopal Relief & Development and ADDRO were "embedded" in the registration and distribution processes; they attended district-level planning meetings to ensure that they were operating as intended, observed the registration of households by volunteers organized through Ghana Health Services, and observed selected distribution points (details in footnote).31 There was no "embedded" monitoring in AMF's distributions in Malawi and DRC since AMF worked with a single non-profit partner for those distributions.
- Distribution reports: Distribution reports provide narrative summaries of activities implemented and challenges encountered by distribution partners. Concern Universal has provided distribution reports for four of the six large-scale distributions it has completed in Malawi.32 IMA World Health has provided a distribution report for the Kasaï-Occidental, DRC 2014 distribution.33 We have not yet seen a distribution report from the Nord Ubangi, DRC distribution, which was completed in May 2016. From the three distributions in Ghana in 2016, we have seen a full distribution report from Episcopal Relief & Development from the Northern Region distribution, and short-term "Activity Reports" covering some periods of the pre-distribution and distribution activities in Greater Accra and the Upper West Region.34 It is our understanding that full distribution reports will be available for the Greater Accra and Upper West distributions later.
- Post-distribution validation tracing: After distributions were complete, AMF's distribution partners in Ghana checked that a random sample of households (100 households per district) had actually received the number of LLINs they were allocated by calling or visiting the households.35 Episcopal Relief & Development noted that, through this process, it discovered that a significant number of residents of two districts in Greater Accra, Ghana had not picked up the LLINs they had been allocated; we have not seen results beyond that.36 It is our understanding that similar processes have not been used in AMF's distributions in Malawi and DRC.
- Impact monitoring (i.e., the activities used to assess the long-term impact of the distribution):
- Post-distribution check ups (PDCUs): Distribution partners conduct follow-up surveys (called post-distribution check-ups, or PDCUs) in 5% of households at 6-month intervals for 2.5 years after a distribution. PDCUs estimate the presence, usage, and condition of LLINs from the AMF-funded distribution (details below).
We summarize which PDCUs have been completed and whether they were completed on time in this spreadsheet (see "PDCUs" sheet), and summarize the results of PDCUs we have seen in this spreadsheet.
Most scheduled PDCUs have been completed in Malawi. In 2016, AMF published data and reports from its first two PDCUs from Kasaï-Occidental, DRC, its first PDCUs completed outside of Malawi.37 We believe these PDCUs were poorly implemented and do not provide an accurate estimate of the proportion of AMF's LLINs that reached their intended destinations or the impact of AMF's distribution on LLIN usage over time in Kasaï-Occidental (see this blog post for details). PDCUs from Ghana are expected later.
- Malaria case rate data: Previously, AMF expected to collect data on malaria case rates from the regions in which it funded LLIN distributions: in the distribution agreements for several Malawi distributions, AMF asked Concern Universal to provide monthly malaria case rate data from before and after the distributions from all health centers in the three districts.38 In 2016, AMF shared malaria case rate data from Malawi covering January 2010 through March 2015 for Balaka, Ntcheu, and Dedza, and for Dowa from July 2012 to March 2015, but we have not prioritized analyzing it closely.39 AMF believes that this data is not high quality enough to reliably indicate actual trends in malaria case rates, so we do not believe that the fact that AMF collects malaria case rate data is a consideration in AMF’s favor, and do not plan to continue to track AMF's progress in collecting malaria case rate data.40
- Post-distribution check ups (PDCUs): Distribution partners conduct follow-up surveys (called post-distribution check-ups, or PDCUs) in 5% of households at 6-month intervals for 2.5 years after a distribution. PDCUs estimate the presence, usage, and condition of LLINs from the AMF-funded distribution (details below).
In previous versions of our review of AMF, we noted that AMF used "105%" registration data collection in some distributions. For distributions with "105%" registration data collection, after staff or volunteers have collected registration data from households, a different group of staff or volunteers revisit 5% of households and independently collect registration data.41 We do not believe this process, as it has been carried out, has been a useful form of monitoring. AMF and its partner in Malawi, Concern Universal, have not matched the two sets of registration data in time to correct potential errors in the distribution process (e.g., distributing the incorrect number of LLINs to a particular household). We have seen the 5% registration check data from one distribution in Malawi which AMF partially matched to the original data several months after the distribution was completed,42 and AMF has told us that 105% registration data was collected for other Malawi distributions but that it has not prioritized analyzing it.43 AMF has told us it sees the primary purpose of 105% data collection as being a tool to motivate data collectors, and that it does not need to check the data to achieve that goal.44
In previous versions of our review of AMF, we also discussed AMF's use of photos and videos as evidence that nets had reached their intended destinations. Because AMF has told us that it does not now consider photo and video footage part of its monitoring and evaluation activities, and because photos are missing for many recent distributions and there are fairly small numbers of photos available for other distributions, we no are longer tracking whether AMF is collecting photos and videos from distributions.
PDCU process
Section last updated: November 2016
"Post-distribution Check-Ups (PDCUs) are carried out every 6 months for 2.5 years (so 5 checks) with 5% of households in each Health Area selected at random and visited unannounced. These data are put in electronic form."45
Malawi
AMF and Concern Universal are in the process of revising the methodology used for PDCUs in Malawi. Concern Universal previously used the following procedure for its PDCUs:46
- From a list of all villages that received nets in a distribution, Concern Universal's project manager arbitrarily picks 5% of the villages to carry out PDCUs in. For each PDCU, the project manager tries to not select villages that are geographically close to each other or villages that have been visited in the previous PDCU. (Note that this selection procedure is not random and may be subject to bias.)
- For each selected village, the project manager arbitrarily selects 25 households to be surveyed during the PDCU. (Note that this selection procedure does not appear to be random and may be subject to bias.)
- On the dates of the PDCU, a supervisor or HSA from the local health facility accompanies the enumerator (a contracted employee of Concern Universal) to the village being surveyed. The HSA and enumerator meet with the village leader to discuss the purpose of their visit. The health facility staffer and enumerator arrive at the village unannounced to avoid knowledge of the PDCU influencing beneficiary behavior.
- The village leader (or a representative of the village leader) then accompanies the enumerator and health facility staffer to each household selected for the PDCU. The village leader knows the location of each selected household, and introduces the household head to the PDCU team.
- The household head gives consent for the PDCU team to enter the household and perform its check. If a household head refuses, the enumerator informs the project manager and the project manager selects another household. If the household head is away when the PDCU team arrives, the enumerator will return later in the day to perform the check. Household heads are frequently away when the PDCU team arrives, often because they are doing fieldwork far from their home.
- During the PDCU check, the enumerator:
- assesses the number of regularly used sleeping spaces.
- asks how many people are in the household, and how many have had blood-diagnosed malaria in the past month.
- checks the brand of the bed nets. Nets are recorded either as 'Permanet,' 'Olyset' (the two brands AMF distributes), or 'other.'
- records whether the net was distributed by an AMF-funded distribution. AMF-funded nets have a tag with the AMF logo and the date of the distribution.
- assesses the net condition. Nets are assessed as 'very good,' 'okay,' or 'poor.'
- A net in very good condition has two or fewer holes of less than 2 cm.
- A net in okay condition has fewer than 10 small holes (less than 2 cm).
- A net in poor condition has more than 10 small holes, or one large hole (greater than 10 cm).
Net quality is assessed by physical examination by the enumerator. If a net is not hung, the net is unfolded and physically assessed.
- asks who sleeps under each net, categorized by children under 5, children ages 6-18, pregnant women, or other people.
- asks how many nets were originally distributed to the household during the AMF distribution, and also notes how many of these nets are not present, or worn out.
- records how many nets in the household are hung, and how many are not hung.
- assesses whether nets are being used correctly. For this assessment, the enumerator makes a judgment call about whether the nets being are used properly. Often, nets will not be hung during the day due to limited space, but still may be used properly at night.
Ghana
Episcopal Relief & Development agreed to carry out post-distribution check-up surveys for the three distributions in Ghana in 2016.47
AMF has told us that, for the first PDCU for the Northern Region in Ghana in November 2016, there will be "5% of the 5%" data collection—a separate set of data collectors re-visiting a random selection of 5% of the households selected for the PDCU to re-collect data. AMF also told us that it expects to be able to quickly match the "5% of the 5%" data to the original PDCU data, and intends to use this check to assess the accuracy of PDCU data collection.48
PDCU definitions
Section last updated: November 2016
The following definitions are based on our understanding from multiple sources and may not have been used fully consistently for each post-distribution survey (PDCU). Definitions:
- Nets hung: AMF told us that interviewers are asked to observe whether nets are hung by entering interviewees' houses (rather than simply asking interviewees if they are hung). Concern Universal told us that PDCU surveyors use their judgment to assess whether nets are being hung properly when they are interviewing recipients and inspecting their nets and sleeping spaces.49 It is our understanding that none of the nets included in the "nets hung" category are "worn out" (i.e., all are in "viable", "good", or "very good" condition).50
- "Very good" condition: LLIN has fewer than 2 holes of less than 2 cm in size.51
- "Good" condition: LLIN has fewer than 10 small holes.52
- "Viable" condition: LLIN has more than 10 small holes or has one large hole.53
- "Worn out": Survey reports from Concern Universal did not provide a definition of this term. AMF has told us in the past that an LLIN is considered worn out if it has multiple large holes and the LLIN is unrepairable, such that it would not provide protection against mosquitoes.54
- Used correctly: According to a survey form from 2012, interviewers are asked to "ask the householder to demonstrate how the nets are used at night" and to then select yes or no to the question "are the nets being used correctly."55 In 2014, AMF told us that the procedure had not changed in more recent surveys.56 Note that some of the nets included in this category may not be AMF nets and that it is possible that some "used correctly" nets are also "worn out" nets.57 In this blog post, we discuss the possibility of bed nets being used for other purposes than covering sleeping spaces (e.g., being used as fishing nets). We have not seen evidence that AMF-distributed nets are being misused at a meaningful scale, and the PDCU results we have seen to date indicate that, at least in Malawi, a large majority of AMF-distributed nets are being used appropriately.
- People covered by nets: We are unsure of how to interpret this metric. We previously believed that this included all people who sleep under a properly hung and not worn out net. However, the data for some post-distribution check-ups is not consistent with this interpretation.58
What's included in our cost per LLIN estimates
The estimate for completed distributions in Malawi uses:59
- The purchase price per LLIN, calculated from AMF's total LLIN budget and the number of LLINs distributed.
- Actual costs for five completed large-scale distributions in Malawi as reported by Concern Universal60 and a combination of budgeted and actual costs for post-distribution check-up surveys following these distributions.61 AMF told us that the non-net budgets include all monetary costs incurred by Concern Universal, as well as the use of vehicles and managerial staff time;62 they may exclude non-monetary costs such as the use of office space.
- An estimate of non-monetary costs incurred by Concern Universal and local governments, based on estimates provided in 2012 by Concern Universal for its distribution in Ntcheu province.63
- A calculated estimate of the non-monetary costs incurred by Concern Universal and local governments due to post-distribution check-ups (calculated based on the Ntcheu 2012 estimate discussed above).64
- An estimate of AMF's organizational costs per net, including an estimate of the value of donated services and volunteer time. Because these costs generally do not vary with the number of LLINs distributed and remain roughly constant on a per year basis, the per LLIN estimate is highly dependent on the time period used and the number of LLINs distributed in that period. We estimate AMF's organizational costs as $0.29 per LLIN for July 2011 through the beginning of October 2016.65
For DRC, the estimate is rougher. We constructed the estimate similarly, although the information we have for DRC is likely of lower quality.66 The estimate for the DRC is based on:67
- The purchase price per LLIN, calculated from AMF's total LLIN budget and the number of LLINs distributed.
- Actual costs for the Kasaï-Occidental 2014 and Nord Ubangi 2015 distributions.
- Actual costs for the first three post-distribution check-ups following the Kasaï-Occidental 2014 distribution, an estimate of the costs of the remaining Kasaï-Occidental post-distribution check-ups based on the costs of the completed post-distribution check-ups, and budgeted costs for the post-distribution check-ups for the Nord Ubangi 2015 distribution.
- The same estimate of AMF organizational costs as we used for Malawi.
- Calculated estimates of non-monetary costs incurred by IMA World Health and local governments, based on the estimate from the 2012 distribution in Ntcheu, Malawi.68
The higher cost per net in the DRC may be explained by a considerably more difficult operating environment in DRC,69 or our data for DRC may simply be of lower quality.
The estimate for Ghana is also rough; it is based on:70
- The purchase price per LLIN, calculated from AMF's total LLIN budget and the number of LLINs distributed.
- Budgeted costs for pre-distribution, distribution, and post-distribution activities (primarily monitoring activities, including post-distribution check-ups) from AMF's non-profit distribution partner in Ghana, Episcopal Relief & Development.71
- A rough estimate of costs incurred by Ghana Health Services for its implementation of the distributions.72
- The same estimate of AMF organizational costs as we used for Malawi and DRC.
- Calculated estimates of non-monetary costs incurred by Episcopal Relief & Development, ADDRO, and local governments, based on the estimate from the 2012 distribution in Ntcheu, Malawi.73
Our estimates of the cost per net for planned distributions that AMF has signed or is considering are also rough; they are based on:
- The purchase price per LLIN, based on what AMF told us it expects the cost to be.
- AMF's estimate of the non-net costs associated with pre-distribution surveys, distribution days, and post-distribution check-ups for each upcoming distribution.74 These estimates are consistent with what we would expect based on past distributions in Malawi, DRC, and Ghana.75
- An estimate of AMF's organizational costs per net in the future, including increases in AMF's salary costs in FY 2017 and FY 2018 due to planned hiring of additional staff.76 We estimate AMF's organizational costs will be about $0.08 per net in the future, compared with $0.29 in our estimates for AMF's past distributions in Malawi, DRC, and Ghana.77
We are uncertain about the use of marginal funding to AMF this year. Our best guess at the cost per LLIN in distributions AMF may fund with additional funding this year is an average of the cost per LLIN for all of AMF's past large-scale distributions and its planned distributions discussed above (weighted by AMF's spending in each country). We calculate that the weighted average is $4.85 per LLIN.78
Global estimate of cost per LLIN
The Roll Back Malaria Partnership, which estimates funding gaps for LLINs across all African countries,79 uses an estimate of $2.70 to purchase an LLIN and $2.50 for all non-net costs of a distribution, for a total of $5.20 per LLIN.80 This assumption is used in cases where country-specific data is not available. Note that we have not seen details about where this estimate comes from, so we put little weight on it. The Co-Chair of the Roll Back Malaria Harmonization Working Group told us that non-net costs can vary significantly by country and that non-net costs in DRC may be more than $5 per LLIN.81 We have little information on how reliable the Roll Back Malaria Partnership's estimate is likely to be. Broadly, this estimate is consistent with the data we have seen from AMF. We have used estimates based on data from AMF in our cost per life saved calculations, because we have more information on their reliability. We do not believe our estimate of AMF's cost-per-net ought to be compared directly to the Roll Back Malaria Partnership's estimate, given how uncertain we are about the latter.
Specific country funding gaps
Section last updated: November 2016
We spoke to individuals who are involved in national planning for mass LLIN campaigns in two countries where AMF is interested in working in the future. They provided details on why other funding sources are unlikely to be sufficient to replace LLINs when the next campaign is due.
In one case, the next LLIN campaign will be due before it is likely feasible for Global Fund funding to be secured and used to order and ship LLINs. AMF may be able to move up the distribution by a year or more. There are also expected to be funding gaps for the next campaign, regardless of when it occurs, because, due to government regulations and systems, the country has been slow to spend Global Fund resources and, as a result, the Global Fund has reduced how much funding it allocates to the country. Note that the Global Fund has not yet announced its allocation to the country for the next round, so it is possible that the allocation will be higher than expected.
In another case, past campaigns have had major funding gaps, and recently two major funders of previous mass campaigns, the President's Malaria Initiative and the UK Department for International Development, have shifted their focus to other malaria interventions, such as indoor residual spraying, treatment for pregnant women, strengthening health systems, and private sector case management. These two funders are not expected to provide as much funding for mass LLIN campaigns in the future as they have in the past.
Counterfactual in countries where AMF funded distributions
Section last updated: November 2017
Ghana 2016
Ghana's National Malaria Control Program told us in August 2016 that, without AMF providing funding for the distributions in the Upper West, Greater Accra, and Northern Regions in 2016:82
- No LLINs may have been available to deliver in the Upper West Region in 2016. The Upper West Region had been deprioritized relative to other regions for mass LLIN distribution, since other malaria control programs (Seasonal Malaria Chemoprevention and Indoor Residual Spraying) were ongoing.
- Distributions in Greater Accra may have been targeted to slums and rural areas only. Note that we have seen some limited evidence that LLIN usage is generally higher in rural areas of Greater Accra than in semi-urban areas.83
- LLINs originally intended for continuous distributions (e.g., LLINs distributed through antenatal clinics) may have been re-directed to mass campaigns, which may have later led to shortages in continuous distribution channels.84
Overall, our understanding from our site visit and our discussions with Ghana's National Malaria Control Program is that the counterfactual impact of AMF funding LLINs in Ghana in 2016 is complex, but that Ghana's National Malaria Program may have been appropriately triaging its malaria control needs prior to AMF's involvement, making the remaining opportunities (e.g., distributing LLINs in regions with other ongoing malaria control strategies, distributing LLINs in semi-urban areas) somewhat less impactful in expectation.
Togo 2017
AMF and the Global Fund each funded roughly half of the costs of Togo's 2017 campaign.
A representative of the health ministry in Togo told us that the Global Fund is the only major funder of net campaigns in Togo; DFID works only in certain countries and Togo is not one of them, and the U.S. government does not provide malaria funding in Togo. The World Bank has provided some funding for routine net distribution and the Chinese government has provided some funding for treatment; neither have funded net campaigns. He told us that, without AMF's funding, the campaign would have been delayed while the government sought out other funders and that he was unsure who else might have been able to provide funding.85
AMF told us that its best guess as to what would have happened without AMF funding is that Togo would have had to use the funding from the Global Fund to prioritize certain populations, such as people in highly endemic areas and/or young children and pregnant women. The Global Fund and the Alliance for Malaria Prevention told AMF that they were not aware of other donors who might have been able to contribute a significant amount of funding to the campaign.86
All sources for AMF review
Section last updated: November 2017
- 1
"LLINs will be allocated to each household based on 1 net for every 2 persons, rounding up in the case of an uneven number of people in the household. The maximum number of LLINs that any household can receive is 4." AMF Distribution Plan, Malawi, 2018, Pg 10.
- 2
Nelson Coelho, conversation with GiveWell, April 15, 2016 and AMF Distribution Plan, Malawi, 2018.
- 3
- @Rob Mather, AMF Founder, conversation with GiveWell, September 28, 2016@
- Concern Universal Balaka 2015 registration data 100% vs 5%
- 4
- Concern Universal Ntcheu 2012 pre-distribution registration survey data
- Concern Universal Balaka 2013 pre-distribution registration survey data
- Concern Universal Dedza 2014 pre-distribution registration survey data
- Concern Universal Dowa 2015 pre-distribution registration survey data
- Concern Universal Balaka 2015 registration data 100% vs 5%
- Concern Universal Ntcheu 2015 pre-distribution registration survey data
- 5
- AMF: "DRC, West Kasaï Province: Distribution Report and separate Technology Report":
- "In West Kasaï the fight against malaria involves the distribution and hang‐up of LLINs, a so‐called ‘Hang‐Up’ strategy, in which volunteer Community Health Workers (CHWs) distribute the nets and hang them at the same time. This is in contrast to strategies that simply hand out the nets. A modified version of the Hang‐Up strategy, called a ‘Hang‐Up and Track’ (HUT) strategy, was used on this distribution. With the HUT approach, in addition to distribution and hang‐up, information on each net hung is captured using smartphones equipped with a data collection program." Distribution Report, Pg 4.
- "Teams of three or four CHWs [Community Health Workers] visited households sequentially.
This was a crucial stage of the net distribution that consisted of collecting relevant household data for every single household in the distribution catchment area including, but not limited to, the following ‘must have’ information:
- Name of household head
- Number of people in the household
- Number of sleeping spaces (‘for the purpose of nets’*)
- Number of perfectly good nets with at least two years of life left in them
- Location of household (GPS coordinates)
- Household identifier**
The last two pieces of information were to facilitate later identification of the individual household for post‐distribution check‐up work.
* When assessing the number of sleeping spaces it is important to avoid confusion between the number of separate areas in which individuals might sleep and, given nets distributed are ‘double’ nets that will accommodate two people, the number of sleeping spaces that is equal to the number of nets that are needed. For example, in a household with mum plus dad, four children aged eight, four, six and two and a grandfather there may be six separate sleeping areas: mum plus dad, one each for the four children and one for the grandfather. However, for the purposes of nets, there would be four sleeping spaces: one for mum and dad, one for the two older children, one for the two younger children and one for the grandfather.
**Creating a household identifier is necessary as many households do not have clear addresses. For example, it is frequently the case in rural and semi‐rural locations that street names and household numbers do not exist." Pg 16.
- IMA World Health, Kasaï-Occidental 2014 distribution data
- AMF: "DRC, West Kasaï Province: Distribution Report and separate Technology Report":
- 6
- "Differences [between the Nord Ubangi distribution and the Kasaï-Occidental 2014 distribution]
1. The North Ubangi distribution will be a two-phase distribution – a distinct pre-distribution phase and a subsequent distribution phase. In West Kasaï establishing net need per household and distributing of nets was contemporaneous.
2. Distribution of nets will be done from distribution points rather than at the household
3. The ODK registration form has been re-designed to better capture the required data, to eliminate some types of error and to reduce the chance of others.
4. Operational lessons learned from the West Kasaï distribution will be implemented.
5. An experienced, full time technology manager has been employed by IMA meaning there will be more extensive and easier liaising with AMF on data elements of the distribution. We have met the new technology manger and have spent a week working with him and are impressed.Similarities [between the Nord Ubangi distribution and the Kasaï-Occidental 2014 distribution]
1. The same smartphone-based data collection technology will be used as in West Kasaï.Comment
IMA has agreed there are benefits to conducting a separate, earlier pre-distribution phase during which accurate household-level net need is established. IMA are less able to estimate accurately population and net need in North Ubangi compared to what they felt was possible in West Kasaï.IMA believes point distributions rather than at-household distribution will be more cost-effective." Rob Mather, AMF Founder, email to GiveWell, September 9, 2015
- "IMA will ensure the collection and availability, for independent inspection, of household level data (sleeping spaces and number of perfectly usable nets in place) for all households in the distribution zones identified in section 4." IMA World Health and AMF distribution agreement for Nord Ubangi 2015
- IMA World Health Nord Ubangi 2015-16 registration data
- "Differences [between the Nord Ubangi distribution and the Kasaï-Occidental 2014 distribution]
- 7
- GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016:
- "GHS [Ghana Health Services] volunteers carry out the registration process, which takes place more than one month before the distribution. Volunteers are organized by the Ministry of Health (MoH) and participate in a number of government health programs, such as vaccination campaigns (immunization days). Many of them are long-time volunteers with considerable experience." Pg. 5.
- " Volunteers visit each house in their zone. The head of household, or someone else who is present, is asked how many people are in the household based on the 'people who eat from the same pot' definition. Ghana's policy is to provide one net for every two individuals in a household; as a result, in contrast to previous AMF distributions, volunteers were not required to check households' existing nets or number of sleeping spaces. The only exception to this policy is that household members aged 70 or older are not counted in the household total, and each receive their own net." Pg. 5.
- GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016:
- 8
- AMF Data Entry System, Ghana 2017
- The full registration data we have seen in AMF Data Entry System, Ghana 2017 include names of heads of households and either an address or a written description of the location of a household (e.g., "Next to the church").
- 9
AMF Data Entry System, Uganda 2017
- 10
- 11
Under "Challenges," the report says "Registration activations were not conducted hence most people were not registered." AMF Distribution Report, Eastern Uganda, Wave 2, 2017, Pg 32.
- 12
Comment provided in response to a draft of this review in November 2017
- 13
"Figure 1 shows that for the 22 wave two districts, Parish Chiefs’ validation was conducted in 89.3% of 261 sub counties, 72.7% of 1,461 parishes and 52.6% of 10,802 villages. Since the sample size of administrative units greatly increases from sub-county level to parish and village level, it’s probable that Parish Chiefs sampled villages within their reach hence not reaching far distant villages." AMF Distribution Report, Eastern Uganda, Wave 2, 2017, Pg 16.
- 14
Comment provided in response to a draft of this review in November 2017.
- 15
In AMF's Data Entry System it is possible to drill down to entries for individual households and see the data collected from that household.
- 16
"The major change to the survey methodology instigated by AMF is that previously a village meeting was called after a survey was carried out where names of family members was read out to ensure that all households had been surveyed. Instead of this process, a summary of every village surveyed is posted for 24 hours in a public place so that villages can ensure that all their family members have been included in the survey. As a further verification, RAM and local health staff team members carry out random house surveys of 6% of households in every village to ensure that the survey process has been carried out correctly," AMF Distribution Report, Morobe, Papua New Guinea, 2017, Pg 5.
- 17
AMF Distribution Report, Togo, 2017. Note that this distribution report is in French.
- 18
"Chaque ménage dénombré a reçu un coupon sur lequel a été inscrit le nombre de MILDA nécessaire à recevoir lors de la distribution." (English: "Each enumerated household received a coupon with the number of LLINs required to be received at the time of distribution.") AMF Distribution Report, Togo, 2017, Pg 20, quotation translated from French by Google Translate.
- 19
"Les superviseurs ont visité les ménages qui ont déjà reçu la visite des agents dénombreurs/distributeurs pour vérifier si ces ménages ont bénéficié des interventions selon les directives. Au cas où les prestations n’étaient pas conformes aux directives, les équipes étaient interpellées pour corriger les insuffisances. L’insuffisance majeure constatée a été l’inadéquation entre le nombre de couchage et le nombre de MILDA requises pour le ménage dans certaines localités." (English: "Supervisors visited households that have already been surveyed by enumerators / distributors to see if these households have benefited from the interventions as directed. In the event that the services did not comply with the directives, the teams were called upon to correct the deficiencies. The major deficiency noted was the mismatch between the number of beds and the number of LLINs required for the household in some localities.") AMF Distribution Report, Togo, 2017, Pg 18, quotation translated from French by Google Translate.
- 20
"Les évaluations rapides ont été conduites dans les trois jours qui ont suivi la fin du dénombrement et de la distribution… Au total 80 grappes ont été évaluées en raison de 10 ménages par grappe dans les régions Savanes, Kara, Centrale, Plateaux et Maritime (sans Golfe)." (English: "Rapid assessments were conducted within three days of the completion of enumeration and distribution ... A total of 80 clusters were evaluated due to 10 households per cluster in the Savannah, Kara, Central, Plateaux and Maritime regions (without Golfe).") AMF Distribution Report, Togo, 2017, Pg 26, quotation translated from French by Google Translate. See also results on Pg 27 for the items assessed by surveyors; the results show 98% of households reported being visited by a surveyor and 98% reported receiving a coupon for LLINs.
- 21
- 22
GiveWell's non-verbatim summary of a conversation with Nelson Coelho, April 15, 2016, pg. 4.
- 23
- 24
- 25
Rob Mather, AMF Founder, email to GiveWell, September 9, 2015
- 26
"Following the survey, all data was entered into a computer where possible or entered into a master sheet of the area concerned. For each village, nets were allocated to each family on a needs basis and this is also entered into the computer or master sheet. This process ensured that all villages received nets and that there would be no shortfalls.
"The needs basis in PNG is based on the assumption that children under the age of six sleep with their parents, after which boys and girls are grouped separately from their parents. Nets are also allocated based on the number of children in the house their ages, and whether there are other dependents living in the households such as grandparents. A full description of the allocation system can be obtained from RAM officers but suffice to say a family of four could receive between one to four nets depending on the sex and age of family members.
"Nets are allocated to each LLG based on assumed populations of each LLG and it was important prior to the distribution to ensure that when allocations had been carried out to families and villages that there were enough nets available to cover each and every family.
"Distribution
"There was no official launch for the LLINs distribution. However, prior to distribution there was a social awareness phase where the project teams informed communities about the arrival of the distribution teams, especially those accessible by road.
"The LLINs were then distributed to teams depending on the needs of each village. The LLINs were then distributed by village or wards depending on whether villages were close together. In this latter case, a central location was identified in the ward and distribution of nets was carried out. In terms of distribution sites, villages were organised in different ways to make the flow of nets easy and effective. However, the actual organisation depended on each team and the structure of the various groups receiving nets.
"All net packaging was opened at distribution with a district code printed on each net e.g. ENG17 for Enga 2017. This code is added to identify nets to assist in future monitoring exercises as well as discouraging householders from selling the nets on the open market.
"Some examples of the process used to control the crowd includes;
- Arranging the people according to the serial numbers in the survey books e.g. 72001 - 72050 and then calling the names and the serial numbers.
- Call the names of the owners of the households, collect the white copy and give the nets.
- Wait for everyone to come to the distribution site, collect all the white copies and sort the copies with the pink copies. Call the names on the white copies and give the nets. Some of the following control methods were used.
- Everyone who presented with a white copy was given LLINs at the distribution site with LLINs only given to the owner of the house or a member of the house whose name is written on the back of the survey form.
- Those people who lost the white receipts were left to the end of each distribution, with nets only being issued if the name of the person collecting the nets was on the back of the pink copies.
- For those families who were absent at the time of distribution, their names were written in a form – MOSQUITO NETS HELD IN TRUST BY VILLAGE CHIEFS FORM and the nets allocated to missing families were given to chiefs or any person entrusted by the community with the forms for these families to collect their LLINs later.
- Householders are now being asked to make a thumb print as proof of delivery of nets to a household. The Global Fund had formally asked that householders should sign for delivery of nets. This was considered problematic as many householders are illiterate and even where householders can write, this process takes a long time in what is often a very chaotic and charged environment where speed is of the essence.
"It should also be noted, that due to the very remote locations found in some LLGs of the province, surveys and distributions took place at more or less the same time in some places. This is particularly true for mountainous villages or small islands which might take several hours or days to reach. In these instances, estimations based on the 2000 Census data and local knowledge to pre-allocate nets to these areas. Teams were then dispatched to these areas with nets. On arrival in these remote places, surveys took place and were immediately followed by a distribution based on the survey results."
Papua New Guinea Distribution Report, Enga Province, August 2017, Pgs 8-9.
- 27
Papua New Guinea Distribution Report, Enga Province, August 2017, Pg 9.
- 28
- "A copy of the beneficiary list for their village or community will be passed to the relevant leaders to check for duplications, omissions, or inaccuracies. This will include reading out the list publically so the community is involved and engaged in the verification and the net distribution preparations more generally. CU field staff supervise this process." Concern Universal, Dowa 2015 planning document, pg. 7.
- Based on a conversation with AMF, it is our understanding that this same process is used for all of AMF's distributions in Malawi with Concern Universal. Rob Mather, AMF Founder, Conversation with GiveWell, September 28, 2016
- The results of the registration data verification process in the Dedza 2014 distribution:
- "During the verification process, CU staff roll-called the beneficiaries from the printed registers whilst the HSAs cross-checked with their list compiled from the registration forms in order to detect any mismatch. This verification exercise revealed that a total of 32,183 households had been skipped during the registration.
[…]
"After the village household verifications, the corrected and edited registers were referred back to the data entry team and the corrections made to the database." Concern Universal Dedza 2014 distribution report Pg 5.
- "During the verification process, CU staff roll-called the beneficiaries from the printed registers whilst the HSAs cross-checked with their list compiled from the registration forms in order to detect any mismatch. This verification exercise revealed that a total of 32,183 households had been skipped during the registration.
- Results of verification process for other large-scale AMF distributions in Malawi:
- Concern Universal Ntcheu 2012 distribution report, Pgs 5-7.
- Concern Universal Balaka 2013 distribution report, Pg 6.
- We have not yet seen the results of the registration data verification process for AMF's other large-scale distributions in Malawi (Dowa 2015, Balaka 2015, and Ntcheu 2015). We expect that the results of the registration data verification process will be reported in the full distribution reports for these distributions.
- 29
- IMA World Health, Kasaï-Occidental 2014 distribution report:
- "Household data went through the following steps:
- Data collected at the household by a CHW
- Independent supervisor uploaded the data from the smartphone to a laptop.
- The supervisor reviewed the data and corrected obvious mistake or inconsistencies."
Pg 18.
- "After the distribution had been completed in a Health Area and all data uploaded from the smartphones, the data team compared the electronic data with the paper data collected by each distribution site in each health area. At the end of the day each CHW returned undistributed nets to the supervisor and depot manager and reported the number of nets they distributed. These numbers were recorded on paper." Pg 18.
- "Household data went through the following steps:
- We have not seen the results of the supervisors' validation activities (e.g., the number of household records corrected for obvious errors and for discrepancies between electronic and paper records).
- We have seen a high level analysis of the quality of the registration and distribution data, which finds that some data is available for 91% of the LLINs distributed (data is missing for the other 9% of LLINs). For households for which data is available, 96% had internally consistent data and 4% had apparent errors or anomalies. AMF distribution verification Kasaï-Occidental 2014, pg. 3.
- We are not certain when AMF completed this analysis, or if this analysis was used in-country (e.g., to check whether households with inconsistent data had received the appropriate number of LLINs).
- IMA World Health, Kasaï-Occidental 2014 distribution report:
- 30
- "Step 7 - Pre-distribution validation
In all three regions with AMF-supported distributions, all coupons underwent a pre-distribution validation process by the NMCP and non-profit partners.
At the community level, total estimated nets (based on 2010 census data) are compared to the total number of nets allocated during registration. This gives a sense of the variance between these figures; communities with more than 2% variance might be spot-checked.
The validators also look for other potential problems with the coupons, including:- illegible entries
- households with over 20 members
- mismatches between the numeral and text versions (both are required) of figures in the 'nets allocated' field
- miscalculation of figure in 'nets allocated' field
If there are problems, the representative follows up with the volunteer who collected the data. The volunteer might return to the household to correct mistaken entries; if not, the representative will do so in person, or if that is too costly, by phone. If the information cannot be verified or corrected, the coupon is discarded." GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, pg. 6.
- "This activity [pre-distribution validation] has been completed in the Northern and Greater Accra regions. In both regions, ADDRO observed a few mismatches in terms of number of members registered as a HH and the number of LLINs allocated. Following discovery of this misinformation, validation teams returned to the communities to verify valid information." Episcopal Relief & Development Ghana Activity Report 1 2016, pg. 6.
- We have not seen comprehensive data on the results of the pre-distribution data validation process (e.g., exact numbers of household records corrected by this process).
- "Post-distribution validation
"After the grace period, coupon books are returned to the district level for the post-distribution validation process. This is carried out by the NMCP and non-profit partners, and involves analyzing the coupon books in order to:- Check for coupons that were not crossed out; for example, this scenario might arise if a household did not pick up their nets.
- Check that the number of nets allocated matches the number of nets given.
- Check that the number of nets allocated is accurate based on household size.
- Add total values for household size, nets allocated, and nets given, and record them on the inside cover of each coupon book."
- We have not seen comprehensive data on the results of the post-distribution data validation process (e.g., exact numbers of household records corrected or households re-visited during this process).
- "As it might be confusing to use a 5% figure in too many contexts, AMF and Episcopal Relief & Development decided to run verifications on 6% of the data entered by clerks. This data is reentered by another individual and checked against the original data. If there are discrepancies, the coupon book is verified." GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, pg. 13.
- We have seen results from the 6% data entry checks for the Greater Accra distribution, but (as of November 2016), not from the Northern Region or Upper West Region distributions. As of November 6, 2016 33,269 "6% check" records had been entered (5.4% of the 566,947 total records). 2,586 of these 6% check records had not yet been matched to the original electronic records. Of the 30,683 matching pairs of records, 10,632 (35%) were an exact match without correction from a supervisor, an additional 19,598 (64%) had "key data" (number of people in a household, LLINs allocated, and LLINs give) match without correction from a supervisor, and 247 (0.8%) had mismatches of key data. AMF Data Entry System, Ghana 2016
- "Step 7 - Pre-distribution validation
- 31
- "Instead, Episcopal Relief & Development and ADDRO carried out 'light monitoring,' and considered themselves 'embedded' in the pre-distribution and distribution work. Monitoring activities included:
- Attending meetings, including planning meetings at the national level, as well as a sample of district meetings and training sessions. This was done to observe whether activities were proceeding according to the NMCP's checklist and, if there were issues, to follow up with the NMCP.
- ADDRO monitors observed the registration process in a randomly selected sample of 50% of the 20 districts in the Northern Region and all districts in the other two regions. Names of all of the sub-districts in these districts were written on slips of paper and randomly selected. Episcopal Relief & Development reported that it did not skip any of the randomly selected districts, even those that were far away. The order of sub-district visits was determined by what made sense geographically.
- Monitors called the supervisors before arrival and visited some households to see whether they were aware that registration was happening in their area that day; almost everyone was aware.
- Monitors also observed the work of multiple community-based agents (CBAs); sometimes, they would do a cross-check by performing this task without the supervisor. They listened to the CBA's conversations with households, verified that the correct information had been recorded in the book, and addressed any issues with the CBA. They then went back through the route traveled by the CBA to ensure that households that should have been registered were registered, and that the coupons had been correctly filled out. This process is facilitated by the practice of marking registered houses. Monitors also asked households if they were aware of the date and location of the distribution.
- Monitors met with supervisors, analyzed completed coupon books to identify any problems, and filled out forms summarizing their daily observations.
- Monitors might address issues by:
- Debriefing with supervisors at the end of the day, so that learnings could be applied to the next day's work;
- Communicating directly with CBAs if they encounter the CBA before meeting with the supervisor;
- Observing CBAs without informing the supervisor, if they observed a number of issues the previous day." Pg. 7.
- "Monitoring of the distribution process by ADDRO
ADDRO has two or more monitoring teams per region; each of these has a vehicle. The NMCP has 2-3 evaluators per district; each has a vehicle. ADDRO representatives attend the NMCP planning meeting. ADDRO monitors introduce themselves to the municipal health directorate, and are taken by a representative to each distribution point in a sub-district to ensure that their presence will be accepted by those whose work they will be monitoring.In the Greater Accra Region, the malaria focal person for the district took monitors to a distribution point. From there, they went to other distribution points in the sub-district. They asked the focal person which areas were the least accessible and attempted to focus on those; in total, they visited 60% of the distribution points. They had the distributors' phone numbers so they could ask for directions if necessary.
Monitors verified that:
- nets were being distributed
- distributors were using the tally sheet
- nets were kept in the shade
- coupons were marked when nets were distributed
- a Ghana Health Services staff member was present
- distributors were giving helpful messages to the recipients
NMCP monitors were also present during the distribution." Pgs. 10-11.
- We do not have a clear understanding of exactly how ADDRO monitors choose which distribution points to observe, but note that the selection process described above does not appear to be random.
- Examples of challenges encountered and actions taken during distribution by ADDRO monitors in Zabzugu district, Northern Region:
- "A lot of household members did not understand the distribution method of giving one LLIN to two persons per household according to the universal coverage strategy. Although this was explained to them and most of them understood, there were complaints by a few.
- On the first day of the distribution, the DP attendants could not locate some few booklets. Apparently, these had been mistakenly taken to the district, making it difficult for some households to redeem their nets. Exercise books were procured to capture the information of those households as they presented their coupons to enable them redeem their nets. Arrangements were made to get the booklets from the region to the distribution point.
- Some households who were not registered turned up at the distribution sites. For example, all the households in the Zabzugu police barracks were not registered due to misinformation from their command about the registration procedure (i.e., that they don’t need to register as the nets will be given to them without prior registration). All these households did not benefit from the nets because of the no coupon, no net policy.
- At a distribution point in Zabzugu sub-district, DP attendants detected that some households altered the household numbers on their coupons, ostensibly to receive more nets. The DP attendants noted the right numbers on the coupon counterfoil (validated data) and gave out the correct number of nets to the households.
- Some beneficiaries waited all day to receive their nets because the booklets from which they were issued coupons could not be found on the day of the distribution. They booklets were mistakenly sent to different distribution point but were eventually brought to the DP and the people received their nets."
Episcopal Relief & Development Ghana Northern Region distribution report 2016, Pg 6.
- Challenges encountered and actions taken by ADDRO monitors for other districts in the Northern Region reported in Episcopal Relief & Development Ghana Northern Region distribution report 2016, Pgs 6-10.
- Challenges encountered and actions taken during distribution point monitoring for the distribution in Greater Accra reported in Episcopal Relief & Development Ghana Activity Report 2 2016, Pg 6.
- The distribution reports we have seen so far from Episcopal Relief & Development do not cover challenges encountered and actions taken during distribution point monitoring for the Upper West Region distribution.
-
GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016:
- "Instead, Episcopal Relief & Development and ADDRO carried out 'light monitoring,' and considered themselves 'embedded' in the pre-distribution and distribution work. Monitoring activities included:
- 32
- AMF page on Ntcheu 2012 distribution
- AMF page on Balaka 2013 distribution
- AMF page on Dedza 2014 distribution
- AMF page on Dowa 2015 distribution
We have not yet seen distribution reports for the Ntcheu 2015 or Balaka 2015 distributions in Malawi.
- 33
- 34
- 35
- Episcopal Relief & Development Ghana Activity Report 3 2016:
- "The validation was to establish precisely how many LLINs were distributed during the distribution campaign in all 12 districts. Apart from validating counterfoils, the validators undertook 'End-User Verification'. The end-user verification (EUV) is a rapid check-up to determine whether the beneficiaries really received the number of LLINs allocated for the households and are using the LLINs for the intended purpose. This involved randomly selecting 100 households in each district to verify LLINs received, LLIN use by household members, etc. Validators randomly sampled 100 booklets and from each booklet, randomly sampled one coupon counterfoil for the EUV visit. Validators then called the beneficiaries of the sampled coupon counterfoils and followed up to the households for the end-user verification exercise." Pgs 4-5.
- "24 NMCP validators were assigned to work in the 12 AMF supported districts (2 validators per district). Two teams from ADDRO (a team from the ADDRO HQ and a team from the ADDRO Greater Accra office) visited all the 12 AMF districts to monitor the validation process and to provide support for the packaging of coupon counterfoils for transportation to ADDRO headquarters in Bolgatanga. Key findings were as follows:
- It was comparatively easier for the validators/monitoring team to enter into bedrooms/sleeping places of beneficiaries in the rural areas to inspect or observe net usage (LLINs hanging and being used) than it was in the urban areas. Residents in the urban areas felt very reluctant to allow ‘strangers’ to observe their sleeping places.
- The Ashaiman and Ningo Prampram districts had significant numbers of nets not redeemed. For example, Ashaiman had 127 bales (12,700 pieces) at the District Health Directorate not distributed. The reasons given were that during the distribution in July, there was a LLIN shortage (because the districts had not received all their nets) so later when the districts finally had their nets, beneficiaries did not turn up to claim them. The GHS staff had called most of the beneficiaries per the contact numbers in the coupon counterfoils but only few came for the LLINs. This issue was reported to NMCP to take action."
Pgs 4-5.
- We have not seen comprehensive results from the post-distribution tracing exercises in Ghana (e.g., the proportion of selected households that had originally received the appropriate number of LLINs).
- "In Greater Accra, post-distribution validation tracing was implemented for a random sample of households. The same process is planned for the Upper West Region. The Global Fund imposed this requirement." GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, pg. 10.
- Episcopal Relief & Development Ghana Activity Report 3 2016:
- 36
- Episcopal Relief & Development Ghana Activity Report 3 2016:
- "The validation was to establish precisely how many LLINs were distributed during the distribution campaign in all 12 districts. Apart from validating counterfoils, the validators undertook 'End-User Verification'. The end-user verification (EUV) is a rapid check-up to determine whether the beneficiaries really received the number of LLINs allocated for the households and are using the LLINs for the intended purpose. This involved randomly selecting 100 households in each district to verify LLINs received, LLIN use by household members, etc. Validators randomly sampled 100 booklets and from each booklet, randomly sampled one coupon counterfoil for the EUV visit. Validators then called the beneficiaries of the sampled coupon counterfoils and followed up to the households for the end-user verification exercise." Pgs 4-5.
- "24 NMCP validators were assigned to work in the 12 AMF supported districts (2 validators per district). Two teams from ADDRO (a team from the ADDRO HQ and a team from the ADDRO Greater Accra office) visited all the 12 AMF districts to monitor the validation process and to provide support for the packaging of coupon counterfoils for transportation to ADDRO headquarters in Bolgatanga. Key findings were as follows:
- It was comparatively easier for the validators/monitoring team to enter into bedrooms/sleeping places of beneficiaries in the rural areas to inspect or observe net usage (LLINs hanging and being used) than it was in the urban areas. Residents in the urban areas felt very reluctant to allow ‘strangers’ to observe their sleeping places.
- The Ashaiman and Ningo Prampram districts had significant numbers of nets not redeemed. For example, Ashaiman had 127 bales (12,700 pieces) at the District Health Directorate not distributed. The reasons given were that during the distribution in July, there was a LLIN shortage (because the districts had not received all their nets) so later when the districts finally had their nets, beneficiaries did not turn up to claim them. The GHS staff had called most of the beneficiaries per the contact numbers in the coupon counterfoils but only few came for the LLINs. This issue was reported to NMCP to take action."
Pgs 4-5.
- "In Greater Accra, post-distribution validation tracing was implemented for a random sample of households. The same process is planned for the Upper West Region. The Global Fund imposed this requirement." GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, pg. 10.
- AMF notes that: "Our understanding is this was a temporary logistical issue and we asked for this to be addressed in the next ERD report. We have now written to ERD and asked for a quicker, specific update on this issue." Comment provided in response to a draft version of this review in November 2016.
- Episcopal Relief & Development Ghana Activity Report 3 2016:
- 37
- 38
- We discuss case rate data from Ntcheu in our February 2013 update on AMF.
- "[AMF:] Please confirm you are able to provide monthly malaria case rate data going back at least 12 months for each health centre/clinic in the distribution area and will continue to provide monthly data for a period of four years post‐distribution. This ensures we understand pre‐distribution malaria levels and can monitor them post‐distribution.
[Concern Universal:] Malaria case rate information for each of the 16 Health Centres is available from the Balaka District Health Office using the Malawi Government’s national Health Management Information System (HMIS). This information will be sent to AMF separately." Concern Universal Balaka 2013 distribution proposal, pg. 2 - Concern Universal Dedza 2014 distribution proposal, pg. 2, contains the same language as above
- 39
- Balaka 2010-2015 MCRD
- Dowa 2012-2015 MCRD
- Ntcheu 2010-2015 MCRD
- Dedza 2010-2015 MCRD
- 40
AMF told us that the data is likely to be poor quality, and that it will put little to no weight on the results. Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015
- 41
"After the HSAs have collected data from all villages, spot-checkers independently collect the same data from 5% of households in each village. The spot-checkers are employed by the government and receive lunch allowances paid by Concern Universal during the time they are engaged in the exercise. This data is later compared to the HSA data to check for consistency. The two sets of data are entered into a Microsoft Access database separately. Reconciliation of the data is done after data entry. The data is used to calculate the number of nets required in each village, health center catchment area, and district. A printed register including all information recorded during registration is created for each village." GiveWell's non-verbatim summary of a conversation with Nelson Coelho, April 15, 2016, pg. 3.
- 42
Concern Universal Balaka 2015 registration data 100% vs 5%
- 43
AMF told us that 105% registration data was collected for the Dowa 2015 and Ntcheu 2015 distributions. @Rob Mather, AMF Founder, conversation with GiveWell, September 28, 2016@
- 44
@Rob Mather, AMF Founder, conversation with GiveWell, September 28, 2016@
- 45
- AMF Summary features of an AMF distribution
- PDCUs every six months for 2.5 years are included in the distribution agreements AMF signed for its distributions in Ghana and Uganda. Ghana: AMF Ghana 2016 distribution agreement; Uganda: AMF Uganda 2016 distribution agreement
- AMF used to require that distribution partners conduct follow-ups every 6 months for up to 3 or 4 years.
- "[AMF:] Post-Distribution Surveys (see Note 3). Please confirm you will carry out Post-Distribution Surveys (PDSs) every 6 months post-distribution for a period of up to four years to assess the level of net usage (hang-up %), correct usage and condition of the nets and you will provide us with the findings. Each survey would cover approximately 5% of households.
[Concern Universal:] I can confirm Concern Universal and Dedza District Council will conduct a Post-Distribution Survey every six months following the distribution." Concern Universal Balaka 2013 distribution proposal, pg. 2 - Concern Universal Dedza 2014 distribution proposal, pg. 2 contains the same language.
- IMA World Health agreed to conduct surveys every six months for 3 year post-distribution. IMA World Health and AMF distribution agreement for Kasaï-Occidental 2014, pgs. 3-4 and IMA World Health and AMF distribution agreement for Nord Ubangi 2015, pgs. 3-4.
- "[AMF:] Post-Distribution Surveys (see Note 3). Please confirm you will carry out Post-Distribution Surveys (PDSs) every 6 months post-distribution for a period of up to four years to assess the level of net usage (hang-up %), correct usage and condition of the nets and you will provide us with the findings. Each survey would cover approximately 5% of households.
- 46
Nelson Coelho, conversation with GiveWell, April 15, 2016
- 47
"Post-distribution Check-Ups (PDCUs) will be carried out every six months for 2.5 years with five PDCU to be conducted in total. With the PDCUs, all households will be visited by trained volunteers who will collect household data. A different set of volunteers from the ones who will collect data on all households will be trained to visit unannounced, 5% of households in each community and collect data." Episcopal Relief & Development Planning Document 2016, Pg 23.
- 48
Rob Mather, AMF Founder, email to GiveWell, October 31, 2016
- 49
- Rob Mather, AMF Founder, email to GiveWell, June 30, 2014
- A representative of Concern Universal told us that, due to limited indoor space, some recipients take down their nets during the day and hang them up each night. When assessing whether a net is being used, enumerators observe the net and the sleeping space, ask the recipient questions, then make a judgment call. A net that is being used regularly will likely show signs of wear that an unused net would not; and a sleeping space over which a net is hung nightly may show evidence of the hanging even when the net is not present. Nelson Coelho, conversation with GiveWell, April 15, 2016
- 50
In the data for AMF's post-distribution check ups, it looks to us like "Nets Received" = "Hung" + "Present Not Hung" + "Missing" +" Worn out/not usable", implying that "nets hung" and "nets that are worn out" are exclusive categories. See, for example, Concern Universal Ntcheu 2012 33-month post-distribution check-up data
- 51
"'very good condition' (fewer than 2 holes of up to 2cms in size)." Concern Universal Ntcheu 2012 33-month post-distribution check-up report, pg. 4
- 52
- "‘good condition’ (fewer than 10 small holes on them)" Concern Universal Ntcheu 2012 33-month post-distribution check-up report, pg. 3.
- Note that nets with fewer than 10 small holes used to be considered in "OK" condition.
"'OK' (fewer than 10 small holes on them)" Concern Universal Ntcheu 2012 24-month post-distribution check-up report, pg. 4.
- 53
- "‘viable condition’, with more than 10 holes or 1 hole larger than 10 cm," Concern Universal Ntcheu 2012 33-month post-distribution check-up report, pg. 3.
- Note that nets with more than 10 small holes used to considered in "poor" condition. "'Poor' condition with more than 10 holes or 1 hole larger than 10 cm." Concern Universal Ntcheu 2012 24-month post-distribution check-up report, pg. 4.
- 54
"Worn out means not usable. Typically, holes will be 10 cms of cms in size and numerous and the net considered effectively unrepairable and/or structural integrity will have gone. If hung the net would not provide protection against mosquitoes given the extent of holes or tears or other loss of structural integrity." Rob Mather, AMF Founder, email to GiveWell, June 30, 2014.
- 55
- @AMF how we work with distribution partners@, pg. 4.
- According to a representative of Concern Universal, when assessing whether a net is being used, enumerators make a judgment call. Nelson Coelho, conversation with GiveWell, April 15, 2016
- 56
Rob Mather, AMF Founder, email to GiveWell, June 30, 2014
- 57
For example, consider data from the Ntcheu 2012 33-month PDCU:
- [GiveWell]: "Second, I'm trying to better understand the outcomes measured for post-distribution check-ups. In particular, I'm curious what the "People - Covered", "People sleeping under nets", and "Nets Used - Correctly/Not Correctly" metrics mean (under the "Usage of nets" tab). In the Ntcheu 33-month PDCU results , the "Presence of AMF Nets" tab lets me see how many nets are "hung", "present but not hung, "missing", or "worn out": the numbers of "hung" and "present but not hung" nets are 10,667 and 630, respectively, for a total of 11,297 nets that aren't missing or worn out. However, under the "Usage of nets" tab, I see that 11,498 nets are used "Correctly." Since 11,498 > 11,297, I believe that some of the "worn out" nets must be included in the "used correctly" metric. If anyone sleeping under a net that is "used correctly" is considered to be "covered", then some of the people counted in the "covered" category are actually sleeping under worn out nets. Is my understanding correct?"
[AMF]: "This is because the latter figure includes non-AMF nets i.e. there are 462 non-AMF nets included in the ‘Usage of the nets’ tab as we wished to have an overall figure for sleeping space coverage (52%). We may add a note to this tab to indicate there are some non-AMF nets included in the data on this tab." Rob Mather, AMF Founder, email to GiveWell, October 13, 2015. Note that we did not ask AMF where the 462 figure comes from. - 10,664 AMF nets were found to be "very good", "good", or "viable" condition. 10,664 + 462 non-AMF nets = 11,126 nets, which is less than the 11,497 nets "used correctly", implying that some AMF nets that are "worn out" are included in the "used correctly" category. Concern Universal Ntcheu 2012 33-month post-distribution check-up data
- [GiveWell]: "Second, I'm trying to better understand the outcomes measured for post-distribution check-ups. In particular, I'm curious what the "People - Covered", "People sleeping under nets", and "Nets Used - Correctly/Not Correctly" metrics mean (under the "Usage of nets" tab). In the Ntcheu 33-month PDCU results , the "Presence of AMF Nets" tab lets me see how many nets are "hung", "present but not hung, "missing", or "worn out": the numbers of "hung" and "present but not hung" nets are 10,667 and 630, respectively, for a total of 11,297 nets that aren't missing or worn out. However, under the "Usage of nets" tab, I see that 11,498 nets are used "Correctly." Since 11,498 > 11,297, I believe that some of the "worn out" nets must be included in the "used correctly" metric. If anyone sleeping under a net that is "used correctly" is considered to be "covered", then some of the people counted in the "covered" category are actually sleeping under worn out nets. Is my understanding correct?"
- 58
For example, in the Concern Universal Ntcheu 2012 33-month post-distribution check-up data, we see that:
- 10,664 AMF nets are hung ("Presence of AMF Nets" tab)
- 10,664 AMF nets are also in "viable", "good", or "very good" condition ("Condition of AMF Nets tab"). It seems odd to us that these numbers match.
- 11,497 nets are "used correctly." ("Usage of the nets" tab) AMF told us 462 non-AMF nets were included in this tab, meaning that 11,035 AMF nets are included.
- [GiveWell]: "Second, I'm trying to better understand the outcomes measured for post-distribution check-ups. In particular, I'm curious what the "People - Covered", "People sleeping under nets", and "Nets Used - Correctly/Not Correctly" metrics mean (under the "Usage of nets" tab). In the Ntcheu 33-month PDCU results , the "Presence of AMF Nets" tab lets me see how many nets are "hung", "present but not hung, "missing", or "worn out": the numbers of "hung" and "present but not hung" nets are 10,667 and 630, respectively, for a total of 11,297 nets that aren't missing or worn out. However, under the "Usage of nets" tab, I see that 11,498 nets are used "Correctly." Since 11,498 > 11,297, I believe that some of the "worn out" nets must be included in the "used correctly" metric. If anyone sleeping under a net that is "used correctly" is considered to be "covered", then some of the people counted in the "covered" category are actually sleeping under worn out nets. Is my understanding correct?"
[AMF]: "This is because the latter figure includes non-AMF nets i.e. there are 462 non-AMF nets included in the ‘Usage of the nets’ tab as we wished to have an overall figure for sleeping space coverage (52%). We may add a note to this tab to indicate there are some non-AMF nets included in the data on this tab." Rob Mather, AMF Founder, email to GiveWell, October 13, 2015. Note that we did not ask AMF where the 462 figure comes from.
- [GiveWell]: "Second, I'm trying to better understand the outcomes measured for post-distribution check-ups. In particular, I'm curious what the "People - Covered", "People sleeping under nets", and "Nets Used - Correctly/Not Correctly" metrics mean (under the "Usage of nets" tab). In the Ntcheu 33-month PDCU results , the "Presence of AMF Nets" tab lets me see how many nets are "hung", "present but not hung, "missing", or "worn out": the numbers of "hung" and "present but not hung" nets are 10,667 and 630, respectively, for a total of 11,297 nets that aren't missing or worn out. However, under the "Usage of nets" tab, I see that 11,498 nets are used "Correctly." Since 11,498 > 11,297, I believe that some of the "worn out" nets must be included in the "used correctly" metric. If anyone sleeping under a net that is "used correctly" is considered to be "covered", then some of the people counted in the "covered" category are actually sleeping under worn out nets. Is my understanding correct?"
- 11,035 nets "used correctly" is more than the 10,664 nets "hung", implying that nets that were "present but not hung" or "worn out/not usable" could be included. ("Presence of AMF Nets" tab).
- 11,035 nets "used correctly" is also more than the 10,664 nets found in "viable", "good", or "very good" condition, implying that some "worn out" nets are "used correctly."
- 24,980 people were found to be "sleeping under nets" ("Usage of the nets" tab). These are the people who are also labelled as "Covered" by nets.
- 24,980 people / 11,035 nets "used correctly" = 2.26 people per net, which seems reasonable, though higher than we would have expected. 24,980 people / 10,664 nets "hung" = 2.34 people per net, which also seems reasonable but higher than we would have expected. We are uncertain about whether "sleeping under a net" means sleeping under a "used correctly" net, a "hung" net, or a non-"worn out" net.
Concern Universal Ntcheu 2012 33-month post-distribution check-up data
- 59
Full details in our AMF cost per net spreadsheet.
- 60
- The estimate includes actual costs for the Balaka 2013, Dedza 2014, Dowa 2015, Ntcheu 2015, and Balaka 2015 distributions. The only completed large-scale AMF distribution in Malawi not included is Dowa 2012. Concern Universal paid for the non-net costs of the Dowa 2012 distribution; we have not asked for CU's actual or budgeted costs.
- Full details in our AMF cost per net spreadsheet.
- 61
Full details in our AMF cost per net spreadsheet.
In estimating the overall cost per LLIN in Malawi, we have used an average that weights distribution costs by distribution size (i.e. costs associated with larger distributions are given proportionally more weight). Another option would be to put more weight on the actual costs to account for the fact that actual costs are often more reliable than budgeted costs. - 62
Rob Mather, AMF Founder, conversation with GiveWell, May 23, 2014. Managerial staff time is listed in the budget.
- 63
See the “Ntcheu 2012 other costs” sheet in in our AMF cost per net spreadsheet.
- 64
Full details in our AMF cost per net spreadsheet.
- 65
- See our AMF cost per net spreadsheet, "AMF org" sheet for details.
- AMF's fiscal year runs from Jul 1 - Jun 30 (e.g. FY 2012 is Jul 1 2011 - Jun 30 2012).
- 66
For example:
- The spreadsheet IMA shared of actual costs for the Kasaï-Occidental distribution is hard to make sense of and appears to contain several errors. Also, about 20% of the costs are "unconfirmed charges," which are not listed individually.
- For Malawi, Concern Universal shared, in 2012, information on non-monetary costs (such as office space) and the value of time provided by government employees. IMA's actual cost data includes an estimate of indirect costs (as a percentage of total direct costs), but it's unclear if it includes costs paid for by the government or other actors. As a rough approximation, we have added $0.13 per net, which was the estimate of non-monetary partner and local government costs from the 2012 Ntcheu, Malawi distribution.
- 67
Full details in our AMF cost per net spreadsheet.
- 68
Full details in our AMF cost per net spreadsheet.
- 69
- AMF told us that in DRC net costs and non-net costs are very roughly the same. Rob Mather, AMF Founder, conversation with GiveWell, May 23, 2014
- "In other cases, non-net costs are assumed to $2.50 per LLIN, which is the historical average. Non-net costs can vary significantly across countries. For example, in Angola and DRC, non–net costs may be more than $5." Melanie Renshaw, African Leaders Malaria Alliance Chief Technical Advisor, phone conversation with GiveWell, May 23, 2014, pg. 3.
- 70
Full details in our AMF cost per net spreadsheet.
- 71
- Major categories of activities included in Episcopal Relief & Development's budget: attending planning meetings and workshops, monitoring the household registration process, pre-distribution and post-distribution data validation, distribution exercises, post-distribution check-up surveys, data entry, and community sensitization activities. A budget for equipment, office costs, and staff time is also included. Episcopal Relief & Development Ghana non-net costs budget 2016
- See our AMF cost per net spreadsheet for our calculations.
- 72
- We have not yet seen budgets or actual costs from Ghana Health Services from these distributions.
- In June 2016, AMF estimated that standard non-net costs (other than PDCUs) would cost around $1.50 or $2.00 per net for AMF's future distributions (listed as "Addntl non-net costs (per net)" in source). AMF funds status (June 2016)
- We have chosen to use $1.75 per LLIN as an estimate of the costs incurred by Ghana Health Services for its pre-distribution planning activities and distribution implementation activities, though we are highly uncertain of this estimate.
- 73
Full details in our AMF cost per net spreadsheet.
- 74
- 75
Full details in our AMF cost per net spreadsheet
- 76
- "AMF Medium Term Strategy… Increase AMF central staffing by four people before the end of 2016: two program directors, one operations assistant and a technology assistant. Keep under review to assess hiring ahead of need. Hire one to two permanent staff in each country in which large scale distributions are funded." AMF medium term strategy discussion document (May 2016).
- AMF provided us with its estimates for increases in salary costs due to additional hiring in FY 2017 and FY 2018. Rob Mather, AMF Founder, email to GiveWell, November 14, 2016
- 77
For our estimate of AMF's organizational costs for future distributions, we use a weighted average of our annual estimates of AMF's organizational costs per net beginning in AMF's FY 2015, through our projected organizational cost per net for FY 2018. We exclude earlier years, when AMF was operating at a smaller scale and its organizational cost per net was higher than we expect it to be in future years. See our cost per net spreadsheet, "AMF org" sheet for calculations.
- 78
See our most recent cost-effectiveness analysis, "Bednets" sheet, cell J71.
- 79
See for example Roll Back Malaria Partnership gap analysis (September 2013).
- 80
- 81
"The gap analysis includes both the cost to purchase LLINs and non-net costs, such as shipping and distribution. In cases where national governments provided data on non-net costs, this data was used in the analysis. In other cases, non-net costs are assumed to $2.50 per LLIN, which is the historical average. Non-net costs can vary significantly across countries. For example, in Angola and DRC, non–net costs may be more than $5." Melanie Renshaw, African Leaders Malaria Alliance Chief Technical Advisor, phone conversation with GiveWell, May 23, 2014, pgs. 2-3.
- 82
"In November or December 2015, the NMCP learned that the Against Malaria Foundation (AMF) would be providing funding for Ghana's 2016 net distribution. In the absence of this funding:
- There might not have been any nets available for mass distribution in the Upper West Region this year, where the NMCP carries out both IRS and SMC activities. The NMCP believes nets should still be used in regions with IRS and SMC, but prioritizes distribution in regions without them.
- There might not have been enough nets to cover the Greater Accra Region. As a result, the NMCP might have only pursued a targeted mass campaign in the region, and focused on slums and rural areas.
- Continuous distribution of nets, which are carried out in health clinics and schools, might have faced net shortages due to continuous distribution nets being redirected to the mass campaign."
- 83
- "The Greater Accra GHS told us that they expected that rural areas were likely to
have a higher uptake level of nets than semi-urban areas." GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, pg. 19. - In our visits in August 2016 to a few households in semi-urban and rural communities targeted by the July 2016 AMF-funded distribution, we found several households in rural communities with LLINs from the recent AMF distribution hung over sleeping spaces, but none in semi-urban areas. GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016:
- "Semi-urban community visits
GiveWell staff visited 13 households in Ga South and La Dadekotopon districts. The Greater Accra distribution began on July 5th and lasted approximately one week; we are not sure exactly when the distributions in these two communities took place.
Main observations
None of the households we visited were sleeping under an AMF net: some were sleeping under an older net, and some had received AMF nets but were not yet sleeping under them." Pg. 15. - "Our main observations from visiting two rural communities included:
- All eight households we talked to registered for and received nets from the 2016 AMF distribution.
[…] - Four households told us that everyone in their household had slept under 2016 AMF nets the previous night." Pg. 17.
- All eight households we talked to registered for and received nets from the 2016 AMF distribution.
- Our selection process for choosing households to visit included in GiveWell's notes from a site visit to a bed net distribution program funded by the Against Malaria Foundation in Greater Accra, Ghana, August 15-18, 2016, Pg 15.
- "Semi-urban community visits
- "The Greater Accra GHS told us that they expected that rural areas were likely to
- 84
"Continuous Distribution of Nets
This involves nets given for free at health facilities (antenatal clinics for pregnant women and child welfare clinics i.e. during vaccinations) and in schools.
Pregnant women receive a free net when they register their pregnancy at a health center; most women are aware, and take advantage, of this opportunity. Nets are given to children between 18 and 36 months who visit clinics for the measels. Two booster doses are used as an incentive to improve vaccination coverage.
Nets are also distributed to children in selected classes in primary schools in most regions of the country. In 2016, nets were distributed in six regions." GiveWell's non-verbatim summary of a conversation with Ghana's National Malaria Control Program, August 16-18, 2016, pg. 4. - 85
Stephane D'Almeida, conversation with GiveWell, April 14, 2017.
- 86
Rob Mather and Peter Sharratt, conversation with GiveWell, April 18, 2017.