Against Malaria Foundation - August 2014 Update

Published: August 2014

Summary

Future distributions and room for more funding (More)

    AMF currently has $6.8 million available for future distributions, having committed $8 million to distributions planned for 2014-2017. $2.7 million of its allocated funds are allocated to distributions scheduled to occur more than 2 years from now. Since 2012 (the period in which AMF has negotiated larger-scale distributions), about $8 million of the $10.3 million spent or committed has been in partnership with Concern Universal in Malawi. We continue to believe that AMF has limited room for more funding (though we believe there are ways in which AMF's actions are being affected by funding constraints) and are not planning to re-recommend it at this time.

Updates on past distributions

  • AMF, with Concern Universal, completed a distribution in Balaka district, Malawi in late 2013. The distribution in Dedza, Malawi is now scheduled for late 2014. The distributions have been delayed and expanded. Total expected cost is about $1.46 million. (More)
  • Two years of follow up data from Ntcheu, Malawi is now available – the first of its kind for an AMF distribution – as well as one round of follow up data from Balaka. AMF is the only organization we know of that is regularly and extensively (sampling ~5% of households that received LLINs) collecting post-distribution data on LLIN condition and use. The Ntcheu two-year survey found that 79% of individuals in households that had received LLINs were sleeping under a useable LLIN, and 77% of LLINs were in "very good" or "ok" condition. The Balaka survey suggests that LLINs may be decaying more quickly in that district. Note that we have not seen full technical details for this survey and have therefore not carefully vetted its methodology. (More)
  • We have re-analyzed the total cost to distribute a LLIN funded by AMF. The cost varies considerably by country. We estimate $5.20-$5.40 in Malawi and very roughly $6.70 in DRC. (More

Table of Contents

Distribution in Balaka and Dedza, Malawi

In our February 2013 update on AMF, we wrote that AMF planned to distribute 235,000 nets in two districts of Malawi, Balaka and Dedza, in January-July 2013. This distribution has been delayed and significantly expanded, with Balaka having taken place in October-November 2013 and Dedza scheduled for September-October 2014. AMF told us that the delay was due to the registration survey in Balaka being carried out incorrectly, following a change in leadership at Concern Universal. The survey had to be repeated.1

Balaka distribution

Concern Universal, AMF's distribution partner for this distribution, originally estimated that 80,000 nets would be needed to reach universal coverage in the district. After completing the household registration survey, the number of nets nearly doubled to about 154,000.2 It appears that the difference was due to:

  1. The household registration survey found fewer usable nets than expected: 123,627 nets were expected from a previous distribution in May 2012,3 but the survey found only 79,655 useable nets,4 increasing the number of nets needed by about 44,000.
  2. The household registration survey found a greater population than was previously assumed,5 increasing the number of nets needed by about 10,500.
  3. The average number of people per sleeping space was found to be about 1.7,6 rather than 1.8, as was found in the previous distribution in Ntcheu, increasing the number of nets needed by about 14,000.7
  4. 9,500 nets were added to the total during meetings with villages to verify the results of the household registration survey. These households and sleeping spaces were not included during the pre-distribution registration process.8
  5. 4,575 of the LLINs that were required, according to the pre-distribution registration, were not distributed, reducing the total from 158,805 to 154,230.9

The distribution was originally scheduled for January to April 2013,10 but did not take place until October to November 2013.11 The household registration data was somewhat outdated by the time of the distribution, which caused some logistical difficulties during the distribution.12

AMF has publicly shared the following information from the distribution:

  • Sample data from the pre-distribution household registration (25 households), and totals from the registration, including number of households, people, useable nets, and sleeping spaces.13
  • Weekly reports on the progress of the distribution with number of nets distributed, problems encountered, and lessons learned,14 plus a summary report describing the full distribution process.15 Based on this report, it seems that the distribution followed similar processes to those used by Concern Universal in the Ntcheu distribution, which we detail in our full review of AMF.

Concern Universal did not take photos and video during the distribution. AMF has confirmed with Concern that it will do so during the Dedza distribution.16

A 6-month follow up survey was completed in Balaka in April 2014.17 Details below.

Dedza distribution

The distribution in Dedza was originally scheduled for April to June 2013.18 It is now scheduled for September to October 2014.19 Concern Universal estimates that 245,000 nets will be required for the distribution.20

This brings the total nets for the two districts to about 405,000, or about 170,000 more than previously expected. The total cost to AMF of the Balaka and Dedza distributions is expected to be about $1.46 million.21

Follow up surveys from Ntcheu and Balaka, Malawi

In December 2011 to February 2012, AMF, with Concern Universal, distributed about 270,000 nets in the district of Ntcheu, Malawi. Concern Universal has now completed three follow up surveys in the district: at 6, 15, and 24 months post-distribution.

6-month post-distribution data is also available from the Balaka distribution discussed above.

We have not seen technical details at the level of a high quality academic study on how the surveys were carried out,22 and we therefore have not been able to fully vet the results.

About the Ntcheu 6-month survey

While data from the 6-month report is available, there is no accompanying narrative report (AMF did not ask for narrative reports until before the Ntcheu 15-month survey23 ), so we do not have details on how the survey was carried out. AMF told us that similar methods were used in this survey as in subsequent surveys.24

About the Ntcheu 15-month survey

Concern Universal reported using the following methodology for the 15-month post-distribution survey:

  • Ten villages in each of 37 health facility catchment areas were randomly selected for inclusion in the survey. Concern Universal noted, "Random sampling gave a chance to villages that were in remote areas to be visited [that] would not have been visited if random sampling was not done."25
  • Within each village, 25 households were randomly selected from the pre-distribution registration list. Only households that received nets during the distribution were included in the sample.26
  • Surveys were carried out by "Environmental Health Officers" (EHOs) – officials responsible for supervising Health Surveillance Assistants (HSAs), Malawi's community health workers. They were overseen by the district Malaria Coordinator and his deputy.27

In total, the survey cost about $6,800. Most of the cost was staff time, particularly for the data collectors and their supervisors. Most of the rest of the cost was for fuel.28 Concern Universal noted that it experienced some difficulty determining where the funds for the survey would come from.29

Data entry for the survey took about three times as long as planned, in part because AMF asked Concern Universal to submit data via an online system and internet connectivity at the office in Malawi was limited.30

About the Ntcheu 24-month survey

Concern Universal reported using similar methodology for this survey as with the 15-month survey. 10 villages were randomly selected in each of the 37 health facility catchment areas in Ntcheu. The pre-distribution registration list was used to randomly select 25 households in each village. If no one was home at the time of the survey, a randomly selected alternative household was surveyed.31

Unlike in the 15-month survey, the survey was carried out by HSAs (rather than HSA supervisors).32 Supervisors were asked to visit a randomly selected 5% of the households to check the accuracy of the data collectors’ work.33 We have not seen data on what portion of audits were carried out or what portion found errors in data collection.

As before, Concern Universal noted that AMF's requirement to use its web-based data entry system caused some problems due to internet connectivity outages.34 However, the web-based system improved the efficiency of the data cleaning process.35

The total cost of the survey was $7,650, or about $0.85 per household surveyed.36

For both the 15-month and 24-month surveys, we have not seen technical details at the level of a high quality academic study, and we therefore have not been able to fully vet the results.

About the Balaka 6-month survey

While data from the 6-month report is available, AMF has not yet published an accompanying narrative report, so we do not have details on how the survey was carried out. AMF has told us that this survey used similar methods to those used in Ntechu surveys.37 AMF expects to publish the survey details by October 2014.38

Results from follow up surveys

Data sources in footnote.39

Ntcheu: 6-month Ntcheu: 15-month Ntcheu: 24-month Balaka: 6-month
# of households surveyed 7,689 9,250 8,939 4,536
# of AMF nets received by surveyed households 15,814 18,580 18,448 7,977
% of nets surveyed of total distributed 6% 7% 7% 5%
% of nets hung 90% 85% 81% 87%
% of nets missing 1% 3% 4% 2%
% nets in very good condition 99% 89% 49% 70%
% of nets in OK condition 1% 6% 28% 25%
% of nets in poor condition 0% 2% 15% 4%
% of nets worn out 0% 3% 8% 1%
% of nets used correctly 98% 98% 90% 94%
% of people covered by nets 90% 84% 79% 90%

Definitions:

  • Nets hung: We have not seen a precise definition of this term. AMF told us that interviewers are asked to observe whether nets are hung by entering interviewees' houses (rather than simply asking interviewees if they are hung).40
  • Very good condition: LLIN has fewer than 2 holes of less than 2 cm in size.41
  • OK condition: LLIN has fewer than 10 small holes.42
  • Poor condition: LLIN has more than 10 small holes or has one or more large holes.43
  • Worn out: Survey reports from Concern Universal did not provide a definition of this term. AMF told us that an LLIN is considered worn out if it has multiple large holes and the LLIN is unrepairable, such that it would not provide protection against mosquitos.44
  • Used correctly: According to a survey form from 2012, interviewers are asked to "ask the householder to demonstrate how the nets are used at night" and to then select yes or no to the question "are the nets being used correctly."45 AMF told us that the procedure has not changed in more recent surveys.46

79% of individuals in households that received LLINs sleeping under a useable LLIN (at 24-month follow up in Ntcheu) is roughly in line with usage rates of the trials of bednet efficacy documented in our page on Long Lasting Insecticide Treated Nets, in which small-scale studies of bednet efficacy had net usage rates generally in the 60%-80% range.47

The "decay model" we use to anticipate the lifespan of LLINs assumes that 80% of LLINs are effective at 24 months.48 AMF found that 77% LLINs in Ntcheu were in "very good" or "OK" condition at 24 months. LLINs appear to have decayed more quickly in Balaka, where 70% of nets were in "very good" condition at 6-months (and 25% in "OK" condition), compared with 99% in "very good" condition at 6-months in Ntcheu. It is not clear to us whether these rates of decay are comparable to the assumptions in the decay model, in part because it is not clear whether the definitions of useable are similar in the model and AMF's data. We may do further analysis to more directly compare the model and AMF's data in the future.

Malaria case rate data

In early March 2014, AMF told us that additional malaria case rate data from Ntcheu would be available around the end of March.49 In May, AMF told us that it had received some of this data and expected to post an update on malaria cases in Ntcheu in June.50

Note that in August 2013 AMF and its distribution partner found that there were some problems in the quality of the malaria case rate data in Ntcheu. AMF told us that it would be assessing the credibility of this data.51

Future distributions and room for more funding

As of June 2014, AMF held about $14.8 million. Of this amount, it has committed $8 million to future distributions over the next three years. The remaining $6.8 million is uncommitted.52

Kasaï Occidental, Democratic Republic of the Congo

On March 25, 2014, AMF announced that it had agreed to fund 676,000 nets in the province of Kasaï Occidental in DRC, at a cost of about $1.94 million53 – its largest distribution to date. The UK's Department of International Development (DFID) will fund the non-net costs of the distribution and the distribution will be carried out by IMA World Health.54

AMF notes that DRC is a very difficult country to work in, but that the agreement has met its high standards for monitoring and evaluation.55

Four districts in Malawi

AMF, with Concern Universal, has distributed LLINs, or will soon do so, in three districts in Malawi: Ntcheu (early 2012), Balaka (late 2013), and Dedza (planned for late 2014). AMF has recently announced that is will add a fourth district, Dowa, where it plans to conduct a distribution in November 2014. AMF has also committed funds to replenishing LLINs in the first three districts over the next three years. In total, AMF has committed $5.6 million to distribute 1.4 million LLINs in Malawi in 2014-2017 (not including the previously planned distribution in Dedza in late 2014). Number of LLINs and expected timing by district:56

  • Dowa District: 400,000 LLINs in November 2014
  • Ntcheu District: 350,000 LLINs in October 2015
  • Balaka District: 235,000 LLINs in October 2016
  • Dedza District: 430,000 LLINs in October 2017

North Idjwi, Democratic Republic of the Congo

AMF plans to fund 60,000 LLINs for a universal coverage campaign in the northern part of Idjwi island. It will work in partnership with Amani Global Works. Once this distribution is complete, AMF expects to fund another 60,000 LLINs for the southern part of the island.57 AMF expects each portion of the distribution to cost $186,000, for a total of $372,000.58

Summary of recent and confirmed distributions

Year Distributions planned (# of LLINs) Cost
2012 Ntcheu (268,420)59 $1,110,890
2013 Balaka (154,230)60 and Dedza (delayed to 2014, 245,000)61 $1,210,744
2014 Kasaï Occidental (676,000), Dowa (400,000) and North Idjwi (60,000) $3,726,000
2015 Ntcheu (350,000) and South Idjwi (60,000) $1,586,000
2016 Balaka (235,000) $940,000
2017 Dedza (430,000) $1,720,000

Total funding for distributions in partnership with Concern Universal: $7.98 million.

Total funding for distributions with other partners: $2.31 million.

Distribution discussions that did not succeed

AMF was previously involved in negotiations for two further distributions that would have cost a total of $12.85 million. It is no longer proceeding with these negotiations. The majority of the $12.85 million would have funded a distribution in DRC. There were two main partners involved in the DRC negotiations: a co-funding partner who would have funded the non-net costs of the distribution and a distribution partner who would have carried out the distribution. According to AMF, the distribution did not proceed because "after extensive discussions we did not reach agreement [with the distribution partner] on the detailed aspects of verification of the distribution or monitoring and evaluation within the timeframe necessary to secure a firm commitment from our co-funding partner. During the negotiation, significant budget cuts were also placed on our co-funding partner."

Distribution pipeline

It is our understanding that AMF currently has a limited pipeline for future distributions. It previously was in discussions for several large distributions and if all discussions had succeeded, it would have had a small funding gap.62 AMF told us that it is, as of June 2014, "looking at building a distribution pipeline to replace this."63

Funding constraints

We believe that funding constraints are affecting the actions AMF has taken recently and will take in the short-term. Examples include:

  • AMF would like to be a major partner in a large distribution, which would cost perhaps $10 million or more. This would allow AMF more leverage to influence how the distribution and monitoring was carried out.
  • AMF feels that it can only negotiate distributions for which it has high confidence that it will have enough funds available. This means that AMF is limited in the number of distributions it will explore at one time.
  • AMF would like to have a planning horizon of at least three years. Limited funds now could affect AMF's distribution pipeline for several years into the future.

Our determination that AMF has limited room for more is based largely on our level of confidence in AMF's ability to negotiate further distributions. This determination may change as AMF's negotiations proceed.

Re-estimation of cost per net

In our most recent review of AMF, we relied on an estimate from the Roll Back Malaria Partnership's Harmonization Working Group for the cost to purchase and deliver an LLIN ($5.80). In this update we have attempted to improve on that estimate by: (a) talking to the Harmonization Working Group about its estimate; (b) comparing this estimate to AMF's financial history and plans.

We spoke to Melanie Renshaw, Co-Chair of the Harmonization Working Group. Dr. Renshaw told us that the group uses $2.50 per net for non-net costs because it has found this to be the historical average. Actual costs can vary significantly across countries.64

Using budgets and estimates provided by AMF, we estimate that the total cost to purchase, distribute and follow up on an AMF LLIN is $5.20-$5.40 in Malawi and very roughly $6.70 in DRC. We focus on these two countries because they are where AMF has completed recent distributions or is planning distributions.

The estimate for Malawi is based on:

  • The purchase price per LLIN, calculated from AMF's total LLIN budget and the number of LLINs for the distributions in Dedza ($3.02 per LLIN) and Balaka ($3.06 per LLIN).
  • Concern Universal's budget for non-net costs for the Dedza and Balaka distributions, which include the costs of six follow up surveys. AMF told us that that non-net budget includes all monetary costs incurred by Concern Universal and use of vehicles;65 non-monetary costs such as use of office space and managerial staff time do not appear to be included.
  • An estimate of non-monetary costs incurred by Concern Universal and local governments, based on estimates provided in 2012 by Concern Universal for its distribution in Ntcheu province.
  • An estimate of AMF's organizational costs per net, including an estimate of the value of donated services and volunteer time. Because these costs generally do not vary with the number of LLINs distributed and remain roughly constant on a per year basis, the per LLIN estimate is highly dependent on the time period used and the number of LLINs distributed in that period. We estimate AMF's organizational costs as $0.58 for 2013-2015. This estimate assumes that AMF will complete the distributions it currently plans for that period and does not account for any additional distributions.

We shared our analysis with AMF before publishing and AMF wrote the following in response.

Response from AMF: Using actual costs to assess cost effectiveness

The AMF view is it more relevant to its donors to look at AMF’s actual costs when calculating its cost-effectiveness, rather than adding in costs where either no cash cost exits or support is provided pro bono.

This matters as not to do so distorts cost-effectiveness figures. AMF’s overhead costs per annum are $120,000. Using this figure the cost per delivered and monitored net would be $4.50 - $4.80 (midpoint $4.65).

GiveWell uses an overhead figure of $300,000 due to adding in a CEO salary of $100,000 (AMF’s CEO does not draw a salary) and $80,000 of other overhead costs for services provided pro bono to AMF and therefore where no cash cost exists (the organisations providing pro bono support are very happy to work for no immediate financial return since they benefit in other ways from their involvement, e.g. in improved staff morale. This means they incur no net cost from providing the pro bono service).

As a result, GiveWell calculates the cost per net for AMF at between $5.20 - $5.40 (midpoint $5.30).

GiveWell has calculated the cost per net for other organisations as $5.80.

At $5.30 per net, AMF is 10% more cost-effective than other organisations. At $4.65 per net, 25% more effective. That is a material difference.

We believe our donors are more interested in a comparison derived from AMF’s actual costs rather than penalising it (reducing its assessed effectiveness) by not considering the no-cost and pro bono cost savings they may achieve. AMF as an organisation will continue to seek pro bono support believing this is more cost effective than paying for services.

For DRC, the estimate is much rougher. It is based on the assumption of $3 to purchase an LLIN, $3 in non-net costs (based on AMF's rough estimate that purchase and non-net costs for DRC are roughly equal), and the same estimates of non-monetary and AMF organizational costs used for Malawi.

Full details in this spreadsheet.

We have not yet updated our analysis of the cost per life saved with LLIN distributions. Our previous analysis used $6.13 as the total cost to distribute an LLIN and estimated that this translated to roughly $3,400 per life saved (with a number of caveats). Our revised cost per LLIN would translate into a lower cost per life saved in Malawi and perhaps a slightly higher cost per life saved in DRC. These adjustments do not take into account any differences in the effectiveness across countries and it is not straightforward how to account for such differences.

Source

Document Source
AMF future distributions Unpublished
AMF How we work with distribution partners Source
AMF non-net cost budget for Balaka and Dedza (October 2013) Source (archive)
AMF page on Balaka 2013 distribution Source (archive)
AMF page on Dedza 2014 distribution Source (archive)
AMF page on Kasaï Occidental 2014 distribution Source (archive)
AMF page on North Idjwi distribution Source (archive)
AMF page on Ntcheu 2012 distribution Source (archive)
AMF page on South Idjwi distribution Source (archive)
AMF: "676,000 LLIN distribution agreed in the Democratic Republic of Congo (DRC)" Source (archive)
AMF: "Distributions update, June 2014" Source (archive)
Balaka 6-month post-distribution check up data (April 2014) Source (archive)
Balaka 6-month post-distribution check up plan (April 2014) Source (archive)
Balaka distribution proposal (2013) Source (archive)
Balaka distribution report (2013) Source (archive)
Balaka distribution weekly report (October 7-12, 2013) Source (archive)
Balaka household survey Source (archive)
Cost per LLIN analysis Source
GiveWell's non-verbatim summary of a conversation with Rob Mather, August 15, 2013 Source
Melanie Renshaw, conversation with GiveWell, May 23, 2014 Source
Ntcheu 12-month post-distribution check-up report Source (archive)
Ntcheu 15-month post-distribution check-up data Source (archive)
Ntcheu 24-month post-distribution check-up report Source (archive)
Ntcheu 24-month post-distribution check-up data Source (archive)
Ntcheu 6-month post-distribution check-up data Source (archive)
Rob Mather, conversation with GiveWell, March 6, 2014 Source
Rob Mather, conversation with GiveWell, May 23, 2014 Unpublished
Rob Mather, email to GiveWell, May 22, 2014 Unpublished
Rob Mather, email to GiveWell, June 30, 2014 Unpublished
Rob Mather, email to GiveWell, August 22, 2014 Unpublished