Malaria Consortium — Support for Monitoring and Evaluation of LLIN Distribution Campaign in Ondo State, Nigeria (July 2021)

Note: This page summarizes the rationale behind a GiveWell grant recommendation to Malaria Consortium. Malaria Consortium staff reviewed this page prior to publication.

Published: September 2021

The organization

Malaria Consortium's seasonal malaria chemoprevention program is one of GiveWell's top charities. In March 2021, we recommended funding for the first time to Malaria Consortium for LLIN campaigns. The March grant followed a 2020 grant to Malaria Consortium for scoping LLIN funding gaps.

Planned activities and budget

The grant is for $2.1 million to evaluate the impact of a $12.8 million LLIN campaign in Ondo state, Nigeria, that we recommended funding to support in March 2021.1 The campaign is scheduled to start sometime between November 2021 and January 2022.2

Note that in the budget that Malaria Consortium provided, 36% of the costs are assigned to specific study components, while the remainder is primarily for cross-component costs such as staff time and overheads.3 In the budget breakdown below, we've assigned cross-component costs to study components proportionately. In reality, staff time, which is a large portion of the budget, may be split in different proportions than the component-specific budget is.

This grant will support:4

• Malaria incidence trend analysis ($0.58 million—28% of the budget): Malaria Consortium will select 32 public health facilities (primary health centers) from which to collect data on malaria cases. Only facilities that have reported malaria data consistently for the past few years will be included in the sample. Data clerks based at the health facilities will compile data on malaria cases from patient registers stored at the clinic. Data will be compiled from stored registers for the three years before the LLIN campaign (corresponding to 1-4 years after the prior LLIN campaign, in 2017) and then for the three years following the new campaign.5
• Tests of local mosquitoes' susceptibility to insecticide used in LLINs ($0.49 million—24% of the budget): Mosquitoes will be collected at two times, once before the LLIN campaign and once two years after the campaign, to test their degree of resistance to the chemicals used to treat LLINs. Lab tests will be conducted to measure what proportion of mosquitoes are killed by varying intensities of pyrethroid insecticides used on standard LLINs (both with and without the addition of the piperonyl butoxide [PBO] synergist, a chemical that is added to some LLINs, including the ones that will be distributed in Ondo), and what proportion have genetic markers known to confer some resistance to the insecticide.6
• Net durability monitoring ($0.47 million—23% of the budget): A portion of nets will be labeled prior to the campaign and tracked over time. Once a year, Malaria Consortium will visit households to measure what proportion of the cohort nets are still in place and the extent of physical damage (number and size of holes). A portion of cohort nets will also be tested in a laboratory to measure the efficacy of the insecticide.7
• Tracking exposure to infected mosquitoes ($0.26 million—13% of the budget): Malaria Consortium will combine several sources of data to estimate what proportion of exposure, without nets, would occur outside of the times when people are in bed (and can be protected with a net). The first component is "How many mosquitoes would come to bite humans, if a human is present, at each hour of the evening/night?" Malaria Consortium will measure this by installing mosquito traps inside and outside of houses, having a volunteer sleep under a net next to the traps, and counting the mosquitoes in the traps hourly. This will be done at four houses every six months.8 The second component is "What proportion of the mosquitoes carry the type of malaria parasite that is responsible for most deaths?" Collected mosquitoes will be sent to a laboratory to determine the species of Plasmodium parasite (since only one malaria parasite species is particularly deadly in Africa)9 and whether they carry malaria parasites that could infect humans.10 The third component is to determine what times people are outside, inside but not in bed, and in bed; Malaria Consortium will collect some informal data on this question.11
• Surveys to track the number of nets in each household and reported use rates ($0.17 million—8% of the budget): A random sample of households12 will be surveyed to observe the number of nets and to ask what proportion of people who slept in the house the previous night slept inside a net.13 Surveys will be conducted one month prior to the campaign and three months after the campaign. Additional information on net ownership and use will be collected during the durability monitoring surveys at one, two, and three years after the campaign.14
• Full cost accounting ($0.05 million—2% of the budget): The full costs of the campaign, not only costs borne by Malaria Consortium, will be compiled. One or more health economists will be recruited as consultants to review existing financial records and visit Ondo to collect additional information.15
• Meteorological tracking ($0.03 million—2% of the budget): Malaria Consortium will compile and analyze data on rainfall, temperature, and humidity from weather stations in Ondo to use in interpreting the malaria incidence trends.16

How we plan to use the results

We have several goals in recommending this grant:

1. Help us to evaluate the cost-effectiveness of the Ondo LLIN campaign in retrospect
2. Inform parameters in our cost-effectiveness analyses for future campaigns we're considering supporting
3. Learn about what metrics we might want to track for future campaigns we support and how best to measure them

​​

Study component	Current situation	How we think this grant will help
Malaria incidence trend analysis	Our experience with trying to use malaria incidence data to track the impact of an intervention is primarily from following AMF's work in one country, where AMF ultimately decided that the data was too low-quality to draw conclusions from,17 and Malaria Consortium's efforts in several countries, where Malaria Consortium also concluded that the data was not reliable.18 In both cases, they were drawing data from central government databases.	Learn about metrics we might want to track for future campaigns. More specifically, answer: (1) does the proposed method (which collects data from clinic registers directly) yield reliable trend data, and (2) if so, is it possible to draw conclusions about the impact of LLINs on malaria incidence with trend data and without a control group?
Tests of local mosquitoes' susceptibility to insecticide used in LLINs	There appears to be a relatively large number of data points on pyrethroid resistance from Nigeria,19 with a very small number measuring the impact of PBO on susceptibility.20 We have not seen a study of insecticide susceptibility from Ondo specifically.21 We have used existing data from Nigeria to estimate an insecticide resistance adjustment in our cost-effectiveness analysis for LLIN campaigns in Nigeria.	Help us to evaluate the cost-effectiveness of the Ondo 2021 (or 2022) LLIN campaign in retrospect. The pre-campaign tests will yield a data point on both pyrethroid and pyrethroid + PBO resistance levels for Nigeria, and the first data point we're aware of for Ondo specifically. We can compare this to what we assumed in our cost-effectiveness analysis. Improve the accuracy of parameters in our cost-effectiveness analyses for future LLIN campaigns. We and other funders may use the second data point (from two years after the campaign) to help with selecting the most cost-effective type of LLIN for Ondo and nearby states in future campaigns. We also plan to use both data points in the future in the insecticide resistance adjustment in our cost-effectiveness analysis for LLIN campaigns in Nigeria.
Net durability monitoring	We have relied on a large number of data points in estimating the effective lifespan of a standard LLIN (without PBO). There appears to be significant variation found across studies and locations. We are unsure about the durability of the PBO component when it is added to nets, and our impression is that limited evidence exists on this subject.22 Our understanding is that there are not yet any durability studies specifically of the DuraNet Plus net, the type of net that Malaria Consortium will distribute in Ondo, which contains PBO.23	Help us to evaluate the cost-effectiveness of the Ondo 2021 (or 2022) LLIN campaign in retrospect. It will also be helpful in evaluating the campaign in Anambra state, where DuraNet Plus nets will also be used. We plan to use this data to update the parameter for effective years of coverage24 in our cost-effectiveness analyses. Improve the accuracy of parameters in our cost-effectiveness analyses for future LLIN campaigns, particularly for campaigns that will use the DuraNet Plus net and somewhat for other PBO net campaigns.
Tracking exposure to infected mosquitoes	It may be the case that a greater proportion of exposure to infected mosquitoes happens now at times when people are not in bed, perhaps through mosquitoes adapting to the widespread use of LLINs.25 We don't currently have an adjustment in our cost-effectiveness analysis for changes in exposure compared to when the key studies of LLIN impact were conducted. We have not conducted a literature review on this subject.	We are unsure of the specifics of whether/how we will use the findings from this research. Possibilities include updating on the importance of this factor to cost-effectiveness based on this work and using it as a starting point for further exploration, using the results to make an update to our cost-effectiveness model, and deciding not to use the results or pursue further work on the subject. We decided to include this component based on Malaria Consortium's suggestion to do so, its relatively low cost, and the possibility of it being important to future cost-effectiveness analyses.
Surveys to track the number of nets in each household and reported use rates	Based on past experience with LLIN campaigns and with Malaria Consortium's work on SMC in Nigeria, we expect that rates of pre-campaign ownership of functional nets will be very low (<10%, because it has been over four years since the last campaign),26 and that the campaign will deliver nearly all of the purchased LLINs to households (>90%), resulting in high coverage. We expect moderately high reported use rates.27 We could be wrong about low pre-campaign ownership if there are other sources of LLINs, such as LLINs given out at clinic visits or through schools.	Help us to evaluate the cost-effectiveness of the Ondo 2021 (or 2022) LLIN campaign in retrospect. We will update the parameter in our model for "adjustment for net coverage years lost due to residual nets from previous distributions." We may also update the "net use adjustment."
Full cost accounting	Malaria Consortium has shared a detailed budget and will be able to report back on its own spending. Without further study, it would not be in a position to report on spending by other contributors, such as the state and central governments and their employees and volunteers. We have also seen information on the costs of LLIN campaigns from AMF, but AMF has limited information on costs of LLIN campaigns other than the costs of purchasing and shipping LLINs, which are the components it pays for directly.28	Help us to evaluate the cost-effectiveness of the Ondo 2021 (or 2022) LLIN campaign in retrospect. Improve the accuracy of parameters in our cost-effectiveness analyses for future LLIN campaigns. This work will provide a data point for third-party costs that we will likely use to estimate such costs for future campaigns in other locations. It will also provide a high-quality estimate of distribution costs against which we can triangulate budgets for future campaigns.
Meteorological data monitoring	We haven't used this type of data before.	We expect to look at these indicators in combination with malaria incidence trends. They may help to explain variation in malaria incidence over time that's due to ecological factors rather than net coverage. We are unsure whether this data will be helpful in interpreting results from the malaria incidence trend analysis, but we've decided to collect it because it is relatively inexpensive to do so, and it may be an important control variable for this analysis given the lack of a control group.

Risks and reservations

• We haven't created a cost-effectiveness model of the value of the information from this grant. We are relying on an intuitive sense that the learning in this case is worth the cost of the research.
• It may be time-consuming to maximize learning from this grant, and capacity constraints among GiveWell staff may limit our future takeaways. Capacity constraints limited the amount of planning that went into this grant. We do not have staff formally trained in malaria epidemiology or entomology, and we likely would have changed the plan for this research to some degree if we had spent more time getting context from published research.
• There was additional research that Malaria Consortium suggested and that we ultimately decided not to include. The timeline for deciding on the scope of research was fairly short due to the timing of the campaign, and we felt that it would have taken more time than we had to agree on a plan for these components that we were reasonably confident was worth the cost. It is possible that we made a mistake in missing the opportunity to collect this data; Ondo is an unusual case in that there has been a four-year gap between campaigns,29 so baseline measures pre-campaign are closer to the counterfactual than for typical three-year intervals between campaigns (when there are more LLINs still in place from the last campaign and less time for malaria to rebound to the no-LLINs level). We may also be making a mistake in not collecting data that could be used to help interpret the malaria incidence trends. The main components that we discussed with Malaria Consortium and that are not included in this grant are:
  • Prevalence of malaria and anemia in young children pre-campaign and annually post-campaign. We concluded that data on changes in malaria prevalence was less useful than incidence because:
    • It's possible for incidence to decrease by substantially more than prevalence. Even if the number of people experiencing symptoms of malaria was reduced meaningfully, we might have measured little change in prevalence (the percentage of people with malaria parasites in their blood). The same is true of anemia, which can be caused by many factors, not only malaria,30 and, like prevalence, is at least one causal step away from the primary benefits of preventing malaria in our model (reducing morbidity and mortality). Additional desk-based research could have allowed us to understand the relationship between symptomatic disease and these intermediate measures, but we opted not to do that work.
    • The survey to measure prevalence and anemia was expected to be two to three times as expensive as the incidence study, so we're less likely to decide to incorporate it as a standard part of evaluating future campaigns.
  • Additional collection and laboratory analysis of mosquitoes. We discussed including monthly mosquito collections to analyze species, infectivity, age, and whether they had had a blood meal recently. Similar to tracking prevalence and anemia, we didn't have a strong understanding of what trend we should expect to see if the impact of the campaign was in line with our expectations. This component was also relatively expensive.

Internal forecasts

Confidence	Prediction	By time
40%	We recommend one or more additional grants that fund tracking of malaria incidence over time, with the goal of monitoring an LLIN or SMC campaign.	End of 2026
40%	Conditional on us completing the analysis, our retrospective cost-effectiveness analysis for the Ondo campaign will change by more than 20%.	End of 2025
10%	Conditional on us completing the analysis, we find at least one factor that leads us to update on the cost-effectiveness of future net distributions by at least 20%. This may incorporate additional desk research or data collection beyond this project.	End of 2027

Our process

The study design discussed above was developed iteratively in discussions between us and Malaria Consortium. The nearly final version was reviewed by a Senior Researcher at GiveWell who was not involved in the discussions. We have not yet sought out external feedback, beyond the discussions with Malaria Consortium, on this grant.

Sources

Document	Source
Carrasco et al. 2019	Source (archive)
GiveWell, "Against Malaria Foundation: Supplementary Information"	Source
GiveWell, "Against Malaria Foundation"	Source
GiveWell, "Malaria Consortium — Support for LLIN Distribution Campaigns in Ondo and Anambra States, Nigeria (March 2021)," August 2021	Source
GiveWell, "Malaria Consortium—Seasonal Malaria Chemoprevention"	Source
GiveWell, Counterfactual scenarios on pre-existing nets, 2021	Source
GiveWell, Insecticide Resistance (IR) Mapper synergist bioassay data for Nigeria, 2021	Source
GiveWell, Ondo and Anambra LLIN budget as of March 19, 2021 (redacted)	Source
GiveWell, Ondo and Anambra LLIN distribution CEA, 2021	Source
Gleave et al. 2021	Source (archive)
IR Mapper	Source (archive)
Malaria Consortium, Ondo campaign M&E budget, June 2021 (redacted)	Source
Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021	Source
Malaria Consortium, Ondo state campaign timeline, 2021	Source
PMI VectorLink Project, "Where we work: Nigeria"	Source (archive)
WHO, "Anaemia"	Source (archive)
WHO, Fact sheet detail, "Malaria," April 1, 2021	Source (archive)
WHO, Malaria Threats Map, Insecticide resistance status	Source (archive)

• 1

  See here for the budget for the LLIN campaigns in Ondo and Anambra states. This grant will not cover costs of evaluating the campaign in Anambra state, which we also recommended funding for in March 2021. We expect to recommend a smaller grant for that purpose in the next few months.

• 2

  Unpublished conversations with Malaria Consortium on July 14, 2021, and September 8, 2021. The campaign was originally scheduled to start in October or November (see Malaria Consortium, Ondo state campaign timeline, 2021), but was delayed due to social distancing requirements imposed at the net manufacturing plant in response to COVID-19, and a shortage of shipping containers.

• 3

  See Malaria Consortium, Ondo campaign M&E budget, June 2021 (redacted), "Summary" tab.

• 4

  See here for the list of study components and their percentage of the overall budget, and here for the cost of each study component including proportional cross-component costs.

• 5
  • "Monthly malaria incidence rates will be calculated based on number of RDT-confirmed cases, clinically diagnosed (presumptive cases), test positivity rates, treatment-seeking behaviour and catchment populations of the sampled health facilities. The malaria data will be compiled directly from patient registers, by age, gender and residence for the period 1 November 2018 – 31 October 2024 [if the campaign is postponed to January 2022, data from 1 January 2019 – 31 December 2024]. This will provide three years of baseline for comparison with data the post-campaign period. A stringent data compilation quality control system will be put in place through frequent supervisory visits from the LGA level. Only publicly owned primary health centers or equivalent facilities that have been reporting malaria data regularly during the past few years will be included in the sampling frame, to ensure similarity of the sampling units and representative coverage. [...] A total of 32 primary health centers or equivalent facilities will be selected across the state." Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021, p. 2
  • Malaria Consortium told us that the start of the replacement campaign may be postponed to January 2022. In line with this potential delay, the malaria incidence data collection period may be from January 2019 to December 2024. Malaria Consortium, comments on a draft of this page, September 9, 2021

• 6

  "Insecticide susceptibility studies will be conducted in six sites at baseline and two years after the campaign, to understand changes in resistance levels and make recommendations on the types of ITNs for the subsequent replacement campaign. The WHO and/or CDC bottle bioassays, resistance intensity tests, synergist assays, and molecular analysis (target-site resistance markers and other mechanisms as necessary) will be carried out." Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021, pp. 5-6

• 7

  "Durability of the campaign nets will be studied over three years using standard procedures. The study includes collection of data on attrition (due to loss of nets), physical durability (measured by number and size of holes), and insecticide effectiveness (measured by cone bioassay using susceptible mosquitoes and chemical analysis).

  "Durability monitoring will include identification of a cohort of campaign nets and following them up longitudinally during annual surveys, and additional bioassays of a sample of campaign nets outside the main cohort in each location. The nets for this specific campaign are labelled at the manufacturing stage for ease of identification, with additional unique labelling of the cohort nets. At the end of the third year, a sample of surviving nets from the main cohort will be used for bioassay and chemical analysis. Standard procedures will be used to study ITN durability, including attrition, calculations of the hole index, bio-efficacy, and chemical contents over time." Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021, p. 5

• 8

  "Human biting rates will be estimated using Centers for Disease Control and Prevention (CDC) light traps installed in and around two houses indoors and outdoors at each of the two entomological surveillance sites, over three nights, twice a year. The traps will be installed by a bed or sleeping place where a volunteer (household member) will sleep under an ITN at the indoor and outdoor locations. Mosquitoes will be collected hourly by entomology assistants." Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021, p. 5

• 9

  "In 2018, P. falciparum accounted for 99.7% of estimated malaria cases in the WHO African Region…. [P. falciparum is] the most deadly malaria parasite globally and the most prevalent in Africa." WHO, Fact sheet detail, "Malaria," April 1, 2021

• 10

  "Collected mosquitoes will be stored dry on silica gel (after morphological identification) for subsequent laboratory analyses to determine species, sporozoite infection, and resistance markers." Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021, p. 5

• 11

  "The data will also provide estimates of human biting rates per night, estimated from biting patterns and night time habits of local residents. The length of time villagers spend outdoors and indoors during the night will be observed to understand exposure to mosquito bites and use the information for human-vector contact and biting rate calculations." Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021, p. 6

• 12

  "We will use a stratified multi-stage cluster sampling of households. The first-stage sampling of wards will use PPS [probability proportional to size], followed by the second-stage sampling of households. Households within selected wards will be sampled using simple random sampling. Wards and households will be re-sampled during each survey round.

  "All households in selected wards will be visited and numbered consecutively. The numbers and names of household heads will be recorded. The list will be sent to Abuja and London for sampling to avoid bias. Simple random sampling will be done using computer-generated random numbers. Large wards will be divided into segments before selecting one segment randomly for listing of households to save costs of listing and numbering." Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021, pp. 3-4

• 13

  "Survey staff will record names, sex and age of each household member, whether they stayed in the house the previous night, and whether they slept under a mosquito net. Data on each mosquito net found in sampled households will be gathered, including: type, brand, identity of persons who slept under the net, and where and when the net was obtained." Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021, p. 4

• 14
  • Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021, Table 1, "Durability monitoring (DM) and household survey rounds," p. 3
  • "All durability monitoring household interviews in cohort households will be also used to collect data on ITN ownership and use." Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021, p. 3

• 15

  "The costing analysis will involve review of financial cost data obtained from various partners involved in the ITN campaign, and use of multiple sources to estimate economic costs incurred by each partner. Both desk-based studies and field visits will be required. Consultants (health economists) will be recruited to undertake the work, assisted by Malaria Consortium staff, national and state malaria programme staff and field assistants." Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021, p. 6

• 16

  "Monthly rainfall, minimum and maximum temperatures, and relative humidity data will be collected from all functional weather stations in Ondo State, for up to 10 years before the ITN campaign and during the three years period post-campaign. The data will be analyzed in conjunction with selected indicators.

  "Time series methods will be used to study seasonal and annual trends in meteorological variables in conjunction with trends in malaria morbidity (incidence rates), entomological variables, and data related to ITN use rates and durability data. The meteorological data will be used to explain some of the trends in the other variables and for deeper understanding of the impacts of the intervention." Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021, p. 6

• 17

  See discussion under "Malaria case rate data" here: "In 2016, AMF shared malaria case rate data from Malawi covering January 2010 through March 2015 for Balaka, Ntcheu, and Dedza, and for Dowa from July 2012 to March 2015, but we have not prioritized analyzing it closely. AMF believes that this data is not high quality enough to reliably indicate actual trends in malaria case rates, so we do not believe that the fact that AMF collects malaria case rate data is a consideration in AMF’s favor, and do not plan to continue to track AMF's progress in collecting malaria case rate data."

• 18

  See discussion here.

• 19

  The World Health Organization (WHO)'s Malaria Threats Map identifies 1,024 bioassay tests of pyrethroid resistance status conducted in Nigeria between 2010 and 2019. See more in this spreadsheet.

• 20

  The Insecticide Resistance (IR) Mapper Anopheles map identifies two studies that conducted pyrethroid-PBO synergist bioassays in Nigeria, Awolola et al. 2005 and Djouaka et al. 2016. (See here and here in the data we retrieved from IR Mapper.) We use results from these studies, reported by IR Mapper, in our insecticide resistance adjustment for Nigeria here.

• 21

  No measurements for Ondo state show up in any of the large insecticide resistance summary sources we're aware of, including the IR Mapper Anopheles map and the WHO Malaria Threats Map for vector insecticide resistance status. Ondo is not among the states that the President's Malaria Initiative (PMI) has selected for ongoing entomological monitoring (see here).

• 22

  The recent Cochrane review on pyrethroid-PBO nets states: "Questions remain about the durability of PBO on nets, as the impact of pyrethroid-PBO nets on mosquito mortality was not sustained over 20 washes in experimental hut trials, and epidemiological data on pyrethroid-PBO nets for the full intended three-year life span of the nets is not available." Gleave et al. 2021, "Authors' conclusions"
  

• 23

  See the "Durability of protection provided by an LLIN" section in our write-up of the grant for the Ondo and Anambra LLIN campaigns here.

• 24

  Labeled "Equivalent coverage-years for an LLIN over a 36-month distribution" in the model.

• 25

  See Carrasco et al. 2019 for one discussion of this possibility.

• 26

  Our cost-effectiveness model uses an estimate of 8% of LLINs distributed during a mass distribution campaign existing and effective after four years. Additional nets may be available if purchased by households or received through other programs. Our model assumes that non-campaign sources of nets are negligible. The survey described above will test that assumption.

• 27

  Net use in the trials on which we base our cost-effectiveness model found imperfect usage, often in the range of 60-70%. Our model assumes that usage in current contexts is 90% of what it was in the trials. For more discussion, see the latest version of our cost-effectiveness analysis ​​here, "AMF" tab, "Net use adjustment" row.

• 28

  See here for more detail.

• 29

  See here in our write-up of the Ondo and Anambra LLIN campaign grant: "The last LLIN campaign in Ondo specifically was at the end of 2017, and in Anambra in 2014."

• 30

  "The most common causes of anaemia include nutritional deficiencies, particularly iron deficiency, though deficiencies in folate, vitamins B12 and A are also important causes; haemoglobinopathies; and infectious diseases, such as malaria, tuberculosis, HIV and parasitic infections." WHO, "Anaemia"