Note: This page summarizes the rationale behind a GiveWell grant to IDinsight. Staff at IDinsight and AMF reviewed this page prior to publication.
In a nutshell
In March 2024, GiveWell recommended a $94,500 grant from GiveWell’s All Grants Fund to IDinsight to review and provide input on the Against Malaria Foundation (AMF)’s proposals for enhanced monitoring of its mass distribution of insecticide-treated nets (ITNs) program. AMF is one of GiveWell’s Top Charities, but we have some uncertainty about the quality of its monitoring data. (more)
We are recommending this grant because:
- We believe IDinsight is well placed as a monitoring and evaluation (M&E) partner for AMF and we think it’s likely that implementing IDinsight’s recommendations will lead to improvements in AMF’s monitoring.
- Working with IDinsight enables AMF to prioritize M&E improvements that the organization has not previously prioritized due to capacity constraints. (more)
Our main reservations about this grant include:
- AMF may not be receptive to implementing all of IDinsight’s recommendations.
- IDinsight’s feedback may not meaningfully improve AMF’s M&E proposals. (more)
Published: October 2024
Table of Contents
The organizations
Against Malaria Foundation (AMF)
The Against Malaria Foundation (AMF)'s mass distribution of insecticide-treated nets (ITNs) program is one of GiveWell's top charities. While we have confidence in AMF’s implementation of ITN campaigns overall, we have some uncertainty about the quality of AMF’s monitoring data (see below). Additional content on our view of and grants to AMF can be found here.
IDinsight
IDinsight is an international NGO that partners with clients to generate and apply rigorous evidence to maximize social impact. Depending on client needs, IDinsight helps diagnose social sector challenges, design and test potential solutions, and operationalize those solutions found to be most impactful.1 GiveWell partners with IDinsight to provide monitoring and evaluation support to grantees.
What we think this grant will do
AMF and its partners collect a large volume of information about the ITN campaigns it supports. This includes distribution data, backchecks on the distribution data, and post-distribution monitoring (PDM) data at 9, 18, and 27 months after campaigns. AMF is transparent about sharing this data with GiveWell. While this information allows us to better understand AMF’s work, we have some reservations about its quality and completeness.2 The types of data we receive from AMF and our key uncertainties around it are outlined in more detail on our top charity page for AMF.
This grant will fund a ten-week engagement with IDinsight to review and provide input on proposals AMF has developed for enhanced monitoring of its programs. This will consist of two, potentially three, workstreams:
- Automated checks and analyses to bolster confidence in distribution data. AMF has proposed a series of automated checks on its distribution data, aiming to catch anomalies that may indicate misallocation of nets.3 After familiarizing itself with the format of AMF’s data, IDinsight will review AMF’s proposal and provide recommendations on how to prioritize the suggested checks, or any additional checks that should be prioritized leveraging the technology that AMF already uses.4
- Post-distribution monitoring (PDM)-0 methodology. AMF has proposed a methodology for a randomized post-distribution monitoring exercise shortly after net campaigns (PDM-0).5 IDinsight will review AMF’s proposed methodology for PDM-0 and provide recommendations on what to add, what to remove, and whether and how processes should be modified in order to ensure the methodology is both technically sound and feasible to implement.6
- Pairing rates between distribution and audited data.7 If time allows, IDinsight may also explore innovative, easy-to-implement methods to improve matching rates (the share of households whose records get successfully paired between data collection rounds).8
The budget for this grant is $94,500 and covers staff time, software, and communication costs.9
The case for the grant
We are recommending this grant for the following reasons:
- We believe IDinsight is well placed as an M&E partner for AMF and that their recommendations are likely to lead to improvements in AMF’s monitoring. GiveWell has partnered with IDinsight to advise on and support M&E for a number of our top charities since 2017, including a prior collaboration with AMF that we believe led to improved accuracy in AMF’s post-distribution surveys.10 Reducing our uncertainties around the quality of AMF’s monitoring data will give us more confidence in parameters such as the number of nets being distributed during campaigns, which is a key input in our cost-effectiveness model.
- We think bringing in IDinsight will enable AMF to prioritize M&E improvements that the organization has not previously prioritized due capacity constraints. AMF has a small headcount given the scope of its operations, and while it has put proposals together for monitoring improvements, our sense is that the team has not had the time to implement them.
Risks and reservations
Our main reservations about this grant are:
- AMF may not be receptive to implementing all of IDinsight’s recommendations. We have historically judged AMF to be less concerned than we’ve been on issues around data quality. However, we expect that IDinsight and AMF will work collaboratively to co-create solutions that not only improve AMF’s M&E systems but are also feasible to implement.
- IDinsight’s feedback may not meaningfully improve AMF’s M&E proposals. It is possible that IDinsight will largely agree with AMF’s proposals and will not have very much feedback to offer. However, it may still be valuable to have IDinsight’s M&E experts confirm that AMF’s proposals for improved M&E are sound.
Plans for follow up
- IDinsight will share each deliverable as it is completed, first with AMF to ensure the recommendations accurately reflect AMF’s on-the-ground context, and then with GiveWell.11
- After we receive each deliverable, we plan to have feedback calls with IDinsight and AMF.
- After the IDinsight engagement is finished, we will work with AMF to track progress around implementation of IDinsight’s recommendations.
Our process
In investigating this grant, we reviewed AMF’s monitoring materials and our analyses of them to date. We also had a series of conversations with AMF leadership, AMF’s operations team, and IDinsight.
Relationship disclosures
Two members of GiveWell’s malaria team are former employees of IDinsight, though this project was conceived before either had joined GiveWell.
Sources
- 1
- “IDinsight is a mission-driven global advisory, data analytics, and research organization that helps global development leaders maximize their social impact. We tailor a wide range of data and evidence tools, including randomized evaluations and machine learning, to help decision-makers design effective programs and rigorously test what works to support communities. We work with governments, multilateral agencies, foundations, and innovative non-profit organizations in Asia and Africa.” IDinsight, “About us.”
- “IDinsight is on-site with our clients to translate data into actionable insights. We provide rigorous evidence to improve social sector programs and help clients understand how to use data to inform decisions large and small, within time, budget, operational, and political constraints.Our teams begin with a careful diagnosis of a client’s most pressing questions and challenges.” IDinsight, “Services.”
- 2
Our sense is that more recent data from AMF includes some methodological improvements, better correspondence rates between distribution and post-distribution data, and more realistic net retention figures. Correspondence from AMF to GiveWell, Sept 25, 2024 (unpublished).
- 3
AMF, “Proposed analysis for registration/distribution and PDM geospatial data & timestamps.” Unpublished.
- 4
“IDinsight [will] review [AMF’s proposal], give recommendations on prioritization,
additional checks/ways to leverage the technology already in use, and how to implement
These … We expect this work to entail primarily: review of AMF and GiveWell documents, review of
(de-identified) AMF data and data collection instruments, and conversations with GiveWell and
with AMF." IDinsight, “Scope of Work: IDinsight review of AMF monitoring.” - 5
AMF, “PDM-0 proposal.” Unpublished.
- 6
“AMF has put together a document proposing a methodology for the PDM-0s and
outlining key questions to consider. Instead of using the exact same methodology as the
other PDMs, we’d like IDinsight’s input on tweaking the methodology to serve the
PDM-0’s unique purpose. We’d like IDinsight to weigh in on what to cut, what to add,
whether processes should be modified, etc.” IDinsight, “Scope of Work: IDinsight review of AMF monitoring.” - 7
This workstream is lower priority than the above two workstreams and may only be conducted if IDinsight has additional time before the end of the engagement after completing the above two workstreams. As AMF has indicated that this is a priority for them, we may also conduct a follow-on engagement with IDinsight beyond the initial ten weeks to complete this workstream if needed.
- 8
“We think IDinsight may have ideas for innovative, easy-to-implement ways to improve
household matching in the absence of unique identifiers. This is a lower priority area for
us, but if AMF does succeed in substantially improving pairing rates, it would give them
and us a stronger signal on data quality.” IDinsight, “Scope of Work: IDinsight review of AMF monitoring.” - 9
“Our proposed budget for this 10 week Phase 1 is $94,500. This budget includes staff time, as well as
software and communication costs.” IDinsight, “Concept Note: Estimating ITN Usage in DRC,” June 2023. (Unpublished). (This is a deprecated IDinsight proposal, however the budget included still holds.) - 10
- Information on IDinsight M&E work for GiveWell’s Top Charities can be found here.
- “In 2017, GiveWell commissioned a project with IDinsight to better understand the survey methods used in several countries and to provide suggestions for AMF for the future … AMF notes that its current monitoring procedures implement the majority of these recommendations.” GiveWell, “Against Malaria Foundation – December 2022 version,” section on “Monitoring.”
- 11
IDinsight in conversation with GiveWell and AMF, July 30, 2024. (unpublished)