SCI Foundation - November 2018 Version
We have published a more recent review of this organization. See our most recent report on the SCI Foundation, formerly known as the Schistosomiasis Control Initiative.
More information: What is our evaluation process?
Published: November 2018
Summary
What do they do? SCI (https://schistosomiasiscontrolinitiative.org/) works with governments in sub-Saharan Africa to create or scale up programs that treat schistosomiasis and soil-transmitted helminthiasis (STH) ("deworming"). SCI's role has primarily been to identify country recipients, provide funding to governments for government-implemented programs, provide advisory support, and conduct monitoring and evaluation on the process and outcomes of the programs. (More)
Does it work? We believe that there is strong evidence that administration of deworming drugs reduces worm loads but weaker evidence on the causal relationship between reducing worm loads and improved life outcomes; we consider deworming a priority program given the possibility of strong benefits at low cost. SCI has conducted studies to monitor (a non-representative sample of) about 50% of the treatments it has funded to determine whether its programs have reached a large proportion of children targeted. These studies have generally found moderately positive results, but have some methodological limitations. (More)
What do you get for your dollar? Our best guess is that deworming is generally highly cost-effective. We estimate that children are dewormed for a total of around $0.99 per child in SCI-supported programs. This figure relies on several difficult-to-estimate inputs including how to account for (a) donated drugs and (b) in-kind contributions from governments with which SCI works. Excluding drugs and government contributions, we estimate that SCI's cost per treatment is $0.32. The number of lives significantly improved by deworming is also a function of a number of difficult-to-estimate factors, which we discuss in detail in a separate report. (More)
Is there room for more funding? We estimate that SCI will have opportunities to spend $31 million more than we expect it to receive over the next three years (or $24 million if reserve funding is considered to be available to fund this work if necessary). This estimate is highly dependent on the amount of funding SCI receives from a major funder. (More)
Note: In March 2019, we updated our estimate of SCI's room for more funding from $16.8 million to $28.7 million as a result of incorporating SCI's updates to its projections for how much funding it expects to receive from other funders over the next few years. We had previously updated this figure to account for $2.5 million that we recommended Good Ventures donate to SCI.
SCI is recommended because of its:
- Focus on a program with a strong track record and excellent cost-effectiveness. (More)
- Track record – SCI has repeatedly demonstrated success at starting and expanding national deworming programs.
- Standout transparency – it has shared significant, detailed information about its programs with us.
- Room for more funding – we believe SCI will be able to use additional funds to deliver additional treatments. (More)
Major open questions:
- The evidence we have seen on SCI's track record of reaching those it has targeted for treatment is somewhat limited. SCI has conducted studies to monitor (a non-random sample of) about 50% of the treatments it has funded, to determine whether its programs have reached a large proportion of children targeted. We are somewhat uncertain about the results we have seen because of the methodological limitations of the studies.
- SCI may use additional funding to, in part, treat adults in areas with moderate or high risk of schistosomiasis infection, as part of a strategy to stop transmission of the parasite. While this strategy has the potential to be cost-effective by decreasing infection rates in children over the long-term, we have not seen sufficient evidence to support it and do not currently believe it meets our criteria for evidence and cost-effectiveness.
Table of Contents
- Summary
- Our review process
- What do they do?
-
Does it work?
- Is there independent evidence that the program is effective?
- Are deworming pills delivered to and ingested by recipients?
- Are SCI's monitoring results representative of its work overall?
- What is the likely impact per treatment in SCI's programs compared with the independent studies on the impact of deworming?
- Are there any negative or offsetting impacts?
- What do you get for your dollar?
- Is there room for more funding?
- SCI as an organization
- Sources
Our review process
We began reviewing SCI in 2009. Our review has consisted of:
- Extensive communications with SCI staff, including leadership, program managers, and monitoring and evaluation and finance staff.
- Reviewing documents SCI sent in response to our queries.
- Visiting a national schistosomiasis control program meeting and demonstration mass drug administration in Malawi in October 2011 (notes and photos from this visit).
All content on the SCI Foundation, including past reviews, updates, blog posts and conversation notes, is available here. We have also published a page with additional, detailed information on SCI to supplement some of the sections below.
What do they do?
SCI works with governments in sub-Saharan Africa to create or scale up mass drug administration programs (MDAs) for neglected tropical diseases (NTDs), particularly schistosomiasis and soil-transmitted helminthiasis (STH), in school-aged children and other groups determined to be at high risk.1 SCI's role has primarily been to identify country recipients, provide funding to governments for government-implemented programs, provide advisory support, and conduct research on the process and outcomes of the programs.
SCI's model involves both (a) employing staff for program management, technical assistance, and capacity building and (b) funding governments to carry out infection mapping and treatment programs.
SCI’s role in mass drug administration programs
SCI's role in mass drug administrations (MDAs) in general is to:2
- Advocate for the benefits of mass drug distributions to government officials.
- Assist with planning and budgeting.
- Deliver funding and drugs to governments.
- Provide financial management and technical support.
- Develop procedures for monitoring and evaluation, analyze data, and write reports.
In 2014, we spoke with four of SCI's program managers to better understand SCI's role in four countries. These conversations were mostly consistent with our general understanding of SCI's work. We selected Côte d'Ivoire, Ethiopia, and Mozambique because SCI has spent significant unrestricted funds, the type of funding GiveWell has recommended, in these countries (more below). SCI selected Uganda as the fourth case study. We have summarized SCI’s work in these countries on a separate page with additional information on SCI .
Major funding sources for SCI’s work
The major sources of SCI’s funding are (a) unrestricted funds—67% of revenue available to support programs in its 2018/19 budget year3 —and (b) a large, multi-year grant from the UK Department for International Development (DFID), which is due to end at the end of 2018. SCI also has a number of smaller grants from other funders.
For SCI's 2017/18 budget year, we estimate that over 90% of the unrestricted funding that SCI received was due to GiveWell’s recommendation of SCI.4
More detail on SCI’s current and past major sources of funding is available on a separate page with additional information about SCI.
Breakdown of SCI’s spending
Financial reporting and management
We wrote in a 2016 blog post about issues in SCI's financial data. Since 2016, we have found that SCI's financial documents have improved significantly (details in footnote).5
Despite these improvements, we have some remaining concerns about SCI's financial systems and reporting:
- SCI is housed within Imperial College London, a university. Because it is not a standalone charity, it does not publish annual financial statements or undergo annual organization-wide audits (as U.S. charities are required to do).6 In 2017, accounting firm KPMG carried out a piece of "assurance work" commissioned by SCI to look at the way in which SCI manages its donor funding and the way in which it is spent. Our interpretation of the report is that KPMG did not identify major concerns about SCI's financial management.7
- SCI's financial reports have been prone to containing errors. We detail the errors we have learned about in this footnote (including some errors we found in 2016).8 Of particular note: (1) a July 2015 grant from GiveWell for about $333,000 was misallocated within Imperial College until we noticed it was missing from SCI's revenue in March 2016; and (2) in 2015, SCI provided inaccurate information about how much funding it would have from other sources in 2016, leading us to overestimate its room for more funding by $1.5 million.9 The errors we have found largely pre-date SCI's current Finance & Operations Senior Manager, who started in September 2015.10
Spending breakdown by country
April 2017 to March 2018 expenditures by country (in millions USD)11
| Restricted | Unrestricted | Total | % of total | |
|---|---|---|---|---|
| Central expenditure | $2.7 | $1.1 | $ 3.8 | 23% |
| DRC | $0.7 | $1.1 | $ 1.8 | 11% |
| Ethiopia | $0.7 | $1.0 | $ 1.7 | 10% |
| Malawi | $0.3 | $1.2 | $ 1.5 | 9% |
| Tanzania | $1.2 | $0.2 | $ 1.4 | 8% |
| Madagascar | $0.4 | $1.0 | $ 1.3 | 8% |
| Cote D'Ivoire | $0.6 | $0.3 | $ 0.9 | 5% |
| Zambia | $0.8 | $0.1 | $ 0.9 | 5% |
| Niger | $0.3 | $0.3 | $ 0.6 | 4% |
| Uganda | $0.3 | $0.3 | $ 0.6 | 4% |
| Mozambique | $0.6 | $ - | $ 0.6 | 3% |
| Zanzibar | $0.2 | $0.4 | $ 0.6 | 3% |
| Liberia | $0.2 | $0.3 | $ 0.5 | 3% |
| Sudan | - | $ 0.3 | $ 0.3 | 2% |
| Burundi | - | $ 0.1 | $ 0.1 | 1% |
| Mauritania | - | $ 0.1 | $ 0.1 | 1% |
| Total | $8.9 | $7.8 | $16.6 | 100% |
For breakdowns of SCI's spending by country in previous years, see our past reviews of SCI:
- 2016-17 in our November 2017 review
- 2015-16 in our November 2016 review
- 2013-14 and 2014-15 in our November 2015 review
Spending breakdown within country programs
In 2015, SCI began to use a system of country cashbooks, which compare monthly in-country actual spending to budgets.12 Below we have summarized the data from the country cashbooks covering April 2016 to March 2017 for Tanzania (excluding Zanzibar), Uganda, Zanzibar, Niger, Malawi, Côte d'Ivoire, and Burundi.13
SCI also sent us data from its 2015-16 budget year, which is summarized here. The 2015-2016 documents included data from Niger, Côte d'Ivoire, Democratic Republic of the Congo, Malawi, Tanzania, and Uganda.
We also received 2016-2017 cashbooks from Ethiopia, Madagascar, and Democratic Republic of the Congo (DRC). We have excluded those countries from the overall analysis for reasons that are explained in GiveWell's analysis of SCI cashbooks 2016-2017, "Summary" tab, row 18.
SCI also sent us data from its 2015-16 budget year, which is summarized here. The 2015-2016 documents included data from Niger, Côte d'Ivoire, Democratic Republic of the Congo, Malawi, Tanzania, and Uganda.14
Notes about the data in these cashbooks:
- Activity categories in the country cashbooks sometimes appear to be loosely defined and overlapping. For example, expenditure on drug distribution materials is sometimes classified as "drug distribution" and sometimes as "drug logistics."15
- There are some differences between expenses recorded in country cashbooks and expenses recorded in SCI budget vs. actuals 2015-16 Redacted and SCI Budget vs Actuals April 2016-March 2017 Redacted. For example, for Malawi, the total expenditure recorded for the 2015-16 budget year in both the country cashbook and SCI budget vs. actuals 2015-16 Redacted was $0.8 million, but total expenditure for Tanzania differed substantially in the two sources.16 SCI notes that the discrepancy for Tanzania is due to SCI recording the expenditure when the funds were sent to the country, while the cashbooks record when the funds are spent in-country.17 It is our understanding that SCI planned to change this system.
- The total amount of spending recorded in the seven country cashbooks we have included in the table below is $3.5 million, approximately one-third of the $10.9 million in total spending within country programs in SCI's 2016-17 budget year.18
In-country SCI spending in seven countries, April 2016 – March 201719
Activity % of total spending Range across countries Description Drug distribution training 34% 0% to 80% Per diem payments for teachers and officers during MDA training, accommodation and meals Drug distribution 32% 0% to 65% Per diem payments for teachers and officers, fuel costs, communications costs, drug distribution materials (dose poles, registers, etc.) Country management 8% 1% to 34% Per diem payments and salaries for national NTD program staff, fuel costs, communications Social mobilization 5% 0% to 14% Per diem payments, advertising, fuel costs Drug distribution supervision 4% 0% to 10% Per diem payments for MDA supervisors, fuel costs Drug distribution registration 4% 0% to 7% Per diem payments for teachers, drug distribution materials Strategic planning 3% 0% to 13% Expenses related to strategic planning meetings, including per diem payments, travel costs, and accommodation and meals Drug logistics 3% 0% to 8% Drug distribution materials (dose poles, registers, etc.), fuel costs Monitoring & evaluation 2% 0% to 10% Per diem payments for teachers, MDA report writers, and drivers, fuel costs Mapping 2% 0% to 26% Per diem payments for teachers, travel costs, laboratory supplies Global management 2% 0% to 37% Travel costs to an international conference, miscellaneous items (e.g. laboratory supplies, per diem payments for teachers) Advocacy 1% 0% to 2% Per diem payments, fuel costs, communications Other projects
In addition, SCI has received some smaller grants for a variety of projects, including:
- Research. SCI has received a number of smaller grants to carry out research related to NTD control.20
- Other NTD-related activities. SCI has also used funding from individuals for surgeries for hydrocele (a symptom of lymphatic filariasis) in Niger, and health education and water and sanitation programs in Burundi.21
Does it work?
We believe that there is strong evidence that administration of deworming drugs reduces worm loads but weaker evidence on the causal relationship between reducing worm loads and improved life outcomes; we consider deworming a priority program given the possibility of strong benefits at low cost.
To evaluate SCI's track record at executing programs, we have considered:
- Coverage surveys from fourteen of the countries SCI has worked in, including many of the countries where SCI's work has been focused in the past five years. These household surveys estimate what percentage of individuals who were targeted for treatment actually received treatment. Overall, coverage of school-aged children was above 75% (the WHO-recommended minimum threshold) in the majority of districts surveyed by SCI. We note some limitations of these surveys below.
- Prevalence and intensity studies of infection over time from six of the countries SCI has worked in over the past few years. The surveys show substantial improvements following SCI treatment programs. These surveys have a number of limitations and represent a small proportion of the total number of treatments delivered in SCI's programs.
- Academic papers that might reflect the treatment coverage achieved by SCI's programs by directly measuring deworming drug uptake or by measuring worm prevalence in countries where SCI has worked. The papers did not provide a clear case for or against programs being executed well. We discuss this analysis on a separate page with additional information about SCI.
We have now seen some recent monitoring results from approximately eighty percent of the countries in which SCI works. For those countries from which we have seen monitoring results, we have often seen one year of results from the past few years. We are somewhat uncertain about the results we have seen because of methodological limitations of the studies.
In this section, we also discuss how the disease burden in the areas SCI works in compares to the places where the independent studies that form the evidence base for the impact of deworming were conducted. While SCI's programs generally target areas that require mass treatment according to World Health Organization (WHO) guidelines, the disease burden in SCI areas is on average lower than in the study areas, so our expectation is that the average impact per child treated is lower in SCI areas. We adjust our cost-effectiveness analysis accordingly.
Is there independent evidence that the program is effective?
SCI supports mass school-based deworming programs, the independent evidence for which we discuss extensively in our intervention report on deworming programs. In short, we believe that there is strong evidence that administration of the drugs reduces worm loads but weaker evidence on the causal relationship between reducing worm loads and improved life outcomes; we consider deworming a priority program given the possibility of strong benefits at low cost.
There are some important differences between the type and severity of worm infections in the places SCI works and the places where the key studies on improved life outcomes from deworming took place, which we discuss below.
Are deworming pills delivered to and ingested by recipients?
Coverage surveys
We have seen results from coverage surveys that SCI has conducted, or worked with partners to conduct, in Côte d'Ivoire (in 2014, 2016 and 2018), Malawi (2012, 2014, 2016 and 2017), Uganda (2014 and 2017), Mozambique (2015 and 2016), Zanzibar (2015), Zambia (2015), Ethiopia (2015, 2016 and 2017), Madagascar (2016 and 2017), Burundi (2017), Liberia (2017), Mauritania (2017), Niger (2017), Nigeria (2017) and Tanzania (2017). We summarize the methods and results in this spreadsheet and discuss themes in the sections that follow.
Methods
In each of the surveys, surveyors visit a sample of households and ask children, or in some cases their parents on their behalf, whether they received treatment in the most recent MDA. Villages and households for these surveys are generally selected randomly or quasi-randomly.22 Other survey questions, such as questions about age, gender, where the respondent received the treatment, and why the respondent did not take the drug(s), are often included as well.23 The main results are reported as "survey coverage" figures (the number of school-aged children interviewed who ingested the drug(s) divided by the total number of school-aged children interviewed), and are intended to check the accuracy of governments' "reported coverage" figures (the number of treatments delivered to school-aged children according to government administrative data divided by the estimated number of eligible school-aged children in the area).24
The methodology used in SCI's coverage surveys has differed somewhat across surveys. We have summarized the details of the methodologies used in the surveys in this spreadsheet ("Methods" sheet).
"Survey coverage" and "reported coverage" estimates of the proportion of school-aged children that ingested deworming pills in SCI programs can differ substantially.25 We generally believe that coverage surveys provide more reliable information on the proportion of school-aged children that received and ingested deworming pills than "reported coverage" figures calculated by governments of countries with SCI-supported programs (justifications in footnote).26 However, we note some limitations to SCI's coverage surveys:
- Accuracy of survey responses: We have a few concerns about the accuracy of responses from school-aged children, particularly young children (e.g., 5-6 year olds), on whether they received deworming treatments. Some coverage surveys include "verification questions" (e.g., asking children if they recognize pills), but we have not yet seen evidence from these questions that raises our general confidence in the accuracy of SCI's coverage surveys.
- Length of time between MDA and survey: This varied between one and six months in the coverage surveys we have seen from SCI. Intuitively speaking, the more time that passes, the less likely children are to remember accurately and the more likely they are to confuse past MDAs. Mozambique (2015) and Côte d'Ivoire (2016) had the shortest interval at 1-2 months and Zambia had the longest at 5-6 months, of the surveys for which we have seen information about the length of time between the MDA occurring and the survey occurring. Other surveys were generally carried out 2-4 months after the MDA.27
- Verification questions: In Côte d'Ivoire, Malawi, Uganda, Zanzibar, Zambia, Ethiopia, Madagascar, Mauritania, Liberia, Burundi, Niger and Nigeria surveyors asked some verification questions in at least one of the coverage surveys that we have seen, such as whether respondents recognized pills or dose poles presented by the interviewers, what they thought of the pills (praziquantel is very large and tastes bitter), and how many pills they took.28
- In Mozambique (2015), respondents were asked whether they recognized the dose pole used in schistosomiasis MDAs. However, in that survey parents were surveyed on their children's behalf29 and most children (79%) received drugs at school,30 presumably when parents were not present. We do not know how to interpret the result that a very high percentage of parents (median 90%, ranging 61-94% across provinces) reported recognizing the dose pole.31 SCI hypothesized that parents may either recognize the dose pole from publicity efforts prior to the MDA or remember a similar dose pole from previous MDAs for lymphatic filariasis.32 In either case, this may indicate that the coverage survey was not measuring actual delivery of drugs to children in the 2015 MDA. The Mozambique (2016) coverage survey does not indicate that any verification questions were asked.33
- Recognition of pills and dose poles was low in some coverage surveys. This is surprising given the high survey coverage rates in those surveys, and reduces our confidence in the accuracy of the survey coverage rates. For example, from the most recent round of coverage surveys we have seen, from 2017 and 2018, children seem to have particularly low recognition in Mauritania (2017) and Ethiopia (2017). In Mauritania (2017), recognition of praziquantel pills is approximately 10% or less in two out of three districts, and yet praziquantel survey coverage rates are greater than 80% in both districts.34 In Ethiopia (2017), recognition of both praziquantel and albendazole pills is approximately 40% or less in at least three out of four regions, and yet survey coverage rates for both treatments are greater than 75% in all regions.35
- Supervision and auditing of surveys: Many of SCI's reports on coverage surveys mention that teams of surveyors were overseen by supervisors. It is our understanding that supervisors provide teams with guidance on the logistics of implementing coverage surveys36 but do not implement any specific data quality or auditing procedures, such as re-surveying a sample of households to check the accuracy of the data collected.37
Additionally, there are limitations that apply only to some of SCI's coverage surveys:38
- Selection of geographic target area: All of the surveys were limited to specific geographic areas (such as districts). In Uganda (2014), Zambia (2015), Côte d'Ivoire (2014), Ethiopia (2015), Madagascar (2016), Nigeria (2017), Tanzania (2017) and Liberia (2017) these were selected randomly or nearly randomly. In Malawi (2012, 2014, 2016 and 2017), Mozambique (2015), Ethiopia (2016 and 2017), Burundi (2017), Niger (2017) and Uganda (2017) the districts were purposefully selected and not intended to be nationally representative. The selection procedures for Zanzibar (2015), Mozambique (2016), Côte d'Ivoire (2016 and 2018), Mauritania (2017) and Madagascar (2017) were not given in the documents we have seen on the surveys or were unclear.39 We believe the criteria used for the non-random sampling of districts in Ethiopia (2016 and 2017) and Malawi (2017) are particularly likely to lead to bias in the coverage results, though we are uncertain how many districts were excluded as a result of the criteria used.40 Villages and households are randomly selected within each geographic target area.41
- Independence from the government: In Mozambique (2015 and 2016), the survey was carried out by government health staff, who may have had an incentive to bias the results. SCI told us, "[M]ost of the interviews in one district were done by the other district officers with no connection with the district."42 We believe the Ethiopia (2016 and 2017) and Mauritania (2017) surveys were also carried out by government health workers. The reports on the Malawi (2014, 2016 and 2017), Uganda (2017), Burundi (2017) and Madagascar (2017) surveys note that the surveyors were independent of the government. University students or staff conducted the surveys in Malawi (2012), Uganda (2014), Zanzibar (2015), Zambia (2015), Côte d'Ivoire (2014, 2016 and 2018) and Niger (2017).43 It is unclear to us who conducted the Ethiopia (2015), Madagascar (2016), Nigeria (2017), Tanzania (2017) and Liberia (2017) surveys.
- Whether parents or children were interviewed: In Mozambique (2015 and 2016), parents were interviewed about whether their children took the drugs. In both Côte d'Ivoire and Malawi (both 2012 and 2014 surveys), if children in a household were not available, then their parents were interviewed about whether the children had received deworming drugs. SCI made different choices about whether to include these responses in the results, which slightly inflated the results overall.44 SCI told us that parents were not asked to answer on behalf of their children in Uganda (2014), Zanzibar (2015), Ethiopia (2015), and Madagascar (2016), and that going forward, SCI no longer plans to have parents answer on the behalf of children.45 We have seen no further mention of this in the reports for the 2017 and 2018 coverage surveys. The Ethiopia (2016) coverage survey protocol indicates that caretakers were interviewed when children were unavailable only about non-school-based MDAs.
Results
Results from the coverage surveys we have seen are available here. For coverage of praziquantel (to treat schistosomiasis), the median point estimate for coverage achieved by SCI, according to coverage surveys, was 85% and the average was 83%. For coverage of albendazole (to treat STH), the median point estimate was 87% and the average was 81%.
The fact that the surveys identified low coverage in several cases increases our confidence in their reliability. Given the smaller sample size, government involvement in the survey, and question about parents recognizing the dose pole noted above, we are more skeptical about the results from Mozambique (2015) than those from other surveys. We have low confidence in the results from Mauritania (2017) because of government involvement, uncertainty about how districts were selected and children's low recognition of the pills. We also have low confidence in the results from Mozambique (2016) and Ethiopia (2016 and 2017) because of government involvement and uncertainty/concerns about how districts were selected. Additionally, we have limited confidence in the results of Côte d'Ivoire (2016) due to concerns about the reliability of the data.46
Prevalence and intensity studies
SCI has conducted surveys to track changes in schistosomiasis and STH prevalence and intensity rates following SCI-supported treatment programs. In each of these studies, SCI tracked infection rates at the same schools ("sentinel sites") each year. In general, prevalence and intensity of the parasites decreased over time in each of the countries studied. We note several methodological limitations of these surveys below.
Below, we discuss results from studies of schistosomiasis and STH prevalence and intensity from eight countries: Niger (2004-2006),47 Burundi (2007-2010) and (2007/8-2017),48 Liberia (2012-2013),49 Malawi (2012-2017), Madagascar (2015-2017), Tanzania (2016-2017), Côte d'Ivoire (2013-2016) and Ethiopia.50 We summarize the methods and results in this spreadsheet and discuss themes in the sections that follow.
Results from prevalence and intensity studies
The studies in Niger, Malawi (2012-17), Burundi (2007-10) and Liberia (2012-13) were originally designed as cohort studies, in which the same individuals are repeatedly surveyed over time.51 In the 2015 Malawi and 2013 Liberia studies, SCI switched to a cross-sectional sample, where random children from the same schools were surveyed, rather than the same individuals.52 The studies in Burundi (2007/8-2017), Madagascar (2015-17), Tanzania (2016-17), Côte d'Ivoire (2013-16) and Ethiopia also sample a random cross-section of children from the same schools in the baseline and follow-up surveys.53 There is no control group for these studies due to the ethical implications of withholding treatment from infected children.54
Results from the impact surveys we have seen are available here. Prevalence and intensity for the two main types of parasites that cause schistosomiasis, S. haematobium and S. mansoni, decreased over time in each of the countries studied other than Côte d'Ivoire, in which there was a small increase in prevalence for both types of infection, and Tanzania, in which there was an increase in prevalence and intensity for S. mansoni.55 The prevalence and intensity of S. mansoni increased between the second and third follow-up surveys in Burundi (2007/8-2017), but remained significantly lower than in the baseline.56
The prevalence and intensity of two of the three soil-transmitted helminths, hookworm and ascaris, also decreased over time in most of the countries studied. In Madagascar, the prevalence and intensity of ascaris increased, and the intensity of hookworm and ascaris increased in Ethiopia.57
Though it is possible that other factors besides the treatment program caused these declines (such as improved sanitation infrastructure), the fact that the declines occurred over a short period following treatment strongly suggests that treatment caused or contributed to it.
The prevalence and intensity of the third soil-transmitted helminth, trichuris, was generally low and the changes over time were more mixed. Prevalence and intensity decreased in approximately the same number of countries as it increased.58
In some countries, there were large differences in the change in prevalence across schools, with some schools experiencing changes in the opposite direction to the average trend for the country as a whole. In Côte d'Ivoire, despite little change in the prevalence of S. mansoni or S. haematobium on average, several schools experienced increases in the region of 10%.59 In Madagascar, whilst the prevalence of S. haematobium infection and heavy intensity infection decreased overall, there were large differences across schools.60 In Ethiopia, the prevalence of S. mansoni increased in nearly one third of schools, despite an overall reduction.61
Some of the studies also report results for other indicators of disease, such as anemia. We omit discussion of these other indicators because they are more likely to be influenced by external factors than are prevalence and intensity (see our previous review of SCI for discussion of these indicators and SCI Liberia impact survey dashboard 2012-13 for results from Liberia).
Limitations of the prevalence and intensity study data include:
- Monitoring of selected locations in some countries. It appears that, in the Niger and Burundi pilot studies, locations included in the study were selectively chosen rather than selected to be a representative sample of treated areas. In the Côte d'Ivoire, Ethiopia, Liberia, Madagascar, Malawi and Tanzania studies, schools were sampled randomly from locations at medium and high risk of infection in line with a stratified statistical approach. We are uncertain how locations in the Burundi (2007/8-2017) study were chosen.62
- Attrition of schools from the sample. In Liberia, nearly one-third of the schools surveyed during the baseline year could not be re-visited in the follow-up year (which SCI attributes to inaccurate school identity numbers, school closures, and/or inaccurate recording of GPS coordinates), and were replaced by nearby schools.63 In Madagascar, two out of twenty-nine locations could not be revisited and were not replaced.64
- Low follow-up rates in cohort studies. Follow-up rates were low in two of the three studies using a cohort model for follow-up (89% at the first year follow-up in Niger, 33%-50% in the pilot survey and 53%-80% in the other schools survey in Burundi (2007-2010), and 52% in the first follow-up in Malawi).65 To be included in follow-up surveys, children must be present in school when the surveys are done.66 If those who are present in school are less likely to be infected than those who are not present, this could lead to overstating the impact of the program. The connection between infection status and absenteeism could be a direct relationship (infection could cause absenteeism) or an indirect one (a third factor, such as poverty, could cause both higher levels of infection—perhaps through poor sanitation infrastructure—and absenteeism). As previously noted, SCI has moved away from the use of cohort studies.
- Different methods of measurement for the prevalence of S. mansoni lead to different prevalence rates in Burundi (2007/8-2017). In Burundi (2007/8-2017), both the rapid urine-based Circulating Cathodic Antigen (CCA) and Kato-Katz methods are used to measure the prevalence of S. mansoni. Prevalence using the CCA method is always higher and for one school was nearly fifty times higher. It is unclear which method is used in the reported results, or whether the same method is used in the baseline and each follow-up survey.67
- The procedure used to sample children changed between the baseline and follow-up surveys in Malawi. In the second, third and fourth follow-up surveys in Malawi, children were sampled by age, not by grade as in the baseline. It is unclear whether the age distribution of children in these follow-ups is comparable to the baseline, and so whether prevalence rates are comparable.68
Are SCI's monitoring results representative of its work overall?
We have now seen monitoring results (coverage or pre- and post-treatment prevalence and intensity surveys) that cover about 50% of the treatments that SCI delivered in 2014-2017. There are several large country programs for which we have not seen any monitoring results (or the only monitoring results we have seen were from 4 or more years ago).
Details in this spreadsheet.
SCI told us that it is sometimes unable to share results because programmatic data about populations is owned by governments and therefore, in line with data sharing agreements between SCI and the countries, it is necessary to gain permission from the governments before data can be shared. It can also take some time for data, once collected, to reach SCI because in some countries it is cleaned and analyzed by country program staff before being shared with SCI.69 For example, for Mozambique, one of SCI's largest recipients of both restricted and unrestricted funds, SCI shared a report from a consultant who visited the country in May 2015 to assist with data cleaning and analysis for prevalence data from 2012, 2013, and 2014. The report notes major problems with this data and the refusal of the government to allow SCI and other international partners to have access to the data outside of Mozambique.70
Based on conversations with SCI, we understood that it planned to carry out more coverage surveys than we have received results from. Therefore, the results we have seen could overstate SCI's impact if coverage surveys are more likely to be skipped or the results withheld in countries with lower coverage rates. There are couple of reasons this might be the case:
- Country programs that have more capacity and experience are likely to be those that both carry out high-quality distributions and complete all the steps necessary for coverage surveys to be implemented and the results shared with SCI (and thus with GiveWell).
- Coverage surveys are more likely to be skipped or results withheld if implementers recognize that the surveys are likely to show low coverage results and reflect poorly on them. We have no evidence that this has occurred for SCI-supported programs, and note this only as a general possibility.
What is the likely impact per treatment in SCI's programs compared with the independent studies on the impact of deworming?
In general, mass deworming programs treat everyone in a targeted demographic, regardless of whether each individual is infected (more). Because of this, the benefits (and therefore the cost-effectiveness) of a program are highly dependent on the baseline prevalence of worm infections.
In this section, we discuss how the disease burden in the areas SCI works in compares to the places where the independent studies that form the evidence base for the impact of deworming were conducted. While it is our understanding that SCI's programs generally target areas that require mass treatment according to WHO guidelines, the disease burden in SCI areas is on average lower than in the study areas, so our expectation is that the impact per child treated is lower in SCI areas. We adjust our cost-effectiveness estimate (more below) accordingly.
We have seen baseline data on the prevalence and intensity of schistosomiasis and STH infections for countries that account for about three-quarters of the treatments SCI has delivered in recent years. Schistosomiasis and STH prevalence and intensity in these countries was generally fairly low compared to the studies providing the best evidence for the benefits of deworming (Croke 2014 and Miguel and Kremer 2004).
Baseline data was collected in schools that had been selected for prevalence and intensity studies. The baseline reports use methodologies that seem similar to the other SCI prevalence and intensity studies discussed above. With the exception of the study discussed above from Malawi, we have not fully vetted the methodology used in these studies.
In Malawi an error in data collection may have resulted in prevalence being underestimated.71 In Zanzibar, treatment has been ongoing,72 so the study does not reflect pre-treatment conditions.
Detailed results and sources are available in this spreadsheet.
Are there any negative or offsetting impacts?
We discuss several possible considerations but do not see significant concerns.
Administering deworming drugs seems to be a relatively straightforward program.73 However, there are potential issues that could reduce the effectiveness of some treatments, such as:
- Drug quality: For example, if drugs are not stored properly, they may lose effectiveness or expire.
- Dosage: If the incorrect dosage is given, the drugs may not have the intended effect and/or children may experience additional side effects.
- Replacement of government funding: We have limited information about whether governments would pay for the parts of the program paid for by SCI in its absence. We also have little information about what governments would use the resources they put toward deworming for if they did not choose to implement deworming programs.
- Diversion of skilled labor: Drug distribution occurs only once every year or two and is conducted by volunteers in communities or teachers in schools. Given the limited time and skill demands of mass drug distribution, we are not highly concerned about distorted incentives for skilled professionals. Planning for the program can take senior government staff time. While we have limited information on what these staff would spend their time on in the absence of deworming programs, we would guess that they would support other education or health initiatives.
- Adverse effects and unintended consequences of taking deworming drugs: Our understanding is that expected side effects are minimal and there is little reason to be concerned about drug resistance in the near team (more information from our report on deworming). We are somewhat more concerned about potential side effects during integrated NTD MDAs, since multiple drugs are taken within a short time period. However, it is our understanding that organizations follow protocols to space out the treatments so as to avoid adverse effects.
- Popular discontent: We have heard a couple of accounts of discontent in response to SCI's mass drug administration campaigns, including one case that led to riots.74 SCI notes that following episodes of popular discontent, it has worked with governments to improve public education about the programs.75 Additionally, during deworming activities supported by Evidence Action's Deworm the World Initiative in Ogun State, Nigeria in December 2017, rumors of students collapsing reportedly generated panic that led some parents to take their children out of school; the Ogun State government denied that any students collapsed.76
What do you get for your dollar?
We estimate that on average the total cost of a schistosomiasis treatment (which is often combined with STH treatment) delivered in SCI's programs is $0.99. Excluding the cost of drugs (which are often donated) and in-kind government contributions to the programs, we estimate that SCI's cost per treatment is $0.32.
We make a number of assumptions and judgments in developing our estimate. Our process could introduce errors that overstate or understate the actual cost, and there are some significant sources of uncertainty. More on our approach below.
Note that our estimate of the number of lives significantly improved by SCI's programs is a function of a number of difficult-to-estimate factors. We discuss how the cost per treatment figure relates to how much it costs to improve a child's health and development in our report on mass treatment programs for schistosomiasis and STH. We incorporate our estimates into a cost-effectiveness model which is available here.
SCI's estimates
In October 2017, SCI estimated that it would cost £0.18 on average per additional treatment delivered to a school-aged child, or about $0.24 USD at the exchange rate at the time. It estimated £0.27 per child in the first year of a new country program.77 For community-based distributions (used when adults are also targeted for treatment), SCI also estimated £0.27 per person in delivery costs, but noted that it would need to purchase praziquantel for adults at a cost of £0.24 per treatment. Praziquantel is currently donated by a pharmaceutical company for school-aged children but not for adults.78
Our approach
Our general approach to calculating the cost per treatment is to identify comparable cost and treatment data and take the ratio. We prefer to have a broadly representative selection of treatments in order to mitigate possible distortions, such as using data from a new program, which may incur costs from advocacy, mapping, etc. before it has delivered any treatments.
It is our understanding that SCI generally intends to treat for STH in all places where it treats for schistosomiasis, so the treatments SCI reports can generally be interpreted as combination schistosomiasis and STH treatments,79 though we are aware of several cases in which schistosomiasis-only treatments were delivered either by design or due to problems with implementation, and of some cases where SCI delivered STH-only treatments (SCI told us that STH-only treatments are not counted in its treatment numbers).80
To get the total cost, we attempt to include all partners (not just SCI), such that our cost per treatment represents everything required to deliver the treatments.81 In particular, we include these categories of costs:
- SCI’s funding to country programs (e.g., to fund drug delivery).
- SCI headquarters' costs (e.g., for management and technical salaries), including an estimate of costs paid by Imperial College (e.g., office space and some legal and administrative expenses).
- Cost of drugs. We include the full market cost of all praziquantel that is needed to deliver the treatments, regardless of whether SCI purchased it or used donated drugs. It is our understanding that DFID funds praziquantel for some countries and that in recent years SCI has not purchased drugs beyond what is funded by DFID and donated by a pharmaceutical company.
- Costs incurred by the government implementing the program (e.g., for staff salaries when working on treatment programs).
We start with this total cost figure and apply adjustments in our cost-effectiveness analysis to account for cases where we believe the charity's funds have caused other actors to shift funds from a less cost-effective use to a more cost-effective use ("leverage") or from a more cost-effective use to a less cost-effective use ("funging"). (Further discussion in this blog post.)
SCI notes that cost per treatment calculations should include sensitivity analysis82 —i.e., analysis on the degree to which the cost per treatment varies when various assumptions vary. We have not yet completed such an analysis.
Our analysis
We analyzed several sources of data, which cover different country programs and time periods between April 2015 and March 2018, and developed several different cost per treatment estimates based on the inclusion or exclusion of different types of costs. Full details in this spreadsheet. Our estimates are:
- SCI’s cost per schistosomiasis treatment, including government costs, drug costs and research costs: $0.99,
- SCI's cost per schistosomiasis treatment, including drug costs and research costs, but excluding government costs: $0.69,
- SCI's cost per schistosomiasis treatment, including government costs and research costs, but excluding drug costs: $0.46, and
- SCI's cost per schistosomiasis treatment, including research costs but excluding drug costs and government costs: $0.33.
- SCI's cost per schistosomiasis treatment, excluding government costs, drug costs and research costs: $0.32.
Shortcomings of our analysis
While we believe the estimates described above are reasonable, we want to highlight specific reasons to interpret them with caution.
- We rely on reported treatment data. Our understanding is that these data can often be inaccurate. We have discounted the number of treatments (by 5%) based on the differences between reported treatment rates and treatment rates found in the coverage surveys discussed above (see footnote for why this is an imperfect comparison).83
- We rely on an estimate that 30% of overall program costs are attributable to the government. We derived this estimate from an analysis of a single program in Niger (this footnote elaborates on this estimate).84
- We have learned about some cases where SCI reported treatments for a program where it played a limited role.85 We have adjusted for the cases we know of, but are not confident that we have learned about all such cases.
- We do not have data that indicate what proportion of drugs are wasted. We expect that in some cases drugs are purchased or donated but expire before use. We do not know how common this is. In our analysis, we have assumed that 10% of drugs are wasted.
- We do not have data on the expenses Imperial College incurs to support SCI. Based on a conversation with SCI, we have roughly estimated these expenses as 10% of SCI's expenses (excluding drugs and government contributions).86
- In DRC, SCI is one of several donors contributing to a program with pooled funding. We are not sure how closely the treatment numbers reported in DRC by SCI alone reflect the contribution that it has made. We believe that using data on the total costs and treatments across all actors in DRC is more likely to give a representative estimate of SCI's cost per treatment there. We have used this pooled analysis to estimate SCI's cost per treatment in DRC.87
Is there room for more funding?
Note: In March 2019, we updated our estimate of SCI's room for more funding (accounting for the $2.5 million that we recommended Good Ventures donate to SCI) from $16.8 million to $28.7 million as a result of incorporating SCI's updates to its projections for how much funding it expects to receive from other funders over the next few years.88
We believe that SCI could productively use more funding than it expects to receive and that is it likely to be constrained by funding, but that there is significant uncertainty in this estimate.
In short:
- Estimated needs: We estimate that SCI will have opportunities to spend $59 million over the next three years (April 2019 to March 2022) to treat school-aged children (and some adults) in its 15 current countries of operation. SCI would also like to add $1.7 million to its reserve funding.
- Cash on hand: SCI currently holds $9.5 million in uncommitted funding available to allocate to future budget years. It also holds $5.0 million in reserve.
- Expected additional funding: We roughly estimate that SCI will receive an additional $4-6 million to support its budget in each of the next three years. SCI's grant from DFID (for ~$6.7 million per year) ends at the end of 2018 and so revenue in 2019-2021 is highly dependent on whether that funding is renewed. We have assumed that it will continue at a similar level with 60% likelihood but there is a wide range of possibilities.
In summary, we estimate that SCI will have opportunities to spend $31.2 million more than we expect it to receive over the next three years, or $24.4 million if we consider reserve funding to be available to support this work if necessary. This estimate is highly dependent on the amount of funding SCI receives from DFID.
Update: In November 2018, we recommended that Good Ventures grant $2.5 million to SCI; our updated estimate of its room for more funding is approximately $28.7 million.
More detail and calculations in this spreadsheet.
Uncommitted and expected funding
SCI had $22.1 million in unrestricted funds in the bank as of June 30, 2018. Of this, $7.6 million was committed to programs for the April 2018 to March 2019 budget year and $5.0 million was held in reserve against unexpected shortfalls. $9.5 million was uncommitted.89 In addition, GiveWell had allocated $3.2 million to SCI, which it was due to receive in August 2018 or soon after.90
We project that SCI will receive funding from the following sources for its work in the next three budget years:91
- Unrestricted support independent of GiveWell: In the past year, SCI has received $1 million in unrestricted funding that we do not attribute to GiveWell's recommendation.
- Unrestricted funding due to being a GiveWell top charity: GiveWell maintains both a list of all top charities that meet our criteria and a recommendation for which charity or charities to give to in order to maximize the impact of additional donations, given the cost-effectiveness of remaining funding gaps. We estimate that SCI will receive $1 million from donors who use our top charity list but don't follow our recommendation for marginal donations.92
- Restricted funding from DFID: Through 2018, SCI will receive about $6.7 million per year from DFID. As of August 2018, DFID had announced that it will continue to fund work on neglected tropical diseases past 2018, but it had not yet announced how that funding will be structured, how much will be allocated to deworming, or who the implementing agencies will be. SCI will be applying for funding from DFID in late 2018.93 SCI told us that it believes there is a 50-70% chance that it will receive continued funding from DFID and that the amount of funding would most likely be in the range of £3 to 6 million in the next year and £3 to 5 million the following two years. For our analysis of SCI's room for more funding, we have estimated an expected value of this funding of £2.4-2.7 million (about $3.2-$3.6 million) per year.
- Restricted funding from other sources: There are two other funders that have provided restricted funding to SCI in the past. Based on SCI's estimate of the likelihood of continuing to receive funding from these sources and accounting for known commitments, we estimate SCI will receive $0.6 million per year in restricted grants in the next two years and $0.2 million in the third year.
In total, we project that SCI will raise about $4-6 million per year in each of the next three years. This is highly sensitive to funding from DFID, which accounts for 59-74% of projected revenue.
More detail and calculations in this spreadsheet.
Additional spending opportunities
We estimate that SCI will have opportunities to spend $59 million over the next three years (April 2019 to March 2022) to treat school-aged children (and some at-risk adults) in its 15 current countries of operation. SCI is not currently proposing to expand its work to additional countries. SCI would also like to add $1.7 million to its reserve funding in accordance with its reserve policy.94
SCI provided estimates of the costs of maintaining programs in each of the 15 countries it currently works in over the next three years. SCI notes that these budgets were based on requested from countries, are fairly rough estimates and will be refined annually.95
These budgets include funding to treat adults, in addition to school-aged children, in countries where treatment of at-risk adults is ongoing.96 SCI has told us that would be difficult for it to convince these countries to cease using funding from SCI to treat at-risk adults as this is in line with the WHO's current guidance.97 About 82% of total treatments would go to school-aged children, with the remainder to adults at high risk of infection.98
SCI is also interested in using funding to expand its support for treatment of additional at-risk adults, in line with WHO guidance.99 To meet all requests from countries for this work, SCI would need to spend a total of $109 million over three years.100
We expect to exclude funding used for the treatment of adults (except in cases where treatment of adults is ongoing) in considering how much funding to recommend donors give to SCI. We believe the strongest case for the cost-effectiveness of mass deworming is based on the possibility of long-term developmental impacts in children who are dewormed. It is our understanding that treating adults may decrease infection rates in children, but we have not yet investigated how large this effect might be. Our best guess at this point is that it would be significantly less cost-effective than treating children. Additionally, we do not feel confident that we will be able to learn in the future what portion of SCI's funding was used to treat adults. SCI's monitoring processes do not include checks on the number of treatments delivered to adults and to children. SCI has indicated that they are planning to monitor the relative cost of treating adults versus children in the next year's treatments.
Note that for our room for more funding analysis we ask top charities for rough estimates of their ideal budgets for the next three years. Because GiveWell-directed funding can fluctuate significantly year-to-year, our preference is for the organizations to which we direct funding to treat the funding as a multi-year grant to help smooth funding year-to-year and allow for longer-term planning.
In past years, we have noted that we have had challenges communicating with SCI about its room for more funding.101 Our communication with SCI and the quality of SCI's financial information has improved in 2016-2018 (see above), so we have more confidence than we have had in the past in the accuracy of the information SCI provided us on its room for more funding, though as discussed below, there are major sources of uncertainty in our conclusions, due primarily to uncertainty about future funding from DFID.
Major sources of uncertainty
DFID funding
SCI estimates a range of potential annual funding from DFID of £3-6 million in the next year and £3-5 million the following two years. If SCI does not receive funding from DFID after its current grant ends in 2018, its funding gap, at $39 million, will be considerably larger than we currently estimate and this would have significant implications for SCI's ability to maintain its programs.102 SCI told us that it believes DFID is likely to provide a similar level of support for deworming as it has in the past few years, and that SCI is well-positioned to compete for that funding because it has a track record of implementing programs for DFID.103
Reserve funding
We have not yet applied a consistent approach to how we account for organizations holding and building up funding reserves in our analyses of their room for more funding. We expect to improve on our approach in the future. We consider room for more funding on a three-year basis in order to mitigate the risk to the organization of fluctuations in GiveWell-directed funding. Some organizations have a policy of holding reserves in addition to this three-year funding. Information about SCI's reserve policy is available in this footnote.104 Possibilities of approaches we could take in considering reserves in the context of our room for more funding analyses include:
- Refer to the organization's reserve policy to determine how much funding it should set aside as reserves. SCI is seeking $1.7 million in additional reserves in the next year. Using this approach, our best guess of SCI's room for more funding over the next three years is $31.2 million.
- Consider reserve funding to be available for supporting program costs in the third year of our room for more funding analysis. SCI currently holds $5.0 million in reserves that could potentially serve as a backstop against funding shortfalls in its 2021-2022 budget year. Using this approach, our best guess of SCI's room for more funding over the next three years is $24.4 million.
- Vary the time period included in our room for more funding analysis based on organization-specific factors such as what portion of an organization's spending are costs that take time to wind down (for example, core staff salaries) and costs that can be scaled down quickly (for example, drug purchases), and/or the percent of the organization's budget that is supported by GiveWell-directed funds. An approach like this, if designed well, would better achieve the goal of mitigating risk to the organization, but would likely add significant complexity to the analyses.
Global need for treatment
There appears to be a substantial unmet need for STH and schistosomiasis treatment globally.
In 2017, the World Health Organization (WHO) released a report on 2016 treatments stating that:105
- 69% of school-age children in need of treatment were treated for STH in 2016, up from 63% in 2015 and 45% in 2014. Coverage was 65% in African countries in 2016.
- 52% of school-age children in need of treatment were treated for schistosomiasis in 2016, up from 42% in 2015. Coverage was 58% in African countries in 2016.
We have not vetted these data.
SCI as an organization
- Track record: SCI has an extensive track record of aiding governments to start and scale-up national deworming programs, as discussed above.
- Self-evaluation: SCI’s self-evaluation is strong compared to the vast majority of organizations we have considered. We discuss limitations to this evidence above.
- Transparency: SCI has consistently been strong in its commitment to transparency. It has generally provided the information we’ve asked for and has never hesitated to share it publicly (unless it had what we felt was a good reason). It has allowed a lot of public dialogue about its work that other charities may have been uncomfortable with.
- Communication: In the past, we have had some challenges communicating effectively with SCI representatives. Our communication with SCI has improved in 2016-2018.
More on how we think about evaluating organizations in our 2012 blog post.
Sources
- 1
"Objectives of SCI
- To encourage development of sustainable schistosomiasis and STH control programmes in sub-Saharan Africa.
- In the selected countries: to reach at least 75% of school-aged children (which in most countries would be from 6 to 15-year-old) and other high-risk groups with chemotherapy, namely PZQ and ALB; and thereby reducing prevalence and intensity of schistosomiasis and STH infections; as well as reducing schistosomiasis-related morbidity in high risk groups; and burdens due to STH infections in the targeted populations.
- To create a demand for sustained schistosomiasis and STH control.
- To promote access to anthelmintic drugs and good case management in the regular health system.
- To develop a rigorous monitoring and evaluation plan which will generate the information required to determine whether or not the objectives have been met."
Fenwick et al. 2009, Pg. 3.
- 2
- GiveWell's non-verbatim summary of a conversation with Alan Fenwick, SCI Director, June 17, 2010.
- SCI Summary sheet of treatments instigated and overseen by SCI.
- SCI Proposal by SCI, Imperial College to manage the Program for Integrated Control of Neglected Tropical Diseases in Côte d'Ivoire.
- Conversations with four program managers, discussed below.
- 3
GiveWell's analysis of SCI 2018-19 budget May 2018 Redacted, "summary" sheet, cells C18 and C19. Calculation: 14,048,651 / 21,091,731 = approximately 67%.
- 4
SCI received 9.9 million GBP in the fiscal year April 1, 2017 to March 31, 2018 (unpublished source: SCI unrestricted income update as of July 31, 2018). Using an average exchange rate in this period of 1.33 GBP per USD (from https://www.ofx.com/en-us/forex-news/historical-exchange-rates/), we estimate that SCI received $13.1 million in unrestricted funds in this period. GiveWell tracked $12,082,412 in donations to SCI that were attributed to GiveWell's recommendation of SCI in that period (internal donation records), or 92% of the total. Note that there is uncertainty in this comparison due to (a) delays in some cases between when GiveWell granted funds to Imperial College Foundation and when SCI received a corresponding grant from Imperial College Foundation and recorded the revenue in its records, and (b) the possibility of donors making donations to SCI based on GiveWell's recommendation without telling SCI about the reason for their gifts.
- 5
- In several of our reviews and updates on SCI before 2016, we noted that we were uncertain about how much funding SCI held.
- See our November 2015 review of SCI.
- See our August 2015 update on SCI.
- Since 2016, SCI has sent us documents indicating how much funding it held at the start of each budget year:
- SCI allocation table 2016-2017, "Summary" sheet
- GiveWell's analysis of SCI 2017-18 budget May 2017 Redacted, "Income Analysis USD" sheet.
- GiveWell's analysis of SCI 2018-19 budget May 2018 Redacted, "summary" sheet.
- SCI sent us information on in-country spending breakdowns for several of its country programs for its 2015-16 and 2016-17 budget years (see here). We wrote in our November 2015 review that we had not seen recent information on how SCI spent funding within country programs.
- In several of our reviews and updates on SCI before 2016, we noted that we were uncertain about how much funding SCI held.
- 6
SCI notes, "Imperial College has charitable status as a UK Higher education establishment and is audited in line with the requirements of the UK charity commission." Comment provided in response to our draft of this page in June 2016.
We have looked at Imperial College's financial statements and did not find that they helped us understand SCI's finances.
- 7
KPMG's work consisted of reviewing controls in place at SCI to manage the receipt of grant funds from donors, the passing of funding through to partners in the countries where SCI programs operate, and the recording and monitoring of expenditure against the specific budget for each grant. It also involved undertaking a series of tests related to the expenditure incurred in delivering the grants, covering different categories of expenditure for the period April 1, 2016 to March 31, 2017. We do not have permission to publish KPMG's report, but interested parties can contact SCI to request access to the report.
- 8
- In our November 2012 update on SCI (see footnote 1 of that report), we noted inconsistencies in the spending data SCI shared with us. In one case, we asked SCI about a discrepancy we have noticed and SCI's reply was implausible.
- In our next update on SCI, in October 2013, we reported having a fuller view of how SCI had spent funds, though noted a discrepancy in funds held: "We reported (after checking this number with SCI) that as of October 2012, SCI held $2.65 million in unrestricted funds. As part of this update, we sought to understand SCI's use of unrestricted funds between October 2012 and August 2013. SCI told us that it had held $1.96 million in unrestricted funds as of October 1, 2012, about $660,000 less than we previously thought. We have assumed that we miscommunicated with SCI last year; one possibility is that the numbers we received included some restricted funds and we may not have fully excluded all restricted funds in our adjustment. We have used the more recent figure for this update."
- In October 2014, SCI shared reports of spending since our last update and balance of unrestricted funds at the end of the period. Combining these figures with our previously published figure for balance of unrestricted funds as of our last update implied negative revenue over the period. We asked SCI to explain the discrepancy. SCI first told us that the discrepancy was due to counting some restricted funds in the unrestricted balance. At the same time, SCI provided data on unrestricted revenue over the period, giving us a double check on these figures. There was a discrepancy between actual revenue and revenue implied by the starting and ending balances and spending over the period of about £528,000. When we asked SCI about this discrepancy, SCI told us that there was an additional expenditure of £595,420 that was missing from the report previous, as well as a small amount of additional income, which reduced the discrepancy to £11,651.
- In June 2015, SCI shared financial statements for 2013/14 with us that showed no spending in Ethiopia in that year (SCI draft financial statements for 2013/14 and 2014/15). We asked SCI about this because other communications with SCI indicated that SCI had been active in Ethiopia in that year. SCI noted that this was an error and was likely due to confusion by newer staff on how to interpret older financial data: "The three transfers which were made to Ethiopia during 2013/2014 were recorded to the L‐account which is an account the Imperial College currently use to record unrestricted income. In the past however, the L‐account was also used to record expenditures ‐ but later we changed our recording system whereby expenditures were accredited to 'NX' accounts which were created to capture country expenses. Because the transfers went straight from the L account [SCI staff member] did not identify them as being for Ethiopia." Alan Fenwick, email to GiveWell, October 1, 2015.
- "In October 2015, SCI shared estimates for its April 2016-March 2017 budget year with GiveWell on target treatment numbers by country, amounts of funding available from DFID and other large donors, and the amounts of additional funding required to deliver the targeted number of treatments in each country and cover SCI's central expenditures (http://www.givewell.org/files/DWDA%202009/SCI/SCI_planned_SCH_treatment_numbers_by_country_by_year_(October_2015).xlsx). GiveWell interpreted the estimates for SCI's projected central expenditures in the document as part of SCI's funding gap for its 2016-2017 budget year.
"Documents SCI sent GiveWell in March 2016 indicate that around $1.5 million more in funding from DFID is available to allocate for SCI's 2016-2017 budget year than GiveWell had previously expected. This is because the document from October 2015 included funding available from DFID that could be allocated to in-country programmatic expenditures, but did not include the funding from DFID that SCI planned to allocate to central expenditures (included in the March 2016 documents)." GiveWell's non-verbatim summary of a conversation with Dr. Wendy Harrison, Najwa Al Abdallah, and Dr. Lynsey Blair, April 6, 2016.
- In July 2015, GiveWell granted $333,414 to SCI, which included all donations received between February and March 2015. In March 2016, after reviewing a preliminary version of SCI L-account 2016, GiveWell told SCI that the July 2015 funding did not appear to be accounted for. SCI then learned that the funding had been misallocated by Imperial College. SCI received the funding in April 2016. Dr. Wendy Harrison and Najwa Al Abdallah, conversation with GiveWell, May 4, 2016.
- 9
See previous footnote.
- 10
Wendy Harrison, email to GiveWell, September 8, 2015.
- 11
- SCI Budget vs Actuals April 2017-March 2018. Conversion to USD in unpublished sheets.
- These spending totals include funding budgeted for SCI's 2017-18 budget year and funding rolled over from SCI's 2016-17 budget year.
- SCI's payments to Imperial College ("IC overheads") are counted as "Central expenditure" in this table.
- In SCI Budget vs Actuals April 2017-March 2018, LSTM is the Liverpool School of Tropical Medicine. For this table, we have added spending labeled "LSTM" to the total for Mozambique.
- SCI told us that all LSTM spending for its April 2017 to March 2018 budget year was in support of the program in Mozambique. Najwa Al Abdallah, email to GiveWell, August 15, 2018
- 12
"In 2015, SCI implemented the use of cashbooks by its country programs to report monthly spending. Countries where SCI receives funding from the Department for International Development (DFID) implemented this system first, and it has now been rolled out to all of SCI's country programs. A few countries are currently delayed in submitting cashbooks to SCI because of limited capacity." @GiveWell's non-verbatim summary of a conversation with Dr. Wendy Harrison and Najwa Al Abdallah, February 17, 2016@.
- 13
- Cashbooks:
- SCI cashbook Tanzania 2016-17.
- SCI cashbook Uganda 2016-17.
- SCI cashbook Zanzibar 2016-17.
- SCI cashbook Niger 2016-17.
- SCI cashbook Malawi 2016-17.
- SCI cashbook Côte d'Ivoire 2016-17.
- SCI cashbook Burundi 2016-17.
- We also received 2016-2017 cashbooks from Ethiopia, Madagascar, and Democratic Republic of the Congo (DRC). We have excluded those countries from the overall analysis for reasons that are explained in GiveWell's analysis of SCI cashbooks 2016-2017, "Summary" tab, row 18.
- SCI cashbook Ethiopia 2016-17
- SCI cashbook Madagascar 2016-17
- SCI cashbook DRC 2016-17
- 14
- SCI cashbook Niger 2015-16.
- SCI cashbook Côte d'Ivoire 2015-16.
- SCI cashbook DRC 2015-16.
- SCI cashbook Malawi 2015-16.
- SCI cashbook Tanzania 2015-16.
- SCI cashbook Uganda 2015-16.
- SCI cashbook summary 2015-16.
- 15
SCI notes, "SCI carries out capacity development of and actively trains in-country accountants on the activity terminology and types of expenses that fall under each which are clearly defined e.g Drug distribution is for expenses related to conducting MDA at a site while drug logistics refer to the transportation of drugs from the source to MDA sites." Comment provided in response to our draft of this page in June 2016.
- 16
- Total expenditure in 2015-16 was $0.8 million for Malawi and $1.4 million for Tanzania. GiveWell's analysis of SCI budget vs. actuals 2015-16 Redacted, "BVA in USD" sheet, cells D16 and D24.
- Country costs sub-total for Malawi is 528,675 GBP ($767,239). GiveWell's analysis of SCI cashbook summary 2015-16, "Malawi" sheet, cell F23.
- Country costs sub-total for Tanzania is 265,973 GBP ($385,993). GiveWell's analysis of SCI cashbook summary 2015-16, "Tanzania" sheet, cell F23.
- 17
"The reason for the differences: SCI BvA is prepared based on imperial college information system (ICIS) -expenditure is recorded when a transfer is made to a country while the cash book record when the money is being spent. Example of Tanzania 2 transfers were made: on 16 Dec. 2015 for the amount GBP 344,018 from DFID and GBP 406,696 from unrestricted. In the cash books Tanzania didn’t spend all the money received in Dec. 2015 as the MDA was delayed and took place during Jan-March 2016." Comment provided in response to a draft of this page in June 2016.
- 18
- $3,524,923 in total spending. GiveWell's analysis of SCI cashbooks 2016-2017, "Summary" tab, cell M19.
- $10,888,087 in total country programs expenditure. SCI Budget vs Actuals April 2016-March 2017 Redacted, "summary USD" tab, cell N11.
- 19
- GiveWell's analysis of SCI cashbooks 2016-2017 "Summary" sheet.
- Descriptions for these activities drawn from line items in SCI cashbooks from 2015-2016 and 2016-2017.
- 20
SCI's summary of active accounts as of June 2014 lists six research grants totaling £2.6 million, or about $4.1 million over 2010-2014. SCI advisory board financial report (June 2014), Pg. 5.
- 21
- "Once we have people that want to give at least $100,000, we talk to them directly. [...]
Someone wanted to do something special with his money, so we're doing hydrocele surgery in Niger. He gave us $200,000 and we told him we could do ~1000 hydrocele surgeries." Alan Fenwick, SCI Director, phone conversation with GiveWell, February 16, 2011.
See also Alan Fenwick, SCI Director, phone conversation with GiveWell, September 15, 2011. - Note: "SCI generally doesn't do water and sanitation programs because of the expense. In Burundi they're doing water and sanitation programming because they have been successful there with running a program and treating schistosomiasis, but soil-transmitted helminth infections remain persistent." GiveWell's notes from visit to Malawi on October 17-19, 2011.
- "Once we have people that want to give at least $100,000, we talk to them directly. [...]
- 22
- For a full description of the methodology used in each survey (and sources for the statements below), see this spreadsheet, "Methods" sheet.
- SCI has told us (in reports on the coverage surveys or through personal communication) that villages are selected randomly within districts that the coverage survey is implemented in (often with adjustments to account for differing population sizes of villages).
- SCI has told us (in reports on coverage surveys or in personal communication) that households are either chosen through random selection from a village register or through a "random walk" method. A description of a "random walk" method for choosing households:
- "Households were selected using the random walk procedure. A central point in the village was designated and a bottle was spun to randomly select a direction of walk. All households along the direction of walk were counted. A sampling fraction was calculated and the households selected." SCI Madagascar coverage survey recommendation report 2016, Pgs 4-5.
- Note that the selection process for villages and households is not fully clear to us for all coverage surveys we have seen. Also note that coverage surveys are implemented in a selection of districts covered by the MDA program, and the selection of districts for coverage surveys is not always random or intended to be representative of the MDA program as a whole. See here, "Methods" sheet for sources and details.
- For a full description of the methodology used in each survey (and sources for the statements below), see this spreadsheet, "Methods" sheet.
- 23
For a full description of the methodology used in each survey, see this spreadsheet, "Methods" sheet.
- 24
- "The primary objectives of this coverage survey were to:
- Quantify and validate PZQ and MBD treatment coverage for SCH and STH, respectively;
- Assess coverage rates disaggregated by school attendance and gender for SAC;
- Collect information on why targeted eligible individuals did not receive or accept treatment.
Reported coverage was defined as, number of SAC ingesting drugs / eligible SAC population x 100
Survey coverage defined as, number of SAC interviewed that ingested the drug / total number of interviewed
SAC x 100" SCI Madagascar coverage survey recommendation report 2016, Pg. 4. - For a comparison between survey coverage and reported coverage rates, see this spreadsheet.
- Our understanding (formed over several conversations with SCI) is that the governments of countries with SCI-supported programs calculate "reported coverage" figures. The numerator of these figures (the number of treatments delivered to school-aged children according to government administrative data) is calculated by aggregating data from each school in the program on the number of reported treatments delivered. The denominator of this figure (estimated number of eligible school-aged children in the area) is calculated by referencing the most recent government census in the area, which may be adjusted to estimate the effect of population growth since the last census, or by using school enrollment data.
- "The primary objectives of this coverage survey were to:
- 25
For a comparison between survey coverage and reported coverage rates, see this spreadsheet.
- 26
- It seems plausible to us that there may be an incentive for schools to over-report the number of treatments delivered, or for districts or regions to over-report aggregated treatment figures.
- We have not seen calculations used by governments to find the denominator of the reported coverage calculation (estimated number of eligible school-aged children in the area), but it seems that these estimates are sometimes substantially inaccurate. For example, note that in this spreadsheet, "Results" sheet, that some reported coverage estimates are over 100% (which means that the aggregated reported treatment figures to school-aged children are greater than the government's estimate of the total number of school-aged children in the area).
- Several people have told us that it is difficult to get accurate government administrative data and that data is often missing from some portion of schools/clinics/etc.
- 27
See this spreadsheet, "Methods" sheet, column D.
- 28
- Côte d'Ivoire (2014): “The surveyors may ask additional questions to remind the children and see if the answers seem believable, for example:
- Show children the dose poll and ask if the children remember it
- Ask the children what they remember about the pills (it is normal to remember praziquantel's bitter taste, size and smell)
- Ask the children how many pills they took (while trying to judge how many they should have received by getting them to stand near the dose pole)."
GiveWell's non-verbatim summary of a conversation with Sarah Nogaro, October 16, 2014, Pgs. 5-6.
- For Malawi (2014), SCI told us that verification methods were used, but the data was not included in the survey report. Fiona Fleming, email to GiveWell, November 5, 2015.
- For Uganda, Zanzibar, and Zambia, SCI told us that the survey "asked if [respondents] recognised pills / dummy pills and dose poles." Fiona Fleming, email to GiveWell, November 5, 2015.
- "Verification methods, including the demonstration of pills and dose poles, were used to minimize recall bias." SCI Ethiopia coverage survey 2015, Pg. 4.
- For Madagascar (2016), SCI told us that similar verification methods to those in Ethiopia were used. Fiona Fleming, conversation with GiveWell, September 19, 2016.
- For Côte d'Ivoire (2016 and 2018), Ethiopia (2016 and 2017), Liberia (2017), Madagascar (2017), Mauritania (2017), Malawi (2017) and Niger (2017), it appears that individuals were asked whether they recognized the pills and dose pole. See SCI Côte d'Ivoire coverage survey dashboard 2016, SCI Ethiopia coverage survey dashboard 2016, SCI Liberia coverage survey dashboard 2017, SCI Madagascar coverage survey dashboard 2017, SCI Niger coverage survey dashboard 2017, SCI Côte d'Ivoire coverage survey recommendations report 2018, SCI Ethiopia coverage survey recommendations report 2017, SCI Mauritania coverage survey recommendations report 2017, and SCI Malawi coverage survey dashboard 2017.
- For Burundi (2017) and Nigeria (2017), we also believe that children were asked whether they recognized the pills and dose pole because these questions appear in the Child Questionnaire (see question 25 on p.28 of SCI Burundi coverage survey protocol 2017, and question 42 on p.22 of SCI Nigeria coverage survey protocol 2017 (with Sightsavers)). However, data was not included in the survey reports that we have seen in either case.
- Côte d'Ivoire (2014): “The surveyors may ask additional questions to remind the children and see if the answers seem believable, for example:
- 29
"The actual sample included caretakers of 578 children." SCI Mozambique coverage survey 2015, Pg. 5. Note that we have only received permission to publish a summary of this report. Quotation is from the full, unpublished report.
- 30
SCI Mozambique coverage survey 2015, Pg. 23. Note that we have only received permission to publish a summary of this report. Quotation is from the full, unpublished report.
- 31
SCI Mozambique coverage survey 2015, Pg. 24. Note that we have only received permission to publish a summary of this report. Quotation is from the full, unpublished report.
- 32
"Pole doses were used as a proxy for the medication – rather than showing the medication we showed them the poles. Only parents (not teachers) can authorize medication to kids in Mozambique and they need to be involved and informed during the social mobilization which is conducted usually by the district officers and the activists, in the week preceding the distribution of the medication. The activists use posters and poles when they inform the community and the poles are very recognizable. The poles are also used in the integrated campaigns – and in most of these districts campaigns have been conducted for a number of years. So…they may be recognized as part of a preventive medication campaign rather than related to the PZQ only." Fiona Fleming, conversation with GiveWell, November 5, 2015 quoting from an email from FPSU.
- 33
The interview protocol (provided in Portuguese) does not include any verification questions. See SCI Mozambique coverage survey 2016, Pgs 28-29.
- 34
See the top left figure on p.7 of SCI Mauritania coverage survey recommendations report 2017 for the recognition rates, and this spreadsheet for the survey coverage rates.
- 35
See the top right figure on p.6 of SCI Ethiopia coverage survey recommendations report 2017 for the recognition rates, and this spreadsheet for the survey coverage rates.
- 36
- Fiona Fleming, conversation with GiveWell, September 19, 2016
- Email from SCI's partner FPSU, quoted to us by SCI:
"Further to our conversation, here is a recount of the supervision of the survey:- Initially there were 6 teams with Cabo Delgado and Nampula divided into two. All of them were supervised by Don and myself during 2 days of intense data collection in Nampula Cidade to ensure the areas selected were visited, the random selection of houses and respondent and the correct administration of the questionnaire plus the team organization and supervision.
- Teams were then reduced to 4 as it made more sense to complete districts before traveling to the next.
- Each team had one or two supervisors, one being the provincial NTD or M&E staff in the case of Zambezia, the national supervisor being NTD or other MISAU Departments (Cabo Delgado, Zambezia). The district officers worked in teams so that most of the interviews in one district were done by the other district officers with no connection with the district. In fact in most cases the district officer of that particular district was solving logistical and administrative issues which are quite intense in Mozambique (permission by the provincial delegate, permission by the district and the locality chief letters of introduction all signed and stamped, motorcycles rental and bills etc).
- Nampula was identified as the most complex area and the team was supervised by me during the completion of Ilha and Mossuril districts plus Cidade Nampula while Don supervised District Nampula and again the finalization of Cidade Nampula which was pretty complicated. In other areas the survey was easier but in Chinde Islands where Dr Xose went from one island to the next to find the selected village and got lost for a few days because the district has been divided into two and we had to select again the villages. All other supervisors provided daily reports to Don. They also wrote a report of the questionnaires received every day, reviewed and sent.
- In most cases, data was sent to the database and reviewed every day. Once the supervisor had sent the questionnaire editing was only allowed to Don Whitson. Some areas did paper based questionnaires and this was again submitted and reviewed."
Fiona Fleming, email to GiveWell, November 5, 2015.
- 37
See notes in the "data quality control" column of this spreadsheet, "Methods" sheet.
- 38
Citations for all statements in this list can be found in this spreadsheet, "Methods" sheet.
- 39
For the Zanzibar survey, SCI told us, "Shehias were stratified by MoH treatment programme and by elimination programme; required sample size selected from each startum." Fiona Fleming, email to GiveWell, November 5, 2015. We do not know how to interpret this.
- 40
- In Ethiopia (2016 and 2017): "Criteria for purposive selection of the Districts include:
- Treatment was given for both SCH and STH
- Safety of data collectors while in the district
- Districts that have been treated within the last 3 months
- Too high or low reported coverage (either from the current round or historical)",
- In Malawi (2017): "Districts were purposively selected for this survey, four of which were surveyed in 2016, based on safety and whether they were surveyed in previously." "...in the Southern districts of Malawi, health workers, local leaders and medical services were attacked in villages due to superstition surrounding witchcraft. It was decided for the security of survey teams that the Southern districts would not be visited", SCI Malawi coverage survey protocol 2017, p.12.
- In Ethiopia (2016 and 2017): "Criteria for purposive selection of the Districts include:
- 41
See this spreadsheet, “Methods” sheet, “How villages chosen” column for details.
- 42
Fiona Fleming, email to GiveWell, November 5, 2015 (quoting from email from FPSU, which runs the program in Mozambique).
- 43
- See this spreadsheet, “Methods” sheet, “Who conducted the survey” column for details.
- For Malawi (2012), Uganda (2014), Zanzibar (2015) and Zambia (2015), see Fiona Fleming, email to GiveWell, November 5, 2015.
- 44
- For Côte d'Ivoire, coverage reported by parents was lower than coverage reported by children. In Malawi, coverage reported by parents was similar to coverage reported by children. In Côte d'Ivoire, parents' answers were excluded from the reported results: "Calculation of validated coverage rates initially included answers from both proxy and direct interviews across all 4 districts. As shown in Figure 2, coverage rates calculated based on direct interviews were higher than those which included responses given by proxy (p < 0.001), perhaps due to the parent erring on the side of caution when giving their answer. As direct interviews are believed to be more robust, we omitted data from proxy interviews when calculating final coverage rates (Table 4)." SCI Côte d'Ivoire coverage survey 2014, Pg. 10.
- In Malawi, parents' answers were included in reported results. Proxy results were similar to results from direct interviewing in Malawi, and were included in the headline analysis. SCI Malawi coverage survey 2012, Pg. 11, Figure 4.
- About 9% of the responses in Côte d'Ivoire were from parents. SCI Côte d'Ivoire coverage survey 2014, Pg. 9, Table 3.
- Number of yes/no answers obtained in person (summed across four districts): 2178.
- Number of proxy yes/no answers obtained (summed across four districts): 228.
- 228 / (228 + 2178) = ~.09.
- 45
- Fiona Fleming, email to GiveWell, November 5, 2015.
- Fiona Fleming, conversation with GiveWell, September 19, 2016.
- 46
"Some of the data for the Côte d'Ivoire coverage survey conducted in 2016 may not
be reliable for the following reasons.
Possibility of other NTD MDAs
SCI found that interviewees were more likely to report receiving ALB than PZQ, even though the two drugs were both meant to be administered together in the latest MDA. It is possible that a treatment for lymphatic filariasis (LF) may have been administered between the time of the PZQ and ALB treatment and the time of the survey. This would affect the interviewees' answers because ALB is also used to treat LF. It is also possible that the amount of drugs available was insufficient, so people received only ALB and not PZQ. SCI plans to investigate this by asking further questions of the national program and of the district officials to find out whether other NTD MDAs may have happened during that time period. This will help SCI know how reliable their data are. SCI believes that the data from one of the three districts surveyed, Prikro, does not suffer from this issue.
District-specific problems
Bangolo
SCI was told before the survey that Bangolo was one of the treated districts, and conducted its survey there based on this information. However, SCI later found out that Bangolo had not been a part of the latest MDA. This leads SCI to believe that interviewees in this district may have been responding about some treatment other than the PZQ and ALB.
Aboisso
In Aboisso, SCI had difficulty finding people to interview, since many people there live in camps such as internally displaced people's camps. Since the district lacks a regular community structure, there were fewer people available to interview than usual. This resulted in very large confidence intervals for the district."
Conversation with Dr. Fiona Fleming, August 1, 2017. - 47
Tohon et al. 2008. In 2014, SCI sent us a more recent report, SCI Niger panel study 2011. SCI Niger panel study 2011 is our only source of data on the second to fifth year followups. This analysis excludes three of the eight schools that were originally in the study as well as the 56% of participants lost to follow-up in the remaining five schools. Participants who were followed up for some years but lost to follow-up by the end of the study are completely excluded rather than being included in a separate analysis of the earlier follow-ups. The 3 schools were apparently dropped because they were not surveyed in some years:
- "Of these eight sentinel sites, five had data available in all six years of the study up to 2010." SCI Niger panel study 2011, Pg. 19.
- "455 out of 1024 (44%) children recruited at baseline from November 2004 to April 2005 were successfully followed up for the full duration of the 6-year sentinel site monitoring." SCI Niger panel study 2011, Pg. 25.
For this reason, we feel that the results from SCI Niger panel study 2011 are hard to interpret, and we don't present them here, relying instead on Tohon et al. 2008.
- 48
- We have seen two reports on the Burundi (2007-2010) study, Styles 2011 and Koukounari 2011, which use different methods of analysis and included different numbers of participants. Which individuals they included in their analyses is not always clear, however the fact they report similar results for prevalence provides some evidence that the results are not highly dependent on these choices. (Both Dr. Koukounari and Dr. Styles are statisticians who formerly worked at SCI.)
- Pilot study: Styles 2011 and Koukounari 2011 have strengths and weaknesses in defining the sample for the pilot study. Styles 2011 includes a much larger number of individuals (Styles 2011, Pg. 3, Figure 1b compared to 710 in Koukounari 2011, Pg. 7, Figure 2). However, it is unclear whether the individuals in the follow-up were all in the original cohort or whether some were added to the sample later. The two reports indicate different numbers of students included in the study and retraced at each followup. We have seen two explanations for the differences:
(1) "Dr. Koukounari only included students who were in first grade during the first year of the study and who were successfully surveyed every year of the study. In addition to the children counted by Dr. Koukounari, Dr. Styles included students who entered first grade and were added into the study in subsequent years, as well as students who were missing data from some years. Each of these strategies for data analysis has benefits and drawbacks. SCI initially planned to do a cross-sectional evaluation of sixth grade students every year, because each year the current sixth grade class would have received more rounds of treatment over the course of elementary school than the previous year. SCI did not complete this plan, but Dr. Koukounari included the data from the sixth grade students in the baseline data. Dr. Styles did not include this data." GiveWell's non-verbatim summary of a conversation with Giuseppina Ortu on June 20, 2014.
(2) "In addition to the longitudinal studies at each follow-up newly recruited children were added to these surveys. At 1st follow-up (2008) 2288 newly recruited children were added to these surveys with range age: 6-21 years old and median age: 12 years old. Of these 2288, only 210 i.e. (9.18 %), were of age 6 and eligible to be included in the specific cross sectional data analysis. At 2nd follow-up (2009) 2311 newly recruited children were added to these surveys with range age: 5-20 years old and median age: 11 years old. Of these 2311, only 160 i.e. (6.92 %), were of age 6 and eligible to be included in this specific data analysis. Finally, at 3rd follow-up (2010) 2224 newly recruited children were added to these surveys with range age: 6-20 years old and median age: 12 years old. Of these 2224 only 189 i.e. (8.50 %) were of age 6 and eligible to be included in this specific data analysis." Koukounari 2011, Pg. 6. Koukounari 2011 notes that it only includes the 20% of participants in the pilot study who were tracked through all follow-ups: "Finally at 3rd follow-up (2010) there were 713 children successfully followed-up (i.e. follow-up rate=19.71%). Longitudinal analyses for the 4 years are presented in the next pages for these 713 children." Koukounari 2011, Pg. 6. - Other schools: Koukounari 2011 claims that 5,700 participants were recruited at baseline: "At baseline (2008) there were recruited 5700 children while the follow-up rate one year later was 53.42 % (3045/5700)." Koukounari 2011, Pg. 16. Styles 2011 Pg. 13, Table 8 claims there were 3,781 participants recruited at baseline. SCI later told us that this may be because Koukounari 2011 included data from a group of students who were surveyed for a concurrent cross-sectional study, which was not completed: "SCI initially planned to do a cross-sectional evaluation of sixth grade students every year, because each year the current sixth grade class would have received more rounds of treatment over the course of elementary school than the previous year. SCI did not complete this plan, but Dr. Koukounari included the data from the sixth grade students in the baseline data. Dr. Styles did not include this data." GiveWell's non-verbatim summary of a conversation with Giuseppina Ortu on June 20, 2014. Both studies include a similar number of participants at the first follow up: "At baseline (2008) there were recruited 5700 children while the follow-up rate one year later was 53.42 % (3045/5700)." Koukounari 2011, Pg. 16; Styles 2011 Pg. 13, Table 8 shows about 3030 students were included in the first year follow-up analysis.
- Pilot study: Styles 2011 and Koukounari 2011 have strengths and weaknesses in defining the sample for the pilot study. Styles 2011 includes a much larger number of individuals (Styles 2011, Pg. 3, Figure 1b compared to 710 in Koukounari 2011, Pg. 7, Figure 2). However, it is unclear whether the individuals in the follow-up were all in the original cohort or whether some were added to the sample later. The two reports indicate different numbers of students included in the study and retraced at each followup. We have seen two explanations for the differences:
- Details on the 2017 survey, evaluating the period 2007/8-2017, can be found in this spreadsheet, "Methods" sheet.
- We have seen two reports on the Burundi (2007-2010) study, Styles 2011 and Koukounari 2011, which use different methods of analysis and included different numbers of participants. Which individuals they included in their analyses is not always clear, however the fact they report similar results for prevalence provides some evidence that the results are not highly dependent on these choices. (Both Dr. Koukounari and Dr. Styles are statisticians who formerly worked at SCI.)
- 49
- We have seen reports on a baseline impact survey from 2012 (SCI Liberia panel study baseline report) and one follow-up survey from 2013 (SCI Liberia impact survey follow up recommendations report 2013) and (SCI Liberia impact survey dashboard 2012-13).
- Implementation of the follow-up survey as described in SCI Liberia impact survey follow up recommendations report 2013:
- "The survey aimed to revisit all of the schools that were visited in 2012 for the baseline survey to determine changes in prevalence and intensity over time. The schools were randomly selected at baseline and the sampled children within each school were also randomly selected at baseline, with stratification by gender and grade." Pg 3.
- "There were some difficulties in matching pupils to schools, and also matching schools between years. This was because the school identity numbers became muddled making matching not possible. This was fixed by going back to the paper records to determine the data associated with each school." Pg 4.
- "Eleven of the baseline schools (with code 1, 7, 9, 16, 17, 18, 20, 23, 24, 25, 28) could not be re-visited in 2013 and were substituted by schools as close as possible. Three of these new schools (code 39, 41, 48) were deemed to be too far away from the corresponding baseline schools to serve as their follow up and were added to the analysis as independent new schools." Pg 4.
- "The original protocol was originally designed as a ‘cohort’ study where the same children are surveyed repeatedly across a number of years. Issues with identification numbers meant that children could not be matched between the years and consequently we analysed the data as a cross-sectional study. Any future surveys of these schools in Liberia will be cross-sectional and SCI has moved away from cohort studies across all its programmes. In addition, as time that has passed since the original baseline study in 2012 many of the original children surveyed will no longer be at school." Pg 4.
- 50
- Details on the methods for the surveys in Malawi, Madagascar, Tanzania, Côte d'Ivoire and Ethiopia can be found in the "Methods" sheet of this spreadsheet.
- Note that we have excluded studies from Uganda (Kabatereine et al. 2007) and Burkina Faso (Touré et al. 2008 and Koukounari et al. 2007) because participants in these studies received separate, more intensive treatment than other children in those countries (discussed in blog posts in 2013 and 2014). Therefore, we believe that the results from Uganda and Burkina Faso do not reflect the quality of the national programs which were supported by SCI.
- It is our understanding that, in the studies that we have included, study participants received treatment in the same manner as other children in the country, and thus that those studies reflect the performance of the national MDAs:
GW: "In our current review, when discussing prevalence and intensity sentinel site surveys, we write, 'It is our understanding that, in the Niger, Burundi, and Malawi studies, study participants received treatment in the same manner as other children in the country, and thus that those studies reflect the performance of the national MDAs.' Would it be accurate to include the Liberia study in the statement above?
Fiona Fleming: "Yes it would, participants in all these surveys receive treatment as part of the national programme and not at the time of the survey. Participants only received treatment in surveys in Uganda in the first 2 years back in 2003 and no surveys have treated participants since that time."
Fiona Fleming, email to GiveWell, October 11, 2016. This is also noted in several specific studies; see the "Methods" sheet of this spreadsheet.
- 51
- Niger: “Praziquantel (using dose-pole corresponding to 40 mg/kg) and Albendazole (400 mg) were given to the target population regardless of infection status, during the mass drug administration campaign that took place 3–4 weeks after the surveys were conducted.” Tohon et al. 2008, Pg. 3. “A total of 89% of the initial sample group were re-examined one year after baseline data collection and the first round of treatment with praziquantel and albendazole.” Tohon et al. 2008, Pg. 4.
- Burundi pilot: “At baseline (2007) there were recruited 3616 children. At 2008 the 1st follow-up took place where 1188 children were retraced since baseline (i.e. follow-up rate=32.85 %). At 2nd follow-up (2009) there were 1004 children successfully followed up since baseline (i.e. follow-up rate=27.77%). Finally at 3rd follow-up (2010) there were 713 children successfully followed-up (i.e. follow-up rate=19.71%). Longitudinal analyses for the 4 years are presented in the next pages for these 713 children.” Koukounari 2011, Pg. 6.
- Burundi other schools: “At baseline (2008) there were recruited 5700 children while the follow-up rate one year later was 53.42% (3045/5700).” Koukounari 2011, Pg. 15.
- Malawi: “A longitudinal survey design requires baseline data collection from schools prior to the initiation of large-scale distribution of praziquantel and albendazole or mebendazole through the school-based platform. Follow up surveys will be conducted immediately prior to subsequent rounds of treatment for the life of the programme to monitor the impact of the health intervention.” SCI Malawi panel study, Pg. 3. “During the baseline survey, cohorts of 125 children from Standards 1, 2 and 3 (aged approximately 6, 7 and 8 years) were randomly selected in each of the schools and enrolled into the study. [...] This group of selected children, now in standards 2, 3, and 4, as well as a new group of 40 Standard 1 children, were re-tested to measure the same indicators during the 1st follow-up.” SCI Malawi panel study, Pg. 4.
- Liberia: "The original protocol was originally designed as a ‘cohort’ study where the same children are surveyed repeatedly across a number of years. Issues with identification numbers meant that children could not be matched between the years and consequently we analysed the data as a cross-sectional study. Any future surveys of these schools in Liberia will be cross-sectional and SCI has moved away from cohort studies across all its programmes." SCI Liberia impact survey follow up recommendations report 2013, Pg 4.
- For all four studies, the methodology does not discuss a control group, and with context, it is sufficiently clear that there was not one.
- 52
- "This survey year saw a switch from a longitudinal survey design to a cross-sectional design which occurred following internal SCI reviews of the data and issues arriving from field surveys. The change in survey design led to the ages of children included in the study to be altered slightly to allow for like for like comparison over time and to capture those with the highest burden of infection." SCI Malawi impact study – second follow up, Pg. 3.
- "The original protocol was originally designed as a ‘cohort’ study where the same children are surveyed repeatedly across a number of years. Issues with identification numbers meant that children could not be matched between the years and consequently we analysed the data as a cross-sectional study. Any future surveys of these schools in Liberia will be cross-sectional and SCI has moved away from cohort studies across all its programmes." SCI Liberia impact survey follow up recommendations report 2013, Pg 4.
- 53
See column O in the "Methods" sheet of this spreadsheet.
- 54
SCI notes that because children in a control group would be tested for infection, ethical guidelines would require that those found to be infected receive treatment and thus would no longer serve as controls.
- 55
- See the "Results" sheet in this spreadsheet.
- SCI report that the increase in prevalence of S. mansoni in Côte d'Ivoire and Tanzania and the increase in prevalence of S. haematobium in Côte d'Ivoire is not statistically significant, though we have not verified these statistical tests. The statistical significance of the increase in intensity of S. mansoni in Tanzania was not reported.
- 56
"For S. mansoni both prevalence and intensity increased between FU2 and FU3. All differences are significant", SCI Burundi impact survey recommendations report 2017, p.6. See the notes in the cells for the Burundi 2017 study in this spreadsheet.
- 57
See the "Results" sheet in this spreadsheet.
- 58
See the "Results" sheet in this spreadsheet.
- 59
- See SCI Côte d'Ivoire impact survey recommendations report 2016 (Phase 1 schools follow-up and Phase 2 schools baseline).
- "There was an insignificant increase in overall prevalence of S. mansoni infections. However, there were big differences between schools. The biggest increase in S. mansoni prevalence was registered for schools 2 (+8.9%), 9 (+9.3%), and 20 (+11.5%)", p.2.
- "The overall prevalence in S. haematobium increased insignificantly although there were big differences between schools. The biggest increase was registered at the schools 3 (+13.1%), and 11 (+11.3%)", p.2.
- 60
"The prevalence of S. haematobium infections and heavy intensity infection decreased overall. However, there is significant heterogeneity between schools. In many schools prevalence is still high following two rounds of MDA", SCI Madagascar impact survey recommendations report 2017, p.3.
- 61
"Increase in prevalence of S. mansoni (42 out of 146 schools) and heavy intensity infection (5 out of 146 schools) despite overall reduction in both", SCI Ethiopia impact survey recommendations report 2016-17, p.3.
- 62
- Niger:
- "Eight villages located in schistosomiasis endemic regions were randomly selected to represent the two main transmission patterns in Niger: six villages located near permanent (Tabalak, Kokorou) or semi-permanent (Kaou, Mozague, Rouafi, and Sabon Birni) ponds and two (Saga Fondo, Sanguile) located along the Niger River. The villages represented the south-western region (Tillabe´ry) and the central-northern region (Tahoua) of the country, with four villages from each region. One village is located in the Sudanian climatic zone and the seven others are in the Sahelian climatic zone." Tohon et al. 2008, Pg. 2.
- SCI told us that these locations "are not representative of the treatment population as a whole. They were selected to indicate the impact of treatment in schools with varying prevalence and intensity of both [types of schistosomiasis]." Anna Phillips, SCI Country Program Manager for Burkina Faso and Niger, email to GiveWell, October 13, 2011.
- Burundi pilot survey: “More precisely, the 12 schools were chosen based on 3 zones-believed at the time that they would have the majority of NTDs. 4 schools were selected randomly so that they represent the ‘STHs +Schisto +oncho’ zone (these were Musenyi, Nyamibu, Munyika, Rukinga); then another 4 schools were selected randomly so that they represent the ‘STHs +oncho’ zone (Mirombero, Kizuga, Ruzibira, Mudende) and finally 4 schools were selected randomly so that they represent STHs only endemic areas (Gatwe, Ruko, Condi, Gitobo). Such decisions were based on available historic data. Thus, SCI Programme Manager advised not to stratify the statistical analysis by province and so such results (i.e. stratification by province) are not presented anywhere in this report.” Koukounari 2011 Pg. 6.
- Burundi other schools: it appears that schools were selected to be representative, though this is not fully clear in the reports we have seen.
- Styles 2011 says, "The selection of schools was done randomly from the non-pilot provinces; taking into account 11 separate ecological zones." Pg. 13.
- Koukounari 2011 says, “At baseline (2008) there were recruited 5700 children while the follow-up rate one year later was 53.42% (3045/5700). For these set of studies as they were designed to cover almost all of the country, it is worthwhile to also examine stratifications of analyses by district and such results are also presented in the following subsections. However, in most of the districts the children were coming only from 1 school (see relevant graphs for district whenever n<200; when this is the case then this is only 1 school per district and thus results should be treated there with caution and programmatic decisions to be taken with reservations). Whenever/wherever this is the case, results should be interpreted with caution as just 1 school would be quite ‘risky’ to represent inference/decisions for a whole district.” Pg. 16.
- Malawi: The schools seem to have been selected in a way that makes them representative of districts with moderate to high prevalence, which are those districts that receive annual treatment (low prevalence districts receive more limited treatment). The schools were selected only from districts found to have moderate schistosomiasis prevalence in SCI's mapping: "Method of sentinel site selection: SCI’s protocol is to monitor only in those districts where prevalence of schistosomiasis is moderate or high i.e. SCI does not monitor in non-endemic or low prevalence districts where a full control program is not implemented. All districts except Mzuzu City surveyed in the mapping in February 2012 were determined to have moderate prevalence of S. haematobium, and consequently all districts except Mzuzu City were included in the selection of sites to be monitored for this species (see Table 1). S. mansoni infection was more focal and only present at moderate prevalence in Chiradzulu, Blantrye Rural, Lilongwe City and Lilongwe Rural East. Following district stratification by S. mansoni infection, such that the number of schools selected for S. mansoni monitoring, reflected the frequency of moderate risk areas in the monitoring areas, 22 schools were selected that would be monitor S. haematobium infection with a subset of 9 schools which also monitor S. mansoni infection. Due to the low prevalence of STH’s, STH infection was only monitored in those schools where the Kato-Katz slides were already prepared for S. mansoni." SCI Malawi panel study, Pg. 4; SCI's senior biostatistician told us that the sampling method would produce a sample representative of the treated districts. Michelle Clements, SCI Senior Biostatistician, conversation with GiveWell, October 15, 2014.
- Details on the sampling procedure for all other studies are given in the "Methods" sheet in this spreadsheet.
- Niger:
- 63
- "Baseline Impact M&E surveys took place in 38 schools in three provinces of Liberia (Bong, Nimba, Lofa) in Nov-Dec 2012, as part of the ICOSA programme." SCI Liberia panel study baseline report, Pg 3.
- SCI Liberia impact survey follow up recommendations report 2013:
- "Eleven of the baseline schools (with code 1, 7, 9, 16, 17, 18, 20, 23, 24, 25, 28) could not be re-visited in 2013 and were substituted by schools as close as possible. Three of these new schools (code 39, 41, 48) were deemed to be too far away from the corresponding baseline schools to serve as their follow up and were added to the analysis as independent new schools." Pg 4.
- Survey recommendations table, Pg 6:
- "School identity numbers were not always correct."
- "Many of the GPS coordinates were not recorded correctly."
- "Schools closed between years of data collection."
- 64
See the "Methods" sheet, cell Q10 in this spreadsheet.
- 65
- Niger: "A total of 89% of the initial sample group were re-examined one year after baseline data collection and the first round of treatment with praziquantel and albendazole." Tohon et al. 2008, Pg. 4.
- Burundi: Pilot schools: 33%: “Without taking into consideration the parasitological exams, at baseline (2007) there were recruited 3616 children. At 2008 the 1st follow-up took place where 1188 children were retraced since baseline (i.e. follow-up rate=32.85 %).” Koukounari 2011, Pg. 16. 50%: Styles 2011 Pg. 1, Table 1. Other schools: 53%: “At baseline (2008) there were recruited 5700 children while the follow-up rate one year later was 53.42 % (3045/5700).” Koukounari 2011, Pg. 15. 80%: Styles 2011 Pg. 13, Table 8. The discrepancy between the populations included in Koukounari 2011 and Styles 2011 is described in more detail below.
- Malawi: "Overall, the drop-out rate was higher than expected for both species of schistosomiasis. 48% of those pupils monitored for S. haematobium dropped out the study between baseline and follow-up, and 64% of those pupils monitored for S. mansoni dropped out of the study." SCI Malawi panel study, Pg. 17. Since the most meaningful results from Malawi were for Schistosoma haemotobium, we focus on the follow-up rate for that species.
- Liberia: Although the Liberia study was originally designed as a cohort study, the survey implementers switched to a cross-sectional methodology at the first follow-up: "The original protocol was originally designed as a ‘cohort’ study where the same children are surveyed repeatedly across a number of years. Issues with identification numbers meant that children could not be matched between the years and consequently we analysed the data as a cross-sectional study." SCI Liberia impact survey follow up recommendations report 2013.
- 66
Benjamin Styles, SCI Senior Biostatistician, phone conversation with GiveWell, August 12, 2011.
- 67
"Prevalence with CCA is always higher (or equal if 0) than with KK. Prevalence using CCA with trace positive is between 3 and nearly 50 times larger than with KK." SCI Burundi impact survey recommendations report 2017, p.7.
- 68
For further details, see the "Methods" sheet, cell S12 in this spreadsheet.
- 69
- 70
LSTM Mozambique trip report (May 2015)
- "With authorization of Dra Olga, I am sharing with you the EXCEL file with results (not the databases), and she reminds me that the data belongs to MISAU." Pg. 2.
- "The Team reviewed the results for 2012-2013-2014 and they confirmed that only 2 sentinel sites were strictly comparable for 2013-2014 (Mecula and Mandiba) and none of the Sentinel Sites for 2015 are going to be comparable to 2014. They decided to review the location of the new sentinel sites, to be able to have a better and stronger comparisons." Pg. 2.
- 71
"In Malawi, urine volumes were not accurately recorded thus it is possible that data is indicating lower overall prevalence in sentinel sites. ICOSA will be undertaking further data analysis to quantify underestimates using mapping data from 2012 and baseline data in appropriate districts." SCI report to DFID (October 2013), Pg. 15.
- 72
"The NTD programme in Zanzibar has recently completed the 3rd round of MDA." SCI report to DFID (October 2013), Pg. 17.
- 73
- Our intervention report discusses this briefly.
- Other conversations and observations have reinforced our impression that administering deworming drugs is fairly straightforward.
- The WHO factsheet on STH: "The WHO recommended medicines – albendazole (400 mg) and mebendazole (500 mg) – are effective, inexpensive and easy to administer by non-medical personnel (e.g. teachers)." WHO STH factsheet.
- 74
- "In Tanzania matters came to a head in places around Morogoro in 2008. Distribution in schools of tablets for schistosomiasis and soil-transmitted helminths provoked riots, which had to be contained by armed police. It became a significant national incident, and one of the consequences has been the delay in Tanzania adopting a fully integrated NTD programme, and the scaling back some existing drug distributions." Allen and Parker 2011, Pg. 109.
- "From these reports a number of problems with the MDA were raised which included fear of side effects from the tablets, particularly following the mass hysteria and death in Blantyre and Rumphi respectively and may explain some of the geographic heterogeneity seen. Furthermore most districts reported that MDA occurred after standard 8 students had finished exams and left school, and due to having inadequate resources for drug distribution...The side-effects incident in Blantyre and death in Rumphi had a large effect on districts and with many district reports stating that after the incidence many families refused to participate." SCI Malawi coverage survey 2012 Pgs 5, 21.
- 75
Fiona Fleming, conversation with GiveWell, November 5, 2015.
- 76
"There was confusion on Wednesday in some public primary and secondary schools in Ogun State, over the administration of anti-worm tablets. Nigerian Tribune gathered that some students reportedly collapsed in the cause of administering the tablets on them. This resulted into rumour that spread like wildfire across the length and breadth of the state, as parents stormed various school to withdraw their wards. When the Nigerian Tribune visited Egba High School, Asero and Asero High School both in Abeokuta South Local Government Area of the state, some parents were sighted at the school gate, who had come to confirm the incident and probably withdraw their wards. There was calmness in both schools as students in the Senior Secondary Classes were said to be preparing for their examinations. Meanwhile, the Ogun State Government through the State Commissioner for Health, Dr Babatunde Ipaye, has denied any case as a result of the anti-worm drug. Ipaye in a statement made available to the Nigerian Tribune in Abeokuta, said that no pupil or student to the best of his knowledge had reacted to the drug in the state. He explained that the exercise was done by his Ministry in collaboration with Evidence Action." Nigerian Tribune, "Panic in Ogun schools over deworm exercise," December 2017
- 77
- In October 2018, SCI described its calculations:
- "The financial reports on in-country expenditure and reported treatment data allow SCI to perform cost analyses on the cost per treatment by year and by country. The £0.18 is derived from average of the country cost per treatment for a school-age child over several financial years. The £0.27 for community-based treatments is based on the same overall dataset, but with the cost data disaggregated by community-based delivery.
"The £0.27 per child in the first year of a new country program is also derived from cost analyses on the same financial data when looking at the first year of MDA across multiple countries. The cost is higher due to baseline prevalence mapping in the country to determine treatment strategy, as well as the purchase of capital goods and because countries tend to be less efficient in the first year of implementation. Subsequently efficiency increases." Wendy Harrison, email to GiveWell, October 30, 2018
- 78
- For cost per school-aged child see, for example, cell D12 on sheet "Ideal Scenario 2018-2021."
- For cost in community-based distributions, see, for example, cell D10 on sheet "Ideal Scenario 2018-2021."
- For cost to purchase drugs for adults, see, for example, cell E10 on sheet "Ideal Scenario 2018-2021" showing that 3 tablets are needed per adult and cell F10 showing a cost per tablet of £0.08.
- For cost the first year of a new country program, see, for example, cell D33 on sheet "Expansion_Scenario 2018-2021."
- 79
This understanding is from undocumented conversations with SCI from early in our investigation of SCI.
- 80
- "Ordering of ALB is carried out by the Ministry Of Health Programme Manager for Lymphatic Filariasis (LF) Elimination. Previously the ALB had been used for the LF program and stocks had not been replenished in time for the SCH campaign which contributed to the low ALB coverage. Treatments were also carried out in conjunction with the child health days which may have caused confusion with who was eligible for treatment and prioritising the younger children. Furthermore due to the complexities of distributing PZQ compared to ALB there may have been more focus during the training and distribution on reporting and dispensing PZQ." SCI Malawi coverage survey 2012, Pgs 20-21.
- "Miscommunication between the national Program Managers for lymphatic filariasis and schistosomiasis led many districts to believe that the ALB which they received should have been made available for the MDA but was not used." SCI Malawi coverage survey 2014, Pg. 37.
- "Unlike in Sub-Saharan Africa, where SCI distributes praziquantel for schistosomiasis and albendazole for STH in equal amounts, Sudan has many areas where schistosomiasis is a problem but STH are not, and SCI distributes more praziquantel than albendazole. However, a small number of areas including Kassala and eastern Sennar have STH and no schistosomiasis. In these areas, the Ministry distributes more albendazole than praziquantel." GiveWell's non-verbatim summary of a conversation with Alan Fenwick and Najwa Al Abdallah, September 14, 2015.
- "Albendazole (ALB) was also distributed in some of the 30 districts under different partners, however ICOSA programme did not support its procurement, distribution or data collection in 2013/2014." SCI Uganda coverage survey 2014, Pg. 4.
- Schistosomiasis Control Initiative, conversation with GiveWell, September 6, 2016:
- In Ethiopia, the SCI-supported program has delivered STH-only treatments in some districts where schistosomiasis prevalence was low.
- SCI told us that its reported treatment numbers were total numbers of schistosomiasis treatments.
- 81
We explain why we take this approach in this blog post.
- 82
Alan Fenwick, SCI Director, email to GiveWell, November 24, 2014.
- 83
See this spreadsheet for details. Note that this is not an ideal comparison because reported coverage rates are calculated from reported treatment numbers and an assumed target population. We would like to adjust the reported treatment numbers, rather than the reported coverage rate, but don't have the data to do so. It is possible that the difference in the reported coverage rate and the coverage rate from the coverage surveys is due to errors in the numbers used for the target population rather than the reported treatment numbers.
- 84
We used Leslie et al. 2011, a study of the costs of a SCI-funded deworming program in four districts of Niger in 2004-2006, to estimate non-SCI contributions to SCI's deworming programs. Three of the authors of the study were affiliated with SCI. The study aimed to account for all costs of the program, including costs funded by the government and non-financial costs such as the value of volunteers' time:
“This was a retrospective study which covered a two year period from April 2004 to May 2006, including the first and second years of MDA and related programme activities in four health districts. All data on first year costs at national, regional, district, and sub district levels were taken from the PNLBG accounts and receipts and records of staff missions or activities. Second year cost data for national and regional level activities were taken from receipts. District and sub district, school and community MDA resource use data for 2005 were collected in June 2006 through a retrospective survey…
The main cost elements include: the programme specific expenditure; the opportunity cost or value of government contributions related to in-kind costs of using local government staff and vehicles and the value of CDD’s time (taken as the daily agricultural labour rate); and the international costs of programme co-ordination, reporting and technical support." Leslie et al. 2011, Pgs. 2-3.The study is of a single country, looked at a program that was carried out years ago, and the program may differ in some ways from current programs, but overall it is of high quality and provides us with a sense for the portion of resources contributed by SCI versus non-SCI parties.
Two examples of how the area where the study was conducted may not be representative of all areas in which SCI works:
- Due to low school enrollment rates, a substantial portion of the program was through community distribution. Current SCI programs focus on school-based distribution. "The primary school net enrolment rate (NER) in 2004 in Niger was 41%... To achieve high treatment coverage in targeted school age children and at risk adults two treatment strategies, school-based and community-based distribution, were established." Leslie et al. 2011, Pg. 2.
- "The cost per treatment and prevalence figures relate to the study sample of four districts located in the Niger River Valley. This was and is an area of high disease prevalence and high population density relative to other parts of the country. The costs per person treated may be higher in lower density and more remote areas." Leslie et al. 2011, Pg. 8.
Non-SCI costs were 18% of the total cost of the program and 33% of the cost of school-based deworming (the program also included community-based deworming).
- ”Programme cost: 75%
- Government cost: 18%
- International tech. support: 7%”
Leslie et al. 2011, Pg. 5, Table 2.
It is our understanding from the paper and our past conversations with SCI that "programme expenditure" was fully funded by SCI. We believe that "international tech. support" refers to SCI staff time and travel costs; we're somewhat less confident in this than in our understanding of “programme expenditure.” Government costs are "related to in-kind costs of using local government staff and vehicles and the value of CDD’s time (taken as the daily agricultural labour rate)." Leslie et al. 2011, Pg. 3.
Calculating non-SCI costs of school-based delivery:
- The average cost/treatment in the study was $0.58: “The total economic cost per treatment was $0.58. This includes programme, government and international costs.” Leslie et al. 2011, Author Summary. At 7% of the total cost, international tech. support accounts for $0.04/treatment.
- “The full economic delivery cost of school based treatment in 2005/06 was $0.76, and community treatment was $0.46. If only programme costs are included these figures are $0.47 and $0.41 respectively.” Leslie et al. 2011, Pgs 7-8.
- Therefore, non-program costs (government and international tech. support) are $0.29 ($0.76 - $0.47) of the $0.76 cost of each school based treatment. Since $0.04 is international tech. support, that leaves $0.25 of government costs, or 33% of the total cost.
It is our understanding that in recent programs SCI has continued to do some community-based deworming but that most of its treatments are delivered through schools. Therefore, we conservatively estimate that non-SCI actors contribute 30% of the cost of a SCI deworming program.
- 85
For example, SCI provides technical assistance to the program in Yemen and does not pay other costs. It reported the full number of treatments delivered in that program.
Najwa Al-Abdallah and Lynsey Blair, conversation with GiveWell, August 11, 2017.
- 86
"Imperial College pays the majority of SCI’s overhead costs, including rent and utilities, and offers free services such as legal assistance. Imperial’s legal department prepares all of SCI’s contracts. The college also covers some risks and liabilities that SCI may face. In return, SCI pays Imperial 6% of its funding from DFID, but this does not cover the full cost of the services that SCI receives. The 6% of the DFID funding that is paid to the college is not included in SCI’s budget, but is included in the total size of the DFID grant. SCI may perform an analysis of the costs covered by Imperial College, but this is not currently a high priority." GiveWell's non-verbatim summary of a conversation with Wendy Harrison and Najwa Al Abdallah on September 8, 2015, Pg. 4.
- 87
The data on total costs and treatments across all actors in DRC can be found in this spreadsheet, "GW summary" tab. In this spreadsheet, in the "Countries with pooled funding" tab, we then divide SCI's implementation costs in DRC by the overall cost per treatment estimate in DRC to back out an estimate of SCI's total treatments there.
- 88
In November 2018, we estimated SCI's room for more funding at approximately $19.3 million. This estimate relied on information SCI provided to us in July 2018. In August 2018, we asked SCI to revise its estimates of how it would spend additional funding (to account for GiveWell not including expansion of treatment for adults into our room for more funding estimates). In the revised spreadsheet, SCI also revised its estimates of the amount of funding it expected to receive from DFID. We did not realize at the time that SCI had made this update to its projected revenues; we discovered the change later through the process we use to internally vet our work. While this update changed our estimate of the total amount that SCI could absorb, it did not have implications for the amount of funding we recommended that Good Ventures and other donors give to SCI. We have since updated the review to incorporate SCI's updated budget projections (from August 2018 rather than July 2018).
See the following sources for more detail:
- 89
See this spreadsheet, sheet "Funds on hand and projected revenue"
- 90
See note in this spreadsheet, sheet "Funds on hand and projected revenue," cell E25.
- 91
Notes and sources in this spreadsheet, sheet "Funds on hand and projected revenue."
- 92
See this spreadsheet, sheet "Funds on hand and projected revenue," row 26.
- 93
Wendy Harrison and Najwa Al Abdallah, conversation with GiveWell, July 27, 2018. We also spoke with Dr. Delna Gandhi at DFID on October 6, 2017 about the process.
- 94
- Calculations in this spreadsheet. See sheets "SCI budget projections" and "Funding gaps."
- SCI, comments on a draft of this review, September 3, 2018.
- 95
- "The changes in our approach from last year:
- We asked the Ministries of Health to submit their Preventive Chemotherapy Plans for SCH and STH and their 5 year budget at a summary level using 5 main budget lines.
- We have reviewed these budgets in the light of existing data on actual cost per treatments but did not request detailed itemisation of any additional costs that may be associated with re-assessments etc. The 5 year plans are therefore our best estimates and are subject to change as more data becomes available to us.
- On an annual basis we do a very detailed analysis of all the items requested and do further analysis against the actual cost per treatments.
- This year we have reduced the treatment numbers to 75% of the target numbers provided by the countries. This is based on data from previous years where we see the actual reported treatment numbers are approximately 75% of the proposed targets. This we feel will help us to arrive at a more realistic cost per treatment delivered and gives a slight increase in the cost per treatment for previously budgeted figures.
- Our Value for Money Officer will be working this year on further analysis of the differentials between costs per treatment of a child versus treatment of an adult. Our expectation is that the adults costs will be lower than the per child cost and we are collecting data to verify this.
- We have not factored in inflations and fluctuations in exchange rates due to Brexit." Wendy Harrison, SCI Executive Director, email to GiveWell, August 24, 2018.
- SCI, comments on a draft of this review, September 3, 2018.
- "The changes in our approach from last year:
- 96
SCI initially provided a budget that included full scale-up of adult treatments to the levels requested by country governments. We requested that SCI provide a version of the budget that included adult treatments only in locations where adult treatment was ongoing.
- 97
- Wendy Harrison, SCI Director, email to GiveWell, October 16, 2017.
- SCI, comments on a draft of this review, September 3, 2018.
- 98
This is based on the number of treatments for schistosomiasis. SCI also provided data on the number of treatments for STH. We have not used the STH numbers in our estimate here because they may be skewed by a very large number of STH treatments estimated for DRC in April 2019-March 2020, which, given the budget for that work, appears implausible.
- 99
SCI, comments on a draft of this review, September 3, 2018.
- 100
Total costs in FY19/20 to FY21/22 of £82.6 million, converted to USD using exchange rate from Google of 1.32 USD/GBP as of June 30, 2018. SCI Full 5 year budget (July 2018), Redacted
- 101
- Comparing our conversations from 2013 to what happened in 2014 provides some context. In October 2014, SCI told us that if it had received an additional £1 million (about $1.6 million) at the beginning of 2014 it would have used the money to increase the size of each country's budget by 10-20%, which would have been used to make the programs more comfortable but would not have led to more treatments. Alan Fenwick, SCI Director, conversation with GiveWell, October 15, 2014. (As of November 4, 2014, Google stated that £1 is worth $1.6.) In November 2013, SCI told us that it would likely use additional funding to fund two existing programs and to start two additional programs. Expectations as of November 2013 for how it would spend an additional $4 million:
- Côte d'Ivoire: expected to need $400,000 and actually spent about $350,000.
- Mozambique: expected to need as much as $3 million and actually spent $1.4 million.
- Mauritania: expected to need $1 million and didn’t spend anything.
- Ugandan islands: expected to need about $600,000 and didn’t spend anything.
- The $1.75 million spent in Côte d'Ivoire and Mozambique represents about 80% of the spending.
- Expectations as of 2013 are listed in our November 2013 review of SCI.
- Actual spending is through mid-September 2014 from GiveWell summary of SCI finances (October 2014).
- See also our discussion of budget revisions in 2015 in our August 2015 update on SCI.
- In October 2015, SCI sent us estimates of its room for more funding for its next three budget years. (SCI planned SCH treatment numbers by country by year (October 2015)) We later learned that these estimates did not include the full amount of restricted funding that DFID planned to provide, which caused us to overestimate its room for more funding for its 2016-2017 budget year in our November 2015 review.
- "In October 2015, SCI shared estimates for its April 2016-March 2017 budget year with GiveWell on target treatment numbers by country, amounts of funding available from DFID and other large donors, and the amounts of additional funding required to deliver the targeted number of treatments in each country and cover SCI's central expenditures ( http://www.givewell.org/files/DWDA%202009/SCI/SCI_planned_SCH_treatment… ). GiveWell interpreted the estimates for SCI's projected central expenditures in the document as part of SCI's funding gap for its 2016-2017 budget year.
- "Documents SCI sent GiveWell in March 2016 indicate that around $1.5 million more in funding from DFID is available to allocate for SCI's 2016-2017 budget year than GiveWell had previously expected. This is because the document from October 2015 included funding available from DFID that could be allocated to in-country programmatic expenditures, but did not include the funding from DFID that SCI planned to allocate to central expenditures (included in the March 2016 documents)." GiveWell's non-verbatim summary of a conversation with Dr. Wendy Harrison, Najwa Al Abdallah, and Dr. Lynsey Blair, April 6, 2016.
- Comparing our conversations from 2013 to what happened in 2014 provides some context. In October 2014, SCI told us that if it had received an additional £1 million (about $1.6 million) at the beginning of 2014 it would have used the money to increase the size of each country's budget by 10-20%, which would have been used to make the programs more comfortable but would not have led to more treatments. Alan Fenwick, SCI Director, conversation with GiveWell, October 15, 2014. (As of November 4, 2014, Google stated that £1 is worth $1.6.) In November 2013, SCI told us that it would likely use additional funding to fund two existing programs and to start two additional programs. Expectations as of November 2013 for how it would spend an additional $4 million:
- 102
See this spreadsheet, sheet "Funding gaps."
- 103
Wendy Harrison and Najwa Al Abdallah, conversation with GiveWell, July 27, 2018.
- 104
"SCIs Reserve is set aside to ensure SCI is able to meet its liabilities and commitments (mainly to staff as per local laws and regulations). Reserve will be used as a last resort after exploring all options of reducing operations in case of drying [up] of resources. SCI will not commit funds that are not secured. In addition to the reserve, SCI is building its strategic funding and treating the funds that it will receive from GiveWell as multi-year grant and will ensure that the funds available at the bank at 31 March 2019 will be to cover future years as much as possible as per the allocation criteria shared with GiveWell previously." SCI, comments on a draft of this review, September 3, 2018.
- 105
- WHO, Summary of global update on preventive chemotherapy implementation in 2016, Pg 590, Table 1.
- WHO, Summary of global update on preventive chemotherapy implementation in 2015, Pg 456, Table 1.
- WHO Weekly epidemiological record, 18 December 2015, Pg. 707, Table 1.
