Summary
- Our work on re-evaluating SCI's evidence of effectiveness is ongoing.
- SCI has provided details of how it has spent unrestricted funding and how it expects to spend unrestricted funding in 2014. Notable among its plans is a new opportunity to work with Sudan to create a national deworming program.
- SCI has provided baseline prevalence and intensity rates from six countries and validated treatment coverage rates from two countries. Baseline prevalence and intensity rates are generally lower than baseline rates in SCI's previous studies, suggesting that future activities may be somewhat less impactful than past activities, though there is considerable variation across counties and data is not yet available for all countries SCI is working in. Coverage validation surveys have the potential to address some of our concerns about SCI's evidence of effectiveness. We have seen headline figures from two countries; we have not seen the details of how the data was collected. By the end of 2014, SCI expects to have collected coverage validation data from six countries; it is not clear when this data will be available to GiveWell.
- We have a somewhat better understanding than we did previously of why SCI has not shared more of its monitoring and evaluation results from recent programs.
- We do not have a full update on SCI's room for more funding. SCI plans to provide this information in the next few months. SCI's rate of spending suggests that funding may be a limiting factor in expanding its work.
Published: June 2014
Table of Contents
SCI's evidence of effectiveness
In November 2013, we posted on our blog about concerns we had about SCI's evidence of effectiveness. These concerns stemmed from new information about studies of national deworming programs supported by SCI. Our review of SCI relied heavily on these studies, and the new information made us less confident in SCI's track record.
Over the past few months, we have continued to investigate the potential problems in the studies and revisit the case for SCI's past effectiveness. This work is ongoing. To date, we have:
- Raised our concerns with SCI and asked SCI to share any research on the effectiveness of its programs not yet included in our review.
- Revisited all of the studies of SCI's effectiveness that we are aware of to see if we could better understand the methods used and the potential sources of bias.
- Contacted the authors of some of the studies to ask for more information on particular parts of the methodology used in the studies.
Financial update: unrestricted funds
Spending between September 2013 and March 2014
Between our last update on SCI, which reported on spending through August 2013, and March 2014, SCI spent about $1.75 million in unrestricted funds. 79% of this was spent on country programs, split evenly between supplementing programs that receive funding from the UK's Department for International Development (DFID) and supporting programs for which SCI does not have restricted funding.1 Details of how unrestricted funds were spent follows.
Country programs
- Mozambique ($604,000, 35% of spending): In August 2013, SCI committed 400,000 GBP to fund treatments in Mozambique by March 2014, and since then has spent 371,000 GBP (about $604,000).2 SCI's grant from DFID also funds some treatments in Mozambique, but at significantly less than national scale – in part because SCI has shifted DFID funding to other countries with the expectation of using unrestricted funds in the country.3
- Ethiopia ($479,000, 27% of spending): Disease mapping was recently completed in Ethiopia4 and SCI expects that Ethiopia will complete a large-scale round of treatment in 2014. SCI expects to treat 3.7 million individuals in 2014. These treatments will be primarily funded by unrestricted funds, with a small contribution from DFID.5
- Zimbabwe ($179,000, 10% of spending): Funds were granted to the University of Edinburgh for disease mapping in Zimbabwe.6
- Cote d'Ivoire ($86,000, 5% of spending): SCI allocated unrestricted funds to supplement DFID funds for disease mapping and monitoring and evaluation.7
Other costs
- General organization costs ($191,000, 11% of spending): This included spending in the categories "SCI development" (including office support, travel, salaries, and training), "advocacy and fundraising," and "student support."8
- Enhanced monitoring and evaluation ($170,000, 10% of spending): SCI made three grants in this category:9
- $89,600 to University of Cambridge for "assessing strategies to increase coverage."
- $46,128 to the Ugandan Ministry of Health for "enhanced M&E in Uganda to explore elimination strategies."
- $34,308 to the Technical University of Munich to match funding from the Bill and Melinda Gates Foundation "to support cysticercosis project in Malawi."
Spending since November 2011
GiveWell first recommended SCI in November 2011.10
Since then, SCI has raised $7.6 million in unrestricted funds,11
of which we can attribute $4.4 million to GiveWell's recommendation.12
It has spent about $4.0 million in unrestricted funds, in the following ways:13
| Country/Type | % of total |
|---|---|
| Countries not funded by restricted funds | 53.5% |
| Ethiopia | 36.2% |
| Zimbabwe | 8.7% |
| Senegal | 5.6% |
| Mauritania | 1.5% |
| DRC | 1.4% |
| Madagascar | 0.1% |
| DFID-funded countries | 24.9% |
| Mozambique | 16.6% |
| Cote D'Ivoire | 4.5% |
| Liberia | 2.3% |
| Niger | 0.8% |
| Zanzibar | 0.5% |
| Malawi | 0.1% |
| Uganda | 0.1% |
| Countries funded by other restricted funds | 4.0% |
| Yemen | 3.9% |
| Rwanda | 0.1% |
| General organization costs | 13.1% |
| Enhanced monitoring and evaluation | 4.8% |
Funding commitments
SCI's most recent round of funding commitments were set in August 2013 and reported on in our October 2013 update on SCI. The table below shows the remaining balances on the commitments from August 2013:14
| General organization costs | $10,091 |
| Enhanced monitoring and evaluation | $39,709 |
| Burundi – capacity building | $73,310 |
| DRC – mapping | $80,394 |
| Mozambique – treatment | $47,739 |
| Senegal – M&E support for USAID program | $259,447 |
| Zimbabwe – mapping and treatment | $48,873 |
| Total | $0.56 million |
In addition, SCI shared its plans for allocating unrestricted funds to country programs in 2014.15 It is our understanding that these are less formal plans than the commitments listed above, and that these plans are subject to change.
| Sudan | $814,550 |
| Mozambique | $785,226 |
| Ethiopia | $651,640 |
| Cote D'Ivoire | $325,820 |
| Liberia | $40,728 |
| Niger | $26,473 |
| Total | $2.64 million |
See our conversation with SCI in April 2014 for a discussion of how SCI allocates DFID and unrestricted funding across the countries it works in. It is our understanding that, within the countries that receive funding through SCI's grant from DFID, unrestricted funds are largely fungible with DFID funds.
Current reserves
SCI told us that as of March 31, 2014, it held £2.59 million, or about $4.22 million in unrestricted funding.16 Between January and March, GiveWell received $242,000 for SCI (and granted these funds to SCI on May 14), bringing SCI's reserves to about $4.46 million.
| Amount in millions17 | |
|---|---|
| Balance of unrestricted funds as of Sep 1, 2013 | $2.77 |
| Unrestricted spending between Sep 2013 and Mar 2014 | $1.75 |
| Unrestricted revenues between Sep 2013 and Mar 2014 | $3.20 |
| Balance of unrestricted funds as of Mar 31, 201418 | $4.46 |
| Balance of spending commitments as of Mar 31, 2014 | $0.56 |
| Country program spending plans for 2014 | $2.64 |
| Balance as of Mar 31, 2014 less commitments and country program spending plans | $1.25 |
We can confidently attribute about $2.26 million of SCI's $3.2 million in unrestricted revenues in this period to GiveWell's recommendation (more on how we attribute donations to GiveWell's recommendations here).19
Evaluation of previously funded activities
Most of the monitoring we have seen of SCI's programs was carried out in 2003-2007, with funding from the Gates Foundation, with the addition of one study from Burundi carried out in 2007-2011.20 SCI has told us that it has conducted more recent monitoring in many of the countries it works in.21 In our conversation with SCI in April, we asked why we have not seen more of this recent monitoring. SCI told us that:22
- Data collected under its grant from USAID (2006-2011) is currently being prepared for publication in an academic journal.
- Third parties (e.g. governments, WHO, funders) often need to give permission before data can be shared.
- It can take some time for data, once collected, to reach SCI because in many countries it is cleaned and analyzed by country program staff before being shared with SCI.
In its October 2013 report to DFID (which SCI shared with us in May 2014), SCI notes that it will track impact in the countries supported by DFID through:23
- Average disease intensity. As of October 2013, baseline data was available, or partially available, from sentinel schools in six countries: Liberia, Malawi, Zanzibar, Mozambique (partial data), Tanzania (partial data), and Niger (historical data).24 SCI reported summary figures from these studies. Prevalence and intensity rates are generally lower than baseline rates in SCI's previous studies and vary considerably across countries (details in footnote25 ). In Malawi an error in data collection may have resulted in prevalence being underestimated.26 In Niger, SCI is focusing on low prevalence districts because it is supplementing an existing USAID NTD program.27 In Zanzibar, treatment has been ongoing.28 The first round of follow up data for these countries will be collected in 2014.29
- Validated treatment coverage. SCI will conduct surveys to monitor what percentage of targeted children are reached by the programs. As of October 2013, surveys had been completed in Malawi (76-80% validated coverage) and Zanzibar (83-92% validated coverage).30 We have not seen technical details of how these studies were carried out. The government of Zanzibar carried out the survey there.31 We do not know who carried out the Malawi survey or how coverage rates from routine reporting compared to validated coverage rates. Coverage surveys are planned for four more countries in the next year.32
We have not seen details of how the surveys were carried out for the above surveys, with the exception of the baseline surveys in Liberia (see our October 2013 update on SCI) and Zanzibar.33
Room for more funding
How do SCI's current plans compare to its room for more funding analysis from November 2013?
SCI told us in November 2013 that it could effectively absorb up to $10 million in additional unrestricted funds in the next year and it provided plans for how it would expect to spend the first $4 million it received. Between September 2013 and March 2014, it received about $3.2 million in unrestricted funds, of which we can confidently attribute about $2.26 million to GiveWell's recommendation (see above). Its plans for this funding as of April 2014 differ somewhat from its plans as of November 2013:34
| Country | Plans for 2014 as of November 2013 for first $4 million in additional funding - in order of priority | Plans for 2014 as of April 201435 |
|---|---|---|
| Cote D'Ivoire | $400,000 | $411,786 |
| Mozambique | $2,000,000 | $785,226 |
| Mauritania | $333,333 (SCI expected to raise $1 million for 3 years before commiting to program) | Not in plan |
| Ugandan islands | $200,000 | Not in plan |
| Sudan | Not in plan | $814,550 |
| Ethiopia | Not in plan | $651,640 |
| Liberia | Not in plan | $40,728 |
| Niger | Not in plan | $26,473 |
| Zanzibar | Not in plan | $19,305 |
| Total | About $4 million | $2,749,707 |
There are three main differences between the November and April plans:
- SCI has allocated about $800,000 to Sudan. The government of Sudan recently approached SCI to request its help in setting up a program in the country. Most of the allocated funds would likely be used for disease mapping.36
- SCI has continued to allocate unrestricted funding to Ethiopia. While it expects DFID to begin supporting treatments in Ethiopia in 2014, it plans to allocate only GBP 50,000 (about $81,000) of DFID funding to Ethiopia in 2014.37
- SCI has not allocated funding to support treatment on the Ugandan islands or to Mauritania. In the case of Mauritania, SCI told us in November that it would not allocate this funding until it had enough to commit to three years of the program, at a cost of $1,000,000.38
We have limited insight into the reasons for these changes. However, one of the reasons we recommend unrestricted funding to SCI is because we want it to have the flexibility to adjust for changing circumstances. The decision to allocate funding to Sudan, in particular, seems to be a case in which SCI reallocated funding to an unexpected opportunity.
Does SCI have room for more funding?
We asked SCI for an update on how it would use additional funding. SCI told us that it is in the middle of a process to upgrade its financial planning processes and would be able to update us on this by July.39
As of March 31, 2014, SCI held about $3.9 million in uncommitted unrestricted funds. As of April 2014, it expects to spend about $2.6 million on country programs in 2014.40 While not conclusive, this rate of spending is consistent with the idea that SCI has more opportunities to expand programs than it has funding to do so. As discussed below, we will follow up with SCI about room for more funding later this year.
We noted in our October 2013 update on SCI that DFID was considering a second grant to SCI for £25 million over five years. As of early April 2014, DFID was in the final stages of approval of the grant.41
Our plans for our next update on SCI
Our top priority will be to continue our investigation for SCI's evidence of effectiveness (see above).
In addition, in the next few months (by September 2014), SCI has told us that it will share:
- Reports to SCI's advisory board on programmatic and financial progress
- Decisions made at its biannual meeting to allocate unrestricted funds
- Plans for how SCI would spend additional funding
This information, as well as conversations with SCI, will help us to answer questions that are important to our understanding of SCI and which we have not attempted to answer in the current update:
- How much has SCI spent in total since we recommended the organization (both restricted and unrestricted funds) and how many treatments has it delivered? What has the average cost per treatment delivered been since we started recommending SCI?
- How much additional funding could SCI productively use in the next year and what is the likely impact of this funding?
Sources
- 1
SCI financial update (April 2014), Sheet 'Summary.'
- 2
SCI report to GiveWell (April 2014), Pg 2.
- 3
Conversation with Wendy Harrison and Sarah Knowles in April 2014.
See also table in SCI report to GiveWell (April 2014), Pg 5. - 4
"Ethiopia: The mapping of SCH, STH, and water and sanitation indicators across the country has been completed and the data are currently being analysed. Completed maps are expected in April 2014 and will inform where large-scale MDA will be conducted in 2014 and beyond. SCI is also offering technical assistance on deworming approaches on a Partnership for Child Development-led school health and nutrition programme." SCI newsletter (April 2014).
- 5
SCI report to GiveWell (April 2014), Pg 5.
- 6
- 7
"Funds for Cote D’Ivoire were used to leverage DFID funded activities and support provided by a private donor to map Schistosomiasis and STH in the 49 districts where disease endemicity is unknown. Additionally a monitoring and evaluation strategy was developed and 11 sentinel sites selected which were evaluated post-mapping." SCI report to GiveWell (April 2014), Pg 3.
- 8
SCI report to GiveWell (April 2014), Pg 1.
- 9
Description of grants from SCI report to GiveWell (September 2013), Pg 7.
Payment of grants documented in SCI report to GiveWell (April 2014), Pg 2. Amounts converted to USD from GBP in SCI financial update (April 2014), Sheet 'Summary from SCI – Apr 2014.' - 10
See GiveWell blog post: Top charities for holiday season 2011: Against Malaria Foundation and Schistosomiasis Control Initiative.
- 11
SCI financial update (April 2014), Sheet 'Combined with previous updates.'
- 12
SCI financial update (April 2014), Sheet 'Due to GiveWell.'
- 13
SCI financial update (April 2014), Sheet 'Combined with previous updates.'
- 14
SCI financial update (April 2014), Sheet 'Summary'
- 15
SCI report to GiveWell (April 2014), Pg 5
- 16
SCI report to GiveWell (April 2014), Pg 4
- 17
SCI financial update (April 2014), Sheets 'Summary,' 'Combined with previous updates.'
- 18SCI told us that as of March 31, 2014, it held GBP 2.59 million, or about $4.22 million in unrestricted funding (SCI report to GiveWell (April 2014), Pg 4). Between January and March, GiveWell received $242,000 for SCI (and granted these funds to SCI on May 14), bringing SCI's reserves to about $4.46 million.
- 19
SCI financial update (April 2014), Sheet 'Due to GiveWell.'
- 20
See our review of SCI. Footnote 40, Table "Funding for mass treatment | Have we seen disease prevalence/intensity or morbidity results?" provides a summary of the monitoring we have seen.
- 21
- 22
Conversation with Wendy Harrison and Sarah Knowles in April 2014
- 23
- 24
"Baseline data has been received as follows:
- Data complete and analysed -Malawi, Liberia, Zanzibar (see Annex 1).
- Data partially available – Tanzania, Mozambique (see Annex 1). Due to restrictions on data flow outside the country, intensity data has not been received and analysis will have to be conducted in country. The SCI biostatistician is working with the NTD Programme Manager in Tanzania and Mozambique to identify the optimal time to visit each country to undertake any necessary training and provide statistical support for existing MoH NTD staff. Mozambique will be scheduled during the last quarter of FY4. Tanzania is scheduled for April 2014 (due to the limited availability of the NTD Programme Manager who has specifically requested to be present).
- Data collected, currently being entered – Zambia, Uganda
- Data collection scheduled - Cote d’Ivoire (November 2013)
- Historical baseline data available – Niger"
- 25
All non-null values are statistically significant at the 95% level, with the exception of those values marked with a "*" for which the 95% confidence interval was not reported. Only data on infections in children is reported. Summary of SCI data from previous programs is based on our summary of data from SCI's past programs in our review of SCI.
Summary of SCI data from previous programs Liberia Malawi Mozambique (partial data) Zanzibar Tanzania (partial data) Any schistosome infection Prevalence N/A 39.0% 10.5% N/A N/A N/A Mean intensity N/A N/A N/A N/A N/A N/A Proportion heavily infected N/A N/A N/A N/A N/A N/A S. haematobium Prevalence Very low in two countries, 59% and 75% in two others 20.2% 9.9% 19% * 4.3% in Unguja and 8.9% in Pemba N/A Mean intensity 94.2 egges/10ml in one country, not reported for others 42.4 eggs/10ml 2.3 eggs/10ml N/A N/A 18 eggs/10ml Proportion heavily infected 22.8% in one country, not reported for others 9.3% 1.0% N/A 0.6% in Unguja and 3.2% in Pemba N/A S. mansoni Prevalence 3%, 6-13%, and 42% in three countries, not reported in fourth 26.2% 1.9% 0.038 N/A N/A Mean intensity 4.6 and 220 epg in two countries, not reported in others 16.3 epg 18 epg N/A N/A 109 epg Proportion heavily infected N/A 0.2% 0.1% N/A N/A N/A Ascaris Prevalence N/A 12.8% 0.1% 4.3% * N/A N/A Mean intensity N/A 4.3 epg 0.1 epg * N/A N/A 0 epg Proportion heavily infected N/A 0.0% 0.0% N/A N/A N/A Hookworm Prevalence 15-18% and 51% in two countries, not reported in others 15.2% 0.2% 3.6% * N/A N/A Mean intensity 309 epg in one country, not reported in others 5.1 epg 0.3 epg * N/A N/A 218 epg Proportion heavily infected N/A 0.0% 0.0% N/A N/A N/A Trichuris Prevalence N/A 0.7% 0.0% 1% * N/A N/A Mean intensity N/A 0.2 epg N/A N/A N/A 0 epg Proportion heavily infected N/A 0.0% 0.0% N/A N/A N/A Data from SCI report to DFID (October 2013), Pgs 26-28, except for Zanzibar.
Data for Zanzibar is from Knopp et al 2013: "The baseline Schistosoma haematobium prevalence in school children and adults was 4.3% (range: 0-19.7%) and 2.7% (range: 0–26.5%) in Unguja, and 8.9% (range: 0–31.8%) and 5.5% (range: 0–23.4%) in Pemba, respectively. Heavy infections were detected in 15.1% and 35.6% of the positive school children in Unguja and Pemba, respectively." This study also measured disease rates in adults and the treatment program will include adult treatments. - 26
"In Malawi, urine volumes were not accurately recorded thus it is possible that data is indicating lower overall prevalence in sentinel sites. ICOSA will be undertaking further data analysis to quantify underestimates using mapping data from 2012 and baseline data in appropriate districts." SCI report to DFID (October 2013), Pg 15.
- 27
"In Niger, a strategic plan for schistosomiasis 2012-2016 has been developed and is being implemented. ICOSA activities are focused in 7 low prevalence districts which takes the programme to 100% geographical coverage alongside the USAID support for NTDs. Hot-spot areas of transmission have been identified in these low prevalence districts as a result of the re-assessment surveys." SCI report to DFID (October 2013), Pg 17.
- 28
"The NTD programme in Zanzibar has recently completed the 3rd round of MDA." SCI report to DFID (October 2013), Pg 17.
- 29
"As determined by the treatment schedules within each country, the project will have follow-up data available from Mozambique, Liberia, Malawi and Zanzibar during 2014 to determine progress against the health impact milestone for end 2014." SCI report to DFID (October 2013), Pg 5.
- 30
"Zanzibar: Unguja 91.9% overall; Pemba 82.6% overall. Malawi: 75.9% female; 79.9% male." SCI report to DFID (October 2013), Pg 5
- 31
"The Government of Zanzibar undertake their own coverage survey." SCI report to DFID (October 2013), Pg 5.
- 32
"To date, Zanzibar and Malawi have undertaken coverage validation surveys, as distinct from reported coverage captured during routine reporting in all countries. During FY4, coverage surveys will be undertaken in Liberia, Uganda and Tanzania (December 2013) and Cote d’Ivoire (January 2014)." SCI report to DFID (October 2013), Pg 6.
- 33
- 34
Summary at SCI financial update (April 2014), Sheet 'Compared to Nov 2013 plan.'
For November 2013 expectations, see our review of SCI.
For April 2014 expectations, see SCI report to GiveWell (April 2014), Pgs 5-6. - 35Also includes spending in Cote D'Ivoire (£52,769) and Zanzibar (£11,850) that was not yet committed as of our last update. SCI report to GiveWell (April 2014), Pg 3.
- 36
Conversation with Wendy Harrison and Sarah Knowles in April 2014
- 37
For April 2014 expectations, see SCI report to GiveWell (April 2014), Pgs 5-6.
- 38
See our review of SCI.
- 39
Conversation with Wendy Harrison and Sarah Knowles in April 2014
- 40
Summary at SCI financial update (April 2014), Sheet 'Summary.'
- 41
Conversation with Wendy Harrison and Sarah Knowles in April 2014