VillageReach - April 2011 Review

Published: April 2011

Summary

VillageReach aims to improve the systems that distribute medical supplies to rural areas in Africa, so that life-saving supplies get to those who need them. Its programs include both technical support staff and changes in logistical setups (such as moving from a system in which health clinics collect their own supplies to a centralized delivery system). It ran its pilot program in a single province in Mozambique from 2004 to 2007 and is now working to scale up the project across the entire country. As of November 2011, it operates in 4 of Mozambique's 9 provinces.

We believe that VillageReach is a standout organization because:

• Its pilot project and current scale-up focuses on increasing immunization coverage among children. Immunizations are a proven, cost-effective way to prevent child mortality.
• It produced a credible evaluation of its pilot project that, while not a randomized controlled trial, makes a compelling case for impact.
• As an organization it has demonstrated a commitment to sharing results of its work publicly enabling it and others to learn from both its successes and failures.
• Because of the above commitment, if its approach to health logistics succeeds, we believe there is significant possibility of upside, i.e., a positive effect that significantly surpasses the direct benefits of vaccinating the children its program serves. This would occur if its project succeeds and other governments choose to implement VillageReach's approach to health logistics.

Our full review, below, discusses our full assessment of VillageReach, including what we see as its strengths and weaknesses as well issues we have yet to resolve.

We also continue to follow VillageReach's progress closely. We post regular updates on its progress to our VillageReach updates page.

What do they do?

Broadly, VillageReach aims to improve health care access in remote, underserved places by improving the medical supply delivery systems.1 A key component of VillageReach's system focuses on shifting health-products distribution from a "pull" system (in which health clinics collect their own supplies) to a "push" system (in which dedicated teams deliver supples to clinics directly).2

VillageReach implemented a pilot project in the province Cabo Delgado in Mozambique from 2002 to 2007 focused on delivering vaccines, and also briefly ran a similar program in the neighboring province of Nampula. It then handed off the projects. In 2010, it resumed its vaccine delivery system in Cabo Delgado, and is working to expand the system to other provinces in Mozambique and to use the system to deliver additional supplies. VillageReach has also contracted with other organizations to improve logistical systems in other countries, but generally hasn't used, and doesn't plan to use, funding from individuals for projects outside of Mozambique (with a few exceptions discussed below).

Pilot project in Cabo Delgado, Mozambique (2002-2007)

VillageReach, which was founded in 2000,3 conducted a demonstration of its vaccine delivery system in the province Cabo Delgado in Mozambique from April 2002 to March 2007.4 The Cabo Delgado project officially became the local government's responsibility in 2007,5 but VillageReach resumed responsibility in mid-20106 due to problems under government management.

VillageReach's pilot project in Cabo Delgado is the focus of our review because it is similar to the future activities most likely to be funded with donations. The pilot project delivered supplies to all of the facilities in the province that offered vaccinations. In addition to vaccines, VillageReach delivered propane gas, medicines, and other medical supplies to the facilities each month.7 The components of the delivery system included:8

• Transportation vehicles: "Created multi-modal transport network including land cruisers, motorcycles and bicycles. Staff inspects and repairs equipment on monthly visits."
• Cold chain: "Introduced reliable, low maintenance and cost-effective refrigerators in clinics."
• Injection safety equipment: "Installed propane burners for sterilization, incineration points and needle removers to ensure safe disposal of used syringes."
• Supplies-tracking: "Partnered with Iridium to utilize their global satellite system, introduced communication system in trucks to enable near, real-time inventory tracking."
• Training: "Trained community representatives to provide basic health care."
• Creating a for-profit enterprise to improve energy supply: "Established VidaGas, a Mozambican propane distribution company to reliably supply energy to clinics, businesses and households."

Nampula expansion (2006-2007)

In 2006, VillageReach began to replicate the project in a second province in Mozambique, Nampula.9 It handed off the project to a local nonprofit in January 2007.10

Project scale-up in Mozambique (2010-present)

VillageReach's current and planned future activities in Mozambique focus on expanding/replicating the model of the pilot project. These are the main activities for which VillageReach is seeking donations from individuals.

VillageReach expects to have an active presence in each of eight provinces for three years, after which point it is hoped that the government health system in that province will maintain its model without further support.11 VillageReach will emphasize the health supply delivery system detailed above,12 while also applying it to additional supplies such as rapid diagnostic tests for malaria and other diseases.13

Current activities

VillageReach resumed activities in Cabo Delgado in June 2010, has been delivering vaccines to health facilities on a monthly basis, and has started to distribute diagnostic tests.14 As of February 2011, it had acquired two vehicles, completed a baseline survey, trained coordinators, and begun regular data collection.15

In another province, Niassa, it completed a baseline survey and began distributions in early 2011 (we do not have further details on activities in Niassa).16

Future plans

As of February 2011, the project scale-up plan was 6-12 months behind the schedule set out in July 2010 due to delays partly related to dialogue with the government but primarily due to delays in receiving sufficient funding (though revenue in 2010 closed VillageReach's entire "stretch funding gap" for the year, the vast majority of that revenue was not received or anticipated until December).17 In Maputo, VillageReach originally planned to begin full implementation of its program in the final quarter of 2010, and now intends to begin full implementation in mid-2011.18

In addition to Cabo Delgado, Niassa, and Maputo, VillageReach is currently hoping to implement its model across five more of Mozambique's provinces, for a total of eight out of ten provinces in the country.19 For each province, implementation is scheduled to start in either 2012 or 2013.20 As of February 2011, VillageReach was in discussions with the Gaza province to begin work there,21 and planned to begin talks with the Inhambane province once it had completed a key hire.22

Additional activities

VillageReach is also involved in other projects, specifically:

• A variety of contract engagements. VillageReach has contracted with specific parties to carry out projects in different parts of the world, all on the theme of improving health system logistics but sometimes different in many ways from the pilot project discussed above. With one exception, all of these projects are fully funded by the party in question, and VillageReach does not seek donations for them. As such, we have not deeply examined these programs (with the exception of the one for which donations are sought).
• A number of other initiatives, detailed below.

Contract engagements

Specific parties have offered VillageReach funds to carry out projects in different parts of the world, all on the theme of improving health system logistics but sometimes different in many ways from the pilot project discussed above. Notes on these engagements follow. Projected expenses for these contract engagements have not been updated since mid-2010 and therefore will not fully reconcile with other figures given below.

As noted in the table above, all contract projects have been fully funded by the main project funder, with the exception of the South Africa project, which requires VillageReach to contribute $100,000 toward the costs of the project. Some additional notes on this project, since it is relevant to individual donors:

• VillageReach seeks to improve the general health system capacity in the Zululand district of the KwaZulu Natal province of South Africa23 through "business process mapping and optimization, streamlining data collection, staff training, mentoring, and supervision, expanding quality improvement methodology, and the identification of innovative uses for information and communication technology in patient care coordination at the clinic level."24 The project is projected to last three years.25
• VillageReach's specific plans for assessing its impact are unclear to us. It states that it plans on "A baseline and endline survey to gauge the outcomes and impact of the project" as well as "track[ing] output and outcome indicators that could include stock levels of key commodities, accuracy and frequency of data collection in the clinics, frequency of supervision visits, reports of data use and understanding by clinic staff, and satisfaction of clinic staff with the program."26 We have not seen details of how baseline and end-of-project data will be collected, or of which indicators are included in VillageReach's long-term goal of "80% improvement in identified primary healthcare indicators."27 On a more short-term basis, it is also unclear to us how VillageReach plans on assessing "accuracy and frequency of data collection in the clinics."28

Other plans

As of mid-2010, VillageReach's other planned activities included continuing to develop and share its management information system, pursuing new contract engagements, and ongoing support to VidaGas, a gas delivery business developed as part of its pilot program.

The following is its overall budget for the year to September 31, 2010 (projected for June-September).29 Note that these figures are not taken from exactly the same source, time period and date as the contract engagement figures above and have not been updated since mid-2010.

Including the $100,000 matching requirement over three years detailed above (we count this as $33,333 for the year), there were a total of $810,844 in expenses or projected expenses in the year to September 31, 2010 that were neither part of the Mozambique expansion nor included in contract engagements.

Does it work?

This section focuses on VillageReach's pilot project in Cabo Delgado, then briefly discusses its hand off of its work in Cabo Delgado to the government and in Nampula to another organization. It also reports on what we know to date about its current and planned future scale-up of the project in Mozambique.

Pilot project in Cabo Delgado

One of VillageReach's primary methods of evaluating success is through tracking the progress made in administering basic immunizations. Such immunizations are a proven, cost-effective way to improve health and save lives in the developing world (more at our report on immunization), and so success in increasing immunization coverage - alone - likely constitutes, in our view, success in saving lives.

Below we examine evidence provided by VillageReach on its pilot project in Cabo Delgado. This evidence includes: (a) reporting on vaccines and equipment delivered, progress in vaccination coverage rates, and health clinic inventories; (b) an independent evaluation30 of VillageReach's impact. We conclude that VillageReach's pilot project was effective in increasing vaccine coverage.

Delivery of vaccines and medical supplies

A key component of VillageReach's model is a shift from a "collection-based" to a "delivery-based" supply system: rather than clinics' being responsible for picking up their own supplies, VillageReach's logistics team delivers supplies and provides other logistical support.31

The tables below provide data on the goods VillageReach delivered to Cabo Delgado between 2004 and 2007.32 (Note that the project began in April 2002; we aren't sure why data has not been provided pre-2004.)

Increases in immunization coverage

The charts below show that (a) the number of children receiving DTP-3 (third dose of diphtheria-tetanus-pertussis vaccine34 ) immunizations increased; (b) the number of children who "dropped out" during the DTP-3 sequence - that is, they received one, but not all doses - fell; (c) reported "stock-outs" - centers with no inventory of the vaccine - fell significantly over the course of VillageReach's pilot project, which ran from April 2002 to March 2007.35 . Charts are taken from Kane 2008.36

Were improvements attributable to VillageReach?

The evaluation of the project notes that during this time period, "Mozambique and most sub-Saharan African countries achieved significant improvements in their DTP-3 coverage, probably due to GAVI [a global funder of immunization programs] and its support for infrastructure development and provision of new vaccines and safe injection equipment."37

To evaluate the question of VillageReach's role in improvements, we look at three types of information:

1. Reports from VillageReach about the problems interfering with immunizations in Cabo Delgado before their arrival.
2. An evaluation report published by VillageReach explicitly focused on addressing this question, comparing the change in immunization coverage rates in Cabo Delgado to that of another province in Mozambique, Niassa.
3. Examining the changes in immunization coverage in several countries in Africa over this period to put the observed change in context.

We also observe that the charts showing "stock-outs" above have stock-outs falling from a very high level prior to the start of the project to a very low level shortly after the project began. While it is possible that this change occurred for some reason other than VillageReach's involvement in the area, such an outcome seems unlikely.

VillageReach reports of pre-arrival conditions: VillageReach provides a report of the obstacles to immunization coverage in Cabo Delgado before its arrival. We would prefer to have better documentation of these conditions, but nevertheless, we believe the report offers some support to the idea that VillageReach's services were needed in Cabo Delgado.

VillageReach reports,

In 2002, before starting work, VillageReach performed an assessment of access to vaccines in Cabo Delgado province.39 This report claims that, in 2002, there were 22 health facilities in the three districts in Cabo Delgado that VillageReach assessed.40 Of these 22, 4 did not offer access to vaccination services:41 two facilities because they did not have access to a cold chain; 1 because it lacked personnel; and 1 for other reasons.42

VillageReach evaluation document: VillageReach compared improvements in vaccine delivery in Cabo Delgado to improvements in the nearby Niassa province, which was not served by VillageReach.43

• Baseline data collection: Because VillageReach did not have baseline data (i.e., data from a time prior to the start of the project) available for Niassa, it used data from the 1997 and 2003 Demographic and Health Surveys, which surveyed a large number of households and provided data on DTP-3 coverage.44
• Outcome data collection: Evaluators randomly selected households in the treatment area (Cabo Delgado province), including 474 children.45 Evaluators then also randomly selected households in the comparison area (Niassa province), thereby including 571 children in the evaluation.46
• Results: The study found that DTP-3 coverage rates increased substantially in both provinces during this period, but rates improved more in Cabo Delgado (treatment area) than Niassa (control area). The chart below shows the change in coverage rates in the two areas during this time period; the 1997 and 2003 data points are based on Demographic and Health Surveys, while the 2008 data points are from VillageReach's independent data collection.47

On one hand, we believe that this chart creates an inflated picture of VillageReach's impact. We have reason to believe that there were significant improvements in immunization coverage between 1997-2001 that were more related to Cabo Delgado's recovery from the aftermath of a civil war than to VillageReach's activities.48 However, the jump to extremely high levels of coverage as of 2008 - a change not mirrored in the nearby province - give some reason to attribute impact to VillageReach.

The evaluation report is forthright about many limitations of this comparison analysis, including limited sample size, uncertainty about the appropriateness of Niassa as a "comparison province," and issues with taking baseline and endpoint data from different sources.49 However, it says that its way of comparing the results of surveys done with different methodology "is consistent with international practice," and it concludes that it appears that the Project is responsible for the immunization coverage rising more in the treatment province, Cabo Delgado, than in the comparison province Niassa; although "additional information about the conditions in Niassa compared to those in Cabo Delgado is needed to better understand and interpret the comparison data."50 It also notes that "it is ... unlikely that the activities of other NGO’s, which are not very involved in immunization activities in Cabo Delgado, were responsible for the improvement."51 (Note, however, that it does not discuss the confounding effect of the civil war recovery that we discussed above.)52

Our comparison of Cabo Delgado to other areas in the developing world: We took a broader look at changes in African immunization coverage over the time period in question in order to further investigate the idea that Cabo Delgado's improvements may simply have reflected a wider phenomenon. Using the Demographic and Health Surveys (Measure DHS),53 we collected data on DTP-3 immunizations for countries in Sub-Saharan Africa.54 The table below summarizes this data, sorted by the country's arithmetic percentage change in immunization coverage.

This table provides information at a country level rather than province level, and variation within countries could be significant. There does not appear to be strong evidence of a continent-wide positive trend in immunization rates, but it does not appear, on its own, to rule out the idea that the observed change in Cabo Delgado purely reflected a broader (country-wide or continent-wide) change. We note, however, that Cabo Delgado's 2003 coverage rate was slightly below the Mozambique overall rate, while its post-project rate was above the Mozambique overall rate (and above every other country's overall rate).

Bottom line on the Cabo Delgado program between 2001-2008

We do not feel that any of the pieces of evidence above is highly compelling by itself. But we are persuaded of VillageReach's impact by the combination of the observations that VillageReach's program (a) entered an area with clearly documented logistics problems; (b) reduced stockouts - one of the clearest measures of the logistics improvement it was aiming for - to near-zero levels; (c) brought Cabo Delgado from an "average" (for the country) level of coverage to an extremely high level of coverage; (d) was reported not to have been supplemented by other nonprofits' programs.

Hand off of the Mozambique projects

Cabo Delgado

Based on the evidence above, we feel that the VillageReach program improved capacity to deliver vaccines in Cabo Delgado. However, the Cabo Delgado project officially became the local government's responsibility in 2007,55 and a later report stated:56

VillageReach's representatives stated to us, "When we turned it over to the government they let it slide back into the old collection-based system."57

The fact that VillageReach monitored the program's continuing performance, and has been open about setbacks, is encouraging; but news of program deterioration is cause for concern.

As discussed above, VillageReach has now reactivated its support role in Cabo Delgado, and conducted a baseline survey there.58

Ultimately, we are skeptical about VillageReach's ambitions of handing over its model to the government. However, we note that VillageReach could be making lasting differences in individuals' lives even if its effects on health care are only temporary, since 1-3 doses of most vaccines are sufficient to immunize children against diseases. (Details in our report on immunization.) Our recommendation of VillageReach is made under the assumption that it will not succeed in getting its model adopted by the government, while recognizing that its ultimate cost-effectiveness would be much higher if it could.

Nampula

As described above, VillageReach briefly worked in the Nampula province of Mozambique before handing its activities off to a local organization in January 2007.

VillageReach provided us with internally collected data from this project through August 2008,59 and stated to us that the data became unreliable (due to internal contradictions) after that point.60 We have not received clearance to post the data publicly. Overall, it showed encouraging trends that resemble the trends outlined above: increasing numbers of immunizations and substantial drops in the rates of stockouts and other logistical problems. However, because of the fact that the data terminates at an apparently arbitrary point, we have serious doubts about the impact of this project.

This project's expenses, overall, were equal to about 20% of the expenses associated with the pilot project (details in our cost-effectiveness section). Based on multiple conversations with VillageReach representatives, it is our impression that the very limited role VillageReach took is not representative of VillageReach's typical or future activities.

Project scale-up in Mozambique

Project plans

As discussed above, the current and planned future activities in Mozambique will be fundamentally similar to the pilot project, and evaluation will be similar as well, in that VillageReach will conduct before and after surveys of immunization coverage,61 and plans to track a set of key indicators such as deliveries and "stock-outs" over time, as it did with its pilot project.62

The full six-year plan does not call for surveys in a "comparison province" without VillageReach's support (which the original evaluation had). However, they do call for surveys of immunization coverage in each province, both when VillageReach enters and when it exits. VillageReach will be entering different provinces in different years,63 and depending on exactly when implementation and measurement end up occurring, we feel that the final figures could be quite revealing even without "comparison" provinces.

Because there has only been one demonstrated success, these activities should be considered to have a reasonable risk of failure, but they are - to us - clearly good investments because they are highly similar to activities that have worked before, and we believe VillageReach has made a credible commitment to continue documenting their success or failure.

Progress to date

VillageReach shared with us a 2010 baseline vaccination coverage survey from Cabo Delgado and a six-month report on progress on key indicators. It has told us that it has also conducted a baseline survey in a second province, Niassa, though we have not yet seen this. (VillageReach earlier told us that baseline surveys would be publicly available for 3 provinces by year-end 2010,64 but is behind schedule on project roll out - reasons for this are discussed above.) From these documents, we note:

• Data collection seems to have improved,65 despite a major data collection issue in October 2010 that VillageReach has informed us may have been due to an ongoing election.
• Other indicators, including the particularly important "stockouts" indicator (VillageReach aims to reduce the frequency with which health centers run out of vital medical supplies), don't show noticeable improvement.66 We don't find this cause for major concern, as VillageReach's successful (in our view) pilot project also didn't show major improvement at the six-month mark (above).
• A survey has been conducted in Cabo Delgado to assess the existing level of vaccination coverage. Overall immunization has fallen only slightly since the 2008 end-of-project survey for VillageReach's pilot project in the province (which ended in 2007), but it has fallen significantly for children under the age of 12 months.67 The fact that the overall coverage rate (which includes some people immunized under VillageReach's model) has fallen slightly while the under-12-month coverage rate (only those immunized recently) has fallen a lot is consistent with the hypothesis that the switch away from VillageReach's model has led to inferior immunization coverage (though the previous, more effective system still affects the overall coverage rate).
• There appears to have been a problem with the flow of funds from a funder to the government. In the short run, this has resulted in VillageReach's lending money to the government to cover for a delay.68 This problem does not appear to directly involve VillageReach, and VillageReach representatives told us they are unclear on the exact nature of the problem, but they stated that they expect a relatively quick resolution (within a few months) and have included a mention of the issue in the six-month evaluation in the hopes of raising the profile of the problem and raising the odds of its quick resolution.
• Supply of pentavalent vaccine (which includes diphtheria-tetanus-pertussis, the vaccinations we focus on in our calculation of lives saved) has become disrupted due to a global problem. VillageReach provided this quote from a GAVI (The Global Alliance for Vaccines and Immunisation) representative:
  
  Yes, there is a global shortage of pentavalent vaccines which is expected to be resolved by Q3 of 2011. Countries may be required to use their buffer stocks this year. UNICEF is trying their best to manage the situation. The shortage has occurred due to withdrawal of WHO pre-qualification status of one of the suppliers last year due to quality issues.

  Unfortunately, this supplier has not been successful in regaining that status. New suppliers have been pre-qualified by WHO this year but they require lead time to produce the volume necessary to match the demand. So, in short, Mozambique is not the only country affected but all GAVI countries are affected by this.

  This issue may be partly responsible for the particularly high stock out rates for this particular vaccine.69

• Overall, we found the provided materials to be exceptionally clear and straightforward about VillageReach's progress, struggles and future plans.
• The six-month report does not present a chart of each of VillageReach's core indicators; for example, rather than showing the percentage of health centers with a stock-out of at least one vaccine (which is VillageReach's official definition of its "stock outs" indicator70 ), it breaks out stockouts by individual vaccine and does not include data on non-vaccine supplies.71 VillageReach made the case to us over the phone that the charts shown are more informative at this stage of the project, while still making it clear that there has not been progress on key indicators to date, and we find this a reasonable argument, but we have requested that in the future, VillageReach chart all standard indicators along with whatever other charts it finds to be informative.

Contract engagements

We are less positive on VillageReach's contract engagements, many of which we have very little information about. The South Africa project that VillageReach must raise $100,000 in "matching donations" for is particularly worrisome to us, as we find the proposal for activities and impact assessment relatively vague.

Possible negative/offsetting impact

As stated at our discussion of impact analysis, we are generally concerned about charities' potential diversion of skilled labor and/or interference with government responsibilities. However, we believe these concerns are smaller with VillageReach than with other charities we've seen.

VillageReach's focus is on improving logistics rather than on increasing the available resources in an area. Its cost analysis argues that its program ultimately ends up saving the government money (more below), and a conversation with its representatives implies that it does not attempt to repurpose skilled labor from other areas or sectors.72 In addition, it appears to be seriously committed to handing off its programs to the government over time, as it did previously in Cabo Delgado. It does not appear to grant funds directly to governments.

What do you get for your dollar?

Past cost-effectiveness: pilot program

We do not attempt to quantify the full benefits of the VillageReach program. Instead, we observe that even a relatively conservative estimate of its cost per child vaccinated would imply quite strong cost-effectiveness (in terms of cost per death averted).

The Disease Control Priorites report (Jamison et al. 2006) estimates the cost per fully-immunized child with a basic set of vaccines at $14.21 in sub-Saharan Africa.73 According to Jamison et al. (2006), this implies a cost per death averted of approximately $200.74

• VillageReach's estimate: VillageReach sent us a draft of its internal review of the cost effectiveness of the Cabo Delgado pilot project. VillageReach estimates that its program is significantly more cost-effective than the government's program, at a cost of $5.76 per child receiving three doses of each DTP and hepatitis B vaccines (which VillageReach asserts is a proxy for a fully immunized child), including both VillageReach and government costs.75 This would imply a cost-per-death averted that is significantly lower than the DCP's estimate of $200, and thus easily within the range we consider highly cost-effective.
• GiveWell's conservative estimate: We assumed that all VillageReach costs are attributable to the Cabo Delgado project (including costs associated with Nampula activities, due to our uncertainty about these activities' impact), and that there is no impact of VillageReach on immunization coverage beyond 2008 (the last year for which we have data). We also assumed that starting in 2003, the difference in immunization coverage between Cabo Delgado and Niassa can be attributed to VillageReach's program. These assumptions yield an estimate of one additional child fully immunized for every ~$41 of VillageReach's expenses.76 If the Disease Control Priorities Report is correct to estimate that $15 per fully immunized child corresponds to $200 per death averted, this would imply that VillageReach is averting a child death for every ~$545 it spends, still well within the range discussed on our overview of cost-effectiveness estimates. (This estimate ignores government costs entirely, in order to give a sense of what is accomplished for donor money.)77

Note: In September 2011, we confirmed a number of errors in the estimates for the cost-effectiveness of deworming published in the Disease Control Priorities report. Based on those findings, we are currently rethinking our use of cost-effectiveness estimates, like the DCP2's, for which the full details of the calculations are not public. For more information, see our blog post on the topic.

Cost-effectiveness of future activities?

Further activities in Mozambique

VillageReach has provided estimates of the "incremental children vaccinated" projected for its six-year project.78 These assume that "Total children forecasted to be vaccinated with new system is 80% in the first year, 85% in the second year, and 90% in the third year. The exception is Cabo Delgado where it is 90% for all 3 years."79 These assumptions do not strike us as overaggressive, seeing as the pilot project achieved 95% coverage in Cabo Delgado (see above). Also note that these estimates count only incremental children vaccinated while VillageReach is active in a given area, and thus could substantially underestimate impact if VillageReach's work has lasting effects (as intended).

We have calculated the "cost per additional child vaccinated" based both on the Mozambique-only costs and on VillageReach's overall costs as an organization, excluding contract engagements (i.e., all activities that unrestricted funds support).

The aggregate cost-per-vaccination over the duration of the project is $31.88 looking at only Mozambique costs,80 and $58.28 when including other costs.81 If, as discussed above, $15 per additional immunized child corresponds to $200 per life saved, these two estimates would imply around $400 or $800 per life saved, respectively. Note that the latter estimate assumes that VillageReach activities such as IT development will have zero impact aside from its contribution to VillageReach's main logistics program in Mozambique.

The baseline survey in Cabo Delgado found existing coverage in-line with the VillageReach's projections that we have used in our cost-effectiveness estimate. Since the project has just begun, we don't know whether the project will come in over/under budget. We'll continue to watch these figures and update the review as new information becomes available.

Room for more funds?

As discussed above, VillageReach's current priority for unrestricted funds is its work in Mozambique. As of April 2011, we believe that VillageReach has "room for more funding" for the Mozambique project of approximately $1.03 million in above-expectations revenue in 2011, and we would cease recommending it to donors if it reached that threshold. Note that VillageReach's expected revenues do not include donations that might result from GiveWell's research.82

Details:

• VillageReach stated in February 2011 that "in order to commit to starting work in Inhambane in early FY2012 (i.e. Jan-March 2012 time frame), we would like to recognize funding by December 2011 that supports the program for the subsequent 12 months."83 According to our calculations, this would require that VillageReach receives about $200,000 in unrestricted 2011 revenue,84 and as of April 2011, we had tracked over this amount (about $360,000) in unrestricted donations since the start of the year.
• VillageReach hopes to expand to three more provinces as well, beginning in late 2012.85 Doing so will require an additional ~$3.2 million in funding;86 incorporating conservative projections of incoming revenue, we estimate an overall funding gap of ~$1.03 million.87 VillageReach states that additional revenue of $500,000-$1,000,000 would accelerate discussions with these additional provinces.88

In discussions with VillageReach, we have learned more about the factors that go into expanding to a new province, aside from funding. In addition to funding its own operations and getting sign-off from the local health authorities, VillageReach must ensure that there is sufficient government funding earmarked to helping to carry out its program89 and that it has sufficient capacity of its own, which can involve a hiring process.90 VillageReach representatives have stated to us that these factors are relatively straightforward to address when funding on its own end is available, but these issues are worth keeping in mind as potential non-financial bottlenecks to expansion.

It appears that donors who gave to VillageReach as a result of GiveWell's recommendation have played a significant role in removing certain financial barriers. Our understanding is that VillageReach's work in Gaza and Inhambane has been made possible by such donors.91 VillageReach received about $2 million in unrestricted revenue in 2010, of which a little over $1 million can be attributed to GiveWell's recommendation.92

Financials/other

All data comes from VillageReach's IRS form 990s for 2002-2008 and its audited financial statements for 2009 and 2010.93

Revenue and expense growth (about this metric): VillageReach reached a large five-year, $3.3 million grant agreement with the Gates Foundation in 2004,94 which explains the large jump reported revenues in 2004.95

Both revenues and expenses have grown since a low point in 2007 (possibly due to the fact that VillageReach had completed its work in Mozambique by that time and was largely focused on reviewing and evaluating that project).

Assets-to-expenses ratio (about this metric): VillageReach's assets to expenses ratio was low in 2009-2010. At the end of the 2010 fiscal year (which ends in September), VillageReach had only half a year's expenses on-hand. However, as discussed above, VillageReach raised substantially more revenue than expected in the final months of 2010, primarily as a result of GiveWell's recommendation.

Expenses by program area (about this metric): See discussion above.

Expenses by IRS-reported category (about this metric): VillageReach maintains a reasonable "overhead ratio," spending approximately 75-85% of its budget on program expenses.

Sources

• Barrett, Leah. VillageReach Program Manager. Email exchange with GiveWell (PDF), June 2009.
• Beale, John. VillageReach Director of Strategic Development. Email to GiveWell, December 23, 2009.
• Beale, John. VillageReach Director of Strategic Development. Email to GiveWell (PDF), June 21, 2010.
• Beale, John. VillageReach Director of Strategic Development. Email to GiveWell (PDF), July 2, 2010.
• Beale, John, Allen Wilcox, and Becca Miller. VillageReach Director of Strategic Development, President, and Finance and Program Administration Manager. Phone conversation with GiveWell, May 21, 2009.
• Bill and Melinda Gates Foundation. VillageReach. http://www.gatesfoundation.org/Grants-2004/Pages/VillageReach-OPP30874… (accessed April 26, 2010). Archived by WebCite® at http://www.webcitation.org/5pHHJonys.
• GAVI Alliance. Country information: Mozambique. http://www.gavialliance.org/performance/country_results/index.php?count… (accessed April 15, 2011). Archived by WebCite® at http://www.webcitation.org/5xxyuAW9t.
• GiveWell. Expanding immunization coverage for children.
• Government of Mozambique. Phase 2 Proposal to GAVI Alliance (2006) (PDF).
• Jamison, Dean T., et al., eds. 2006. Disease control priorities in developing countries (PDF). 2nd ed. New York: Oxford University Press.
• Kane, Mark. 2008. Evaluation of the project to support PAV (expanded program on immunization) in northern Mozambique, 2001-2008: An independent review for VillageReach with program and policy recommendations (PDF). Seattle: VillageReach.
• Leach-Kemon, Katie, Mariana Dionísio, and Nelia Taimo. 2008. Evaluation of the project to support PAV (expanded program on immunization) in northern Mozambique, 2001-2008: Statistical analysis (PDF). Seattle: VillageReach.
• Miller, Becca. VillageReach Finance and Program Administration Manager. Email exchange with GiveWell, December 23, 2009.
• Miller, Becca. VillageReach Finance and Program Administration Manager. Email to GiveWell (PDF), December 18, 2009.
• Measure DHS. Statcompiler. http://www.statcompiler.com (accessed June 30, 2009). Archived by WebCite® at http://www.webcitation.org/5tvz9ansa.
• UNICEF. Mozambique: Immunization plus. http://www.unicef.org/mozambique/activities_1869.html (accessed April 15, 2011). PDF "print out".
• VillageReach. About VillageReach. http://villagereach.org/about-us/about-villagereach/ (accessed April 23, 2010). Archived by WebCite® at http://www.webcitation.org/5pCYUtlLc.
• VillageReach. Audited financial statements:
  • 2009 (PDF).
  • 2010 (PDF).
• VillageReach. Board of Directors and Advisors. http://villagereach.net/about-us/board-of-directors-advisors/ (accessed April 23, 2010). Archived by WebCite® at http://www.webcitation.org/5pDGzQgrO.
• VillageReach. Cabo Delgado: Six-month process evaluation (February 2011). VillageReach has not yet cleared this document for public consumption.
• VillageReach. Cabo Delgado: Vaccine coverage baseline survey (April 2010).
• VillageReach. Comparison of costs incurred in dedicated and diffused vaccination logistics systems: Cost-effectiveness of vaccine logistics in Cabo Delgado and Niassa provinces, Mozambique. Summary available (PDF). VillageReach has asked us not to post the full text online.
• VillageReach. Cost estimates (August 8, 2009) (XLS).
• VillageReach. Field programs. http://villagereach.org/what-we-do/field-programs/ (accessed July 27, 2010). Archived by WebCite® at http://www.webcitation.org/5rXZ0bVIB.
• VillageReach. Five year project report (PDF).
• VillageReach. Funding gap memo (PDF).
• VillageReach. Health system strengthening in Mozambique (PDF).
• VillageReach. IRS form 990:
  • 2002 (PDF).
  • 2003 (PDF).
  • 2004 (PDF).
  • 2005 (PDF).
  • 2006 (PDF).
  • 2007 (PDF).
  • 2008 (PDF).
• VillageReach. John Snow proposal. Currently withheld due to confidentiality request due to discussion of pending contracts. Interested individuals should contact VillageReach.
• VillageReach. Key indicator descriptions (PDF).
• VillageReach. Medicines for Malaria Ventures proposal. Currently withheld due to confidentiality request due to discussion of pending contracts. Interested individuals should contact VillageReach.
• VillageReach. Milestones (PDF).
• VillageReach. Mission report - VillageReach: Logistics support to health services - MISAU Mozambique (PDF).
• VillageReach. Monthly key indicator tabular report (PDF).
• VillageReach. Mozambique budget with GiveWell additions (February 2011, updated April 2011) (XLS).
• VillageReach. Mozambique expansion budget (JPG).
• VillageReach. Mozambique funding gap (February 2011) (XLS).
• VillageReach. Mozambique revised project plan (February 2011) (XLS).
• VillageReach. Mozambique update (February 2011) (DOC).
• VillageReach. Nampula indicators. We have not received permission to publish this document.
• VillageReach. Nigeria budget. Currently withheld due to confidentiality request due to discussion of pending contracts. Interested individuals should contact VillageReach.
• VillageReach. Northern Mozambique project. http://web.archive.org/web/20080726084956/http://www.villagereach.org/M… (accessed January 11, 2010).
• VillageReach. Organization budget (2010) (PDF).
• VillageReach. President's report (September 12, 2009). Currently withheld due to confidentiality request due to discussion of pending contracts. Interested individuals should contact VillageReach.
• VillageReach. Revenue attribution (2010) (XLS).
• VillageReach. South Africa proposal. Currently withheld due to confidentiality request due to discussion of pending contracts. Interested individuals should contact VillageReach.
• VillageReach. Supply chain. http://web.archive.org/web/20080630001351/www.villagereach.org/supply_c… (accessed January 11, 2010).
• Wilcox, Allen, and Becca Miller. VillageReach President, and Finance and Program Administration Manager. Phone conversation with GiveWell, November 10, 2009.
• World Health Organization. Glossary. http://www.who.int/immunization_monitoring/glossary/en/index.html (accessed April 23, 2010). Archived by WebCite® at http://www.webcitation.org/5pDJPLQ56.

• 1

  "VillageReach improves access to healthcare for remote, underserved communities around the world...VillageReach’s model improves access to healthcare by providing a logistics platform to facilitate delivery of medical supplies and by starting and managing social businesses to improve local infrastructure." VillageReach, "About VillageReach."

• 2

  "An efficient logistics system not only ensures that appropriate, high quality medicine and vaccines are available to those who need them most, but it also allows health workers to focus on provision of care for the community rather than logistics. A recent analysis of the VillageReach model in Mozambique found that in comparison to the ad hoc logistics system which is the default in most low-resource countries, VillageReach’s dedicated logistics system frees up 216 days of staff time per month, allowing for significant improvements in health worker productivity by removing unproductive time spent on vaccine logistics." VillageReach, "Field Programs."

• 3

  "VillageReach was founded in Seattle, Washington in 2000 by Blaise Judja-Sato." VillageReach, "About VillageReach."

• 4

  "In March 2002, a 5-year pilot project (extending from April 2002 to March 2007) was initiated. In the project, the FDC and VillageReach, in coordination with MISAU, the Expanded Program on Immunization (PAV4) and the Provincial Directorate of Health (DPS) of Cabo Delgado, distributed vaccines, gas, medicines and other essential medical supplies each month to all the heath facilities providing immunization. This included rural health facilities, which were often isolated from the normal distribution systems due to insufficient public infrastructure." Kane 2008, Pg 10.

• 5

  "April 2007 - We have transitioned the VillageReach model and program to the local Ministry of Health in Cabo Delgado province, home to 88 vaccination clinics. VillageReach and FDC, our local implementation partner, will continue to provide technical assistance and data reporting." VillageReach, "Milestones."

• 6

  "The Dedicated Logistics System (DLS) was reinitiated in the province of Cabo Delgado in June 2010." VillageReach, "Cabo Delgado Dedicated Logistics System Six-Month Process Evaluation (February 2011)."

• 7

  "In the project, the FDC and VillageReach, in coordination with MISAU, the Expanded Program on Immunization (PAV4) and the Provincial Directorate of Health (DPS) of Cabo Delgado, distributed vaccines, gas, medicines and other essential medical supplies each month to all the heath facilities providing immunization. This included rural health facilities, which were often isolated from the normal distribution systems due to insufficient public infrastructure." Kane 2008, Pg 10.

• 8

  VillageReach, "Northern Mozambique Project."

• 9
  • "The Project has already been replicated in Nampula, where it is currently being implemented by the FDC." Kane 2008, Pg 8.
  • "August 2006: Began deliveries in 85 health centers (in North and Central zones in12 districts) in Nampula." Kane 2008, Pg 16.

• 10

  "We officially transitioned Nampula province to FDC in January 2007, and they officially ended the project in August 2009- which means the technical assistance and support ended but the Ministry continues to do the activities." Becca Miller, email to GiveWell, December 18, 2009.

• 11

  See VillageReach, "Health System Strengthening in Mozambique," Pg 8.

• 12

  "Based on the documented success of the demonstration project, the Mozambique Minister of Health has formally directed officials in each of Mozambique’s provinces to pursue implementation of the model." VillageReach, "Health System Strengthening in Mozambique," Pg 7.

• 13

  "Specific program objectives [include] … Integrate additional key commodities – such as rapid diagnostic tests – into the dedicated logistics system." VillageReach, "Health System Strengthening in Mozambique," Pg 7.

• 14

  VillageReach, "Cabo Delgado: Six-Month Process Evaluation (February 2011)," Pgs 3-4.

• 15

  VillageReach, "Cabo Delgado: Six-Month Process Evaluation (February 2011)," Pg 3.

• 16

  "Baseline assessments have been completed in both Cabo Delgado and Niassa and distributions are well underway." VillageReach, "Mozambique Update (February 2011)," Pg 1. The timeline in VillageReach, "Mozambique Revised Project Plan (February 2011)" indicated that the "implementation phase" in Niassa began in the first quarter of 2011.

• 17

  "VillageReach is active in Cabo Delgado, Niassa and Maputo. The initial start last year in these provinces was slower than had been anticipated, primarily due to concerns about funding and some delays in dialogue with the government, but we now have good progress in all three provinces. Baseline assessments have been completed in both Cabo Delgado and Niassa and distributions are well underway ... In the last six weeks of 2010 VillageReach received funding from individual donors well beyond expectations. Until this time it had not been clear what level of activity would be possible for the Mozambique program in 2011.
  Because of this lack of visibility, we had not pursed any new DPS discussions late least year, except for Gaza, where [a funder] had already made it clear that they can provide funding for the Gaza provincial governments activities. Note that it can take some months of discussion with a provincial government before permission to work in the province is received and finances and personnel are allocated by the government." VillageReach, "Mozambique Update (February 2011)." Timeline for expansion in VillageReach, "Mozambique Revised Project Plan (February 2011)."

• 18

  VillageReach, "Mozambique Revised Project Plan (February 2011)."

• 19

  "The program, started in January 2010, is expected to cover eight of ten provinces over six years." VillageReach, "Health System Strengthening in Mozambique," Pg 3.

• 20

  VillageReach, "Mozambique Revised Project Plan (February 2011)."

• 21

  "We had not pursed any new DPS discussions late least year, except for Gaza, where [a funder] had already made it clear that they can provide funding for the Gaza provincial governments activities. Note that it can take some months of discussion with a provincial government before permission to work in the province is received and finances and personnel are allocated by the government." VillageReach, "Mozambique Update (February 2011)," Pg 1.

• 22

  "After Gaza, we will work on Inhambane next, with outreach to the government there expected once we have hired a new program manager; we have identified a strong candidate. We estimate the start of those discussions with Inhambane Province in early summer 2011." VillageReach, "Mozambique Update (February 2011)," Pg 2.

• 23

  VillageReach, "South Africa Proposal," Pg 4.

• 24

  VillageReach, "South Africa Proposal," Pg 7.

• 25

  VillageReach, "South Africa Proposal," Pg 11.

• 26

  VillageReach, "South Africa Proposal," Pg 12.

• 27

  VillageReach, "South Africa Proposal," Pg 7.

• 28

  VillageReach, "South Africa Proposal," Pg 12.

• 29

  VillageReach, "Organization Budget (2010)."

• 30

  "This report was compiled by Mark Kane, MD, MPH, a consultant, following review of these materials and extensive discussions with VillageReach staff. Many of the opinions and viewpoints in this report are those of the reviewer, and do not necessarily represent the views of the VR, its staff, or Project implementation partners." Kane 2008, Pg 12. Dr. Kane is listed on VillageReach's Board of Advisors. See VillageReach, "Board of Directors and Advisors."

• 31

  "Under the previous system of distribution, clinic workers in need of vaccines and other medical supplies were required to travel many miles, often on foot, to a provincial or district warehouse to obtain supplies that were not always available. Today, in Cabo Delgado, health workers at 90 rural clinics receive monthly deliveries from one of VillageReach's three delivery trucks, specially outfitted to navigate the difficult terrain of rarely maintained roads and sustain the cold chain necessary for the safe transport of vaccines. As the VillageReach drivers leave from the provincial warehouse for two-week excursions, they bring with them the necessary vaccines, medical supplies, and energy needed by each clinic to serve their communities." VillageReach, "Supply Chain."

• 32

  Kane 2008, Pgs 17-18.

• 33

  VillageReach reports this as "DPTHpB," however, the more common way of referring to this vaccine seems to be "DTP/HepB" or the "Tetravalent" vaccine, i.e. a combination vaccine for diphtheria, tetanus, pertussis, and hepatitis B. (See UNICEF, "Mozambique: Immunization Plus," which notes, "The national EPI programme was recently bolstered by the introduction of the DPT-HepB vaccine (diptheria, pertussis, tetanus and hepatitis B)."

  A proposal submitted by the government of Mozambique to the GAVI Alliance in 2006 requested the following assistance:
  "Please list the vaccines to be introduced with support from the GAVI Alliance (and presentation):
  1. Pentavalent vaccine – DPT/Hep B + Hib – begin introduction (national target objective of 20%), using two doses lyophilized for Hib and two doses liquid for DPT/Hep B to be used also as a diluent.
  2. Continue Tetravalent – DPT/Hep B during 2007 (national target objective of 55%), using a 10 doses vial liquid for DPT/Hep B." Government of Mozambique, "Phase 2 Proposal to GAVI Alliance (2006)," Pg 18.
  According to GAVI Alliance, "Country Information: Mozambique," this proposal was approved for "NVS-Penta" in 2008.

• 34

  World Health Organization, "Glossary."

• 35

  Kane 2008, Pg 10.

• 36

  Kane 2008, Pgs 19-20.

• 37

  Kane 2008, Pg 24.

• 38

  Kane 2008, Pg 14.

• 39

  VillageReach, "Mission Report - VillageReach: Logistics Support to Health Services - MISAU Mozambique."

• 40

  VillageReach, "Mission Report - VillageReach: Logistics Support to Health Services - MISAU Mozambique," Pg 9, Table 1.1.

• 41

  VillageReach, "Mission Report - VillageReach: Logistics Support to Health Services - MISAU Mozambique," Pg 10, Table 2.1.

• 42

  VillageReach, "Mission Report - VillageReach: Logistics Support to Health Services - MISAU Mozambique," Pg 10, Table 2.2.

• 43

  "Comparison data was obtained from a 2007 immunization coverage cluster survey conducted by DPS in the neighboring province of Niassa, in which the Project did not operate." Kane 2008, Pg 6.

• 44

  "This study used data from the 2003 Mozambique Demographic and Health Survey (DHS 2003) as baseline data. DHS surveys are cross-sectional household surveys that are representative on both a national and provincial level...12,315 households were included in the study...The variable 'DTP 3' was computed using the following variables from the Mozambique DHS 2003 dataset: 'received DTP 1,' 'received DTP 2,' and 'received DTP 3.' 'DTP 3' was defined as those children who received all DTP doses (DTP 1-3) according to card or history." Leach-Kemon, Dionísio, and Taimo 2008, Pg 9.

• 45

  Leach-Kemon, Dionísio, and Taimo 2008, Pg 10.

• 46

  Leach-Kemon, Dionísio, and Taimo 2008, Pg 16.

• 47

  Kane 2008, Pg 23.

• 48

  Leah Barrett, email exchange with GiveWell, June 2009.

• 49

  "The Statistical Analysis for the quantitative surveys carefully describes the factors that could bias the evaluation results:

  • Baseline surveys were not done at the inception of the project
  • A “comparison” Province (or Provinces) was not designated at the initiation of the project
  • Comparing the results of surveys done with different methodologies (DHS and EPI cluster surveys) creates certain potential biases
  • Relatively small sample sizes made it difficult to detect small changes in coverage between the two age groups
  • Uncertainty about the reasons districts were chosen for the Niassa survey
  • Uncertainty about the comparability of Niassa as a “comparison” province."

  Kane 2008, Pg 23.

• 50

  Kane 2008, Pg 24.

• 51

  Kane 2008, Pg 24.

• 52

  "You are right that a lot happened in Cabo Delgado between 1997-2001 and the coverage rates reflect that. Cabo Delgado was particularly hard hit by Mozambique’s civil war from 1977-1992, which the health system in a very poor state and landmines prevented people from traveling to the facilities that did exist. In 1994, Mozambique had their first multi-party elections and major rehabilitation efforts followed. In the 1997-2001 time period, there was a lot of effort put into building new health centers in Cabo Delgado, which greatly increased access to immunization services in the province." Leah Barrett, email exchange with GiveWell, June 2009.

• 53

  Measure DHS, "Statcompiler." We accessed data through the StatCompiler tool and looked at all available surveys for Sub-Saharan Africa.

• 54

  We downloaded data on DTP-3 immunizations. The percentages in the table reflect reports either from (a) the child's vaccination card or (b) a mother's report (called "either source" in the Measure DHS tables). We only include countries that had at least one survey during or before 1997 and at least one survey during or after 2003; 1997-2003 was the period over which VillageReach provided Measure DHS surveys for Cabo Delgado.

• 55

  "We have transitioned the VillageReach model and program to the local Ministry of Health in Cabo Delgado province, home to 88 vaccination clinics. VillageReach and FDC, our local implementation partner, will continue to provide technical assistance and data reporting." VillageReach, "Milestones."

• 56

  Kane 2008, Pg 26.

• 57

  John Beale, Allen Wilcox, and Becca Miller, phone conversation with GiveWell, May 21, 2009.

• 58

  "Baseline assessments have been completed in both Cabo Delgado and Niassa and distributions are well underway." VillageReach, "Mozambique Update (February 2011)," Pg 1.

• 59

  VillageReach, "Nampula Indicators."

• 60

  John Beale, email to GiveWell, December 23, 2009.

• 61

  "Measurement is a critical component of our methodology and model. An initial baseline survey and a concluding endline survey will be conducted for each provincial deployment. Data from these surveys will be publicly available upon approval from the government of Mozambique." VillageReach, "Health System Strengthening in Mozambique," Pg 3.

• 62

  See:

  • VillageReach, "Monthly Key Indicator Tabular Report," for a sample (not using real data) of the data it plans to collect.
  • VillageReach, "Key Indicator Descriptions," for a guide to the meanings of these indicators.
  • John Beale, email to GiveWell, June 21, 2010, for how frequently the indicators will be tracked.

• 63

  See individual province budgets in VillageReach, "Mozambique Expansion Budget."

• 64

  John Beale, phone conversation with GiveWell, July 23, 2010.

• 65

  See chart at VillageReach, "Six-Month Process Evaluation (February 2011)," Pg 5.

• 66

  See charts beginning at VillageReach, "Cabo Delgado: Six-Month Process Evaluation (February 2011)," Pg 11. The chart on stockouts is particularly core to the question of whether VillageReach is succeeding in improving the efficiency of the local health system.

• 67

  "The vaccine coverage rate was 89% for all children in the survey and 74.9% for children fully immunized below the age of 12 months This represent a reduction in vaccine coverage from 2008 to 2010 (children below 12 months of age) of nearly 18 percentage points, although the vaccine coverage including children above 12 months of age is only slightly lower than the high level of 92.6% in 2008 (endline survey for the phase of the 2002-2007 programme) and is above pre 2002 levels." VillageReach, "Cabo Delgado: Vaccine Coverage Baseline Survey (April 2010)," Pg 6.

• 68

  "A key requirement of the Dedicated Logistics System is that the monthly funding mechanism to support the field coordinators and the distribution schedule are in place and operating without the need for external support. The distribution costs are covered by a partnership with [a funder], which results in full funding for the previously identified operational needs. However, discussions during the evaluation revealed that there are some challenges in the flow of funds from [a funder] to DPS related to the larger agreement between the partners (i.e. not limited to Dedicated Logistics System activities) that affected the availability of per diems for the November distribution. Additionally, in February the funds were not available for the distributions and DPS relied upon a loan from VillageReach to support the costs. Prior to this problem, the funding of the monthly distributions was working well. Without fixing the problem, there is a significant risk to the operations and outcomes of the Dedicated Logistics System." VillageReach, "Cabo Delgado: Six-Month Process Evaluation (February 2011)," Pg 5.

• 69

  VillageReach, "Cabo Delgado: Six-Month Process Evaluation (February 2011)," Pg 11.

• 70

  "Stock outs: Reports the number and percentage of health units having a stock level of 0 (termed a stock out event) of any particular vaccine, EPI-related supplies, or rapid diagnostic tests. This indicator measures how well the logistics system is functioning to supply the health unit with vaccines and associated commodities to meet communities’ need for a full-month. The target for stock outs is 0% of the health units reporting, but 5% is an acceptable level. We expect stock outs to increase up to 40% at the start of the implementation of the dedicated logistics system as demand for vaccines increases (due to increased supply). After 6 months of implementation, we expect stock outs to decrease to 5% or below." VillageReach, "Cabo Delgado: Six-Month Process Evaluation (February 2011)," Pg 21.

• 71

  See chart at VillageReach, "Cabo Delgado: Six-Month Process Evaluation (February 2011)," Pg 11.

• 72

  "The people that we hired initially were retired MOH employees; in Nampula they remained MOH employees and were just under our management for a period of time." John Beale, Allen Wilcox, and Becca Miller, phone conversation with GiveWell, May 21, 2009.

• 73

  Jamison et al. 2006, Pg 401. For more, see our discussion of the cost-effectiveness of immunization programs.

• 74

  Jamison et al. 2006, Pg 401. For more, see our discussion of the cost-effectiveness of immunization programs.

• 75

  VillageReach, "Comparison of Costs Incurred in Dedicated and Diffused Vaccine Logistics Systems."

• 76

  For Cabo Delgado, we have (a) the number of children receiving 3 doses of DTP (which Kane 2008, following GAVI, asserts is a proxy for "fully immunized") in 2001-2007 (Kane 2008, Pg 19) and (b) the percentage of children immunized in 2003 and 2008 (Kane 2008, Pg 23). For Niassa, we have the percentage of children immunized in 2003 and 2008 (Kane 2008, Pg 23).

  The use of DTP-3 as a proxy for full immunization is supported by data from Cabo Delgado (reported by Leach-Kemon, Dionísio, and Taimo 2008, Pg 63, Table 1-11). Coverage rates of BCG, polio 3, and measles vaccinations were at least as high as DTP-3 coverage. In addition, 92.8% of children in the 24-35 months age group were "fully-vaccinated," while 95.4% were immunized with DTP-3.

  We credit VillageReach with each child immunized in Cabo Delgado above and beyond the percentage immunized in Niassa for the same year, starting in 2003 (the year after the VillageReach program began), interpolating data (linearly) when it is missing. While far from ideal, we find this a reasonably appropriate comparison, as we estimate the Cabo Delgado coverage rate prior to VillageReach's involvement to have been close to the Niassa coverage rate (see Leah Barrett, email exchange with GiveWell, June, 2009).

  Year	CD: # of children vaccinated	CD: % of children immunized	Niassa: % of children immunized	Children in Cabo Delgado needing immunizations	Calculated: Additional children immunized in CD
  2003	50,588	69%	55%	73,316	10,264
  2004	54,584	76%	58%	72,297	12,652
  2005	58,861	82%	61%	71,782	15,074
  2006	64,772	89%	64%	73,189	17,931
  2007	71,044	95%	67%	74,783	20,939
  2008	71,250	95%	70%	75,000	18,750

  • Immunization rates: It appears that the program deteriorated after VillageReach handed it over to the government in 2007 (see above). We don't know how much immunization rates fell, so to err on the conservative side, we assign the same rate to 2007 that is assigned to 2008. We linearly interpolate the immunization rate between 2003-2007 (in Cabo Delgado) and 2003-2008 (in Niassa).
  • Children in need of vaccination: Calculated as follows: (# of children immunized in Cabo Delgado) / (% of children immunized in Cabo Delgado).
  • Addtional children immunized: Calculated as follows: [(% of children immunized in Cabo Delgado) - (% of children immunized in Niassa)] * (Total number of children in need of immunizations in Cabo Delgado).

  Our total impact estimate comes to 95,610 additional children immunized (we do not apply a discount rate; a moderate discount rate would make little difference due to the short time period under discussion).

  Costs are detailed in VillageReach, "Cost Estimates (August 17, 2009)." We include all costs, even those allocated to the Nampula activities, for which we assume no impact. Between 2001 and 2006 (the year VillageReach stopped financially supporting the program in Cabo Delgado), VillageReach spent a total of $3,910,411, implying (with the impact estimate above) $40.90 spent by VillageReach for every additional child immunized between 2003-2008.

• 77

  If one assumes that the government spends an additional $15 per child - consistent with the Jamison et al. (2006) estimate of the costs for a standard expansion program, and probably an overstatement (since some of VillageReach's costs likely substitute for government costs) - the implied total cost per death averted rises to ~$745, still well within the range discussed on our overview of cost-effectiveness estimates.

• 78

  VillageReach, "Health System Strengthening in Mozambique," Pg 7.

• 79

  VillageReach, "Health System Strengthening in Mozambique," Pg 7.

• 80
  • Total projected cost for Mozambique: $5,874,588. VillageReach, "Mozambique Budget with GiveWell Additions (February 2011, updated April 2011)."
  • Total projected incremental children vaccinated: 184,281. VillageReach, "Health System Strengthening in Mozambique," Pg 7.
  • Cost per incremental child vaccinated: $5,874,588 / 184,281 = $31.88

• 81
  • Total projected cost for Mozambique: $5,874,588. (VillageReach, "Mozambique Budget with GiveWell Additions (February 2011, updated April 2011)") plus $810,844 per year for six years in non-Mozambique, non-contract costs (see above) = $10,739,652.
  • Total projected incremental children vaccinated: 184,281. VillageReach, "Health System Strengthening in Mozambique," Pg 7.
  • Cost per incremental child vaccinated: $10,739,652 / 184,281 = $58.28

• 82

  As noted in VillageReach, "Mozambique Budget with GiveWell Additions," GiveWell-influenced donations are not factored into the funding gap estimate (with the exception of donations that have already been made). VillageReach notes, "As you have advised us, because of GiveWell’s interest in identifying other leading charities, we should not expect to retain our #1 GiveWell rating through this year. We certainly understand and respect this and accordingly assume funding via GiveWell supporters will be much less for 2012 vs. 2011."

• 83

  VillageReach, "Mozambique Update (February 2011)," Pg 2.

• 84

  See VillageReach, "Mozambique Budget with GiveWell Additions (February 2011, updated April 2011)," cell H17, which shows costs for 2011-2012 including only the provinces of Cabo Delgado, Niassa, Maputo, Gaza and Inhambane, a total of $1,771,717. VillageReach, "Mozambique Funding Gap (February 2011)" shows about $1,592,491 available at the start of the year, implying a gap of $179,226 for 2011.

• 85

  "Note we would also expect to start work in Manica Province before the end of FY2012. Sofala and Tete would follow at the beginning of FY2013 (ie. late CY2012/early 2013)." VillageReach, "Mozambique Update (February 2011)," Pg 2.

• 86

  VillageReach, "Mozambique Budget with GiveWell Additions (February 2011, updated April 2011)," cell H18.

• 87

  See VillageReach, "Mozambique Budget with GiveWell Additions (February 2011, updated April 2011)," cell G22.

• 88

  "If we raise up to $500k more than is forecast by the end of CY2011: VillageReach will likely still pursue the program as presented in the project plan, although there might be some modest acceleration.

  If we raise $500k - $1M more than is forecast by the end of CY2011: VillageReach will likely accelerate planned discussions with the 7th and potentially 8th provinces.

  If we raise any amount beyond $1M more than is forecast by the end of CY2011:
  VillageReach may be able to accelerate discussions with the 8th province." VillageReach, "Mozambique Update (February 2011)," Pg 2.

• 89

  For example: "we had not pursed any new DPS discussions late least year, except for Gaza, where [a funder] had already made it clear that they can provide funding for the Gaza provincial governments activities. Note that it can take some months of discussion with a provincial government before permission to work in the province is received and finances and personnel are allocated by the government." VillageReach, "Mozambique Update (February 2011)," Pg 1.

• 90

  "After Gaza, we will work on Inhambane next, with outreach to the government there expected once we have hired a new program manager; we have identified a strong candidate. We estimate the start of those discussions with Inhambane Province in early summer 2011." VillageReach, "Mozambique Update (February 2011)," Pg 2.

• 91

  VillageReach states,

  You asked us for our thinking as to what funding amounts trigger what levels of activity …

  • If VillageReach had raised $500,000 less in late 2010:
    • VillageReach would still be active in Cabo Delgado, Niassa and Maputo.
    • VillageReach would still be planning to pursue expansion in Gaza when the opportunity arises, probably a few months from now.
    • VillageReach would not be planning on pursuing expansion beyond Gaza until/unless significant further funding came in.
  • If VillageReach had raised $1,000,000 less in late 2010:
    • VillageReach would still be active in Cabo Delgado, Niassa and Maputo.
    • VillageReach would not be planning on any further expansion unless significant further funding came in and the program would have been at serious risk of ending prematurely."

  VillageReach, "Mozambique Update (February 2011)," Pg 2.

  This is consistent with our analysis of VillageReach's budget, which is similar to the analysis we show in VillageReach, "Mozambique Budget with GiveWell Additions (February 2011, updated April 2011)."

• 92

  VillageReach, "Revenue Attribution (2010)."

• 93

  VillageReach, "IRS Form 990 (2002-2008)" and VillageReach, "Audited Financial Statements (2009-2010)."

• 94

  Bill and Melinda Gates Foundation, "VillageReach."

• 95

  Organizations report income on their tax forms in the year a grant agreement is reached. VillageReach received funds from this grant over the five-year period.