VillageReach - July 2010 Review

We have published a more recent review of this organization. See our most recent report on VillageReach.

Note: This review was published in July 2010. On December 20, 2010, we updated our discussion of VillageReach's funding needs.

VillageReach aims to improve the systems that distribute medical supplies to rural areas in Africa, so that life-saving supplies get to those who need them. Its programs include both technical support staff and changes in logistical setups (such as moving from a system in which health clinics collect their own supplies to a centralized delivery system).

VillageReach is a relatively small and young organization. We believe its activities have had and will have significant impact, under $1000 per infant death averted.

What do they do?

Broadly, VillageReach aims to improve the logistics - particularly delivery of medical supplies - for health systems in rural areas.1

Pilot project

VillageReach, which was founded in 2000,2 conducted a demonstration of its core logistics program in the Cabo Delgado province of Mozambique from April 2002 to March 2007.3 The Cabo Delgado project officially became the local government's responsibility in 2007,4 but VillageReach may soon resume responsibility due to problems under government management.

VillageReach's pilot project in Mozambique is the focus of our review because it is similar to the future activities most likely to be funded with donations. It included:5

• Transportation vehicles: "Created multi-modal transport network including land cruisers, motorcycles and bicycles. Staff inspects and repairs equipment on monthly visits."
• Cold chain: "Introduced reliable, low maintenance and cost-effective refrigerators in clinics."
• Injection safety equipment: "Installed propane burners for sterilization, incineration points and needle removers to ensure safe disposal of used syringes."
• Clinics' energy access: "Provided lighting for nighttime care, refrigerators, and sterilizers at clinics."
• Supplies-tracking: "Partnered with Iridium to utilize their global satellite system, introduced communication system in trucks to enable near, real-time inventory tracking."
• Training: "Trained community representatives to provide basic health care."
• Creating a for-profit enterprise to improve energy supply: "Established VidaGas, a Mozambican propane distribution company to reliably supply energy to clinics, businesses and households."

Nampula expansion

In 2006, VillageReach began to replicate the project in a second province in Mozambique, Nampula.6 It handed off the project to a local nonprofit in January 2007.7

Future activities

VillageReach's planned activities consist of:

• Further activities in Mozambique, expanding/replicating the model of the pilot project. These are the main activities for which VillageReach is seeking donations from individuals.
• A variety of contract engagements. VillageReach has contracted with specific parties to carry out projects in different parts of the world, all on the theme of improving health system logistics but sometimes different in many ways from the pilot project discussed above. With one exception, all of these projects are fully funded by the party in question, and VillageReach does not seek donations for them. As such, we have not deeply examined these programs (with the exception of the one for which donations are sought).
• An number of other initiatives, detailed below.

Further activities in Mozambique

VillageReach is currently hoping to implement its model across eight of Mozambique's ten provinces, over the next six years.8 For each province, it expects to have an active presence for three years, after which point it is hoped that the health system in that province will maintain its model without further support).9 VillageReach will emphasize the alternate health logistics system detailed above,10 while possibly applying it to new non-immunization-related supplies such as rapid diagnostic tests for malaria and other diseases.11

Currently, VillageReach has begun work in three provinces: Cabo Delgado (the province where the pilot project was carried out), Niassa and Maputo.12 An email sent in June 2010 from a VillageReach representative states that "Distribution has started in Cabo Delgado province ... Niassa Province distribution is scheduled to start in July."13

VillageReach states that funding will be the primary factor in expansion to other provinces,14 and that the national-level Ministry of Health has encouraged province-level Ministries of Health to adopt the VillageReach model.15

The total projected cost of this six-year project is about $5.6 million.16

Contract engagements

Specific parties have offered VillageReach funds to carry out projects in different parts of the world, all on the theme of improving health system logistics but sometimes different in many ways from the pilot project discussed above. Notes on these engagements follow. Projected expenses have not been updated since mid-2009 and therefore will not fully reconcile with other figures given below.

Some additional notes on the South Africa project, as this project is relevant to individual donors:

• VillageReach seeks to improve the general health system capacity in the Zululand district of the KwaZulu Natal province of South Africa17 through "business process mapping and optimization, streamlining data collection, staff training, mentoring, and supervision, expanding quality improvement methodology, and the identification of innovative uses for information and communication technology in patient care coordination at the clinic level."18
• VillageReach's specific plans for assessing its impact are unclear to us. It states that it plans on "A baseline and endline survey to gauge the outcomes and impact of the project" as well as "track[ing] output and outcome indicators that could include stock levels of key commodities, accuracy and frequency of data collection in the clinics, frequency of supervision visits, reports of data use and understanding by clinic staff, and satisfaction of clinic staff with the program."19 It is unclear to us exactly what the impact survey will consist of and, thus, how VillageReach will assess whether it has achieved its long-term goal of "80% improvement in identified primary healthcare indicators."20 On a more short-term basis, it is also unclear to us how VillageReach plans on assessing "accuracy and frequency of data collection in the clinics."21

Other plans

VillageReach's other planned activities include continuing to develop and share its management information system, pursuing new contract engagements, and ongoing support to VidaGas, a gas delivery business developed as part of its pilot program.

The following is its overall budget.22 Note that these figures are not taken from exactly the same source, time period and date as the contract engagement figures above.

Including the $100,000 matching requirement over three years detailed above (we count this as $33,333 for the next year), there is a total of $810,844 in expenses over the next year that are neither part of the Mozambique expansion nor included in contract engagements.

Does it work?

This section focuses on VillageReach's pilot project in Cabo Delgado, then briefly discusses its work in Nampula and what can be expected of its future activities.

One of VillageReach's primary methods of evaluating success is through tracking the progress made in administering basic immunizations. Such immunizations are a proven, cost-effective way to improve health and save lives in the developing world (more at our report on immunization), and so success in increasing immunization coverage - alone - likely constitutes, in our view, success in saving lives.

Below we examine evidence provided by VillageReach including (a) reporting on vaccines and equipment delivered, progress in vaccination coverage rates, and health clinic inventories; (b) an independent evaluation23 of VillageReach's impact. We conclude that VillageReach's pilot project has been effective in increasing vaccine coverage.

Delivery of vaccines and medical supplies

A key component of VillageReach's model is a shift from a "collection-based" to a "delivery-based" supply system: rather than clinics' being responsible for picking up their own supplies, VillageReach's logistics team delivers supplies and provides other logistical support.24

The tables below provide data on the goods VillageReach delivered to Cabo Delgado between 2004 and 2007.25 (Note that the project began in April 2002; we aren't sure why data has not been provided pre-2004.)

Increases in immunization coverage

The charts below show that (a) the number of children receiving DTP-3 (third dose of diphtheria-tetanus-pertussis vaccine26 ) immunizations increased; (b) the number of children who "dropped out" during the DTP-3 sequence - that is, they received one, but not all doses - fell; (c) reported "stock-outs" - centers with no inventory of the vaccine - fell significantly over the course of VillageReach's pilot project, which ran from April 2002 to March 2007.27 . Charts are taken from Kane 2008.28

Were improvements attributable to VillageReach?

The evaluation of the project notes that during this time period, "Mozambique and most sub-Saharan African countries achieved significant improvements in their DTP-3 coverage, probably due to GAVI and its support for infrastructure development and provision of new vaccines and safe injection equipment."29

To evaluate the question of VillageReach's role in improvements, we look at three types of information:

1. Reports from VillageReach about the problems interfering with immunizations in Cabo Delgado before their arrival.
2. An evaluation report published by VillageReach explicitly focused on addressing this question, comparing the change in immunization coverage rates in Cabo Delgado to that of another province in Mozambique, Niassa.
3. Examining the changes in immunization coverage in several countries in Africa over this period to put the observed change in context.

We also observe that the charts showing "stock-outs" above have stock-outs falling from a very high level prior to the start of the project to a very low level shortly after the project began. While it is possible that this change occurred for some reason other than VillageReach's involvement in the area, such an outcome seems unlikely.

VillageReach reports of pre-arrival conditions

VillageReach provides a report of the obstacles to immunization coverage in Cabo Delgado before its arrival. We would prefer to have better documentation of these conditions, but nevertheless, we believe the report offers some support to the idea that VillageReach's services were needed in Cabo Delgado.

VillageReach reports,

In 2002, before starting work, VillageReach performed an assessment of access to vaccines in Cabo Delgado province.31 This report claims that, in 2002, there were 22 health facilities in the three districts in Cabo Delgado that VillageReach assessed.32 Of these 22, 4 did not offer access to vaccination services:33 two facilities because they did not have access to a cold chain; 1 because it lacked personnel; and 1 for other reasons.34

VillageReach evaluation document

VillageReach compared improvements in vaccine delivery in Cabo Delgado to improvements in the nearby Niassa province, which was not served by VillageReach.35

• Baseline data collection: Because VillageReach did not have baseline data (i.e., data from a time prior to the start of the project) available for Niassa, it used data from the 1997 and 2003 Demographic and Health Surveys, which surveyed a large number of households and provided data on DTP-3 coverage.36
• Outcome data collection: Evaluators randomly selected households in the treatment area (Cabo Delgado province), including 474 children.37 Evaluators then also randomly selected households in the comparison area (Niassa province), thereby including 571 children in the evaluation.38
• Results: The study found that DTP-3 coverage rates increased substantially in both provinces during this period, but rates improved more in Cabo Delgado (treatment area) than Niassa (control area). The chart below shows the change in coverage rates in the two areas during this time period; the 1997 and 2003 data points are based on Demographic and Health Surveys, while the 2008 data points are from VillageReach's independent data collection.39

On one hand, we believe that this chart creates an inflated picture of VillageReach's impact. We have reason to believe that there were significant improvements in immunization coverage between 1997-2001 that were more related to Cabo Delgado's recovery from the aftermath of a civil war than to VillageReach's activities.40 However, the jump to extremely high levels of coverage as of 2008 - a change not mirrored in the nearby province - give some reason to attribute impact to VillageReach.

The evaluation report is forthright about many limitations of this comparison analysis, including limited sample size, uncertainty about the appropriateness of Niassa as a "comparison province," and issues with taking baseline and endpoint data from different sources.41 However, it says that its way of comparing the results of surveys done with different methodology "is consistent with international practice," and it concludes that it appears that the Project is responsible for the immunization coverage rising more in the treatment province, Cabo Delgado, than in the comparison province Niassa; although "additional information about the conditions in Niassa compared to those in Cabo Delgado is needed to better understand and interpret the comparison data."42 It also notes that "it is ... unlikely that the activities of other NGO's, which are not very involved in immunization activities in Cabo Delgado, were responsible for the improvement."43 (Note, however, that it does not discuss the confounding effect of the civil war recovery that we discussed above.)44

Our comparison of Cabo Delgado to other areas in the developing world

We took a broader look at changes in African immunization coverage over the time period in question in order to further investigate the idea that Cabo Delgado's improvements may simply have reflected a wider phenomenon. Using the Demographic and Health Surveys (Measure DHS),45 we collected data on DTP-3 immunizations for countries in Sub-Saharan Africa.46

The table below summarizes this data, sorted by the country's arithmetic percentage change in immunization coverage.

This table provides information at a country level rather than province level, and variation within countries could be significant. There does not appear to be strong evidence of a continent-wide positive trend in immunization rates, but it does not appear, on its own, to rule out the idea that the observed change in Cabo Delgado purely reflected a broader (country-wide or continent-wide) change. We note, however, that Cabo Delgado's 2003 coverage rate was slightly below the Mozambique overall rate, while its post-project rate was above the Mozambique overall rate (and above every other country's overall rate).

Bottom line on the Cabo Delgado program between 2001-2008

We do not feel that any of the pieces of evidence above is highly compelling by itself. But we are persuaded of VillageReach's impact by the combination of the observations that VillageReach's program (a) entered an area with clearly documented logistics problems; (b) reduced stockouts - one of the clearest measures of the logistics improvement it was aiming for - to near-zero levels; (c) brought Cabo Delgado from an "average" (for the country) level of coverage to an extremely high level of coverage; (d) was reported not to have been supplemented by other nonprofits' programs.

Cabo Delgado after 2008

Based on the evidence above, we feel that the VillageReach program improved capacity to deliver vaccines in Cabo Delgado. However, the Cabo Delgado project officially became the local government's responsibility in 2007,47 and a later report stated, "The data suggests that following the discontinuation of field coordinator teams delivering supplies and performing supervision, the districts and health centers are having difficulty reliably picking up supplies, stock-outs of vaccines are beginning to occur again, there is some (not statistically significant) evidence that immunization coverage is beginning to fall, and district level budgets are not being maintained for these activities."48 (Emphasis ours.) VillageReach's representatives stated to us that "When we turned it over to the government they let it slide back into the old collection-based system."49

The fact that VillageReach has been monitoring the program's continuing performance, and has been open about setbacks, is encouraging; but news of program deterioration is cause for concern. As discussed above, VillageReach has now reactivated its support role in Cabo Delgado.

Ultimately, we are skeptical about VillageReach's ambitions of handing over its model to the government. However, we note that VillageReach could be making lasting differences in individuals' lives even if its effects on health care are only temporary, since 1-3 doses of most vaccines are sufficient to immunize children against diseases. (Details in our report on immunization.) Our recommendation of VillageReach is made under the assumption that it will not succeed in getting its model adopted by the government, while recognizing that its ultimate cost-effectiveness would be much higher if it could.

Nampula activities

As described above, VillageReach briefly worked in the Nampula province of Mozambique before handing its activities off to a local organization in January 2007.

VillageReach provided us with internally collected data from this project through August 2008,50 and stated to us that the data became unreliable (due to internal contradictions) after that point.51 We have not yet received clearance to post the data publicly. Overall, it showed encouraging trends that resemble the trends outlined above: increasing numbers of immunizations and substantial drops in the rates of stockouts and other logistical problems. However, because of the fact that the data terminates at an apparently arbitrary point, we have serious doubts about the impact of this project.

This project's expenses, overall, were equal to about 20% of the expenses associated with the pilot project (details in our cost-effectiveness section). Based on multiple conversations with VillageReach representatives, it is our impression that the very limited role VillageReach took is not representative of VillageReach's typical or future activities.

Possible negative/offsetting impact

As stated at our discussion of impact analysis, we are generally concerned about charities' potential diversion of skilled labor and/or interference with government responsibilities. However, we believe these concerns are smaller with VillageReach than with other charities we've seen.

VillageReach's focus is on improving logistics rather than on increasing the available resources in an area. Its cost analysis argues that its program ultimately ends up saving the government money (more below), and a conversation with its representatives implies that it does not attempt to repurpose skilled labor from other areas or sectors.52 In addition, it appears to be seriously committed to handing off its programs to the government over time, as it has done in Cabo Delgado. It does not appear to grant funds directly to governments.

What can be expected of future activities?

As discussed above, the activities in Mozambique will be fundamentally similar to the pilot project, and evaluation will be similar as well, in that VillageReach will conduct before and after surveys of immunization coverage.53

The full six-year plan does not call for surveys in a "comparison province" without VillageReach's support (which the original evaluation had). However, they do call for surveys of immunization coverage in each province, both when VillageReach enters and when it exits. VillageReach will be entering different provinces in different years,54 and depending on exactly when implementation and measurement end up occurring, we feel that the final figures could be quite revealing even without "comparison" provinces. VillageReach has stated that baseline surveys will be publicly available for 3 provinces by year-end 2010.55

In addition, VillageReach plans on tracking a set of key indicators such as deliveries and "stock-outs" over time, as it did with its pilot project.56

Because there has only been one demonstrated success, these activities should be considered to have a reasonable risk of failure, but they are - to us - clearly good investments because they are highly similar to activities that have worked before, and we believe VillageReach has made a credible commitment to continue documenting their success or failure.

We are less positive on VillageReach's contract engagements, many of which we have very little information about. The South Africa project that VillageReach must raise $100,000 in "matching donations" for is particularly worrisome to us, as we find the proposal for activities and impact assessment relatively vague.

What do you get for your dollar?

Past cost-effectiveness: pilot program

We do not attempt to quantify the full benefits of the VillageReach program. Instead, we observe that even a relatively conservative estimate of its cost per child vaccinated would imply quite strong cost-effectiveness (in terms of cost per death averted).

The Disease Control Priorites report (Jamison et al. 2006) estimates the cost per fully-immunized child with a basic set of vaccines at $14.21 in sub-Saharan Africa.57 According to Jamison et al. (2006), this implies a cost per death averted of approximately $200.58

• VillageReach's estimate: VillageReach sent us a draft of its internal review of the Cabo Delgado project's cost-effectiveness. VillageReach estimates that its program is significantly more cost-effective than the government's program, at a cost of $5.76 per child receiving three doses of each DTP and hepatitis B vaccines (which VillageReach asserts is a proxy for a fully immunized child), including both VillageReach and government costs.59 This would imply a cost-per-death averted that is significantly lower than the DCP's estimate of $200, and thus easily within the range we consider highly cost-effective.
• GiveWell's conservative estimate: We assumed that all VillageReach costs are attributable to the Cabo Delgado project (including costs associated with Nampula activities, due to our uncertainty about these activities' impact), and that there is no impact of VillageReach on immunization coverage beyond 2008 (the last year for which we have data). We also assumed that starting in 2003, the difference in immunization coverage between Cabo Delgado and Niassa can be attributed to VillageReach's program. These assumptions yield an estimate of one additional child fully immunized for every ~$41 of VillageReach's expenses.60 If the Disease Control Priorities Report is correct to estimate that $15 per fully immunized child corresponds to $200 per death averted, this would imply that VillageReach is averting a child death for every ~$545 it spends, still well within the range discussed on our overview of cost-effectiveness estimates. (This estimate ignores government costs entirely, in order to give a sense of what is accomplished for donor money.)61

Cost-effectiveness of future activities?

Further activities in Mozambique

VillageReach has provided estimates of the "incremental children vaccinated" projected for its six-year project.62 These assume that "Total children forecasted to be vaccinated with new system is 80% in the first year, 85% in the second year, and 90% in the third year. The exception is Cabo Delgado where it is 90% for all 3 years."63 These assumptions do not strike us as overaggressive, seeing as the pilot project achieved 95% coverage in Cabo Delgado (see above). Also note that these estimates count only incremental children vaccinated while VillageReach is active in a given area, and thus could substantially underestimate impact if VillageReach's work has lasting effects (as intended).

We have calculated the "cost per additional child vaccinated" based both on the Mozambique-only costs and on VillageReach's overall costs as an organization, excluding contract engagements (i.e., all activities that unrestricted funds support).

The aggregate cost-per-vaccination over the duration of the project is $30.58 looking at only Mozambique costs,64 and $56.98 when including other costs.65 If, as discussed above, $15 per additional immunized child corresponds to $200 per life saved, these two estimates would imply around $400 or $800 per life saved, respectively. Note that the latter estimate assumes that VillageReach activities such as IT development will have zero impact.

Room for more funds?

As discussed above, VillageReach's current priority for unrestricted funds is its work in Mozambique. As stated in the 2010 budget, this work is fully funded for the next year, but faces a funding gap of $804,848 for 2011 and $4,391,732 for the six years as a whole.66

VillageReach has stated to us that it can productively absorb up to $1.5 million in 2010, and that if it did take in this much revenue, it would likely accelerate the start of three new provinces by 3-6 months.67

December 20, 2010 update:

As of December 20, VillageReach has received $509,109 towards its total fiscal year 2011 budget of $1,029,848, leaving VillageReach with a 2011 funding gap of $520,739. (Note that as discussed above, VillageReach has stated it can productively absorb up to about $1.5 million in 2010 - i.e., it could productively absorb more than the budgeted amount for 2011.) VillageReach has a $4,060,371 funding gap for the entire six-year projected Mozambique expansion.

Financials/other

All data comes from VillageReach's IRS form 990s for 2002-2008.68

Revenue and expense growth (about this metric): VillageReach reached a large five-year, $3.3 million grant agreement with the Gates Foundation in 2004,69 which explains the large jump reported revenues in 2004.70

In 2007, both revenues and expenses fell. It's possible that this is because VillageReach had completed its work in Mozambique and was largely focused on reviewing and evaluating that project.

Assets-to-expenses ratio (about this metric): VillageReach has maintained an assets:expenses ratio of between approximately 1:1 and 2:1, aside from the year (and year after) they received the Gates Foundation grant.

Expenses by program area (about this metric): See discussion above.

Expenses by IRS-reported category (about this metric): VillageReach maintains a reasonable "overhead ratio", spending approximately 70-80% of its budget on program expenses.

Sources

• Barrett, Leah. VillageReach Program Manager. Email exchange with GiveWell (PDF), June 2009.
• Beale, John. VillageReach Director of Strategic Development. Email to GiveWell, December 23, 2009.
• Beale, John. VillageReach Director of Strategic Development. Email to GiveWell (PDF), June 21, 2010.
• Beale, John. VillageReach Director of Strategic Development. Email to GiveWell (PDF), July 2, 2010.
• Beale, John, Allen Wilcox, and Becca Miller. VillageReach Director of Strategic Development, President, and Finance and Program Administration Manager. Phone conversation with GiveWell, May 21, 2009.
• Bill and Melinda Gates Foundation. VillageReach. http://www.gatesfoundation.org/Grants-2004/Pages/VillageReach-OPP30874… (accessed April 26, 2010). Archived by WebCite® at http://www.webcitation.org/5pHHJonys.
• GiveWell. Expanding immunization coverage for children.
• Jamison, Dean T., et al., eds. 2006. Disease control priorities in developing countries (PDF). 2nd ed. New York: Oxford University Press.
• Kane, Mark. 2008. Evaluation of the project to support PAV (expanded program on immunization) in northern Mozambique, 2001-2008: An independent review for VillageReach with program and policy recommendations (PDF). Seattle: VillageReach.
• Leach-Kemon, Katie, Mariana Dionísio, and Nelia Taimo. 2008. Evaluation of the project to support PAV (expanded program on immunization) in northern Mozambique, 2001-2008: Statistical analysis (PDF). Seattle: VillageReach.
• Miller, Becca. VillageReach Finance and Program Administration Manager. Email exchange with GiveWell, December 23, 2009.
• Miller, Becca. VillageReach Finance and Program Administration Manager. Email to GiveWell (PDF), December 18, 2009.
• Measure DHS. Statcompiler. http://www.statcompiler.com (accessed June 30, 2009). Archived by WebCite® at http://www.webcitation.org/5tvz9ansa.
• VillageReach. About VillageReach. http://villagereach.org/about-us/about-villagereach/ (accessed April 23, 2010). Archived by WebCite® at http://www.webcitation.org/5pCYUtlLc.
• VillageReach. Board of Directors and Advisors. http://villagereach.net/about-us/board-of-directors-advisors/ (accessed April 23, 2010). Archived by WebCite® at http://www.webcitation.org/5pDGzQgrO.
• VillageReach. Comparison of costs incurred in dedicated and diffused vaccination logistics systems: Cost-effectiveness of vaccine logistics in Cabo Delgado and Niassa provinces, Mozambique. Summary available (PDF). VillageReach has asked us not to post the full text online.
• VillageReach. Cost estimates (August 8, 2009) (XLS).
• VillageReach. Field programs. http://villagereach.org/what-we-do/field-programs/ (accessed July 27, 2010). Archived by WebCite® at http://www.webcitation.org/5rXZ0bVIB.
• VillageReach. Five year project report (PDF).
• VillageReach. Funding gap memo (PDF).
• VillageReach. Health system strengthening in Mozambique (PDF).
• VillageReach. IRS form 990:
  • 2002 (PDF).
  • 2003 (PDF).
  • 2004 (PDF).
  • 2005 (PDF).
  • 2006 (PDF).
  • 2007 (PDF).
• VillageReach. John Snow proposal. Currently withheld due to confidentiality request due to discussion of pending contracts. Interested individuals should contact VillageReach.
• VillageReach. Key indicator descriptions (PDF).
• VillageReach. Medicines for Malaria Ventures proposal. Currently withheld due to confidentiality request due to discussion of pending contracts. Interested individuals should contact VillageReach.
• VillageReach. Milestones (PDF).
• VillageReach. Mission report - VillageReach: Logistics support to health services - MISAU Mozambique (PDF).
• VillageReach. Monthly key indicator tabular report (PDF).
• VillageReach. Mozambique expansion budget (JPG).
• VillageReach. Nampula indicators. We have not received permission to publish this document.
• VillageReach. Nigeria budget. Currently withheld due to confidentiality request due to discussion of pending contracts. Interested individuals should contact VillageReach.
• VillageReach. Northern Mozambique project. http://web.archive.org/web/20080726084956/http://www.villagereach.org/M… (accessed January 11, 2010).
• VillageReach. Organization budget (2010) (PDF).
• VillageReach. President's report (September 12, 2009). Currently withheld due to confidentiality request due to discussion of pending contracts. Interested individuals should contact VillageReach.
• VillageReach. South Africa proposal. Currently withheld due to confidentiality request due to discussion of pending contracts. Interested individuals should contact VillageReach.
• VillageReach. Supply chain. http://web.archive.org/web/20080630001351/www.villagereach.org/supply_c… (accessed January 11, 2010).
• Wilcox, Allen, and Becca Miller. VillageReach President, and Finance and Program Administration Manager. Phone conversation with GiveWell, November 10, 2009.
• World Health Organization. Glossary. http://www.who.int/immunization_monitoring/glossary/en/index.html (accessed April 23, 2010). Archived by WebCite® at http://www.webcitation.org/5pDJPLQ56.

• 1

  "For nearly ten years, VillageReach's health systems strengthening programs have maintained a clear focus on the last mile healthcare access. Our approach focuses on the logistics of medical supply and service delivery to increase reliable access to healthcare in the poorest, most remote locations." VillageReach, "About VillageReach."

• 2

  "VillageReach was founded in Seattle, Washington in 2000 by Blaise Judja-Sato." VillageReach, "About VillageReach."

• 3

  Kane 2008, Pg 10.

• 4

  "We have transitioned the VillageReach model and program to the local Ministry of Health in Cabo Delgado province, home to 88 vaccination clinics. VillageReach and FDC, our local implementation partner, will continue to provide technical assistance and data reporting." VillageReach, "Milestones."

• 5

  VillageReach, "Northern Mozambique Project."

• 6
  • "The Project has already been replicated in Nampula, where it is currently being implemented by the FDC." Kane 2008, Pg 8.
  • "August 2006: Began deliveries in 85 health centers (in North and Central zones in12 districts) in Nampula." Kane 2008, Pg 16.

• 7

  "We officially transitioned Nampula province to FDC in January 2007, and they officially ended the project in August 2009- which means the technical assistance and support ended but the Ministry continues to do the activities." Becca Miller, email to GiveWell, December 18, 2009.

• 8

  "The program, started in January 2010, is expected to cover eight of ten provinces over six years." VillageReach, "Health System Strengthening in Mozambique," Pg 3.

• 9

  See VillageReach, "Health System Strengthening in Mozambique," Pg 8.

• 10

  "Based on the documented success of the demonstration project, the Mozambique Minister of Health has formally directed officials in each of Mozambique's provinces to pursue implementation of the model." VillageReach, "Health System Strengthening in Mozambique," Pg 7.

• 11

  "Specific program objectives [include] ... Integrate additional key commodities – such as rapid diagnostic tests – into the dedicated logistics system." VillageReach, "Health System Strengthening in Mozambique," Pg 7.

• 12

  "The three provinces of Cabo Delgado, Maputo and Niassa are already engaged in the preparation and implementation phases of the program. We expect to extend the program to Gaza, Inhambane, Manica, Sofala and Tete provinces." VillageReach, "Health System Strengthening in Mozambique," Pg 7.

• 13

  John Beale, email to GiveWell, June 21, 2010.

• 14

  "The program is being launched in three provinces in 2010. Funding is the chief determinate of deploying the model in future provinces." VillageReach, "Health System Strengthening in Mozambique," Pg 3.

• 15

  "Based on the documented success of the demonstration project, the Mozambique Minister of Health has formally directed officials in each of Mozambique's provinces to pursue implementation of the model." VillageReach, "Health System Strengthening in Mozambique," Pg 7.

• 16

  VillageReach, "Mozambique Expansion Budget."

• 17

  VillageReach, "South Africa Proposal," Pg 4.

• 18

  VillageReach, "South Africa Proposal," Pg 7.

• 19

  VillageReach, "South Africa Proposal," Pg 12.

• 20

  VillageReach, "South Africa Proposal," Pg 7.

• 21

  VillageReach, "South Africa Proposal," Pg 12.

• 22

  VillageReach, "Organization Budget (2010)."

• 23

  "This report was compiled by Mark Kane, MD, MPH, a consultant, following review of these materials and extensive discussions with VillageReach staff. Many of the opinions and viewpoints in this report are those of the reviewer, and do not necessarily represent the views of the VR, its staff, or Project implementation partners." Kane 2008, Pg 12. Dr. Kane is listed on VillageReach's Board of Advisors. See VillageReach, "Board of Directors and Advisors."

• 24

  "Under the previous system of distribution, clinic workers in need of vaccines and other medical supplies were required to travel many miles, often on foot, to a provincial or district warehouse to obtain supplies that were not always available. Today, in Cabo Delgado, health workers at 90 rural clinics receive monthly deliveries from one of VillageReach's three delivery trucks, specially outfitted to navigate the difficult terrain of rarely maintained roads and sustain the cold chain necessary for the safe transport of vaccines. As the VillageReach drivers leave from the provincial warehouse for two-week excursions, they bring with them the necessary vaccines, medical supplies, and energy needed by each clinic to serve their communities." VillageReach, "Supply Chain."

• 25

  Kane 2008, Pgs 17-18.

• 26

  World Health Organization, "Glossary."

• 27

  Kane 2008, Pg 10.

• 28

  Kane 2008, Pgs 19-20.

• 29

  Kane 2008, Pg 24.

• 30

  Kane 2008, Pg 14.

• 31

  VillageReach, "Mission Report - VillageReach: Logistics Support to Health Services - MISAU Mozambique."

• 32

  VillageReach, "Mission Report - VillageReach: Logistics Support to Health Services - MISAU Mozambique," Pg 9, Table 1.1.

• 33

  VillageReach, "Mission Report - VillageReach: Logistics Support to Health Services - MISAU Mozambique," Pg 10, Table 2.1.

• 34

  VillageReach, "Mission Report - VillageReach: Logistics Support to Health Services - MISAU Mozambique," Pg 10, Table 2.2.

• 35

  "Comparison data was obtained from a 2007 immunization coverage cluster survey conducted by DPS in the neighboring province of Niassa, in which the Project did not operate." Kane 2008, Pg 6.

• 36

  "This study used data from the 2003 Mozambique Demographic and Health Survey (DHS 2003) as baseline data. DHS surveys are cross-sectional household surveys that are representative on both a national and provincial level...12,315 households were included in the study...The variable 'DTP 3' was computed using the following variables from the Mozambique DHS 2003 dataset: 'received DTP 1,' 'received DTP 2,' and 'received DTP 3.' 'DTP 3' was defined as those children who received all DTP doses (DTP 1-3) according to card or history." Leach-Kemon, Dionísio, and Taimo 2008, Pg 9.

• 37

  Leach-Kemon, Dionísio, and Taimo 2008, Pg 10.

• 38

  Leach-Kemon, Dionísio, and Taimo 2008, Pg 16.

• 39

  Kane 2008, Pg 23.

• 40

  Leah Barrett, email exchange with GiveWell, June 2009.

• 41

  "The Statistical Analysis for the quantitative surveys carefully describes the factors that could bias the evaluation results:

  • Baseline surveys were not done at the inception of the project
  • A “comparison” Province (or Provinces) was not designated at the initiation of the project
  • Comparing the results of surveys done with different methodologies (DHS and EPI cluster surveys) creates certain potential biases
  • Relatively small sample sizes made it difficult to detect small changes in coverage between the two age groups
  • Uncertainty about the reasons districts were chosen for the Niassa survey
  • Uncertainty about the comparability of Niassa as a “comparison” province."

  Kane 2008, Pg 23.

• 42

  Kane 2008, Pg 24.

• 43

  Kane 2008, Pg 24.

• 44

  "You are right that a lot happened in Cabo Delgado between 1997-2001 and the coverage rates reflect that. Cabo Delgado was particularly hard hit by Mozambique's civil war from 1977-1992, which the health system in a very poor state and landmines prevented people from traveling to the facilities that did exist. In 1994, Mozambique had their first multi-party elections and major rehabilitation efforts followed. In the 1997-2001 time period, there was a lot of effort put into building new health centers in Cabo Delgado, which greatly increased access to immunization services in the province." Leah Barrett, email exchange with GiveWell, June 2009.

• 45

  Measure DHS, "Statcompiler." We accessed data through the StatCompiler tool and looked at all available surveys for Sub-Saharan Africa.

• 46

  We downloaded data on DTP-3 immunizations. The percentages in the table reflect reports either from (a) the child's vaccination card or (b) a mother's report (called "either source" in the Measure DHS tables). We only include countries that had at least one survey during or before 1997 and at least one survey during or after 2003; 1997-2003 was the period over which VillageReach provided Measure DHS surveys for Cabo Delgado.

"We have transitioned the VillageReach model and program to the local Ministry of Health in Cabo Delgado province, home to 88 vaccination clinics. VillageReach and FDC, our local implementation partner, will continue to provide technical assistance and data reporting." VillageReach, "Milestones."

Kane 2008, Pg 26.

John Beale, Allen Wilcox, and Becca Miller, phone conversation with GiveWell, May 21, 2009.

VillageReach, "Nampula Indicators."

John Beale, email to GiveWell, December 23, 2009.

"The people that we hired initially were retired MOH employees; in Nampula they remained MOH employees and were just under our management for a period of time." John Beale, Allen Wilcox, and Becca Miller, phone conversation with GiveWell, May 21, 2009.

"Measurement is a critical component of our methodology and model. An initial baseline survey and a concluding endline survey will be conducted for each provincial deployment. Data from these surveys will be publicly available upon approval from the government of Mozambique." VillageReach, "Health System Strengthening in Mozambique", Pg 3.

See individual province budgets in VillageReach, "Mozambique Expansion Budget."

John Beale, phone conversation with GiveWell, July 23, 2010.

See:

• VillageReach, "Monthly Key Indicator Tabular Report," for a sample (not using real data) of the data it plans to collect.
• VillageReach, "Key Indicator Descriptions," for a guide to the meanings of these indicators.
• John Beale, email to GiveWell, June 21, 2010, for how frequently the indicators will be tracked.

Jamison et al. 2006, Pg 401. For more, see our discussion of the cost-effectiveness of immunization programs.

Jamison et al. 2006, Pg 401. For more, see our discussion of the cost-effectiveness of immunization programs.

VillageReach, "Comparison of Costs Incurred in Dedicated and Diffused Vaccine Logistics Systems."

For Cabo Delgado, we have (a) the number of children receiving 3 doses of DTP (which Kane 2008, following GAVI, asserts is a proxy for "fully immunized") in 2001-2007 (Kane 2008, Pg 19) and (b) the percentage of children immunized in 2003 and 2008 (Kane 2008, Pg 23). For Niassa, we have the percentage of children immunized in 2003 and 2008 (Kane 2008, Pg 23).

The use of DTP-3 as a proxy for full immunization is supported by data from Cabo Delgado (reported by Leach-Kemon, Dionísio, and Taimo 2008, Pg 63, Table 1-11). Coverage rates of BCG, polio 3, and measles vaccinations were at least as high as DTP-3 coverage. In addition, 92.8% of children in the 24-35 months age group were "fully-vaccinated," while 95.4% were immunized with DTP-3.

We credit VillageReach with each child immunized in Cabo Delgado above and beyond the percentage immunized in Niassa for the same year, starting in 2003 (the year after the VillageReach program began), interpolating data (linearly) when it is missing. While far from ideal, we find this a reasonably appropriate comparison, as we estimate the Cabo Delgado coverage rate prior to VillageReach's involvement to have been close to the Niassa coverage rate (see Leah Barrett, email exchange with GiveWell, June, 2009).

• Immunization rates: Evidence suggests that immunization rates fell slightly between 2007-2008 as VillageReach handed over responsibility for the program (see the Can Improvements be Maintained section on this page). We don't know how much rates fell, so to err on the conservative side, we assign the same rate to 2007 that is assigned to 2008. We linearly interpolate the immunization rate between 2003-2007 (in Cabo Delgado) and 2003-2008 (in Niassa).
• Children in need of vaccination: Calculated as follows: (# of children immunized in Cabo Delgado) / (% of children immunized in Cabo Delgado).
• Addtional children immunized: Calculated as follows: [(% of children immunized in Cabo Delgado) - (% of children immunized in Niassa)] * (Total number of children in need of immunizations in Cabo Delgado).

Our total impact estimate comes to 95,610 additional children immunized (we do not apply a discount rate; a moderate discount rate would make little difference due to the short time period under discussion).

Costs are detailed in VillageReach, "Cost Estimates (August 17, 2009)." We include all costs, even those allocated to the Nampula activities, for which we assume no impact. Between 2001 and 2006 (the year VillageReach stopped financially supporting the program in Cabo Delgado), VillageReach spent a total of $3,910,411, implying (with the impact estimate above) $40.90 spent by VillageReach for every additional child immunized between 2003-2008.

If one assumes that the government spends an additional $15 per child - consistent with the Jamison et al. (2006) estimate of the costs for a standard expansion program, and probably an overstatement (since some of VillageReach's costs likely substitute for government costs) - the implied total cost per death averted rises to ~$745, still well within the range discussed on our overview of cost-effectiveness estimates.

VillageReach, "Health System Strengthening in Mozambique," Pg 7.

VillageReach, "Health System Strengthening in Mozambique," Pg 7.

• Total projected cost for Mozambique: $5,635,439.49. VillageReach, "Mozambique Expansion Budget."
• Total projected incremental children vaccinated: 184,281. VillageReach, "Health System Strengthening in Mozambique," Pg 7.
• Cost per incremental child vaccinated: $5,635,493.49 / 184, 281 = $30.58

• Total projected cost: $5,635,439.49 for Mozambique (VillageReach, "Mozambique Expansion Budget") plus $810,844.00 per year for six years in non-Mozambique, non-contract costs (see above) = $10,500,503.49.
• Total projected incremental children vaccinated: 184,281. VillageReach, "Health System Strengthening in Mozambique," Pg 7.
• Cost per incremental child vaccinated: $10,500,503.49
  / 184,281 = $56.98

See VillageReach, "Organization Budget (2010)" and VillageReach, "Mozambique Expansion Budget." The total 2011 cost of the Mozambique expansion (cells I34-L34 in "Mozambique Expansion Budget") is $1,029,848.00, with $225,000 in reserves available for it ("Organization Budget (2010)").

"Considering what we believe are the limitations of the provinces, we think it's reasonable that we could accelerate the start of three new provinces by 3 – 6 months, which would require approximately $6-700k in additional funds, for a total of $1.5M for next year. It's reasonable to think the remaining two provinces might react to seeing the accelerated deployments and requesting to start earlier also, which would translate into additional need for funds." John Beale, email to GiveWell, July 2, 2010.

VillageReach, "IRS Form 990 (2002-2008)."

Bill and Melinda Gates Foundation, "VillageReach."

Organizations report income on their tax forms in the year a grant agreement is reached. VillageReach received funds from this grant over the five-year period.