Results for Development — Phase I Grant Evaluation

Published: January 2019

Note: This page summarizes our analysis of a GiveWell Incubation Grant to Results for Development in May 2016. Results for Development staff reviewed this page prior to publication.

Summary

In May 2016, Results for Development (R4D) received a GiveWell Incubation Grant of $6,400,000 to support its pneumonia treatment program in Tanzania. R4D's goal for this grant was to increase the treatment coverage rate of pediatric amoxicillin, which is the World Health Organization-recommended first-line treatment for childhood pneumonia.1 This grant was intended to support the first phase of a project that R4D estimated would cost a total of $19 million. On this page, we analyze the outcomes of Phase I of the program, in part as an input into our decision about whether to recommend a second grant to R4D to support Phase II of the program.

What did they do? R4D has implemented Phase I of its pediatric amoxicillin program to treat childhood pneumonia in Tanzania. As part of Phase I, R4D directly funded the procurement and clearance of amoxicillin dispersible tablets (amoxicillin DT), worked with the Government of Tanzania to develop a co-financing strategy and create a budget allocation for amoxicillin DT, provided technical assistance to improve guidelines, forecasting, and supply planning, and conducted monitoring and evaluation, including measuring the rate of accurate diagnosis of childhood pneumonia and the rate of prescription of pediatric amoxicillin for children with pneumonia seeking care in the public sector.

Did it work? We believe that R4D's program likely increased the availability of pediatric amoxicillin in the public sector relative to the counterfactual in 2017 and 2018 by increasing the level of procurement of amoxicillin DT and improving forecasting and supply planning. However, we are highly uncertain about the extent of its impact on these factors because we have little evidence with which to estimate the counterfactual, and therefore rely strongly on R4D's guess about what the Government of Tanzania would have done in its absence. R4D's monitoring activities identified that clinicians are correctly diagnosing pneumonia in a minority of cases in Tanzania's public health facilities.

How cost-effective was Phase I of the program? We have not conducted a formal cost-effectiveness analysis of Phase I of the program. However, we have conducted a forward-looking cost-effectiveness analysis of Phase II of the program, which we believe is likely to be in the range of cost-effectiveness of our top charities, although this is highly uncertain. We think that Phase I was slightly more cost-effective than we project Phase II will be.

Table of Contents

What did they do?

Note: All of the content below is based on communication that GiveWell had directly with R4D.

The program

Results for Development (R4D)'s pediatric amoxicillin program to treat childhood pneumonia in Tanzania is based on the following theory of change:

  1. Increases in the volume of pediatric amoxicillin procured for distribution to public health facilities and improvements in the timeliness of procurement through improved forecasting and supply planning contribute to an increase in the availability of pediatric amoxicillin in public health facilities.2
  2. Increased availability of pediatric amoxicillin leads to increases in the proportion of children with pneumonia who receive the appropriate treatment, assuming a certain minimum level of a) public sector care-seeking, b) correct diagnosis by clinicians, and c) prescription of pediatric amoxicillin for pneumonia cases, regardless of whether pneumonia was correctly diagnosed.

Phase I of the program, which took place in 2016-2018, involved the following components:

  • R4D aimed to increase the volume of amoxicillin dispersible tablets (amoxicillin DT) procured by directly funding the purchase of amoxicillin DT and working with the Government of Tanzania to encourage it to create a budget allocation for amoxicillin DT.
  • R4D provided technical assistance to the Ministry of Health, Community Development, Gender, Elderly and Children to improve its methodology for forecasting demand for pediatric amoxicillin through the 2020 fiscal year.
  • R4D provided technical assistance to the government agency responsible for procurement of medical supplies (the Medical Stores Department (MSD)) to support its initiation of procuring amoxicillin DT and to improve the timeliness of pediatric amoxicillin procurement through better supply planning.
  • R4D conducted monitoring to determine whether the pediatric amoxicillin that was procured was consistently reaching health facilities and therefore successfully increasing availability of amoxicillin for children who need it.
  • R4D and several partners conducted a clinical study to determine how accurately clinicians are diagnosing pneumonia in children and prescribing pediatric amoxicillin in the public sector.
  • R4D identified and tested a few interventions to improve the knowledge of public health facility clinicians and private sector drug shop dispensers about childhood pneumonia treatment.
  • R4D supported the government in developing a co-financing agreement to finance amoxicillin DT through the 2020 fiscal year, with the ultimate goal of transitioning funding for amoxicillin DT procurement to the Government of Tanzania in the medium to long term.

Spending breakdown

A detailed breakdown of the budget for Phase I can be found in R4D Budget, Phase I & II, November 2018. The total cost of Phase I was $6,400,000, which was allocated as follows:

  • 41% ($2,625,000) for pediatric amoxicillin procurement
  • 26% ($1,654,136) for monitoring and evaluation (inclusive of staff costs)
  • 25% ($1,577,307) for staff costs and expenses
  • 8% ($543,557) for public and private programmatic activities

Some of the funding allocated for procurement was not spent during Phase I; this will be used to support procurement during Phase II of the program.

Did it work?

  • Did R4D increase the total volume of pediatric amoxicillin procured relative to the counterfactual? We believe so, but we are highly uncertain about the size of its impact because we have little evidence with which to estimate the counterfactual. We rely strongly on R4D's guess for what the Government of Tanzania would have done in its absence.
  • Did R4D improve the timing of pediatric amoxicillin procurement relative to the counterfactual through improved supply planning? We believe that R4D was able to improve supply planning more quickly than it would likely have been improved in the counterfactual, for example by providing technical assistance to MSD. Our assessment here is based on R4D documents and conversations with R4D staff.
  • Did R4D improve availability of amoxicillin relative to the counterfactual? We believe so, because we believe that R4D increased the amount of amoxicillin procured and made improvements to supply planning. However, we are highly uncertain about the extent of its impact because we have little evidence with which to estimate the counterfactual.
  • Did R4D assess the level of clinician accuracy in diagnosing pneumonia? The R4D-facilitated pneumonia diagnosis and prescription study found that clinicians in public health facilities in Tanzania correctly diagnosed pneumonia in about 18% of pneumonia-positive cases, as identified by a lung ultrasound examination.
  • Did R4D assess the level of clinician accuracy in prescribing pediatric amoxicillin? The R4D-facilitated pneumonia diagnosis and prescription study found that clinicians in public health facilities in Tanzania prescribed pediatric amoxicillin in 72% of cases in which they correctly diagnosed a child as having pneumonia, as identified by a lung ultrasound examination. However, they also prescribed pediatric amoxicillin in approximately half of cases in which a child with pneumonia was incorrectly given a different diagnosis.
  • Did R4D make improvements to help sustain the gains in treatment availability in the event that the Government of Tanzania takes over the financing of amoxicillin DT in the longer term? We believe that R4D has better positioned the Government of Tanzania to take over financing of amoxicillin DT in the longer term, including by agreeing to a cost-sharing plan for product costs through the 2020-21 fiscal year.

Details follow.

Did R4D increase the total volume of pediatric amoxicillin procured relative to the counterfactual?

We believe that R4D increased the total volume of pediatric amoxicillin procured relative to the counterfactual, but we are unsure about the likely size of the increase because we have little information with which to estimate the counterfactual. Our guess relies strongly on R4D's own guess about the volume of pediatric amoxicillin the Government of Tanzania would have procured in its absence. We think that guess is reasonable but highly uncertain.

Some of the considerations we took into account include:

  • Procurement under the program: R4D told us that under the program in 2017, approximately 29 million tablets, amounting to approximately 1.5 million treatments, were procured.3 R4D told us this was equivalent to slightly more than one year's worth of demand.4 We have not seen data on the number of tablets procured in 2018, and 2017 was the first year in which R4D directly administered financing for the procurement of amoxicillin DT in Tanzania.5
  • Procurement in the counterfactual: R4D told us that had it not intervened, the Government of Tanzania would have made efforts to reallocate funding from the Global Financing Facility to procure an emergency shipment of three months' worth of amoxicillin DT for 2017.6 R4D told us that it expects that in its absence, and given the available funding sources and needs at the time, the Government of Tanzania could have procured another three months' worth of amoxicillin DT in 2018 by reallocating funds in its health commodities budget away from the procurement of other child health commodities, such as zinc and oral rehydration solution (the recommended combination treatment for diarrhea7 ), because diarrhea and pneumonia are the two largest causes of child mortality in Tanzania and the government does not appear to prioritize treatments for one over the other.8 If these guesses are correct, the amount of pediatric amoxicillin procured under the program in 2017 was more than four times greater than it would have been in the counterfactual. R4D told us that it planned for a similar level of procurement in 2018 as in 2017, but we have not yet seen data on the actual level of procurement in 2018 and therefore do not know by how much procurement increased during Phase I overall.
  • Uncertainty over the counterfactual estimate due to a lack of data: We have very little data with which to estimate the counterfactual and are relying on R4D's guess, so we are highly uncertain about the size of the program's impact on procurement.
  • Uncertainty over the counterfactual estimate due to the possibility of alternative funders: R4D's guess for counterfactual availability assumes that no alternative funder other than the Government of Tanzania would have intervened in its absence. We believe this is a reasonable assumption.9 Amoxicillin DT had previously been funded by the Canadian government and the Reproductive, Maternal, Newborn and Child Health Trust Fund, and administered by UNICEF. However, this funding ended as of mid-2016, leaving a large funding gap around the beginning of Phase I.10

Did R4D improve the timing of pediatric amoxicillin procurement relative to the counterfactual through improved supply planning?

Improvements to supply planning can increase the availability of pediatric amoxicillin for a given volume of the drug procured by increasing the likelihood that it is stocked in the health facilities where it is needed at the right time. We believe that R4D was able to improve supply planning under the program by providing technical assistance, including to MSD. While R4D believes that the Government of Tanzania would have sustained efforts to improve supply planning on its own in the counterfactual, we think it is likely that this occurred at a faster rate due to R4D's assistance. R4D told us about three specific improvements it made, or is making, to supply planning:

  1. Measuring seasonality of pneumonia incidence in 2017 and its impact on supply planning: R4D told us that previously no analysis had been performed on the number of people with pneumonia and other respiratory illnesses seeking care in the Tanzanian public health system in a given year, nor on the distribution of cases throughout the year. MSD previously did not take seasonality into account when conducting distribution planning of pediatric pneumonia treatments and assumed constant demand for pediatric amoxicillin through the year.11 In practice, actual demand is not constant through the year, for example due to the measured seasonality of pneumonia incidence in 2017.12 R4D measured a seasonality trend in pneumonia cases in 2017 by analyzing patient register data collected as part of a three-round health facility survey. R4D shared this analysis with the Government of Tanzania and is co-developing strategies to systematically improve distribution planning at the central level and supply planning at the health facility level.13
  2. Improving the consistency of procurement: The Government of Tanzania's health commodity procurements are often reactive, with stockouts occurring before additional volumes are ordered.14 R4D told us that it has provided support to the Government of Tanzania to order pediatric amoxicillin more consistently and proactively.15 We have not asked for more details on how it has helped the Government of Tanzania to do this, but R4D told us that it made regular orders itself in 2017.16
  3. Updating health facilities' request and requisition forms to facilitate ordering of amoxicillin DT:17 R4D advocated for the Government of Tanzania to update its request and requisition forms to include amoxicillin DT.18 This reduces the effort required by health staff to order amoxicillin DT, as they would otherwise have to write out their orders manually on the form.19 We think that reducing this barrier will make it more likely that health facilities will request amoxicillin DT from the MSD stores when it is needed.

Did R4D improve availability of amoxicillin relative to the counterfactual?

We believe that R4D likely increased availability of pediatric amoxicillin relative to the counterfactual (i.e., what would have happened during the same period if R4D were not present) because it achieved a higher level of procurement and made improvements to supply planning. However, we are unsure about the size of its impact because we have little information with which to estimate counterfactual availability.

Availability under the program

Methodology:

  • R4D used a stratified two-stage sampling procedure to randomly select a sample of 624 public health facilities that we believe are likely to be fairly representative of mainland Tanzania,20 with the exception that district hospitals and possibly health facilities that serve a small number of patients are over-represented.21 The sample consisted of 53 district hospitals, 50 health centers, and 521 dispensaries, which are the three types of public health facility in Tanzania.22 More details on the sampling procedure can be found in R4D, Summary of Monitoring, Evaluation & Learning Survey Methodologies, May 2018.
  • Health facility surveys were conducted in March, July, and November 2017.23
  • All surveys were conducted after the first shipment of amoxicillin DT had entered Tanzania in January 2017 to avert national stockouts, so R4D does not have data on baseline availability of pediatric amoxicillin. R4D notes that amoxicillin DT availability in facilities that are designated as part of Group A in the Government of Tanzania's Integrated Logistics System (ILS) during Round 1 or amoxicillin OS availability in Round 2 can be used to approximate baseline data (more detail in footnote).24
  • A health facility is measured as having pediatric amoxicillin available if at least one unit of either amoxicillin DT or amoxicillin OS is available in the health facility on the day of the survey.25

Results:

  • Pediatric amoxicillin availability was measured at 67% in the November 2017 survey.26
  • We focus on the results from the third survey because by this point all six shipments scheduled for 2017 had entered Tanzania.27

Availability in the counterfactual

  • R4D's guess is that counterfactual availability would have been 25% in 2017 and 2018.28 This estimate is based on R4D's best guess that the Government of Tanzania would have procured three months' worth of stock in each year (calculated as a proportion of annual demand), and the assumption that the percentage of the annual need that is procured is equivalent to the level of availability.29
  • R4D's guess may be an overestimate due to challenges in supply planning. For the reasons explained above, challenges in supply planning in the counterfactual might mean that a procurement of three months' worth of supply translates into less than 25% availability.30
  • R4D's guess may be an underestimate due to amoxicillin OS stocks left over at the start of 2017. In Round 1 of the health facility survey in March 2017, amoxicillin OS availability was measured at 25%.31 By Round 3 in November 2017, amoxicillin OS availability was measured at 16%.32 R4D told us that amoxicillin OS was not procured during 2017,33 so these stocks were left over from the previous year and would have been available in the counterfactual as well. R4D's estimate of counterfactual availability takes into account only amoxicillin DT, though the measured availability of pediatric amoxicillin includes both amoxicillin DT and amoxicillin OS. Based on amoxicillin OS alone, availability of pediatric amoxicillin would have been between 16% and 25% in 2017, so it seems plausible to us that R4D's guess underestimates availability. The extent to which a government procurement of amoxicillin DT would have increased availability beyond this level would likely depend on the degree of overlap between facilities that had amoxicillin OS stocks and facilities that would have ordered amoxicillin DT.

Did R4D assess the level of clinician accuracy in diagnosing pneumonia?

A pneumonia diagnosis and prescription study undertaken in three regions of Tanzania by R4D, the Government of Tanzania, several Tanzanian partners, and IDinsight suggests that clinicians in public health facilities in Tanzania are not accurately diagnosing pneumonia in most cases. With less accurate diagnosis, increases in the availability of pediatric amoxicillin in health facilities will translate less effectively into increases in the proportion of children with pneumonia who receive treatment.34

Methodology:

  • Three regions were purposefully selected. The diagnosis and prescription study sample is not nationally representative. The Pwani, Dodoma, and Tabora regions were sampled because they include a mix of rural and urban areas and a mix of high- and low-performing regions, they are easy to access but geographically dispersed within Tanzania, and they are areas where R4D could potentially pilot interventions to improve diagnosis.35 We are unsure whether diagnosis rates in this sample are likely to under- or overestimate diagnosis rates in Tanzania as a whole.
  • Health facilities were randomly selected within those regions. A total of 83 health facilities were randomly selected, using a stratified sampling procedure, from these three regions. See this footnote for details.36
  • In each facility, approximately one provider was randomly selected from a list of all eligible providers.37 A total of 93 healthcare providers were surveyed.38 Enumerators observed all patients seen by those providers.39
  • Misdiagnosis was measured by comparing clinicians' diagnoses to results from lung ultrasound examinations.40

Results:

  • The pneumonia diagnosis and prescription clinical study found that only 18% of children with lung ultrasound-confirmed pneumonia were correctly diagnosed, out of a sample size of 847 children.41 We have not vetted this conclusion.

Did R4D assess the level of clinician accuracy in prescribing pediatric amoxicillin?

The pneumonia diagnosis and prescription clinical study suggests that clinicians prescribed pediatric amoxicillin in the majority of cases in which they had correctly diagnosed a child as having pneumonia. However, they also prescribed pediatric amoxicillin in nearly half of cases in which they had incorrectly diagnosed a child as having something other than pneumonia. Without accurate prescription, increases in the availability of pediatric amoxicillin in health facilities will have a reduced impact on increasing the proportion of children with pneumonia who receive treatment.

Methodology:

  • R4D's prescription rate estimates were based on the same study that was used to estimate the diagnosis rate (see above).
  • Prescription rates were estimated based on provider behavior in health facilities that had pediatric amoxicillin in stock.42

Results:

  • The diagnosis and prescription study found that in health facilities that had pediatric amoxicillin in stock, clinicians prescribed pediatric amoxicillin in 72% of cases where they had correctly diagnosed a child as having pneumonia.43 We have not vetted this estimate.
  • The diagnosis and prescription study found that in health facilities that had pediatric amoxicillin in stock, clinicians also prescribed pediatric amoxicillin in 46% of cases in which they had incorrectly diagnosed a child as having something other than pneumonia.44 We have not vetted this estimate. We are unsure why clinicians were prescribing pediatric amoxicillin to children with pneumonia whom they had incorrectly diagnosed as having something other than pneumonia in such a high proportion of cases.45
  • Based on these results, R4D calculates that about 51% of children under the age of five with lung ultrasound-confirmed pneumonia who visited a public health facility were prescribed pediatric amoxicillin.46

Did R4D make improvements to help sustain the gains in treatment availability in the event that the Government of Tanzania takes over procurement and financing in the longer term?

R4D told us that it has taken several steps during Phase I to prepare for the transition of all pediatric amoxicillin procurement and financing to the Government of Tanzania in the medium to long term, increasing the chance that gains in treatment availability made during Phase I can be sustained in the future. These steps included:

  • Supporting the Government of Tanzania in improving the robustness of its methodology for forecasting demand for pediatric amoxicillin. The previous methodology for estimating the demand underestimated pediatric amoxicillin demand in two ways. First, the methodology accounted for only two of three WHO-recommended age/weight bands, and did not include the age/weight band that requires the largest number of tablets per treatment.47 Second, it assumed that children with severe pneumonia would not receive pediatric amoxicillin, which did not align with WHO recommendations and health provider practices in Tanzania.48 R4D was involved in the forecasting process in 2016 and advocated for the Government of Tanzania to update the methodology to reduce the likelihood of underestimation. R4D told us it has estimated that this led to a 27% increase in forecast demand;49 we have not vetted this estimate. The level of demand forecast by the Government of Tanzania is an important input into the amount of funding it budgets for pediatric amoxicillin.
  • Developing a co-financing plan with the Government of Tanzania to finance the product costs of amoxicillin DT. In November 2017, R4D and the Government of Tanzania agreed to a co-financing plan under which the Government of Tanzania will pay for approximately 10%, 30%, 50%, and 60% of the product costs of amoxicillin DT over the fiscal years 2017-18, 2018-19, 2019-20, and 2020-21, respectively.50 We have not seen evidence of the actual amount the Government of Tanzania has spent so far, and we are uncertain whether the Government of Tanzania will meet this commitment in the future.

How cost-effective was Phase I of the program?

We have not conducted a formal cost-effectiveness analysis of Phase I of the program. We have conducted a forward-looking cost-effectiveness analysis of Phase II of the program, which can be found in this spreadsheet. We believe that Phase II is in the range of cost-effectiveness of our top charities, though our estimates are highly uncertain, particularly due to our uncertainty about the counterfactual. Our best guess is that Phase I was slightly more cost-effective than we project Phase II will be, as we expect that the counterfactual level of availability would grow over time (and therefore be higher in Phase II compared to Phase I) as the Government of Tanzania’s role in financing and procuring pediatric amoxicillin increases.

Sources

Document Source
GiveWell cost-effectiveness analysis, R4D Phase II Source
GiveWell's non-verbatim summary of a conversation with Kanika Bahl, Cammie Lee, and Thayer Rosenberg on January 27, 2016 Source
GiveWell's non-verbatim summary of a conversation with Kanika Bahl, Cammie Lee, Thayer Rosenberg, and Aileen Palmer on February 19, 2016 Source
R4D Budget, Phase I & II, November 2018 Source
R4D estimate of the counterfactual level of procurement, September 2018 Unpublished
R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018 Unpublished
R4D, Summary of Monitoring, Evaluation & Learning Survey Methodologies, May 2018 Source
WHO Fact sheet, diarrhoeal disease Source (archive)
WHO Fact sheet, pneumonia Source (archive)
  • 1

    "Pneumonia should be treated with antibiotics. The antibiotic of choice is amoxicillin dispersable [sic] tablets." WHO Fact sheet, pneumonia

  • 2

    We define "availability" as the proportion of facilities that have at least some pediatric amoxicillin in stock, such that a child with pneumonia who entered the facility could receive treatment immediately.

  • 3

    "R4D-procured amoxicillin DT shipments (~29 million tablets or ~1.5 million treatments)," R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 12.

  • 4

    "In 2017, 14.25 months of amoxicillin DT stock was procured." R4D estimate of the counterfactual level of procurement, September 2018, p. 6.

  • 5

    The first shipment entered Tanzania in January 2017. R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 12.

  • 6

    R4D estimate of the counterfactual level of procurement, September 2018, p. 2.

  • 7

    "Key measures to treat diarrhoea include the following:

    • Rehydration: with oral rehydration salts (ORS) solution. ORS is a mixture of clean water, salt and sugar. It costs a few cents per treatment. ORS is absorbed in the small intestine and replaces the water and electrolytes lost in the faeces.
    • Zinc supplements: zinc supplements reduce the duration of a diarrhoea episode by 25% and are associated with a 30% reduction in stool volume."

    WHO Fact sheet, diarrhoeal disease

  • 8

    R4D estimate of the counterfactual level of procurement, September 2018, p. 5.

  • 9

    "R4D believes it is unlikely to find funding for its Tanzania program apart from GiveWell and Good Ventures.

    "In the past, Amox DT in Tanzania has been procured through support from the Canadian government and the Reproductive, Maternal, Newborn and Child Health (RMNCH) Trust Fund, both administered through UNICEF. The Canadian government funding came via a three year project to scale up access to Amox DT, Zinc and ORS in four countries, including Tanzania. While the project had many successes, at this time, there is no commitment to continue funding Amox DT.

    "Meanwhile, the RMNCH Trust Fund is sunsetting in mid-2016; its country grants have been finalized and countries have already made line-item allocations for this funding, so it is unclear whether this is still a potential source of additional funding for Amox DT.

    "The Gates Foundation has already committed programmatic funding to R4D for the initial market-shaping work in Tanzania. It is unlikely that this will increase due to funding priorities.

    "Ms. Bahl's impression is that it is not uncommon for potentially cost-effective interventions (other than for particularly prominent diseases, such as AIDS, tuberculosis, malaria, and some vaccines) to be unavailable due to lack of funding."

    GiveWell's non-verbatim summary of a conversation with Kanika Bahl, Cammie Lee, Thayer Rosenberg, and Aileen Palmer on February 19, 2016, pp. 3-4.

  • 10

  • 11
    • R4D estimate of the counterfactual level of procurement, September 2018, p. 3.
    • R4D comment on a draft of this page, November 6, 2018.

  • 12

    "The number of recorded acute respiratory infection (ARI) and pneumonia cases was about twice as high from March 2017 to June 2017 as it was during the rest of the year." R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 13.

  • 13
    • "R4D has been supporting the Government of Tanzania in accounting for periods when health facilities reported experiencing stockouts based on the Government of Tanzania’s eLMIS [electronic logistic management information system] data to augment the average monthly forecasted quantity. This has led to more accurate supply planning...and procurement, and likely better availability trends." R4D estimate of the counterfactual level of procurement, September 2018, p. 3.
    • R4D comment on a draft of this page, November 6, 2018.

  • 14

    "R4D estimate of the counterfactual level of procurement, September 2018, p. 3.

  • 15

    "Even with financing support, we have been providing substantial support around consistency of procurement." R4D estimate of the counterfactual level of procurement, September 2018, p. 4.

  • 16

    "Prior to the Government of Tanzania initiating procurement, R4D worked closely with an external procurement agent and made a total of 6 regular orders in 2017." R4D estimate of the counterfactual level of procurement, September 2018, p. 4.

  • 17

    Conversation with R4D staff, August 24, 2018.

  • 18

    In a conversation on August 24, 2018, R4D staff told us that amoxicillin DT is relatively new to Tanzania, and as such there was previously only a tick box for amoxicillin oral suspension (amoxicillin OS).

  • 19

    R4D told us that without this tick box, health staff members have to fill out an open section of the form, listing details such as the formulation, dosage, and pack size. Conversation with R4D staff, August 24, 2018.

  • 20
    • "Prior to Round 1 data collection, a stratified two-stage sampling method was used to select a representative sample of public health facilities across the 26 mainland regions of Tanzania to be included in the baseline and subsequent rounds of data collection." R4D, Summary of Monitoring, Evaluation & Learning Survey Methodologies, May 2018, p. 1. Note that this document is based on a document prepared by IDinsight, which we have not seen and which may include more detail.
    • "The sample was therefore comprised of 53 district hospitals, 50 health centers, and 21 dispensaries, for a total of 624 public health facilities – out of a total of 61 district hospitals, 477 health centers, and 4,471 dispensaries in Tanzania (according to Tanzanian Government Open Data information from July 2016)." R4D, Summary of Monitoring, Evaluation & Learning Survey Methodologies, May 2018, p. 2.

  • 21
    • "The sample was therefore comprised of 53 district hospitals, 50 health centers, and 21 dispensaries, for a total of 624 public health facilities – out of a total of 61 district hospitals, 477 health centers, and 4,471 dispensaries in Tanzania (according to Tanzanian Government Open Data information from July 2016)." R4D, Summary of Monitoring, Evaluation & Learning Survey Methodologies, May 2018, p. 2.
    • "Councils were weighted according to the proportion of public health facilities in each council so that councils with a higher number of public health facilities would have a greater probability of being selected." R4D, Summary of Monitoring, Evaluation & Learning Survey Methodologies, May 2018, p. 1. Since the sampling methodology weights councils by the number of public health facilities located there but does not mention weighting each health facility by the number of patients it serves, we think it is plausible that this methodology would cause small health facilities to be overrepresented compared to the size of the populations they serve.

  • 22

    "The sample was therefore comprised of 53 district hospitals, 50 health centers, and 521 dispensaries, for a total of 624 public health facilities – out of a total of 61 district hospitals, 477 health centers, and 4,471 dispensaries in Tanzania (according to Tanzanian Government Open Data information from July 2016)." R4D, Summary of Monitoring, Evaluation & Learning Survey Methodologies, May 2018, p. 2.

  • 23

    R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 12 includes a timeline of the surveys and shipments.

  • 24
    • R4D told us that amoxicillin OS availability in Round 2 can be used as a rough proxy for baseline availability of pediatric amoxicillin in Tanzania at the end of 2016. They told us that amoxicillin OS levels are a proxy for pediatric amoxicillin levels overall because amoxicillin DT stocks had depleted by the end of 2016.
    • Health facilities in ILS Group A had placed orders for amoxicillin DT before R4D's procurement had entered Tanzania at the start of 2017. Our understanding is that these facilities did not receive any of the first shipment of R4D-funded amoxicillin DT due to the routine scheduled delivery cycle of this ILS group. Amoxicillin DT stocks in these facilities in Round 1 may therefore constitute a rough proxy for baseline availability of amoxicillin DT.

    R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 16.

  • 25

    "Survey question: Medication is available in facility if at least one unit (tablet or bottle) available at either the dispensing window or in the stock room on the day of the survey." R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 18.

    We think this is a reasonable measure because it corresponds with the probability that a random child with pneumonia in Tanzania who visits their nearest health facility would be able to receive pediatric amoxicillin immediately.

  • 26

    R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 18, left panel.

  • 27

    R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 12 includes a timeline of the surveys and shipments.

  • 28

    R4D estimate of the counterfactual level of procurement, September 2018, Table 1, p. 1.

  • 29

    "[W]e have made a simplistic assumption that procurement of a certain number of months of stock equates to availability levels (i.e. 6 months of stock procured in aggregate in a given year would lead to a 50% availability in health facilities...6 months procured/12 months in a year = 50% availability)." R4D estimate of the counterfactual level of procurement, September 2018, p. 5.

  • 30

    R4D estimate of the counterfactual level of procurement, September 2018:

    • "[W]e believe the assumption of the months of stock procured equalIing availability very likely leads to an overestimate of availability." p. 6.
    • "[T]here is a complex relationship between the amount of amoxicillin DT procured and availability levels at health facilities because of many of the factors listed above." p. 5.

  • 31

    R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 18, left panel.

  • 32

    R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 18, left panel.

  • 33

    Conversation with R4D staff, August 7, 2018.

  • 34

    As explained in this section, clinicians are more likely to prescribe pediatric amoxicillin to a child with pneumonia if they are correctly diagnosed than if they are not. Taking those prescription rates as given, improvements to the diagnosis rate would therefore increase the proportion of children with pneumonia who receive treatment (when treatment is available).

  • 35

    "This study was conducted...in the Pwani, Dodoma, and Tabora regions in Tanzania...These regions were selected because they: 1) give a mix of rural and urban areas, 2) give a mix of high-and low-performing regions, 3) are accessible but geographically dispersed, and 4) are areas where R4D plans to pilot interventions to address misdiagnosis." R4D, Summary of Monitoring, Evaluation & Learning Survey Methodologies, May 2018, p. 3.

  • 36

    R4D, Summary of Monitoring, Evaluation & Learning Survey Methodologies, May 2018:

    • "This study was conducted with...83 facilities (5 hospitals, health centers, and 67 dispensaries)." p. 3.
    • "Within each region, we employed a stratified sampling strategy to select the facilities to be included in the study. Three districts were randomly selected within each region, and within each district, we randomly selected one district hospital, one public health center, and seven public dispensaries...We were unable to complete data collection at all facilities selected for the study; where this occurred we randomly selected replacement facilities (of the same facility type where possible) within the same district." p. 4.

  • 37

    "Providers were eligible to be included in the study if they: 1) worked full time at the facility; 2) were responsible for the assessment, diagnosis, and treatment of children under 5; and 3) would be available for observation in the following days. One provider per facility was randomly selected from a list of eligible providers...In some cases, we were unable to observe a provider for the full length of a facility visit (e.g., due to illness or family emergencies). Where this occurred, another eligible provider was randomly identified from the original list." R4D, Summary of Monitoring, Evaluation & Learning Survey Methodologies, May 2018, p. 4.

  • 38

    "This study was conducted with 93 healthcare providers." R4D, Summary of Monitoring, Evaluation & Learning Survey Methodologies, May 2018, p. 3.

  • 39

    "Enumerators observed all patients seen by the providers during their time at the facility and completed observation checklists for all children under 5. Following the consultation, enumerators asked the caregiver of any child who was under 5 and had respiratory symptoms (cough or difficulty breathing) to consent for LUS verification of the provider’s diagnosis. Children who were under two months old were ineligible due to technical difficulties with performing LUS on such small children. Children with severe illness, who were immediately referred to a higher-tier facility were also ineligible for LUS." R4D, Summary of Monitoring, Evaluation & Learning Survey Methodologies, May 2018, p. 4.

  • 40

    "R4D observed actual provider practice in diagnosing children under five with respiratory conditions and measured misdiagnosis rate by comparing provider Dx vs. ultrasound, a more advanced technology." R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 26.

  • 41

    "N = 847 children under 5." R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 27.

  • 42

    Sample: "100 children under 5 with ultrasound-confirmed pneumonia seeking care at public health facilities with pediatric amoxicillin in stock." R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 32.

  • 43

    R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 32.

  • 44

    R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 32.

  • 45

    The 82% of children with pneumonia who were incorrectly diagnosed were diagnosed as having either an acute respiratory infection (ARI), upper respiratory tract infection (URTI), or a cough/cold in most cases. "Provider diagnoses in these instances: 42% ARI, 27% URTI, 10% cough/cold, 3% other." See R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 32.

  • 46

    (18% x 72%) + (82% x 46%) = 51%

  • 47

    "The quantification only accounted for two age/weight bands requiring 10 and 20 tablets per course of treatment, instead of the WHO-recommended three age/weight bands, with the third band requiring 30 tablets per course of treatment." R4D estimate of the counterfactual level of procurement, September 2018, p. 1.

  • 48

    "Additionally, the forecast assumed that the 20% of children under five estimated to have severe pneumonia would not receive any amox DT, while according to WHO and Tanzania practice, severe pneumonia patients are given amox DT as well." R4D estimate of the counterfactual level of procurement, September 2018, p. 1.

  • 49

    "R4D participated in the national quantification for 2016 and successfully advocated for updating the methodology. ~22M tablets of amox DT 250mg was forecasted as the total need for 2016, a 27% increase over 2015’s estimate of ~16M, after accounting for annual population growth and a methodological adjustment made to the quantification calculation." R4D estimate of the counterfactual level of procurement, September 2018, p. 1.

  • 50

    "GoT agreed in November 2017 to an amox DT co-financing plan, which will usher in a more sustainable financing structure. Agreed upon levels: 2017/18: GoT 10%, 2018/19: GoT 30%, 2019/20: GoT 50%, 2020/21: GoT 60%." R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 34.