Results for Development — Childhood Pneumonia Treatment Program (2019)

Published: October 2019

Note: This page summarizes the rationale behind a GiveWell Incubation Grant to Results for Development. Results for Development staff reviewed this page prior to publication.

In July 2020, Results for Development staff shared its May 2020 pre-analysis plan for Phase II of its pneumonia treatment program, which is available here. We spoke with Results for Development about updates on this grant in August 2020, April 2021, July 2021, October 2021, and May 2022.

The program to date

R4D's pneumonia treatment program in Tanzania aims to increase the proportion of children with pneumonia who are treated with pediatric amoxicillin in health facilities. Though the activities conducted during Phase I (funded by the previous Incubation Grant to R4D) and the activities planned for Phase II (funded by the current grant) differ in some ways, both primarily focus on increasing the availability of amoxicillin dispersible tablets (amoxicillin DT) in health facilities so that children with pneumonia who visit a health facility are more likely to receive the appropriate treatment.

Based on our evaluation of R4D's work during Phase I of the program, we believe that its work led to increases in the availability of pediatric amoxicillin and that the monitoring it conducted was of high quality.2 The project team has also been highly communicative with us on the phone and over email and has engaged extensively with our cost-effectiveness analysis of this program.

Planned activities

R4D plans to continue the following activities during Phase II of the program:

• Advocating for increasing the amount the Government of Tanzania budgets for purchasing amoxicillin DT. R4D and the government have agreed to a cost-sharing plan in which the government will contribute 30% of procurement costs in 2018-19, 50% in 2019-20, and 60% in 2020-21;3 they later agreed to include more of a focus on sharing contributions to procurement volumes, since there are regular updates to forecasted amoxicillin DT volumes, as well as fluctuations in product pricing and foreign exchange rates. The original commitment to cover 30% of the costs in 2018-19 is equivalent to procuring about 6.7 million amoxicillin DT tablets; R4D reports that the government exceeded this target and financed the procurement of about 11.5 million tablets during this time period.4 Based on our conversations with R4D, we believe that continued advocacy will increase the likelihood that the government will continue to meet this cost-sharing agreement. R4D's goal is for the government to assume the full procurement costs after Phase II ends, while R4D continues to provide technical assistance in the longer term.
• Providing technical assistance to the Government of Tanzania in order to reduce the incidence and length of stockouts of pediatric amoxicillin at health facilities. R4D's activities include work to improve supply planning at both (a) the national level so that orders from health facilities can be fulfilled more consistently, and (b) the health facility level so that facilities more consistently order amoxicillin DT before stockouts occur.5 One challenge is that pneumonia incidence varies seasonally.
• Monitoring availability of pediatric amoxicillin in health facilities. R4D will conduct surveys to measure the availability of pediatric amoxicillin in health facilities. R4D told us it has not yet finalized the details of its plans for monitoring and evaluation in Phase II, but we expect that it will be broadly similar to R4D's monitoring during Phase I, which included three surveys at different times of year wherein enumerators checked whether a health facility had at least one unit of amoxicillin DT or amoxicillin oral suspension (OS) available (for more detail, see here).

In Phase I, R4D also worked on monitoring and piloting interventions to improve the level of clinician accuracy in diagnosing pneumonia and prescribing pediatric amoxicillin. At the government's request, R4D plans to continue some work to pilot interventions to improve clinician accuracy in Phase II. R4D initially requested grant funding to scale up this work further; we decided not to fund scale-up due to our best guess that it would have limited impact.6

During Phase II of the program, R4D will continue to directly fund the procurement of amoxicillin DT, as it did during Phase I.7 This funding will augment the government's funding for amoxicillin DT, as per the cost-sharing plan that R4D and the government have agreed upon (see above).

R4D expects its total budget of $5,605,398 to break down as follows:

• Technical assistance: $3,273,6518
• Monitoring and evaluation: $1,806,7479
• Support for procurement costs: $525,00010

The case for the grant

The case for this grant rests mainly on our expectation that the availability of pediatric amoxicillin in health facilities will be significantly higher under the program than it would be in its absence.11

Treatment availability under the program

This project began in 2016. At three points in 2017, R4D conducted health facility surveys to measure the availability of pediatric amoxicillin in health facilities. The last survey, conducted in November 2017, found that amoxicillin DT was available in 67% of health facilities.12 We discuss these surveys in more detail in our Phase I grant evaluation.

R4D predicts that, with the continuation of its technical and financial assistance, availability will increase to 77% by 2022.13 This estimate is based on:

• A guess that the increased availability achieved during Phase I will cause national-level procurement of amoxicillin DT to increase. Increased availability may cause clinicians to prescribe amoxicillin DT more frequently. This could lead to increased demand for amoxicillin DT from health facilities, which, in turn, may lead the Government of Tanzania to procure a larger amount of amoxicillin DT.14
• A guess that R4D's activities will be effective at further increasing availability at a given level of national-level procurement. Though R4D's technical assistance was ongoing at the time of the 2017 surveys, R4D believes that it will see further gains in amoxicillin availability due to improved supply planning at the national and health facility levels. This seems plausible to us due to the timing of the surveys (fairly soon after R4D began technical assistance) and the possibility that R4D, alongside the government, will identify and address additional challenges in the supply chain.15
• Comparison with availability of cotrimoxazole. Cotrimoxazole is another antibiotic used in health facilities in Tanzania. Prior to R4D's work on amoxicillin DT, there was more consistent external support for procurement of cotrimoxazole in Tanzania than for amoxicillin DT, primarily from PEPFAR and The Global Fund to Fight AIDS, Tuberculosis and Malaria.16 R4D believes that, with its support, availability of amoxicillin DT can reach similar levels to those seen with cotrimoxazole. Across three rounds of health facility surveys in 2017 (described above), treatment availability of cotrimoxazole was measured at 75%, 72%, and 74%, with an average treatment availability of 74%.17

We believe it is reasonable to expect that, with R4D's intervention, availability of amoxicillin DT will reach the current level of treatment availability of cotrimoxazole (74%) by 2022.18

Treatment availability in the counterfactual

R4D's best guess is that in the absence of its program, treatment availability would be 29%, 31%, 33%, and 35% in 2019, 2020, 2021, and 2022, respectively.19 This guess is based on a number of factors:

1. Underestimation of demand. R4D told us that in 2016, the Government of Tanzania's methodology for forecasting demand for pediatric amoxicillin would have underestimated demand by 14%.20 During Phase I of the program, R4D provided assistance in the forecasting process, helping the government identify and address the calculations that were leading to the underestimation.
2. Incomplete disbursement of budgeted funds. In Tanzania in 2015, the average proportion of funds budgeted for procurement that were actually disbursed was 77%.21 For this reason and the above reason, the maximum that the government would be expected to procure in the counterfactual is 8 months' worth of annual demand.22
3. Room for improvement in supply planning. R4D told us about several ways in which it believes the quality of the government's supply planning would be lower in its absence.23 Supply planning challenges decrease the likelihood that if the government were to procure, for example, 50% worth of demand, then amoxicillin DT would be available a corresponding 50% of the time that a child with pneumonia enters their local health facility..

The first and third of these factors relate to issues that R4D already started addressing during Phase I of its program. We still place some weight on those factors when estimating counterfactual availability in Phase II, since we believe it is possible that without continued partnership with R4D, the Government of Tanzania would return to its previous approaches to forecasting or supply planning.

Additionally, we have continued to place some weight on these factors because we want to avoid disincentivizing charities we support from addressing challenges shortly before a grant renewal decision, which could occur if charities believe that we would exclude any such improvement from a forward-looking model because the improvement has already been made.

We analyzed each of the explanations that R4D gave us to underpin its best guess for counterfactual availability and came up with our own best guess by placing weight only on the explanations that we found persuasive. Our best guess is that in the counterfactual, treatment availability would be 42%, 44%, 45%, and 46% in each year from 2019-2022.24

Cost-effectiveness

Based on the assumptions described above, our best guess is that Phase II of R4D's pediatric amoxicillin program is roughly 5 times as cost-effective as cash transfers to people living in extreme poverty, which is in the range of cost-effectiveness of our top charities.25 However, this estimate is more uncertain than the cost-effectiveness estimates for our top charities.

Note that our cost-effectiveness analyses are simplified models that do not take into account a number of factors. There are limitations to this kind of cost-effectiveness analysis, and we believe that cost-effectiveness estimates such as these should not be taken literally due to the significant uncertainty around them. We provide these estimates (a) for comparative purposes and (b) because working on them helps us ensure that we are thinking through as many of the relevant issues as possible.

Other benefits

Other benefits of this grant include:

• When Phase II of the program ends, we may decide to recommend an R4D program as a top charity. This program could be a continuation of the existing pediatric amoxicillin program in Tanzania, perhaps in which the Government of Tanzania has assumed the full procurement costs of amoxicillin DT and R4D provides only technical assistance. Alternatively, R4D could start a new, similar program to increase the availability of other life-saving drugs in either Tanzania or other countries.
• We believe that there is potential for us to recommend other organizations that implement the type of program that R4D implements—assisting a government in implementing a health program—as top charities in the future. However, we have less experience analyzing this type of program, which is fairly different than most of the programs we have analyzed previously. For this reason, this grant presents us with a learning opportunity. Based on our communication with R4D in the past, we believe it is a strong partner for learning about this type of program.

Reservations about the grant

Our estimate of this program's cost-effectiveness rests mainly on our estimate of treatment availability in the counterfactual, which is based on R4D’s best guess of what the Government of Tanzania would have done in its absence. While we find R4D’s reasoning persuasive, it is possible that in R4D's absence, the government would have identified and addressed the issues in its forecasting process, or made significant improvements to its approach to supply planning.26 If our best guess underestimates treatment availability in the counterfactual, then we may be overestimating the cost-effectiveness of the program.

Our cost-effectiveness analysis also rests heavily on our assumptions about how R4D's program activities influence the government's actions, about which we are highly uncertain. We assume that R4D's assistance in supply planning improves treatment availability for a given level of procurement, but we have little evidence about the size of this effect. Our uncertainty about the effects of these activities is notable because R4D will spend a large proportion of its Phase II budget on technical assistance.27

Although we have worked to arrive at our best estimates for the above parameters, they are based on quite limited evidence.

While our estimate of this program's cost-effectiveness is in the range of cost-effectiveness of our top charities, we do not consider it sufficient justification for this grant on its own. Part of the case for this grant is its other potential benefits, and it is difficult to predict the extent to which those benefits will be realized. While this grant is a learning opportunity for us, we are uncertain about the specifics of what we can expect to learn. In addition, while it is possible that we may decide to recommend an R4D program as a top charity after Phase II, this will depend upon how our cost-effectiveness analyses and total money moved change over the next four years, which is difficult to predict.

Other reservations about this grant include:

• R4D has not yet finalized the details of its plans for monitoring and evaluation. We are uncertain about what evidence for the impact of R4D's work will be available at the end of Phase II.
• Conducting follow-up work and investigating R4D as a potential top charity will likely require a significant amount of staff time.

Plans for follow-up

• As stated above, we plan to investigate R4D as a potential top charity when Phase II of the program ends. This will involve significant follow-up work.
• As stated above, this grant is a learning opportunity for us. For this reason, we plan to conduct significant follow-up work, but we do not yet know the details of what such work will involve.
• We expect to discuss plans for monitoring and evaluation with R4D in more detail in September 2019, and to continue the discussion as R4D iterates on its process.

Our open questions include:

• To what extent will R4D's technical assistance continue to impact the availability of amoxicillin DT in health facilities?
• What monitoring and evaluation will R4D conduct to measure its impact?
• How, in general, do governments budget for and procure drugs and health commodities?
• What bottlenecks limit the availability of drugs in health facilities for a given level of national-level procurement, and how can a non-governmental organization (NGO) help mitigate those bottlenecks?
• How should we model the impact of such an NGO in a cost-effectiveness analysis?
• When randomized controlled trials are not possible, how can we gather evidence about the counterfactual of policy or technical assistance interventions?

Internal forecasts

For this grant, we are recording the following forecasts:

Confidence	Prediction	By time
40%	R4D or an R4D program is a top charity	December 2023
35%	R4D or an R4D program is a top charity and we estimate that donations to that program are at least half as cost-effective as the most cost-effective unfunded giving opportunity among top charities (i.e. where we recommend donors give on the margin)	December 2023
5%	R4D or an R4D program is a top charity and we estimate that donations to that program are at least twice as cost-effective as the most cost-effective unfunded giving opportunity among top charities (i.e. where we recommend donors give on the margin)	December 2023

Our process

We previously made a grant to support Phase I of this program. In May 2018, R4D shared full results with us from Phase I. Based on those results and several phone calls with R4D to discuss our major questions, particularly about the counterfactual, we updated and restructured our cost-effectiveness analysis of the program.

Sources

Document	Source
Althaus et al. 2017	Source (archive)
GiveWell cost-effectiveness analysis, R4D Phase II	Source
R4D Budget, Phase II	Source
R4D estimate of the counterfactual level of procurement, September 2018	Unpublished
R4D Quantification Calculations	Source
R4D, Marginal costs incurred by the government of Tanzania	Source
R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018	Unpublished
Uwemedimo et al. 2017	Source
WHO Fact sheet, pneumonia	Source (archive)

• 1

  "Pneumonia should be treated with antibiotics. The antibiotic of choice is amoxicillin dispersible tablets." WHO Fact sheet, pneumonia

• 2

  For example, R4D addressed a major concern we had when making the Phase I grant by assessing the level of clinician accuracy in diagnosing pneumonia.

• 3

  "GoT agreed in November 2017 to an amox DT co-financing plan, which will usher in a more sustainable financing structure...Agreed-upon funding levels: 2017/18: GoT 10%, 2018/19: GoT 30%, 2019/20: GoT 50%, 2020/21: GoT 60%." R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 34.

  We have considered two possible scenarios in our cost-effectiveness analysis. In Scenario 1, the government of Tanzania meets its obligations under the cost-sharing plan. We then assume that the government's contribution continues to increase to 70% of procurement costs in 2021-22 and 80% in 2022-23 (see sheet "Amoxicillin procurement P2," rows 6-13). We also consider an alternative scenario, Scenario 2, in which the government does not increase its contribution beyond what we believe is its current level (30% in 2018-19) (see sheet "Amoxicillin procurement P2," rows 15-22).

• 4

  Cammie Lee, Senior Program Director, R4D, email to GiveWell, July 12, 2019.

• 5

  In Tanzania, procurement of amoxicillin DT occurs via a "pull" supply chain, i.e. health facilities submit orders for a quantity that they determine, rather than a central agency deciding the quantity to deliver to each facility. This means that the amount procured is determined by the level of demand from health facilities, which receive only the amount that they order: "Tanzania’s supply chain system works on a pull procurement mechanism with health facilities only receiving deliveries of the supplies they order." R4D estimate of the counterfactual level of procurement, September 2018, Pg 4.

  To improve supply planning, R4D has conducted the following technical assistance activities: improving the drug request forms used by health facilities to make it easier to order amoxicillin DT, and working with the Medical Stores Department (the government agency responsible for procurement of medical supplies) to improve the consistency of procurement and to start to account for the seasonality of pneumonia incidence. For more details, see our evaluation of Phase I of the program.

• 6

  In Phase I, R4D piloted three fairly light-touch approaches to improving clinician accuracy: clinical mentoring, mobile messaging, and visual aids. Our decision not to fund scale-up is largely based on a guess that causing sustained behavior change with these approaches is unlikely. "R4D started exploring interventions to improve provider knowledge and after conducting focus groups, R4D tested the potential of three solutions: clinical mentoring, mobile messaging, and visual aids." R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 37.

  We also did a very shallow review of the evidence that light-touch training programs improve the accuracy of diagnosis and prescription by clinicians in health facilities in low-resource contexts. We relied on studies that R4D sent us as well as a search for meta-analyses on the subject. None of the studies we looked at directly addressed our question; among those we considered, we found Uwemedimo et al. 2017 to be the most relevant and, to a lesser degree, Althaus et al. 2017.

  • Uwemedimo et al. 2017 is an observational study that looks at associations between various factors and accurate pneumonia diagnosis in Malawi. Our understanding from a brief review of this study is that:
    1. There were modest differences in diagnosis accuracy between clinicians assessed as being in the 25th and 75th percentiles on adherence to guidelines: "Clinical quality was strongly associated with correct diagnosis: sensitivity was 23% in providers at the 75th percentile for guideline adherence compared with 14% for those at the 25th percentile." p. 1
    2. There was no statistically significant association found between sensitivity and clinicians being at clinics that self-reported providing Integrated Management of Childhood Illness training or supportive supervision in the prior six months: "Contextual factors, facility structural readiness, and training or supervision were not associated with sensitivity." p. 1. See also Table 3 on page 7: "Training in past 6 months" is associated with 23% sensitivity, while "No recent training" is associated with 20% sensitivity; "Supportive supervision in past 6 months" is associated with 22% sensitivity, while "No supportive supervision" is associated with 18% sensitivity.
  • Our understanding from a brief review of Althaus et al. 2017 is that it reports on a randomized controlled trial with a very small sample that found no statistically significant effect of two versions of a short intervention to improve pneumonia diagnosis.

  We put little weight on these studies because we have not reviewed the literature on the subject beyond this very shallow review and because of the issues noted above with these specific studies (the first is observational and does not evaluate an intervention, and the second has a very small sample size).

• 7

  See our Phase II cost-effectiveness analysis for more detail.

• 8

  R4D Budget, Phase II, sum of cells B3 and B12

• 9

  R4D Budget, Phase II, cell B35

• 10

  R4D Budget, Phase II, cell B4

• 11

  We define availability as the proportion of health facilities that have at least some pediatric amoxicillin in stock, such that a child with pneumonia who visited the facility could be treated immediately.

  In our cost-effectiveness analysis, our best guess is that during Phase II, treatment availability in the counterfactual would be 35-41% lower than expected for a given level of national-level procurement compared with what we expect to see with R4D's technical assistance (see sheet "Amoxicillin procurement P2," row 54).

• 12

  R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 18, left panel.

• 13

  Conversation with R4D staff, September 7, 2018

• 14

  In Tanzania, procurement of amoxicillin DT occurs via a "pull" supply chain, i.e. health facilities submit orders for a quantity that they determine, rather than a central agency deciding the quantity to deliver to each facility. This means that the amount procured is determined by the level of demand from health facilities, which receive only the amount that they order: "Tanzania’s supply chain system works on a pull procurement mechanism with health facilities only receiving deliveries of the supplies they order." R4D estimate of the counterfactual level of procurement, September 2018, Pg 4.

• 15

  Conversation with R4D staff, September 7, 2018

• 16

  Cammie Lee, R4D Market Shaping Program Director, email to GiveWell, September 25, 2018

• 17

  R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018, slide 18.

• 18

  We've estimated treatment availability in the intermediate years of the program by assuming a constant growth rate from 2017 to 2022.

  See our cost-effectiveness analysis, sheet "Amoxicillin procurement P2," row 45.

• 19

  "Months of stock procured and corresponding availability in the counterfactual, 2017-2022...Availability: 2019: 29%, 2020: 31%, 2021: 33%, 2022: 35%." R4D estimate of the counterfactual level of procurement, September 2018, Pg 1.

• 20

  In R4D Quantification Calculations, R4D calculates a 19% underestimate of demand (cell C24). However, we note an error in row 19 of this spreadsheet, where R4D uses the percentage point change rather than percent change. After correcting this error, we believe that forecasted demand in the counterfactual would have been 1.16 x 16.4 million tablets (rather than 1.08 x 16.4 million tablets), which leads us to a final estimate of 14%.

• 21
  • "Similarly, due to financing constraints, actual disbursements from GoT to purchase medicines generally fall short of budgeted amounts. On average, only 75% of the annual GoT-budgeted amounts for health products are released." R4D estimate of the counterfactual level of procurement, September 2018. Pg 2.
  • We recalculated this percentage based on R4D's data and arrived at an estimate of 77%.

• 22

  0.86 x 12 months x 0.77 = about 8 months

• 23

  For more details on improvements made to supply planning due to R4D's technical assistance, see here.

• 24

  To arrive at this best guess, we use the first and second factors (underestimation of demand and incomplete disbursement of budgeted funds) to estimate the counterfactual level of national-level procurement:

  • R4D estimates that in 2016, the Government of Tanzania's forecasting methodology would have underestimated demand by 14%. Because R4D already addressed this issue during Phase I, we apply a 10% discount to this factor. Therefore, we estimate that in the Phase II counterfactual, the government would forecast (1 - (0.14 x 90%)) = 87.4% of actual demand.
  • However, because the average proportion of budgeted funds that are actually disbursed is 77%, we estimate that the Government of Tanzania would procure 87.4% x 77% = 67% or 8.08 months’ worth of annual demand.

  We then use the third factor (room for improvement in supply planning) to estimate the counterfactual availability of amoxicillin DT in health facilities:

  • R4D told us about several improvements it made to the Government of Tanzania's supply planning. We only model two specific improvements—R4D's assistance in 1) improving the consistency of procurement and 2) accounting for the seasonality of pneumonia incidence—as discounts to counterfactual availability in our cost-effectiveness analysis. Because R4D has already begun making these improvements during Phase I, we apply a discount to these factors.
  • We assume that in the absence of R4D's program, the government would gradually improve its supply planning, so we decrease these supply planning discounts over time. For each year of Phase II, we multiply our estimate of counterfactual national-level procurement (8.08 months’ worth of annual demand) by these supply planning discounts, resulting in availability estimates of 42%, 44%, 45%, and 46% from 2019-2022.

  For more details, see cell notes in rows 53 and 55, sheet "Amoxicillin procurement P2" in our cost-effectiveness analysis.

• 25

  See our cost-effectiveness analysis for more details.

  In brief, the structure of the model is:

  • The government of Tanzania would have spent X and achieved G benefits in the counterfactual.
  • R4D now spends Y, causing the government to spend (X+Z), and the program overall achieves (G+R) benefits.
  • In the "Amoxicillin procurement P2" sheet in our cost-effectiveness analysis, we calculate the cost-effectiveness of the program as (Y+Z) costs to achieve R benefits.
  • However, Z of these costs are spent by the government and would otherwise have been spent on some other program achieving C benefits. Therefore, in the "Leverage P2" sheet, we subtract Z from the total costs, but also subtract C from the total benefits. This gives the final estimate of cost effectiveness: Y costs to produce (R-C) benefits.

• 26

  We do assume that the government makes some improvements in the counterfactual (for example, to supply planning), which is why our best guess for counterfactual availability grows over time during the course of Phase II. However, we may have underestimated that rate of improvement.

• 27

  In total, between 48% ($3,273,651/$6,783,498 in Scenario 2) and 52% ($3,273,651/$6,258,498 in Scenario 1) of R4D's expenditures in Phase II will be spent on technical assistance. See our cost-effectiveness analysis, sheet "Amoxicillin procurement P2."