Evidence Action — Syphilis Screening and Treatment in Pregnancy

Note: This page summarizes the rationale behind a GiveWell Incubation Grant to Evidence Action. Evidence Action staff reviewed this page prior to publication.

Published: April 2021

The organization

Evidence Action is the parent organization of one of GiveWell’s current top charities (Deworm the World Initiative) and one standout charity (Dispensers for Safe Water). GiveWell also partners with and has recommended grants to Evidence Action Accelerator to support a dedicated incubator portfolio focused on GiveWell-aligned, evidence-backed, cost-effective interventions.

The intervention

There is reasonably strong evidence showing that treating syphilis in pregnancy with benzathine penicillin G (BPG) is effective in substantially reducing adverse outcomes that can result from syphilis in pregnancy, such as neonatal mortality, stillbirth, preterm birth, and incidence of congenital syphilis. See our report on screening and treatment of syphilis in pregnancy for our review of this evidence.

In Liberia, 98% of pregnant people attend antenatal care, and the majority of those attend antenatal care early in their pregnancy.3 HIV screening and treatment rates at antenatal care visits are high.4

However, fewer than 10% of those attending antenatal care are screened for syphilis,5 even though an estimated 2.2% of people attending antenatal care in Liberia have an active syphilis infection.6

In 2015, WHO prequalified a dual HIV/syphilis rapid diagnostic test that can be used to simultaneously diagnose both HIV and syphilis in under 20 minutes.7 With this grant, Evidence Action will provide technical assistance to the Liberian government to facilitate a switch from HIV rapid tests to dual HIV/syphilis rapid tests and to scale up syphilis treatment in routine antenatal care nationally. This presents an opportunity to leverage the existing antenatal HIV screening platform to increase screening and treatment of syphilis in pregnancy.

Why Evidence Action chose to work in Liberia

Evidence Action determined that Liberia is a country particularly well-suited to this intervention because of its relatively high baseline prevalence and low screening rates of syphilis in pregnancy, high rate of antenatal care attendance, and high HIV screening rates at antenatal care visits. Additionally, Evidence Action told us that the Liberian government expressed interest in including dual HIV/syphilis rapid tests in Liberia’s antenatal care protocols.8 We have not confirmed this claim independently.

Through initial work conducted through the Evidence Action Accelerator, Evidence Action built relationships with Liberian policymakers responsible for implementing the country’s National AIDS and STI Control Program (NACP). In early 2020, Evidence Action staff partnered with the NACP to conduct a pilot of the dual test for HIV and syphilis, while supporting Liberia in integrating the rapid dual test into its national guidelines.9 This grant will enable Evidence Action to provide technical assistance to support Liberia in rolling out the dual HIV/syphilis test and strengthening syphilis treatment for pregnant people nationally.

Planned activities

This five-year, $3.9 million grant will allow Evidence Action to support the roll out of the rapid dual syphilis/HIV test in Liberia. Evidence Action has told us that its activities will include:10

• Continuing to support Liberia in updating its national antenatal care guidelines to include guidelines on dual HIV/syphilis rapid test administration and treatment of syphilis
• Continuing to support Liberia in the procurement of both the dual rapid tests and penicillin for treatment of syphilis, via the Global Fund and via direct procurement11
• Supporting Liberia in managing and monitoring the supply chain for both dual rapid tests and penicillin
• Continuing to support the revision of facility reporting templates to ensure integration of syphilis screening and treatment
• Working with the NACP to train providers to use the dual rapid test and provide treatment for syphilis
• Working with the NACP to provide mentorship to health facilities (see footnote for details)12
• Monitoring program activities and the impact of the program on coverage of syphilis screening and treatment among pregnant people

Budget

Evidence Action's budget for the program breaks down as follows:13

• Personnel – approximately $1,425,000, including both local and global staff.
• Program implementation – approximately $925,000, including the costs of procuring dual tests and BPG, health provider training and mentorship, printing materials, and stakeholder meetings. Around $300,000 of those costs are considered an “as-needed” expense.14 These funds will be released in the event that there are gaps in the supply of key commodities and additional procurement is needed.15
• Monitoring and evaluation – approximately $575,000, including the cost of monitoring and evaluation staff and local enumerators for the comprehensive facility survey.
  • We may recommend an additional grant of approximately $500,000 to cover an enhanced monitoring and evaluation plan if the data obtained through the comprehensive facility survey is insufficient to establish whether providers are adhering to treatment protocols, which we believe is likely.
• Operating costs and travel – approximately $225,000
• Finance, administrative, and indirect costs – approximately $750,000.16 Of this, $94,000 are associated with the potential additional procurement of dual tests and will only be disbursed as needed.17

The case for the grant

We are recommending this grant for the following reasons:

• We consider this program to be cost-effective. More below.
• We think that it’s likely that the Liberian government will switch to procuring dual tests instead of single HIV tests for use in antenatal care settings.
• We think it’s likely that the government of Liberia will continue to use the dual test in antenatal care clinics with little or no support from Evidence Action after five years.
• In the absence of additional philanthropic support, we believe it is unlikely that there is another organization that would partner with the Liberian government and provide technical assistance on this program. We also believe it is unlikely that the Liberian government would transition to using the dual rapid tests nationwide in the next five years without support.
• We are fairly confident that we will be able to learn from this grant because Evidence Action has outlined an initial lighter-touch evaluation plan with the option to execute a much more rigorous evaluation if data gathered under the initial plan is judged to be inadequate.
• If the intervention is successful, that may lead to future opportunities for GiveWell or others to direct funding toward technical assistance work for similar transitions from HIV rapid tests to dual HIV/syphilis rapid tests in other countries with relatively high syphilis burdens.
• We expect the grant to provide GiveWell with an opportunity to learn more about technical assistance programs in general, which may put GiveWell in a better position to evaluate other technical assistance programs in the future.

Cost-effectiveness

Based on our cost-effectiveness analysis of Evidence Action’s syphilis screening and treatment program in Liberia, we believe the program is in the range of cost-effectiveness of programs we expect to direct funding to, as of 2021.18

Note that our cost-effectiveness analyses are simplified models that do not take into account a number of factors. There are limitations to this kind of cost-effectiveness analysis, and we believe that cost-effectiveness estimates such as these should not be taken literally due to the significant uncertainty around them. We provide these estimates (a) for comparative purposes and (b) because working on them helps us ensure that we are thinking through as many of the relevant issues as possible.

There are a number of uncertainties in the parameters we use in this analysis, described below. In particular, we rely on highly uncertain assumptions about the value of preventing adverse pregnancy outcomes and neonatal deaths as well as uncertain estimates of the treatment rates that will be achieved by the program.

Additionally, since this program involves providing technical assistance to a government rather than directly delivering a good or service, we are less confident than usual about our estimate of the number of individuals that will be impacted as a result of Evidence Action’s activities. We are also uncertain about what proportion of the impact of the intervention to attribute to Evidence Action and how much to attribute to other actors.

Internal and external validity adjustments

In our cost-effectiveness model, we have applied several adjustments to academic estimates of the treatment effectiveness of BPG (taken from Blencowe et al. 2011) and rates of adverse pregnancy outcomes due to syphilis (taken from Gomez et al. 2013). We have applied adjustments for internal validity (i.e., the likelihood that the results of the studies we rely on were accurate) but have decided not to apply adjustments for external validity (i.e., the likelihood that the academic estimates are applicable to the Liberian context).19

• Internal validity – We make several internal validity adjustments. For example, some of the studies in Gomez et al. 2013 had unusually high proportions of patients with latent and asymptomatic cases,20 for which we expect the likelihood of adverse pregnancy outcomes to be lower. Because of this, we believe that Gomez et al. 2013 may have underestimated the likelihood of adverse pregnancy outcomes for typical syphilis cases. Accordingly, we apply an upward internal validity adjustment. We also apply a downward adjustment to account for the fact that many of the studies reviewed in Blencowe et al. 2011 did not control for potential confounding factors.21 See this spreadsheet for more detail on the reasoning behind our internal validity adjustments.
• External validity – We do not apply an external validity adjustment because there is high consistency across studies in various contexts22 and because we have no particular reason to believe the program will be more or less effective in Liberia. See this spreadsheet for more detail on our reasoning for not making an external validity adjustment.

Key judgment calls in our cost-effectiveness analysis

In our cost-effectiveness analysis of this program, we make difficult judgment calls about the following:

• Timing of treatment. The timing of syphilis treatment within a pregnancy may impact the likelihood of various relevant health outcomes, but we are unsure about the size of this impact.23
• Benefits to subsequent pregnancies. We have made a number of simplifying adjustments to attempt to estimate the effect of syphilis treatment during pregnancy on the future pregnancies of those treated; however, these effects are complex, and we are unsure whether we have modeled them accurately.24
• The impact of averting congenital syphilis on a child’s long-term quality of life. We are uncertain about the long-term burden on children who are born with congenital syphilis, so we are unsure of how to value averting that burden.
• The comparative moral value of relevant outcomes. We are highly uncertain about the moral value of averting neonatal mortality, stillbirth, miscarriage, and congenital syphilis compared to the moral value of outcomes of other programs we would consider recommending funding.25
• The prevalence of syphilis. We considered two sources of syphilis prevalence estimates in Liberia: (1) the NACP sentinel surveillance survey in 2017, which takes a convenience sample of geographically representative facilities in Liberia26 and (2) Korenromp et al. 2019, which generates estimates using the Spectrum-STI model based on facility-reported data from Liberia prior to 2016.27 We believe that the 2017 NACP sentinel surveillance survey provides the best available estimate for prevalence of syphilis in pregnancy in Liberia. However, this estimate may be biased due to the lack of randomization, and our cost-effectiveness analysis is highly sensitive to this input.28
• Adjustments for diagnostic test type. Our understanding is that the most accurate way to detect active syphilis infections is to administer two types of diagnostic tests: a treponemal test and a non-treponemal test.29 The studies that we have relied on to form our conclusions on this program do not consistently use the same type of diagnostic test.30 We make large and uncertain adjustments to account for the differences in performance of the diagnostic tests used in these studies and the rapid dual HIV/syphilis tests that will be used in Liberia.31
• Screening rates. We have assumed that, once the switch from HIV rapid tests to dual HIV/syphilis rapid tests is complete, 95% of pregnant people we would expect to be screened for HIV in the absence of this program will be screened for both HIV and syphilis.32
• Treatment rates. Our current best guess is that 40% of pregnant people who test positive for syphilis in the first year of the program will be treated and that this rate will increase to 52% over the five years of the grant. This guess is highly uncertain.33
• The duration of the program and whether it fully transitions to the government. It seems possible to us that the program could eventually be shut down or that Evidence Action may never transition the program to the government. We believe that a shutdown of this program is unlikely, as Evidence Action plans to maintain an ongoing presence in Liberia and we expect there to be momentum in procuring dual HIV/syphilis tests once the switch has been integrated into the Global Fund’s procurement plan for Liberia.34
• Whether there is a decline in quality during or after a potential transition to the government. For the sake of simplicity, we have accounted for the risk of a decline in quality during and after transitioning the program to the government by using more conservative treatment rate estimates for that time period.35
• Cost of diagnostic tests. The Global Fund and UNICEF did a one-off procurement of some dual HIV/syphilis rapid tests in 2019 for a pilot of this intervention in Liberia, at a cost of $1.50 per test. An additional procurement occurred last year via the Global Fund at this price, and additional procurement is expected via this channel in the coming years at similar or lower costs per test. The Global Fund is currently procuring HIV single rapid tests at $0.80 per test on behalf of Liberia.36 We are unsure whether the Global Fund will completely switch to procuring dual HIV/syphilis tests for antenatal care. It seems likely that it will start procuring the dual tests in the next year, though there is some chance it will never switch to procuring the dual tests. We have assumed that the cost to the Global Fund of taking on procurement of the dual HIV/syphilis test would be the marginal cost of purchasing a dual test over an HIV single test—that is, $0.70 per test.37
• Leverage adjustments. In many of our cost-effectiveness analyses, we adjust for "leverage" (influencing other actors to spend resources in ways that we think are better than what they would have done otherwise) and/or "funging" (crowding out funding that other actors would have otherwise donated). As part of this program, Evidence Action plans to leverage (a) funds from the Global Fund for commodity procurement and (b) resources and capacity from the Liberian government and health care system.
  • We’ve accounted for leveraging the Global Fund by assuming that its spending in the absence of this program would be 38% as cost-effective as our recommended grants to the Against Malaria Foundation.38
  • We have chosen not to include a leveraging adjustment for the government’s spending. Evidence Action has told us that the NACP is already planning to conduct training for current and new health providers to incorporate recent changes to HIV screening and treatment protocols (unrelated to the addition of syphilis screening). Evidence Action is partnering with NACP to integrate training and mentorship for syphilis screening and treatment in this upcoming HIV training as well as in the typical supervision visits that NACP already plans to conduct.39 Our impression from Evidence Action is that incorporating syphilis training and mentorship into these existing plans will not incur significant additional costs to NACP, but we are uncertain whether this assumption is accurate.
• Adjusting for reduced value of future benefits. In some of our cost-effectiveness analyses, we apply a discount to adjust for the value of benefits that occur at different times. For a number of reasons, we might expect a benefit that occurs in the future to be less valuable than the same benefit today. For this particular grant, we’ve chosen not to apply a discount for the potentially reduced value of future benefits.40 We are particularly uncertain about this input and do not see a feasible way to obtain better data on it.
• What would happen without Evidence Action’s support? In our cost-effectiveness model, we assume that Evidence Action’s support will speed up the switch from HIV rapid tests to dual HIV/syphilis tests and improve the coverage of treatment for syphilis in Liberia, relative to what we expect would happen without Evidence Action’s support. Our best guess is that, without Evidence Action’s support:
  • Rates of syphilis screening in pregnancy in Liberia would increase by three percentage points per year for the next five years.
  • Five years from now, the Liberian government would begin scaling up the syphilis screening and treatment program.
  • Eight years from now, the program would be operating at full scale, with slightly lower treatment coverage than we would expect with Evidence Action’s support.

  These assumptions imply that Evidence Action’s support will result in full implementation of the program five years earlier than without its support. However, we think it’s possible that, in Evidence Action’s absence, the Liberian government would have implemented and scaled up the program much earlier or later than we have guessed.41

We will update our cost-effectiveness analysis periodically, as new information becomes available.

Room for more funding

What we learn from Evidence Action’s program in Liberia could inform our views of how promising providing technical assistance to switch to dual HIV/syphilis tests is in general, potentially unlocking significant room for more funding in this area.

Evidence Action anticipates that there are additional countries where (a) syphilis screening and treatment in pregnancy has not yet been implemented at scale, (b) the political climate is favorable to such implementation, and (c) the baseline prevalence of syphilis is sufficiently high to make this a potentially worthwhile investment.42 We have not yet vetted these claims.

Risks and reservations

• Implementation risks. There are several ways in which the implementation of the program could go less well than we expect:
  • We have assumed that the program will be acceptable to providers and patients at antenatal-care clinics in Liberia, but there is some evidence of resistance to penicillin treatment during pregnancy in other countries due to fears of allergic or adverse reactions.43
  • We have assumed that providers will be able to distinguish between the HIV single test and the dual HIV/syphilis test. Our understanding is that HIV single tests can be used to test any patient for HIV, regardless of whether the patient is pregnant, but dual HIV/syphilis tests would only be used for pregnant patients and their sexual partners. The added complexity of using a different test for pregnant people than for the rest of the primary care population could pose a larger problem than we are assuming.
  • More broadly, training health providers to change their behavior may be more difficult than we have assumed due to potential issues such as high staff turnover, staff shortages, or inadequate government capacity to provide ongoing supervision.
  • We expect Evidence Action’s support to mitigate problems with the dual test and BPG supply chains at the national and regional levels, but it could be that commodity shortages and stockouts will disrupt syphilis screening and treatment services. We are currently assuming that supply shortages will result in roughly 30% of pregnant people not being treated. 44
  • It is possible that the Global Fund may never support the procurement of dual HIV/syphilis tests in Liberia. In such a case, full transition of the program to the government will likely not be feasible. Technical assistance for the program could be considerably less cost-effective than we’ve modeled if the program is never transitioned to the government.
• Monitoring and evaluation. We may be overestimating Evidence Action’s ability to successfully integrate syphilis screening and treatment into facility reporting. We may find that facility-reported data is not accurate or reliable enough for us to estimate screening and treatment coverage rates among pregnant people. We could also find that a direct observation survey is infeasible due to a lack of government buy-in or concerns over disruption to facilities. In such a case, we may not have an accurate estimate of coverage to use to model the impact of the program at the end of the grant.
• Budget. We have not included any additional training costs in our estimates of the cost of the program (beyond the training and mentorship line included in the budget) because it is our understanding that training for administering the dual HIV/syphilis tests can be integrated into Liberia’s routine training platform for prevention of mother-to-child transmission (PMTCT) of HIV, and thus the government trainings that occur will not be made significantly more expensive due to the inclusion of syphilis content. However, this assumption could be incorrect. We have also assumed that Evidence Action is accurately forecasting its budgetary needs.
• Choice of partner. It is possible that there are other organizations that are better suited to do this work. Evidence Action had no existing presence in Liberia before it began scoping out this program, and the relationships that Evidence Action has built within the Liberian government are relatively new. However, Evidence Action has told us that its team has experience providing technical assistance on the transition to new health products and the scale up of new health programs in national-level health systems.45 Our impression is that there are currently no non-governmental actors working in Liberia to support the rollout of the dual HIV/syphilis rapid tests.46 However, it seems plausible that implementing organizations who have supported PMTCT interventions might also be capable of supporting this program.

Plans for follow up

An assessment of changes in syphilis screening and treatment coverage in Liberia after the introduction of dual HIV/syphilis tests could have meaningful implications for how we estimate the cost-effectiveness of future grants for similar programs. Evidence Action has developed a default monitoring plan and an optional enhanced monitoring plan to assess the impact of this program. We believe we will be able to retrospectively evaluate the success of the program after five years.

Monitoring and evaluation

The main outcomes of this program that we are interested in are coverage among pregnant people of (1) syphilis screening and (2) syphilis treatment. Evidence Action views coverage as a function of supply availability, provider knowledge, and provider and patient behavior. It has developed a monitoring and evaluation plan to measure each of these components of coverage.47

Evidence Action’s default evaluation plan includes two primary components:

• Comprehensive facility surveys
• Triangulation of national, county, and facility records

The comprehensive facility surveys are intended to assess supply availability and provider knowledge.48 Evidence Action plans to contract a survey firm to conduct these surveys approximately four months after each phase of training.49 The surveys will use simple random sampling, likely stratified by county.50 Facility surveys will involve:

• Assessing supply availability of dual tests and BPG by (a) directly observing and counting facility inventories, and (b) reviewing documentation of a facility’s supply requests, supply receipt records, and dispensing logs51
• Reviewing facility registers and/or patient charts for data on screening and treatment52
• Surveying providers, which will include (a) assessing provider knowledge of screening and treatment protocols through skills tests and oral knowledge tests, and (b) conducting qualitative interviews to understand barriers to patient or provider adherence to the treatment guidelines as well as other challenges53

Evidence Action’s current plan is to assess provider and patient behavior by triangulating facility register data, comprehensive facility survey data, and supply data at the facility, county, and national levels.54 However, it will be more difficult and costly to measure this factor compared to supply availability and provider knowledge. Evidence Action believes that, within the first year of the program, it will be able to develop a better understanding of the quality of available data related to patient and provider behavior.55

It is possible that the available data will be inadequate for estimating screening and treatment coverage. To account for this possibility, Evidence Action has developed the option of an enhanced monitoring plan.56 The enhanced plan involves the addition of a direct observation survey in which enumerators are stationed within health facilities to observe screening and treatment.57 We have planned for the possibility of recommending an additional grant of approximately $500,000 to support the enhanced monitoring plan.

We believe that this option-based approach carries some inherent risks. We think the most likely potential issue is that GiveWell and Evidence Action disagree on whether there is a need to conduct more rigorous direct observation work. Evidence Action does not currently have enough information to concretely specify how it will triangulate screening and treatment rates, so Evidence Action and GiveWell have not yet articulated and agreed upon the precise findings that would necessitate enhanced monitoring.

Internal forecasts

For this grant, we are recording the following forecasts:

Confidence	Prediction	By time
50%	We will assess this work as being more than 10 times as cost-effective as cash transfers at the end of the grant.	2026
60%	Within five years, Evidence Action will have largely transitioned this work to the Liberian government and will be spending less than $150,000 per year in-country on the program (including ongoing political support, monitoring support, etc.).	2026
25%	By the end of this grant, GiveWell will recommend at least one other grant for syphilis screening and treatment in pregnancy.	2026

Our process

We previously recommended a grant to Evidence Action Accelerator to build out a dedicated portfolio focused on scoping and bringing to scale evidence-backed, cost-effective interventions. As part of this work, Evidence Action investigated opportunities to address maternal syphilis, spoke with experts, and conducted initial in-country scoping visits to Indonesia, Liberia, and Ghana.58 In early 2020, Evidence Action also partnered with the NACP to conduct a pilot of the program in Liberia.

We have had several calls and many email exchanges with Evidence Action to understand its proposed activities, programmatic risks, monitoring and evaluation plan, and budget. We also had one call with an external expert on congenital syphilis.59

Using our cost-effectiveness analysis of syphilis screening and treatment in pregnancy as a base, we created a program-specific cost-effectiveness analysis of Evidence Action’s program in Liberia. This cost-effectiveness analysis was based partly on our conversations with Evidence Action and partly on our research.

Sources

Document	Source
Althabe et al. 2019	Source (archive)
Ansbro et al. 2015	Unpublished
Asundep et al. 2014	Source
Betran et al. 2018	Source (archive)
Blencowe et al. 2011	Source (archive)
Bronzan et al. 2007	Source
Clinton Health Access Initiative, 2018 slide deck	Source (archive)
Das 2011	Source (archive)
Evidence Action, 5-yr plan for maternal syphilis program in Liberia, 2020	Unpublished
Evidence Action, Back-of-the-envelope-calculation cost-effectiveness analysis, 2020	Unpublished
Evidence Action, Cost-effectiveness analysis of syphilis screening and treatment, 2020	Source
Evidence Action, Liberia maternal syphilis program budget, 2020	Unpublished
Evidence Action, Liberia maternal syphilis program budget, 2020 (public)	Source
Evidence Action, Liberia Scoping Visit Report Addendum, 2020	Unpublished
Evidence Action, Maternal Syphilis M&E in Liberia: Description of the Overall Approach, 2020	Source
Evidence Action, Maternal syphilis research overview, 2020	Source
Evidence Action, Maternal Syphilis: Monitoring Deep Dive slide presentation, 2020	Unpublished
Evidence Action, Scoping Visit to Indonesia: Summary of Findings, 2019	Unpublished
Evidence Action, Scoping Visit to Indonesia: Summary of Findings, 2019 (public)	Source
Evidence Action, Scoping Visit to Liberia: Summary of Findings, 2020	Unpublished
Evidence Action, Scoping Visit to Liberia: Summary of Findings, 2020 (public)	Source
Flores et al. 2015	Source
Ghanem, Ram, and Rice 2020	Source (archive)
GiveWell, Cost-effectiveness analysis of Evidence Action technical assistance to Liberia for syphilis screening and treatment, 2020	Source
GiveWell, Cost-effectiveness analysis, 2020, Version 2	Source
GiveWell, Internal and external validity adjustments for Evidence Action's Liberia maternal syphilis program, 2020	Source
GiveWell, Reweighting of sentinel surveillance prevalence, 2020	Source
GiveWell, Summary of maternal syphilis diagnostic test adjustments, 2020	Source
GiveWell, Syphilis in pregnancy discount rate explanation, 2020	Source
GiveWell's non-verbatim summary of a conversation with Dr. Melanie Taylor, June 2, 2020	Source
Gomez et al. 2013	Source (archive)
Ham et al. 2015	Source (archive)
Hawkes et al. 2013	Source
Holmes et al. 2017	Source (archive)
Joint United Nations Programme on HIV/AIDS, AIDSinfo, Coverage of pregnant women who receive ARV for PMTCT	Source (archive)
Joint United Nations Programme on HIV/AIDS, AIDSinfo, HIV testing among pregnant women	Source (archive)
Korenromp et al. 2019	Source (archive)
Korenromp et al. 2019, Supplementary Materials S2	Source (archive)
Liberia Institute of Statistics and Geo-Information Services, Liberia Demographic and Health Survey 2013	Source (archive)
Liberia Institute of Statistics and Geo-Information Services, Liberia Demographic and Health Survey 2019-2020	Source (archive)
Liberian Ministry of Health, Liberia Service Availability and Readiness Assessment (SARA) and Quality of Care Report, 2016	Unpublished
Liberian Ministry of Health, Service Availability and Readiness Assessment (SARA) & Quality of Care (QOC) 2018 Final Report	Unpublished
Munkhuu et al. 2009	Source
National AIDS and STI Control Program, 2017 HIV Sentinel Surveillance Survey for Women Attending Antenatal Care Services in Liberia	Unpublished
Nkamba et al. 2017	Source (archive)
Nnko et al. 2016	Source
Nurse-Findlay et al. 2017	Source (archive)
Raatikainen, Heiskanen, and Heinonen 2007	Source (archive)
Roush 2017	Source (archive)
Smith et al. 2015	Source (archive)
Strasser et al. 2012	Source
Swartzendruber et al. 2015	Source (archive)
Wang et al. 2015	Source (archive)
World Health Organization, "Women screened for syphilis during pregnancy: Data by country"	Source (archive)
World Health Organization, Information Note on the Use of Dual HIV/Syphilis Rapid Diagnostic Tests, 2017	Source (archive)
World Health Organization, Public Report for STANDARD Q HIV/Syphilis Combo Test, 2020	Source (archive)
Young et al. 2018	Source (archive)

• 1

  $3,894,342-$100,724 in unspent funding = $3,793,618 spent from this grant. $3,793,618+$1,214,089 = $5,007,707 in total funding including the exit grant. Note: the total funding figure differs slightly from the sum of these two grants due to rounding of sub-dollar amounts.

• 2

  Evidence Action confirmed that they have $100,724 remaining in an email to GiveWell, July 16, 2024 (unpublished).

• 3
  • “Table 12 shows that 98% of mothers reported receiving prenatal care from a health professional at least once for their most recent birth in the 5-year period before the survey, which is slightly higher than the 96% figure reported in 2013.” Liberia Institute of Statistics and Geo-Information Services, Liberia Demographic and Health Survey 2019-2020, p. 24.
  • “Table 9.2 also shows that the majority of women (67 percent) had their first visit at less than four months of pregnancy, as recommended. Twenty-four percent of women had their first visit in the fourth to the fifth month of pregnancy, 6 percent had their first visit in the sixth to the seventh month of pregnancy, and 1 percent had their first visit from the eighth month onwards.” Liberia Institute of Statistics and Geo-Information Services, Liberia Demographic and Health Survey 2013, p. 120.

• 4
  • Coverage of pregnant people who receive antiretrovirals for PMTCT in Liberia has increased from 28% in 2014 to 90% in 2019. See Joint United Nations Programme on HIV/AIDS, AIDSinfo, Coverage of pregnant women who receive ARV for PMTCT.
  • HIV screening of pregnant people in Liberia was 99% in 2019. See Joint United Nations Programme on HIV/AIDS, AIDSinfo, HIV testing among pregnant women.

• 5
  • In 2015, 7.8% of pregnant people accessing antenatal care services were tested for syphilis. See World Health Organization, "Women screened for syphilis during pregnancy: Data by country".
  • Note that the data show a large spike in syphilis screening rates in 2018. Evidence Action has confirmed that this spike was due to a data entry error. Evidence Action, Scoping Visit to Liberia: Summary of Findings, 2020 (unpublished).

• 6

  We considered two sources of syphilis prevalence estimates in Liberia: National AIDS and STI Control Program, 2017 HIV Sentinel Surveillance Survey for Women Attending Antenatal Care Services in Liberia (unpublished) and Korenromp et al. 2019.

  • The Liberian National AIDS and STI Control Program (NACP) sentinel survey reported that, among pregnant people attending antenatal care services who were screened for syphilis, 2.7% tested positive. This is an unweighted average of the prevalence rates for the 35 surveyed facilities.
  • Korenromp et al. 2019 generated estimates for maternal syphilis using the Spectrum-STI model. The model estimated that there was an 8.5% prevalence of maternal syphilis in Liberia in 2016.
    • “Maternal syphilis estimates were generated using the Spectrum-STI model, fitted to sentinel surveys and routine testing of pregnant women during antenatal care (ANC) and other representative population data.” Korenromp et al. 2019, p. 1.
    • See Korenromp et al. 2019, Supplementary Materials S2, column "Maternal syphilis prevalence," row "Liberia, 2016."
  • Both the NACP sentinel survey and Spectrum-STI estimates are subject to potential biases, but we believe the NACP sentinel survey provides the best available prevalence estimate. We re-weighted the NACP sentinel survey estimates by clinic catchment area and county population to get an estimate of 2.2% syphilis prevalence among pregnant people attending antenatal care. See our calculations in this spreadsheet.
  • See the cost-effectiveness section of this page for more detail on the prevalence estimates.

• 7
  • “In 2015, the first multiplex rapid diagnostic test (RDT) for detection of anti- HIV and anti-treponema pallidum (dual HIV/syphilis RDT) was listed on the WHO list of prequalified in vitro diagnostic products (IVD).” World Health Organization, Information Note on the Use of Dual HIV/Syphilis Rapid Diagnostic Tests, 2017, p. 1.
  • "Time to result: 15 minutes." World Health Organization, Public Report for STANDARD Q HIV/Syphilis Combo Test, 2020, p. 6.

• 8

  Jeff Grosz, Accelerator Senior Director and Anna Konstantinova, Accelerator Research Manager, Evidence Action, multiple conversations with GiveWell (unpublished),

• 9

  On August 13, 2020, Evidence Action forwarded to GiveWell email exchanges between Evidence Action and the Liberian government to support these claims. See also Evidence Action, Liberia Scoping Visit Report Addendum, 2020 (unpublished).

• 10

  Jeff Grosz, Accelerator Senior Director and Anna Konstantinova, Accelerator Research Manager, Evidence Action, multiple conversations with GiveWell (unpublished).

• 11

  Our impression from multiple conversations with Evidence Action is that it plans to assist the Liberian government in applying to the Global Fund to receive HIV/syphilis dual tests rather than HIV-only tests for use with pregnant people. Until the Global Fund makes this switch, which may take some time because of its lengthy procurement cycles, Evidence Action will assist Liberia through direct procurement of the dual rapid tests and penicillin. Evidence Action would also continue to assist the Liberian government if Liberia's application to the Global Fund is unsuccessful.

• 12
  • The mentorship component of the program will include (a) integrating mentorship for syphilis screening and treatment in pregnancy into county health team visits to county health facilities, and (b) targeting additional mentoring to facilities where coverage for syphilis screening and treatment in pregnancy fall below expectations.
  • For the facilities that require additional mentoring, Evidence Action hopes, where possible, to leverage local NGOs or existing government supervision site visits that already provide mentorship to Liberian health facilities. However, Evidence Action is uncertain of how many facilities will need additional mentoring for syphilis screening and treatment and whether it will be feasible to conduct targeted mentoring primarily through local NGOs.
  • Evidence Action plans to target facilities that need additional mentorship by comparing syphilis screening rates in pregnancy to both historical and current data on HIV testing in pregnancy; triangulating syphilis screening rates with facility dual test orders; and triangulating syphilis treatment rates with testing volumes, numbers treated, and estimated prevalence of syphilis in pregnancy.
  • If Evidence Action is unable to effectively target facilities due to missing or unreliable data, it may instead expand mentorship more broadly across all health facilities.
  • Anna Konstantinova, Accelerator Research Manager, Accelerator, Evidence Action, conversation with GiveWell, July 27, 2020 (unpublished).

• 13

  See Evidence Action, Liberia maternal syphilis program budget, 2020 (public).

• 14
  • "The dual test procurement we consider ‘as needed’ is the $293k for Y2." Jeff Grosz, Accelerator Senior Director, Evidence Action, Slack conversation with GiveWell, August 4, 2020 (unpublished).
  • Procurement costs are the bulk of the "direct program support" costs listed in Evidence Action, Liberia maternal syphilis program budget, 2020 (public).

• 15
  • Evidence Action, Liberia maternal syphilis program budget, 2020 (public)
  • Evidence Action, Liberia maternal syphilis program budget, 2020 (an unpublished version of the budget with additional detail about procurement costs)
  • Jeff Grosz, Accelerator Senior Director, Evidence Action, Slack conversation with GiveWell, August 4, 2020 (unpublished)

• 16

  This amount is the total of "Regional F&A Support" ($160,926) and "IDC @ 18%" ($594,052). Evidence Action, Liberia maternal syphilis program budget, 2020 (public)

• 17
  • "The dual test procurement we consider ‘as needed’ is the $293k for Y2. . . . When we add regional finance/admin and global indirect costs, that comes to $387k." Jeff Grosz, Accelerator Senior Director, Evidence Action, Slack conversation with GiveWell, August 4, 2020 (unpublished)
  • $387,000 - $293,000 = $94,000

• 18
  • This year, we expect the marginal dollar we recommend will be 10 times as cost-effective as GiveDirectly. For an example of the cost-effectiveness of our recommendations, see this page. Assuming that GiveWell recommends an additional grant of roughly $500,000 for enhanced monitoring (which we think is likely), we estimate that this program will be 12 times as cost-effective as GiveDirectly.
    • See our cost-effectiveness analysis, “Cost per equivalent life saved,” “With enhanced monitoring.”
  • If we don't recommend the additional grant of $500,000, we estimate that this program will be about 14 times as cost-effective as GiveDirectly.
    • See our cost-effectiveness analysis, “Cost per equivalent life saved,” “Without enhanced monitoring.”
  • Read more about Evidence Action’s monitoring plans in the “Monitoring and Evaluation” section of this page.

• 19

  See the ”treatment effects” tab of our cost effectiveness analysis.

• 20

  For example, Ingraham, 1940-49, studied women who were asymptomatic. Characteristics of each of the studies analyzed in Gomez et al. 2013 are described in Table 1, pp. 220-221.

• 21
  • Blencowe et al. 2011, p. 10, Table 1 reports that the studies used to estimate treatment effectiveness of BPG for each of our outcomes of interest (stillbirth, neonatal mortality, preterm birth, and congenital syphilis) had “no or insufficient controlling for important potential confounding variables.”
  • We expect that pregnant people who attend early antenatal care and are screened and treated for syphilis may differ from those who do not in key ways that affect their risk for adverse pregnancy outcomes.
    • “Under- or non-attendance associated with social and health behavioral risk factors: unmarried status, lower educational level, young maternal age, smoking and alcohol use.” Raatikainen, Heiskanen, and Heinonen 2007, Abstract.
  • We also expect that attending antenatal care provides additional benefits outside of syphilis screening and treatment that are likely to reduce the likelihood of adverse pregnancy outcomes. More frequent antenatal care attendance appears to be correlated with fewer adverse pregnancy outcomes, and the difference appears to be large, though we have not looked deeply into the literature.
    • “After logistic regression analyses, controlling for confounding, there were significantly more low birth weight infants in under- and non-attenders (OR:s with 95% CI:s: 9.18 (6.65–12.68) and 5.46 (3.90–7.65), respectively) more fetal deaths (OR:s 12.05 (5.95–24.40) and 5.19 (2.04–13.22), respectively) and more neonatal deaths (OR:s 10.03 (3.85–26.13) and 8.66 (3.59–20.86), respectively). . . . Even when birth takes place in hospital, non- or under-attendance at antenatal care carries a substantially elevated risk of severe adverse pregnancy outcome.” Raatikainen, Heiskanen, and Heinonen 2007, Abstract.

• 22

  For example, Blencowe et al. 2011 found that the effect estimates of BPG treatment "were large and remarkably consistent across studies." Abstract.

• 23

  "Our review has shown that the timing of antenatal care interventions makes a significant difference in the risk of having an adverse outcome due to syphilis. Women who sought care in the first two trimesters of their pregnancy, and received the appropriate intervention, were more likely to have a healthy infant, compared to women screened and treated in the third trimester." Hawkes et al. 2013, Abstract.

• 24

  For more information about these adjustments, see this spreadsheet, "Assumptions" tab, "Subsequent pregnancies" section.

• 25

  See this spreadsheet, “Results” tab, “Moral weights” section, for the numeric values that our model assigns to various outcomes. These values represent current best guesses for which we are highly uncertain.

• 26

  National AIDS and STI Control Program, 2017 HIV Sentinel Surveillance Survey for Women Attending Antenatal Care Services in Liberia (unpublished).

• 27
  • “Maternal syphilis estimates were generated using the Spectrum-STI model, fitted to sentinel surveys and routine testing of pregnant women during antenatal care (ANC) and other representative population data.” Korenromp et al. 2019, p. 1.
  • “Country estimates of maternal syphilis prevalence were based on data extracted from the Spectrum-STI adult prevalence database developed by Avenir Health with support from WHO. The database synthesizes earlier multi-country databases and country data reported to the WHO, UNAIDS and UNICEF, through GAM. The vast majority of adult prevalence data comes from sentinel surveys and routine testing in pregnant women in ANC.” Korenromp et al. 2019, p. 4.
  • Sentinel surveillance obtains data from selected healthcare facilities: "Sentinel surveillance involves a limited number of recruited participants, such as healthcare providers or hospitals, who report specified health events that may be generalizable to the whole population." Roush 2017.
  • "Based on data from a variety of sources covering the period 2008 to 2016, we developed a database of the three key service indicators: national ANC1 coverage, the proportion of pregnant women attending ANC screened for syphilis, and the proportion of pregnant women diagnosed with syphilis during ANC who were adequately treated." Korenromp et al. 2019, p. 5.

• 28
  • Because the impact of syphilis treatment is limited to people who have a syphilis infection, whereas the costs are fixed based on the number of people screened, whether or not they have an infection, the cost-effectiveness scales roughly linearly with the prevalence of syphilis among the targeted population.
  • The NACP conducted the sentinel survey at 35 sentinel facilities where 14,432 pregnant people were screened for syphilis, and 2.7% tested positive. The NACP survey takes a convenience sample of geographically representative facilities, but the results may still be biased from the lack of a randomization. In addition, the NACP report weights the prevalence from each site equally in the overall prevalence estimate. We have re-weighted the results by clinic catchment area and county population to get an estimate of 2.2% syphilis prevalence among pregnant people (see our calculations in this spreadsheet). Although this estimate of 2.2% is still likely subject to biases due to the sentinel survey not using a random population sample, we believe it is the best available estimate of syphilis prevalence among pregnant people in Liberia.
    • From National AIDS and STI Control Program, 2017 HIV Sentinel Surveillance Survey for Women Attending Antenatal Care Services in Liberia (unpublished)
  • Korenromp et al. 2019, using the Spectrum-STI model, takes facility-reported data from Liberia and applies a smoothing curve to generate a prevalence trend estimate. Because fewer than 10% of pregnant people are screened for syphilis each year, we think it is likely that health providers are choosing to test patients who show signs of having syphilis. If that is the case, the facility-reported data, as well as the Spectrum-STI-generated estimate that relies on this data, could be substantially overestimating syphilis prevalence among pregnant people.
    • “Country estimates of maternal syphilis prevalence were based on data extracted from the Spectrum-STI adult prevalence database developed by Avenir Health with support from WHO. The database synthesizes earlier multi-country databases and country data reported to the WHO, UNAIDS and UNICEF, through GAM. The vast majority of adult prevalence data comes from sentinel surveys and routine testing in pregnant women in ANC.” Korenromp et al. 2019, p. 4.
    • “Maternal syphilis prevalence was estimated using one of five approaches, depending on data availability: . . . A. Countries with at least one study either in ANC women (sentinel survey or routine syphilis screening in ANC, with a sample size of >100) or in the general population from 2011 or later AND 3 or more syphilis prevalence measurements in ANC women or general adult populations from 2000 and 2017: Spectrum-STI model was used to generate a prevalence trend estimate for pregnant women. Data were fitted with smoothed-splines polynomial regression, using second order segmented polynomials. Both the number and positions of the knots were estimated, and the Akaike Information Criterion was used for model selection.” Korenromp et al. 2019, Pg 4.
    • Korenromp et al. 2019, Supplementary Materials S2 reports that Liberia is in group A for maternal syphilis prevalence estimation (see row for Liberia, 2016; column “Group of maternal prevalence estimation”).
    • In 2015, 7.8% of pregnant people accessing antenatal care services in Liberia were tested for syphilis. See World Health Organization, "Women screened for syphilis during pregnancy: Data by country".

• 29

  Non-treponemal tests test for active infections, but are non-specific to syphilis (i.e., they may show positive results for other infections such as HIV, malaria, and pneumonia). Treponemal tests will only show positive results for syphilis, but cannot distinguish between different stages of the infection and may show positive results even for patients who have already been treated for syphilis. Thus, an active syphilis infection is generally confirmed through testing positive on both a non-treponemal and treponemal test.

  • "A reactive, but unconfirmed, non-treponemal test may represent a biological false-positive result, whereas a reactive treponemal test alone may represent an old or previously treated infection that poses little exposure risk for the fetus." Ham et al. 2015, p. S10.
  • "PAS [Probable active syphilis] (defined as seroreactivity for both non‐treponemal and treponemal tests) is used as the reporting measure by WHO since surveillance data typically do not include clinical information. Currently, no single test or combination of tests accurately predicts the extent to which maternal syphilis infection in pregnancy will affect the fetus. However, serologic tests can be suggestive; the combination of a reactive non-treponemal test (e.g. rapid plasma regain [RPR], venereal disease research laboratory [VDRL]) and a reactive treponemal test (e.g. Treponema pallidum particle agglutination [TP-PA], T. pallidum hemagglutination assay), defined in the 2008 WHO estimates as PAS, is compelling evidence for an infection that may result in MTCT." Ham et al. 2015, p. S10.

• 30
  • The studies from which we derived our estimates for rates of adverse pregnancy outcomes vary in the types of tests used for diagnosis.
    • In our cost-effectiveness analysis, we rely on the estimates of the rates of adverse pregnancy outcomes due to syphilis from Gomez et al. 2013. The studies included in the meta-analysis vary in the type of tests used for diagnosis, including the Wasserman test, Khan test, and rapid plasma reagin (RPR); some studies use two tests to confirm a syphilis infection. See Gomez et al. 2013, p. 220, Table 1, “Syphilis diagnostic test” column.
  • The sentinel survey from which we take our estimate of maternal syphilis prevalence in Liberia used both RPR and confirmatory TPHA diagnostic tests.
    • National AIDS and STI Control Program, 2017 HIV Sentinel Surveillance Survey for Women Attending Antenatal Care Services in Liberia (unpublished).
  • Rapid dual HIV/syphilis tests will be used to screen women in Liberia.

• 31

  See this document for a summary of different types of syphilis diagnostic tests and our adjustments based on the type of screening tests used in the literature.

• 32
  • Our impression from the relevant literature is that syphilis screening in pregnancy consistently reaches high levels when rapid syphilis tests are available at a clinic.
    • For example, Swartzendruber et al. 2015, a review of six observational studies, found that “[a]ll studies reported substantial increases in antenatal syphilis testing following introduction of RST [rapid syphilis testing].” Abstract.
  • Thus, we expect that if the Global Fund shifts all procurement of HIV tests for antenatal care visits to dual HIV/syphilis tests, it is plausible that the country will achieve almost the same syphilis screening rates as HIV screening rates.
  • To account for possible shortages in dual HIV/syphilis rapid tests, we have assumed that the syphilis screening rate will reach 95% of the expected HIV screening rate rather than 100%.
  • Based on Evidence Action’s proposed training plan, we assume that it will take three years to implement dual HIV/syphilis testing in all Liberian health clinics. Evidence Action, 5-yr plan for maternal syphilis program in Liberia, 2020.

• 33
  • See this spreadsheet, "Assumptions" tab, "Treatment rate" section.
  • We have estimated that roughly half of pregnant people who test positive for syphilis will be treated, based on high stockout rates of BPG in Liberia in the past (based on data as of 2016), persistent issues with the BPG supply chain at the global and national levels, and likely gaps in provider adherence to treatment protocols.
    • According to the Liberian Ministry of Health, Liberia Service Availability and Readiness Assessment (SARA) and Quality of Care Report, 2016 (unpublished), only 59% of Liberian health facilities had BPG in stock at the time of the assessment.
    • Nurse-Findlay et al. 2017 discusses several issues with the BPG supply chain at the global and national levels:
      • “In all, 95 of 114 unique countries and territories responded to the surveys. Of these, 39 reported a BPG shortage and 56 reported no BPG shortage.
      • “The team discovered 3 major issues. First, countries often obtain their product from a single wholesaler, which often obtains its products from a single final dose formulator, which often obtains its active ingredient from a single manufacturer. This 'sole sourcing,' combined with a highly consolidated market, makes alternative supply difficult if there are production, quality, regulatory, or specification changes within a country’s supply chain.
      • “Second, as an off-patent medication, BPG commands a market price of pennies per dose. However, as a sterile injectable, it is also expensive to manufacture. These economics have led manufacturers to either abandon BPG production or implement stringent ordering protocols that compromise supply for low- and middle-income countries.
      • “Third, inaccurate country forecasts, weak procurement systems, and clinical knowledge gaps about syphilis treatment have compromised demand for and procurement of BPG.” P. 3.
    • Additionally, our impression is that there may be a significant gap between what health providers know and what they do in practice, though we have not investigated this in detail. We believe it is possible that providers who receive training for HIV/syphilis dual testing and correctly test for syphilis in pregnancy may still fail to treat with BPG.
      • “Recent research documents a large gap between medical knowledge and practice: doctors, in countries ranging from Tanzania to India to The Netherlands, do a lot less with real patients than they say they would in similar hypothetical scenarios (vignettes). . . . Because of the know–do gap, medical training has a small impact on the actual care a patient receives.” Das 2011, p. e1000432.
  • We completed a light review of relevant literature, which provides fairly convincing evidence that screening rates increase substantially with the introduction of rapid diagnostic tests and BPG and some basic training. For example, Swartzendruber et al. 2015, a review of six observational studies, found that “All studies reported substantial increases in antenatal syphilis testing following introduction of [rapid syphilis testing] RST.” Abstract.
  • The literature also points to increases in treatment rates, but is somewhat less convincing on this point. The programs in the literature we reviewed on treatment rates guaranteed the supply of tests and BPG at each clinic studied, and all involved some sort of supervision, patient pre- and post-test counseling, or remedial training. In particular, Althabe et al. 2019, a cluster-randomized trial, compared treatment rates achieved in clinics that only received syphilis screening and treatment supplies with those that received both supplies and a behavioral intervention (consisting of supportive supervision, audits and feedback, etc.). The study found that clinics that only received supplies achieved an average treatment rate of 43%, while clinics that received behavioral supports in addition to supplies achieved a treatment rate of 100%. Based on this study, we think it’s likely that implementing the program without the behavioral components included in the academic literature would lead to lower treatment rates.
    • See our review of syphilis screening and treatment adherence literature.
  • Because we do not expect Evidence Action’s program to include the same intensity of mentoring activities as provided to programs in the literature, we’ve adjusted our estimate of the treatment rate Evidence Action will be able to achieve in Liberia downward from the rates observed in the literature cited above.

• 34

  Our understanding is based on multiple conversations with Evidence Action staff and on Evidence Action, 5-yr plan for maternal syphilis program in Liberia, 2020.

• 35

  We have projected that treatment rates will increase only during the period during which Evidence Action is providing in-country support. See this spreadsheet, "Assumptions" tab, "Treatment rate" section.

• 36
  • “The Global Fund and UNICEF did a one-off procurement of some dual tests in 2019 on behalf of Liberia to be used in the pilot. An additional procurement occurred last year via the Global Fund and additional procurement is expected via this channel in the coming years." Anna Konstantinova, Accelerator Research Manager, Evidence Action, email to GiveWell, February 9, 2021 (unpublished)
  • The HIV/syphilis rapid tests were previously procured at a price $1.50 per test, but Evidence Action expects future procurement to range from $0.90 to $1.50 per test. Evidence Action, Comments on a draft of this page, February 5, 2021 (unpublished).
  • The Global Fund is currently procuring HIV single rapid tests at $0.80 per test on behalf of Liberia. Anna Konstantinova, Accelerator Research Manager, Evidence Action, email to GiveWell, June 22, 2020 (unpublished).
  • In its budget, Evidence Action has added 30% to the cost of the commodities to account for procurement and supply management costs as well as for some wastage of commodities. We think this is a reasonable adjustment. Jeff Grosz, Accelerator Senior Director and Anna Konstantinova, Accelerator Research Manager, Evidence Action, conversation with GiveWell, May 8, 2020 (unpublished).
  • Note that a cheaper dual HIV/syphilis rapid test was pre-qualified by the WHO this year: “STANDARD Q HIV/Syphilis Combo Test with product code 09HIV20D, manufactured by SD Biosensor, Inc, Rest-of-World regulatory version, was accepted for the WHO list of prequalified in vitro diagnostics and was listed on 26 May 2020. . . . According to the claim of intended use from SD Biosensor Inc, 'STANDARD Q HIV/Syphilis Combo Test is a rapid chromatographic immunoassay for the qualitative detection of antibodies specific to HIV-1 including subtype O, HIV 2 and Syphilis (Treponema pallidum) in human serum, plasma or whole blood.'” World Health Organization, Public Report for STANDARD Q HIV/Syphilis Combo Test, 2020, p. 1.
  • The costs of both tests may decline over the five-year period of the grant due to market competition and increased procurement volumes. Anna Konstantinova, Accelerator Research Manager, Evidence Action, email to GiveWell, June 18, 2020 (unpublished).
  • Evidence Action has said that it worked to include screening instructions in the new screening and treatment protocols that will apply to any dual HIV/syphilis test, in order to allow Liberia to more easily switch to the new, cheaper test when it is widely available. Anna Konstantinova, Accelerator Research Manager, Evidence Action, email to GiveWell, June 18, 2020 (unpublished).

• 37

  $1.50 (cost of dual rapid test) - $0.80 (cost of HIV single rapid test) = $0.70.

• 38

  This is the standard Global Fund leveraging factor used in GiveWell analyses. See this spreadsheet for more details on how we decided on this leveraging adjustment.

• 39
• Anna Konstantinova, Accelerator Research Manager, Accelerator, Evidence Action, conversation with GiveWell, July 27, 2020 (unpublished).

• 40

  See this document for our reasoning behind not discounting future benefits.

• 41

  Our best guess of a five-year speedup is based on the following:

  • Liberia already recommends syphilis screening as part of routine antenatal care, but rates of syphilis screening in pregnancy have remained low.
    • Evidence Action has told us that syphilis testing in antenatal care is already mandatory in Liberia: “Policy wise, syphilis testing is already a mandatory component of antenatal care.” Evidence Action, Scoping Visit to Liberia: Summary of Findings, 2020 (public), p. 8. We have not vetted this claim.
    • According to the WHO’s Global Health Observatory data repository, syphilis screening rates remained under 20% during 2010-2015. World Health Organization, "Women screened for syphilis during pregnancy: Data by country".
    • Note that the same data show a sudden spike in syphilis screening rates in 2018. Evidence Action has investigated and found that the spike was due to a data entry error. Evidence Action, Scoping Visit to Liberia: Summary of Findings, 2020 (unpublished). We have not vetted this claim.
  • NACP procured commodities for a pilot of the program, but the pilot did not begin until Evidence Action became involved.
    • Information from Evidence Action, Scoping Visit to Liberia: Summary of Findings, 2020 (unpublished) and Evidence Action’s answers to GiveWell questions, April 2020 (unpublished).
  • Evidence Action supported the Liberian government in an initial rollout and pilot of 11,300 dual HIV/syphilis rapid tests in five clinics in Montserrado County.
    • “The National AIDS and STI Control Program (NACP) received 11,300 SD Bioline Duo HIV/Syphilis Combo Tests (10,300 from Global Fund and 1,000 from UNICEF) in July 2019. NACP intends to use these tests to ​do a small pilot in several facilities in Montserrado County.” Evidence Action, Scoping Visit to Liberia: Summary of Findings, 2020 (public), p. 9.
    • Evidence Action has told us that it supported the Liberian government in this pilot, and on August 13, 2020, Evidence Action forwarded to GiveWell email exchanges between Evidence Action and the Liberian government to support this claim.
  • While the government has signaled strong interest in this work, Evidence Action believes the government does not have the capacity to implement the program on its own.
    • “Based on conversations and feedback shared by key officials within the Liberian Ministry of Health, local NGOs, and major development partners, there are several areas where a non-profit organization’s support in Liberia could lead to meaningful increases in syphilis screening and treatment. As of now, syphilis screening has remained stagnant in Liberia for the last decade or more. The government (namely, the National AIDS and STI Control Program or NACP) has taken preliminary steps to introduce dual HIV/syphilis rapid test that will enable higher screening rates. As it moves forward, NACP would benefit from a partner that is able to provide additional capacity to develop, execute, and monitor introduction and scale-up of dual test coupled with syphilis treatment.” Evidence Action, Scoping Visit to Liberia: Summary of Findings, 2020 (public), p. 2.
    • “[N]umerous stakeholders including the NACP identified that there were challenges in ensuring all women who test positive are treated with benzathine penicillin-G (BPG). The challenges are twofold:
      • "1. There may be inconsistent procurement of BPG at the national level. It was frequently identified that insufficient BPG was being procured at the national level and that its future procurement could not be guaranteed.
      • "2. Clinics, which likely provide a large percentage of antenatal care, have to request BPG in order to receive it from the County Depot but many do not. There is low salience of syphilis treatment because many women are not being screened for syphilis at the clinic level. As a result, clinics are not requesting BPG stocks and gaps in treatment are likely to arise when dual test screening is brought down to the clinic level.” Evidence Action, Scoping Visit to Liberia: Summary of Findings, 2020 (public), pp. 8-9.
  • The only other organization we have identified working on syphilis screening and treatment in pregnancy is the Clinton Health Access Initiative (CHAI), which is focusing its efforts in Nigeria, India, Kenya, South Africa, and Uganda.
    • A primary goal of CHAI’s maternal syphilis program is to “[a]chieve significant, rapid and sustained improvement in syphilis testing and treatment coverage at ANC in five focus countries (Nigeria, India, Kenya, South Africa and Uganda).” Clinton Health Access Initiative, 2018 slide deck, Slide 14.
    • We vetted this finding by speaking with Dr. Melanie Taylor, Medical Officer for the Department of Reproductive Health and Research at WHO: “CHAI is currently the primary major global health organization advocating strongly for the elimination of congenital syphilis at the national level.” GiveWell's non-verbatim summary of a conversation with Dr. Melanie Taylor, June 2, 2020, p. 3.
  • We have not identified any non-governmental organizations (NGOs) working on this issue in Liberia.
  • Since WHO prequalified the first dual HIV/syphilis rapid test in 2015, a few countries have attempted to make the switch from HIV single rapid tests to dual HIV/syphilis rapid test without an NGO leading the effort, but a number of challenges seem to have hindered adoption.
    • “In 2015, the first multiplex rapid diagnostic test (RDT) for detection of anti- HIV and anti-treponema pallidum (dual HIV/syphilis RDT) was listed on the WHO list of prequalified in vitro diagnostic products (IVD).” World Health Organization, Information Note on the Use of Dual HIV/Syphilis Rapid Diagnostic Tests, 2017, p. 1.
    • For example, “Officials within Ethiopia's Ministry of Health are actively championing the adoption of dual HIV/syphilis RDTs, but are currently still working to change the country's maternal HIV testing algorithm to include the RDT.” GiveWell's non-verbatim summary of a conversation with Dr. Melanie Taylor, June 2, 2020, p. 3.
  • However, there have been a number of changes in the past year that have reduced the barrier to countries making the switch. We have not looked into the details of these changes, but based on our preliminary understanding, it seems likely that Liberia would eventually roll out the dual tests on its own in the absence of external support.
    • “WHO prequalified the first dual HIV/syphilis RDT in 2015 and has since prequalified an additional two dual RDTs. Interest in national-level adoption of these tests has been gradually increasing, primarily due to:
      • "Updated global guidance – In December 2019, WHO released updated guidelines for HIV testing recommending the use of dual HIV/syphilis RDTs. Additionally, WHO is now prioritizing scale-up of syphilis screening and treatment at the global and regional level, as part of its guidance on the elimination of mother-to-child transmission of HIV and syphilis.
      • "Reduced cost – The dual HIV/syphilis RDT that was most recently prequalified by WHO can be procured at a similar cost as a single HIV RDT.
      • "Increased access – Governments are now able to procure dual HIV/syphilis RDTs using funding from the Global Fund to Fight AIDS, Tuberculosis and Malaria (The Global Fund) and the President's Emergency Plan for AIDS Relief (PEPFAR).” GiveWell's non-verbatim summary of a conversation with Dr. Melanie Taylor, June 2, 2020, p. 1.

• 42

  Evidence Action, Back-of-the-envelope-calculation cost-effectiveness analysis, 2020 (unpublished)

• 43
  • “However, several of key barriers to implementation of universal prevention of congenital syphilis screening have been identified. These barriers range from patient-level factors (e.g., late entry into ANC, costs borne by patients, and the need to return at a later time for results) to facility-level constraints (e.g., health worker absence or insufficient training; provider fears around side effects of benzathine penicillin treatment despite their rarity.” Nkamba et al. 2017, p. 2.
  • Additional information from Evidence Action, Scoping Visit to Indonesia: Summary of Findings, 2019 (unpublished).

• 44
  • Given that there currently aren't reliable data on treatment rates, we believe lack of drug availability will limit the treatment rate. A 2016 assessment of the availability of key commodities in Liberia found that only 59% of health facilities had benzathine penicillin in stock. Liberian Ministry of Health, Liberia Service Availability and Readiness Assessment (SARA) and Quality of Care Report, 2016. It's possible that supply availability has improved since 2016, so we assume that around 30% of people will not be treated due to supply shortages.
  • Additionally, our impression is that there may be a significant gap between what health providers know and what they do in practice, though we have not investigated this in detail. We believe it is possible that providers who receive training for HIV/syphilis dual testing and correctly test for syphilis in pregnancy may still fail to treat with BPG.
    • “Recent research documents a large gap between medical knowledge and practice: doctors, in countries ranging from Tanzania to India to The Netherlands, do a lot less with real patients than they say they would in similar hypothetical scenarios (vignettes). . . . Because of the know–do gap, medical training has a small impact on the actual care a patient receives.” Das 2011, p. e1000432.

• 45

  Jeff Grosz, Accelerator Senior Director and Anna Konstantinova, Accelerator Research Manager, Evidence Action, syphilis case study meeting with GiveWell, January 28, 2020 (unpublished).

• 46
  • There are a number of implementing and funding partners like UNAIDS, UNICEF, the Global Fund, and PEPFAR who are important for providing technical assistance for certain aspects of the program or handling procurement of the dual tests. We’re uncertain about the exact role these implementing partners play in Liberia, but our impression from talking with Dr. Melanie Taylor from WHO is that besides CHAI, there are no major actors actively advocating for or leading the rollout of the dual tests.
  • “The national offices of WHO, UNAIDS, and UNICEF can serve as implementation partners to provide technical assistance to governments transitioning to dual HIV/syphilis testing. . . . CHAI is currently the primary major global health organization advocating strongly for the elimination of congenital syphilis at the national level.” GiveWell's non-verbatim summary of a conversation with Dr. Melanie Taylor, June 2, 2020, p. 3.

• 47

  From Evidence Action, Maternal Syphilis M&E in Liberia: Description of the Overall Approach, 2020, p. 2:

  • "M&E Objective 1: Estimate the fraction (and number) of ANC-going pregnant women who are tested for syphilis and the fraction (and number) of ANC-going syphilis-positive pregnant women who receive appropriate treatment
  • "Screening and treatment coverage (the output of Objective 1) is a function of two factors:
    1. "Supply availability: Without tests and BPG available at the facility, no pregnant women can be screened and/or treated.
    2. "Provider adherence: When products are present, providers must still elect to follow the clinical guidelines related to screening and treatment. Provider adherence is influenced by two components:
       • "Provider knowledge: Providers are unlikely to administer the dual test or to give BPG when a patient’s test comes back positive if they are not aware that these are the recommended courses of action.
       • "Provider/patient behavior: Once knowledgeable, providers must choose to deliver the appropriate services. Patient consent is also required for any testing or treatment to take place."

• 48

  “Supply availability and provider knowledge will be estimated through the Comprehensive Facility Survey taking place on a roughly annual basis (every 12-18 months).” Evidence Action, Maternal Syphilis M&E in Liberia: Description of the Overall Approach, 2020, p. 2.

• 49

  From Evidence Action, Maternal Syphilis: Monitoring Deep Dive slide presentation, 2020 (unpublished):

  • "Evidence Action will be contracting with an enumeration firm with experience in Liberia to administer the survey."
  • “Comprehensive Facility Surveys will be conducted approx. 4 months after the end of training.”

• 50

  “Sampling approach for the Comprehensive Facility Survey . . . Recommendation: Use simple random sampling stratified by counties.” Evidence Action, Maternal Syphilis: Monitoring Deep Dive slide presentation, 2020 (unpublished).

• 51

  “Supply availability will be assessed by a mix of quantitative and qualitative data collection, which will include inventory counting of non-expired syphilis tests and BPG injections available at the facility, review of the facility’s supply requests and delivery receipts, review of dispensing logs, and interviews with pharmacists to assess challenges in supply management.” Evidence Action, Maternal Syphilis M&E in Liberia: Description of the Overall Approach, 2020, p. 2.

• 52

  “Facility records reviewed during the Comprehensive Facility Survey, to include HIV counseling and testing registers, ANC registers, and patient charts.” Evidence Action, Maternal Syphilis M&E in Liberia: Description of the Overall Approach, 2020, p. 2.

• 53
  • “Provider knowledge will be estimated via clinical vignettes administered to a sample of providers in the course of the Comprehensive Facility Survey. Clinical vignettes are a form of a 'knowledge test' in which providers are assessed on their awareness of clinical guidelines via a mock ANC visit.” Evidence Action, Maternal Syphilis M&E in Liberia: Description of the Overall Approach, 2020, p. 2.
  • “An assessment of provider training will be done via clinical vignettes and qualitative interviews administered to health providers in which we will directly measure knowledge retention following the training.” Evidence Action, Maternal Syphilis M&E in Liberia: Description of the Overall Approach, 2020, p. 3.
  • From Evidence Action, Maternal Syphilis: Monitoring Deep Dive slide presentation, 2020 (unpublished).
    • “Additional facility survey components beyond supply availability . . .
      • ""Survey of providers, to include:
        • "Clinical vignette (oral knowledge test to assess awareness of ANC guidelines on syphilis screening and treatment)
        • "Skills test to assess how accurately providers use the dual test and interpret the results
        • "Qualitative interview to assess comfort level with using dual test, providers’ trust in the results, comfort with administering treatment, and challenges encountered in providing recommended maternal syphilis care”
      • "Qualitative interviews may include discussions about providers’ perceptions of barriers to patient adherence to treatment."

• 54
  • From Evidence Action, Maternal Syphilis M&E in Liberia: Description of the Overall Approach, 2020, p. 2.
    • “Provider and patient behavior will be initially estimated via a triangulation approach on a roughly annual basis as well (every 12-18 months). The triangulation method itself will be developed once more learning takes place in-country on the quality and availability of all potential data components. However, it is likely we will rely on a combination of the following data components:
      • "Supply data related to stock on hand, dispensing, supply issues, and supply orders taken from the Comprehensive Facility Survey, LMIS, and SCMU/CMS records;
      • "Facility records reviewed during the Comprehensive Facility Survey, to include HIV counseling and testing registers, ANC registers, and patient charts;
      • "Administrative data, namely HMIS;
      • "Qualitative interviews with providers.
    • "Should the triangulation approach prove infeasible or provider/patient behavior is found to be a large contributor to low coverage, we will explore an alternative approach of direct clinical observation.”
  • “We will use data from ANC and HIV counseling and testing (HCT) registers, as well as national, county, and facility supply data to triangulate an estimate of provider/patient behavior.” Evidence Action, Maternal Syphilis: Monitoring Deep Dive slide presentation, 2020 (unpublished).

• 55

  "Within the first year, we will also have better data on supply availability, we will see how well supply issue data correlates to ultimate facility-reported coverage, and we will have feedback from clinical vignettes and qualitative interviews with providers to assess whether there are provider-level barriers to screening and treatment unrelated to supply. At that time, we will also have data from the ongoing dual test pilot that may also be suggestive of potential adherence gaps." Evidence Action, Maternal Syphilis: Monitoring Deep Dive slide presentation, 2020 (unpublished).

• 56

  “Should the triangulation approach prove infeasible or provider/patient behavior is found to be a large contributor to low coverage, we will explore an alternative approach of direct clinical observation.” Evidence Action, Maternal Syphilis M&E in Liberia: Description of the Overall Approach, 2020, p. 2.

• 57

  From Evidence Action, Maternal Syphilis: Monitoring Deep Dive slide presentation, 2020 (unpublished).

  • “Outline of a Direct Observation Survey (if we determine it is necessary to conduct during the first year of the program)
    • "Purpose: To assess provider adherence to testing and treatment guidelines when commodities are available, either due to providers’ behavior in providing care or other reasons.
    • "Activity Description: Quantitative data collection from a sample of facilities consisting of direct clinical observation of ANC providers.”

• 58

  From Evidence Action, Maternal syphilis research overview, 2020, pp. 11-12.

  • “To assess whether investing in efforts to increase screening and treatment of syphilis among pregnant women is potentially highly cost-effective, and what role Evidence Action might consider playing in these efforts, we have worked on three main fronts:
    • "Literature review: We reviewed all literature cited in GiveWell’s interim intervention report, in addition to numerous other resources. . . . Throughout the research, we looked for promising intervention examples that could inform our potential efforts to prototype and pilot a model for boosting screening and treatment.
    • "Expert interviews: We spoke to a wide range of experts to clarify our learnings from the literature review, including experts interviewed by GiveWell and numerous others. . . .
    • "In-country scoping visits: We completed initial scoping visits in Indonesia, Liberia, and Ghana.”

• 59

  GiveWell's non-verbatim summary of a conversation with Dr. Melanie Taylor, June 2, 2020.