Evidence Action — Liberia Syphilis Screening and Treatment in Pregnancy Exit Grant (April 2024)

Note: This page summarizes the rationale behind a GiveWell grant to Evidence Action. Evidence Action staff reviewed this page prior to publication.

Published: October 2024

Background

In August 2020, GiveWell recommended a $3.9 million grant to Evidence Action to support the roll out of the rapid dual syphilis/HIV test in Liberia over five years.

In September 2023, Evidence Action alerted us to the fact it sought additional funds from GiveWell for this grant.1 This would cover (a) cost overruns for the existing five year period of performance, since Evidence Action has drawn down its initial grant funds more quickly than it expected, and (b) the extension of the program for a sixth year.

Evidence Action has attributed its cost overruns to a wide range of unanticipated programmatic costs,2 as well as inflation, which it had not accounted for in its initial budget.3 It attributed the extension of the program as necessary to account for the impact of programmatic delays caused by COVID-19, and to increase the likelihood of a successful transition of programming from Evidence Action to Liberia’s Ministry of Health, as intended in the original grant.4

Evidence Action’s overall funding request was $2,046,826, of which $831,412 would cover cost overruns within the original grant period, and $1,215,414 would cover extending the program into year six.5

The grant

We are recommending a grant of $1,214,089 to Evidence Action. This is designed to cover around 70% of Evidence Action’s cost overruns for the existing five-year period of performance, and around 50% of the budget for its proposed sixth year. This will take the overall budget for this program to $5,007,705. This is $100,724 less than the combined cost of the initial $3.9m grant and this $1.2m exit grant, because Evidence Action did not need to use that amount of the $300,000 contingency funding in the original grant agreement hypothecated specifically for gaps in the supply of key commodities.6

From our perspective, this grant amount represents:

• A partial contribution towards cost overruns ($606,382).7 This represents the portion of Evidence Action’s budget overruns that we believe could reasonably be attributed to unanticipated or surprise costs, plus omissions from the original budget that we think GiveWell should have caught at the time we made the initial grant in 2020 (such as not accounting for inflation). This is roughly 70% of Evidence Action’s $831,412 request for this portion of the grant.8 We think Evidence Action may be able to fundraise or use its unrestricted funding for the remainder.
• A partial contribution towards year 6 costs ($607,708).9 We are recommending a grant for around 50% of Evidence Action’s proposed budget for year 6.10 We acknowledge that an additional year of programming may help to address unexpected issues that came up over the course of our initial grant period, such as COVID-19 delays leading to delayed project rollout. However, we are not convinced that just one additional year will meaningfully increase the likelihood of a smooth transition to the Ministry of Health, and we are unwilling to fund this program indefinitely. We think Evidence Action may be able to fundraise for the remainder of these costs.

In practice, Evidence Action will use these exit grant funds at its discretion. For example, it may decide to use this grant to fully cover its cost overruns, and then fundraise for year 6, or it may use its own unrestricted funds to cover its cost overruns, and use our grant to fully fund year 6. If Evidence Action decides to put our grant towards its cost overruns, but does not extend the program to year 6, the remaining grant amount ($382,679) will be returned to GiveWell.11

This $1.2 million exit grant increases the overall cost of the program by 31%, reducing our back-of-the-envelope cost-effectiveness estimate for the program to around 11 times the cost-effectiveness of unconditional cash transfers.12 However, we believe this estimate likely overstates cost-effectiveness, as it only accounts for changes to the cost of the grant, and not for any reductions in the grant’s impact, which we anticipate may be associated with the delays and challenges reported by Evidence Action.13 We have not yet investigated the cost-effectiveness of the overall program further, as we (a) do not believe we have sufficient monitoring and evaluation data to meaningfully update our estimate of syphilis screening and treatment rates, and (b) do not think the range of uncertainty around this is large enough to affect the case for awarding this exit grant either way.

The case for the grant

We are recommending this grant because:

• We think this grant is necessary for us to be a responsible funding partner. Although the overall program likely now falls below our cost-effectiveness bar, we believe making an exit grant is the right thing to do. This is because:
  • We want to protect Evidence Action’s relationship with the Liberian government. Without our exit grant, Evidence Action may have needed to close the program early, after year 4. We think it is plausible that Evidence Action doing so would have permanently affected its ability to work in Liberia. At present, without philanthropic support, we think it is highly unlikely that Liberia’s Ministry of Health would implement the program.
  • Premature program closure could harm syphilis screening and treatment rates. In order to sustain the current rate of screening and treatment, Evidence Action believes it needs to provide supervision in the counties that had delayed rollouts due to COVID-19, and to work on the government’s supply chain issues.14 This grant provides Evidence Action with funding for the additional time associated with this work, thereby plausibly preventing a decline in the screening and treatment rate.
  • We believe it’s right for GiveWell to absorb some of the impact of the program’s delays and challenges. In the past, GiveWell has encouraged Evidence Action to expand to new geographies and program areas, and therefore we think we should shoulder some of the risk associated with it doing so. Furthermore, we think we should have caught the fact that Evidence Action’s original budget did not account for inflation risk or include a buffer to account for the programmatic risk associated with a new program. This makes us more inclined to make an exit grant in this case.
• Evidence Action has improved the monitoring and evaluation of its program over time, which slightly increases our confidence that we will understand the program’s effect on syphilis screening and treatment rates. We also expect to learn about the impact of Evidence Action’s technical assistance by whether it is able to obtain government administrative data on syphilis screening and treatment rates from 2025 onwards.
  • Initially, it was difficult to quantify the impact of Evidence Action’s technical assistance.
    • Evidence Action had originally planned to monitor syphilis screening and treatment rates through (a) government administrative data (Health Management Information System, or HMIS), since it believed it could ensure that routine reporting on syphilis screening and treatment was included on the next HMIS form, and (b) its own comprehensive facility surveys (CFS), to be triangulated with HMIS.15
    • However, while syphilis reporting was added to the HMIS form, syphilis treatment was omitted from the facility recording registers, creating challenges in tracing and validating treatment data. Additionally, Evidence Action’s first CFS had key limitations, including (a) a small sample size, (b) reliance on self-reporting, and (c) reliance on proxy variables, such as medicine stockouts and provider knowledge, to estimate treatment rates.16
  • Evidence Action improved its second CFS, and it now plans to measure dual test and penicillin stock quarterly, rather than annually.17 This increases the likelihood that we understand how its program is going, and therefore allows us to be more confident about the expected impact of our total grant on the syphilis screening and treatment rate.
  • Furthermore, Evidence Action will have a second opportunity to ensure that syphilis screening and treatment data are included in the HMIS form, when it is revised again in 2025.18 This will give us another opportunity to observe whether Evidence Action is able to achieve a concrete goal of its technical assistance work, and, if it is successful, this additional data will improve our ability to understand syphilis screening and treatment rates during the period of Evidence Action’s involvement and thereafter.

Risks and reservations

Our main reservations about this grant are:

• We are unsure whether $1.2 million is the right grant amount. In particular, we do not have a satisfying way of (a) triangulating Evidence Action’s claims about the seriousness of ending the program early, (b) knowing whether a sixth year of grant activity is likely to be sufficient to fully transition the program over to the Ministry of Health, and (c) determining whether we are covering the right amount of Evidence Action’s cost overruns. This makes it difficult to gauge whether our grant is commensurate with the "true" level of programmatic need.
• We are wary that this grant could create perverse incentives on behalf of Evidence Action and other prospective grantees. We think there is a risk that in approving part of Evidence Action’s funding request, we are implying to Evidence Action and other organizations that GiveWell is willing to provide additional funding whenever there could be negative consequences associated with not doing so. This could encourage organizations to intentionally under-budget at the outset in order to artificially inflate the cost-effectiveness of their programs. This is particularly pertinent to this grant, given the extent to which Evidence Action initially underestimated the cost of this program.
• The cost-effectiveness of the overall program likely falls beneath our bar. Without any changes to the benefit side of our calculation, the new total budget implies a cost-effectiveness figure of 11x cash transfers. However, our initial model for this grant assumed that coverage levels would reach 71% and remain there after the program transitions from Evidence Action to Liberia’s Ministry of Health.19 Given the delays and challenges with grant, this feels implausible to us, which suggests that the program likely falls below our bar.
• Despite the improvements that Evidence Action has made to its monitoring and evaluation, there is a risk that we will still not get reliable information on syphilis screening and treatment rates. Although Evidence Action has improved its second CFS relative to the first, its surveys will still rely on self-reported data and proxy variables to triangulate syphilis screening and treatment rates. Even if treatment indicators are added to the faculty reporting registers (which would be highly valuable, particularly if the two methods give consistent results), this program will not have as authoritative monitoring and evaluation data as many other programs that we fund.

Plans for follow up

• We receive ongoing formal reporting from Evidence Action, including routine updates on their challenges and progress, quarterly data on stockouts, and annual survey data on health facilities. We plan to monitor this closely for the remainder of the grant period.
• We are viewing this as an exit grant, so it is not eligible for renewal.

Internal forecasts

For this grant, we are recording the following forecast:

Confidence	Prediction	By time
10%	We will assess this work as being more than 10 times as cost-effective as cash transfers by August 2026	August 2026

Our process

• We reviewed Evidence Action’s budget request and narrative, as well as the ongoing monitoring and evaluation data that it has provided us as part of the initial grant.20 We then exchanged several calls and emails with Evidence Action to discuss its progress on this grant and its funding request.
• We also spoke with a senior representative of Liberia’s Ministry of Health, in order to understand how it views Evidence Action’s progress to date.

Sources

Document	Source
Evidence Action, Comprehensive Facility Survey Report for the Maternal Syphilis Program in Liberia	Source
Evidence Action, email to GiveWell, July 16, 2024	Unpublished
Evidence Action, Liberia Top-Up & Extension – Unanticipated Cost Overages	Unpublished
Evidence Action, Maternal Syphilis Liberia Y4-Y5 Top Up and Y6 Extension Proposal	Source
Evidence Action, responses to GiveWell’s questions, October 13, 2023	Unpublished
GiveWell, CEA of 2023 top-up grant - Evidence Action technical assistance to Liberia for syphilis screening and treatment	Source
GiveWell, conversation with Evidence Action, March 6, 2024	Unpublished
GiveWell, Evidence Action — Syphilis Screening and Treatment in Pregnancy, 2021	Source
GiveWell, review of initial monitoring and evaluation	Unpublished

• 1

  See Evidence Action, Maternal Syphilis Liberia Y4-Y5 Top Up and Y6 Extension Proposal for details of Evidence Action's funding request.

• 2

  These include office space as a requirement for registering in-country, and a variety of additional personnel costs. Evidence Action, Liberia Top-Up & Extension – Unanticipated Cost Overages (unpublished).

• 3

  "In a budgeting oversight, Evidence Action did not include any inflation in its original budget proposal. Although certain costs are not subject to inflation (namely the cost of donated commodities), staff salaries and certain other variable costs would increase year-on-year. Due to the instability in the economy and the ongoing conflicts in Europe over the last few years, we have witnessed even more increased prices related to inflation and fuel prices, especially in the Africa Region…In an attempt to address these fluctuating costs, Evidence Action included a 6% year-on-year inflation and cost-of-living rate projection. This subsequently increased all applicable individual expenses that made up our projections for Y4 and Y5 budgets." Evidence Action, Maternal Syphilis Liberia Y4-Y5 Top Up and Y6 Extension Proposal, pp. 6-7

• 4
  • "As we were about to launch the first phase of the training-of-trainers cascade, the Delta variant reached Liberia and COVID cases started increasing dramatically. In line with the MoH and Evidence Action’s internal COVID policies, we decided at the time to delay the training cascade until COVID cases declined and ended up training the first cohort of master trainers in mid-September, offsetting our program timeline by approx. 6 months. As a consequence, we have only just concluded the training of the final targeted health facilities in May-July of 2023." Evidence Action, Maternal Syphilis Liberia Y4-Y5 Top Up and Y6 Extension Proposal, p. 7
  • "There are a number of program activities which Evidence Action has been leading and/or funding that would ultimately need to be integrated into NACP workplans such that any emerging gaps in syphilis screening and treatment coverage are identified and addressed efficiently by NACP going forward. Most of these activities relate either to supply chain (as discussed), to facility mentorship / supportive supervision, or to monitoring and evaluation. Based on our assessment, an additional year of programming will be crucial to ensuring these activities are integrated appropriately." Evidence Action, Maternal Syphilis Liberia Y4-Y5 Top Up and Y6 Extension Proposal, p. 9

• 5
  • See here in the "Cost analysis (March 2024)" tab of our CEA for the funding request amount. Note: the exact dollar amount varies by one dollar due to how we've rounded the calculations in our CEA.
  • See Table 1 in Evidence Action's proposal for the cost overrun amount, and Table 2 for the Year 6 budget. Evidence Action, Maternal Syphilis Liberia Y4-Y5 Top Up and Y6 Extension Proposal, pp. 4-5

• 6

  Per our 2020 grant page, “Around $300,000 of [program] costs are considered an “as-needed” expense. These funds will be released in the event that there are gaps in the supply of key commodities and additional procurement is needed.”
  Evidence Action confirmed that they have $100,724 remaining in an email to GiveWell, July 16, 2024 (unpublished).

• 7

  See the "Cost Analysis" sheet in GiveWell, CEA of 2023 top-up grant - Evidence Action technical assistance to Liberia for syphilis screening and treatment.

• 8

  Evidence Action requested $831,412 for Years 4 and 5. Evidence Action, Maternal Syphilis: Liberia Y4-Y6 Budget Proposal, 2023 p. 4.

• 9

  See the "Cost Analysis" sheet in GiveWell, CEA of 2023 top-up grant - Evidence Action technical assistance to Liberia for syphilis screening and treatment.

• 10

  Evidence Action requested $1,215,415 for Year 6. Evidence Action, Maternal Syphilis: Liberia Y4-Y6 Budget Proposal, 2023 p. 5.

• 11

  $1,214,089 - $831,410 = $382,679. Note that there is a slight discrepancy between Evidence Action's total ask amount in its proposal versus in our CEA ($831,412 vs. $831,410). This is due to how we've rounded the calculations in our model.

• 12

  See GiveWell, CEA of 2023 top-up grant - Evidence Action technical assistance to Liberia for syphilis screening and treatment, "Results: overall" sheet. Note that a) our cost-effectiveness analyses are simplified models that are highly uncertain, and b) our cost-effectiveness threshold for directing funding to particular programs changes periodically. As of April 2024, our bar for directing funding is 10 times the cost-effectiveness of unconditional cash transfers. See GiveWell’s Cost-Effectiveness Analyses webpage for more information about how we use cost-effectiveness estimates in our grantmaking.

• 13

  Per the next footnote, the main delays and challenges cited by Evidence Action are around the impact of COVID-19, and supply chain issues. Evidence Action, Maternal Syphilis Liberia Y4-Y5 Top Up and Y6 Extension Proposal.

• 14
  • On COVID-19, Evidence Action wrote: “In line with the MoH and Evidence Action’s internal COVID policies, we decided at the time to delay the training cascade until COVID cases declined and ended up training the first cohort of master trainers in mid-September [of 2021], offsetting our program timeline by approx. 6 months. As a consequence, we have only just concluded the training of the final targeted health facilities … in some of the hardest to reach regions of the country … Furthermore, many of these areas demonstrated the highest prevalence of syphilis in the 2017 sentinel survey … it is critical to ensure we have sufficient time (preferably three years rather than two) to fully integrate syphilis screening and treatment services within these facilities.” Evidence Action, Maternal Syphilis Liberia Y4-Y5 Top Up and Y6 Extension Proposal, p. 7
  • On supply chain issues, Evidence Action wrote: “Evidence Action had identified the supply chain as the one of the key potential failure points of the program and hypothesized that gaps in syphilis screening and treatment following scale-up of dual testing would be due primarily to inconsistent availability of dual tests and/or benzathine penicillin in health facilities. Since Year 1, we have implemented a number of approaches to mitigate stock out issues among health facilities. Though much of this has been successful, additional work is required to reach our targets for availability of dual tests and benzathine penicillin and then maintain availability at those target levels. … Given the difficult nature of these challenges, and that Evidence Action cannot independently implement solutions to them but must rather collaborate with partners, we will require an additional program year to iterate and implement different solutions to stabilize the availability of dual tests and benzathine penicillin among health facilities.” Evidence Action, Maternal Syphilis Liberia Y4-Y5 Top Up and Y6 Extension Proposal, pp. 7-8

• 15
  • “Evidence Action’s default evaluation plan includes two primary components:
    • Comprehensive facility surveys
    • Triangulation of national, county, and facility records
  • The comprehensive facility surveys are intended to assess supply availability and provider knowledge”
  • “Evidence Action’s current plan is to assess provider and patient behavior by triangulating facility register data, comprehensive facility survey data, and supply data at the facility, county, and national levels”

  GiveWell, “Evidence Action — Syphilis Screening and Treatment in Pregnancy”, 2021.

• 16

  The sample size was small because Evidence Action had only reached three counties at the time the survey was conducted. GiveWell, review of initial monitoring and evaluation (unpublished).

• 17

  GiveWell, conversation with Evidence Action, March 6, 2024 (unpublished).

• 18

  Evidence Action, responses to GiveWell’s questions, October 13, 2023 (unpublished).

• 19

  See the "Percent of ANC attendees who are screened for syphilis - intervention" row on the "Benefits" tab of our CEA."

• 20

  For example, see Evidence Action’s Facility Survey Report from February 2023 here.