Short Message Service (SMS) Reminders to Increase Infant Vaccination

Published: August 2024. (Previous versions of this page: June 2017 version)

Summary

What does this program do?

A SMS reminders program for childhood vaccinations collects mobile numbers for pregnant women or caregivers of young infants and sends Short Message Service (SMS) messages to them prior to each routine immunization visit, reminding caregivers of an upcoming vaccination date. Key features of SMS reminders programs may vary across organizations, including the method and timing of enrollment, the frequency, timing and content of messages, and whether caregivers are able to communicate with the service (e.g. by responding to questions). (more)

How cost-effective is it?

We think an SMS reminders program for childhood vaccinations could be highly cost-effective because SMS reminders are an inexpensive way to modestly decrease the number of children who do not receive some or all of the routine childhood vaccinations. We estimate that in Nigeria, an example of a context with low existing childhood vaccination coverage, a generic SMS reminders program is roughly 12 times as cost-effective as unconditional cash transfers (GiveWell’s benchmark for comparing different programs).

In simple terms, we estimate that an SMS reminders program for childhood vaccinations is cost-effective because:

• Sending SMS reminders is inexpensive. We expect the cost per child for SMS reminders to vary by a large amount, depending mainly on whether the system can source enrollments through an existing system or needs to hire staff to enroll caregivers, and on how many children the core costs of running the program can be spread across. We have estimated that it costs roughly $2.90 per caregiver/child pair reached for an example SMS program, including both direct SMS message and operational costs, over the course of all routine childhood immunizations. SMS reminder programs have the potential to be very inexpensive per child if they are large-scale and leverage existing sources of caregiver phone numbers, since the messaging costs (roughly $0.01 per SMS message) are low. (more)
• Sending SMS reminders may lead to modest increases in the number of children who receive routine childhood vaccinations. Getting a child fully immunized requires caregivers to remember to bring their children in for multiple visits on a somewhat complicated schedule. By reminding caregivers of upcoming appointments around the time of their child’s due date, we think SMS reminders can reduce the number of children who are not brought in for vaccinations. We estimate that an SMS reminders program decreases the number of children who are not vaccinated by around 15 percent. This estimate is based on the results of a meta-analysis of 16 randomized controlled trials in low- and middle-income countries, to which we apply downward adjustments to account for concerns around the internal validity and generalizability. (more)
• Absent vaccinations, many children could die from vaccine-preventable causes. In Nigeria, specifically, we estimate that unvaccinated children have a roughly 3% probability of dying from vaccine-preventable causes before their fifth birthday. This includes an adjustment for all-cause mortality effects because we think vaccines avert about 0.75 additional deaths for every direct death averted from the disease they target. This is based on evidence that vaccination programs sometimes have larger impacts on mortality than would be expected from their impact on the diseases they target alone. (more)
• Increased childhood vaccinations lead to large reductions in childhood mortality as well as reductions in later life mortality. We have high confidence that vaccination can reduce child mortality, as well as mortality later in life. In Nigeria, specifically, based largely on RCT evidence, we estimate that receiving the full set of vaccines considered in this report can reduce a child’s mortality risk from vaccine-preventable diseases by roughly 52%. We also think it’s likely that vaccination during childhood will provide longer term protection against disease later in life. (more)

• We think vaccination during childhood will have other impacts, including:
  • Increased income for children in later life, by averting disease in a sensitive developmental period of childhood. This accounts for 22% of the benefits we model (more).
  • Various other benefits and downsides that we don’t model and apply as percentage best guesses. For example, we think childhood vaccination reduces morbidity from disease and averts costs that would have been spent on treatment of disease. These increase our cost-effectiveness estimate by 20% (more).

A sketch of our cost-effectiveness analysis is below. This analysis requires making estimates for several uncertain parameters that could change cost-effectiveness substantially. These key parameters are in orange.

You can see our cost-effectiveness analysis for the program here and a simple version here.

How do SMS reminders for vaccination compare to other interventions GiveWell thinks are promising?

Compared to one of the programs run by a GiveWell top charity, New Incentives’ conditional cash transfers (CCTs) for vaccination program in northern Nigeria:

• We think SMS reminders are much less expensive. We estimate that it costs roughly $2.90 per infant sent SMS reminders, with the potential to be considerably less expensive than that, compared to $21 per infant enrolled for CCTs.1
• We think SMS reminders are much less impactful at increasing vaccination uptake. We estimate that SMS reminders can decrease the number of unvaccinated children by roughly 15%, compared to roughly 29% for CCTs provided by New Incentives.2 While we would expect these estimates to vary in different contexts, we think it makes intuitive sense that SMS reminders programs would generally be less impactful because New Incentives’ program is more intensive (e.g. in addition to providing cash transfers, New Incentives conducts awareness-raising and supply-side strengthening activities) and likely addresses more barriers to vaccination than a typical SMS program by providing information about and alleviating costs for attending vaccination visits.3
• We’re significantly more uncertain about the quality of the evidence for SMS reminders and how we should apply the evidence in a given setting. We estimate the effect of an SMS reminders program on immunization uptake using a meta-analysis of 16 randomized controlled trials (RCT) in low- and middle-income countries, which were of moderate quality overall. We have not investigated the individual studies in detail. For New Incentives, we base our estimate on a reasonably strong RCT of New Incentives’ program in northern Nigeria, which we followed closely and reviewed in-depth.4
• In a given setting, we think SMS reminders and CCTs for vaccination would largely increase uptake of the same set of vaccines, which we think are highly effective at reducing mortality. The exact impact of vaccination uptake on mortality would depend on the specific program and setting in which it’s implemented.

Our key open questions

• How should we combine the results from different studies when applying them to our model? We estimate the effect of an SMS reminders program on vaccination based on a meta-analysis of 16 RCTs in low- and middle-income countries. Our meta-analysis estimates the effect size of an SMS reminders program by combining several different measures, largely capturing completion of vaccinations later in the routine schedule, into a single effect size (more). This may lead us to over or underestimate a program’s impact on mortality and other outcomes, especially because receiving earlier vaccines contributes to a larger share of the benefits in our model. We expect to investigate this further when considering specific giving opportunities. (more)

• How effective are SMS reminder programs in different contexts? We think effectiveness is likely highly dependent on context-specific factors such as the local barriers to vaccination and mobile phone coverage. We apply a -20% adjustment to account for the likelihood that a program is less effective outside of a study context, but do not make other adjustments. We expect to incorporate context-specific information when we investigate specific giving opportunities. (more)

• How effectively do different programmatic models impact vaccination uptake? SMS reminders programs include a heterogenous class of interventions, and we think the uptake in vaccination rates caused by an SMS reminder program likely differs based on a specific program’s characteristics, including the successful message received rates, whether communication is bidirectional, the timing and frequency of messages sent, and when caregivers are recruited. We expect to investigate how program design impacts effectiveness in the context of specific giving opportunities. (more)
• How much does it cost specific SMS reminders programs to reach each caregiver/infant pair? We expect that messaging costs are likely to be low, however, we have significant uncertainty about the non-messaging costs required for an SMS reminders program. We estimate non-messaging costs based on the costs from a digital immunization registry (that also sends SMS reminders) which GiveWell has supported in Pakistan. Cost per caregiver/infant pair may vary significantly based on the scale and design of a program. (more)

• How effective are vaccines at reducing mortality and improving other outcomes? We use Nigeria as an illustrative example of a location where we might fund an SMS reminders program; but the vaccines targeted, their efficacy, and the underlying disease burden expected to be impacted will vary based on the specific program. We’ve seen some evidence that vaccine efficacy is lower in low-income settings where we expect to fund programs, and we have a large degree of uncertainty about the available estimates of disease burden. (more)
• What are other outside perspectives that might not be captured explicitly in our cost-effectiveness estimates? Our cost-effectiveness analysis intends to capture the total impact of a program per dollar spent; however, focusing only on our model may mean we miss other factors which affect the impact of a program that are more difficult to quantify. We have not deeply investigated these other perspectives at the time of writing, but plan to consider them more when investigating specific grant opportunities. (more)

Is there room for more funding?

GiveWell has funded two organizations which provide SMS reminders for vaccinations (alongside other interventions in some areas): Suvita in India, and IRD Global in Pakistan. We have not yet completed a full analysis of room for more funding for SMS reminders for vaccination programs. There may be opportunities to expand operations, fund other programs implemented by NGOs, or support government-led programs via technical assistance. (more)

1. The basics of the program

1.1 Why do childhood vaccination rates fall short of full coverage?

Child vaccination reduces the risk of illness, disability, and death from vaccine-preventable diseases including diphtheria, hepatitis B, measles, pertussis (whooping cough), pneumonia, polio, rotavirus diarrhea, rubella, mumps, and tetanus.5 However, vaccination rates in many settings are estimated to fall short of full coverage.6 Full immunization for many childhood vaccines requires attending multiple appointments on a somewhat complicated schedule, which caregivers may be unaware of or experience difficulty remembering.7

1.2 What is an SMS reminders program?

A Short Message Service (SMS) reminder for vaccination program sends text messages to caregivers of young children shortly before the child is due for a routine vaccination visit. Programs may also include follow up reminders if a child is not vaccinated at the recommended time.8 By doing this, SMS reminder programs aim to increase uptake and timeliness of child vaccination. We think this likely occurs by reducing the likelihood that caregivers forget or delay vaccination visits, which may be an important barrier to infants receiving vaccinations in low- and middle-income contexts.9

SMS reminders programs include a variety of different programmatic models, and we believe the effectiveness of SMS reminder programs could vary considerably depending on the specific programmatic design, so our report focuses on SMS programs with the following characteristics:

• Recruitment method: The implementer collects phone numbers from caregivers at health clinics and routine immunization visits.
• Timing of enrollment: Caregivers are initially enrolled to receive SMS reminders at birth or around the time of the child’s first routine immunization visit.10
• Timing and frequency of messages: Caregivers would receive between 1-3 SMS reminders in the week preceding each subsequent routine vaccination due date. They may also receive follow-up messages for missed visits.
• Content of messages: We expect messages would include the child’s name and subsequent immunization due date, but we do not otherwise make assumptions about how reminders are framed.11 We assume no additional health messaging about immunizations is included with the reminders.
• Whether caregivers can respond to messages: Messages are one-directional (from the organization to caregivers only).

We think the features above broadly characterize the SMS programs we reviewed evidence for, and broadly characterize many of the SMS reminders interventions we could consider funding. SMS programs may also differ along other characteristics not explicitly outlined in this report, and we would investigate how program design is likely to impact the program’s effectiveness when assessing specific funding opportunities. (more)

In addition, this report focuses on a standalone SMS program, but we think it’s likely that organizations (or governments) who implement an SMS reminders program would bundle other interventions alongside SMS reminders because it would be relatively costly to set up a system solely intended to enroll and send SMS reminders for immunizations. We would consider how a program’s design is likely to impact effectiveness in the context of a specific funding opportunity. (more)

2. How GiveWell estimates cost-effectiveness

GiveWell recommends programs that we believe save or improve lives as much as possible for as little money as possible. To estimate this, we produce a cost-effectiveness analysis (“CEA”) which aims to produce a best guess of the overall impact of a program per dollar donated.

We use moral weights to quantify the benefits of different impacts (e.g., increased income or reduced deaths). We benchmark to a value of 1, which we define as the value of doubling someone’s consumption for one year. The main moral weights we use for our analysis of SMS reminders programs are in the table below.

For more about how GiveWell thinks about cost-effectiveness, see our discussion on this page.

3. What impact do SMS reminders programs have on childhood vaccination uptake?

3.1 Summary

We have moderate confidence that SMS reminders programs can reduce the number of children who do not receive some or all routine childhood immunizations. This is based on moderate quality evidence from 16 RCTs in low- and middle- income countries. We performed a meta-analysis to estimate that an SMS reminders program could reduce the share of children who are unvaccinated by 15%. We’re highly uncertain how the results of the studies we reviewed would apply for a specific giving opportunity, and how estimates would differ across contexts and programmatic designs.

3.2 Impact of SMS reminders on vaccination rates

Our best guess is that a typical SMS reminders program results in a 15% reduction in the share of children who do not receive routine childhood vaccinations.

• This is based on a systematic review (Eze, Lawani and Acharya 2021) and four additional RCTs of SMS reminders in low- and middle-income countries. We view these studies as providing moderate quality evidence for the effectiveness of SMS reminders programs at increasing vaccination rates in contexts similar to those where we would expect to fund programs. (more)
• We conducted a meta-analysis of 16 RCTs in low- and middle-income countries and estimated that SMS reminders programs increase vaccination rates by 19%. To estimate the impact of a reminders program at different levels of baseline coverage, we calculate that this corresponds to a 34% decrease in the unvaccinated population. (more) Because the interventions studied include a range of specific programs and the effect sizes were highly heterogeneous, we’re highly uncertain how the results of the studies would apply to any particular program. (more)
• We adjust for internal as well as external validity concerns to arrive at our best guess for a 15% reduction in non-vaccination rates. We’re highly uncertain how the exact magnitude of the estimated program impact is likely to differ depending on the context in which it’s implemented. (more)
• We also think it’s likely that program effectiveness varies meaningfully depending on the programmatic design, but we have not estimated the impact of different program models at this time. (more)

Summary of literature

Eze, Lawani and Acharya 202112 conducted a systematic review and meta-analysis on the effect of SMS reminder interventions on child immunization in low-income and middle-income countries. Of the 18 studies included in the review, 13 were RCTs; and the majority of these RCTs were conducted in sub-Saharan Africa (8 studies), while the remainder were conducted in Guatemala (3 studies) and South Asia (2 studies).13

We conducted a literature search for studies released since Eze, Lawani and Acharya 2021 and identified 5 other RCTs of SMS reminders for vaccination conducted in low- and middle-income countries.14 We considered 4 of these additional RCTs (Yunusa et al. 2024, Chandir et al. 2022, Kagucia et al. 2021, and Mekennon et al. 2021)15 to be largely representative of SMS programs considered in this report and include the results in our meta-analysis (see footnotes for caveats).16 (more below)

The fifth study (Banerjee et al. 2021) reported on a large-scale RCT conducted in Haryana, India which cross-randomized SMS reminders about upcoming vaccination drives, incentives to caregivers for vaccinating their children, and community “ambassadors” to spread the word about immunization in their community.17 Because Banerjee et al. 2021 was not designed to evaluate SMS programs in isolation (more details in the footnotes)18 and we think the intervention being studied meaningfully differs from the program we’re evaluating in this report (because it involves a bundle of interventions to increase vaccination rather than SMS reminders only), we decided not to include the results in our meta-analysis.

• We’re uncertain whether to include the study in our meta-analysis, and we may incorporate the results if we investigate a specific giving opportunity that includes a bundle of interventions similar to those studied.
• When we include the SMS-only results from Banerjee et al. 2021 as estimated in Heidmann and Sankar 201919 in a version of our meta-analysis, the estimated effect size is very similar (see below).

For detailed summaries of all studies included in our meta-analysis, see GiveWell, Characteristics of SMS for vaccination studies.20

Meta-analysis of SMS reminder programs’ impact on vaccination uptake

We conducted our own meta-analysis to incorporate the additional studies identified. We estimate that a generic SMS reminders program increases vaccination uptake by roughly 19%. To estimate the impact of a program at different levels of baseline coverage, we calculate that this corresponds to a 34% reduction in the number of children who are not vaccinated. We have a moderate level of confidence that SMS reminders programs could be effective across a variety of contexts, but because of significant heterogeneity across the interventions studied and results, we’re very uncertain how the results would apply to any particular program, and we expect we would revise our estimates when investigating a specific opportunity.

Our approach

To estimate the effect of an SMS reminders program, we conducted our own meta-analysis incorporating the additional studies we identified (with the exception of Banerjee et al. 2021). We also excluded one study from Eze, Lawani and Acharya 2021 that did not publish relevant results.21 As per the methodology described in Eze, Lawani and Acharya 2021, we used the results of SMS reminders on full immunization coverage22 (and if full immunization results were not available, we used coverage of the third DTP or PENTA dose).23 For three studies where full coverage and DPT3 or PENTA3 results were not reported, we used other measures of immunization coverage as reported.24 For some studies, we were unable to corroborate the results from cited papers that were presented in Eze, Lawani and Acharya 2021, and we have noted discrepancies between any study-specific results cited in their paper and those we used in our meta-analysis in the cell notes here.25

We estimate that SMS reminders increased vaccination uptake in the included studies by roughly 19%.26 When including the results of Banerjee et al. 2021, we estimate roughly an 18% increase.27 We extrapolate from these results to estimate that an SMS reminders program leads to a 34% reduction in the unvaccinated population.28

• This is because we assume that SMS reminders programs will have different effects at different levels of baseline vaccination coverage. In particular, we assume a downward sloping, linear relationship between baseline vaccination coverage and the program's effect.
• We use this approach because it’s relatively simple to model, and we think this assumption captures the idea that a program's impact will be smaller in areas where there are fewer children who are unvaccinated (i.e. higher levels of baseline vaccination coverage) and larger in areas with more unvaccinated children.
• While we think it’s likely that a program’s impact would decline with higher levels of baseline coverage, we have some uncertainty about whether extrapolating using a linear relationship is the right approach because we did not observe a strong correlation when comparing effect sizes and baseline coverage across the RCTs. However, we think there are likely sources of heterogeneity across studies which make such a comparison imperfect.

Shortcomings and uncertainties

While the majority of studies found positive results of SMS reminders on vaccination coverage, and the pooled estimate was positive and statistically significant, we’re highly uncertain how the estimates would apply to any particular program. We view the estimate that includes all studies (with the exception of Banerjee et al. 2021) as being illustrative of the effectiveness of an SMS reminders program; but it’s likely we would update our best guess, for example by including only a subset of studies or adjusting the weights for the included studies, to give more weight to interventions which are more similar to a specific giving opportunity.

In addition, we’re uncertain about how other sources of heterogeneity should impact how we combine the study results and apply the estimate to any specific giving opportunity. We expect to investigate these more in the future.

• Differing study quality. We have not systematically conducted robustness tests, and it’s possible that we would adjust our estimates after a more in-depth review to exclude or give less weight to studies we view as lower quality. For example, the inclusion of one study (Yunusa et al. 2024) had a large impact on the results, and we aren’t sure how much weight to place on the study due to quality concerns and the inclusion of phone calls in addition to SMS reminders in the intervention (more details in the footnotes).29 Rather than exclude the study, we have applied a moderately large discount to our estimates (more).
• Combining different outcome measures. To estimate the effect size, our meta-analysis combines different measures of immunization uptake, largely based on full immunization or DTP3 completion. Where studies did not report either of these outcomes, we use the main outcome reported: “cumulative number of returns by 3 months” from Kawakatsu et al. 2020 and “median immunization coverage” from Seth et al. 2018. We’re uncertain whether it’s appropriate to combine different outcomes into one estimate, particularly for the studies that did not measure full immunization or DPT3 or PENTA3 completion.30 We also have not carefully reviewed all the individual studies for consistency in follow-up times and endline measurement of vaccination status, and it’s possible that an in-depth review would cause us to adjust our best guess of the program’s impact (more details in the footnotes).31 We expect that we would look into this in more depth when investigating specific programs.

Adjustments to meta-analysis results

We discount the estimated effect size of SMS reminders on vaccination rates due to the risk of bias from the inclusion of lower quality studies and publication bias, and concerns about the generalizability of RCT results to programs operating outside of a study context.

After these adjustments, we estimate that SMS reminder programs could lead to a 15% decrease in the number of children who are unvaccinated.

Adjustment for internal validity (study quality)

We apply a 45% downward adjustment for internal validity, primarily because of moderate concerns about study quality and the risk of publication bias. It’s possible that we would adjust this estimate when investigating a specific giving opportunity.

• Study quality. As noted above, inclusion of one study (Yunusa et al. 2024), which we have significant concerns about due to evidence of baseline imbalance, meaningfully increased the estimated effect size. We discounted our best guess of the program’s impact to account for these concerns. In addition, we think there’s some risk of response bias in a number of other studies included in our meta-analysis (more details in the footnotes).32
• Skeptical prior. In general, we expect published studies are more likely to overstate an intervention’s efficacy than understate it. In this case, we find it unintuitive that an SMS reminders program could cause increases in vaccination uptake that are comparable to our estimates from more intensive interventions such as New Incentives’ cash incentives program (we estimate that New Incentives’ program results in a 33% decrease in the share of unvaccinated children, prior to IV adjustments. For more details, see our intervention report). We also reviewed evidence presented in Eze, Lawani and Acharya 2021, which we think is suggestive of possible publication bias in the studies reviewed (more details in the footnotes).33

Adjustment for external validity (generalizability)

We apply a 20% downward adjustment for external validity (i.e. generalizability) because, in general, we expect interventions to have relatively smaller effects outside of a study context.

Since our findings are based on operational trials across a variety of settings,34 we have some confidence that SMS reminders are not highly context-specific, but we are highly uncertain what makes SMS reminders for vaccination more or less effective in different contexts. We will take contextual factors into account if we are considering a specific funding opportunity and may revise our external validity estimate for context-specific factors such as:

• Barriers to vaccination. SMS messages help alleviate one bottleneck to vaccination: remembering when vaccinations are due and, in some cases, providing information on when and where to access vaccinations. Whether alleviating this one bottleneck is enough to cause a meaningful increase in vaccination will vary by context, and is difficult to assess given the many factors that lead to someone receiving or not receiving a vaccine. For example, even if a caregiver receives a reminder and brings their child in for an immunization visit that they otherwise would have missed, it’s possible that the child may still not be vaccinated that day because of a vaccine stockout.
• Prevalence of cell phone ownership. An SMS reminder program inherently relies upon the target population being reachable via text messages. We have not incorporated this into our model because we assume that caregivers give a phone number at the time of enrollment. However, in locations with rates of phone ownership significantly below ownership rates in the settings where the RCTs were conducted, we might expect smaller benefits of the program (e.g. because phone numbers are shared within a household and caregivers responsible for bringing their children to get immunized do not receive the reminders, or because the population with phone numbers may be healthier and more likely to get immunized).
• Mobile coverage. We believe that successful message received rates are an important factor for program effectiveness because they heavily impact the effective population reached by any intervention. 6 out of 16 RCTs included in our meta-analysis reported message received rates, with an average rate of 82%. We have considerable uncertainty about this estimate given potential recall bias35 and the limited subset of studies reporting message received rates. We would lower our estimate of program effectiveness for a program which reported lower rates of messages received.
• Literacy rates. We expect an SMS program to be most effective in a population where caregivers who receive messages are able to read the messages received. If literacy rates are much lower than the contexts where the RCTs were conducted, we might expect text reminders to be less effective.

Other factors which may impact program effectiveness

SMS reminders programs encompass a variety of different models, and we think variations in program design are likely to impact effectiveness. We do not currently model the impact of different programs. If we investigate a specific funding opportunity, we would update our estimates for the effectiveness of an SMS program based on its specific features.

Method of recruitment. We currently assume the organization collects phone numbers from caregivers at a health clinic. Phone numbers may be collected from other sources such as general phone lists or other digital registries. We think the source of phone numbers is likely to impact a program’s effectiveness in two key ways:

1. Rates of successfully received messages. Actual received rates may vary considerably depending on from where the organization sources phone numbers, and the local provider and local mobile coverage rates. For example, we would expect a lower share of reachable phone numbers in a general phone list. Reaching a lower share of the population would likely decrease a program’s effectiveness, and/or significantly increase the cost per caregiver reached.
2. Population reached. Caregivers who don’t actively sign up for reminders may also systematically differ from those who choose to enroll in a program. We think most of the studies we reviewed are more representative of caregivers who actively enroll for messages because RCTs typically require informed consent. We’re uncertain how different populations of caregivers would respond to SMS reminders, and would take these factors into consideration when reviewing a specific giving opportunity.
   • Caregivers who do not actively enroll to receive SMS reminders may be less likely to modify their behavior in response to messages received, resulting in lower program effectiveness. A program which collects phone numbers from caregivers who enroll at a health clinic may also be more likely to reach members of the household who are responsible for bringing a child in for immunizations, and thus would be more likely to respond to SMS reminders.
   • Alternatively, a program which reaches a broader population may include a larger share of caregivers who would most benefit from SMS reminders because they have a lower counterfactual likelihood of bringing their children for vaccinations.

Whether the program allows for two-way communication. We currently model a program which only allows for one-directional communication (from the organization to the caregivers). A program may be designed to encourage two-way communication (for example, by sending questions alongside reminders), where caregivers could respond to messages or proactively contact providers (e.g. via text message responses, phone calls, a chatbot, etc.).

Our best guess is that two-way communication would have larger effects on vaccination rates than one-directional communication. This is because we believe two-way communication allows people to request information on immunizations in a relatively low cost way, and potentially increases caregiver motivation and trust in healthcare providers. In addition, we’re aware of some studies which suggest a larger effect of 2-way interventions in the context of other health services.36

We expect two-way SMS reminders programs to cost more per caregiver/infant reached than a one-way program. The relative effectiveness of a two-way SMS reminder program is also likely highly dependent on various factors such as literacy rates, the cost (if any) to households of sending messages, who in the household holds the phone, and what information is included.

Timing of recruitment. We currently assume that caregivers are recruited at the time of the child’s birth or near the time of the first immunization visit.37 Caregivers who are recruited for reminders earlier may experience larger benefits. For example, reminders could increase uptake of BCG, which we do not currently model benefits for because the dose is typically scheduled at birth (more). We’re also unsure whether effects would be as large in contexts where caregivers may be reached later in the child’s infancy period.

The content of messages. We expect messages would include the child’s name and subsequent immunization due date, but do not currently model differential effects from how reminders are framed.38 It’s possible that messages which are better tailored to caregivers, or messages which provide additional health messaging related to immunization (for example, about the benefits of immunization) would be more effective at increasing vaccination rates. Alternatively, shorter messages may be more likely to be read and remembered.

The timing and number of messages.39 We do not currently model differences in the effectiveness of reminders depending on the frequency or exact timing of messages. While it’s possible that more frequent or specifically timed messages may increase the effectiveness of SMS reminders, Eze, Lawani and Acharya 2021 did not find a significant difference in effects based on the number or timing of messages.40 In several studies caregivers also received a follow-up reminder within a week of missing an appointment, but we have not investigated whether additional follow-up increases immunization uptake.

4. What is the impact of increased vaccination uptake on mortality and other outcomes?

4.1 Summary

Our cost-effectiveness analysis models three main benefits from an increase in vaccination uptake, largely drawing on our prior work on New Incentives:41

1. Reduced mortality for children under age five from vaccine-preventable diseases. (more)
2. Reduced mortality for older children and adults. (more)
3. Long-term income increases from averting disease in a sensitive developmental window of childhood. (more)

For this report, we use Nigeria as an example of a context with low childhood vaccination coverage to illustrate the impact of a SMS reminders program. We estimate that in Nigeria, the probability of death from vaccine-preventable diseases for unvaccinated children under 5 is ~3% (more), and that receiving all childhood vaccinations could reduce this probability of death by ~52% (more).

In addition, we expect vaccines to provide some longer-term protection against the same set of diseases and reduce vaccinated children’s mortality risk later in life. Our analysis uses the same approach that we use to model averted mortality for children under five. We roughly adjust for declining vaccine efficacy over the course of a person's life and an assumption that deaths from vaccine-preventable diseases will likely decline over time as health generally improves.

Some of the main uncertainties in our estimates are:

• We do not make any adjustments to account for the fact that caregivers with access to phone numbers may be systematically different from the overall population (e.g. more or less at risk of death from a vaccine-preventable disease; more or less likely to have an effective immune response due to vaccination). We expect to take these factors into consideration when investigating a specific giving opportunity.
• We use estimates of mortality from IHME's Global Burden of Disease Project. These are based on a number of modeling assumptions that we have not reviewed in detail, and we have some doubts about their reliability.
• We have high confidence that vaccination reduces child (and later life) mortality, but have some uncertainty about the exact magnitude, particularly in low-income settings because we’ve seen some evidence that vaccine efficacy is lower in low-income countries (including Nigeria specifically).42

4.2 Reduced mortality for children under five and later in life

Probability of death among unvaccinated children

Based on earlier work on New Incentives, we estimate that receiving vaccines in the Nigerian child immunization schedule may lower the risk of dying from lower respiratory infections, meningitis, whooping cough, diphtheria, tetanus, measles, and diarrheal diseases.43 We estimate that an unvaccinated child’s risk of death before age five from the set of vaccine-preventable diseases considered here is 2.8% in Nigeria. This estimate also includes an assumption that the vaccine-preventable diseases we consider lead to 0.75 deaths from other causes for every death directly attributed to vaccine-preventable disease.

We rely primarily on estimates of vaccine-preventable disease mortality from the Institute of Health Metrics and Evaluation (IHME)'s Global Burden of Disease (GBD) project. We adjust for the following:

• Proportion of deaths from vaccine-preventable diseases caused by specific pathogens (i.e., disease etiology). We follow a similar approach as we did for New Incentives, which is described here. This estimate also includes an assumption that vaccine-preventable diseases cause 0.75 deaths from other causes for every death directly attributed to vaccine-preventable disease.
• Deaths that occur before vaccines are administered. We follow a similar approach as we did for New Incentives, which is described here.
• Vaccination coverage. Because the IHME data include children who are vaccinated and unvaccinated, we adjust baseline mortality to reflect a higher-than-average probability of death for unvaccinated children specifically. We follow a similar approach as we did for New Incentives, which is described here.

In addition, we expect vaccines to confer some longer-term protection against the same set of diseases. Our analysis follows a similar approach for estimating the vaccine-preventable burden that could be averted later in life, but we make rough downward adjustments based on the assumption that deaths from vaccine-preventable diseases will likely decline over time because we expect overall human health to improve in the future.

While we believe the vaccine-preventable disease burden is high in this context, we have a number of major uncertainties about the specific estimates we use, including uncertainty about the reliability of GBD’s estimates and whether our adjustments for unvaccinated children and all-cause mortality are correct.

See our New Incentives (Conditional Cash Transfers to Increase Infant Vaccination) report for more detail.

Effects of vaccination uptake on mortality

Based on earlier work on New Incentives, we estimate the impact of the following vaccines in the Nigerian child immunization schedule: pentavalent vaccine (Penta), pneumococcal conjugate vaccine (PCV), measles vaccine (MCV) and the rotavirus vaccine.44 Some of these vaccines provide near full protection from the corresponding diseases that they target45 , while others provide partial protection (e.g. rotavirus vaccine is partially protective against rotavirus diarrhea, and not protective against other causes of diarrhea). We estimate that in Nigeria fully vaccinating a child reduces the incidence of severe illness caused by this set of diseases by roughly 52%, relative to receiving no vaccines.46 We assume that vaccines reduce mortality from these causes by the same percent that they reduce incidence.

In addition, we would expect vaccines to provide some longer-term protection against the same set of diseases. We follow a similar approach to estimating vaccine efficacy later in life, but we make rough downward adjustments based on the assumption that the protection conferred by vaccines received during childhood declines over the course of a person's life.47

We have some uncertainty about our efficacy estimates in Nigeria, and how they would differ in other settings.

We generally follow a similar approach as we did for New Incentives, but make minor adjustments to our previous work given the different contexts and program types modeled:

• Vaccine efficacy estimates. We use a slightly higher vaccine efficacy for measles compared with our New Incentives model to reflect a two-dose vaccine efficacy estimate.48 The second measles dose was introduced into the Nigerian vaccination schedule in 202049 after we first modeled the impact of New Incentives’ program. At the time of writing, New Incentives’ program incentivizes caregivers to bring their children in to receive the second measles dose, but we have not prioritized updating the model to account for the additional benefits of a two-dose measles vaccine because we believe it is likely to have a small impact on cost-effectiveness.
• Exclusion of BCG vaccine effects. The BCG vaccine is scheduled to be received at birth and reduces the risk of death from tuberculosis. Because we assume SMS programs typically solicit phone numbers for caregivers at the time of a child’s birth or first immunization visit, we think it is unlikely that SMS reminders would target or cause a substantial behavioral change in BCG vaccine coverage. We exclude benefits (as modeled in our New Incentives CEA) that accrue from BCG vaccine uptake.

See our New Incentives (Conditional Cash Transfers to Increase Infant Vaccination) report for more detail.

4.3 Long-term income increases

Our best guess is that vaccination leads to small income/consumption increases in adulthood. We call these long-run economic benefits "development effects". Based on previous work on New Incentives, we estimate that the income benefits of an SMS reminders program are 31% as large as the benefits from deaths averted.

See our report on New Incentives (Conditional Cash Transfers to Increase Infant Vaccination) for more information.

4.4 Additional program benefits and downsides

Our cost-effectiveness analysis includes a number of additional benefits and downward adjustments related to this intervention that we have opted not to explicitly model. Instead, we incorporate them as rough percentage best guesses. These adjustments increase our estimate of the impact of an SMS reminders’ program by 20%.50

We believe that increasing vaccination coverage may have additional benefits, such as reducing morbidity from vaccine-preventable diseases, averting treatment costs, and reducing disease transmission.51 We also think that there could be some additional mortality benefits due to a) increased timeliness of vaccination, which may cause the child to be susceptible to disease for a shorter period of time,52 and b) increased coverage of vaccinations that we have not explicitly included in our cost-effectiveness analysis.53 We have used a rough estimate for additional benefits as per our work on New Incentives (see here for more detail). We adjusted these estimates where they did not apply to a general SMS reminder program.54 We may revise these estimates further if we investigate a specific giving opportunity.

We similarly use rough estimates for offsetting effects and downside adjustments as per our work on New Incentives (see here for more detail). In particular, we currently make an -18% adjustment in our model to account for the possibility that childhood vaccination rates are increasing even absent intervention in many parts of the world, which may reduce the impact of an SMS reminders program. We may revise these estimates if we investigate a specific giving opportunity.

4.5 Grantee-level adjustments

Our cost-effectiveness analyses also include adjustments relating to the specific organizations we recommend rather than the intervention itself. These reflect aspects of the organization’s delivery of a program (such as risk of fraud or quality of the monitoring and evaluation) that might have an impact on cost-effectiveness. Rather than explicitly model these, we apply them as rough percentage best guesses.

Based on our work on New Incentives, we estimate that these factors reduce the cost-effectiveness of an SMS reminders program by 7% on net. These estimates are illustrative, and grantee-level adjustments would be updated in the context of a specific organization.

See our New Incentives (Conditional Cash Transfers to Increase Infant Vaccination) report for more detail.

4.6 How does the program affect other actors’ spending?

Part of our cost-effectiveness analysis involves asking what impact funding a program has on other actors’ spending. We think an SMS reminders program may lead other organizations or governments to spend more (we refer to this as "leveraging" funding, or “crowding in”) or less (we refer to this as "funging," from “fungibility,” or “crowding out”) on vaccines than they otherwise would.

Based on our work on New Incentives, we include “leverage and funging” adjustments in our cost-effectiveness analysis to account for the possibility that GiveWell funding would cause other actors to spend more on vaccines than they otherwise would, and to account for the possibility of other actors’ spending less on an SMS reminders program than they otherwise would. These lead to roughly a 21% decline in cost-effectiveness.

See our report on New Incentives (Conditional Cash Transfers to Increase Infant Vaccination) for more information.

5. How cost-effective is the program?

5.1 Summary

We created a preliminary cost-effectiveness analysis of an SMS reminders program for childhood vaccinations. As of August 2024, we estimate that SMS reminders programs are roughly 12 times as cost-effective as unconditional cash transfers, which is in the range of cost-effectiveness of programs we expect to direct funding to.55 This cost-effectiveness analysis is for a generic program that provides SMS reminders to caregivers for vaccinations in Nigeria. We expect our bottom line cost-effectiveness estimate will change for specific giving opportunities.

Note that our cost-effectiveness analyses are simplified models that do not take into account a number of factors. There are limitations to this kind of cost-effectiveness analysis, and we believe that cost-effectiveness estimates such as these should not be taken literally due to the significant uncertainty around them. We provide these estimates (a) for comparative purposes and (b) because working on them helps us ensure that we are thinking through as many of the relevant issues as possible.

This cost-effectiveness analysis is at an early stage, and we think it’s likely that our bottom line cost-effectiveness estimate will change with research into specific funding opportunities.

5.2 Sketch of our cost-effectiveness analysis

Intuitively, we think SMS reminders for vaccination are cost-effective because sending SMS messages is inexpensive and leads to modest reductions in the number of children who do not receive routine childhood vaccinations. We think that increasing the number of children who receive childhood vaccinations significantly reduces mortality, both in childhood and later in life, and likely leads to long-term income gains and other benefits. This leads to high cost-effectiveness in contexts with relatively low vaccination coverage. We expect that costs and program models will differ significantly across programs and lead to large differences in cost-effectiveness.

A sketch of the cost-effectiveness model is below:

• We estimate that an SMS reminders program for childhood vaccination in Nigeria is around 12 times as cost-effective as an unconditional cash transfers program.
  • We consider a hypothetical $100,000 donation to an organization operating an SMS reminders program.
  • We estimate that vaccination promotion via SMS costs around $2.90 per caregiver receiving SMS messages, in part based on data we’ve seen from a program we’ve supported in Pakistan which sends SMS reminders for immunizations through a digital immunization registry.56 This assumes that two reminders are sent per scheduled vaccination.57 With a hypothetical donation of $100,000, an SMS reminders program is expected to reach approximately 34,200 caregivers.58
  • Using national-level vaccination coverage estimates from UNICEF, we estimate that 65% of children would be vaccinated absent the intervention.59 We estimate that SMS reminders may plausibly decrease the number of children who are not vaccinated by 15%, or roughly an additional 1,800 children who are vaccinated.
  • We estimate that the probability of death from vaccine-preventable diseases among unvaccinated children under 5 is around 2.8% in Nigeria.60 This includes an adjustment for mortality averted from other causes in addition to the specific infections that the vaccines are intended to prevent.
  • We estimate that being vaccinated reduces a child’s mortality risk from vaccine-preventable causes by approximately 52%.
  • As a result, we estimated that around 26 deaths would be averted for children under 5.
  • In addition, we think unvaccinated children are at an elevated risk of dying from vaccine-preventable causes later in life,61 and we expect vaccination during childhood to reduce mortality later in life by providing long-term protection against diseases, but we expect these effects to be smaller.62
  • In addition to averting mortality, we think the program likely results in the following benefits:
    • We estimate that vaccination in a sensitive development period of childhood results in increased income for children later in life. This accounts for around 22% of the benefits we model.
    • We assume that supplemental grantee-level adjustments would reduce the overall value of the program by 7%. The grantee-level adjustments are illustrative and based on our work on New Incentives.
    • We include various other unmodeled benefits and downsides, and apply best guesses. These adjustments increase cost-effectiveness by an additional 20%.
    • We also make adjustments for leveraged costs incurred by the government and Gavi (for vaccine purchases) as a result of the program, and the possibility of funging the government. These lead to roughly a 21% decline in cost-effectiveness.
• This implies that an SMS reminder program for vaccination could avert a death for an average of ~$4,700, which is approximately 12 times as cost-effective as an unconditional cash transfer.

5.3 How could our model be wrong?

We have significant uncertainty about an SMS program’s effectiveness at increasing vaccination rates, particularly in different settings, and the costs of specific SMS reminders programs.

• In our model, cost-effectiveness is driven by increases in vaccination uptake for earlier doses because we assume that a significant portion of protection is conferred by the first or second dose of two-dose or three-dose vaccines, respectively.63
• The effect size we estimate is primarily based on measures of full immunization or DTP3 uptake, which typically reflect vaccines received at visits later in the routine schedule. It’s possible that an SMS reminders program would have different effects on coverage at different vaccination visits (for example, a program may have smaller effects on DTP1 than DTP3 coverage because infants are more likely to receive at least one dose even absent intervention. Alternatively, a reminders program may have larger effects on earlier doses because it’s easier to get caregivers to return for one or two visits than to persist through all doses).

What is the program’s impact in different contexts? We would expect a program’s effectiveness to differ based on context-specific factors such as location-specific barriers to vaccination, characteristics of the population reached, and the vaccines targeted by the program. We have estimated a general effect of SMS reminders on vaccination using Nigeria as an example, but if we investigated a specific giving opportunity, we would consider location-specific adjustments. For examples of specific factors we might consider, see above.
How does effectiveness vary by program design? We think the uptake in vaccination caused by an SMS reminder program is likely to differ based on a number of program modalities, including how and when caregivers are recruited, the quality and content of messages, and whether the program is combined with other vaccination interventions. We have estimated a general effect of SMS reminders as an example, but if we investigated a specific giving opportunity, we would consider how variations on program design impact effectiveness. For examples of specific factors we might consider, see above.
How much does an SMS reminders program cost? Our estimates are currently based on costs from a digital immunization registry GiveWell has supported in Sindh province, Pakistan, which records immunization data and other information in addition to sending SMS reminders.64 We think it’s most likely that a SMS reminders program would rely on existing databases (e.g. an electronic or paper registry at a health facility) because it would be relatively costly to set up a system and hire staff solely to collect phone numbers and send SMS reminders for immunizations. However, we are highly uncertain about how applicable these costs may be to SMS programs in other contexts, in particular standalone SMS programs operating at a smaller scale. We expect to be able to refine our estimates if we investigate a specific giving opportunity.

• Estimated costs for SMS programs in the literature appear to be lower in general,65 however we have not conducted an in-depth investigation of the costs from these programs.
• We expect a smaller scale program to have higher costs per child because the fixed costs for running a standalone program would be spread out over fewer recipients.

6. Other Considerations

In theory, our cost-effectiveness analysis intends to capture the total impact of a program per dollar spent. But we recognize that our cost-effectiveness calculations are not able to capture every factor that could make a program more or less impactful. Focusing only on our cost-effectiveness model may mean we’re missing things that are difficult to quantify.

As a result, we think it’s helpful to look at other perspectives and types of evidence that may not be captured in our bottom line cost-effectiveness number. We have not engaged deeply with this set of questions at the time of writing, but some additional perspectives that we have considered include:

• Would an SMS program for immunizations be bundled with other interventions? This report focuses on a standalone SMS reminders program, but we haven’t seen data from standalone programs like this. For example, we’ve supported IRD Global to provide SMS reminders on top of an electronic immunization registry in Sindh, Pakistan (for more, see our separate report on IRD); and we’ve supported Suvita to send SMS reminders for routine childhood vaccinations, breastfeeding, and healthcare during pregnancy (for more, see our grant page on Suvita). We expect that it would be relatively uncommon for an organization (or government) to solely enroll caregivers into a system intended to send SMS reminders for immunizations. We think it’s most likely that an SMS reminders program we investigate would be built on top of an existing system or be bundled with other interventions, and we’re uncertain how this would affect the impact of an SMS reminders program.
  • Interaction with other reminder content. Digital tools could combine reminders for vaccination with reminders for other health-related activities. Depending on the content of other reminders, we might expect larger benefits accruing from vaccination reminders, or due to benefits from additional health services. We will consider the interactions of different reminder types if we investigate any specific organizations.
  • Interaction with other vaccination interventions. Banerjee et al. 2021 found that SMS reminders alone did not increase vaccination coverage for measles vaccines,66 but that it may be effective in combination with other promotion activities such as incentives or information hubs.67 We are uncertain about the cost-effectiveness of SMS reminders alone, compared with the cost-effectiveness of a bundles approach in any given context, and when one should be preferred over the other.
• Are other digital reminders similarly effective to SMS? How does adoption of other communications technologies affect the impact of SMS messages? This report focuses on SMS reminders, but similar reminders messages could be sent through other digital communications platforms such as WhatsApp68 or Telegram.69 These digital tools could also be used to deliver other forms of context such as videos or images, which may make them more effective tools to increase immunization uptake. Adoption of other digital communications platforms may also lower the effectiveness of SMS reminders if people favor those technologies and ignore messages from SMS. We have not reviewed evidence on the effectiveness of other digital tools.
• Who is best suited to implement an SMS reminders program in a given context? For example, an SMS reminders program could be delivered by the government, or implemented by non-governmental organizations in conjunction with the routine healthcare system. Who the implementer is may impact the quality and reach of an SMS reminders program, and the ability of an SMS reminders program to scale.
• What do other experts and practitioners think about SMS reminders programs? We’re more confident in programs which have wide support from experts and government authorities in the countries where they’re delivered and the global health community more widely. At the time of writing, we have not spent significant amounts of time engaging with other experts and practitioners about SMS programs.
• Have there been national or other wide-scale success stories for SMS reminders? To estimate the impact of an SMS program, we extrapolate from RCTs that typically operated on a smaller scale and only measured impacts over a short window of time. We have not looked for evidence from programs operating SMS reminders at large-scale and for an extended duration, which could cause us to update our best guess for the impact of an SMS program.

7. Is there room for more funding?

We have not yet estimated total room for more funding for this intervention. Based on the large target population and relatively low child vaccination rates in some settings, there may be substantial room for more funding, though we are highly uncertain about this and have not investigated the degree to which financial resources are a limiting factor in scale-up for SMS reminder programs.70 There may be opportunities to support government-led programs via technical assistance, as well as directly fund programs implemented by NGOs. We plan to continue to investigate potential funding opportunities.

8. Key questions for further investigation

Our biggest uncertainties for an SMS reminders program are around the effectiveness of a particular program in a specific context.

• What would a specific SMS reminders program look like? Which studies would be most representative of a particular SMS program?
• How do context-specific factors like the barriers to vaccination or phone ownership rate impact program effectiveness?
  • For any given vaccine, what is vaccination coverage, vaccine efficacy, and the vaccine-preventable disease burden in a specific context?
• How do variations on program design like the method of recruitment and direction of communication impact the effectiveness of SMS reminders programs?
• How effective are SMS reminders programs at increasing uptake for different vaccines?
• What are the costs of specific SMS reminder programs?
• Would there be other interventions included alongside SMS reminders programs?
• What organizations and governments have room for more funding to scale SMS reminders programs?
• What do other experts and practitioners think about SMS reminders programs?
• Are there examples of large-scale SMS reminders programs which have successfully increased immunization uptake?

9. Our process

We reviewed GiveWell’s previous work on SMS reminders for vaccination and conducted a search for more recent evidence on the effect of SMS reminders on vaccination coverage. We conducted our own meta-analysis incorporating more recent RCTs conducted in LMICs that we were aware of. We also updated our cost-effectiveness analysis to align with GiveWell’s existing work on the impact of vaccinations.

Sources

• 1

  For more, see our New Incentives report

• 2

  This is based on the results of a randomized controlled trial (RCT) of New Incentives’ program in three states in northern Nigeria:

  • The RCT estimated that New Incentives’ program caused a 27 percentage point increase in vaccination rates. We extrapolate from this to estimate that the program leads to roughly a 33% reduction in the number of unvaccinated children.
  • We then apply a -12% internal validity adjustment to incorporate evidence from other studies of vaccine incentives we have reviewed which found smaller impacts. 33% * (1 - 12%) = 29%

  For more details, see our New Incentives report.

• 3

  For more details, see this section of our review of New Incentives' program.

• 4

  For more see, our New Incentives report.

• 5

  “Immunization has reduced the number of deaths from infectious diseases dramatically. Vaccines also prevent disability, which can impair children’s growth and cognitive development, so that they not only survive but also flourish. Vaccines benefit not only infants and children but also older people. They can prevent infection-related cancers and protect the health of the elderly and the vulnerable, allowing people to live longer, healthier lives. In addition, fewer infections mean less risk of transmitting disease to relatives and other members of the community.” World Health Organization, "Immunization Agenda 2030: A Global Strategy to Leave No One Behind", p. 12.
  "Vaccines protect against many different diseases, including: cervical cancer, cholera, COVID-19, diphtheria, hepatitis B, influenza, Japanese encephalitis, malaria, measles, meningitis, mumps, pertussis, pneumonia, polio, rabies, rotavirus, rubella, tetanus, typhoid, varicella, yellow fever” [modified from a bulleted list in the original source] World Health Organization, "Vaccines and immunization: Impact

• 6

  “An estimated 25 million children under the age of 1 year did not receive basic vaccines, which is the highest number since 2009…In 2021, the number of completely unvaccinated children increased by 5 million since 2019.” World Health Organization, “Immunization coverage”

• 7

  “Childhood immunisations are often delayed or missed either due to caregivers’ lack of awareness about the vaccines or their due dates.10 Many LMICs do not have functioning primary healthcare systems and routine well childcare services, making a somewhat complicated primary childhood vaccine series (with multiple appointments at various ages) difficult for caregivers to remember.8 9” Eze, Lawani and Acharya 2021

• 8

  For example, Eze and Adeleye 2015 reported on a study where follow-up messages were sent to caregivers who did not take their child for a vaccination visit: “Recall messages were sent to defaulters and their responses (presence at immunization session) assessed the next RI [routine immunization] session.”

• 9

  For example, Patel and Pandit 2011 found that a lack of reminders was a major reason reported by caregivers for missed vaccination visits in a rural area of Anand District, Gujarat: “Reasons for 'missed' vaccination were prior reminder not given (32.9%, P<0.01); mother's forgetfulness (26.6%); unavailability of vaccine (15%).”

• 10

  The routine immunization schedule differs depending on the country, but we expect the first immunization visit after birth to occur at around 6 weeks.

• 11

  See this column of our detailed summary of SMS for vaccination studies for examples of message text.

• 12

  We are aware of two other meta-analysis, but focus on Eze, Lawani and Acharya 2021 for the following reasons:

  • Mekonnen et al 2019 contains several studies conducted in the US, while all the included studies conducted in LMICs were also included in Eze, Lawani and Acharya 2021.
  • Yunusa et al 2021 analyzed both phone call and SMS reminders and included fewer SMS reminder studies than Eze, Lawani and Acharya 2021.

• 13

  See Table 1, Eze, Lawani and Acharya 2021

• 14

  We searched the first five pages of Google Scholar for the terms “SMS reminders vaccination” and “SMS vaccination RCT” (results filtered to be since 2020). We considered RCTs that targeted infant vaccination in low- and middle-income countries.
  We also identified two additional quasi-experimental studies. Although we did not incorporate these in our meta-analysis, they further provide support that SMS reminders increase vaccination coverage:

  • Yunusa et al. 2022 (conducted in Nigeria): “Three local government areas were each allocated to the intervention (reminder) and control groups of the study. Mobile phone reminders (SMS and follow-up calls) were sent to mothers in the reminder group three days to and on the due date of their child's schedule for the 1st, 2nd and 3rd doses of the pentavalent vaccine…A total of 541 mothers (271 in the intervention group and 270 in the control group) participated in the study. Completion rates for the three doses of the pentavalent vaccine were observed to be higher for children in the reminder group (n = 161, 59.4%) compared to those in the control group (n = 92, 34.1%).”
  • Ibraheem et al. 2021 (conducted in Nigeria): “Mother-infant pairs presenting for the first vaccination appointment were randomized into four (three interventions, one control) groups, each consisting of 140 participants. Intervention groups were reminders via calls (A), SMS reminders (B), immunization fact SMS messages (C) and controls on usual care (D). Reminders were made a day before the appointment while SMS immunization facts were sent at five weeks, nine weeks and eight months…The immunization completion rates after the nine months’ visit were 99.2%, 99.3%, 97% and 90.4% for Groups A, B, C and D respectively.”

• 15
  • Yunusa et al. 2024 (conducted in Nigeria): “We conducted a parallel arm cluster randomized controlled trial in four primary health care facilities in Nigeria. Reminders were sent to eligible participants in the intervention group at specific intervals when their children were scheduled to receive the vaccines administered at the sixth, 10, and 14 weeks after birth. Immunization records of all participants’ children were then tracked to assess their immunization status. The immunization status of the intervention (n = 275) and control (n = 261) arms was analyzed. Completeness and timeliness of the vaccine series were significantly higher (p < .001) among children of participants in the intervention (n = 169, 61.5% and n = 138, 50.2%) than those in the control group (n = 35, 13.4% and n = 13, 5%) arm.”
  • Chandir et al. 2022 (conducted in Pakistan): “Participants in Karachi, Pakistan, were individually randomized into a seven arm, factorial open label study with five mCCT arms, one reminder (SMS) only arm, and one control arm… SMS had a marginally statistically significant impact on FIC [full immunization coverage] versus control (OR: 1.16, 95% CI: 1.00-1.35; p = 0.046). Findings were similar for up-to-date coverage of penta-3, measles-1 and measles-2 at 18 months.”
  • Kagucia et al. 2021 (conducted in Kenya): “Caregivers of eligible infants aged 6–8 months were enrolled into an individually randomised controlled trial and assigned to receive either: no intervention (control), two SMS reminders (SMS) sent 3 days, and 1 day before the scheduled MCV1 date, or SMS reminders coupled with a Kenya Shilling (KES) 150 incentive (SMS +150 KES) sent 3 days before the scheduled MCV1 date…Between 6 December 2016 and 31 March 2017, 179 infants were enrolled into each of the three study arms. Follow-up visits were completed between 19 April 2017 and 8 October 2017 for control (n=170), SMS (n=157) and SMS + 150 KES (n=158) children. MCV1 timely coverage was 68% among control arm infants compared with 78% in each intervention arm. This represented a non-statistically significant increase in the SMS arm (adjusted relative risk 1.13; 95% CI 0.99 to 1.30; p=0.070; adjusted risk difference 9.2%; 95% CI: −0.6 to 19.0%; p=0.066)”
  • Mekennon et al. 2021 (conducted in Ethiopia): “Participants assigned to the intervention group received mobile phone text message reminders one day before the scheduled vaccination visits… A total of 426 participants were included for the analysis. We found that a higher proportion of infants in the intervention group received Penta-3 (204/213, 95.8% vs 185/213, 86.9%, respectively; P<.001), measles (195/213, 91.5% vs 169/213, 79.3%, respectively; P<.001), and full vaccination (176/213, 82.6% vs 151/213, 70.9%, respectively; P=.002; risk ratio 1.17, 95% lower CI 1.07) compared to infants in the usual care group.”

• 16

  We considered excluding Kagucia et al. 2021 because the intervention studied a somewhat different intervention than the one we broadly outlined in this report: caregivers of infants aged 6-8 months were included in the study as opposed to infants around the time of birth. The main outcome measure was completion of the first dose of the measles-containing vaccine (MCV1).
  Kagucia et al. 2021: “Caregivers of eligible infants aged 6–8 months were enrolled into an individually randomised controlled trial and assigned to receive either: no intervention (control), two SMS reminders (SMS) sent 3 days, and 1 day before the scheduled MCV1 date, or SMS reminders coupled with a Kenya Shilling (KES) 150 incentive (SMS +150 KES) sent 3 days before the scheduled MCV1 date. Study staff conducted a household follow-up visit at age 12 months to ascertain vaccination status.”

• 17

  From Banerjee et al. 2021,

  • “Our empirical application is a large-scale experiment covering seven districts, 140 Primary Health Centers (PHCs), 2,360 villages involved in the experiment including 915 at risk for all the treatments, and 295,038 children in the resulting data base, which we conducted in collaboration with the government of Haryana, India.”
  • “The policies under consideration include the two most frequently discussed tools—reminders and incentives—as well as an intervention inspired by the networks literature. We cross-randomize whether (a) individuals receive SMS reminders about upcoming vaccination drives; (b) individuals receive incentives for vaccinating their children; (c) influential individuals (information hubs, trusted individuals, or both) are asked to act as “ambassadors” receiving regular reminders to spread the word about immunization in their community. By taking into account different versions (or “dosages”) of each intervention, we obtain 75 unique policy combinations.”

• 18
  • The study does not report the results of SMS in the absence of other cross-randomized interventions. One of the study authors cautioned against estimating treatment effects for interventions in isolation because the paper was about interaction effects and underpowered for analysis on very specific combinations.
    Email from Anirudh Sankar to GiveWell, June 23, 2023 (unpublished): “I understand its very tempting in this setting with 75 treatment combinations to want to run regressions for a very specific combination (such as "gossips alone") you are most interested in for various reasons. However, from the perspective of the paper this is not kosher. Not only is one underpowered for this analysis, but there is also [an]overfitting issue. With 75 cells and being able to run any contrast you increase the chance of seeing flukes, both positive and negative. This is why we don't do the full unroll of every comparison, and instead [run] this pruning and pooling procedure. Through it, we tie our hands in an important way. [...] I believe the senior coauthors have tons of papers for the treatments in isolation (maybe in different settings). This was about the interaction effects. Which policy combinations pack a punch -- especially if they can be considered the same modulo some dosage levels. This is why, of all that you have explored, only one question seems to lean more kosher within the perspective of the paper : does gossips + SMS emerge as a potent policy?”
  • The study used SMS reminder arms where 0%, 33%, and 66% of caregivers received reminders. No treatment arms involved all caregivers receiving SMS reminders.

• 19

  Heidmann and Sankar 2019 estimates the effect of reminders on vaccination rates using a set of controls for the incentives and a set of controls for ambassadors interventions. The results found in Table 47 did not report a significant effect of reminders on vaccination coverage, but we think it’s likely that the study was underpowered for SMS-only results. However, power calculations weren’t reported in the results, so we don’t have a good sense of how underpowered the trial was.

• 20

  This builds on a previous set of notes on the evidence for SMS reminders for vaccination shared by Charity Science Health.

• 21

  Ceballos et al. 2020 did not report data on vaccination coverage.

• 22

  The definition of full immunization coverage may differ depending on context and which vaccines are included in a specific country’s national immunization schedule. One common measure of being fully immunized is receipt of the following vaccines: BCG, 3 doses of DPT-containing vaccine, 3 doses of polio vaccine (excluding polio vaccine given at birth), and 1 dose of MCV.
  “Fully vaccinated (basic antigens):
  BCG, 3 doses of DPT-containing vaccine, 3 doses of polio vaccine (excluding polio vaccine given at birth), and 1 dose of MCV.” Guide to DHS Statistics DHS-8

• 23

  “Where studies reported data for both overall immunization and DPT-3 data, we extracted data for overall immunization. However, we also included studies that reported only DPT-3 outcomes, given the conventional use of DPT-3 to monitor progress of interventions aimed at improving vaccine delivery services.12 37 DPT-3 coverage—defined as the proportion of children receiving complete (three) doses of diphtheria, pertussis and tetanus—is a particularly valuable measure of the childhood immunization coverage and countries’ vaccine delivery effectiveness.” Eze, Lawani and Acharya 2021

• 24

  The three alternative measures of immunization coverage were median immunization coverage (Seth et al. 2018), cumulative number of returns by 3 months after original appointment dates (Kawakatsu et al. 2020), and first-dose measles-containing vaccine (Kagucia et al. 2021).

  • Seth et al. 2018: “"Immunization coverage is defined as the proportion of the total number of immunizations received by a child divided by the total number of immunizations required for the child for their age at the time of measurement.”

  “Median immunization coverage at enrollment was 33% in all groups and increased to 41.7% (interquartile range [IQR]: 23.1%–69.2%), 40.1% (IQR: 30.8%–69.2%), and 50.0% (IQR: 30.8%–76.9%) by the end of the study in the control group, the group with mobile phone reminders, and the compliance-linked incentives group, respectively.”

  • Kawakatsu et al. 2020: "The return rate for child vaccinations in the intervention group was significantly higher (p < 0.001) by 4.8% - 6.0% than that in the control group, consistently across all the five different timings (on time as scheduled, and by 7 days, 14 days, 30 days, and 3 months after appointment dates)."
  • Kagucia et al. 2021: "The primary outcome was the proportion of infants receiving MCV1 by age 10 months (304 days; ie, MCV1 timely coverage)."

• 25

  We were unable to find the underlying data in the case of Ceballos et al. 2020. In some other situations, we believe they may have used unrounded data (Bangure et al. 2015), rounded data (Gibson et al. 2017), or used a different set of results (as in the cases of Dissieka et al. 2019, Domek et al. 2016 (pilot), Domek et al. 2019, and Eze et al. 2015). We are unsure why our estimates for Schlumberger et al. 2015 differ.

• 26

  See our cost-effectiveness analysis, “Uptake” sheet, for our calculations of the effect of these interventions on vaccination coverage. Using the random effects model, RR was calculated as 1.1938, with a 95% confidence interval of [1.0523; 1.3543]. We use the random effects model because we think that the impact of SMS reminders on vaccination uptake is likely to vary significantly across study contexts..

• 27

  Including the results of Banerjee et al. 2021 leads to an RR of 1.1798, with a 95% confidence interval of [1.0483; 1.3279]. See the figure to the right of the "Uptake" sheet in our cost-effectiveness analysis.

• 28

  In the studies we use to calculate our effect size, we roughly estimate a baseline vaccination rate of 64%, or that 36% (= 1 - .64) of infants were unvaccinated. We estimate that SMS reminders programs increase vaccination rates by roughly 19.4% or 12.4 (= .64 * .194) percentage points. This equals a 34% reduction (=.124 / .36) in the non-vaccinated population.

• 29

  Excluding the results from Yunusa et al. 2024 lowered the estimated effect size to 1.11 (95% CI: 1.06, 1.17). See our results in this worksheet. We think the program might have larger effects than a typical SMS reminders program because phone calls were included in addition to the messages.
  In addition, we have some concerns about study quality due to baseline imbalances between groups that we think might bias the results upwards. In particular, there were significant differences in the average monthly income between groups, with a larger share of control group participants (45.2%) in the lowest income bracket of <30,000/38.4 (Naira/USD) compared to the intervention group (2.9%). Including the next income brackets, a total of 75.9% of control group participants were in the lowest two income brackets compared to the 61.4% in the intervention group (45.2% + 30.7% = 75.9% and 2.9% + 58.5% = 61.4%). In general, we think lower socio-economic households are less likely to get their children vaccinated (e.g., because it’s more difficult to cover transportation costs), so by not controlling for these differences between groups, we expect that the estimates are biased towards the intervention group. See Table 1 in Yunusa et al. 2024.

• 30

  See this column of our cost-effectiveness analysis for the outcome measures we used in our meta-analysis.

• 31

  SMS reminders may increase the number of children who receive their scheduled vaccines at all (i.e. vaccination completion) and/or increase the number of children who receive vaccines when they are first scheduled to receive particular vaccines (i.e. vaccination timeliness). If SMS reminders cause more infants to receive their vaccinations on time, we think studies which measure completion using a shorter follow-up window (e.g., endline measurement at 14 weeks when a DTP3 or Penta3 vaccine is scheduled compared to a follow-up at 6 months) are likely to estimate larger differences in immunization completion between treatment and control groups because some children would likely receive the vaccines after the scheduled date. Our analysis primarily considers the effect of SMS reminders on vaccination completion, so including effect sizes from studies with a relatively short follow-up window may cause us to overestimate impact.
  We have not reviewed the results on timeliness in-depth, but we think there is some evidence that SMS reminders may increase timeliness based on the results in Eze, Lawani and Acharya 2021. “Of the 12 included studies that evaluated the effect of SMS reminders on childhood immunisation timeliness, 10 studies demonstrated that SMS reminders significantly improve timely receipt of vaccines in children in the intervention group compared with those in the control group with usual care. Meta-analysis of data from included interventions (n=12, sample size=25 257 participants) showed that SMS reminders significantly improved timely receipt of childhood vaccines; RR=1.21; 95% CI: 1.12 to 1.30; I2=87.3%—figure 2B. However, the predictive interval for this effect overlaps the null (0.93, 1.56), indicating some uncertainty about the distribution of effects in similar populations.”

• 32

  Endline data on vaccination rates was collected using a mix of written and digital immunization records (at clinics and households) and verbal reports. See our summary of program characteristics here.
  For the studies using verbal reports to collect some or all of their endline data, we think there’s some risk that caregivers report immunizations that did not take place. While there’s likely also some risk of bias in the control groups, we think it’s likely that the risk of response bias is higher in the intervention groups because these were the caregivers being targeted with immunization reminders and thus, they may be more inclined to respond favorably when asked about their child’s vaccination status.

• 33

  Eze, Lawani and Acharya 2021 concludes there is an absence of evidence for publication bias through examination of symmetry of a funnel plot and a Harbord test. After examining the funnel plot presented in the supplemental materials, we think there is some evidence for possible publication bias due to our subjective judgment that there is imbalance to the right of center. We also believe that a statistical analysis for publication bias is likely underpowered due to too few studies. We did not make a new funnel plot to incorporate the additional trials in our meta-analysis because we do not believe it would update us significantly.
  “Funnel plot presents graphical diagnostics of small study effects based on subjective visual inspection. Symmetric location of individual plots in the funnel plots indicates absence of publication bias. Graphical assessment of the funnel plot suggests absence of publication bias (online supplemental file 7). Objective assessment of publication bias using the Harbord test also indicated there is no evidence of publication bias (p value=0.2088) (online supplemental file 8).” Eze, Lawani and Acharya 2021

• 34

  See GiveWell, Characteristics of SMS for vaccination studies

• 35

  For “received rate”, we rely on rates of messages that are recalled as being received by targeted caregivers, regardless of whether they’re opened. We think there’s some risk that participants who are asked to retroactively report whether they received an SMS reminder may over (or under) report the actual received rates.
  See GiveWell, Characteristics of SMS for vaccination studies for the rates we identified

• 36

  Some studies we are aware of (but have not reviewed in depth) that have informed this prior include:

  • Unger et al 2018, which found that two-way SMS messages had a larger effect on exclusive breastfeeding rates than 1-way (although there was no clear difference with facility delivery rates or contraceptive use).
    • From Table 2: Probability of eBF to 24 weeks - Control 0.41, 1-way SMS 0.49, 2-way SMS 0.62; Probability of contraceptive use 24 weeks postpartum - Control 0.77, 1-way SMS 0.83, 2-way SMS 0.83
  • Wald, Butt and Bestwick 2015 found that “two-way text messaging is associated with substantially improved medication adherence compared with 1-way text messaging." An author of one of the included trials flagged that their study had been mischaracterised, and there were errors in the study (see here). However, our understanding based on the author's note is that the broad results still hold after those errors are resolved.
  • Finitsis et al 2014 looked at an overlapping set of studies specifically related to ART adherence and found that “Sensitivity analyses of intervention characteristics suggested that studies had larger effects when interventions…supported bidirectional communication”
  • The following two meta-analyses look specifically at one-way and two-way communication respectively:
    • Linde et al 2019 found “One-way SMS improved appointment attendance, OR:2·03; 95% CI:1·40–2·95 (12 trials, 6448 participants), but not medicine adherence, RR:1·10; 95% CI:0·98–1·23 (nine trials, 4213 participants). Subgroup analyses showed that one-way SMS had the highest impact on childhood immunization attendance, OR:3·69; 95% CI:1·67–8·13 (three trials, 1943 participants). There was no clear evidence of one-way SMS improving facility delivery, knowledge level (reproductive/antenatal health, hypertension), diabetes- and hypertension management.”
    • Ødegård et al 2022 found that “Two-way text messages did not improve appointment attendance, RR: 1.03; 95% CI: 0.95–1.12, I2 = 53% (5 trials, 4374 participants) but improved medicine adherence compared to standard care, RR: 1.14, 95% CI: 1.07–1.21, I2 = 8% (6 trials, 2783 participants).”
  • Kinuthia et al 2021 found no effect of either 1-way or 2-way SMS reminders on ART adherence (or most outcomes of interest) in pregnant and postpartum women living with HIV.

• 37

  Across the included studies, caregivers were generally recruited close to the time of birth or first immunization visit. See column “Timing of Recruitment” here.

• 38

  See GiveWell, Characteristics of SMS for vaccination studies for examples of message text.

• 39

  The studies included in a meta-analysis by Eze, Lawani and Acharya 2021 differed significantly by how many reminders were sent, and when. For example:

  • Bangure et al 2015 reported on a study where the intervention group received messages 7 days, 3 days and 1 day before the immunization appointment: “The first message was sent 7 days before the due date for the immunization as a reminder. The second message was sent 3 days before the due date. The last message was sent a day before immunization appointment date. The messages were sent for the 6th, 10th and 14 weeks appointments.”
  • Dissieka et al 2019 reported on a study where the intervention group received a message 2 days before the immunization appointment: “Mothers were randomized to receive a voice or text reminder message two days prior to each scheduled visit and two additional reminders for missed doses (n=798; intervention group), or no phone reminder messages (n=798; control group).”

• 40

  Eze, Lawani and Acharya 2021: “In subgroup analysis, we found substantial differences in intervention effect size by country’s income status and the study’s quality and a marginal difference by study design. We found no difference in effects by study setting, outcome measure, number of SMS reminders sent or the timing of the SMS reminder—table 2.” We have not vetted or replicated this analysis.

• 41

  New Incentives operates a conditional cash transfer program in Northern Nigeria which aims to increase childhood immunization rates. We first recommended a grant to New Incentives in 2014. See our page on New Incentives for more details.

• 42

  For more details, see this section of our report on New Incentives' program.

• 43

  This list includes diseases that are targeted by the Pentavalent vaccine (Penta), Pneumococcal conjugate vaccine (PCV), Measles vaccine (MCV) and the Rotavirus vaccine. The routine immunization schedule also includes vaccines which reduce the risk of tuberculosis, polio, yellow fever, hepatitis B, and meningitis A.

• 44

  The routine immunization schedule also includes the BCG, Hepatitis B, oral and live polio, yellow fever, and meningitis A vaccines. See the Nigeria routine immunization schedule here.

• 45
  • Pentavalent vaccine (Penta) protects against diphtheria, tetanus, pertussis (whooping cough), hepatitis B, and Haemophilus influenzae type b
  • Pneumococcal conjugate vaccine (PCV) protects against pneumococcal disease
  • Measles vaccine (MCV) protects against measles

  See this section of our cost-effectiveness analysis for our estimates of vaccine efficacy.

• 46

  See this cell of our cost-effectiveness analysis.

• 47

  See this section of our cost-effectiveness analysis.

• 48

  We assume efficacy is higher for two doses, relative to one dose. For New Incentives, we use a value of 0.85 for one dose. Sudfeld et al. 2010 estimates the efficacy of two doses of measles vaccines would be around 0.98, but this is based on observational evidence: “The effect of a second dose of measles vaccine on measles disease or mortality compared with no vaccination has not been evaluated on individual children in prospective randomized studies as this type of trial would be unethical. Therefore, the best estimate of the effect of a two-dose measles vaccine schedule on measles mortality must be extrapolated from serology data, studies looking at effectiveness of two dose vs one-dose measles vaccination, and observational studies. Caution should be taken when using serology data to estimate the impact on mortality. A recent WHO review of serology studies determined that a median 97% [inter-quartile range (IQR) 87–100%] of children that failed to seroconvert to first dose measles vaccine developed immunity after a second dose. If 85% efficacy is assumed for single dose measles vaccine, these serology results would correlate to an efficacy of 99.6% for two dose measles vaccine with a range of 98.1–100% based on the IQR of the review. In addition, the effectiveness of two doses of measles vaccine will vary by setting based on the age of vaccination. Epidemiologic studies comparing the effectiveness of early two dose vaccination vs single dose have found varying results in developing country settings; a study in Niger found two doses (first dose at 6–8 months and second at 9 months) was 23% less effective than single dose whereas studies in India (first dose at 9–12 months and second at 15–18 months) and Guinea Bissau (first dose at 6–8 months and second at 9–12 months) determined two doses of vaccine were respectively 83 and 90% more effective than one dose of measles vaccine. In order to produce a conservative estimate of the efficacy of two dose measles vaccine per LiST rules, we felt an input of 98% based on the lower quartile of the WHO two dose measles vaccine serology review was reasonable.”
  We discount this and use a value of 0.90 for two doses of measles. We have not thoroughly vetted this estimate, but it’s unlikely to change our bottom line.

• 49

  See the WHO’s immunization portal.

• 50

  See this row of our cost-effectiveness analysis.

• 51

  We use additional adjustments as per our New Incentives CEA except where adjustments are specifically related to the cash transfer component of New Incentives’ program. In these cases (‘Inflation’ and ‘Investment of income increases’) we set the adjustment to be 0%. We also set the 'Herd immunity' adjustment to 0% because we estimate that SMS reminders programs lead to smaller increases in vaccination coverage rates compared to New Incentives' program. We guess this would make SMS reminders programs less likely to result in community-wide benefits in a given context.

• 52

  Eze, Lawani and Acharya 2021 found that SMS reminders improved timeliness of vaccination: “Meta-analysis of 12 included studies involving 25 257 infants showed that SMS reminders significantly improved timely receipt of childhood vaccines (RR=1.21; 95% CI: 1.12 to 1.30; I2=87.3%).”
  We think that it is plausible that a reduction in delayed vaccination would have additional benefits:

  • “Delay in vaccination increases the susceptibility window for developing VPDs at individual level and reduces herd immunity at population level. Evidence from previous studies have demonstrated that delayed vaccination may increase the risk of Pertussis, Measles and Haemophilus influenzae B infections up to 6 folds and lead to outbreaks. In case of certain vaccines with fixed upper age limit (e.g. rotavirus) delayed vaccination leads to reduced coverage for the vaccine.” Choudhary et al. 2019
  • “children whose parents delay vaccinations may be at increased risk of not receiving all recommended vaccine doses and may be more vulnerable to VPDs [vaccine-preventable diseases].” Smith et al. 2010

• 53

  In Nigeria, this includes polio, yellow fever, rubella, mumps, and varicella meningitis A.

• 54

  This includes removing the adjustments for all supplemental adjustments related to cash transfers: inflation and investment of income increases. In addition, we set the herd immunity adjustment to zero given the relatively lower increase in vaccination coverage rates we expect would be caused by SMS reminders.

• 55

  For an example of the cost-effectiveness of our recommendations, see this page. As of August 2024, we estimate that the cost-effectiveness of opportunities we direct funding to is 10 times as cost-effective as unconditional cash transfers.

• 56

  We estimate that direct SMS costs would be around $0.10, and operational and other costs would be around $2.82. $0.10 + $2.82 = $2.92.

  • SMS costs are based on an assumption that texts cost $0.01 per SMS, and would be sent twice ahead of each scheduled vaccine visit. We guess two texts would be sent as a best guess, but an actual program could include fewer or more SMS reminders than this.
    • According to the WHO vaccine schedule tables for Nigeria, vaccines are recommended in five months during infancy or young childhood (excluding vaccines such as BCG that are recommended at birth, and vaccines that are currently being piloted). ($0.01 per SMS x 10 messages = $0.10)
      • Nigeria's vaccine schedule: month 1 includes DTP/Hib/HepB, PCV, OPV, RV1 and IPV; month 2 includes DTP/Hib/HepB, PCV, OPV and RV1; month 3 includes DTP/Hib/HepB, PCV, OPV and IPV; month 9 includes measles, MENA and YF; month 15 includes measles.
  • We have proxied operational costs based on data from a digital immunization registry operated by IRD Global (IRD) which GiveWell has supported in Pakistan. The registry records immunization data and other information in addition to sending SMS reminders. We use our estimate of IRD's total non-incentive costs per infant eligible to receive SMS reminders. See here.
    • The costs are based on a budget provided by IRD, and the majority are for technical support and implementation of a digital immunization registry, followed by procurement, operations, and other indirect costs. Phone and laptop costs are also included.

• 57

  This is a guess, based on the rough numbers of SMS reminders we have seen in the trials we have reviewed. See GiveWell, Characteristics of SMS for vaccination studies for a summary of this parameter across the trials.

• 58

  Unlike other digital health programs we have modeled (e.g., our report on breastfeeding promotion programs), we do not include a multiplier for the possibility that SMS reminders lead to higher vaccination rates for subsequent children. This is because we guess that vaccination appointment reminders are unlikely to be habit-forming, and the benefit comes largely from reminding caregivers about specific dates.

• 59

  We estimate vaccine coverage by starting with 2021 Multiple Indicator Cluster Survey (MICS) data and making the following adjustments to reach our final aggregate vaccine coverage estimates:
  1. Rotavirus vaccine: We do not have data on the rotavirus vaccine because it was not introduced in Nigeria’s immunization schedule until August 2022. We assume rotavirus coverage is the same average of Penta and PCV coverage, as both vaccines are administered at the same time.
  2. Adjustment for self-report bias: Our estimates rely on caregivers reporting on their child's vaccination status. We believe caregivers may be likely to overreport whether their children are vaccinated because of social desirability bias (the tendency of survey participants to overreport “good” behaviors), which we do not believe MICS factors into its data. We estimate a ~1% rate of overreporting based on BCG scarring from our previous work on New Incentives.

  3. Weighting vaccine coverage by contribution to reducing mortality: To convert coverage estimates for individual vaccine doses to a single aggregate estimate of baseline coverage, we weight the proportion of children who receive each vaccine dose by that dose’s contribution to reducing under-five mortality. This approach is based on our previous work on New Incentives.

  Our calculations are in this sheet.

• 60

  This includes adjustments from baseline IHME data described above and an additional .75 deaths averted from other causes per vaccine-preventable death. (see here)

• 61

  Our estimates for probability of death from vaccine-preventable diseases in older age groups includes subjective adjustments for expected improvements in health more generally over time.

  • We estimate that the probability of death from vaccine-preventable diseases for unvaccinated individuals aged 5-14 years old is roughly 0.2% (see here).
  • We estimate that the probability of death from vaccine-preventable diseases for unvaccinated individuals aged 15-49 years old is roughly 0.3% (see here).
  • We estimate that the probability of death from vaccine-preventable diseases for unvaccinated individuals aged 50-74 years old is roughly 1.1% (see here).

• 62

  We expect childhood vaccines to be less effective at reducing later deaths due to declines in efficacy over time.

  • For 5-14 year olds, we estimate that vaccination in childhood reduces the probability of death from vaccine-preventable diseases by 49% (see here).
  • For 15-49 year olds, we estimate that vaccination in childhood reduces the probability of death from vaccine-preventable diseases by 29% (see here).
  • For 50-74 year olds, we estimate that vaccination in childhood reduces the probability of death from vaccine-preventable diseases by 14% (see here).

• 63

  For example, for the pentavalent and PCV 3-dose vaccines, we estimate that 90% of the total efficacy is reached by the second dose, which is typically administered at 10 weeks at Visit 3.

• 64

  We based our estimate of costs for a digital delivery program on our cost estimates for IRD's mobile conditional cash transfers for immunizations program.

• 65

  See this column of our review of SMS for vaccination studies. Three trials report that the full cost per additional child immunized or reminded is less than $1 USD, while we estimated a cost of $4.35 per person enrolled in a fourth trial (Kawakatsu et al. 2020). We are uncertain what’s driving the differences in costs across programs, although the sophistication of the system used to send and capture SMS reminders may be a key factor. In particular, while both Eze et al. 2015 and Kawakatsu et al. 2020 appear to include start-up costs for setting up SMS delivery systems in Nigeria, Kawakatsu et al. 2020 used a system developed for a wider range of maternal and child health services, which may have been more costly than a system solely intended to send and track reminders for immunizations.
  Kawakatsu et al. 2020, p. 6601: “The System was developed with the aim of increasing utilizations of a continuum of maternal and child health services, i.e. child vaccinations (incl. vitamin A supple-mentations), ANC and FP. [...] In middle of January 2019, a pre-test of the System was con-
  ducted at two PHCs to examine and confirm the communication and functionality between the mobile application, API portal, and administrative website portal. Some necessary adjustments were made in programming for the System during the pre-test period, prior to its formal piloting.”

• 66

  “However, the more pertinent analysis, given the design of this experiment, is at the child level. Here, we don’t see any significant effect of being reminded (Table 15), and only a weak effect in the isolated sample of 33 per cent treated villages (with regard to receiving the measles shot in particular, which entails a special reminder [Table 16]). Furthermore, we don’t see any significant spillover effects.” Banerjee et al. "Evaluating the impact of interventions to improve full immunisation rates in Haryana, India," 2020, p. 44.

• 67

  “The policy that has the largest impact (information hubs, SMS reminders, incentives that increase with each immunization) increases the number of immunizations by 44 % relative to the status quo. The most cost-effective policy (information hubs, SMS reminders, no incentives) increases the number of immunizations per dollar by 9.1%.” Banerjee et al. 2021

• 68

  “WhatsApp started as an alternative to SMS. Our product now supports sending and receiving a variety of media: text, photos, videos, documents, and location, as well as voice calls. Some of your most personal moments are shared with WhatsApp, which is why we built end-to-end encryption into our app. Behind every product decision is our desire to let people communicate anywhere in the world without barriers.” https://www.whatsapp.com/about

• 69

  “Telegram is a messaging app with a focus on speed and security, it’s super-fast, simple and free. You can use Telegram on all your devices at the same time — your messages sync seamlessly across any number of your phones, tablets or computers. Telegram has over 700 million monthly active users and is one of the 10 most downloaded apps in the world.” https://telegram.org/faq

• 70

  A rough estimation of theoretical total funding suggests ~$108 million could be devoted annually to SMS reminders for vaccination.

  • Total annual children born in countries with less than 70% coverage of the third dose of DTP-containing vaccine: ~37 million. (using 2021 estimates of number of children and 2022 estimates of vaccination coverage)

  Calculation: 37 million children covered x $2.92 = ~$108 million