This is an interim intervention report. We have spent limited time to form an initial view of this program and, at this point, our views are preliminary. See our most recent report on SMS reminders for vaccination.
Summary
- What is the program? Short Message Service (SMS) messages sent to caregivers' cell phones are intended to remind caregivers of young children's upcoming vaccination dates.
- What is its evidence of effectiveness? There is some limited evidence of the effectiveness of SMS reminders for vaccination within developing countries. Due to the diversity of settings and characteristics of available studies, we cannot draw broad conclusions about the effectiveness of SMS reminders for vaccination in developing countries.
- How cost-effective is it? Our best guess is that sending SMS reminders for vaccination is within the range of cost-effectiveness of programs we would consider directing funding to, but there are several key factors about which we have limited information.
- Does it have room for more funding? Charity Science Health, a GiveWell Incubation Grant recipient, is developing an SMS vaccination reminder program. [Update: we are no longer supporting Charity Science Health as of 2020. See this page.] We have not performed an analysis of the funding landscape for such programs, of other implementing organizations, or of other potential funders.
- Bottom line: This program appears promising, and we may continue our investigation.
Published: June 2017
Table of Contents
What is the problem?
Child vaccination prevents illness, disability, and death from vaccine-preventable diseases including diphtheria, hepatitis B, measles, pertussis (whooping cough), pneumonia, polio, rotavirus diarrhea, rubella, mumps, and tetanus. However, vaccination rates in many settings fall short of full coverage.1 Among children who are vaccinated, some may be vaccinated later than recommended (delayed vaccination), leaving the child susceptible to disease for a longer period of time.
What is the program?
Short Message Service (SMS) reminders for vaccination are a demand-creation and behavior-change intervention aiming to increase uptake and timeliness of vaccination. Caregivers of young children who enroll in an SMS reminder program receive SMS messages soon before the child is due for a vaccination visit. Caregivers may enroll with a personal cellular phone or a shared cellular phone. The timing and number of messages may vary by program. The program may also include other health messaging delivered via SMS.
Does the program have strong evidence of effectiveness?
We are aware of six randomized controlled trials (RCTs) of SMS reminders for vaccination, five in sub-Saharan Africa and one in Guatemala. See Characteristics of SMS for vaccination studies, which draws on Charity Science Health's notes on evidence for SMS reminders for vaccination.
There is some limited evidence that SMS messages can notably increase vaccination rates in some sub-Saharan populations that already access vaccination services or clinic delivery2 and have moderate baseline vaccination rates. Of two studies with high baseline vaccination rates, both in Kenya, one found a notable effect on vaccination rates and the other found no effect. The four RCTs that found an effect of SMS reminders on vaccination rates (8.7 - 20 percentage point increase in third visit at age 14-16 weeks) measured effects only on early infant vaccination, and not on measles vaccination at nine months.
The pilot RCT in Guatemala did not find statistically significant effects of SMS reminders on vaccination rates.
The six RCTs vary in population characteristics and program implementation. We are hesitant to draw strong conclusions about the effectiveness of SMS reminders in other contexts based on the evidence available. Below, we summarize the findings of these six RCTs. For more information, see Characteristics of SMS for vaccination studies.
- Gibson et al. 2017, the "M-SIMU" RCT in Kenya in a rural population with high baseline vaccination rates, found no effect of SMS reminders on vaccination rates.3 The study population was recruited by field-based community interviewers prior to the first scheduled vaccination visit at six weeks,4 and vaccination rates in the control group were 98-100% for the first three pentavalent visits, and 84% for the nine-month measles visit.5 This study had fairly high (23%) loss to follow-up.6
- Haji et al. 2016, an RCT in Kenya in a population with fairly high baseline vaccination rates, found that SMS reminders significantly increased vaccination rates, from 91% to 98% (7 percentage points, 95% CI 0.3–14 percentage points) for the second vaccination visit at 10 weeks, and from 83% to 96% (13 percentage points, 95% CI 5.6–21.26 percentage points) for the third vaccination visit at 14 weeks.7 The study did not extend to SMS reminders for measles vaccination at nine months. Study participants were recruited when presenting at the six-week vaccination visit, a population more likely to be receptive to vaccination than the population at large.8
- Bangure et al. 2015, an RCT in urban Zimbabwe found that SMS reminders significantly increased vaccination rates, from 82% to 97% (15 points) for the visit at six weeks, 80% to 96% (16 points) for the visit at 10 weeks, and from 75% to 95% (20 points) for the visit at 14 weeks.9 The study did not extend to SMS reminders for measles vaccination at nine months. Study participants were recruited prior to the six-week vaccination visit, "soon after delivery" or at first-week clinic visits.10
- Eze et al. 2015, an RCT in urban Nigeria, found that SMS reminders increased vaccination rates for the third vaccination visit, from 60.3% to 69.0% (8.7 points); however, this study did not report the statistical significance of the effect on vaccination rates.11 Study participants were recruited when presenting at BCG visit or first DTP vaccination visit, a population more likely to be receptive to vaccination than the population at large.12 The study did not extend to SMS reminders for measles vaccination at nine months.
- Schlumberger et al. 2015, an RCT in urban Burkina Faso in a population with relatively low vaccination rates, found that SMS reminders significantly increased vaccination rates, from 55.7% to 73.3% (17.6 points) for a visit at two months, 53.6% to 71.3% (17.7 points) for a visit at three months, and from 42.3% to 60.3% (18 points) for a visit at four months. All effects were significant at p<0.001.13 Infants in this context followed this vaccination schedule rather than the 6/10/14-week schedule.14 Study participants were recruited at birth at an urban health clinic, a population which may be more receptive to vaccination than the population at large.15 The study did not extend to SMS reminders for measles vaccination at nine months.
- Domek et al. 2016, an RCT in urban Guatemala, found a small non-statistically significant increase in vaccination rates for the second vaccination visit (95.0% vs. 90.1%, 4.9 points, p = .12) and the third vaccination visit (84.4% vs. 80.7%, 3.7 points, p = .69).16 Study participants were recruited when presenting at first vaccination visit, a population more likely to be receptive to vaccination than the population at large.17 The study did not extend to SMS reminders for measles vaccination at nine months.
How informative is the existing evidence for estimating the effect of Charity Science Health's program?
Charity Science Health is a GiveWell Incubation Grant recipient that is developing an SMS reminders for vaccination program in India. Our evaluation of the evidence for SMS reminders for vaccination is relevant to evaluating what magnitude of effect we expect that Charity Science Health might achieve in India. We expect that the population targeted by Charity Science Health is urban, has access to hospitals, and has relatively high vaccination rates. However, we are uncertain about vaccination rates in the Charity Science Health population. We expect that Charity Science Health will aim to recruit enrollees prior to six-week vaccination, but may also enroll participants at the six-week vaccination visit. We believe that the existing literature on the effect of SMS reminders on vaccination rates has low generalizability to the Charity Science Health context.18
Evidence of effectiveness of SMS reminders for other health or behavior interventions
We have not reviewed the body of literature on effects of SMS reminders on health behaviors other than infant vaccination. Some such studies may evaluate interventions similar enough to SMS reminders for infant vaccination (such as prenatal maternal vaccination or infant vitamin A supplementation) that we would find them somewhat relevant. Charity Science Health's review of the literature includes some non-comprehensive notes on some such studies.19
How cost-effective is the program?
Based on a cost-effectiveness model of Charity Science Health we put together in May 2017 and updated with our most recent moral weights as of February 2021, we estimate that sending SMS reminders for immunization is within the range of cost-effectiveness of the opportunities that we expect to direct marginal donations to (about 10x cash or higher, as of 2021).20 However, there are several key factors about which we need more information. Cost-effectiveness of such programs is likely to be highly dependent on context-specific factors, such as baseline vaccination rates, disease incidence and mortality, reasons for non-vaccination, and a population's responsiveness to SMS reminders.
We are highly uncertain about the magnitude of effect on vaccination rates to expect in the population targeted by Charity Science Health. We are also highly uncertain about the baseline vaccination rate in this population, and disease incidence and mortality rates in this population. We are somewhat uncertain about the costs of implementing an SMS reminders for vaccination program.
Charity Science Health has published Charity Science Health's preliminary CEA, early 2017 and Charity Science Health's preliminary CEA, early 2017, companion document. Our understanding is that Charity Science Health is continuing to update its analysis and that this published CEA no longer reflects Charity Science Health's current thinking.
There are limitations to cost-effectiveness analyses, and we believe that cost-effectiveness estimates such as these should not be taken literally, due to the significant uncertainty around them. We provide these estimates (a) for comparative purposes and (b) because working on them helps us ensure that we are thinking through as many of the relevant issues as possible.
Organizations and room for more funding
Charity Science Health is in the early stages of developing an SMS reminders for vaccination program.21 We are uncertain about Charity Science Health's room for more funding at scale. We have not searched for additional organizations that implement SMS reminders for vaccination.
Focus of further investigation
Below, we list areas we expect to research as we continue this investigation:
- GiveWell Incubation Grants previously recommended gifts to the Abdul Latif Jameel Poverty Action Lab (J-PAL) at the Massachusetts Institute of Technology (MIT) and to Interactive Research and Development (IRD) to support two randomized controlled trials (RCTs) in India and Pakistan, which include testing the effect of SMS reminders on vaccination rates. We expect early results of these studies in fall of 2017.
- Charity Science Health may consider conducting an RCT of its program, although we would be surprised if such a study began before late 2018.22
Our process
Charity Science Health previously identified studies of SMS reminders for vaccination (Charity Science Health's notes on evidence for SMS reminders for vaccination). Since this work, one additional study (Gibson et al. 2017) has been published. We reviewed these studies. We searched for additional studies or meta-analyses and identified none.
Sources
- 1
- "Immunization prevents illness, disability and death from vaccine-preventable diseases including cervical cancer, diphtheria, hepatitis B, measles, mumps, pertussis (whooping cough), pneumonia, polio, rotavirus diarrhoea, rubella and tetanus." WHO Immunization coverage fact sheet.
- For country-specific estimates of vaccination coverage, see WHO/UNICEF Estimates of National Immunization Coverage.
- 2
We note that the four studies that found effects were in urban or mixed settings and recruited participants with prior vaccination or health system contact; the one study that did not find an effect took place in a high-vaccination rural setting and recruited participants from the community. See "Recruitment" column on the “SMS for immunization” tab of Characteristics of SMS for vaccination studies.
- 3
Gibson et al. 2017, Table 2, p. e433 shows number of fully immunized children at 12 months (the study's primary outcome) in the control group (82%) and the SMS-only arm of the trial (86%), p=0.29. Table 2 also presents the proportion of children who received each vaccine in each arm of the trial; there are no differences between the SMS arm and the control group significant at p=0.05. The lowest p-value is for receipt of the third dose of pentavalent vaccine (PENTA3), 98% in the control group compared to 97% in the SMS group, p=0.20.
- 4
"The M-SIMU trial recruited HDSS village reporters to identify eligible caregivers and their infants. Village reporters used mobile phones to send birth notification text messages to the RapidSMS server. Birth notifications were relayed to field-based community interviewers who then screened caregivers of newborns for eligiblity into the study. Inclusion criteria for participation included being a caregiver of an infant aged 0–34 days and being a current resident of one of the randomised study villages. Exclusion criteria included being a caregiver planning to migrate from the study area in the next 6 months, if the infant received vaccination other than Bacillus Calmette–Guérin (BCG) or polio birth dose, or if the caregiver was not willing to vaccinate the child at an M-SIMU staffed clinic. Caregivers of infants aged 35 days and older were excluded because of the close temporal proximity to the first pentavalent visit scheduled at 6 weeks (ie, 42 days). M-SIMU health facility recorders were stationed at 24 clinics whose catchment area overlapped with all study villages. Mobile phone ownership was not an inclusion or exclusion criterion. Participants only needed to have access to a mobile phone, whereby access was defined by the caregiver." Gibson et al. 2017, pp. e429-e430.
- 5
Gibson et al. 2017, Table 2, p. e433.
- 6
"Between Oct 14, 2013, and Oct 17, 2014, we enrolled 2018 caregivers and their infants (infant-caregiver pairs) from 152 villages (figure 1) to meet sample size requirements. The number of infant-caregiver pairs enrolled in control, SMS only, SMS plus 75 KES, and SMS plus 200 KES groups were 489 infant-caregiver pairs (38 villages), 476 infant-caregiver pairs (38 villages), 562 infant-caregiver pairs (38 villages), and 491 infant- caregiver pairs (38 villages), respectively. After accounting for loss to follow-up, which included outmigration and death, and excluding caregivers who verbally reported their child’s immunisation history, the primary analytic sample contained 1600 infant–caregiver pairs who completed a 12-month follow-up survey and provided MCH booklet information for immunisation history; 360 (74%) infant–caregiver pairs from 37 villages (control arm), 388 (82%) pairs from 38 villages (SMS only), 446 (79%) pairs from 38 villages (SMS plus 75 KES), and 406 (83%) pairs from 38 villages (SMS plus 200 KES). Approximately 1600 (94%) of 1707 children who were alive and had not outmigrated had an MCH booklet present at 12-month follow-up." Gibson et al. 2017, pp. e432-e433. The sample of infant-caregiver pairs at enrollment of 489 + 476 = 965 in the control + SMS only arms compared to the sample at follow-up of 360 + 388 = 748 yields a loss to follow-up of (965 - 748)/945 = 23%. The authors note: "There were some differences in sociodemographic characteristics between the analytic sample and those who were excluded from the analysis. Excluded caregivers were more likely to be less than 25 years old, have one child younger than 5 years old in the household, and reside in Gem. In this area, there is a cultural practice of caregivers migrating to their home village to give birth, which might be more common in younger mothers who are having their first child." Gibson et al. 2017, p. e437.
- 7
"In the SMS intervention group, at 10 weeks of age 365 (98%) children had received their second dose of pentavalent vaccine. At 14 weeks of age, 359 (96%) had received their third dose of pentavalent vaccine (p = 0.4)... In the control group at 10 weeks of age, 340 (91 %) children had received their second dose of pentavalent vaccine. At 14 weeks of age, 309 (83 %) of children had received their third dose of pentavalent vaccine (p ≤ 0.001) (Fig. 2)... At 10 weeks, the risk difference for those who received SMS reminders and the control group was seven percent (95% CI: 0.3–14). At 14 weeks, the risk difference for those who received SMS reminders and the control group was 13% (95% CI: 5.6–21.26)." Haji et al. 2016, p. 5.
- 8
"Children <12 months of age who were brought to the selected vaccinating health facilities in the three districts for their first dose of pentavalent vaccine were recruited on a first come basis until the strategy-level target sample sizes was reached." Haji et al. 2016, p. 2.
- 9
"At 6 weeks OPV1, Penta1 and PCV1the immunization coverage in the intervention group was 97% and in the non intervention group was 82%. (p < 0.001). At 10 weeks the immunization coverage for OPV2, Penta2 and PCV2 was 96% in the intervention group and 80% in the non intervention (p < 0.001). Immunization coverage at 14 weeks for OPV3, Penta3 and PCV3 was 95% in the intervention group and 75% in the non intervention group (p < 0.001)." Bangure et al. 2015, p. 3.
- 10
"A Randomized Control Trial was conducted at Kadoma City Clinics in Mashonaland West province of Zimbabwe namely Rimuka, Waverley Chemukute and Kadoma General Hospital. Woman or caregiver was recruited into the study soon after delivery or during the 3rd and 7th day visits after delivery of the baby." Bangure et al. 2015, p. 2.
- 11
See Eze et al. 2015, Figure 3, p. 7, which presents results for the control group and intervention group. Results are discussed on pp. 6-7, including statistical analysis of the effect of SMS reminders on timeliness, but not including statistical significance for the effect on vaccination rate: "Logistic regression showed that receiving SMS reminders resulted in an earlier receipt of DPT3 in the intervention group, OR 1.47 (95% CI: 1.1 – 2.0) (Table 2); and 8.7% increase in coverage (Figure 3)."
- 12
"A total of 1001 consenting caregivers who brought their wards for routine immunization were selected using multistage sampling method from 8 health facilities in Egor local government area of Edo State. The facilities included 1 tertiary hospital, 2 primary health centres and 5 privately owned health facilities – all of which provided routine immunization services. They were selected proportionate to the relative distribution of these various types of health facilities in the study area. Selection criteria was bringing child for routine immunisation for the first or second schedules of RI, and consenting to participating in the study. The second schedule was added because some babies get their first shots at their bedside and as such, at their first visits to the RI centre they are given the second schedule of vaccines." Eze et al. 2015, pp. 4-5. As the "first shots" or "first schedule of RI" is referred to as shots that infants can receive at "bedside" (birth), this refers to Bacille Calmette-Guerin (BCG) vaccination against tuberculosis, and other vaccinations given at birth, and not to the first of three vaccinations on the pentavalent schedule. In settings with low delivery volume at facilities, infants may receive BCG and other birth vaccinations at birth, or may return at a later date for this first vaccination visit. Hence, the "second schedule" refers to the six-week visit and the first dose of pentavalent vaccine.
- 13
See Schlumberger et al. 2015, Table 2, p. 4.
- 14
The vaccination schedule is presented in Schlumberger et al. 2015, Table 1, p. 3.
- 15
- "Les nouveau-nés du CSPS de Colma 1 étaient inclus dans l’étude lors du premier contact PEV à leur naissance (Tableau 1), à partir de février 2014." Schlumberger et al. 2015, p. 2. "PEV" refers to the Expanded Programme on Immunisation (EPI), and CSPS refers to the health clinic: "L’impact dans le Programme élargi de vaccination (PEV) d’un registre informatisé de vaccination (RIV), avec l’envoi de SMS aux parents avant les sessions vaccinales, n’a jamais été estimé en Afrique sub-saharienne. Un Centre de santé et de promotion sociale (CSPS) urbain prodiguant le PEV a été tiré au sort (Colma 1, district de Do) à Bobo- Dioulasso (Burkina Faso), puis a été doté d’un RIV." p. 1.
- GiveWell translation: "The newborns of CSPS of Colma 1 were included in the study as of the first contact EPI at their birth (Table 1), starting in February 2014." ... [Abstract:] "The impact in the Expanded Programme on Immunisation (EPI) of a computerized vaccination register (RIV), with the sending of SMS to parents before vaccination visits, has never been evaluated in sub-Saharan Africa. An urban Center of Health and Social Promotion (CSPS) providing EPI was randomly selected (Colma 1, district of Do) in Bobo- Dioulasso (Burkina Faso), then was provided with a computerized vaccination register." We have translated close to the text rather than into native-sounding English in order for readers to more easily check our translation against the original.
- 16
See Domek et al. 2016, Table 3, p. 2441.
- 17
"Parents of infants between the ages of 8 and 14 weeks presenting for the first dose of the 3-dose infant primary immunization series were eligible if they owned a mobile phone with SMS text messaging capability. At least one parent had to be literate and able to use SMS technology." Domek et al. 2016, p. 2438.
- 18
In addition to the studies above, we are aware of Uddin et al. 2016, a quasi-experimental study of the effect of SMS reminders for vaccination in hard-to-reach rural populations and urban street-dweller populations of Bangladesh, which observed large effects of SMS reminders on vaccination rates. Due to geographic similarities, some may consider this study informative for the estimation of the expected effect of Charity Science Health's program in India. However, because Charity Science Health targets urban women who deliver in hospitals, rather than especially hard-to-reach populations, we believe that this study has limited applicability to the Charity Science Health context. Additionally, we believe there are limitations to the design and implementation of this study. More:
- Uddin et al. 2016 report vaccination coverage according to (a) EPI card and/or maternal recall (Table 2) and also according to (b) EPI card only (Table 4). According to EPI card and/or maternal recall, baseline rate of full vaccination (BCG, PENTA3, and measles) in the hard-to-reach rural intervention group was 58.9%, in the urban street-dweller intervention group was 40.7%, in the rural control group was 65.9%, and in the urban control group was 44.5%. (Table 2, p. 280) According to EPI card only, baseline rate of full vaccination was 72% (n=106) in the intervention group and 70% (n=116) in the control group. (Table 4, p. 281)
- This study observed a notable effect of SMS reminders on vaccination rates, measured as a difference-in-difference effect between cross-sectional surveys in intervention and control areas before and after intervention. The study found a 29.5% difference-in-difference effect in rural areas, and a 27.1% effect in urban areas. (Table 2, p. 280)
- These effect magnitudes are partially due to unexplained decrease in vaccination rates in control areas.
- Different children were evaluated pre- and post-intervention, and the samples differed significantly in characteristics other than vaccination rate, such as maternal age, education, mobile phone ownership, and (in urban control areas) sex of child. See Uddin et al. 2016, Table 1, p. 279.
- Additionally, the study reports the following implementation challenges: "...because the mTika intervention was implemented by the existing public health system, we were unable to rigorously standardize implementation procedures, randomize EPI centers or service providers, or use contemporaneous controls... Using maternal recall for vaccination history is not as accurate as EPI cards, but is a necessary reality in many LMICs. More than one-third of parents could not show EPI cards during our surveys either because they lost the cards or their children were never vaccinated and never received cards... mTika implementation challenges included difficulties with developing entirely new software, multiple other groups working in this space, growing capacity of health workers and field staff to use smart phones, low active mTika usage by mothers, and inability to track SMS notifications by different vaccination coverage categories. Frequent initial trainings were held to improve staff competence and capacity early in implementation. Low active engagement with the system by mothers, as evidenced by low rates of birth notifications, is a challenge also observed in other mHealth field sites." Uddin et al. 2016. p. 282.
- 19
Charity Science Health's notes on evidence for SMS reminders for vaccination, tab "Related studies".
- 20
See our cost-effectiveness analysis of Charity Science Health, “Charity Science Health” sheet, “Charity Science Health vs GiveDirectly (unconditional cash transfers)” row.
- 21
- 22
"Our broad steps are: ... 2. Run an RCT on the program to ensure previous studies are externally valid." Charity Science Health webpage: Our Plan