Suvita — SMS Reminders and Ambassadors for Immunization (April 2023)

Note: This page summarizes the rationale behind a GiveWell grant to Suvita. Suvita staff reviewed this page prior to publication.

Published: January 2024

Summary

What we think this grant will do

Suvita is a small nonprofit that operates in India and implements two programs aimed at increasing the take-up of vaccinations:

• SMS reminders: Suvita collects telephone numbers for caregivers or other contacts of recently born infants, then sends SMS reminders to those contacts. These reminders are for routine childhood vaccinations, breastfeeding, and healthcare during pregnancy.
• Immunization ambassadors: Immunization ambassadors are volunteers recruited by Suvita who agree to receive regular SMS messages and phone calls about vaccination, and to spread information about vaccination in their communities.

With this grant, Suvita plans to:

• ~$2 million: Scale up its SMS reminders program in Maharashtra, from seven districts to all 36 districts.
• ~$0.85 million: Scale up its immunization ambassadors program, from approximately 5,000 ambassadors to approximately 35,000 ambassadors across Bihar and Maharashtra.
• ~$0.45 million: Maintain the current scale of its SMS reminders program in Bihar, covering two districts.

We think this grant will lead to increased uptake of routine childhood immunizations in the districts where Suvita plans to operate, which will, in turn, lead to reductions in deaths from vaccine-preventable diseases. We also incorporate some additional benefits from SMS reminders for breastfeeding and healthcare during pregnancy in our cost-effectiveness analysis.

Why we made this grant

• We estimate Suvita’s program will be highly cost-effective during the grant period.

  Intuitively, we think this program is likely cost-effective because:

  • Suvita’s program leads to a slight to modest increase in vaccination rates at a low cost. Our best guess is that SMS reminders increase vaccination rates by around 2 percentage points at a current cost of $0.40 per child reached in Maharashtra and $1.71 per child reached in Bihar, and the immunization ambassadors program increases vaccination rates by around 7 percentage points and costs $1 to $2 per child reached.
  • Vaccine-preventable diseases are a major cause of death among children. We estimate that unvaccinated children have a 1.2% risk of mortality before age 5 in Bihar and a 0.4% risk in Maharashtra.
  • Vaccines are effective at preventing disease. We estimate that vaccination reduces the risk of targeted diseases by 90% overall.
  • We think increased vaccination also leads to additional benefits, including averting deaths of individuals older than 5 and increasing income later in life.

  Our best guess is the overall cost-effectiveness of the grant is 23 times as cost-effective as GiveDirectly’s unconditional cash transfers, which is above our funding bar (10x cash). A sketch of our cost-effectiveness analysis for the SMS program in Maharashtra, the largest program in the grant, is below.

You can see our full cost-effectiveness analysis, including estimates for the other programs and the grant overall, here.

• We think this grant may open up additional cost-effective funding opportunities with Suvita in the future. By providing Suvita an opportunity to scale up its SMS reminders program in Maharashtra and, to a lesser extent, its ambassadors program in Bihar and Maharashtra, this grant may enable Suvita to absorb more funding for these programs in the future. In addition, we expect that if the Bihar program (~9x cash during the grant period) were to scale up at a later date, costs per person would be lower (due to spreading out fixed costs among more program participants). As a result, we believe funding the SMS program in Bihar preserves the option to fund a more cost-effective scale-up at a later time. Our cost-effectiveness estimates do not account for this option value from giving Suvita the opportunity to expand and absorb more funding in the future.
• We have a favorable qualitative impression of Suvita as an organization. We have a strong overall qualitative impression of Suvita's leadership, in terms of transparency, communication, impact-orientation, and willingness to respond to feedback.

Main reservations

• We have not yet completed up-to-date intervention-level research on the impact of SMS reminders or community ambassadors on immunization rates. Our estimate for the effect of SMS reminders on immunization rates (~2 percentage points) is similar to the value we use in our cost-effectiveness analysis (CEA) for IRD’s Zindagi Mehfooz program, which has similar baseline levels of vaccination. Our estimate for the effect of immunization ambassadors on immunization rates (~7 percentage points) is based on the rough estimate provided by Suvita, based on Suvita's analysis of the results of a randomized controlled trial (RCT). We plan to complete additional intervention-level research before considering additional grants to Suvita.
• Suvita doesn’t have much of a track record yet, and it’s a relatively young organization with a small current budget. As a result, we’re not sure about its ability to scale with additional funding, and may need to consider additional non-funding bottlenecks it might face on the path to scale. This grant would result in a ~3x increase in Suvita's annual budget—we have some uncertainty about Suvita's ability to absorb this level of funding. Suvita has told us that it does not believe that this grant would require unmanageable increases in Suvita's organizational or management capacity, since the funding will primarily be directed to fees for SMS messages, which are relatively easy to scale.
• There could be other organizations better positioned to scale this type of program or who are likely to scale similar programs in the future. We haven’t investigated what other organizations are working in this space in India.

The organization

Suvita aims to increase uptake of childhood vaccinations in India.1 It was founded through Charity Entrepreneurship's Incubation Program in 2019 and merged with another organization focused on vaccinations, Charity Science Health, in 2020.2 It operates primarily in two states in India, Bihar and Maharashtra.3

This is the first GiveWell grant to Suvita.4

The problem

Immunizing children against vaccine-preventable diseases, including diphtheria, measles, pertussis (whooping cough), Haemophilus influenzae type b (HiB) infections, pneumococcal disease, rotavirus diarrhea, rubella, mumps, and tetanus, prevents child death, illness, and disability.5 However, in many places, a large number of children have missed some or all of their recommended vaccinations.6

Based on data from India’s National Family Health Survey 2019-21 (NFHS-5), Suvita estimates that 25% to 30% of infants in Bihar and Maharashtra are not fully immunized each year (around 850,000 infants in Bihar and 500,000 in Maharashtra).7

The intervention

Suvita implements two programs aimed at increasing uptake of childhood immunizations: an SMS reminders program and an "immunization ambassadors" program.8 Suvita also sends SMS reminders about breastfeeding and healthcare during pregnancy.9

For the SMS reminders program in Maharashtra, Suvita partners with the state government to use the state's Reproductive Child Health database to enroll telephone numbers of women with registered pregnancies and caregivers of infants born in public health facilities.10 Suvita’s SMS messages in Maharashtra during pregnancy include reminders for TD (tetanus and diphtheria) vaccination, iron and folic acid supplementation, antenatal care (ANC) visits, and encouragement to give birth in a public facility. After birth, caregivers receive breastfeeding reminders and reminders to bring children to a health facility for immunizations at 6 weeks, 10 weeks, 14 weeks, 9 months, and 15 months, in line with India's national immunization schedule.11

Bihar’s Reproductive Child Health database is not currently maintained at the same coverage and cadence, so Suvita collects phone numbers under a partnership with the state government by digitizing paper records of births from public health facilities.12 Its messages in Bihar currently include breastfeeding reminders and reminders to bring children to a health facility for immunizations at 6 weeks, 10 weeks, 14 weeks, 9 months, and 15 months.13

Suvita's immunization ambassadors are volunteers who agree to share information about immunization in their communities.14 Suvita recruits ambassadors and follows up with the ambassadors regularly through SMS and voice messages.15

Do SMS reminders and ambassadors work to increase childhood immunization rates?

Suvita's combination of SMS reminders and immunization ambassadors programs was informed by interventions tested in a randomized controlled trial (RCT) in Haryana, India, discussed in Banerjee et al. 2021 (working paper).16

Our best guess is that SMS reminders and immunization ambassadors increase rates of childhood immunization, but we have only conducted limited research so far.

We last updated our intervention report on SMS reminders for vaccination in 2017. Since then, we have completed (but not yet published) a preliminary review of updated evidence for SMS reminders, including reviewing a recent meta-analysis of 18 studies (including 13 RCTs) on SMS reminders for immunization in low and middle income countries.17 The majority of studies in this meta-analysis show that SMS reminders increase the immunization rate among intervention groups; however, there is substantial heterogeneity between study sample population characteristics, settings, and intervention designs.18

Banerjee et al. 2021 is the only direct evidence we have seen on immunization ambassador programs, and we have not yet reviewed the paper in depth.

We plan to conduct more research on these interventions before considering additional grants to Suvita (see below).

The grant

This three-year grant will fund Suvita to maintain its scale for SMS reminders in two districts in Bihar and scale up its SMS reminders in Maharashtra from seven districts to all 36 districts in the state.19 Suvita estimates that the program in Maharashtra will reach an estimated 4.8 million children for the full schedule of reminders. An additional 1.3 million older children will be back-enrolled to receive any remaining reminders in the schedule based on their age.20

This grant will also fund Suvita to recruit 30,000 additional immunization ambassadors over three years. As a result, the scale of its immunization ambassadors program will increase roughly sevenfold, from approximately 5,000 ambassadors before the grant to approximately 35,000 ambassadors at the end of the grant, split across Bihar and Maharashtra.21

The grant also includes funding for Suvita to iterate and improve upon its ambassadors program and expand its monitoring and evaluation activities.22

Budget for grant activities

The total budget for this grant is $3,305,079. This breaks down as follows: ~$2 million for the SMS program in Maharashtra, ~$0.45 million for the SMS program in Bihar, and ~$0.85 million for the immunization ambassadors program.23

The case for the grant

We are recommending this grant because:

• We think this grant is a cost-effective use of funding. Our cost-effectiveness analysis for Suvita is based on the model we built for New Incentives,24 a GiveWell top charity in Nigeria that provides conditional cash transfers to caregivers who bring their children to clinics for routine immunizations. On average, we think Suvita’s programs funded through this grant are 23 times as cost-effective as unconditional cash transfers (more).25
• Suvita has high potential for future growth. Suvita has identified four additional Indian states that it believes would be promising to work in.26 Suvita estimates that it could absorb tens of millions of dollars per year on its programs in these six states.27 Suvita also believes there may be potential to expand its programs outside of India, which would further increase the room for more funding.28
• Scale-up of Suvita’s SMS program in Maharashtra and its ambassadors programs in Bihar and Maharashtra will allow us to learn more about its impact. Suvita plans to accompany its expansion with a monitoring and evaluation strategy (more), which we expect will allow us to learn more about the impact of its program as well as its ability to expand into additional states in India.
• Suvita believes there is a current “window of opportunity” for scaling up this work. Suvita has told us that now may be a particularly good time to work on scaling up SMS reminders in Maharashtra. In recent conversations with Suvita, officials from Maharashtra's state government have expressed interest in expanding the SMS reminders program to the entire state.29
• We have overall strong qualitative impressions of Suvita's leadership, in terms of transparency, communication, and focus on impact.
  • In conversations with us, Suvita has framed its decisions, such as which states in India to focus on for future expansion, in terms of maximizing impact. It has also been very transparent about potential risks and difficulties it could face in the future.
  • Our impression is that Suvita provided clear and reasonable answers to our questions throughout our investigation of this grant.

Cost-effectiveness

• Cost-effectiveness sketch: We think the grant overall is ~23x cash.30 This section provides a sketch of our cost-effectiveness estimate for the SMS program in Maharashtra, which we expect to receive the majority of funding from our grant. Our cost-effectiveness estimate for the SMS program in Maharashtra is ~26x cash.31
  • We’re recommending a grant of ~$3 million to Suvita, ~$2 million of which we expect to be spent on the SMS program in Maharashtra.32
  • We estimate this will lead to an additional ~5 million infants receiving SMS reminders.33
  • We estimate a mortality rate of 0.4% before the age of 5 among unvaccinated infants in Maharashtra.34
  • We think Suvita’s SMS programs increase vaccination rates by ~2 percentage points.35
  • We think vaccination lowers the probability of vaccine-preventable disease among those infants by ~90% before age 5.36
  • As a result, we think this program averts ~460 under-5 deaths.37
  • With our moral weights, this implies a cost-effectiveness of ~8x cash.38
  • While we think that 37% of expected benefits of these programs will be to avert deaths of children under the age of 5, we think there are several additional benefits as well: reduction in deaths beyond the age of 5 due to continued vaccine protection (32% of total benefits) and development effects leading to increases in income later in life (31% of total benefits).39
  • These estimates include an adjustment for the benefits of breastfeeding and other unmodeled benefits (+57%).40 We also include an adjustment for leveraging costs covered by governments and other funders (-15%), and other downside adjustments (-7%).41
  • Combining these, we get a bottom line cost-effectiveness estimate of ~26x cash.42
• Intuitive comparison: Suvita’s SMS program in Maharashtra is similarly cost-effective to New Incentives because:43
  • New Incentives is more expensive per infant in the area where the program is operating (~$25 for New Incentives vs. ~$0.40 for Suvita) because it provides cash incentives, whereas Suvita only provides SMS reminders.44
  • But this is counterbalanced by New Incentives’ impact on vaccination being higher (~20 percentage points for New Incentives vs. ~2 percentage points for Suvita),45 since New Incentives has a more intensive intervention and works in areas with lower baseline vaccination rates and higher mortality among under-5s who are unvaccinated (~4% in the parts of Nigeria where the program is most cost-effective vs. ~0.4% for Maharashtra).46

• Key judgment calls: Factors that we’re uncertain about and that would change the bottom line a lot (+/- 20%):
  • We have not conducted up-to-date intervention-level research on the impact of SMS reminders or community ambassadors on immunization rates. Our estimate for the impact of SMS reminders (~2 percentage points) is similar to the value we use in our CEA for IRD Global’s Zindagi Mehfooz (ZM) program in Pakistan,47 which operates in areas with roughly similar baseline levels of vaccination.48 Our estimate for the effect of immunization ambassadors (a ~7 percentage point increase in vaccination rate) is based on the rough guess provided by Suvita, based on the results of the J-PAL Haryana immunization RCT.49
  • The Suvita CEA is based on our New Incentives CEA. We may prioritize additional work on the New Incentives CEA in the near future, including researching the durability of vaccine protection. In our Suvita and New Incentives CEAs, we assume vaccine protection lasts beyond age five, but we have not thoroughly interrogated this assumption.50
  • By providing Suvita an opportunity to scale up its SMS reminders program in Maharashtra and, to a lesser extent, its ambassadors program in Bihar and Maharashtra, this grant may enable Suvita to absorb more funding for these programs in the future. Our cost-effectiveness estimates do not account for the value of giving Suvita the opportunity to expand and absorb more funding in the future.

Risks and reservations

Our main reservations about this grant are:

• We have not yet completed up-to-date intervention-level research on the impact of SMS reminders or community ambassadors on immunization rates. We currently base our estimate for the effect of Suvita's SMS reminders program on immunization rates (an increase of 2 percentage points) on our cost-effectiveness analysis for SMS reminders for immunization through IRD Global’s ZM program.51 The effect of immunization ambassadors (an increase of 7 percentage points) in our cost-effectiveness analysis is a rough estimate provided by Suvita, based on Suvita's analysis of the results of a J-PAL randomized controlled trial (RCT) in Haryana, India discussed in Banerjee et al. 2019.52
  We plan to complete and publish additional intervention-level research on both SMS reminders for immunization (as well as reminders for breastfeeding and healthcare during pregnancy) and immunization ambassadors before considering additional grants to Suvita.53 Our best guess is that we will continue to believe that this grant to Suvita is above our cost-effectiveness bar (10x as cost-effective as cash transfers) following this research, but that there is some chance that our intervention-level research will lead to substantial decreases in our estimate of Suvita's cost-effectiveness (see below).
• Suvita is a relatively young organization with a small current budget. We expect this grant to result in a roughly 3x increase in Suvita's annual budget.54 We have some uncertainty about Suvita's ability to absorb this level of funding. Suvita has told us that it does not believe that this grant would require unmanageable increases in Suvita's organizational or management capacity, since the funding will primarily be directed to Suvita's SMS reminders program in Maharashtra, which Suvita expects to be relatively straightforward to scale up.55
• There could be other organizations better positioned to scale this type of program. We haven’t done a comprehensive review of all the organizations working in this space in India.

Plans for follow up

We plan to meet quarterly with Suvita to discuss monitoring, scale-up progress, and general updates.

Additionally, we plan to conduct additional research on the effectiveness of SMS reminders and ambassadors in 2023. We expect this research will contribute to our decisions about whether to renew this grant or support Suvita to expand into new areas.

Monitoring, evaluation, and learning

The main outcomes of this program that we are interested in are the effects of Suvita’s SMS reminders and ambassadors program on immunization rates. The key questions Suvita plans to investigate through its monitoring, evaluation, and learning under this grant are:56

• Does Suvita accurately collect phone numbers from eligible program participants?
• Do Suvita's SMS reminders reach participants' phones?
• Are participants able to read and recall information from Suvita's SMS messages?
• Do participants say that Suvita's messages made a difference in whether vaccinations are administered on time?
• Is Suvita able to acquire accurate contact information for nominees to be immunization ambassadors?
• Is Suvita able to contact and recruit volunteer immunization ambassadors by phone?
• Do immunization ambassadors read and engage with Suvita's SMS messages?
• Do ambassadors take action to encourage vaccination in their communities?
• Do ambassadors' actions increase the number of caregivers who bring their infants in for vaccinations?

Internal forecasts

For this grant, we are recording the following forecasts:

Our process

• We adapted the New Incentives cost-effectiveness analysis to build a cost-effectiveness analysis for Suvita’s program.57
• We have had six calls with Suvita since May 2022 to discuss its program, plans for scale-up, our cost-effectiveness analysis, and monitoring and evaluation.
• We asked Suvita to send us a proposal and detailed budget for a grant.
• We had conversations with three outside experts familiar with Suvita’s work.

Sources

• 1

  "​​Suvita implements two rigorously tested solutions which have been found to increase uptake of vaccinations in India:
  1) Building a network of carefully selected volunteer immunisation ambassadors, who share information with new parents in their community about local vaccine clinics
  2) Sending personalised SMS reminders to caregivers informing them when their child is due for a vaccination" Suvita, "Our work"

• 2
  • "Varsha and Fiona founded Suvita in 2019, after meeting at the Charity Entrepreneurship incubation programme, which supports entrepreneurs to launch impact-focused and evidence-driven nonprofits. After evaluating the results of the landmark study by Nobel Prize laureates Banerjee et al. (2019), we decided to focus on replicating and scaling up the researchers' proven approach for increasing uptake of vaccines: immunisation ambassadors.

  In 2020, Suvita merged with Charity Science Health (CSH) - another nonprofit working to increase uptake of immunisation in India. CSH had been working since 2017 to send SMS reminders about immunisations to caregivers." Suvita, "Our story so far"

  • See more at Charity Entrepreneurship, “Incubation Program.”

• 3

  Suvita, 3-year grant proposal for GiveWell:

  • "Our SMS reminders programme involves sending SMS messages to the parents of new children, along the sequence of their vaccination schedule. We currently deliver the programme in two states, Bihar and Maharashtra." p. 1.
  • "Our ambassadors programme involves conducting community-level surveys to identify community ‘influencers’, then asking these influencers to act as volunteer immunisation ambassadors in their communities. Ambassadors then engage parents and community stakeholders to support and encourage childhood immunisation in whichever way they see as most valuable for their own community. After they are recruited, we send ambassadors fortnightly motivational SMS messages.

    So far we have delivered the programme in several blocks of Saran district, Bihar and Satara district, Maharashtra - our current capacity is to cover ~1% of a state per year.” p. 2.

• 4

  GiveWell previously recommended funding for Charity Science Health.

• 5
  • See GiveWell, New Incentives (Conditional Cash Transfers to Increase Infant Vaccination) for our analysis of the impact of vaccines' effects on disease for children under 5.
  • See World Health Organization, "Vaccination schedule for India" for a list of recommended childhood vaccinations and targeted diseases for India.

• 6

  UNICEF, "Immunization," 2022:

  • "Despite ongoing efforts towards recovery, a staggering 20.5 million children remained either unvaccinated or under-vaccinated in 2022. Furthermore, the number of children who did not receive any vaccines, often referred to as zero-dose children, reached 14.3 million – a significant increase of 1.4 million compared to 2019."
  • "The percentage of children receiving DTP3 is often used as a measure to gauge how well countries are providing routine immunization services and has a global target of 90 per cent as set by the Immunization Agenda 2030."
  • "Global coverage of the third dose of diphtheria-tetanus-pertussis (DTP3) fell from 86 per cent in 2019 to 81 per cent in 2021 – its lowest level since 2008. The latest WHO/UNICEF estimates of national immunization coverage (WUENIC) also show that 112 countries experienced stagnant or declining DTP3 coverage since 2019 with 62 of those countries declining by at least 5 percentage points. As a result, 25 million children were un or under-vaccinated in 2021 where more than 60 per cent live in just 10 countries (India, Nigeria, Indonesia, Ethiopia, Philippines, Democratic Republic of the Congo, Brazil, Pakistan, Angola, and Myanmar) and 18 million did not receive any vaccines (zero-dose children), an increase of 5 million from 2019." UNICEF, "Immunization," 2022

• 7

  See the “Full immunisation gap (total)” column, “Bihar” and “Maharashtra” rows of Suvita, state selection shortlist.

  GiveWell separately estimated immunization coverage in Bihar and Maharashtra and arrived at a similar figure (~80% effective coverage rate, or a coverage gap of ~20%), taking vaccine efficacy and probability of death from diseases prevented by each vaccine into account. See the “Baseline vaccination coverage” sheet, “Effective coverage rate” row of our Supplementary calculations for Suvita CEA.

• 8

  "​​Suvita implements two rigorously tested solutions which have been found to increase uptake of vaccinations in India:
  1) Building a network of carefully selected volunteer immunisation ambassadors, who share information with new parents in their community about local vaccine clinics
  2) Sending personalised SMS reminders to caregivers informing them when their child is due for a vaccination" Suvita, "Our work"

• 9

  Included messages: Bihar: “Welcome message; breastfeeding reminders; reminders for vaccination visits at: 6 weeks (Penta-1 etc.), 10 weeks (Penta-2 etc.), 14 weeks (Penta-3 etc.), 9 months (MR-1 etc.), 15 months (MR-2 etc.)
  Included messages: Maharashtra: Pregnancy sequence: “Welcome message; TD vaccination reminders; message about government-provided pregnancy supplements / medications (iron, folic acid, calcium, deworming); reminders for ANC visits at 12, 28 and 36 weeks; information about high-risk pregnancy symptoms; encouragement to give birth in a public facility (with information about baby’s birth-dose vaccines); breastfeeding reminder” Suvita, 3-year grant proposal for GiveWell, p. 1.

• 10
  • “Maharashtra: We enrol children and pregnant women from the government’s existing (digital) Reproductive Child Health database, which primarily contains children born in public health facilities” Suvita, 3-year grant proposal for GiveWell, p. 1.
  • Fiona Conlon, Suvita Co-founder, conversation with GiveWell, August 17, 2022 (unpublished).

• 11
  • “Pregnancy sequence: Welcome message; TD vaccination reminders; message about government-provided pregnancy supplements / medications (iron, folic acid, calcium, deworming); reminders for ANC visits at 12, 28 and 36 weeks; information about high-risk pregnancy symptoms; encouragement to give birth in a public facility (with information about baby’s birth-dose vaccines); breastfeeding reminder.
    Vaccination sequence: breastfeeding reminder; reminders for vaccination visits at: 6 weeks (Penta-1 etc.), 10 weeks (Penta-2 etc.), 14 weeks (Penta-3 etc.), 9 months (MR-1 etc.), 15 months (MR-2 etc.)” Suvita, 3-year grant proposal for GiveWell, p. 1.
  • This timeframe includes the vaccinations: oral poliovirus vaccine 1-3 (OPV); Penta1-3; inactivated poliovirus vaccine (IPV); measles, mump, and rubella (MMR); diphtheria, tetanus, and pertussis (DTP); and Japanese encephalitis (JE). Government of India, Ministry of Health & Family Welfare, National Immunization Schedule (NIS) for Infants, Children and Pregnant Women.

• 12
  • Fiona Conlon, Suvita Co-founder, conversation with GiveWell, August 17, 2022 (unpublished).
  • Bihar: Enrollment strategy: "We digitise relevant data to enrol children from labour wards in public health facilities" Suvita, 3-year grant proposal for GiveWell, p. 1.

• 13

  Bihar: Included messages: “Welcome message; breastfeeding reminders; reminders for vaccination visits at: 6 weeks (Penta-1 etc.), 10 weeks (Penta-2 etc.), 14 weeks (Penta-3 etc.), 9 months (MR-1 etc.), 15 months (MR-2 etc.)” Suvita, 3-year grant proposal for GiveWell, p. 1.

• 14

  "Immunization ambassadors are local influencers nominated by people in their community and then selected to share health information with their neighbors. The idea of systematically identifying nodes in a local social network and engaging with them to improve the uptake of health interventions is relatively new. Suvita planned to replicate the approach taken in the J-PAL Haryana study, then to modify the program to make it more effective, scalable, and cost-effective." Suvita, conversation with GiveWell, November 22, 2021, p. 3.
  "After agreeing to become immunisation ambassadors, ambassadors received a sequence of SMS messages and phone calls, with one contact point every two weeks, encouraging them to advocate for vaccinations in their community." Suvita, Using a remote model to identify and recruit volunteer immunization ambassadors, October 2020-January 2021, p. 10.

• 15

  “Within villages where Suvita already has a user base for its SMS reminders programme in Bihar state, a random sample of existing users were called by phone and asked to nominate people in their locality “who share information in a way that many people come to know about that information”. These community influencers were then called by phone and invited to become immunisation ambassadors.” Suvita, Using a remote model to identify and recruit volunteer immunization ambassadors, October 2020-January 2021, p. 1.
  "After agreeing to become immunisation ambassadors, ambassadors received a sequence of SMS messages and phone calls, with one contact point every two weeks, encouraging them to advocate for vaccinations in their community." Suvita, Using a remote model to identify and recruit volunteer immunization ambassadors, October 2020-January 2021, p. 10.

• 16

  "The Immunization Ambassador: Network-Based Seeding. The goal of the immunization ambassador intervention was to leverage the social network to spread information. In particular, the objective was to identify influential individuals who could relay to villagers both the information on the existence of the immunization camps, and, wherever relevant, the information that incentives were available. Existing evidence shows that people who have a high centrality in a network (e.g., they have many friends who themselves have many friends) are able to spread information more widely in the community (Katz and Lazarsfeld, 1955; Aral and Walker, 2012; Banerjee et al., 2013; Beaman et al., 2018; Banerjee et al., 2019). Further, members in the social network are able to easily identify individuals, whom we call information hubs, who are the best placed to diffuse information as a result of their centrality as well other personal characteristics (social mindedness, garrulousness, etc.)(Banerjee et al., 2019)." Banerjee et al. 2021 (working paper), p. 10.
  "Our programmes are supported by high-quality, recent, local evidence from Haryana, India. A large-scale randomised controlled trial, conducted by a team from MIT's Abdul Latif Jameel Poverty Action Lab (J-PAL) including two recent Nobel laureates - Esther Duflo and Abhijit Banerjee - found that a combination of immunisation ambassadors and SMS reminders significantly increased the number of children attending government-run immunisation camps by 25%." Suvita, "Our work"

• 17

  We referenced six of the studies included in this meta-analysis in our existing intervention report: Gibson et al. 2017, Haji et al. 2016, Bangure et al. 2015, Eze and Adeleye 2015, Schlumberger et al. 2015, Domek et al. 2016

• 18
  • "Results: 18 studies, 13 RCTs and 5 non-RCTs involving 32 712 infants (17 135 in intervention groups and 15 577 in control groups) from 11 LMICs met inclusion criteria. Pooled estimates showed that SMS reminders significantly improved childhood immunisation coverage (RR=1.16; 95% CI: 1.10 to 1.21; I2=90.4%). Meta-analysis of 12 included studies involving 25 257 infants showed that SMS reminders significantly improved timely receipt of childhood vaccines (RR=1.21; 95% CI: 1.12 to 1.30; I2=87.3%). Subgroup analysis showed that SMS reminders are significantly more effective in raising childhood immunisation coverage in lower middle-income and low-income countries than in upper middle-income countries (p<0.001) and sending more than two SMS reminders significantly improves timely receipt of childhood vaccines than one or two SMS reminders (p=0.040)." Eze, Lawani and Acharya 2021, p. 1.
  • “Twelve studies showed that SMS reminders significantly improved childhood immunisation coverage in children in the intervention group compared with those in the control group with usual care. Three studies showed that childhood immunisation coverage in the control group was relatively high (compared with intervention group). Two other studies evaluated the utility of SMS reminders in one arm of a three-arm study including compliance-linked monetary incentives in other arms. Both studies reported insignificant effectiveness for SMS reminders alone but statistically significant effect when coupled with monetary incentives. Finally, in another study, countrywide vaccine shortages precluded the evaluation of SMS reminders on overall childhood immunisation coverage. However, SMS reminders demonstrated statistically significant improvement in childhood immunisation timeliness.” Eze, Lawani and Acharya 2021, p. 5.
  • “Observed heterogeneity could be due to the methodological and clinical differences between the studies, although the magnitude and direction of the intervention effects mitigates this concern. Substantial heterogeneity remained when we assessed the effect by country income status, study design, study setting, number of SMS reminders sent and timing of last SMS reminder, but it dropped when we limited the analysis to non-RCTs and in participants that received one or two SMS reminders.” Eze, Lawani and Acharya 2021, p. 10.

• 19
  • “Currently in Bihar, we are delivering SMS reminders in 2 large districts (of 38 total), Patna and Saran, representing roughly 10% of the state’s population. In Maharashtra, we are delivering SMS reminders in 2 districts (of 35 total), Satara and Jalna, representing roughly 5% of the state’s population.

  We propose to target the following scale during a 3-year grant:

  • By the end of 2023, grow from our current (~5%) scale in Maharashtra to deliver SMS reminders across 100% of districts in the state
  • Continue covering ~10% of Bihar”

  Suvita, 3-year grant proposal for GiveWell, p. 1.
  Suvita informed us that it expanded its SMS program from 2 districts to an additional 5 districts in Maharashtra between the time Suvita wrote a grant proposal (Suvita, 3-year grant proposal for GiveWell) and GiveWell made the grant, and was thus covering about 12% of Maharashtra prior to this grant recommendation. It also informed us that there is a typo in the grant proposal: Maharashtra has a total of 36 districts, not 35. Suvita, comments on a draft of this grant page, October 2023 (unpublished)

• 20

  “We estimate that in total, this will mean enrolling approximately 4.8 million children for SMS vaccination reminders (plus a further 1.3 million already-born children who will be “back-enrolled” at the time of expansion in Maharashtra to receive any remaining messages of their reminder sequence - at minimum the 9-month and 15-month reminders). It will also mean enrolling approximately 4.9 million women for pregnancy reminders (plus a further 1.4 million “back-enrolled”).” Suvita, 3-year grant proposal for GiveWell, p. 2.

• 21
  • “We propose to target the following scale during a 3-year grant: Between Bihar and Maharashtra, double our total ambassadors programme capacity (from currently ~1% of a state per year) to cover ~2% of a state per year, and maintain this capacity. Concretely, we will aim to recruit 30,000 additional immunisation ambassadors over 3 years across Bihar and Maharashtra, who we estimate will reach 455,000 families to support and encourage them to access immunisation.” Suvita, 3-year grant proposal for GiveWell, p. 2.
  • Suvita informed us that the term "capacity" in its grant proposal referred to recruitment capacity or the rate of recruitment of ambassadors each year. Prior to this grant, Suvita had approximately 5,000 ambassadors and was recruiting ambassadors at a rate of roughly 5,000 a year. With this grant, Suvita expects to double its recruitment rate to roughly 10,000 ambassadors a year. Concretely, that results in recruiting 30,000 new ambassadors over the three year grant. Suvita, comments on a draft of this grant page, November 2023 (unpublished)
  • Suvita also noted during a review of this page that it has updated its estimate of the number of children reached per ambassador since the grant proposal was written. It now estimates these additional ambassadors will reach 1.2 million families rather than 455,000 families. Suvita, comments on a draft of this grant page, October 2023 (unpublished)

• 22

  “Conduct 3 experiments to iterate on our model of operational delivery of the ambassadors programme to make it more ready-for-scale.” Suvita, 3-year grant proposal for GiveWell, p. 2.
  “We will standardise and scale our monitoring systems for both the SMS reminders and ambassadors programmes, to ensure reliable delivery of both programmes as they scale.” Suvita, 3-year grant proposal for GiveWell, p. 3.

• 23

  Suvita, detailed budget, April 2023 (unpublished)

• 24

  See our New Incentives CEA.

• 25
  • We estimate a cost-effectiveness of ~26x cash for the Maharashtra SMS program, ~9x cash for the Bihar SMS program, and ~22x cash for the immunization ambassador programs. See the “Overall Summary” sheet, “Cost-effectiveness (in multiples of cash transfers)” column of our Suvita cost-effectiveness analysis.
  • Note that a) our cost-effectiveness analyses are simplified models that are highly uncertain, and b) our cost-effectiveness threshold for directing funding to particular programs changes periodically. As of mid-2023, our bar for directing funding is about 10 times as cost-effective as unconditional cash transfers. See GiveWell’s Cost-Effectiveness Analyses webpage for more information about how we use cost-effectiveness estimates in our grantmaking.

• 26

  The other states Suvita has identified are Uttar Pradesh, Rajasthan, Madhya Pradesh, and Gujarat. See Suvita, state selection shortlist.

• 27

  Suvita, comments on a draft of this grant page, October 2023 (unpublished)

• 28

  Suvita, comments on a draft of this grant page, October 2023 (unpublished)

• 29

  Fiona Conlon, Suvita co-founder, conversation with GiveWell, February 22, 2023 (unpublished).

• 30

  See the "Overall summary" sheet, “Weighted average” row of our Suvita CEA for this estimate. See "Suvita - SMS" and "Suvita - Ambassadors" sheets of the CEA for full details.

• 31

  See the "Overall summary" sheet, “Maharashtra SMS reminders” row of our Suvita CEA for this estimate.

• 32

  See the "Overall summary" sheet of our Suvita CEA for the allocation of total spending to each of Suvita’s programs.

• 33

  See the “Simple version - SMS” sheet, “cohort of children” row of our Suvita CEA for our calculation of the number of infants receiving SMS reminders in Maharashtra.

• 34

  See the “Suvita - SMS” sheet, ”Adjustment for all-cause mortality effect” section of our Suvita CEA for how this mortality rate was estimated.

• 35

  We currently base our estimate for the effect of Suvita's SMS reminders program on immunization rates (an increase of 2 percentage points) on our cost-effectiveness analysis for SMS reminders for immunization through IRD Global’s Zindagi Mehfooz (ZM) program in Pakistan.
  "Our best guess after applying these adjustments is that ZM increases vaccine coverage by 2 percentage points in high-coverage districts and 3 percentage points in low-coverage districts. (This percentage point increase is a weighted average of increases in vaccination rates across each visit, weighted by the percentage of vaccine-preventable deaths that can be averted through vaccines delivered at each visit.)" GiveWell, IRD Global (Electronic Immunization Registry and Mobile-Based Conditional Cash Transfers to Increase Vaccination)

• 36

  See the “Suvita - SMS” sheet, “Effect of vaccination on disease incidence” section of our Suvita CEA for this calculation.

• 37

  See the “Suvita - SMS” sheet, “Number of deaths of children under five averted in cohort” row of our Suvita CEA for this calculation.

• 38

  See the “Overall summary” sheet, “Initial cost-effectiveness, under-5 deaths averted only, in multiples of cash transfers” row in our Suvita CEA for the 8x cost-effectiveness estimate, and the “Moral weights and discount rate” sheet, “Value assigned to averting the death of an individual under 5 from vaccine-preventable diseases” row here for an explanation of how moral weights were calculated.

• 39

  See the “Overall summary” sheet, “Primary benefit streams (as % of total)” section of our Suvita CEA for the percentage breakdown of the program's benefits.

  See the discussion of development effects due to increases in vaccination rates in our intervention report for New Incentives' program.

• 40

  See the “Suvita - SMS” sheet, “Supplemental intervention-level adjustments” section of our Suvita CEA for a list of all supplemental adjustments.

• 41

  See the ​​“Overall summary” sheet, “Additional adjustments” section of our Suvita CEA for the additional adjustments to the cost-effectiveness estimate.

• 42

  See ​​the “Overall summary” sheet, “Final cost-effectiveness estimate, in multiples of cash transfers” row of our Suvita CEA for the bottom line cost-effectiveness of the Maharashtra SMS program.

• 43

  See our most recent (as of the publication of this page) New Incentives CEA.

• 44

  See the “Overall summary” sheet, “Suvita's cost per child enrolled” row of our Suvita CEA and the “Summary” sheet, “Cost of the program per enrolled infant, New Incentives” row of our New Incentives cost per infant immunized spreadsheet.

• 45

  See “Adjusted increase in vaccination rates due to New Incentives (in percentage points)” row of our New Incentives CEA and the “Suvita - SMS” sheet, “Adjusted increase in vaccination rates due to intervention (in percentage points)” row of our Suvita CEA.

• 46
  • See the “Baseline vaccination coverage” sheet, “2019” column of our New Incentives CEA supplemental information for baseline vaccination rates in Nigeria and the “Baseline vaccination coverage” sheet, “Maharashtra” column of our Supplementary calculations for Suvita CEA for baseline rates in Maharashtra.
  • See the “Probability of death from vaccine-preventable diseases for unvaccinated children under 5” row of our New Incentives CEA for under-5 unvaccinated mortality rates by province in Nigeria and the “Overall Summary” sheet, “Under-5 mortality rate among unvaccinated children” row of our Suvita CEA for the rate in Maharashtra.

• 47

  See the “CEA Main” sheet, “Increase in vaccination rates from ZM, excluding mCCTs (percentage points), weighted by vaccines' contributions to deaths” row of our Zindagi Mehfooz CEA.

• 48

  See the “Baseline vaccination coverage in Sindh” sheet of our Zindagi Mehfooz CEA and the “Baseline vaccination coverage” sheet of our Supplementary calculations for Suvita CEA.

• 49

  See the “Suvita - Ambassadors” sheet, “Unadjusted increase in vaccination rates due to the intervention (in percentage points)” row of our Suvita CEA.

• 50
  • "Risk of vaccine-preventable diseases also occurs at ages beyond five. Vaccines may confer immunity beyond the age of five, reducing risk of mortality as vaccinated infants age." GiveWell, New Incentives (Conditional Cash Transfers to Increase Infant Vaccination)
  • See the “Mortality reduction for individuals older than 5” sections of our New Incentives CEA here and our Suvita CEA here.

• 51

  "Our best guess after applying these adjustments is that ZM increases vaccine coverage by 2 percentage points in high-coverage districts and 3 percentage points in low-coverage districts. (This percentage point increase is a weighted average of increases in vaccination rates across each visit, weighted by the percentage of vaccine-preventable deaths that can be averted through vaccines delivered at each visit.)" GiveWell, IRD Global (Electronic Immunization Registry and Mobile-Based Conditional Cash Transfers to Increase Vaccination).
  GiveWell recommended a three-year grant of up to $25 million to IRD Global for its mobile conditional cash transfer and SMS reminders programs for immunization in Pakistan in October 2021.

• 52
  • Suvita, Answers to GiveWell questions, August 2022 (unpublished).
  • See Banerjee et al. 2020 for another resource on the same study.
  • We also note that GiveWell recommended a small grant ($100,000) to J-PAL to support the immunization RCT in Haryana in 2015.

• 53

  We note that we published an intervention report on SMS reminders for immunization in 2017, but the report is not up-to-date.

• 54

  Suvita’s expenses from April to June 2022 totaled ~$88,000. On the basis of this quarter, we estimate Suvita's annual budget prior to this grant at around $350,000 per year (~$88,000 * 4). Suvita's budget per year for this grant is around $1.1 million. $1.1 million is roughly 3x as high as $350,000 (not counting funding Suvita may receive from sources other than GiveWell). Suvita, detailed budget, April 2023 (unpublished)

• 55

  Fiona Conlon, Suvita Co-founder, conversation with GiveWell, February 22, 2023 (unpublished).

• 56

  See Suvita, MEL indicators summary for details on Suvita’s monitoring, evaluation, and learning.

• 57

  See our most recent (as of the publication of this page) New Incentives CEA.