New Incentives' Coverage Assessments: Plans as of October 2021

Addendum added: March 2023

Published: November 2021

Plans for vaccination coverage assessments

New Incentives plans to group the Local Government Areas (LGAs) to which it is expanding into cohorts. For each cohort, it plans to:

1. Conduct a baseline assessment of vaccination coverage.10 New Incentives will use the results of the baseline assessment when considering which LGAs to work in.
2. Conduct follow-up assessments of vaccination coverage in the LGAs it works in, one year after the baseline assessment and every six months after that.11

For the first cohort for which it will conduct these assessments, New Incentives plans to do the baseline assessment in September to November 2021,12 and to begin enrolling infants in the program in these LGAs in the three months following the baseline assessments.13

While the RCT collected data on children 12 to 16 months of age,14 the vaccination coverage assessments will collect data on children 6 to 12 months of age,15 in order to generate coverage indicators more quickly. Children are scheduled to receive all routine immunizations covered by the New Incentives program, except the measles vaccine, by 14 weeks of age.16 Thus, we expect that a high percentage of children who are going to receive vaccinations (other than measles) will have done so by 6 months of age. We don't expect the assessments to track measles coverage as reliably as coverage for other vaccines.17

More details about the study methodology are available here.

GiveWell's plans for interpreting and using results from the vaccination coverage assessments

In this spreadsheet, we have recorded our current guesses for the increase that New Incentives' vaccination coverage assessments will find over baseline coverage. We have set these guesses to be consistent with the coverage increases found in the New Incentives RCT, but expect coverage increases to be lower at earlier follow-up rounds because the New Incentives program won't yet be fully phased in. We may modify our guesses after seeing the baseline results if they show that the coverage assessment cohorts differ from the RCT cohort in key ways.

Across follow-up assessment rounds, we expect to place the most weight on changes in coverage for the BCG vaccine and the Penta 1 vaccine. The visits at which the BCG and Penta 1 vaccines are given provide a substantial portion of the benefits modeled in our cost-effectiveness analysis (CEA) of New Incentives' program.18 Additionally, we expect these two vaccines to have the most accurate coverage estimates because of (a) higher statistical power because more infants will have reached this point in the vaccination schedule by the time of the survey than for vaccines later in the schedule, (b) the ability to verify BCG vaccinations by examining infants' arms for BCG scars, and (c) better caregiver recall and verification.19 In addition, for infants who receive incentives for the BCG vaccine and are thus enrolled in the program, New Incentives can track retention rates through its regular operations.20

We will not have data from baseline and follow-up vaccination coverage assessments for LGAs that New Incentives already works in because it is not possible to do baseline assessments there. For LGAs without baseline assessments, we will put relatively more weight on the RCT results.

More details on how we expect to use the results from each time period of the vaccination coverage assessments to update our CEA of New Incentives' program follow.

Baseline assessments

There are three main ways we plan to use results of the baseline assessment in our CEA:21

• Cost per additional child vaccinated: New Incentives provides incentives to all vaccinated infants, including those who would have been vaccinated without the incentives (since it is not feasible to distinguish who would and would not have received vaccines without the incentives). We plan to use baseline data from New Incentives' vaccine coverage assessments to inform our estimate of the proportion of infants in surveyed cohorts who would have received vaccinations without the program. We use this to calculate the cost per additional infant incentivized to be vaccinated.
• Expected impact of incentives on vaccination rates based on baseline coverage level: If baseline coverage in the entry cohorts is substantially lower or higher than the control group coverage in the RCT endline survey, we may adjust our expectations about the increase in coverage we expect to see in the follow-up assessments. Baseline coverage in the new areas that is higher than the control group coverage in the RCT endline survey suggests a smaller pool of caregivers of unvaccinated children to influence with cash incentives, and thus a smaller likely effect of the intervention.22
• Expected impact of incentives on vaccination rates based on reasons for not vaccinating: If the proportion of caregivers reporting that they have strongly held beliefs against vaccinating is higher in new cohorts than in the population surveyed during the RCT, this suggests that there will be a smaller pool of caregivers of unvaccinated children who will respond to cash incentives.23

Coverage at 12-month follow-up

We expect to use the results of the first follow-up assessment to make preliminary within-cohort comparisons of baseline coverage to coverage at the time of the follow-up. At the first follow-up assessment, we expect to see an increase in coverage of the BCG, Penta 1, and measles vaccines compared to the baseline. We do not expect the increases to be statistically significant at the first follow-up.24 We expect that the increase will be roughly one-third lower than the increase observed in the RCT because most of the children surveyed at this juncture will have been born before New Incentives was fully operational in these areas.25

Coverage at 18-month follow-up

We expect to use the results of the second follow-up assessment to make within-cohort comparisons of coverage at the time of that assessment to baseline coverage. At the second follow-up survey, we expect to see increases since baseline in coverage of the BCG, Penta 1, and measles vaccines that are 10% to 15% lower than those observed in the RCT.26 We expect to see increases in coverage for the BCG and Penta 1 vaccines over baseline that are statistically significant.27 We expect the increase to be lower than in the RCT because the survey cohort will not yet have been fully exposed to the New Incentives program.28

Coverage at 24-month follow-up

We expect to use the results of the third follow-up assessment to make within-cohort comparisons of coverage at the time of those surveys to baseline coverage. At the third follow-up, we expect coverage increases roughly equivalent to the coverage increases observed in the RCT.29

Additional follow-ups

With increasing time between baseline and follow-up, the baseline vaccination coverage rate will become a worse proxy for the counterfactual coverage rate (i.e., the level of vaccination coverage that would be the case if New Incentives' program did not operate in that area). We plan to primarily use additional follow-ups to track whether coverage remains at a similar or higher level than at 24 months. Falling coverage rates would be a negative update on New Incentives' ongoing impact.

Predicting off-trend results

As of this writing, we are in discussions with New Incentives about how it will record unusual circumstances that it expects to impact vaccination coverage rates, before conducting each round of follow-up surveys. For cases where there are good reasons to believe a result will not be representative, the goal is to allow us to decide, independently of seeing the results, to put more weight on the RCT results or on the coverage assessment results from another cohort, and less weight on what we believe to be an unrepresentative result for this cohort. At this stage, the plan for recording these predictions is under development and may include:

• Using routinely collected data on vaccine supply disruptions, security risks, staff availability (missed immunization days or missed clinics), vaccination campaigns in the area, severe weather events, and rapid changes in population due to migration to inform predictions.
• Asking field managers and other senior members of the New Incentives operations team to identify other circumstances that could have a major impact on vaccination coverage in the relevant period of time.
• New Incentives' senior staff combining this information into a prediction, for each follow-up round, of what coverage rate the survey will find and to what degree that coverage rate is driven by unusual circumstances.30

Limitations to the assessments

No control group

The vaccination coverage assessments do not include a control group—that is, a comparison group that is not receiving New Incentives' program. Changes in vaccination rates over time may be caused by other factors, and we may incorrectly ascribe them to New Incentives. As a result, we expect to continue placing significant weight on the results of the RCT, which did include a randomly assigned control group.

It may be possible to use a difference-in-differences study design for future cohorts to partially address concerns about the lack of a control group. To do so, additional planned cohorts could be surveyed around the same time as the first entry cohort, then changes in coverage over time compared between areas where New Incentives began offering the program earlier and those that entered the program later. Such a research design would entail additional expenses and logistics; we are unsure whether the benefits of such an approach would be worth the expense. We may pursue this design in the future after learning from the initial round of vaccination coverage assessments.

COVID-19

The COVID-19 pandemic may impact immunization coverage in the areas where New Incentives is expanding and thus affect the baseline coverage estimates. If, following baseline data collection, people's behavior changes as the effects of the pandemic ease, we could falsely attribute such changes to the New Incentives program.

Self-reporting bias

The vaccination coverage assessments will rely on caregivers' reports of vaccines. They may misremember which vaccinations their infant received or report what they believe the interviewer wants to hear. We discuss this potential source of bias in our review of New Incentives' program. As in the case of the RCT, we plan to compare reported vaccination coverage to BCG scar rates to estimate the size of the bias. (Receiving a BCG immunization commonly leaves a scar on a child's arm.)31

Spillover effects

Since the RCT was completed, the New Incentives program has scaled up substantially in the three states where it operates.32 It is possible that baseline coverage results could include some children who already received incentives from New Incentives in adjacent LGAs.

Potential staff biases

The coverage monitoring surveys will be conducted by New Incentives staff.33 It is possible that New Incentives staff will be motivated either consciously or unconsciously to generate results that overestimate the impact of the program.

IDinsight, the organization that conducted the New Incentives RCT, is helping to design the study and is creating training materials for New Incentives staff,34 which increases our confidence in the survey's quality. In addition, New Incentives has implemented quality-control procedures,35 including the following:

• Verifying survey locations using GPS
• Back-checking by revisiting a portion of households to check accuracy of data collection
• Having supervisors conduct spot checks of enumerator survey technique in the field
• Auditing audio recordings of the surveys
• Remote checks of self-reported vaccination against photos of Child Health Cards, when the cards are available36

Sources

Document	Source
GiveWell, "New Incentives (Conditional Cash Transfers to Increase Infant Vaccination)"	Source
GiveWell, "New Incentives"	Source
GiveWell, Estimated coverage time trends for New Incentives rapid assessments, 2021	Source
GiveWell, New Incentives - impact of Penta visits on program benefits, 2021	Source
GiveWell, New Incentives CEA supplemental information, 2020	Source
GiveWell, New Incentives monitoring and evaluation 2017-2020 and plans for future evaluation	Source
GiveWell, Studies of incentives for immunization, 2020	Source
IDinsight, Coverage Monitoring Analysis Plan, 2021	Source
IDinsight, Impact Evaluation of New Incentives, Final Report	Source
IDinsight, Insights from New Incentives' Coverage Monitoring Data, December 2022	Source
IDinsight, New Incentives Coverage Monitoring Protocol, 2021	Source
World Health Organization, BCG vaccines: WHO position paper – February 2018	Source (archive)

• 1

  These sixteen cohorts reflect all of the baseline coverage results New Incentives has shared with us through February 27, 2023. Twelve of these sixteen cohorts are included in IDinsight, Insights from New Incentives' Coverage Monitoring Data, December 2022.

• 2
  • IDinsight, Insights from New Incentives' Coverage Monitoring Data, December 2022, table 3. Enumeration area statistics, p. 8.
  • "Cohorts 3 and 4 had the highest percentage of enumeration areas replaced along with having the highest number of chain replacements … These cohorts also had the most significant differences in coverage between discarded and replacement surveys." IDinsight, Insights from New Incentives' Coverage Monitoring Data, December 2022, p. 8.

• 3 "In conclusion, we believe that the higher percentage of replacements in cohorts 3 and 4, as well as the fact that replacement enumeration areas were, on average, less populated than the original cohort resulted in lower coverage after replacement." IDinsight, Insights from New Incentives' Coverage Monitoring Data, December 2022, p. 12.

• 4

  "New Incentives could use the coverage estimate that includes the discarded surveys as an upper bound and the coverage estimate that includes the replacement surveys as a lower bound to calculate a range of possible impacts in these two cohorts." IDinsight, Insights from New Incentives' Coverage Monitoring Data, December 2022, p. 14.

• 5
  • "[W]e found that the replacement surveys in cohort 4 resulted in a meaningfully different estimated coverage across multiple vaccine categories, particularly for BCG and any vaccines which had differences of more than 10 percentage points." IDinsight, Insights from New Incentives' Coverage Monitoring Data, December 2022, p. 7.
  • See BCG and Penta 1 vaccine coverage estimates for Cohort 4 in IDinsight, Insights from New Incentives' Coverage Monitoring Data, December 2022, Table 2, p. 6.

• 6

  We've forecasted expected effect sizes of an 11.5 percentage point increase in vaccination for the first follow-up to a 17 percentage point increase for the third follow-up.

• 7

  See BCG and Penta 1 vaccine coverage estimates for Cohort 3 in IDinsight, Insights from New Incentives' Coverage Monitoring Data, December 2022, Table 2, p. 6.

• 8

  "While there were differences in coverage between these groups, most of these differences were less than 5 percentage points in absolute magnitude. However, we found that the replacement surveys in cohort 4 resulted in a meaningfully different estimated coverage across multiple vaccine categories, particularly for BCG and any vaccines which had differences of more than 10 percentage points. The replacement surveys in cohort 3 resulted in meaningfully different estimated coverage of BCG and measles vaccines, but these differences were less than 10 percentage points. Therefore, we can conclude that with the exception of cohorts 3 and 4, the replacement surveys did not meaningfully affect the baseline coverage estimates." IDinsight, Insights from New Incentives' Coverage Monitoring Data, December 2022, p. 7.

• 9
  • IDinsight, funded by a GiveWell Incubation Grant, conducted an RCT to measure the impact of New Incentives' program on vaccination rates in three states in North West Nigeria. For information about the RCT, see here. See IDinsight, Impact Evaluation of New Incentives, Final Report for the results of this RCT.
  • The plan for coverage monitoring can be found here.

• 10

  "New Incentives will group local government areas (LGAs) it expands to within a given state at a given point in time into ‘expansion groups’. New Incentives will then collect coverage data in these expansion groups once before the start of operations to establish baseline coverage rates." IDinsight, Coverage Monitoring Analysis Plan, 2021, Pg. 1.

• 11
  • "The first follow-up will occur approximately 12 months after baseline." IDinsight, Coverage Monitoring Analysis Plan, 2021, Pg. 1.
  • "Once operating in all LGAs in an expansion group (usually within six months of completion of baseline), New Incentives will obtain follow-up RI coverage estimates in these LGAs in six months intervals." IDinsight, Coverage Monitoring Analysis Plan, 2021, Pg. 1.

• 12

  New Incentives, call with GiveWell, September 7, 2021 (unpublished).

• 13

  New Incentives, call with GiveWell, September 7, 2021 (unpublished).

• 14

  "Only children aged 12 to 16 months were included in the study." IDinsight, Impact Evaluation of New Incentives, Final Report, Pg. 17.

• 15

  "The unit of analysis is the individual 6-12-month-old infant." IDinsight, Coverage Monitoring Analysis Plan, 2021, Pg. 4.

• 16

  "BCG vaccine . . . 1 [dose], At birth; Penta vaccine . . . 3 [doses], At 6 weeks, 10 weeks, and 14 weeks; PCV vaccine . . . 3 [doses], Same as for Penta vaccine; Measles vaccine . . . 1 [dose], 9 months." IDinsight, Impact Evaluation of New Incentives, Final Report, Table 1, Pg. 11.

• 17

  Many in the cohort of children 6 to 12 months old will not yet be eligible for the measles vaccine, which is scheduled to be given to children who are 9 months old, thus reducing the sample size (and statistical power) of the coverage assessment for the measles vaccine. "Measles vaccine . . . 1 [dose], 9 months." IDinsight, Impact Evaluation of New Incentives, Final Report, Table 1, Pg. 11.

• 18
  • Together, the BCG vaccine visit at birth and the Penta 1 vaccine visit at 6 weeks (at which children receive the DTP and HiB vaccines as part of Penta 1, as well as the PCV vaccine, and which is expected to include the rotavirus vaccine in 2022) account for roughly half of program benefits. See this spreadsheet for calculations. We learned about the expected timeline for introduction of the rotavirus vaccine in the states where New Incentives works from an unpublished conversation with New Incentives on April 13, 2021.
  • An analysis of the program benefits that are provided by the vaccines given at the visit for the Penta 1 vaccine can be found here.

• 19

  In many cases, receipt of the BCG vaccine can be confirmed by the presence of a physical scar on the child's arm. Additionally, we hypothesize that caregiver recall may be better for this vaccine because it is the first one offered in the schedule.

  While Penta 1 does not have as clear a verification marker as BCG, analysis of the rate of agreement between self-reports and alternative measures of vaccination status from the RCT suggested that caregiver recall was substantially better for Penta 1 than Penta 2 and 3. The analysis also indicated that the gap in recall accuracy between caregivers exposed to New Incentives and those in the control group was substantially larger for Penta 2 and 3, so comparisons to baseline coverage may be less accurate than for Penta 1 due to improved recall after exposure to New Incentives.

  • "Sensitivity was relatively high for BCG vaccine, Penta 1 vaccine, and Measles 1 vaccine and was higher in treatment than in control. Sensitivity was highest for BCG vaccine and lowest for any Penta vaccine (Table 20). BCG vaccine sensitivity was higher when using scars than when using either cards or CIRs. Overall, these relationships among the estimated sensitivities for the different vaccines were of reasonable magnitude and in line with expectations. The BCG vaccine commonly leaves a scar and is also amongst the first vaccines children receive. This should make it relatively easy for caregivers to remember. We would also have expected that it is easier for a caregiver to remember that their child received Penta 1 vaccination but that it can be difficult to recall the exact number of Penta vaccinations received." IDinsight, Impact Evaluation of New Incentives, Final Report, Pgs. 67-68.
  • Estimated total sensitivity was 93% for Penta 1, 83% for Penta 2, and 72% for Penta 3. IDinsight, Impact Evaluation of New Incentives, Final Report, Table 20, Estimate using Cards + CIR, Pg. 68.
  • Estimated treatment group sensitivity was 8 percentage points higher than the control group for Penta 1, 16 percentage points higher for Penta 2, and 14 percentage points higher for Penta 3. IDinsight, Impact Evaluation of New Incentives, Final Report, Table 20, Estimate using Cards + CIR, Pg. 68.

  We expect to place less weight on coverage changes for other incentivized vaccines, such as PCV, offered at the same time as the Penta vaccines because we expect that their coverage rates will be similar to rates for Penta and that it is difficult for caregivers to distinguish these vaccines from each other.

  • "PCV, while not a primary outcome, is part of NI-ABAE’s incentives conditions and is generally given at the same visits as Penta vaccine. It is not as prominent in the evaluation in large part because we expected it to be both highly correlated with and easily confused with Penta vaccine, making self-reported coverage data for it somewhat more difficult to interpret." IDinsight, Impact Evaluation of New Incentives, Final Report, footnote 27, Pg. 13.

  Additionally, measles vaccines are administered beginning when children are 9 months old. Many in the cohort of children 6 to 12 months old will not yet be eligible for the vaccine, thus reducing the sample size (and statistical power) of the coverage assessment for the measles vaccine. Full vaccination coverage rates include the measles vaccine and thus are also based on a smaller sample.

  • "Measles vaccine . . . 1 [dose], 9 months." IDinsight, Impact Evaluation of New Incentives, Final Report, Table 1, Pg. 11.
  • "The minimum detectable effect sizes (MDES) to establish trends in measles and full immunization coverage are 12, 14, and 16 percentage points in Katsina, Jigawa, and Zamfara, respectively. These MDES indicate the possibility that this design is underpowered for New Incentives to detect meaningful differences over time for Measles coverage and full immunization outcomes." IDinsight, Coverage Monitoring Analysis Plan, 2021, Pg. 3.

• 20

  See discussion of retention rate tracking in our full review of New Incentives.

• 21

  See the most recent version of our cost-effectiveness analysis here, "New Incentives" tab.

• 22

  See also GiveWell's literature review of incentives for immunization, which indicates that programs offering incentives for vaccination tend to show larger impacts in contexts with low baseline coverage rates.

• 23

  New Incentives notes, "We will be less confident in reasons for not vaccinating until we add these to data quality checks starting in Q1 2022. We haven’t added this yet to maximize focus on other key questions." Comment provided in response to reviewing a draft of this page in October 2021.

• 24
  • The coverage assessment is powered to detect an increase over baseline of approximately 11 percentage points for the BCG and Penta 1 vaccines and an increase of approximately 16 percentage points for the measles 1 vaccine. IDinsight, Coverage Monitoring Analysis Plan, 2021, Table 1, Pg. 3, "Trend*" columns. The power estimates cited here are averaged across all three states.
  • At the first follow-up, we expect increases of 11 percentage points for the BCG vaccine, 14 percentage points for the Penta 1 vaccine, and 10 percentage points for the measles vaccine. See this spreadsheet, "Expected effect sizes at 12-month follow up: combined across entire survey age range." These expected increases are close to the limit of the changes that can be detected by the monitoring assessment.

• 25
  • See this section of our review of New Incentives' program: "The RCT finds a 14 to 21 percentage point increase in the study's three pre-specified primary outcomes — i.e. self-report of having received 1) the BCG vaccine, 2) the measles vaccine, and 3) any of the three doses of PENTA vaccine. Relative to control households, households in the catchment area for treatment clinics were:
    • 16 percentage points more likely to report receiving the BCG vaccination (95% CI 0.12-0.21, 63% control group mean),
    • 21 percentage points more likely to report receiving any of the three doses of PENTA (95% CI 0.16-0.26, 54% control group mean), and
    • 14 percentage points more likely to report any dose of measles (95% CI 0.10-0.18, control group mean 59%).
  • We expect increases of 11 percentage points for the BCG vaccine, 14 percentage points for the Penta 1 vaccine, and 10 percentage points for the measles vaccine. See this spreadsheet, "Expected effect sizes at 12-month follow up: combined across entire survey age range."

• 26

  See previous footnote for increases observed in the RCT. We expect increases of 14 percentage points for the BCG vaccine, 18 percentage points for the Penta 1 vaccine, and 12 percentage points for measles. See this spreadsheet, "What effect size could we expect to obtain on vaccinations at 18-month follow up?"

• 27
  • We expect increases of 14 percentage points for the BCG vaccine, 18 percentage points for the Penta 1 vaccine, and 12 percentage points for measles. See this spreadsheet, "What effect size could we expect to obtain on vaccinations at 18-month follow up?"
  • The coverage survey is powered to detect a time trend increase of approximately 11 percentage points for the BCG and Penta 1 vaccines and an increase of approximately 16 percentage points for the measles 1 vaccine. IDinsight, Coverage Monitoring Analysis Plan, 2021, Table 1, Pg. 3, "Trend*" columns. The power estimates cited here are averaged across all three states.
  • Because the expected increases exceed the minimum detectable effect sizes, we expect the increases in the BCG and Penta 1 vaccines to be statistically significant.

• 28
  • ​​Not all LGAs will begin operating at the same time: "We understand from New Incentives that there is a ~6 month 'ramp up' period once operations start in a given clinic / LGA. LGAs will introduce operations between months 1-6. This means that LGAs that start in month 1 will undergo the 'ramp up' period between months 2-7 and will be operating at 'status quo' by month 8. But LGAs that start operating in month 6 will undergo the 'ramp up' period between months 7-12 and will only be operating at 'status quo' at month 13." IDinsight, New Incentives Coverage Monitoring Protocol, 2021, Pg. 12.
  • Because of this, not all children in the relevant age group will have been able to be vaccinated by the second follow-up: "The 6-12 month olds who are surveyed at follow-up 1 will have been born as clinics are being activated - in some cases, they will have been born before their clinic was actually activated. The 6-12 month olds who are surveyed at follow-up 2 will have been born as some clinics are ramping up, so operations may not yet be stable. It is not until follow-up 3 that the 6-12 month olds will have been born when all clinics in the expansion group were operating according to 'status quo'." IDinsight, New Incentives Coverage Monitoring Protocol, 2021, Pg. 12.

• 29

  We expect increases of 16 percentage points for the BCG vaccine, 21 percentage points for the Penta 1 vaccine, and 14 percentage points for measles. See this spreadsheet, "What effect size could we expect to obtain on vaccinations at 24-month follow up?"

• 30

  Svetha Janumpalli, Founder and CEO, New Incentives, emails to GiveWell, October 1 and 5, 2021 (unpublished)

• 31
  • The World Health Organization (WHO) reports that about 90% of infants vaccinated with BCG develop a scar. "However scar formation is not a marker for protection and approximately 10% of vaccine recipients do not develop a scar." World Health Organization, BCG vaccines: WHO position paper – February 2018, Pg. 84.
  • Data that New Incentives has collected indicates that 97% of infants had a BCG scar when they came for subsequent vaccinations. In June 2020, New Incentives started collecting data on the BCG scars of infants returning after the BCG vaccine. We have seen data covering roughly four months (June to October 2020). This data indicated that roughly 97% of infants had one BCG scar, roughly 3% had no BCG scar, and a small percentage (0.04%) had two or more scars. See GiveWell, New Incentives monitoring and evaluation 2017-2020 and plans for future evaluation, "Results" sheet, "Repeat enrollments - BCG scars on returning infants," for details.

• 32

  As of August 2021, the program now covers 20 LGAs. New Incentives, email to GiveWell, August 27, 2021 (unpublished).

• 33
  • "NI-ABAE will use current staff to conduct the household surveys according to this protocol." IDinsight, New Incentives Coverage Monitoring Protocol, 2021, Pg. 8.
  • "A multi-day, remote training will be designed that all enumerators will be required to complete. This will include knowledge tests to confirm retention of knowledge and may be used to ultimately select enumerators. This training will cover modules including (but not limited to): Purpose of activity; best practices of enumeration; informed consent and ethics; use of the CAPI; sampling process and household identification; age screening; and question-by-question review and explanation of the surveys. A brief retraining will be conducted before every survey activity." IDinsight, New Incentives Coverage Monitoring Protocol, 2021, Pg. 10.

• 34

  For example, IDinsight created IDinsight, Coverage Monitoring Analysis Plan, 2021.

• 35

  "There are several steps that will be taken to ensure data quality:

  1. Comprehensive enumerator training and periodic re-training: A multi-day, remote training will be designed that all enumerators will be required to complete. This will include knowledge tests to confirm retention of knowledge and may be used to ultimately select enumerators. This training will cover modules including (but not limited to): Purpose of activity; best practices of enumeration; informed consent and ethics; use of the CAPI; sampling process and household identification; age screening; and question-by-question review and explanation of the surveys. A brief retraining will be conducted before every survey activity.
  2. Collection of GPS data: GPS data will be collected for each survey conducted to both help with relocating the household if selected for backchecks and to confirm that enumerators are conducting the surveys in the sampled area. [Additional clarity on this point provided by New Incentives in response to reviewing a draft of this page in October 2021: "GPS is used for reviewing survey patterns, starting points, enumeration areas, and missed households in addition to verifying survey locations."]
  3. Back-checks: Back-checks will be conducted by managers and auditors only. The current plan is to randomly select a minimum of 10% of households that were surveyed (both screening and RI surveys). Back-checkers will then re-ask a subset of the survey questions to the household to confirm that the answers are consistent. The target number of back-checks may change based on the final sample sizes. [Update from New Incentives, November 5, 2021: "We found that there were not enough advantages to limiting back checks to managers and auditors so the back checks are being conducted by other staff members as well so that we can conduct more back checks. We select these staff members, managers, and auditors for back checks based on their performance in training surveys. We do not think there is significant room for collusion because surveyors do not back check surveys within a batch where they conducted one or more work packages. Back checkers are usually informed about their back check assignments no more than 2 days in advance and do not know the name of the surveyor. Furthermore, GPS checks, audio checks, and child health card checks would help us identify potential concerns."]
  4. Spot-checks: NI-ABAE intends to have supervisors in the field to do live spot-checks, depending on the location of the enumerator. These spot checks will be used to observe whether enumerators are following protocols and to share feedback.
  5. Audio Audits: NI-ABAE aims to record critical sections of the screening (number of children and their ages) and RI survey (self-reported vaccination status) and have managers and auditors verify whether the information entered by data collectors matches the recording. NI-ABAE will obtain explicit informed consent from respondents for these audio recordings. Resulting audio recordings will be stored in line with the data security measures described below."

  IDinsight, New Incentives Coverage Monitoring Protocol, 2021, Pg. 10.

• 36

  "We also check self-reported vaccination against a photo of the Child Health Cards, where available. We expect availability to be low at first but to be higher starting at the 12-month follow-up." Comment provided by New Incentives when reviewing a draft of this page in October 2021.