Innovations for Poverty Action — Randomized Controlled Trial on the Effect of Face Masks on COVID-19

Note: This page summarizes the rationale behind a GiveWell-recommended grant (and two follow-up grants) to Innovations for Poverty Action. This grant relies on more subjective judgment calls than GiveWell’s typical Incubation Grants or top charities.

Innovations for Poverty Action staff and other Principal Investigators of the study reviewed this page prior to publication.

We typically publish our write-ups in the months after making a grant. This page primarily describes our understanding at the time we recommended the grant (July 9, 2020), although we refer to more recent information where it helps to provide a clearer explanation of the goals of the study. We expect some characteristics of the trial, policy environment, and our understanding of the COVID-19 pandemic will have changed by the time the trial is completed.

Summary

In July 2020, GiveWell recommended a grant of $2,000,000 from the Effective Altruism Global Health and Development Fund to Innovations for Poverty Action (IPA). This grant was to support a randomized controlled trial (RCT) of an intervention to distribute face masks and promote their use to reduce the transmission of SARS-CoV-2, the virus that causes COVID-19. The grant resulted from GiveWell’s work to identify giving opportunities related to COVID-19 that could be more cost-effective than support for our top charities.

GiveWell later recommended two additional grants from the Effective Altruism Global Health and Development Fund to support the trial. These grants provided an additional $890,000 of core funding and $250,000 in contingency funding.1 $50,000 of that contingency funding has been released.

In total, as of July 2021, GiveWell has recommended $2,940,000 in funding that has been released, and an additional $200,000 in contingency funding that has not yet been released.2

Published: February 2021, Last updated: July 2021

Table of Contents

The study

The study is a cluster RCT.3 It will take place in Bangladesh. IPA will implement the intervention, which consists of (a) one time mask distribution and promotion at households, (b) mask distribution and promotion in public spaces, markets, and mosques, (c) asking local leaders to serve as role models and advocate for mask wearing. In addition, the researchers will test other interventions such as public commitments, incentives to wear masks (monetary and non-monetary), text reminders, and verbal commitments.4

The study will seek to answer whether an intervention to promote mask-wearing: (i) increases mask-wearing, (ii) reduces the spread of respiratory disease and symptomatic COVID-19 infection in the community, and (iii) protects the wearer from COVID-19 infection.5 It will compare an intervention group in which masks are distributed and their use is encouraged with a control group to which the project will neither distribute nor promote masks (i.e., behavior as usual). Individuals in the control group will continue to wear masks as they usually would—there are no restrictions placed upon them.6 The treatment and control groups together are expected to be approximately 600 villages, with a population of about 1,000 each. Half the villages will be assigned to receive the intervention, and half the villages will be assigned to the control group.7

The investigators plan to assess the effect of masks on community transmission (i.e., whether increased mask-wearing reduces incidence of SARS-CoV-2 in the community) as well as the individual protection they provide separately (details in footnote).8

At the time we recommended the grant, the investigators were planning to distribute only cloth masks. The grantees later decided to also distribute surgical masks to some of the intervention group to test whether cloth or surgical masks are more effective in reducing the spread of COVID-19.9 We believe comparing the two types of masks is valuable enough to justify the additional complexity.

At the time we recommended the grant, the investigators planned to conduct the intervention and collect data between August and November 2020.10 Subsequent delays, in order to obtain the necessary permissions and approvals in Bangladesh, mean that as of February 2021, we expect preliminary results on the effect of mask use on COVID-19 infection rates to be available in March 2021. The grantees may release results on which of the strategies tested, if any, promote mask-wearing prior to March 2021 if they believe those results would be helpful to decision makers.

The team

The investigators are:11

  • Professor Jason Abaluck, Associate Professor of Economics, Yale School of Management, Yale University
  • Professor Ahmed Mushfiq Mobarak, Professor of Economics, Yale University
  • Professor Stephen Luby, Professor of Medicine (Infectious Diseases), School of Medicine, Stanford University
  • Md Alamgir Kabir, Senior Operations Manager, Bangladesh, Innovations for Poverty Action
  • Dr. Laura H Kwong, Woods Institute for the Environment, Stanford University
  • Dr. Ashley Styczynski, Infectious Diseases Fellow, School of Medicine, Stanford University
  • Professor Meerjady Sabrina Flora, Director, Institute of Epidemiology, Disease Control and Research
  • Professor Peter Winch, Professor in the Social and Behavioral Interventions (SBI) Program in the Department of International Health, Johns Hopkins Bloomberg School of Public Health
  • Dr. Jade Benjamin-Chung, Lecturer, Epidemiology & Biostatistics, University of California, Berkeley
  • Dr. Mohammad Ashraful Haque, Country Director, Bangladesh, Innovations for Poverty Action
  • Dr. Muhammad Maqsud Hossain, Director, NSU Genome Research Institute, North South University

The intervention will be implemented by IPA, a non-profit that conducts RCTs on programs aimed at helping the poor.12

How will we assess the success of this grant?

We will assess the success of this grant on the basis of:

  • The measured impact of the intervention.
  • Our assessment of the quality of the study.
  • Whether policymakers or other implementers use the results of the study to inform critical policy decisions on mask usage.

Case for the grant

The case for the grant is:

  • Wearing a face mask is a low-cost behavior change that may have substantial positive health impacts, and is relevant to most countries in the world during the COVID-19 pandemic. Reusable face masks can cost less than a dollar each and may have a substantial effect on transmission of SARS-CoV-2 (see footnote below for a shallow review of the existing evidence).13 When we made this grant (July 9, 2020), the Coronavirus Resource Center at Johns Hopkins University estimated there had been 11.7 million confirmed cases of COVID-19 globally (with at least some confirmed cases in nearly all countries), and 539,000 deaths from COVID-19.14
  • There is little direct evidence on the effectiveness of face masks in community settings. This study will provide such evidence. Face masks likely reduce transmission of SARS-CoV-2, the virus that causes COVID-19. The existing literature primarily consists of observational studies, laboratory studies, and common-sense reasoning, all of which have important limitations.15 If successful, this study would address these limitations. It will be: (1) randomized rather than observational, allowing us to infer a clearer causal connection between the masks and their effect; (2) in a real-world setting rather than a lab, allowing us to account for considerations like incorrect or inconsistent mask use, (3) larger than all previous experiments combined, meaning it has a higher chance of finding an effect if there is one, and (4) designed to assess both community and individual protective effects.16
  • In July 2020, when we made this grant, global mask use appeared to be low, suggesting it had room to grow. Surveys of self-reported mask use suggested that there were more people wearing masks globally in July than there were at the start of the pandemic. However, observed mask use remained low, suggesting that self-reported mask-wearing overstated actual mask-wearing, perhaps due to social desirability bias (providing responses that will be perceived favorably by others—for example, avoiding appearing irresponsible). Additionally, self-reported mask use remained low in some countries as of July.17 In January 2021 (shortly before we published this page), it appears that self-reported mask use has risen since July in many countries, but there is still substantial variation between countries.18
  • Decision makers have shown strong interest in this evidence. Our understanding, based on conversations with scientific advisers for the World Health Organization (WHO) and UK government, is that high-quality evidence on the effectiveness of mask-wearing would have a high chance of informing policy decisions about the extent to which governments should invest in mandating, promoting, or distributing face masks to the public.19

Taking these four factors into consideration, we calculated the cost-effectiveness of the grant by looking at its potential benefits relative to its expected costs. Our estimate of the cost-effectiveness of this grant suggests it may be more cost-effective than our top charities. We believe this estimate is directionally informative, but due to time limitations (we aimed to provide grant funding sooner than later to have more impact during the pandemic), the complexity of modelling the cost-effectiveness of a study with uncertain outcomes, and uncertainty about the projected number of deaths from COVID-19, the calculations are substantially less rigorous and more uncertain than cost-effectiveness analyses for our top charities.20

In addition to being informative during the current COVID-19 pandemic, we see a plausible case that the study will generate substantial value by contributing to longer-term understanding and attitudes towards mask-wearing, reducing the burden of other respiratory diseases. However, we have not deeply investigated this possibility, and the primary reason we recommended this grant was because of its potential impact on behavior during the current pandemic.

Risks and reservations

We investigated a number of risks and reservations related to this grant. Of these, we believe the following are most important:

  • If the increase in mask use from the intervention is lower than expected, or if COVID-19 infection rates are lower than expected, the study will lose power and may fail to find a statistically significant effect of masks on COVID-19, even if such an effect exists. This result could be misinterpreted as providing evidence of no effect (i.e., that masks don't work).
    • Given the existing evidence base for the effectiveness of masks, we expect the study to find a statistically significant reduction in COVID-19. However, its ability to do so depends on the true size of the effect of masks against COVID-19, the degree to which the intervention can successfully increase mask use, and how widespread COVID-19 is during the study period.
    • We expect that the RCT would not be capable of distinguishing small effects from no effect. Given the low cost of masks, we believe that most individuals should wear a mask in public places, even if it reduces the chance of contracting or transmitting SARS-CoV-2 by only 1%. We do not believe it would be feasible to run a trial large enough to detect or rule out an effect this small.21 However, given the uncertainty about the size of the effect of masks, we believe this trial could provide valuable information to help governments and other organizations decide how to prioritize their limited resources for addressing COVID-19 and actions they should take to promote mask use.
    • If mask use was already high in Bangladesh, then it would be particularly difficult for the intervention to increase mask use in the intervention group relative to control groups. However, we believe mask use was low in Bangladesh based on an observation in 1,441 locations across the country between June 19-22. That observation found 26% of people were wearing masks, in contrast to self-reported estimates suggesting mask use was substantially higher.22 However, it is possible that the intervention could fail to increase mask use for other reasons, or that mask use might increase in the control group over the study period as a result of increased government enforcement. We see this as a major risk to the validity of the study.

  • The results from the study may come too late to meaningfully influence the trajectory of the COVID-19 pandemic. At the time we made the grant, we expected preliminary results from the RCT to be available by the end of 2020. 23 Subsequent delays mean we now expect results to be available some months later. Our best guess is that stronger evidence of mask effectiveness would lead to an increase in mask use through a combination of government policy and individual behavior change, preventing many deaths from COVID-19. It is possible that governments will implement additional interventions to increase mask use before the results of the study are available, or that other interventions, such as vaccines, become available sooner than expected. This would reduce the impact of the study over the course of the pandemic.

Other questions we investigated included:

  • Is it ethical to run this study? RCTs typically compare two randomly assigned groups: a treatment group, which receives the intervention, and a control group, which does not.24 Because face masks likely reduce transmission of SARS-CoV-2 (see previous footnote for a shallow review of the existing evidence), any RCT that asks the control group not to wear masks would be unethical.25

    However, the efficacy of masks can be tested without asking the control group not to wear masks. In IPA's study, individuals in the control group will continue to purchase and wear masks as usual, with no restrictions placed upon their behavior. The study is to compare the impact of additional encouragement to wear masks with typical behavior.26

    Additionally:

    • The study has been approved by the Directorate General of Health Services (DGHS) in Bangladesh, the Bangladesh Medical Research Council (BMRC), and the Yale University Institutional Review Board (IRB).27
    • This study is similar in execution to previous RCTs examining public health interventions to change behavior. There have been multiple RCTs testing the effect of interventions to provide soap and promote handwashing on diarrheal disease,28 as well as RCTs attempting to study the effect of masks on respiratory disease.29
    • The surveyors will ensure they obtain informed consent from potential participants in the study.30
  • How are the masks designed? At the time we recommended funding for the RCT, the authors were planning to distribute polyester masks with three layers (more details in footnote).31 The grantees are also planning to distribute surgical masks to part of the intervention group to test whether cloth or surgical masks have greater efficacy against COVID-19.32

Plans for follow-up

We are checking in with the investigators every two weeks for updates on the RCT design and implementation. In July, we expected that we might make a smaller grant at a later date to assist in the dissemination of study results. We have since done so.

Our process

  • The investigators approached us in early April with a proposal to run an RCT on mask use in Bangladesh.
  • We reviewed the existing evidence base for masks at a shallow level, spoke with relevant decision makers about the potential for policy impact, conducted desktop research on current mask use and policies, and investigated projections of the spread of COVID-19 in Bangladesh to determine if the RCT would be well-powered. We had multiple conversations with the study authors to probe at elements of the study design and to confirm that it would be conducted in an ethical manner.

Internal forecasts

We made a number of forecasts related to this grant. These forecasts were made by the lead grant investigator and were informed by the material contained in this grant page, as well as intuitive reasoning. Because of the uncertainty around how this grant will resolve, we felt that making explicit forecasts would help us to determine the extent to which we exercised good judgment in making this grant after the fact. The resolution dates listed below are the dates by which we expect to evaluate the predictions associated with them.

Prediction Credence Resolution date
The RCT will NOT find a statistically significant33 result on self-reported respiratory symptoms, AND it will not find a statistically significant result on COVID-19.34 15% June 1, 2021
The RCT will find a statistically significant result on self-reported respiratory symptoms, but NOT COVID-19. 20% June 1, 2021
The RCT will NOT generate a statistically significant result on AT LEAST ONE of the outcomes, and the authors will say in their first preprint that an important reason was lower than expected incidence of COVID-19 or self-reported respiratory symptoms. 25% June 1, 2021
The RCT will NOT generate a statistically significant result on AT LEAST ONE of the outcomes, and the authors will say in their first preprint that an important reason was a failure to increase mask-wearing. 20% June 1, 2021
Results on respiratory symptoms (Conditional on a paper being published in a top 200 academic journal)
The RCT will NOT find a statistically significant (p>0.05) impact on self-reported respiratory symptoms. 13% June 1, 2021
Statistically significant result and reduction in respiratory symptoms of 0-10%. 5% June 1, 2021
Statistically significant result and reduction in respiratory symptoms of 10-20%. 20% June 1, 2021
Statistically significant result and reduction in respiratory symptoms of 20-30%. 30% June 1, 2021
Statistically significant result and reduction in respiratory symptoms of 30-40%. 22% June 1, 2021
Statistically significant result and reduction in respiratory symptoms of >40%. 10% June 1, 2021
Results on COVID-19 outcome (Conditional on a paper being published in a top 200 academic journal)
The paper will not report an objectively measured disease outcome confirmed with diagnostic tests. 10% June 1, 2021
Conditional on a COVID-19 outcome being reported in the published paper
The RCT will NOT find a statistically significant (p>0.05) impact on preferred measure of COVID-19. 25% June 1, 2021
Statistically significant result and reduction in COVID-19 of 0-10%. 5% June 1, 2021
Statistically significant result and reduction in COVID-19 of 10-20%. 14% June 1, 2021
Statistically significant result and reduction in COVID-19 of 20-30%. 28% June 1, 2021
Statistically significant result and reduction in COVID-19 of 30-40%. 20% June 1, 2021
Statistically significant result and reduction in COVID-19 of >40%. 8% June 1, 2021
Attention as a path to impact
At least one academic paper will be published in a top 200 academic journal on the basis of this RCT. 85% June 1, 2021
Conditional on the study getting a statistically significant result on EITHER respiratory symptoms OR COVID-19, it will be mentioned in over 50,000 tweets according to altmetrics. 80% June 1, 2021
Conditional on a preprint being published (i.e., a draft paper is published on the internet), the study will receive justified criticism (as bad as this; determined by a panel of three GiveWell researchers if it's not obvious) questioning its conclusions. 1% June 1, 2021
Impact
Conditional on the study getting a statistically significant result on EITHER respiratory symptoms OR COVID-19, the WHO will publish an update to its guidance on masks mentioning this study. 65% June 1, 2021
Conditional on the study getting a statistically significant result on EITHER respiratory symptoms OR COVID-19, we will see at least one country with a population >20 million update its guidance on mask-wearing, explicitly referring to EITHER this evidence OR updated WHO guidance that refers to this evidence. 60% June 1, 2021
Conditional on the study getting a statistically significant result on EITHER respiratory symptoms OR COVID-19, it will be mentioned in the minutes of the UK's SAGE discussions. 40% June 1, 2021
Timing
A preprint is published (i.e., a draft paper is published on the internet) on the basis of this RCT by October 1, 2020. 20% October 1, 2020
A preprint is published (i.e., a draft paper is published on the internet) on the basis of this RCT by December 1, 2020. 55% December 1, 2020
A preprint is published (i.e., a draft paper is published on the internet) on the basis of this RCT by February 1, 2021. 65% February 1, 2021
A preprint is published (i.e., a draft paper is published on the internet) on the basis of this RCT by April 1, 2021. 75% April 1, 2021

Additional funding

In November 2020, we recommended an additional $90,000 from the Effective Altruism Global Health and Development Fund to support management of communications related to the study. We also recommended an additional $50,000 in contingency funding for communications activities that may or may not be necessary. The contingency funding will be held by the Global Health and Development Fund, but may be deployed quickly to IPA if it is needed. If it is not needed, it will remain in the Global Health and Development Fund for another use.35

In December 2020, we recommended an additional $800,000 and $200,000 in contingency funding from the Effective Altruism Global Health and Development Fund to support the completion of the trial. This additional funding is due to a combination of unexpected costs, opportunities to improve the study design, and the grantees being unable to raise additional funding they expected to receive from other sources. As above, the contingency funding will be held by the Global Health and Development Fund.36

We made these grants because we continue to believe the results of this trial would be valuable, and we expected that without this additional funding, the grantees would have had to compromise important aspects of the study design.

In May 2021, we recommended the release of the $50,000 in contingency funding from the November 2020 grant for communications activities related to the grant.

Sources

Document Source
Abaluck and Mobarak 2020 Source (archive)
Begley, "Influential Covid-19 model uses flawed methods and shouldn’t guide U.S. policies, critics say," 2020 Source (archive)
BMRC, National Research Ethics Committee, Ethical Clearance for "Can Face Masks Reduce Transmission of SARS-CoV-2 in Bangladesh? A Cluster-Randomized Controlled Trial," November 10, 2020 Source
Chu et al. 2020 Source (archive)
COVID-19 Dashboard by the Center for Systems Science and Engineering at Johns Hopkins University, World Map, July 29, 2020 Source (archive)
COVID-19 Dashboard by the Center for Systems Science and Engineering at Johns Hopkins University, World Map, July 9, 2020 Source (archive)
DGHS, Approval for "Serological test of blood samples for Bangladesh Mask Project," October 10, 2020 Source
Ejemot-Nwadiaro et al. 2015 Source (archive)
Good Judgment, About Source (archive)
Good Judgment, COVID Recovery Dashboard (accessed May 31, 2020) Source (archive)
Haber et al. 2020 Source (archive)
Heneghan and Jefferson 2020 Source (archive)
Higgins, Eldridge, and Li 2020 Source (archive)
Howard, "FAQs For Mask Usage Skeptics," 2020 Source (archive)
IHME, "Maps of Mask Use," 2020 Source (archive)
IHME, Infographic: COVID-19 Mask Usage Meta-Analysis, 2020 Source (archive)
IHME, News release, 2020 Source (archive)
Innovations for Poverty Action, What We Do Source (archive)
IPA, Proposal for communications support on masks RCT, 2020 Source
Jefferson et al. 2020 (preprint) Source (archive)
Konda et al. 2020 Source (archive)
LaMorte 2020 Source (archive)
Lyu and Wehby 2020 Source (archive)
MacIntyre and Chughtai 2020 Source (archive)
MacIntyre et al. 2015 Source (archive)
MacIntyre et al. 2020 Source (archive)
Masks RCT additional budget request, 2020 Unpublished
Masks RCT Proposal, July 6, 2020 Source
Mitze et al. 2020 (preprint) Source (archive)
SCImago Journal & Country Rank, 2019, Ranks 151-200 Source (archive)
The Institute for Health Metrics and Evaluation Source (archive)
U.S. Food and Drug Administration, Institutional Review Boards Frequently Asked Questions, 1998 Source (archive)
UK Government Office for Science and Scientific Advisory Group for Emergencies 2020 Source (archive)
WHO 2020 Source (archive)
Yale Human Research Protection Program, IRB Approval Document for: "An RCT for Masks to Slow the Spread of COVID-19 in Bangladesh," July 15, 2020 Unpublished
YouGov, COVID-19 behaviour changes tracker: Wearing a face mask when in public places, 2020 Source (archive)
Zhang et al. 2020 Source (archive)
Zhang et al. 2020, Altmetric Twitter Metrics Source (archive)
  • 1
    • Core funding is funding that we strongly expect will be necessary to complete the trial and has been transferred to IPA.
    • Contingency funding is funding we are unsure will be necessary to complete the trial. GiveWell will recommend this funding to the Effective Altruism Global Health and Development Fund if the grant investigator believes it is necessary. The contingency funding will remain in the Global Health and Development Fund unless it is necessary, at which point it can be quickly deployed to IPA. If it is not needed, it will remain in the Global Health and Development Fund for future grantmaking.

  • 2
    • In November 2020, GiveWell recommended a grant of $90,000 of core funding and $50,000 in contingency funding to support IPA's work managing communications related to the trial.
    • In December 2020, GiveWell recommended a grant of $800,000 of core funding and $200,000 in contingency funding to support the completion of the trial.
    • In May 2021, GiveWell recommended the release of the $50,000 in contingency funding from the November 2020 grant.

  • 3

    "In cluster-randomized trials, groups of individuals rather than individuals are randomized to different interventions. We say the ‘unit of allocation’ is the cluster, or the group. The groups may be, for example, schools, villages, medical practices or families. Cluster-randomized trials may be done for one of several reasons. It may be to evaluate the group effect of an intervention, for example herd-immunity of a vaccine. It may be to avoid ‘contamination’ across interventions when trial participants are managed within the same setting, for example in a trial evaluating training of clinicians in a clinic. A cluster-randomized design may be used simply for convenience." Higgins, Eldridge, and Li 2020.

  • 4

    Communications lead of the RCT, comments on a draft of this page, January 29, 2021 (unpublished)

  • 5
      "We have three main questions we seek to answer:
    1. Can we successfully change community mask-wearing norms?
    2. Can community mask-wearing reduce the spread of respiratory disease and (symptomatic) COVID-19 infection?
    3. Can individual mask-wearing provide protection against COVID-19 infection?"

    Masks RCT Proposal, July 6, 2020, p. 4.

  • 6

    "Study participants (including those in the control groups) may decide to wear other locally available masks or face coverings besides the ones provided through this project." Masks RCT protocol, July 1, 2020 (unpublished), p. 15.

  • 7

    "The "unit" of the experiment is now a village, or more precisely, a village + market + mosque combination. Each village has about 250 households (or a population of about 1,000). There are 600 villages in the experiment." Principal Investigators of the RCT, comments on a draft of this page, January 11, 2021 (unpublished)

  • 8
      To test community transmission:
    • "Before the intervention begins, the research team will obtain informed consent and collect data and blood samples following appropriate public health protocols and expert recommendations. The investigators will conduct a phone survey in weeks five and nine to understand if participants have respiratory symptoms consistent with COVID-19. Twelve weeks after the intervention began, blood samples will be collected from the 25,000 people who provided blood at baseline and people who report respiratory symptoms at any point in the study. We will then test the blood using serology (blood) tests to confirm whether they have COVID-19. Additionally, the research team will systematically and unobtrusively observe mask-wearing and physical distancing during the eight weeks of study time" Principal Investigators of the RCT, comments on a draft of this page, January 11, 2021 (unpublished)
      To test individual protection:
    • "For the individual-level experiment, the primary outcome is seroconversion (development of antibodies against SARS-CoV-2) over the course of the study. Masks will be given to 2,500 vendors who work at indoor markets in the control villages. Similar to the community-level experiment, they will be asked about respiratory symptoms. However, they will receive serology tests for COVID-19 at baseline as well as at 12 weeks, allowing for the assessment of individual-level protection of mask use." Principal Investigators of the RCT, comments on a draft of this page, January 11, 2021 (unpublished)

  • 9
    • Professor Mushfiq Mobarak, Professor of Economics, Yale University; Professor Jason Abaluck, Associate Professor of Economics, Yale School of Management, conversation with GiveWell, July 24, 2020 (unpublished)
    • "Surgical Mask Specifications:
      • Exterior layer: 100% spunbond, non-woven polypropylene
      • Middle layer: 100% meltblown, non-woven polypropylene
      • Inner layer: 100% spunbond, non-woven polypropylene
      • Elastic ear loops; nose bridge
      • ~95% filtration efficiency of 0.3 um particles" Dr. Laura Kwong, Postdoctoral Researcher, Woods Institute for the Environment, Stanford University, email to GiveWell, January 16, 2021 (unpublished)

  • 10

    Masks RCT Timeline Spreadsheet, 2020 (unpublished)

  • 11

    Principal Investigators of the RCT, comments on a draft of this page, January 11, 2021 (unpublished)

  • 12

    "In partnership with top researchers in the field, we design and implement randomized evaluations to measure the effectiveness of programs and policies aimed at helping the poor. We specialize in randomized controlled trials (RCTs) because this rigorous methodology allows us to isolate the effects of a program from other factors." Innovations for Poverty Action, What We Do

  • 13

    "Latest version of paper says cloth masks were $0.40 and surgical were $0.13." Grantee, comments on a draft of this page, January 29, 2021 (unpublished)

  • 14

    See COVID-19 Dashboard by the Center for Systems Science and Engineering at Johns Hopkins University, World Map, July 9, 2020.

  • 15
    • We have reviewed the evidence on the effect of cloth and medical face masks on virus transmission at a shallow level, but have not deeply reviewed each paper. Our understanding is that the preponderance of existing evidence, combined with common sense reasoning, suggests that wearing a mask is likely effective at reducing transmission of viruses in general and SARS-CoV-2 in particular. However, the existing experimental evidence is limited, which leaves us uncertain as to the size of the effect.
      • The evidence that cloth masks reduce transmission of SARS-CoV-2 comes from:
        • Observational studies using quasi-experimental methods (e.g., Zhang et al. 2020, Mitze et al. 2020 (preprint), Lyu and Wehby 2020). Unlike experimental studies, observational studies do not involve intervention by the researchers. Instead, the researchers analyze data on existing behavior and trends. It can be difficult to separate the effect of masks from other factors in these studies. We have not reviewed these studies in depth, although at least one (Zhang et al. 2020) has been heavily criticized (see Haber et al. 2020).
        • Laboratory studies (e.g., Konda et al. 2020), in which the filtration efficiency of cloth masks is tested under laboratory conditions. These studies provide evidence that masks are likely to be effective but may not translate to real-world use cases.
          • "Importantly, there is a need to evaluate filtration efficiencies as a function of aerosol particulate sizes in the 10 nm to 10 μm range, which is particularly relevant for respiratory virus transmission. We have carried out these studies for several common fabrics including cotton, silk, chiffon, flannel, various synthetics, and their combinations. Although the filtration efficiencies for various fabrics when a single layer was used ranged from 5 to 80% and 5 to 95% for particle sizes of <300 nm and >300 nm, respectively, the efficiencies improved when multiple layers were used and when using a specific combination of different fabrics. Filtration efficiencies of the hybrids (such as cotton–silk, cotton–chiffon, cotton–flannel) was >80% (for particles <300 nm) and >90% (for particles >300 nm). We speculate that the enhanced performance of the hybrids is likely due to the combined effect of mechanical and electrostatic-based filtration. Cotton, the most widely used material for cloth masks performs better at higher weave densities (i.e., thread count) and can make a significant difference in filtration efficiencies. Our studies also imply that gaps (as caused by an improper fit of the mask) can result in over a 60% decrease in the filtration efficiency, implying the need for future cloth mask design studies to take into account issues of “fit” and leakage, while allowing the exhaled air to vent efficiently. Overall, we find that combinations of various commonly available fabrics used in cloth masks can potentially provide significant protection against the transmission of aerosol particles." Konda et al. 2020, Pg. 6339.
        • One experimental study (MacIntyre et al. 2015) comparing medical masks with cloth masks in a healthcare setting. This study found that the group using fabric masks had significantly higher rates of influenza-like illness (particularly illness caused by rhinovirus) than the group using medical masks, implying cloth masks were less effective than medical masks in reducing the spread of viruses. The authors have published a note suggesting that this study should not be interpreted as strong evidence that cloth masks are ineffective against COVID-19 if medical masks are unavailable.
          • "The rates of all infection outcomes were highest in the cloth mask arm, with the rate of [influenza-like illness (ILI)] statistically significantly higher in the cloth mask arm (relative risk (RR)=13.00, 95% CI 1.69 to 100.07) compared with the medical mask arm. Cloth masks also had significantly higher rates of ILI compared with the control arm. An analysis by mask use showed ILI (RR=6.64, 95% CI 1.45 to 28.65) and laboratory-confirmed virus (RR=1.72, 95% CI 1.01 to 2.94) were significantly higher in the cloth masks group compared with the medical masks group. Penetration of cloth masks by particles was almost 97% and medical masks 44%." MacIntyre et al. 2015, Pg. 1.
          • "Among the 68 laboratory-confirmed cases, 58 (85%) were due to rhinoviruses. Other viruses detected were hMPV (7 cases), influenza B (1 case), hMPV/rhinovirus co-infection (1 case) and influenza B/rhinovirus co-infection (1 case) (table 3). No influenza A or RSV infections were detected." MacIntyre et al. 2015, Pg. 4.
          • "Health workers are asking us if they should wear no mask at all if cloth masks are the only option. Our research does not condone health workers working unprotected. We recommend that health workers should not work during the COVID-19 pandemic without respiratory protection as a matter of work health and safety. In addition, if health workers get infected, high rates of staff absenteeism from illness may also affect health system capacity to respond. Some health workers may still choose to work in inadequate PPE. In this case, the physical barrier provided by a cloth mask may afford some protection, but likely much less than a surgical mask or a respirator." MacIntyre et al. 2020
      • Our impression from two systematic reviews is that there is limited experimental evidence of the effectiveness of surgical masks at reducing COVID-19, but observational evidence provides suggestive evidence that surgical masks are effective.
        • Jefferson et al. 2020 (preprint), an update to a Cochrane Collaboration systematic review of experimental evidence on the effectiveness of facemasks, concludes that 15 RCTs find no evidence that face masks are effective. However, the review concludes that most trials were limited by poor design, reporting, and sparse events. The study authors recommend the use of masks combined with other measures.
          • "We included 15 randomised trials investigating the effect of masks (14 trials) in healthcare workers and the general population and of quarantine (1 trial). We found no trials testing eye protection. Compared to no masks there was no reduction of influenza-like illness (ILI) cases (Risk Ratio 0.93, 95%CI 0.83 to 1.05) or influenza (Risk Ratio 0.84, 95%CI 0.61-1.17) for masks in the general population, nor in healthcare workers (Risk Ratio 0.37, 95%CI 0.05 to 2.50)." Jefferson et al. 2020 (preprint), Pgs. 2-3.
          • "Most included trials had poor design, reporting and sparse events. There was insufficient evidence to provide a recommendation on the use of facial barriers without other measures. We found insufficient evidence for a difference between surgical masks and N95 respirators and limited evidence to support effectiveness of quarantine. Based on observational evidence from the previous SARS epidemic included in the previous version of our Cochrane review we recommend the use of masks combined with other measures" Jefferson et al. 2020 (preprint), Pg 3.
        • Chu et al. 2020 is a systematic review of 172 observational studies on the effect of face masks and other interventions to reduce the transmission of viruses, funded by the World Health Organization. It concludes that face masks could result in a large reduction in risk of infection, but that robust RCTs are needed to better inform the evidence base.
          • "We did a systematic review and meta-analysis to investigate the optimum distance for avoiding person-to person virus transmission and to assess the use of face masks and eye protection to prevent transmission of viruses… Our search identified 172 observational studies across 16 countries and six continents, with no randomised controlled trials and 44 relevant comparative studies in health-care and non-health-care settings (n=25697 patients)." Chu et al. 2020, Pg. 1973.
          • "Face mask use could result in a large reduction in risk of infection (n=2647; aOR 0·15, 95% CI 0·07 to 0·34, RD −14·3%, −15·9 to −10·7; low certainty), with stronger associations with N95 or similar respirators compared with disposable surgical masks or similar (eg, reusable 12–16-layer cotton masks; pinteraction=0·090; posterior probability >95%, low certainty)." Chu et al. 2020, Pg. 1973.
          • "The findings of this systematic review and meta-analysis support physical distancing of 1 m or more and provide quantitative estimates for models and contact tracing to inform policy. Optimum use of face masks, respirators, and eye protection in public and health-care settings should be informed by these findings and contextual factors. Robust randomised trials are needed to better inform the evidence for these interventions, but this systematic appraisal of currently best available evidence might inform interim guidance." Chu et al. 2020, Pg. 1973.
          • "Funding: World Health Organization." Chu et al. 2020, Pg. 1973.

  • 16
    • "In 2010, at the end of the last influenza pandemic, there were six published randomised controlled trials with 4,147 participants focusing on the benefits of different types of masks. Two looked at healthcare workers and four at family or student clusters. The face mask trials for influenza-like illness (ILI) reported poor compliance, rarely reported harms and revealed the pressing need for future trials. Despite the clear requirement to carry out further large, pragmatic trials a decade later, only six had been published: five in healthcare workers and one in pilgrims. This recent crop of trials added 9,112 participants to the total randomised denominator of 13,259 and showed that masks alone have no significant effect in interrupting the spread of ILI or influenza in the general population, nor in healthcare workers. The design of these twelve trials differed: viral circulation was usually variable; none were conducted during a pandemic. Outcomes were defined and reported in seven different ways, making comparison difficult. It is debatable whether any of these results could be applied to the transmission of SARs-CoV-2. Only one randomised trial included cloth masks. This trial found ILI rates were 13 times higher in Vietnamese hospital workers allocated cloth masks compared to medical/surgical masks, and over three times higher when compared to no masks." Heneghan and Jefferson 2020
    • "We expect to successfully survey directly 500 people per mouza per wave across 700 mouzas, or 350,000 people for each of the 3 waves of the survey." Masks RCT Proposal, July 6, 2020, Pg. 7.

  • 17
    • A YouGov survey of a limited number of countries suggests that in most of the countries, self-reported mask-wearing has increased substantially since the start of the pandemic. In some of the countries, however, mask use remains relatively low. At the time we recommended this grant, the latest self-reported mask use estimate was 31% in the UK (June 25), compared to over 80% in most Asian countries. The latest estimates for Norway, Denmark, and Sweden as of July 28 were under 10%.
    • The Institute for Health Metrics and Evaluation (IHME), a global health research center, estimated—based on internet surveys conducted on June 13—that mask use was highly variable between countries: "We considered more than 20 studies and ran a meta-analysis. The results show that masks work, but survey data reveal that mask use varies." IHME, Infographic: COVID-19 Mask Usage Meta-Analysis, 2020.
    • We expect that these self-reported mask use estimates overstate the prevalence of mask-wearing due to self-reporting bias and potentially selection bias (we are unsure how the surveys were conducted). IPA conducted a direct observation of mask use in Bangladesh, which found a lower rate of mask-wearing than suggested by phone surveys.
      • In April, the study authors conducted a telephone survey in Bangladesh in which 97% of people said that they regularly wore a mask outdoors. "We surveyed 2,500 people in both urban and rural areas throughout Bangladesh by phone on 22 April 2020: 97% reported owning a mask, with near-uniform adoption across every region of the country." Abaluck and Mobarak 2020
      • In late June, the study authors ran a direct observation of 84,000 people in markets and mosques in Bangladesh, finding only 26% were in fact wearing masks.
        • "We have some initial results from our latest mask surveillance, conducted over the past 2 days.
          Average mask use was 25.5%..." Professor Jason Abaluck, Associate Professor of Economics, Yale School of Management, email to GiveWell, June 20, 2020 (unpublished).
        • Mask surveillance data, June 19-20, 2020 (unpublished)
      • It is possible that mask use declined sharply over this period, but we expect a substantial proportion of the difference was caused by self-reporting bias.

  • 18

    See these estimates from IHME on how self-reported mask-use has changed over time. We continue to believe that self-reported mask use likely overstates real mask-use because of self-reporting bias.

  • 19
    • Professor Sir Mark Walport, Chief Executive, UK Research and Innovation, conversation with GiveWell, June 22, 2020 (unpublished)
    • Professor Benedetta Allegranzi, Coordinator, Infection Prevention and Control Global Unit, World Health Organization; Dr. Alessandro Cassini, Technical Officer, Infection Prevention and Control Global Unit, World Health Organization; Professor John Conly, Expert Adviser, World Health Organization, conversation with GiveWell, July 3, 2020 (unpublished)
    • Dr. Soumya Swaminathan, Chief Scientist, World Health Organization, email to Professor Jason Abaluck, Associate Professor of Economics, Yale School of Management, July 7, 2020 (unpublished)

  • 20
    • The Institute for Health Metrics and Evaluation (IHME) estimates that there would be close to 20% fewer COVID-19 deaths in the United States over a particular time period if 95% of people wore masks, compared to no mask use. This suggests that each percentage point increase in mask use may reduce deaths from COVID-19 by 0.2%. We have not vetted this model, and we are unsure about the extent to which we should trust its conclusions. We used the output of this model because it seemed intuitively reasonable and because explicit cost-effectiveness estimates played less of a role in our decision-making for our grant to IPA than it would for our top charities. We note that the IHME COVID-19 projections have received substantial criticism.
      • "SEATTLE (June 24, 2020) – In its first projections comparing different actions to control COVID-19 transmission, the Institute for Health Metrics and Evaluation (IHME) at the University of Washington is forecasting nearly 180,000 in the United States will die by October 1.

        The forecast shows 179,106 deaths (with a range of 159,497 to 213,715). Those numbers drop to 146,047 (with a range of 140,849 to 153,438), if at least 95 percent of people wear masks in public." IHME, News release, 2020

      • 20% reduction in deaths divided by 95% mask-wearing = ~0.2% reduction in COVID-19 deaths associated with each additional percentage point of mask-wearing. This assumes a linear relationship between mask-wearing and COVID-19 deaths, which may not hold in practice.
      • See this article for a description of some of the criticism that IHME's COVID-19 projections have received.
    • If this study finds a significant and positive result, our best guess is that it could lead to a 0.1 to 1 percentage point increase in global mask use. This is based on variable mask use between countries, the current limitations of the evidence base, and interest from policymakers. We therefore estimate that if this study finds a significant reduction in COVID-19, the global reduction in deaths could be between 0.02% and 0.2%.
      • 0.2% reduction in COVID-19 deaths associated with each additional percentage point of mask-wearing * between 0.1 and 1 percentage point increase in global mask use = 0.02% to 0.2% global reduction in COVID-19 deaths
    • Our best guess (at the time we made the grant) was that the results of the study would become available at the end of 2020. These results could impact the number of deaths from COVID-19 that would otherwise occur from the end of 2020 onwards, which we estimate at ~3 million deaths. This guess was informed by forecasts from Good Judgment, a platform that aggregates the best guesses of different forecasters.
      • Our best guess is 1 million deaths from COVID-19 between January 1 and March 31, 2021. Between the start of the pandemic and March 2021, Good Judgment estimates a 44% chance of fewer than 2.5 million deaths, a 43% chance of 2.5-8 million deaths, and a 13% chance of more than 8 million deaths. Our estimation is that by March 31, 2021, there will have been 4 million deaths from COVID-19. Although we expect less than 4 million deaths is more likely, our mean estimate is skewed upwards because of a smaller chance of many more deaths. Of these 4 million deaths, we roughly guess that 1 million will occur in the first quarter of 2021 because that represents a quarter of the time period between March 31, 2020 and March 31, 2021. The number of confirmed cases of COVID-19 prior to March 31, 2020 was relatively low.
      • Our best guess is 2 million deaths from COVID-19 after March 2021. We do not know what a reasonable estimate would be for the number of deaths from COVID-19 that will occur after March 2021. Based on our understanding of likely vaccine timelines in the United States (as of July 2020), and the additional time required to distribute the vaccine more widely, we roughly guess that there will be an additional 2 million deaths from COVID-19 after March 2021 (~50% of the deaths that we estimate will have occurred from the beginning of the pandemic to March 2021). Please note that this estimate in particular is highly uncertain.
      • Our best guess based on the estimates above is that there will be roughly 3 million deaths from COVID-19 from the end of 2020 onwards.
        • 1 million deaths (between January 1 and March 31, 2021) + 2 million deaths (after March 2021) = 3 million deaths from COVID-19 from the end of 2020 onwards
    • We forecast a 65% chance that the study finds a statistically significant effect of distributing and promoting masks on COVID-19. This is based on the likely effectiveness of masks against COVID-19, the likely power of the study, and the possibility that the intervention may not substantially increase mask use. This implies that the study will prevent about 400 to 4,000 deaths in expectation, which suggests our grant to IPA could be more cost-effective than our top charities.
      • 3 million deaths * 0.02 to 0.2% reduction in deaths = 600 to 6,000 deaths prevented
      • 600 to 6,000 deaths prevented * 0.65% = ~400 to 4,000 deaths prevented in expectation
      • $2,000,000 grant cost divided by 400 to 4,000 deaths prevented in expectation = grant cost-effectiveness of $500-5,000 per life saved. For comparison with our top charities, see our cost-effectiveness analysis (sheet "Results", section "Cost per life saved estimates").

  • 21

    Based on our conversations with the Principal Investigators of the study and our past experience evaluating RCTs, we believe it would be prohibitively difficult to conduct a study capable of detecting a very small (e.g., 1%) effect of masks on COVID-19. We have not conducted a formal analysis.

  • 22
    • We received information on the June survey from Professor Jason Abaluck, Associate Professor of Economics, Yale School of Management, email to GiveWell, January 28, 2021 (unpublished)
    • In April, the study authors conducted a telephone survey in Bangladesh in which 97% of people said that they regularly wore a mask outdoors. "We surveyed 2,500 people in both urban and rural areas throughout Bangladesh by phone on 22 April 2020: 97% reported owning a mask, with near-uniform adoption across every region of the country." Abaluck and Mobarak 2020

  • 23

    The trial is scheduled to begin on August 9, with endline data collection scheduled for 12 weeks later. Masks RCT Timeline Spreadsheet, 2020 (unpublished).

    Our understanding, based on multiple conversations with the Principal Investigators of the study, is that preliminary results will be available shortly after endline data collection is completed.

  • 24

    See our blog post for more on RCT study design.

  • 25

    "It is not always ethical to use a placebo. If there is already a standard treatment or method of care, it would be unethical to withhold it. A new treatment should be compared to the standard therapy rather than to a placebo." LaMorte 2020

  • 26
    • The ability to measure the causal effect of masks comes from the difference in mask-wearing between the intervention group (which receives an intervention designed to increase mask use) and the control group (which does not receive the intervention but may wear masks as usual). We expect IPA's study to measure a significant difference in mask-wearing between the treatment and control group, due to the efficacy of the intervention, even if mask-wearing in the control group is not zero.
    • "Study participants (including those in the control groups) may decide to wear other locally available masks or face coverings besides the ones provided through this project." Masks RCT protocol, July 1, 2020 (unpublished), p. 15.

  • 27

  • 28

    See Ejemot-Nwadiaro et al. 2015, a Cochrane Collaboration systematic review.

  • 29

    See MacIntyre and Chughtai 2020.

  • 30

    "Our surveyors will communicate with individuals in local languages and will ensure participants are fully aware of all aspects of the study before consenting to participate." Masks RCT protocol, July 1, 2020 (unpublished), p. 10.

  • 31

    "After an extensive review of the literature on fabrics for low-cost masks and testing at Stanford University, we have selected to use a mask that has inner and outer layer made of Pellon 931 polyester fusible interface ironed onto interlocking knit and a middle layer of interlocking knit. The fabric will be sewn into a mask with a flat front panel that bunches near the ears and is affixed to the head with ear straps and a metal wire nose brace. The mask fully covers the nose and mouth even with animated talking. The mask is comfortable and does not hold hot air over the face, which discourages use. We estimate this mask is approximately 60% efficient at blocking particles 300 nm in size at a flow velocity of 10 cm/s (the particle size and approximate flow velocity specified by the U.S. National Institute of Occupational Safety and Health). This compares to bandanas that are <10% effective at a similar flow velocity and surgical masks which have filtration efficacies ranging from 40-70%." Masks RCT Proposal, July 6, 2020, Pgs. 3-4.

  • 32
    • Professor Mushfiq Mobarak, Professor of Economics, Yale University; Professor Jason Abaluck, Associate Professor of Economics, Yale School of Management, conversation with GiveWell, July 24, 2020 (unpublished)
    • See previous footnote for details on surgical mask specifications.

  • 33

    Statistically significant = p < 0.05. See more on p-values and statistical significance in our blog post.

  • 34

    The COVID-19 measure is currently intended to refer to the combination of respiratory disease symptoms and a positive serology test: "In the mouza experiment, serological tests will be conducted 12 weeks after baseline for individuals who reported respiratory disease symptoms during our intervention. We will follow-up with these individuals using household interviews and conduct serology tests using blood spots obtained from finger pricks. The number of positive serology tests in this population will tell us rates of “symptomatic COVID-19”." Masks RCT Proposal, July 6, 2020, Pg. 9. If this changes, then this prediction refers to the primary outcome involving an objective test.

  • 35

    For a more detailed description of the planned communications activities, see IPA, Proposal for communications support on masks RCT, 2020.

  • 36

    Masks RCT additional budget request, 2020