WAW Statistical Consulting Ltd — Development of an R Package for Bayesian Evidence Aggregation

Note: This page summarizes the rationale behind a GiveWell-recommended grant to WAW Statistical Consulting Ltd. WAW Statistical Consulting Ltd. reviewed this page prior to publication.

Published: May 2022

In March 2022, GiveWell recommended that Open Philanthropy grant $66,400 to WAW Statistical Consulting Ltd. This grant is to support Rachael Meager and Witold Więcek over two years to continue development of baggr, a Bayesian statistical package for R.1

We recommended this grant because:

• Baggr has been used to support important research in global health and development, some of which has informed GiveWell funding decisions.2
• We have a generally positive view of the usefulness of Bayesian approaches to statistical analysis.
• Dr. Meager and Dr. Więcek told us there is demand for future collaboration using baggr, but it has been challenging to raise funding to develop core features to enable these collaborations.3

We note that we have relatively little experience using baggr, and haven't prioritized deeply understanding the rationale for the additional development the grantees proposed. One of our Program Officers recommended this grant from their discretionary budget based on a one-hour conversation and a brief proposal.

Sources

• 1

  “A grant would cover development up to about v1.0, at which point baggr could be considered a mature software. We plan to do the following before reaching v1.0:

  • Develop new functions to encourage better workflow for meta-analysis, potentially with new default approach to priors; this will be in collaboration with other Bayesian statisticians who use the package and non-technical end users.
  • Implement specific models for common but non-standard aggregation problems including: ‘spike and slab’/zero-inflated data (e.g. the microcredit quantiles analysis), survival models (necessary for modeling e.g. child mortality data), network meta-analysis models (typical in comparing drugs) and difference-in-difference modeling (typical in economics quasi-experiments, natural experiments and some RCTs)
  • Develop methods for mixing individual-level data with summary-level data; this occurs when only some studies make ‘micro’ data available (a common issue in evidence synthesis) or when decision-makers have substantial information not encoded in published studies.
  • Improve integration with other R packages, with a focus on visualisations, interpretation of results and accessibility to non-Bayesian or non-technical audiences.

  The above list is aspirational and we will continue to make development decisions based on feedback we receive from users. In parallel to continuing core development of the package, we will continue to collaborate with researchers on particular use cases (for which we are not seeking any extra funding).

  A grant would also cover some aspects of maintenance: the issues page in our code repository lists another 30+ features and bugs that need work, with new ones constantly added based on user feedback.”

  Baggr project proposal, pgs. 1-2

• 2“We have been supporting researchers and implementers in conducting meta-analyses primarily for program evaluations. The following is a non-exhaustive list of 8 notable projects that have used baggr:
  • A new working paper on meta-analysis of under-5 mortality and water interventions (Michael Kremer, Witold Więcek and others), which has been used by GiveWell’s recent CEA [cost-effectiveness analysis] of DSW and ILC uses baggr to fit Bayesian models.
  • Digital Agricultural Advice (Raissa Fabregas, Michael Kremer, Frank Schilbach); we were able to provide some bespoke modelling for them on top of the standard baggr models, and this is now published in Science with baggr cited
  • Financial education in developing countries (Tim Kaiser, Annamaria Lusardi, Lukas Menkhoff, Carly J. Urban); correspondence with Tim Kaiser indicates they were dissatisfied with their current approach and are using baggr for the next version. Tim tweeted about it here.
  • Teaching at the Right Level (TARL; Noam Angrist at Young1ove, Botswana, joint with Rachael), uses baggr’s basic hierarchical models to aggregate evidence across countries, this forms the first set of deliverables on the CEDIL / DIFD grant to Young1ove, and informs Young1ove’s strategy for scaling up TARL in Botswana and Namibia.
  • BRAC Ultra Poor Graduation Program (Dean Karlan and Chris Udry, joint with Rachael, Witold and Andrew Gelman), uses baggr’s models to initially assess heterogeneity across countries in average effects; finding large heterogeneity motivates further work on individual effect estimation.
  • ‘Meta-Analysis and Public Policy: Reconciling the Evidence on Deworming’ (Michael Kremer, Ted Miguel, Witold Więcek and others; in submission) uses baggr for Bayesian robustness checks
  • MDMA for treatment of PTSD (Scott Cunningham, Priyasmita Ghosh and Rebecca Thornton), uses baggr’s basic hierarchical models to aggregate Phase 2 trial data (work in progress). Scott tweeted about it here.
  • Early prototypes of baggr’s core hierarchical models developed by Rachael Meager were used to assist with Vitamin A supplementation meta-analysis for Andrew Martin with GiveWell.”

  Baggr project proposal, pgs. 2-3

• 3
  • "Here are potential use cases. Once again we emphasise that we are not looking for funding to engage in particular collaborations but only to develop the core features and continue maintaining the package.
    • Witold is working with Development Innovation Lab at UChicago to incorporate baggr as the default choice for all meta-analyses produced by the lab, including follow up work on water interventions and child mortality.
    • Young1ove expressed interest for greater meta-regression capacity within baggr, as their focus for the future will be understanding correlates of higher treatment effects rather than absolute quantification of averages or variances across settings.
    • Eventually we hope that organizations such as GiveWell and Open Philanthropy may be able to incorporate some aspect of baggr into their workflow or general approach."
  • "The package was funded initially through LSE and subsequently by a Schmidt Futures grant, but as of January 2022 we do not have any funding for development available."

  Baggr project proposal, pg. 3