Nick Otis — Forecasting Research
Published: June 2018
Note: This page summarizes the rationale behind a GiveWell Incubation Grant to Nick Otis. Nick reviewed this page prior to publication.
Summary
In May of 2018, Nick Otis received a GiveWell Incubation Grant of $10,000 to support forecasting work relevant to GiveWell's top charities. This is a relatively small grant to support research that we expect to be potentially useful for our decisionmaking.
Table of Contents
About the grantee
Nick Otis ("Nick" throughout this page) is a first-year PhD student in Health Economics at UC Berkeley who has done some forecasting research previously. We first considered funding this project after a conversation between Nick and GiveWell staff at a conference (“EA Global 2017”). Nick has also received about $20,000 in funding for this project from the Harvard Weiss Fund.
Nick asked us to share that he can be contacted at notis@berkeley.edu.
About the grant
This grant will support:
- ~$6,000: Collection of forecasts from academics ($4,500) and Mechanical Turk respondents ($1,500) on the outcomes of studies relevant to GiveWell's top charities, which may include the following:
- GiveDirectly's General Equilibrium study,
- IDinsight's randomized controlled trial (RCT) of New Incentives,
- a new RCT of No Lean Season at a larger scale, and
- UC Berkeley's KLPS-4 study on the long-term impacts of deworming.
- ~$4,000: Collection of forecasts from a population similar to GiveDirectly recipients in Kenya about the outcomes of the GiveDirectly General Equilibrium study.
Nick also received $20,000 from the Weiss Fund to collect forecasts from beneficiaries in Kenya about other studies; he hasn't received any other funding for surveying academics or Mechanical Turk respondents. We expect it to be valuable to collect forecasts from multiple groups and see which group's forecasts are most accurate.
The forecasting is set to happen within the next several months. The planned output of this project is a paper by Nick summarizing his work and results.
Because of its small size, we vetted this grant somewhat less thoroughly than usual.
Goals for the grant
We expect our grantmaking decisions would be improved by having a method for collecting external forecasts on 1) the outcomes of planned RCTs and other studies, and 2) other key questions relevant to our cost-effectiveness analyses. Our cost-effectiveness analyses are an important input to our decisionmaking and frequently involve difficult judgment calls; in the long-term, we think it could be valuable to have a system for quickly getting external input on key questions, particularly from people working in development economics and global health whom we’d expect to have useful perspectives. We also expect this project to be useful in the near-term by piloting potential forecast collection methods on studies that are relevant to our top charities.
Plans for follow-up
We plan to follow up when Nick's paper is complete (roughly a year from now). Key questions for follow-up include:
- What were respondents' forecasts on the outcomes of the key studies?
- How accurate were those forecasts?
- Did these forecasts update our views on the expected cost-effectiveness of any programs?
- Has Nick developed methods that GiveWell could use for collecting forecasts about other key parameters of our cost-effectiveness analyses and grantmaking decisions?
Internal forecasts
For this grant, we are recording the following forecasts:
| Confidence | Prediction | By Time |
|---|---|---|
| 90% | Nick produces a paper summarizing his work on this project. | End of 2019 |
| 60% | Nick collects forecasts from at least 10 academics on at least four studies. | End of 2019 |
| 65% | The academics' pooled forecast of the probability that New Incentives' intervention increases vaccine coverage by 15 percentage points differs from GiveWell's internal forecast by at least 10 percentage points (for instance, the academics give a 45% chance while we give a 60% chance). | End of 2019 |