Center for Effective Global Action at UC Berkeley — Scoping RCTs for Long-Term Follow-Ups

Published: September 2017

Note: this page summarizes the rationale behind a GiveWell Incubation Grant to the Center for Effective Global Action (CEGA) at the University of California, Berkeley. CEGA staff reviewed this page prior to publication.

The organization

The Center for Effective Global Action (CEGA) at the University of California, Berkeley is a global development research center with a network of researchers anchored at the University of California, Stanford University, and the University of Washington.1

The case for the funding

Randomized controlled trials (RCTs) play an important role in our decisions about which giving opportunities to recommend, often providing important evidence about the effectiveness of an intervention.2 However, many RCTs have relatively short follow-up periods—often between one and five years—in which the impact of the studied intervention is measured. Oftentimes, there may be important long-term effects of interventions that are not assessed. For example, for many education-related programs, it may be valuable to look at how incomes change for beneficiaries over the long term. At present, relatively few studies attempt to estimate the long-term effects of many health and development interventions.

Using this grant, CEGA will aim to identify promising RCTs conducted 10+ years ago that could be followed up on to assess the long-term impact of specific programs. If CEGA identifies such RCTs, we could consider funding the follow-up studies to better understand if a particular intervention had important long-term effects. These studies could affect our views about global health and development interventions and potentially alter our charity recommendations and/or funding allocations.

We are uncertain whether CEGA will be able to identify promising RCTs that would be plausible for long-term follow-up (see below). However, if they are able to identify such studies, they could be highly valuable and would improve our understanding of the long-term effects of programs more quickly and more economically than funding an entirely new RCT.

Project details

With this funding, CEGA faculty and staff will advise postdoctoral fellows on identifying and assessing promising completed RCTs that may be suitable for long-term follow-up. They will begin with the fields of cash transfers and child health, and are open to moving into other fields where possible. The identification and assessment phase is expected to take approximately one year. Professor Ted Miguel, Faculty Director at CEGA, will be an advisor to the project.

We expect to have ongoing input on the project, with regular meetings to help prioritize which intervention areas to look at and how the research might be tailored to be most useful to our work. After the identification and assessment phase, we and other funders could potentially determine whether to fund long-term follow-ups on any identified RCTs from the list created by CEGA.

Goals for the project

The primary output we expect from this project is a list of previously completed RCTs that could be promising candidates for follow-up studies on long-term effects. This list will include information to help assess the potential value of funding follow-up studies on any of the identified RCTs, such as estimated cost, importance, and research design. We tentatively expect this list to be completed in early 2019, approximately one year after postdoctoral researchers begin working on the project.

Additional expected outputs from the project include:

• A “database” (in table or spreadsheet form) of all major trials that were considered and excluded from the list of promising candidates, along with key characteristics of those studies and rationale for why they were excluded.
• A “synthesis paper” documenting the results of the project, which will address such topics as the value of long-term follow-up studies in different contexts, the typical intervals over which data are collected, explanation of why certain studies cannot be followed up on, and the attrition rate for study participants to expect over certain periods of time.
• A “toolkit” to help investigators design studies for long-term follow-up. This toolkit will include a list of “identifiers” (e.g. national identity numbers) that investigators can use to match study participants with administrative data that already exist on income or health outcomes, and suggestions for using newer data collection tools—such as mobile or satellite sources—to track long-term outcomes.

Remaining questions and risks of the project

The main risk we perceive with this project is that CEGA may be unable to identify RCTs that are promising enough for long-term follow-up. At present, we do not know how many studies might exist with potential for follow-up, and this project may find that there are simply very few, if any, strong candidates. However, it seems possible to us that strong candidates could exist and thus that this project is highly valuable to pursue.

Internal forecasts

For this project, we are recording the following forecasts. Note that these forecasts assume two postdoctoral researchers assigned to the project by CEGA.

• 40% chance that by the end of 2019 we will fund at least one long-term follow-up study because of this project.
• 20% chance that by the end of 2019 we will fund at least two long-term follow-up studies because of this project.
• 35% chance that by the end of 2019 we renew the grant to fund further research into intervention areas where we could potentially fund long-term follow-up studies.
• 5% chance that by the end of 2020 we will fund at least five long-term follow-up studies because of this project.
• 40% chance that by the end of 2020 non-GiveWell funder(s) will fund at least three long-term follow-up studies because of this project.
• 5% chance that by the end of 2020 a long-term follow-up study funded because of this project (by us or another funder) will affect how we allocated more than $10 million in funding from 2018-2020.
• 25% chance that by the end of 2025 a long-term follow-up study funded because of this project (by us or another funder) will affect how we allocate more than $10 million in funding from 2018-2025.
• 10% chance that by the end of 2025 a long-term follow-up study funded because of this project (by us or another funder) will affect how we allocate more than $40 million in funding from 2018-2025.

Follow-up

We expect to have regular meetings with CEGA to get updates on progress and to provide feedback on the type of information to collect and intervention areas to consider. We plan to write publicly about the results of the project after its conclusion, likely about two years from now, and we will also consider publishing updates or conversation notes in the interim if we think anything is of particular public interest.

Key questions for follow-up

Our key questions for follow-up include:

• Are there long-term follow-up studies identified by this project that are worth funding?
• Does it seem valuable to fund additional research of this kind to identify more studies to potentially follow up on?
• If we fund long-term studies, do they end up affecting our view on how to allocate funds?
• Do other funders support long-term studies because of this project?

Our process

The investigation for this grant consisted mainly of conversations with CEGA and others in our network, as well as some internal analyses on existing RCTs that might make strong candidates for follow-up studies. We spent some time researching whether there might be promising studies to follow up on in nutrition and education. Our preliminary analyses suggested that there might be some promising nutrition trials focused on weight gain as well as some promising education intervention trials to follow up on, but this was not researched to the point of establishing confidence.

Sources

Document	Source
CEGA, About, 8/29/2017	Source (archive)

• 1
  • “The Center for Effective Global Action (CEGA) is a hub for research on global development, with a network of 80 academic researchers anchored at the University of California, Stanford University, and the University of Washington.” CEGA, About, 8/29/2017
  • “Our faculty affiliates use rigorous evaluations, tools from data science, and new measurement technologies to assess the impacts of large-scale social and economic development programs. CEGA brings a scientific lens to global development, integrating empirical economic analysis with expertise in agriculture, public health, education, engineering, and the environment.” CEGA, About, 8/29/2017

• 2For more on how we assess evidence, see our 2012 series on how we assess studies: Our Principles for Assessing Evidence, How We Evaluate a Study, Surveying the Research on a Topic.