Update
After reviewing IDinsight’s revised research ethics and publications policies, we confirmed that its updated policies address the concerns raised in this inquiry and establish a foundation to avoid similar issues in the future.
In December 2020, we advised that Open Philanthropy release the remaining funding for IDinsight’s work on the main RCT of New Incentives. We had asked Open Philanthropy to hold this funding pending completion of our review.
At the time of this review (December 2020), we also updated our own grant agreement to require grantees to abide by standards for ethical research in the grantee’s country of registration as well as the standards for ethical research in the country where the research is being conducted.
Since this inquiry, GiveWell and GiveWell-funded grantees have partnered with IDinsight on various projects. For example, in August 2022, GiveWell made a grant of $71,000 to IDinsight to fund desk-based research on approaches to assessing the number of people reached by mass distribution programs. (At other times, for logistical reasons, we have structured funding to IDinsight as contracts rather than grants and therefore haven't written about these projects in our online list of grants.)
We plan to consider other grants to IDinsight in the future, and we don’t consider this inquiry to be an impediment to future grantmaking to IDinsight. As of October 2023, IDinsight has not received further official communication from NHREC or a formal resolution to the inquiry.
Updated: October 2023
Summary
This page provides information about a research ethics inquiry into a pilot study that was recommended by GiveWell and subsequently funded by Open Philanthropy. The pilot study was conducted by IDinsight to inform the operations of New Incentives, an independent organization that provides cash transfers to incentivize routine childhood immunizations in Nigeria.
IDinsight conducted two studies of New Incentives' program:
- A small pilot study in Anambra, Akwa Ibom, and Nasarawa States in Nigeria.
- A large randomized controlled trial (RCT) in Katsina, Jigawa, and Zamfara States in Nigeria. This RCT (the "main RCT") informed our evaluation of New Incentives as a potential GiveWell top charity.
The questions raised in June 2020 related to the pilot study, and regarded the pilot's ethical approvals timing and the authorship of IDinsight's paper on the pilot study. Similar questions have not been raised about the main RCT. IDinsight told us that the main RCT received full ethical approvals and permissions in advance of data collection, and it shared the signed approvals it received from the National Health Research Ethics Committee, Nigeria (NHREC).
Published: November 2020
Table of Contents
What happened
On June 29, 2020, IDinsight informed us about concerns raised on social media regarding a paper it published using data from a small pilot study it conducted in Anambra, Akwa Ibom, and Nasarawa States in Nigeria. The paper described results from work to assess if measles immunization completion, achieved by New Incentives, varied with different incentive amounts. The paper was published in the PLOS ONE journal on June 18. On June 30, IDinsight received a set of questions from the NHREC.
The inquiry centers on questions about whether IDinsight had proper ethical approvals prior to starting the pilot study and whether IDinsight appropriately acknowledged its local collaborators. The NHREC has not yet issued a formal response.
IDinsight's response
IDinsight posted a public statement on June 30 and requested a retraction from the journal. The retraction was published on July 14 and included an apology from the IDinsight authors.1 IDinsight has revised its research ethics and publications policies in response to this incident. In addition, IDinsight informed us it has responded to all questions received to date from Nigerian authorities.
GiveWell's response
GiveWell is following the NHREC inquiry. We also asked to review IDinsight's revised research ethics and publications policies, which IDinsight shared in early November. Our review is to confirm that the policies address the concerns raised this summer and that they establish a foundation to avoid similar issues in the future.
We have asked Open Philanthropy to hold off on releasing the remaining tranche of funding for IDinsight's work on the main RCT of New Incentives until we have completed our review of the policies.
Finally, when making recommendations to donors for GiveWell Incubation Grants in the future, we will suggest that grant agreements include a confirmation that grantees will abide by ethical research standards in both their home and host countries, if these countries are different. If grant funding is from GiveWell directly, we will include this confirmation in our grant agreement.
GiveWell grantmaking to IDinsight beyond the RCT
We made a grant to IDinsight for its COVID-19 response work in August 2020, prior to receiving IDinsight's revised research ethics policies. The work we funded IDinsight to do related to the COVID-19 pandemic was:
- Analytical, and did not involve running RCTs or gathering new experimental evidence.
- Outside of Nigeria.
- Time-sensitive.
We made this grant because we believed the benefits of IDinsight's work to reduce the effects of COVID-19 outweighed the costs of providing funding before the NHREC had issued a formal response.
We are currently considering other potential grants to IDinsight. However, given that they are not time-sensitive, we plan to complete our review of IDinsight's revised policies before we make funding decisions. If the NHREC still has not released its recommendations when we have completed our review of IDinsight's revised policies, we will consider whether the lack of resolution should delay our grant decisions.
Relationship disclosure: GiveWell's former Managing Director Neil Buddy Shah was CEO of IDinsight until May 31, 2020. In 2016, as IDinsight CEO, Buddy was involved in setting up the IDinsight-GiveWell embedded team, in which IDinsight would provide analytical and evaluation services to improve the monitoring of GiveWell top charities and to support the development of future top charities—which, in 2017, came to include work with New Incentives. After helping to conceptualize and structure the team's overarching work arrangement, Buddy had no managerial or direct work responsibility on the New Incentives project. Buddy served as GiveWell's Managing Director from July 2020 to April 2022. He remains on IDinsight's Board of Directors.
- 1
"The authors apologize for omitting acknowledgment of key contributors and for not working more closely with the Nigerian research community to co-produce this study. We also apologize for an error in judgement regarding the timing of the ethical application. At the time of retraction, the authors are working with the National Health Research Ethics Committee of Nigeria to resolve this issue." Brownstone et al. 2020, p. 2.