Precision Development (PxD) — Scoping Grant (March 2022)

Summary

GiveWell recommended a grant of $539,864 to Precision Development (PxD) to scope and plan a three- to five-country implementation and impact evaluation of its core agriculture program. We recommended this grant because:

  • We think there’s a good chance it will lead to us making an implementation and evaluation grant to PxD.
  • We roughly estimate that PxD’s agriculture program is within the range of cost-effectiveness of programs we would recommend funding, but we are very uncertain about the size of the effect and how it generalizes to different geographies.
  • A high-quality, large-scale evaluation of PxD’s agriculture program in three to five countries could update us substantially on the cost-effectiveness of PxD’s agriculture program in and across different geographies.
  • If we think PxD’s agriculture program is within the range of cost-effectiveness of programs we’d fund after seeing the evaluation results, this could cause us to allocate more than $50 million a year across the five countries PxD is considering.
  • The evaluation may identify ways to increase the cost-effectiveness of PxD’s agriculture program, improving the cost-effectiveness of any future grants we might make to PxD.

We have the following primary reservations about this grant:

  • There may be a risk that PxD’s core agriculture program or other components the organization is considering to test (e.g., weather forecasting) make farmers worse off. PxD plans to incorporate a risk analysis and a risk mitigation plan in its proposal.
  • We remain highly uncertain about our cost-effectiveness estimates and potential room for more funding for this program, which means we are highly uncertain about the value of information of this grant.

This grant was funded by Open Philanthropy.

Published: May 2022

Table of Contents

Background

Precision Development (PxD), formerly called Precision Agriculture for Development, provides locally customized information to people living in poverty in low- and middle-income countries via their mobile phones. PxD’s agriculture program provides smallholder farmers with evidence-based, livelihood-enhancing agricultural information (e.g., recommendations to use specific farming practices or inputs, such as seeds and fertilizers, or answers to specific questions posed by farmers).1

PxD was previously a GiveWell standout charity (charity page, intervention report), a designation which we decided to discontinue in October 2021.2 We initially deprioritized further investigation into PxD because we estimated it to be somewhat less cost-effective than our top charities, and because we were highly uncertain about the evidence base and our cost-effectiveness analysis for the program.3 We decided to revisit this program because we think there could be an opportunity to improve confidence in the evidence and update our cost-effectiveness estimate with an implementation and evaluation grant to PxD. Our cost-effectiveness bar is also somewhat lower than when we first evaluated PxD.

Planned activities during scoping grant

During its six-month scoping period (April-September),4 PxD plans to conduct two “phases” of activities:

  • Identify the specific countries, farming practices, and interventions it would like to test during the operational trial (Phase 1). PxD will do this through discussions with its global and in-country teams, literature reviews and other desk research, and phone and in-person surveys.5 It plans to identify three to five countries where it will recommend implementation and evaluation (current short list: India, Pakistan, Kenya, Nigeria, and Ethiopia). PxD also plans to explore promising interventions (e.g., weather forecasts, leaf color charts) to pilot alongside its “core” program in at least one of these settings, which we believe could enhance cost-effectiveness of the core program.6
  • Create an implementation and evaluation plan for each country and submit a proposal to GiveWell (Phase 2).7 At the end of the six-month scoping phase, PxD will provide a proposal covering implementation and evaluation across three to five countries over two to three years.8 The proposal will include discussion of research design and implementation rollout plan for each selected geography, power calculations, an intended analysis plan, plan for third-party replication, and budget.9 We also plan to discuss with PxD during the scoping phase whether and how to engage research collaborators to provide separation between the implementation team and research team.

Budget

The total cost for the scoping grant is $539,864. This breaks down as follows:

  • Staffing: $274,447. This includes 9.5 full-time equivalent employees over six months across PxD leadership and its global research team, as well as in-country personnel across the five countries.10
  • In-person and phone surveys and staff travel: $195,000.11
  • Indirect costs (15% of direct costs): $70,417.12

Case for the grant

We recommended this grant because we think that there is a good chance it will lead to us making an implementation and evaluation grant to PxD. Specifically:

  • Our preliminary best guess is that PxD’s agriculture program is within the range of cost-effectiveness of programs we would recommend funding (approximately six times as cost-effective as unconditional cash transfers13 ). As a result, if we did fund implementation with evaluation, we guess the implementation would be a cost-effective use of funding.
  • We think the evaluation has a good chance of updating our estimate of the cost-effectiveness of PxD’s agriculture program. Our current cost-effectiveness model relies on limited evidence, and we have high uncertainty about how much findings would generalize to settings where PxD would work in the future.14 By providing a high-quality, large-scale evaluation on PxD’s agriculture program in three to five countries where it could expand in the future, the evaluation could update us on both.15
  • If we think PxD is within the range of cost-effectiveness of programs we would recommend funding after the evaluation, this could cause us to allocate a moderate amount of funding to PxD. Our rough guess is that PxD’s agriculture program could absorb $54 million per year in the next five years in the five countries it currently has shortlisted for scoping.16 PxD estimates a total addressable market of 400 million smallholder farmer households in these five countries, which we estimate to cost $720 million per year to reach.17 We have not vetted these estimates.
  • The evaluation may identify ways to increase the cost-effectiveness of PxD’s agriculture program. The evaluation will pilot several approaches that we believe could possibly increase the impact of PxD’s agriculture program (e.g., weather forecasting, leaf color charts).18 If these are found to increase impact and are ultimately integrated into the program going forward, it could improve PxD's cost-effectiveness and the cost-effectiveness of future grants we make to PxD.
  • We have a positive qualitative impression of the organization. It has high interest in evaluating its impact/conducting randomized controlled trials and has standout transparency.19

Risks and reservations

  • There may be a risk that PxD’s core agriculture program or other components it's planning to test (e.g., weather forecasting) make farmers worse off. It seems possible that if the advice provided by PxD or weather forecasts are incorrect, it could lower farm profits, either on average or for a subset of farmers. Part of the purpose of the evaluation is to test for this, but there is a risk that this worsens farmers’ outcomes during the course of the implementation and evaluation grant. PxD plans to incorporate “an analysis of project and evaluation risks and a risk mitigation plan” to understand how much of a risk there is of making farmers worse off and will share this with us in its proposal.20
  • There are many details we don’t yet know about the evaluation, and it’s possible that doing a high-quality evaluation is infeasible. During scoping, our understanding is that PxD will decide on the evaluation design, sample size, timeframe, outcome measures (e.g., how to measure yields or household consumptions), other data it could collect that could inform cost-effectiveness (e.g., household size, percentage of household income impacted by PxD’s program), and other specifics of the evaluation.21 As a result of this, we may find that doing an evaluation that fits our needs isn’t feasible. We plan to mitigate this by checking in with PxD during scoping to stay in sync.
  • We also have not dug deeply into how much room for more funding there will be in settings where the results from the proposed evaluation are likely to generalize. After further investigation, we may find that room for more funding in settings where we think the results of the impact evaluation generalize is much more limited (e.g., maybe they apply only to rice farmers in one state in India) or that there are other funders lined up for many of the geographies where results generalize. We plan to look further into this as we review the implementation and evaluation proposal.
  • We haven’t estimated the cost-effectiveness of this scoping grant, and our estimate of the cost-effectiveness of PxD’s program is based on a shallow review.

Plans for follow-up

We plan to check-in with PxD during the scoping phase to help ensure the proposal aligns with what we’re looking for. We expect to receive a proposal within six months and make a funding decision for the operational trial within nine months.22

As we decide whether to make a grant for implementation and evaluation, we’ll consider the following:

  • How reliable will the evaluation be? This will be based on identification strategy (e.g., RCT vs. other approaches), outcomes measured (e.g., how will it measure yield or whether it'll be able to estimate profits and household consumption), power calculations, empirical strategy for estimating causal impacts, and other factors.
  • How generalizable will the results be to geographies where PxD reports room for more funding in the future, and how much room for more funding does PxD have in these geographies?
  • How likely is it the evaluation will identify ways to optimize cost-effectiveness (by adding components like weather forecasting)?
  • Will we be able to learn about other parameters that affect our cost-effectiveness estimates? This includes household size and percentage of household income coming from plots affected by PxD’s advice.

Internal forecasts

We think there’s a 70% chance we will provide a grant to fund implementation and evaluation of PxD’s agriculture program by the end of December 2022. Conditional on receiving the evaluation results, we think there’s a 40% chance we’ll provide a grant of $30 million or more to PxD by the end of December 2026.

Sources

Document Source
GiveWell, Charity review of PxD, 2020 Source
GiveWell, Cost-effectiveness analysis of PxD, 2020 Source
GiveWell, Interim intervention report on PxD (mobile-based agricultural advice), 2020 Source
PxD, Scoping proposal for GiveWell, March 2022 Source
  • 1

    More on our charity page.

  • 2

    See more about why we decided to discontinue the standout charity designation in this blog post.

  • 3

    More in our intervention report.

  • 4

    See "Appendix: Gantt Chart" section, Pg. 3 of PxD, Scoping proposal for GiveWell, March 2022 for a complete timeline.

  • 5

    PxD, Scoping proposal for GiveWell, March 2022

    • "This will be an organization-wide staffing effort drawing from Global and In-Country teams that will be coordinated by a project management team (PMT) over the 6 months of the scoping grant." Pg. 2.
    • "We’ve budgeted for 500 in-person surveys and 800 phone surveys in each of the five countries, at a total cost of $185,000 to conduct farmer needs assessments and validate the demand for different innovations that we’d like to test." Pg. 2.

    PxD indicated that the activities will also include literature reviews and other desk research. Precision Development (PxD), comments on a draft of this page, April 2022.

  • 6

    “In the first phase from March through July, we will establish the Project Management Team; identify 3 to 5 geographies (current short list: India, Pakistan, Kenya, Nigeria, and Ethiopia), key value chains, and farming practices that will be the focus of the implementation and impact evaluation work; and identify promising interventions (e.g. weather forecasts, agro-dealer phone book, leaf color charts, etc.) that complement PxD’s digital advisory to address barriers facing farmers. We will also collect data and conduct other due diligence to identify high impact areas and questions, and inform the experimental design.” PxD, Scoping proposal for GiveWell, March 2022, Pg. 1.

  • 7

    “In the second phase, from July to the end of September, we will combine insights and data from Phase 1 with feedback from GiveWell to develop an integrated impact evaluation and implementation plan and budget for each geography. This will include an analysis of project and evaluation risks and a risk mitigation plan.” PxD, Scoping proposal for GiveWell, March 2022, Pg. 1.

  • 8

    "We expect scoping activities to be launched by the first week of April 2022, and be completed by September 30, 2022. The main output of the scoping process will be the submission of an impact evaluation and implementation proposal covering activities in 3 to 5 countries over 2-3 years. We expect to deliver this to GiveWell by September 30, 2022." PxD, Scoping proposal for GiveWell, March 2022, Pg. 1.

  • 9
    • "Phase 2: Collation [...] Develop appropriate research designs for each geography [...] Create an implementation roll out plan in each geography [...] Conduct risk analysis and develop mitigation plan [...] Proposal writing and budget creation" PxD, Scoping proposal for GiveWell, March 2022, Pg. 3.
    • Jonathan Lehe, Chief Development Officer, PxD, email to GiveWell, March 29, 2022 (unpublished)
    • PxD expects to develop and update a pre-analysis plan throughout the initial phase of implementation, with the first draft registered on the American Economic Association's Trial Registry platform before PxD begins rolling out the treatment intervention. Precision Development (PxD), comments on a draft of this page, April 2022.

  • 10

    "Overall we expect to allocate approximately 9.5 FTE over 6 months to complete the activities outlined in the work plan above. The total cost of all staffing will be $274,447." PxD, Scoping proposal for GiveWell, March 2022, Pg. 2. See the "Staffing Plan" section, Pg. 2 of the scoping proposal for additional details.

  • 11

    "We’ve budgeted for 500 in-person surveys and 800 phone surveys in each of the five countries, at a total cost of $185,000 to conduct farmer needs assessments and validate the demand for different innovations that we’d like to test. To supervise surveys and meet stakeholders we’ve also budgeted $10,000 for domestic travel by country staff." PxD, Scoping proposal for GiveWell, March 2022, Pg. 2.

    • $185,000 + $10,000 = $195,000

  • 12

    "Indirect costs (calculated at 15% of direct costs) include accounting, legal, auditing, administrative functions, communications, organizational IT services and support, program and project development. They amount to $70,417." PxD, Scoping proposal for GiveWell, March 2022, Pg. 2.

  • 13

    We often use GiveDirectly’s unconditional cash transfers as a benchmark for comparing the cost-effectiveness of different programs. When discussing cost-effectiveness, we generally refer to the cost-effectiveness of a program in multiples of "cash." Thus, if a program is estimated to be "10x cash," this means it is estimated to be ten times as cost-effective as unconditional cash transfers.

    Note that our cost-effectiveness analyses are simplified models that do not take into account a number of factors. There are limitations to this kind of cost-effectiveness analysis, and we believe that cost-effectiveness estimates such as these should not be taken literally due to the significant uncertainty around them. We provide these estimates (a) for comparative purposes to other grants we have made or considered making, and (b) because working on them helps us ensure that we are thinking through as many of the relevant issues as possible. In addition, our process for estimating cost-effectiveness focuses on determining whether a program is cost-effective enough that it is above our threshold to consider funding; it isn't primarily intended to differentiate between values that are above that threshold.

  • 14

    More in our intervention report.

  • 15

    “The objective of this larger 2-3 year scope of work will be to generate evidence that is as generalizable as possible across multiple geographies, crops and livestock value chains, recommended farming practices, and farmer populations. We hope to achieve this high level of generalizability through the breadth of services included in the evaluations, the selection of settings that are representative of much larger farmer populations, and through detailed analysis before and after the evaluations of a variety of sub-populations within each geography.” PxD, Scoping proposal for GiveWell, March 2022, Pg. 1.

    Based on conversations with PxD, we expect it to select geographies where it would like to expand its program.

  • 16

    This is based on PxD estimating being able to reach 30 million farming households in the next 5 years and a cost of $1.80 per farming household.

  • 17

    This is based on PxD estimating a total addressable market of 400 million farming households and a cost of $1.80 per farming household.

  • 18

    “In the first phase from March through July, we will establish the Project Management Team; identify 3 to 5 geographies (current short list: India, Pakistan, Kenya, Nigeria, and Ethiopia), key value chains, and farming practices that will be the focus of the implementation and impact evaluation work; and identify promising interventions (e.g. weather forecasts, agro-dealer phone book, leaf color charts, etc.) that complement PxD’s digital advisory to address barriers facing farmers. We will also collect data and conduct other due diligence to identify high impact areas and questions, and inform the experimental design.” PxD, Scoping proposal for GiveWell, March 2022, Pg. 1.

  • 19

    More on our charity page.

  • 20

    “In the second phase, from July to the end of September, we will combine insights and data from Phase 1 with feedback from GiveWell to develop an integrated impact evaluation and implementation plan and budget for each geography. This will include an analysis of project and evaluation risks and a risk mitigation plan.” PxD, Scoping proposal for GiveWell, March 2022, Pg. 1.

  • 21
    • "Phase 2: Collation [...] Develop appropriate research designs for each geography [...] Create an implementation roll out plan in each geography [...] Conduct risk analysis and develop mitigation plan [...] Proposal writing and budget creation" PxD, Scoping proposal for GiveWell, March 2022, Pg 2.
    • Conversations with PxD, March 1, 2022, and April 1, 2022 (unpublished)

  • 22

    See "Appendix: Gantt Chart" section, Pg. 3 of PxD, Scoping proposal for GiveWell, March 2022 for a complete timeline.