Precision Development (Mobile-based Agricultural Advice)

Published: November 2020

This is an interim intervention report. We have spent limited time to form an initial view of this program and, at this point, our views are preliminary. We plan to consider undertaking additional work on this program in the future.

What is the problem?

In low-income countries, employment is concentrated in the agricultural sector,1 and the poorest households are more likely than others to work in agriculture, according to data compiled and reported by the World Bank.2 Improvements in agricultural productivity may therefore have the potential to substantially raise incomes among poor households.

Agricultural researchers and policymakers believe many households are not using optimal farming practices (such as use of chemical fertilizer or improved seed varieties) and that improved use of these practices has the potential to raise income for farm households.3

Researchers have proposed many potential explanations for lack of use of improved practices.4 One is lack of information about certain improved practices. If information is a constraint, then providing information to farmers may lead to improved adoption of these practices. Researchers and policymakers have suggested mobile phones can provide a way to deliver information on improved farming practices to individuals in low-income countries.5

What is the program?

Precision Agriculture for Development (PAD) provides personalized agricultural advice to farmers through their mobile phones.6 PAD provides advice through a combination of short message service (SMS) messages (i.e., short, text-based messages) and voice calls.7 This advice includes recommending farmers use specific inputs (e.g., agricultural lime to reduce soil acidity)8 and answering specific questions posed by farmers.9

PAD’s programs vary in intensity and comprehensiveness across settings. “Light-touch” programs generally provide messages to farmers about a single crop or topic. “High-touch” programs provide advice on a wider set of agronomic issues and, in addition to SMS and voice messages “pushed” to farmers, may include providing a hotline that farmers can call to receive advice from agronomists, access market prices for agricultural commodities, and listen to questions asked by other farmers.10 PAD has developed software, which is used in its program in Kenya, that leads farmers through a series of questions (and may also provide decision-support tools) to assist them in making informed decisions based on local conditions.11

Programs also differ in terms of whether they’re implemented as a stand-alone program or in conjunction with existing programs run by the government or other nonprofits. Most often, PAD develops mobile-based programs to support government programs or programs run by other organizations (PAD reaches about 55% of its farmers through these types of programs). In other cases, PAD advises partners with existing digital agricultural extension services on how to improve them (PAD reaches approximately 40% of its farmers this way). For about 5% of the farmers it serves, PAD acquires its own users and operates the product independently.12

PAD’s general process when starting a new program is to:

• source agricultural advice from agronomic experts,
• gather farmers’ mobile phone numbers,
• conduct focus groups to learn more about what content would be most helpful,
• put the advice in a format that is easy to understand,
• share with farmers via SMS or voice messages at regular intervals, and
• conduct monitoring and evaluation activities to measure impact.13

PAD has told us that it is considering ways it could provide locally-tailored information to farmers about pests and upcoming weather conditions in the future. It is also considering delivering messages through apps, Telegram, or WhatsApp in the future in areas with high smartphone usage. This would allow for sharing of video and photos, in addition to text or audio messages.14

As of the fourth quarter of 2019, PAD’s programs reached an estimated 3.5 million farmers across eight countries: India, Kenya, Rwanda, Ethiopia, Pakistan, Uganda, Bangladesh and Zambia.15

What is the evidence of effectiveness?

There have been several randomized controlled trials (RCTs) estimating the effect of mobile-phone based agricultural advice on farmer knowledge, practices, and yields. We put the highest weight on a subset of these RCTs that measure the effect on farmer yields, as summarized in the meta-analysis by Fabregas, Kremer, and Schilbach 2019. We focus on these RCTs because yield is most closely related to household income and consumption, which we model as the main outcome of PAD’s program in our cost-effectiveness analysis.

We have a skeptical prior that mobile-based agricultural programs have a limited effect on farmer behavior and yields (see below); overall, we view the evidence from Fabregas, Kremer, and Schilbach 2019 as a modest positive update on this prior view.

Fabregas, Kremer, and Schilbach 2019 find mobile-based agricultural advice yields modest increases in farm yield, though confidence intervals are wide. We view this evidence as moderate quality. We have a high level of uncertainty about the extent to which these findings will generalize, given the relatively small number of studies that test an intervention similar to PAD’s programs and given our impression that the effect of mobile-based agricultural advice is likely to depend critically on context-specific factors like crops, baseline agricultural knowledge, the effectiveness of the advice being disseminated, and local partners. We also have major questions about whether PAD’s monitoring and evaluation activities will be able to show that its programs are achieving the same effects observed in the RCTs, unless its programs are highly similar to the few programs evaluated in RCTs.

In addition to RCTs measuring effects on yield directly, we did a superficial review of studies that measure changes in lime and fertilizer use as a result of mobile-based agricultural advice, then estimated the effects on yield that would result from those changes. We guess these changes would result in a smaller effect on yield than what Fabregas, Kremer, and Schilbach 2019 find in studies directly showing an effect on yields. However, because this is a rough first estimate, we are not sure how much to weight these findings.

There may also be additional benefits (such as spillovers to other farmers) as well as offsetting negative effects (such as the possibility that agricultural advice leads to financial losses to some farmers or in some settings) that we do not currently incorporate into our cost-effectiveness analysis. We also haven’t done a comprehensive review of the literature, and it’s possible we could update our views based on reviewing additional studies.

In our cost-effectiveness analysis, we attempt to translate yield gains estimated in Fabregas, Kremer, and Schilbach 2019 into increases in household consumption. This requires us to make several assumptions related to: the extent to which gains observed in trials apply to household-level farm output, the extent to which increases in input costs offset increases in yield, what share of household income is accounted for by farm profits, and how increases in income translate to increases in consumption (details below).

Meta-analysis of RCTs on the effect of mobile-based agricultural advice on yields

Main findings

Fabregas, Kremer, and Schilbach 2019 provide meta-analyses of RCTs that measure the effect of mobile-based agricultural advice. They provide two meta-analyses: a meta-analysis of six trials reporting an effect on lime use (a recommended agricultural input) and a meta-analysis of seven trials reporting an effect on farmer yields.16 We focus on the trials reporting an effect on yields, since yields are most closely related to household income and consumption, which are the main outcomes we examine in our cost-effectiveness analysis.

Across these seven trials measuring the effect on yield, Fabregas, Kremer, and Schilbach 2019 find treated farmers had yields that were 4% higher (95% confidence interval: 0%-8%).17 These trials include those that provide advice exclusively through mobile phones, as well as trials that include an in-person component. Some trials also include video or a mobile software application, rather than purely SMS or voice messages.18 Limiting only to three trials measuring an effect on yield where mobile-based agricultural advice is delivered directly to farmers, the effect size is 4% (95% CI -3%-10%).19 These trials come from Casaburi et al. 2019 (draft paper), which reports on two trials of a program that provided SMS reminders to sugarcane farmers in Kenya, and Cole and Fernando 2018 (working paper), which reports on the effect of a voice-based service for cotton farmers in India that provided messages to farmers and also let them receive answers to questions they posed.20 We believe these trials more closely resemble the programs implemented by PAD than the other four that measure the effect on yield.

Quality of evidence

We view Fabregas, Kremer, and Schilbach 2019 as providing moderate quality evidence for the effect of programs like those implemented by PAD on farmer yields, though we have not conducted a thorough review of the meta-analysis or the studies included.

The meta-analysis relies exclusively on RCTs, and the search strategy for component studies is clearly articulated, which gives us more confidence in the results.21 However, we have a few concerns. Fabregas, Kremer, and Schilbach 2019 do not describe the quality of evidence for the studies included in their meta-analysis;22 all trials are reported in what appear to be working papers and so have not been subject to full peer review;23 and the meta-analysis does not appear to be pre-registered. We note that one of the authors, Michael Kremer, is on the board of PAD, a relationship which is disclosed in the paper.

In addition to these concerns, the wide confidence interval in the overall yield effect size we use from the meta-analysis gives us some uncertainty about the true effect size and causes us to downweight the estimated yield effect toward our skeptical prior view of lower yield effects. The confidence interval is also likely underestimated because it does not take into account variation in weather across settings.24

Generalizability

We also have a high degree of uncertainty about the extent to which the effect size found in the meta-analysis will generalize to other settings. Our impression is that the effectiveness of mobile-based agricultural advice depends critically on factors like crops grown, farmers’ current knowledge of best practices, soil conditions, weather patterns, method of delivery of advice, local partners, and type of advice given. The three trials of programs similar to PAD and measuring effect on yield in Fabregas, Kremer, and Schilbach 2019 encompass just two settings: cotton farmers in India and sugarcane farmers in Kenya. Since we guess that effects vary across contexts, showing positive effects across a large number of contexts seems especially important.

Monitoring and evaluation data from PAD’s programs could help alleviate some concerns about generalizability. For example, if we were to see evidence that PAD’s programs caused changes in farmers’ practices and those practices led to increases in yields, we would have more confidence that findings from Fabregas, Kremer, and Schilbach 2019 would generalize. Our understanding from PAD is that sufficient evidence for all steps in its theory of change is available for a small portion of PAD's programs.25 One possible next step in working with PAD would be to focus on learning more about the locations where a) there is the most evidence for the impact of the program or b) the programs implemented and specifics of the setting (e.g., crops planted, farming practice being promoted) are similar to those in Fabregas, Kremer, and Schilbach 2019.

Skeptical prior view

Our impression is that behavior is difficult to change, including for small-scale farmers, and that SMS or voice based interventions, since they are relatively light-touch, are unlikely to have a large effect. Moreover, we’d guess that there are several cases where adoption of a particular agricultural intervention might lead to increases in yields but not be profitable.26 As a result, we began our investigation of PAD with a skeptical view of the effect of mobile-based agricultural advice on farmer income and have updated this view as we interpret evidence from PAD and build our cost-effectiveness model.

To make this process more formal, we specified a “prior” (i.e., an initial estimate) for the effect of PAD on yields, based on relevant literature beyond the main results on yields from Fabregas, Kremer, and Schilbach 2019, then used Bayes’ rule to update on the prior with the results of Fabregas, Kremer, and Schilbach 2019. We also gut checked against what internal and external validity adjustments seemed appropriate, based on benchmarking against similar interventions.

Our prior is that PAD’s programs would have a roughly 1% effect on yields (90% CI -2%-4%). This prior is based on our guess that PAD would have a 3 percentage point effect on adoption of yield-improving farm practices and that those practices, once implemented, would lead to yield gains of 35% for farmers who implemented them.27

We update this prior using the results of Fabregas, Kremer, and Schilbach 2019, which find an effect of about 4%. We use a 90% CI of -3%-11%, which is wider than what is implied by the 95% CI reported in Fabregas, Kremer, and Schilbach 2019 (95% CI -3%-10%; this implies a 90% CI of -2%-9%). The wider CI accounts for additional uncertainty, due to questions about study quality and additional variation due to weather, neither of which are incorporated into the standard errors reported in the meta-analysis.

Based on this updating process, our best guess is that we should apply a 0.41 adjustment factor (i.e., downweight yield effects from Fabregas, Kremer, and Schilbach 2019 by 59%). This suggests a yield effect of 1.6%.28

However, this adjustment is necessarily subjective, so we are highly uncertain about it. Broadly, we think yield effects in the range of 0% to 4% seem plausible.

We discuss our updating process and adjustment in more detail in this write-up.

Additional evidence we have not considered

There are three pieces of additional evidence that could change our understanding of the effect of mobile-based agricultural advice but that we have not currently incorporated into our best guess about the effect of PAD’s programs:

• Our understanding is that there is a broad literature on the effect of various agricultural interventions (e.g., increased fertilizer use, increased lime use, use of improved seed varieties). This literature might help us understand the potential effects we might expect from mobile-based agricultural advice programs aimed at promoting the use of these practices.
• There also appear to be other reviews of mobile-based agricultural programs, and we have not reviewed these.29
• We also have not conducted our own search to identify other studies measuring the impact of mobile-based agricultural advice that are not included in Fabregas, Kremer, and Schilbach 2019. It’s possible there are other studies that we have not identified but that could also change our assessment of PAD.30

Potential additional benefits

Mobile-based agricultural advice may also have additional benefits. Three examples are below:

• Spillover effects. Mobile-based agricultural advice may have “spillover effects” to farmers not directly targeted by the SMS or voice messages (e.g., through farmers observing each other's plots or by sharing information with one another).
• Biofortified crops. PAD has shared with us that some of its programs include promotion of biofortified crops.31 In addition to increasing yields, these may have additional benefits through improved nutrition.
• Persistent changes in behavior. In our cost-effectiveness analysis, we model PAD as changing farmer practices in the same year farmers receive messages. It’s also possible that changes in agricultural practices caused by PAD’s programs may persist beyond one year.

We have not reviewed evidence related to these additional effects in depth or sought to model them. We include rough guesses for these effects in our cost-effectiveness analysis.32

Potential offsetting/negative effects

We have not explored the potential for offsetting or negative effects of mobile-based agricultural advice in depth. We have not reviewed evidence related to these offsetting/negative effects in depth or sought to model them. We include rough guesses for these effects in our cost-effectiveness analysis.33

How cost-effective is it?

A preliminary cost-effectiveness model for this intervention is available here. Note that our cost-effectiveness analyses are simplified models that do not take into account a number of factors. There are limitations to this kind of cost-effectiveness analysis, and we believe that cost-effectiveness estimates such as these should not be taken literally due to the significant uncertainty around them. We provide these estimates (a) for comparative purposes and (b) because working on them helps us ensure that we are thinking through as many of the relevant issues as possible.

As of November 2020, our best guess is that PAD’s program is below the range of cost-effectiveness of opportunities that we expect to direct marginal donations to (about 10x cash or higher, as of 2020).34 However, we have high uncertainty about several parameters used in our cost-effectiveness analysis, and it seems plausible our view of the intervention may change if we get more information. Based on sensitivity analyses, we guess that a range of cost-effectiveness for this program is plausible.35

We model the benefits of PAD’s programs as accruing through increases in profits (and, in turn, household consumption) as a result of farmers adopting improved farming practices that raise farm revenue more than costs. While our best guess is that PAD's effect on household income is small (approximately 0.4%), the cost per farmer of PAD’s programs is also small (we estimate $1.80 per household per year, or $0.27 per person). However, this estimate relies on a number of assumptions about which we have a high degree of uncertainty and that could change our cost-effectiveness estimates substantially if revised.

A sketch of our calculations is below:

• The meta-analysis in Fabregas, Kremer, and Schilbach 2019 finds mobile-based agricultural advice programs lead to a 4% increase in yields.36
• We view this as a modest update to our skeptical prior that this type of program has an effect of roughly 1% on yield. Our best guess of the effect after this update is 1.6%. We reach this best guess by applying an adjustment factor of .41 to Fabregas, Kremer, and Schilbach 2019’s 4% effect estimate.37
• We assume 45% of this increase in revenue is offset by increases in input costs (e.g., increased use of fertilizer, lime, or labor), so we multiply the increase in yields by 0.55 to calculate our best guess of increase in profits.38
• We assume profits from plots affected by PAD account for 45% of households’ annual income.39
• Combining these, we estimate an increase of roughly 0.4% in household income and consumption.40
• We assume a cost of $1.80 per household41 and 6.7 individuals per household,42 or roughly $0.27 per person.
• We assume increases in household income are spread equally across household members, so we estimate a 0.4% increase in income and consumption per person for each $0.27 spent per person.43

Key uncertainties in our cost-effectiveness analysis, which we would guess have a high potential to change our bottom line on PAD, are:

• Translating yield increases into increases in household income. Because the meta-analysis in Fabregas, Kremer, and Schilbach 2019 does not report effects on farm profits or household income, we have to extrapolate yield gains to profit and income gains. This requires answering the questions below, and we view all of our answers as speculative at this point.
  • To what extent do yield gains observed in trials apply to household-level farm output? We assume that the 4% yield gain reported in Fabregas, Kremer, and Schilbach 2019 applies to all plots farmed by a household. If only a subset of plots sees yield gains (e.g., because the advice is specific to crops that are planted only on certain plots), then we would want to apply an adjustment for the share of output accounted for by plots affected by changes in practices induced by PAD. This would lower cost-effectiveness.
  • How much do increases in input use offset increases in yield? Farmers may achieve higher yields by using more inputs (e.g., increased expenditures on fertilizer, lime or seed; increased labor). These should be accounted for in order to estimate increase in farm profits or farm income. We currently assume 45% of the increase in yield is offset by an increase in input costs.44 Given lack of data on farm profits in Fabregas, Kremer, and Schilbach 201945 and several challenges in estimating farm profits,46 we view this parameter as highly speculative.47
  • What share of annual household income is accounted for by farm profits on plots affected by PAD’s programs? This share would be 100% if PAD leads to changes in practices across all plots across all seasons and all income is derived from the production of farmers' own plots. However, households may generate income and consumption from other activities, such as off-farm labor. In addition, PAD’s advice may be specific to certain crops or seasons. As a rough guess, we assume households receive 45% of income from profits affected by PAD.48
  • How do increases in income translate to increases in consumption? We assume increases in income translate 1:1 to increases in consumption.
• Adjustments for study quality and generalizability of findings. We also adjust the yield effect size reported in Fabregas, Kremer, and Schilbach 2019 to reflect concerns about replicability (internal validity) and generalizability (external validity). We make downward adjustments for internal validity (mostly reflecting the wide confidence interval for the effect on farm yields) and external validity (which reflects uncertainty about the extent to which the findings from two settings will translate to new settings where PAD plans to operate in the future). This uncertainty causes us to adjust our best guess downward to reflect a skeptical prior that mobile-based agricultural programs are likely to have a relatively small effect. We adjust the effect reported in Fabregas, Kremer, and Schilbach 2019 downward by a factor of 0.41.49 However, this parameter is necessarily subjective, and we therefore view it as speculative as well.
• Cost per farmer. PAD has provided estimates of cost per farmer for its programs. While PAD estimates an average cost of $1.38 per farmer per year, the cost varies across programs.50 In addition, we guess that expanding PAD's programs to new regions may entail some fixed cost, and in the past we have found that charities' cost estimates are sometimes underestimated. As a result, our best guess of PAD’s average cost is $1.80 per farmer per year. However, we’re highly uncertain about this estimate and it is possible that we would update our assessment based on reviewing PAD’s costing in more detail.

Our cost-effectiveness analysis explores how sensitive the results are to adjustments in the above parameters, and we may prioritize further investigation of these parameters as part of continued investigation into PAD.

Is there room for more funding?

PAD reports that it can absorb up to $9 million per year for new programs that are similar to the programs we have used to develop our cost-effectiveness estimate. We discuss room for more funding further in our charity review page for PAD.

Key questions for further investigation

• How reliable is our current approach for translating yield gains into increases in profits and household income?
• How will PAD’s costs change over time? Would our estimate of PAD's costs change upon further review of PAD’s cost figures?
• What are the details of either new or current programs that have funding gaps, and how would our cost-effectiveness estimates change (based on either estimated effectiveness or cost) for those programs?
• What monitoring and evaluation data would we need to understand whether PAD’s program is achieving the same yield gains observed in the meta-analysis by Fabregas, Kremer, and Schilbach 2019?
• How likely is it that PAD will secure funding for its current and proposed new programs?
• Are there additional studies on mobile-based agricultural advice, and how should they update our best guess on the effects of these programs?
• Would our estimate of the effect of mobile-based agricultural advice programs on yield change if we conducted a more thorough review of the specific studies in the meta-analysis by Fabregas, Kremer, and Schilbach 2019?
• How much weight should we give to different studies in the meta-analysis by Fabregas, Kremer, and Schilbach 2019, given variation in program implementation and setting?

Sources

• 1

  In 2019, 59% of employment in low-income countries was in agriculture, based on International Labor Organization estimates. World Bank, World Bank Open Data: Employment in agriculture as percentage of total employment in low-income countries, International Labour Organization, ILOSTAT database (accessed November 6, 2020).

• 2

  See Castañeda et al. 2016 (working paper), p. 13, figure 6. We have not vetted the data presented in this figure or in the analysis as a whole.

• 3

  See for example:

  • “Modern inputs have expanded rapidly but have lagged in Sub-Saharan Africa.” World Bank, World Development Report 2008: Agriculture for Development, p. 52, figure 2.2.
  • “Exploitable yield gaps are especially high in medium- to high-potential areas of agriculture-based countries. Onfarm demonstrations using available 'best bet' technologies suggest a wide yield gap for maize in Sub-Saharan Africa.” World Bank, World Development Report 2008: Agriculture for Development, p. 67.
  • “While the Green Revolution benefited many farmers, the adoption of promising agricultural technologies has been far from ubiquitous, and has remained particularly low among the poor—leading to concerns that the Green Revolution may have increased both intra-and inter-regional inequalities in South Asia (Freebairn 1995). In sub-Saharan Africa, adoption of new technologies has lagged behind that of Asia. For example, by 2000, adoption of modern varieties of maize was estimated to be 17 percent of total area harvested in sub-Saharan Africa compared to 90 percent in East and South East Asia and the Pacific, and 57 percent in Latin America and the Caribbean (Gollin et al. 2005). Increased technology adoption—broadly defined to include adoption of improved agricultural practices, crop varieties, inputs, and associated products such as crop insurance—has the potential to contribute to economic growth and poverty alleviation amongst the poor, particularly in sub-Saharan Africa.” Jack 2013 (white paper), p. 2.

• 4

  One possibility is that the improved practices may not be profitable for all farmers (e.g., due to differences in soil conditions). Even if an improved practice would be profitable, farmers may face several constraints that prevent them from taking up the practice. These constraints include the following, from Jack 2013 (white paper), p. 7:

  • “Externalities – Some technologies create spillovers that affect others. If farmer decisions ignore these spillovers, then technologies that create benefits for others may not be adopted, while technologies that impose costs on others may be adopted too widely.
  • Input and output market inefficiencies – Problems with infrastructure and with supply chains, compounded by weak contracting environments, make it more costly for farmers to access input and output markets and access the benefits from technology adoption.
  • Land market inefficiencies – In settings where land tenure is weak and property rights insecure, farmers may not have an incentive to invest in beneficial technologies.
  • Labor market inefficiencies – New technologies need different types and timing of labor input. Restrictions on labor mobility and high costs in the labor market will interfere with adoption opportunities.
  • Credit market inefficiencies – Many farmers have difficulty accessing credit and face high interest rates, which prevents investment in profitable technologies. Financial decisions may be difficult for farmers without high levels of financial literacy.
  • Risk market inefficiencies – Technologies that carry a small risk of a loss may not be worth large expected gains if risks cannot be offset. Psychological issues around risky decisions further lower levels of adoption.
  • Informational inefficiencies – If an individual does not know that a technology exists, does not know about its benefits or does not know how to use it effectively, then the technology will not be adopted.”

• 5
  • “Sustainably raising agricultural productivity for the 2 billion people living in smallholder farming households in the developing world is critical for reducing world poverty and meeting rising food demand in the face of climate change. Nevertheless, most smallholder farmers have no access to science-based agricultural advice. The widespread adoption of basic mobile phone technology presents opportunities to improve upon existing in-person agricultural extension efforts that are expensive and fraught with accountability problems.” Fabregas, Kremer, and Schilbach 2019, p. 1.
  • “Information and communication technology (ICT) and digital tools are fundamentally transforming the operating environment for agricultural knowledge and information systems. These technologies and tools can expand access to information and knowledge, and promote communication and cooperation among the actors in agriculture. Mobile phones in particular can drive participatory communication, including communication with those on the margins of traditional research-extension processes, and phones are often the key instruments enabling organizations to deliver services to larger numbers of rural people than they could reach before. ICT is also integral to the business models of the public and private 'info-mediaries' and 'information brokers'—such as extension agents, consultants, and companies contracting farmers—that are emerging to broker advice, knowledge, collaboration, and interaction among groups and communities throughout the agricultural sector. All of these developments offer opportunities to significantly enhance the effectiveness and reach of agricultural research, extension and advisory services, and learning programs, as well as opportunities for profound and transformational changes in how such programs are structured.” World Bank, ICT in Agriculture, 2017, pp. 127-128.

• 6

  “PAD is a non-profit organization with a mission to support smallholder farmers in developing countries by providing customized information and services that increase productivity, profitability, and environmental sustainability. We are establishing a new model for agricultural extension: reaching farmers with personalized agricultural advice through their mobile phones.” PAD, "Our model"

• 7

  “PAD aims to improve the livelihoods of farmers in developing countries by providing them with advice about evidence-backed farming practices that enable them to increase their yields and net income. PAD primarily disseminates this advice through voice calls and SMS messaging.” GiveWell's non-verbatim summary of conversations with Precision Agriculture for Development, February 25, 2020 and March 5, 2020, p. 1.

• 8

  “This research revealed that farmers who received recommendations and were advised to use agricultural lime to combat local soil acidity were 10-24% more likely to experiment with the input. Farmers who received the service and were advised that they did not need lime were 25% less likely to purchase it.” PAD, "Kenya: Programs"

• 9

  “Building on these successes, we rolled-out our initial service in April 2016. The service, called Krishi Tarang—which means 'agriculture wave' or 'vibe' in Gujarati and Hindi—started with only 200 farmers, and grew to over 40,000 farmers by mid-2017. Krishi Tarang provides farmers with free, customized information in two ways: via weekly voice messages sent to a farmer’s mobile phone and a direct response to any agricultural question that a farmer logs.” PAD, "India: Programs"

• 10
  • “PAD classifies its program on a range from "light-touch" to "high-touch". PAD's India programs are its most high-touch, and its programs in Kenya and Rwanda in partnership with OAF are especially light-touch. Most of its programs fall somewhere in between these levels of intensiveness.” GiveWell's non-verbatim summary of conversations with Precision Agriculture for Development, February 25, 2020 and March 5, 2020, p. 3.
  • “For its light-touch programs, PAD focuses on identifying high-impact practices backed by agronomic evidence that are underutilized by farmers and designing a campaign to address one specific such topic.” GiveWell's non-verbatim summary of conversations with Precision Agriculture for Development, February 25, 2020 and March 5, 2020, p. 3.
  • “PAD's high-touch programs are essentially an accumulation of many light-touch programs, providing a comprehensive set of advice to farmers about all of the best practices for a particular crop.” GiveWell's non-verbatim summary of conversations with Precision Agriculture for Development, February 25, 2020 and March 5, 2020, p. 3.
  • “In India, where PAD uses an interactive voice response (IVR) system that pushes calls out to farmers, it is able to collect data on how many farmers answer each type of call and how long they listen to the call.” GiveWell's non-verbatim summary of a conversation with Precision Agriculture for Development, February 12, 2020, p. 7.
  • “For PAD India, our advisory content is disseminated through a two-way mobile phone-based platform that delivers customized content to farmers in audio form and in their local language. Farmers receive i) a weekly push call with agricultural advisory information that is customized in line with the crop cycle, each farmer’s location, and land characteristics; and ii) access to a hotline where farmers can record questions that are answered within 48 hours by our team of agronomists, with answers disseminated through pre-recorded voice calls. Farmers who dial into our hotline can also choose to listen to a library of advisory messages and, in some instances, access crop and commodity price information.” The Journey to One Million: PAD in India
  • “Q&A: Farmers can call in and ask questions, answered by expert. Farmers can listen to questions asked by other farmers.” Precision Agriculture for Development presentation June 2017

• 11
  • “In just two years, MoA-INFO – which we deliver in partnership with the Kenyan Ministry of Agriculture, Livestock, and Fisheries (MoALF) – has developed from a relatively simple one-way information advisory system, into a set of dynamic, mutually reinforcing, two-way communications tools which empower farmers to make informed agriculture choices across a range of decision points. . . . The service allows farmers to both pull content, when they are looking for information about crop production or pest control, and receive push messages containing information about farming practices that are timed to coincide with cropping schedules in different parts of the country. The service also provides farmers with Decision Support Tools (DSTs) to help farmers make evidence-based decisions. PAD has developed tools to help farmers decide on which maize seeds to plant, which fertilizers to apply, and whether pesticides are appropriate based on the scale of pest infestation. Using a farmer’s location, MoA-INFO asks the user filter questions – such as their land size, budget and preferences – and then provides them with recommendations for inputs and practices based on their responses.” Bakirdjian 2020
  • Additional information from Owen Barder, Chief Executive Officer, PAD, comments on a draft of this page, October 9, 2020 (unpublished)

• 12

  “In 2019, PAD's work reached 3.5 million farmers. This number includes:

  • those reached directly through services run by PAD (less than 5%)
  • those reached by services PAD runs for or in partnership with state governments or other actors (over 50%)
  • those reached by the services of another organization that PAD is partnering with to add value to the service (about 40%)"

  GiveWell's non-verbatim summary of a conversation with Precision Agriculture for Development, February 12, 2020, p. 1.

• 13

  “When starting a new program, PAD begins by forming partnerships with groups that have agronomic research and content about particular crops in the country, to ensure that it is leveraging the best evidence-backed content that has been validated at the trial level. PAD does not primarily develop content itself; rather, its agronomists identify and refine the best available content and develop a set of messages to deliver that content to farmers. . . . PAD then looks for a source for farmer phone numbers and data. . . . Early in its process, PAD tries to hold farmer focus groups to learn about farmers' current farming practices and their most significant needs in terms of advice and information. . . . PAD works with economists and behavioral scientists, and field tests its content with focus groups of farmers, to ensure its content is framed in an understandable and actionable way. PAD then digitizes its content for delivery. This involves distilling the material into individual messages that focus on a limited amount of content (so as not to overwhelm farmers with too much information at once), condensing those messages into e.g. one- or two-minute voice recordings or 160-character SMS messages, and setting a schedule for when in the season various pieces of content will be delivered. Before the season starts, PAD decides which messages it will send on which weeks. . . . PAD uses surveys to collect data on farmers' knowledge before and after an intervention is implemented. . . . Where possible, PAD tries to obtain administrative datasets that include data on farmer behaviors.” GiveWell's non-verbatim summary of conversations with Precision Agriculture for Development, February 25, 2020 and March 5, 2020, pp. 4-7.

• 14
  • “PAD is also doing research aimed at enhancing its impact, including research into new topics (e.g. lime, biofortified foods), new types of content (e.g. weather alerts), and new technologies for implementing its programs (currently PAD primarily uses two-way voice calls and SMS, but it is working on incorporating more advanced technologies to deliver content, such as smartphone apps and WhatsApp messaging).” GiveWell's non-verbatim summary of conversations with Precision Agriculture for Development, February 25, 2020 and March 5, 2020, p. 1.
  • Additional information from Owen Barder, Chief Executive Officer, PAD, comments on a draft of this page, October 9, 2020 (unpublished).

• 15

  “PAD is currently working in eight countries: India, Bangladesh, Kenya, Rwanda, Ethiopia, Zambia, Uganda, and Pakistan. In 2019, PAD's work reached 3.5 million farmers.” GiveWell's non-verbatim summary of a conversation with Precision Agriculture for Development, February 12, 2020, p. 1.

• 16

  See Fabregas, Kremer, and Schilbach 2019, p. 3, figure 2 and p. 4, figure 3 for summaries of these meta-analyses.

• 17

  Fabregas, Kremer, and Schilbach 2019, p. 4, figure 3, row for “Overall.”

• 18

  “Figure 3 reports on a complementary meta-analysis measuring the impact of experimentally evaluated digital agricultural extension interventions on farm yields or harvest value (unfortunately we do not have sufficient data on farm costs to estimate impacts on profits). This analysis encompasses four trials of messages delivered purely through mobile phones: two text message interventions with sugarcane farmers in Kenya and two season measures for an interactive voice response (IVR) intervention with cotton farmers in India. It also includes four studies with an in-person component: two video interventions with maize and rice farmers in Uganda implemented via in-person visits, a program providing customized information on rice cultivation to Nigerian farmers offered through extension workers, and a program in Ghana delivered by community extension workers who relied on a mobile software application.” Fabregas, Kremer, and Schilbach 2019, p. 4.

• 19

  Fabregas, Kremer, and Schilbach 2019, p. 4, figure 3, subtotal for “Delivered directly to farmers.”

• 20

  Brief descriptions of the interventions:

  • Kenya: “We collaborate with one of the largest agri-business companies in East Africa. The partner company runs a sugarcane large contract farming scheme. Farmers’ plots are mostly below one hectare. In the contract farming arrangement, the company provides inputs on credit that are recouped at harvest through payment deductions. The paper evaluates an intervention that leveraged on the growing penetration of mobile phones in the region to improve agricultural productivity, Farmers receive a set of text messages that inform them about agricultural tasks to be performed right around the time they need to complete such tasks on the plot.” Casaburi et al. 2019 (draft paper), p. 2.
  • India: “Specifically, we evaluate Avaaj Otalo (AO), a demand-driven, mobile phone-based technology that allows farmers to call a hotline, ask questions, and receive a recorded response from agricultural scientists and local extension workers. Callers can also listen to answers to questions posed by other farmers. Working with the Development Support Centre (DSC), an NGO with extensive experience in delivering agricultural extension, the research team randomly assigned toll-free access to AO to 400 households (hereafter, ‘AO group’) and to test the hypothesis that ICT would not be effective without at least some in-person element, an additional 400 households received both AO and an annual in-person extension session (hereafter, ‘AOE group’). A further 400 households served as a pure control group. To test the importance of making access to this information salient, among the 800 treated households, 500 were randomized into receiving bi-weekly reminder calls. The households were spread across 40 villages in Surendranagar district in Gujarat, India, and randomization occurred at the household level. The AO service also included weekly push content, delivering time-sensitive information such as weather forecasts and pest planning strategies directly to farmers. An important difference from prior ICT-based agricultural extension programs is that the information was exclusively delivered through voice messages as opposed to text-based approaches that may be less suited to semi-literate environments.” Cole and Fernando 2018 (working paper), pp. 1-2.

• 21

  “We focus our review on experimental studies of agricultural extension programs delivered by mobile technologies in developing countries (low-income or middle-income as classified by the World Bank), that collected information on yields or harvest value and were written in English. We employed Proquest, JSTOR, Google, and Google Scholar and used the following keywords: 'digital agriculture,' 'phone agriculture,' 'ICT agriculture' combined with 'yields' or 'output.'” Fabregas, Kremer, and Schilbach 2019, Supplementary material, p. 2.

• 22

  This concern applies to both a) the three trials focusing on programs that seem most similar to PAD’s programs, as well as b) the four additional trials reporting yield effects.

• 23

  This concern applies to both a) the three trials focusing on programs that seem most similar to PAD’s programs, as well as b) the four additional trials reporting yield effects.

• 24
  • “Another common feature of existing studies is that standard errors are computed based on sampling error exclusively. It is well-known in the econometrics literature, however, that the confidence intervals around single-period cross-sectional parameters estimates are too conservative in the presence of time-varying stochastic shocks that are common across cross sectional units and that alter the parameter estimates themselves (Andrews, 2005).” Rosenzweig and Udry 2020, p. 2.
  • “We find that in both agricultural settings, the returns to planting stage investments are very sensitive to rainfall realizations and that rainfall realizations are themselves quite variable. Therefore, agricultural profits are highly variable over time in both contexts. Our calculations of lower-bound confidence intervals based on both our parameter estimates of rainfall sensitivity and the actual distribution of rainfall, based on a long time-series of rainfall outcomes, are substantially larger than the standard confidence intervals computed from any single year realization that are based solely on population sampling variability.” Rosenzweig and Udry 2020, p. 5.

• 25

  “PAD would like to monitor all four of the components in its theory of change for all of its programs, though that is not always possible in practice. The degree of evidence PAD has regarding each of these components varies between programs. . . . PAD has especially complete monitoring at scale for its Gujarat program.” GiveWell's non-verbatim summary of conversations with Precision Agriculture for Development, February 25, 2020 and March 5, 2020, p. 8.

• 26

  For example, our impression, based on a light review of the literature on using fertilizer, is that there are some cases in which researchers have found that using more fertilizer is more profitable. However, this conclusion seems dependent on a) the specific fertilizer recommendation and b) the specific farmer, including that farmer’s soil conditions and other similar factors.

  • Duflo, Kremer, and Robinson 2008 conduct trials on the profitability of fertilizer and find that yield increases due to fertilizer do not always result in profit increases: “The Kenyan Ministry of Agriculture recommends the use of hybrid seed and fertilizer for maize, the staple crop in most of Eastern and Southern Africa. This recommendation is based on evidence from experimental farms that fertilizer and hybrid seeds increase yield from 40 percent to 100 percent. . . Our mean estimates of yield increases due to fertilizer use are in the range of the estimates found on model farms. We find that the mean rate of return to using the most profitable quantity of fertilizer we examined was 36 percent over a season, or 69.5 percent on an annualized basis. However, other levels of fertilizer use, including the combination of fertilizer plus hybrid seed recommended by the Ministry of Agriculture, are not profitable for farmers in our sample.” p. 483
  • Beaman et al. 2013 conduct an RCT in Mali. The authors find that fertilizer increased yields, but they detect a near-zero effect on profits (though the confidence interval includes the findings from Duflo, Kremer, and Robinson 2008):
    • “We use a simple field experiment to provide free fertilizer to women rice farmers in southern Mali to measure how farmers choose to use the fertilizer, what changes they make to their agricultural practices, and the profitability of this set of changes.” p. 1
    • “[W]hile the increase in inputs led to a significantly higher level of output, we find no evidence that profits increased.” p. 6
    • “The point estimate of the effect of this increased input intensity on profits is very close to zero, but the precision of the estimates is low. The confidence interval includes the estimates of the most profitable quantity of fertilizer examined by Duflo, Kremer and Robinson (2008).” p. 2

• 27

  0.03 multiplied by .35 equals 0.0105.

• 28

  0.04 multiplied by 0.41 equals 0.0164.

• 29

  See, for example, Fabregas et al. 2019 (working paper): “The evidence we present complements previous work focused on the role of information and communication technologies (ICT) on agricultural development. Existing review articles, such as Aker et al. (2016) and Nakasone et al. (2014), provide an in-depth overview of different approaches and services related to agriculture. These reviews have called for additional evidence on the effectiveness of ICT-based extension services (Nakasone et al., 2014), but have also noted that while these systems appear to increase knowledge, they 'have little to no impact on agricultural practices, production, or farm-gate prices' (Aker, 2017).” p. 3.

• 30

  For example, in a conversation PAD, Michael Kremer noted two studies that Fabregas, Kremer, and Schilbach 2019 did not include in their meta-analysis that would suggest even larger effects: “Another factor which may have resulted in a lower estimated yield gain is the fact that the meta-analysis did not include one study with larger impact estimates because the analysis of that study was not ready in time for the article to be published. Once that work has been incorporated into the meta-analysis, Professor Kremer believes that the estimate of impact may be 5% or 6%.” GiveWell's non-verbatim summary of a conversation with Precision Agriculture for Development, February 12, 2020, p. 6.

• 31

  “PAD aims to scale up programs that have proven effective (e.g. its content on lime, fall armyworm, biofortified beans and leafy vegetables)” GiveWell's non-verbatim summary of conversations with Precision Agriculture for Development, February 25, 2020 and March 5, 2020, p. 8.

• 32

  See our cost-effectiveness analysis of PAD, “Inclusion/Exclusion” sheet, “PAD” section.

• 33
  • See our cost-effectiveness analysis of PAD, “Inclusion/Exclusion” sheet, “PAD” section.
  • One possibility is that the advice may be harmful to some farmers, leading to decreases in yields and farm profit. In addition, if advice leads to an increase in farm investment (e.g., through increases in fertilizer purchases), this may make farming riskier by increasing losses in the event of poor rainfall. If these offsetting effects are present, they could cause us to decrease our best guess of the effect of PAD’s programs. There may also be general equilibrium effects that offset the program’s positive impacts (i.e., increased yields from improved practices could increase supply and lower market prices). However, we would guess that these effects are small, given the small effects on yield we estimate the program achieves.

• 34See our cost-effectiveness analysis of PAD, “PAD” sheet, “PAD vs cash” row.
  
• 35

  See our cost-effectiveness analysis of PAD, “PAD” sheet, “Sensitivity” columns.

• 36

  See our cost-effectiveness analysis of PAD, “PAD” sheet, “Percentage increase in yields per farmer” row.

• 37
  • 0.04 multiplied by 0.41 equals 0.0164.
  • See our cost-effectiveness analysis of PAD, “PAD” sheet, “Replicability and generalizability adjustment for yield effect” row.
  • See GiveWell, Replicability and generalizability adjustment for PAD, 2020 for an explanation of how we calculated an adjustment of 0.41.

• 38

  See our cost-effectiveness analysis of PAD, “PAD” sheet, “Adjustment for increased costs of inputs (to translate yield/revenue into profit)” row.

• 39

  See our cost-effectiveness analysis of PAD, “PAD” sheet, “Percentage of household consumption coming from higher farm profit induced by program” row.

• 40

  See our cost-effectiveness analysis of PAD, “PAD” sheet, “Percentage increase in household consumption” row.

• 41

  See our cost-effectiveness analysis of PAD, “PAD” sheet, “Cost per farmer” row.

• 42

  See our cost-effectiveness analysis of PAD, “PAD” sheet, “Multiplier for resource sharing within households” row.

• 43

  See our cost-effectiveness analysis, “PAD” sheet, “Percentage increase in household consumption” row.

• 44
  • See our cost-effectiveness analysis of PAD, “PAD” sheet, “Adjustment for increased costs of inputs (to translate yield/revenue into profit)” row.
  • We subtract the adjustment factor of .55 from 1 to estimate that 45% of the increase in revenue is offset by increases in input cost.

• 45

  “Figure 3 reports on a complementary meta-analysis measuring the impact of experimentally evaluated digital agricultural extension interventions on farm yields or harvest value (unfortunately we do not have sufficient data on farm costs to estimate impacts on profits).” Fabregas, Kremer, and Schilbach 2019, p. 4.

• 46

  Specifically, it’s not clear to us how to properly value inputs for which markets don’t exist (like household labor or land).

• 47

  Note that it’s also possible for this adjustment to be positive.

  • For example, suppose a farmer increases revenue from $100 to $150 by increasing fertilizer expenditure from $10 to $12. In that case, the percentage increase in revenue would be 50% (equal to ($150-$100)/$100) and percentage increase in profit would be 53% (equal to ([$150-$12]-[$100-$10])/($100-$10).
  • Arouna et al. 2020 report profit effects that are higher than yield effects. “[W]e find that households who were only given personalized advice increase their yield by around seven percent and increase their profit from rice by around 10 percent.” p. 3.

• 48

  See our cost-effectiveness analysis of PAD, “PAD” sheet, “Percentage of household consumption coming from higher farm profit induced by program” row.

• 49

  See our cost-effectiveness analysis of PAD, “PAD” sheet, “Replicability and generalizability adjustment for yield effect” row.

• 50
  • PAD, Slide presentation, 2020, slide 14 shows the variation of costs across PAD’s programs.
  • PAD estimates that it spent $1.55 per farmer in 2019. However, it also estimates that it spends $0.53 per farmer per year for current programs with room for more funding and $1.36 per farmer per year for new programs with room for more funding. PAD, Slide presentation, 2020, slide 13 and 23.
  • In August 2020, PAD shared with us an updated average per farmer cost of $1.38 per farmer per year.

• 51S. A. Cole, A. N. Fernando, “‘Mobile’izing Agricultural Advice: Technology Adoption, Diffusion and Sustainability,” Working paper (2018)