Population Services International (PSI) - 2007 review

We have published a more recent review of this organization. See our most recent review of PSI.

What do they do?

PSI sells and markets a variety of products intended to combat disease, as well as (in some cases) improve nutrition and prevent unwanted pregnancies. The following table gives a relatively complete picture of PSI's activities based on 2005 expenses; links go to relevant section of our problems and solutions overview. Data was provided by PSI, using materials that cannot be posted publicly but are available upon request (see below for details).

HIV/AIDS prevention - marketing condoms and promoting abstinence, monogamy, and condom use - accounts for about 50% of PSI's expenses. Combined with marketing insecticide treated nets (ITNs), we have about 73% of PSI's budget; so we focus on these two activities in later sections of this review.

It's not practical for us to examine the rest of PSI's products in depth, but generally, these activities match with our understanding of the more cost-effective ways to help people, addressing many of the biggest problems for those in Africa: maternal and perinatal death (which PSI aims to help prevent through family planning), diarrhea (which PSI fights with oral rehydration therapy, an extremely cheap and simple way to save lives), etc. We do not have a strong understanding of the Safe Water System, which promotes products that people can add to water to make it safe; we haven't seen this strategy employed in any other context, but we do not focus on it since it is a small part of PSI's budget (for the record, PSI does have materials available gauging this strategy's impact).

PSI is a charity; the reason it charges for products and services is not to profit, but rather to distribute them more efficiently. As shown below, revenues only end up recovering around 15% of PSI's costs.

Through informal conversations with them and our own reasoning, we'd summarize the logic behind selling (as opposed to giving) as follows:

• Selling may help products get into the hands of people who will use them. White Man's Burden (review with the relevant excerpt is available here) gives the anecdote of free bednets' being used as fishnets and wedding veils. Selling materials may help prevent this phenomenon and thus distribute materials more efficiently.
• Purchasing goods and services (rather than receiving them for free) may actually make a person more likely to use and/or value them. The logic here is psychological and informal: people tend to attribute value to what they pay for, and may thus be more likely to save it and use it appropriately. (This point relates to the effect of selling on a particular individual, whereas the above point relates to the effect on distribution across individuals.)
• Selling helps reduce costs. This is not a major factor (PSI recoups about 10-15% of its costs, as shown below), but by charging people what they can easily pay, PSI may end up spending less to serve more.

On the other hand, we can see many reasons to give rather than sell, including:

• Selling may prevent products from reaching those most in need. Though the examples we've seen (detailed below) imply that PSI's pricing isn't too steep for most customers, it may be too steep for some of them.
• Selling may give people one more reason to choose short-term pleasure over long-term safety. For example, condoms already have a reputation for reducing the pleasure of sex; giving them away for free may reduce hesitation to use them, and send a stronger message that the giver is trying to help, not profit.
• Giving may have beneficial side effects, such as reducing opportunities for infection. For example, insecticide-treated bednets kill mosquitoes; distributing enough of them could significantly reduce the risk even for those who don't use nets.

The merits of selling vs. giving are often debated. We don't take a general position on which is better. Rather, we evaluate what we know about individual charities' abilities to save lives consistently and cost-effectively. We believe (as should emerge from this review as a whole) that PSI is strong on these counts.

PSI's general strategy

Across its strategies, PSI employs the following general approach (detailed more fully in Attachment B-18, Pgs 4-7):

• Strategic pricing. One example of how PSI sets its price comes from its ITN program in Madagascar, where it conducted a survey on how much people would be willing to pay for ITNs. PSI set its price at a level where close to 100% of respondents would pay (Attachment B-8, Pg 17). Though its approach in other cases isn't necessarily identical, this represents an approach committed not to maximizing profit, but rather to reaching as many people as possible while still realizing the benefits of selling (detailed immediately above).
• Promotion. PSI states an explicit strategy it uses to approach promotion. It starts with a goal - for example, preventing HIV/AIDS for a specific, high-risk target group of people in a certain area (Attachment B-18, pg 5). It then surveys the target group to assess which people practice risky behaviors (i.e., don't use a condom); identifies the significant differences between those who do and do not practice these behaviors (Attachment B-18, pg 10); and uses this information to plan its marketing campaign around the specific attitudes that correlate to healthy behavior. PSI uses two types of promotional strategies (examples of these strategies are described in TRaC Studies conducted by PSI, in Attachments B-6 - B-17, available on PSI's website):
  • Mass media. PSI advertises on TV, radio, in newspapers, and on billboards. Some examples of advertisements they use are available on their website, here.
  • Interpersonal communication (IPC). PSI trains individuals to talk to their friends/people they know about, for example, safe sex, condom use, HIV testing, etc. They also provide those they train with materials that support their ability to convince others. See below for examples of - and data on - this strategy. A general description of the strategy is also available, on PSI's website.
• Continuing monitoring and evaluation. Part of PSI's model is continued surveys and examinations to see whether its plans are working as well as possible. Attachment B-30, Pg 7 lays out PSI's organization-wide approach to monitoring, with two key components:
  • TRaC (Tracking Results Continuously) reports. These are surveys on the attitudes and behaviors that PSI seeks to change, such as condom and bednet utilization.
  • MAP (Measuring Access and Performance) reports. These are reports on whether PSI's products are sold and advertised as intended (an example is given below, in discussing the Mozambique project).

  Despite PSI's stated goal of conducting TRaC and MAP reports comprehensively, its coverage is far from complete. Attachment B-31 shows its "scorecard," summarizing what monitoring it has and hasn't done; when looking for these reports on its research website, we can only find some of them (accounting for about 10% of its total activities). This raises the question of whether there may be publication bias at work. Still, while many of our applicants state a commitment to organization-wide monitoring and evaluation, PSI is one of the few that demonstrates both a comprehensive strategy for doing so and actual execution of reports for a large portion of its activities.

Regions

PSI mostly serves Sub-Saharan Africa, but also does significant work in Southeast Asia, Central Asia, Eastern Europe and Latin America. Activities in both Sub-Saharan Africa and Southeast Asia have grown significantly in recent years, and we expect new money to go towards programs in those regions. (All information in the tables below comes from PSI's Unit Cost Analyses - see below.)

Does it work?

Most of the items PSI markets, as discussed in our problems and solutions overview, can be expected to save lives if they are both acquired and used by people who need them. In this section, we examine the evidence that PSI is making these two things happen, focusing on its largest two programs: HIV/AIDS prevention and insecticide treated net (ITN) marketing. In both cases, we find reason to believe that (a) PSI is a leading, if not the leading, supplier in the regions it serves; (b) the people served likely use its products. We conclude with reasonable, though not complete, confidence that PSI's activities accomplish their goal of saving lives. Details follow.

PSI as a major supplier

The United Nations Population Fund estimates that 2.1 billion condoms were provided by "by bilateral donors, UNFPA and social marketing organizations" to poor countries worldwide in 2004; that same year, PSI sold 850 million condoms (Attachment B-4, Pg 4). This implies that PSI alone accounted for over 1/3 of aid-based condom distribution worldwide.

Based on this observation and the examination of more specific projects (immediately below), we believe that in its regions of focus, PSI is a leading, if not the leading, supplier of the products it markets. We give two region-specific examples, using projects that are particularly well documented: Mozambique, where PSI markets condoms (and promotes safe sexual behavior) in high-risk areas, and Madagascar, where it markets insecticide-treated nets (ITNs).

Mozambique: PSI's role in supplying condoms

As Attachment B-9, pg 1 states, PSI operates in districts of Mozambique that were recommended by the Mozambique Ministry of Health as being "high risk" in terms of HIV/AIDS; it sells condoms and uses both advertising and interpersonal communication to promote safe sexual behavior (not just condom use but also abstinence and monogamy).

The table below shows the total number of condoms sold, the estimated number of urban males aged 15-49 in Mozambique as a whole, and the implied number of "condoms per [relevant] person" sold by PSI from 1997-2006. Data on condoms sold comes from Unit Cost Analyses and Sales Reports (see below). Population and demographic information comes from Attachment B-21.

Note that we are dividing condoms sold by the total population of urban males aged 15-49 - including those who are outside PSI's districts of focus and those who don't use condoms at all. If we had the data to adjust for these factors, we would guess it would mean a substantially higher "condoms per person" figure. Knowing how understated these numbers are, we find them to provide some evidence that PSI is a highly significant provider of condoms in its areas of focus.

Further evidence comes from PSI's direct surveys of condom coverage:

• PSI conducted a sample of 21 random zones in its areas of focus, and found that PSI condoms were available in at least 2 outlets in 17 of them (Attachment B-22, pg 12); by contrast, it appears (though PSI doesn't explicitly point to this) that non-PSI condoms were only available in 1 of the 21 zones surveyed (Attachment B-22, pg 12). This survey was conducted as a check on outlets' compliance with PSI's policies; it also found that all outlets that sold PSI condoms sold them for the recommended price, though they were often not as visible as desired and not accompanied by the promotional materials that PSI encouraged (Attachment B-22, pg 12).
• In 2005, PSI measured condom coverage in the Beira district, the 2nd largest urban district in Mozambique (Attachment B-22, pg 6). It found that condoms were sold in 11 of the 19 residential neighborhoods it evaluated, while non-PSI condoms were only available in one (Attachment B-22, pg 8). As in the survey above, all outlets that sold PSI condoms sold them for the recommended price, though they were often not as visible as desired and not accompanied by the promotional materials that PSI encouraged (Attachment B-22, pg 12).

From this, we take that PSI has sold a large number of condoms - and likely has been a leading supplier of them - in its regions of focus in Mozambique.

Madagascar: PSI's role in supplying insecticide treated nets (ITNs)

As Attachment B-8 details, PSI markets insecticide-treated nets (ITNs) in Madagascar. It also promotes ITN use through television and radio ads, as well as mobile video units (Attachment B-8, Pg 7); advertising specifically targets pregnant women and caregivers to children under 5 (Attachment B-8, pg 1), the people who are generally most at risk from malaria.

The following table is taken from PSI's Unit Cost Analyses and Sales Reports (see below).

According to the World Health Organization, long-lasting ITNs and Olysets - 50% of the ITNs PSI sells (see Attachment B-9, Pg 28) - last between 4-5 years without needing to be retreated by insecticide. (Regular ITNs last the same amount of time but require retreatment.) In the last three years, PSI has sold 1.7 million ITNs, implying about 1.7 million direct or indirect customers - in a country with a total population of 20 million people, not all of whom are at risk from malaria or interested in owning an ITN.

We did a rough estimate of how many people in Madagascar actually gained ownership of an ITN over this time period, using PSI's survey data from Attachment A-12. This data estimates that 45% of households owned an ITN in 2006, up from 22% in 2004; combined with PSI's estimates of the total number of households and likely ITNs per household (Attachment A-11, Pg 42), we come up with about 3 million ITNs owned in 2006, vs. about 1.5 million owned in 2004. Even if all people who owned an ITN replaced it over this period, that implies that PSI sold about half of all the ITNs sold in Madagascar.

Utilization of PSI's products

It isn't enough to sell materials; relevant people must use them in order for PSI's activities to be saving lives. As discussed above, part of PSI's approach is systematic use of surveys to track changes in behavior; not all of the surveys it refers to are available, but from what we've seen, we have reason to believe that PSI's products are being used. We first look at Mozambique and Madagascar, the two countries described above, and then give a general overview of the other survey data we've seen.

Mozambique: changes in condom use

PSI's survey data from Mozambique indicate large changes in condom utilization in the areas it works in - much larger than across Africa as a whole. Given PSI's large role in these areas (discussed above), we're inclined to believe that PSI was largely responsible for increased condom use in high-risk areas, and therefore saved lives from sexually transmitted diseases. We don't know whether it also increased the practice of other safe behaviors (abstinence and monogamy). Details follow.

The table below summarizes PSI surveys on condom use in Mozambique from 1996, 2001 and 2004, according to PSI surveys. Between 340 and 830 people responded to each question. PSI did not provide the exact wording of the questions asked. (This data is taken from Attachment B-9, Pg 6 (table) and Pg 22 (chart)).

We believe that it is somewhat dangerous to trust survey data about sexual activity, particularly questions about "consistent condom use" (the more specific question, "Did you use a condom in your last sexual encounter?" is less vague and easier to answer concretely). However, across the board the surveys imply large increases in condom use.

We examined the USAID Demographic and Health Surveys to get some sense of how much of this change was simply the result of the general environment (i.e., condom use increasing across Africa as a whole). While these surveys do show that condom use increased across Africa, the changes are not nearly on the scale noted above. Data from these surveys is taken from 28 countries, 16 of which offered condom use data for 2 or more years, and is available at the HIV/AIDS Survey Indicators Database.

Men reporting condom use at last sexual activity:

Men reporting condom use at last higher-risk sexual activity:

PSI's regions of focus, drawn from a smaller but still significant sample, show changes closer to 40%. This difference could still reflect many factors independent of PSI, and is not conclusive. However, we believe that this comparison gives some reason to believe that PSI's large-scale activities impacted behavior, through some combination of changing attitudes and making condoms more available.

Madagascar: utilization of insecticide treated nets (ITNs)

PSI conducted surveys in both 2004 and 2006 to assess utilization of ITNs in Madagascar. The surveys selected a random set of households across all the "Fokontanys" (districts in Madagascar) where PSI worked (methodology in Attachment B-8, pg 25) and questioned pregnant women and caregivers to children under five. The sample size was 1367 in 2004 and 1553 in 2006.

Note that this data implies that the change in ownership of nets was dominated by a change in insecticide-treated nets (ITNs), the kind PSI sells, while ownership of untreated nets remained relatively constant. This is consistent with (though again, not implying) PSI's leading role in the change in ITN ownership.

We would guess that surveying people about last night's activities, as PSI did, likely provides more accurate data than surveying them about their habits would, since the former is less vague and easier to recall. Having slept under a bednet last night is not the same as regularly sleeping under a bednet, but as we explain in a later section, using different possible interpretations of this number has virtually no effect on our estimate of the number of lives saved.

The larger point is that the above data suggests that owners of ITNs in Madagascar largely use them, and that utilization did not fall as ownership rose - implying that some combination of PSI's selling and promotion resulted in more people's using ITNs.

More data on PSI's interpersonal communication strategy

Aside from looking in depth at the two regions above, we also examined several available evaluations of PSI's interpersonal communication strategy (IPC), which consists of small-group discussions aimed at changing behavior. We find the evidence for this strategy's effectiveness to be moderately positive but not strongly convincing.

We found seven PSI reports that directly studied the IPC strategy and surveyed people's reports of condom use over time. These reports, Attachments B-10 through B-16, are summarized below.

In PSI's favor, 6 out of 7 of these studies show large changes with large sample sizes (and the exception is the study with the smallest sample size). On the other hand, these results don't address the question of whether PSI was mostly responsible for this change, or whether it was more due to broad changes independent of PSI.

Answering this question would be difficult, especially because PSI aims to reach as many people as possible with its promotion campaigns, and presumably cannot ensure that any "comparison group" is completely unexposed to its strategies. In the one case where PSI did attempt to use a comparison group (the 2005 Romania study), the comparison group (people without explicit exposure to PSI's strategy) saw a similar improvement to the people PSI targeted, although it's questionable how comparable these two groups really were (for example, the comparison group had twice as high a proportion of married people - see Attachment B-14, pg 5).

Weighing the evidence

Selling life-saving materials en masse is, by nature, both likely to be cost-effective if it works (more on this below) and difficult to measure with great confidence or precision. There are many concerns about using the evidence above to conclude that PSI's activities are effective:

• Changes in survey responses don't necessarily reflect actual changes in behavior, although we generally feel that the questions are specific and concrete enough to make this a relatively minor concern.
• All of the reports we've seen are internal reports (i.e., PSI - not an external evaluator - conducted them). This raises the question of whether the reports we've seen are representative; it's possible that reports with more negative conclusions are withheld from publication, and we have no way of knowing this for sure. This is a concern with any nonprofit organization that does not regularly employ an external evaluator (including all of our other finalists). The concern is mitigated somewhat by the fact that some reports (including the one detailed in Attachment B-6) are critical of the program and find that it had no impact. PSI has also recently informed us that it does have some externally conducted evaluation reports available, but we haven't had a chance to review them.
• Even if PSI does not purposefully withhold less favorable reports, there may be a relationship between the countries in which it conducts good evaluations and the countries in which it performs well: certain countries may be better environments to work in and thus simpler places to carry out all these activities; or the relevant staff might be better in these countries. (This concern also applies to all our other finalists, though it is less serious for PIH and Interplast since their activities are so simple and straightforward as to seemingly require less confirmation from reports.)
• Neither PSI's sales figures nor its survey results conclusively demonstrate an impact beyond what would have happened without PSI's intervention. It is possible that PSI simply uses its subsidized prices to outcompete more expensive sellers of similar materials, and ends up reducing people's costs but not increasing their ownership or utilization of these materials. A similar concern applies to all aid projects, including all of our finalists: people served does not equal lives affected, and it is difficult to attribute a change to the activities of a particular organization. In PSI's case, many of the observations above mitigate this concern somewhat (for example, the change in reported condom use for PSI's areas vs. other countries; the doubling of ITN ownership in Madagascar, which occurred without a correspondingly drastic economic boom).