Our reasoning on "outclassed" charities/programs

A note on this page's publication date

The content we created in 2010 appears below. This content is likely to be no longer fully accurate, both with respect to the research it presents and with respect to what it implies about our views and positions.

This page explains the reasoning behind our determinations that some charities/programs are "outclassed" by others (see our process).

Table of Contents

Charter school networks

We reviewed six national charter school networks:

  • Achievement First
  • Green Dot Public Schools
  • KIPP
  • Replications
  • The SEED Foundation
  • Uncommon Schools

We did not review each of these organizations in depth. Rather, we reviewed each organization's website and did literature searches to determine whether there are external evaluations of their programs and impact.1 We particularly looked for evaluations that were randomized controlled trials of students' academic achievement (as measured by standardized test scores), or in the absence of randomized controlled trials, attempts to use non-experimental designs to evaluate student academic achievement. (More on our preference for randomized controlled trials and other high-quality studies.)

KIPP was the only organization that had any of what we considered compelling evaluations. Details are at our full review of KIPP. Highlights:

  • KIPP retained Mathematica to conduct a randomized controlled trial of its impact, the results of which are expected to be published in the future.
  • Mathematica released a report in the summer of 2010 that provided non-randomized data on a large number of KIPP academies nationwide, attempting to evaluate the full impact of KIPP (as opposed to evaluating just one or two schools).

Early childhood care programs: NFP

We considered seven programs that consisted of home visits for parents and their young children. Of these, there is substantially stronger evidence for the impact of the Nurse Family Partnership program than for the other programs.

  • Nurse-Family Partnership: For full details, see our report on the Nurse Family Partnership program. In short, this program has been studied in three randomized controlled trials in which families have been followed for between 4 and 19 years. The studies make the strongest case of any U.S. equality of opportunity program we have seen that the program is improving the lives of the people served.
  • Healthy Steps for Young Children: The Healthy Steps for Young Children program provides specialized case managers (nurses, nurse practitioners, early childhood educators, and social workers) within pediatric practices to newborns. Case managers provide home visits and other support.2 The randomized controlled trial that studied the program, unlike the studies of the Nurse Family Partnership program, had limited follow up times.3 In addition, while participating families reported receiving higher quality pediatric services, there was a statistically significant difference between the participating and non-participating families for only one other outcome (using covers on electrical outlets).4
  • Home Instruction Program for Preschool Youngsters (HIPPY): The HIPPY program provides home visits to 4- and 5-year-old children with the aim of promoting school-readiness.5 The randomized controlled trial that studied the program, unlike the studies of the Nurse Family Partnership program, had a limited follow up time of one year.6 In addition, the evaluation looked at two cohorts of children and statistically significant program effects were only found in one of the cohorts.7
  • Nurturing Program for Teenage Parents and Their Families: The Nurturing Program for Teenage Parents and Their Families program provides home visits to adolescent mothers with the aim of improving their mental health and parenting behaviors. The randomized controlled trial that studied the program, unlike the studies of the Nurse Family Partnership program, had a limited follow up time of 15 months.8 The evaluation found statistically significant differences between the treatment and control groups on some of the indicators it measured, but the precise meaning and significance of these indicators is unclear to us and we have some concern that the difference may have reflected survey psychology rather than tangible changes in participants' lives.9
  • Parents as Teachers: The Parents as Teachers program provides home visits to parents of young children with the aim of improving parenting practices and school readiness, preventing abuse, and detecting development delays and health issues.10 The program was studied in a number of evaluations, with the longest follow up time being 3 years.11 This three-year study did not find statistically significant differences between treatment and control groups.12
  • Project CARE: The Project CARE program provides either (a) a combination of educational daycare and home visits by teachers, social workers, or nurses, or (b) home visits only.13 The randomized controlled trial that studied the program, unlike the studies of the Nurse Family Partnership program, had a limited follow up time of 42 months.14 The evaluation of the program did not find significant effects of the home visit component of the program at any point, and the evidence for the impact of the combined program was mixed, finding no statistically significant effect compared to the control group at the last test (42 months).15
  • Transactional Model of Early Home Intervention: The Transactional Model of Early Home Intervention program provides one year of home visits to families with low birth-weight infants with the aim of improving the child's development.16 Project follow up time was limited (16 months after birth).17 The evaluation found mixed effects, substantially less compelling than those of the Nurse-Family Partnership program.18

Preventing substance abuse programs: Big Brothers Big Sisters

We considered six mentoring programs. Of these, the Big Brothers Big Sisters program has the largest body of rigorous research evaluating its impact.

  • Big Brothers Big Sisters: For full details, see our review of Big Brothers Big Sisters. In short, Big Brothers Big Sisters has commissioned independent randomized controlled trials on both of its main programs. The studies found statistically significant effects of one of the programs at 18 months (including participants' reports of illegal drug use), while results from the other study were less encouraging.19
  • Brief Strategic Family Therapy: The Brief Strategic Family Therapy program provides six months of weekly family therapy sessions to families of adolescents suspected of using drugs.20 We have not seen randomized controlled trial data on the impact of the program on participants' drug usage (only data on treatment group drug usage before and after the program).21
  • Creating Lasting Connections: The Creating Lasting Connections program consists of organizing Church Advocacy Teams, and providing parent and youth training sessions and case management for at-risk adolescents with the aim of preventing substance abuse.22 There was no evidence in the evaluation that the program reduced substance abuse.23
  • Friendly PEERsuasion: The Friendly PEERsuasion program provides about 25 group discussion sessions to adolescent girls with the aim of preventing substance abuse.24 The evaluation of the program only reported outcomes for immediately after program end and only found sub-group effects.25
  • Keep a Clear Mind: The Keep a Clear Mind program provides 4 weekly at-home activities for children and parents with the aim of preventing drug abuse.26 The evaluation of the program followed participants for only two weeks after project end, did not find significant differences in intention to use substances, and does not seem to have measured reported substance use.27
  • Sembrando Salud: The Sembrando Salud program provides 8 small-group discussion sessions to low-income Hispanic adolescents with the aim of preventing alcohol and tobacco use.28 The randomized controlled trial of the program followed participants for only 2 months after project completion and did not measure tobacco or alcohol use.29

Employment-focused education for students: National Academy Foundation

We considered two job skills programs for adolescents/young adults. The Career Academies program (implemented by the National Academy Foundation) found statistically significant effects, while the JobStart program did not.

  • Career Academies (National Academy Foundation): For full details, see our report on the Career Academies program. In short, Career Academies are small high schools that combine vocational/technical training with a focus on internships and work experiences aiming to improve students' job prospects. A rigorous study of the program found participants experienced more than a 10% increase in annual earnings, an effect that persisted as of eight years after high school graduation (the most recent follow up).
  • JobStart: The JobStart program aims to provide academic and occupational skills to individuals aged 17-21.30 Although JobStart focuses on a slightly different demographic than the Career Academies program (individuals aged 17-21 who are not in school as opposed to slightly younger students, currently in school),31 the program approaches and intended effects are extremely similar. The JobStart program did not identify statistically significant impacts on earnings at its two-year (or later) follow ups.

Note that JobStart did have a large (and statistically significant) effect on GED attainment.32

Reading assistance: Experience Corps

We considered two reading assistance programs. Of these, the Experience Corps program has the largest body of rigorous research evaluating its impact.

  • Experience Corps: For full details, see our report on the Experience Corps program. In short, the Experience Corps program provides adult tutors (generally age 55+) to struggling elementary schools students. The program has been the subject of two randomized controlled trials, one of which we find to be a high-quality study. The study assessed students' reading ability at the beginning of the school year and again at the end of the school year and found a statistically significant effect on one of four measures: "grade-specific reading."
  • Ready to Learn: Ready to Learn is a program administered by pre-school teachers to students to prepare students for reading.33 The Ready to Learn evaluation was relatively small (145 students)34 and, because it was a pre-school focused program, it did not directly measuring reading ability.35 Given that the ultimate aim of reading assistance programs is to improve students' reading ability, we prefer programs with evidence of impact on that outcome, something Experience Corps provides.

Improving behavior in children: Invest in Kids

We considered two programs aiming to improve behavior among young children. Of these, the Incredible Years program, run by the charity Invest in Kids, has the largest body of rigorous research evaluating its impact.

  • Incredible Years (run by the charity Invest in Kids): For full details, see our report on the Incredible Years program. In short, Incredible Years is a set of group-based training programs for parents, teachers, and children that aims to promote 0-12 year old children's social and emotional competence, and reduce their aggression and emotional problems. There have been a large number of rigorous evaluations of Incredible Years, finding positive effects on children's behavior, though we know of no long-term follow ups assessing whether these results persist throughout the child's life.
  • Families First: The Families First program provides 6 weeks of intensive home-based assistance to families of children with behavioral problems.36 The program has been evaluated in a single, small-scale randomized controlled trial with limited follow up (3 months after program completion). 37

Sources

  • 1

    In addition to reviewing the website of each organization we searched for the following terms in the following databases:

    Green Dot Public Schools:
    Searched for "Green Dot" and "Green dot public schools" in Google and Google Scholar. Searched in Social Sciences Citation Index (SSCI) for "Green Dot" and "Green Dot public schools" in topic or title.

    The SEED Foundation:
    Searched for 'SEED Foundation,' "SEED Foundation," 'SEED school,' "SEED school," 'SEED schools,' and "SEED schools" in Google, Google Scholar, and in SSCI (in SSCI topic or title for all search terms).

    Uncommon Schools:
    Searched for 'Uncommon Schools' and "Uncommon Schools" in Google, Google Scholar, and SSCI.

    Achievement First:
    Searched for 'Achievement First' and "Achievement First" in Google, Google Scholar, and SSCI.

    Replications:
    Searched for 'Replications Incorporated,' "Replications Incorporated," ''Replications, Inc.,' "Replications, Inc.," 'Replications school,' "Replications school," 'Replications schools,' and "Replications schools" in Google, Google Scholar, and SSCI.

  • 2

    "Healthy Steps provided six home visits in the first three years, a direct phone line to the Healthy Steps specialist, developmental assessments, written materials emphasizing prevention and health promotion, parent support groups, and links and referrals to community resources. The Healthy Steps program also provided enhanced well-child care, in which parents' questions and concerns about child development and behavior are addressed with a focus on positive parent-child interactions. Mother's health was also emphasized." ChildTrends, "Healthy Steps for Young Children."

  • 3

    "Of the 5565 enrolled families, 3737 (67.2%) responded to an interview at 30 to 33 months (usual care, 1716 families; Healthy Steps, 2021 families)." Minkovitz, et al. 2003, abstract.

  • 4

    "Intervention families were significantly more likely than control families to see increased effectiveness of pediatric service, e.g. receiving four or more services, receiving home visits, discussing more than six topics, and becoming informed about community resources. Intervention families were also significantly more likely than control families to report that someone in the practice went out their way for them. Intervention families were significantly more likely than control families to receive age-appropriate vaccinations and age-appropriate care at visits at 1, 2, 6, 12, and 24 months. Among parenting outcomes, the intervention families were only significantly more likely than control families to use covers on electrical outlets. All other outcomes concerning parent responses to child misbehavior, perceptions of child behavior, and promotion of child development and safety showed no significant differences between groups." ChildTrends, "Healthy Steps for Young Children."

  • 5

    "Target population: Families with 4- and 5-year-old children.
    The Home Instruction Program for Preschool Youngsters program is a home visitation program that serves families with 4- and 5-year-old children. It consists of home visits by paraprofessionals and meetings of groups of parents in order to teach parents how to carry out educational activities with their children so that they will be ready for school. Parents are given books and activity packets to use with their children, and are instructed to work for 15 minutes each day. The activity packets are designed to improve language and critical thinking skills, such as talking about a text and vocabulary building (Baker, Piotrkowski, & Brooks-Gunn, 1999)." ChildTrends, "Home Instruction Program for Preschool Youngsters (HIPPY)."

  • 6

    "Five variables were collected at the one-year follow-up: attendance, standardized reading, standardized math, grades, and classroom adaptation." ChildTrends, "Home Instruction Program for Preschool Youngsters (HIPPY)."

  • 7

    "Results: Cohort II had no significant impacts. Cohort I had four significant impacts: parental expectations of performance at post-test (d=.72), classroom adaptation at post-test (d=.76), standardized reading at one-year follow-up (d=.69), and classroom adaptation at one-year follow-up (d=.73). The following variables were not significant at post-test: number of types of literacy materials in the home, number of play materials in the home, parental expectations of child's educational attainment, Cooperative Preschool Inventory, attendance, standardized reading, and standardized math. The following variables were not significant at one-year follow-up: attendance, standardized math, grades, and academic self-image." ChildTrends, " Home Instruction Program for Preschool Youngsters (HIPPY)."

  • 8

    "Structured interviews were conducted at baseline and at 15 months' follow-up by research staff blinded to group assignment." Barnet et al. 2002, Pg 1217.

  • 9

    "Among participating adolescents, those who received home visits had significantly lower levels of parent-child dysfunction and inappropriate expectations of the child. Furthermore, although the differences were not large enough to be significant, adolescents receiving home visits had lower levels of lack of empathy and physical punishment. No differences in adolescent's reports of satisfaction with social support, parenting stress levels, or mental health were found." ChildTrends, "Nurturing Program for Teenage Parents and Their Families."

  • 10

    "The goal of the PAT program is to increase parent knowledge of early childhood development, improve parenting practices, prevent child abuse and neglect, increase children's school readiness, and detect developmental delays and health issues." ChildTrends, "Parents as Teachers (PAT)."

  • 11

    "Because of the high attrition in the other two sites, the three-year evaluation was only conducted in one site." ChildTrends, "Parents as Teachers (PAT)."

  • 12

    "With regard to child outcomes, children in the PAT program experienced several non-significant gains. Specifically, they showed gains in the self-help domain of child development at the 3-year assessment (effect size=0.26). They also were more likely to be fully immunized (effect size=0.81) and less likely to be treated for injury (effect size=0.36) at the 3-year follow up." ChildTrends, "Parents as Teachers (PAT)."

  • 13

    "The families were randomly assigned to one of three groups: Child Development Center Plus Family Education, Family Education only, or a Control group... The educational daycare component was organized according to a systematic developmental curriculum and emphasized intellectual and social development. The daycare center maintained low child/teacher ratios and employed staff with an average of seven years experience in early childcare. The family education component was intended to help parents foster both the cognitive and social development of their child. Home visitors provided families with various supports, information, advice and referrals. They emphasized problem-solving methods as well as components of the same basic child curriculum used by the daycare center. Home visitors were traditionally teachers, social workers and nurses." ChildTrends, "Project CARE."

  • 14

    "At the 30 and 42 month tests, however, the combined group differed significantly from the family education group but not from the control group." ChildTrends, "Project CARE."

  • 15

    "The family education component, whether combined with the educational daycare or alone, did not appear to have a significant impact on any of the measured outcomes...At the 12, 18, 24 and 36 month tests, the combined group differed significantly from the other two groups on cognitive development outcomes. At the 30 and 42 month tests, however, the combined group differed significantly from the family education group but not from the control group." ChildTrends, "Project CARE."

  • 16

    "The Transactional Model of Early Home Intervention is designed for mothers who give birth to low birth-weight infants. The program focuses on improving child development, which is often slower for low birth-weight infants, by improving mother-child interaction and parent responsiveness through the first year of life... The intervention lasts for one full year." ChildTrends, "Transactional Model of Early Home Intervention."

  • 17

    "The infants were assessed at 4, 8, 12, and 16 months on measures of infant development, home environment, and parent-child interactions." ChildTrends, "Transactional Model of Early Home Intervention."

  • 18

    "Up to 16 months, the parent-child interaction, or Transactional, program had a larger impact. In the second year of life, the parent-child interaction program enhanced children's security... relative to a no-treatment control group. The intervention had limited or no impacts on child temperament, parent interactive behaviors, and child interactive behaviors." ChildTrends, "Transactional Model of Early Home Intervention."

  • 19

    See our discussion of the community mentoring program and of the school mentoring program in our review of Big Brothers Big Sisters.

  • 20

    "Brief Strategic Family Therapy is a family-systems approach which is designed for adolescents with behavior or drug related problems. Through family therapy, therapists identify and change maladaptive relationship patterns within families. Family therapy sessions are also manualized and are designed to be attended weekly for up to six months." ChildTrends, "Brief Strategic Family Therapy."

  • 21

    "Results: To measure the fidelity of treatment, the researchers computed an engagement score for each participant, in the experimental condition, participants received an average level of 2.8 on the engagement scale defined above (0 to 5) while the control condition received 0.8. Participants in the experimental condition were far more likely to attend therapy sessions (92.9%) than those in the control condition (42.3%). Participants in the SSSE condition were also far more likely to reach completion of therapy (77%) compared with those in the EAU condition (25%). Treatment also had significant impacts on the subscales and total scale of the PSS. Participants were substantially less likely to abuse drugs (20%) at the conclusion of treatment, compared with at the outset of treatment (93%)." ChildTrends, "Brief Strategic Family Therapy."

  • 22

    "The program consists of two main components and runs for approximately one year. The system level component works to mobilize the community by engaging church staff and volunteers to advocate for substance abuse prevention programs, resources, and services. The program organizes Church Advocate Teams that receive approximately 8 to 10 weeks of training. Upon completion of training, teams are responsible for identifying and recruiting high-risk families in the community, assisting with program implementation and evaluation, and ultimately ensuring ongoing family participation and engagement. The client level component of CLC consists of parent and youth training modules, early intervention services, and follow-up case management." ChildTrends, "Creating Lasting Connections."

  • 23

    "Adolescent-reported bonding with mothers increased among program participants; however this effect only approached significance when compared with the control group (p.10). All other findings on adolescent outcome comparisons were nonsignificant." ChildTrends, "Creating Lasting Connections."

  • 24

    "Friendly PEERsuasion program is designed to change and prevent substance abuse in at-risk girls, grades 6-8. The program, based on the social influence and life skills model, examines the patterns, causes, and consequences of substance abuse in middle school girls. PEERsuasion encourages girls to learn how to make their own decisions, resist peer pressure, create positive peer groups, and become role models through specified program activities. Participants complete a two-phase program, which consists of 14-hour long sessions in the first phase and eight to ten half-hour sessions in the second. For both sessions, an adult leader led group discussions and hands-on, interactive activities." ChildTrends, "Friendly PEERsuasion."

  • 25

    "Of the younger participants, no girls in the treatment group reported substance abuse immediately post-treatment, compared to an 8-percentage point increase in the comparison group...Impacts for the older girls were very small." ChildTrends, "Friendly PEERsuasion."

  • 26

    "The Keep A Clear Mind (KACM) program is a four-week take-home drug education program designed for upper elementary school students. The program contains four weekly sets of activities to be completed by students and their parents together...Keep a Clear Mind is an early-intervention drug education program that includes several weekly lessons aimed at helping students develop skills to refuse the use of drugs (Werch, 1991). " ChildTrends, "Keep a Clear Mind."

  • 27

    " Surveys administered two weeks after the program showed that mothers and fathers in the treatment group reported significantly more recent and frequent communication with their children about resisting peer pressure and the effects of drug use, but there were no significant differences found between the treatment and control groups on drug-related knowledge or beliefs. In addition, students in the treatment group perceived less peer pressure to use tobacco and less peer use of alcohol, tobacco, and marijuana then students in the control group. No significant differences were found between groups on measures of self-efficacy, knowledge of substances, and intention to use alcohol, tobacco, and marijuana in the future." ChildTrends, "Keep a Clear Mind."

  • 28

    "Target population: Hispanic children ages 11 to 16 and their parents. Sembrando Salud is an alcohol and tobacco prevention program that strives to improve healthy decision-making skills among adolescents. The intervention consists of eight weekly two-hour sessions where adolescents meet in small groups... Presentations involve a variety of techniques such as group discussion, demonstrations, skills practice, role-playing and behavioral modeling." ChildTrends, "Sembrando Salud."

  • 29

    "Furthermore, the follow-up only assessed the short-term impact of SS, as it was conducted within two months of program completion. Finally, outcome measures did not capture actual tobacco or alcohol utilization or decision-making practices, two of the primary goals of the program. The evaluation did, however, specifically choose to focus on parent-child communication, due to its association with youth substance use and abuse." ChildTrends, "Sembrando Salud."

  • 30

    "Each program was required to provide — or provide access to — the following services:

    • Instruction in basic academic skills that allowed participants to progress at their own pace toward goals related to their reading, communication, and basic computational skills
    • Occupational skills training that combined theory and hands-on experience to prepare participants for jobs in high-growth occupations
    • Training-related support services, including assistance with transportation and child care, counseling, and, where possible, supports such as work-readiness training and financial incentives contingent on program performance
    • Job placement assistance to help participants find training-related jobs"

    MDRC, "JobStart."

  • 31

    "About 2,300 low-income 17- to 21-year-olds without a high school diploma or GED were judged eligible for the program." MDRC, "JobStart."

  • 32

    38.4% of men in the treatment group versus 21.7% of men in the control group and 36.8% of women in the treatment group versus 20.6% of women in the control group received their GED in the 48 months following the intervention. Cave et al., 1993, Pg 111.

  • 33

    "The Ready to Learn program focuses on three target skill areas: listening comprehension, attending skills, and social skills. These skills are taught using five strategies: modeling-coaching-cuing, positive peer reporting, student story telling, student story retelling, and encouragement council. Pre-school teachers systematically embed these strategies into daily classroom routines." ChildTrends, "Ready to Learn."

  • 34

    "145 four- and five-year old children constituted the study sample for this investigation. These children were drawn from 10 classrooms at three inner-city preschool centers." ChildTrends, "Ready to Learn."

  • 35

    "Compared with students from control classrooms, students from treatment classrooms showed greater improvement over time on understanding of story structure, teacher ratings of behavior, and observer ratings of attending behavior. Treatment students did not show significantly greater improvement than control students on the measure of listening comprehension." ChildTrends, "Ready to Learn."

  • 36

    "The Families First program is a home-based intervention for families that have difficulties managing severe behavioral problems in their children... The program focuses on teaching skills such as communication, rule-setting, and providing praise and consequences for child behaviors. The program also provides concrete services, such as helping with finances and improving community support in the schools and elsewhere...The program has an intensive curriculum; it takes place over a period of six weeks, and treatment specialists meet with families for as many as 15 hours a week." ChildTrends, "Families First."

  • 37

    "Evaluated population: 164 families with children between the ages of 3 and 17 who had been identified by their schools or juvenile courts as having severe behavior problems...The survey was administered to the families twice more over the course of the study: once immediately after the Families First intervention had been completed, and again three months later." ChildTrends, "Families First."