EvidenceBased Programs

1
Evidence-Based Programs

• An Introduction
• Presentation to the JWB Childrens Services
  Council Board of Directors July 19, 2007
• Starr Silver, Ph.D.
• Director of Research and Evaluation

2
Programs Funded by JWB Childrens Services Council

• Goals
• Evidence-based demonstrated success
• High return on investment
• Improve measurable objectives

3
Terms

• evidence-based
• research-based
• science-based
• blueprint programs
• model programs
• promising programs
• effective programs

4
Definition

• For a program to be considered evidence-based, it
  must have been a tested, well-designed and
  methodologically sound study, shown to produce
  significant reductions in poor outcomes or
  associated risk factors.
• (Center for Prevention, Penn State University)

5
Standards Vary
6
Strong Evaluation Designs

• Allow you to infer that the outcomes are due to
  the program
• Voluntary Pre-K children are ready for school
• Parents have improved parenting skills
• And NOT due to other factors
• --pre-K children were doing great before program
• --Parents with poor parenting skills dropped out
  of program

7
Strong Evaluation Designs

• Experimental randomly assign children to groups
• Quasi-experimental make the groups as similar as
  possible 1 gets the new service, the other gets
  the standard service

8
Did the Program Produce the Outcome?

• Consider two groups of 3 year-old children
• Group 1 attends preschool from Sept to June
• Group 2 does not attend any preschool
• We measure both groups recognition of letters
• in June.
• Group 1the children who attended preschool all
  year--scores worse than Group 2
• Therefore, the preschool didnt help!!!

9
Strong Evaluation Designs

• Experimental randomly assign children to groups
• Quasi-experimental make the groups as similar as
  possible 1 gets the new service, the other gets
  the standard service

10
(No Transcript)
11
Group Differences

• Group 1 low income children
• Group 2 high income children
• Group 1 scores worse than group 2 because group 1
  started out with higher scores.
• If you had done a pre-test in September
• High income group would have shown high scores
• Low income group would have shown low scores
• You have to do a pre-test to see whether the 2
  groups are equivalent before 1 of the groups gets
  the services.

12
Standards for judging evidence-based programs
vary

• Rand Corp -Promising Practices Network
• Ctr Study Prevention Violence Blueprints
• WA State Policy Inst Cost/Benefits
• Substance Abuse MH Administration National
  Registry of Evidence-Based Programs

13
Rand Corporation

• Promising Practices Network
• The PPN site features descriptions of evaluated
  programs that improve outcomes for children
• 3 Evidence Levels
• Proven Programs
• Promising Programs
• Proven/Promising Programs

14
Promising Practice Network Pgms

• Proven
• At least one outcome is changed by 20.
• Strong study design to identify program impacts
• At least 30 people in treatment group and
  comparison group

• Promising
• Change in outcome is more than 1.
• Design weaknesses (e.g., groups not comparable on
  pre-test or demographics)
• At least 10 people per group

15
PPN-Proven Programs

• Early Head Start
• Perry Preschool Project
• Healthy Families New York
• Incredible Years
• Life Skills Training
• Multi-Systemic Therapy
• Project ALERT
• Project STAR/Midwestern Prevention Project
• Partial list

16
Center for the Study and Prevention of Violence
Blueprints for Violence Prevention

• Focus programs that prevent or reduce violence,
  delinquency, drug use

17
Criteria

• Experimental or quasi-experimental design
• The outcomes last for at least 1 year
• A sufficient of people were measured
• Valid and reliable instruments used
• Program has been replicated
• Program has been evaluated by someone other than
  program developer
• Clear instructions exist for how to replicate
  this program

18
Blueprints Model Programs

• Midwestern Prevention Project
• Big Brothers Big Sisters of America
• Functional Family Therapy
• Life Skills Training
• Multisystemic Therapy
• Nurse-Family Partnership
• Multidimensional Treatment Foster Care
• Olweus Bullying Prevention Program
• Promoting Alternative Thinking Strategies (PATHS)
• The Incredible Years Parent, Teacher Child
  Training Series
• Project Towards No Drug Abuse

19
WA State Institute for Public PolicyBenefits
Costs Prevention Early Intervention

• High Return on Investment (ROI)
• Effective programs for juvenile offenders
• Some forms of home visiting programs
• Early childhood education for low income 3- and
  4-year-olds
• Some youth development programs

20
WA State Institute for Public PolicyBenefits
Costs Prevention Early Intervention

• Lower Returns, but Worthwhile
• Many substance use prevention programs for youth
• Poor Return
• Few programs are effective at reducing teenage
  pregnancy.
• Some prevention and early intervention programs
  very expensive and produce few benefits.

21
(No Transcript)
22
Substance Abuse Mental Health Administration

• National Registry of Evidence-Based Programs
  Example
• Multisystemic Therapy (MST) for Juvenile
  Offenders.
• Outcomes Post-treatment arrest rates Long-term
  arrest rates
• Long-term incarceration rates
  Self-reported criminal activity
• Alcohol and drug use Perceived family
  cohesion
• Peer aggression
• Keywords   Criminal/juvenile justice,  Violence
  prevention,  Experimental, 
• 6-12 (Childhood),  13-17 (Adolescent), 
  American Indian/Alaska Native,  Asian
  American,  Black or African American,  Hispanic
  or Latino, Other/unspecified,
  White,  Female,  Male,  Home,  Other
  community settings,  Outpatient, 
  Rural and/or frontier,  School, 
  Suburban,  Urban,  Proprietary

23
Cautions

• Community-based more difficult to find
• Narrow, versus more general, population
• High fidelity
• Need for outcome evaluation

24
References

• Evidence-based Programs (April 2005). Technical
  Assistance Fact Sheet Number 1. Prevention
  Research Center for the Promotion of Human
  Development. Downloaded 7/16/07 from
  http//www.prevention.psu.edu/pubs/documents/EBP_f
  actsheet.pdf.
• Rand Corp operates the Promising Practices
  Network on Children, Families and Communities.
  Rand Corporation conducts research in several
  areas, some of which relate to services for
  children. Use the search term evidence at the
  Rand Corporation website (http//www.rand.org/)
  and/or link directly to the Promising Practices
  Network on Children, Families and Communities at
  http//www.promisingpractices.net/.
• Blueprints Model Programs Overview. Center for
  the Study and Prevention of Violence.
  http//www.colorado.edu/cspv/blueprints/model/over
  view.html.
•  Benefits and Costs of Prevention and Early
  Intervention Programs for Youth(July 2004).
  Steve Aos, Roxanne Lieb, Jim Mayfield, Marna
  Miller, Annie Pennucci. 04-07-3901.Washington
  State Institute for Public Policy.
  http//www.wsipp.wa.gov/.
• National Registry of Evidence-Based Programs and
  Practices. http//www.nrepp.samhsa.gov/.