EvidenceBased Programs and Practices: What, Why and How

1
Evidence-Based Programs and PracticesWhat, Why
and How?

• Patricia San Antonio
• CSR, Incorporated

2
Outline

• What are Evidence Based Programs (EBP)?
• Reasons to Support, Promote, and Increase the use
  of EBP
• Applying EBP
• EBP for AIAN Youth and Families
• Issues in Using EBP
• Looking Ahead to Evaluating Your Program
• Resources

3
What are Evidence Based Practices/Programs?

• A program or differentiated set of program
  elements, many combinations of which are
  associated with positive outcomes
• Also called
• Best practices
• Effective programs
• Research based programs
• Model programs

Source Lipsey, 1995 Lipsey Wilson, 2000
4
OJJDPs Definition of EBP

• Programs and practices that have been shown,
  through rigorous evaluation and replication, to
  be effective at preventing or reducing juvenile
  delinquency or victimization, or related risk
  factors. Evidence based programs or practices can
  come from many valid sources (OJJDPs Model
  Programs Guide).
• Also includes practices generally recognized as
  best practice based on research literature
  and/or the degree to which it is based on a
  clear, well-articulated theory or conceptual
  framework for delinquency or victimization
  prevention and/or intervention.

5
Reasons to Support/Promote/Increase the Use of EBP

• OJJDPs authorizing legislation
• State level funders increasingly requiring it
• Reinvigorated federal focus on what works
• Current economic realities

6
Juvenile Justice and Delinquency Prevention Act
of 2002

• . . . to the extent practicable give priority
  in funding to programs and activities that are
  based on rigorous, systematic, and objective
  research that is scientifically based.

42 U.S.C. 5633 Sec. 223. (a)
7
EBP at the State Level

• A Number of State level Funders Encourage the Use
  of EBP
• South Carolina
• Pennsylvania

8
South Carolina Department of Alcohol and Other
Drug Abuse Services (DAODAS)

• South Carolina Toolkit for Evidence-Based
  Prevention Programs and Strategies
• (http//www.daodas.state.sc.us/toolkit.asp)
• July 2006- Required prevention providers to
  implement at least one evidence-based program or
  strategy selected from the Tool Kit.

9
PA Resource Center for Evidence-Based Prevention
and Intervention Programs and Practices
.

• Provide technical assistance for start-up,
  operation, sustainability of prevention or
  intervention programs proven to be effective.
• Ensure that PCCD- and DPW-funded Evidence-Based
  Program (EBP) models are implemented with
  fidelity
• Serve as a resource to the field related to the
  implementation of evidence-based programming.
• Promote proliferation of EBP by educating
  practitioners and providers about practical and
  economic benefits of EBP

10
Renewed Federal Focus on What Works
10

• Performance/Accountability a priority
• Economic Realities make this necessary
• New White House position
• Chief Performance Officer

11
Evidence based Programs and Practices Save Money

• Cost to provide every student in U.S. a model
  drug prevention program like LST is 550 million
  per year
• Current national drug control spending is
  approximately 40 billion per year
• This represents 1.5 of our current drug control
  spending

12
Applying Evidence Based Practices

• Replicate an existing Model Program
• The Program must be
• Based on a sound theory or model
• Evidence based
• Well defined
• Well documented
• Replicated with fidelity

13
Things to Look for When Reviewing Programs

• Risk and protective factors targeted by program
• Population(s) targeted in the demonstration/test
• Level of effectiveness
• Guidelines for implementation do they exist?
• Cost
• Adapted for Native American/Alaska Native youth

14
EBP That Show Effectiveness for American Indian
and Alaska Native Youth and Families

• Project Venture (Details in following
  presentation)
• American Indian Life Skills Development/Zuni
  Life Skills Development
• Strengthening Families Program
• Families and Schools Together (FAST)
• Not-On-Tobacco (N-O-T)
• Native American Prevention Project Against AIDS
  and Substance Abuse

15
American Indian Life Skills Development/Zuni
Life Skills Development

• 13-17 years old, boys and girls
• Reduced feelings of hopelessness
• Increased suicide prevention skills
• Zuni Life Skills Development curriculum
  originally for high school students in Zuni
  Pueblo. Has been adapted and replicated with
  several other tribes (American Indian Life Skills
  Development curriculum)
• Stanford University

16
Strengthening Families Program

• Ages 3-17, boys and girls
• Improved parenting skills
• Reduced conduct disorders among children
• Improved social competencies for children
• Extensively replicated and found effective for
  families of diverse backgrounds (including
  AI/NA).
• University of Utah

17
Families and Schools Together (FAST)

• Ages 4-12, boys and girls
• Displayed more positive social skills
• Lower aggressive behavior
• Higher academic performance
• One random control experiment focused on AI/NA
  youth and families. Results consistent.
• University of WI-Madison

18
Not-On-Tobacco (N-O-T) Smoking Cessation Program

• Ages 14-19, boys and girls
• More likely to quit smoking
• Increase in positive attitudes, behaviors
  (managing stress, feel better about themselves)
• Curriculum was adapted for Native American youth
• West Virginia University

19
Native American Prevention Project Against AIDS
and Substance Abuse

• 12-18 years old, boys and girls
• Reduced incidence/support of risky sexual
  behavior
• Less favorable attitudes about alcohol, drugs
• Stronger refusal skills
• National Institute of Child Health and Development

20
Issues in Using EBP

• Fidelity versus Adaptation
• Some say the need for local adaptation is over
  estimated
• Adaptations must fit with program rationale
• Language/cultural adaptations most justified
• Little evidence for race/ethnicity, gender, or
  class differences in school program effects
• Most frequent threats to fidelity
• Frontline implementers
• Disseminating Agency

Source Elliot, D. 2008, Blueprints Conference
21
Issues in Using EBP

• Adaptation is as likely to reduce effects as
  enhance them
• Local adaptation may increase buy in but also
  creates uncertainty about program effects
• Program success must be judged by real changes in
  behavior, not number of adaptations or survival

Source Elliot, D. 2008, Blueprints Conference
22
Examine Experiences of EBPs(Lessons Learned)

• Conduct a needs assessment before implementation
• Have strong administrative support and program
  champion
• Have a stable agency/shared vision
• Obtain support, motivation, buy-in of staff
• Ensure implementation fidelity
• Use paid staff, not volunteer

Source Mihalic, et al., 2004. Successful
Program Implementation Lessons From Blueprints
(OJJDP Bulletin)
23
(No Transcript)
24
Commit to (Eventually) Evaluating Your Program

• Most (but not all) EBPs were established in a
  University setting or with University partners
• Make sure you are internally tracking the data
  needed to determine the programs effect
  focusing on those who receive the services
• Make linkages to existing resources in evaluation
  design and assistance

25
Moving Forward Becoming an EBP?

• Start by focusing on Best Practices Principles
• Continue to update/refresh your logic model
• Make performance measures a priority
• Learn more about those programs that have been
  through the EBP process
• What have been their successes and challenges,
  both expected and not
• Seek partners in evaluation (including OJJDP)
• Use the Tools and Resources available

26
Update/Refresh Your Logic Model Periodically

• Are goals and objectives still relevant?
• Are activities occurring as intended?
• Do you need to revisit outcomes or activities?
• Where are you experiencing challenges?
• Has something new entered the mix?
• Other changes?
• Use this process as a way to keep staff and
  partners engaged

27
Make Performance Measures a Priority
27

• Performance measures and data collection are
  building blocks of evaluation
• Hard proof of what/how/when/why your program is
  doing
• Documentation supports sustainability

28
Resources EPB Websites

• Model Programs Guide (MPG), Office of Juvenile
  Justice and Delinquency Prevention
• Blueprints for Violence Prevention, OJJDP and
  Centers for Disease Control and Prevention
• What Works Clearinghouse (WWC), U.S. Department
  of Educations Institute of Education Sciences
• National Registry of Evidence-based Programs and
  Practices (NREPP), Substance Abuse and Mental
  Health Services Administration
• Promising Practices Network on Children, Families
  and Communities (PPN), RAND corporation with many
  partners
• Find Youth Info web site, Federal Interagency
  Working Group on Youth Programs

29
OJJDP Model Programs Guide
www.ojjdp.ncjrs.gov/programs/mpg.html
An easy-to-use tool to search for evidence-based
programs State-of-the-art literature reviews
in dozens of program areas Over 200 program
descriptions
30
Additional Resources

• OJJDPs Contractors
• Education Development Center (EDC)
• CSR, Inc.
• National Training and Technical Assistance Center
  (NTTAC)
• Each other share your experiences, successes,
  challenges
• Tell us what else you need training, TA
  materials, resources, assistance?

31
Contact Information

• CSR, Incorporated
• Pattie San Antonio, psanantonio_at_csrincorporated.co
  m
• [phone]