Michael Prendergast

1
Issues in Defining and Applying Evidence-Based
Practices Criteria

• Michael Prendergast
• UCLA Integrated Substance Abuse Programs
• Steven Schinke
• Columbia University School of Social Work
• Judith Sachwald
• American Probation and Parole Association
• Implementing and Sustaining Evidence-Based Drug
  Treatment in Criminal Justice Settings
• CEICA Conference
• Philadelphia, PA
• December 6-7, 2006

2
What is evidence-based practice for treatment in
criminal justice settings?

• Definition (provisional)
• The use of the best available evidence from
  systematic research, combined with clinical
  experience and judgment, in making decisions
  about delivering care to clients.

3
What is evidence-based practice for treatment in
criminal justice settings?

• The intent of EBP is to shift decision-making
  about what interventions to offer (and fund) from
  authority, intuition, rules of thumb, tradition,
  and anecdote to a rational, transparent,
  systematic process based on evidence from
  scientific research and on consensus among
  experts.

4
What is evidence-based practice for treatment in
criminal justice settings?

• EBP assumes that there are treatments that do not
  meet evidence-based criteria
• Some of these have not been evaluated yet, but
  may be effective (Lack of evidence does not
  mean lack of effectiveness)
• Others have been evaluated, but can only be
  judged promising (best practices?)
• Others have not been shown to be effective in
  several trials, and may even be harmful

5
What are the criteria for determining which
practices are evidence based?

• Several different sets of criteria for EBP are
  available
• Maryland Scientific Methods Scale
• SAMHSA National Registry of Evidence-Based
  Programs and Practices
• DOE What Works Clearinghouse
• OMB Program Assessment Rating Tool
• Society for Prevention Research
• Coalition for Evidence-Based Policy
• Blueprints for Violence Prevention
• Washington State Institute for Public Policy
• Oregon Evidence-Based Practices Criteria

6
What constitutes evidence and the best ways to
evaluate the quality and applicability of
evidence?

• Evidence vs. anecdote
• Experimental studies Quasi-experimental
  studies Qualitative studies
• Internal validity External validity
• Efficacy vs. Effectiveness
• Publication bias
• Methodological quality

7
Balancing effect sizes and statistical
significance with clinical relevance and cost
factors

• Statistical significance vs. clinical
  significance
• Research in controlled settings vs.
  Implementation in community settings
• Effectiveness vs. cost effectiveness

8
What are major EBP cataloging efforts?

• National Registry of Evidence-based Programs and
  Practices
• Coalition for Evidence-Based Policy
• Blueprints for Violence Prevention
• Society for Prevention Research
• Washington State Institute for Public Policy
• Campbell Collaboration
• Individual meta-analysis projects (e.g., Lipsey)

9
Cautions with RTCs

• Often not possible
• Lower external validity (generalizibility)
• Implementation problems with RTCs
• Recruitment, randomization
• Compliance
• Delivery
• Follow-up
• Limited qualitative findings

10
Are different levels of review possible to
address immediate needs?

• Specific conditions and situation may require
  flexibility in application of evidence-based
  practice
• Need to decide quickly
• Public safety
• Decisions about overcrowding
• Response to court orders
• Civil rights issues
• Less rigorous process can be developed, provided
  that they are also systematic, transparent, and
  objective.

11
Key Issues and Action Steps

• Which EBP criteria should be adopted for
  treatment within criminal justice settings?
• To what extent do evidence-based treatments
  (often based on studies of specific populations
  and settings) generalize to other populations and
  settings?
• How can EBP be misused in policy and funding
  decisions?
• What are the challenges of implementing
  evidence-based practices?

12
Discussion

• Steven Schinke
• Judith Sachwald
• Audience