Overview: Evidence Based Practices

1
OverviewEvidence Based Practices

• Ann Garland, Ph.D.
• Mary Baker, Ph.D.
• Kristin Hawley, Ph.D.
• Child and Adolescent Services Research Center
  (CASRC)
• www.casrc.org

2
Overview Evidence Based Practices

• Promises and Pitfalls of Evidence Based Practices
  
• Highlighting Evidence Based Mental Health
  Treatment Models
• Adapting an Evidence Based Approach for Every
  Program
• Four steps to Evidence Based Practice
• Becoming Informed Consumers of Research

3
Evidence Based Practices Promises and Pitfalls

• Ann Garland, Ph.D.
• Associate Professor
• UCSD Dept. of Psychiatry
• Associate Director, CASRC
• Supervising Psychologist
• CHHC Out-patient Psychiatry Clinic

4
Outline

• Why all this attention to evidence based
  practices?
• What is the evidence regarding the effectiveness
  of usual care?
• What are the challenges of integrating evidence
  based practices into usual care?
• How do we bridge the gap between evidence and
  practice?
• How do we learn more about evidence based
  treatment models?

5
Buzz Words Associations?

• Evidence-based practice
• Best practice guidelines
• Manualized treatments
• Quality of Care/Quality Improvement
• Empirically supported/validated treatments
• Efficacy vs. Effectiveness

6
Efficacy Effectiveness
Highly controlled Laboratory setting Specific
inclusion/exclusion criteria Participants
recruited Selected providers Targeted
outcomes
Less controlled Real world setting Few exclusion
criteria Heterogeneous population Real world
providers Diverse outcomes
7
Why all this attention to evidence based practice?

• Pervasive across all areas of health care
• Managed Care Influence Accountability
• Limited Resources Cost/Benefit Focus
• Concerning data on lack of effectiveness of
  community based services
• Quality of care monitoring systems
• Tension between clinical and statistical
  decision making

8
Evidence for the Effectiveness of Usual Care?

• Relatively strong evidence for the efficacy of
  specific treatment programs from controlled
  studies
• gt1500 trials of out-patient treatment
• Of approx. 300 well designed studies, 60 show
  significant positive treatment effects
• However, findings on effectiveness of usual care
  practice are concerning

9
Challenges in integrating evidence and practice

• Clinicians may perceive that much of the research
  on evidence based treatment models is not
  relevant to their realities, e.g.
• Patients and families very different
• Clinicians working under different
  incentives/motivations may have different
  training backgrounds
• Treatment settings, organizations very different

10
(No Transcript)
11
Challenges (continued)

• Limited opportunities for clinicians to learn
  about research findings and/or participate in
  training on evidence based treatments
• Clinicians already overloaded
• Research findings often not well translated or
  disseminated in useful ways
• Few opportunities for training in e-b
  interventions
• Few incentives for changing treatment practices
• Coercive mandates to change may not be
  constructive

12
Challenges (continued)

• Cultures of Research and Practice are very
  different and there has been limited
  collaboration or cross-fertilization to date
• Communication/Language
• Time frames/Incentives
• Assumptions (e.g., individual differences vs.
  average effects)

13
Communication Problems
14
Without clinical expertise, practice risks
becoming tyrannized by external evidence, for
even excellent external evidence may be
inapplicable to, or inappropriate for, an
individual patient. Without current best
external evidence, practice risks becoming
rapidly out of date, to the detriment of
patients. David Sackett, 1997
15
Bridging the Gap

• The gap between researchers and practitioners
  limits effectiveness of care as well as the
  clinical utility and social validity of research.
• Bridging the gap requires improved collaboration
  with providers and more balanced attention to
  empirically-derived and practitioner-derived
  experience/ knowledge

16
New Paradigms for Integrating Evidence Based
Practice

• Values driven evidence based practice
• Adaptations of evidence based treatment models
• Identification of principles or components of
  evidence based practice for dissemination
• Collaboration between researchers and providers
  to develop and test treatment models
• Collaborative research on existing practice to
  identify effective practice strategies

17
Balancing Evidence-Based Practice with
Practice-Based Evidence
18
(No Transcript)
19
Evidence Based Treatment Models

• Highlighting models with strong evidence (so far)

20
Selected Psychotherapeutic Interventions

• Disruptive behavior problems
• Parent Management Training (PMT) (Patterson,
  Reid, Chamberlain)
• Parent Child Interaction Therapy (PCIT) (Eyberg)
• Videotape PMT (Webster-Stratton)
• Problem Solving Skills/Anger Coping
  (Shure,Kazdin, Lochman)
• Anxiety disorders
• CBT (Kendall, Barrett)
• Mood disorders
• Cognitive Behavior Therapy (Clarke)
• Interpersonal Therapy (Mufson)

21
Additional Evidence Based Treatment Models

• Intensive Case Management (Evans, Burns)
• Therapeutic Foster Care (Chamberlain, Reid)
• Multisystemic Therapy (MST) (Henggeler)
• Stimulant medication for ADHD (MTA study Jensen
  et al.)

22
Websites for more information about Evidence
Based Practices

• http//www.effectivechildtherapy.com/
• http//www.sp-ebi.org/
• http//w-w-c.org/index.html
• http//www.challengingbehavior.fmhi.usf.edu/index.
  html
• http//www.strengtheningfamilies.org/index.html
• http//www.modelprograms.samhsa.gov/template.cfm?p
  agedefault

23
(No Transcript)
24
(No Transcript)
25
To Be or To BecomeEvidence Based

• Mary J. Baker-Ericzen, Ph.D.
• Research Scientist, CASRC
• Adjunct Clinical Instructor, UCSD Departments of
  Psychiatry and Psychology
• Clinical Psychologist, Private Practice

26
Steps to Selecting and Implementing an EBP

• Step 1 Conduct Program Needs Assessment
• Step 2 Survey the Available EBP Treatment
  Models Review Literature
• Step 3 Adopt and Adapt a Model for Use in your
  Organization-Trying it out!
• Step 4 Evaluate for Effectiveness

27
STEP 1 Needs Assessment
28
Step1 Needs Assessment

• Who are your clients? What are your/childs
  needs?
• Children/youth, parents, family members, case
  workers, courts,
• What is your setting? Where are services needed?
• Clinic based, school based, home based, group
  homes, residential, juvenile hall, camps, space
  constraints
• What is the funding mechanisms? Who will pay for
  it?
• MediCal, AB2726, Title VIIII, grants, insurance,
  private

29
Step1 Needs Assessment-cont.

• How is time spent in your organization? How much
  time is available? When can services be received?
• Average length of treatment, average staff time
  per case, productivity loads, time spent in
  supervision, time spent in case recording, time
  spent in evaluation, hours of operation
• What are the policies and bureaucracies your
  program may be tied to? What are potential
  limitations? What are barriers to receiving
  services?
• Legislation, mandates, programming tied to
  funding

30
Step 1 Needs Assessment-cont.

• What is your programs mission? What goals do you
  have for yourself/child?
• Goals, values, directions,
• What are the values and goals of your family
  members-consumers? What are goals of other
  caregivers, teachers, clinicians?
• Family choices, priorities, preferred outcomes

31
STEP 2 Survey the EBP Models
32
Step 2 Survey the Available EBP Models

• Conduct a thorough review of Models
• Search by Population
• Search by Service type
• Search by Setting type
• Compare population, service, setting etc. to
  those identified in your Needs Assessment
• Review information critically
• Compare each response from Needs Assessment to
  data/information reported on the model
• Review and compare all aspects of programming (no
  detail too small)

33
Step 2 Survey the Available EBP Models

• Conduct a review of the literature
• Tested through efficacy trials
• Tested through effectiveness trials
• Interpret study limitations
• Critically appraise the evidence and determine
  the implications of evidence for clinical practice

34
Step 2 Survey the Available EBP Models

• Consumer Fit
• Questions
• Are these results applicable to my local
  population?
• Are the results applicable to my particular
  client?
• Are these practices right for my child and family?

35
STEP 3 Adopt and Adapt a Model
36
Step 3 Adopt and Adapt a Model

• Implementing changes in practice based on
  evidence
• Clearly define problem and population want to
  address
• Develop stakeholder commitment
• Facilitate stakeholder input Select
  interventions with stakeholders when appropriate
• Examine fit of intervention to community/organizat
  ion/self and list needed accommodations
• Evaluate the program in your setting

37
Step 3 Adopt and Adapt a Model

• Needs to be tailored to meet local population
  needs
• How do you deal with the tension between
  adherence to models and maximizing the local
  fit?
• Start with the common elements
• Figure out how to get the necessary supports and
  resources in place before you start
• Collect data that can be fed back in real time
  to program and then collect more data
• Go for continuous quality improvement

38
STEP 4 Evaluation
39

Step 4 Evaluation of Effectiveness
Moving From Regulations to Effectiveness Evaluatin
g impact on clinical practice

• From
• Compliance driven data collection
• Rule and regulation driven administration
• Best-guess decision-making
• Preference given to distinct
• professional roles
• System reacts to need
• Information is withheld

• To
• Outcome-based monitoring
• Goal driven management
• Data-based decision-making
• Cooperation across professionals is a priority
• Need is anticipated
• Information is disseminated, transparent

40
Conceptualizing the Service System Levels of
Data Collection

• Demographics (individual community)
• Risk Factors (individual community)
• Diagnosis, Chronicity, Acuity, Severity,
  Comorbidity
• Symptomatology
• Functional Impairment, stay out of trouble
• Hopefulness, Self Confidence, Self Esteem
• Safe, non-victim,
• Living in Home or home-like environment
• School Success

Child Family

• Clinician / Provider Characteristics
• Treatment Modality, type, amount
• Therapeutic Alliance
• Resistance to Therapy

Clinician Provider
Outcome

• Service Mix (client preference and service
  program design)
• Service Intensity and Duration
• Level of Participation -- Utilization Patterns
• Treatment Adherence -- Was treatment plan carried
  out?
• Location of services
• Informal Networks -- families, community
  supports, etc.

Service Mix
41
Outcomes

• Early, intermediate, end results of treatment
• Increase or decrease in symptomatology
• Increase or decrease in functional status
• Increase or decrease in risk factors (prevention)
• Increase or decrease in quality of life
• Outcome should be reported at individual level,
  clinical profiles, and aggregated at system or
  subgroup levels for understanding of
  effectiveness
• Outcomes may be reported concurrently, or by time
  intervals (annually intake to 12 months, etc.)

42
Other Components of Measurement

• Characteristics of the treatment-service
  environment
• Service array and professional staff mix
• Service location, ease of access, convenience
• Sensitivity to diverse service populations
• Reimbursement practices

43
Measurement System Objectives

• A measurement system needs to provide
• A comprehensive and integrated system that uses
  all available data consumer survey and
  administrative data
• Less burdensome and non-duplicative assessment
  measures that attend to client service issues
• Measures that provide a foundation for standard
  setting and benchmarking

44
Choosing MeasuresRelevance Meaningfulness

• Measures that are meaningful to providers AND
  consumers for making treatment choices
• Measures that stimulates internal improvement
  efforts
• Measures that assist in understanding the
  clinical significance-client improvements
• Measures that assess economic value treatment is
  most efficient service to maximize behavioral
  health
• Measures that assess at least one process that
  can be controlled that has important effects on
  the outcome.
• If the measure is a process measure, there should
  be a strong link between the process and desired
  outcomes

45
Choosing Measures Feasibility Practicability

• Precise specification measures should have
  clear specifications for data sources and methods
  for data collection and reporting
• Reasonable cost measures should not impose an
  inappropriate burden on health care systems
• Confidentiality the collection of data for the
  measures should not violate any accepted
  standards of consumer confidentiality
• Logistical feasibility the data required for
  the measure should be available (administrative
  or consumer survey tools)
• Auditability measures should not be susceptible
  to manipulation or gaming that would be
  undetectable in an audit.

46
Choosing MeasuresScientific Soundness

• Measures should accurately measure what is
  actually happening reliable and valid
• Measures should not be affected by any variables
  that are beyond the mental health care systems
  control
• Measures should have documented links between the
  clinical processes and the outcomes addressed by
  the measure
• Measures should produce the same results when
  repeated in the same population and settings
• Measures should make sense logically, clinically,
  and financially
• Measure should correlate well with other measures
  of the same aspects of care and capture
  meaningful aspects of this care.
• Measure should not be affected if different
  systems have to use different data sources for
  the measure.

47
Range of Outcome Criteria to Evaluate Treatment
Effectiveness

• 1) Child Functioning
• Symptom reduction
• Functional Impairment improvements
• Prosocial competence, strengths of youth
• improvements/maintenance
• Academic functioning improvements/maintenance
• Living environments improved to/maintained in
  homelike settings

48
Range of Outcome Criteria to Evaluate Treatment
Effectiveness

• 2) Parent and Family Functioning
• Family dysfunction reduction
• Contextual stress reduction
• Quality of Life improvements/maintenance
• Family relations improvements/maintenance
• Coping skill improvements/maintenance
• Social support system improvements/maintenance

49
Range of Outcome Criteria to Evaluate Treatment
Effectiveness

• 3) System Functioning Social Impact Measures
• School attendance/ activities improvement/maintena
  nce
• Truancy/delinquency reductions
• Arrests, Probation status reduction
• Service Use changes from restricted to community
  based to family supported

50
Critical Consumer

• Becoming Informed Consumers
• of Research

51
Critical Users Consumers

• How does a consumer-family know if they are
  receiving high quality-effective care?
• How does a provider know if they are delivering
  high quality-effective care?

52
Questions to Ask regarding a specific Treatment

• Will the Treatment result in Harm to the child?
  What are potential risks?
• How will Treatment failure affect my child
  family? What are the goals? How will we know if
  it is working or not working?
• Has the Treatment been researched and found
  empirically validated?
• Are there assessment procedures specified?
• How will the Treatment be integrated into the
  childs current program?

53
Questions to Ask regarding a specific Treatment

• How successful has the program been for other
  children?   
•  How is progress measured? Will my child's
  behavior change be observed and recorded? 
•  Will my child be given tasks and rewards that
  are personally motivating? 
• Will the program prepare me to continue the
  therapy at home? 
•  What is the cost, time commitment, and location
  of the program?

54
Principles of Evaluating Treatments

• Beware of any program or strategy that claims
  effectiveness for EVERY person with a similar
  disorder or profile
• Beware of any program that thwarts
  individualization potentially results in
  potentially harmful decisions
• Be Aware that any treatment represents 1 of
  several options
• Be Aware that treatment should always depend on
  individual assessment info. that points to it as
  an appropriate choice for a particular child
• Be Aware that often new treatments have NOT been
  validated scientifically (inquire about in
  process evaluation)

55
4 Questions Answered in Research Articles

• What was studied?
• Who participated in the study?
• How and where was the research done?
• What did the researchers find?
• Secondary Questions
• How similar are your clients or your child to
  the individuals studied?
• Are the findings likely to be applicable to
  others or my family?
• What are the study limitations?

56
Understanding Research in Literature or Media

• 1) Consider the source
• Investigate the author qualifications,
  reputation, affiliations, agenda
• 2) Media is also a source to be evaluated
• - Media coverage may not fully or accurately
  summarize the original research, can oversimplify
  leading to misinterpretation
• 3) Has the research been published, and where?
• - Research published in peer reviewed journals
  is more trustworthy

57
Understanding Research in Literature or Media

• 4) Research results are about the topic as
  measured, not as any one persons definition
• Topic studied is measured and defined in a
  specific way understand study definition
• 5) Different types of research have different
  strengths
• - Learn the research design inquire about
  replications
• 6) Sampling is more important than sample size
• -Size of sample sample selection, response rate
• 7) Statistical significance explained
• -Means effect is unlikely to be due to chance
  meaningfulness

58
Understanding Research in Literature or Media

• 8) Research findings are about groups
• -Usually involve comparisons between groups
• 9) All research is not created equal
• - Higher quality studies are given more weight
• 10) Any one study is not the whole story
• - More valuable when results of multiple studies
  are viewed together Examine past research,
  understand differences between studies with
  different results, evaluate quality of studies.
• Understanding Research Top Ten Tips for
  Advocates and Policymakers. Washington, DC
  National Association of Child Advocates, 2001.
  www.childadvocacy.org