EvidenceBased Practice: Whats the Bar for Implementing a New Therapy

1
Evidence-Based Practice Whats the Bar for
Implementing a New Therapy?

• Andrea Sartori Don Labbe
• The University of Alabama at Birmingham
• July 17, 2008

2
Evidence Based Practice in PsychologyEBPP

• Evidence based practice in psychology is the
  integration of the best available research with
  clinical expertise in the context of patient
  characteristics, culture, and preferences.
• EBPP starts with the patient and asks what
  research evidence will assist the psychologist in
  achieving the best outcome from assessment to
  treatment. Empirically supported treatments start
  with the treatment and ask whether it works for a
  certain disorder or problem under specified
  circumstances.
• APA Presidential Task Force (2005)

3
EBPP Historical Perspective

• Despite years of research on psychological
  treatments, public sentiment remained that
  psychology was inferior to medication in treating
  mental illness and generally ineffective.
• 1995 APA Division 12 Task Force develops
  template for developing practice guidelines.
  Identifies 18 ESTs.
• 2002 Criteria for Evaluating Treatment
  Guidelines based on efficacy and clinical
  utility.
• 2005 APA Presidential Task Force on EBPP
  incorporates aspects of efficacy, clinical
  expertise, and patient characteristics.

4
Chambless et al.(1996) An Update on Empirically
Validated Therapies

• Purposes
• To add to the list compiled by the 1995 Division
  12 Task Force
• To raise several issues regarding use and
  limitations of empirically supported treatments
• To caution about the use of the list of
  empirically validated treatments

5
Cautions

• 1. List intended to facilitate education by
  identifying treatments with a scientific basis
  NOT a complete list, treatments not on the list
  are not necessarily ineffective
• 2. List is not a substitute for clinicians own
  decisions about appropriate treatments
  flexibility is required
• 3. List created based on criteria set forth by
  Task Force- some may disagree relied on
  meta-analyses
• 4. Current list contains new entries as well as
  original entries
• 5. Brand names of interventions not the critical
  identifiers the manuals are (e.g. relaxation
  training and exposure response prevention are
  behavioral therapies)
• 6. Depending on the problem, the treatments on
  the list may have been used in combination with
  other treatments (e.g. medication) or employed as
  the sole treatment

6
2 Major Variables That May Affect Response to
Treatment

• Empirically validated treatments (EVTs) with
  minority clients
• Aptitude x Treatment Interactions (ATIs)

7
EVTs with Minorities

• We know of no psychotherapy treatment research
  that meets basic criteria important for
  demonstrating treatment efficacy for ethnic
  minority populations
• Most investigators did not specify ethnicity of
  subjects in their samples not one used ethnicity
  as a variable of interest
• How do we know these treatments are efficacious
  for minority clients?
• Most therapists will see clients from different
  cultural backgrounds
• Mental health needs of minorities are high
• Lack of research in this area is poor science
  cannot assume the treatments will generalize

8
Recommendations

• 1. Ethnicity should be specified in all studies
• 2. Investigators should be given incentives for
  studies of ethnicity and treatment (i.e. simply
  the need is not enough?)
• 3. Researchers should report effect sizes on
  major outcome variables by ethnicity
• 4. Barriers that are preventing this kind of
  research should be more clearly defined
• 5. Given the lack of efficacy data, should
  psychotherapy continue to be offered to diverse
  populations? Yes (duh) clinicians have a
  responsibility to provide services to the best of
  their abilities

9
Aptitude x Treatment Interactions

• Client characteristics and personality qualities
  may affect which treatment will be the most
  beneficial
• Research in this area is difficult because
  psychotherapy studies usually lack the power to
  detect these interactions, and replication of
  ATIs is difficult
• As such, Chambless et al. sets forth criteria for
  identifying ATIs
• A) The aptitude (client) variable should be
  assessed with valid and reliable measures
• B) differential efficacy must be shown (those who
  are high on the aptitude do better or worse with
  a specified treatment than those who are low on
  the aptitude)
• Also can use single case design experiments
• Examples of characteristics/aptitudes reactance
  (benefit more from paradoxical instructions),
  impulsivity (cognitive therapy seems to be
  especially useful in these cases)

10
Considerations of Ethics

• Clinical psychologists are active consumers of
  the research literature inclusion of a
  treatment on The List is not a necessary and
  sufficient reason to assume overarching validity
  judgment is required
• Clinical psychologists must remain current and
  read beyond their specific area of research or
  interest
• Informed consent to treatment reasonable
  expectations of treatment, treatment options,
  negative effects of treatment, costs

11
Seligman (1995) The Effectiveness of
PsychotherapyThe Consumer Reports Study

• In 1995, Consumer Reports magazine surveyed its
  readers about experiences with Mental Health
  issues over the past 3 years. Readers were asked
  to respond if at any point during that period
  they experienced stress or emotional problems for
  which they sought help from friends, family,
  mental health professionals, family doctor, or
  support groups. 7000 people responded with 2900
  seeing a mental health professional during that
  time.

12
The Consumer Reports Study

• The survey asked 26 questions about their
  experience covering the following areas
• Type of therapist Presenting problem
• Emotional state (outset vs. current) Cost and
  Coverage
• Group vs. Individual therapy Duration and freq.
• Modality Therapist Competency
• How much therapy helped Satisfaction
• Reason for terminating

13
The Consumer Reports Study Results

• Treatment by MH professionals worked. Readers
  reported significant improvements.
• Long-term therapy was reported to be more
  effective than short-term therapy.
• Psychotherapy was equal to psychotherapy and meds
  in effectiveness.
• Psychologists, psychiatrists and social workers
  did equally well. All did better than marriage
  counselors.
• Family doctors did as well as MH professionals in
  the short run, worse long term.

14
The Consumer Reports Study Results

• Active Shoppers did better in treatment than
  passive recipients.
• AA did especially well
• No modality of psychotherapy did better than any
  other
• Individuals whose duration on treatment was
  limited by their insurance coverage did worse.

15
Efficacy VS. Effectiveness

• Efficacy Studies contrast therapy to a comparison
  group under well-controlled conditions.
• Effectiveness Studies investigate how patients
  fare under the actual conditions of treatment in
  the field.

16
Characteristics of Efficacy Studies

• Random assignment
• Rigorous controls
• Manualized treatments (fidelity)
• Fixed number of sessions
• Well-operationalized outcomes (usually DSM-IV
  based)
• Raters and diagnosticians blind
• Patients meet criteria for a single disorder
• Patients followed for a fixed period after
  termination with a thorough assessment battery

17
What Efficacy Studies Leave Out

• The Inertness Assumption Any treatment that
  cannot be studied using efficacy methodology must
  be inert.
• Long-term, dynamic treatments are too difficult
  to examine, but are commonly used and often
  clinically effective.
• Efficacy studies omit too many crucial elements
  of what is done clinically in the field. These
  include
• Not of fixed duration Self-Correcting
• Active shopping Multiple presenting problems
• Goals of improvement in general function in
  addition to symptom relief

18
CR Effectiveness Study Virtues

• Sampling large sample representative of educated
  middle class Americans, weighted toward people
  who believe in therapy and are problem solvers.
• Treatment Duration varied, more realistic,
  terminated when patient was better, left
  treatment unimproved, or by lack of insurance.
• Self-Correction treatment as it naturally
  occurs, dynamic.
• Multiple Presenting Problems individuals not
  turned away due to co-morbidity or subclinical
  presentation.

19
CR Effectiveness Study Virtues

• Outcomes involved improvements in general
  functioning as well as improvements in DSM-IV
  symptoms.
• Clinical Significance Outcomes were meaningful
  to the patient based on their responses as
  opposed to statistical significance from efficacy
  studies.
• Unbiased no theory guiding the study or axe to
  grind.

20
CR Effectiveness Study Flaws

• Sampling Bias 38 responded, participants
  recognized problems and chose treatment
• No control group would time alone have revealed
  similar results?
• Self-report cannot verify type of therapy,
  provider, real gains, or even nature of
  improvement
• Blindness demand characteristics
• Inadequate outcome measures very subjective
  (how much better do you feel?)
• Retrospective cant truly infer improvement
  across time
• Therapy Junkies people committed to therapy as
  a way of life.
• Nonrandom assignment

21
Take Home Message

• Effectiveness studies allow us to examine how and
  to whom treatment is actually delivered and to
  what end.
• While it may not be able to tease apart the
  effective components of therapy, the CR study
  answers the question Do people have fewer
  symptoms and a better quality of life after
  therapy?
• With improved methodological design, future
  effectiveness studies may provide valuable
  information on treatment utility above and beyond
  that of efficacy studies.

22
Weston Bradley(2005) Empirically Supported
Complexity

• Issues regarding the empirically supported
  therapies (EST) movement include
• A restricted view of evidence
• Brief treatments for discrete disorders
• Empirically underqualified conclusions

23
EST Movement

• EST movement operationalizes the construct of
  evidence-based practice
• EST movement adopted a FDA model treatments are
  classified as supported or unsupported based on
  outcomes of randomized controlled trials of
  active vs. control treatments
• EST movement focused on brief, focal treatments
  for specific disorders as defined by the DSM-IV

24
A Restricted View of Evidence

• 1. EST research arose as a psychotherapeutic
  analog to drug trials psychological compounds
  are created in the lab, compared to placebos, and
  disseminated to clinicians are these superior
  to homebrewed compounds of successful,
  experienced clinicians? Interventions emerge from
  practice, not the laboratory- use clinical
  practice as a natural laboratory
• 2. Researchers have compared their treatments to
  no treatment or other intent-to-fail conditions
  not a viable control condition. Something
  intended to be effective works better than
  something intended to be ineffective

25

• 3. Science should be about examining all the
  evidence, yet EST movement minimizes the
  importance of basic science/research in
  evidence-based practice (dont we need to
  understand underlying processes to implement
  problem-solving?)
• Randomized controlled trials may be the best way
  to assess causal relationships, but the tradeoff
  of external to internal validity is too high

26
Brief Treatments for Discrete Disorders
(minimizes within-group variability)

• EST movement makes assumption that most patients
  have (or can be treated as if they have) one
  primary syndrome in reality, comorbidity among
  disorders is quite high
• Many patients seen in clinical practice do not
  meet criteria for a DSM-IV specified disorder
  (e.g. NOS) how do treatment manuals apply to
  these patients?
• Growing role of personality in psychopathology
  suggests limits to these brief, focal treatments
  shared diatheses suggested, not coincidental
  occurrence of unrelated disorders
• Brief treatment lengths assume that
  psychopathology is mutable over short intervals
  not true for most disorders with repeated
  reoccurrences and residual symptoms

27
Empirically Underqualified Conclusions

• Different criteria can influence outcome
• Effect size a moderate effect size could be
  achieved through a very large effect for only a
  small subset of patients
• Percent recovered and percent improved out of
  what number? Different denominators can be used,
  liberal or conservative. Also, a treatment could
  lead to substantial improvement in patients,
  though they remain highly symptomatic (residual
  post-treatment symptomatology)
• Quantifying generalizability when using a
  homogenous sample, researchers need to qualify
  the population to whom their results can best
  generalize (describe inclusion/exclusion
  criteria, indicate percent of patients excluded
  of those screened)
• Limited data on sustained efficacy

28
Important Points

• If the goal is to identify best practice, all
  available data should be considered, not just
  data using one kind of research design
• The burden of proof for a new treatment should
  be that its outcomes compare favorably to the
  outcomes obtained by experienced clinicians, not
  that it survives the test of the null hypothesis
  (i.e. that it works better than nothing, or
  better than something intended to fail)
• EBP gt EST evidence-based practice is more than
  a list of empirically supported treatments for
  specific disorders
• EST movements definition of evidence eliminates
  too much of what is available to us from science

29
APA Presidential Task Force Evidence-Based
Practice in Psychology (2006) Efficacy Best
Available Research Evidence

• Includes scientific research in the laboratory
  (applied and basic) and in the field addressing
  intervention strategies, assessment, clinical
  problems and patient.
• Treatment Efficacy The systematic and
  scientific evaluation of whether a treatment
  works. Ranges from RCTs to quasi-experimental
  designs to clinical observation and expert
  opinion. Meta analysis to synthesize research
  findings.
• Clinical Utility Is the treatment feasible,
  applicable, and useful in the setting it is to be
  offered. Does it generalize across settings,
  therapists and patient populations?

30
Clinical Expertise

• Defined by the Task Force as competence attained
  by psychologists through education, training, and
  experience resulting in effective practice.
• Clinical expertise also arises from
  self-reflection (know your biases) and knowledge
  of current research and clinical strategies from
  continuing professional education and training.
• Psychologists should use clinical expertise to
  identify and integrate the best research evidence
  with clinical data (patient info obtained during
  treatment).

31
Components of Clinical Expertise

• Assessment, diagnostic judgment, case
  formulation, and treatment planning
• Clinical decision making, treatment
  implementation, and monitoring progress
• Interpersonal expertise
• Continual self-reflection and acquisition of
  skills

• Evaluation and use of research evidence
• Understanding the influence of individual,
  cultural, and contextual differences on tx
• Seeking available resources as needed
  (consultation, adjunctive tx, etc.)
• A cogent rationale for clinical strategies

32
Patient Characteristics

• EBPP must involve consideration of the patients
  values, religious beliefs, world views, goals,
  and preferences for treatment.
• Clinicians must address the issue of how to
  provide the best available treatment to patients
  whose characteristics and problems differ from
  those in study samples.

33
Individual Differences Affecting Treatment
Outcomes

• Variation in the presenting problem
• other symptoms
• co-morbidity personality factors
• Age, developmental status and history
• Current environmental factors and stressors

• Sociocultural factors, religion, social class,
  ethnicity
• Family context
• Cultural influences
• Gender and gender identity
• Sexual orientation
• Personal values and treatment expectations

34
Future Directions

• Research should investigate Patient X Treatment
  interactions, diverse patient populations,
  distinctions between common and specific factors
  for change, models of treatment based on
  clinician practices in the field,
  generalizability of treatments
• Clinical expertise identify specific practices
  and expertise in the clinic leading to positive
  outcomes, improve diagnosis, observe outcomes for
  more unique and diverse patients
• Patient Characteristics PCs as moderators for
  treatment outcome, increase information on
  cultural influences to clinicians, identify
  treatment decision making best suited to diverse
  patients

35
The Big Picture

• Evaluate all forms of evidence for treatment
  effectiveness including experimental and
  clinical.
• Develop your expertise through ongoing education
  and training in the latest evidence.
• Continue to develop an understanding of unique
  patient characteristics and how they will affect
  treatment.
• Incorporate knowledge from research, personal
  expertise, and patient preferences and goals into
  flexible treatments to achieve the best outcomes.

36
Kazdin Chapter 11 The Single Case in Clinical
Work

• How to improve the quality of inferences that can
  be drawn in clinical practice
• Methods for assessment and evaluation

37
Improving Quality of Inferences

• Collect systematic data standardized, not
  simply self-report
• Assess on multiple occasions can decrease
  plausibility of threats to internal validity
• Consider past and future projections of
  performance knowing the course or history of
  the problem increases likely predictions for
  outcome
• Consider the type of effect associated with
  treatment immediacy and magnitude of change
• Use multiple and heterogeneous subjects
• See Table 11.1 on Page 304

38
Methods for Assessment and Evaluation in Clinical
Practice

• The key to all evaluation is careful assessment
• The primary goal is to assess, evaluate, and
  demonstrate change
• Key steps
• Specify and assess treatment goals (keep in mind
  these can change along the way)
• Specify and assess procedures and processes the
  means to achieving goals can be the procedures
  used in treatment and emergent processes/relations
  hip issues
• Select appropriate measures
• Assessment on occasions measure performance
  before treatment begins and on an ongoing basis
• Design and data evaluation has change occurred
  and is it reliable? Can employ graphical displays
  and nonstatistical (descriptive) methods such as
  changes in means and slopes

39
Food For Thought

• How can clinical practice better inform research
  design?
• Given the criticisms and complexities regarding
  empirically-supported treatments, how willing are
  you to embrace them?
• Are we really trained for this?