Measuring Clinical Skills: A Unique Challenge in Fidelity Assessment

1
Measuring Clinical Skills A Unique Challenge in
Fidelity Assessment

• Kim T. Mueser
• Dartmouth Psychiatric Research Center

2
Gary Bond Man Without a Planet or Brother from
Another Planet?

• Innovative thinker
• Math major
• Beloved by students colleagues alike
• Unflappable
• Force to be reckoned with in psychiatric
  rehabilitation
• Honest, direct, modest, sensitive, humorous
• Clearly not of this world, but is he without a
  planet or

3
(No Transcript)
4
(No Transcript)
5
Fidelity Adherence Competence

• Adherence
• The extent to which the defining characteristics
  of a standardized intervention are implemented in
  practice
• Competence
• The skillfulness effectiveness with which a
  standardized intervention is implemented

6
Potential Data Elements of Fidelity Assessments

• Record reviews
• Clinician/administrator interviews
• Consumer interviews
• Knowledge tests
• Role play tests
• Direct observation
• Audio/videotape observation

7
Methods for Assessing Fidelity Depend on

• Definition of the treatment provider
• Individual
• Team
• Agency
• Defining elements of practice
• Organization staffing (e.g., ACT team, voc
  unit)
• Structure, aim, scope of services (e.g., rapid
  job search, services provided in community)
• Clinicianconsumer interactions (e.g., structure,
  content, skill of clinician working with
  consumer)

8
Practices That Have Emphasized Organization
Structure

• Assertive Community Treatment
• 110 staffclient ratio
• Shared caseloads
• Daily team meetings
• Services provided in community
• Direct, not brokered, service provision
• Supported Employment
• Zero exclusion
• Focus on competitive jobs
• Rapid job search
• Follow-along supports
• Consumer preferences

9
Practices That Have Emphasized ClinicianClient
Interactions

• Integrated Dual Disorder Treatment, Illness
  Management Recovery
• Psychoeducational skills
• Motivational interviewing
• Cognitive-behavioral teaching skills

10
Relevance of Clinical Skills to All
Evidence-Based Practices

• Importance of the relationship between clinician
  consumer
• Utility of specific skills in changing behavior
  or achieving specific objectives
• Differences between clinicians in their
  effectiveness despite adherence to organizational
  structural aspects of practice
• Potential to improve quality of services through
  targeted training of specific skills

11
Assessment of Clinical Skill at Implementing a
Practice

• Limited utility of interviews with clinicians or
  consumers as skills must be inferred
• Knowledge tests informative about clinicians
  understanding of practice, but not ability to do
  it
• Role play tests informative about clinicians
  ability to use specific clinical skills in
  simulated situations
• Direct observation informative about clinicians
  use of clinical skills during actual
  interactions, but may be reactive to presence of
  observer
• Audio/videotaped observation informative about
  clinicians use of clinical skills during actual
  interactions, but requires dealing with concerns
  about recording interactions

12
Advantages of Formal Assessment of Clinical Skill
at Practice

• Identifies areas in need of subsequent training
• Facilitates understanding of differences in
  effectiveness of clinicians implementing same
  practice
• Useful for research focused on understanding
  differences between consumers in benefit from a
  practice
• May lead to certification of clinicians in
  practices for which fidelity is defined mainly in
  terms of the interaction between the clinician
  client
• Illness Management Recovery
• Family Psychoeducation (e.g., multi-family
  groups, behavioral family therapy)
• CBT (e.g., for psychosis, PTSD)

13
Two Examples of Clinical Skill-Based Fidelity
Scales

• IMR Clinical Competency Scale
• CBT for PTSD Fidelity Scale

14
IMR Clinical Competency Scale

• Developed by Gingerich, Mueser, Meyer
• Initial step of brainstorming core skills
  necessary to implement IMR, based on principles
  specific strategies outlined in manual
• Initial draft of competency items, behavioral
  anchors, multiple choice questions
• Feedback on initial draft from 25 experienced IMR
  clinicians trainers
• Revised draft additional feedback obtained
• Scale piloted with experienced IMR clinicians
  trainers
• Feedback obtained, revisions made

15
Overview of Scale

• 19 items, 5-point ratings, behaviorally anchored
• Administered to individual providers with
  experience treating at least 3 individual clients
  with IMR or leading/co-leading 1 group
• Ratings based on
• Observation of 2 IMR sessions
• Interview with clinician, including role plays
• Review of written materials

16
Overview of Scale (cont.)

• Administered by experienced IMR clinician,
  trained in implementing scale
• Requires about 6 hours to complete
• Organization of scale
• Process skills
• Motivational enhancement strategies
• Educational strategies
• Cognitive-behavioral strategies
• IMR group leader skills
• IMR curriculum

17
Materials in the Evaluators Manual

• 1. Steps for Evaluators (in checklist form)
• 2. Competency Items in the MN IMR Clinical
  Competency Scale
• Item description
• Skill requirements
• Primary measurement
• Missed Opportunity/Comment Boxes

18
Materials in Manual, contd

• 3. Interview Guide
• Questions for the interview section
• Examples of role plays for evaluation
• 4. Scoring Manual
• 5-point behaviorally anchored scale
• Descriptors for 3 anchor points
• Workshop materials

19
Steps for Evaluators Completing the Scale

• Clinician meeting-preparation for evaluation
• Observe or listen to audiotapes of 2 sessions
  (group or individual or both)
• Review written materials
• Conduct clinician interview-evaluate skills not
  demonstrated in observation and follow-up any
  questions from observation
• Feedback session-offer praise for strengths and
  suggestions for areas of improvement

20
11 Items to be rated using behavioral observation

• 1. Therapeutic relationship
• 2. Structure of sessions
• 3. Efficient use of time
• 4. Motivational enhancement strategies
• 6. Goal follow-up
• 7. Educational strategies

• 8. Use of reinforcement
• 9. Home assignments
• 11. Relaxation Training (if one of sessions
  includes this topic)
• 15. Group skill Involving all members of group
• 16. Group skill Enlisting support between group
  members
• 17. Group skill Tailoring IMR materials to
  experiences and goals of each group member

21
Items to be rated using Written Materials

• 5. Goal Setting
• 6. Goal Follow-up
• 10. Relapse Prevention Training
• 18. Comprehensive Use of IMR Curriculum
• Questions about written materials should be
  followed up during the clinician interview

22
How to evaluate the goal tracking sheets?

• What to look for?
• Recovery goal
• Short-term goals-how many and are they related to
  the long-term goal
• Steps-how many and are they small enough
• Any modifications?
• Remaining questions for Clinician Interview
• See Interview Guide
• Additional questions for clarification
• Review how the client decided on the goal
• Why did the client choose the short-term goals
  listed?

23
(No Transcript)
24
Items to be rated using the Interviewer Guide (
Role Plays)

• 11. Relaxation Training
• 12. Behavioral Tailoring and other medication
  adherence strategies
• 13. Social Skills Training
• 14. Coping Skills Training
• 19. Knowledge of IMR modules

• Followed up from written materials review
• 5. Goal Setting
• 6. Goal Follow-up
• 10. Relapse Prevention Training
• 18. Comprehensive Use of IMR Curriculum

25
Feedback Meeting With Practitioner Supervisor

• Conducted as soon as possible after completion of
  the interview
• Explain purpose of meeting is to review results
  of evaluation
• Give each person a copy of the Scoring Summary
  Sheet
• Start by noting practitioners strengths,
  followed by areas that might be improved
• Briefly review ratings and the basis for them
• Provide suggestions for addressing problems areas

26
CBT for PTSD Program

• 12-16 individual sessions, for consumers with SMI
  PTSD
• Components of intervention
• Orientation, crisis plan, breathing retraining (1
  session)
• Psychoeducation about PTSD (2 sessions)
• Cognitive restructuring, taught as a skill to
  manage distressing feelings address
  trauma-related thoughts beliefs (9-13 sessions)
• Supported by one RCT in SMI multiple other
  studies underway in other special populations
  (e.g., addiction, disaster survivors,
  adolescents)

27
CBT for PTSD Fidelity Scale

• Ratings based on audio recordings
• 17 items rated on 5-point Likert scales
• Narrative feedback on specific ratings
• Used to facilitate training certification of
  clinicians in treatment model
• Used to verify fidelity to model over treatment
  study

28
Items on CBT for PTSD Fidelity Scale

• 1. Agenda setting
• 2. Homework review
• 3. Overview of program
• 4. Crisis plan
• 5. Breathing retraining
• 6. Use of Educational materials
• 7. Psychoeducation
• 8. Cognitive restructuring

• 9. Problem solving
• 10. Trauma focus
• 11. Assign homework
• 12. Manual adherence
• 13. Teaching effectiveness
• 14. Interpersonal effectiveness
• 15. Pacing efficient use of time
• 16. Reduction of client distress
• 17. Overall session quality

29
NJ-Dartmouth PTSD Study

• Kim Mueser, Dartmouth, PI
• Stan Rosenberg, Dartmouth, Co-PI
• Jen Gottlieb, Dartmouth, Project Manager
• Steve Silverstein, UMDNJ, Site PI
• Weili Lu, UMDNJ, Site Project Manager
• Phil Yanos, UMDNJ, Clinical Supervisor
• Stephanie Marcello, UMDNJ, Clinical Supervisor
• Stephanie Dove-Williams, UMDNJ, Interviewer
• Danielle Hawthorne, UMDNJ, Interviewer
• Danielle Paterno, UMDNJ, Interviewer
• Lindy Fox, Dartmouth, Assessment Supervisor
• Rosemarie Wolfe, Dartmouth, Data Manager
• Eric Slade, University of Maryland, Health
  Economist

30
Training Model
Annual Training
Practice Cases Assigned Sessions Recorded
If fidelity for sessions 4-16lt3.5
Weekly On-site Group Supervision
Fidelity Monitoring Via Tapes
31
Training Model Monitoring treatment fidelity

• Frontline clinicians were provided with fidelity
  ratings for all sessions for their first practice
  cases
• If the overall fidelity ratings for sessions 4-16
  was above 3.5, clinician was certified to take on
  protocol case
• If overall fidelity was below 3.5, then the
  clinician took a second practice case
• 17/20 clinicians were certified after 1 case, 3
  after a second practice case

32
Participants
33
CBT Session Progress (N26)
34
Conclusions

• Clinical skills are useful to rate for both
  training purposes to verify fidelity
• Audio-based fidelity assessments are feasible to
  conduct with frontline clinicians working with
  regular clients
• Clinical skills assessments have potential to
  improve other practices (e.g., ACT, supported
  employment)