HFMG Template 607

1
Demystifying the Competencies
Barbara L. Joyce, Ph.D.
2
Objectives

• Identify examples of educational content for each
  of the competencies
• Identify an assessment system for your program
• Identify a program improvement plan for your
  program

3
What does the ACGME expect?

• Your program should document and demonstrate
• Learning opportunities in each competency domain
• Evidence of multiple assessment methods
• Use of aggregate data to improve the educational
  program

4
The Six Competencies

• Medical Knowledge
• Patient Care
• Practice-based Learning and Improvement
• Systems-based Practice
• Interpersonal and Communication Skills
• Professionalism

5
Practice-Based Learning and Improvement

• Quality Improvement
• Evidence based medicine
• Teaching others (patients, students, junior
  residents)
• Developing habits of life long learning and self
  reflection

PDSA Cycle Quality Improvement
6
What does this look like in different specialties

• Evidence-based Medicine (EBM)
• Internal Medicine
• Course in Basics of EBM and Journal Club with
  specific format residents evaluated
• Online modules through Sladen Library on
  Information Literacy
• Teaching Others
• Emergency Medicine
• Didactics and workshop in being an effective
  teacher
• Residents evaluated on teaching skills by medical
  students
• Lectures given by residents evaluated by
  participants

• QI Activities
• Pathology
• Residents form teams and actively engage in
  process improvement to reduce wait time for
  patients getting results from breast biopsy
  Participate in Quality Expo
• OB/GYN
• M M drives quality improvement activities of
  residents
• Internal Medicine
• PIM Modules
• Development of learning plans
• Review by PD at biannual meeting

7
Practice-Based Learning and ImprovementSelf
Assessment

• Do residents actively participate in Quality
  Improvement initiatives?
• Do you have an EBM based Journal Club? Do you
  use EBM when teaching?
• Do you have structured activities where residents
  learn to teach others?

• How do residents reflect on and improve their
  practice?
• Do residents develop learning plans to review
  with mentor/PD?
• Do you have an assessment tool for assessing a
  residents teaching skills?

Reflect
8
Systems-Based Practice

• Understanding the Health Care Delivery System
• Patient Advocacy
• Working in Inter-professional Teams
• Patient Safety

9
What does this look like in different specialties?

• Cost effective health care
• Allergy Immunology
• Review of prescribing patterns of faculty and
  fellows discussion and recommendations to
  pharmacy re formulary
• Family Medicine
• Practice management curriculum
• Interprofessional Teamwork
• Internal Medicine
• Collaborative Rounding
• General Surgery
• In Situ Simulation
• Emergency Medicine
• Mock codes

• Patient Safety and Advocacy
• All
• Residents present root cause analysis of near
  miss or sentinel event residents attend
  sentinel event committees
• Link M M conferences to all six competency
  domains or QI projects
• Understanding the healthcare system
• All
• Inform and engage residents in the national
  healthcare debate
• Didactics on insurance, billing coding,
  performance metrics (pay for performance),
  disease/case management

10
Systems-Based PracticeSelf Assessment

• Do residents have a learning activity that
  addresses interprofessional teamwork?
• Do residents have a learning activity that
  addresses patient advocacy?

• Do residents actively participate in M M?
• Do residents have a learning activity that
  addresses
• Cost containment
• Risk-benefit analysis

Reflect
11
Interpersonal and Communication Skills

• Communication with patients and families
• Communication with colleagues and other health
  professionals
• Written and scholarly communication

12
What does this look like in different specialties?

• Communication with Colleagues
• Internal Medicine
• Standardized handoff (ISWITCH)
• Diagnostic Radiology
• Clinical and didactic online teaching about
  providing effective consultations
• Written Communication
• All

• Communication with Patients and Families
• Institutional Curriculum
• Informed Consent
• Delivering Bad News
• Disclosure of Errors
• Hand off
• Family Medicine
• Workshop role plays around setting an agenda for
  the visit
• Emergency Medicine
• Didactics with role play on abuse screening,
  managing angry patients and families, screening
  for depression, alcohol

13
Interpersonal and Communication SkillsSelf
Assessment

• What experiential learning activity do residents
  engage in to learn communication skills with
  patients and families?
• What experiential learning activity do residents
  engage in to learn teamwork

• How do residents learn scholarly communication ?
• Do you routinely review residents medical
  records and give written feedback?

Reflect
14
Professionalism

• Demonstrating respect, compassion and integrity
• Commitment to ethical principles
• Being respectful of diversity

15
What does this look like in different specialties?

• AMA Virtual Mentor
• Ethics Conference
• Clinical teaching which actively embeds ethical
  issues (futility of treatment) into discussion
• Role modeling mentoring

16
ProfessionalismSelf Assessment

• Do residents have an experiential activity that
  addresses ethics?
• How is the development of professional behavior
  taught?
• What kinds of conversations do faculty have with
  residents re professional behavior?
• Describe the remediation process

Reflect
17
Assessment

• How do I know a resident
• is competent?

18
Why Is This Important?
19
Assessment Pearls

• Assess residents longitudinally
• Use multiple evaluations
• Use multiple evaluators who give different
  perspectives

20
And in the end
You have enough data to paint a picture of
competence
21
What is an assessment system?

• An assessment system
• Is a collection of assessment tools that measure
  a residents performance
• Defines who the evaluators are
• Describes what performance will be evaluated
• Indicates how often the evaluation occurs

22
A Core Assessment System
23
A Core Assessment System
24
Global Clinical Performance Ratingsaka End of
Rotation Evaluation

• Most common form of assessment in residency
  programs
• Align questions with key components of the
  competencies
• Most often ratings are inaccurate

• Add behavioral anchors
• Train faculty
• Least effective in predicting a good doctor

25
Behavioral Anchors
26
Multisource Feedbackaka 360

• Evaluation of resident by multiple individuals
  such as self, peers, nurses, technicians, allied
  health professionals, and patients
• Provides multiple perspectives of residents
  performance
• Provides opportunity for residents to improve
  self-assessment skills

27
Direct Focused Observation The Most Important
Form of Assessment!!!!!!

• Observation of Patient Care Encounter
• SEGUE
• Communication Checklist
• Mini-CEX
• Observation of Procedural Skill
• OSATS

28
CognitiveProcedure/Case Logs

• Cognitive
• In-training exams
• Multiple Choice exams
• Procedure/Case Logs

29
Portfolio

• A collection of authentic and diverse evidence
  representing what a person has learned over
  timeused to demonstrate competence and
  reflection on learning.
• Lewis 2007

30
Portfolio ? File
31
Portfolio

• Structured component
• External assessments of performance
• Reflective component
• Creative piece(s) that are learner driven
• Reflection and self-assessment of contents
• Used as a learning tool
• Formative feedback
• Resident and faculty-driven process
• Continuous

32
Learning Plans

• Promote self reflection
• Encourages the resident participation in
  defining their learning goals
• Promotes life long self directed learning

33
Other Forms of Assessment

• Peer Evaluation
• Chart audit
• Simulation/OSCEs/In Situ Simulation
• Mock orals
• .and many others

34
Program Improvement
Using aggregate data
Where does this data come from?
Aggregate what?
Lets not re-invent the wheel
I dont have time!!

35
Phase 3Outcome Project
36
Why is program improvement important?

• Focuses faculty and residents on a shared vision
  for the educational program
• Identifies effective educational practices within
  the program
• Identifies educational practices in need of
  improvement

• Communicates educational effectiveness to
  external stakeholders (GMEC, hospital
  administration, ACGME)
• Metrics are useful for recruitment

37
What is Program Evaluation and Improvement?

• Program evaluation
• Systematic evaluation of the residency program
  using multiple sources of data
• Program improvement
• Determining an improvement plan
• Measuring improvement over time

38
Two Components
One Process

• Program evaluation
• Program improvement

39
Next Steps

• Identify program performance indicators
• Organize the indicators
• Determine performance standards
• Develop an action plan for improvement

40
Departmental Performance Indicators

• Core Performance Indicators
• Performance on in-training exams compared to
  national average
• Rolling board certification pass rate
• Resident evaluation of educational program
• Faculty evaluation of educational program
• Resident evaluation of faculty
• Alumni surveys (1 year)
• Case and/or procedure logs
• Cycle length and number of citations by GMEC or
  ACGME
• Faculty Development
• Recommended or required by ACGME

41
Additional Departmental Performance Indicators

• Program evaluations done by residents (aggregate)
• Group performance on specialty specific or
  program performance standards
• of procedures of specific cases overall in
  training exam scores
• ACGME Resident survey
• Institutional Surveys

• Areas in need of improvement
• (rotation evaluations faculty evaluations focus
  group)
• Match rate
• Number of scientific presentations/publications
• Patient satisfaction surveys

42
Clinical Outcome Measures

• The Pulmonary/Critical Care Medicine fellowship
  at Henry Ford Hospital compared their ICU
  clinical outcomes to national benchmarks
• Patient severity of illness
• Severity-adjusted mortality rates
• Sepsis-specific mortality rates
• Iatrogenic pneumothorax rates
• Ventilator-associated pneumonia rates
• ICU length-of-stay
• Catheter-related blood-stream infections

43
Methods

• Reinforce practice patterns and knowledge for
  outcomes better than benchmark data
• Incorporate a PDSA cycle to clinical processes
  where outcomes are worse than benchmark data
• Identify roles of key elements
• Educational components
• Personnel
• Environmental factors
• Involve learners in PDSA cycle

44
Iatrogenic Pneumothorax

• PLAN
• Operational analysis Residents place CVCs
• Resources were directed toward improving
  technical performance of this procedure
• DO
• Expand educational curriculum
• Add simulation training
• STUDY
• Rates improved
• Specific program elements reviewed
• ACT
• Expand use of ultrasound guidance, PICCs

45
Pneumothorax rate Occurences / Patients-at-risk
46
Sepsis-specific Mortality Rates

• PLAN
• Review of provider knowledge Good
• Strategies were focused on implementation of
  therapy
• DO
• Incorporate a sepsis bundle of five
  interventions
• STUDY
• Bundle compliance initially low, but feedback
  helped
• Rates improved
• ACT
• Rapid Response Team now initiates sepsis bundle
  on the general practice units

47
Henry Ford Hospital Sepsis Mortality
Sepsis Bundled Initiated June 2006
July 2007
Sepsis Bundle Compliance

• Mich. Hosp Assoc. expected 13.5
• Henry Ford Hospital
• July 2007 11.5
• 2004 vs. 2007(YTD) 29

48
Performance Better Than Benchmarks
49
Develop an action plan

• Programs required to have yearly curriculum
  review
• Programs required to document action plan
• Document that action plan!!

50
Questions?

• Barbara Joyce, Ph.D. bjoyce2_at_hfhs.org
• 313-916-9161