Residents In Trouble

1
Residents In Trouble

• M. Rebecca Hoffman, MD, MSPH
• December 8, 2008
• Academy for Scholarship in Education

2
Overview

• Scope of the Problem
• Literature Review
• Study Design
• Study Results (in progress)

3
Scope of the Problem

• The goal of Family Medicine Residency
• Residents backgrounds
• No guidance

4
What does the RRC say?

• Must have advising system
• Must give regular feedback
• BUT no real guidance regarding what/how

5
Consequences of RIT

• For the individual RIT
• Psychological stress
• Financial stress
• Eventual job security
• For the program
• Adequate coverage of duties
• Morale
• Faculty time/energy
• Dismissal

6
Consequences

• Other consequences
• Recruiting
• Impact on patient care

7
Literature Review

• Not much on the prevalence of RIT
• Reamy et al (1) 25 year in-depth study of one
  family medicine residency program
• 9.1
• Yao and Wright (2) Survey of Internal Medicine
  PDs
• 6.9
• Williams et al (3) SIU Gen Surgery Residency
• 22

8
Kinds of Problems

• Knowledge Deficits
• Attitudinal Problems
• Interpersonal Conflict
• Psychiatric Illness
• Substance Abuse
• Family Stress
• Relationship Disruption

9
Other Classification

• Academic Performance
• Classroom, test
• Clinical Performance
• Applied knowledge/skills
• Professional Behavior
• Workplace behavior, relationships with other
  professionals, etc
• Others
• Difficult to classify

10
So we can name it

• But what do we do with the information?
• Literature reviews
• Need to do better
• Core content review, quizzes
• Counseling
• Treatment of illnesses
• Repetition (rotation, year)
• Probation
• Dismissal
• Others

11
Summary of the problem

• We know its there, but we arent sure exactly
  how common it is
• We arent good at predicting who will struggle
• Once we identify them, we dont know what to do
  with them

12
RIT Family Medicine Study

• Primary Aims
• 1) Determine the prevalence of RIT in the SIU
  Family Medicine Residency Programs
• 2) Describe and categorize the types of troubles
  encountered in the FCM residency programs

13
RIT Aims...

• 3) Identify and describe the kinds of processes
  the programs employ to address deficiencies with
  RIT
• 4) Describe outcomes of the residents who
  underwent remediation
• 5) Identify early indicators to predict RIT

14
RIT Study Design

• Chart review/resident record -based
• Prevalence over significant timeframe
• 10 years of entering classes
• Entering class of 1993-entering of 2003
• Why?
• De-identified data from all 4 programs
• To further assure confidentiality
• No one program singled out

15
RIT Study

• Initially modeled after Reamy et al study (1)
  after discussion with Dr. Reamy about the process
  in his residency program
• Assistance from Reed Williams, PhD and Nicole
  Roberts, PhD

16
Methods

• Rosters of all entering residents for the 10
  years from all programs
• Random ID numbers assigned
• Records from each program reviewed in their
  entirety
• Mostly paper
• Some electronic

17
Methods

• Data collection instrument created based on
  personal experience, literature, and discussion
  with Reed Williams and Nicole Roberts
• Feedback from academic faculty
• Suggestions incorporated as needed

18
Collected for all residents

• US Medical School Grad (Y/N)
• USMLE or NBOME or COMLEX scores
• Step 1, Step 2, Step 3 (when available)
• Gender (M/F)
• ITE Scores
• PGY1, PGY2, PGY3 (when available)
• Did the resident have substantial difficulties?
  (Y/N)

19
For those who struggled

• Time frame- when first identified
• Categorization of the main problem
• Academic, Clinical, Professionalism, Other
• Main performance area involved
• See list
• Breadth of performance areas
• Academic, Clinical, Professional, Other
• All performance areas involved
• See list

20
For those who struggled

• What remediation attempted?
• See list
• What was the final resolution?
• Graduated with concerns
• Graduated without concerns
• Voluntarily left
• Dismissed
• Finished on probation
• Not documented

21
Severity Rating

• Compared to other residents with performance
  problems, this residents problems were
• Likert scale, 1-5
• 1 among the least serious
• 5 among the most serious
• Narrative descriptions of overall case, time
  course, etc

22
Example of Survey Instrument

• Survey Monkey for data entry

23
Data Collection

• 2 raters
• Independently reviewed same 5 charts at first
  program to establish inter-rater reliability,
  then 3 charts at each other program

24
Data Collection

• Then independent record review
• Any case the reviewer identified as a RIT was
  then reviewed by the other reviewer
  independently, ratings discussed after each case
• Borderline cases also reviewed by both

25
Data Analysis

• Data entered into Survey Monkey for ease of
  collection
• Analysis descriptive and basic statistics
• When all data complete, analysis using SPSS

26
Results

• 96 Residents reviewed (as of 12/4/2008)
• US Med School Grads 69 (66)
• Male 65 (62)
• Female 35 (34)
• Prevalence of documented RIT 38.5 (37)

27
RIT (N37)

• US graduates 59.5 (22)
• Foreign graduates 40.5 (15)
• Male 70.3 (26)
• Female 29.7 (11)
• Male, US 40.5 (15)
• Male, Non-US 29.7 (11)
• Female, US 18.9 (7)
• Female, Non-US 10.8 (4)

28
RIT
29
Most Important Single Problem Area?

• Knowledge (10)
• Putting everything together (3)
• Data interpretation/diagnosis (2)
• Data collection (2)
• Incomplete paperwork/charts (2)
• Lack of motivation/interest (2)
• Treatment/management (2)

30
Breadth of Problems
31
Breadth of problems

• Number of performance areas involved ranged from
  1 to 23
• Average 7
• See breakdown of individual RIT on handout

32
Remediation

• Ranged from none or told to improve to 12 or
  more interventions for a single resident
• Most commonly reported Increased meetings with
  advisor/mentor

33
Outcomes of RIT (residency)

• 26 graduated from the program (70.3)
• 11 did not graduate (29.7)
• Left voluntarily 6
• Changed specialty 2
• Dismissed 3
• More complete follow up not complete
• Have initiated licensure status and board
  certification F/U but too little data to report

34
Outcomes
35
Conclusions?

• RIT more common than suspected based on lit.
• Increased prevalence compared to other studies
• ? Over-diagnosis
• For mild cases
• ? Truly higher prevalence
• Due to population?
• Due to program characteristics?
• All RIT who had concerns in more than 8 areas had
  ongoing concerns at graduation or did not
  complete program
• RIT who struggled in all 3 performance areas also
  either did not complete or graduated with ongoing
  concerns
• Professionalism issues very common

36
Problems Encountered

• Information in files varies between programs
• Missing data
• Esp. test scores (fortunately, may be able to
  acquire)
• Even within programs, varying data
• Classification of single most important area for
  major problem residents
• We collected data for an additional group of
  residents those with ITE performance less than
  10th ile, even if no other problems, will follow
  those as well (n4 right now)

37
To come!

• Full data reporting (approx 270)
• Including USMLE and ITE data
• Predictive model?
• Follow up data (post-graduation)

38
Special Thanks to

• Nicole Roberts, PhD data spelunker
  extraordinaire
• Reed Williams, PhD
• Steve Verhulst, PhD
• Jerry Kruse, MD, MSPH
• SIU Family Medicine Residency Program Directors
• Janet Albers, MD
• John Bradley, MD
• Tom Miller, MD
• Penny Tippy, MD
• Wiley Jenkins, PhD

39
Referenced Articles

• 1. Reamy BV, Harman JH. Residents in trouble an
  in-depth assessment of the 25-year experience of
  a single family medicine residency. Fam Med
  200638(4)252-7.
• 2. Yao DC, Wright SM. National survey of internal
  medicine residency program directors regarding
  problem residents. JAMA 20002841099-1104.
• 3. Williams RG et al. The nature of general
  surgery resident performance problems
  (publication pending)