Application screening tool as a predictor of resident performance

1
Application screening tool as a predictor of
resident performance

• Anne Campbell Larkin, MD FACS
• Jeannine St. Pierre
• Giles Whalen, MD FACS
• UMass Memorial Department of Surgery

2
Background

• Methodology of reviewing resident applications is
  imprecise
• Difficult to weigh certain aspects of an
  application
• Process is time-consuming and tedious

3
Background

• Medical school honors and election to AOA
  correlated with several variables at the junior
  level
• EE Erlandson Surgery 92(2) 270-75
• Number of honors grades correlates with overall
  performance, but not performance on standardized
  tests
• DR Dirschl Clin Ortho Rel Res 399265-71
• High variability between institutions in grade
  assignment
• H Takayama Curr Surg 63 391-96

4
Application Scoring Tool

• Developed by the Program Director
• Consensus by faculty and members of the Surgical
  Education Committee
• Represented specific qualities deemed important
  to the department

5
Items

• Transcript
• Quality of medical school
• USMLE scores
• Research
• Letters of recommendation
• Student known to us

• Service ethic
• Balance
• Leadership skills
• Achievement in other fields
• Character and integrity
• Personal statement

6

• Standardized approach to applicants offered
  interviews
• Unclear benefit in terms of predictive value

7
Methods

• Application scores collected for all categorical
  residents entering in 2004 and 2005
• Other data points collected
• Interview scores
• ABSITE scores
• Performance scores
• Subjective rankings
• Approved by the IRB

8
Interview Scores

• Likert scale 0-10
• Aspects of interview
• Motivation
• Appearance
• Personality
• Friendliness
• Poise
• Non-blinded to application

9
Trainee Rank Report

• EValue web-based evaluation system
• Evaluations are competency-based
• Completed by faculty and peers
• Report was generated on a 0-100 percent scale

10
Subjective Ranking

• Completed by 17 faculty/residents
• Numerical rank assigned to each junior resident
• Based on opinion of overall performance
• Given no additional materials on which to base
  this decision
• Rank numbers were averaged for final subjective
  rank

11
Class Entering in 2004
12
Class entering in 2005
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
Conclusion

• An application scoring tool has value in
  predicting resident performance at the junior
  levels.
• Limitations include small sample size, single
  institution
• Continued study and analysis is warranted

24
Acknowledgements

• Jeannine St. Pierre, Program Coordinator
• Giles Whalen, MD, Vice Chair for Education
• Mary Zanetti, PhD and colleagues
• UMass Memorial Surgical Residents