Objective Structured Assessment of Technical Skills OSATS in Orthopedic Hand Surgery

1
Objective Structured Assessment of Technical
Skills (OSATS) in Orthopedic Hand Surgery

• Ann Van Heest, M.D.
• University of Minnesota
• ARCOS 2009

2
Outline

• Goals
• Background / Literature Review
• Research Study Methods
• Future Research
• Potential Applications

3
Goals

• An objective test(s) that assess trainee
  technical proficiency.
• Feasible
• Reliable
• Valid

4
Background

• Apprenticeship Model
• Observation
• Performance under supervision of preceptor
• Subjective evaluation
• Direct Observation - unreliable (Reznick, 1993)
• Retrospective - recall bias (Norcini, 2003)

5
Background

• Surgical Proficiency
• Knowledge
• Decision making ability
• Communication skills
• Formally appraised in written, clinical and oral
  examinations

6

• Reznick et al. (Am J Surg, 1996)
• Objective Structured Assessment of Technical
  Skills (OSATS)
• Standardized surgical tasks
• Inanimate models
• Scoring - task specific checklist and Global
  Rating Scale (GRS)
• N 48
• Interstation reliability 0.78 checklist, 0.85 GRS
• Evidence of Construct Validity

7
Background

• Datta et al. (Am J Surg, 2002)
• Imperial College Surgical Assessment Device
  (ICSAD)
• Motion analysis with electromagnetic tracker
• Number of movements made
• Retrospective video analysis
• Decreased hand movements, increased GRS,
  decreased time to completion with increasing
  seniority

8

• Datta et al. (J Am Coll Surg, 2004)
• No difference between bench test and OR
• GRS positively correlated with experience
• Videotape / Retrospective analysis
• Inter-rater reliability high gt0.80

9

• Dubrowski et al. (JBJS, 2004)
• Bone drilling
• N 11 expert surgeons
• N 15 PGY1 residents
• Artificial bone model with force-torque sensor
• Right arm of each participant instrumented with
  magnetic marker
• Significantly less plunging by experts

10
Methods

• University of Minnesota Department of
  Orthopaedics
• The Ramon B. Gustilo Center for Medical Education
  and Biomechanics Laboratory
• Upper Extremity Surgical Competency Testing
• May 2009
• Participants Medical students, PGY1-5,
  fellowship trained experts
• IRB approved

11
May 2, 2008 Hand Surgery Motor Skills Assessment

• WebVista Carpal Tunnel Pre-test
• Detailed Checklist (100 points)
• Carpal Tunnel Release Pass/Fail
• Global Rating
• Carpal Tunnel
• Trigger Finger
• Volar Plating
• Anatomic Structure Identification

12
HCMC Lab May 2, 2008
13
(No Transcript)
14
WebVista Carpal Tunnel Pre-Test
n 2 8 5 7 4 2
Program Year
p lt 0.001
15
WebVista Knowledge Pre-test

• However, although the absence of knowledge on
  knowledge testing can predict failure on OSAT
  testing, the presence of knowledge on knowledge
  testing does not predict success in OSAT testing.
  
• The knowledge test is a useful screen for those
  with sufficient knowledge (over 68 points) to
  warrant cadaveric testing.

16
OSA Pass / Fail
n 0 / 2 1 / 7 5 / 0 7 / 0 4 / 0 2 / 0
Program Year
p lt 0.001
17
Carpal Tunnel Check and re-check
18
Where is Guyons canal?
19
Carpal Tunnel Release Detailed Checklist (100pts)
n 2 8 5 7 4 2
p 0.002
Program Year
20
(No Transcript)
21
Global Rating Carpal Tunnel
n 2 8 5 7 4 2
Program Year
p 0.040
22
Global Rating Trigger Finger
n 2 8 5 7 4 2
Program Year
23
(No Transcript)
24
OSA Global Rating Volar Plating
n 2 8 5 7 4 2
Program Year
25
(No Transcript)
26
Anatomic Structure Identification(21 Items)
n 2 8 5 7 4 2
Program Year
27
May 2, 2008 Hand Surgery Motor Skills Assessment

• Grand Rounds Distal Radius Results
• Validation of Assessment Tools
• High stakes test after validated
• Repeat again May 1, 2009

28
Conclusions

• Construct validity
• There is significant correlation between level of
  training and
• Knowledge Testing
• Pass/Fail
• Global Rating Scale
• Detailed checklist
• A minimum Knowledge was necessary for passing
  cadaveric testing, but did not predict success at
  surgical skills.

29
Potential Applications

• Assessment of individual technical skills
• Establish goal related learning
• Competence Based Advancement
• Identify outliers
• Assessment of resident programs
• Development of curriculum
• Simulators
• Certification / Licensure

30
Proposed HCMC Lab May 1, 2009
Adverse Event
Debrief
Debrief
2 faculty reviewers 2 reviewers blinded
Debrief
31
Bibliography

• Reznick RK. Teaching and testing technical
  skills. Am J Surg. 1993165358-361.
• Norci JJ, Blank LL, Duffy FD. The mini-CEX a
  method for assessing clinical skills. Ann Intern
  Med. 2003138476-481.
• Reznick RK, Regehr G, MacRae H, Martin J,
  McCulloch W. Testing technical skill via an
  innovation bench station examination. Am J
  Surg. 1997173226-230.
• Datta V, Chang A, Mackay S, Darzi A. The
  relationship between motion analysis and surgical
  technical assessments. Am J Surg. 2002 184
  70-73.
• Datta V, Bann S, Beard J, Mandalia M, Darzi A.
  Comparison of bench test evaluations of surgical
  skill with live operating performance
  assessments. Am Coll Surg. 2004 603-606
• Doyle JD, Webber EM, Sidhu RS. A universal
  global rating scale for the evaluation of
  technical skills in the operating room. Am J
  Surg. 2007 193 551-555
• Dubrowski A, Backstein D. The Contributions of
  Kinesiology to Surgical Education. JBJS 2004
  86 2778-2781.

32
Thanks to Drs. Putnam, McPherson, House, Kuzel,
Shanedling, Schmitz, and the U of M Residents