ABAASA JOINT COUNCIL ON INTRAINING EXAMINATIONS

1
ABA/ASA JOINT COUNCIL ON IN-TRAINING EXAMINATIONS

• 2005 REPORT TO SAAC/AAPD
• Glenn P. Gravlee, MD
• Chair, Joint Council
• The Ohio State University Medical Center
• Department of Anesthesiology
• Gravlee.1_at_osu.edu

2
2005 JOINT COUNCIL MEMBERS

• ABA
• David Chestnut
• Douglas Coursin
• Glenn Gravlee
• Patricia Kapur
• - Jim Di Nardo
• Raymond Roy
• Kenneth Tuman

• ASA
• Arnold Berry
• John Rowlingson
• Jeffrey Gross (V.C.)
• Philip Lebowitz
• Charles Otto
• Patricia Petrozza
• Mark Rosen (Robert Gaiser)

3
IN-TRAINING EXAMINATION

• 3 exams in 1
• In-Training Exam
• Requalifying Exam
• ABA Written Exam
• Subset of 300 questions from ITE
• Plus Recertification Exam
• Well-performing A-types from ITE form the
  question pool

4
Demographics of In-Training Examinations
Total Number of Examinees
Registrants
5
Test Center Issues

• Tulane and Univ of S. Alabama cancelled due to
  storm (not Katrina)
• Two alternate dates offered most took exam on
  July 30

6
DEMOGRAPHICS OF IN-TRAINING EXAMINATIONS - AMG
IMG
Number of Residents (n)
7
Exam Content 2005340 Questions 100 are Links
19
13
4
8
10
45
Number of Questions
8
Disease States Breakdown163 Questions
Number of Questions
7.2
5.7
5.7
4.9
5.1
4.6
4.6
4.6
4.3
9
2005 ITE Question Distribution
Percent of Questions ()
10
Item Difficulty (Mean Logit SE)
11
In-training ExamAverage Scaled Score
Scaled Score
All CA3 Residents
12
Gaps in Knowledge CA3s

• 10 Surprising Gaps Identified
• Report sent before ASA

13
Correctly Answered 2005 ITECalibration Group
Percent Correct Answers ()
14
Survey of Program Directors

• Sent to all residency programs in early 2005
• 95 of programs responded by 5/26/2005

15
Joint Council 2005 Survey of Residency Programs
Question Assuming that the In-training
Examination will be administered once a year, I
would prefer that the exam be given in
Percent ()
16
Joint Council 2005 Survey of Residency Programs
Question The importance of having the
in-training Examination be equivalent in content
to the ABA written examination is
Percent ()
17
Joint Council 2005 Survey of Residency Programs
Question Having the opportunity to administer
the In-training Examination in one, 3.5-hr, test
session instead of two, 3.5-hr test sessions
would be
Percent ()
Rationale May need to shorten for computerized
format
18
Joint Council 2005 Survey of Residency Programs
Question Having the opportunity to administer
the In-training Examination twice a year would
be
Percent ()
19
If the In-training examination should be
administered twice a year, it would be best to
Joint Council 2005 Survey of Residency Programs

• Have all residents take it both times 44
• Limit access on the second administration to
  residents with scores below a particular level
  (e.g. 20th percentile of their cohort) 34
• Have the second administration be optional but
  available to all residents 19
• Dont do it 1

20
Joint Council 2005 Survey of Residency Programs
The ideal way for residents to take the
In-training examination is

• On the same day at the same time 54
• During an eight-day testing window beginning and
  ending on a Saturday 46

21
Joint Council 2005 Survey of Residency Programs
The ideal way to deliver the In-training
examination to residents is via

• Printed test books 27
• Computer at a commercial computer-based testing
  center 21
• Internet to secure, proctored centers, which may
  be at the training institution 45
• Both B C 2
• No answer 5

22
Internet Based Exam

• ABA survey of directors and Joint Council members
  strongly suggests that internet-based secure exam
  capacity of most teaching institutions remains
  quite low
• Cant determine that anyone has yet done an
  internet-based exam on this scale
• AB Derm does it on a much smaller scale, and
  implementation was UGLY in 1st year

23
Future of ITE

• Computerized Exams target date 2008
• Logistic Issues
• Cost (seat fees, capacity limitations)
• Leads to separation of dates of ABA, ITE
• Leads to security concerns
• Requires larger question pool and larger number
  of exam forms while keeping content equivalent
  drives development cost up

24
Future of ITE

• Computerized Exam advantages
• Images more easily used and updated
• Fewer concerns about proctor consistency (?),
  enforcement of rules (cell phones/pagers,
  cheating)
• Facilitates interactive questions (expensive to
  develop)
• No exam booklets to print, ship, track, or lose
• Bottom Line We want to do it, but cost and
  other issues may hold us back. Dec meeting in
  Phil.

25
Future of ITE

• Consideration of grading or peer comparison for
  different subject areas, e.g.
• Physiology, Pharmacology, Anes Procedures
• Cardiovascular, Respiratory, OB anesthesia
• Section needs to have a critical mass of
  questions to achieve reliability
• (Some questions will fall into more than one
  category)

26
ABA Written Exam Failure Rate
Failure Rate ()
27
2005 ABA WRITTEN EXAMINATION PASS RATES1st Time
Takers
28
Pass Rates for 1st time takers of 2005 ABA
Written Exam Grad Class 2003 or 2004)
29
Joint Council UpdateSummary

• Exam population still growing Max now?
• 2005 resident exam performance EXCELLENT
• Computerized Exam On the way (?)
• Taking the ABA exam ASAP appears better than
  delaying it

30
(No Transcript)
31
(No Transcript)
32
Examination Registration/Behavior

• Government Issue Picture ID
• NO Cell Phones or Pagers
• Calculators are NOT OK
• Consequences of Cheating
• Proctors are in charge

33
Joint Council Finances

• Joint Oversight by ASA/ABA
• Largely sustained by Exam fees
• In the Red (50-100K) when N was low

34
Joint Council Finances

• Still losing 20-50K/year with strong N
  increased proctor compensation, NBME fees mainly
• Exam cost _at_ 85 since 1996
• Joint Council approved increase in exam fee to
  100 effective for 2005 exam

35
Content Outline Revision

• 1996 version used for 2004 Exam
• 2003 version used for 2005 Exam
• New Format Organocentric
• Both versions on ASA website
• 2004 version disappears 12/31/04
• www.asahq.org/publicationsandservices/contentoutli
  nerev2003

36
FAQs/Future Issues

• Computerized Exam?
• Phase out K-types?
• Introduce R-types and question clusters (G types)
  in 2005

37
50 Question Writers

• Title Junior Editors

38
IN-TRAINING EXAMINATION EDITORS

• Steven Allen, Audree Bendo, James Di Nardo,
  Carter Dodge, Sylvia Dolinski, John Ebert, John
  Emhardt, Robert Gaiser, Eric Kitain, Bruce
  Kleinman, Elliott Krane, Larry Krenis, Larry
  Kushins, Cynthia Lien, Vinod Malhotra, Donald
  Martin, Roger Mecca, John Moyers, Julia Pollock,
  Lindo Jo Rice, Robert Sladen, Richard Stypula,
  Richard Teplick, Helen Westman, Thomas Wolfe