USMLE: Today and Tomorrow

1
USMLE Today and Tomorrow

• Donald E. Melnick, MD
• President, NBME

2
The National Board of Medical Examiners (NBME)

• Founded 1915 -- 93 years old
• Non-profit - 501(c)(3) - organization
• Initially focused on NBME Certification
• examinations of such high quality that they
  would meet the requirements of all state
  licensing authorities

3
NBME

• Based in Philadelphia
• Staff of 400
• Supported by examination fees
• Approximately 95 million annual revenues
• 350,000 tests annually

4
NBME Mission To protect the health of the
public through state of the art assessment of
health professionals. While centered on the
assessment of physicians, this mission
encompasses the spectrum of health professionals
along the continuum of education, training and
practice and includes research in evaluation in
well as development of assessment instruments.
5
NBME Mission To protect the health of the
public through state of the art assessment of
health professionals. While centered on the
assessment of physicians, this mission
encompasses the spectrum of health professionals
along the continuum of education, training and
practice and includes research in evaluation in
well as development of assessment instruments.
6
NBME Products and Services

• USMLE?
• Services for medical schools and students
• Subject tests (domestic and international)
• Customized tests
• Self-assessments
• Services for healthcare organizations
• Services for practicing doctors
• International collaboration
• Research development

7
USMLE Step 1 Pharmacology Committees
2007 Committee
8
Pharm Committee Members2005-2008

• Joseph Awad, MD
• Lynn Crespo, PhD
• Thomas Garrett, MD
• Stephen Geraci, MD
• Edward Ishac, PhD
• David Lehmann, MD
• Lionel Lewis, MD

• Michael Miyamoto, PhD
• Edward Morgan, PhD
• Patrick Murray, MD
• Norman Pedigo, Jr, PhD
• Sandra Roerig, PhD
• A. Shepherd, MD, PhD
• Eugene Silinsky, PhD

9
Other Committee MembersStep, IRC, CEUP, Step 2/3
TMDC, etc.
10
Retention Study - Method

• Representative sample of 502 scored items on
  2004-05 Step 1 test forms (550-1350 Step 1
  examinees/item)
• Embedded in unscored slots on 2004-05 Step 2
  test forms each examinee responded to a small
  number of randomly selected items (300 Step 2
  examinees/item)
• Shifts in examinee performance from Step 1 to
  Step 2 were analyzed in relation to item
  characteristics (content area, stem format,
  committee of origin)

11
Retention Study - Results

• Correlation between Step 1 Step 2 Item
  Difficulties 0.61

12
Sample Study ItemPharmacology Experimental
Vignette
13
Differences in RetentionVignette-Based vs
Non-Vignette Items
14
Differences in Retention Vignette-Based vs
Non-Vignette Formats
15
Changes in Performance by Committee of Origin
16
Basic Science Retention Pharmacology
100

80
60
Basic Science Item Difficulty on Step 2
40
Experimental Vignette
Non Vignette
20
Patient Vignette
0
0
20
40
60
80
100
Basic Science Item Difficulty on Step 1
17
Summary of Study Results

• Performance of 2004-05 Step 2 examinees was 6
  lower than 2004-05 Step 1 examinees
• Shifts in performance varied by
• Stem format (eg, patient vignette vs
  non-vignette)
• Discipline of the committee in which items were
  written
• Performance declines were largest in Biochemistry
  and Microbiology
• Behavioral science was the only discipline in
  which performance improved (9) from Step 1 to
  Step 2

18
Results of Five Studies Done Over the Past 30
Years
Change in Performance (Step 2 Step 1)
19
Discussion of Results

• Performance decrement from Step 1 to Step 2 was
  roughly twice that observed in previous studies
• Patterns of retention in relation to basic
  science discipline and item format were similar
  to previous studies
• Explanations for results are unclear without
  further study, but we dont appear to be
  achieving what we want to achieve

20
USMLE

• MISSION
• provide state licensure boards valid reliable
  independent assessments needed to support
  granting a primary license to practice medicine.

21
USMLE

• SECONDARY USES
• medical schools use results for curriculum
  assessment, promotion, graduation decisions.
• results considered by residency program directors
  during residency selection process.
• international graduates must pass Step 1, Step2
  CK, Step 2 CS for ECFMG certification.

22
USMLE Evolution

• Created by merging NBME, FLEX, FMGEMS in 1992
• Computer based testing 1998-9
• Step 2 CS 2004
• Comprehensive review planned in 2004
• Committee to Evaluate the USMLE Program (CEUP)
  formed in 2006

23
CEUP

• Students, residents, fellows, deans associate
  deans, basic science clinical faculty,
  international medical graduates, state medical
  board members, practicing physicians, the
  public.
• Some (but not all) have experience in USMLE
  program as item writers, reviewers, or test
  committee members.

24
CEUP Guiding Principles

• First priority of USMLE is to assure licensing
  authorities that candidates possess knowledge
  skills for safe effective patient care in both
  supervised unsupervised settings
• Secondary uses of USMLE results should be
  supported as long as the primary purpose of USMLE
  is not compromised
• USMLE assessment instruments should be valid
  reliable measures of competencies required for
  medical practice
• USMLE must continue to reflect evolving national
  consensus of competencies required for practice

25
Data Acquisition

• 20 focus groups
• State medical boards
• AMA, AAMC
• Regional groups of schools
• International graduates
• Basic science organizations
• Surveys
• State medical board presidents executives
• Deans, Academic Deans, Student Affairs Deans
• Students/residents
• International medical school deans
• Leaders of national medical organizations

26
Themes Decision Points

• Designed to support licensure decisions at two
  points
• Readiness to begin direct patient care under
  supervision (at interface between undergraduate
  graduate medical education).
• Readiness to provide unsupervised patient care
  to obtain a license for unrestricted practice.

27
Themes Integration Synthesis

• The science of medicine is essential to good
  medical practice.
• A separate examination of fundamental sciences
  creates an artificial, isolating gap.
• Weight of opinion favors integration of science
  concepts throughout all examination components.
• Widely expressed sentiment, including both basic
  clinical sciences faculty members.

28
USMLE Competencies

• USMLE should assess competencies relevant to safe
  and effective care wherever possible.
• Good Medical Practice-USA (National Alliance for
  Physician Competence)
• Patient Care
• Medical Knowledge Skills
• Practice-based Learning and Improvement
• Interpersonal Communications Skills
• Professional Behavior
• Systems-based Practice

Tr Tr
29
Proposed Model

• Gateway A Supervised medical practice
• Integrative and competency based
• Probably modular
• Pass/fail decisions likely at the level of
  competency
• Gateway B Independent practice
• Integrative and competency based
• May include new testing modalities under
  development targeting poorly tested competencies
• Includes focus (and perhaps pass/fail hurdle)
  related to medical knowledge

30
Possible Model
Gateway A
Gateway 2
Patient care
Patient care
Practice-based learning
Practice-based learning
Medical knowledge
Medical knowledge
Interpersonal commun
Interpersonal commun
Systems-based practice
Systems-based practice
Professional behavior
Professional behavior
S S S
31
Misconceptions

• This is merely a merger of current Step 1 2,
  subsequent administration of current Step 3 NOT
  the case.
• Results will be reported only as pass/fail NOT
  yet determined.
• Medical science will be averaged into clinical
  content NOT likely.
• No USMLE test will be available at the end of the
  pre-clinical curriculum NOT yet determined,
  derivative tests will continue.

32
Concerns from Basic Science Community

• Potential dilution of the value of basic sciences
• Loss of the opportunity for student integration
  of learning
• How will changes in the exam change curriculum?
• Can basic science departments meet the potential
  demand for more teaching in years 3-4 and during
  graduate education?
• Will a national assessment tool specific to the
  basic sciences be available?

33
Questions or Comments?