Standard Setting for a PerformanceBased Examination for Medical Licensure

1
Standard Setting for a Performance-Based
Examination for Medical Licensure

• Sydney M. Smee
• Medical Council of Canada

Presented at the 2005 CLEAR Annual
Conference September 15-17 Phoenix,
Arizona
2
MCC Qualifying Examination Part II

• OSCE format - 12 short stations
• 5 or 10 minutes per patient encounter
• Physicians observe and score performance
• Required for medical licensure in Canada
• Prerequisites
• Passed MCCQE Part I (Knowledge Clinical
  reasoning)
• Completed 12 months of post-graduate clinical
  training
• Pass/Fail criterion-referenced examination
• Multi-site administration - twice per year
• Overall fail rate 10-30
• Implemented 1992

3
Why do it?

• Requested by licensing authorities, largely in
  response to two issues
• Increase in complaints, many centered around
  communication skills.
• Public accountability - OSCE to serve as an
  audit of training of all candidates seeking
  licensure in Canada.

4
Blueprint Considerations

• Four domains
• History-taking Patient
  Interaction Physical Examination
  Management
• Multi-disciplinary / multi-system content
• Patient demographics
• Two formats
• 55 couplets 10 minute
• Each case based on an MCC Objective

5
Standard for MCCQE Part II

• Acceptably competent for entry to independent
  practice
• Conjunctive standard
• Pass by total score
• AND
• Pass by minimum number of stations
• High performance in a few stations does not
  compensate for overall poor performance
• Just passing enough stations does not compensate
  for overall poor performance

6
Translating a Standard to a Pass Mark

• Pilot exam Ebel method
• Items rated for relevance and importance
• Pass based on most relevant and important items
• Failed 40
• First two administrations Angoff method
• Estimated score for the minimally competent
  candidate
• Pass based on average of estimates per instrument
• Pass marks varied more than the test committee
  liked
• Test committee did not like the task
• 1994 Adopted borderline group method

7
Physicians as Scorers

• Three Assumptions
• Clinicians do not require training to judge
  candidate behaviour according to checklists for
  basic clinical skills
• Most clinicians can make expert judgments about
  candidate performance
• Being judged by clinicians is vital for a
  high-stakes examination

8
Physicians as Standard Setters
9
Global Rating Question

• Did the candidate respond satisfactorily to the
  needs/problems presented by this patient?

• Borderline Unsatisfactory
• Unsatisfactory
• Inferior

• Borderline Satisfactory
• Good
• Excellent

10
Numbers....

• ?1,000-2,200 candidates per administration
• Examiners each observe 16-32 candidates
• 20-60 examiners per case
• Number of candidates identified as borderline per
  case ranges from ?150-500
• Collect gt99 of data for global rating item

11
Modified Borderline Group Method

• Examiners (content experts) identify borderline
  candidates based on the 6-point scale
• Scores of borderline candidates define
  performance that describes the pass standard
• Examiner judgments are translated into a pass
  mark by taking the mean score for the borderline
  candidates for each case

12
Pass Marks by Case Across Exams

• Challenge to assess pass marks over multiple
  administrations
• Scoring instruments are revised post-exam
• Rating scale items have been revised
• Rating scale items have been added to cases
• As competency and difficulty of cases changes, so
  do cut scores

13
Setting Total Exam Pass Mark

• Pass marks for cases are summed
• Add one standard error of measure (?3.2 )
• Pass mark falls between 1 to 1.5 SD below mean
  score
• Station performance is reviewed by Central
  Examination Committee
• Then the standard for the number of stations
  passed is set
• Standard has been 8/12 since 2000

14
Outcomes

• 15,331 candidates became eligible in 2000 2005
• 6,099 have yet to attempt MCCQE Part II
• 8,514 have passed
• 718 or 7.7 failed
• 2,243 candidates were eligible prior to 2000 and
  also took MCCQE Part II in 2000 2005
• 2,166 have passed
• 77 or 3.4 failed and are likely out of the
  system
• Fail rates do not reflect impact on repeat takers
• Focused hundreds of candidates on remediation

15
Limitation

• Current approach is easy to implement but it
  relies upon
• Large number of standard setters per case
• Large number of test takers in borderline group
• Smaller numbers would lead to more effort
• Increase training of examiners
• Impose stricter selection criteria on standard
  setters

16
Whats ahead?

• Increasing number of candidates to be assessed
  each year
• Modifications to the administration are needed
• Predictive validity study currently in progress
• Use non-physician examiners?
• Which type of cases, who sets standard?
• Add more administrations?
• Case development / challenge of piloting content

17
A Sample of References

• Burrows, P. J., Bingham, L., Brailovsky, C. A.
  (1999). A modified contrasting groups method used
  for setting the passmark in a small scale
  standardized patient examination. Advances in
  Health Sciences Education Theory and Practice,
  4, 145-154.
• Cohen, A. S., Kane, M. T., Crooks, T. J.
  (1999). A generalized examinee-centered method
  for setting standards on achievement tests.
  Applied Measurement in Education, 12, 343-366.
• Dauphinee, W. D., Blackmore, D. E., Smee, S. M.,
  Rothman, A. I., Reznick, R. K. (1997). Using
  the judgments of physician examiners in setting
  the standards for a national multi-center high
  stakes OSCE. Advances in Health Sciences
  Education Theory and Practice, 2, 201-211.
• Haladyna, T. M. Hess, R. (2000). An evaluation
  of conjunctive and compensatory standard-setting
  strategies for test decisions. Educational
  Assessment, 6, 129-153.
• Hambleton, R. K., Jaeger, R. M., Plake, B. S.,
  Mills, C. (2000). Setting performance standards
  on complex educational assessments. Applied
  Psychological Measurement, 24, 335-366.
• Jaeger, R. M. (1995). Setting standards for
  complex performances An iterative, judgemental
  policy-capturing strategy. Educational
  Measurement Issues and Practice, 14, 16-20.
• Kane, M. T., Crooks, T. J., Cohen, A. S.
  (1999). Designing and evaluating standard-setting
  procedures for licensure and certification tests.
  Advances in Health Sciences Education Theory and
  Practice, 4, 195-207.
• Kaufman, D. M., Mann, K. V., Muijtjens, A. M. M.,
  van der Vleuten, C. P. M. (2001). A comparison
  of standard-setting procedures for an OSCE in
  undergraduate medical education. Academic
  Medicine, 75, 267-271.
• Plake, B. S. (1998). Setting performance
  standards for professional licensure and
  certification. Applied Measurement in Education,
  11, 65-80.
• Smee, S. M. Blackmore, D. E. (2002). Setting
  standards for an objective structured clinical
  examination The borderline group method gains
  ground on Angoff. Medical Education, 35,
  1009-1010.
• Southgate, L., Hays, R. B., Norcini, J. J.,
  Mulholland, H., Ayers, B., Woolliscroft, J.,
  Cusimano, M. D., MacAvoy, P., Ainsworth, M.,
  Haist, S., Campbell, M. (2001). Setting
  performance standards for medical practice A
  theoretical framework. Medical Education, 35,
  474-481.
• Wilkinson, T. J., Newble, D. I., Frampton, C.
  M. (2001). Standard setting in an objective
  structured clinical examination use of global
  ratings of borderline performance to determine
  the passing score. Medical Education, 35,
  1043-1049.
• Zieky, M. J. (2001). So much has changed How the
  setting of cutscores has evolved since the 1980s.
  In G.J.Cizek (Ed.), Setting performance
  standards Concepts, methods, and perspectives
  (pp. 19-52). Mahwah, NJ Lawrence Erlbaum
  Associates.

18

• Medical Council of Canada
• Ottawa
• Sydney M. Smee, M.Ed.
• Manager, MCCQE Part II
• www.mcc.ca