Performancebased standard setting methods

1
Performance-based standard setting methods

• Katharine Boursicot
• Trudie Roberts

2
Performance-based standard setting methods

• Borderline group method
• Contrasting group method
• Regression based standard method

Kramer A, Muijtjens A, Jansen K, Düsman H, Tan L,
van der Vleuten C Comparison of a rational
and an empirical standard setting procedure for
an OSCE, Medical Education, 2003 Vol 37 Issue 2,
Page 132 Kaufman DM, Mann KV, Muijtjens AMM, van
der Vleuten CPM. A comparison of standard-setting
procedures for an OSCE in undergraduate medical
education. Acad Med 2000 75267-271.
3
(No Transcript)
4
The examiners role in standard setting

• Uses the examiners clinical expertise to judge
  the candidates performance
• Examiner allocates a global judgement based on
  the candidates performance at that station
• Remember the level of the examination

Pass
Borderline
Fail
5
Borderline Group Method
Test score distribution
Checklist
1. Hs shjs sjnhss sjhs sjs sj 2. Ksks
sksmsiqopql qlqmq q q qkl 3. Lalka kdm ddkk
dlkl dlld 4. Keyw dd e r rrmt tmk 5. Jfjfk
dd 6. Hskls skj sls ska ak akl ald 7.
Hdhhddh shs ahhakk as TOTAL
?
?
?
?
Borderline score distribution
?
Pass, Fail, Borderline
Passing score
6
Contrasting groups method
Test score distribution
Checklist
1. Hs shjs sjnhss sjhs sjs sj 2. Ksks
sksmsiqopql qlqmq q q qkl 3. Lalka kdm ddkk
dlkl dlld 4. Keyw dd e r rrmt tmk 5. Jfjfk
dd 6. Hskls skj sls ska ak akl ald 7.
Hdhhddh shs ahhakk as TOTAL
?
?
?
Pass
Fail
?
?
Pass, Fail, Borderline P/B/F
Passing score
7
Regression based standard
Checklist
X passing score
1. Hs shjs sjnhss sjhs sjs sj 2. Ksks
sksmsiqopql qlqmq q q qkl 3. Lalka kdm ddkk
dlkl dlld 4. Keyw dd e r rrmt tmk 5. Jfjfk
dd 6. Hskls skj sls ska ak akl ald 7.
Hdhhddh shs ahhakk as TOTAL
?
Checklist Score
?
X
?
?
?
Overall rating 1 2 3 4 5
Clear Borderline Clear Excellent
Outstanding fail pass
1 Clear fail 2 Borderline 3 Clear pass 4
Excellent 5 Outstanding
8
Performance-based standard setting

• Advantages
• Utilises the expertise of the examiners
• they are observing the performance of the
  students
• they are in a position to make a (global)
  judgement about the performance
• based on
• their clinical expertise
• expected standards for the level of the test
• knowledge of the curriculum/teaching

9
Borderline/ contrasting/ regression-based methods

• Advantages
• Large number of examiners set a collective
  standard while observing the candidates not
  just an academic exercise
• Reliable cut-off score based on large sample of
  judgments
• Credible and defensible based on expert judgment
  in direct observation

10
Performance-based standard setting

• Disadvantages
• Requires large cohort of candidates to achieve
  enough numbers in the borderline group
• Passing score not known in advance
• Judgments not independent of checklist scoring
• Requires expert processing of marks immediately
  after the exam
• Checking of results
• Delay in producing results