Principles of Good Assessment and Standard Setting

1
Principles of Good Assessment and Standard Setting

• Katharine Boursicot
• Reader in Medical Education
• St Georges, University of London

2
Principles

• Examine what is being assessed
• Review methods/ blueprinting
• Consider if formats suited to needs
• Standard setting

3
Critical questions in assessment

• WHY are they doing the assessment?
• WHAT are they assessing?
• HOW are they assessing it?
• HOW WELL is the assessment working?

4
1. WHY are they doing the assessment?

• Is its purpose
• Formative?
• Summative?

5
2. WHAT are they testing?

• Elements of competence
• Knowledge
• factual
• applied clinical reasoning
• Skills
• communication
• clinical
• Attitudes
• professional behaviour
• Tomorrows Doctors, GMC 2003

6
3. How are they doing the assessment? Test
formats
Does
Shows how
Knows how
Knows
7
4. HOW WELL is the assessment working?

• Evaluation of assessment systems
• Is it valid?
• Is it reliable?
• Is it doing what it is supposed to be doing?
• To answer these questions, we have to consider
  the characteristics of assessment instruments

8
Utility function
U wrR x wvV x weE x waA x wcC

• U Utility
• R Reliability
• V Validity
• E Educational impact
• A Acceptability
• C Cost
• W Weight

9
Utility function
Weight
U wrR x wvV x weE

• U Utility
• R Reliability
• V Validity
• E Educational impact
• A Acceptability
• C Cost
• w Weight

10
Utility function
Weight
U wrR x wvV x weE

• U Utility
• R Reliability
• V Validity
• E Educational impact
• A Acceptability
• C Cost

11
Setting the pass mark characteristics of
credible standards

• Are they using a recognised method?
• Evidence....
• How are the standards applied?
• The method has to be
• defensible
• credible
• supported by a body of evidence in the literature
• feasible
• acceptable to all stakeholders
• Norcini, J. J. (2003). Setting standards on
  educational tests. Medical Education, 37,
  464-469.

12
Principles of Assessment

• There is no perfect assessment compromise is
  always required
• The compromise depends on the context of the
  assessment
• The Quality of assessments is a matter of the
  integral assessment programme, rather than of
  the individual instruments