Life beyond the OSCE: Assessment in the workplace

1
Life beyond the OSCEAssessment in the workplace

• Teacher Development Day
• Thursday 9 March 2006
• Manchester
• Cees van der Vleuten
• University of Maastricht

2
This presentation can be found on www.fdg.unimaa
s.nl/educ/cees/wba
3
Overview of presentation

• Where are we with assessment?
• What do we know?
• Where should we be going?

4
Climbing the pyramid
Does
Shows how
Knows how
Knows
Miller GE. The assessment of clinical
skills/competence/performance. Academic Medicine
(Supplement) 1990 65 S63-S7.
5
Climbing the pyramid
Does
Shows how
Knows how
Knows
Miller GE. The assessment of clinical
skills/competence/performance. Academic Medicine
(Supplement) 1990 65 S63-S7.
6
Climbing the pyramid
Paper/written/computer based technologies using
(simple) simulations that require application
of knowledge or judgment (key feature, MCQ, EMQ.)
Does
Shows how
Knows how
?
Knows
Miller GE. The assessment of clinical
skills/competence/performance. Academic Medicine
(Supplement) 1990 65 S63-S7.
7
Climbing the pyramid
?
Does
OSCE-ology
Shows how
Knows how
?
Knows
Miller GE. The assessment of clinical
skills/competence/performance. Academic Medicine
(Supplement) 1990 65 S63-S7.
8
From subjective to objective

• Clinical supervisor rating (does level)
• Subjective
• Unstructured
• 2nd 3rd hand information
• Holistic
• Uninformative

?

• OSCE (shows how level)
• Objective
• Structured
• Directly observed
• Analytical scoring

?
9
Reliability of OSCEs1
1Studies taken from Petrusa, E. R. (2002).
Clinical performance assessments. In G. R.
Norman, C. P. M. Van der Vleuten D. I. Newble
(Eds.), International Handbook for Research in
Medical Education (pp. 673-709). Dordrecht
Kluwer Academic Publisher (Table 1).
10
Reliability as a function of testing time
Case- Based Short Essay2 0.68 0.73 0.84 0.82
Practice Video Assess- ment7 0.62 0.76 0.93 0.93
Mini CEX6 0.73 0.84 0.92 0.96
In- cognito SPs8 0.61 0.76 0.82 0.86
Testing Time in Hours 1 2 4 8
MCQ1 0.62 0.76 0.93 0.93
PMP1 0.36 0.53 0.69 0.82
Oral Exam3 0.50 0.69 0.82 0.90
Long Case4 0.60 0.75 0.86 0.90
OSCE5 0.47 0.64 0.78 0.88
1Norcini et al., 1985 2Stalenhoef-Halling et al.,
1990 3Swanson, 1987
4Wass et al., 2001 5Petrusa, 2002 6Norcini et
al., 1999
7Ram et al., 1999 8Gorter, 2002
11
From subjective to objective

• Clinical supervisor rating (does level)
• Subjective
• Unstructured
• 2nd 3rd hand information
• Holistic
• Uninformative

?

• OSCE (shows how level)
• Objective
• Structured
• Directly observed
• Analytical scoring

?
12
Checklist/rating reliability
Van Luijk van der Vleuten, 1990
13
Analytical versus holistic

• Global rating scales in OSCEs
• Correlate (very) highly
• Show slightly lower inter-rater reliabilities
• Show higher inter-case reliabilities
• (Rothman et al., 1997 Cunnington et al., 1997
  Regehr et al., 1998, 1999 Hodges et al, 1999
  Govaerts et al., 2002 Hodges McIlroy, 2003)

14
From subjective to objective

• Clinical supervisor rating (does level)
• Subjective
• Unstructured
• 2nd 3rd hand information
• Holistic
• Uninformative

?

• OSCE (shows how level)
• Objective
• Structured
• Directly observed
• Analytical scoring

?
15
Varieties of OSCEs
Patient-based
Written task
Clinical task
16
Assessment drives learning

• Isolated skills lead to fragmentation
• Checklists lead memorization
• Leading to Chickens without a head (Van Luik et
  al., 1990)

17
From subjective to objective

• Clinical supervisor rating (does level)
• Subjective
• Unstructured
• 2nd 3rd hand information
• Holistic
• Uninformative

?

• OSCE (shows how level)
• Objective
• Structured
• Directly observed
• Analytical scoring

?
18
Assessment drives learning
Curriculum
Assessment
Teacher
Student
19
Assessment drives learning

• Assessment may drive learning through
• Content
• Format
• Programming/scheduling
• Regulations
• ..........

20
Assessment drives learning

• Summative assessments are poor feedback carriers
• Pure formative assessments are not very long
  lasting
• Separation of learning and assessment is asking
  for trouble
• Strategic use of assessment is on the agenda
• In-training assessment
• Embedded assessment (learning task assessment
  task)
• Combining summative and formative functions
• Reinforcing desirable learning and performance
  outcomes

21
Climbing the pyramid
Work-based assessment
Does
?
Shows how
Knows how
Knows
Miller GE. The assessment of clinical
skills/competence/performance. Academic Medicine
(Supplement) 1990 65 S63-S7.
22
Lessons learned

• Wide sampling across content (and assessors) is
  always required (much more an issue than
  objectivity and standardization)
• Holistic judgements may do just as well or better
  (provided performance is concretely observed)
• Strategic in-training assessment is the way
  forward

23
Work-based assessment

• The venue has moved from the relatively
  controlled and homogeneous settings in education
  to the uncontrolled and heterogeneous world of
  work
• (John Norcini, 2005)

24
Wise strategies for WBA

• Having a framework of competencies structuring
  the work-based learning environment

25
Competency categorizations

• CanMeds
• roles
• Medical expert
• Communicator
• Collaborator
• Manager
• Health advocate
• Scholar
• Professional

• ACGME
• competencies
• Patient care
• Medical knowledge
• Practice-based learning improvement
• Interpersonal and communication skills
• Professionalism
• Systems-based practice

• Dundee
• outcomes
• Clinical skills
• Practical procedures
• Patient investigation
• Patient management
• Health promotion and disease prevention
• Communication
• Information management skills
• Principles of social, basic clinical sciences
• Attitudes, ethics legal responsibilities
• Decision making, clinical reasoning, judgement
• Role as a professional
• Personal development

26
But you need a lot more
General competencies
Curriculum structure
Sub- competencies
Courses
Professionally meaningful tasks
Professional activities
Performance level standards
Assessment sources
Assessment decisions
27
Wise strategies for WBA

• Having a framework of competencies structuring
  the work-based learning environment

• Having a toolbox of assessment instruments or
  information resources

28
WBA methods

• Indirect methods
• Chart audit
• Practice audits
• Prescription data
• Direct methods
• Patient reports
• Peer reports
• Clinical work samples
• Multisource feedback (3600)
• Mini-CEX
• Clinical (supervisor) ratings
• Undercover standardized patients
• Video assessment

29
Data-gathering change

• Conditions are never standard, never similar,
  never homogeneous (patient mix)
• Information gathering will therefore rely more on
  descriptive and qualitative judgemental
  information requiring professional and expert
  judgement
• To interprete/judge observed clinical performance
• To interpret aggregated assessment information
  across sources of (assessment) information

30
Quantitative or qualitative

• Quantitative
• Scores
• Grades
• Statistical computations
• Cut-off scores
• Algorithmic

• Qualitative
• Narratives
• Professional judgements
• Combination of information
• Performance standards
• Judgemental

31
Wise strategies for WBA

• Having a framework of competencies structuring
  the work-based learning environment

• Having a toolbox of assessment instruments or
  information resources

• Procedures and evidence that makes promotion
  decisions credible or trustworthy

32
Criteria in Research Approach(Crefting, 1991)
Strategies for establishing trustworthiness
Prolonged engagement Triangulation Peer
examination Member checking Structural
coherence Time sampling Stepwise
replication Dependability audit Thick
description Confirmability audit
Procedural measures and safeguards Assessor
training benchmarking Appeal procedures
Triangulation across sources, saturation
Assessor panels Intermediate feedback cycles
Decision justification .
Quantitative Qualitative Criterion approach a
pproach Truth value Internal validity Credibilit
y Applicability External validity Transferability
Consistency Reliability Dependability Neutrali
ty Objectivity Confirmability
33
Threats to WBA

• Implementation (involving, convincing, committing
  and training the stakeholders)
• Lots of threats to validity of WBA due to the
  heterogenous and uncontrolled character of WBA
  (but lots of opportunity for research!)

34
It will be a long journey..
35
This presentation can be found on www.fdg.unimaa
s.nl/educ/cees/wba