Work based assessment [e-mail]

1
Work based assessmenth.davies_at_shef.ac.uk

• Challenges and opportunities

2
Choosing assessments?

• What do you want to assess - e.g.
• Professionalism
• Clinical reasoning
• Technical skills
• Clinical evaluation
• Who do you want to assess
• Med student
• Resident
• Practising physician

3
Choosing assessments

• Patient outcome gold standard but attribution a
  problem
• Judgements largely based on process measures
• Case mix and reliability an issue

4
Assessors

• If sufficient subjective judgements are combined
  the collated judgement about performance can be
  reliable
• i.e objectivity and reliability are not the same
  thing
• Assumes that assessor has both observed
  competence in question and can make a judgement
  about its quality
• How many assessors is enough?

5
MSF in healthcare settings

• Feedback about observable behaviors is provided
  by some or all of
• Physician colleagues (peers, referring MDs,
  referral MDs)
• Co-workers (e.g., nurses, pharmacists,
  dieticians)
• Patients
• Self
• If trainee borderline or in difficulty sample
  more

6
Insight
7
MSF

• One to one feedback
• Problems may need further diagnostic work
• Need support mechanisms in place

8
Bias

• Consider sources of bias such as
• Gender
• Working relationship
• Working environment
• Ethnic group
• University of graduation

9
F1/F2
SHO
SpR
AHP
GP
Nurse
Cons
SASG
F524.1 p lt0.001
10
Mini-CEX case complexity
11
Validity

• Extent to which a test assesses what it purports
  to
• Content validity
• Criterion validity
• Construct validity
• Face validity
• Consequential validity
• Recognised need for further validity data in
  relation to MSF and other work based assessments1
• Evans, R., G. Elwyn, and A. Edwards, Review of
  instruments for peer assessment of physicians.
  BMJ, 2004. 328(7450) p. 1240.

12
Validity

• Content validity
• Blueprint assessments to the curriculum
• Evidence from a range of sources supporting the
  hypothesis that the score really measures what it
  is meant to

13
Relationship with patient ratings
SHEFFPAT vs SPRAT r0.12 NS SHEFFPAT vs
PATSCORE r0.45 plt0.01
14
Predictive and consequential validity
15
Centralisation

• Facilitates standardisation and robust QA
• Facilitates movement between locations
• Economies of scale
• Strategies to enhance local ownership important

16
Practicalities

• Is it feasible?
• Electronic vs paper
• Centralised vs localised
• Who assesses?
• How is training organised?
• Who will do the QA?

17
Acknowledge legitimate concerns

• Healthy scepticism
• Resources - especially time
• Conflicting demands espy CLINICAL
• Need for sampling
• Lack of standardisation
• Training needs
• Loss of local ownership

18
Support doctors in difficulty
19
Be patient

• Op-ti-mist n
• Somebody who tends to feel hopeful and positive
  about future outcomes
• Recognise extent of cultural change
• Ensure work based assessment is done as well as
  possible

Every problem is just an opportunity waiting to
be made use of
20
Who assesses?
14.8
19.4
7.1
57.6
21
The programme should be quality assured
importance
100
0
R
V
E
C
A
aspect
F
van der Vleuten C. The assessment of professional
competence developments, research and practical
implications. Advances in Health Sciences
Education. 1996141-67.
22
Sampling

• Content specificity
• Being good at one thing doesnt mean you are good
  at everything
• Must sample clinical content widely
• Map to curriculum - blueprint
• Sources of variance
• Assessors significant source of variance
• Use lots of assessors

23
Classification Scheme for Work-Based Assessment
Norcini BMJ 2003
24
Quality assurance
Quality assure assessment system
Modify assessment system in response to QA
25
Effective (work based) assessment

• RESOURCES and
• CO-ORDINATION
• TIME
• FUNDING

26
(No Transcript)
27
Key messages

• Sampling is crucial
• Consider sources of bias
• Have as many different clinical problems and as
  many assessors as possible
• Subjectivity ? unreliability
• Train the raters

Holmboe, E. S., R. E. Hawkins, et al. (2004).
"Effecs of training in direct observation of
medical residents' clinical competence a
randomized trial." Ann Intern Med 140 874-881.
28

• Attributes of physician
• Personality
• Background
• Aptitude for specialty

• Attributes of training
• Post
• Programme
• Supervisor

• Context of training
• Workload
• Colleagues
• Patients

MSF
PERFORMANCE OF PHYSICIAN

• Personal pressures
• Home
• Health