The Summative Assessment of Clinical Performance in the Workplace

1
The Summative Assessment of Clinical Performance
in the Workplace

• Centre for Medical and Health Sciences Education
• Monash University

Professor Gordon Page February 22 2007
2
(No Transcript)
3
(No Transcript)
4
Outline of Presentation

• A brief history of the summative assessment of
  clinical performance
• Recent Canadian and Australian studies
• Conclusions

5
Outline of Presentation

• A brief history of the summative assessment of
  clinical performance
• Recent Canadian and Australian studies
• Conclusions

6
Performance Assessment Strategies The Menu

• Long and short case clinical orals (some observe
  trainee with a patient)
• In-training (end of term) assessments
• OSCEs (standardized patients, skills stations, )
• Mini-CEX, DOPS, CBD, 360 degree assessments,

7
A 30-Year Story of Performance Assessment

• Up to 1970s Long and short case clinical orals
  disenchantment with subjectivity of examiners
  ratings notoriously unreliable scores
• 1980s Development of the OSCE the same set of
  simulated cases for all examinees, objective
  checklists for scoring better reliability of
  scores

8
A 30-Year Story of Performance Assessment

• 1990s Increased reliability shown to be a
  function of better sampling across cases and not
  of the objective checklists.
• 1990s Global ratings (professional judgments) on
  OSCEs shown to provide more reliable scores than
  checklist scores

9
A 30-Year Story of Performance Assessment

• Late 1990s Renewed interest in assessment with
  real patients, with a focus on better sampling
  focus on Mini-CEX, DOPS, CBD, and 360 degree
• 2000 Global ratings on less structured/less
  standardized workplace assessments shown to
  possess reliability comparable to or better than
  OSCEs, for similar testing times.

10
Measurement Qualities of the Mini-CEX

• Norcini (2003) 95 CI of .6 on a 9-point scale
  with 4 observations
• During (2002) generalizability coef. of .90
  with 7 observations
• Kogan (2003) generalizability coef. of .77 with
  8 observations
• ECFMG (2002) strong correlations between IMGs
  OSCE and mini-CEX scores
• Several studies -- Inter-observer reliability
  high, construct validity high

11
A 30-Year Story of Performance Assessment

• Conclusions
• Sampling across cases, not objectivity provided
  by checklists, is the main contributor to better
  reliability
• Global (professional) judgements, in contrast to
  checklist scores, seem to be more reliable.
• Reliable, formal assessment in real clinical
  situations is achievable (and desirable)

12
and desirable
Assessment in real clinical situations is
achievable
13
The NSW False Positives Experience with IMGs

• A 2004-5 study 35 of IMGs were underperforming
  in their PGY1 placements (vs 2 for Australian
  graduates)
• Clinical judgment (54)
• Communication skills (30)
• Professionalism (12)

14
Competence versus Performance
Correlations between Mini-CEX and MCQ scores
.335 Mini-CEX and OSCE scores .372
15
Outline of Presentation

• A brief history of the summative assessment of
  clinical performance
• Recent Canadian and Australian studies
• Conclusions

16
Recent Studies

• Study 1
• Canadian Mini-CEX study with general practice
  doctors
• Study 2
• Canadian Mini-CEX Study with IMGs
• Study 3
• AMC Mini-CEX Studies with IMGs in NSW, QSLD,
  Vic

17
What is a Mini-CEX?

• Mini-CEX is a Mini Clinical Evaluation
  Exercise
• A direct observation of a doctors core clinical
  skills in a patient encounter
• Based on a real patient in real clinical
  situation more of an assessment of performance
  (as distinct from competence)

18
ic Rounds                                       
                         
19
(No Transcript)
20
Mini-CEX Studies -- Purpose

• To evaluate the utility (U) of the Mini-CEX,
  where
• U R x V x E x A x C
• R reliability
• V validity
• E educational impact
• A acceptability
• C cost

21
Study 1 The Utility of the Mini-CEX in
Assessing the Performance of General and Family
Practice Doctors

• Gordon Page
• Faculty of Medicine
• University of British Columbia
• November 2006

22
Purpose/Research Questions

• To evaluate the utility of the mini-CEX as a tool
  for assessing family and general practice doctors
  in an office setting
• How may encounters must be observed to provide a
  reliable mean rating of overall clinical
  competence?
• What level of agreement exists between
  independent assessors?
• What is the face validity, educational impact and
  acceptability of the mini-CEX in this setting, as
  perceived by those assessed and those assessing?
• What is the cost of the mini-CEX in this context?

23
Study Design

• 15 volunteer subjects (family and general
  practice doctors)
• 6 trained assessors (family and general practice
  doctors) -- 4-hour training workshop
• 10 office encounters per doctor observed over a
  half day, with feedback provided
• 12 doctors observed by one assessor 3 doctors by
  a pair of assessors
• Assessors and doctors complete a form eliciting
  their evaluations of mini-CEX face validity,
  acceptability, and educational impact
• Ethics Board approvals doctors and patients
  sign consent letters

24
How many encounters must be observed to provide a
reliable mean rating of overall clinical
competence?
25
Level of agreement between independent assessors?

• Intraclass correlations of ratings across
  independent assessors exceeded .90.
• SD of ratings across independent assessors were
  in the range .50 to 1.00 on a 9-point scale

26
Face validity, educational impact and
acceptability

• What is the
• face validity,
• educational impact and
• acceptability of the mini-CEX in this setting,
  as perceived by those assessed and those
  assessing?

27
Face (Content) Validity
28
Face Validity
29
Face Validity/Acceptability
30
Acceptability
31
Acceptability
32
Educational Impact
33
What is the cost of the mini-CEX in this context?

• For assessing 15 practitioners
• Assessor time 400 per half-day x 24 9,600
• Staff-- scheduling, data retrieval, preparation
  and analysis, estimated at 5,000
• Leadership workshop presentation, planning,
  estimated at 5,000
• Assuming one assessor per assessment, cost per
  doctor assessed appears to approximate 1,500

34
Conclusions
In the context of assessing family and general
practice doctors in an office setting, the
Mini-CEX has demonstrated

• An acceptable level of reliability with as few as
  6 encounters
• Acceptance by both assessors and those assessed.
  Both groups support its face validity and
  educational efficacy
• Costs that are modest relative to other commonly
  used tools (OSCEs, orals, )

35
Study 2 A Study of the Structured Assessment
of International Medical Graduates (IMGs) in
Supervised Practice Settings

• Dr. Gordon Page
• Professor
• Dr. George Pachev
• Research and Assessment Director
• Division of Educational Support and Development
• Faculty of Medicine, University of British
  Columbia

36
IMG Study -- Data

• 633 Mini-CEX assessments from 106 assessors over
  24 IMGs
• Forms evaluating the Mini-CEXs face validity,
  educational impact and acceptability completed by
  12 (of 24) IMGs and 18 (of 106) supervising
  physicians

37
Reliability of the Mini-CEX Average Composite
Ratings
38
Reliability of ITER Average Composite Ratings
39
Study Results -- Reliability

• Reliability Coefficients
• gt .80 with 6 observations
• gt .90 with 12 observation
• gt .95 with 24 observations
• Enhanced further by sampling performance in
  different clinical contexts

40
Study Results -- Validity

• Strong evidence of
• Content (sampling) Validity
• Face validity
• Weak evidence of construct validity
• Correlations between
• Mini-CEX and MCQ scores .335
• Mini-CEX and OSCE scores .372
• Mini-CEX and ITER scores .724

41
Study Results Educational Impact and
Acceptability

• IMGs and supervising physicians ratings of the
  Mini-CEX
• Highly effective for stimulating learning and
  providing feedback
• Strongly endorse the continued use of the
  Mini-CEX in future practice-based assessment
  programs

42
Study Conclusions and Recommendations

• With modest effort, composite scores from
  Mini-CEX assessments can contribute reliable and
  valid data for summative judgments of an IMGs
  clinical competence (performance?)
• The Mini-CEX should be an integral component of
  the assessment toolbox for IMG assessment

43
Study Conclusions and Recommendations

• In the context of making decisions to grant a
  license, deliberate sampling of encounters over
  the full spectrum (problems, settings) of
  clinical practice is important

44
Study 3 An AMC Study of the Mini-CEX Assessment
of IMGs

• Heather Alexander, Queensland
• Kichu Nair, NSW
• Barry McGrath, Victoria
• (G. Page and G. Pachev)

45
Background to the Study

• Many IMGs enter practice in Australia without AMC
  (or any other) assessment.
• Australia has classified some countries as
  Competent Authorities IMGs to be fast
  tracked, and assessed in practice settings.
• The challenge how to arrive at a defensible
  assessment in the workplace?

46
Study Data
47
Study Results -- Reliability
48
Other Results

• Inter-rater reliability was good data not yet
  available on effects of assessor training.
• Face validity, Acceptability, Educational Impact
  Data highly supportive and comparable to
  Canadian data
• Mini-CEX was judged superior to other assessment
  strategies ITERs, written examinations, oral
  examinations, OSCEs

49
Comparison to OSCE

• IMG Gives a more natural and relaxed environment
  where you can live up to your true abilities.
• IMG You deal with real patients which is much
  closer to the clinical reality. The feedback is
  really excellent and individualised
• IMG miniCEX provides much better feedback than
  OSCE exams and its assessment on a candidate's
  clinical skills is more reasonable
• Assessor I like OSCEs because of their
  objectivity and standardisation but the miniCEX
  adds an invaluable dimension of dealing with a
  real patient and I feel it tests the real
  clinical performance of the candidate more
  appropriately

50
Should it continue?

• IMG Yes as long as barriers with time management
  are improved and theoretical aspects are tested
  as well.
• IMG It is not exam orientated and will help a
  lot of IMG's to work in Australia

51
Should it continue?

• Assessor It is overdue to get away from most
  times, very unrealistic role players
• Assessor It allows assessment, feedback, an
  opportunity for learning and improving skills and
  performance.
• Assessor Not sure too many variables and too
  many personnel involvement may bring in wide
  variability and discrepancy in outcomes.

52
Should it be used summatively?

• Assessor I found the tool to be somewhat
  burdensome, but it is no different to other
  assessment tools
• Assessor Need more information on its utility as
  a test instrument.
• Assessor To assist not as a stand alone.

53
Comments- further concerns

• IMG MiniCEX will be a good assessment tool if it
  can maintain its quality in the long run
• IMG Organise all cases on single day so that it
  does not take unnecessary time from daily
  routine
• Assessor I found the whole process labour
  intensive but good training often is. Not sure
  who is best to administer this.
• Assessor The process must be time consuming for
  the admin staff.

54
Outline of Presentation

• A brief history of the summative assessment of
  clinical performance
• Recent Canadian and Australian studies
• Conclusions

55
Conclusions So Where does this lead?

• Workplace-based assessment
• Is desirable
• Is feasible
• Can provide a basis for defensible decisions on
  competence
• Ideally is an add-on, not a substitute for
  existing competency assessments
• Is an important addition to the competency
  assessment toolbox!

56
(No Transcript)