Assessment of conscientiousness and its relation to professionalism

1
Assessment of conscientiousness and its
relation to professionalism

• John C. McLachlan
• School of Medicine and Health
• Durham University
• j.c.mclachlan_at_durham.ac.uk

2
Jane Macnaughton
Marina Sawdon
Gabrielle Finn
3
Metrics
The Conscientiousness Index
Validity

Practicality
Reliability
4
Durham Undergraduate Medicine Programme

• 2 year Phase 1 programme
• 102 students
• Case based with early exposure to patients
• Subsequently join with Newcastle students and go
  to Base Units across the North East

5
Assessment in the Professional Behaviour Domain

• Objective testing of cognitive knowledge through
  MCQs and EMIs
• Subjective testing of knowledge and awareness in
  SAQs and Data Interpretation questions
• Analysis of reflection in written assignments and
  reports
• Formative assessment of portfolio
• No assessment of behaviours

6
Approaches currently being explored in the Durham
programme

• MultiSource Feedback
• Significant Event Recording
• Conscientiousness Index (CI)
• (In press in Academic Medicine)

7
There is evidence that concerns about
undergraduate programme performance are a risk
factor for subsequent disciplinary action

• 1. Papadakis MA, Hodgson CS, Teherani A, Kohatsu
  ND. Unprofessional behavior in medical school is
  associated with subsequent disciplinary action by
  a state medical board. Acad Med. 2004 79244-9.
• 2. Papadakis MA, Teherani A, Banach MA, Knettler
  TR, Rattner SL, Stern DT, Veloski JJ, Hodgson CS.
  Disciplinary action by medical boards and prior
  behavior in medical school. N Engl J Med. 2005
  3532673-82.
• 3. Teherani A, Hodgson CS, Banach M, Papadakis
  MA. Domains of unprofessional behavior during
  medical school associated with future
  disciplinary action by a state medical board.
  Acad Med. 2005 80(10 Suppl)S17-20.

8
Conscientiousness Index1 point awarded if
student has

• Respond on immune status after 2nd request
• Provided information on criminal record status
  after 2nd request
• Provided photograph after 2nd request
• Completed each Edpac examination sheet correctly
  after induction session and formative practice

9
Conscientiousness Index1 point awarded if
student has

• Attended each compulsory class or provided
  satisfactory explanation
• Submitted each assignment on time or provided
  satisfactory explanation
• Completed each module evaluation form when
  requested, after induction session
• Completed other required activities (e.g. Base
  Unit Allocation form) when requested

10
CI points can be deducted

• If an individual student fails to carry out an
  action unique to them which they could reasonably
  be expected to do
• e.g. Failure to respond to staff e-mails
  requiring a response after the third e-mail has
  been opened and read.
• e.g. Failure to keep an appointment with a staff
  member after it has been explicitly agreed
• e.g. Failure to return library material after
  repeated requests
• (Serious incidents are the subject of a
  Significant Event Procedure or Fitness to
  Practice Procedure rather than deduction of a CI
  point)

11
Additional CI points can be awarded

• If a student displays unusual professionalism in
  difficult circumstances
• e.g. responding well to a medical emergency
• e.g. volunteering to assist others when unaware
  of implications for CI points

12
Metrics
The Conscientiousness Index
Validity

Practicality
Reliability
13
Typical Distribution
14
CI parameters
15
Metrics
The Conscientiousness Index
Validity

Practicality
Reliability
16
Validity 1 staff estimates

• Take ten top, middle and bottom scores from each
  year
• Give experienced staff a randomised list and ask
  to rate professionalism
• Ratings are I am happy with this students
  approach to professionalism, I have concerns
  about this students approach to professionalism
  and Dont know this student well enough to
  comment

17
(No Transcript)
18
Validity 1

• More unknowns in Year 1 than Year 2
• (77 versus 61)
• More unknowns in middle group
• (43 vs 56 vs 39)
• Combining both year groups, 67 out of 79 Concerns
  in lowest group, 9 in middle and 3 in top
• Of 14 Critical Incident forms, 10 in lowest 10
• Correlates moderately with exam performance but
  not with current Professional Behaviour measure

19
Validity 2 student estimates

• Invite students to name most professional and
  least professional peers
• Correlate most professional and least
  professional scores with CI
• No correlation between most professional
  rankings and CI
• Highly significant correlations between least
  professional scores and CI
• Spearmans Rho -.257 (p 0.002) for Year 1 and
• Spearmans Rho -.314 (p 0.006) for Year 2)

20
Metrics
The Conscientiousness Index
Validity

Practicality
Reliability
21
Reliability 1

• When the academic year is split into two, and
  performance on the first half year compared to
  the performance on the second half year, the
  Spearman Rank Correlation Coefficient is 0.60 for
  Year 2 and 0.59 for Year 1

22
Reliability 2

• How stable are findings from year to year?

23
Is there a correlation between year 1 and year 2
CI scores?

• R 0.5
• P0.01

24
How do students change from year 1 to year 2?
25
Is there a difference between males and females
in CI scores?
26
Is there a difference between school leavers and
graduate entrants in CI scores?
27
Metrics
The Conscientiousness Index
Validity

Practicality
Reliability
28
Practicality

• Objective
• Can largely be done by admin staff
• Does not require a decision to report
• Runs continuously all year
• Requires registers to be taken
• Requires centralisation of information previously
  gathered but not collated

29
Summary

• Low CI is valid by 2 independent measures as a
  measure of lack of professionalism
• It is reliable within and between years as a
  strong educational effect
• It is practical (low cost and easy to
  administer)
• The property it measures is objective and scalar

30
Potential uses

• Summative
• Used to determine progression. Can identify
  outliers and major changes. Can be used to
  explore failure to remediate
• Formative
• Used for feedback to student
• Informative
• Used to inform the organisation of potential
  issues which may lie far in the future