Assessing clinical reasoning of poorly performing physicians: Does the Script Concordance Test SCT m

1
Assessing clinical reasoning of poorly performing
physiciansDoes the Script Concordance Test
(SCT) make a difference?

• Dr Marguerite Dupré, MD
• Robert Gagnon, MSc
• François Goulet, MD, MA
• André Jacques, MD

2
Potential Conflict of Interests Disclosure
Dr André Jacques declares
I do not have any affiliation or financial
interests of any nature in a business
corporation, or I do not receive remuneration,
royalties or research grants from a business
corporation
3
System of monitoring and enhancement of
physicians performance
Monitoring
Enhancement
Level 3
Personalized educational prescription
In-depth performance appraisal
Enhancement
Monitoring
Evaluation of physicians at risk or in difficulty
(professional inspection)
Level 2
Recommendations of the Professional Inspection
Committee
Monitoring
Enhancement
Screening of all physicians (clinical or
administrative indicators)
Feedback given to all physicians
Level 1
4
Practice Enhancement Division (PED) at the CMQ,
assessed practising family physicians

• Assessment tool SOI-OSCE
• SOI-OSCE is valid and reliable
• SOI-OSCE does not provide objective and
  quantitative data about clinical reasoning

5
Objectives of the study

• 1) To quantify and gather data on clinical
  reasoning of practising family physicians who
  have to be assessed by the
• SOI-OSCE.
• 2) To evaluate the contribution of the Script
  Concordance Test (SCT) to the SOI-OSCE.

6
Population studied

• 25 practising family physicians referred to the
  PED for an assessment (Level 3, MEPP).

7
The SOI-OSCE

• Evaluation ? Exam
• Evaluation done by 2 peers
• Length 1 day
• Expectations Practice competency judged in the
  main stream
• No time duration required to complete each
  vignette

8
The SOI-OSCE

• Evaluation Domains
• A. Interview Vignettes
• Data gathering
• Data interpretation
• Problem-solving
• Clinical reasoning (subjective) and qualitative
  data
• B. Simulated Patients
• Questionnaire
• Physical Examination
• Patient-Doctor Communication

9
The SOI-OSCE

• Psychometric Data
• Face validity candidates recognized and
  validated the representativity of the clinical
  vignettes in terms of their practice
• Internal consistency (36 cases) ? Cronbach 0.91
• Inter-rater reliability (7 raters) ? 91.2
• Mean score obtained by candidates ? 50.5

10
The SOI-OSCE

• Domain assessed (20 vignettes)
• Questionnaire (12)
• Physical examinations (13)
• Diagnosis (12)
• Investigation plan ( 9)
• Drug prescriptions (13)
• Treatment plan and counselling ( 8)
• Follow-up ( 6)

11
The SOI-OSCE Simulated Patients

• Based on simulated standardized patients
• The candidate must question a patient or perform
  a physical examination
• Checklist completed by an evaluator
• Items on checklist are validated by peers
• Scoring
• Score for each item depending on its importance
• Total score for each station

12
Script Concordance Test (SCT)

• Construction
• Each vignette describes a clinical situation (3
  or 4 sentences)
• The question format differs depending on the
  objective of the assessment
• Diagnosis
• Investigation Plan
• Treatment Plan

13
Script Concordance Test (SCT)

• Clinical reasoning assessment and the capacity to
  make micro-decisions
• Paper-Pen tool
• Measures the level of concordance between the
  candidates decision and the modal answer of a
  panel of peers
• Probes the organization of clinical reasoning

14
Script Concordance Test (SCT)
For diagnostic assessment
15
Script Concordance Test (SCT)
16
SCT

• 64 QUESTIONS (Alpha 0,90)
• Evaluation domain
• Diagnosis (questionnaire)
• Diagnosis (physical exam)
• Investigation plan
• Treatment plan
• Panel 13 family physicians randomly selected
• Table of specification

17
The SOI-OSCE/SCT METHOD

• COMPARISON OF BOTH METHODS
• Classification by 3 raters of quality of clinical
  reasoning in the SOI report
• Reliability of independent rating K 0,75
• Reliability of combined scores K 0,89
• Classification of candidates of SOI and SCT
• Below median score
• Over median score

18
RESULTSSOI-OSCEDistribution of mean rating of
the three raters
19
RESULTSSCT
20
RESULTSClinical ReasoningComparison of
SOI-OSCE VS SCT(Subjective and qualitative
data)vs Objective and quantitative data n 20
21
The SOI-OSCE/(SCT)Conclusion

• Adding the SCT in the SOI-OSCE in assessing
  clinical reasoning of poorly performing
  physicians provides
• Some common information with both measures
• Discordance in 40 of cases ( 8/20 )
• The SCT revealed a problem not measured by
  SOI-OSCE assessors

22
DIRECTIONS

• Dr. André Jacques
• 2170 René-Lévesque West bvd
• Montréal, Québec H3H 2T8
• Canada
• Tel. [phone] ext 5322
• Toll free 1-888-633-3246 ext. 5322
• Fax [phone]
• jacques.cmq_at_sympatico.ca
• Assistant mcharette_at_cmq.org

23
QUESTIONS ?