Title: Assessing clinical reasoning of poorly performing physicians: Does the Script Concordance Test SCT m
1Assessing clinical reasoning of poorly performing
physiciansDoes the Script Concordance Test
(SCT) make a difference?
- Dr Marguerite Dupré, MD
- Robert Gagnon, MSc
- François Goulet, MD, MA
- André Jacques, MD
2Potential Conflict of Interests Disclosure
Dr André Jacques declares
I do not have any affiliation or financial
interests of any nature in a business
corporation, or I do not receive remuneration,
royalties or research grants from a business
corporation
3System of monitoring and enhancement of
physicians performance
Monitoring
Enhancement
Level 3
Personalized educational prescription
In-depth performance appraisal
Enhancement
Monitoring
Evaluation of physicians at risk or in difficulty
(professional inspection)
Level 2
Recommendations of the Professional Inspection
Committee
Monitoring
Enhancement
Screening of all physicians (clinical or
administrative indicators)
Feedback given to all physicians
Level 1
4Practice Enhancement Division (PED) at the CMQ,
assessed practising family physicians
- Assessment tool SOI-OSCE
- SOI-OSCE is valid and reliable
- SOI-OSCE does not provide objective and
quantitative data about clinical reasoning
5Objectives of the study
- 1) To quantify and gather data on clinical
reasoning of practising family physicians who
have to be assessed by the - SOI-OSCE.
- 2) To evaluate the contribution of the Script
Concordance Test (SCT) to the SOI-OSCE.
6Population studied
- 25 practising family physicians referred to the
PED for an assessment (Level 3, MEPP).
7The SOI-OSCE
- Evaluation ? Exam
- Evaluation done by 2 peers
- Length 1 day
- Expectations Practice competency judged in the
main stream - No time duration required to complete each
vignette
8The SOI-OSCE
- Evaluation Domains
- A. Interview Vignettes
- Data gathering
- Data interpretation
- Problem-solving
- Clinical reasoning (subjective) and qualitative
data - B. Simulated Patients
- Questionnaire
- Physical Examination
- Patient-Doctor Communication
-
9The SOI-OSCE
- Psychometric Data
- Face validity candidates recognized and
validated the representativity of the clinical
vignettes in terms of their practice - Internal consistency (36 cases) ? Cronbach 0.91
- Inter-rater reliability (7 raters) ? 91.2
- Mean score obtained by candidates ? 50.5
10The SOI-OSCE
- Domain assessed (20 vignettes)
- Questionnaire (12)
- Physical examinations (13)
- Diagnosis (12)
- Investigation plan ( 9)
- Drug prescriptions (13)
- Treatment plan and counselling ( 8)
- Follow-up ( 6)
11The SOI-OSCE Simulated Patients
- Based on simulated standardized patients
- The candidate must question a patient or perform
a physical examination - Checklist completed by an evaluator
- Items on checklist are validated by peers
- Scoring
- Score for each item depending on its importance
- Total score for each station
12Script Concordance Test (SCT)
- Construction
- Each vignette describes a clinical situation (3
or 4 sentences) - The question format differs depending on the
objective of the assessment - Diagnosis
- Investigation Plan
- Treatment Plan
13Script Concordance Test (SCT)
- Clinical reasoning assessment and the capacity to
make micro-decisions - Paper-Pen tool
- Measures the level of concordance between the
candidates decision and the modal answer of a
panel of peers - Probes the organization of clinical reasoning
14Script Concordance Test (SCT)
For diagnostic assessment
15Script Concordance Test (SCT)
16SCT
- 64 QUESTIONS (Alpha 0,90)
- Evaluation domain
- Diagnosis (questionnaire)
- Diagnosis (physical exam)
- Investigation plan
- Treatment plan
- Panel 13 family physicians randomly selected
- Table of specification
17The SOI-OSCE/SCT METHOD
- COMPARISON OF BOTH METHODS
- Classification by 3 raters of quality of clinical
reasoning in the SOI report - Reliability of independent rating K 0,75
- Reliability of combined scores K 0,89
- Classification of candidates of SOI and SCT
- Below median score
- Over median score
18RESULTSSOI-OSCEDistribution of mean rating of
the three raters
19RESULTSSCT
20RESULTSClinical ReasoningComparison of
SOI-OSCE VS SCT(Subjective and qualitative
data)vs Objective and quantitative data n 20
21The SOI-OSCE/(SCT)Conclusion
- Adding the SCT in the SOI-OSCE in assessing
clinical reasoning of poorly performing
physicians provides - Some common information with both measures
- Discordance in 40 of cases ( 8/20 )
- The SCT revealed a problem not measured by
SOI-OSCE assessors
22DIRECTIONS
- Dr. André Jacques
- 2170 René-Lévesque West bvd
- Montréal, Québec H3H 2T8
- Canada
- Tel. 514 933-4441 ext 5322
- Toll free 1-888-633-3246 ext. 5322
- Fax 514 933-4668
- jacques.cmq_at_sympatico.ca
- Assistant mcharette_at_cmq.org
23QUESTIONS ?