The Clinical Skills Assessment a case study of case writing

1
(No Transcript)
2
The Clinical Skills Assessment a case study of
case writing

• Kamila Hawthorne
• Amar Rughani
• Pauline Hernandez

3
Table of Contents

• Dust cover development of the CSA
• What the critics say
• Prologue how cases are developed
• Acts write your own case
• Illustrations discuss your cases
• Appendices an essay on case difficulty
• Going on tour how to disseminate information
  about the CSA and help candidates prepare
• About the authors Paulines plea

4
Coverage of the curriculum by the licensing
components
5
Development of the CSA

• nMRCGP curriculum
• Curriculum Blueprint
• Curriculum Statements
• CSA mapped to nMRCGP curriculum, able to test in
  some domains
• Some triangulation possible with AKT and WBA

6
What is the assessment score?

• MCP-knowledge test
• CSA clinical skills
• WA-Embedded periodic summative assessments e.g.
  Mini CEX, DOPS, MSF
• WA-Baseline continuo of portfolio assessment
• measuring performance over
  time
• Timeline of learning and assessment

7
Dust cover - Purpose of the CSA

• An assessment of a doctors ability to integrate
  and apply appropriate clinical, professional,
  communication and practical skills in general
  practice
• Integrative skills assessment - tests a doctors
  abilities to gather information and apply learned
  understanding of disease processes and
  person-centred care appropriately in a
  standardised context, making evidence-based
  decisions, and communicating effectively with
  patients and colleagues.

8
Why a Clinical Skills Assessment?

• Criticism of current MRCGP that there is no
  clinical consulting skills component
• Provides external validation / triangulation with
  the other testing methods used
• Using simulated patients is a validated and
  reliable method for testing clinical skills, so
  long as quality assurance of case production,
  role player and assessor training is carried out.
• Able to offer a standardised, pre-determined
  level of challenge to candidates and to vary this
  level of challenge as needed by the assessment
  requirements

9
Development of the CSA

• First planning meeting Summer 2005
• Creating and developing a case bank, recruiting
  and training assessors, recruiting and training
  roleplayers, finding and fitting a central venue,
  setting up electronic booking systems and case
  bank through Integra
• First diet October 2007
• Now run October 07, February, April, May 08
• Numbers of candidates

10
What the CSA can and cant do

• Can do
• Test to pre-set criteria and standards
• Test integrated ability to consult with a range
  of patients
• Realistic cases, representative of British GP
• Test across many parts of the Curriculum
• Spot check clinical examination skills

• Cant do
• Test comprehensively across the Curriculum
• Eg. Management issues
• Medical emergencies
• Test ability to pick up physical signs
• Test invasive or intimate examinations
• Still a simulated exam, doesnt test at apex of
  Millers triangle

11
What a just passing candidate looks like

• Based on generic grade descriptors
• Critically, the difference between a MP and MF
• Examiner uses clinical experience, experience of
  examining methodology and ability to make
  justifiable marking decisions, knowledge of
  expected standard of Drs at the end of their
  training in GP, to make their decision
• Applied to and helped by case specific marking
  schedule

12
The critical decision rests on

• Marginal Pass
• The candidate demonstrates an adequate level of
  competence, displaying a clinical approach that
  may not be fluent but is justifiable and
  technically proficient.
• The candidate shows sensitivity and tried to
  involve the patient

• Marginal Fail
• The candidate fails to demonstrate adequate
  competence, with a clinical approach that is at
  times unsystematic or inconsistent with accepted
  practice. Technical proficiency may be of
  concern.
• The patient is treated with sensitivity and
  respect, but the doctor does not sufficiently
  facilitate or respond to the patients
  contribution.

13
What the critics say

• Candidates
• Cases representative of British GP, role players
  very realistic
• Communication (93) and problem solving skills
  tested (83)
• Not like real surgeries as no time to catch up,
  or ask for help. Time issues in approx. half
  comments from February exams.
• In general, cases moderately difficult, 28
  thought some cases too difficult, 5 thought some
  cases too easy!
• So wrapped up in the case, dont notice the
  assessor most of the time
• Generally happy with organisation and paperwork,
  but 30 unhappy with dates/ times of CSA

14
What the critics say

• Assessors and role players
• More details in RP briefing, especially when to
  give cues
• Include the character of the patient for RP
  authenticity
• More specific marking schedules, tailoring
  generic indicators to the case, to help assessors
• Physical examinations need more prominence if
  they are included in the case
• Time for case calibration at the start of each
  day
• Comments about test equivalence day to day (cases
  selected electronically to specified algorithm).
  Some cases too easy/ difficult.
• Realistic, feasible cases, mostly discriminating
  between good and poor candidates

15
What the critics say

• Visitors to the CSA
• Range CE Academy of Royal Medical Colleges,
  Directors of PG GP Education, some trainers.
• Overall, well organised and robust assessment
• Fair selection of cases, reflecting everyday
  practice
• 10 minute consultations reasonable as no
  distractions
• ? How to define and apply the standard in
  marking??
• More detail in candidate feedback needed

16
Prologue how cases are developed

• Identify area of the CSA blueprint
• Think of cases seen in real life
• Consult relevant Curriculum Statement to find
  linking Learning Outcomes
• Construct case to reflect LOs

17
How cases are written

• Mostly during case writing residentials
• Working with a buddy
• Review each others cases, 3-4 each per
  residential.
• Additional information and reflection on the
  process from plenaries
• Role players come in to trial cases
• Taken home for polishing

18
How cases are written

• Focussing the case (finding the nub)
• What the candidate needs to do to pass
• Marking schedule
• Checklist
• Writing a thumbnail
• Writing the case
• Role player briefing
• Assessor briefing
• Props/ letters/ pathology results etc
• Marking schedule
• References or relevant evidence base that
  underpins the case
• Checklist/ SOCKS

19
Put on your Socks..(preparing the marking
schedule)

• Specific
• is the indicator case-specific?
• Observable
• can the assessor see whether the indicator is
  met?
• Correct domain
• is the indicator correctly located?
• Key issue
• is this indicator necessary
• Singular
• does it describe one action or behaviour..

20
What happens next

• Case sent to cell group leader for checking
• Banked in the case bank
• Case assured prior to piloting
• Piloted during a live CSA
• Case assured post piloting
• Every case has case evaluation by assessors and
  candidates at the end of every day in the CSA
• Ongoing regular checks to make sure up to date

21
Monitoring the process CSA structure for case
writing and case quality assurance
Case writing production lead and deputy (2)
Psychometric support
Case Bank lead and deputy (2)
Case Hanging Committee (6)
Case Assurance Group (10)
Case writing panel (50)
Admin support and storage/filing Of case
materials
New case writers
Case writer trainer
22
Acts

• Take a thumbnail proforma
• Think about a case you have seen recently, that
  you think would make a good case
• Do-able in 10 minutes
• Clear nub to the case that could be linked with
  a learning outcome
• Representative of British GP
• We can give you ideas if necessary, but better to
  use your own
• Spend 15 minutes completing the proforma
• Pair up for 15 minutes, discuss your cases and
  refine them as necessary

23
Illustrations

• A few presentations of your work
• Demonstration that creating cases representing
  British GP is feasible
• But most of the work lies ahead of you
• Fine tuning, thinking about cues for RPs, all the
  possible ways a candidate might consult, making
  examiner instructions watertight
• Making sure no discrepancies, collecting all
  data/ props needed for the case
• Writing the marking schedule in detail, keeping
  to the nub of the case (to help examiners as much
  as possible).

24
Appendix To be, or not to bedifficult

• Good performance in one case doesnt predict good
  performance overall so EACH case needs to be
  discriminating ( only 12 cases, so each needs to
  contribute to the reliability of the exam).
• Developing better understanding of what makes a
  case difficult
• Bread and butter cases are not necessarily easy
  but are about commonly presenting problems eg
  allergic reactions, joint pains, dyspepsia
• Can increase level of challenge in these cases by
  tweaking one of a number of pivotal indices (see
  later).

25
In an ideal world we would/ could

• Understand what makes a case difficult
• Write cases of reliable predictive difficulty
• Accurately index case difficulty
• Pick cases in such a way that the overall
  difficulty can be predicted and ensure test
  equivalence
• We needed to work out the nature of difficulty
  in our cases and how to measure it.

26
Methods we used to learn about difficulty of our
cases

• Development of theory and modelling at
  casewriting residentials
• Pivot between clinical and emotional challenge
• Opinions of assessors, RPs and candidates on case
  difficulty
• Standard setting exercises around the country
• Paired comparisons exercise with cases and
  assessors
• Performance of cases limited data

27
How to keep control of the case difficulty

• Casewriter can dictate point of fulcrum
• Case difficulty can be modulated

28
Paired comparisons exercise and what it taught us

• What the exercise consisted of
• Outcomes
• (Health Warning marketing tool giving
  qualitative inferences only!!)
• 40 assessors able to rank order 8 cases very
  similarly
• Assessors able to define what it was about each
  pair that helped them decide which was the most
  difficult

29
What makes cases difficult

• Pt is a Dr/nurse
• Task volume
• Type of clinical challenge
• Perceived high stakes case
• Hidden agendas
• Emotional challenge involving Drs feelings
• More than one problem
• Physical examination included
• Unexpected context eg phone call, home visit
• Ethical decisions/ legal frameworks to apply
• Intractable problems

30
Case difficulty whats the answer for case
writers?

• Think carefully about the case and what you want
  it to test focus, focus, focus
• Dont ask the case to test too many things
• Only use case relevant indicators that test what
  you want it to test in the marking schedule -
  otherwise too much background noise
• Use role players during writing
• Buddy writers specifically give opinion on case
  feasibility and difficulty (for Dr entering GP)
• Pilot the case
• Case assurance and performance

31
Types of data we can get on case performance

• How candidates performed on each individual case,
  compared with their overall performance (would
  expect a good candidate to pass most cases and
  vice versa).
• How many times the case was passed compared to
  the other cases.
• How reliable the case was at predicting the
  passing/ failing candidate
• Item correlation with other cases in the circuit
• Anecdotal information from evaluation forms

32
(No Transcript)
33
(No Transcript)
34
Going on tour

• How to disseminate information about the CSA and
  help candidates prepare
• What the audience can do
• Visit the CSA as observers
• Become CSA examiners
• Use this seminar to write your own cases for
  local practice
• What we are doing
• Information on website, College accredited
  courses, publications, standard setting exercises
  and presentations around the country

35
About the authors

• Paulines plea.