Standard Setting for Clinical Skill Assessment

1
Standard Setting for Clinical Skill Assessment

• Dr Graeme Horton
• Examples taken from presentation by Dr Onishi
  (University of Tokyo) at 12th International
  Ottawa Conference

2
Aims of Assessment

• These include
• Motivating Learners
• Providing Feedback
• Assessing Excellence
• Assessing Competency
• This presentation will focus on ways which OSCE
• examinations can be marked to assess competence

3
Reliability vs Validity

• When considering different assessment
• instruments, it is important to consider their
• Reliability stability of measurement
• Validity whether it measures what is relevant or
  targeted

4
Reliability, Validity and Standards

• In written examination, reliability can often be
  high as a wide range of knowledge is assessed in
  the one test
• In short case and long case exams, validity is
  high they test more of the day to day skills
  which doctors perform, however reliability can be
  poor as the result can depend on what patients
  are available on the day
• With OSCE stations reliability is improved and we
  are able to focus more on what determines whether
  a candidates performance is satisfactory.

5
The Optimum Standard

• This is worth determining as
• False fails waste resources and demotivate
  students
• False passes leave learners who require
  additional education

6
Criterion vs Norm-referenced

• Whilst norm- referenced assessment has the
  following
• advantages
• Less faculty intensive
• Able to predetermine the number of passing
  examinees
• Criterion referenced methods focus on whether the
  students performance is satisfactory, so that
• If all examinees are competent all can pass
• Each examinee does not target at competition, but
  at passing criterion

7
Example of OSCE station

• In order to demonstrate different methods of
  scoring OSCE examinations, we will consider the
  following example of an OSCE station.
• This question might form part of the final
  summative assessment in an orthopaedics term

8
(No Transcript)
9

• Following are the marking sheets of two students
  one performed poorly and the other performed
  well.
• There are two types of scores given
• A checklist score which looks at different skills
  within the assessment separately
• A global score which rates the students overall
  performance.

10
(No Transcript)
11
(No Transcript)
12
Global vs Itemised Scoring Systems

• Checklist scoring advantages
• Helps examiners know what the station writer is
  looking for
• Helps examiners be objective
• Facilitates the use of non experts as markers
• Checklist scoring disadvantages
• It can just reward the process or thoroughness of
  the student rather than integrative processes
• It may not sufficiently reward excellence
• Does not utilise examiner expertise who may not
  enjoy just ticking boxes

13

• Global Scoring advantages
• Utilises the expertise of examiners
• Examiners are in a position to make global
  judgment
• Global Scoring disadvantages
• Examiners have to be expert and well trained in
  conducting OSCEs
• Examiners must be familiar with the expected
  standard of candidates at that level
• The Global rating is used for two of the three
  rating systems we will look at later.

14

• Fifty candidates sat our sample OSCE.
• Their marks are written in the first table
• The right hand table ranks the scores with
  cumulative totals of the marks.
• In otherwords,
• No students scored 8.5
• One student scored 9.0
• One student scored 9.5
• The cumulative total at 9.5 is therefore 2

15
(No Transcript)
16
This cumulative table is then plotted
17
Hofsee Method

• Panelists decide lowest and highest acceptable
  percentage of fail students
• Panelists decide lowest and highest acceptable
  cut-off scores
• A curve is plotted accumulative of students for
  each score (as per previous chart)
• The score where the diagonal line and the curve
  cross will be the cut-off

18
Our panelists decided that they were prepared to
fail between 0 and 19 students They thought a
reasonable pass score would fall between 9 and 13
The diagonal between the extremes of these
ranges therefore gave a cut off score of 11.5
19
Borderline Regression Method

• Accumulated score (checklist score) is plotted
  against global rating score for each candidate
• Find line of best fit
• Pass mark is where line crosses borderline on X
  axis
• In the following example, 10 of the 50 students
  in our OSCE are tabled and plotted

20
(No Transcript)
21
Angoff Method

• Expert judges imagine how borderline examinees
  competently perform the skill item by item.
• For each skill item the proportion of borderline
  students who would pass the item is written next
  to the item.
• This percentage is then multiplied by the maximum
  score for each item to give the pass subscore for
  that item.
• Note This is all usually done before the
  candidates sit the test

22
Here the examiners have decided that the pass
mark for this station will be 9.7 out of 20 Some
examiners manipulate these pass marks once the
actual results from the test are known.
23
Considerations

• Hofsee
• A mix of normative and criterion based a
  compromise
• Acceptable ranges of pass mark might be difficult
  to determine numbers
• Borderline Regression
• Accuracy/validity of global rating scale is
  questionable
• Angoff
• Availability of judges is limiting step
• Difficulty in imagining borderline examinees can
  be a problem

24
Guidelines for Writing High Quality Multiple
Choice Questions

• Dr Graeme Horton
• Based on Material from Dr Gordon Page, presenter
  at 12th International Ottawa Conference and
• IDEAL Faculty Development materials

25
Evaluating an Item

• Does it test an important concept ?
• Is it clear and Unambiguous ?
• Is the item flawed?
• Technical flaws undermine validity!
• You want the knowledgeable students to do better
  not the exam-wise ones.

26
The problem should be in the stem, not in the
options

• Cover the Options Rule the candidate should be
  able to think of possible answers to the stem
  without having to see the options.
• This is said to be the most important point in
  item writing and if adhered to, many common flaws
  are avoided

27

• Example of a question which adheres to the
• Cover the Options Rule
• A 62 year old man with alcohol dependence is
  admitted to hospital for TURP. The following
  morning while being transported to the operating
  room, he has two generalised seizures within five
  minutes. Neurological exam shows no focal
  abnormalities. Which of the following is the
  most likely diagnosis?
• Alcohol withdrawal
• Korsakoffs psychosis
• Partial complex motor seizure
• Wernickes Encephalopathy
• Correct A

28
Plausible distracters are vital

• Students who do not know the material increase
  their chances of guessing the correct option by
  eliminating implausible distracters

29
Avoid repeating words in the stem and in the
correct option

• Similar wording allows students to identify the
  correct option without knowing the material

30

• Example of repeated words
• A 58 year old man with a history of heavy alcohol
  use and previous psychiatric hospitalisation is
  confused and agitated. He speaks of experiencing
  the world as unreal. This symptom is called
• Depersonalisation
• Derailment
• Derealization
• Focal memory deficit
• Signal anxiety
• Correct C

31
Avoid convergence cues!

• Question writers tend to use the correct answers
  more frequently across all options.
• Students will identify as correct the answer in
  which all components appear most frequently

32

• Example of convergence cue
• Local anaesthetics are most effective in the
• Anionic form , acting from inside the nerve
  membrane
• Cationic form, acting from inside the nerve
  membrane
• Cationic form, acting from outside the nerve
  membrane
• Uncharged form, acting from inside the nerve
  membrane
• Uncharged form, acting from outside the nerve
  membrane
• Correct B

33
Options should be similar in length and amount of
detail

• If one is longer or contains more detail or it
  contains more complex language, students can
  usually correctly assume that this is the correct
  answer.

34
Avoid use of never, always, only, all

• Students are taught that there are often no
  absolute truths in most health science subjects
  and they can therefore eliminate these
  distracters.

35

• For example
• In patients with advanced dementia, Alzheimers
  type, the memory deficit
• Can be treated adequately with phosphatidylcholine
  (lecithin)
• Could be a sequelae of early Parkinsonism
• Is never seen in patients with neurofibrillary
  tangles at autopsy
• Is never severe
• Possibly involves the cholinergic system
• E Correct

36
Avoid use of vague terms

• such as frequently, occasionally, rarely,
  usually, commonly
• These terms lack precision and there is seldom
  agreement on the actual meaning of often or
  frequently

37

• Which of the following statements is true about
  cleft palate?
• It may result in speech, hearing and dental
  abnormalities
• Repair is frequently deferred until the child is
  two years old
• It is often associated with mental retardation
• Children with cleft palate commonly have mood
  disorders

38
Avoid the use of negatives

• They poorly assess students actual knowledge.
• If teachers wish to assess contraindications, the
  questions should be worded clearly to indicate
  that this is what is being assessed.

39
Avoid use of all of the above

• Students can easily identify if this is correct
  if at least two options are correct.
• They can eliminate it by knowing if only one of
  the options is incorrect.

40
Avoid none of the above

• It only measures students ability to detect
  incorrect answers.
• If used, and it is the correct option, the
  teacher must be certain that there are no
  exceptions to any of the options that the student
  may detect.

41

• For example
• A 62 year old man with alcohol dependence is
  admitted to hospital for TURP. The following
  morning while being transported to the operating
  room, he has two generalised seizures within five
  minutes. Neurological exam shows no focal
  abnormalities. IV administration of which of the
  following drugs is most appropriate.
• Diazepam
• Haloperidol
• Phenobarbital
• Phenytoin
• None of the above

42
Interprofessional Learning

• that works!
• Examples for 12th International Ottawa Conference
  
• Dr Graeme Horton

43
Interprofessional Learning

• Can enhance collaboration
• Can reduce service fragmentation
• Can promote high quality care

44
Ward Rounds

• Important part of the everyday routine of doctors
  and nurses
• Complex task that requires interpersonal skills
  and collaboration
• Are where different professional groups can offer
  unique and essential contributions

45
Copenhagen University Hospital

• Trained medical students and nursing students in
  the workings of a ward round
• One day training course
• Using actors as simulated patients

46
Purpose of Course

• To develop understanding of knowledge, skills,
  roles and duties of the other profession
• To stress the importance of good communication
  for teamwork and patient care
• Suggested structure for conducting ward rounds
• To increase the consciousness of each
  professionals own role and duties

47
Structure of Course

• Preparatory manual with a structured model for
  teamwork in conducting ward rounds relating to
  discharge planning
• Medical and nursing students interviewed each
  other in pairs
• Proceeded to role playing of ward round scenario
  this was videotaped
• Discussion in plenary

48
Feedback

• Students appreciated gaining the perspective of
  their colleagues
• Realised that they both had important and
  complementary roles

49
OSCE station

• Two parts , each six minutes
• Part 1 student receives case story and prepares
  discharge of patient
• Part 2 station set up with nurse and patient
  actor waiting for ward round

50
Evaluation

• Statistical difference between students in
  control and project group in relation to
  collaboration with the nurse and patient during
  the session and therefore OSCE performance

51
Other models of Interprofessional Learning

• Example from Sweden
• Combined Care model in Emergency Department
• Orthopaedic patients triaged for combined care
• Seen by medical, nursing , physio, OT students
• Management plan devised and reviewed by treating
  doctor