Constructing a test

1
Constructing a test
2
Aims

• To consider issues of
• Writing assessments
• Blueprinting

3
Methods of assessment MCQs Multiple
true/false Best of five Extended
matching questions Short answer
questions Modified essay questions Objective
Structured Examination Mini-sampling
examinations Prolonged observation Summative
at end of year Multiple throughout
years Oral examination Non-structured vs.
structured Assessment of professional
behaviours Utility?
4
The professional progress Knows Knows
how Shows
5
The professional progress Knows Knows
how Shows
Knowledge Can state how
Demonstrates information is used
competence MCQs MEQs MEQs EMQs
EMQs EMQs Multi-station exam
Multi-station exam OSCE
Workbased test Multi-source
data Professionalisms
6
DANGER
Subjects
7
DANGER
Subjects Many curricula
Assessments
8
DANGER
Subjects
Assessments
Learning
Subjects Many curricula
9
DANGER
Subjects
Assessments
Learning
Subjects Many curricula
Disconnected teaching leads to disconnected
assessments
Leads to disconnected learning
10
Future directions in assessment
Testing competencies by ability to handle complex
tasks van der Vleuten and Schurworth
2005 Assessment in context of practice
work-place based assessments Rethans
et al, 2002 Multiple sampling by multiple
assessors Petrusa, 2002 Peer
review Do OSCEs do all that you
need? Professional behaviours assessed
longitudinally and at many times
11
Methods of assessment MCQs Multiple
true/false Best of five Extended
matching questions Short answer
questions Modified essay questions Objective
Structured Examination Mini-sampling
examinations Prolonged observation Summative
at end of year Multiple throughout
years Oral examination Non-structured vs.
structured Assessment of professional
behaviours Utility?
12
Methods of assessment in use MCQs Multiple
true/false Best of five Extended
matching questions Short answer questions Key
feature case Modified essay
questions OSCE Mini Clinical Examination
(Mini-CEX) Long case Summative at end of
year Multiple throughout years Oral
examination Non-structured vs.
structured Assessment of professional
behaviours Multiple throughout course
13

• Methods of assessment in use in Sheffield
• Extended matching questions
• Modified essay questions
• OSCE
• Mini Clinical Examination (Mini-CEX)
• Observed long cases - multiple throughout years

14

• Methods of assessment in use in Sheffield
• Extended matching questions
• Modified essay questions
• OSCE
• Mini Clinical Examination (Mini-CEX)
• Observed long cases - multiple throughout years

All need Resources Staff who are
trained and able Exams prepared months in
advance Evidence that the exams are good
Student understanding Good statistics support
15
Validity linking to the curriculum by
blueprinting

• Blueprinting
• aims to ensure that the defined curriculum (and
  only that) is assessed
• ensures that a reliable sample of the curriculum
  is selected for testing
• enables the most appropriate test methods to be
  used for different aspects to be tested

16
Assessment blueprinting
Subjects/ modules taught
Core problems
17
Assessment blueprinting
Subjects/ modules taught
Core problems
Components to be tested
18
Assessment blueprinting
Subjects/ modules taught
Core problems
Exam components and weighting
Components to be tested
19
Assessment blueprinting
Subjects/ modules taught
Core problems
Exam components and weighting
Components to be tested

• Questions
• Balanced sampling between subjects,
• teaching components and exam
• components
• Marking appropriately weighted
• Increases validity
• Guides students and staff

20
Validity linking to the curriculum by
blueprinting
Core clinical problems
etc
RS
GI
ENDO
CVS
21
Validity linking to the curriculum by
blueprinting
Core clinical problems
etc
RS
GI
ENDO
CVS
History
Examination
Case Management
Communication skills
Data interpretation
22
FeasibilityCan the test be delivered?

• Time management
• Blueprinting / writing questions in time
• External validation of the test
• Educational
• Can what you want to test be tested in the format
  proposed?
• Practical
• Sufficient space examiners, patients

23
Test securityKeeping it confidential

• Who sees the examination paper?
• Reuse of previous papers/questions

24
CostsHuman financial

• Human resources
• Academic and administration staff time
• Simulated patients/actors
• Financial costs
• OSCEs
• Manikins
• consumables

25
Information to candidates

• Ensure candidates know the purpose
• of the assessment
• Information about the test
• Results and (remediation/resit)

26
Feedback

• All assessments must lead to feedback
• All assessments are formative in part
• Without feedback, the educational loop is left
  open

27
Characteristics of good assessments

• Valid - test what they purport to test
• - link to the curriculum
• Reliable - reproducible
• Feasible - can be delivered
• Impact - preparation of students and
  feedback to them
• Leadership - with an overview

28
Question writing

• Link one question to one topic

29
Question writing

• Link one question to one topic
• Decide what you wish to test
• eg knowledge / reasoning / psychomotor skill /
• communication / response to emergencies
• Decide best format of question

30
Question writing

• Link one question to one topic
• Decide what you wish to test
• eg knowledge / reasoning / psychomotor skill /
• communication / response to emergencies
• Decide best format of question
• Write - accurately
• - unambiguously and no negatives
  single or double!
• - without trying to confuse or
  mislead
• - at the right level
• - to produce an exam that links
  directly to the curriculum
• - and show your work to colleagues
  / assessment group
• - with non-specialists
• - using the appropriate clinical
  context

31
Assessment is enhanced if it is

• Authentic
• Related directly to practice
• Seen by students to be fair, have a utility,
• be related to the curriculum and be a source
• of feedback
• Is more than an exercise in factual recall