Test Writing Workshop

1
Test Writing Workshop

• Chris Martin, MD, MSc
• Assistant Professor
• Department of Community Medicine

2
Outline

• Take the exam
• Introduction
• Multiple choice questions
• Writing the stem
• Writing the distracters
• Analysis of questions

3
What is the goal of any examination?
4
A Teachers View
Curriculum
Teacher
5
A Students View
Examination
Student
6
A Teachers View vs. A Students View
Curriculum
Examination
Communicates Content
Communicates Values
Teacher
Student
_____________ Melnick D. Presentation at
Marshall U. February 12 2001.
7
If you say in a lecture, this isnt on the exam
8
What must be achieved in an examination?
9
What must be achieved in an examination?

• Need to scatter test-takers
• Particularly important to discriminate around
  pass cut-off

10
Examination Formats

• True/False
• Essay
• Short answer
• Observed encounter (OSCE)
• Oral examination

11
MCQs - Advantages

• Objective
• Anonymous
• Rapidly graded

12
MCQs - Disadvantages

• Can only test recognition
• Limitations in testable material
• Difficult to write

13
Writing Good MCQs

• Plan ahead try to think of questions before the
  lecture
• Ensures material has been covered
• Ensures that you will have a enough testable
  material

14
Anatomy of an MCQ

• Dr. Chris Martin is from what country?
• a. Cameroon
• b. China
• c. Central African Republic
• d. Canada

15
What is the hardest part of an MCQ to write?
16
What is the hardest part of an MCQ to write?

• The stem
• The correct answer
• The distracters

17
What is the hardest part of an MCQ to write?

• The stem
• The correct answer
• The distracters
• Answer c

18
Writing Good MCQs General Format

• Single, best answer
• Forget K type questions
• 3 distracters

19
Writing the stem

• Clear and concise
• Avoid teaching in the stem
• Each item should focus on one important concept,
  avoid rote recall.
• Think about the one testable concept you are
  incorporating

20
Bad MCQ

• Carpal tunnel syndrome involves which nerve?
• Median
• Ulnar
• Radial
• Brachial

21
Better version

• A 40 year old woman employed in poultry
  processing presents to you with a 6 month history
  of pain and tingling involving the thumb, index,
  and middle fingers which has been waking her up
  at night. Which of the following is the most
  likely cause of her symptoms?
• Ulnar nerve entrapment at the elbow
• Median nerve entrapment at the wrist
• Ulnar artery occlusion
• Radial artery occlusion

22
Writing the stem

• Clear and concise
• Avoid teaching in the stem
• Each item should focus on one important concept,
  avoid rote recall.
• Do not use negative questions

23

• Which of the following is NOT a risk factor for
  infectiosum incognito?
• Excessively long hair
• Lack of vitamin H
• Body odor
• An absence of the Zapatista haplotype

24
Writing the stem

• Clear and concise
• Avoid teaching in the stem
• Each item should focus on one important concept,
  avoid rote recall.
• Do not use negative questions
• Items should be answerable without looking at
  options
• Incorporate as much information in the stem as
  possible.

25

• Which of the following is TRUE?
• With respect to back pain, the MRI has low
  sensitivity.
• With respect to back pain, the MRI has low
  specificity.
• With respect to back pain, the MRI has a high
  positive predictive value.
• With respect to back pain, the MRI has a high
  number of false negatives.

26

• What is the main limitation in the use of MRI for
  the investigation of a patient with low back
  pain?
• Low sensitivity.
• Low specificity.
• High positive predictive value.
• High number of false negatives.

27
Writing the distracters

• Avoid none of the above and all of the above

28

• What is the capital of Canada?
• Montreal
• Toronto
• Vancouver
• None of the above.
• (Student, knowing the correct answer is Moose
  Jaw, selects d.)

Ottawa (Rideau Canal - Largest skating rink in
the world)
29
Distractors

• Avoid none of the above and all of the above
• Should be plausible (yet clearly incorrect!)
• Make sure they are grammatical correct
• Should resemble the correct answer
• Differ on single dimension all labs, all
  diagnoses, etc.)
• Keyed answer on a single continuum from most
  likely to least likely

30

• What is the positive predictive value of a test?
• 1. The likelihood that a positive test result
  represents a true positive.
• 2. The likelihood that someone with disease will
  have a positive test result.
• 3. The likelihood that someone without disease
  will have a negative test result.
• 4. The likelihood that someone with disease will
  have a negative test result.

31
How can I tell if my question is good?

• Analyze !
• A precondition is a secure bank of questions to
  reuse
• Statistics vary, but include two common elements
• achieving correct score
• Some indication of discrimination

32
Discrimination 0.01Mean Score 98.9

• Hill's criterion of "biological gradient" refers
  to
• 1. differences in disease rates comparing
  different animals used in laboratory experiments
• 2. patients becoming rapidly more sick as a
  disease progresses
• 3. patients fearing diseases such as cancer, even
  though not all cancers are fatal
• 4. differences in disease rates with different
  amounts of exposure

33
The worst MCQ I have ever written
34
Discrimination - 0.09Mean Score 24.2

• Which of the following is NOT an example of an
  application of the Public Health Model?
• 1. A mother bringing her 1 year old child to a
  clinic for vaccinations.
• 2. Development of a rapid response paramedic team
  for trauma victims.
• 3. A research study of smoking patterns in high
  school students.
• 4. State laws requiring the use of seat belts.
• 5. A physician advising her patient to lose
  weight.

35
Discrimination 0.30Mean Score 29.5

• Using death certificates for cholera in September
  1854 and maps to investigate the Broad Street
  pump outbreak, John Snow performed what type of
  epidemiological study?
• 1. A retrospective cohort study
• 2. A prospective cohort study
• 3. A cross-sectional study
• 4. A correlational study

36

• Penzak SR Gubbins PO Stratton SL Anaissie EJ.
  Investigation of an outbreak of gram-negative
  bacteremia among hematology-oncology outpatients.
  Infect Control Hosp Epidemiol 2000
  Sep21(9)597-9.

37

• This study was an investigation "to identify risk
  factors associated with an outbreak of
  gram-negative bacteremia bacteria in the blood
  (GNB)" among hematology-oncology outpatients in a
  university hospital. "Thirty-eight patients
  developed GNB 13 patients experienced more than
  one episode, and eight blood cultures grew more
  than one gram-negative organism. The most
  frequently isolated organisms were
  Stenotrophomonas maltophilia, Klebsiella
  pneumoniae, Acinetobacter baumannii, and
  Acinetobacter johnsonii. When the GNB patients
  were compared with randomly selected
  hematology-oncology patients, central venous
  catheter (CVC) self-care (71 vs 39 P.02), and
  duration of recent hospital stay (median, 15 vs 4
  days P.01) were identified as risk factors. In
  a logistic regression model, duration of recent
  hospital stay was the only risk factor
  significantly associated with GNB (odds ratio,
  1.05 95 confidence interval, 1.01-1.08 Plt.02).
  CONCLUSIONS Hematology-oncology patients
  providing their own CVC care who have recently
  been hospitalized for more than 2 weeks may be at
  increased risk of GNB. CVCs should be protected
  from possible environmental contamination in
  hematology-oncology patients. Patients providing
  their own CVC care should undergo continued
  rigorous education regarding proper CVC care."

38
Discrimination 0.56Mean Score 68.1

• This type of study is
• 1. A retrospective cohort study
• 2. A randomized clinical trial
• 3. A cross-sectional study
• 4. A case-control study
• 5. A prospective cohort study

39
Not all your questions will be great performers

• Blend to develop a balanced exam
• Treasure highly discriminatory questions
• Psychometricians may provide helpful input
• A question which performs badly is a failure of
  the teacher, not the students
• I have yet to administer an exam in which
  questions were not dropped
• For dropped questions, add a point to everyones
  grade, rather than make all options correct