Question Writing Workshop Stanford University Medical Education Faculty

1
Question Writing WorkshopStanford University
Medical Education Faculty

• Lynn C. Webb, Ed.D.
• August 15, 2007

2
Agenda

• Introductions
• Examination goals
• Giving away the answers
• 3 formats for questions
• Investing in future examinations
• The joy of feedback
• Questions / discussion

3
1. Introductions

• Lynn Webb Testing consultant
• Previous affiliations
• American Board of Psychiatry Neurology
• National Board of Medical Examiners
• ETS (Princeton)
• ACT (Iowa City)

4
Examples of Tests

• AANs RITE
• ACPs PRITE
• APAs FOCUS
• AOAs UHM
• ABPM
• AAPMRs SAE-R
• WIP
• CBNC

• ABPN
• Certification
• Recertification
• Subspecialty
• Added qualifications
• Non-medical exams
• NCLEX
• Solar Energy

5
2. Examination Goals

• Test students learning
• Validate your teaching
• Prepare students for future testing
• Your questions can contribute to these goals, or
  detract from them.

6
Contribute to Goals

• Questions should be aligned with the specific
  content
• Questions should be technically sound
• Clearly written
• Straight-forward, NOT tricky

7
Detract from Goals

• Questions dont address teaching or learning
• Questions are convoluted or confusing, testing
  something other than the content
• The focus shifts to the question instead of the
  content

8
3. Giving away the answers

• Medical students tend to be test-wise
• When they dont know an answer, they retrace your
  steps in writing the question to guess the
  correct answer
• Even if YOU are test-wise, blind-spots exist when
  its your material
• Multiple-choice questions are especially
  vulnerable

9
Lets take a quiz

• Our content will be highly esoteric facts about
  the Alaskan Malamute
• Each quiz question will contain a flaw in its
  structure that will assist test-wise examinees.
  You wont need to study the Alaskan Malamute to
  pass this quiz.

10
Boris Badenov Webb
11
1. The primary function of sled dogs such as the
Alaskan Malamute, Samoyed, Siberian Husky, and
Eskimo Dog is to

• rescue drowning humans
• retrieve tossed objects
• guard property
• pull sleds

12
Test-wise examinees

• Look for hints that the question writer
  overlooked between the question and the correct
  option
• Sled dogs Pull sleds (no other options about
  sleds)

13
2. Sled dogs provided vital transport of a
life-saving serum during a diphtheria epidemic in
Nome, Alaska in

• 1492
• 1776
• 1925
• 1952

14
Several ways to approach question

• First, eliminate options that arent plausible
  1492 and 1776 are too early for question content
• Now you have 50/50 chance between C and D.
• Is D plausible? Why not fly in 1952?
• Numeric options when in doubt, choose C

15
3. Credit for the introduction of the Alaskan
Malamute in the lower 48 is usually given to

• Arthur Treadwell Walden and The Seeleys
• Admiral Byrd and Arthur Treadwell Walden
• The Seeleys and Commodore Perry
• Arthur Treadwell Walden and Jack London

16
Look for repeats

• Arthur Treadwell Walden and the Seeleys
• Admiral Byrd and Arthur Treadwell Walden
• The Seeleys and Commodore Perry
• Arthur Treadwell Walden and Jack London

17
More repeats?

• Arthur Treadwell Walden and The Seeleys
• Admiral Byrd and Arthur Treadwell Walden
• The Seeleys and Commodore Perry
• Arthur Treadwell Walden and Jack London

18
Some guys listed only once

• Arthur Treadwell Walden and The Seeleys
• Admiral Byrd and Arthur Treadwell Walden
• The Seeleys and Commodore Perry
• Arthur Treadwell Walden and Jack London

19
Where do repeats converge?

• Arthur Treadwell Walden and The Seeleys
• Admiral Byrd and Arthur Treadwell Walden
• The Seeleys and Commodore Perry
• Arthur Treadwell Walden and Jack London

20
Why do test-wise examinees do that?

• They are retracing your steps in creating the
  question
• Thats why you shouldnt submit your first draft
• Its easy to fix this problem

21
4. Champion Coldfoot Oonanik, the worlds most
titled dog, was how many inches at the withers?

• 2.75
• 25.0
• 27.5
• 275

22
Test-wise examinees compare options

• 3 options have the digits 2 7 5
• One option has different digits (B), but its
  close in size to C
• C must be the answer
• (Author had the answer 27.5, then moved the
  decimal place for 2 distractors, then chose
  another plausible one near answer)

23
Alaskan Malamutes are known for a

• wooly undercoat, from 1-2 in depth when the dog
  is in full coat
• early warning system concerning burglaries or
  robberies
• interest in jumping on furniture and sleeping in
  beds with humans
• eye color of blue, brown, or one blue and one
  brown

24
Which option fits with stem?

• Grammar issue a only works with wooly
• Also, author qualified answer in A when the
  dog is in full coat not all year round. The
  other options arent qualified

25
6. Alaskan Malamutes tails are

• wiry
• short
• tightly curled
• plumed and carried over their backs most of the
  time

26
Where did author spend time?

• Correct answer is longest
• Correct answer is most detailed
• Correct answer is qualified

27
7. Which of the following is char-acteristic of
Alaskan Malamutes?

• They dont shed
• They are always strong and silent
• They can survive extreme cold and usually
  tolerate warm climates
• They are always one man dogs

28
Where did author spend time?

• Correct answer is longest
• Correct answer is qualified
• Other answers include absolutes

29
Could these clues exist in your examinations?
30
Thrombotic microangiopathies are associated
with

• Thrombocytopenia
• Reduced coagulation factors
• Antibodies to glomerula basement membranes
• Anti-neutrophilic cytoplasmic antibodies
• Necrotizing vasculitis

31
A patient carefully reads the labels on all her
food products and calculates that she is taking
in about 3.5 grams of sodium. This is closest to

• 60 mEq of sodium
• 100 mEq of sodium
• 160 mEq of sodium
• 200 mEq of sodium

32
Numeric options

• You may use C as the correct answer
• It shouldnt ALWAYS be the correct answer

33
4. 3 formats for questions

• Lets talk about the 3 formats that you use for
  the test questions
• Multiple choice questions (MCQs)
• True/False
• Matching

34
Multiple-choice questions

• The examinees task is to choose the one best
  answer of 5 options (A-E)

35
Multiple Choice

• Idea
• Correct answer
• Distractors

36
Multiple Choice Qs Ideas

• The idea for the question is sometimes called the
  teaching point
• You need questions for each lecture hour you
  present
• Test essential concepts

37
Source of ideas

• Essential content
• Questions asked during lecture
• Conversations overheard
• Any points upon which you will build

38
How to start?

• Review syllabus notes
• What are the most essential concepts in the
  lecture?
• The purpose of this lecture is to teach
  _______________________
• You might have 6 concepts, or you might have 2
  concepts and some sub-topics

39
Sampling

• Testing is always a sampling of knowledge
• Even the USMLE samples knowledge
• If you make a list of 12 things that MUST be
  tested from a lecture, perhaps ½ of them could be
  used now and ½ of them could be used later.
  (More on this in Topic 5 Investing in the
  Future)

40
Characteristics of GREAT questions

• Clearly written
• Examinee knows what is being asked before reading
  the options
• All the information needed to answer is in the
  question (options dont build upon each other)
• Simplest phrasing possible
• No extraneous information

41
MCQs Correct answer

• Each question should have one answer that is best
• When youre writing, the best sequence is
• Teaching point (What you want to ensure they
  know)
• Correct answer
• Distractors

42
Other Sequence

• (Not recommended)
• Vague concept
• List of options
• Choose one to be correct OR revise some options
  to be either correct or incorrect

43
Example FK506 (Tacrolimus)

• is an anti-proliferative drug
• binds and inhibits the stimulation of a calcium
  activated phosphatase, thereby blocking the
  induction of the early response genes in the
  immune response
• is less commonly used than cyclosporin because of
  its lower potency
• is preferred over cyclosporin since it has no
  nephrotoxicity
• is most commonly used to treat auto-immune
  disorders

44
If you didnt know anything about FK506.

• Which option would you choose
• Hint retrace the steps of the question writer

45
FK506 (Tacrolimus)

• is an anti-proliferative drug
• binds and inhibits the stimulation of a calcium
  activated phosphatase, thereby blocking the
  induction of the early response genes in the
  immune response
• is less commonly used than cyclosporin because of
  its lower potency
• is preferred over cyclosporin since it has no
  nephrotoxicity
• is most commonly used to treat auto-immune
  disorders

46
What to do?

• Choose a specific teaching point from the lecture
  notes what is it that you want to be sure the
  students know about FK506?
• Form that teaching point into a question
• Write the correct answer
• Write the distractors

47
Multiple Choice Qs Distractors

• The distractors should be
• - plausible
• - less correct than the answer
• They are called distractors because their
  intent is to DISTRACT examinees who do not know
  the concept

48
Sources for Distractors

• Common misconceptions (Why do some of the
  students think that ___________ )
• Logical thinking, but it doesnt apply in this
  case

49
FK506 (Tacrolimus) binds and inhibits the
stimulation of

• calcium activated phosphatase

50
Better format

• But is it an essential teaching point?

51
(or) What happens when FK506 (Tacrolimus) binds
and inhibits the stimulation of calcium activated
phosphatase?

• The induction of the early response genes in the
  immune response is blocked

52
Your item here?
53
Why change format?

• When questions are written clearly, its more
  likely that responses will reflect examinees
  knowledge on this topic, rather than their care
  in reading the question
• Typical of the type of questions they will see on
  USMLE and specialty certification examinations

54
USMLE others do NOT use

• Which of the following is TRUE?
• Which of the following is FALSE?
• Each of the following statements about X is true
  EXCEPT

55
Which one is true (or false)

• Is not a multiple-choice question format
• Its a multiple true/false format
• Options tend to be heterogeneous (mixing apples
  and oranges) because the teaching point isnt
  focused

56
True/False Questions

• Much easier/faster to write
• - Less information about what the students know
• They have a 50/50 chance of answering correctly,
  just by answering (MCQs were only 20 chance)

57
When to write a T/F question?

• When the content is essential, but shallow
• When the multiple choice question didnt work

58
FK506 (Tacrolimus)

• is an anti-proliferative drug
• binds and inhibits the stimulation of a calcium
  activated phosphatase, thereby blocking the
  induction of the early response genes in the
  immune response
• is less commonly used than cyclosporin because of
  its lower potency
• is preferred over cyclosporin since it has no
  nephrotoxicity
• is most commonly used to treat auto-immune
  disorders

59
True/False Qs Ideas

• (same as Multiple Choice Qs)
• Essential content
• Questions asked during lecture
• Conversations overheard
• Any points upon which you will build

60
True/False Qs Correct answer

• Be clear
• Dont be tricky
• If something is generally true (say in 999 of
  1,000 cases) do you want them to say TRUE for the
  999 cases or FALSE for the 1 case?

61
Examples

• Menarche is usually the first sign of puberty in
  females
• Endometrial adenocarcinoma typically presents as
  high stage disease.

62
Great material for T/F

• Can be found in unfocused multiple choice
  questions

63
Which statement is true about Bladder carcinoma?

• Bladder carcinoma is not associated with smoking
• Recurrence and progression are very rare after
  local excision of low grade papillary carcinomas
• Field effect refers to the concept that toxins
  in wheat grain pesticides may cause prostate
  carcinoma
• Bladder carcinoma is usually adeno-carcinoma
• Squamous cell carcinoma is associated with
  chronic irritative conditions such as chronic
  catheterization

64
Any questions about the True/False format?
65
Matching

• A series of 2 or more questions that use the same
  set of up to 10 possible answers (listed once as
  a heading).
• The answers may be used once, more than once, or
  not at all
• There is only ONE correct answer per question

66
Matching Ideas

• Essential content
• Several concepts that are alike typically
  testing recognition more than application of
  knowledge
• Lists of diseases, microorganisms, medications,
  etc.

67
Matching Correct answers

• This format requires more thought about correct
  answers
• After preparing a matching set, review each
  description again (left side) comparing to each
  of the options on the right. You must be sure
  that only ONE is the best answer

68
Matching Distractors

• There are typically fewer distractors to generate
  for Matching than for MCQs, because the other
  correct answers are included in the distractor
  list
• Additional distractors that are added should be
  common misconceptions about descriptions on left
  or plausible-sounding alternatives from within
  the category of things listed.

69
Any questions about the matching format?
70
5. Investing in Future Examinations

• You might decide to work ahead for future
  examinations when youre working on your assigned
  questions.
• Work ahead when you have an abundance of ideas
• Clone your questions

71
Cloning Multiple Choice Qs

• Especially easy to do when starting from a
  question that had a negative context (NOT,
  EXCEPT, etc.)

72
Which one of the following obstetric risks is NOT
commonly associated with mullerian anomalies?

• Intrauterine growth restriction (IUGR)
• Pre-eclampsia
• Malpresentation
• Preterm labor

73
4 options

• 3 are associated with mullerian anomalies
• 1 is NOT associated with mullerian anomalies

74
Create a positive item

• Which of the following obstetric risks is
  commonly associated with mullerian anomalies?
• Pre-eclampsia (now a distractor, instead of
  correct answer)
• Intrauterine growth restriction (IUGR) (now the
  correct answer)
• (add distractor here)
• D. (add distractor here)

75
Clone new item

• Same item, use same distractors, but insert new
  correct answer

76
Which one of the following obstetric risks is
commonly associated with mullerian anomalies?

• Pre-eclampsia (still a distractor, instead of
  correct answer)
• Malpresentation (new correct answer)
• (add same distractor here)
• D. (add same distractor here)

77
Clone second item

• Same process
• Keep same distractors
• Insert new correct answer

78
Which one of the following obstetric risks is
commonly associated with mullerian anomalies?

• Pre-eclampsia (still a distractor, instead of
  correct answer)
• Preterm labor(new correct answer)
• (add same distractor here)
• D. (add same distractor here)

79
Another example

• Change negative context Multiple choice question
  to positive, then clone it.

80
Negative context question

• Post-streptococcal glomerulonephritis is
  associated with all EXCEPT
• Subepithelial humps
• Pharyngitis
• Low serum complement levels
• Low anti-streptolysin antibody titers
• Hematuria

81
Post-streptococcal glomerulonephritis is
associated with

• Low anti-streptolysin antibody titers (now a
  distractor)
• Subepithelial humps (now correct)
• (add a distractor)
• (add a distractor)
• (add a distractor)

82
Clone 3 more items

• Keep the 4 distractors (1 old, 3 new) the same
• Change the correct answer each time,
  alternatively using
• Pharyngitis
• Low serum complement levels
• Hematuria

83
Do students share exams?

• Cloning items is a great way to discourage
  memorizing test materials. The correct answer
  changes across administrations of the test.

84
Many topics have more than one compenent

• Think of the questions you used to write as All
  of the above - Those kinds of topics are great
  for cloning
• Write the question with only one of the possible
  correct answers
• Work hard to develop plausible distractors
• Keep the question and distractors alternate the
  correct answers

85
Use 1 per test

• Store the others for the future

86
Cloning True/False

• Write the True/False question
• Clone it to the opposite answer

87
TRUE Original

• The most common cause of congenital adrenal
  hyperplasia is 21-hydroxylase deficiency
• Now, clone to FALSE

88
Clone to FALSE

• The most common cause of congenital adrenal
  hyperplasia is 3-beta-HSD deficiency
• Or
• The most common cause of congenital adrenal
  hyperplasia is 11-hydroxylase deficiency

89
Cloning Matching

• Not as easy to do as for Multiple Choice or
  True/False
• Easiest when the list of possibilities is very
  long
• Try to utilize the options that were not used in
  the original set, if possible.

90
6. The joy of feedback

• Writing questions is difficult
• Writing questions alone is even more difficult
• Its unlikely that your first draft of any
  question should be submitted

91
(Clinical presentation was here)

• Which one of the following is the best
  explanation for the serum sodium concentration
  and the concentrated urine?

92
Options in first draft

• Arginine vasopressin release caused by hypotonic
  serum.
• Arginine vasopressin release caused by hypertonic
  urine.
• Arginine vasopressin release caused by low
  effective blood volume.
• Arginine vasopressin release caused by a
  combination of hypertonic urine and low effective
  blood volume

93
Your colleague recommends tidying the options

• The phrase Arginine vasopressin release caused
  by repeats in all the options
• Move it up to the question

94
The best explanation for the serum sodium
concentration and the concentrated urine is that
arginine vasopressin release was caused by

• hypotonic serum
• hypertonic urine
• low effective blood volume
• a combination of hypertonic urine and low
  effective blood volume

95
1st drafts can include too many details

• When its your patient, many of the details will
  come to mind when writing the test question
• Its easier for your colleague to weed out any
  information that isnt needed to answer the
  question

96
Sample first draft question

• Miss Smith, a previously healthy 22-year-old
  woman, is brought to the ER by her
  overly-protective mother. Miss Smith has a one
  day history of dysuria, urinary frequency,
  urgency, and supra-pubic tenderness. She noted
  cloudy urine, but denied fever. Her urinalysis
  showed many polymorphonuclear leukocytes and
  bacteria. Which of the following best
  characterizes her condition?

97
Edited question

• Miss Smith, a previously healthy 22-year-old
  woman, presents to the ER with a one day history
  of dysuria, urinary frequency, urgency, and
  supra-pubic tenderness. She noted cloudy urine,
  but denied fever. Her urinalysis showed many
  polymorphonuclear leukocytes and bacteria. Which
  of the following best characterizes her condition?

98
Questions?
99
Should we write some questions together?
100
What problems have you faced when working on
test assignments?
101
THANK YOU

• Lynn C. Webb, Ed.D.
• testing_at_lwebb.com