Writing MCQs

1
Writing MCQs

• Susan Chamberlain
• Dept of OBS/GYN
• Queens University

2
Writing MCQs

• At the end of this session, the participant will
  be able to
• Identify different types of MCQ items
• Describe the structure of an MCQ
• List the technical flaws common to MCQs
• write one well-structured MCQ testing appropriate
  content

3
Writing MCQs

• At the end of this session, the participant will
  be able to
• Identify different types of MCQ items
• Describe the structure of an MCQ
• List the technical flaws common to MCQs
• write one well-structured MCQ testing appropriate
  content
• All of the above

4
MCQ Check-list

• Why we write them
• What type is best
• What flaws are to be avoided
• What content to test
• How to construct it

5
Why do we test?

• Communicate to the students what material is
  important (formative vs summative)
• Motivate students to study
• Identify areas of deficiency in need of
  remediation or further learning
• Determine final grades or make promotion
  decisions
• Identify areas where the course/curriculum is weak

6
What Should Be Tested?

• Exam content should match course objectives
• Important topics should be weighted more heavily
  than less important topics
• The testing time devoted to each topic should
  reflect the relative importance of the topics
• The sample of items should be representative of
  the instructional goals

7
What Should Be Tested?

• Sampling
• Performance on the sample provides a basis for
  estimating achievement in the broader domain that
  is actually of interest.
• Questions sample
• TOPICS
• SKILLS ie determining diagnosis, next step in
  management, etc.

8
MCQ Items

• Should test important information either common
  or life-threatening
• Should be well-structured
• Flaws benefit test-wise examinees
• Irrelevant difficulty does not test examinees
  specific knowledge

9
Anatomy of an MCQ Item

• Stem
• Lead-in
• Options
• One correct answer
• Distractors

10
Types of MCQs

• True/False family
• Assess recall of isolated facts
• Stems must be clear and unambiguous
• Options must be absolutely true or false or
  examinee must decide how true options are
• Generally not recommended usually have subtle
  flaws

11
Types of MCQs True/False

• Which of the following statements is true
  regarding contraception?
• oral contraceptives should not be initiated until
  six weeks after therapeutic abortion to avoid the
  risk of hypercoagulability
• an intrauterine device can provide good
  contraceptive protection 5 years after insertion
• oral contraceptives should not be prescribed to
  individuals under the age of 15 without parental
  consent
• lesbian women do not require counseling about
  contraception or sexually transmitted diseases
• oral contraceptives should not be prescribed for
  women over the age of 35

12
Types of MCQs

• One-Best-Answer family (A type)
• Options may be diagrammed
• D C A E
  B
• Correct answer is the most likely correct
• Better application of knowledge, integration,
  synthesis and judgment

Least correct
Most correct
13
Types of MCQs One-Best-Answer family

• A 24 year old university student came to see
  you because she is missing several days of school
  each month because of severe dysmenorrhea
  unresponsive to the OCP and NSAIDS. She also
  complains of deep dyspareunia and has a tender
  uterosacral nodule on the pelvic examination.
  Which one of the following conditions is most
  likely responsible for her symptoms?
• Chronic pelvic inflammatory disease
• Endometriosis
• Adenomyosis
• Fibroids

14
MCQ Check-list

• Why we write them
• What type is best
• What flaws are to be avoided
• What content to test
• How to construct it

15
MCQ Technical Flaws

• Issues Related to Test-Wiseness
• Issues Related to Irrelevant Difficulty

16
MCQ Technical Flaws

• Issues Related to Test-Wiseness
• provide clues to the right answer
• examinee can eliminate options based on flaws and
  not on knowledge
• Issues Related to Irrelevant Difficulty

17
Item Flaws Test-wiseness

• Grammatical cues
• Logical cues
• Absolute terms
• Long correct answer
• Word repeats
• Convergence strategy

18
Item Flaws Test-wiseness

• Grammatical cues
• One or more distractors dont follow
  grammatically from the stem

19
Item Flaws Test-wiseness

• Grammatical cues
• A 24 year old patient presents with a one year
  history of dysmenorrhea, mid-cycle pain,
  dyspareunia, and painful bowel movements with her
  periods. The most likely diagnosis is
• Fibroid uterus
• Adenomyosis
• Endometriosis
• Psychosomatic condition

20
Item Flaws Test-wiseness

• Logical cues
• A subset of the options are collectively
  exhaustive

21
Item Flaws Test-wiseness

• Logical cues
• Arterial blood gases in a normal pregnant
  woman, compared to her non-pregnant state may
  show which of the following?
• Decreased pO2
• Increased pCO2
• Decreased pCO2
• Decreased pH
• No difference in pCO2

22
Item Flaws Test-wiseness

• Logical cues
• One or more distractors dont follow logically
  from the stem

23
Item Flaws Test-wiseness

• Logical cues
• An important tool for assessing fetal well-being
  is the Non Stress test. A well baby demonstrates
  the following fetal heart rate pattern
• A baseline heart rate above 120 bpm
• Evidence of fetal movement
• Baseline variability
• Two fetal heart rate accelerations in 20 minutes
• Fetal breathing movements

24
Item Flaws Test-wiseness

• Absolute terms
• Always, never
• Not a problem if stem is long and lead-in asks a
  precise question

25
Item Flaws Test-wiseness

• Absolute terms
• In patients with a previous Cesarean section
• Vaginal birth is NEVER an option
• Repeat Cesarean section is less dangerous to the
  mother
• The uterine scar may rupture during labour
• Care should ALWAYS be supervised by an
  obstetrician
• No further pregnancies should EVER be considered

26
Item Flaws Test-wiseness

• Long correct answer
• Correct answer is longer, more specific, or more
  complete than other options

27
Item Flaws Test-wiseness

• Long correct answer
• Which of the following problems in vaginal
  breech delivery is most likely to cause serious
  long-term complications?
• Failure to descend
• Cord prolapse
• Nuchal arms
• Entrapment of the after coming head with an
  incompletely dilated cervix
• 1 minute Apgar score of 4

28
Item Flaws Test-wiseness

• Word repeats
• A word or phase is included in the stem and in
  the correct answer

29
Item Flaws Test-wiseness

• Word repeats
• A 24 year old with a previous history of
  chronic PID presents with primary infertility.
  Select the procedure that would be most
  appropriate to evaluate the status of her tubes
  and pelvic anatomy
• Post-coital test
• Endometrial biopsy
• Laparoscopy with dye insufflation of the
  fallopian tubes
• Hysterosalpingogram

30
Item Flaws Test-wiseness

• Convergence strategy
• The correct answer includes the most elements in
  common with the other options

31
Item Flaws Test-wiseness

• Convergence strategy
• Normal bladder neck mobility is measured by
• A Q-tip angle measurement under 40 degrees
• A Q-tip angle measurement under 30 degrees
• A bladder neck ultrasound angle less than 30
  degrees
• A Q-tip angle measurement under 20 degrees

32
MCQ Technical Flaws

• Issues Related to Test-Wiseness
• Issues Related to Irrelevant Difficulty
• do not assess knowledge of intended topic

33
Irrelevant Difficulty

• Options are long, complicated or double
• Extraneous details in stem
• Numeric data not stated consistently
• Vague frequency terms
• Non-parallel language in options
• Stems are tricky or unnecessarily complicated
• None of the above

34
Irrelevant Difficulty

• Options are long, complicated or double
• Extraneous details in stem
• Shifts difficulty from content knowledge to speed
  reading

35
Irrelevant Difficulty

• Options are long, complicated or double
• Mrs. Brown is a 70 year old woman who
  had a stroke 6 months ago and is still impaired.
  Her daughter tells you that her mothers recent
  urinary incontinence makes her care unmanageable.
  She is seeking permanent placement. Mrs. Brown
  complains of urinary frequency, nocturia, and
  urge incontinence. Which of the following
  statements is true?
• Neurological conditions are involved in the
  majority of women with symptoms like those
  presented by Mrs. Brown
• Diabetes has a greater impact on quality of life
  than Mrs. Browns condition.
• The pharmacological blockage of muscarinic
  receptors in the detrusor muscle controls
  symptoms compatible with Mrs. Browns condition
• Overactive bladder is defined as urinary
  incontinence accompanied by or immediately
  preceded by urgency
• Alpha stimulants are effective in treating this
  condition

36
Irrelevant Difficulty

• Extraneous details in stem
• You see in your office a 24 year old
  woman for her routine postpartum visit, at 6
  weeks. Her baby is irritable and cries all the
  time (including during her visit.) As she is
  changing the diaper during her visit, you notice
  that the child has a very red bum but upon
  questioning, she tells you she did not notice
  anything abnormal. Her partner is away in
  Afghanistan and she does not have a lot of
  friends, as they just recently were posted in
  Kingston. She states that she has a hard time
  getting dressed in the morning, cries easily and
  feels like she does not love her child. She
  denies any medical or psychiatric history. Her
  exam is essentially normal. She weighs 2 kg less
  than at her first antenatal visit at 10 weeks.
  You most likely clinical diagnosis is
• Postpartum hypothyroidism
• Hyperthyroidism
• Postpartum blues
• Postpartum depression
• Psychotic depression

37
Irrelevant Difficulty

• Numeric Data
• Not stated consistently
• Mixed format
• Illogical order

38
Irrelevant Difficulty

• Numeric Data not stated consistently
• Following a second episode of salpingitis,
  what is the likelihood that a woman is infertile?
• Less than 20
• 20 30
• Greater than 50
• 90
• 75

39
Irrelevant Difficulty

• Vague frequency terms
• Often, usually, rarely, frequently
• Not consistently defined
• may be used in stem if otherwise carefully
  constructed unambiguous question

40
Irrelevant Difficulty

• Vague frequency terms
• Gonorrhea is
• A) sometimes asymptomatic
• B) treated with cefixime 450 mg orally
• C) caused by a gram positive intracellular
  diplococcus
• D) diagnosed by a DNA test
• E) usually associated with vulvar ulcers

41
Irrelevant Difficulty

• Non-parallel language in options
• Non-logical order

42
Irrelevant Difficulty

• Non-parallel language in options
• In a vaccine trial, 200 2-year-old boys were
  given a vaccine against a certain disease and
  then monitored for five years for occurrence of
  the disease. Of this group, 85 never contracted
  the disease. Which of the following statements
  concerning these results is correct?
• No conclusion can be drawn, since no follow-up
  was made of non-vaccinated children
• The number of cases (ie 30 cases over five year)
  is too small for statistically meaningful
  conclusions
• No conclusions can be drawn because the trial
  involved only boys
• Vaccine efficacy () is calculated as 85-15/100

43
Irrelevant Difficulty

• None of the above
• Problematic where judgment is involved or if
  options not absolutely true or false
• Turns question into true/false question
• Confuses very knowledgeable students

44
Irrelevant Difficulty

• None of the above
• The main reason that fertility is reduced
  in a woman in her early 40s is
• Irregular cycles make it hard to time intercourse
• Chromosomal damage adversely affects oocytes
  genetic complement
• Male erectile difficulties increase
• Hormone imbalance leads to poor endometrial
  development
• None of the above

45
Irrelevant Difficulty

• Stems are tricky or unnecessarily complicated
• Arrange the parents of the following
  children with Downs syndrome in order of the
  highest to lowest risk of recurrence. Assume
  that the maternal age in all cases is 22 years
  and that a subsequent pregnancy occurs within 5
  years. The karyotypes of the daughters are
• I 46, XX, 14,T (14q21q) pat
• II 46, XX, 14,T (14q21q) de novo
• III 46, XX, 14,T (14q21q) mat
• IV 46, XX, 21,T (14q21q) pat
• V 46, XX, 21,T (21q21q) (parents not typed)
• III, IV, I, V, II
• IV, III, V, 1, II
• III, I, IV, V, II
• I, V, III, I, V, II
• III, IV, I, II, V

46
MCQ Check-list

• Why we write them
• What type is best
• What flaws are to be avoided
• What content to test
• How to construct it

47
What to test?

• Items classified by cognitive processes required
  to answer the question
• Recall (Memory) tests knowledge of isolated
  facts
• Interpretation (Comprehension) review
  information to reach conclusion, eg diagnosis
• Problem solving (Reasoning) situation requires
  action, eg next step in management
• Process depends on background of trainee

48
What to test?

• Simpler classification based on task
• Application of knowledge item
• Reach conclusions
• Make predictions
• Select course of action
• Recall item
• Test knowledge of isolated facts

49
What to test?

• Test application of knowledge using clinical
  vignettes
• Adds face-validity
• Usually selects more important, less trivial
• Identifies student whove memorized but do not
  understand
• BUT experience in patient care should not be
  necessary

50
What to test - summary

• Focus item on key concepts and principles that
  are essential information (without access to
  references) for all examinees to understand
• Test material that is relevant to learning in
  clinical clerkships and beyond
• Avoid items that only require recall of isolated
  facts
• Avoid esoteric or interesting topics that are not
  essential

51
MCQ Check-list

• Why we write them
• What type is best
• What flaws are to be avoided
• What content to test
• How to construct it

52
Writing the Questions

• Construct stem include all necessary information
  to arrive at the right answer
• Choose lead-in pose a clear question
• Choose distractors

53
Stem

• Usually a clinical scenario
• Clear, unambiguous
• Should be long relative to options
• Includes all pertinent information
• Patients age/gender
• Clinical setting
• Complaint
• Other important info history, physical
  findings, test results

54
Lead-in

• Follows stem
• In the form of a question
• Should relate to the stem
• Should be answerable without looking at options
• Cover the Options test

55
Distractors

• Each should be selected by some, therefore all
  plausible, none obviously incorrect
• 3 or 4 choices
• Common misconceptions
• Faulty reasoning

56
Distractors

• Homogeneous in content to the correct answer
• Should be in same category as correct answer
  diagnosis, treatment, diagnostic test
• Clearly incorrect or inferior to the correct
  answer
• Plausible and attractive to the uninformed
• Similar to the correct answer in construction and
  length
• Grammatically consistent and logically compatible
  with the stem

57
General Guidelines for Item Construction

• Include as much of the item as possible in the
  stems Long stems, short options
• Avoid negatively phrased items (except or not in
  the lead-in). If you must, use only with short
  options
• Avoid writing question of the form
• Which of the following statements is correct?
• Each of the following statements is correct
  EXCEPT
• Unfocussed
• Heterogeneous options

58
Top-down versus Bottom-up

• Top-Down
• Knowledge or recall question
• Cite a disease and then ask what patient findings
  are expected
• Structured like textbooks
• Clinically backward

59
Top-down versus Bottom-up

• Bottom-up
• Application of knowledge
• Gives findings and asks examinee to indicate the
  disease
• Examinees need to be able to synthesize
  information from several pages of a textbook (or
  notes) to answer these questions
• Clinically realistic

60
Guidelines for Clinical Sciencesie Clerkship and
beyond

• Test application of knowledge using clinical
  vignettes to pose medical decisions in patient
  care situations
• Focus items on common or potentially catastrophic
  problems
• Pose clinical decisions that would be expected of
  a successful examinee
• Avoid clinical situations that would be handled
  by a (sub)specialist

61
MCQ Check-list

• Why we write them
• What type is best
• What flaws are to be avoided
• What content to test
• How to construct it

62
Basic Rules for MCQs (A type)

• Focus item on important concept
• Item should assess application of knowledge, not
  recall of an isolated fact
• Stem must pose clear question Cover the
  options test
• All distractors should be homogeneous
• Avoid technical item flaws

63
Reference

• Constructing Written Test Questions For the Basic
  and Clinical Sciences
• Case Swanson, NBME 1996