Constructing Written Test Questions for Basic and Clinical Sciences Aaron McGuffin, M.D. Senior Associate Dean for Medical Education

1
Constructing Written Test Questions for Basic and
Clinical SciencesAaron McGuffin, M.D.Senior
Associate Dean for Medical Education

• Adapted from NBME 3rd Edition Manual 2002
• Susan M. Case, PhD and David B. Swanson, PhD

http//www.nbme.org/publications/item-writing-manu
al-download.html
2
Purpose of Testing

• Communicate to students what is important
  information for them to know as practicing
  physicians
• Provide an outcome measure to verify students
  have achieved course objectives
• Identify areas of deficiency in students for
  purposes of remediation or to further learning
• Identify areas of deficiency in curriculum
• Teach students to interpret data and critically
  think to make decisions
• Motivate students to study

3
Multiple Choice Item Formats

• In order for a test question to be a good one, it
  must satisfy two basic criteria
• The test question must address important content
• Test question must be well structured and avoid
  flaws that benefit the test-wise examinee

4
What should be tested?

• Exam content should match course objectives
• Important topics should be weighted more heavily
  (more questions) than less important topics
• Congestive heart failure versus sarcoidosis
• Need to sample topics and sample skills
  (determining the diagnosis, deciding on the next
  best step in management)
• A test is a biopsy. You cannot ask everything!
  (at least not on the examination)

5
Where do I start?

• Start by writing an objective for the question
• Ask yourself
• What is it that I am wanting to see that the
  students know?
• Is it important that they know that?
• Does my question objective line up with the
  institutional objectives, course objectives, and
  my lecture objectives?
• Focus on important concepts dont waste time
  testing trivial facts
• Review on-line question bank

6
Single Best Answer in Clinical Vignette Format

• Stem A 32-year-old man has a four day history of
  progressive weakness in his extremities. He has
  been healthy except for an upper respiratory
  tract infection 10 days ago. His temperature is
  37.80 C (1000 F), blood pressure is 130/80 mm Hg,
  pulse is 94/min, and respirations are 42/min and
  shallow. He has symmetric weakness on both sides
  of the face and the proximal and distal muscles
  of the extremities. Sensation is intact. No deep
  tendon reflexes can be elicited and the plantar
  responses are flexor.
• Lead in Which of the following is the most
  likely diagnosis?
• Options
• A. Acute disseminated encephalomyelitis
• B. Guillain-Barre syndrome
• C. Myasthenia gravis
• D. Poliomyelitis
• E. Polymyositis
• Correct Answer B

7

• Stem A 32-year-old man
• Starts with a patient
• has a four day history
• Gives a time frame
• of progressive weakness in his extremities.
• Provides a chief complaint
• He has been healthy except for an upper
  respiratory tract infection 10 days ago.
• Provides some past medical history
• His temperature is 37.80 C (1000 F), blood
  pressure is 130/80 mm Hg, pulse is 94/min, and
  respirations are 42/min and shallow. He has
  symmetric weakness on both sides of the face and
  the proximal and distal muscles of the
  extremities. Sensation is intact. No deep tendon
  reflexes can be elicited and the plantar
  responses are flexor.
• Provides some physical exam

8

• Lead in Which of the following is the most
  likely diagnosis?
• Should have a question with this stem for every
  disease covered, or make the diagnosis be part of
  a two step question, i.e. the student must figure
  out its Guillain-Barre syndrome, and then answer
  a question about the change in respiratory
  physiology that accompanies it.
• Options
• A. Acute disseminated encephalomyelitis
• B. Guillain-Barre syndrome
• C. Myasthenia gravis
• D. Poliomyelitis
• E. Polymyositis
• Answers are capitalized, in alphabetical order,
  close to the same length, all reasonable
  answers, i.e. answer E was not Tourettes
  syndrome

9
General Guidelines for Item Construction

• COVER TEST Make sure the item can be answered
  without looking at the options
• Include as much of the item as possible in the
  stem. Stems should be long and options short
• Avoid superfluous information on Step 1
  questions. Step 2 and 3 level questions should
  have some additional information to teach
  students to filter.
• Avoid tricky or overly complex items

10
General Guidelines for Item Construction

• Write options that are grammatically consistent
  and logically compatible with the stem
• Write plausible distracters
• Never use absolutes always, never, all
• Avoid negatively phrased items, excepts, all of
  the following are TRUE
• Which of the following is not
• Which of the following is the least likely
• All the following are true EXCEPT
• Which of the following is true.

11
Lead-in examples

• Which of the following is the most likely
  diagnosis?
• Which of the following is the next best step in
  management?
• Which of the following enzymes does this drug
  inhibit?
• Which of the following nerves has most likely
  been damaged?

12
Two items written to assess same topic

• Question 1

• Question 2 (Preferred)

• Acute intermittent porphyria is the result of a
  defect in the biosynthetic pathway for
• A. Collagen
• B. Corticosteroid
• C. Fatty acid
• D. Glucose
• E. Heme
• F. Thyroxine

• A 33-year-old man presents with mild weakness and
  intermittent episodes of steady, severe crampy
  abdominal pain. He denies vomiting or diarrhea.
  One aunt and a cousin have had similar episodes.
  During an episode, his abdomen is distended, and
  bowel sounds are decreased. Neurologic
  examination shows mild weakness in the upper
  arms. These findings suggest a defect in the
  biosynthetic pathway for which of the following?
• A. Collagen
• B. Corticosteroid
• C. Fatty acid
• D. Glucose
• E. Heme
• F. Thyroxine

13
How do I write questions about clinical entities
students are not familiar with?

• 1. Use diseases they know
• Common cold, strep throat, pneumonia,
  appendicitis, gastroenteritis, diarrhea
• 2. If it involves a disease they do not know,
  describe it in the vignette and tell them the
  disease. By reading the vignette they are still
  learning.
• 3. Provide a short snapshot of the disease in the
  lecture material, even as read on your own
  material which begins to enhance their life long
  learning skills

14
How do I write questions about clinical entities
I am not familiar with?

• 1. Read about them
• www.uptodate.com
• 2. Engage clinical faculty in actual course or a
  discussion about the topic
• In general we all love to talk about what we do
• Let them review the questions for clinical
  accuracy
• 3. Attend Grand Rounds presentations on topics
  you teach
• http//musom.marshall.edu/cme/cme-calendar.htm

15
Review of Submitted Questions
16
Immunology

• A 9-year-old boy is admitted to the hospital for
  an evaluation of a suspected underlying immune
  deficiency because of a life-threatening
  infection.  Careful history reveals that two
  years ago he underwent an emergency operation in
  which his spleen was removed following its
  rupture in a motor vehicle accident.  Which of
  the following is a major characteristic
  associated with splenectomy?
•  
• A.            Low IgM levels with increased
  mucosal IgA
• B.            Increased problems with blood-born
  pathogens
• C.            Progressive deterioration of the
  immune system
• D.            Transient depression of innate
  immunity
• Answer B

17
Microbiology

• An 18-year-old college freshman presents to the
  emergency department with fever, rash and severe
  headache. A lumbar puncture is performed and
  shows numerous white blood cells and
  intracellular gram-negative diplococci.
  Antibiotic therapy is started. Which of the
  following organisms is most likely causing the
  patients symptoms?
•  
• A. Neisseria gonorrhoeae
• B. Neisseria meningitidis
• C. Haemophilus influenzae type b
• D. Mycoplasma pneumoniae
• E. Streptococcus pneumoniae
•  

18
Pathology

• A 17-year-old woman presents to the emergency
  department with a 12 hour history of nausea,
  vomiting, right lower quadrant abdominal pain and
  an elevated white blood cell count. She is taken
  to surgery and her appendix is removed.
  Histologic examination of the appendix revealed
  each of the following features. Which of the
  following best indicates tissue necrosis?
• A. Dilation of blood vessels
• B. Interstitial edema
• C. Loss of cell nuclei
• D. Presence of neutrophils
• E. Swelling of cell cytoplasm

19
Physiology

• Which of the following would still be secreted in
  a patient who had his stomach removed as a
  treatment for gastric cancer?
• A. Insoluble mucus
• B. Intrinsic factor
• C. Pepsin
• D. Secretin
• E. Trypsin

20

• A 64-year-old man presents to his primary care
  doctor complaining of three months of nausea,
  early satiety and persistent, dull epigastric
  abdominal pain. He has also noticed a 10 kg
  (22lb) weight loss. His primary care doctor
  refers him to a gastroenterologist who performs
  an esophagogastroduodenoscopy and finds a gastric
  mass. Biopsies of the mass reveal gastric
  carcinoma. A total gastrectomy is performed to
  remove the mass and chemotherapy is begun. The
  patient is lost to follow-up but returns to your
  office approximately 15 months later with
  numbness and weakness of his lower extremities.
• 34. Deficiency of which of the following
  substances explains this patients neurological
  symptoms?
•  
• A. Hydrochloric acid
• B. Intrinsic factor
• C. Pepsin
• D. Secretin
• E. Trypsin
•  
• 35. Which of the following substances is still
  secreted in a patient who has undergone a
  complete gastrectomy?
•  
• A. Hydrochloric acid
• B. Intrinsic factor
• C. Pepsin
• D. Secretin
• E. Trypsin