David DiBattista, Ph.D. Brock University Psychology Department

1
David DiBattista, Ph.D.Brock UniversityPsycholog
y Department
Getting the Most Out of Multiple-choice Questions
2
Overview

• About multiple-choice testing
• Two challenges in MC testing
• Addressing the challenges

3
About MC Testing

• Most common selected-response format
• Permits broad coverage of material
• Practical even with large classes

4
(No Transcript)
5
About MC Testing

• Most common selected-response format
• Permits broad coverage of material
• Practical even with large classes

6
Some Important Terminology

• The stem is a question or incomplete statement
  that is followed by several alternatives.
• One alternative is the correct answer (or key)
  and the others are distracters.
• The correct-answer format is typical, with the
  best-answer format less common.

7
Overview

• About multiple-choice testing
• Two challenges in MC testing
• Addressing the challenges

8
Two Challenges in MC Testing
Challenge 1 Constructing high-quality MC items
takes time, effort, and skill. Challenge 2 MC
items are too often used to assess what students
remember, with higher-level cognitive skills
getting little attention.
9
Overview

• About multiple-choice testing
• Two challenges in MC testing
• Addressing the challenges
• 1. Constructing high-quality items
• 2. Assessing higher-level thinking

10
Tips for MC Item Construction

• Use question format whenever possible and
  sentence-completion format only when necessary.
  In either case, use appropriate punctuation and
  capitalization.

11
When hungry rats are given small bits of food at
fixed intervals, excessive water consumption
occurs. What is this excessive consumption
called? A. primary drinking B. osmometric
drinking C. psychogenic polydipsia D.
schedule-induced polydipsia
12
When hungry rats are given small bits of food at
fixed intervals, excessive water consumption
occurs. This excessive consumption is known as A.
primary drinking. B. osmometric drinking. C.
psychogenic polydipsia. D. schedule-induced
polydipsia.
13
Tips for MC Item Construction

• If you use completion format, always place the
  blank at the very end of the stem.

14
Poor Generally speaking, ___________ testing
supports content sampling better than do other
testing formats. A. essay B. performance C.
multiple-choice D. paper-and-pencil
15
Improved Generally speaking, content sampling is
best supported by A. essay testing. B.
performance testing. C. multiple-choice
testing. D. paper-and-pencil testing.
16
Better still Generally speaking, which type of
testing best supports content sampling? A. essay
testing B. performance testing C.
multiple-choice testing D. paper-and-pencil
testing
17
Tips for MC Item Construction

• The stem should present the issue under
  consideration clearly and contain as much
  information as possible.
• Do not include irrelevant information in the stem
  unless it plays a role in the assessment
  procedure.

18
Poor South America A. imports coffee from
Australia. B. is where the Gobi Desert is found.
C. was colonized mostly by people from
Spain. D. has a larger population than the
United States of America.
19
Even worse South America, which has an area of
more than 17 million square kilometres, A.
imports coffee from Australia. B. is where the
Gobi Desert is found. C. was colonized mostly by
people from Spain. D. has a larger
population than the United States of America.
20
Better Most of South America was colonized by
people from what country? A. Spain B. France C.
Holland D. England
21
A distribution of 100 test scores is positively
skewed, with a mean of 70 and a standard
deviation of 10. Johns test score is 80. What is
his standard score? A. -1.0 B. 1.0 C. -10.0
D. 10.0
22
A distribution of 100 test scores is positively
skewed, with a mean of 70 and a standard
deviation of 10. Johns test score is 80. What is
his standard score? A. -1.0 B. 1.0 C. -10.0
D. 10.0
23
Tips for MC Item Construction

• Check carefully for spelling errors, giving
  special attention to distracters.
• If you use sentence-completion format, check
  carefully for grammatical consistency of stem and
  alternatives.

24
Poor Stamp collectors use stamp hinges to A. pick
up stamps. B. fold learge stamps in half. C.
mount stamps in albums. D. joining two stamps
together.
25
Improved Stamp collectors use stamp hinges to A.
pick up stamps. B. fold large stamps in half. C.
mount stamps in albums. D. join two stamps
together.
26
Better still What do stamp collectors use stamp
hinges for? A. picking up stamps B. folding large
stamps in half C. mounting stamps in albums D.
joining two stamps together
27
Tips for MC Item Construction

• Whenever possible, avoid negative wording in the
  stem, and be sure to emphasize it when it does
  occur.

28
Poor Which term is not usually associated with
Sigmund Freud? A. id B. extinction C. repression
D. latent content
29
Better Which term is NOT usually associated with
Sigmund Freud? A. id B. extinction C. repression
D. latent content
30
Also better Which term is usually associated with
behaviourism? A. id B. extinction C. repression
D. latent content
31
Tips for MC Item Construction

• All distracters should be plausible.
• Four alternatives will usually be quite adequate,
  but the number used is best determined by the
  number of plausible distracters you can supply.

32
Which river flows through the city of Edmonton?
A. North Saskatchewan River B. Athabasca
River C. Peace River D. Milk River
33
Which river flows through the city of Edmonton?
A. North Saskatchewan River B. Mississippi
River C. Amazon River D. Rhine River
E. Seine River F. Nile River
34
Tips for MC Item Construction

• To generate plausible distracters

• Use students most common errors on
  constructed-response tests.
• Use distracters that are similar to the correct
  answer in content, length, and complexity.
• Use words that sound important or have
  associations to the stem.
• Use distracters that are true, but do not
  correctly answer the question.

35
Name the river that flows through the city of
Edmonton. Athabasca River
What do stamp collectors use stamp hinges for? To
join stamps together
36
Tips for MC Item Construction

• To generate plausible distracters

• Use students most common errors on
  constructed-response tests.
• Use distracters that are similar to the correct
  answer in content, length, and complexity.
• Use words that sound important or have
  associations to the stem.
• Use distracters that are true, but do not
  correctly answer the question.

37
In severe cases of obesity, there may be a
substantial increase in the number of adipocytes.
What term is used to refer to this increase? A.
hyperbole B. hyperplasia C. hypertrophy D.
hypertonicity
38
Tips for MC Item Construction

• To generate plausible distracters

• Use students most common errors on
  constructed-response tests.
• Use distracters that are similar to the correct
  answer in content, length, and complexity.
• Use words that sound important or have
  associations to the stem.
• Use distracters that are true, but do not
  correctly answer the question.

39
In responding to a lengthy survey, a man answers
yes to every yes-no question asked. It is
reasonable to suspect that his responses may be
influenced by which of the following? A. response
acquiescence B. opportunistic characterization C.
the partial reinforcement effect D. the
conspicuous agreement predisposition
40
Tips for MC Item Construction

• To generate plausible distracters

• Use students most common errors on
  constructed-response tests.
• Use distracters that are similar to the correct
  answer in content, length, and complexity.
• Use words that sound important or have
  associations to the stem.
• Use distracters that are true, but do not
  correctly answer the question.

41

• Which of the following events caused the Prime
  Minister of Canada to proclaim the War Measures
  Act?
• Newfoundland was occupied by German
• forces in 1940.
• B. The October Crisis occurred in 1970.
• C. The Quebec Referendum was held in
• 1980.
• D. The Meech Lake Accord was defeated in
• 1990.

42
Tips for MC Item Construction

• Avoid patterns in the length and location of
  correct answers that could provide clues that are
  unrelated to content.
• Balance the answer key so that the correct
  response appears in each position about the same
  number of times.

43

• Your baseball team has a runner on first base.
  Under what circumstances might it be a good idea
  for the batter to lay down a sacrifice bunt?
• when it is a night game
• B. when the pitcher is a relief pitcher
• C. when the batter is a right-handed hitter
• D. when it is late in the game, the score is
• tied, there are no outs, and the batter is a
• weak hitter with less than two strikes

44
In a four-alternative multiple-choice test, about
how often should the correct answer appear in
each of the four locations? A. 10 of the time B.
25 of the time C. 40 of the time D. 60 of the
time
45
Who invented the binaural recording system
commonly known as stereo? A. Xxxxxxxxxxxxxx B.
Xxxxxxxxxxxxxx C. Xxxxxxxxxxxxxx D. Xxxxxxxxxxxxxx
46
Who invented the binaural recording system
commonly known as stereo? A. Alan Dower
Blumlein B. Alan Dower Blumlein C. Alan Dower
Blumlein D. Alan Dower Blumlein
47
Tips for MC Item Construction

• Do not use none of the above.

48
None of the above as a distracter
Poor Which of these 19th century authors wrote
Wuthering Heights? A. Emily Bronte B. George
Eliot C. Charles Dickens D. None of the above
49
None of the above as a distracter
Better Which of these 19th century authors wrote
Wuthering Heights? A. Emily Bronte B. George
Eliot C. Charles Dickens D. William Thackeray
50
None of the above as the key
Poor Which of these 19th century authors wrote
Middlemarch? A. Jane Austen B. Anne Bronte C.
Willkie Collins D. None of the above
51
None of the above as the key
Which of these 19th century authors wrote
Middlemarch? A. Jane Austen B. Anne Bronte C.
Willkie Collins D. Charles Dickens
Hey! Wait a minute! Dickens didnt write
Middlemarch!
52
None of the above as the key
Better Which of these 19th century authors wrote
Middlemarch? A. Jane Austen B. Anne Bronte C.
Willkie Collins D. George Eliot
53
Tips for MC Item Construction

• Do not use all of the above unless there are
  only two distracters.

54
Poor Which of these terms is associated with
Sigmund Freud? A. superego B. repression C.
latent content D. all of the above
55
Poor Which of these terms is associated with
Sigmund Freud? A. superego ? B. repression C.
latent content D. all of the above
56
Poor Which of these terms is associated with
Sigmund Freud? A. superego ? B. repression ? C.
latent content D. all of the above
57
Poor Which of these terms is associated with
Sigmund Freud? A. superego ? B. repression ? C.
latent content ??????? D. all of the above
58
Better Which of these terms is associated with
Sigmund Freud? A. superego B. bulimia nervosa C.
cognitive dissonance D. fixed-interval schedule
59
Also better Which of these terms is associated
with Sigmund Freud? A. superego B. latent content
C. both of the above
60
Tips for MC Item Construction

• Do not use structurally complex items (e.g.,
  Type K), which may call heavily on test-taking
  skills.

61

• Poor
• Which of the following may threaten the internal
  validity of an experiment?
• 1. Instrumentation changes
• 2. Homogeneity of variance
• 3. Differential participant mortality
• 4. Non-random participant selection
• A. 1 and 3 only
• B. 2 and 4 only
• C. 1, 2, and 3
• D. 4 only
• 1, 2, 3, and 4

62

• Poor
• Which of the following may threaten the internal
  validity of an experiment?
• 1. Instrumentation changes Threat
• 2. Homogeneity of variance No threat
• 3. Differential participant mortality ???
• 4. Non-random participant selection ???
• A. 1 and 3 only
• B. 2 and 4 only
• C. 1, 2, and 3
• D. 4 only
• 1, 2, 3, and 4

63

• Poor
• Which of the following may threaten the internal
  validity of an experiment?
• 1. Instrumentation changes Threat
• 2. Homogeneity of variance No threat
• 3. Differential participant mortality ???
• 4. Non-random participant selection ???
• A. 1 and 3 only
• B. 2 and 4 only X (from knowing 2)
• C. 1, 2, and 3 X (from knowing 2)
• D. 4 only
• 1, 2, 3, and 4 X (from knowing 2)

64

• Poor
• Which of the following may threaten the internal
  validity of an experiment?
• 1. Instrumentation changes Threat
• 2. Homogeneity of variance No threat
• 3. Differential participant mortality ???
• 4. Non-random participant selection ???
• A. 1 and 3 only
• B. 2 and 4 only X (from knowing 2)
• C. 1, 2, and 3 X (from knowing 2)
• D. 4 only X (from knowing 1)
• 1, 2, 3, and 4 X (from knowing 2)

65
Tips for MC Item Construction

• Avoid trivia. Test only important information,
  concepts, and abilities.

66
Peace talks were held in Paris after World War I.
How long did these talks last? A. 3 months B. 6
months C. 9 months D. 12 months
67
Which of the following countries DID NOT join the
League of Nations in the period following World
War I? A. United States B. Germany C. France D.
Japan
68
Tips for MC Item Construction
The last suggestion

• Ignore any of the preceding suggestions when you
  have a good reason to do so.

69
It's time to practice!
70
Overview

• About multiple-choice testing
• Two challenges in MC testing
• Addressing the challenges
• 1. Constructing high-quality items
• 2. Assessing higher-level thinking

71
Two Challenges in MC Testing
Challenge 1 Constructing high-quality MC items
takes time, effort, and skill. Challenge 2 MC
items are too often used to assess what students
remember, with higher-level cognitive skills
getting little attention.
72
The Original Blooms Taxonomy
Evaluation Synthesis Analysis Application Comprehe
nsion Knowledge
73
The Revised Blooms Taxonomy
Factual Knowledge Dimension Conceptual Procedural Knowledge Dimension Conceptual Procedural Metacognitive
Cognitive Process Dimension Remember
Cognitive Process Dimension Understand
Cognitive Process Dimension Apply
Cognitive Process Dimension Analyze
Cognitive Process Dimension Evaluate
Cognitive Process Dimension Create
74
The Revised Blooms Taxonomy
Factual Knowledge Dimension Conceptual Procedural Knowledge Dimension Conceptual Procedural Metacognitive
Cognitive Process Dimension Remember 47 14 --- ---
Cognitive Process Dimension Understand
Cognitive Process Dimension Apply
Cognitive Process Dimension Analyze
Cognitive Process Dimension Evaluate
Cognitive Process Dimension Create
75
The Revised Blooms Taxonomy
Factual Knowledge Dimension Conceptual Procedural Knowledge Dimension Conceptual Procedural Metacognitive
Cognitive Process Dimension Remember 47 14 --- ---
Cognitive Process Dimension Understand 6 22 --- ---
Cognitive Process Dimension Apply --- 3 3 ---
Cognitive Process Dimension Analyze --- --- --- ---
Cognitive Process Dimension Evaluate --- --- --- ---
Cognitive Process Dimension Create --- --- --- ---
76
Assessing Higher-level Thinking
Although MC tests too often focus on what
students can remember, this can usually be
avoided by

• using challenging individual MC items
• ? Ramp it up! (Blooms Revised Taxonomy)
• ? Item shells
• using interpretive exercises

77
According to the Empirical Rule, about how many
scores will typically lie within one standard
deviation on either side of the mean of a
mound-like distribution? A. one-tenth B.
one-quarter C. one-half D. two-thirds
78
The mean of a mound-like distribution of scores
is 65 and the standard deviation is 12. You can
be pretty sure that about two-thirds of the
scores lie between what two values? A. 59 and
71 B. 53 and 77 C. 47 and 83 D. 41 and 89
79
The mean of a mound-like distribution of scores
is 65 and the standard deviation is 12. You can
be pretty sure that about two-thirds of the
scores lie between what two values? A. 59 and
71 B. 53 and 77 C. 47 and 83 D. 41 and 89
80
The mean of a mound-like distribution of scores
is 65 and the variance is 144. You can be pretty
sure that about two-thirds of the scores lie
between what two values? A. 59 and 71 B. 53 and
77 C. 47 and 83 D. 41 and 89
81
What characteristic of a distribution does the
standard deviation measure? A. dispersion B.
skewness C. normality D. kurtosis
82
Median is to variance as central tendency is
to A. kurtosis. B. normality. C. skewness. D.
dispersion.
83
In classical conditioning, what name is give to a
stimulus that elicits a particular response even
in the absence of any prior training? A.
conditioned stimulus B. unconditioned stimulus C.
activational stimulus D. discriminative stimulus
84
Right after a rat smells menthol, it is always
given Drug X, which causes osmometric drinking.
Eventually, the rat drinks whenever it smells
menthol, even when it is not injected with Drug
X. In this situation, what is the role of Drug
X? A. conditioned stimulus B. unconditioned
stimulus C. activational stimulus D.
discriminative stimulus
85

• Which name is most closely associated with
  classical psychoanalytic therapy?
• B.F. Skinner
• Abraham Maslow
• Leon Festinger
• D. Sigmund Freud

86

• What must be done to the following sentence to
  make it accurate?
• Classical psychoanalytic therapy, which was
  developed by Freud and involves extensive use of
  free association, is more widely used today than
  it was 60 years ago.
• Nothing needs to be done.
• Change the word Freud to Skinner.
• Change the words free association to
• behaviour modification techniques.
• D. Change the word more to less.

87
Claim Captain Queeq was a coward. Which of the
following most strongly supports this claim? A.
his bullying of Stillwell B. his behaviour at
Saipan C. the cutting of the towline D. the
affair of the strawberries
88
Your friend Anne reads that census data indicate
that people are having fewer children nowadays
than they did 40 years ago. She tells you that
she does not believe this because the young
couple who live next door to her are both under
30 and already have four children. If Keith
Stanovich were told about this, what might you
reasonably expect him to say?
89
A. The census data must be wrong. B. Annes
comment illustrates valid probabilistic
reasoning. C. Annes comment illustrates the use
of person-who statistics. D. The young
couple provide an exception that actually
serves to prove the rule.
90
To learn about the attitudes of Canadian voters
toward a government bill limiting auto emissions,
Gary gave a questionnaire to 200 members of a
group called Friends of a Cleaner Environment. He
found that every single one of them strongly
supported the bill. Based on this research, which
of these is the most appropriate conclusion?
91
A. The questionnaire is very reliable. B. The
questionnaire items are not clearly written. C.
The majority of Canadian voters strongly support
the bill. D. The results tell us very little
about the attitudes of Canadian voters.
92
Assessing Higher-level Thinking
Although MC tests too often focus on what
students can remember, this can usually be
avoided by

• using challenging individual MC items
• ? Ramp it up! (Blooms Revised Taxonomy)
• ? Item shells
• using interpretive exercises

93
Item Shells
Haladyna and Shindoll (1989) describe an item
shell as a hollow MC item that has a syntactic
structure, but no content. The test writer can
insert important concepts into the item shell to
construct challenging MC items.
94

• Which of these is an example of values found in
  the Fibonacci series?
• 34, 55, 89, 144
• 49, 57, 65, 73
• B. 64, 128, 256, 512
• 81, 100, 121, 144

95

• Which of these is an example of values found in
  the Fibonacci series?
• 34, 55, 89, 144
• 49, 57, 65, 73
• B. 64, 128, 256, 512
• 81, 100, 121, 144

96
Which is a difference between r and ?? A. r is a
parameter, and ? is a statistic. B. ? is a
parameter, and r is a statistic. C. r is a
continuous variable, and ? is a discrete
variable. D. ? is a continuous variable, and r is
a discrete variable.
97
Which is a difference between r and ?? A. r is a
parameter, and ? is a statistic. B. ? is a
parameter, and r is a statistic. C. r is a
continuous variable, and ? is a discrete
variable. D. ? is a continuous variable, and r is
a discrete variable.
98

• Which of the following is a major shortcoming of
  Freuds theory of personality?
• Many aspects of the theory cannot be
• falsified.
• B. The theory overemphasizes the concept of
• self-actualization.
• C. Freud ignores sexuality as a cause of
• neurotic behaviour.
• D. Freud did not present his ideas in sufficient
• detail to permit evaluation of the theory.

99

• Which of the following is a major shortcoming of
  Freuds theory of personality?
• Many aspects of the theory cannot be
• falsified.
• B. The theory overemphasizes the concept of
• self-actualization.
• C. Freud ignores sexuality as a cause of
• neurotic behaviour.
• D. Freud did not present his ideas in sufficient
• detail to permit evaluation of the theory.

100
Nina wants to reduce the probability of making a
Type I error in her research study. Which of
these strategies could she use? A. Increase the
sample size. B. Decrease the value of alpha. C.
Use a one-tailed rather than a two-tailed
test. D. Use a repeated-measures design rather
than an independent-groups design.
101
Nina wants to reduce the probability of making a
Type I error in her research study. Which of
these strategies could she use? A. Increase the
sample size. B. Decrease the value of alpha. C.
Use a one-tailed rather than a two-tailed
test. D. Use a repeated-measures design rather
than an independent-groups design.
102
Assessing Higher-level Thinking
Although MC tests too often focus on what
students can remember, this can usually be
avoided by

• using challenging individual MC items
• ? Ramp it up! (Blooms Revised Taxonomy)
• ? Item shells
• using interpretive exercises

103

• In an interpretive exercise, a number of MC items
  follow the presentation of novel introductory
  material, such as a reading, scenario, data set,
  chart, or map. The MC items can be answered
  correctly only by referring to the introductory
  information.

104
Tips for Interpretive Exercises

1. Ensure that the introductory material is relevant
   to the course objectives and appropriate to
   students curricular experience.
2. Ensure that the introductory material is novel,
   and is as brief and clearly written as possible.

105
Tips for Interpretive Exercises

1. Construct MC items that require analysis and
   interpretation of the introductory material, not
   simple fact-finding.
2. Let the number of MC items be roughly
   proportional to the length of introductory
   material.

106
Tips for Interpretive Exercises

1. Interpretive exercises can be used not only with
   MC items, but also with other response formats,
   including constructed-response formats.
2. To reuse an interpretive exercise, create new
   introductory material, and make whatever minimal
   changes are necessary to the individual MC items.

107
Overview

• About multiple-choice testing
• Two challenges in MC testing
• Addressing the challenges
• 1. Constructing high-quality items
• 2. Assessing higher-level thinking

108
David DiBattista, Ph.D.Brock UniversityPsycholog
y Department
Getting the Most Out of Multiple-choice Questions