The Design and Use of Multiple Choice Tests

1
"The Design and Use of Multiple Choice Tests"
2
Multiple Choice Exams

• Written properly, multiple choice exams
  correlate strongly with assessments by
  descriptive tests
• Brown, Robert, Multiple Choice Versus
  Descriptive Tests, Frontiers in Education
  Conference, Reno NC 2001.

3
Use of Multiple Choice Exams

• Not limited to Factual Knowledge
• Making inferences
• Application of models
• Evaluation

4
Multiple Choice vs. Descriptive Examinations
5
Structure of a Multiple Choice Question

• Stem
• Answers

6
Formulating GoodMultiple Choice Questions

• Wording the stem so that there is a single
  correct answer
• Finding plausible wrong answers
• Sensitivity to ESL students
• Negative voice in the stem
• Conditional if would structures
• Cultural contexts for questions

7
Stems

• Statements
• A river that flows through a desert is
  called..
• Questions
• Which of the following descriptions best fits
• the character of Alsatian wines?
• Negative voice
• Which of the following grapes is not used to
• make Amarone?

8
Stems

• Conditional Voice
• If you want a VQA Niagara red wine that
• was very fruity and required only a short
• ageing period you would select

9
Answer Options

• Single correct answer options
• Collective answer options
• (a) and (c) are both correct
• all of the above are correct
• none of the above are correct

10
Plausible Answer Strategies

• Good questions contain several plausible answer
  options
• Grey area between designing tricky questions
  and providing answer options that test knowledge
  thoroughly.
• Inadvertently giving away the answers

11
Giving away the answer

• A verbal question
• The forbidden fruit that Eve offered Adam was
  an
• (a) apple (b) pear
• (c) banana (d) melon
• (e) carrot

12
Giving away the answer

• i.e. a numerical question
• It costs a vendor 1.00 each time s/he sells a
  hot dog to a customer. If the price of the hot
  dog is 3.00, what is the contribution margin
  percentage?
• (a) 17.5 (b) 22.5
• (c) 47.5 (d) 66.7
• (e) 82.0

13
Plausible Answer Strategy

• It costs a vendor 1.00 each time s/he sells a
  hot dog to a customer. If the price of the hot
  dog is 3.00, what is the contribution margin
  percentage?
• (a) 300 (b) 200
• (c) 67.7 (d) 50
• (e) 33.3

14
Plausible Answer Strategies

• Use Nonsense Words
• Colio Wines in Ontario produce an interesting
  product classified as Vin de Curé in much the
  same way as Amarone is made in Veneto. The
  Italian word describing the drying process used
  to desiccate the grapes is
• a) Ripasso b) Appasimento
• c) Raisinato d) Pommace
• e) Prunello

15
Plausible Answer Strategies

• Use related, but different terms.
• The word that refers to the addition of sugar to
  a finished wine is
• (a) Amelioration
• (b) Chaptalization
• (c) Edulcoration
• (d) Elaboration
• (e) Oxidation

16
Adding Uncertainty..

• The term "fortified wine" means
• a) the bottle had been reinforced
• b) the wine has had tannins and acid
• added
• c) the wine has had sugar added
• d) the wine has had SO2 added
• e) none of the foregoing

17
Adding difficulty..

• Choose the best answer.
• Roothams, a well-known local manufacturer of
  brandied fruit preserves sells their product
  through gourmet food shops and at craft fairs
  throughout Canada. Their recipes use genuine
  brandy and all natural ingredients. They are
  able to sell these legally because
• (a) the alcohol used is produced by fermentation
• (b) the product is made with ethyl alcohol
• (c) the alcohol used is purchased from the LCBO
• (d) the alcohol concentration is less than 1.2
• (e) the label says made with genuine brandy

18
Making Multiple Choice Questions Interesting

• A gracious, but parsimonious hostess wants to
  receive guests in the private dining room of your
  restaurant with a glass of champagne just to
  tantalise the taste buds and set the mood for the
  evening.
• 1.You suggest that if the wine is being poured at
  the bar, she should consider serving a Canadian
  sparkling wine that is very similar to Champagne
  in quality but considerably less expensive. Such
  a wine would be a

19

• 2. The cheapest sparkling wine would be obvious
  to sophisticated guests because they would know
  that in the best sparkling wines
• 3. If the gracious hostess insists on having
  French Champagne and you are left to
  rationalise costs, the most cost effective
  solution is to serve Champagne from..

20

• 4. Occasionally, a patron asks you to do
  something that will distinguish their event from
  others. One client wants to obtain a large
  bottle of Champagne to celebrate a particular
  event. Which is the largest bottle of champagne
  on the list below?
• 5. If someone wanted to serve a "glass of bubbly"
  during a reception, which of the following would
  be the most inappropriate type of wine to select
  as the pre-prandial (before dinner) libation?

21
Evaluating Multiple Choice Questions

• Difficulty Factor R/N
• R of times the correct answer is
• selected
• N of people taking the exam

Too Easy
Too Difficult
0.5
1
0
22
Difficulty Factor

• Values greater than 0.8
• The question is too easy
• Stem wording gives away the answer
• Not enough plausible incorrect answer options
• Values less than 0.2
• Guessing effect can be a factor
• The question is defective
• Material wasnt covered
• Stem wording is unclear, confusing
• Answer options need work

23
Evaluating Multiple Choice Questions

• Discrimination Index
• Proportion of capable students that select an
  answer option proportion of weak students
  that select the answer option
• i.e. D Ftop quarter Fbottom quarter

Good Discrimination
Reverse Discrimination
0
1
- 1
24
Discrimination Index

• For the Correct Answer Option
• Values 0.5 give good discrimination
• lt 0.2 consider dropping
• negative drop the question
• For Incorrect Answer Options
• Values should be negative.

25
Knowledge or Successful Guessing?

• Multiple Choice Exam Strategies
• Improve odds by eliminating 1 or more infeasible
  or unlikely answer options
• Descriptive Exam Strategies
• Brain dumping
• Part marks
• Consideration for perfect answers to questions
  that were not asked

26
Possibility of a Random Pass

• Depends on the number
• of answer options per question
• and the number of questions!

27
from Brown, 2001
28
Adjustment for Guessing

• Negative Marking
• Elimination strategy reduces odds of wrong answer
  penalty
• Subtracting a percentage of the number of wrong
  answers obtained from the final grade
• Give a grade of 4 for a correct answer and a
  score of -1 for a wrong answer on a 4 choice
  question

29
Negative Marking..

• A score of less than zero is possible
• Students hate negative marking
• Negative marking is not practised in descriptive
  examinations
• A poor substitute for a test that is too short
  with too few answer options

30
Teaching WithMultiple Choice Questions

• Prepare lectures incorporating all material
  covered in the questions to be used
• After each lecture, check coverage by reviewing
  questions
• Provide students with multiple choice questions
  for self-evaluation at the end of readings

31
Teaching WithMultiple Choice Examinations

• Use stem questions to teach additional or
  ancillary terms not covered in the course
• i.e. . For post-prandial (after dinner) drinks
  you want to stock a supply of fortified wines and
  brandy. The term "fortified wine" means.

32
Provide Students the Stem Questions Prior to the
Exam!

• Exam questions should cover all the course
  material being tested
• Provides more detail than a course outline
• Students focus their study effort on the topics
  they understand the least
• Increases learning
• Reduces stress

33

• Thanks for your interest in my topic!
• I look forward to your questions and ideas.
• J.E. (Joe) Barth
• School of Hospitality and Tourism Management
• University of Guelph, Ontario CANADA