Best Practices for Building Tests and Quizzes

1
Best Practices for Building Tests and Quizzes

• League for Innovation
• November 2004

2
7 question types

• multiple choice
• true/false
• essay/short answer
• matching
• ordering
• fill in the blank
• multiple answers

3
Part II-Instructional Design Considerations for
Creating Tests
4
Selected Response Questions

• Selected response questions require students to
  select from a predetermined list of potential
  answers.
• multiple choice
• true false
• Matching
• fill-in-the blank questions

5
Created Response Questions

• Ex. Essays measure the students ability to
  communicate effectively, not just their
  understanding of content.
• Easier to write but harder and (more subjective)
  to grade.

6
Match Question Type to Level of Assessment Desired

• Multiple choice and matching questions
• offer the most flexibility in terms of content
  covered
• thinking skills that can be assessed
• True/false are usually limited to fact recall.

7
Match Instructional Objectives

• Use lesson plans or teacher notes to accurately
  reflect content that was covered in class
• Choose the most important objectives to assess
  and use these as the outline for your test

8
Cover Important Material

• Facts, definitions, comprehension, analysis,
  applications
• Trivial items result in trivial studying and
  learning.

9
Items Should be Independent

• Do not give away answers via information in
  other questions
• Independence maximizes breadth of coverage

10
Write Simply and Clearly

• Measure knowledge of material and concepts, not
  vocabulary
• Ambiguous questions create error, frustration,
  and compound biases related to language and
  disability

11
Clearly Specify What Type of Response is Sought

• How long or short an answer is sought?
• Should they show their work?
• Do you want description? comparisons?
  application? evaluation?
• Whose opinion do you want (book, lecture, their
  own)?

12
Good Tests Take Time to Write

• Give yourself enough time to evaluate items after
  a day or two
• Revise, edit, and ask others to read before
  administering the test

13
Good Tests Have High Degrees of Reliability and
Validity

• Reliability refers to the extent to which
  measurement is consistent
• Validity refers to the extent to which a test
  measures what it should.

14
Poor Example
15
Construct Questions Carefully

• The most important test-making process is
  selecting the wording of each question and
  answer.
• Question Stem and Distractors

16
Construction Tips

• Construct evaluation items with a single correct
  answer
• Use plausible distracters
• Arrange options in a logical sequence
• Include a minimum of 3 and a maximum of 5 options
• Alternate the order of the correct responses
  randomly among all options

17
Construction Tips

• Refrain from using the choices all of the above
  or none of the above
• Identify an initial difficulty level

18
Construction Tips

• Increasing item difficulty can be achieved by
  paying attention to the distracters (options)
• The more homogeneous the options and
  distracters, the higher the degree of difficulty

19
Higher Degree of Difficulty

• The first James Bond movie was released in what
  year?
• a) 1960
• b) 1962
• c) 1964
• d) 1966

20
Mid-Level Difficulty

• The first James Bond movie was released in what
  year?
• a) 1960
• b) 1962
• c) 1964
• d) 1970

21
Low-Level Difficulty

• The first James Bond movie was released in what
  year?
• a) 1958
• b) 1962
• c) 1970
• d) 1975

22
Points of Concern for Different Formats

• Writing True-False Items
• Choose words with precise, definite meanings
• Avoid tricks and trivia
• Avoid easy clues
• Follow guidelines for multiple choice

23
Writing Multiple Choice Questions

• Present the problem, including qualifying
  statement
• There should be only one correct answer
• Distracters should be plausible but clearly
  incorrect
• Avoid negative wording (especially double
  negatives)

24
Writing Multiple Choice Questions

• When item is controversial, indicate whose
  opinion is sought
• Avoid irrelevant cues to correct answer (length,
  grammar)
• Items should test one central idea or concept

25
Writing Multiple Choice Questions

• Watch out for patterns in alternatives (overuse
  of "C" as correct answer)
• Choose appropriate level of difficulty
• Assure even coverage of material and types of
  knowledge

26
Writing Matching Items

• Keep each matching set short (no more than 5
  stimuli/responses)
• Use longer items as stimuli, shorter as responses
• Arrange responses in alphabetical or logical
  order

27
Writing Short Answer and Completion Items

• Clearly indicate type of answer you want
• Do not use more than two blanks per completion
  item
• Make a key before scoring
• Periodically re-score early tests to detect
  shifting criteria

28
Writing Essay Questions

• Limit questions to vitally important material
• Clearly define task, scope, and directions for a
  "good" answer
• Allow time for thought
• Use multiple medium-length essays rather than one
  long one

29
Writing Essay Questions

• Use questions that have a limited number of good
  answers
• Allow choice between alternatives (e.g., "answer
  3 of the 4 questions")

30
Part III- Evaluating your Assessment or Assessing
Your Evaluation
31
Evaluating the Test

• Once scored, spend some time reviewing the test
  and observing patterns that may be present.
• Were there any questions that every student got
  wrong?
• If so, can you deduce whether it was due to poor
  test item construction or to instruction?

32
Evaluating the Test

• If poorly written, you may want to consider
  canceling it out and recalibrating the test.
• If students didnt understand the concept,
  perhaps the content needs to be retaught.

33
A Checklist for Teachers

• Are your directions CLEAR and CONCISE?
• Are your objectives clear?
• Is there a logical connection between questions
  and answers?
• Are you being specific in regard to the area you
  are covering?
• Are your answers simple and factual?

34
Higher Order Thinking

• What later events best affirm the ideas set forth
  in the Declaration of Independence?
• A) Emancipation Proclamation19th Amendment
• B) Eminent Domain, Manifest Destiny
• C) Civil War

35
References

• Orlich,Donald C, Harder,Robert J.,Callahan,Richard
  C, and Gibson,Harry W.(1998), Teaching
  Strategies A guide to better instruction, 5th
  Ed, Houghton Mifflin, New York
• Royse, David (2001) Teaching Tips for College and
  University Instructors A Practical Guide, Allyn
  and Bacon, Boston

36
References

• Dominowski, Roger L. (2002),Teaching
  Undergraduates, Lawrence Erlbaum Associates, New
  Jersey
• Paul Eggen, Don Kauchak (2001),Educational
  Psychology, Windows on Classrooms, 5th Ed, Pub.
  Merrill- Prentice Hall, Upper Saddle River, New
  Jersey.

37

References

• Bloom, B.S. (Ed.) (1956) Taxonomy of educational
  objectives The classification of educational
  goals Handbook I, cognitive domain. New York
  Toronto Longmans, Green
• Chase, C.I. (1999).  Contemporary Assessment for
  Educators.  New York Addison Wesley Longman,
  Inc. 
•  

38

References

• Dominowski, R.L. (2002).  Teaching
  Undergraduates.  Mahwah, NJ  Lawrence Erlbaum
  Associates. 
• Friedenberg, L. 1995. Psychological testing
  Design, analysis, and use.  Needham Heights
  Allyn Bacon.  
•  

39
References

• Kay Burke (1999), The Mindful School How to
  Assess authentic Learning Pub.SkyLight
  Professional Development , Arlington Heights,
  Illinois
• Heinich, R., Molenda, M., Russell, J. D., and
  Smaldino, S. E. (1996). Instructional media and
  technologies for learning (5th ed.). (pp. 40-41).
  Englewood Cliffs Prentice-Hall.

40
References

• Jacobs, L. C. Chase, C. I.  (1992). Developing
  and using tests effectively  A guide for
  faculty.  New York  Jossey-Bass.
• Kopeikin, Hal S. (n. d.) UC Los Angeles
  lecturer, unpublished notes on writing effective
  questions.
• http//web.wi.mit.edu/sive/pub/Lab20Resources20D
  ocuments/Thoughts20on20Exam20Quests.html

41
References

• http//www.rpi.edu/verwyc/chap2tm.htm
• http//www.glencoe.com/sec/teachingtoday/education
  upclose.phtml/print/40
• http//taesig.8m.com/createv.html
• http//www.utc.edu/Administration/WalkerTeachingRe
  sourceCenter/FacultyDevelopment/Assessment/test-qu
  estions.html
• http//www.psych.ucsb.edu/taap/writingtests.html
  (defunct)

42
Writing Exam Questions and Constructing a Test

• Anthony Foster
• Senior Instructional Designer

43
The Task at Hand

• Creating effective tests is an essential task for
  all classroom teachers
• While Course management tools allow for ease of
  distributing quizzes and tests online, they do
  nothing to help with sound practices.
• Lets examine the importance of good test
  question construction and
• Provide a variety of solid suggestions you can
  follow when constructing your next classroom test

44
The Importance of Good Tests

• A well-written test allows you to accurately and
  consistently measure students mastery of
  specific content taught in class
• Results of well-written tests also allow teachers
  to measure, to some degree, how effective their
  instruction has been

45
The Danger of Bad Tests

• Poorly designed test questions can lead to
• inaccurate measurements of learning and provide
  false information regarding student performance
• decreased effectiveness of instruction
• unintended measurements of skills not taught

46
I. Guidelines for Good Tests
47
Cover important material

• Facts
• Definitions
• Comprehension
• Analysis
• Applications
• Trivial items result in trivial studying and
  learning

48
Strive for proportional representation

• Make sure lectures, readings, and the key
  elements of the course are measured without too
  much emphasis on any one area

49
Items should be independent

• Do not give away answers via information in other
  questions
• Independence maximizes breadth of coverage

50
Write simply and clearly

• Measure their knowledge of material, not
  vocabulary or "mind reading"
• Tricky or ambiguous questions create error,
  frustration, and compound biases related to
  language and disability

51
Clearly specify what type of response is sought

• How long or short an answer is sought?
• Should they show their work?
• Do you want
• description?
• comparisons?
• application?
• evaluation?
• Who's opinion do you want (book, lecture, their
  own)?

52
Good items and therefore good tests take time to
write

• Give yourself enough time to evaluate items after
  a day or two
• Revise, edit, and ask others to read before
  administering

53
Good tests have high degrees of reliability and
validity

• Reliability refers to the extent to which
  measurement is consistent
• Validity refers to the extent to which a test
  measures what it should

54
Types of Test Questions

• The test questions most teachers use in
  classrooms fall into two broad categories
• selected response and
• created response

55
Selected response questions

• Require students to select from a predetermined
  list of potential answers
• Multiple choice
• True/false
• Matching
• Fill-in-the-blank questions

56
Selected response questions

• Often viewed as less challenging in terms of the
  thinking skills required to answer them
• When well written, they can measure higher levels
  of thinking, not simply the recalling of facts
• The writing of these test items can be
  challenging, and they frequently take more time
  to construct
• When well written they are easier to score and
  can provide a more objective method of assessment
  than do created response items

57
Created response questions

• include extended answer and essays
• measure the students ability to communicate
  effectively, not just their understanding of
  content

58
Created response questions

• Several considerations when choosing what to
  include in a test
• fair assessment of knowledge or skills,
• number of students taking the test
• amount of time available to score the test
• This type is often easier to write, but can
  require more time to score
• Scoring often less reliable, more subjective

59
Writing Better Tests

• There are a number of steps you can take to
  improve the quality of the tests you write

60
Match Instructional Objectives and Teacher Notes
to the Test

• When creating your test items...
• Refer back to your instructional objectives for
  the content and skills you want to assess
• Use any lesson plans or teacher notes regarding
  what was covered and assigned to ensure your test
  items accurately reflect content that was covered
  in class.
• Choose the most important objectives to assess
  and use these as the outline for your test.

61
Match Question Type to Level of Assessment Desired

• Carefully choose the type of questions
• Multiple choice and matching questions offer the
  most flexibility in terms of content that can be
  covered and thinking skills that can be assessed
• T/F are usually limited to fact recall
• Try to balance the number of question types and
  limit their number to no more than three types on
  one test for middle school and high school
  students

62
Construct Questions Carefully

• The next most important test-making process is
  selecting the wording of each question and answer
  
• The stem or question portion of the test item
  should be carefully worded using straightforward
  language
• Careful attention should also be given to writing
  the correct and incorrect answers (also known as
  distractors)

63
II. Advantages and Disadvantages of Different
Item Formats
64
Multiple-Choice Advantages

• Possible to cover broad range of knowledge in
  limited time
• Skillfully written items can measure higher-order
  cognitive skills
• A MC test can be scored quickly and objectively
• Strengths and weaknesses can be detected easily

65
Multiple Choice Disadvantages

• Individual differences in skill at such items
  equals bias
• Measure recognition rather than recall
• More likely to measure recognition of content
  than integration or application of information
• It is difficult and time-consuming to write good
  items
• It is possible to assess higher-order cognitive
  skills, but most assess only knowledge
• Some correct answers can be guesses

66
True/False Advantages

• Same as multiple-choice, i.e.
• Possible to cover broad range of knowledge in
  limited time
• Skillfully written items can measure higher-order
  cognitive skills
• A T/F test can be scored quickly and objectively
• Strengths and weaknesses can be detected easily

67
True/False Disadvantages

• Same as multiple-choice
• In addition, often too "black white" or clear
  cut for subject matter (psychology often is not
  so clear cut
• random guessing equals 50 correct

68
Short Answer and Completion Items Advantages

• Requires recall of ideas
• Can measure integration and application more
  easily than multiple-choice
• Takes less time per item than essays (advantage
  of broader sampling)

69
Short Answer and Completion Items Disadvantages

• Moderately time consuming
• Less breadth of sampling than multiple-choice
• Consistent scoring is difficult and time consuming

70
Essay Questions Advantages

• Best measure of organizing, integrating, and
  evaluating information
• Can approximate application of knowledge, easy to
  write

71
Essay Questions Disadvantages

• Time consuming (reduces breadth of sampling)
• Scoring is very time consuming
• Scoring consistently is very hard
• Writing skill and speed are often major sources
  of bias

72
III. Writing Exam Questions
73
Writing Multiple-Choice Items

• Stem should present the problem, including
  qualifying statements
• There should be only one correct answer
• Distractors should be plausible but clearly
  incorrect
• Avoid negative wording (especially double
  negatives)
• Use "all of the above," "none of the above" and
  "A and B" sparingly
• When item is controversial, indicate whose
  opinion is sought

74
Writing Multiple-Choice Items

• Avoid irrelevant cues to correct answer (length,
  grammar)
• Items should test one central idea or concept
• Watch out for patterns in alternatives (overuse
  of "C" as correct answer)
• Choose appropriate level of difficulty--best to
  write item figuring 50-75 of students would
  choose the correct answer
• Assure even coverage of material and types of
  knowledge

75
More on MC

• Multiple-choice questions seem to be many
  teachers' preferred testing method in that they
  are relatively easy to correct and grade
• The construction of good MC questions does
  require some thought and planning The following
  are simple guidelines

76
Writing True-False Items

• Choose words with precise, definite meanings
• Avoid tricks and trivia
• Avoid "always" and "never" (easy clues)
• Follow guidelines for multiple choice

77
Writing Matching Items

• Keep each matching set short (no more than 5
  stimuli/responses)
• Each item should have only one correct match
• Use longer items as stimuli, shorter as responses
• Arrange responses in alphabetical or logical order

78
Writing Short Answer and Completion Items

• Phrase item to elicit one correct answer (makes
  scoring possible)
• Clearly indicate type of answer you want
• Do not use more than two blanks per completion
  item
• Try to place blank near the end of a completion
  item
• Make a key before scoring
• Periodically re-score early tests to detect
  shifting criteria

79
Writing Essay Questions

• Limit questions to vitally important material
• Clearly define task, scope, and directions for a
  "good" answer
• Allow time for thought
• Use multiple medium-length essays rather than one
  long one

80
Writing Essay Questions

• Use questions that have a limited number of good
  answers
• Allow choice between alternatives (e.g., "answer
  3 of the 4 questions")
• Make a key before scoring
• Periodically re-score early tests to detect
  shifting criteria

81
Presenting the question or problem

• The 'stem' of a MC question is the statement for
  which a choice of response is to be made The stem
  has to be
• Clearly stated
• Composed of complete statements
• Comprehensive enough so that the subsequent
  choices can be brief
• Succinctly stated

82
Examples of well-stated stems

• The term 'Native Americans' refers to which of
  the following groups?"
• The US president's election is determined by"

83
Examples of poorly stated stems

• Fluctuations in the stock market
• Unsupervised school children

84
Determining response options

• Typically, MC questions will include 4-5 response
  options To be effective, response options should
• Include only one completely correct and
  defensible option
• Include only choices that are logically connected
  to the stem
• Include choices that represent misconceptions
  previously expressed by students
• Include choices that are mutually exclusive
• Avoid 'all of the above' or 'none of the above'

85
Formatting of response options

• Approximate uniform length of items
• Use uniform grammatical structures (all
  sentences, all phrases, etc)
• Use a consistent number of options for all
  questions
• State options positively, whenever possible
• Distribute the correct option randomly among the
  choices (not all correct answers should be 'B',
  etc)
• Eliminate verbal cues flowing directly from the
  stem

86
Formatting the Test

• Are your directions CLEAR and CONCISE?
• Are your objectives clear?
• Are you being specific in the area you are
  covering?
• Are your answers simple and factual?
• Are your questions on the left and numbered?

87
Formatting the Test

• Are your answers on the right and lettered?
• Are your answers in some specific order? Ex.
  Alphabetical or Chronological
• Is there a logical connection between questions
  and answers?
• Are there more answers than questions?
• Are your answers and questions concise?

88
Top Ten Rules for Writing Good Questions

• Remember your tests purpose
• If in doubt, throw it out
• Keep your questions simple
• Stay focused - avoid vague issues
• If a question can be misinterpreted, it will be

89
Top Ten Rules for Writing Good Questions

• Include only one topic per question (avoid
  "double-barreled" questions)
• Avoid leading questions
• Consider alternate ways to ask sensitive
  questions
• Make sure the respondent has enough information

90
Things to remember when writing an essay test
91
Do your essay tests

• Have Clear and Specific Directions
• Contain Any Ambiguous Words
• Have a Specific Task for Each Question
• Avoid Factual Knowledge Questions
• Avoid Giving Students Options
• Account for Class Size and Time

92

• MC questions that ask for the best answer out of
  choices that incorporate other answers can be
  confusing and should be limited

93
Punctuation

• The MC question should be constructed so that the
  stem, when complemented by the option, makes up a
  complete, correct sentence
• If the stem of a MC question is a question, then
  each option should begin with a capital letter 
• Each option should not have a period unless it is
  a complete sentence in and of itself

94
Punctuation

• If the stem is a question or statement with a
  blank in it that one option is intended to fill,
  all options should be lowercase with no period
  after them
• If the stem is an incomplete question, and one
  option is meant to be added to the end in order
  to complete the sentence, the options should be
  lowercase and have a period after each