Test Development

1
Test Development

• PSY 614
• Fall 2007
• Instructor Emily E. Bullock, Ph.D.

2
Five Stages of Test Development

1. Test Conceptualization
2. Test Construction
3. Test Tryout
4. Item Analysis
5. Test Revision

3
Stage 1 Test Conceptualization

• There ought to be a test designed to
• measure __________ in _______ way.
• Some Test Conceptualization Questions
• What is the test designed to measure?
• What is the objective of the test?
• Is there a need for this test?
• Who will use this test?
• Who will take this test?
• What content will the test cover
• How will the test be administered?

4
Stage 1 Test Conceptualization

• (More Test Conceptualization Questions)
• What is the ideal format of the test?
• Should more than one form of the test be
  developed?
• What special training will be required of test
  users for administering or interpreting this
  test?
• What types of responses will be required by test
  takers?
• Who benefits as a results of an administration of
  this test?
• Is there any potential for harm as a result of an
  administration of this test?
• How will meaning be attributed to scores on this
  test?

5
Stage 2 Test Construction

• Selecting Item Types
• Scale Construction
• Rational
• Empirical
• Rational-Empirical
• Response Sets
• Social Desirability
• Random Responding
• Dissimulation
• Response Style

6
Stage 2 Test Construction

• Writing the Items
• What range of content should the items cover?
• How many items should be written?
• A typical standardized, multiple choice test
  should begin with an item pool two times the
  intended size of the the final test items
• Considerations should be made for reading level,
  grammar used, bias in wording, length of the
  item, and more.
• Scoring the Items

7
Stage 3 Test Tryout

• Use people similar in critical respects to the
  people for whom the test was designed
• 5 to 10 test takers per item
• Begin Item Analysis process to create final
  version of test

8
Step 4 Item Analysis

• Item Analysis set of procedures for analyzing
  responses to test items and the relationships
  between item characteristics and test
  characteristics
• What is a good test item?

9
A Good Test Item

• Is reliable and valid
• Helps to discriminate test takers
• High scorers on the test as a whole get the item
  correct
• Low scorers on the test as a whole get the item
  incorrect

10
Step 4 Methods of Item Analysis

• Item Difficulty Index
• Item Validity Index
• Item Reliability Index
• Item Discrimination Index
• Item Response Theory
• Test Item Bias
• Qualitative Item Analysis

11
Step 4 Item Analysis Item Difficulty Index

• Item Difficulty Index an index of an items
  difficulty is obtained by calculating the
  proportion of the total number of test takers who
  got the item correct
• p7 would be read item difficulty index for item
  7
• p ranges from 0 (no one got item correct) to 1
  (everyone got item correct)
• If 75 of 100 test takers got item 7 correct then
• p7 .75 because 75/100 equals .75

12
Step 4 Item Analysis Item Difficulty Index

• Difficulty is relative to the item and the test
  taking population
• Provides a common measure of the difficulty of
  test items that measure completely different
  domains
• Has little impact on reliability or validity of a
  test
• Item Difficulty impacts the variability of test
  scores and precision with which test scores
  discriminate among different groups of test
  takers
• p of .5 is optimum for test items

13
Step 4 Item Analysis Item Discrimination Index

• Item Discrimination Measures indicate how
  adequately an item separates or discriminates
  between high scorers and low scorers on an entire
  test
• It is a measure of the difference between the
  proportion of high scorers answering an item
  correctly and the proportion of low scorers
  answering the item correctly

14
Item Validity and Reliability Indices

• Item Validity Index method of assessing the
  criterion-related validity through a formula that
  utilizes the item-score standard deviation and
  the correlation between the item score and the
  criterion score
• Item Reliability Index method of assessing the
  internal consistency of a test using the
  item-score standard deviation and the correlation
  between the item score and the total test score

15
Step 4 Item Analysis Item Response Theory

• Provides informative and interpretable
  information about
• Item characteristics
• Ability levels needed to pass test items
• Test scores
• Does not address reliability and validity of a
  test but information for understanding how
  individual differences affect the behavior of an
  individuals performance on a specific item
• Provides information that is sample non-dependant

16
Step 4 Item AnalysisTest Item Bias

• Consensus that it is virtually impossible to
  assess item bias by examining item content
• Use the concepts of validity to determine whether
  the test measures the same thing for different
  groups in the population

17
Step 4 Item Analysis Qualitative Item Analysis

• Gaining information about a test takers
  experience taking the test or an experts opinion
  of the test
• Types of Qualitative Item Analysis
• Test-taker questionnaire
• Think-aloud test administration
• Expert Panels

18
Stage 5 Test Revision

• Based on your item analysis, reliability
  estimates, and validity information determine
  which items form the final test
• The test may be revised multiple times as it is
  goes through the test tryout, item analysis, test
  revision loop.
• Norm and Standardize
• Cross Validation

19
After the Test is Developed

• Writing the Manual
• Test Publication
• Professional journals
• Publishing Companies
• Marketing and Rights
• Additional Revision
• Implications of revision