Technical Scaling Decisions: Do They Matter

1
Technical Scaling Decisions Do They Matter?
The Psychology of Pattern Scoring The Tradeoff
between Accuracy and Transparency

• Jennifer L. Dunn
• Center for Assessment

2
Does the method used to estimate scale scores
matter?

• But first.
• Where do scale scores come from?
• If using the 3PL IRT Model
• Difficulty, Discrimination, Guessing
• There are a couple of options
• Total test Number Correct
• Item Response Patterns

3
Number Correct

• Based on characteristics of the total test
• Count the correct responses
• Find the scale score that corresponds to the count

4
Using Number Correct

• Advantages
• Simplicity
• Transparency
• One to one relationship between scale scores and
  raw scores

• Disadvantages
• Does not use all available information
• Theoretically less reliable
• Potentially biased
• Getting scale scores below guessing can be
  problematic
• Getting a scale score for perfect number correct
  scores is problematic

5
Response Patterns

• Based on the characteristics of the items
• Use the response pattern of each student to
  estimate the scale score
• Statistical optimal weighting of item responses
• The properties of the items that a student gets
  correct/incorrect matter

6
Using Response Patterns

• Advantages
• Maximal use of available information
• Theoretically more reliable
• Theoretically less biased
• MLE
• Minimize bias
• EAP
• Scale score estimates for all students

• Disadvantages
• Students with the same raw score can have
  different scale scores
• More complex
• Not transparent
• MLE
• No estimates for perfect and 0 scores
• EAP
• Biased estimates for extreme abilities

7
Does it Matter?

• What are the differences between methods
• Random Error
• Systematic Error
• If the characteristics of the items change do the
  differences between methods change?
• Difficulty
• Discrimination

8
Simulation

• Need to know TRUTH
• Create Students
• 10,000 Students
• Create Items
• 29 Multiple Choice Items
• State Science Assessment
• Estimate Scale Scores
• Number Correct (TCC)
• Response Pattern (MLE)
• Response Pattern (EAP)
• Compare Estimates to TRUTH

9
(No Transcript)
10
(No Transcript)
11
Does it Matter?

• What are the differences between methods
• Random Error
• Systematic Error
• If the characteristics of the items change do the
  differences between methods change?
• Difficulty
• Discrimination

12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
Thoughts and Conclusions

• What are the differences between methods?
• Very little difference between the approaches
• TCC has similar RMSE and Bias as MLE
• EAP has lower RMSE but higher Bias
• Because of transparency, TCC seems the best
  option
• Do the characteristics of the items matter?
• Increasing item discrimination
• Decreases RMSE
• Increases extreme ability bias of TCC estimates
• The changes are not meaningfully significant
• Improving item Discrimination is a good thing
• Recommend TCC Scale Scores for simplicity