Threats to Validity in Largescale Writing Performance Tests

1
Threats to Validity in Large-scale Writing
Performance Tests

• Tom Haladyna Arizona State University
• Bob Olsen, San Juan Unified School District
  (Carmichael, California)

2
Writing Performance Test

• A single prompt
• Holistic or analytic prompts
• Human, subjective scoring
• Might be augmented with multiple-choice items

3
Main premise of this essay

• Too many threats to validity
• These are enumerated and discussed here and in
  the paper
• We need to be very circumspect about how we use
  writing test scores.

4
Validity

• Reasonableness of a test score interpretation
• Reasonableness of a test score use

5
Validation

• Define writing
• Specify interpretations and uses
• Argument and claim for validity
• Construct validation
• Investigative process
• Requires evidence
• Judgment

6
What is writing?

• A cognitive ability
• A mode of communication
• Open-ended and closed-ended aspects
• Opencreative
• Closerules to guide us in effective writing
  (spelling, punctuation, etc.)
• At least five modes
• At least six analytic traits
• Thousands of writing tasks comprise the domain of
  writing
• Simple email note or text message
• Complicated essayThomas Friedmans The World is
  Flat

7
Types of Validity Evidence

• Content-related
• Reliability
• Item Quality
• Test Design
• Test Administration
• Scoring
• Scaling for Comparability
• Standard Setting
• Security/Cheating/Ethics

8
Points to Ponder About Validation

• As stakes go up, so does the need for evidence.
• Performance testing provides us with greater
  challenges.
• Validation can provide evidence that supports or
  weakens validity.

9
Content

• Fidelity of task to construct
• Cognitive demand
• Sampling adequacy
• Role of SR items in a writing performance test
• Role of computer ability and handwriting
  (construct relevant or construct irrelevant)

10
Reliability

• Hard to find coefficients
• Standard errors can be very large
• Coefficients can be inflated due to halo scoring
• How consistent are raters?

11
Prompt Quality

• How do you validate a prompt?
• How representative is a single prompt?
• What about modes?
• What about traits?
• How do we validate rubrics?
• What about DIF?
• P and D???
• Sparse matrix

12
Test Design

• Single prompt
• Two prompts
• Portfolio
• Prompts Selected Response Items
• Choice?
• Handwriting or computer writing?

13
Administration

• Is it uniformstandardized?
• Do teachers vary administration to gain an
  advantage for their students?
• Handwriting or computer writing

14
Scoring

• Rater training?
• Rater consistency
• Rater effects
• Scoring errorsquality control
• Wordiness
• Automated essay scoring

15
Scaling for Comparability

• Vertical growth scales
• Improvement over time
• Obtaining accurate estimates of prompt difficulty
  in field tests

16
Standard Setting

• Methods?
• Set for a prompt?
• Set on a scale for all prompts?
• Is there a proven technology?

17
Remedies

• Better, public technical reports
• Validity studies
• Independent evaluations

18
Implications for Practice

• Use writing scores in the company of other valid
  information about student writing.
• Limit its use to low-stakes uses.
• We do not want to eliminate writing tests.
• But we have to do a better job of combatting
  threats to validity.