The English Language Development Assessment ELDA

1
The English Language Development Assessment (ELDA)

• Technical Characteristics, Warranted Inferences
• Steve Ferrara
• Jun 27, 2006

2
Overview for this talk

• Roles and responsibilities in the test design and
  development process
• Technical characteristics of ELDA components

3
System for assuring technical adequacy and
supporting intended inferences

• Oversight and management
• Steering Committee, LEP SCASS, CCSSO
• Design and development
• AIR
• Focused validation studies
• CSAVE
• Implementation
• MI
• Project evaluation, focus on rigor of development
  process and outcomes
• Jamal Abedi

4
Design and development process

• Proposed design?refinements
• Design specifications
• Conventional development process
• Item reviews, item data reviews
• Pilot testing, scoring study, field testing
• Data quality checks, analysis
• Classical
• IRT scaling/linking
• Adjustments along the way

5
English Language Development Assessment (ELDA)

• NCLB requirements
• Annual assessment of English proficiency
• Reading, listening, writing, speaking,
  comprehension
• Progress toward full English proficiency
• ELDA
• Four grade-cluster assessments, grades K-12
• Vertically linked
• Aligned with ELDA English proficiency standards

6
ELDA (cont.)

• Developed, evaluated, and implemented with the
  LEP-SCASS of CCSSO
• Enhanced Assessment Grant, Title VI of NCLB
• Nevada, 18 other states, CCSSO
• AIR, CSAVE, Jamal Abedi, MI
• School-related topics
• ELA, math/science/technology, social studies,
  school-environmental
• Avoid content knowledge requirements
• Offer opportunity to use academic language

7
Analyses and results

• Test development and evaluation process
• Pilot test 2003
• Field tests 2004 2005 (five states)
• Operational administration 2005 (six states)
• Operational administration 2006
• Validity studies as part of 2004 field test
• CSAVE

8
Psychometric analyses

• Classical item analyses
• Item difficulty, discrimination, etc.
• DIF Male-female, Spanish-other, current
  ELLs-exited ELLs
• IRT analyses
• Simultaneous calibration
• Parameter stability, linking items, etc.
• All analyses plausibility checks, parallel
  independent analyses

9
Scaling

• Masters Partial Credit model
• Support vertical linking, standard setting, score
  reporting, inferences about student proficiency
  and growth over time
• Simultaneous calibration, stability and linking
  checks

10
Selected results

• Not just the good news

11
Mean classical item statistics
12
Coefficient alpha
13
Non-response rates, in percentages
14
Items flagged for DIF
15
Item bank calibration, forms assembly

• 2004 field test?operational form 1
• 2005 field test Link to operational form
  1?operational forms 2 and 3
• 2005 bank calibration
• Recalibrate form 1 simultaneous cross-grade,
  common items
• Field test items Fix parameters for common items
  in field test, estimate freely

16
Items flagged for misfit
17
Special analyses, TAC deliberations

• Two writing CR linking items, 6-8gt9-12
• Three follow-up analyses, decision to retain the
  CR items in the linking
• Linking with and without flagged linking items
  (reading and listening)
• Little difference in item parameters use them
  for linking

18
Special analyses, TAC deliberations (cont.)

• Calibrations of form 1 using operational and
  field test states data separately
• Virtually identical item parameter (0.01) and fit
• Parameter stability analyses
• 2004 field test and 2005 operational data
• Link the 2004 to the 2005 form (stepwise
  selection of linking items)
• 24, 24, 40, and 20 of items drifted
• Supports decision to recalibrate in 2005

19
Warranted inferences

• Student proficiency in English? Yes
• Coverage of ELP standards
• Classification consistency and accuracy
• Fidelity, scoring
• Student growth? Yes
• Drift analyses

20
Conclusions

• Groups completed important analyses to guide
  development and support implementation of ELDA
• Bank replenishment and refinement
• Ongoing validity research agenda
• ELDA project, individual states, and others
• New states

21
Speaking operational form 1 (2005)
22
Scaling results Speaking
23
TCCs Speaking, operational form 1