DARA Research and Next Steps

1
DARA Research and Next Steps

• Cara Cahalan-Laitusis Linda Cook
• Educational Testing Service

2
Presentation

• Experimental Study of Read Aloud
• Psychometric Research
• Research Plans for Year 3
• Psychometric analysis of experimental data
• Tailored Test Design
• Cognitive labs
• IEP Decision Making for read aloud

3
Differential Boost from Read Aloud (Non-disabled
vs. RLD)

• Is there a Differential Boost from read aloud?
• How well do test scores (standard, audio, and
  fluency) predict variance in teacher ratings of
  reading comprehension?
• Are teachers able to predict which students will
  benefit from read aloud?

4
Prior Research

• No Differential Boost
• Kosciokek Ysseldyke (2000)- Small sample size
  (n31)
• Meloy, Deville, and Frisbie (2002) Between
  subjects design (n260, 76 non-disabled,
  randomly assigned to audio or standard)
• McKevitt Elliott (2003)-Small sample size
  (n39)
• Differential Boost
• Crawford and Tindal (2004)-(n338, 78
  non-disabled)
• Fletcher, et. al (2006)-Between subjects design
  (randomly assigned to audio or standard). Sample
  included 91 Dyslexic (poor decoder) and 91
  average decoders

5
Data Collected

• GMRT Forms S and T
• Extra Time
• Extra Time with Read Aloud via CD
• 2 Fluency Measures
• WJ Reading Fluency
• Test of Silent Word Reading Fluency
• 2 Decoding Measures (4th grade only)
• WJ Letter Word ID
• WJ Word Recognition
• Demographic and Survey Data

6
Sample

• 1170 4th Graders
• 522 Students with RLD
• 648 Students without a disability
• 855 8th Graders
• 394 Students with RLD
• 461 Students without a disability

7
Design
8
Means for Grade 4
9
Means for Grade 8
10
Scores by RLD and Grade
11
Is there a Differential Boost from read aloud?

• Repeated Measures ANOVA and ANCOVA
• Dependent Variables
• GMRT Standard
• GMRT Audio
• Independent Variables
• Disability Status (RLD vs. NLD)
• Form/Order (STSA, STAS, TSSA, TSAS)
• Covariate Decoding and Fluency Measures

12
ANOVA Findings

• Yes, students with reading-based learning
  disabilities have larger gains (on average) from
  read aloud than students without disabilities
• Finding consistent at both grades 4 and 8, but
  boost is larger at grade 4
• Controlling for Decoding and/or Fluency as a
  covariate did not alter findings

13
How well do test scores predict reading
comprehension?

• Multiple regression analyses to determine how
  much variance in teachers rating of reading
  comprehension (5-point scale) were predicted by
  three test scores
• Standard
• Audio
• Fluency

14
Regression Findings

• Audio score does not significantly predict
  variance in Teacher Ratings of Reading
  Comprehension (beyond standard and fluency) for
  Grade 8 RLD
• Audio score adds to prediction of reading
  comprehension (beyond standard and fluency
  scores) for three groups (NLD grade 4, NLD grade
  8, and RLD grade 4), but incremental change is
  small

15
Are teachers able to predict which students will
benefit from read aloud?

• Analyses
• Analysis of variance in boost by teacher
  predictions
• Cross-tabulations of teacher ratings by degree of
  boost (more than on SEM, less than one SEM,
  neither)

16
Accuracy of Teacher Prediction

• For this study each student took a reading
  comprehension test that was read aloud by a CD
  player and another reading comprehension test
  that they read to themselves. Which test do you
  predict the student did better on?
• ? Test read aloud by CD player
• ? Test the student read to themselves
• ? No difference

17
Findings from Teacher Predictions

• ANOVA indicated that on average teachers were
  able to predict score gain from audio at grade 4
  but not grade 8
• At the individual level teachers accurately
  predicted if a student would benefit from the
  audio version about 35 of the time and were
  completely wrong about 5 of the time

18
DARA Psychometric Research

• Purpose of psychometric research To help us
  understand how an examinee's disability or the
  accommodations he or she receives impacts the
  psychometric properties of a reading test

19
Results of This Years Psychometric Analyses

• Psychometric Analyses
• Factor analyses
• Differential item functioning analyses
• Populations
• Students with learning disabilities who took the
  test with and without accommodations
• Test
• Grade 4 and grade 8 English-language arts (ELA)
  assessment
• Focus
• Determine if the test measures the same
  constructs for
• Examinees without disabilities
• Examinees with disabilities who took the test
  with and without accommodations

20
(No Transcript)
21
(No Transcript)
22
STAR ELA Grade 4 and Grade 8 Summary Statistics
23
Factor Analyses of ELA Assessment

• Exploratory analyses (separately in each group)
• how many factors
• Confirmatory (multi-group)
• Establish base-line model
• Confirm number of factors needed to describe data
  across all groups

24
Differential Item Functioning (DIF)Analyses

• The purpose of this study was to examine
  differential item functioning on the same
  English-Language Arts assessment that was used
  for the factor analyses
• DIF is a statistical observation that involves
  matching test takers from different groups on the
  characteristic measured by the test and then
  looking at performance differences on an item.
  (Sireci, 2006)

25
Method

• Mantel-Haenszel Categorization3 Levels
• A ? Negligible DIF
• B ? Slight to Moderate DIF
• C ? Moderate to Large DIF
• Directions of DIF Flags
• - ? Favors reference group
• ? Favors focal group

26
Comparisons Made in the Study
27
DIF Categories ELA Grade 4 LD Without
Accommodations
Easy
Difficult
Favors Students Without Disabilities
Favors LD Students
28
DIF Categories ELA Grade 4 LD With Accommodations
(IEP/504)
Easy
Difficult
Favors Students Without Disabilities
Favors LD (IEP/504)
29
DIF Categories ELA Grade 4 LD With Accommodations
(Read-Aloud)
Easy
Difficult
Favors Students Without Disabilities
Favors LD (Read-Aloud)
30
DIF Categories ELA Grade 8 LD Without
Accommodations
Easy
Difficult
Favors Students Without Disabilities
Favors LD Students
31
DIF Categories ELA Grade 8 LD With Accommodations
(IEP/504)
Easy
Difficult
Favors Students Without Disabilities
Favors LD (IEP/504)
32
DIF Categories ELA Grade 8 LD With Accommodations
(Read-Aloud)
Easy
Difficult
Favors Students Without Disabilities
Favors LD (Read-Aloud)
33
Interpreting the Results of the DIF Study

• Grade 4
• 1 C DIF item, 8 B DIF items
• Grade 8
• 1 C DIF item, 6 B DIF items
• Majority of flagged items were reading items that
  favored students who took test with read-aloud
  accommodation
• Consistent with Factor Analysis Results

34
Next Steps

• Psychometric Research
• Examination of Tailored Testing
• Cognitive Labs
• IEP decision making

35
Psychometric Research
36
Plans for Next Years Psychometric Analyses

• Psychometric analyses
• Factor Analyses
• Differential item functioning analyses
• Populations
• Students with learning disabilities who took the
  test with and without an audio accommodation
• Test
• Gates-McGinitie Reading Test
• Focus
• Aid in interpretation of results of differential
  boost study
• Increase understanding of impact of disability
  and audio accommodation on reading test scores

37
Factor Analyses We Plan to Carry Out

• Aid in interpretation of results of differential
  boost study
• Compare factor structures for students without
  disabilities who took test with and without
  accommodation
• Compare factor structures for students with
  disabilities who took test with and without
  accommodation

38
Factor Analyses We Plan to Carry Out

• Increase understanding of impact of disability
  and accommodation on reading test scores
• Compare factor structures of test given to
  examinees with and without disabilities under
  standard conditions
• Compare factor structure of test given to
  examinees with disabilities who take test with
  accommodations and examinees without disabilities
  who take test without accommodations

39
Purpose of Doing DIF and DDF Analyses on Data
From the Differential Boost Study

• Aid in interpretation of results of differential
  boost study
• Increase understanding of impact of disability
  and accommodation on reading test scores

40
Possible Comparisons for DIF Analyses
41
Procedures for Analyzing Data

• Differential Item Functioning Mantel-Haenszel
• Differential Distractor Analysis Standardized
  Distractor Analysis

42
Two Staged Tailored Testing
43
Operational Data
44
GMRT Data
45
DARA Tailored Testing Model

• Two (or three) stages of testing
• Students subtests on stage 2 are determined by
  performance on routing test administered in stage
  1
• Ideally computer administered but can be paper
  administered
• Some parts could be individually administered
  (e.g., decoding) if only a few students are
  routed into a decoding measure and this format
  reduces the number of students receiving
  individualize testing accommodations (e.g., read
  aloud by human)

46
(No Transcript)
47
Advantages of Model

• Score is more reliable estimate since items are
  targeted to students ability level
• Students may feel less frustrated if they can do
  some of the items on the routing test
• Teacher receives more information on low
  performing students strengths and weaknesses
• Fundamental Skills and Comprehension are not
  confounded for students with poor fundamental
  skills (some LD) or poor comprehension (some LD
  and ELL)
• Growth can be more accurately measured in
  students working significantly below (or above)
  grade level

48
Disadvantages of Model

• Requires computer administration or teacher
  scoring of items after stage 1
• Students who are routed to fluency test may be
  embarrassed
• Routing decision is made before test is scaled or
  standard setting is completed
• Design could route more that 2 of students to
  modified test

49
Questions for Year 3

• How many items (and of what difficulty) are
  needed for an accurate routing test?
• Can we equate the audio extended and standard
  extended using the routing test?
• What portion of students would be routed to
  fluency measure and what portion would be routed
  to decoding?

50
Questions for Year 3 (continued)

• Are the 2 alternate routes highly correlated with
  the standard administration?
• What is the impact audio, fluency, and decoding
  scores on total test score.
• If student is not a fluent reader should the
  total test score be non-proficient?
• Is the routing test accurate for all students?
• Do some students do better on hard items?
• Do some students having trouble with the first
  few items on the test?

51
Questions for Year 3 (continued)

• How should we weight different measures and what
  impact will this have on subpopulations?
• Could we compose a tailored test from a states
  current operational item pool?
• If not how many additional items would be
  required and at what difficulty level?

52
Cognitive Labs
53
Background

• Cognitive labs using the think aloud method on
  reading comprehension questions
• Build off the findings of last years large scale
  differential boost study
• Gates MacGinitie Reading Comprehension Test
• Use items found in preliminary findings of the
  DIF analysis of the GMRT data

54
Cognitive Labs Advantages

• Beneficial to learn about components of mental
  processes of reading (Afflerbach Johnston,
  1984, Alavi, 2005 Pressley Afflerbach, 1995)
• Beneficial in the development of assessments
  (Caspar, Lessler, Willis, 1999 Desimone
  LeFloch, 2004 Willis, 2005)
• Open flexible procedure can be catered to the
  specific situation and activity (Davison, Vogel
  Coffman, 1997)
• May use a small sample size
• Procedure has been successfully conducted with
  children as young as 3rd grade (Laing Kamhi,
  2002 Paulsen Levine, 1999 Trambasso
  Magliano, 1996)

55
Cognitive Labs Disadvantages

• Thinking aloud is an unnatural step which may
  affect or interfere ones normal mental processes
  
• Students with disabilities may have difficulty
  with the procedure (Johnstone, Miller,
  Thompson, in press)
• Responses have the potential to be incomplete or
  incorrect
• Lack of desire/motivation
• Embarrassment
• Inability to understand the task

56
Purpose of Study

• This study is being conducted to serve the
  following
• purposes
• How do students with and without reading-based
  learning disabilities differ as they approach a
  reading comprehension assessment?
• Is this type of information gathering and data
  quality worthwhile to conduct in future large
  scale studies considering
• Age of students
• Students with disabilities

57
Research Questions

• In what way do students with reading-based
  disabilities respond differently to reading
  comprehension questions compared to students
  without disabilities?
• What errors occur while reading the
  passage/reading the items?

58
IEP Decision Making
59
IEP Decision Making

• What factors contribute to boost?
• Low standard score
• WJ Reading Measures
• Teacher Predictions
• Student Preference

60
Analyses planned

• Regression analyses to predict boost for RLD
  students using
• WJ scores
• Standard score
• Use of read aloud in class or on tests
• Teacher predictions
• NJ ASK from prior year