Developing a Severity-of-Disability Scale and Modeling Early Reading and Math Performance in a Longitudinal Study of Preschoolers with Disabilities

1
Developing a Severity-of-Disability Scale and
Modeling Early Reading and Math Performance in a
Longitudinal Study of Preschoolers with
Disabilities

• Elaine Carlson
• Tamara Daley
• Frank Jenkins
• Westat
• The Pre-Elementary Education Longitudinal Study
  is funded by the U.S. Department of Education.

2
PEELS design

• 6-year national longitudinal study of
  preschoolers with disabilities, currently in its
  fourth year.
• Designed to describe
• the children,
• the services they receive,
• their transitions from early intervention to
  preschool and preschool to elementary school, and
• their early academic and functional skills.
• Sample includes 3,104 children who were ages 3-5
  and were receiving special education services in
  2003-04.
• Produces statistical estimates that generalize to
  the national population of children with
  disabilities ages 3-5.

3
PEELS design, contd

• Data collection
• a one-on-one assessment of early academic skills,
  
• a parent telephone interview,
• mail questionnaires to childrens teachers and
  school-, district-, and state-level
  administrators, and
• indirect assessments of students social skills,
  problem behaviors, academic skills, and motor
  skills

4
Sample design

• Nationally representative sample of 223 LEAs (208
  in Wave 1, plus 15 added in Wave 2)
• LEAs stratified by
• enrollment size
• geographic region
• wealth

5
LEA participation

• Main sample
• 709 released for recruiting
• 245 agreed in 2001
• 189 participated in 2003-04 (Wave 1)
• Nonresponse sample
• 32 of 464 non-recruited LEAs selected
• 19 participated in 2003-04 (Wave 1)
• Supplemental sample to add 1 previously
  nonparticipating SEA
• 15 of 24 participated in 2004-05 (Wave 2)
• Total Sample 223 LEAs

6
Family participation

• Wave 1 main and nonresponse samples
• (ages 3-4 in 2003-04)
• 52,871 on sampling frame
• 5,330 selected
• 4,072 eligibility determined
• 2,906 agreed (81 of those eligible)

7
Family participation

• Wave 2 supplemental sample
• (ages 4-6 in 2004-05)
• 7,727 on sampling frame
• 542 selected
• 433 eligibility determined
• 198 agreed (63 of those eligible)
• Total sample 3,104 families

8
PEELS schedule
9
Unweighted response rates, by wave
Wave
1 2 3
SEA questionnaire 100 - -
LEA questionnaire 84 - -
Principal/program director questionnaire 73 77 56
Teacher questionnaire 79 84 84
Parent interview 96 93 88
Child assessment 96 94 93
English direct assessment 84 87 88
Alternate assessment 11 7 5
-Not applicable QED data were used to
supplement these data, bringing percentage of
children with some school data in waves 1, 2, 3
to 94, 95, 94, respectively
10
Demographic characteristics
Gender
Male 70.7
Female 29.3
Race/ethnicity
American Indian/Alaska Native 3.3
Asian 3.2
Black or African American 14.2
White 84.6
Pacific Islander 0.8
Hispanic, any race 21.0
NOTE Demographic and household characteristics
are for the unweighted sample. SOURCE U.S.
Department of Education, National Center for
Special Education Research, Pre-Elementary
Education Longitudinal Study (PEELS), Parent
Interview (Wave 1 for main and nonresponse
samples, Wave 2 for supplemental sample),
previously unpublished tabulation.
11
Household characteristics
Mothers education
ltH.S diploma 18.9
H.S. diploma or GED 31.0
Some college 29.4
4-year degree or higher 21.0
Household Income
20,000 or less 26.3
20,000-40,000 28.3
gt40,000 45.3
SOURCE U.S. Department of Education, National
Center for Special Education Research,
Pre-Elementary Education Longitudinal Study
(PEELS), Parent Interview (Wave 1 for main and
nonresponse samples, Wave 2 for supplemental
sample), previously unpublished tabulation.
12
Primary disabilities
Speech or language impairment 49.8
Developmental delay 26.4
Autism 6.7
Mental retardation 4.0
Learning disability 2.5
Other health impairment 2.4
Orthopedic impairment 1.6
Emotional disturbance 1.1
Low incidence disability 5.5
NOTE Low incidence includes visual impairment,
hearing impairment, deaf-blindness, traumatic
brain injury, multiple disabilities. SOURCE U.S.
Department of Education, National Center for
Special Education Research, Pre-Elementary
Education Longitudinal Study (PEELS), Early
Childhood and Kindergarten Teacher Questionnaires
(Wave 1 for main and nonresponse samples, Wave 2
for supplemental sample), previously unpublished
tabulation.
13
Developing a Severity-of-Disability Scale

14
Severity

• Under IDEA, children classified according to one
  of 13 disability categories
• Children in each category have a range of
  abilities
• Heterogeneity within categories often overlooked
• Categorical labels especially problematic for
  preschoolers

15
Approaches to measurement

• Pediatric health conditions (PEDI, WeeFim)
• Self-care, mobility, and social cognition
• Require a certain threshold of impairment
• Single disability conditions in children
• Hearing, vision, language, depression
• ICIDH (WHO, 1980)
• Published as a research tool
• Not widely adopted for use

16
Approaches to measurement

• ABILITIES Index (Simeonsson Bailey, 1991)
• Severity in 9 domains
• Audition
• Behavior and social skills
• Intellectual function
• Limbs
• Intentional communication
• Tonicity
• Integrity of physical health
• Eyes
• Structural status

17
Approaches to measurement (contd)

• Since its original publication, 6 new domains
  added (Simeonsson, 2006)
• Regulation of attention,
• Regulation of activity level, and
• Regulation feeling/emotions,
• Academic skills,
• Motivation, and
• Impulse control

18
ABILITIES index

• In Special Education Expenditures Project (SEEP)
  (Chambers et al., 2004)
• Federal disability category able to explain only
  10 of the variation in total expenditures
• In contrast, the ABILITIES Index alone was able
  to account for 40 of the variation in total
  educational expenditures for special education
  students
• Including the federal disability categories
  increased the variance accounted for by 2

19
Question of current study

• To what extent do functional markers of severity
  of childhood disability predict measurement of
  child outcomes?

20
Overall method

• Identify items from parent interview similar to
  ABILITIES domains
• Create composite domains where necessary
• Examine distribution of severity within the 15
  domains
• Examine regressions of the 15 domains on multiple
  outcomes
• Create short version of index using 6 domains
• Compare 6 domain and 15 domain versions of index
  to outcomes to select better version
• Compare 6 domain version of index with other
  indicators of severity

21
Outcome measures
Wave 1 data

• Cognitive
• PPVT-III
• Woodcock-Johnson III Letter-Word Identification
• Woodcock-Johnson III Applied Problems
• Social/Behavioral
• PKBS Social Competence
• PKBS Problem Behaviors
• Alternate Assessment
• ABAS Conceptual Domain
• ABAS Practical Domain
• ABAS Social Domain

22
Creating the severity measure

• Used parent report from Wave 1 CATI
• Created 4-point scale for each domain
  (normal/typical, mild, moderate, severe)
• 11 domains already with 4 point scales
• 4 levels of severity
• 4 domains with 3 point scales
• 1 ? 1, 2 ? 2, 3 ? 4 to preserve range

23
Creating the severity measure

• Some domains matched to a single question
• Use of arms (within limbs domain) How well does
  child use his/her arms and hands for things
  like throwing, lifting, or carrying?
• Motivation Some children try to finish things,
  even if it takes a long time. How much does this
  sound like child
• For domains with multiple questions, worked with
  ABILITIES Index author to best reflect original
  intention of domain

24
Example Combining Items to Create Communication
with Others
When CHILD talks to people he/she doesnt
know well, is he/she 1 Very easy to
understand 2 Fairly easy to understand 3
Somewhat hard to understand 4 Very hard to
understand 5 DOES NOT OR WILL NOT TALK AT ALL
Compared with other children about the same age,
how well does CHILD make his/her needs known
to you and others? Communication can be any form,
for example crying, pointing or talking. Would
you say he/she 1 Communicates just as well as
other children 2 Has a little trouble
communicating 3 Has a lot of trouble
communicating 4 Does not communicate at all?
1 Communicates just as well as other children
and very easy to understand 2 Some difficulties
communicating or being understood 3 Moderate
difficulties communicating or being understood 4
Does not communicate at all or very hard to
understand
25
ResultsPopulation weighted percentages (n
2,986)

• Audition, vision, use of arms, legs, regulation
  of emotions
• 80-95 in normal/typical category
• Inappropriate or unusual behavior, overall
  health, social skills, use of hands,
  understanding
• 50-60 in normal/typical, 20-30 mild

26
ResultsPopulation weighted percentages (n
2,986)

• Motivation, regulation of attention, regulation
  of activity level,
• 20-30 in normal/typical, 33-42 mild, 16-30
  severe
• Communicating with others
• 29 in normal/typical, 9 mild, 45 moderate, 17
  severe
• Intellectual function
• 10 in normal/typical, 43 mild, 32 moderate,
  14 severe

27
Regressions on outcome variables

• All predictors entered simultaneously examined
  beta weights and bivariate correlations between
  predictors and each outcome
• Significant predictors of at least three
  outcomescognition, communicating with others,
  understanding, overall health, and regulation of
  activity level
• Significant predictor of two outcomesregulation
  of attention
• The remaining severity domains generated a
  mixture of significant and nonsignificant
  associations
• No significant loading on any outcome variables
  use of hands and arms and use of legs

28
Creation of indices

• Index A, 15 items Sum of all domains
• Index B, 6 items cognition, communicating with
  others, understanding, overall health, regulation
  of activity, regulation of attention
• No significant differences in correlations
  between Index A and outcomes and Index B and
  outcomes

29
Index A and B correlations with outcomes
Index A 15 var Index B 6 var
PPVT -.32 -.36
Letter-Word Identification -.22 -.26
Applied Problems -.40 -.45
PKBS Social Skills Composite -.47 -.43
PKBS Problem Behavior Composite .35 -.35
ABAS Conceptual Domain -.53 -.46
ABAS Practical Domain -.53 -.43
ABAS Social Domain -.40 -.35
The associated p-values are less than .0001 for
all coefficients.
30
Final severity measure items

• Cognition
• Communicating with others
• Understanding
• Overall health
• Regulation of activity
• Regulation of attention

31
Distribution of final severity measure
32
Validation correlations between Index B and
other indicators

• Wave 1 Parent report Age at which children
  began receiving special education or therapy
  servicer (2,802) -.22, p lt .0001
• Wave 1 Teacher report Amount of modification
  needed to curriculum materialsr (248) .42, p
  lt .0001
• Wave 1 Teacher report Number of services the
  child receives in school r (2,014) .37, p lt
  .0001

33
Validation comparison of mean scores on severity
measure

• From teacher/parent declassification measure,
  Wave 1 Children remaining in special educationM
  13.2Children no longer receiving special
  education M 10.7 , p lt .0001
• From assessment measures, Wave 1Children taking
  the alternate assessment M 16.3Children
  completing the direct assessmentM 12.4, p lt
  .0001

34
Summary

• To what extent do functional markers of severity
  of childhood disability predict measurement of
  child outcomes?
• Of the 15 domains examined, most were significant
  predictors of at least two outcomes
• An index of only six variables was as effective
  as the longer version
• Severity was significantly correlated with
  intervention variables
• Severity differentiated children in two groups

35
Modeling Early Reading and Math Performance

36
Hierarchical analysis

• PEELS has children in naturally-occurring
  hierarchies
• time points within children
• children within districts
• districts

37

• What we want to know
• What factors relate to childrens cognitive
  growth over 3 years?
• What we have
• observations of PEELS children over 3 years.
• Yearly information about childs SES, health,
  severity, and services received
• Yearly measures of 3 academic outcomes
• Adapted Peabody Picture Vocabulary Test (PPVT)
• Woodcock-Johnson III Letter-Word Identification
• Woodcock-Johnson III Applied Problems

38
Problem How to make sense out of longitudinal
data

• Hierarchical data is clustered i.e., repeated
  measures are not independent observations.
  Standard regression assumes independent
  observations.
• Ordinary repeated measures analyses do not allow
  for missing time points or clustering.
• Repeated cross-sectional analyses ignore the
  growth of individual children. Mean growth is not
  the same as growth of individuals.

39
Solution Hierarchical linear modeling

• Data is modeled at 3 levels of hierarchy at the
  same time.
• Most of the clustering in the sample is accounted
  for, leading to correct statistical tests.
• Focus is on individual growth profiles. Modeling
  seeks to explain differences in growth between
  children.

40
Hierarchical structure of data
41
HLM model

• Level 1 Repeated observations within child
• is the outcome for individual i measured at
  wave t .
• is the age of the child at time at time t.
• is the growth intercept average achievement for
  the individual
• is the growth curve slope How much the outcome
  changes over years.
• is a random error term.

42
HLM modelLevel 2 Children nested within
districts.
is an intercept for child ji,
is the individual growth curve for child ji
are child factors that predict growth.
to
is the deviation from average achievement for
each child.
is the deviation from average growth for each
child.
43
HLM modelLevel 3 Districts

• Where,
• is the achievement status of district j ,
• is the grand mean of achievement, and
• is the district effect on achievement.

44
Hypothetical growth curve HLM

• Individual slopes estimated for high low
  severity children
• 
  
• 
  lt- Low
  severity

45

• Individual slopes estimated for high low
  severity children
• 
  
• 
  lt- Low
  severity
• 
  
  lt- High severity

46
Four Sets of Predictors

• Predictors are added in four sets of similar
  variables. These sets include
• SES e.g., mothers education, SES scale
• Severity e.g., severity scale, age services
• started
• Health e.g., childs general health, health
  scale
• Services e.g., time in a regular classroom,
  parent
• involvement scale

47
Predictors of PPVT growth
Predictor Factor Effect Prob
Intercept 229.55 .00
Slope 54.24 .00
Years in special education Severity -1.37 .01
Severity of disability scale Severity -.78 .00
Ease of transition Service -1.85 .01
Time in a regular classroom Service .03 .03
Accounted for 22 of PPVT growth
Note Controlled for age and cohort
48
Predictors of Letter-Word growth
Predictor Factor Effect Prob
Intercept 318.18 .00
Slope 37.76 .00
Years in high poverty school SES 2.10 .00
Mother had some education after HS SES 2.92 .00
Years in special education Severity -1.62 .00
Age services started Severity .09 .02
Severity of disability scale Severity -.69 .00
Ease of transition Service -2.61 .00
Time in regular classroom Service .08 .00
Parent involvement scale Service 1.49 .03
Accounted for 57 of Letter-Word growth
Note Controlled for age and cohort
49
Predictors of Applied Problems growth
Predictor Factor Effect Prob
Intercept 393.26 .00
Slope 24.00 .00
Household Income SES -1.26 .03
Child is Hispanic SES 1.53 .03
Mother had some education after HS SES 1.42 .02
Parent SES scale SES 1.38 .01
Problems with health Health 2.50 .00
Health scale Health 1.18 .05
Childs general health Health 2.49 .00
50
Predictors of Applied Problems growth (continued)
Predictor Factor Effect Prob
Years in special education Severity -1.02 .00
Age services started Severity .09 .00
Severity of disability scale Severity -.24 .02
Time in regular classroom Service .03 .00
Parent involvement scale Service .75 .06
Accounted for 12 of Applied Problems growth
Note Controlled for age and cohort
51
Summary

• Scores increased from 1½ to 2 standard deviations
  over 3 years
• SES, severity, health, service predictors
  accounted for 12 to 57 of growth

Percent of growth accounted for Percent of growth accounted for

PPVT 22
Letter-Word 57
Applied Problems 12
52
Summary (cont.)

• Service-related predictors of growth
• Time in a regular classroom
• Positively related to growth for all outcomes
• Parent involvement
• Positively related for 2 of 3 outcomes
• (Letter-Word Applied Problems)
• Ease of transition
• Positively related for 2 of 3 outcomes
• (PPVT Letter-Word)

53
Summary (cont.)

• Other predictors of growth
• Years child was in special education
• Predicted lower growth for all outcomes
• Parents rating of severity
• Predicted lower growth for all outcomes
• Mothers education
• Positively related to growth for 2 of 3 outcomes
  
• (Letter-Word Applied Problems)

54
WEBSITE WWW.PEELS.ORG