Measuring Early Literacy Skills and Monitoring Progress: A Comparison of Online and TeacherAdministe

1
Measuring Early Literacy Skills and Monitoring
Progress A Comparison of Online and
Teacher-Administered Assessments

• James R. McBride
• Renaissance Learning, Inc.
• Jim Ysseldyke
• Minnesota Center for Reading Research
• University of Minnesota
• Presented at the 36th National Conference on
  Large Scale Assessment, San Francisco, June 27,
  2006

2
Overview

• Looked at reliability, validity, and usability of
  four assessments used in Reading First programs
• Reliability (test-retest)
• Validity (convergent and discriminant)
• Utility (cost and ease of use)

3
Reading First Requires Assessments for

• Screening
• Diagnosis
• Progress Monitoring
• Outcomes Assessment

4
Skills that Need to be Assessed

• One or more of the 5 National Reading Panel
  Identified Skills
• Phonemic Awareness (PA)
• Phonics (PH)
• Vocabulary (VO)
• Reading Comprehension (CO)
• Fluency (FL)

5
Test Selection

• States are Free to Choose
• Must meet Federal Approval
• Tests are unlikely to be approved if they are not
  on state lists, most of which are derived from or
  based on the list of tests viewed favorably in a
  2002 report by an IDEA panel/University of Oregon
  project (Kameenui, 2002)

6
The Argument

• Reading First approved assessments are
  meritorious, but some require extensive testing
  time, or teacher time to score, record, and
  interpret.
• Educational professionals whine significantly
  about the amount of testing time that comes with
  Reading First.
• Schools want the most bang (technically adequate
  and instructionally sensitive tests) for the buck.

7
Assessments Studied

• DIBELS (Dynamic Indicators of Basic Early
  Literacy Skills), a system of individually
  administered tests of various aspects of fluency,
  published by the University of Oregon and/or
  Sopris West.
• GRADE (Group Reading Assessment Diagnostic
  Evaluation), a battery of group-administered
  early literacy assessments, published by American
  Guidance Service.

8
Assessments Studied

• SEL (STAR Early Literacy), a computerized
  adaptive test of early literacy skills published
  by Renaissance Learning.
• TPRI (Texas Primary Reading Inventory), an early
  reading instrument consisting of separate
  screening and inventory tests, developed by the
  Texas Education Agency and the University of
  Texas.

9
One-on-one Assessments

• DIBELS is a system of timed, individually
  administered tests
• Two formats available paper booklets and
  handheld computers, but we could not identify a
  school using the handheld format for the study
• TPRI is an individually administered instrument
  consisting of screening and inventory tests
  most tests are not timed
• Two formats available paper booklets and
  handheld computer versions, both included in the
  study

10

• Two models for administering one-on-one tests
• In class, classroom teacher administers
• Benefit Teachers see student performance
  first-hand
• Drawback Very time consuming. Testing a few
  students a day took between one and four weeks.
  Also, testing is frequently interrupted by other
  students in class.
• Pull out, team of teachers/consultants
  administers
• Benefit More time-efficient a school can test
  most students in a few days
• Drawback Teachers dont see their students
  performance first-hand. Also, some reports of
  test anxiety if a student is testing with someone
  unfamiliar to them.

11
STAR Early Literacy

• A computer adaptive test that can be administered
  in a group setting
• 25 adaptively selected test items
• from either five (Grade K) or all seven (Grades 1
  and 2) literacy domains represented in the item
  bank.
• Initial item difficulty level based on grade
  placement
• ability estimate updated following each item
  response.
• Items 2 25 selected adaptively, based on the
  updated ability estimates.
• SEL reports a single scale score, plus
  proficiency scores in each of the seven domains
  and 41 sub-domains.

12
Assessments at the Grade Levels See handout
13
Research Questions

• How do the measures compare in terms of
• Correlations with the NRP components
• Reliability
• Convergent validity
• Discriminant validity
• What is the comparative usefulness of the
  measures?

14
Subjects

• Students in K, 1, and 2 in
• 8 districts
• 7 states

15
Subjects

• N 633
• 51 female
• 75 white, 14 black, 7 Hispanic, 4 Asian or
  Native American
• 29 eligible for free/reduced price lunch
• 20 Title 1
• 6 special education
• Four ESL/ LEP students
• 10 gifted/talented

16
The Design

• Test administration Oct Dec 2004
• All students were to take all 4 assessments
• DIBELS and SEL were to be given twice
• Planned analysis
• Correlations between subtests, by grade
• Factor analysis
• To confirm the 5 NRP components
• To establish construct validity of SEL

17
Selected Results

• Correlations among all assessments/subtests
• See handout
• Correlations of SEL with NRP component measures
• Reliability of SEL and DIBELS subtests
• Convergent-discriminant validity of the measures
• Utility of the measures

18
Convergent and Discriminant Validity Grade 1
19
Convergent Validity of DIBELS, GRADE, SEL
and TPRI Grade 1
20
Discriminant Validity of DIBELS, GRADE, SEL
and TPRI Grade 1
21
Reliability by Grade and OverallSTAR Early
Literacy and DIBELS Tests (less Oral Reading
Fluency)
22
Utility/Usefulness-- Cost-- Ease of use

23
Cost Figures for Criterion-Referenced Tests
24
Cost Figures for Norm-Referenced Tests
25
What are the true costs of these early literacy
assessments?

• Resource costs, including money time, are
  critical for determining an assessments
  usability
• Study conducted in 2005 by Christensen Associates
  (an independent economics consulting firm) to
  document the costs of STAR Early Literacy,
  DIBELS, and TPRI

26
Study Methodology

• Interviews conducted with seven schools using
  STAR Early Literacy plus either DIBELS or TPRI
• Two categories of costs measured
• Direct costs, annual over 3 years
• testing materials, licensing fees, and/or fees
  for access to scoring and reporting services
• Average respondent school (268 students, 15
  classes, 18 students per class)
• Opportunity costs
• time to administer, score, and report results
  time that could be spent on instruction if
  testing was not taking place

27

• Administrators were asked to provide costs for
• Purchasing tests and materials
• Hiring substitutes or others to assist with test
  administration
• Teacher salary
• Teachers other staff were asked to
• Quantify amount of time spent per student and per
  class for each assessment preparing,
  administering, using results
• Reflect on the usefulness of each assessments
  results and the positive negative features of
  the usability of each test
• Costs for training were not included in the
  analysis because they varied widely among the
  schools in the study
• SEL requires comparatively very little
  training/experience

28
Analysis

• Direct costs collected for each test and averaged
• Because direct costs are a function of school
  size, an average school was used to summarize
  costs based on the schools in the study
• Opportunity costs calculated based on average
  time spent by teachers and staff
• Teacher average salary benefits data used to
  assign a dollar value to the costs

29
Results

• Direct Costs

Direct Costs for Assessment Materials

Annual, over three years

Direct Costs, annually

STAR

DIBELS

TPRI




Early Literacy

paper

paper

handheld







Per test kit/software

1,299

69

150

167

annual, over three years

433

1,035

2,250

2,505

Per student fees

0.39

1.00

0

6.50

average annual fees per

105

268

0

1,739

school year

Total Di
rect Costs / Year

538

1,303

2,250

4,244


30

• Opportunity Costs (Time)

Opportunity Costs


Based on average respondent school
Opportunity Costs

STAR

DIBELS

TPRI

(per round of testing)

Early Literacy

paper

handheld

paper







Per student (minutes)

15

10.56

17.80

27.80

per classroom (mins.)

30

190

320

500

Per school (mins.)

450

2851

4,806

7,506

Per school (hours)

7.5

47.5

80.1

125.1

Avg. hourly compensation

28.91

28.91

28.91

28.91

Total Costs per round

434

1,374

2,315

3,616


31

• Total costs (direct opportunity)

32

• Total costs (direct opportunity)

Total per student costs per year
50
48.87
40
41.75
30
20
20.24
15.41
14.70
10
6.86
0
STAR Early
DIBELS
TPRI
TPRI (paper)
GRADE
GRADE
Literacy
(paper)
(handheld)
(manual)
(scanner)
33
Discussion

• This study is among the first to compare
  assessments used in Reading First programs with
  one another, and to focus on the validity of the
  assessments as measures of the five important
  literacy skills identified in the National
  Reading Panels 2000 report.
• The data suggest that STAR Early Literacy scores
  correlate as highly or more highly with
  NRP-relevant measures than these current Reading
  First assessments correlate among themselves.

34

• Almost all of the assessments subtests were
  moderately to highly intercorrelated.
• SEL is about as highly intercorrelated with the
  other batteries subtests as they are with one
  another. E.g., average correlations at Grade 2
• STAR Early Literacy 0.58
• DIBELS 0.55
• GRADE 0.61
• TPRI 0.50 .

35

• For NRP components, SEL correlates about as
  highly with the relevant DIBELS, GRADE, and TPRI
  subtests as they do with one another.
• SEL also correlates highly with fluency so did
  GRADE.
• Strong correlations of non-fluency measures with
  fluency suggest that the tests of fluency,
  comprehension, vocabulary, and word reading are
  measuring the same underlying construct.

36
Conclusions

• STAR Early Literacy, a computerized adaptive
  test, has comparable technical adequacy to paper
  and pencil group- or teacher-administered
  assessments of Reading First skills.
• Its
• Reliable
• Valid
• Efficient test administration, scoring, and
  report preparation in 10 minutes

37
Conclusions

• STAR Early Literacy (SEL) is the most cost
  effective early literacy assessment among those
  studied
• DIBELS is about 3 times the cost of SEL TPRI is
  about 6-7 times the cost of SEL
• Most teachers felt that reports contained the
  most information of the three and were most
  suitable for parent conferences

38
Conclusions

• Teachers also appreciated the immediacy of
  results from SEL
• Paper versions of DIBELS TPRI need to be
  hand-scored and hand-entered into online
  reporting systems some teachers reported that
  they would not see their results for several
  weeks
• The handheld computer version of TPRI was more
  efficient than the paper version
• Overall, teachers felt that the assessments were
  generally consistent with each other, but that
  SEL was by far the most efficient

39

• For information
• About STAR Early Literacy, go to
  www.renlearn.com.
• About the presentation, contact
• jmcbride_at_renlearn.com
• or
• emstickn_at_renlearn.com