Application of Growth and Value-Added Models to WASL

1
Application of Growth andValue-Added Modelsto
WASL

• A Summary of Issues, Developments and Plans for
  Washington
• WERA Symposium on Achievement Growth Models
• June 2, 2006
• Joe Willhoft, OSPI

2
Overall Impressions

• Shaws Rule
• For every complex problem there is a simple
  solution that is wrong.
• -- G.B. Shaw
• Willhofts Corollary
• For every complex problem there is a correct
  solution that cant be understood.

3
Issues Associated with Growth and Value-Added
Models

• Students must be presented with off-grade-level
  items. Younger students may not even have
  studied them older students may not have studied
  them recently. Neither seems a fair
  representation of student performance.
• If the curriculum includes blocks of content that
  are not taught at or before the earlier grade
  level but are taught at the higher grade level,
  then the lower grade level test has questionable
  validity for inferences to the domain of the
  trait across the two grade levels.
• In using the scale, performance on
  off-grade-level items is estimated from
  performance on on-grade-level items, presenting a
  validity concern.
• Growths in different regions of a vertical scale
  developed across several grade levels are not
  comparable.
• It is possible that students in different grades
  achieve the same scores. However, their
  educational experiences are different and
  therefore, appropriate achievement level
  descriptions differ.

4
Issues Associated with Growth and Value-Added
Models (Cont.)

• Students can show negative growth. Since this is
  possible, given enough replications, it will
  happen. Explanations likely will be developed
  that depend on the differences between the
  content at the two grade levels, and that begs
  the question of why the two tests were put on the
  same scale.
• External achievement standards may be disordinal.
  For example, the cut score for proficient may
  be lower on the scale for grade five than it is
  for grade four. Since this can happen, given
  enough replications it will happen. Clearly some
  heroic fudging will be needed before the scale
  can be used.
• Students from different grade levels with the
  same score will not have the same growth
  expectations.

5
Additional Concerns with Growth and Value-Added
Models

• Record keeping systems must be more robust
• Missing records are usually not random
• Implementation of variables is inconsistent
  across units in VAM
• Modeling growth and VA is intuitively simple, but
  technically complex

6
All that may be true, but.

• We will have tests in Reading and Math across
  grades 3 through 8
• Parents, principals, superintendents,
  policymakers and the public at-large will not
  accept that we cannot or should not measure growth

7
Isnt Value-Added an Improvement over Growth
Models?

• This depends on audience and purpose
• Parents probably more interested in growth from
  one year to the next
• Policymakers probably more interested in
  improvements conditioned on input factors
  (value-added)
• More precise models are more complex and
  difficult to explain

8
How is ED Using Growth for AYP

• Improvement already used as Safe Harbor
• States invited to apply as pilots in December
  2005 (No more than 10 would be approved)
• In May, Secy Spellings approved North Carolina
  and Tennessee as pilot states
• Alaska, Arkansas, Arizona, Delaware, Florida, and
  Oregon will be given early consideration in the
  next round.
• A total of no more than 10 pilot states will be
  approved.

9
ED Use of Growth Models for AYP

• States required to meet the following
• All students proficient by 2014 with annual state
  goals to close ach. gap
• Annual ach. expectations based on grade-level
  proficiency, not on student background or school
  characteristics
• Schools accountable for achievement in reading/LA
  and mathematics
• All students are included in the
  assessment/accountability system
• Assessments in grades 3-8 and high school have
  been operational for more than one year, and have
  received approval through the NCLB peer review
  process for the 2005-06 school year.
• The assessment system must also produce
  comparable results from grade to grade and year
  to year
• Track student progress as part of the state data
  system and
• Continue to include student participation rates
  and student achievement as separate academic
  indicators in the state accountability system.

10
What Are OSPI Plans?

• 2005 Pilots in in grades 3, 5, 6, 8 and
  Operational forms in 4 and 7 included vertical
  forms
• Pilot item locations contained items from
  preceding or following grade levels
• National TAC requested technical review of
  scaling, growth and VAM
• OSPI contracted for development of technical
  treatment of the topic for NTAC (Available upon
  request)

11
What Are OSPI Plans?

• 2007 tests in 3 thru 8 will include vertical
  forms
• Pilot item locations will contain items from
  preceding grade levels
• Did not use 2006 per NTAC recommendation of new
  test
• Results of vertical scaling to be presented to
  NTAC in January 2008

12
What Are OSPI Plans?

• Hope to develop at least three-year spans
• 3-4 3-4-5 4-5-6 5-6-7 6-7-8 OR
• A 3-4-5 scale and a 6-7-8 scale
• May have to settle for paired grades 3-4, 4-5,
  etc.
• OSPI Technology shop is developing longitudinal
  database for tracking student scores across time
• Monitor developments in other states for ED
  approval to use Growth Models or VAM for NCLB

13
Questions/Discussion
14
Status Model(e.g., Adequate Yearly Progress
15
Improvement Model(e.g., Safe Harbor)
16
Growth Model(Simplified generic model)
17
Value-Added Model(Simplified generic model)
18
Isnt Growth Better than Status for Measuring
Student Achievement?

• Which Group(s) demonstrate the desirable
  achievement?