An Integrated Process of Setting Consistent Coherent Performance Standards

1
An Integrated Process of Setting Consistent
(Coherent) Performance Standards

• Liru Zhang, Delaware DOE
• Paper presented at the 2006 CCSSO Large-Scale
  Assessment Conference
• San Francisco, CA, 25-28 June, 2006

2
Background

• Under NCLB, the statewide assessment system will
  be the primary means for determining whether
  schools and school districts are making adequate
  yearly progress (AYP) toward educating students
  to high standards. (Peer Review Guidance, 2004)

3
Background

• In a well functioning educational system,
  curriculum and instructional practice are
  articulated across grades. Performance standards
  for a subject area represent the expectations
  that students should know and be able to do from
  grade to grade. Students who meet or exceed the
  expectations at a given grade have the requisite
  knowledge and skills. If they maintain continued
  effort, receive instructional support, and
  demonstrate reasonable growth, they should meet
  the expectations in the next grade.

4
Why Set Consistent Performance Standards?

• Under the requirements of NCLB, states must
  measure student progress yearly to demonstrate
  compliance to the goal of adequate yearly
  progress. Therefore, performance standards set at
  contiguous grades must demonstrate a new and
  higher level of consistency across grades than
  ever before (Mitzel, 2004).
• With the advent of AYP, it is essential that
  performance standards across grades be reasonably
  consistent. Performance standards that become
  suddenly stringent in a particular grade level
  may affect the percentage of students at or above
  the proficient level and thus adversely impact
  AYP ratings for schools and states (Huynh and
  Schneider, 2004).

5
Why Set Consistent Performance Standards?

• To track student progress across grades and to
  predict student performance from one grade to the
  next
• To have reasonably stable data for AYP and
  accountability across grades and over time
• To improve the interpretation of assessment
  results for teachers, parents, and general public
  

6
How to Set Consistent Performance Standards?

• In 2004, the Delaware Department of Education
  started a project of revisiting and resetting the
  performance standards for the statewide
  assessment (DSTP) in Reading, Writing, and
  Mathematics for grade 2 through grade 10. There
  were three connected components (alignment and
  vertical alignment, performance descriptors, and
  revisiting/setting cut-scores). The results of
  each component contributed to the consistency of
  performance standards.

7
Delaware Experience Conducting Alignment and
Vertical Alignment Studies

• Webbs criteria (2002) for alignment was used to
  determine the relationships between the
  assessments and the state Content Standards and
  Grade-Level-Expectations. The alignment results
  provided information about misalignment and
  developed a better understanding of the
  limitations of using assessment data to develop
  the performance descriptions.
• To achieve a reasonable level of consistency of
  performance standards and meaningful
  interpretation of student achievement from grade
  to grade, a vertical alignment was conducted in
  English language arts and Mathematics (Wise,
  2005). The vertical alignment targeted on (a)
  the nature of content linkages from grade to
  grade and (b) the quality of the linkages.

8
Delaware Experience Conducting Alignment and
Vertical Alignment Studies

• The Grade-Level-Expectations (GLE) were used in
  the vertical alignment which examined the
  cross-grade alignment for content objectives and
  performance expectations. The analyses
  identified misalignment and suggested improving
  the clarity of some expectations and the
  connections of expectations from grade to grade.
• The results of vertical alignment also provided
  useful information for the development of the
  Performance Level Descriptors (PLD).
• The alignment studies created the content basis
  for setting consistent/coherent performance
  standards.

9
Delaware Experience Developing Performance
Level Descriptors

• The Performance Level Descriptors (PLD) were
  primarily based on the Content Standards and
  Grade-Level-Expectations. Ordered-item test
  booklets and item-level statistics provided
  empirical data about the assessment in the
  process of developing PLD.
• The Performance Level Descriptors
• - mirror the grade by grade expectations
• - demonstrate articulated curriculum and
  standards
• - reveal instructional practice and student
  performance
• - reflect reasonable expectations of student
  growth

10
Delaware Experience Developing Performance
Level Descriptors

• Experienced teachers and content specialists were
  assigned to grade-cluster groups (e.g., 2-4, 5-7,
  8-10) to facilitate communications between grade
  levels and aligned the performance descriptions
  from grade to grade within a grade-cluster.
• The alignment of PLD across all grades (2 to10)
  achieved the
• - Consistency across grades
• - Consistency across performance levels within
  grade
• - Consistency in each performance level across
  grades
• - Consistency across tests

11
Delaware Experience Process for Setting
Performance Standards

• Established review panels
• Trained Panels
• Identified cross-grade performance patterns
• Reviewed Process
• Interpolated and extrapolated cut-scores for
  grades 2, 4, 6, 7, and 9

12
Delaware Experience Establishing Review Panels

• Established panels for revisiting/setting
  performance standards with overlapping grade
  level spans. For instance, each panel consisted
  of on-grade teachers and teachers from adjacent
  grades.
• Used some common panelists who participated in
  multiple activities of the project, such as
  alignment studies, PLD development, review of
  existing cut-scores, and set cut-scores.
• Had a cross-grade discussion to facilitate the
  consistency of performance standards.

13
Delaware Experience Panel Training

• The majority of the first day was spent for
  training. It consisted of three sections (1)
  General training (2) Content-related training
  and (3) Ordered-item test booklets.
• The general training was conducted for all panels
  together in order to minimize the variations.
• A pre-training was conducted for Table
  Facilitators to develop a better understanding of
  the process, the training materials, and the
  ordered-item test booklets a post-session was
  held with Consultants and Table Facilitators to
  share the information across subject areas and
  make adjustment accordingly.

14
Delaware Experience Panel Training

• The first session focused on why and how to set
  consistent/coherent cut-scores. The trainer
  demonstrated the student performance patterns and
  the impact data using the existing cut-scores,
  described the process and methodology, and
  explained how to identify the cross-grade
  performance patterns.
• The second session was a content-related training
  which focused on the Grade-Level-Expectations and
  the Performance Level Descriptors with sample
  items from the state assessments.
• The third session focused on reviewing the test
  forms and training for the ordered-item test
  booklets. The items were rearranged based on
  item difficulty. The content and cognitive
  categories of each item, the corresponding scale
  scores, and existing cut-scores were also
  provided.

15
Delaware Experience Identifying the Cross-Grade
Performance Patterns

• The impact data, that is, the percent of students
  at or above the cut-score for Meets the Standard
  and the knowledge and skills expected to be
  mastered for this level was introduced to the
  panels as part of the training. More
  specifically, for each percentile rank the
  associated scale score, number of items in the
  ordered item booklet, and the number correct
  score were included. Using existing cut-scores
  for grades 3, 5, 8, and 10, panels identified a
  decreasing trend of percent of students meeting
  the standard in mathematics and a slightly
  decreasing/flat trend in reading.
• The identified performance patterns across grade
  levels would serve as the guideline in the
  process of reviewing and setting cut-scores.

16
Delaware Experience Identifying the Cross-Grade
Performance Patterns

• The discussion was the process of collecting
  evidence to support the existing performance
  patterns or alternative patterns
• - Do these observed patterns seem reasonable?
• - Do these trends represent what participants
• experienced in the classrooms around the
  state?
• - Do their observations support this data?
• The discussion was first conducted in grade-group
  and then in the content-group across grades of 3,
  5, 8, and 10.

17
Delaware Experience Identifying the Cross-Grade
Performance Patterns

• In reading, the increases of the percentage of
  items that are of a higher cognitive level as
  students proceed up the grades was cited as a
  reasonable explanation as to the slight decline
  in student performance across grades with the
  existing cut-scores for meeting the standard.
• The mathematics panels had a general consensus
  that rising standards, or rising expectations,
  across grade levels were reasonable and desirable
  for the decreasing patterns for Delaware
  students.

18
Delaware Experience Review Process

• Independent review of test item and existing
  cut-scores followed by grade-group discussion.
• Cross-grade discussion was arranged before the
  final individual decisions.
• Performance level descriptors and
  grade-level-expectations provided the framework
  during the discussions on the cut-scores.
• The panels were asked to provide evidence based
  on the PLD and the GLE for any adjustments of
  existing cut-scores or preliminary cut-scores.

19
Delaware Experience Interpolation and
Extrapolation

• A simple interpolate/extrapolate procedure was
  applied to the proposed cut-scores for grades 2,
  4, 6, 7, and 9 using revised cut-scores for
  grades 3, 5, 8, and 10.
• The interpolated cut-scores were smoothed by
  comparing the percentile locations of cut-scores
  across grades and using the cross-grade
  performance patterns and impact data across
  grades to achieve consistency.
• The preliminary cut-scores were reviewed by the
  panels.

20
Final Cut-Scores for Reading
21
Impact Data for Reading Percent of Students at
or above Meets the Standard
22
Final Cut-Scores for Mathematics
23
Impact Data for Mathematics Percent of Students
at or above Meets the Standard
24
Final Cut-Scores for Writing