Title: An Integrated Process of Setting Consistent Coherent Performance Standards
1An Integrated Process of Setting Consistent
(Coherent) Performance Standards
- Liru Zhang, Delaware DOE
- Paper presented at the 2006 CCSSO Large-Scale
Assessment Conference - San Francisco, CA, 25-28 June, 2006
2Background
- Under NCLB, the statewide assessment system will
be the primary means for determining whether
schools and school districts are making adequate
yearly progress (AYP) toward educating students
to high standards. (Peer Review Guidance, 2004)
3Background
- In a well functioning educational system,
curriculum and instructional practice are
articulated across grades. Performance standards
for a subject area represent the expectations
that students should know and be able to do from
grade to grade. Students who meet or exceed the
expectations at a given grade have the requisite
knowledge and skills. If they maintain continued
effort, receive instructional support, and
demonstrate reasonable growth, they should meet
the expectations in the next grade.
4Why Set Consistent Performance Standards?
- Under the requirements of NCLB, states must
measure student progress yearly to demonstrate
compliance to the goal of adequate yearly
progress. Therefore, performance standards set at
contiguous grades must demonstrate a new and
higher level of consistency across grades than
ever before (Mitzel, 2004). - With the advent of AYP, it is essential that
performance standards across grades be reasonably
consistent. Performance standards that become
suddenly stringent in a particular grade level
may affect the percentage of students at or above
the proficient level and thus adversely impact
AYP ratings for schools and states (Huynh and
Schneider, 2004).
5Why Set Consistent Performance Standards?
- To track student progress across grades and to
predict student performance from one grade to the
next - To have reasonably stable data for AYP and
accountability across grades and over time - To improve the interpretation of assessment
results for teachers, parents, and general public
6How to Set Consistent Performance Standards?
- In 2004, the Delaware Department of Education
started a project of revisiting and resetting the
performance standards for the statewide
assessment (DSTP) in Reading, Writing, and
Mathematics for grade 2 through grade 10. There
were three connected components (alignment and
vertical alignment, performance descriptors, and
revisiting/setting cut-scores). The results of
each component contributed to the consistency of
performance standards.
7Delaware Experience Conducting Alignment and
Vertical Alignment Studies
- Webbs criteria (2002) for alignment was used to
determine the relationships between the
assessments and the state Content Standards and
Grade-Level-Expectations. The alignment results
provided information about misalignment and
developed a better understanding of the
limitations of using assessment data to develop
the performance descriptions. - To achieve a reasonable level of consistency of
performance standards and meaningful
interpretation of student achievement from grade
to grade, a vertical alignment was conducted in
English language arts and Mathematics (Wise,
2005). The vertical alignment targeted on (a)
the nature of content linkages from grade to
grade and (b) the quality of the linkages.
8Delaware Experience Conducting Alignment and
Vertical Alignment Studies
- The Grade-Level-Expectations (GLE) were used in
the vertical alignment which examined the
cross-grade alignment for content objectives and
performance expectations. The analyses
identified misalignment and suggested improving
the clarity of some expectations and the
connections of expectations from grade to grade. - The results of vertical alignment also provided
useful information for the development of the
Performance Level Descriptors (PLD). - The alignment studies created the content basis
for setting consistent/coherent performance
standards.
9Delaware Experience Developing Performance
Level Descriptors
- The Performance Level Descriptors (PLD) were
primarily based on the Content Standards and
Grade-Level-Expectations. Ordered-item test
booklets and item-level statistics provided
empirical data about the assessment in the
process of developing PLD. - The Performance Level Descriptors
- - mirror the grade by grade expectations
- - demonstrate articulated curriculum and
standards - - reveal instructional practice and student
performance - - reflect reasonable expectations of student
growth -
10Delaware Experience Developing Performance
Level Descriptors
- Experienced teachers and content specialists were
assigned to grade-cluster groups (e.g., 2-4, 5-7,
8-10) to facilitate communications between grade
levels and aligned the performance descriptions
from grade to grade within a grade-cluster. - The alignment of PLD across all grades (2 to10)
achieved the - - Consistency across grades
- - Consistency across performance levels within
grade - - Consistency in each performance level across
grades - - Consistency across tests
11Delaware Experience Process for Setting
Performance Standards
- Established review panels
- Trained Panels
- Identified cross-grade performance patterns
- Reviewed Process
- Interpolated and extrapolated cut-scores for
grades 2, 4, 6, 7, and 9
12Delaware Experience Establishing Review Panels
- Established panels for revisiting/setting
performance standards with overlapping grade
level spans. For instance, each panel consisted
of on-grade teachers and teachers from adjacent
grades. - Used some common panelists who participated in
multiple activities of the project, such as
alignment studies, PLD development, review of
existing cut-scores, and set cut-scores. - Had a cross-grade discussion to facilitate the
consistency of performance standards.
13Delaware Experience Panel Training
- The majority of the first day was spent for
training. It consisted of three sections (1)
General training (2) Content-related training
and (3) Ordered-item test booklets. - The general training was conducted for all panels
together in order to minimize the variations. - A pre-training was conducted for Table
Facilitators to develop a better understanding of
the process, the training materials, and the
ordered-item test booklets a post-session was
held with Consultants and Table Facilitators to
share the information across subject areas and
make adjustment accordingly.
14Delaware Experience Panel Training
- The first session focused on why and how to set
consistent/coherent cut-scores. The trainer
demonstrated the student performance patterns and
the impact data using the existing cut-scores,
described the process and methodology, and
explained how to identify the cross-grade
performance patterns. - The second session was a content-related training
which focused on the Grade-Level-Expectations and
the Performance Level Descriptors with sample
items from the state assessments. - The third session focused on reviewing the test
forms and training for the ordered-item test
booklets. The items were rearranged based on
item difficulty. The content and cognitive
categories of each item, the corresponding scale
scores, and existing cut-scores were also
provided. -
15Delaware Experience Identifying the Cross-Grade
Performance Patterns
- The impact data, that is, the percent of students
at or above the cut-score for Meets the Standard
and the knowledge and skills expected to be
mastered for this level was introduced to the
panels as part of the training. More
specifically, for each percentile rank the
associated scale score, number of items in the
ordered item booklet, and the number correct
score were included. Using existing cut-scores
for grades 3, 5, 8, and 10, panels identified a
decreasing trend of percent of students meeting
the standard in mathematics and a slightly
decreasing/flat trend in reading. - The identified performance patterns across grade
levels would serve as the guideline in the
process of reviewing and setting cut-scores.
16Delaware Experience Identifying the Cross-Grade
Performance Patterns
- The discussion was the process of collecting
evidence to support the existing performance
patterns or alternative patterns - - Do these observed patterns seem reasonable?
- - Do these trends represent what participants
- experienced in the classrooms around the
state? - - Do their observations support this data?
- The discussion was first conducted in grade-group
and then in the content-group across grades of 3,
5, 8, and 10.
17Delaware Experience Identifying the Cross-Grade
Performance Patterns
- In reading, the increases of the percentage of
items that are of a higher cognitive level as
students proceed up the grades was cited as a
reasonable explanation as to the slight decline
in student performance across grades with the
existing cut-scores for meeting the standard. - The mathematics panels had a general consensus
that rising standards, or rising expectations,
across grade levels were reasonable and desirable
for the decreasing patterns for Delaware
students.
18Delaware Experience Review Process
- Independent review of test item and existing
cut-scores followed by grade-group discussion. - Cross-grade discussion was arranged before the
final individual decisions. - Performance level descriptors and
grade-level-expectations provided the framework
during the discussions on the cut-scores. - The panels were asked to provide evidence based
on the PLD and the GLE for any adjustments of
existing cut-scores or preliminary cut-scores.
19Delaware Experience Interpolation and
Extrapolation
- A simple interpolate/extrapolate procedure was
applied to the proposed cut-scores for grades 2,
4, 6, 7, and 9 using revised cut-scores for
grades 3, 5, 8, and 10. - The interpolated cut-scores were smoothed by
comparing the percentile locations of cut-scores
across grades and using the cross-grade
performance patterns and impact data across
grades to achieve consistency. - The preliminary cut-scores were reviewed by the
panels.
20Final Cut-Scores for Reading
21Impact Data for Reading Percent of Students at
or above Meets the Standard
22Final Cut-Scores for Mathematics
23Impact Data for Mathematics Percent of Students
at or above Meets the Standard
24Final Cut-Scores for Writing