Performance

1
Performance
Standards
A Map to the Future
Kentucky Department of Education
2
June 2001
Commonwealth Accountability Testing System
Kentucky Board of Education
C A T S
3
June 2001
Commonwealth Accountability Testing System
Kentucky Board of Education
4
New Student Performance Standards
Why ?
5
1998
Commonwealth Accountability Testing System
C A T S
6
Two Types of Standards
Content Standards
What Content
7
Two Types of Standards
Performance Standards
how well
8
National Technical Advisory Panel for Assessment
and Accountability
9
KIRIS
National Technical Advisory Panel for Assessment
and Accountability
10
CATS
The New Test Required New Standards
11
KIRIS
1992
1992
One Method to set Performance Standards
Modified Angoff Method
12
KIRIS
1992
1992
One Method to set Performance Standards
Modified Angoff Method
Six Teachers from each content area
Three Open Response Items from KIRIS
13
KIRIS
1993
1993
Same Method to set Standards
Modified Angoff Method
Six Teachers from each content area
Arts Humanities Practical Living/Vocational
Studies
14
KIRIS Descriptions of NAPD
Novice

• Student may recall some relevant information but
  shows minimal understanding of core concepts.
• Student communication is unclear and ineffective

15
KIRIS Descriptions of NAPD
Apprentice
Student demonstrates a knowledge of many core
concepts and skills but cannot consistently apply
knowledge. Student communicates directly and
reasonably effectively but sometimes response
lacks substance or detail
16
KIRIS Descriptions of NAPD
Proficient
Student demonstrates understanding of major
concepts even though she/he overlooks or
misunderstands some less obvious ideas or
details. Student can apply core concepts and
skills to solve problems. Student makes
connections among major concepts. Student
communicates ideas effectively.
17
KIRIS Descriptions of NAPD
Distinguished
Student demonstrates in-depth understanding of
concepts and/or processes described in Kentuckys
academic expectations. Student can solve
challenging problems using innovative and
efficient strategies. Student communicates
complex ideas effectively. Student makes
insightful interpretations or extensions
(generalizations, applications, and analogies).
18
KIRIS Descriptions of NAPD
No matter what the grade level or content area
The SAME SET of definitions applied
19
KIRIS Standards
Proficient ?
20
KIRIS Standards
Instruction
Assessment
21
Steps for Setting Performance Standards
Draft Performance Descriptions
22
Steps for Setting Performance Standards
23
Steps for Setting Performance Standards
Draft Performance Descriptions
Jaeger-Mills
Contrasting Groups
CTB Bookmark
Synthesis
KBE Decision
24
NOVICE APPRENTICE PROFICIENT DISTINGUISHED
25
Each Content Area
All Assessment Grades
26
Performance Standards For Writing
Not Revised
27
Draft Performance Descriptions
28
Contrasting Groups Students Classroom Work
29
Contrasting Groups
Using student classroom work and the draft
descriptions,
Classify each students work as Novice,
Apprentice, Proficient, Distinguished
30
Contrasting Groups
Students Classroom Work
Distinguished
Proficient
Borderline Prof./Dist.
Apprentice
Borderline App./Prof.
Novice
Borderline Nov./App.
31
Jaeger-Mills Student Answers 2000 Kentucky Core
Content Test
32
Jaeger-Mills
Student Answers 2000 Kentucky Core Content Test
Distinguished
Proficient
Apprentice
Novice
33
CTB Bookmark 2000 Kentucky Core Content Test
TEST ITEMS
34
CTB Bookmark
Spring 2000 KCCT assessment items
35
CTB Bookmark
Hardest
Spring 2000 KCCT assessment items Ordered Item
Book
Easiest
36
CTB Bookmark
Hardest
Spring 2000 KCCT assessment items Ordered Item
Book
Easiest
37
CTB Bookmark
Hardest
Spring 2000 KCCT assessment items Ordered Item
Book
Easiest
38
CTB Bookmark
Hardest
Easiest
39
CTB Bookmark
40

• Synthesis of Recommendations From All Three
  Procedures

Recommend cut-scores in each content area at each
grade level assessed
Synthesis
All Method Groups represented
41
18 Committees
42
(No Transcript)
43
3 Committees
44
6 Committees
Each Content Area Across Grades
45
Steps for Setting Performance Standards
Draft Performance Descriptions
Jaeger-Mills
Contrasting Groups
CTB Bookmark
Synthesis
KBE Decision
46
18 Months
1600 Teachers
47
KDE Web Survey
48
Raw Scores Scale Scores Cut Scores
49
Raw Scores
6 open-response items
24 multiple-choice items
2 open-response items
8 multiple-choice items
50
Raw Scores - Range
0 - 24
0 - 24
51
Raw Scores
Student 1
(out of 24)
16 multiple-choice items
52
Raw Scores
Student 1
17
X 2 34
(out of 24)
16 multiple-choice items
Raw Score 50
53
Raw Scores - Range
RANGE FROM 0 TO 72
Open response 48 Raw Score Points
(24 X 2 48)
Multiple Choice 24 Raw Score Points
72 Raw Score Points
54
Raw Scores, Scale Scores Cut Scores
Raw Scores, Scale Scores Cut Scores
55
Scale Scores
Raw Score
Scale Score
Student
Form
1 50 Form 1 586 2 50 Form 6 583 3 50 Form
1 586 4 69 Form 1 691 5 65 Form
2 657 6 38 Form 4 536 7 39 Form 3 536 8
70 Form 3 680
1 50 Form 1 586 2 50 Form 6 583
56
Why do we have different test forms?
57
(No Transcript)
58
(No Transcript)
59
Raw Scores
FORM 6
FORM 4
60
Scale Scores
FORM 4
FORM 6
61
Scale Scores
A C T
S A T
FORM 4
FORM 6
62
Scale Scores
63
Scale Scores
Raw Score
Scale Score
Student
Form
1 50 Form 1 586 2 50 Form 6 583 3 50 Form
1 586 4 69 Form 1 691 5 65 Form
2 657 6 38 Form 4 536 7 39 Form 3 536
1 50 Form 1 586 2 50 Form 6 583 3 50 Form 1 586
1 50 Form 1 586 2 50 Form 6 583 3 50 Form
1 586 4 69 Form 1 691 5 65 Form
2 657 6 38 Form 4 536 7 39 Form 3 536
64
Raw Scores, Scale Scores Cut Scores
Raw Scores, Scale Scores Cut Scores
65
CUT-SCORES/CUT-POINTS
Scale score from 325 - 800
66
CUT-SCORES/CUT-POINTS
?
?
?
67
High School Reading
68
High School Reading
Student Scale Score
Distinguished
612
69
(No Transcript)
70
5th Grade Math
71
Reading
72
Science
73
Accountability Index
Accountability Index
74
Accountability Index




Incremental Credits
75
High School Reading
454
537
Proficient
Distinguished
Apprentice
Novice
Incremental Credits
76
High School Reading
454
537
326
325
Proficient
Distinguished
Apprentice
Novice
Incremental Credits
77
High School Reading

Weight
Calculation
0
13
Proficient
Distinguished
Total
67.72
67.7
Estimated Reading Academic Index
78
Accountability Index
Non Academic Data Attendance Retention Dropout Tra
nsition

Schools Accountability Index



79
Raw Scores, Scale Scores Cut Points
80
STANDARDS
IF Apprentice/Proficient Cut-Point
IF Proficient/ Distinguished Cut-Point
Proficient
81
STANDARDS
IF Apprentice/Proficient Cut-Point
IF Proficient/ Distinguished Cut-Point
Proficient
82
(No Transcript)
83
(No Transcript)
84
KIRIS CATS
Student Work
Survey
Different set of definitions for each content
area
Same set of definitions for all content areas
85
June 5th 2001
Kentucky State Board of Education accepts
teachers recommendations
86
Kentucky Board of Education
Performance Standards
Kentucky Department of Education