Accountability: Where Are We Going?

1
Accountability Where Are We Going?

• Martha Musser, Coordinator
• NYS Education Department

2
This may be the Dawning of the Age of
Accountability

• Contract for Excellence
• NCLB Reauthorization
• NYC and other local accountability initiatives

3
Contract for Excellence

• Governor has proposed a Contract for Excellence
  that calls for an enhanced State accountability
  system, including
• New accountability standards based on State
  assessments and other indicators of progress,
  such as graduation rates or college attendance
  and completion rates.
• Growth model by 2008-09.
• Value-added model by 2010-2011 based on new or
  revised state assessments.
• Expanded SURR system, resulting in the
  identification of up to 5 of State schools by
  2011-2012 for restructuring or reorganization.

4
Contract for Excellence Plans for Intervention

• School Review Teams conduct resource, program and
  planning audits of SINI and SRAP schools and
  assist all SINI and SRAP schools in development
  of improvement plans.
• Joint School Intervention Teams, whose members
  are either appointed by Commissioner or educators
  from the district, review and recommend plans for
  reorganizing or reconfiguring schools that are to
  be closed.
• Distinguished Educators assist low-performing
  schools and districts.
• The services of all the above are a charge to the
  school district.

5
Contract for Excellence Now its Personal

• Commissioner shall define in regulation deficient
  district performance.
• School districts hiring a superintendent after
  the effective date of regulation must include a
  provision in superintendents contract that after
  two years of deficient district performance the
  superintendent will fully cooperate with a
  distinguished educator.
• After four years of deficient performance, school
  board must seek to remove a superintendent or
  provide a rationale to the Commissioner for why
  board should not take such action.
• After six years of deficient district
  performance, the Commissioner shall commence
  action to remove the board or board members.

6
Distinguished Educators and Deficient District
Performance

• Consult with Commissioner on removal of
  superintendent.
• Serve as ex officio, non-voting members of Board
  of Education.
• Review all school and district improvement plans
  and either endorse or make recommendations to the
  Board of Education for change.
• Boards of Education must make changes as required
  by Distinguished Educators unless upon petition
  by Board of Education Commissioner decides there
  is a compelling reason not to implement the
  modification.

7
Status, Growth and Value- Added Models Defined
Contract for Excellence requires SED to implement
growth and value-added models and meet any
Federal requirements for such models.
8
Status Models Defined

• Status or improvement models, the current
  requirement for measuring Adequate Yearly
  Progress (AYP) under No Child Left Behind,
  measure progress by tracking the improvement at
  the same grade levels within the school over
  time. (This years grades 3-5 compared to last
  years grades 3-5.)

9
Growth Models Defined

• Growth models generally refer to accountability
  models that assess the progress of a cohort of
  individual students over time with the intent of
  measuring the progress these students have made
  (Performance in fourth grade compared to
  performance in third grade).

10
Value-Added Models Defined

• Value-added models generally refer to a specific
  type of growth model in which student demographic
  data or other statistical controls are used to
  attempt to analyze the specific effects of a
  particular school, program, or teacher on student
  learning.
• These models ask whether the school has increased
  the measured achievement of students more than
  expected based on data from similar schools.

11
Purpose of Status, Growth, and Value Added Models

• Status model Determine whether an increasing
  percentage of students are gaining proficiency
  over time.
• Growth model Determine how much progress groups
  of students are making over time.
• Value-added model Determine success of schools,
  programs, or teachers by measuring student growth
  over time while controlling for non-school
  variables that impact on student performance.

12
Value-Added Models The Holy Grail of
Accountability?

• Much controversy about value-added models.
• Some claim that good value-added models exist
  (Sanders), statistical models not based on
  vertical scales.
• Some claim that the state of the psychometric art
  does not currently support vertical scales.
• Value-added models are not easy
• Less sophistication is required to make accurate
  determinations about status then growth.
• Acquisition of knowledge is not linear.
• Value-added models are heavily dependent upon the
  robustness of their assumptions.
• Value-added models break down more quickly as
  groups become smaller.

13
Growth Model System Requirements

• Growth Model System Requirements
• Annual assessments in successive grades
• A unique student identification system
• Vertically aligned or scaled assessments
• Grade by grade standards
• A minimum of two years of assessment data per
  student
• Value-added systems require additional data on
  those demographic or resource factors for which
  the model seeks to control

14
NY and Growth Models

• NYS Grades 3-8 Testing Program uses Vertically
  Moderated Standards (VMS)
• Student progress is measured from grade-to-grade
  relative to proficiency in meeting the standards
  (rather than in terms of change in scale scores).
• VMS is applicable to some but not all growth
  model designs.
• NY will by the end of 2006-07 have only two years
  of test data, limiting our ability to create
  growth trajectories for individual students.
• SEDs goal is to select a growth model
  architecture by April.

15
NCLB Status Report

• Revised State standards and Annual Measurable
  Objectives (AMOs) established.
• 05-06 Accountability Decisions Released.
• Development of elementary/middle level attendance
  standards in progress.
• Regents continuing deliberations regarding
  raising high school graduation standards.
• Regents engaging in NCLB reauthorization advocacy.

16
State Standards

• State Standards for 06-07 are
• 155 for Grade 3-8 ELA and math
• 165 for HS ELA and math
• State Standards for 07-8 are
• 160 for Grade 3-8 ELA and math
• 170 for HS ELA and math
• State standard is used to determine which schools
  must do Local Assistance Plans (LAPs) and which
  schools and districts may be high performing.

17
Original Annual Measurable Objectives for200203
to 201314

• School Year Elementary-Level Middle-Level Secondar
  y-Level
• ELA Math ELA Math ELA Math
• 2002-03 123 136 107 81 142 132
• 2003-04 123 136 107 81 142 132
• 200405 131 142 116 93 148 139
• 200506 138 149 126 105 154 146
• 200607 146 155 135 117 159 152
• 200708 154 162 144 129 165 159
• 200809 162 168 154 141 171 166
• 200910 169 174 163 152 177 173
• 201011 177 181 172 164 183 180
• 201112 185 187 181 176 188 186
• 201213 192 194 191 188 194 193
• 201314 200 200 200 200 200 200

18
Revised Annual Measurable Objectives for200506
to 201314

• School Year Elementary Middle-Level Secondary-L
  evel
• ELA Math ELA Math
• 200506 122 86 154 146
• 200607 122 86 159 152
• 200708 133 102 165 159
• 200809 144 119 171 166
• 200910 155 135 177 173
• 201011 167 151 183 180
• 201112 178 167 188 186
• 201213 189 183 194 193
• 201314 200 200 200 200

19
Calculating AMOs

• In 2000-2001 and 2001-2002 combined, 20 of NYs
  public school students were enrolled in schools
  with PIs below
• Grade 4 ELA 123
• Grade 8 ELA 107
• Grade 4 Math 136
• Grade 8 Math 81
• In 2004-05, the percent of students below the
  2004-2005 AMOs were
• Grade 4 and 8 ELA combined 13.3
• Grade 4 and 8 math combined 4.2
• 2005-2006 Grade 3-8 ELA results
• AMO at 13.3 122
• AMO at 20.0 133
• 2005-2006 Grade 3-8 math results
• AMO at 4.2 86
• AMO at 20.0 132

20
Limited English Proficient (LEP) Students

• All LEP students in grade K12 must take the
  NYSESLAT annually.
• LEP students in grades 3 through 8 enrolled in
  U.S. schools (not including Puerto Rico) for less
  than one year (enrolled on or after January 2,
  2006) were not required to take the NYSTP ELA
  assessment in January 2007. For such students who
  did not take the ELA assessment, valid scores on
  the NYSESLAT Reading/Writing and
  Speaking/Listening components will meet the ELA
  participation requirement.
• The eligible LEP students must be identified in
  the repository using Program Service 0242.
• NYSESLAT performance levels will not be used in
  calculating the Performance Index. LEP students
  meeting the criteria to use the NYSESLAT in lieu
  of the ELA will not be included in the
  Performance Index calculation.

21
2002 Graduation-Rate Cohort Definition

• This cohort will be used to determine if the
  district or school meets the graduation-rate
  requirements for the 200607 school year. The
  2002 graduation-rate cohort consists of all
  students, regardless of their current grade
  status, who were enrolled in the school on
  October 6, 2005 (BEDS day) and met one of the
  following conditions
• first entered grade 9 (anywhere) during the
  200203 school year (July 1, 2002 through June
  30, 2003) or
• in the case of ungraded students with
  disabilities, reached their seventeenth birthday
  during the 200203 school year.

22
2002 Graduation-Rate Cohort Definition (contd)

• The State will exclude the following students
  when reporting data on the 2002 cohort
• students who transferred to another high school
  (excluded from the high school graduation-rate
  cohort) or district (excluded from the district
  graduation-rate cohort) or criminal justice
  facility after BEDS day 2005
• students who transferred to an approved
  alternative high school equivalency preparation
  (AHSEP) or high school equivalency preparation
  (HSEP) program (CR 100.7) after BEDS day 2005 and
  met the conditions stated on the next slide
• students who left the U.S. and its territories
  after BEDS day 2005 and before August 30, 2006
  and
• students who died after BEDS day 2005 and before
  August 30, 2006.

23
2002 Graduation-Rate Cohort(Transfers to
GEDRemoved from Cohort)
Students will be removed from the cohort for the
school and district from which they transferred
to an AHSEP or HSEP program if the final
enrollment record shows that on June 30, 2006 the
student a) earned a high school equivalency
diploma or b) was enrolled in an AHSEP or HSEP
program. Students will be removed from the school
cohort if the enrollment records showed that the
student transferred to a different high school
and was working toward or earned a high school
diploma. Students will be removed from the
district cohort if the enrollment records show
that the student transferred to a high school in
a different district and was working toward or
earned a high school diploma.
24
2003 Graduation-Rate Cohort

• Beginning with the 2003 graduation-rate cohort
  (used for accountability in 2007-08)
• students are included in the cohort based on the
  year they first enter grade 9 (or for ungraded
  students, the year they turn 17).
• students who have spent at least five months in a
  district/school during year 1, 2, 3, or 4 of high
  school are part of the district/school cohort
  unless they transfer to another diploma-granting
  program.

25
Inclusion Rules for the 2003 Graduation-Rate
Cohort

• A student will be included in the
  district/school cohort if the students last
  enrollment record in the district or school
  shows
• that the student was enrolled for at least five
  continuous (not including July and August) months
  and the ending reason was not one of the
  following transferred to another New York State
  district or school, died, transferred by court
  order, or left the U.S.
• fewer than five months enrollment and an ending
  reason indicating that the student dropped out or
  transferred to a GED program and the students
  previous enrollment record in that
  district/school (assuming one exists)
• indicates that the student dropped out or
  transferred to a GED program, and
• that the student was enrolled in the
  district/school for at least five months.

26
School Status for All State Schools 2005-06 2005-06 2006-07 2006-07
Number Percent Number Percent
Good Standing 3,743 84.4 3,749 84.3
In Improvement Status under Title 1 502 11.3 506 11.4
Requiring Academic Progress 192 4.3 193 4.3
Total in improvement status 694 15.6 699 15.7
Total Schools 4,437 4,448
27
Student Enrollment by Accountability Status
NYC NYC Rest of State Rest of State Total Total
  Schools Enrollment Schools Enrollment Schools Enrollment
Requiring Academic Progress - Year 1 9 10845 36 40043 45 50888
Requiring Academic Progress - Year 2 6 5957 38 53161 44 59118
Requiring Academic Progress - Year 3 10 21515 17 18481 27 39996
Requiring Academic Progress - Year 4 36 82411 15 19412 51 101823
Requiring Academic Progress - Year 5 14 19748 1 1032 15 20780
Requiring Academic Progress - Year 6 5 1335 0 0 5 1335
Requiring Academic Progress - Year 7 6 1759 0 0 6 1759
In Need of Improvement - Year 1 62 49523 46 28107 108 77630
In Need of Improvement - Year 2 58 49617 33 27957 91 77574
In Corrective Action 44 32074 23 223305 67 54379
Planning for Restructuring 46 61551 31 30436 77 91987
Restructuring - Year 1 22 17050 20 20459 42 37509
Restructuring - Year 2 57 52336 7 3741 64 56077
Restructuring - Year 3 47 46862 10 7460 57 54322
Total Title I SINI 336 309013 170 140465 506 449478
Total SRAP 86 143570 107 132129 193 275699
Grand Total 422 452583 277 272594 699 725177
Note Enrollment is based on counts from October 2005 BEDS survey. Note Enrollment is based on counts from October 2005 BEDS survey. Note Enrollment is based on counts from October 2005 BEDS survey. Note Enrollment is based on counts from October 2005 BEDS survey.
28
Statewide Status of SINI and SRAP Schools 2005-06
vs. 2006-07
Status 05-06 Status Improved or Stayed the Same Status Became Worse Total Schools In Status Percent Improving or Staying the Same
SINI 1 61 70 131 46.6
SINI 2 33 51 84 39.3
Corrective Action 19 76 95 20.0
Planning for Restructuring 12 31 43 27.9
Restructuring 1 17 58 75 22.7
Restructuring 2 11 62 73 15.1
SRAP 1 44 28 72 61.1
SRAP 2 28 14 42 66.7
SRAP 3 15 44 59 25.4
SPAP 4 6 9 15 40.0
SRAP 5 or 6 3 1 4 75.0
Total 249 444 693 35.9
Note A schools status would improve or stay the
same if the school made AYP in the area(s) of
identification which gave the school its status.
A schools status becomes worse if it fails to
make AYP in one or more of the area of
identification which gave the school its status.
29
WEIGHTED PI BY ACCOUNTABILITY STATUS - ALL SCHOOLS WEIGHTED PI BY ACCOUNTABILITY STATUS - ALL SCHOOLS WEIGHTED PI BY ACCOUNTABILITY STATUS - ALL SCHOOLS WEIGHTED PI BY ACCOUNTABILITY STATUS - ALL SCHOOLS WEIGHTED PI BY ACCOUNTABILITY STATUS - ALL SCHOOLS WEIGHTED PI BY ACCOUNTABILITY STATUS - ALL SCHOOLS WEIGHTED PI BY ACCOUNTABILITY STATUS - ALL SCHOOLS WEIGHTED PI BY ACCOUNTABILITY STATUS - ALL SCHOOLS WEIGHTED PI BY ACCOUNTABILITY STATUS - ALL SCHOOLS
  3-8 ELA 3-8 ELA 3-8 Math 3-8 Math HS ELA HS ELA HS Math HS Math
Acct Status Acct Enrolgt30 Wtd PI Acct Enrolgt30 Wtd PI Acct Enrolgt30 Wtd PI Acct Enrolgt30 Wtd PI
In Good Standing 920,862 163 914,553 168 109,948 183 109,948 185
In Need of Improvement - Year 1 37,283 132 37,069 137 3,694 152 3,694 159
In Need of Improvement - Year 2 41,845 133 41,132 136 4,386 146 4,386 150
In Corrective Action 24,988 127 24,747 125 3,989 144 3,989 149
Planning for Restructuring 32,143 117 31,639 110 9,971 138 9,971 145
Restructuring - Year 1 20,826 119 20,447 114 1,619 122 1,619 130
Restructuring - Year 2 37,304 116 36,921 116 874 132 874 139
Restructuring - Year 3 41,208 116 40,601 111 867 103 867 124
Requiring Academic Progress - Year 1 9,314 154 9,214 154 8,346 180 8,346 184
Requiring Academic Progress - Year 2 16,516 158 16,364 161 8,700 178 8,700 182
Requiring Academic Progress - Year 3 9,700 150 9,602 145 6317 169 6,317 170
Requiring Academic Progress - Year 4 12,526 142 12,418 141 17,151 149 17,151 150
Requiring Academic Progress - Year 5 3,763 125 3,722 113 3,628 132 3,628 141
Requiring Academic Progress - Year 6 443 114 427 99 419 116 419 118
Requiring Academic Progress - Year 7 1,236 94 1,220 91 212 118 212 118
Has No Status - Regulations Do Not Apply 316 145 308 140 0 0 0 0
Total Title I SINI 235,597 123 232,556 122 25,400 140 25,400 147
Total SRAP 53,498 148 52,967 147 44,773 161 44,773 164
Grand Total 1,210,273 154 1,200,384 158 180,121 171 180,121 175
30
NCLB Reauthorization

• Regents have developed positions regarding
• Schools and districts with multiple federal
  designations
• Growth Models
• Targeted interventions
• AYP Methodologies
• Additional time to meet graduation standards
• ELL testing issues
• Implementation of choice and SES
• Safe Schools

31
NCLB Reauthorization What we are Hearing

• Serious discussions about NCLB are beginning but
  reauthorization more likely to occur in 2009 than
  2007, very unlikely to occur in 2008.
• Key issues being discussed
• Full funding
• Growth models
• N size and confidence intervals
• Required interventions, including SES and choice
• Assessments LEP, SWD, high school
• Highly Qualified Teachers vs. High Quality
  Teaching
• National Standards
• Science Assessments

32
Other Accountability Initiatives NYC

• Children First
• Valued Added Model.
• More Ragu than KISS.
• Each school receives letter grade (A-F) on school
  progress report.
• Progress report consists of four categories
• School Environment (15)
• Student Performance (30)
• Student Progress (55)
• Additional Credit

33
Children First Progress Report

• Schools grade is based upon how well the school
  performs relative to other City schools (1/3 of
  grade) and to a peer group of approximately 40
  schools with similar demographics (2/3 of grade).
• State assessments used to measure performance of
  students over time.
• Additional credit given to improved performance
  among low achieving students and various
  disaggregated groups.
• Factors such as parent, student, and teacher
  surveys credit accumulation PSAT scores used in
  determining school grade.

34
Challenges for SED

• Growing Pains
• Testing Times
• Vision 2020
• A Cart Before the Horse?

35
Challenges Ahead for School Districts

• Immediate
• Single Grade 3-8 Performance Index makes schools
  and districts responsible for more disaggregated
  groups.
• New standard setting for grade 3-8 assessments
  may challenge middle schools even more.
• Changes in testing practices for LEP students
  require that bilingual education programs
  emphasize rapid acquisition of English as well as
  fluency in the native language.
• New graduation standards will raise expectations
  for middle schools and force high schools to
  reveal hidden students.
• Longer term
• If Governor and Legislature show school districts
  the money, they are also going to demand that
  districts show results.

36
Challenges Ahead Lists, Lists, Lists

• How do we make all of these work together
• SURR
• SINI, DINI
• SRAP, DRAP
• IDEA Districts
• Title III AMAOs
• Persistently Dangerous Schools
• High School Initiative
• Highly Qualified Teachers

37
This may be the Dawning of the Age of
Accountability

• Key Questions
• How do we design accountability models that
  compel movement from awful to adequate without
  impeding the movement from good to great?
• How do we move from beating the odds to changing
  the odds?

38
More Information

• Ira Schwartz, Coordinator
• Office of School Improvement and Community
  Services (NYC)
• ischwart_at_mail.nysed.gov
• 718 722-2796
• Accountability PowerPoint for 2006-07
• http//www.emsc.nysed.gov/irts/accountability/2005
  -06/accountability-rules-Nov2006_files/frame.htm
• Manuals for NYS Student Identifier System and the
  Repository System http//www.emsc.nysed.gov/irts/
  SIRS/home.shtml