Reading your Accountability Reports U-PASS AYP

1
Reading your Accountability Reports U-PASSAYP

• Judy Park, Associate Superintendent
• Data, Assessment Accountability
• Utah State Office of Education
• August 2007

2
Utah Performance Assessment System for
StudentsU-PASS Accountability Plan
3
U-PASS

• State Accountability
• Required by State Law
• School level report
• Based on multiple assessments
• Includes attendance graduation rate

4
U-PASSAccountability Plan

• Student Participation on Assessments
• Proficiency Scores
• Percentage of students proficient on the
  collection of assessments and other indicators
• Progress Scores
• Longitudinal measure based on comparing the
  achievement levels of the same student from one
  year to the next

5
Achieved State Level of Performance
Total School Subgroup Low progress Medium progress High progress
Acceptable Proficiency
Unacceptable Proficiency Needs Assistance
6
U-PASSAccountability Plan

• Student Participation
• 95
• Acceptable Proficiency Score
• Schools without a 12th grade 80
• Schools with a 12th grade 75
• Progress Score
• Without 12th grade With 12th grade
• Low 0 189 Low 0 - 179
• Medium 190 214 Med 180 - 214
• High 215 400 High 215 - 400

7
Progress Value TableSchools without a 12th grade
Year 1 Level Year 2 Level Year 2 Level Year 2 Level Year 2 Level Year 2 Level Year 2 Level
Year 1 Level 1a 1b 2a 2b 3 4
1a 0 225 350 375 375 375
1b 0 125 225 350 375 375
2a 0 50 150 225 325 350
2b 0 0 75 175 275 325
3 0 0 0 100 200 275
4 0 0 0 0 125 225
196.0
8
Progress Value TableSchools with a 12th grade
Year 1 Level Year 2 Level Year 2 Level Year 2 Level Year 2 Level Year 2 Level Year 2 Level
Year 1 Level 1a 1b 2a 2b 3 4
1a 0 200 250 300 350 375
1b 0 125 200 300 350 350
2a 0 50 150 175 325 325
2b 0 0 75 175 300 325
3 0 0 0 100 200 300
4 0 0 0 0 125 225
196.0
9
Schools will be identified as

• Achieved State Level of Performance
• Participation is 95 and
• Total School AND Subgroup Proficiency is
  acceptable or progress is medium or high
• Needs assistance
• Participation is less than 95 or
• Total School or Subgroup Proficiency is not
  acceptable and progress is low

10
Schools without a 12th gradeProficiency and
Progress Score

• Math (35)
• Math CRT
• Science (20)
• Science CRT
• Attendance (10)

• Language Arts (35)
• ELA CRT (35)
• Or
• ELA CRT (30)
• DWA 5

11
Schools with a 12th gradeParticipation Score

• Schools with a 9th grade
• CRT 90, DWA 10
• Schools without a 9th grade
• CRT 100

12
Schools with a 12th gradeProficiency Score

• Proficiency
• Language Arts 30
• Math 25
• Science 20
• Attendance 10
• Graduation Rate 15

13
Schools with a 12th gradeProficiency Score

• Language Arts 30
• ELA CRT 80
• Or ELA CRT 75
• DWA 5
• UBSCT Reading 10
• UBSCT Writing 10
• Math 25
• Math CRT 65
• UBSCT Math 35

• Science 20
• Science CRT
• Attendance 10
• Absent fewer than 15 days
• Graduation Rate 15
• AYP Calculation

14
Schools with a 12th gradeProgress Score

• Language Arts CRT 30
• Math CRT 25
• Science CRT 20
• UBSCT 15
• Calculate the difference between the pass rate
  for the 5th (final) attempt in the 12th grade
  and the initial pass rate in the 10th grade
• Attendance Progress 10

15
Schools with a 12th gradeProficiency and
Progress Score

• Proficiency
• Language Arts 30
• Math 25
• Science 20
• Attendance 10
• Graduation Rate 15

• Progress
• Language Arts CRT 30
• Math CRT 25
• Science CRT 20
• Attendance 10
• UBSCT 15

16
U-PASS Accountability Plan

• Aggregate Subgroup Accountability
• Every student who belongs to a subgroup other
  than white
• Individual student proficiencies are added
  together and divided by the number of students to
  determine the subgroup proficiency level
• Any individual subgroup not meeting proficiency
  will appear on the front page of the report

17
U-PASS Accountability Plan

• Subgroup Proficiency Accountability
• Allows greater accountability
• Students in a subgroup with less than 10 are now
  included in the aggregate calculation
• Students are more fairly represented
• Each student only counts once, regardless of the
  number of subgroups in which the student
  qualifies.
• Increased reliability
• We are evaluating a larger group
• More students and more schools are represented
• Increased validity
• We do not overweight small populations.
• This lifts the burden of one subgroup being the
  sole determinant of unacceptable status of the
  school.

18
The U-PASS Report

• Web-based report format
• An increased amount of information available on
  each school
• Allows for drill-down information

19
(No Transcript)
20
(No Transcript)
21
We have currently locked the 2007 U-PASS data for
your own schools only.
22
(No Transcript)
23
1
2
3
24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
U-PASS

• Additional Information
• NRT
• ACT/SAT
• AP
• Concurrent Enrollment
• School Summary Information
• Student Summary Information
• Percent of student reading on grade level
• (reported for grades 1-10)
• Dropout Rate (disaggregated by 08)
• Disciplinary Action
• Course Taking Patterns and Trends

30

• Questions?

31
AYP School Reports
32
AYP

• Federal Accountability
• Required by Federal NCLB Title I Law
• School, District State level Reports
• Based on Language Arts CRT Math CRT
• Includes attendance graduation rate

33
AYP Level 1

• Three parts
• Participation in testing
• Academic achievement
• School attendance / graduation
• All-or-nothing concept

34
1
3
2
35
AYP Level 1

• Did the school make AYP?
• Math vs. Language Arts?
• AMO (Annual Measurable Objectives) increased this
  year
• Consequences only for Title I schools

36
AYP Targets
  2002 2005 2007 2009 2011 2013 2014
Language Arts Grades 3-8 65 71 77 83 89 95 100
Mathematics Grades 3-8 57 64 71 78 85 92 100
Language HS Grade 10 64 70 76 82 88 94 100
Mathematics Grade 10-12 35 47 59 72 84 96 100
37
4
1
3
3
2
38
AYP Level 1

• WHICH STUDENTS ARE IN THE AYP SCHOOL REPORTS?

39
Students in AYP U-PASS School Reports

• Two large data collections
• 1) End-of-Year Clearinghouse,
• 2) Core CRT Files
• These Two Files Need To Match To Be On AYP
  U-PASS School Reports!
• 6 Fields for Matching District, School, Last
  Name, First Name, Date of Birth, Gender
• SSID

40
Full Academic Year
Used for Core CRT tests to be included - New definition beginning in 2007 Student membership gt160
41
Math Core CRTs
Math CRTs Grade 3 8 Mathematics 71 Proficient is the Status Goal (AMO) for 2007 Grades 10-12 Algebra Geometry 59 Proficient is the Status Goal ONLY students in Grades 10,11,12 who took the Core CRTs in either Algebra or Geometry. Schools with a 12th grade class get this AMO/Goal.
42
Language Arts Core CRTs
Language Arts Grades 3 8 Language Arts 77 Proficient. This is the Status Goal (AMO) for 2007. Grade 10 Language Arts 76 Proficient. No other language arts Core CRTs are being used for AYP for schools with a 12th grade class!
43
AYP Level 1

• Questions?

44
AYP Level 2

• Status vs. Safe Harbor (Improvement)
• Safe Harbor
• 10 reduction in of students not proficient

45
AYP Safe Harbor (Improvement)


Proficient
Not Proficient
Status
46
AYP Safe Harbor (Improvement)
Did not make AYP Made AYP

Proficient
Not Proficient
Status
47
AYP Safe Harbor (Improvement)
Shows Improvement
Does not show improvement
Improvement
48
AYPSafe Harbor (Improvement)
Shows Improvement Made AYP
Does not show improvement Did not make AYP
Improvement
49
AYPSafe Harbor (Improvement)


Good for Improvement
Shows Improvement
Good for Status
Improvement
Does not show improvement
Proficient
Not Proficient
Status
50
AYPSafe Harbor (Improvement)


Made AYP
Made AYP
Shows Improvement
Improvement
Made AYP
Does not show improvement
Proficient
Not Proficient
Status
51
Status
52
Safe Harbor
53
10 Reduction in of students not
proficientExample

• 2006 60 proficient
• 2007 66 proficient
• 2006 40 not proficient
• 2007 34 not proficient
• 10 of 40 is 4
• 40 - 4 36
• To make safe harbor, 36 or less must be not
  proficient or 64 or more must be proficient
• Made safe Harbor ( more students proficient than
  required)

54
AYPLevel 2

• Safe Harbor
• Additional Indicator is only used when Safe
  Harbor calculation is needed
• Attendance 93
• Graduation Rate 85.7

55
AYPLevel 2

• Confidence Interval on Status
• 99
• Confidence Interval on Safe Harbor
• 75
• Participation Additional Indicator
• N/A

56
ConfidenceInterval

• Voting exit polls

57
Confidence Interval on Status
58
Confidence Interval on Safe Harbor
59
AYPLevel 2

• N size 10 vs. 40
• Small N Automatic YES

60
AYP Level 2

• N40 participation, attendance, and graduation
• N10 academic achievement

61
(No Transcript)
62
(No Transcript)
63
(No Transcript)
64
AYP

• Graduation rate 85.7
• One year delayed (have until Sep 1)
• 05 data from warehouse
• 04, 03 data from USOE statistical files
• Quality of data has improved
• 40 ways to make (or not make) AYP

65
(No Transcript)
66
Times 10
40 ways to make (or not make) AYP
67
Elementary
68
Elementary
69
District
70
AYP Level 2

• Questions?

71
AYP Level 3

• Definition of safe harbor
• Definition of participation
• Definition confidence intervals
• Definition of graduation rate
• UAA

72
AYP 2006Level 3

• Questions?

73
AYP Did your school make it?

• Participation DID YOU MEET IT?
• Participation Is the N size less than 40?
• Did they have 95 participation?
• Did they average 95 participation over the last
  three (or two) years?

74
AYPDid your school make it?

• Status Is the subgroup size less than 10?
• Status Is the proficient enough?
• Status If the proficient is not enough does
  the confidence interval change it?

75
AYPDid your school make it?

• Safe Harbor Is there a 10 reduction in
  students not proficient?
• Safe Harbor Is the reduction of students not
  proficient within the confidence interval?

76
AYP Did your school make it?

• Attendance/Graduation Rate Is the subgroup less
  than 40?
• Attendance/Graduation Rate Did the school meet
  93 attendance rate or 85.7 graduation rate
• Attendance/Graduation Rate has there been an
  increase from the previous year?

77
Level Three Clarifications
78
AYPLevel 3

• Participation Category Students enrolled on
  test date.
• Participation within tests Students youve had
  for the full academic year. (subset of category
  set)
• Participation grad/attend. Is ALL students

79
95 Participation

• Participation percentage can be averaged over
  the previous two (or three) years, if necessary.
  If this does not help, the current year
  participation percentage will be used

80
LEP Subgroup Definition
LEP students will count in the LEP subgroup for two years past their exit of the ELL program. This is determined by the lep_exit_date in the student_lep table of the warehouse.
81
Additional Indicator AYP Attendance 93
The goal for attendance will be set at 93, consistent with the additional indicator goal for elementary and middle schools. Schools must meet this goal or improve from the previous year. Improvement is defined as any increase in this area. The attendance rate is the sum of days in attendance divided by the sum of days in membership for all students in the unit of accountability who were in membership for at least eleven (11) days and whose membership is equal to or greater than attendance (180 days). Membership includes self-contained SCRAM and YIC systems.
82
Additional Indicator AYPGraduation Rate 85.7
Grad Rate is Yes, if Grad Rate is gt 85.7 The cohort graduation rates by school according to the ED approved algorithm in our AYP plan. Here is the algorithm IF(G2lt40,"too few",IF(L2gt0.857,"pass",IF(L2gtW2,"pass","fail"))) In English If the cohort G contains less then 40 students, there are too few students to make a reliable decision about the graduation rate for the unit of analysis (school or district). NA If the cohort is equal to or greater than 40 students and the 2002 graduation rate L is greater than (the constant) 85.7, the unit passes. Yes If the 2002 graduation rate is less than 85.7 but greater than the 2001 graduation rate W, the unit passes. Yes In all other cases, the unit fails. No
83
UAA TestingSpecial Ed Alternate Assessment

• Grade level is derived based on birthdate
• Core Test Id and grade level will be used to
  determine which Actual Test ID to assign a
  student to. LA and grade level will correspond
  to an Actual Language (Reading) class for that
  student. MA and grade level will correspond to a
  math course.
• UAA students whose birthdate is in 9-12th grades
  will be added to GE0812R Geometry. They are part
  of High School Math AYP!
• UAA in 7-8th grade will be added to 7th grade
  math.

84
Out-of-Level Testing

• Not allowed
• No for participation
• No for academic achievement
• Math Levels
• 6th Math 6 7th Math 7 8th Pre-Alg.
  Anything lower for the grades is Out-of-Level and
  considered Not Proficient.

85
District AYP Reports

• District AYP Report
• Grades 3 8
• Grades 10 12
• Business Rule
• An LEA will receive a report based on both 3-8th
  and 10-12th grade AMO's.  An LEA must fail at
  both the 3-8 AND the 10-12 grade levels to fail
  AYP.  (Either subject)

86
District AYP Reports

• District AYP Report
• Grades 3 8
• Grades 10 12
• Program Improvement
• Business Rule
• An LEA is identified for improvement only when
  it misses AYP in the same subject and in all
  grade spans for two consecutive years, or the
  other academic indicator in all grade spans for
  two consecutive years.
• (It is possible for an LEA to not pass AYP and
  yet not be put into 'Improvement' because
  'Improvement' considers 'subject' in addition to
  grade levels whereas 'not making AYP' does not. 

87
Appeals

• 1. An error in the aggregating and merging of
  data and/or a calculation error, or other errors
  that directly affects school or district status.
• 2. Extreme circumstances that affected data
  and/or test administration that had a significant
  impact on the school (e.g., flood, fire, natural
  disaster).

88
Appeal Process

• September 10
• Deadline for District Superintendent to submit a
  written appeal request for U-PASS, school AYP or
  District AYP, stating the reason for the appeal.
  An E-mail response from USOE will confirm receipt
  of the appeal.
• Data and documentation that directly influences
  the appeal needs to be stored at the district.

89
Appeal Process

• September 17
• Districts release reports to the public
• September 18
• Reports posted on the web
• September 17 November 1
• Districts submit to USOE any error corrections
• November 1 November 30
• New U-PASS AYP reports developed
• November 30
• New U-PASS AYP reports posted on web
• All reports a Yes or a No
• Appeal remains for 2 Extreme Circumstance

90
AMAOs

• Title III Accountability

91
AMAOs

• September 17
• Districts release reports to the public
• September 18
• Reports posted on the web
• September 17 November 1
• Districts submit to USOE any error corrections
• November 1 November 30
• New U-PASS AYP reports developed
• November 30
• New U-PASS AYP reports posted on web
• All reports a Yes or a No
• Appeal remains for 2 Extreme Circumstance