Reading your Accountability Reports U-PASS AYP AMAO

1
Reading your Accountability Reports
U-PASSAYPAMAO

• Judy Park, Assessment Accountability Director
• Russell Klein, Results Coordinator
• Utah State Office of Education
• August 2006

2
Reading your Accountability ReportsU-PASS,
AMAO, AYP

• August 18 Sevier D.O., 180 E 600 N, Richfield
• August 22 Granite D.O., 2500 S. State St., Salt
  Lake
• August 23 Weber D.O., 5320 S. Adams Ave. Pkwy,
  Ogden
• August 25 Nebo Learning Center, 570 S. Main
  St., Springville

3
Utah Performance Assessment System for
StudentsU-PASS Accountability Plan
4
U-PASS

• State Accountability
• Required by State Law
• School level report
• Based on multiple assessments
• Includes attendance graduation rate
• 2006 schools without a 12th grade only
• 2007 all schools

5
U-PASSAccountability Plan

• Student Participation on Assessments
• Status Scores
• Percentage of students proficient on the
  collection of assessments and other indicators
• Progress Scores
• Longitudinal measure based on comparing the
  achievement levels of the same student from one
  year to the next

6
Achieved State Level of Performance
Total School Subgroup Low progress Medium progress High progress
Acceptable status
Unacceptable status Needs Assistance
7
U-PASSAccountability Plan

• Student Participation
• 95
• Acceptable Status Score
• 80
• Progress Score
• Low 0 189
• Medium 190 - 214
• High 215 - 400

8
Progress Value Table
Year 1 Level Year 2 Level Year 2 Level Year 2 Level Year 2 Level Year 2 Level Year 2 Level
Year 1 Level 1a 1b 2a 2b 3 4
1a 0 225 350 375 375 375
1b 0 125 225 350 375 375
2a 0 50 150 225 325 350
2b 0 0 75 175 275 325
3 0 0 0 100 200 275
4 0 0 0 0 125 225
196.0
9
Schools will be identified as

• Achieved State Level of Performance
• Participation is 95 and
• Total School AND Subgroup Status is acceptable
  (80) or progress is medium (190) or high
• Needs assistance
• Participation is less than 95 or
• Total School or Subgroup Status is not acceptable
  and progress is low

10
Schools without a 12th gradeStatus and Progress
Score

• Math (35)
• Math CRT
• Science (20)
• Science CRT
• Attendance (10)

• Language Arts (35)
• ELA CRT (35)
• Or
• ELA CRT (30)
• DWA 5

11
U-PASS Accountability Plan

• Aggregate Subgroup Accountability
• Every student who belongs to a subgroup other
  than white
• Individual student proficiencies are added
  together and divided by the number of students to
  determine the subgroup proficiency level
• Any individual subgroup not meeting proficiency
  will appear on the front page of the report

12
U-PASS Accountability Plan

• Subgroup Status Accountability.
• Allows greater accountability.
• Students in a subgroup with less than 10 are now
  included in the aggregate calculation.
• Students are more fairly represented.
• Each student only counts once, regardless of the
  number of subgroups in which the student
  qualifies.
• Increased reliability.
• We are evaluating a larger group.
• More students and more schools are represented.
• Increased validity.
• We do not overweight small populations.
• This lifts the burden of one subgroup being the
  sole determinant of unacceptable status of the
  school.

13
The 2006 U-PASS Report

• Web-based report format
• An increased amount of information available on
  each school
• Allows for drill-down information

14
(No Transcript)
15
1
16
2
17
(No Transcript)
18
Final Result
19
Subgroups Needing Assistance
20
(No Transcript)
21
Click on Grade Name for Instant Graph
22
Click on Grade Score for Further Drill Down
Second Level Drill Down
23
Third Level Drill Down
Boxes expand on mouseover
24
Progress Drill Down coming in October
25
Additional Reports
26
U-PASS

• Additional Information
• NRT
• ACT/SAT
• AP
• Concurrent Enrollment
• School Summary Information
• Student Summary Information
• Percent of student reading on grade level
• (reported for grades 1-10)
• Dropout Rate (disaggregated by 08)
• Disciplinary Action
• Course Taking Patterns and Trends

27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
AYP Reports
31
Schools with a 12th Grade

• Assessment Drill Downs
• Additional Information

32

• Questions?

33
AYP School Reports

• For 2006

34
AYP

• Federal Accountability
• Required by Federal NCLB Title I Law
• School, District State level Reports
• Based on Language Arts CRT Math CRT
• Includes attendance graduation rate

35
AYP 2006Level 1

• Three parts
• Participation in testing
• Academic achievement
• School attendance / graduation
• All-or-nothing concept

36
1
3
2
37
AYP 2006Level 1

• Did the school make AYP?
• Math vs. Language Arts?
• AMO (Annual Measurable Objectives) same as last
  year
• Consequences only for Title I schools

38
4
1
3
3
2
39
(No Transcript)
40
AYP 2006Level 1

• WHICH STUDENTS ARE IN THE AYP SCHOOL REPORTS?

41
Students in AYP U-PASS School Reports

• Two large data collections
• 1) End-of-Year Clearinghouse
• 2) Core CRT Files
• These Two Files Need To Match To Be On AYP
  U-PASS School Reports!
• 6 Fields for Matching District, School, Last
  Name, First Name, Date of Birth, Gender
• As of Spring 06 SSID!

42
Full Academic Year
Used for Core CRT tests to be included - Full Academic year 2006- 1. Student who was in membership on Sept. 15th 2. Student membership gt1603. Student membership School Membership Special Ed Membership (Self-Contained) YIC membership
43
Math Core CRTs
Math CRTs Grade 3 8 Mathematics 64 Proficient is the Status Goal (AMO) for 2006 Grades 10-12 Algebra Geometry Applied Math 1 2 not included 47 Proficient is the Status Goal ONLY students in Grades 10,11,12 who took the Core CRTs in either Algebra or Geometry. Schools with a 12th grade class get this AMO/Goal.
44
Language Arts Core CRTs
Language Arts Grades 3 8 Language Arts 71 Proficient. This is the Status Goal (AMO) for 2006. Grade 10 Language Arts 70 Proficient. No other language arts Core CRTs are being used for AYP for schools with a 12th grade class!
45
AYP 2006Level 1

• Questions?

46
AYP 2006Level 2

• Status vs. Safe Harbor (Improvement)
• Safe Harbor
• 10 reduction in of students not proficient

47
AYP 2006Safe Harbor (Improvement)


Proficient
Not Proficient
Status
48
AYP 2006Safe Harbor (Improvement)
Did not make AYP Made AYP

Proficient
Not Proficient
Status
49
AYP 2006Safe Harbor (Improvement)
Shows Improvement
Does not show improvement
Improvement
50
AYP 2006Safe Harbor (Improvement)
Shows Improvement Made AYP
Does not show improvement Did not make AYP
Improvement
51
AYP 2006Safe Harbor (Improvement)


Good for Improvement
Shows Improvement
Good for Status
Improvement
Does not show improvement
Proficient
Not Proficient
Status
52
AYP 2006Safe Harbor (Improvement)


Made AYP
Made AYP
Shows Improvement
Improvement
Made AYP
Does not show improvement
Proficient
Not Proficient
Status
53
Status
54
Safe Harbor
55
AYP 2006Level 2

• Safe Harbor
• Additional Indicator is only used when Safe
  Harbor calculation is needed

56
AYP 2005Level 2

• Confidence Interval on Status
• 99
• Confidence Interval on Safe Harbor
• 75
• Participation Additional Indicator
• N/A

57
ConfidenceInterval

• Voting exit polls

58
Confidence Interval on Status
59
Confidence Interval on Safe Harbor
60
AYP 2006Level 2

• N size 10 vs. 40
• Small N Automatic YES

61
AYP 2006Level 2

• N40 participation, attendance, and graduation
• N10 academic achievement

62
(No Transcript)
63
(No Transcript)
64
(No Transcript)
65
AYP 2006

• Graduation rate 85.7
• One year delayed (have until Sep 1)
• 05 data from warehouse
• 04, 03 data from USOE statistical files
• Quality of data has improved
• 40 ways to make (or not make) AYP

66
(No Transcript)
67
Times 10
40 ways to make (or not make) AYP
68
Elementary
69
Elementary
70
District
71
Available on the secure FTP site
72
AYP 2006Level 2

• Questions?

73
AYP 2006Level 3

• Definition of safe harbor
• Definition of participation
• Definition confidence intervals
• Definition of graduation rate
• UAA

74
AYP 2006Level 3

• Questions?

75
AYP 2006Did your school make it?

• Participation DID YOU MEET IT?
• Participation Is the N size less than 40?
• Did they have 95 participation?
• Did they average 95 participation over the last
  three (or two) years?

76
AYP 2006Did your school make it?

• Status Is the subgroup size less than 10?
• Status Is the proficient enough?
• Status If the proficient is not enough does
  the confidence interval change it?

77
AYP 2006Did your school make it?

• Safe Harbor Is there a 10 reduction in
  students not proficient?
• Safe Harbor Is the reduction of students not
  proficient within the confidence interval?

78
AYP 2006Did your school make it?

• Attendance/Graduation Rate Is the subgroup less
  than 40?
• Attendance/Graduation Rate Did the school meet
  93 attendance rate or 85.7 graduation rate
• Attendance/Graduation Rate has there been an
  increase from the previous year?

79
Level Three Clarifications
80
AYP 2006Level 3

• Participation Category Students enrolled on
  test date.
• Participation within tests Students youve had
  for the full academic year. (subset of category
  set)
• Participation grad/attend. Is ALL students

81
95 Participation
For 2004 - 2006, Missing tests will be handled by the testing process. Same as 2003 with the exception that participation percentage can be averaged over the previous two (or three) years, if necessary. If this does not help, the current year participation percentage will be used
82
LEP Subgroup Definition
2004 - 2006, LEP students will count in the LEP subgroup for two years past their exit of the ELL program. This is determined by the lep_exit_date in the student_lep table of the warehouse.
83
Additional Indicator AYP Attendance 93
The goal for attendance will be set at 93, consistent with the additional indicator goal for elementary and middle schools. Schools must meet this goal or improve from the previous year. Improvement is defined as any increase in this area. The attendance rate is the sum of days in attendance divided by the sum of days in membership for all students in the unit of accountability who were in membership for at least eleven (11) days and whose membership is equal to or greater than attendance (180 days). Membership includes self-contained SCRAM and YIC systems.
84
Additional Indicator AYPGraduation Rate 85.7
Grad Rate is Yes, if Grad Rate is gt 85.7 The cohort graduation rates by school according to the ED approved algorithm in our AYP plan. Here is the algorithm IF(G2lt40,"too few",IF(L2gt0.857,"pass",IF(L2gtW2,"pass","fail"))) In English If the cohort G contains less then 40 students, there are too few students to make a reliable decision about the graduation rate for the unit of analysis (school or district). NA If the cohort is equal to or greater than 40 students and the 2002 graduation rate L is greater than (the constant) 85.7, the unit passes. Yes If the 2002 graduation rate is less than 85.7 but greater than the 2001 graduation rate W, the unit passes. Yes In all other cases, the unit fails. No
85
UAA TestingSpecial Ed Alternate Assessment

• Grade level is derived based on birthdate.
• Core Test Id and grade level will be used to
  determine which Actual Test ID to assign a
  student to. LA and grade level will correspond
  to an Actual Language (Reading) class for that
  student. MA and grade level will correspond to a
  math course.
• UAA students whose birthdate is in 9-12th grades
  will be added to GE0812R Geometry. They are part
  of High School Math AYP!
• UAA in 7-8th grade will be added to 7th grade
  math.

86
Out-of-Level Testing

• Not allowed
• No for participation
• No for academic achievement
• Math Levels
• 6th Math 6 7th Math 7 8th Pre-Alg.
  Anything lower for the grades is Out-of-Level and
  considered Not Proficient.

87
District AYP Reports

• Previous District AYP Report
• Grades 3 8 only

88
District AYP Reports

• 2006 District AYP Report
• Grades 3 8
• Grades 10 12
• Business Rule
• An LEA will receive a report based on both 3-8th
  and 10-12th grade AMO's.  An LEA must fail at
  both the 3-8 AND the 10-12 grade levels to fail
  AYP.  (Either subject)

89
District AYP Reports

• 2006 District AYP Report
• Grades 3 8
• Grades 10 12
• Program Improvement
• Business Rule
• An LEA is identified for improvement only when
  it misses AYP in the same subject and in all
  grade spans for two consecutive years, or the
  other academic indicator in all grade spans for
  two consecutive years.
• (It is possible for an LEA to not pass AYP and
  yet not be put into 'Improvement' because
  'Improvement' considers 'subject' in addition to
  grade levels whereas 'not making AYP' does not. 

90
AMAO Reports
91
AMAO

• Federal Accountability
• Required by NCLB Title III Law
• District State Level Reports
• Based on
• IPT progress
• IPT proficiency
• AYP for ELL

92
(No Transcript)
93
(No Transcript)
94

• Enrollment Information
• AMAO 1 Annual increases in the number or
  percentage of children making progress in
  learning English
• AMAO 2 Annual increases in the number or
  percentage of children attaining English
  proficiency
• AMAO 3 AYP for ELL

95
ELL Enrollment

• Grade Span
• Language Proficiency Category
• 2005 ELL Enrollment
• 2006 ELL Enrollment

96
ELL Enrollment

• Grade Span
• The grade span used for AMAO targets is grades
  K-6 and 7-12.

97
ELL Enrollment

• Language Proficiency Category
• A - Non-English proficient in speaking, reading
  and writing
• B - Limited English proficient (student is non in
  at least one of the language modalities of
  speaking, reading and writing or at least limited
  in two of the language modalities of speaking,
  reading and writing)
• C - Fluent in two of the language modalities and
  limited in one

98
ELL Enrollment

• Language Proficiency Category
• D - Monitored student for minimum of two years
  for English proficiency (student is fluent in all
  language modalities).
• E - Exited, Former LEP student who is fully
  proficient in English speaking, reading and
  writing and has been exited from an alternative
  language services program and fully assimilated
  into the mainstream.
• O - Identified as LEP but opted out of services
  by parent.

99
ELL Enrollment

• 2005 ELL Enrollment
• The number of ELL students enrolled in 2005 for
  the full academic year. For E students, only
  those students who have exited within the last 3
  years are included.

100
ELL Enrollment

• 2006 ELL Enrollment
• The number of ELL students enrolled in 2006 for
  the full academic year. For E students, only
  those students who have exited within the last 3
  years are included.

101
AMAO Requirement 1

• Grade Span
• of 2005 students making progress learning
  English in 2006 (AMAO 80)
• District made the AMAO targets for making
  progress learning English

102
AMAO Requirement 1

• Grade Span
• The grade span used for AMAO targets is grades
  K-6 and 7-12.

103
AMAO Requirement 1

• of 2005 Students making progress
• learning English in 2006
• The percentage of ELL students who made progress
  in one or more domains (speaking, reading,
  writing). Progress is defined as moving from non
  speaking to limited or limited to fluent.

104
AMAO Requirement 1

• District made the AMAO targets for
• making progress learning English
• This column is Yes if the district made the AMAO
  target of 80 of students showing progress in one
  or more domains (speaking, reading, writing).

105
AMAO Requirement 2

• of 2005 students Attaining English Proficiency
  in 2006
• K-6, 16.2
• 7-12, 21.6
• District made the AMAO target for Attaining
  English Proficiency

106
AMAO Requirement 2

• of 2005 students Attaining English
• Proficiency in 2006 (K-6, 16.2) (7-12,
• 21.6)
• Percentage of students enrolled for both the
  2005 and 2006 full academic year, with a language
  proficiency of D or E in 2006.

107
AMAO Requirement 2

• District made the AMAO target for
• Attaining English Proficiency
• The column is Yes if the district made the AMAO
  target.
• For grade ranges K-6, 16.2 of the students are
  at the D or E level.
• For grade ranges 7-12, 21.6 of the students are
  at the D or E level.

108
AMAO Requirement 3

• Grade Span
• Participation of ELL for AYP Language Arts
• District made AYP for the ELL subgroup in
  Language Arts
• participation of ELL for AYP Math
• District made AYP for the ELL subgroup in Math

109
AMAO Requirement 3

• Grade Span
• The grade span used for AYP targets is grades
  3-8 and grades 10-12.
• Participation of ELL for AYP Language
• Arts
• The percentage of ELL students who participated
  in the language arts CRT.

110
AMAO Requirement 3

• District made AYP for the ELL subgroup
• in Language Arts
• The ELL subgroup attained 95 participation and
  for the 3-8 grade span attained 71 of the
  students proficient on the language arts CRT and
  for the 10-12 grade span attained 70 of the
  students proficient on the language arts CRT.

111
AMAO Requirement 3

• Participation of ELL for AYP Math
• The percentage of ELL students who participated
  in the Math CRT.

112
AMAO Requirement 3

• District Made AYP for the ELL subgroup
• in Math
• The ELL subgroup attained 95 participation and
  for the 3-8 grade span attained 64 of the
  students proficient on the Math CRT and for the
  10-12 grade span attained 47 of the students
  proficient on the Math CRT.

113
District Made AMAOs

• The District is Yes if there is a Yes in
  every column. One No in any column will result
  in a No for District Made AMAOs.

114
Appeals

• 1. An error in the aggregating and merging of
  data and/or a calculation error, or other errors
  that directly affects school or district status.
• 2. Extreme circumstances that affected data
  and/or test administration that had a significant
  impact on the school (e.g., flood, fire, natural
  disaster).

115
Appeal Process

• September 11
• Deadline for District Superintendent to submit a
  written appeal request for U-PASS, school AYP,
  District AYP, or District AMAO stating the reason
  for the appeal An E-mail response from USOE will
  confirm receipt of the appeal.
• Data and documentation that directly influences
  the appeal needs to be stored at the district.

116

• September 12-15
• USOE reviews appeals and makes changes to the
  U-PASS, AYP AMAO school or district
  designation.
• September 16
• Districts release U-PASS, school AYP, district
  AYP district AMAO to the public.
• September 17
• U-PASS, AYP AMAO reports posted on the web.
  Reports for granted appeals, changed from a No to
  a Yes (by appeal).