Fourth Year Alternative Accountability in California

1
Mapping the Alternative Schools Accountability
Model (ASAM)
Fourth Year Alternative Accountability in
California
Presented by Vicki L. Barber, Ed.D. El Dorado
County Superintendent of Schools November 5,
2005 A collaborative effort of the California
Department of Education, WestEd, and Sonoma State
University
2
Workshop Overview

• Introduction
• Reporting and School Level Results
• Events and Developments
• Utilization and Capacity Building
• ASAM in an NCLB World

3
Introduction

• ASAM Development A brief overview
• Implementation Experience
• Continued Field Involvement

4
Why a separate accountability system for
alternative schools serving very high-risk
students?

• State law The Public Schools Accountability Act
  requires the development of an accountability
  system for these schools.
• Very few ASAM schools have sufficient valid test
  scores to receive a school-level API.
  Accountability data for most ASAM schools will be
  rolled up into the district/county office AYP.
• There are technical limitations of API/AYP for
  most ASAM schools, including constantly changing
  student populations and student population
  characteristics that limit full application of
  those models.
• ASAM data can supplement API/AYP, focusing upon
  appropriate aspects of school performance.

5
What are the roles of ASAM in providing
accountability for alternative schools serving
very high-risk students?

• Make it possible to hold all schools accountable
  for student performance
• Show reliable, valid, feasible, and fair
  accountability for students and schools
• Provide supplemental accountability information
  to API/AYP
• Provide back-up accountability information when
  there are insufficient test data to report an API

6
Accountability for Californias Schools Serving
Very High-Risk Populations
ASAM School Accountability State Law PSAA
API? - Yes
API? - No
ASAM is Supplemental To API/AYP
ASAM is Back-up To API/AYP
7
Principles of ASAM Accountability

• All schools must be held accountable for the
  achievement of their students.
• Accountability model for ASAM schools must
  include data indicators that are
• Consistent with those included in the states
  primary accountability system (API/AYP)both
  student achievement and school characteristics.
• Reliable and valid across a range of alternative
  school populations.
• Feasible at the local level.
• Representative of performance that reflects
  success for attaining goals of alternative
  schools serving very high-risk students.

8
ASAM System Development A Sharp Learning Curve
9
Introduction

• ASAM Development A brief overview
• Implementation Experience
• Continued Field Involvement

10
ASAM Impact Growth of Capacity
Results based upon survey of ASAM principals and
district administrators. Total responses 519
11
ASAM has had an impact on program assessment and
planning practices
Results based on May 2000 survey of ASAM
principals and district administrators.
12
Introduction

• ASAM Development A brief overview
• Implementation Experience
• Continued Field Involvement

13
Lessons from Californias Implementation of the
ASAM

• Bottom-up planning, development, and support is
  mandatory.
• Technical support and staff continuity are
  essential.
• School and district capacity are at times
  extremely limited.

14
PSAA Subcommittee on Alternative Schools
Accountability

• Subcommittee meets regularly
• Active representation by professional
  associations
• District and county staff participate
• Individuals involved in alternative education are
  encouraged to attend and participate

15
Workshop Overview

• Introduction
• Reporting and School Level Results
• Events and Developments
• Utilization and Capacity Building
• ASAM in an NCLB World

16
Number of ASAM Schools Selecting Each Indicator
for 2003-04
17
ASAM Timeline-Important Dates

• January 1, 2005, PATRS goes active for school
  year 2004-05
• June 15, 2005, ASAM ORS goes active for school
  year 2004-05
• June 30, 2005, Public Reporting of ASAM School
  Reports for school year 2003-04
• July 31, 2005, 2004-05 Pre-Post Data Deadline to
  PATRS
• September 30, 2005, Deadline for school year
  2004-05 ASAM ORS reporting
• November 30, 2005, Deadline for CDE receipt of
  school year 2004-05 ASAM ORS Certified Report
  Forms
• December 31, 2005, Public Reporting of ASAM
  School Reports for school year 2004-05

18
Outcomes and Findings

• Data appear stable across years
• When combined, indicators provide meaningful
  performance differentiation
• Very few schools performing poorly

19
Indicators Stable Across Years
Indicator 6 Attendance Rates
100
90
80
70
60
50
40
30
2002-03
2003-04
Reporting Year
20
Indicators Stable Across Years
Indicator 13a Percent HS Units Passed
100
90
80
70
60
50
40
30
2002-03
2003-04
Reporting Year
21
Question Where to Set Performance Thresholds?
ASAM Indicator 6 School Attendance
Number of Schools
65
84
95
Immediate Action
Growth Plan
Sufficient
Commendable
22
Problem What To Do With Indicators for Which
There Were Very Few Schools OR Data Had Little
Variance?
ASAM Indicator 11 Promotion Rate

Number of Schools
90
Did Not Meet Sufficient
Sufficient
23
Solution Some ASAM Indicators Have 2 and Others
Have 4 Performance Thresholds
24
Solution Some ASAM Indicators Have 2 and Others
Have 4 Performance Thresholds
25
Four Performance Levels Defined
26
Two Performance Levels Defined
27
Indicator 1 Expulsion Suspension
Recommendations
28
Indicator 2 Suspensions
29
Indicator 3 Student Punctuality
30
Indicator 4 Sustained Daily Attendance
31
Indicator 6 Attendance
32
Indicator 11 Promotion to Next Grade
33
Indicator 12ab Percent Courses Passed
34
Indicator 12c Average Courses Completed
35
Indicator 13a HS Credit Passage Rate
36
Indicator 13b HS Credit Average Completion Rate
37
Indicator 14 Graduation Rate
38
Indicator 15a GED Passage Rate
39
Indicator 15b GED Section Passage Rate
40
Workshop Overview

• Introduction
• Reporting and School Level Results
• Events and Developments
• Utilization and Capacity Building
• ASAM in an NCLB World

41
Events and Developments in 2004-05

• SBE approval of single year accountability status
• Development of multi-year status
• Final consideration of indicator change procedure
  
• Adoption of annotations and appeals process
• Pre-post assessment indicators and reporting
  2004-05 data to the PATRS

42
ASAM Single Year Accountability Status
Three Step Process Step 1 Performance
Indicators Determine schools status on each
ASAM performance indicator selected Step
2 Pre-post Assessments (if using) Determine
schools status on pre-post assessment
indicator, if selected Step 3 Determine Overall
ASAM Accountability Status
43
ASAM Single Year Accountability Status Step 1
Performance Indicators

• State Board-approved performance standards
  determine schools status on each selected
  performance indicator based on
• Experience in other states on similar indicators
• Two years of collected data from ASAM schools
• Credibility of data
• Research on achievement levels

44
ASAM Accountability StatusStep 1 Performance
Indicators
45
ASAM Single Year Accountability Status
Step 1a Determine overall school status on two
performance indicators
46
ASAM Single Year Accountability Status
Step 1b Determine overall school status on
three performance indicators
47
ASAM Multiple Year Accountability Status
Still In Development
Assumptions   Each school will receive two
accountability scores One-year score based on
combined performance on three selected indicators
using one-year decision rules Multi-year score
based on previous years accountability status
and current year performance on three selected
indicators using multi-year decision rules  
The two scores are essential they represent two
aspects of accountability status and growth and
the multi-year decision has more reliability
because it is based on more data.
48
ASAM System Is Oriented Towards Growth
49
The PSAA Subcommittee approved that in2005-06,
there will be a window within which an ASAM
school can change performance indicators.
The PSAA Subcommittee has approved that the
window to change ASAM indicators will be between
July 1 and December 31, 2005.
50
Parameters for Changing Indicators
Six parameters apply

• A school can change only one of their two
  originally selected indicators chosen in 2000-01
  or 2001-02 except when changing one indicator to
  a pre-post assessment indicator (8, 9 or 10).
• A school cannot change their third indicator
  (selected in 2003-04) until it has been used for
  three years unless the school chooses to select a
  pre-post indicator in place of the third
  indicator.
• 3. A school is not allowed to change an indicator
  on which the performance standard attained is
  Immediate Action.

51
Parameters for Changing Indicators

• A school seeking to change an indicator on which
  the performance standard attained is Growth Plan
  must submit a rationale for changing the
  indicator.
• A school that changes an indicator would be
  required to continue to collect data and report
  on the old indicator for one year following the
  adoption of the new indicator.
•  
• A school that is currently using an indicator for
  which their school does not meet the reporting
  requirements or conditions of that indicator
  should change to another indicator.

52
Appeal and Annotation Process

• Designed to ensure fair treatment of all ASAM
  schools
• Appeal process allows school to petition for
  review of ASAM accountability status when it
  believes data on which decision is made are
  invalid
• Annotation process (voluntary) allows school to
  submit contextual information in conjunction with
  ASAM accountability status, when appropriate
• Annotation does not overturn schools
  accountability status
• Sample conditions for submission of an
  annotation
• Significant growth occurred from previous year,
  but accountability status did not improve
• School characteristics vary widely from other
  ASAM schools or from previous years student
  population

53
Reporting 2004-05 Data to WestEds Pre-Post
Assessment Tracking and Reporting System (PATRS)

• Accessing the PATRS
• Log on to http//asam.wested.org
• Pre-post assessment database for indicators 8
  (writing), 9 (reading), and 10 (mathematics)
• 90-day students
• Data Reporting Options
• Option 1 Importing data from a file
• Option 2 Entering student data into the PATRS
• Submitting Data
• All schools selecting indicators 8, 9, or 10 must
  report pre-post data for 2004-05 to WestEd no
  later than July 31, 2005

54
PATRS Required Data Reporting Elements

• School Information
• School name
• School County District School (CDS) code
• Student Demographic Information
• Local student ID number
• Date of birth
• Current grade level
• Gender
• Race/Ethnicity
• Primary language
• Pre- and Post Assessment Score Information
• Test Name
• Score (Scaled)
• Pre- and post assessment administration date

55
Approved Pre-Post Measures of Achievement
Additional Pre-Post Assessments were reviewed in
2005 and recommendations for approval of
additional tests will be brought to the State
Board in July 2005.
56
PATRS - Help and Resources
http//asam.wested.org/pub/docs/21
57
PATRS Data Reporting Summary
http//asam.wested.org
58
Workshop Overview

• Introduction
• Reporting and School Level Results
• Events and Developments
• Utilization and Capacity Building
• ASAM in an NCLB World

59
Using ASAM to Build Capacity, Programs, and
Public Support
60
Using ASAM Information Locally Building
Capacity, Programs, and Public Support
May 2005 survey of ASAM principals Total
responses 264
61
An Adequate Student Data System is Key

• Prior to ASAM, many alternative schools serving
  high-risk students did not have access to a
  district or county data system.
• Often these schools have been at the bottom of
  the feeding chain when it came to access to
  information.
• Some districts lacked technical capacity or
  resolve.

62
Improved data systems will aid ASAM schools in
meeting reporting requirements.
Better data is essential for meaningful program
planning and assessment.
63
Using ASAM Information Locally Building
Capacity, Programs, and Public Support

• Two important activities
• Program assessment and planning
• Reporting to local boards and the public

64
Key to school improvement is the availability of
better student data
Regional Alliance suggestions on Ten ways to use
data as a lever for change

• Uncover problems that might otherwise remain
  invisible
• Convince people of need for change
• Confirm or discredit assumptions about students
  and school practices
• Pinpoint cause of problems and guide resource
  allocation

65
Key to school improvement is the availability of
better student data
Regional Alliance suggestions on Ten ways of
using data as a lever for change

• Evaluate program effectiveness keeping the focus
  on student results
• Provide feedback to teachers and administrators
• Prevent over reliance on standardized tests (and
  API/AYP measures)
• Prevent one-size-fits-all solutions
• Respond meaningfully to accountability questions
• Build school culture of inquiry and continuous
  improvement

66
The goal is to develop Data-Driven Decision
Making
Collect ASAM and other student data
Determine school improvement needs
Create data-based improvement plan
Review and analyze
67
Features of a data-based improvement plan

• Uses data to link outcomes to instructional
  practices
• Establishes goals and priorities with measurable
  outcomes
• Defines effectiveness in terms of meeting goals
  and targets
• Provides continuous monitoring and feedback to
  teachers and schools

68
Other Uses For ASAM Data

• WASC accreditation
• Parent information packets
• Grant and funding applications
• Local and public information

69
Workshop Overview

• Introduction
• Reporting and School Level Results
• Events and Developments
• Utilization and Capacity Building
• ASAM in an NCLB World

70
Role Of ASAM In An NCLB World

• Provides important information recognized by the
  state
• Allows presentation of a more comprehensive
  picture of school effectiveness
• Recognizes the distinctiveness of ASAM student
  populations and goals

71
Accountability for Californias Schools Serving
Very High-Risk Populations
ASAM School Accountability State Law PSAA
API? - Yes
API? - No
ASAM is Supplemental To API/AYP
ASAM is Back-up To API/AYP
72
Major Problems Using Regular Accountability
System for ASAM Schools

• Relatively few ASAM schools have valid API
• Few schools meet NCLB Math or ELA percent
  proficient requirements
• Even fewer will meet requirements in coming years

73
ASAM Schools With 11 or More Valid Test Scores in
2004
74
API Scores for ASAM Schools are NOT Keeping Pace
with Increase in API Proficiency Levels
75
Few ASAM High Schools Meet NCLB Percent
Proficiency in English Language Arts
76
Few ASAM High Schools Meet NCLB Percent
Proficiency in Math
77
Justification For Continuation of ASAM
Accountability System

• State law The Public Schools Accountability Act
  requires the development of an accountability
  system for alternative school serving very
  high-risk students.
• School-level Accountability The majority of ASAM
  schools (approximately 60 percent) continue not
  have sufficient valid test scores to receive a
  valid school-level API.
• Technical limitations Very high mobility and
  other student characteristics limit the validity
  of the API/AYP model for most ASAM schools.
• Valid and reliable accountability for ASAM
  schools requires development of ASAM-specific
  accountability status.

78
ASAM High Schools Projected to Meet Percent
Proficiency ELA
79
ASAM High Schools Projected to Meet Percent
Proficiency Math
80
Website Information

• ASAM
• http//ww.cde.ca.gov/ta/ac/am
• asam_at_cde.ca.gov
• vkenneth_at_cde.ca.gov (ASAM ORS pin information)
• PATRS
• http//asam.wested.org
• http//asam.wested.org/pub/docs/27 (Workshop
  documents and Webcast archive)
• AYP
• http//www.cde.ca.gov/ta/ac/ay/index.asp
• evaluation_at_cde.ca.gov
• API
• http//www.cde.ca.gov/ta/ac/ap/index.asp
• aau_at_cde.ca.gov
• Program Improvement
• http//www.cde.ca.gov/ta/ac/ti/programimprov.asp

81
Contact Information

• Robert Bakke, CDE
• rbakke_at_cde.ca.gov 916.445.7746
• Rose Loyola, CDE
• rloyola_at_cde.ca.gov 916.323.2564
• Don Dixon, Sonoma State University
• sra_at_sonic.net 707.829.0829
• Joy Lewis, WestEd
• jlewis_at_wested.org 415.615.3286
• Evelina Du, WestEd
• edu_at_wested.org 415.615.3142
• John Burns, CDE
• jburns_at_cde.ca.gov 916.322.5015