State Reform Efforts: Multiple Measures of Accountability

1
State Reform Efforts Multiple Measures of
Accountability Improvement

• Institute for
• Governors Education Advisors
• NGA Association for Best Practices
• Amelia Island, Florida November 3-5, 2001
• Mari Pearlman

2
The current landscape dangers and opportunities

• Dangers
• Single focus on annual test scores
• Lack of useful data to interpret these scores
• Little connection between testing and instruction
• Search for the guilty rather than search for
  success
• Opportunities
• Systemic improvement for all parts of the
  education system
• Useful data collection and interpretation to make
  improvement an empirical rather than rhetorical
  and emotional issue

3
Protection against the dangers first steps

• Align the entire accountability system, with
  particularly careful connections between what is
  taught and what is tested
• Establish direct feedback loops between test
  score information and teacher instructional
  planning
• Use multiple different measures of student
  achievement, including but not limited to
  standardized tests
• Employ sensible data collection and analysis plans

4
Aligning the accountability system - 1

• Criteria or standards for learning
• A clear definition of what students will be able
  to do to demonstrate that learningthe
  specifications for assessments, both ongoing
  (classroom-based) and summative (school,
  district, state-based)
• A coherent continuum for applying those criteria
  by developmental level of students

5
Aligning the accountability system - 2

• A curriculum that supports the criteria for
  learning and includes demonstrations of learning
  that are aligned with the assessments
• Instruction that implements the curriculum, with
  continuing teacher professional development
• A plan for collecting data that defines what
  questions the data should answer and then
  specifies the data to be collected
• A plan for using those data to inform curriculum,
  instruction, teacher professional development,
  and assessment designs

6
Scope and sequence Content, performance
assessment

• The sequence of alignment matters
• standards for what students must know (and
  teachers must teach)content standards
• Standards for how much of the content students
  must know performance standards
• A definition of what learning looks likewhat
  should studentsat each level--be able to do if
  they meet the performance standardsassessment
  specifications

7
Scope and sequenceCurriculum Instruction

• A careful examination and articulation of what
  should be taught, when it should be taught, and
  how the knowledge and skills articulated are
  connected to the final expectations for student
  learningthe curriculum
• Classroom instruction that really focuses on the
  curriculumhelping teachers make good choices
  about what, of all possible things, should be
  taughtteacher mentoring and evaluation and
  professional development

8
Scope SequenceData collection and
interpretation

• What questions do you want to be able to answer
  about student learning and the effectiveness of
  the education system?
• To answer those questions, what data would you
  need to collect?
• How can you use these collected data to close the
  loop and improve the criteria for student
  learning, the curriculum, the instruction, and
  teacher professional development?

9
Realistic Expectations

• Think realistically about milestones for student
  learning and their connections with annual
  testing
• What is the gain you expect to see from year to
  year?
• Given a particular range of performance in year 1
  (say 1st quartile on the test, or lowest 25),
  what is reasonable to expect as a growth goal in
  the next year?

10
Check those expectations against

• What the curriculum says teachers should teach
• How much actual instructional time is devoted to
  these areas of learning at each level
• What kind of assessment data you actually have
  that would be evidence of learning against these
  expectations

11
And against

• Whether assessment data are ever available in
  time for teachers to change the way they teach
  particular students
• Whether teachers are given any structured
  guidance about the use of assessment data,
  changes in instruction to serve student learning,
  or aligning their practice with ultimate goals
  for student learning

12
Using multiple measures of educational
effectiveness outputs

• Output measures
• Multiple formal assessments this is not
  impossible. All teachers in a school or district
  can come up with easy-to-use performance measures
  that could supplement norm-referenced
  standardized results.
• Informal assessments teacher standardized
  assessment at the beginning and end of each
  yearcreate a standard form and keep those data.
• Grades and teacher evaluations of student
  performance
• Student performance itselfthe science fair, the
  music program, the art exhibit, the auto shop,
  the contests and awards from disciplinary
  organizations

13
Multiple measures 2 inputs

• Teacher profiles-
• Number with full certification
• Number with content majors and advanced degrees
• Years of experience
• School leadership profiles
• Teaching experience
• Training and experience as a school leader
• Demographic variables that matter
• of students who come and go within a teaching
  year
• of drop outs
• Absenteeism

14
Realistic Expectations

• Recognize that this is an ongoing, continuous
  processthis is about human growth from birth to
  age 21
• Student learningin particular, not enough of
  itis not like an infection that can be cured
  with a massive dose of antibiotics
• Everyone must be accountablein particular, it is
  not the fault of the students who are struggling.
  Improving student achievement is not just the
  job of teachers. Parents, principals,
  superintendents, legislators, the public all have
  a role and responsibility