The use of formative assessments: Perspectives from district and school leaders

1
The use of formative assessmentsPerspectives
from district and school leaders
Leslie Nabors Oláh Consortium for Policy
Research in Education, University of Pennsylvania
CCSSO Summer Leadership Training
Conference, Boulder, CO
August 2, 2006
leslieno_at_gse.upenn.edu
2
Acknowledgements

• This presentation is based upon work supported
  by the National Science Foundation under contract
  number (REC-0529485). Opinions expressed in this
  presentation are those of the author and do not
  necessarily reflect the views of the National
  Science Foundation.
• Margaret Goertz, Nancy Lawrence, and John
  Weathers (CPRE) have contributed greatly to this
  research. We would like to thank the two
  participating school districts for their time and
  for their thoughtful responses to our questions.

3
Research Overview

• Framework based on the new Center on Continuous
  Instructional Improvement (CCII), funded by the
  Hewlett Foundation.
• The purpose of assessment is to inform decisions
  about how to respond pedagogically to students
  current state in ways that increase learning.

4
Cycle of Instructional Improvement
DISTRICT CONTEXT
Teaching and learning Curriculum Goals Assessm
ent use Incentives for improvement Data systems
Accountability State District School
Professionalization Professional
development Norms about assessment
Interpret evidence
Use evidence to improve instruction
(differentiate, etc.)
Cycle of Improvement
Gather evidence about standard achievement and
student learning
Carry out improved instruction
School-level culture and routines
Data
School leadership
Schedules
Professional development
SCHOOL CONTEXT
Marshall S. Smith, Hewlett Foundation, 2004
5
Common Language

• Formative assessment
• Benchmark assessment
• Coach

6
Research Questions

• How do elementary teachers learn from formative
  assessment data in mathematics and apply that
  knowledge to instructional decisions about
  content and pedagogy?
• What policies and conditions affect the ability
  of teachers to learn from and make use of
  assessment data?

7
Sites

• Nine schools chosen based on AYP status, math
  performance, and socio-demographic
  characteristics.
• Abington, PA
• Small, suburban district (7 elementary schools).
• Has used Everyday Math (EDM) since 1991.
• District-created formative assessments every 4-6
  weeks.
• Philadelphia, PA
• Large, urban district (177 elementary schools).
• Has used Everyday Math (EDM) since 2003.
• One school uses Investigations on waiver.
• Co-created (district and Princeton Review)
  benchmarks given every six weeks.
• Scores posted on SchoolNet Information Management
  System.
• Monthly SchoolStat principal meetings with
  district leaders.

8
Analysis

• Findings based on initial round of interviews
  with district leaders, school principals, and
  math coaches in Spring 2006.
• Qualitative analysis of participant interviews.
• We will observe instruction and interview
  teachers in SY 2006-2007.

9
Cycle of Assessment and Instruction Abington, PA
Day 28-29 Teachers receive practice end-of-unit
test (benchmark) from math science
coordinator (MSC) (graded, but not counted) a
few days before end-of-unit test. Teachers
administer and score practice test.
Day 1-28 Teachers follow EDM curriculum.
Day 30 Low scores may be reported to ECS or Math
Coach to help with remediation.
Day 30 Teachers use flexible grouping based on
test results.
Day 39 ECSers and teachers (with or without MSC)
may meet to discuss instruction, individual
remediation, or further professional development
as a result of test scores.
Day 39 MSC reviews scores with ECSers once a
month, with an eye toward concepts, classes,
individual remediation, enrichment, and
individual items.
Day 30-32 Teachers review material for 2-3 days.
Day 38 Teachers examine data individually for
flexible grouping.
Day 33 Teachers receive end-of-unit test
(benchmark) from math science coordinator and
administer to students.
Day 38 ECSers print out spreadsheets (by class)
and put in binder for MSC.
Day 35-37 Teachers enter scores into customized
spreadsheet and email to their Elementary
Curriculum Specialists (ECS) within one week of
test administration.
Day 34 Teachers score test.
10
Cycle of Assessment and Instruction
Philadelphia, PA
Days 26-30 Teachers are encouraged to review or
extend math content.
Days 1-25 Teachers follow EDM curriculum.
Day 31-35 Teachers either receive and administer
paper-and-pencil tests or students take benchmark
assessments online (Princeton Review/SchoolNet).
Days 37-43 Based on item analysis, teachers
re-teach as needed. Teachers are encouraged to
collaborate on data interpretation. The pacing
guide allows for six days of review before
teachers must proceed with the next instructional
unit.
Day 36 Princeton Review scores benchmark
assessments and scores are posted online
(SchoolNet).
Day 37 Teachers and principals complete their
respective Benchmark Data Analysis Protocols.
Teachers should turn in worksheet to their
principal.
Day 36 Parents can access childrens scores
through SchoolNet.
Day 36 Principals and coaches can access scores
through SchoolNet.
Day 36 Teachers can access their students
scores through SchoolNet. They can perform item
analyses, group by content area or can access
individual student performance.
11
Overview of findings

• What are the expectations for benchmark
  assessment use?
• How are the assessments aligned with the
  curriculum and with other assessments?
• What are the current supports for assessment use?
• How are data used to improve instruction?
• What constraints affect optimal implementation of
  the benchmark assessment system?

12
Expectations for assessment use

• Expectations for district
• Create instructionally actionable assessments
• Identify district-wide PD needs.
• Expectations for principals
• Identify children needing remediation
• Identify school PD needs
• Follow up on teachers data-driven instruction
• Inform School Improvement Plan
• Collaborate with fellow principals on how to
  raise student achievement.

13
Expectations for assessment use

• Expectations for teachers
• Identify children needing remediation and provide
  appropriate instruction
• Identify children qualifying for enrichment and
  provide appropriate instruction
• Review test with class to provide all students
  with feedback
• Form flex groups and provide groups with
  appropriate instruction
• Re-teach to whole group
• Expand instructional repertoire
• Use data as a vehicle for collaboration with
  fellow teachers
• Ask school leaders for instructional support
• Plan or modify curriculum
• Communicate with parents about assessment results
  (if child not doing well)
• Reflect on instruction.

14
Alignment

• District curriculum and assessment directors take
  responsibility for aligning the benchmark
  assessments with state standards and with
  Everyday Math. Test construction occurs over the
  summer (Abington) or in weeks prior to
  administration (Philadelphia).
• In Abington, math coaches contribute to test
  construction, while there is little participation
  from school-based coaches or teachers in
  Philadelphia.
• In Philadelphia, the assessment goes through
  several iterations between the district and
  Princeton Review.

15
Alignment

• Concepts covered on the benchmark call attention
  to areas that teachers need to emphasize
• Math Coach Even the kindergarten and first
  grade teacher didnt realize, Oh, I never
  realized what we do with symmetry was so
  important. Just because its not covered in
  third grade, well thats where theyre getting
  their background for the PSSA, from you.
• Principal They were more aware. They just
  have to really reflect what they do. And I think
  they got to see that by giving the test.

16
Alignment

• Standards-based benchmark assessments have lead
  some teachers to shift from norm-referenced
  evaluation to criterion-referenced feedback.
• Principal Typically here the teachers have been
  pretty, I would say, rigorous graders. But it was
  easier for them to be rigorous graders with A, B,
  C. I think the shift is making it a little bit
  more challenging, because its standard-based.
  You know, A, B, C-- What does a D in English or
  reading or math mean? What does that mean? What
  does that tell me that they know or they dont
  know? You know, its just like-- Like, its good
  as an average, which doesnt give you much. Its
  just like taking all the colors and making it
  grey.

17
Alignment

• Districts have had to find ways to address format
  non-alignment between the benchmark assessment
  and the state test. The Abington curriculum team
  creates additional open-ended items. Philadelphia
  has contracted with a local university to provide
  access to open-ended problems.
• District leaders are looking for open-ended
  supplements to benchmark assessments, but have
  not found a provider who can score items in an
  adequate amount of time.

18
Supports

• In the large district, SchoolStat meetings
  provide support for principal data use, but
  contribute to accountability pressure.
• Principal Benchmarks, when they first came
  out, it was something that was going to be used
  just by the schools. Its grown. Theyve almost
  became lower PSSAs, because everybody looks at
  it. And so, the pressure is on with these
  benchmarks. Whereas before, it was kind of like
  we were going to use it, and nobodys going to
  know about it but us.

19
Supports

• Professional development and support for data use
  is seen as part of the everyday work of the
  school math coach or computer support personnel.
• Coaches, however, have many other
  responsibilities
• Conducting school and district PD
• Teaching (own class, pull-out, covering)
• Collaborate with school psychology team
• Discipline
• Lunchroom duty
• Tech support
• Whatever the principal needs
  jack-of-all-trades.

20
Supports

• Perhaps as a result, professional development for
  data use remains undefined and focused on
  gathering evidence and interpreting data.
• Still, we found some examples of support and
  planning for improved instruction.
• Math coach Sometimes were able to talk it
  through, because Ill say, You know, what do you
  think is happening here? And a lot of times, it
  requires some talking it through. I might say,
  Well, how much practice did they get in this?
  Was it presented in homework or in a daily-- You
  know, in your daily math message in the same way?
  You know, how much practice did they have with it
  in the same way it was presented on the test?
  ...And Ill say, How many times throughout your
  instruction, throughout this story, did these
  kids have to answer, in writing, an open-ended
  question? laughter And thats the response I
  get. So, and it sort of opens their eyes, because
  theyre doing what theyre told, you know, and
  theyre doing it the way the book is telling
  them. And thats sometimes a problem. And when
  you say, you know, when they look at me and they
  dont understand because they did exactly what
  the book told them to do. And sometimes teaching
  requires you to go beyond that and pull from a
  bag of tricks.

21
Data Use

• District and principal reaction to the benchmark
  assessment use tends to be positive overall.
• Philadelphia credits assessments with a large
  part of rising student achievement.
• Seen as a lens into the classroom.
• However, principals in Philadelphia note that
  SchoolStat meetings have changed the low-stakes
  benchmark assessments into high-stakes tests.
• In addition, principals often view benchmark
  assessment data with an eye toward AYP.

22
Data Use

• Data used primarily for remediation, which is
  addressed largely through flex grouping or
  re-teaching.
• Benchmark assessment data used to construct flex
  groups may be based on only a few items.
• Flex groups may actually remain stable over long
  periods of time, even years.

23
Data Use

• Data also used for accountability.
• In some schools, accountability pressures may be
  the driving force behind data analysis.
• Principal Well, we have to be mindful of it
  because-- It is monitored. It is monitored.
  Everything is monitored now. Because everything
  is online. It is monitored .

24
Data Use

• Data also used for accountability.
• But, accountability not an intended use
• Curriculum Director I think benchmark tests are
  being misused at the classroom level. If were
  misusing it and were comparing schools and were
  comparing teachers and were comparing classrooms
  then theres pressure on teachers to perform. And
  so as soon as theres pressure on teachers to
  perform, that frequently is transferred to
  students which is not what this is intended to
  be

25
Data Use

• Principals who are instructional leaders as well
  as data users contribute to a culture of data
  use. Data not seen as an end in itself, but
  rather as a means to an end (e.g., discussing
  item analyses at grade-group meetings).
• But there is variation in the degree to which
  principals directly engage with data. Some
  principals use a designate, while others may
  spend hours looking at data.

26
Data Use

• Three common interpretation mistakes were
  mentioned
• Common practice of using benchmarks as indicator
  of growth over time, in spite of validity and
  reliability problems with doing so.
• Widespread belief that better benchmark
  assessment scores lead to better state-test
  performance in spite of lack of predictive
  validity evidence.
• Errors in aggregation and disaggregation of
  scores.

27
Implementation Issues

• Teachers lack of time restricts attention
  devoted to data.
• Math Coach We dont have enough time for
  collaboration. It either ends up being first
  thing in the morning, on the fly after school,
  trying to catch them on their prep or their
  lunch. And thats not always fair either. You
  know, we have that one collaboration time a
  month.

28
Implementation Issues

• Pacing is seen to have a great impact on the
  cycle of assessment.
• Disagreements have emerged over administration of
  testing (i.e., pencil and paper versus online
  administration). Curriculum leaders and
  principals argue for value of keeping student
  work, while IT argues for expediency of online
  administration.
• District and school leaders noted aspects of the
  benchmark assessment system that have attracted
  the attention of the Philadelphia Federation of
  Teachers.

29
Implementation Issues

• The pedagogical content knowledge of teachers may
  be limited, affecting their ability to interpret
  data and adjust instruction. In Philadelphia,
  math content knowledge of elementary school
  teachers has been classified anywhere from a
  disaster to good.
• Funding for professional development has been cut
  in Philadelphia.
• The regional math coach position has been
  eliminated, to be replaced with school-based
  growth coaches assigned to school not making
  AYP.
• Summer 2006 PD has been cancelled.

30
Summary
Interpret evidence
Gather evidence about standard achievement and
student learning
Use evidence to improve instruction.
Cycle of Improvement
Carry out improved instruction
School and district leaders realize that
data-driven instruction is here to stay. Do
teachers feel the same way?
31
Summary
Interpret evidence
Gather evidence about standard achievement and
student learning
Use evidence to improve instruction.
Cycle of Improvement
Carry out improved instruction
Districts and schools have numerous expectations
for data use, but tend to use data in more
limited ways
32
Summary
Interpret evidence
Gather evidence about standard achievement and
student learning
Use evidence to improve instruction.
Cycle of Improvement
Carry out improved instruction
Support for data-driven instruction seems to wane
during the last two or three steps of the cycle.
33
Summary
Interpret evidence
Gather evidence about standard achievement and
student learning
Use evidence to improve instruction.
Cycle of Improvement
Carry out improved instruction
Perhaps leading to common interpretation errors.
34
Summary
Interpret evidence
Gather evidence about standard achievement and
student learning
Use evidence to improve instruction
Cycle of Improvement
Carry out improved instruction
Still, district and school leaders report
instructional change as a result of benchmark
assessment data use. We will follow this up next
year.
35
The end?