Some Implications of Large Scale Assessment for MSPs

1
Some Implications of Large Scale Assessment for
MSPs
Accountability that only focuses on identifying
schools is not going to improve schools. Brian
Gong

• Marge Petit
• Center for Assessment
• May 2004

2
Some Definitions

• Large-scale assessment
• Standards
• Standards-based large-scale assessment
• Grade level expectations and
• Grade level assessments.

3
NCLB and Challenges (Jim Pelligrino, NRC
Conference 2004)

• Designing and implementing assessment systems to
  respond effectively to the multiple provisions of
  No Child Left Behind
• Defining the appropriate targets for assessment
  relative to content standards
• Determining how targets might be apportioned
  across different elements of a comprehensive
  assessment system and
• Developing and supporting assessment systems that
  provide information to support the enhancement of
  learning and instruction as well as
  accountability.
• The multiple sets of goals and needs is a major
  conceptual and operational design challenge.

4
Primary Goal of MSPs
To improve student learning in mathematics and
science!
5
Givens

• All schools associated with MSPs are responsible
  for improving student learning in mathematics and
  science as they relate to state standards and
  grade level expectations
• All schools associated with MSPs are responsible
  for improving student performance on state
  assessments
• All schools are responsible for making Adequate
  Yearly Progress in mathematics as defined by each
  state consistent with requirements of NCLB
• The state level assessment is currently the basis
  for determining AYP
• One measure of the effectiveness of your MSP will
  probably be an increase in the percentage of
  student at proficient or above on state grade
  level assessments.

6
Implications for MSPs

• The quality of grade level assessments will
  impact your work and mathematics and science
  instruction and learning. Learn about the issues
  facing your state, and get involved in the
  ongoing development of the grade level
  assessments in your state.
• The quality of grade level expectations will
  impact your work and the quality of mathematics
  and science instruction and learning. All that is
  valued in mathematics and science may or may not
  be articulated in state standards and grade level
  expectations. Find the gaps and fill.
• Large scale assessment CANNOT provide ALL the
  information that is needed to improve
  mathematics/science instruction and learning.
  Build a coherent and balanced assessment system
  to support MSP schools.

7
Implications for MSPs

• The quality of grade level assessments will
  impact your work and mathematics and science
  instruction and learning. Learn about the issues
  facing your state, and get involved in the
  development of the grade level assessments in
  your state.
• The quality of grade level expectations will
  impact your work and the quality of mathematics
  and science instruction and learning. All that is
  valued in mathematics and science may or may not
  be articulated in state standards and grade level
  expectations. Find the gaps and fill.
• Large scale assessment CANNOT provide ALL the
  information that is needed to improve
  mathematics/science instruction and learning.
  Build a coherent and balanced assessment system
  to support MSP schools.

8
Some Real time - Practical and Political issues
interacting with technical and educational issues

• Alignment
• Testing time time of year
• Turn-around time and the impact on item types
• Item types and Depth of Knowledge
• Integrity of the content
• Level of information derived
• Access for the greatest number of students
• Science Lite
• Standard setting and
• Implementation cost.

9
Dimensions of Alignment (e.g., Webb, 1997
Achieve, Inc., Porter, 2002 WestEd)

• Content
• Cognitive demand (e.g., Depth of Knowledge
  (Webb) Level of Challenge (Achieve)
• Balance
• Range

10
Porter Alignment on 2 dimensions
Porter alignment dimensions
11
Range of items
Wyoming BOE
12
An Alignment Quiz
13
How much alignment is enough?
A The state standards and state assessment
should be perfectly matched/aligned. (True or
False Why or why not?)
14
How much alignment is enough?
B The state standards should be a subset of the
objectives assessed on the state assessment.
15
How much alignment is enough?
The objectives assessed on the state assessment
should be a subset of the state standards/grade
level expectations.
16
State Standards/Grade Level Expectations

• Mentally add and subtract whole numbers to .
• Design and conduct an experiment to answer a
  student or teacher generated experiment

17
Alignment
Local Curriculum Students will demonstrate
conceptual understanding of linear relationships
as a constant rate of change by determining the
slope of a line, explaining the meaning of slope
and intercept in context from a table,graph or
situation, and by solving routine and non-routine
problems involving the relationship between the
rate of change, slope, and intercept.
18
Two Types of Grade Level Expectations

• Test Specification for the large scale assessment

• Specification for local curriculum and assessment.

Test Specification for large-scale
19
Some Characteristics of Two Types

• Local Curriculum and Assessment
• Can include concepts and skills not easily
  assessable in an on-demand setting
• Can include foundational skills as they develop
  across grades.

• Test Specification
• Must be assessable in an on-demand large-scale
  setting
• Should be a prioritized set.

20
Examples

• Local Curriculum and Assessment
• GLE M26 In response to a teacher or student
  generated question or hypothesis, collects
  appropriate data, organizes the data,
  displays/represents the data and makes
  observations about the data to draw conclusions
  about the questions or hypothesis being tested.
  (Local Vermont GLE)

• Test Specification
• MDSP31 Interprets a given representation (line
  plots, bar graphs, tally charts, or tables) to
  answer questions related to the data, to analyze
  the data to formulate conclusions, or to make
  predictions. (New England Common Assessment
  Program)

21
Why a balance of item types?

• To adequately assess GLEs within a given time
  period
• To appropriately assess the content and cognitive
  demand in GLEs and
• To appropriately reflect instructional and
  assessment practices consistent with GLEs.

Large scale assessments should be substantially
consistent with high quality classroom
assessments though procedurally separate.
Shepard 2000
Link to item type instruction
22
Item Types
(1) you get what you assess... and (2) you do
not get what you do not assess... (Resnick
Resnick, 1992, p. 59)
Generalize a variety of patterns, and represent
the patterns symbolically.
23
Item Types
(1) you get what you assess... and (2) you do
not get what you do not assess... (Resnick
Resnick, 1992, p. 59
Generalize a variety of patterns, and represent
the patterns symbolically.
24
Science Lite Ed Week Jan. 2004

• Science Lite
• Items that dont require science
• Item with technical design flaws

39 state study - Carnine, Miller Metzenberg
25
Science Lite Ed Week Jan. 2004

• Doesnt require science

• A lever is a bar that turns around a fixed point
  called a fulcrum. A pair of scissors is made of
  two levers that move in opposition an
  illustration of a pair of scissors are provided
  with A D. Which of the following points is the
  fulcrum of the two levers?
• Point A
• Point B
• Point C
• Point D

26
Science Lite Ed Week Jan. 2004

• Science Lite

27
Access for the greatest number of students

• provide the greatest number of students the
  opportunity to demonstrate their knowledge and
  skills in relationship to the mathematical or
  science construct being assessed.

Universal Design
28
Conserving the Mathematical Construct (Petit and
Lager 2003)

• Explicitly aligning items with mathematical
  construct being assessed content and process
  demands in GLE
• Make intentional decisions of when and how to use
  context to assess the mathematics construct
• Streamlining language (Lager, 2003) to provide
  access without compromising mathematical
  construct being assessed
• Appropriately using graphics, pictures, graphs,
  tables, diagrams, and models.

29
Implications for MSPs

• The quality of grade level assessments will
  impact your work and mathematics and science
  instruction and learning. Learn about the issues
  facing your state, and get involved in the
  development of the grade level assessments in
  your state.
• The quality of grade level expectations will
  impact your work and the quality of mathematics
  and science instruction and learning. All that is
  valued in mathematics and science may or may not
  be articulated in state standards and grade level
  expectations. Find the gaps and fill.
• Large scale assessment CANNOT provide ALL the
  information that is needed to improve
  mathematics/science instruction and learning.
  Build a coherent and balanced assessment system
  to support MSP schools.

30
At the foundation of any assessment or
accountability system based on student
performance is the framework of concepts and
skills upon which they are to be built.
31
The ultimate validity test
If a school is identified under NCLB for low
performance in mathematics, the response by the
school should be to strengthen the mathematics
instruction based upon a solid set of grade level
expectations. Tim
Kurtz, NH Assessment Director
32
Primary Goal of MSPs
To improve student learning in mathematics and
science!
MSP Responsibility Go beyond the bullets on the
GLEs to support curriculum, instruction, and
assessment with deep and lasting understanding of
concepts and skills.
33
What does it mean to develop deep and lasting
understanding of. ?

• New Jersey Grade 2---
• By the end of Grade 2, students will
• A. Number Sense
• 1. Use real-life experiences, physical materials,
  and technology to construct meanings for numbers
  (unless otherwise noted, all indicators for grade
  4 pertain to these sets of numbers as well).
• Whole numbers through millions
• Commonly used fractions (denominators of 2, 3,
  4,5, 6, 8, 10, 12, 16)
• 2. Demonstrate a sense of the relative magnitude
  of numbers.
• 3. Understand the various uses of numbers
• 4. Use concrete and pictorial models to relate
  whole numbers, commonly used fractions, and
  decimals to each other, and to represent
  equivalent to forms of the same number
• 5. Compare and order whole numbers.
• 6. Compare and order numbers.
• 7. Explore settings that give rise to negative
  numbers.

34
What does it mean to develop deep and lasting
understanding of. ?

• By the end of Grade 4, students will
• A. Number Sense
• 1. Use real-life experiences, physical materials,
  and technology to construct meanings for numbers
  (unless otherwise noted, all indicators for grade
  4 pertain to these sets of numbers as well).
• Whole numbers through millions
• Commonly used fractions (denominators of 2, 3,
  4,5, 6, 8, 10, 12, 16)
• 2. Demonstrate a sense of the relative magnitude
  of numbers.
• 3. Understand the various uses of numbers
• 4. Use concrete and pictorial models to relate
  whole numbers, commonly used fractions, and
  decimals to each other, and to represent
  equivalent to forms of the same number
• 5. Compare and order whole numbers.
• 6. Compare and order numbers.
• 7. Explore settings that give rise to negative
  numbers.

35
An Example Does this sequence promote deep and
lasting understanding
Shade 1/3 of the figure. What is 1/3 of 27?
36

• Proper fractions (denominators of 2, 3, 4, 8, 10)

• Students progress through four levels of
  partitioning sharing algorithmic halving
  evenness oddness and composition. (Pothier and
  Sawada, 1990, cited in Bezuk and Bieck, 1993,
  pp.124 125)
• Fair share means the same number of pieces or
  parts regardless of their size. Partitions are
  drawn in a shape for the correct number of parts,
  disregarding the size of the parts (e.g., Cutting
  a circle with four vertical lines). (Payne, 1975)

37
Shared with permission of the Vermont Mathematics
Partnership
38
Questions about GLEs and local curriculum

• Do the GLEs promote and support quality
  instruction and assessment in mathematics and
  science?
• Have aspects of science and mathematics not
  easily assessed in large scale, but important,
  been included for local curriculum and
  assessment?
• Is local curriculum and assessment aligned with
  state GLEs?
• Are the GLEs and local curriculum prioritized?
• Other?

39
Implications for MSPs

• The quality of grade level assessments will
  impact your work and mathematics and science
  instruction and learning. Learn about the issues
  facing your state, and get involved in the
  development of the grade level assessments in
  your state.
• The quality of grade level expectations will
  impact your work and the quality of mathematics
  and science. All that is valued in mathematics
  and science may or may not be articulated in
  state standards and grade level expectations.
  Find the gaps and fill the, but do not support
  mile wide and inch deep curriculum or
  assessment. Help Prioritize!
• Large scale assessment CANNOT provide ALL the
  information that is needed to improve
  mathematics/science instruction and learning.
  Build a coherent and balanced assessment system
  to support MSP schools.

40
Large Scale Assessments are Not valuable for

Lorrie Shepard, Assessment in Support of Learning

• Providing detailed understanding of individual
  student learning on an ongoing basis!
• Capturing synthesis of knowledge or some concepts
  that are not easily assessed in large-scale
  assessment.

41
Local Assessment Webb, DoK Level 4 Curriculum
Embedded Assessment
42
How balanced is the assessment system in MSP
schools?

43
Characteristics of a Coherent and Balanced
Assessment System

• Comprehensive (Adapted from Knowing What Student
  Know(NRC, 2001)
• Coherent (Adapted from Knowing What Student
  Know(NRC, 2001)
• Continuous (Adapted from Knowing What Student
  Know(NRC, 2001)
• Integrated (Distilled from other reports listed
  on last page)
• Includes quality assessments. (Distilled from
  other reports listed on last page)

44
What do I know? What do I need to find out? What
actions should I take?

• Assessment design issues facing my state
• The design of grade level assessments
• The nature of school and student level
  information derived from large scale assessment
• The status of the grade level expectation
  development and implementation
• The degree to which educators have the range of
  information needed to make program adjustments
  and to make instructional decisions on time

45
References
Bezuk, N. S., and Bieck, M. (1993). Current
Research on Rational Numbers and Common
Fractions Summary and Implications for Teachers.
In D.T. Owens (Ed.), Research Ideas for the
Classroom Middle Grades Mathematics (pp.
118-136). New York Macmillan. Lager, C.,
Petit, M., (2003) Conserving the Mathematical
Construct, TSNE Test Specifications v. 12,,
National Research Council (2001a), Adding it Up
How Children Learn Mathematics. Mathematics
Learning Study Committee, Jeremy Kilpatrick, Jane
Swan, and Bradford Findell (Eds.)., Washington
D.C., National Academy Press. National Research
Council (2003), Assessment in Support of Student
Learning Bridging the Gap Between Large-scale
and Classroom Assessment. Mathematical Science
Education Board, Board on Testing and
Measurement, and Committee on Science Education
Washington D.C., National Academy Press.
National Research Council (2001b), How People
Learn Brain, Mind, Experience, and School.
Committee in the Development of the Science of
Learning. John Bransford, Ann Brown, and Rodney
Cocking (Eds.) Washington D.C., National Academy
Press. National Research Council (2000), Knowing
What Students Know The Science and Design of
Educational Assessment. Committee on the
Foundations of Assessment. James Pelligrino,
Naomi Chudowsky, and Robert Glaser (Eds.)
Washington D.C., National Academy Press. National
Research Council (2000), Learning about
Assessment, Learning Through Assessment,.
Mathematical Sciences Education Board. Mark
Driscol and Deborah Bryant (Eds.) Washington
D.C., National Academy Press. National Research
Council (2000), Classroom assessment and the
national science education standards, . J. Myron
Adkins, Paul Black, and Janet Coffey (Eds.)
Washington D.C., National Academy Press. Payne,
J. N.. (1975) Review of research on fractions. In
R. Lesh and Bradbard (Eds.) Number and
measurement Papers from a research workshop.
(Page 145 187), Columbus, Ohio. Petit, M.,
(2003) Aligning to What?, RILS Annual Meeting.
www.nciea.org Popham, J. In press (2003)
Crafting Curricula Aims for Instructionally
Supportive Assessment Webb, N., (2002) Resnick,
L. B., Resnick, D. P. (1992). Assessing the
thinking curriculum New tools for educational
reform. In B. R. Gifford M. C. O'Conner (Eds.),
Changing assessments Alternative views of
aptitude, achievement and instruction. Boston
Kluwer Academic