Oregon Content Standards and Assessment System Evaluation

1
Oregon Content Standards and Assessment System
Evaluation

• Prepared for the Oregon Department of Education
• by WestEd
• Dr. Stanley Rabinowitz
• Dr. Edynn Sato
• May 18, 2007

2
Background Overview

• One of several key projects funded by the Gates
  Foundation
• Independent evaluation by WestEd, a non-profit
  educational research, development, and service
  agency

3
Content Standards and Assessment System
Evaluation Why?

• Content standards signal what knowledge and
  skills are valued (breadth and depth)
• Current state content standards have undergone
  periodic revision
• Need to continuously revisit and re-evaluate on
  a set schedule

4
Content Standards and Assessment System
Evaluation Why?

• Rigorous standards and assessments are major
  levers for improving student achievement
• Signal goals
• Focus instruction
• Provide information for improvement to the state,
  districts, schools

5
Content Standards and Assessment System
Evaluation What?

• WestEd will
• Review the content standards (all grades,
  academic content areas)
• Evaluate the structure of the content standards
• Evaluate alignment between the state assessments
  and the content standards
• Make recommendations for improvement of
  structures and systems

6
What?

• WestEd will address questions regarding
• the structure and quality the state content
  standards
• the alignment of state assessments to state
  standards
• Recommendations will be based on data from the
  analyses that have practice and policy
  implications
• Research-based information related to assessment
  and accountability models/practices will be
  provided as available

7
Key Deliverables

• Preliminary report of an initial review of
  selected content standards and grades using
  initial protocol/criteria
• Final report for the comprehensive review of the
  content standards
• Final report for the alignment of assessments to
  content standards
• Final report on the structure and quality of
  Oregons content standards and assessments

8
Content Standards and Assessment System
Evaluation How?

• WestEds independent evaluation will be
  conducted in three phases
• Preliminary Review of Selected Content Standards
  (already presented to Board)
• Comprehensive Evaluation of Content Standards
• Alignment Studies of Content Standards and
  Assessments

9
How?

• Comprehensive Evaluation of Content Standards
• English language arts
• Mathematics
• Science
• Social sciences
• Arts
• Second languages
• Physical education
• Health education
• Technology
• English language proficiency

10
How?

• Alignment Studies of Content Standards and
  Assessments
• Content
• English language artsReading and Literature
• Mathematics
• Science
• Items
• Oregons multiple-choice knowledge and skills
  state test items
• Tests
• One test blueprint/specification for English
  language arts, mathematics, and science

11
How?

• WestEds independent evaluation will involve
  analysts with expertise in
• large-scale test development
• standards development
• alignment
• measurement and statistics
• the content areas
• curriculum and instruction
• the K-12 student population
• No Child Left Behind (NCLB)
• Oregon programs and history

12
How?

• Stakeholder engagement
• DiscussionsThese discussions will not influence
  WestEds independent and objective analyses,
  rather they will inform the analyses
• Surveys (online)
• Parents Guardians (427 completed/593 total)
• Educators (286 completed/535 total)
• Policy Makers (18 completed/38 total)

13
How?

• Stakeholder groups include
• ODE staff, such as
• - Management Team
• - Curriculum and Assessment staff
• State Board of Education
• OAESD Instructional Leader Council
• PK-20 Coordination Advisory Group
• Literacy Leadership State Steering Committee
• Critical Friends Advisory Group

14
Content Standards and Assessment System
Evaluation When?

• December 1, 2006 through July 31, 2007
• Report of a preliminary review of selected
  content standards and grades using initial
  protocol/criteria (January 2007)
• Report for the alignment studies (March/April
  2007)
• Report for the comprehensive review of the
  content standards (April-June 2007)
• Final report on the structure of Oregons content
  standards and assessments (July 2007)

15
Status Alignment Studies

• The analyses aimed to address the following key
  questions
• To what degree do the State assessment items
  reflect the concepts and skills embodied in the
  States academic content standards?
• To what degree do the assessment items cover the
  breadth, depth, and range of complexity of
  content intended by the State?

16
Status Alignment Studies (continued)

• ELA Item Pool-Level Analyses
• Categorical Concurrence
• For all grades except for grade 4, the States
  assessment items had a high level of categorical
  concurrence overall as well as with each CCG in
  the grade-level standards.
• At grade 4, items had a high level of categorical
  concurrence overall and with each CCG in the
  grade-level standards, except for DGUL, DAIL, and
  ECSI, which had a low level of categorical
  concurrence.
• Range of Depth of Knowledge
• Overall, at each grade level, items covered a
  range of DoK. However, the range of DoK appeared
  restricted for some CCGs at each grade level.

17
Status Alignment Studies (continued)

• ELA Item Pool-Level Analyses (continued)
• Range of Knowledge (Comprehensiveness/Breadth)
• Overall, items covered the range of content
  represented in the standards at each grade level.
  With the exception of Grade 3, items appeared to
  address the breadth of content of each CCG, where
  the standards were assessable according to test
  specifications.
• Balance of Representation (Emphasis)
• Overall, at each grade level there was a balance
  of representation. However, the actual emphasis
  of content related to specific CCGs in the item
  pool and the intended emphasis of content related
  to the ELA score reporting categories per the
  test specifications were not entirely consistent.
  

18
Status Alignment Studies (continued)

• ELA Test-Level Analyses 
• The examination of items on the Grade 8 ELA
  assessment shows coverage of content across all
  six score reporting categories. The emphasis of
  content appears relatively consistent for VOCA,
  READ, and ECSL, less so for DGU, DAI, and ECSI.

19
Status Alignment Studies (continued)

• Math Item Pool-Level Analyses
•  Categorical Concurrence
• For all grades, the States assessment items had
  a high level of categorical concurrence overall
  as well as with each strand in the grade-level
  standards.
• Range of Depth of Knowledge
• Overall, at each grade level, items covered a
  range of DoK. However, the range of DoK appeared
  restricted for some strands at each grade level.

20
Status Alignment Studies (continued)

• Math Item Pool-Level Analyses (continued)
• Range of Knowledge (Comprehensiveness/Breadth)
• Overall, items covered the range of content
  represented in the standards at each grade, with
  the exception of Grade 6. At the strand level,
  items for Grades 7, 8, and CIM appeared to
  address the breadth of content of each strand
  however, this was not the case for Grades 3, 4,
  5, and 6.
•  Balance of Representation (Emphasis)
• Overall, at each grade level there was a balance
  of representation. However, the actual emphasis
  of content related to specific CCGs in the item
  pool and the intended emphasis of content per the
  test specifications are not entirely consistent.

21
Status Alignment Studies (continued)

• Math Test-Level Analyses
• The examination of items on the Grade 5
  mathematics assessments shows that overall the
  intended and actual distributions of content
  coverage at the strand level were comparable
  that is, no percentage difference exceeded 4.

22
Status Alignment Studies (continued)

• Science Item Pool-Level Analyses
• Categorical Concurrence
• For all grades, the States assessment items had
  a high level of categorical concurrence overall
  as well as with each CCG in the grade-level
  standards.
• Range of Depth of Knowledge
• For all grades and for all strands, except Earth
  and Space Science at CIM, the items represented a
  range of depth of knowledge levels from Recall to
  Strategic Thinking. In CIM, items in the Earth
  and Space Science strand were aligned to two DoK
  levels only Recall and Basic Application.

23
Status Alignment Studies (continued)

• Science Item Pool-Level Analyses (continued)
• Range of Knowledge (Comprehensiveness/Breadth)
• For Grade 5, the items in the test pool generally
  address the entire breadth of the strand content
  as operationalized in the benchmark standards and
  eligible content.
• For Grade 8, the items in the test pool generally
  address the entire breadth of the strand content
  as operationalized in the benchmark standards and
  eligible content.
• For CIM, the items in the test pool generally
  address the entire breadth of the strand content
  as operationalized in the benchmark standards and
  eligible content.

24
Status Alignment Studies (continued)

• Science Item Pool-Level Analyses (continued)
• Balance of Representation (Emphasis)
• For Grade 5, there was generally a balanced
  representation of content, although Life Science
  had slightly more items than either Physical or
  Earth Sciences.
• For Grade 8, each CCG had a balanced
  representation of content in the item pool items
  however, as was the case with Grade 5, Life
  Science had slightly more items than either
  Physical or Earth Sciences.
• For CIM, each CCG had a balanced representation
  of content in the item pool items.
• The content for Physical Science, Life Science,
  and Earth Science were fairly evenly emphasized
  throughout the entire item pool.

25
Status Alignment Studies (continued)

• Science Test-Level Analyses
• A comparison of the intended and actual
  distribution of items found that overall, the
  percentages of items at each strand were
  comparable that is, no percentage difference
  exceeded 6.

26
Status Alignment Studies (continued)

• General Recommendations
• The ODE should evaluate the small set of items
  that WestEd analysts determined had either
  partial or no alignment (categorical concurrence)
  to the standards and consider replacing or
  revising the items as appropriate in order to
  strengthen the relationships between the items
  and standards. Current procedures for item
  development and review should be analyzed and
  modified to limit future occurrences of partial
  or no alignment.
• The ODE should verify that the content coverage
  of the assessed domains is purposeful in terms of
  breadth, depth, emphasis, and complexity.

27
Status Alignment Studies (continued)

• General Recommendations (continued)
• The ODE should consider the assessed content and
  how content interacts with item format (multiple
  choice) in order to articulate more clearly its
  intended range of depth of knowledge (i.e.,
  whether the emphasis and range of DoK reflected
  in its assessments for each grade level and
  content area are consistent with expectations for
  each grade level/content area as well as across
  grade levels for a content area) and to ensure
  that the item format appropriately and
  effectively lends itself to students
  demonstrations of what they know and can do
  vis-à-vis the assessed content.

28
Status Alignment Studies (continued)

• General Recommendations (continued)
• Although the online assessment engine (TESA) is
  designed to select and administer to students a
  set of items that satisfy the test
  specifications, the ODE should consider examining
  a random sample of these assessments to ensure
  that the actual sets of items administered are
  indeed consistent with the test specifications in
  terms of breadth, depth, emphasis, and
  complexity. Additionally, documentation of the
  reliability and accuracy with which the test
  engine adheres to the states test specifications
  should be produced.

29
Status Alignment Studies (continued)

• General Recommendations (continued)
• The ODE should consider examining the full range
  of test blueprints vis-à-vis relevant sets of
  assessment items.
• The DoK definitions should be considered during
  item development to help ensure the full range of
  complexity of the assessed content, as
  appropriate.

30
Status Standards Evaluation Studies

• The analyses aimed to address the following
  key questions
• Do Oregons content standards adequately
  represent the knowledge and skills that all
  students should know and be able to do?
• Do Oregons content standards reflect the
  appropriate breadth and depth of the content
  area?
• Do Oregons content standards have the clarity
  and consistency needed to adequately guide
  instruction and assessment?

31
External Referents
CONTENT AREA LEVELS STATE REFERENT NATIONAL REFERENT
English Language Arts 10 GradesK-8 and CIM Indiana Draft 2009 NAEP Reading Framework and 2011 NAEP Writing Framework (grades 4, 8, and 12), McREL Speaking and Listening (grades K-3, 5-7, CIM)
Mathematics 10 GradesK-8 and CIM Indiana NCTM Principals and Standards for School Mathematics (2000) and Curriculum Focal Points for K-8 Mathematics (2006)
Science 10 GradesK-8 and CIM Indiana AAAS benchmarks (2001)
32
External Referents (continued)
CONTENT AREA LEVELS STATE REFERENT NATIONAL REFERENT
Social Sciences Geography Benchmarks 1 (K-3), 2 (4-5), 3 (6-8), CIM Washington NAEP Geography Framework (2001)
Social Sciences History Benchmarks 1 (K-3), 2 (4-5), 3 (6-8), CIM Washington NAEP History Framework (2006)
Social Sciences Civics Benchmarks 1 (K-3), 2 (4-5), 3 (6-8), CIM Washington NAEP Civics Framework (2006)
Social Sciences Geography Benchmarks 1 (K-3), 2 (4-5), 3 (6-8), CIM Washington NAEP Economics Framework (2006)
33
External Referents (continued)
CONTENT AREA LEVELS STATE REFERENT NATIONAL REFERENT
Arts Benchmarks 1 (K-3), 2 (4-5), 3 (6-8), CIM Washington Compendium Arts Standards developed by McREL (2006)
Physical Education Benchmarks 1 (K-3), 2 (4-5), 3 (6-8), CIM Indiana Moving into the Future National Standards for Physical Education (2006)
Health Benchmarks 1 (K-3), 2 (4-5), 3 (6-8), CIM Indiana National Health Education Standards Achieving Health Literacy
Educational Technology 1 Level (CCGs) Washington ISTE Standards for Students (2000)
34
External Referents (continued)
CONTENT AREA LEVELS STATE REFERENT NATIONAL REFERENT
Second Languages Benchmarks 1 (K-3), 2 (4-5), 3 (6-8), CIM Indiana American Council on the Teaching of Foreign Languages (1999)
English Language Proficiency Forms and Functions 5 Levels of Language Forms and Functions NA Current National Research on Forms and Functions
35
Status Standards Evaluation Studies (continued)

• Overall Quality
• Oregons English Language Arts (ELA) standards
  overall are of good quality and provide breadth
  and depth of coverage of Reading, Literature,
  Writing, Speaking, and Listening. The standards
  generally are clearly written, focus on important
  skills/concepts for instruction, and are
  assessable.
• Oregons mathematics standards overall are of
  good quality and reflect a breadth and depth of
  content coverage. The standards generally are
  clearly written, focus on important
  skills/concepts for instruction, and are
  assessable.

36
Status Standards Evaluation Studies (continued)

• Overall Quality (continued)
• Oregons science standards overall are of good
  quality in terms of clarity and importance for
  instruction. However, in all grades except for
  grade 8, analysts determined that the standards
  did not reflect a range of depth of knowledge.
  For all grades except for grade 1, analysts
  determined that the standards did not reflect a
  breadth of knowledge. With regard to consistency,
  analysts determined that standards were
  inconsistent (language, skills, knowledge) in
  grades 4, 8, and CIM. And, in all grades except
  for grades 3, 5, 8, and CIM, there were issues of
  assessability.
• Oregons social sciences standards overall are of
  good quality in terms of depth, breadth,
  consistency, importance for instruction, and
  assessability. However, overall these standards
  lack clarity.

37
Status Standards Evaluation Studies (continued)

• Overall Quality (continued)
• Oregons arts standards overall are of good
  quality and cover a breadth and depth of content.
  The standards generally are clearly written,
  focus on important skills/concepts for
  instruction and are assessable.
• Oregon's physical education standards overall are
  of good quality in terms of breadth and depth of
  content coverage, consistency across benchmark
  grades, importance for instruction, and degree of
  assessability. However, analysts determined that
  these standards are not easily understood due to
  lack of clarity and precision of language.
• Oregons health standards overall are of good
  quality in terms of depth, consistency,
  importance for instruction, and assessability.
  For all benchmark levels except for Benchmark 2,
  the standards overall are of good quality in
  terms of breadth of content. And, overall these
  standards lack clarity.

38
Status Standards Evaluation Studies (continued)

• Overall Quality (continued)
• Oregons educational technology standards overall
  are of good quality in terms of depth, breadth,
  consistency, importance for instruction, and
  assessability. However, overall these standards
  lack clarity.
• Oregons second languages standards overall are
  of good quality in terms of depth, breadth,
  clarity, consistency, importance for instruction,
  and assessability.
• Generally, Oregons English language proficiency
  (ELP) forms and functions are presented in a
  clear and useful format. The proficiency level
  descriptors clearly describe and differentiate
  the language skills of students at each level.
  However, the definitions of language function and
  forms of language appear incomplete. All the
  language functions except one are
  appropriateliterary analysis is typically
  considered English-Language Arts content rather
  than a language function. All the language forms
  except one are appropriatethe language of
  propaganda is not a form consistent with other
  elements of form presented by the state rather
  it appears to be a genre.

39
Status Standards Evaluation Studies (continued)

• Alignment to External Referents
• Oregons ELA standards generally aligned to the
  Indiana, NAEP and McREL standards in terms of
  overall content order/sequence and depth of
  content. However, while the breadth of the Oregon
  standards appeared comparable to the NAEP and
  McREL standards, they were not as comparable to
  the Indiana standards.
• Oregons mathematics standards were compared to
  Indianas mathematics standards as well as to the
  NCTM Principles and Standards for School
  Mathematics (2000) and Curriculum Focal Points
  for K8 Mathematics (2006). Oregons mathematics
  standards generally aligned to the NCTM standards
  and to the Indiana standards with regard to
  overall depth and breadth of content coverage.
  However, there was not overall congruence between
  Oregon and Indianas content ordering and
  sequencing.

40
Status Standards Evaluation Studies (continued)

• Alignment to External Referents (continued)
• Oregons science standards were compared to
  Indianas science standards and the AAAS
  benchmarks (2001).
• Generally, the Earth and Space Science standards
  aligned in terms of overall depth of content
  however, they did not align in terms of content
  order/sequence or overall breadth of content.
• Generally, the Physical Science standards aligned
  to the Indiana standards in terms of content
  order/sequence, overall depth, and overall
  breadth of content. The Oregon standards also
  aligned to the AAAS standards in terms of overall
  depth and breadth of content however, they did
  not align in terms of content order/sequence.
• Generally, the Life Science standards aligned to
  the Indiana standards in terms of content
  order/sequence, overall depth, and overall
  breadth of content. However, they did not align
  to the AAAS standards in terms of content
  order/sequence, overall breadth, or overall depth
  of content.

41
Status Standards Evaluation Studies (continued)

• Alignment to External Referents (continued)
• Oregons social sciences standards were compared
  to Washingtons social science standards and the
  NAEP frameworks.
• Generally, the Geography standards aligned in
  terms of overall content order/sequence with
  Washingtons standards. However, they did not
  align in terms of content order/sequence with the
  NAEP framework or in terms of the overall depth
  or breadth of content with the Washington
  standards or the NAEP framework.
• Generally, the History standards did not align to
  the NAEP framework in terms of content
  order/sequence, depth or breadth of content. Nor
  did these standards align in terms of overall
  breadth with the Washington U.S. History and
  World History standards. However, the Oregon
  standards aligned to the Washington standards in
  terms of content order/sequence and overall
  depth.

42
Status Standards Evaluation Studies (continued)

• Alignment to External Referents (continued)
• Oregons social sciences standards (continued).
• Generally, the Civics standards aligned in terms
  of overall content order/sequence with
  Washingtons standards and the NAEP framework.
  However, they did not align in terms of overall
  depth or breadth of content with either the
  Washington standards or the NAEP framework.
• Generally, the Economics standards aligned to the
  Washington standards and the NAEP framework in
  terms of content order/sequence, depth and
  breadth of content.
• There was a high degree of alignment between
  Oregons arts standards and those of Washington
  and McREL. In terms of content order / sequence,
  Washingtons standards spiral content across
  different benchmarks in a manner similar to
  Oregon many of McRELs benchmark level standards
  are not spiraled across different benchmarks.
  Both external referents show similar depth of
  content to Oregons standards.

43
Status Standards Evaluation Studies (continued)

• Alignment to External Referents (continued)
• Generally, Oregon's physical education standards,
  the National Association of Sports and Physical
  Education Standards (NASPE), and the Indiana
  standards share a common purpose. In broad terms,
  the nature of the content knowledge/cognitive
  information and the types of skills students are
  expected to learn and be able to demonstrate are
  comparable across the three sets of standards.
  Oregon differs in depth of content coverage from
  the NASPE and Indiana State standards.
• Oregons health standards were compared to
  Indianas health standards as well as to the
  National Health Education Standards (NHES).
  Oregons standards generally aligned to the
  Indiana and NHES standards with regard to overall
  content order/sequence and breadth of content
  coverage. However, there was not overall
  congruence in terms of depth of content.

44
Status Standards Evaluation Studies (continued)

• Alignment to External Referents (continued)
• Oregons educational technology standards
  generally aligned to the Washington and NETS
  standards in terms of overall content
  order/sequence, depth of content and breadth of
  content.
• While Oregon and Indiana both have the same
  number of proficiency levels, a comparison of the
  depth and breadth of the second languages
  standards across levels in both states shows that
  the Indiana standards start at a slightly higher
  level than those in Oregon, and reach a higher
  level. Oregons proficiency levels appear to be
  based on ACTFL levels Novice-Low through
  Intermediate-Mid. The standards statements differ
  in level of detail. Generally, Oregon standards
  are written more broadly, and Indianas are more
  detailed and specific. Oregons set of Functions
  Supporting Standards describe a similar level of
  detail as Indianas standards.

45
Status Standards Evaluation Studies (continued)

• Alignment to External Referents (continued)
• Oregons English language proficiency (ELP) forms
  and functions were compared to current national
  research and literature on forms and functions
  (e.g., TESOL, CALLA, Butler, et al.). All but one
  of Oregons language functions (literary
  analysis) are appropriate and relevant for ELD
  instruction. The external referents include
  functions that are currently not present among
  those in the Oregon ELP standards.

46
Next Steps

• Address questions regarding the states standards
  and assessment systemquality, structure, etc.
• Make recommendations based on data from the
  analyses (standards evaluation and assessment
  alignment) that have practice and policy
  implications
• Make recommendations that are research-based or
  based on promising/best assessment and
  accountability models and practices
• Consider the feedback/information provided key
  stakeholders through surveys, meetings, etc.

47

• For more information about the standards and
  assessment evaluation
• ode.state.or.us/go/real
• For more information about WestEd
• www.wested.org