National Assessment of

1
National Assessment of Educational Progress
(NAEP) 2009 Science Framework and
Specifications Alice Fu, WestEd and CCSSO Arthur
Halbrook, CCSSO January 22, 2005
2
NAEP Organization
CONGRESS (P.L. 107-279 NCLB)
The Nations Report Card
Policy
Operations
NAEP
National Assessment Governing Board (NAGB)
National Center for Education
Statistics (NCES)
Implementation

• Outside Contractors for
• - Framework development - Achievement level
  setting
• Test development administration - Scoring,
  analysis, reporting
• Etc.

3
NAEP Components
4
NAGB Overview

• History
• Established by Congress in 1988
• Reauthorized in 1994
• Composition
• 26 members
• Appointed by the Secretary of Education
• Bipartisan and broadly representative
• Local, state, and federal officials, educators,
  business representatives, general public

5
NAGB Statutory Responsibilities

• Select content areas to be assessed
• Develop assessment content frameworks and
  specifications
• Set student performance levels
• Develop guidelines for reporting NAEP results
• Design assessment methodology
• Approve all test items
• Improve the form and use of NAEP

6
Science NAEP 2009 Why Now?

• Existing framework and specifications developed
  in 1990
• Limited shelf-life
• Some developments since 1990
• National Science Education Standards (NRC)
• Benchmarks for Science Literacy (AAAS)
• Advances in cognitive and measurement sciences
• No Child Left Behind

7
NAEP Assessment Development

• Five Year Cycle (e.g., science)
• Framework Development Process (2005)
• Assessment framework (content)
• Specifications (detailed blueprint)
  
• Preliminary achievement level descriptions
• Test development (2006 - 2007)
• Item reviews (content, bias, etc.) (2006 - 2007)
• Field testing more reviews (2007 - 2008)
• Operational testing (2009)
• Setting achievement levels (2009)
• Report preparation and review ( 2010)
• Release of NAEP results (2010)

8
Assessment Design Sequence
NAGB
specifies the content of the
Assessment Framework
is further elaborated in the
Broad input from scientists, science
educators, policymakers, the public, etc.
Assessment and Exercise Specifications
specify content and item types on the
Assessment
From NAEP Science Issues Paper, 2004
9
Science NAEP 2009

• 1.3M NAGB Contract
• Contractor WestEd
• Promote excellence, achieve equity, improve
  learning
• National nonprofit
• Research, development, and service agency
• Subcontractor Council Chief State School
  Officers
• Leadership, advocacy, and technical assistance
• National nonprofit
• Professional association of state education
  commissioners

10
Overall Purposes

• Develop the recommended framework and
  specifications for NAEP Science 2009
• Recommend important background variables
  associated with student achievement in science
  that should be included in NAEP Science
• Propose a design for one or more small-scale
  special studies

11
2-Tiered Development Process

• Steering Committee Oversight
• Policies, goals, principles
• Planning Committee Production
• Recommended framework
• Recommended test and item specifications
• Recommendations for background variables
• Designs for small-scale studies

12
Steering Committee

• Chair Gerald Wheeler, NSTA
• Co-chair Mary Yakimowski, CCSSO
• Broad participation Representatives of
  education, industry, science organizations
• Three 2-day meetings in Washington, DC
• December 2004
• March 2005
• August 2005

13
Steering Committee

• Key Responsibilities
• Represent perspectives of member organizations
• Identify relevant resources
• Guide Planning Committee (PC)s goals and
  objectives
• Assist PC in finding agreement, if necessary
• Review PCs draft documents
• Obtain feedback on drafts from member
  organizations
• Approve documents before submission to NAGB

14
Planning Committee

• Chair Richard Shavelson, Stanford University
• Co-chair Senta Raizen, WestEd
• Broad participation educators, measurement
  experts, scientists
• Six 2-day meetings in Washington, D.C.
• December 2004 - May 2005
• January 2005 - June 2005
• March 2005 - August 2005

15
Planning Committee

• Key Responsibilities
• Serve as members of PC sub-committees
• Participate in deliberations
• Assist in writing deliverables
• Identify relevant resources
• Present to Planning and Steering Committees
• Review sub-committees draft documents
• Review feedback and incorporate suggestions
• Approve documents before submission to SC

16
Collaborators

• American Association for the Advancement of
  Science (AAAS)
• Scientific community
• Council of State Science Supervisors (CSSS)
• State level users
• National Science Teachers Association (NSTA)
• Science educators

17
Key Staff
18
Broad-Based Input

• Stakeholders include scientists, teachers, state
  education officials, and the public
• Diverse Steering and Planning Committees
• Collaborators AAAS, CSSS, NSTA
• Public Forums
• Critical reviews of draft documents

19
Proposed Schedule of Activities
20
Proposed Schedule of Activities
21
Issues Recommendations

• Paper commissioned by NAGB
• September-November, 2004
• Panel
• Chair Audrey Champagne, University at Albany,
  SUNY
• Kathleen Bergin, National Science Foundation
• Rodger Bybee, Biological Sciences Curriculum
  Study
• Richard Duschl, Rutgers University
• James Gallagher, Michigan State University

22
Issues and Recommendations

• Overview
• U.S. Standards
• National Science Education Standards (NRC)
• Project 2061 (AAAS)
• International Assessment Frameworks
• Trends in Mathematics and Science Study (TIMSS)
• Programme for Intl Student Assessment (PISA)
• State standards and assessments
• Inquiry
• Stakeholder expectations
• Cognitive and Measurement Sciences
• Frameworks Form

23
Where We Are Now

• 2 meetings held in Washington, DC
• Steering Committees preliminary guidelines
• Planning Committee sub-committees
• Earth and Space
• Life
• Physical
• Technical/Assessment group

24
Where We Are Now

• Issues to be resolved
• Cognitive dimensions
• Cross-cutting themes
• Inquiry and Design
• Learning trajectories Grades 4, 8, 12
• Item formats and weightings
• Equity Issues

25
Next Steps Gathering input

• 10 CSSS Regional Meetings
• March/April, 2005
• CCSSO Large-Scale Assessment Conference
• San Antonio, TX, June 2005
• Association Meetings

26
Websites

• NAEP Science Project
• http//www.sciencenaep2009.org
• NAEP
• http//nces.ed.gov/nationsreportcard/
• National Center for Education Statistics
• http//nces.ed.gov/
• National Assessment Governing Board
• http//www.nagb.org/