EIAC Fall Meeting

1
EIAC Fall Meeting

• Permanent Standing Taskforce on Assessment
• Subcommittee Meeting
• October 25, 2004

2
Update on NCES/CCSSO ActivitiesSuzanne Triplett,
NCESMary Yakimowski, CCSSO
3
NAGB UpdateCharles Smith, NAGB
4
NAEP 2009 Science Framework ProjectMary Crovo,
NAGB
5
Science Framework Project Team

• As a result of a competitive bidding process,
  NAGB contracted with
• WestEd in partnership with CCSSO
• There are three collaborating organizations on
  the contract
• American Association for the Advancement of
  Science (AAAS)
• National Science Teachers Association (NSTA)
• Council of State Science Supervisors (CSSS)

6
Key Project Dates

• Contract began late Sept. 04
• Issues Paper developed by science experts
  convened by NAGB Sept. to Nov. 04
• Committees convened (Steering, Planning,
  Technical Advisory subgroup) Winter 04 to
  Summer 05
• Feedback on Framework - ongoing
• Board action on Framework and Specifications
  Nov. 05

7
NCES UpdatePeggy Carr
8
Pending NAEP Legislation (1)

• Senate Bill 2721 American History Achievement
  Act
• Would require NAEP to conduct trial state
  assessments in U.S. History in grades 8 and 12
• NAGB and NCES Commissioner to select 10 trial
  states that represent geographically diverse
  regions of the U.S.
• Status Introduced July 2004 referred to Senate
  HELP Committee
• Co-Sponsors Sen. Lamar Alexander and Sen. Ted
  Kennedy

9
Pending NAEP Legislation (2)

• House Resolution 4969
• Would require NAEP to be subject to certain
  guidance on the release of information to the
  public
• Treats NAEP as principal economic indicator
  subject to OMBs Statistical Policy Directive on
  Compilation, Release, and Evaluation of Principal
  Federal Economic Indicators
• Status Introduced July 2004 referred to
  Subcommittee on Education Reform
• Co-Sponsors Rep. Carolyn Maloney and Rep.
  William Lacy Clay

10
Coordinator Work Loads

• Federally funded NAEP State Coordinators should
  be spending 90 of their time on NAEP
• Task order contract language Perform such
  other related duties as assigned by State
  Superintendent and State Testing Director, not to
  exceed 10 of work hours.

11
NAEP State Coordinator Presentations at NAGB

• NAGB members interested in meeting the
  Coordinators and learning about their role
• Plan Presentations by NCES and Coordinators at
  the next 3 NAGB meetings
• Presentations to include
• Role of NAEP State Coordinator and State Service
  Center
• Promoting understanding of NAEP to gain school
  cooperation
• Use of NAEP data

12
National Indian Education Study

• The schools and their education agencies,
  including Bureau of Indian Affairs, are being
  contacted as part of the NAEP assessment.
• The sampling of schools for the Indian Education
  Survey will conducted in early December 2004.

13
National Assessment of Adult Literacy

• NAAL data collection was completed Spring 2004
• NAAL is based on in-person, household interviews
  of approximately 19,000 Americans. It includes
• the main NAAL
• a prison study
• a background questionnaire
• State Assessment of Adult Literacy (SAAL)
• Health Literacy Component (HLC)
• Fluency Addition to NAAL (FAN)
• Adult Literacy Supplemental Assessment (ALSA)

14
National Assessment of Adult Literacy

• Performance Levels
• NAAL data will be analyzed based on the new
  performance levels, which are being developed by
  the National Academy of Sciences (NAS)
• Reports Publications (Planned release date)
• General Audience Report July 2005
• Comprehensive Survey Report December 2005
• Individual State Reports Spring 2006
• Prison Report Spring 2006
• Technical Report Spring 2006

15
Assessment Division Organization
16
12th Grade IssuesPeggy Carr
17
New 12th Grade Mathematics Scale

• 2005 Mathematics assessment begins a new trend
  for 12th grade
• New framework
• New achievement level setting process
• New scale needed to avoid confusion 0-300
• Previous scale 0-500

18
Proposed 12th Grade State NAEP

• Text of the Perkins Reauthorization Act does not
  include language on 12th grade state NAEP
• Departmental budget request asked for budget
  numbers to conduct state 12th grade NAEP
  beginning in 2006

19
12th Grade NAEP One Subject Roll-Out Approach
20
12th Grade NAEP Two Subject Roll-Out Approach
21
Upcoming Reports Arnold Goldstein, NCES
22
NAEP 2003 Pilot Charter School Study

• Oversampled charter schools in California,
  Michigan, and Texas at grade 4
• Sample size 150 schools
• Conducted 2003 NAEP in reading and mathematics
• Administered follow-up survey to charter school
  directors

23
2003 Follow-up Survey on Charter Schools

• History Year first provided instruction, new or
  pre-existing school
• Primary educational focus, students primarily
  served
• Organization and monitoring chartering entity,
  to whom school reports
• Autonomy waivers or exemptions

24
Reporting Plans

• Two Reports comparing charter and other public
  schools
• Descriptive report, to be released mid-December
  2004
• Bivariate analyses, highlights style
• In-depth report featuring hierarchical linear
  modeling (HLM) analyses, to be released early
  2005
• To account for school and student effects

25
In-Depth Charter School Report

• Student factors associated with performance in
  charter and other public schools
• Aspects of charter schools associated with
  performance
• Use of HLM to control for student and school
  characteristics to determine extent of effect of
  being a charter school on performance differences

26
NAEP Pilot Private School Studies

• Goals
• Provide NAEP data for various types of nonpublic
  schools
• Collect and report more detailed nonpublic school
  data within resource and sampling constraints

27
Objects for Reporting

• Include largest groups of nonpublic schools
• Provide data on gender, race/ethnicity, parents
  highest level of education
• Report category only if sufficient sample for
  subgroups
• Requires participation of at least 25 schools and
  750 students per school type

28
Pilot Study Design

• Oversampled nonpublic schools in 2000, 2002, and
  2003 with goal of providing results for
• Catholic
• Conservative Christian
• Lutheran
• Other religious
• Nonsectarian
• Collected nonpublic data only at national level

29
Forthcoming Private School Reports

• Two reports with results from 2000, 2002, 2003
• Descriptive Report January 2005
• Analytic Report Early 2005

30
Descriptive Private School Report

• Demographic profiles by type of nonpublic school
  and public schools
• 4th, 8th, 12th grade performance results
• Mathematics, reading, science, writing
• Average scores and percentages of students at
  Basic, Proficient, and Advanced achievement
  levels
• Subgroup results gender, race/ethnicity,
  parents highest level of education

31
Analytic Private School Report

• Analyze contributions of various factors to
  performance student, teacher, school
  characteristics
• Use of hierarchical linear modeling (HLM)
  analysis technique
• School-level variables
• Student-level variables
• Will compare performance of private and public
  school students and schools with similar
  characteristics

32
NAEP Results on the Web
http//nces.ed.gov/nationsreportcard/mathematics/r
esults/
33
Lunch New NAEP ToolsSuzanne TriplettNancy
Mead, ETSJeff Jenkins, ETS
34
2005 and Beyond Assessment IssuesDianne Walsh,
WestatSuzanne Triplett
35
NAEP 2005 Assessment Components
36
The NAEP Science Assessment

• About half of the students take a hands-on task
• Additional 20 minutes for grade 4
• Additional 30 minutes for grades 8 and 12
• Science kits remain in the schools
• Brochure describes how to use in classrooms
• Valuable to schools, therefore valuable in
  recruiting

37
The NAEP 2005 Sample

• Almost 20,000 schools and over 1.1 million
  students
• Over 93 in the main assessments of reading,
  mathematics, and science
• About 9,500 schools and 500,000 students at grade
  4
• About 7,500 schools and 500,000 students at grade
  8
• Pilots and science bridge study
• About 300 schools and 17,000 students at grades 4
  and 8
• Twelfth grade
• About 600 schools and 50,000 students, including
  the pilots and science bridge study

38
Trial State Oversampling

• Four large states
• California triple sample
• Texas double sample
• New York 50 increased sample
• Florida 50 increased sample
• Rationale
• Large states with diverse populations
• Will improve the precision of the national (and
  state) estimates, both overall and by demographic
  subgroups

39
Trial Urban District Assessment (TUDA)

• 10 Districts and Washington, DC

• Atlanta
• Austin (New)
• Boston
• Charlotte
• Chicago

• Cleveland
• Houston
• Los Angeles
• New York City
• San Diego

40
High School Transcript Study (HSTS)

• Provides educational policymakers with
  information on
• Current high school curricula
• Course-taking patterns
• Student grade point averages
• Researchers can examine the relationships between
  these variables and student achievement on NAEP
• Involves all public high schools selected for the
  main NAEP assessment

41
NAEP Non-Participation Issues

• National NAEP science and other voluntary
  subjects
• 12th grade assessments
• Pilot and field tests

42
Implications of Non-Participation

• Validity of national and regional results
• Substitute schools