International Benchmarking

1
International Benchmarking
Mark Schneider American Institutes for
Research IES Research Conference June, 2009
2
Two main purposes of international benchmarking

• Comparison of performance
• League tables

3
(No Transcript)
4
2007 TIMSS 8th Grade Math
5
Benchmarking to Gauge Comparative Performance
(effect sizes in math)
USTaipai MississippiMassachusetts
6
Benchmarking to Learn more about Performance
within the United States (effect sizes in math)
7
PISA v. TIMSS

• PISA is a self-proclaimed yield study assessing
  the total literacy of 15 year olds.
• Emphasis on globalization and 21st century skills
  
• assessing the skills young adults will need in
  the emerging global economy.
• TIMSS is more grade and curriculum centered .
• designed to align broadly with mathematics and
  science curricula in the participating countries.
  
• The results suggest the degree to which students
  have learned mathematics and science concepts and
  skills likely to have been taught in school.

8
Who Participates in These Assessments and Does it
Matter?

• PISA consists of every major trading partner and
  competitor of the U.S.
• TIMSS is increasingly skewed toward less
  developed countries.

9
Differences in PISA/TIMSS results

• In TIMSS 2007 eighth grade math students in the
  United States scored above the international
  TIMSS average.
• In PISA 2006 US 15 year olds were 24 points below
  the OECD average math score
• In TIMSS 2007, 3 times the percentage of US
  eighth grade students were in the top 10
  compared to the international median.
• In PISA 2006, only 1.3 of US students were in
  the highest proficiency level in 2006 PISA math
• half the OECD average and in the same range as
  Greece, Mexico, Portugal and Turkey.

10
Policy Advice

• Identify policy solutions to U.S. education
  system shortcomings. (NGA)
• Two basic approaches
• Statistical analysis
• Best practices

11
Generating Policy Advice from International
Assessments

• OECD vs. IEA
• Limits on the data
• Cross sectional data
• Measurement equivalence
• Mixing statistical reports and policy
• Policy advice built on weak data
• Best practices

12
Next Frontier State Participation in PISA or
TIMSS

• Traditional approach
• 1000 or so students sit for the exam at a cost of
  600,000-750,000 per assessment

13
Limits to State Participation

• How will international assessments fit into the
  already complex world of large scale assessments?
  
• Can a state align its curricula to both NAEP and
  PISA?
• Can PISA really inform policy makers about how to
  improve a states schools system?
• Getting schools and students to take no- or
  low-stakes tests is increasingly difficult.
• These tests are not cheap.

14
Are there cheaper ways of getting state scores?

• Small Area Estimation
• Statistical Linkingor how to save around 25
  million.
• The work of Gary Phillips is foundational

15
TIMSS Mean Scores Including Estimated Scores for
Several American States (TIMSS)
16
Caveat emptor