Mathematics and Science Indicators: Deciding on Measures that Matter

1
Mathematics and Science Indicators Deciding on
Measures that Matter

• Val Plisko
• National Center for Education Statistics
• 14th Annual Management Information Systems
  Conference
• February 27, 2001

2
Current Status of Indicators Affecting Student
Learning

• Current research points to three broad aspects of
  school quality that affect student learning
• Training and talent of the teaching force
• Classroom activities
• School culture and atmosphere
• Indicators in each of these three areas are
  currently of differing levels of quality
• From Monitoring School Quality An Indicators
  Report (Dec 2000) http//nces.ed.gov/pubs2001/200
  1030.pdf

3
Determining Factors in Quality of Current
Indicators

• High quality indicators usually result from
  easily assessed dimensions or a long history of
  data collection on the dimension
• Moderate quality indicators generally lack
  information on an important facet of the
  indicator, but still provide some value
• Poor quality indicators generally suffer from the
  indicator being more complex than the data

4
Current Quality for Student Learning Indicators

• High Quality Indicators
• Teacher Assignment
• Teacher Experience
• Teacher Academic Skills
• Class Size
• Moderate Quality Indicators
• Professional Development
• Technology
• Course Content
• Discipline
• Academic Environment
• Poor Quality Indicators
• Pedagogy
• Goals
• School Leadership
• Professional Community

5
Data on Teacher Preparation

• Teacher subject matter preparation in mathematics
  and science is related to student achievement
• Higher student performance associated with more
  experienced teachers than novice teachers
• Higher student learning has been connected with
  teachers who attended higher quality
  undergraduate institutions, as measured by
  admissions selectivity
• Teachers with higher scores on standardized tests
  have been linked to students with higher test
  scores

6
Data on Class Size

• Most analyses of class size have found that
  smaller classes lead to higher student test
  scores, particularly for primary-grade students
  who are minorities or who come from economically
  disadvantaged backgrounds
• Yet lowering class size may not enhance student
  learning without changes to instructional
  practices or if unqualified teachers are used to
  reduce class size

7
Data on Goals

• Researchers agree that successful schools begin
  with identifying and communicating ambitious
  goals, then implementing and institutionalizing
  those goals with broad consensus from key
  stakeholders
• Data on school goals are generally lacking
• Only nationally representative data come from how
  public and private school principals responded on
  a questionnaire to a list of seven general goal
  statements
• Data lack specifics on how these goals are
  identified, communicated, implemented and
  institutionalized

8
International assessments offer a macro
perspective

• Enable the United States to benchmark student
  performance against international averages or
  countries of particular interest (e.g., G-8
  countries)
• Enable us to think outside of the U.S. box for
  alternatives to the content, methods and context
  for teaching

9
Average Eighth-Grade Mathematics and Science
Achievement, by Nation 1999
10
Comparisons of Average Mathematics Achievement,
by Nation 1995 and 1999
11
Comparisons of Average Science Achievement, by
Nation 1995 and 1999
12
Mathematics Achievement Relative to International
Average, Fourth Grade TIMSS 1995 and Eighth Grade
TIMSS-R 1999
13
Science Achievement Relative to International
Average, Fourth Grade TIMSS 1995 and Eighth Grade
TIMSS-R 1999
14
Factors related to mathematics and science
achievement TIMSS and TIMSS-R Findings

• The materials presented in U.S. mathematics
  classrooms is at a lower grade level than that
  presented in German and Japanese classrooms.
• The mathematical content presented to U.S.
  eighth-grade students is of a lower quality than
  that presented to Japanese eighth-grade students.
• U.S. eighth-grade mathematics teachers typical
  goal is to teach students how to do something
  Japanese teachers typical goal is to help
  students understand mathematical concepts.
• U.S. eighth-grade mathematics lessons appear to
  be less coherent than Japanese mathematics
  lessons.
• U.S. curriculum covers more topics and spends
  less time on each topic than in other nations.
• U.S. curriculum retains topics throughout the
  K-12 learning experience, whereas other nations
  introduce and then move to more advanced topics.

15
Percentage of Lessons Rated as Having Low,
Medium, and High Quality of Mathematical Content
16
Eighth-Grade Mathematics Teachers Academic
Preparation 1999
Significant difference between U.S. average and
international average in this category.
17
Eighth-Grade Science Teachers Academic
Preparation 1999
Significant difference between U.S. average and
international average in this category.
18
Factors not related to mathematics and science
achievement among nations in TIMSS

• It is not the implementation of a national
  curriculum.
• It is not the amount of instructional time.
• It is not the amount of homework assigned by
  teachers.
• It is not the presence or use of computers in the
  classroom.
• It is not average class size.
• It is not the amount of time spent watching TV or
  videos.
• This is not to say that such factors do not
  matter nationally.

19
TIMSS-R State and District Benchmarking

• Purpose to provide data on the mathematics and
  science achievement of 8th-grade students in
  participating states and districts in comparison
  to students nationally and in 37 other nations
• Benchmark performance against international
  average
• Compare state/district instructional practices of
  teachers, student attitudes, and curriculum with
  that of other nations
• Allow state/district to assess the rigor and
  effectiveness of local math and science programs
  in an international context

20
TIMSS-R Benchmarking Participants

• States
• Connecticut
• Idaho
• Illinois
• Indiana
• Maryland
• Massachusetts
• Michigan
• Missouri
• North Carolina
• Oregon
• Pennsylvania
• South Carolina
• Texas

• Districts and Consortia
• Academy School District 20 (CO)
• Chicago Public Schools (IL)
• Delaware Science Coalition
• First in the World Consortium (IL)
• Fremont/Lincoln/West Side Public Schools (NE)
• Guilford County Schools (NC)
• Jersey City Public Schools (NJ)
• Miami-Dade County Public Schools (FL)
• Michigan Invitational
• Montgomery County Public Schools (MD)
• Naperville School District 203 (IL)
• Project SMART Consortium (OH)
• Rochester City School District (NY)
• SW Pennsylvania Regional Mathematics and Science
  Collaborative

21
Program for International Student Assessment
(PISA)

• Purpose to measure 15-year-old
• students knowledge, skills, and
• competencies in reading, mathematics
• and science.
• Nations can compare how their students perform on
  reading, mathematics, and science in relation to
  students in other nations
• Nations can compare themselves to other nations
  on contextual variables related to reading,
  mathematics, and science education
• Nations can gauge the impact of policy changes
  and reform efforts over time

22
PISA Nations
Australia Austria Belgium Brazil Canada China Czec
h Republic Denmark Finland France Germany Greece H
ungary Iceland Ireland Italy
Japan Korea, Republic of Latvia Luxembourg Mexico
The Netherlands New Zealand Norway Poland Portugal
Russian Federation Spain Sweden Switzerland Unite
d Kingdom United States
23
Complementing Indicators with Research on What
Matters

• Indicators should be framed by current research
  on areas that affect school and teacher quality
• Indicators can also inform research in these
  areas, but research needed on macrolevel
  policies that contribute to quality teaching and
  learning
• International assessment can help inform the
  development of indicators that are most valuable
  to capture

24
Upcoming International Releases

• April 4, 2001 TIMSS-R Benchmarking Results
• Late 2001 Mathematics Results from the TIMSS-R
  Video Study
• Late 2001 PISA Results
• Late 2002 Science Results from the TIMSS-R Video
  Study

25
For More Information

• http//nces.ed.gov/timss
• http//pisa.oecd.org
• Val Plisko
• T [phone]
• Email vplisko_at_ed.gov