IEA and TIMSS

1
IEA and TIMSS

• Methodological Tools for Accountability Systems
  in Education
• Ispra, 7 February
• By Heiko Sibberns

2
Overview of Presentation

• Introduction to IEA
• Introduction to TIMSS
• Developing the Assessments
• Collecting and Analyzing the Data
• Data Dissemination

3
IEA (International Association for the Evaluation
of Educational Achievement)

• Non governmental, not-for-profit organization
• Founded 1959
• Headquarters Amsterdam, The Netherland
• Data Processing Center Hamburg, Germany
• More than 60 membership countries
• Acting through a network of research institutes
• Funding Participation fees, World Bank, NCES,
  Dutch Ministry of Education, UNDP...

4
IEAs Mission

• Provide international benchmarks that assist
  policy makers in identifying the comparative
  strength and weaknesses of their educational
  systems
• Provide high-quality data that will increase
  policy-makers understanding of key school- and
  non-school-based factors that influence teaching
  and learning
• Provide high-quality data which will serve as a
  resource for identifying areas of concern and
  action, and for preparing and evaluating
  educational reforms

5
IEAs Mission and Current Studies

• Develop and improve educational systems capacity
  to engage in national strategies for educational
  monitoring and improvement
• Contribute to development of the world-wide
  community of researchers in educational
  evaluation
• Current studies
• TIMSS (Trends in Mathematics and Science)
• PIRLS (Progress in Reading Literacy)
• ICCES (Civic and Citizenship Education)
• Sites (Second Information Technology in Education
  Study)
• TEDS (Teacher Education)

6
TIMSS

• Trends in Educational Achievement
• Mathematics and Science
• Trends 1995, 1999, 2003, 2007
• Baseline for New Countries

7
TIMSS (cont.)

• Background Questionnaires
• Student
• Teacher
• School
• Curriculum (Online Data Collection)
• Encyclopedia (New for 2007)

8
TIMSS Grades Assessed

• From 1995, method of data collection has been
  testing classes of students at grade levels
• Economically efficient and less disruptive
• The purpose of conducting TIMSS is to be as
  directly useful as possible in improving student
  achievement
• Link achievement to policies, curriculum, and
  instructional practices
• These are tied to grade level

9
TIMSS Grades Assessed

• Therefore, for TIMSS
• Basis for comparability across countries is
  number of years of schooling
• Amount of schooling is defined by grades, but
  grades are labeled differently across countries
• Grade chosen should represent the same number of
  years of schooling across countries

10
TIMSS Grades Assessed

• TIMSS focuses on comparing achievement at the end
  of
• Four years of schooling
• Eight years of schooling
• Use ISCED classification scheme to identify first
  grade from which to start counting

11
TIMSS Population Definitions

• Definition Four years of schooling
• The target grade should be the grade that
  represents four years of schooling, counting from
  the first year of ISCED Level 1.
• Definition Eight years of schooling
• The target grade should be the grade that
  represents eight years of schooling, counting
  from the first year of ISCED Level 1.

12
TIMSS Population Definitions (cont.)

• Minimum Maturation Level
• Four years of schooling
• at grade level assessed, students average age
  should be at least 9.5 years at time of
  assessment
• Eight years of schooling
• at grade level assessed, students average age
  should be at least 13.5 years at time of
  assessment

13
TIMSS Grades Assessed

• Using grade (determined by years of schooling) as
  the basis of comparison, important contextual
  factors are
• Age at which children start school
• Promotion/retention policies
• Curriculum prior to and during ISCED Level 1
• In reports beginning with TIMSS 2003

14
Developing the TIMSS 2003 Mathematics and Science
Assessments

• International Expert Panel in Mathematics and
  Science Frameworks
• 14 Mathematics and 15 Science Experts
• Science and Mathematics Item Review Committee
  (SMIRC) Items
• 8 Mathematics and 10 Science Experts

15
Assessment Specialists

• Mathematics Coordinator
• Science Coordinator
• Task Forces
• National Research Coordinators
• Staff at TIMSS PIRLS International Study Center

16
TIMSS 2003 Mathematics and Science Frameworks

• Specify Mathematics and Science Content to Be
  Assessed
• Content Domains
• Cognitive Domains
• Grades 4 and 8

17
Framework Content Domains

• Mathematics
• Number
• Algebra
• Geometry
• Measurement
• Data
• Science
• Life Science
• Chemistry
• Physics
• Earth Science
• Environmental Science

18
Within Content Domains

• Topic Areas (Approximately 4 per Content Domain)
• For example at Grade 4,
• Number Whole Numbers
• Algebra Patterns
• Objectives Within Topic Area (Approximately 4-8)
• Demonstrate knowledge of place value
• Extend and find missing terms of numeric and
  geometric patterns

19
Cognitive Domains

• Mathematics
• Knowing Facts and Procedures
• Using Concepts
• Solving Routine Problems
• Reasoning
• Science
• Factual Knowledge
• Conceptual Understanding
• Reasoning and Analysis

20
Drafting TIMSS 2003 Field-test Items and Scoring
Guides

• Task Forces developed prototype items
• Prepared TIMSS 2003 Item-writing Guidelines for
  Multiple-choice and Constructed-response Items
  (20 pages)
• Held an item-writing workshop for NRCs
• Approximately 50 multiple-choice and 50
  constructed-response

21
Reviewing the TIMSS 2003 Field-test Items and
Scoring Guides

• Coordinators and ISC reviewed
• Task Force reviewed and wrote items to fill gaps
• SMIRC reviewed and wrote items to fill gaps
• NRCs reviewed field-test items and scoring guides

22
Developing the TIMSS 2003 Field-test Scoring
Training Materials

• Task Force selected sample responses to
• Illustrate categories in scoring guides
• Create training packets
• Provide sets of practice papers
• NRC meeting to conduct scoring training

23
Conducting the TIMSS 2003 Field Test

• Field-test sampling integrated with main TIMSS
  2003 sampling plan
• Every participating country required to take part
  in field test
• Grade 4 229 Mathematics and 206 Science items
• Grade 8 190 Mathematics and 196 Science items

24
Analyzing the TIMSS 2003 Field-test Data

• Diagnostic statistics
• Difficulty
• Discrimination
• Scoring reliability
• Item-by-country interaction
• For each country and language groups within
  country

25
Reviewing the TIMSS 2003 Field-test Results

• TIMSS PIRLS ISC and Coordinators reviewed
  results
• SMIRC reviewed results, recommended items for
  TIMSS 2003 data collection
• NRCs reviewed and approved final item selection
  Aug. 2002

26
Conducting the TIMSS 2003 Scoring Training

• Task Force met to update sample responses for
  training materials
• NRC meeting to conduct scoring training March
  2003
• Note A separate scoring training conducted
  earlier for Southern Hemisphere Nov. 2002

27
The Design for the TIMSS 2003 Assessment

• Many, many Mathematics and Science items
• Half newly-developed, half from 1995 and 1999 to
  measure trends
• Almost 400 items at Eighth Grade (7 hours)
• More than 300 items at Fourth Grade (5½ hours)
• Limited student testing time
• 90 minutes at Eighth Grade
• 72 minutes at Fourth Grade

28
Matrix Sampling Approach

• Assign items to blocks (10-15 items/block)
• 28 blocks (14 Mathematics 14 Science)
• Construct student booklets from item blocks
• 6 blocks per booklet
• Use IRT scaling to compute student achievement
  scores

29
TIMSS 2003 Item Block Design
30
Aspects of Booklet Design

• Each booklet
• 6 blocks per booklet
• Mathematics and Science blocks
• Trend blocks and new blocks
• Counterbalanced design
• Linking booklets Each block in 2-3 booklets

31
Sampling Design for Students

• Accuracy requirements
• Standard Error of 5 score points on TIMSS scale
• Basic sampling design
• 150 schools
• 1 (or 2) classroom per school
• 4,000 students (approx)

32
Standards for Population Coverage and Sampling
Participation

• 100 Population coverage
• Results annotated if student exclusions greater
  than 5
• 100 Sampling participation
• Results annotated if school and student
  participation less than 85
• Approved and documented sampling method

33
Data Collection

• Data collection a national responsibility
• TIMSS PIRLS ISC provided detailed manuals
• Quality Assurance program
• International (15 schools per grade)
• National (At Least 15 schools per grade)
• A Huge Effort!!

34
Database Construction IEA DPC

• Checked each countrys data files for internal
  consistency and accuracy
• Liaised with countries to resolve data issues
• Provided NRCs with data almanacs for review
• Incorporated all modifications into final
  database
• Formatted database and sent to TIMSS PIRLS ISC
  for analysis and reporting

35
Used IRT (Item Response Theory) Scaling Methods
to Assign Scores to Students

• With IRT scaling,
• Students score does not depend on taking the
  same set of items
• Student can take just part of the test
• IRT scaling is particularly useful when
• Different test blocks have to be linked
• Different samples of students have to be linked

36
Assigning Scores to Students

• TIMSS 2003 used a procedure to assign students
  scores that combined information
• Item difficulty
• Students responses to the items they took
• Students background characteristics
• Scores were located on the TIMSS achievement
  scales (mean of 500 and standard deviation of
  100)
• Mathematics and Science Scales
• Overall and in content areas at each grade
• Five Plausible values (PV) per scale
• representing the most likely distribution per
  student

37
Linking Across Assessments

• Using Grade 8 mathematics as an example
• Created the TIMSS mathematics achievement scale
  in 1995
• All items, all countries
• In 1999, scaled the mathematics data and placed
  it on the 1995 metric
• In 2003, scaled the mathematics data and placed
  it on the 1995 metric

38
Data Dissemination

• International reports
• Secondary analyses reports
• International database and analysis programs
• National reports
• International database seminars
• Online analyses

39
Three International Reports

• Mathematics Eighth and Fourth Grades
• Science Eighth and Fourth Grades
• Mostly descriptive
• Interpretation left to researchers of
  participating countries
• Technical Report

40
Contents of TIMSS 2003 International Reports

• Mathematics and Science achievement
• Grades 4 and 8
• 2003 results for all participants
• Three-cycle trend at eighth grade
• 1995, 1999, 2003
• Two-cycle trend at fourth grade
• 1995, 2003
• Background questionnaires

41

2003, 1999, and 1995
2003 and 1999
2003
Australia Belgium (Flemish) Bulgaria Cyprus Engla
nd Hong Kong, SAR Hungary Iran, Islamic Rep.
of Israel Italy Japan Korea, Rep.
of Latvia Lithuania Netherlands New
Zealand Romania Russian Federation Singapore Slova
k Republic Slovenia South Africa United
States __________________ Ontario Province,
Can. Quebec Province, Can.
Argentina Chile Chinese Taipei Indonesia Jordan M
acedonia, Rep. of Malaysia Moldova, Rep.
of Morocco Philippines Tunisia ______________ Indi
ana State, US

Armenia Bahrain Botswana Egypt Estonia Ghana Leba
non Palestinian National Authority Saudi
Arabia Serbia Syrian Arab Republic Yemen _________
__________ Basque Country, Spain
2003 and 1995
Norway Scotland Sweden
42
(No Transcript)
43
(No Transcript)
44
(No Transcript)
45
(No Transcript)
46
(No Transcript)
47
International database (IDB)

• All data files from IEA surveys available to
  download for free from the internet including
  programs and documentation http//www.iea.nl/iea_
  studies_datasets.htm
• Structured by
• Study
• Cycle
• Population

48
Analyis Program IDB Analyzer

• Allows straight forward analyses of IEA data
  (TIMSS and PIRLS)
• Grapical User Interface in SPSS environment
• No editing of syntax
• Capabilities
• Combines countries
• Combines data from different levels (students,
  teachers, schools)
• Selection of variables
• Calculates correct standard errors for various
  statistics
• Sampling errors
• Imputation errors

49
Analysis Module Running Analysis
Selecting the parameters (here selecting math PV
as dependent variable)
Weight variables have been pre-defined
50
Analysis Result Output Window
Original SPSS report
51
Analysis result Syntax Window
Contains whole analysis syntax allows to re-run
the analysis
52
National Reports and Training

• IEA relies on national researchers to have
  greatest insight in their system and beeing able
  to advice policy makers most skillfully
• To fascilitate national analysis, IEA conducts
  international and regional seminars regulary and
  on request

53
International Database Seminars

• 19.-23. Feb. 2006 TIMSS 2003 international data-
  base training for the arabic region in Amman,
  Jordan
• 17.-19. Oct. 2005 TIMSS 2003 Database
  introduction and HLM workshop
• 9.-11. Jan. 2005 TIMSS 2003 Database and scaling
  technology
• Oct. 2004 PIRLS 2001 Database construction and
  structural equation modelling

54
Online Analyses

• Data can be analysed online under
• www.IEADATA.ORG
• Available February 2006
• TIMSS 2003 data first

55
Online Analysis
56
IEA and TIMSS

• Methodological Tools for Accountability Systems
  in Education
• Ispra, 7 February
• By Heiko Sibberns