IEA and TIMSS - PowerPoint PPT Presentation

1 / 56
About This Presentation
Title:

IEA and TIMSS

Description:

International Expert Panel in Mathematics and Science Frameworks. 14 ... 8 Mathematics and 10 Science Experts. Assessment Specialists. Mathematics Coordinator ... – PowerPoint PPT presentation

Number of Views:477
Avg rating:3.0/5.0
Slides: 57
Provided by: ralp95
Category:
Tags: iea | timss

less

Transcript and Presenter's Notes

Title: IEA and TIMSS


1
IEA and TIMSS
  • Methodological Tools for Accountability Systems
    in Education
  • Ispra, 7 February
  • By Heiko Sibberns

2
Overview of Presentation
  • Introduction to IEA
  • Introduction to TIMSS
  • Developing the Assessments
  • Collecting and Analyzing the Data
  • Data Dissemination

3
IEA (International Association for the Evaluation
of Educational Achievement)
  • Non governmental, not-for-profit organization
  • Founded 1959
  • Headquarters Amsterdam, The Netherland
  • Data Processing Center Hamburg, Germany
  • More than 60 membership countries
  • Acting through a network of research institutes
  • Funding Participation fees, World Bank, NCES,
    Dutch Ministry of Education, UNDP...

4
IEAs Mission
  • Provide international benchmarks that assist
    policy makers in identifying the comparative
    strength and weaknesses of their educational
    systems
  • Provide high-quality data that will increase
    policy-makers understanding of key school- and
    non-school-based factors that influence teaching
    and learning
  • Provide high-quality data which will serve as a
    resource for identifying areas of concern and
    action, and for preparing and evaluating
    educational reforms

5
IEAs Mission and Current Studies
  • Develop and improve educational systems capacity
    to engage in national strategies for educational
    monitoring and improvement
  • Contribute to development of the world-wide
    community of researchers in educational
    evaluation
  • Current studies
  • TIMSS (Trends in Mathematics and Science)
  • PIRLS (Progress in Reading Literacy)
  • ICCES (Civic and Citizenship Education)
  • Sites (Second Information Technology in Education
    Study)
  • TEDS (Teacher Education)

6
TIMSS
  • Trends in Educational Achievement
  • Mathematics and Science
  • Trends 1995, 1999, 2003, 2007
  • Baseline for New Countries

7
TIMSS (cont.)
  • Background Questionnaires
  • Student
  • Teacher
  • School
  • Curriculum (Online Data Collection)
  • Encyclopedia (New for 2007)

8
TIMSS Grades Assessed
  • From 1995, method of data collection has been
    testing classes of students at grade levels
  • Economically efficient and less disruptive
  • The purpose of conducting TIMSS is to be as
    directly useful as possible in improving student
    achievement
  • Link achievement to policies, curriculum, and
    instructional practices
  • These are tied to grade level

9
TIMSS Grades Assessed
  • Therefore, for TIMSS
  • Basis for comparability across countries is
    number of years of schooling
  • Amount of schooling is defined by grades, but
    grades are labeled differently across countries
  • Grade chosen should represent the same number of
    years of schooling across countries

10
TIMSS Grades Assessed
  • TIMSS focuses on comparing achievement at the end
    of
  • Four years of schooling
  • Eight years of schooling
  • Use ISCED classification scheme to identify first
    grade from which to start counting

11
TIMSS Population Definitions
  • Definition Four years of schooling
  • The target grade should be the grade that
    represents four years of schooling, counting from
    the first year of ISCED Level 1.
  • Definition Eight years of schooling
  • The target grade should be the grade that
    represents eight years of schooling, counting
    from the first year of ISCED Level 1.

12
TIMSS Population Definitions (cont.)
  • Minimum Maturation Level
  • Four years of schooling
  • at grade level assessed, students average age
    should be at least 9.5 years at time of
    assessment
  • Eight years of schooling
  • at grade level assessed, students average age
    should be at least 13.5 years at time of
    assessment

13
TIMSS Grades Assessed
  • Using grade (determined by years of schooling) as
    the basis of comparison, important contextual
    factors are
  • Age at which children start school
  • Promotion/retention policies
  • Curriculum prior to and during ISCED Level 1
  • In reports beginning with TIMSS 2003

14
Developing the TIMSS 2003 Mathematics and Science
Assessments
  • International Expert Panel in Mathematics and
    Science Frameworks
  • 14 Mathematics and 15 Science Experts
  • Science and Mathematics Item Review Committee
    (SMIRC) Items
  • 8 Mathematics and 10 Science Experts

15
Assessment Specialists
  • Mathematics Coordinator
  • Science Coordinator
  • Task Forces
  • National Research Coordinators
  • Staff at TIMSS PIRLS International Study Center

16
TIMSS 2003 Mathematics and Science Frameworks
  • Specify Mathematics and Science Content to Be
    Assessed
  • Content Domains
  • Cognitive Domains
  • Grades 4 and 8

17
Framework Content Domains
  • Mathematics
  • Number
  • Algebra
  • Geometry
  • Measurement
  • Data
  • Science
  • Life Science
  • Chemistry
  • Physics
  • Earth Science
  • Environmental Science

18
Within Content Domains
  • Topic Areas (Approximately 4 per Content Domain)
  • For example at Grade 4,
  • Number Whole Numbers
  • Algebra Patterns
  • Objectives Within Topic Area (Approximately 4-8)
  • Demonstrate knowledge of place value
  • Extend and find missing terms of numeric and
    geometric patterns

19
Cognitive Domains
  • Mathematics
  • Knowing Facts and Procedures
  • Using Concepts
  • Solving Routine Problems
  • Reasoning
  • Science
  • Factual Knowledge
  • Conceptual Understanding
  • Reasoning and Analysis

20
Drafting TIMSS 2003 Field-test Items and Scoring
Guides
  • Task Forces developed prototype items
  • Prepared TIMSS 2003 Item-writing Guidelines for
    Multiple-choice and Constructed-response Items
    (20 pages)
  • Held an item-writing workshop for NRCs
  • Approximately 50 multiple-choice and 50
    constructed-response

21
Reviewing the TIMSS 2003 Field-test Items and
Scoring Guides
  • Coordinators and ISC reviewed
  • Task Force reviewed and wrote items to fill gaps
  • SMIRC reviewed and wrote items to fill gaps
  • NRCs reviewed field-test items and scoring guides

22
Developing the TIMSS 2003 Field-test Scoring
Training Materials
  • Task Force selected sample responses to
  • Illustrate categories in scoring guides
  • Create training packets
  • Provide sets of practice papers
  • NRC meeting to conduct scoring training

23
Conducting the TIMSS 2003 Field Test
  • Field-test sampling integrated with main TIMSS
    2003 sampling plan
  • Every participating country required to take part
    in field test
  • Grade 4 229 Mathematics and 206 Science items
  • Grade 8 190 Mathematics and 196 Science items

24
Analyzing the TIMSS 2003 Field-test Data
  • Diagnostic statistics
  • Difficulty
  • Discrimination
  • Scoring reliability
  • Item-by-country interaction
  • For each country and language groups within
    country

25
Reviewing the TIMSS 2003 Field-test Results
  • TIMSS PIRLS ISC and Coordinators reviewed
    results
  • SMIRC reviewed results, recommended items for
    TIMSS 2003 data collection
  • NRCs reviewed and approved final item selection
    Aug. 2002

26
Conducting the TIMSS 2003 Scoring Training
  • Task Force met to update sample responses for
    training materials
  • NRC meeting to conduct scoring training March
    2003
  • Note A separate scoring training conducted
    earlier for Southern Hemisphere Nov. 2002

27
The Design for the TIMSS 2003 Assessment
  • Many, many Mathematics and Science items
  • Half newly-developed, half from 1995 and 1999 to
    measure trends
  • Almost 400 items at Eighth Grade (7 hours)
  • More than 300 items at Fourth Grade (5½ hours)
  • Limited student testing time
  • 90 minutes at Eighth Grade
  • 72 minutes at Fourth Grade

28
Matrix Sampling Approach
  • Assign items to blocks (10-15 items/block)
  • 28 blocks (14 Mathematics 14 Science)
  • Construct student booklets from item blocks
  • 6 blocks per booklet
  • Use IRT scaling to compute student achievement
    scores

29
TIMSS 2003 Item Block Design
30
Aspects of Booklet Design
  • Each booklet
  • 6 blocks per booklet
  • Mathematics and Science blocks
  • Trend blocks and new blocks
  • Counterbalanced design
  • Linking booklets Each block in 2-3 booklets

31
Sampling Design for Students
  • Accuracy requirements
  • Standard Error of 5 score points on TIMSS scale
  • Basic sampling design
  • 150 schools
  • 1 (or 2) classroom per school
  • 4,000 students (approx)

32
Standards for Population Coverage and Sampling
Participation
  • 100 Population coverage
  • Results annotated if student exclusions greater
    than 5
  • 100 Sampling participation
  • Results annotated if school and student
    participation less than 85
  • Approved and documented sampling method

33
Data Collection
  • Data collection a national responsibility
  • TIMSS PIRLS ISC provided detailed manuals
  • Quality Assurance program
  • International (15 schools per grade)
  • National (At Least 15 schools per grade)
  • A Huge Effort!!

34
Database Construction IEA DPC
  • Checked each countrys data files for internal
    consistency and accuracy
  • Liaised with countries to resolve data issues
  • Provided NRCs with data almanacs for review
  • Incorporated all modifications into final
    database
  • Formatted database and sent to TIMSS PIRLS ISC
    for analysis and reporting

35
Used IRT (Item Response Theory) Scaling Methods
to Assign Scores to Students
  • With IRT scaling,
  • Students score does not depend on taking the
    same set of items
  • Student can take just part of the test
  • IRT scaling is particularly useful when
  • Different test blocks have to be linked
  • Different samples of students have to be linked

36
Assigning Scores to Students
  • TIMSS 2003 used a procedure to assign students
    scores that combined information
  • Item difficulty
  • Students responses to the items they took
  • Students background characteristics
  • Scores were located on the TIMSS achievement
    scales (mean of 500 and standard deviation of
    100)
  • Mathematics and Science Scales
  • Overall and in content areas at each grade
  • Five Plausible values (PV) per scale
  • representing the most likely distribution per
    student

37
Linking Across Assessments
  • Using Grade 8 mathematics as an example
  • Created the TIMSS mathematics achievement scale
    in 1995
  • All items, all countries
  • In 1999, scaled the mathematics data and placed
    it on the 1995 metric
  • In 2003, scaled the mathematics data and placed
    it on the 1995 metric

38
Data Dissemination
  • International reports
  • Secondary analyses reports
  • International database and analysis programs
  • National reports
  • International database seminars
  • Online analyses

39
Three International Reports
  • Mathematics Eighth and Fourth Grades
  • Science Eighth and Fourth Grades
  • Mostly descriptive
  • Interpretation left to researchers of
    participating countries
  • Technical Report

40
Contents of TIMSS 2003 International Reports
  • Mathematics and Science achievement
  • Grades 4 and 8
  • 2003 results for all participants
  • Three-cycle trend at eighth grade
  • 1995, 1999, 2003
  • Two-cycle trend at fourth grade
  • 1995, 2003
  • Background questionnaires

41

2003, 1999, and 1995
2003 and 1999
2003
Australia Belgium (Flemish) Bulgaria Cyprus Engla
nd Hong Kong, SAR Hungary Iran, Islamic Rep.
of Israel Italy Japan Korea, Rep.
of Latvia Lithuania Netherlands New
Zealand Romania Russian Federation Singapore Slova
k Republic Slovenia South Africa United
States __________________ Ontario Province,
Can. Quebec Province, Can.
Argentina Chile Chinese Taipei Indonesia Jordan M
acedonia, Rep. of Malaysia Moldova, Rep.
of Morocco Philippines Tunisia ______________ Indi
ana State, US

Armenia Bahrain Botswana Egypt Estonia Ghana Leba
non Palestinian National Authority Saudi
Arabia Serbia Syrian Arab Republic Yemen _________
__________ Basque Country, Spain
2003 and 1995
Norway Scotland Sweden
42
(No Transcript)
43
(No Transcript)
44
(No Transcript)
45
(No Transcript)
46
(No Transcript)
47
International database (IDB)
  • All data files from IEA surveys available to
    download for free from the internet including
    programs and documentation http//www.iea.nl/iea_
    studies_datasets.htm
  • Structured by
  • Study
  • Cycle
  • Population

48
Analyis Program IDB Analyzer
  • Allows straight forward analyses of IEA data
    (TIMSS and PIRLS)
  • Grapical User Interface in SPSS environment
  • No editing of syntax
  • Capabilities
  • Combines countries
  • Combines data from different levels (students,
    teachers, schools)
  • Selection of variables
  • Calculates correct standard errors for various
    statistics
  • Sampling errors
  • Imputation errors

49
Analysis Module Running Analysis
Selecting the parameters (here selecting math PV
as dependent variable)
Weight variables have been pre-defined
50
Analysis Result Output Window
Original SPSS report
51
Analysis result Syntax Window
Contains whole analysis syntax allows to re-run
the analysis
52
National Reports and Training
  • IEA relies on national researchers to have
    greatest insight in their system and beeing able
    to advice policy makers most skillfully
  • To fascilitate national analysis, IEA conducts
    international and regional seminars regulary and
    on request

53
International Database Seminars
  • 19.-23. Feb. 2006 TIMSS 2003 international data-
    base training for the arabic region in Amman,
    Jordan
  • 17.-19. Oct. 2005 TIMSS 2003 Database
    introduction and HLM workshop
  • 9.-11. Jan. 2005 TIMSS 2003 Database and scaling
    technology
  • Oct. 2004 PIRLS 2001 Database construction and
    structural equation modelling

54
Online Analyses
  • Data can be analysed online under
  • www.IEADATA.ORG
  • Available February 2006
  • TIMSS 2003 data first

55
Online Analysis
56
IEA and TIMSS
  • Methodological Tools for Accountability Systems
    in Education
  • Ispra, 7 February
  • By Heiko Sibberns
Write a Comment
User Comments (0)
About PowerShow.com