What Can We Learn from PISA

1
What Can We Learn from PISA?

• Investigating PISAs Approach to Scientific
  Literacy

2
Dissertation Committee

• Mark Wilson
• Maryl Gearhart
• Samuel Lucas

3
Outline

• Scientific Literacy
• Research Questions
• Data
• Measurement Model
• Question 1
• Question 2
• Overview

4
What is PISA?

• Programme for International Student Assessment
• Organization for Economic Cooperation and
  Development (OECD)

5
PISA Administration Cycle
6
PISAs Scientific Literacy

• ability to undertake a number of fundamental
  processes in a range of situations, backed by a
  broad understanding of key concepts
• (OECD, 2000, p. 7)

7
Process

• Describing, explaining, and predicting scientific
  phenomena
• Understanding scientific investigation
• Interpreting scientific evidence and conclusion

8
Content
9
Situation

• Life and health
• Earth and environment
• Technology

10
Scientific Literacy

• The term scientific literacy embodies the desired
  goals of science education reform and in turn,
  reflects the desired components of science
  education.

11
Themes

• Scientific way of thinking
• Everyday relevance of science
• Equity and equality

12
Research Questions

• Q1. To what extent is there evidence of
  PISAs three competencies of scientific literacy
  in students scored responses to the PISA science
  items?

13

• Q2. How does the scoring and selected
  measurement model of student responses to the
  PISA science items reflect the intended structure
  of scientific literacy?

14
Data

• A total of 34109 students across six countries
  responded to 16 to 34 PISA 2003 science items

15
Item by Booklet
16
(No Transcript)
17
PISA Items

• All three components assessed within each item
• Multiple item formats
• Student responses coded and scored

18
Item Response Theory

• Subset of items measure one latent trait
• Student response to one item is independent to
  response to other items
• Missing data

19
Random Coefficient Multinomial Logit Model
20
MRCML
21
Question 1

• Unidimensional approach
• Consecutive approach
• Multidimensional approach

22
Process
23
Content
24
Situation
25
Process Model Fit Statistics
26
Process EAP Reliabilities
27
Process Dimension Correlations
28
Process Wright Map

3
X


X X
X
X X
XX X
X
XX XX X
XXX
XX XX
2 XXX XX
XX
XXXX XXX XXX
XXXXXX
XXX XXX
XXXXXX XXXXX
XXXXXU3
XXXXXXXX XXXXX XXXXXD8 U10 D33

XXXXXXXXX XXXXXX XXXXX
1 XXXXXXXXX
XXXXXXX XXXXXXXXI12
XXXXXXXXXX XXXXXXX
XXXXXXD7 D25
XXXXXXXXX XXXXXXXX XXXXXXXI2 D28

XXXXXXXXXX XXXXXXXXXXXXXXXXXI16
XXXXXXXXX
XXXXXXXX XXXXXXXXI9 D26
XXXXXXXXX XXXXXXXX
XXXXXXXX 6
XXXXXXXX XXXXXXX XXXXXXXXU21 D22
D24 1 0
XXXXXXX XXXXXXXXXXXXXXXXD5 I13 U14 I18 I32
2 3 4 7 XXXXXXX
XXXXXXX XXXXXXXXD27 D29 5
XXXXXX XXXXXXX
XXXXXXX
XXXXX XXXXXX XXXXXXXI1 D4
I17 D34
XXXX XXXXXX XXXXXXD23
XXXX
XXXXXX XXXXXU11 I30
-1 XX XXXXX
XXXXXXU6 I31
XXX XXX XXXXU20

XX XXX XXXD15
X XXX
XX
X XXX XX

XX X
-2 XX
X
X X

X
X

D19




-3



29
Implications

• Items are measuring the same latent trait
• More items are needed to target the differences
  between the dimensions or to capture the range of
  each dimension
• Educational perspective useful to break the
  ability into teachable traits

30
Process Component
31
Question 2

• Multiple choice
• Open constructed response
• Complex multiple choice

32
Multiple Choice

• Code for each distractor
• Score for correct and incorrect

33
Open constructed response

• Code from scoring guide
• Score full credit, partial credit, no credit

34
Complex multiple choice

• Code for each statement within item
• Score correct or incorrect

35
Open Constructed Response

• N 34109
• Eight items
• Scoring guides four to nine codes

36
Scoring OCR Item
37
Alternative Scoring OCR
38
OCR Models
39
PCM Design Matrix
40
Strict Bundle Model Design Matrix
41
OPM Design Matrix
42
OCR Model Results
43
OCR Item Parameter Fit Statistics
44
Implications

• Order partition model
• OCR scoring guides are item specific
• Construct of scientific literacy

45
Complex Multiple Choice

• N 34109
• Seven items
• Student responds to multiple statements within an
  item

46
Scoring CMC Item
47
CMC Models
48
SBM Design Matrix
49
Simple Logistic Bundle Model Design Matrix
50
OPM Design Matrix
51
CMC Model Results
52
CMC Item Parameter Fit Statistics
53
Implications

• Simple logistic model
• Relationship of statements
• Construct of scientific literacy

54
Overview