The International Comparability of Socioeconomic Measures in PISA 2006

1
The International Comparability of Socioeconomic
Measures in PISA 2006
Monika Townsend Stéphane Baldi American
Institutes for Research
2
Introduction

• Results from PISA 2006 indicate a clear
  association of individual-level socioeconomic
  characteristics with science achievement
  consistent with previous research (Blau Duncan,
  1967 Shavit and Blossfield 1993).
• OECD has been revising its index of socioeconomic
  status each time PISA is administered in an
  attempt to provide an accurate measure that is
  comparable across countries.
• While the most recent version allowed for some
  between-country variation in the creation of one
  of its indicators (home possessions), it still
  largely treats many of the key components making
  up socioeconomic status as operating in the same
  way across all participating countries.

3
Introduction (Continued)

• Many researchers and policymakers question this
  assumption as well as the overall validity of the
  PISA socioeconomic index. For example, being a
  doctor in the United States may mean something
  quite different than in Tunisia.
• This paper examines the validity of the PISA
  measure of socioeconomic status by testing an
  alternative model of socioeconomic status in
  which its components are allowed to vary across
  countries.
• None of the models presented here exactly
  reproduces the OECD method for index scaling.
  Therefore, this analysis should not be considered
  as a critique of the OECD Model rather, it
  presents support for an alternative approach.

4
Purpose

• This paper examines the cross-national validity
  of the PISA 2006 index of economic, social and
  cultural status (ESCS).
• We explored the measurement properties of indices
  related to socioeconomic status using multigroup
  confirmatory factor analysis (CFA).
• Analysis focused on a subset of jurisdictions
  with varying levels of economic development,
  including both OECD and partner countries.

5
Research Questions

• Is the OECD assumption that the three key
  indicators of socioeconomic status operate in the
  same ways in each country supported by the data?
• Do specific variables or indices of ESCS
  contribute to measurement problems more than
  others?

6
Approach

• Two CFA models were created, both of which
  approximate the creation of the ESCS variable as
  a latent factor.
• Single factor
• Two-tier factor
• For each of these models, two versions of the
  model were examined to test the assumption of
  factor invariance between countries.
• Paths constrained ESCS components assumed to
  work the same way across countries
• Unconstrained contributions of ESCS components
  allowed to vary by country
• Comparing the two versions tests the assumption
  of factor invariance between countries. If the
  unconstrained version has a better model fit, and
  that improvement is statistically significant, we
  can say that the measure of ESCS can be improved
  in its measurement properties by allowed for
  national variation of its components.
• It is also possible to identify what parts of the
  model might be causing problems, both overall
  from results of the constrained model, and in
  particular countries, from results of the
  unconstrained model.

7
Socioeconomic Status in PISA

• Three main indicators of socioeconomic status
  income, occupation, and education (Gottfried
  1985 Hauser 1994 Mueller and Parcel, 1981).
  Home possessions used as a proxy for income.
• Based on results from Differential Item
  Functioning, OECD used nationally defined
  parameters for scaling when pooling items to
  create the indices related to home possessions.
  The ESCS variable combined home possessions with
  parent education and occupation.
• Index creation was a multistage process indices
  were created from sets of individual items, then
  indices and items were pooled to create the ESCS
  variable.
• Although parameters for initial index creation
  were nationally defined, during the second
  pooling, parameters were not allowed to vary by
  country (i.e. assumed to contribute equally
  across countries).

8
Schematic
Did not vary by country
Varied by country
WEALTH
HOMEPOS
HEDRES
HISEI
ESCS
CULTPOSS
PARED
BOOKS
Items and Indices
Final SES Index
Items
Indices
9
Sample

• Jurisdictions were identified using cluster
  analysis, which identified 3 clusters using the
  centroid linkage method.
• Jurisdictions were selected from each of the
  clusters with a focus on geographical diversity.
• HDI Human Development Index. Created using
  life expectancy, GDP, literacy and education. All
  jurisdictions were grouped in the high
  development category except Tunisia. (
  indicates non-OECD partner country)

10
Variables Contributing to the Model
11
Univariate Summary Statistics
12
Confirmatory Factor Analysis (CFA)

• Conventions
• Observed, measured variables
• Latent, unmeasurable variables
• Regression weights
• Measurement error

13
Single Factor Model Constrained
Model misfit RMSEAlt.05 Good fit RMSEAlt.10
Moderate RMSEAgt.1 Poor fit All factor loadings
standardized

• Residuals were free to vary between countries
  this image shows residuals for Brazil.
• Overall model fit was poor.
• Squared Multiple Correlations are fit indicators
  for individual variables. These values for PARED
  were especially low for Brazil (R20.09) and
  Tunisia (R2.12 all other R2gt.20)

14
Single Factor Model Constrained

• Results for the 6-variable single-factor solution
  under conditions where the model was not allowed
  to vary by country indicate poor overall model
  fit. In addition, the variable PARED fit poorly
  in the countries of Brazil and Tunisia.
• This was the constrained model, where we made the
  same assumption the OECD did that contributions
  to ESCS do not vary across countries.
• Next, the unconstrained model will allow us to
  see if the model fits any better when we allow
  paths to vary across countries.

15
Single Factor Model Unconstrained
Chi-Square7249.57, df72, p-value0.00
RMSEA0.132
16
Single Factor Model Unconstrained

• Chi square difference test between the two models
  (all paths constrained vs. free to vary between
  countries) ?2 ? 9901 ?2cv(42) 58 so the
  unconstrained model is significantly better.
• However, overall Model fit RMSEA.13 still poor.
• Squared Multiple Correlations for WEALTH low in
  Finland (R20.04).
• Standardized factor loadings gt1 for PARED in
  Brazil and Tunisia may indicate poor model fit
  residuals for this index also exceed 1 in both
  countries.
• subsequent attempts to free single paths, or
  combinations of paths, did not improve model fit.

17
Single Factor Model Unconstrained

• Results for the 6-variable single-factor solution
  under conditions where the model was allowed to
  vary by country indicate better, but still poor,
  overall model fit.
• This tests the theory that parameters should be
  allowed to vary by country when creating ESCS.
  Since the model fit was better when paths were
  allowed to vary by country, results again suggest
  that ESCS might be improved by allowing national
  variation in the way the components operate.
• In addition, the WEALTH index variable fit poorly
  in the country of Finland, while PARED fit poorly
  in the countries of Brazil and Tunisia.

18
Two-Tiered Model Constrained

• Overall model fit poor, RMSEA.18
• No communalities low (but none high either)

19
Two-Tiered Model Constrained

• This model represents a different approach to
  ESCS creation as a latent factor using the same
  six variables. In this model the variables
  contributing to ESCS match the OECD method in
  number. However, instead of entering the HOMEPOS
  index directly, a latent factor was estimated
  using the components of HOMEPOS.
• Results where the model was not allowed to vary
  between countries showed poor model fit but no
  irregularities for specific variables.
• Next, as before, we will examine the
  unconstrained model to again test the assumption
  of invariance in factor loadings across
  countries.

20
Two-Tiered Model Unconstrained

• Chi-Square 15,136.29, df127, p-value0.00,
  RMSEA0.14
• Two paths leading into ESCS were allowed to
  vary HOMEPOS PARED

21
Two-Tiered Model Unconstrained
22
Two-Tiered Model Unconstrained

• Chi square difference test between the two models
  (all paths constrained vs. free to vary between
  countries) ?2 ? 14,404 ?2cv(28) 41.34 so the
  unconstrained model is significantly better.
• However, the overall Model fit RMSEA.14 is still
  poor.
• Standardized factor loadings gt1 for PARED in
  Brazil and Tunisia are permissible in the
  presence of collinearity among latent variables.
• A lower factor loading for PARED in Russia
  (r.08) compared to other countries may be
  related to educational policies in Russia
  compared to other countries, but it may also be
  attributable to differences in the nature and
  comparability of educational attainment under
  different systems.

23
Two-Tiered Model Unconstrained

• Results where the model was allowed to vary
  showed better, but still poor, overall model fit.
  
• This again tests the theory that parameters
  should be allowed to vary by country when
  creating ESCS. Again, model fit improved after
  variance across countries was permitted.
• There were fewer indicators of problems with
  specific variables in this model.

24
Conclusions

• Is the OECD assumption that the three key
  indicators for socioeconomic status operate in
  the same ways between countries supported by the
  data?
• No. Our models fit better when parameters were
  allowed to vary between countries.
• What particular variables or indices may be
  contributing to measurement problems more than
  others?
• PARED in Brazil, Tunisia.
• WEALTH in Finland.

25
Discussion

• Recent developments in the fields of education
  research and policy analysis have allowed for
  evolution in our understanding of the importance
  of socioeconomic status and its relationship to
  academic achievement across countries.
• In response to calls for a reliable measure of
  socioeconomic status that is applicable across
  countries, OECD has been working consistently on
  every version of PISA to improve the indicators
  for this measure.

26
Discussion (Continued)

• In PISA 2006, OECD allowed national definition of
  parameters when creating scale indices for
  WEALTH, CULTPOSS, HEDRES, and HOMEPOS, but not
  for ESCS.
• Our research suggests that using similar
  procedures for the generation of ESCS might
  further improve this indicator.
• Poor overall model fit in our models should not
  call into question the OECD model because our
  models do not perfectly reproduce the OECD method
  for generating ESCS.

27
References
Blau, P.M., and Duncan, O. D. (1967). The
American Occupational Structure. New York John
Wiley Sons. Ganzeboom, H.B.G., de Graaf, P.M.
Treiman, D.J. (1992). A Standard International
Socio-economic Index of Occupational Status,
Social Science Research, No. 21, pp. 1-56.
Gottfried, A. (1985). Measures of Socioeconomic
Status in Child Development Research Data and
Recommendations. Merrill-Palmer Quarterly, 31(1)
85-92. Hauser, R.M. (1994). Measuring
Socioeconomic Status in Studies of Child
Development. Child Development, 65(6) 1541-1545.
Mueller, C.W., and Parcel, T.L. (1981). Measures
of Socioeconomic Status Alternatives and
Recommendations. Child Development, 52 13-30.
Organization for Economic Cooperation and
Development. (1999). Classifying Educational
Programmes. Manual for ISCED-97 Implementation in
OECD Countries. OECD Paris. Shavit, Y., and
Blossfield, H.P. (Eds.). (1993). Persistent
Inequality Changing the Educational
Stratification in Thirteen Countries. Boulder,
CO Westview.
28

• Single factor unstandardized estimates
• All loadings constrained

29

• Single factor unstandardized estimates
• Factor loadings unconstrained

30

• Two factor unstandardized estimates
• All loadings constrained

31

• Two factor unstandardized estimates
• Factor loadings unconstrained

32

• Two factor unstandardized estimates
• Factor loadings unconstrained