Assessment of ModelData Fit

1
Assessment of Model-Data Fit

• Stephen Stark and Oleksandr Chernyshenko
• University of Illinois at Urbana-Champaign

2
Conceptual Issues of Fit

• Benefits of IRT methods may not be realized
  unless an IRT model, used for parameter
  estimation, adequately fits the data
• No a priori justification why a model should
  describe data adequately
• More general models having less restrictive
  assumptions will fit better, but require larger
  samples for parameter estimation
• Choice of a model must be based on theoretical
  and empirical grounds

3
Conceptual Issues of Fit

• For cognitive ability data, a multitude of
  studies has shown that the 3PL model fits very
  well.
• Numerous questions remain regarding the fit of
  IRT models to personality and, more generally,
  noncognitive data.

4
Graphical Methods of Assessing Fit
Example Fit Plot for the 3PL Model
5
Graphical Methods of Assessing Fit

• Traditional Method of Creating Fit Plots
• Estimate item parameters in a calibration sample
• Score respondents, and sort into theta bins
• Compute proportion of respondents in each bin who
  responded positively (an empirical point)
• Problems
• Because of estimation error, theta estimates
  rarely equal true thetas
• Even with large samples and long tests, an
  empirical curve may differ systematically from
  the true response function.
• Capitalization on chance
• Same sample used for estimating parameters and
  examining fit

6
Graphical Methods of Assessing Fit

• Levine and Williams Method
• Empirical point is given by ratio of two
  posterior densities (uses thetas, not theta
  estimates)
• Empirical points computed using cross-validation
  sample

7
Statistical Methods for Assessing Fit

• Chi-square statistics are often used to assess
  goodness of fit
• Typically, chi-squares computed for single items

8
Problems with Chi-Square Singles

• Sensitive to sample size
• Insensitive to certain forms of misfit

9
Improved Chi-Square Method

• Compute chi-squares for pairs and triplets of
  items
• Solves problem of small chi-squares due to
  cancellation (previous slide)
• Can detect violations of local independence
• Adjust chi-squares to sample size of 3,000
• Provides standard for comparison across groups
• Divide chi-squares by degrees of freedom
• Allows for comparison of different models

10
Example Chi-Square Results

• Rule of Thumb
• Adjusted chi-square / df ratio of less than 3
  indicates good fit

AdjChi/df lt 3
11
The MODFIT Program

• Excel 2000 application for model-data fit
  analysis
• Computes for Dichotomous and Polytomous Models
• IRFs / ORFs, TCCs
• IIFs, TIF, SE
• Fit Plots
• Chi-Square singles, doubles, and triples
• System Requirements
• Windows 95/98, NT/2000, ME running Excel 2000