PowerPointPrsentation

1
Sensitivity of Multiple Classification Models To
Misspecifications in the Q-Matrix

Co-author Jon Templin, University of Kansas
Prof. Dr. André A. Rupp Humboldt-Universität zu
Berlin (IQB) Unter den Linden 6 Sitz
Jägerstraße 10 /11 10099 Berlin E-mail
andre.rupp_at_iqb.hu-berlin.de
2
Overview of the Presentation

• Introduction to Problem
• Multiple classification models
• Q-matrix specification
• Study Design
• Simulation model
• Simulation design
• Software used
• Statistics measured
• Results
• (a) Effects of misspecifications on parameter
  estimates
• Effects of misspecifications on respondent
  classifications
• Conclusions

3

• Introduction to Problem

4
Multiple Classification Models

• Objectives
• To simultaneously classify respondents according
  to multiple latent
• characteristics (i.e., traits / abilities /
  skills / attributes)
• To allow for multiple criterion-referenced
  interpretations
• To show efficient pathways toward mastery of all
  characteristics
• To learn about the difficulty of individual
  characteristics
• To learn about the difficulty and discriminatory
  properties of items
• To develop diagnostically optimal non-adaptive
  tests
• To develop diagnostically optimal adaptive tests

5
Example of Attribute Profile
6
Example of Attribute Difficulty
7
Multiple Classification Models

• Statistical Background
• The models are restricted latent class models
  (Haertel, 1989), also known as
• multiple classification models (e.g., Maris,
  1999)
• cognitive diagnosis models (e.g., Templin
  Henson, in press)
• structured IRT models (Rupp Mislevy, in
  press)
• cognitive psychometric models (Rupp, in press).
• They are multidimensional factor models with
  binary or ordinal latent variables (see, e.g.,
  Muthén, 2003) that can model a complex item
  structure.
• Examples of such models
• Rule-space methodology (e.g., Tatsuoka, 1983,
  1995)
• DINA and NIDA models (e.g., de la Torre
  Douglas, 2004 Junker Sijtsma, 2001)
• RUM / Fusion model (e.g., Hartz, 2002 Templin
  Henson, 2003)
• General diagnostic models (e.g., von Davier,
  2005)
• Bayesian inference networks (e.g., Williamson,
  Almond, Mislevy, Levy, 2006)

8
Structure of a Q-matrix
The attributes may be required or not required
for responding to a particular item (dichotmous
0-1 code) oder may be required to a certain
degree (polytomous code such as 0-1-2).
9
Subtraction Items in Mathematics
Example
10

• Study Design

11
Simulation Model

• In the DINA (Deterministic-inputs, noisy and
  gate) model, attributes are combined in a
  non-compensatory fashion to form a latent
  response variable.
• This variable characterizes the response process
  at the item level, which is characterized by two
  error probabilities representing slipping and
  guessing behavior. Structurally, it is the
  simplest multiple classification model.
• where

12
Parameter Values for Generating Data (I)
13
Parameter Values for Generating Data (II)
14
Misspecification Conditions
15

• Results

16
Effects on Item Parameters
17
Effects on Respondent Classifications
18
Conclusions

• Dominant effects of misspecifications on item
  parameters
• Slipping parameters were overestimated for an
  item whenever an attribute was deleted
• ? Items were specified as easier than
  they actually were
• Guessing parameters were overestimated for an
  item whenever an attribute was added
• ? Items were specified as harder than
  they actually were
• ? Item Parameters are local measures of
  misspecification
• Dominant effects of misspecifications on
  respondent classifications
• Attribute patterns that were deleted or added
  from the assessment led to misclassifications of
  respondents with these patterns
• ? Attribute patterns are local measures of
  misspecification for respondents

19
The Sensitivity of Multiple Classification
Models To Misspecifications in the Q-Matrix

Co-author Jonathan Templin, University of Kansas
Prof. Dr. André A. Rupp Humboldt-Universität zu
Berlin (IQB) Unter den Linden 6 Sitz
Jägerstraße 10 /11 10099 Berlin E-mail
andre.rupp_at_iqb.hu-berlin.de