A Cognitive Diagnosis Model for CognitivelyBased MultipleChoice Options

1
A Cognitive Diagnosis Model forCognitively-Based
Multiple-Choice Options
Jimmy de la Torre Department of Educational
Psychology Rutgers, The State University of New
Jersey
2
All wrong answers are wrong
But some wrong answers are more wrong than
others.
3
Introduction

• Assessments should educate and improve student
  performance, not merely audit it
• In other words, assessments should not only
  ascertain the status of learning, but also
  further learning
• Due to emphasis on accountability, more and more
  resources are allocated towards assessments that
  only audit learning
• Tests used to support school and system
  accountability do not provide diagnostic
  information about individual students

4

• Tests based on unidimensional IRT models report
  single-valued scores that submerge any distinct
  skills
• These scores are useful in establishing relative
  order but not evaluation of students' specific
  strengths and weaknesses
• Cluster scores have been used, but these scores
  are unreliable and provide superficial
  information about the underlying processes
• Needed are assessments that can provide
  interpretative, diagnostic, highly informative,
  and potentially prescriptive information

5

• Some psychometric models allow the merger of
  advances in cognitive and psychometric theories
  to provide inferences more relevant to learning
• These models are called cognitive diagnosis
  models (CDMs)
• CDMs are discrete latent variable models
• They are developed specifically for diagnosing
  the presence or absence of multiple fine-grained
  skills, processes or problem-solving strategies
  involved in an assessment

6

• Fundamental difference between IRT and CDM A
  fraction subtraction example
• IRT performance is based on a unidimensional
  continuous latent trait
• Students with higher latent traits have higher
  probability of answering the question correctly

7
(No Transcript)
8

• Fundamental difference between IRT and CDM A
  fraction subtraction example
• IRT performance is based on a unidimensional
  continuous latent trait
• Students with higher latent traits have higher
  probability of answering the question correctly
• CDM performance is based on binary attribute
  vector
• Successful performance on the task requires a
  series of successful implementations of the
  attributes specified for the task

9

• Required attributes

(1) Borrowing from whole
(2) Basic fraction subtraction
(3) Reducing

• Other attributes

(4) Separating whole from fraction
(5) Converting whole to fraction
10
(No Transcript)
11
Background

• Denote the response and attribute vectors of
  examinee i by and
• Each attribute pattern is a unique latent class
  thus, K attributes define latent classes
• Attribute specification for the items can be
  found in the Q-matrix, a J x K binary matrix
• DINA (Deterministic Input Noisy And gate) is a
  CDM model that can be used in modeling the
  distribution of given

12

• In the DINA model
• where
• is the latent group classification of examinee
  i with respect to item j
• P(Hg) is the probability that examinees in
  group g will respond with h to item j
• In more conventional notation of the DINA
• guessing,
  slip

13

• Of the various test formats, multiple-choice (MC)
  has been widely used for its ability to sample
  and accommodate diverse contents
• Typical CDM analyses of MC tests involve
  dichotomized scores (i.e., correct/incorrect)
• The approach ignores the diagnostic insights
  about student difficulties and alternative
  conceptions in the distractors
• Wrong answers can reveal both what students know
  and what they do not know

14

• Purpose of the paper is to propose a
  two-component framework for maximizing the
  diagnostic value of MC assessments
• Component 1 Prescribes how MC options can be
  designed to contain more diagnostic information
• Component 2 Describes a CDM model that can
  exploit such information
• Viability (i.e., estimability, efficiency) of the
  proposed framework is evaluated using a
  simulation study

15
Component 1 Cognitively-Based MC Options

• For the MC format, ,
  where each number represents a different option
• An option is coded or cognitively-based if it is
  constructed to correspond to some of the
  latent classes
• Each coded option has an attribute specification
• Attribute specifications for non-coded options
  are implicitly represented by the zero-vector

16
A Fraction Subtraction Example

• A) B)
• C) D)

17
Attributes Required for Each Option of
18

• The option with the largest number of required
  attributes is the key

19
Attributes Required for Each Option of
20

• The option with the largest number of required
  attributes is the key
• Distractors are created to reflect the type of
  responses students who lack one or more of the
  required attributes for the key are likely to give

21
Attributes Required for Each Option of
22

• The option with the largest number of required
  attributes is the key
• Distractors are created to reflect the type of
  responses students who lack one or more of the
  required attributes for the key are likely to
  give
• Knowledge states represented by the distractors
  should be in the subset of the knowledge state
  that corresponds to the key
• Number of latent classes under the proposed
  framework is equal to , the number of
  coded options plus 1

23
0
24
0
1
000
001
010
100
011
101
110
111
25
1
2
3
000
001
010
100
011
101
110
111
26
1
2
4
3
0
000
001
010
100
011
101
110
111
27
Component 2 The MC-DINA Model

• Let be the Q-vector for option h of item
  j, and
• With respect to item j, examinee i is in group
• Probability of examinee i choosing option h of
  item j is

28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
0
1
2
3
4
36
(No Transcript)
37
DINA Model for Nominal Response N-DINA Model
38
A
B
C
D
Group
0
1
39
(No Transcript)
40
P(10) guessing parameter
P(01) slip parameter
Plain DINA Model
41
A Simulation Study

• Purpose To investigate how
• well the item parameters and SE can be estimated
• accurately the attributes can be classified
• MC-DINA compares with the traditional DINA
• 1000 examinees, 30 items, 5 attributes
• Parameters
• Number of replicates 100

42
Results

• Bias, Mean and Empirical SE Across 30 Items

43
Attribute Classification Accuracy

• Percent of Attribute Correctly Classified

89.71
97.43
91.13
69.58
6.30
20.13
44
Summary and Conclusion

• There is an urgent need for assessments that
  provide interpretative, diagnostic, highly
  informative, and potentially prescriptive scores
• This type of scores can inform classroom
  instruction and learning
• With appropriate construction, MC items can be
  designed to be more diagnostically informative
• Diagnostic information in MC distractors can be
  harnessed using the MC-DINA

45

• Parameters of the MC-DINA model can be accurately
  estimated
• MC-DINA attribute classification accuracy is
  dramatically better than the traditional DINA
• Caveat This framework is only the psychometric
  aspect of cognitive diagnosis
• Development of cognitively diagnostic assessment
  is a multi-disciplinary endeavor requiring
  collaboration between experts from learning
  science, cognitive science, subject domains,
  didactics, psychometrics, . . .

46
Thats all folks!