TheoryBased Task Design: Comments on Embretson 1998

1
Theory-Based Task DesignComments on Embretson
(1998)

• Robert J. Mislevy
• University of Maryland
• April 20, 2001

2
Overview

• Background
• Embretsons Design Process
• The Cognitive Theory for this example
• Correspondence to the ECD Models
• The Task model
• The Student model
• The Evidence Model Evidence Identification
• The Evidence Model Evidence Accumulation
• Integrating the Substantive Statistical
  Arguments

3
Background (1)

• Often, ability test item types are suitable for
  studying cognitive theories, because they are
  complex problem-solving tasks. Jack Carroll
• However, applying cognitive concepts to describe
  traditonal psychometric findings misses the real
  potential of cognitve theory namely, cognitive
  theory is useful for test design.
• Cognitive research also provides results on how
  item stimulus properties influence the cognitive
  processes that are involved in item solving.
  (p.380)

4
Background (2)

• Two kinds of arguments in construct validation
• Meaning (construct representation)
• The nature of the construct in characterizing
  behavior operational definitions
• Significance (nomothethic span)
• The relationship of the construct to other
  constructs or variables.
• In our terms, these are two kinds of backing.

5
Background (3)

• Most test authoring in applied work relies on
  intuition tradition for meaning, focuses on
  establishing Significance through correlations of
  test, more or less as given, with other
  variables.
• Cognitive theory has a lot to contribute to
  Meaning, especially when used to build a test
  rather than applied after the fact.

6
Nomothetic Span
C Sue has a large vocabulary.
Ubiquitous means a) Large rodent b)
Its everywhere c) Los Angeles
W(1) Students who are familiar with many words
can identify close synonyms of them, from a
number of choices.
since
On account of
so
Backing Answering mc items correctly is strongly
correlated with counts of unique words used in
spontaneous writing.
D(student) Sue selected the correct response for
ubiquitous
7
Construct Meaning
C Sue has a large vocabulary.
W(1) Students who are familiar with many words
can identify close synonyms of them, from a
number of choices.
Ubiquitous means a) Large rodent b)
Its everywhere c) Los Angeles
W(2) Whatever a students vocabulary level, a
correct response is increasingly likely for
frequently used words.
since
W(3) Whatever a students vocabulary level, a
correct response is less likely as the
alternatives are further from the prompt in
semantic distance.
so
On account of
Features of task condition application of the
warrant How much to modify belief based on
student data.
Backing Cognitive studies on word use word
retrieval, with attention to impact of frequency
and semantic distance
D(task2)The distractors are semantically far away
D(task2) ubiquitous is an infrequently used
word.
8
Embretsons Design Process (1)
A la Messick

• Specify goals of measurement
• Identify design features in task domain
• Common features variable features
• Develop a cognitive model
• Evaluate cognitive model for psychometric
  potential
• Features of tasks vis a vis empirical patterns

Re Task Model
Given above!
Either incorporating implications of cognitive
theory into statistical model explicitly, or if
not, examining it for consistency
9
Embretsons Design Process (2)

• Specify item distributions on cognitive
  complexity
• Distribution of item complexity parameters
• Distribution of item features
• Generate items
• Evaluate cognitive/psychometric models for test
  with empirical response data in light of purpose
• Validation Strong program of hypothesis testing

Rules for Assembly Model
According to Task Models
Strong backing for warrant
10
Cognitive Theory (1) Ravens matrices items

• Rules across rows down columns,
• for modifying icons, whole or in parts.

?
One pairwise progression, one distribution of
three
11
Cognitive Theory (1) Ravens matrices items
?
Identity relationship only
12
Cognitive Theory (2) Carpenter Just Results

• Mainly a test of working memory,
• but also have to know what the rules are
• identity, progression, addition/subtraction,
  distribution of n relations, distribution with
  null values
• More rules gt harder
• Rules ordered in difficulty

13
Correspondence with ECD Models
Assembly Model
14
Embretsons Task Model (1)

• Just one task model proposed.
• Rationale for why behavior in planned contexts
  will evoke behavior that provides evidence about
  the skill/knowledge of interest
• Based on Carpenter Just
• Features that will be common to all items
  produced according to this TM
• Task stimulus format, number format of
  alternatives, work product description
• Notation system
• Tools conditions of administration

15
Embretsons Task Model (2)

• Features that will vary across items
• For each rule, (Rule, icon(s), orientation)
• Total number of rules
• Vector of 0/1 additional-load feature
  indicators
• (Abstraction, distortion, overlay, fusion)
• Memory load (weighted sum of rules used)
• Key position

How were distractors generated?
16
Embretsons Task Model (3)

• Incidental vs. Radical TM variables
• Structural equivalence implies that processing
  is not influenced by the specific item content
  when structures are identical.
• Thus, item difficulties, response times, and
  correlations should be comparable for an item
  structure with different objects and features.

17
Embretsons Student Model

• IRT q (theta)
• Overall proficiency in the domain of items.
• By theory underlying construction, mainly a
  measure of working memory.
• Linear logistic test model used.

18
Embretsons Evidence Model (1)Evidence
Identification

• Work product (option choice, response time)
• Evaluation rule Option choice key?
• Yes --gt Xij1 No --gt Xij0

19
Embretsons Evidence Model (2)Evidence
Accumulation
where
20
Embretsons Student Model (3) Evidence
Accumulation, continued
lt--easier harder--gt
Items
Item with q1 just one identity rule
Item with q6
Item with q10
Jims q
Sues q
Students
lt--less proficient more proficient--gt
21
Comments

• Task model variables play several roles
• Task construction.
• Test assembly (mix of rules, difficulties).
• Evidence accumulation, modeling item difficulties
• Interpretation of q in terms of observable
  behavior in the task domain.
• Connect q tightly with research backing.