Principled Assessment Design, with an Eye Toward HighLevel Expertise

1
Principled Assessment Design, with an Eye Toward
High-Level Expertise

• Robert J. Mislevy
• Measurement, Statistics, Evaluation
• University of Maryland
• March 11, 2004

2
Expertise Research (1)

• The study of expertise seeks to understand and
  account for what distinguishes outstanding
  individuals in a domain from less outstanding
  individuals in that domain.
• Informal evaluations are influenced by
  observations of previous performances, occurring
  under unique circumstances.
• Differential approach proved unsatisfactory.
• Eg, chess memory tests

3
Expertise Research (2)

• Focus on producing and observing outstanding
  performance under reproducible and relatively
  standardized conditions. (E.g., chess positions)
• Aim to analyze and describe the cognitive
  processes critical to producing outstanding
  performance on such tasks.
• More recent research address interactions among
  participants and knowledge in physical
  situations and processes.
• E.g., emergency room airline cockpit factory
  floor

4
Cognitive Task Analysis

• A traditional job analysis focuses on valued
  tasks in a domain, in terms of how often people
  must perform them and how important they are.
• A cognitive task analysis (CTA) focuses on the
  knowledge people use to carry out those tasks.
• More recent research extends to patterns of
  communication and interaction with others, in
  target environment.
• C.f. Task-based language assessment
• Tools of discourse analysis and ethnology as well
  as those of cognitive psychology.

5
Potential Observations
t0 Start of info presentation for specific task
t1 Generation of answer
...
Processing step 1
Processing step 2
Processing step n
Reaction time t1-t0
E1
E2
Ek
Eye fixations
Kq
K1
K2
K3
Keystrokes
V1
V2
Vm
Think-aloud verbalizations during solution
6
Potential Observations
t0 Start of info presentation for specific task
t1 Generation of answer
...
Processing step 1
Processing step 2
Processing step n

• Post-process
• observations
• Memory for task info
• Retrospective report
• Post-experiment interview

Reaction time t1-t0
E1
E2
Ek
Eye fixations
Kq
K1
K2
K3
Keystrokes
V1
V2
Vm
Think-aloud verbalizations during solution
7
Potential Observations
t0 Start of info presentation for specific task
t1 Generation of answer
...
Processing step 1
Processing step 2
Processing step n

• Post-process
• observations
• Memory for task info
• Retrospective report
• Post-experiment interview

Reaction time t1-t0
E1
E2
Ek
Eye fixations
Kq
K1
K2
K3
Keystrokes

• Work Products
• Qualities of solution product
• Qualities of solution process
• Efficaciousness

V1
V2
Vm
Think-aloud verbalizations during solution
8
Some Pervasive Results

• Knowledge structures are key, for
• Perceiving situations
• Solving problems in terms of generative
  principles of the domain
• Generating plans monitoring progress
• Automaticity
• Frees resources for novel aspects of problems
• Extensive reflective practice, with feedback.

9
CTA for Assessment

• CTA can help ground assessment, re validity
  argument and principled/explicated task
  construction.
• Must aim to specify elements of Student-,
  Evidence-, and Task-models,
• (vs. cognitive scientists aims!)
• in light of the intended assessments purpose,
• and constraints of the assessment-building and
  assessment-delivery contexts.

10
DISC Example

• The Dental Interactive Simulations Corporation
  (DISC)
• DISC / ETS / Chauncey Group Intl.
• The DISC Simulator
• The DISC Scoring Engine
• Evidence-Centered Assessment Design
• The Cognitive Task Analysis

11
Expert hygienists helped us to...

• Obtain information about nature of job knowledge
  areas, conditions of work, typical
  problem-solving decision-making
• Devise nine virtual cases, for CTA solution
  protocols, reflecting above
• Elicit open-ended talk-aloud solutions to same
  stimuli from hygienists at different levels of
  expertise
• Analyze protocols to discern overarching patterns
  that characterize problem-solving behavior, again
  in ways that distinguish levels of expertise.

12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)