Designing the Purpose of an Assessment System

1
Designing the Purpose of an Assessment System
Robert J. Mislevy Michelle Riconscente Fielding
Winters Ting Zhang University of Maryland
Amelia Wenk Gotwals Michigan State University
2
The Why of PADI

• Advanced assessment requires coordination of
  varied expertise
• Assessment design is both science and art
• Capitalize on schemas for recurring structures,
  patterns, relationships
• Explicit validity arguments
• Accumulated wisdom sharablenot context bound
• Interoperability of elements processes (esp.
  technological settings)

3
Session Overview

• Some Thinking Tools
• Assessment as Argument
• A Layered Approach
• Design Patterns / Domain Specifications
• Present
• Hands-on Browse
• Templates
• Present
• Hands-on Browse
• Coming up
• Hands-on Create

4
Law as Argument
C
unless
A
since
W
on account of
supports
so
R
B
D
Toulmin (1958)
5
Assessment as Argument
Inferences
C

• Inferences
• Observations needed to ground them
• Situations that will evoke them
• Chain of reasoning connecting them

since
since
W
Observations
D
Situations
6
A Layered Approach

• Leverage varied expertise
• Common structures
• Well use layers to
• iterate through the assessment argument,
• using different knowledge representations,
• moving from knowledge about the domain to the
  nuts and bolts of assessment design and delivery.

7
Evidence-Centered Design Layers

• Thinking about e.g., science learning inquiry

Domain Analysis
Express content as an assessment argument
Domain Modeling
Conceptual Assessment Framework

• Technical elements of assessment design
  delivery

Assessment Implementation
Assessment Delivery
8
Design Patterns / Domain Specs

• PADI uses Design Patterns (aka Domain
  Specifications) to organize information in the
  Domain Modeling layer
• Narrative, not technical, contents
• The Design Pattern schema reflects assessment
  argument structure

• Express content as an assessment argument

Domain Modeling
9
We Live in a World of Patterns
10
Definition of Patterns
Patterns help to establish the rules or
guidelines for what something does or looks like.
A group of patterns on a similar topic is like a
recipe book.
11
Analogues

• Design Patterns in Architecture
• Design Patterns in Software Engineering
• Poltis Thirty-Six Dramatic
• Situations

12
PADI Design Patterns

• Identify in narrative form
• STUDENT MODEL Knowledge, skills, and abilities
• Observations to support inference
• Features of task situations that elicit target
  KSAs
• Related content or inquiry standards
• Do not provide a concrete design or
  implementation of an assessment task

13
(No Transcript)
14
Design Pattern Benefits

• Facilitate decision-making
• Explicate the assessment argument
• Afford flexibility
• Psychological perspectives
• Generality
• Interdependence (i.e., related patterns)
• Scale

15
Browsing PADI Design Patterns

• For starters
• BioKIDS Kids Inquiry of Diverse Species

16
BioKIDS Purpose Population

• Sequence of inquiry-based science curricula
• Systematically follow students over time
• Detroit Public Schools
• 95 Minority
• 70 Free/reduced lunch

17
BioKIDS Areas of Interest

• Main content areas Biodiversity, weather, simple
  machines
• Main inquiry reasoning areas scientific
  explanations, interpreting data, creating
  hypotheses and predictions

18
Browsing PADI Design Patterns

• For starters
• BioKIDS
• Formulating scientific explanations from evidence
  (91)

19
Mid-Flight Check

• So far so good?

20
(Back to) ECD Layers

• Thinking about science learning
• inquiry

Domain Analysis

• Express content as an assessment argument

Domain Modeling
Conceptual Assessment Framework

• Technical elements of assessment design
  delivery

Assessment Implementation
Assessment Delivery
21
PADI Task Templates

• Support the specification of technical details
• Serve as pre-blueprints abstractions of multiple
  assessment tasks
• Become task specifications when all template
  components are specified

Conceptual Assessment Framework

• Technical elements of assessment design
  delivery

Assessment Implementation
Assessment Delivery
22
Domain Modeling
23
Domain Modeling
Conceptual Assessment Framework
TEMPLATE
24
Domain Modeling
Conceptual Assessment Framework
TEMPLATE
Student Model
SM Vars
25
Domain Modeling
Conceptual Assessment Framework
TEMPLATE
Activity
Student Model
SM Vars
26
Domain Modeling
Conceptual Assessment Framework
TEMPLATE
Activity
Student Model
SM Vars
Work Products
Materials Presentation
27
Domain Modeling
Conceptual Assessment Framework
TEMPLATE
Activity
Student Model
Evaluation Procedures
Measurement Models
SM Vars
Obs Vars
Work Products
Evaluation Phases
Materials Presentation
28
Domain Modeling
Conceptual Assessment Framework
TEMPLATE
Activity
Student Model
Evaluation Procedures
Measurement Models
SM Vars
Obs Vars
Work Products
Evaluation Phases
Materials Presentation
Task Model Variables DASHED RED LINE
RUNTIME
TMV
TMV
29
Domain Modeling
Conceptual Assessment Framework
TEMPLATE
Activity
Student Model
Evaluation Procedures
Measurement Models
SM Vars
Obs Vars
Work Products
Evaluation Phases
Materials Presentation
Task Model Variables DASHED RED LINE
RUNTIME
TMV
TMV
30
Domain Modeling
Conceptual Assessment Framework
TEMPLATE
Student Model
SM Vars
Lets browse student models in the PADI Design
System
31
Browsing PADI Templates

• For starters
• BioKIDS
• BioKIDS Main Template (2497)

32
Activity 1 Defining the Assessment Purpose

• Look at existing domain specifications (design
  patterns) as examples, using PADI database
• Fill out Domain Specification Design Form and
  questions 1-4 on Assessment Template Design Form
  (both in your activity packet) for an assessment
  of your choice
• Use the PADI assessment designer to create a new
  template for your chosen assessment
• Create Domain Specification object (Design
  pattern)
• Define Student Model(s)
• Define Student Model Variable(s)