Universal Design for Assessment: Technologys Role

1
Universal Design for Assessment Technologys
Role

• Michael Russell
• Boston College
• Thomas Hofmann
• Nimble Assessment Systems

2
Goals

• Share research on computer-based test
  accommodations
• Develop an understanding of what is a reality
  today
• Trace evolution of thinking on test
  accommodations and test validity in light of this
  reality

3
Test Accommodations

• A test accommodation is a change in the way a
  test is administered or in the way a student
  responds to test questions. Accommodations are
  intended to overcome the effects of the
  disability and provide a student with the
  opportunity to demonstrate knowledge and skills
  during classroom instruction and on statewide
  tests.
• 
  
  -MA DOE 2006

4
Accommodations Test Validity

• Decrease construct irrelevant variability
• No impact on tested construct

5
Common Test Accommodations

• Braille - 48
• Magnification - 48
• Read Aloud - 45
• Signing - 45
• Extended Time - 45
• Calculator - 41
• Auditory Calming/Noise Buffers - 32
• Word Processing - 25
• Masking - 12
• Spell-checker or assistance - 12
• Color Overlays - 11
• Thesaurus - 6

6
Current Challenges

• Alternate forms
• Managing distributing materials
• Alternate locations within schools
• Additional test proctors
• Scheduling
• Quality control
• Equity
• Monitoring and reporting use

7
Student Perspectives

• Bundling
• Group delivery
• Stigmatized
• Reluctant
• Inaccurate pronunciation and signing
• Intentional and unintentional clues
• Not provided

8
How Computers Can Help

• Standardize and Control Delivery
• Alter Presentation
• Simplify Preparations and Distribution
• Reduce Human-power
• Increase Availability
• Reduce Stigmatization
• Reduce Stimuli
• Monitor Use
• Empower Students

9
New England Compact - 2004

• Read Aloud
• Respond Aloud
• Word Processing

10
NEC Read Aloud Study

• 235 Grade 10 and 11 students
• 54 students with Language-related IEP
• 53 ELL students
• 3 conditions (Paper, Human, Computer)
• 12 item MC Math test

See Demo
11
Read Aloud Study
Students with IEP
12
Read Aloud Study
ELL Students
13
Read Aloud Study
Non-IEP/Non-ELL Students
14
NH Statewide Pilot

• Spring 2006 NH Grade 10 Mathematics Test
• Tutorial and Practice Test to all schools
• Student option to use CB
• Data from 2005 and 2006

See Demo
15
NH Statewide Pilot
16
NH Statewide Pilot
17
NH Statewide Pilot
18
NH Statewide Pilot
19
NH Statewide Pilot
20
NSF Usability Efficacy Studies

• Read Aloud
• Magnification
• Masking

See Demo
21
NSF Usability Efficacy Studies

• 54 Students
• Multiple Needs
• Dyslexia
• Reading Challenges
• Dyscalculia
• Moderate Vision
• Attention Deficit Disorders
• Hyperactivity
• Two 10 Item Math Tests ( 9 MC, 1 OE)

22
NSF Usability Key Findings

• Setting student preferences easy for all teachers
• 100 of students found tools very easy or easy to
  use
• 87 of students felt they performed better when
  using CB tools
• 79 of students wanted to use CB instead of paper
  for future tests, 14 no preference

23
NSF Usability Key Findings
24
NSF Efficacy Key Findings
25
NSF Study Suggestions

• Auditory Calming/Music
• Reading Assistant
• Choice of voice
• More calculator functions
• Signing
• Alternate Contrast options
• Easier Items!

26
Building a Universally Designed System

• Accessibility
• Tab/Enter navigation of the interface
• Intellikeys keyboard with custom keyboard
  overlays
• Read aloud of text with choice of human or
  digital reader
• Presentation of signed text in American Sign
  Language or Signed English
• Read back of open-ended responses
• Talking Calculator
• Talking formula sheets

27
Building a Universally Designed System

• Visual Perception
• Magnification of text and images for students
  with moderate visual impairments
• Magnification of text and images for students
  with low vision
• Reverse contrast with selection of contrast color
• Color overlays with selection of overlay color

28
Building a Universally Designed System

• Attention
• Masking of test items
• Masking of answers
• Auditory Calming
• Reading assistant with option of color overlay
  and/or magnification

29
Building a Universally Designed System

• Alternate Language
• Word Prediction/Writing Supports
• Dictionary/Thesaurus
• Extended Time

30
See demo
31
Challenges to Test Validity

• Flexibility at item level vs. Total test
• Multiple needs addressed simultaneously
• Student Choice
• Modal Effect
• Accommodation vs. System

32
Test Validity

• Joint Standards
• test validity is a unified concept that focuses
  on the validity of inferences made about a
  student or group of students based on a test
  score
• test validity focuses on forming an argument or
  claim regarding an inference and then collecting
  evidence from multiple sources to confirm or
  disconfirm the claim

33
Test Validity

• Four sources of validity evidence
• test content
• response process
• internal structure
• relations to other variables

34
Test Validity

• Intended Effects
• alter the way in which the student interacts with
  test items
• improve the internal characteristics of the test
• decrease the influence of non-tested constructs

35
Paper-based vs. Computer-based

• Joint Standards
• Evidence of equivalence between paper- and
  computer-based forms of a test must be provided
• Equivalence requires the same
• Means
• Standard deviations
• Rankings among individuals

36
Paper-based vs. Computer-based

• Multiple-choice Tests
• Writing Tests
• Accommodations?

37
Test Accommodations True Score Theory

• Observed Score True Score Error
• Error is assumed to be random

38
Test Accommodations True Score Theory

• Observed Score True Score Error
• For Students with disabilities and special needs
• Error Systematic Error Random Error

39
Test Accommodations True Score Theory

• Observed Score True Score Error
• Error Systematic Error Random Error

Test Accommodation(s)
40
(No Transcript)
41
Student Choice

• Not allows in students best interest
• Writing prompts
• Writing with Computers
• Use of accommodations

42
Validity Argument Foundation

• Appropriate accommodations should be provided
• Use of appropriate accommodations increases test
  validity
• Appropriate accommodations are not provided
• Students opt not to use them

43
Four Suppositions

• More students make use of CB accommodations
  during testing
• Use of CB accommodations decreases the influence
  of non-tested constructs
• The use of CB accommodations has a positive
  effect on the performance of eligible students
  but has a non-positive effect on ineligible
  students
• Use of the CB accommodations improves a tests
  psychometric properties

44
Validity Claims

• Use of a CB system to deliver a test increases
  the use of test accommodations by eligible
  students
• Provision of CB accommodations increases the
  validity of inferences about student achievement
  by decreasing the influence of non-tested
  constructs on the performance of eligible
  students which in turn improves the psychometric
  properties of the test

45
Key Research Questions

• To what extent does a students choice to use the
  UD CB System for an actual state test affect
• the extent to which students who are eligible for
  specific test accommodations opt to use those
  accommodations during the test?
• the relationship between reading ability and
  performance on a mathematics test?
• the psychometric characteristics of the test?
• To what extent does the use of accommodation
  tools result in a differential effect for
  students who are eligible to use the tools versus
  students who are not eligible?