Universal Design of Assessments

1
Universal Design of Assessments
Christopher Johnstone, National Center on
Educational Outcomes
2
Goals

• Define universally designed assessments within
  the current context of large-scale assessments
  and accountability
• Identify elements of universally designed
  assessments
• Identify relevant research

3
Universal Design Research?

• Pieced together from
• Specific accommodations research
• Policy statements
• Other fields (vision, ergonomics, graphic design,
  Universal Design of architecture research)

4
Universally designed assessments

• are designed from the beginning to be accessible
  and valid for the widest range of students
• provide optimal standard assessment conditions

5
Who Benefits?

• Universal design does not apply exclusively to
  people with disabilities or limited English
  proficiency
• It applies to all individuals, with wide ranging
  characteristics

6
Think about universal design in architecture and
tool design

• Curb cuts and ramps
• Elevators that talk to you
• Door handles rather than knobs
• Special pen shapes that are easier to hold

7

Remember this? OFFICIAL BALLOT, PALM BEACH
COUNTY, FLORIDA
8
Elements of UD Assessments

• Inclusive assessment population
• Precisely defined constructs
• Accessible, non-biased items
• Amenable to accommodations

9
Elements of UD Assessments (continued)

• Simple, clear, and intuitive instructions and
  procedures
• Maximum readability and comprehensibility
• Maximum legibility

10
Preliminary Research in Universal Design

• Sample of 230 students taken from four schools in
  US Southwest.
• Two schools were town schools (pop. 20,000) and
  two were rural schools.
• Students chosen from sixth grade teams that had
  populations of students with disabilities.

11
Research Design

• Two tests were created, one from sample statewide
  test items, the other re-designed using UD
  principles.
• Each student took both tests.
• Students randomly assigned to take a particular
  test first to prevent practice effect.
• Constructs held constant for each item.

12
Community Involvement

• Advisory Board trained in principles of Universal
  Design and asked to comment / suggest
  improvements based on their perspectives.
• Team consisted of three parents of children in
  special education program (one Navajo, one
  Latina, one Anglo) and one community member with
  dyslexia.

13
Sample Item

• Ramón is building a doghouse. He wants the roof
  of the doghouse to be at an angle that is more
  than 90 but less than 110. Which angle below
  could he use for the roof?
• A. B.
• C. D.

14
Revised Item

• Which angle is more than 90 and less than 110?
• A. B.
• C. D.

15
What changed??

• Design element 2 construct more precisely
  defined.
• Design element 3 bias eliminated (dog house,
  Ramón)
• Design element 4 built in accommodations
  un-timed, students circled answer on paper, did
  not bubble
• Design element 5 simple instructions and
  procedures
• Design element 6 more comprehensible language,
  underlined key words
• Design element 7 larger font

16
Results

• Means of two tests were compared and t-tests
  performed.
• A difference of 8.16 (1.67 sig.) was found
  between means, a statistically significant
  finding.
• Effect size calculated using Cohens d. Effect
  of design .61 (or 6/10 Standard Deviation
  difference) a moderate effect

17
Why??

• Students with largest difference between two
  tests were interviewed to determine difference
  for them.
• Students noted that more direct language made it
  easier for them to understand items and
  unlimited time helped them to think better
  about items. Students also said they
  remembered content better on UD test.

18
What have we learned?

• Design matters!! How a test is designed may
  effect how a student scores on that test.
• Items that are better designed appear to aid
  students that are English Language Learners and
  with disabilities show what they know better.
• This leads to more valid assessment of
  traditionally under-performing students.

19
Simplex Sigillum VeriThe simple is the seal
of the true

• Tests that remain true to constructs, are easy to
  understand, and contain language that is
  accessible to all will give the truest readings
  of what students do and do not know. Universal
  Design does not mean dumbing down a test.

20
More information?
NCEO Resources

• Visit http//education.umn.edu/nceo or Search
  for NCEO
• Web site includes
• Topic introduction
• Frequently Asked Questions
• Online and Other Resources including synthesis
  report and policy directions for Universally
  Designed assessments.