Accommodations Research

1
Accommodations Research Universally Designed
Assessments Where Weve Been Where Were Going

• Sandy Thompson
• Christopher Johnstone
• Amanda Blount Morse
• National Center on Educational Outcomes

2
Goals

• Define and describe the purposes of test
  accommodations
• Update and summarize what we know from research
  on accommodations use
• Provide direction for future research on the
  effects of accommodations

3
Goals

• Define universally designed assessments within
  the current context of large-scale assessments
  and accountability
• Identify elements of universally designed
  assessments
• Identify relevant research

4
Educational Accountability

• Driven by federal law states have implemented
  statewide accountability systems.
• One of the most challenging tasks has been
  determining appropriate ways to include students
  with disabilities in statewide assessments.
• One of the most feasible ways to increase the
  participation of students with disabilities in
  assessments is through the use of testing
  accommodations.

5
What are Assessment Accommodations?

• Definition changes in assessment materials or
  procedures
• Purposes
• To provide students with disabilities access to
  assessments, so that more can participate.
• Allow students to demonstrate their knowledge
  and skills to be assessed rather than the
  students disability.
• To comply with federal and state regulations.

6
Types of Accommodations
Setting Study carrel Small group Individualized Presentation Repeat directions Large print edition Braille edition
Timing Extended time Frequent breaks Unlimited time Response Mark test booklet Word processor Use references
Scheduling Specific time of day Subtests in different order Across multiple days Other Test preparation Out-of-level Motivational cues
7
AccommodationsThe National Picture

• All states have accommodation policies.
• States vary in terms of which accommodations they
  consider standard and non-standard.
• Accommodation use is on the rise.
• About 50 of students with learning disabilities
  received an accommodation during testing.

8
Summary of Research on the Effects of Test
Accommodations

• The first effort to summarize all currently
  available research on the topic of assessment
  accommodations was written by Tindal Fuchs and
  published in 1999.
• The current analysis was conducted to summarize
  research on the effects of test accommodations
  published from 1999 through 2001.

9
Summary of Research on the Effects of Test
Accommodations

• Forty-six research studies were reviewed.
• All of the studies are empirical include an
  analysis of data.
• The studies have the following characteristics
• Purpose
• Types of assessment, content areas, and
  accommodations
• Participants
• Research Design
• Findings
• Limitations

10
Purpose of the Research

• Most commonly cited purposes of research
• Determine the effect of the use of accommodations
  on test scores of students with disabilities
• Investigate the effects of accommodations on test
  score validity
• Study institutional factors, teacher judgment, or
  student desirability of accommodation use
• Examine patterns of errors across items or tests

11
Types of Assessment, Content Areas, and
Accommodations

• The majority of the studies used criterion
  referenced tests.
• Mathematics was assessed in half of the studies,
  and reading/language arts was assessed in about
  one third of the studies.
• Presentation accommodations were investigated
  most frequently.
• Oral presentation" was examined in almost half
  of the studies.

12
Participants in Reviewed Research

• The number of participants ranged from 3 to
  nearly 21,000.
• The largest number of studies included elementary
  school students.
• Learning and cognitive disabilities were most
  frequently investigated among studies that
  documented participants types of disabilities.

13
Research Design

• The studies were identified as representing one
  of four group research designs, a single subject
  research design, or a non-experimental or other
  design.
• Over one third of the studies applied
  non-experimental or other designs to the study of
  accommodations effects.

14
Findings from Reviewed Research

• Three accommodations showed a positive effect on
  student test scores across at least four studies
• Computer administration
• Oral presentation
• Extended time
• However, additional studies on each of these
  accommodations also found no significant effect
  on scores or alterations in item comparability.

15
Findings from Reviewed Research

• All of the meta analyses of accommodated
  conditions found a positive effect on scores.
• All of the studies examining differential item
  functioning (DIF) found some items that exhibited
  DIF under accommodated conditions.

16
Limitations Cited in Reviewed Research

• Frequently cited limitations were
• Unknown variations among students included in the
  study
• Sample sizes too small to provide adequate
  statistical support
• Nonstandard administration of the accommodations
  across proctors and schools

17
Recommendations for Future Research

• Recommendations cited by researchers
• Replicating the research for validation and
  generalization
• Investigating associations to specific
  disabilities
• More detailed non-experimental studies be
  conducted to provide richer data
• Increase researcher control of the testing
  process
• Study larger groups of students

18
Conclusions

• Observations and recommendations
• Clear definition of the constructs tested
• Greater clarity in accommodations needed by
  individual students
• Desirability and perceived usefulness of
  accommodations by students themselves- the End
  Users
• Consider Universal Design

19
Universal Design Research?

• Pieced together from
• Specific accommodations research
• Policy statements
• Other fields (vision, ergonomics, graphic design,
  Universal Design of architecture research)

20
Universally designed assessments

• are designed from the beginning to be accessible
  and valid for the widest range of students
• provide optimal standard assessment conditions

21
Who Benefits?

• Universal design does not apply exclusively to
  people with disabilities or limited English
  proficiency
• It applies to all individuals, with wide ranging
  characteristics

22
Think About Universal Design in Architecture and
Tool Design

• Curb cuts and ramps
• Elevators that talk to you
• Door handles rather than knobs
• Special pen shapes that are easier to hold

23
Remember This? Official Ballot, Palm Beach
County, Florida
24
Elements of UD Assessments

• Inclusive assessment population
• Precisely defined constructs
• Accessible, non-biased items
• Amenable to accommodations

25
Elements of UD Assessments(continued)

• Simple, clear, and intuitive instructions and
  procedures
• Maximum readability and comprehensibility
• Maximum legibility

26
Preliminary Research in Universal Design

• Sample of 230 students taken from four schools in
  US Southwest.
• Two schools were town schools (pop. 20,000) and
  two were rural schools.
• Students chosen from sixth grade teams that had
  populations of students with disabilities.

27
Research Design

• Two tests were created, one from sample statewide
  test items, the other re-designed using UD
  principles.
• Each student took both tests.
• Students randomly assigned to take a particular
  test first to prevent practice effect.
• Constructs held constant for each item.

28
Community Involvement

• Advisory Board trained in principles of Universal
  Design and asked to comment / suggest
  improvements based on their perspectives.
• Team consisted of three parents of children in
  special education program (one Navajo, one
  Latina, one Anglo) and one community member with
  dyslexia.

29
Sample Item

• Ramón is building a doghouse. He wants the roof
  of the doghouse to be at an angle that is more
  than 90 but less than 110. Which angle below
  could he use for the roof?
• A. B.
• C. D.

30
Revised Item

• Which angle is more than 90 and less than 110?
• A. B.
• C. D.

31
What Changed??

• Design element 2 construct more precisely
  defined.
• Design element 3 bias eliminated (dog house,
  Ramón)
• Design element 4 built in accommodations
  un-timed, students circled answer on paper, did
  not bubble
• Design element 5 simple instructions and
  procedures
• Design element 6 more comprehensible language,
  underlined key words
• Design element 7 larger font

32
Results

• Means of two tests were compared and t-tests
  performed.
• A difference of 8.16 (1.67 sig.) was found
  between means, a statistically significant
  finding.
• Effect size calculated using Cohens d. Effect
  of design .061 (or 6/10 Standard Deviation
  difference) a moderate effect

33
Why?

• Students with largest difference between two
  tests were interviewed to determine difference
  for them.
• Students noted that more direct language made it
  easier for them to understand items and
  unlimited time helped them to think better
  about items. Students also said they
  remembered content better on UD test.

34
Simplex Sigillum VeriThe simple is the seal
of the true

• Tests that remain true to constructs, are easy
  to understand, and contain language that is
  accessible to all will give the truest readings
  of what students do and do not know. Universal
  Design does not mean dumbing down a test.

35
What Have We Learned?

• Design matters!! How a test is designed may
  effect how a student scores on that test.
• Items that are better designed appear to aid
  students that are English Language Learners and
  with disabilities show what they know better.
• This leads to more valid assessment of
  traditionally under-performing students.

36
Development Techniques for Universally Designed
Assessments
37
Think aloud

• Recently interviewed 90 students using think
  aloud protocol
• 4th and 8th grade
• Used multiple choice and constructed response
  items from state math test

38
Logistics

• 10 researchers
• 5 days
• Worked in pairs
• All sessions videotaped
• Primary accommodations included oral
  administration and sign language interpretation

39
Student Characteristics
Grade 4 Grade 8
Learning Disability 10 10
Deaf/Hard of Hearing 10 10
Mild Cognitive Impairment 5 5
English Language Learner 10 10
No Disability 10 10
40
Overall Observations

• Students who were confident of content did not
  have problems with design
• Students who had no idea how to solve the problem
  did not have problems with design
• Students in the middle not sure of content,
  some reading difficulty, design made a difference

41
Examples of Student Perceptions

• Many students didnt see one of the cities on a
  map
• The name of one of the cities was Independence
  - uncommon meaning
• Box between top and bottom of item some
  students did not read entire item
• Sign for parallel gave away the answer
• Some students read fraction 3 5/8 as 35 divided
  by 8
• Students unfamiliar with settings Glee club
  does number, fitness club

42
Other Observations

• Some students got a lot of help from sign
  language interpreters and teachers who wanted to
  make sure they understood the problem what
  happens on test day?
• Some student forms reported need for oral
  administration when they could clearly read the
  items independently are some accommodations
  inappropriate?
• For English language learners - is oral
  administration in English an appropriate
  accommodation on a Math test?
• Should sign language interpreters have a script
  to follow so they dont give away some answers?

43
Future Plans

• Interview students with vision impairments
• Partner with American Printing House for the
  Blind
• Explore differential item analyses
• Partner with researchers from CTB/McGraw-Hill
• Produce short awareness video
• Partner with Institute on Community Integration
  at University of Minnesota
• Design and pilot training for item reviewers and
  item developers
• Partner with Missouri Department of Education

44
More Information
NCEO Resources

• Visit http//education.umn.edu/nceo or Search
  for NCEO
• Web site includes
• Topic introduction
• Frequently Asked Questions
• Online and Other Resources