Partnership for Accessible Reading Assessment (PARA)

1
.

• Partnership for Accessible Reading Assessment
  (PARA)
• Technical Advisory Committee Meeting
• Tuesday, June 28, 2005
• Washington, D.C.

2
Instructional Access and Assessment Obstacles for
Students with Disabilities
Sheryl Lazarus
3
Overview

• Review of state reading standards.
• How various disabilities may affect the way
  students access reading.
• Analysis of accommodations that have an impact on
  the assessment of reading.

4
What Reading Means to StatesThemes Found in
State Reading Standards
5
Research Questions

• What are the major themes in state literacy
  standards?
• How might these standards be taught or assessed
  for students with disabilities?

6
Themes

• State Academic Content Standards Define
  Reading As
• Acquisition of specific skills
• Knowledge of elements or conventions in language
• Interactive thinking activity
• Problem-solving tool
• Catalyst for personal growth

7
Common Descriptors Found in Themes
Skills Knowledge Thinking Problem Solving Personal growth
Phonics Fluency Lit. Comp. Syntax Literary Elements Expository Elements Respond Critique Infer Find Solve Organize Research Connect Enjoy Reflect
8
Research Questions

• What is the format of reading assessments
  currently used in grades 4 and 8?
• What standards are and are not assessed?

9
Methods

• Review of reading standards in grades 4 and 8,
  for 5 states.
• Review of test specifications in grades 4 and 8,
  for same 5 states.

10
Findings

• States predominately assess student reading by
  having students read passages and answer
  questions.
• Multiple choice tests require that students are
  able to decode print in order to comprehend
  passages.
• Not all standards are assessed. Research,
  interactive, and personal growth standards are
  typically not assessed in statewide assessments.

11
.

• Issues that surround the instruction and
  assessment of reading for students with
  disabilities

12
Students Receiving Special Education Services
Source Education Week Quality Counts 2004
13
Reading and Students with Visual impairments

• Most students with visual impairments are not
  blind.
• Role of tactile (Braille) and auditory methods
  of accessing text.
• Implications for reading.

14
Reading and Students who are Deaf or Hard
of Hearing

• Communication forms
• American Sign Language
• Manually Coded English
• Lip reading
• Cochlear implants
• Implications for how students access text.

15
Reading and Students with Autism

• Many students with Asperger Syndrome can decode
  words well, but may lack comprehension skills
  (Barnhill, 2004).
• Students with autism may find it difficult to
  screen out distractions.

16
Reading and Students with Specific Learning
Disabilities

• 90 of students with learning disabilities
  identify reading as their primary difficulty
  (Presidents Commission on Excellence, 2003).
• Two instructional approaches most commonly used
• Remediation instruction
• Compensatory (assistive) technology

17
Reading and Students with Mental Retardation

• Traditionally, special educators have
  de-emphasized literacy in favor of functional,
  social or motor skills.
• Today, now that students with MR have access to
  the general curriculum, students are achieving at
  much higher and more complex levels than
  expected.
• 2 broad categories to teach reading
• Traditional or direct approach
• Progressive or holistic approach

18
Reading and Students with Speech or Language
Impairments

• Since reading is a language based skill, students
  without strong language skills are at-risk.
• Improve the learning environment by
• Seating away from distractions
• Monitoring background noises
• Establishing a consistent class structure
• Speaking slowly and clearly
• Using sequential words

19
Reading and Students with Emotional or
Behavioral Disorders

• No one learning strategy will be effective for
  every student.
• 2 main instructional approaches
• Direct instruction
• Peer tutoring
• Setting in which instruction is provided will
  affect reading.

20

• What do state
• accommodations
• policies say?

21
Accommodation Read Aloud Directions
Number of States
Allowed without Restriction 35
Allowed in Certain Circumstances 10
Allowed with Implications for Scoring 6
22
Accommodation Read Aloud Directions

• Examples from State Policies
• Allowed except on the reading test
• Considered a non-standard accommodation on the
  reading test

23
Accommodation Read Aloud Questions
Number of States
Allowed without Restriction 3
Allowed in Certain Circumstances 44
Allowed with Implications for Scoring 14
24
Accommodation Sign Interpret Questions
Number of States
Allowed without Restriction 13
Allowed in Certain Circumstances 29
Allowed with Implications for Scoring 8
25
Questions

• When we take into consideration the
  characteristics of
  students with disabilities,
  what are some of the limitations of
  current large-scale
  assessments?
• What are the implications for
  this project?

26
Definition Panel and Focus Group Work
Deborah Dillon
27
Goal 1

• Formulate a definition of reading proficiency,
  analyze the definition in relation to state
  standards, obtain input, and refine the
  definition.
• Definition Panel Work Formulate a working
  definition of key terms for use in collaborative
  efforts with other funded projects.

28
Definition Panel
Leadership Team

• Group co-leaders Deborah Dillon, University of
  Minnesota and John Sabatini, ETS
• Members Ann Clapper, NCEO Laurie Cutting, ETS
  and Kennedy Krieger Institute Lee Galda and
  David OBrien, University of Minnesota other
  colleagues from the University of Minnesota

29
Membership of the Definition
Panel (DP)

• Peter Afflerbach, Professor, University of
  Maryland
• Donna Alvermann, Professor, University of Georgia
• Diane Browder, Professor, University of North
  Carolina Charlotte
• Donald Deshler, Professor and Director,
  University of Kansas Center for Research on
  Learning
• Dave Edyburn, Associate Professor, University of
  Wisconsin Milwaukee
• Russell Gersten, Director, Instructional Research
  Group
• Cay Holbrook, Associate Professor, University of
  British Columbia
• Michael Kamil, Professor, Stanford University
• Peggy McCardle, Associate Chief, Child
  Development and Behavior Branch, National
  Institute for Child Health and Human Development
  (NICHD)
• Susan Rose, Associate Professor, University of
  Minnesota
• Deborah Simmons, Professor, Texas AM University
• Dorothy Strickland, Professor, Rutgers University
• Richard Wagner, Professor and Associate Director,
  Florida Center for Reading Research, Florida
  State University
• Brenda Jane (BJ) Wiemer, Special Services
  Department Chair, Kirk Day School
• Joanna Williams, Professor, Columbia Teachers
  College

30
DP Tasks

• Prior to Face-to Face Meeting/srequest that
  panel members review selected reports and
  identify relevant research literature for a
  position paper/synthesis document developed to
  undergird a working definition of reading,
  reading proficiency, and other related terms.
• Face-to-Face Panel Meeting/sseek panel members
  consensus on a definition of reading and
  reading proficiency draft descriptive
  statements or claims about the reading
  proficiencies of typical learners in grades 4-8.

31
DP Taskscont.

• Help to draft a Position Paper or Synthesis
  Document this paper is comprised of a summary
  of panel meeting minutes, a review of current
  literature and research reports (e.g., RAND NAEP
  Reading Framework for 2009) used to ground and
  support a definition of reading and reading
  proficiency, and explanations of key terms.
• Identify issues and points in the document that
  need clarification.
• Finalize the definition based on external
  feedback from Focus Groups and other review
  processes.

32
Agenda for Definition Panel Meeting 1/29/05

• Session 1--Develop a robust definition of
  reading.
• Session 2--Discuss associated reading
  proficiencies implied by the definition for
  learners in grades 4 8.
• Session 3--Consider the definition of reading
  with respect to the full range of students.
• Session 4Identify key concepts and terms.
• Session 5--Review and provide initial feedback
  on questions for focus groups.

33
An Overview of the DP Task

• Formulate a definition of the construct of
  reading proficiency that can be used as a basis
  for research and development for accessible
  large-scale tests of reading proficiency
  (consistent with the requirements of the No Child
  Left Behind Act of 2001--NCLB) that provides

34
An Overview of the DP Taskcont.

• a) a valid measure of proficiency against
  academic standards and
• b) individual interpretive, descriptive, and
  diagnostic reports for a full range of students
  with disabilities that affect reading.

35
Session 1Guiding Questions

• Tasks
• Draft a definition of reading, possibly combining
  elements from the 5 report definitions (NRP,
  RAND, PISA, PIRLS, NAEP-2009)
• Talk about key aspects that are critical for a
  robust definition of reading proficiency.
• Identify significant issues or concerns of DP
  members
• What do you feel strongly about that must be
  included in the definition and why?
• If we include particular aspects in the
  definition, what are the implications?
• If we exclude particular aspects in the
  definition, what are the implications?

36
Session 2 Discuss associated reading
proficiencies implied by the definition for
learners in grades 4 8

• Tasks In 3 small groups of 4-5 DP members share
  ideas and draft descriptive statements or claims
  about the reading proficiencies of typical
  learners.
• Given the definition of reading that we
  generated, discuss and draft descriptive
  statements or claims about the reading
  proficiencies of typical learners in grades 4
  8. What would these students be able to do as
  readers?
• Identify significant issues/challenges that may
  require reworking of the initial definition of
  reading.
• Identify significant issues and challenges for
  the PARA and DARA development teamswhat
  innovations must occur with RD?

37
Session 3 Consider the definition of reading
with respect to the full range of students

• Tasks In 3 small groups of 4-5 DP members,
  discuss the definition of reading and
  implications for students with disabilities.
  Reconvene as a whole group discuss ideas and
  identify future work.
• Given the definition of reading we developed,
  what are the components that are needed to be
  proficient in reading and what are the
  implications for individuals with different
  disabilities?
• --At this moment, is our definition inclusive?
• --Can the definition accommodate students with
  disabilities?
• --Where are the weak spots in the definition
  that need further work?
• --What might be the implications for
  assessments?

38
Results of January 2005 DP Meeting

• Definition of Reading Reading is decoding and
  understanding written text. Decoding requires
  translating the symbols of writing systems
  (including Braille) into the spoken words they
  represent. Understanding is determined by the
  purposes for reading, the context, the nature of
  the text, and the readers strategies and
  knowledge.

39
DP Small Group Work on
the Definition

• A small group was formed to examine the January
  DP meeting notes, consider the concerns and
  comments related to disabilities, and reconsider
  the definition of reading. Resulta revised
  definition of reading and a draft definition of
  reading proficiency.
• Two other small groups were charged to work on
  4th and 8th grade proficiency statements.
  Resultgroup members were not able to complete
  this work until the definition was finalized.

40
DP Large Group Work/Email
Discussion re the Definition

• Some concerns were expressed over the revised
  definition of reading generated by the small
  group work (Definition B).
• DP members voted to return to the definition
  drafted in January 2005 (Definition A).
• The draft definition of reading proficiency
  (generated by the small group) was placed on
  hold.
• Exec. Committee of NARAP discussed issues and
  offered a third draft definition of reading
  (Definition C).

41
Focus Group Goals

• Main purpose to obtain feedback on the
  definition of reading from members of the
  reading, disability, and educational measurement
  community.
• Secondary purpose to get the word out about the
  NARAP project.

42
Focus Group Process

• Face-to-face (DARA)
• Piggyback on large conferences
• Broader constituency of educators
• Cost effective, convenient, open to all
• Web-based (PARA)
• Not tied to specific conferences
• Focus on specific disability groups
• Will be more targeted by GAC members

43
Targeted Conferences

• Learning Disabilities Association of America
  (LDA) Cancelled, PARA will do as web-based
• Council for Exceptional Children (CEC)
  April Completed
• American Educational Research Association (AERA)
  April Completed
• International Reading Association (IRA)
  May Completed
• Council of Chief State School Officers (CCSSO)
  June 20-21
• Society for the Scientific Study of Reading
  (SSSR) June 25

44
Sample for Initial
Focus Group Results

• Total - 14 sessions, 76 people
• CEC - 6 sessions, 35 people
• AERA - 3 sessions, 17 people
• IRA - 5 sessions, 24 people

45
Definition A

• Reading is decoding and understanding written
  text. Decoding requires translating the symbols
  of writing systems (including Braille) into the
  spoken words they represent. Understanding is
  determined by the purposes for reading, the
  context, the nature of the text, and the readers
  strategies and knowledge.

46
Definition B

• Reading is decoding and understanding text for
  particular reader purposes. Readers decode
  written text by translating text to speech, and
  translating directly to meaning. To understand
  written text, readers engage in constructive
  processes to make text meaningful, which is the
  end goal or product.

47
Definition C

• Reading is the process of deriving meaning from
  text. For the majority of readers, this process
  involves decoding written text. Some individuals
  require adaptations such as Braille or
  auditorization to support the decoding process.
  Understanding text is determined by the purposes
  for reading, the context, the nature of the text,
  and the readers strategies and knowledge.

48
Questionnaire/ Focus
Group Questions

• How well does the definition define reading?
• How well does the definition define reading for
  students with disabilities that affect reading?
• How well will the definition support the
  development of a fair assessment for all
  students?

49
Focus Groups
General Reaction

• Understanding should be the main focus of the
  definition.
• Decoding over-emphasized.
• not as important as understanding
• one facet of larger process
• References to spoken words and speech
  problematic for Deaf/Hearing Impaired.
• Braille as text for Blind/Visually Impaired
  acceptable concerns about auditorization.

50
Focus Groups
General Reaction- cont.

• Assorted issues
• Understanding vs. Comprehension vs.
  Constructing/Deriving Meaning
• Specific lists vs. generalities
• Text-- how broadly defined
• Reading vs. Literacy
• Disconnect between process and product for many
  disabled students
• ELL students not addressed

51
Continued Work Position Paper Paper on
Reading Proficiencies

• Work group 1 currently drafting a
  position/synthesis paper that provides a brief
  overview of our work, presents a definition of
  reading, a review of research, a discussion of
  why the definition contains particular components
  and not others, defines terms used in the
  document, and includes issues related to various
  disability groups (summaries of the PARA
  Disability Papers).
• Work group 2 currently drafting a paper that
  provides a definition of reading proficiency and
  outlines proficiencies for grades 4 and 8.
• DP reviews the two papers in August 2005.

52
TAC Discussion Questions

• 1. The Panel charged with constructing a
  definition of reading sought to draft a
  straightforward, research-based, and complete
  definition. From your perspective, what
  components of the draft definition of
  reading are critical for a definition that
  extends assessment opportunities for students
  with disabilities? 

53
Discussion Questions
cont.

• 2. How could the draft definition of reading push
  current reading assessment practices in new and
  challenging ways for all students? 

54
Discussion Questions
cont.

• 3. Will the collaborative work of the
  projects (PARA and DARA) on the definition create
  any specific advantages or problems for the PARA
  project and for producing an accessible
  assessment?

55
Partnership for Accessible Reading Assessment
(PARA)Research Plans
National Center on Educational Outcomes (NCEO)
National Center for Research on Evaluation,
Standards, and Student Testing (CRESST)
Westat
Years 2 to 4
56
Partnership for Accessible Reading Assessment
(PARA)Research Plans

• Section I Long-Term Research Plans
• Studies Included in the Proposal

57
Four Research and Development Goals Guide the
Work of the PARA Projects

1. Formulate a definition of the construct of
   reading proficiency
2. Conduct a program of research on the assessment
   of reading proficiency
3. Develop research-based principles and guidelines
   for large-scale accessible reading assessments
4. Develop and field-test instruments or methods for
   assessing reading proficiency

58
Goal 2 Research Plans

1. Which students need which kinds of assistance to
   access the type of reading assessments that
   states currently administer? (Identification and
   Sampling Issues)
2. How can a valid assessment be developed that will
   permit students with disabilities to demonstrate
   their reading proficiency? (Validity of
   Accessible Assessments)
3. What common forms of assessment are appropriate
   for group testing that are accessible to students
   with disabilities that affect reading? (Universal
   Design and Individual Student Characteristics)
4. What opportunities do students with disabilities
   that affect reading have to access reading
   through alternatives to print in the classroom?
   (Opportunity for Use of Access Tools)

59
Study 1 Sampling Issues1. Data Collection
Beginning Year 2, data will be collected on

• How the study participants interact with print
• Student statewide test results
• Information about accommodation use
• Data from student IEP

60
Study 1 Sampling Issues2. Field Study Design
As an Ex Post Facto design, this experiment
involves a Randomized Field Study with an
experimental and a control group. Experimental
group students who have difficulty
interacting with print Control group students
with no difficulty with print, matched on
gender, ethnicity, grade level and
performance Students in the experimental group
will be divided into two groups 1. Those with
disabilities 2. Those with no apparent
disabilities
61
Study 1 Sampling Issues2. Field Study
Identification of Subjects
Students in the experimental group may be
identified through the following sources to
determine disabilities that may affect
reading 1. Teacher nomination 2. Interviews
(student, teacher, and parent) 3. Analyses of
IEPs 4. Analyses of previous large-scale
assessment results Only students who know
how to use needed accommodations will be included
in the experimental group with disabilities.
62
Study 1 Sampling Issues2. Field Study
Statistical Design
Performance of students across the three groups
will be compared on following measures of
reading 1. Scores of standardized achievement
tests 2. Grade points in reading 3. Reading
testlets developed using definitions from
Goal 1 A multiple discriminant analysis (or a
multivariate analysis of variance) model will be
applied to the data from this phase. A
mean-structure multiple-group factor analysis
will also be conducted to compare the performance
of the three groups.
63
Study 2 Validity of Accessible Assessments for
Students with Disabilities
Study 2A Effectiveness of assessments Study 2B
Validity of accessible assessments Study 2C
Feasibility of accessible assessments Study 2D
Differential impact of assessments
64
Study 2A Effectiveness of Accessible Assessments
Performance of students using new testlets will
be compared with those using large-scale reading
assessments. The new testlets will be deemed
effective if the performance gap between
experimental and control is significantly less
with the new testlets.
65
Study 2B Validity of Accessible Assessments
Performance of students with and without print
difficulties will be compared. If providing
accessible assessments increases the performance
of students without print difficulty, the
validity of accessible assessment may be in
question.
66
Study 2B Validity Phase I
Qualitative phase Teacher will judge (in a
5-point Likert-scale) whether the accessibility
factor impacts the construct being measured.
67
Study 2B Validity Phase II

• Field-test phase Skilled print readers within a
  classroom will be assigned randomly to
• An experimental group where they receive the same
  access conditions that are used for students with
  print difficulty
• A control group where they receive no access
  conditions
• Student scores on standardized reading and their
  grade points in reading will be used as
  covariates.
• Skilled print readers scoring significantly
  higher under access conditions may be
  indications of validity problems.

68
Study 2C Feasibility of Accessible Assessments
The feasibility of access conditions will be
determined by interviews with

• State assessment directors
• Test publishers/Vendors
• District superintendents
• School principals
• Teachers

69
Study 2D Differential Impact of Accessible
Assessments
Will identify major student background variables
that may have potential impact on accessible
assessments. Students will be grouped based on
these variables. Performance of these subgroups
will be compared. A multiple discriminant model
will be applied.
70
Study 3 Universal Design and Individual Student
Characteristics
The purpose of this study is to find a balance
between individual students need and group
testing. Utilizing the finding of Study 2D,
variables that are found to have no impact will
collectively present justifications for
universally designed accessible assessments.
71
Study 4 Instructional Sensitivity/ Opportunity
to use Access Tools

• Reading for students with print difficulties will
  be the focus of
• Interviews with teachers and students.
• Observation during regular classroom
  instructions.
• OTL questionnaire data from teachers and
  students.
• The OTL questionnaire developed by CRESST
  researchers will provide some backgrounds.

72
Goal 4 Development and Field Test of Accessible
Reading Assessments
Based on input from the Definition Panel,
research findings, and development principles and
guidelines, a series of testlets will be prepared
and field tested. The field test includes the
following phases to be performed by Westat

1. Identify district and schools for sample, grades
   4 and 8
2. Secure District and School participation
3. Identify and train data collection staff
4. Create data system
5. Conduct field assessments
6. Clean data and create an analysis file

73
Goal 4 Development and Field Test of Accessible
Reading Assessments Validation Studies (CRESST)
Validation studies consist of the following
studies

1. Item analysis
2. Estimating reliability
3. Estimating validity
4. Scaling issues
5. Issues concerning composite scores
6. Identifying test items with substantial
   cultural/linguistic biases
7. Issues concerning standard setting

74
Goal 4 Development and Field Test of Accessible
Reading Assessments1. Item Analysis
Classical item analysis approaches will be
applied and the following statistics will be
computed

1. Item difficulty (p-value in MC) and mean in
   open-ended
2. Item standard deviation
3. Frequencies of the distractors
4. Item discrimination power (biserial/point
   biserial correlations)

75
Goal 4 Development and Field Test of Accessible
Reading Assessments2. Reliability
The following approaches for estimating the
reliability of the assessments will be used

1. Internal consistency using KR-20 and Cronbachs
   alpha
2. Principal components analysis to examine the
   dimensionality of items
3. Latent-variable modeling approach to examine the
   possibility of higher order factors
4. Test-retest approach on a subgroup of subjects

76
Goal 4 Development and Field Test of Accessible
Reading Assessments2. Estimating Validity
Multiple approaches for estimating the validity
of assessments will be utilized

1. Criterion-related approach (both concurrent and
   predictive validity
2. A multi-trait/multi method (MTMM) model within
   the structural equation framework will be applied
   to the data
3. Factorial validity approach may also be applied

77
Goal 4 Development and Field Test of Accessible
Reading Assessments2. Estimating Validity
Criterion-related
Multiple criteria will be used in the
criterion-validity approach. Criteria will be in
the form of latent rather than measured
variables. External variables will include

1. Standardized achievement test scores in
   reading/language arts
2. Grade points in reading
3. Teachers rating of student reading level

78
Goal 4 Development and Field Test of Accessible
Reading Assessments4. Scaling Issue

• Issues concerning horizontal versus vertical
  scaling will be examined
• A vertical scaling methodology will be applied
  since reading proficiency is more developmental
• Enough common sets of items for vertical scaling
  will be developed.

79
Goal 4 Development and Field Test of Accessible
Reading Assessments5. Issues Concerning
Composite Scores

• Dimensionality of the accessible reading
  instrument will be examined
• Multi-dimensional scale will be considered if the
  first principal component does not explain a
  substantial amount of the common variance
• If scales are multi-dimensional, alternatives for
  scoring and reporting will be introduced
• If a single dimension is desired, the common
  variance across all test items will be obtained

80
Goal 4 Development and Field Test of Accessible
Reading Assessments6. Identifying Test Items
with Substantial Cultural/Linguistic Biases
Possible interaction between nuisance/extraneous
variables with items will be examined. Items that
are biased will show differential levels of
performance on students with disabilities versus
those with no apparent disabilities. DIF analysis
will be done using some of the following
approaches

1. Quasi-chi-square
2. Log-linear
3. Mantel-Haenszel
4. The standardization procedure
5. The logistic regression

81
Goal 4 Development and Field Test of Accessible
Reading Assessments7. Issues Concerning
Standard Setting
A standard setting is a major component of this
test development project. Cut-scores will be
developed based on different approaches and
consistencies of the results will be
examined. The following approaches will be
considered

1. Modified Angoff
2. Book Mark
3. Contrasting Group

82
Partnership for Accessible Reading Assessment
(PARA)Research Plans

• Section II Short-Term Research Plans
• Studies to Inform Application of the Definition

83
Small-Scale Studies to Inform Application of
Definition
1. Analyses of existing data 2. Examining the
format of tests for possible changes 3. Observing
students, interviewing students, interviewing
teachers 4. Examining background variables of
students
84
Small-Scale Studies onDefinition
ApplicationStudy 1 Plan

• Item-level analyses will be performed on existing
  reading language-arts achievement tests
• Issues related to accessible reading will be
  examined
• Comparisons will be done across SWDs with and
  without print difficulties
• Comparisons will also be done across non-SWDs
  with and without print difficulty
• Different Item Functioning (DIF) analyses will
  reveal items that interact with students
  disability status (particularly those with print
  difficulties)

85
Small-Scale Studieson Definition Application
Study 1 Questions

• Can you secure a data site for us?
• Any other item-level analyses you would suggest
  that can help?

86
Small-Scale Studieson Definition Application
Study 2 Plan

• Existing test booklets will be reviewed by
  experts in SWD assessments
• Test items and test format will be examined for
  issues concerning print difficulties
• The outcome will help to provide suggestions for
  possible revisions
• The revisions will make the tests more accessible
  for students with print difficulties

87
Small-Scale Studies on Definition Application
Study 2, Questions

• What specific features of the instruments would
  you consider in this study?
• What type of instructions would you provide to
  the reviewers?

88
Small-Scale Studieson Definition Application
Study 3 Plans

• Observing instructional settings for students
  having difficulty with print help understand
  their OTL
• Think aloud sessions with students may help
  understanding issues from student perspective
• Interviews with teachers will help understanding
  students needs
• Observing non-mainstream schools (schools for the
  deaf, blind, learning disabilities provides
  insight into factors contributing to reading
  difficulties.

89
Small-Scale Studies onDefinition Application
Study 3 Questions

• What aspects of teacher/student relationships
  would you consider in this study?
• What should be the focus of observations?
• We plan to target the following subgroups of
  disabilities for Study 3 (1) students with
  visual impairments, (2) students with hearing
  impairments, and (3) students with learning
  disabilities?
• What are your thoughts regarding inclusion of
  students with learning disabilities as one of the
  subgroups in Study 3?

90
Small-Scale Studieson Definition Application
Study 4 Plans

• Understanding Student background variables will
  help with designing more accessible assessments
• Major background variables having impact on
  student performance will be identified
• Performance of students with print difficulties
  many be compared across categories of these
  background variables
• Results will help identify variables having
  impact on reading outcomes

91
Small-Scale Studieson Definition
ApplicationStudy 4, Questions

• What background variables would you suggest for
  inclusion in this study?
• How would you identify students with print
  difficulty?
• We plan to identify students having difficulty
  with print through the following sources of
  information (1) Teacher nomination (2)
  Interviews (student, teacher, and parent) (3)
  Analyses of IEPs and (4) Analyses of previous
  large-scale assessment results
• We will include only students who know how to
  use needed accommodations. Any thoughts?

92
Concluding Questions

• Do the four proposed studies help the definition
  panel?
• Do the proposed studies help understand issues
  concerning print difficulty?
• Should the small-scale studies be conducted in
  California alone?
• How large the sample (number of classes/
  students) be for each of the four proposed
  studies?

93
Assessment Options/ConsiderationsMartha Thurlow
June 28, 2005
94
Existing
Universal Design
Accommodations
Modifications
Other

• PP Formats
• Technology
• Performance
• On/Off Demand

EBD
MR
Autism
LD
Hearing
Vision
Before
During
After
95
Design Considerations (Before)
Component Based Assessment

• Separate the test into parts and assess the parts
  in different ways.

Technology Based Assessment

• Present the assessment on the computer, thus
  allowing the integration of text to speech into
  the assessment in certain parts.

96
An Example

• A reading assessment with three sections
• Phonemic awareness and vocabulary
• Passage reading that requires the student to
  read (decode or sight read)
• Text comprehension that requires the student to
  choose the modality of access (self-reading, text
  to speech, multi-modal)

97
Administration Considerations (During)
Classroom-Based Assessments

• Use structured, regularly administered
  classroom-based assessments to provide data on
  reading skills defined in specific ways that may
  not be adaptable to large-scale assessments.

Data-Collecting Reading Pens

• Use the innovative data-collecting pens to allow
  students to have test read if needed, but collect
  data on needs while getting at comprehension.

98
Scoring/Reporting Considerations (After)

• Adjust scores for use of nonstandard
  accommodations or modifications or for special
  issues (skipped items) that arise because of
  disabilities (see Decision-Making Questions).
• Consider compensatory or disjunctive rather than
  conjunctive scoring approaches separate scores
  and require students to meet certain levels on 3
  of 5, for example.

99
Discussion Questions

1. What are the implications of the various
   assessment options for technical adequacy?

100
Discussion Questions

1. Are there other important questions we should be
   asking about assessment options/considerations if
   we want to achieve accessible reading assessments?