What Are We Learning About Including All Students in Assessment and Accountability?

1
What Are We Learning About Including All Students
in Assessment and Accountability?

• Sue RigneyU.S. Department of Education
• No Child Left Behind Positives, Obstacles and
  Solutions
• University of Connecticut
• March 19, 2008

2
Assessments Shall Provide for

• Participation of all students
• Reasonable adaptations and accommodations for
  students with disabilities
• Inclusion of limited English proficient students
  with accommodations, including, if practicable,
  native-language versions of the assessment
• Assessment in English of reading/language arts
  for any student in the US for 3 consecutive
  school years
• (NCLB 1111(b)(3))

3
What Have We Learned About
meeting the assessment needs of English language
learners and students with disabilities?

• Accommodations
• Alternate assessments
• Based on alternate achievement standards
• Based on modified achievement standards
• Impact on accountability

4
LEP Students in the U.S.

• 5 Million enrolled in grades PK-12
• 4.5 million ages 3-21 speak a language other than
  English and speak English less than very well
• Most were born in the U.S.
• Elementary grade ELLs - 24 foreign-born
• Secondary grade ELLs 44 foreign born
• Most common languages Spanish (79.9),
  Vietnamese (2.0), Hmong (1.6), Cantonese
  Korean (1.0 each)
• (Source NCELA)

5
LEP Students in Connecticut

• Approximately 4.5 of total enrollment
• 148 languages other than English
• Spanish - 17,924 students
• Portuguese - 1,34 students
• Polish - 726 students
• Albanian, Chinese, Haitian, Serbo-Croation
  approx. 550 each
• Bilingual/ESL services authorized in state law
• (Source SERC)

6
(No Transcript)
7
Accommodations - LEP

• Review of lit yields mixed results
• Research mainly examines
• simplified English (some students show advantage
  for ELL)
• customized English dictionaries or glossaries
  (depends on format)
• bilingual dictionary (mixed)
• glossary
• extra time (ELL / non-ELL)
• dual-language test booklets (no advantage)
• native language tests (technical and practical
  concerns)
• Source CRESST Report 731

8
Accommodations - LEP

• Translations challenging to create
• Item translation with verification
• Is the item aligned to the same content standard?
• Does the item maintain the intended reading level
  of the item?
• Does the item maintain the intended difficulty
  level of the item by ensuring that the item was
  not simplified or clarified?
• Does the item maintain the essential meaning and
  style in translation?
• Back translation
• Are both translations accurate?
• Transadaptation (dual development)

9
SC EAGAccommodations - LEP

• Access-Based Item Development
• a carefully crafted variation of an item using
  techniques such as plain language, plain
  formatting, reduced reading load, visuals, and
  other appropriate item modifications intended to
  make test content more accessible for ELL and
  students with certain learning disabilities.
• Created by South Carolina
• Handbook on Developing Accessible Assessment
  Items for English Language Learners and Students
  with Learning Disabilities

10
SC EAGAccommodations - LEP

• Access based item development considers
• Contextual factors
• Culturally broad experiences
• Clear and explicit instructions
• Prior learning experiences
• Structural factors
• Simple language structures
• Vocabulary
• Effective visuals
• Effective item format
• Text amount
• Pattern for demonstrated response
• Impact of home language

11
Assessing Students with Disabilities
Source NCEO
12
Accommodations - SWD

• Most students with disabilities are expected to
  participate in regular statewide assessments
• without accommodations
• with appropriate accommodations that are
  consistent with accommodations provided during
  regular instruction

13
Accommodations for SWD

• Review of the literature
• Extended time ( SWD / non-SWD)
• Oral accommodation (SWD only in math)
• Equivalent test structure (supported by data)

14
Accommodations for SWD

• Steve Sireci says
• Many State accommodation policies not based on
  research
• Research is scarce
• Experimental studies recommended

15
Peer Review of State Assessment Systems Found
That

• Many states fail to monitor the delivery of
  accommodations during testing
• Many states have not validated results based on
  accommodations

16
Connecticut has.

• A comprehensive system of monitoring the use of
  accommodations in testing
• Electronic data entry system
• District Test Coordinator Training
• Annual Accommodation training statewide
• State mandated IEP forms
• Irregularities file maintained by CSDE
• IDEA focused monitoring provides follow-up

17
Connecticut also has.

• Agreed to serve as the lead in a 10-state
  consortium to investigate the validity of test
  scores based on accommodations
• 10 states 10 accommodations
• Counterbalanced design special education and
  general education students
• Results to be disseminated nationally

18
Alternate Assessments Should Have

• Clearly defined structure
• Guidelines for which students may participate
• Clearly defined scoring criteria and procedures
• Report format that clearly communicates student
  performance in terms of the academic achievement
  standards defined by the State

19
States May Use More Than One Alternate Assessment

• Alternate assessment based on grade-level
  grade-level standards
• Alternate assessment based on alternate
  achievement standards appropriate only for
  students with most significant cognitive
  disabilities
• Alternate assessment based on modified
  achievement standards

20
AA-AAS

• Communication skills of students with the most
  significant cognitive disabilities
• What proportion of this population does not use
  symbolic language?
• A. 10
• B. 30
• C. 50
• D. 90

21
NAAC 4 States Reported
22
NAAC Learner Characteristics Inventory

• Represent 1 or less of the total assessed
  population
• All disability categories were represented but
  primarily 3 emerge,
• Mental Retardation
• Multiple Disabilities
• Autism
• Highly varied levels of expressive/receptive
  language use
• Most students in the population use symbolic
  communication
• Level of symbolic language distribution is
  similar across grade-bands
• Only about 50 of the pre and emerging symbolic
  language users use ACS
• Pre-symbolic expressive language users are more
  likely to have additional complex
  characteristics.
• Most of the population read basic sight words and
  solve simple math problems with a calculator.
• Lack of skill progression in reading across grade
  bands (elementary, middle high)
• Skill progression apparent in mathematics across
  grade bands but still small

23
The LCI in Connecticut

• Accommodations for CMT or CAPT/ Learner
  Characteristics
• Inventory
• Students who utilize accommodations for the
  standard Connecticut Mastery Test (CMT) or
  Connecticut Academic Performance Test (CAPT) as
  indicated on their Individual Education Program
  (IEP) must have this information entered on the
  accommodations website www.cttestaccommodations.n
  et as soon as possible. This data is collected by
  our test vendor, Measurement Incorporated, to
  provide appropriate materials necessary for test
  administration. Students who will be assessed
  utilizing the CMT or CAPT Skills Checklist must
  have their Learner Characteristics Inventory
  entered on the same accommodations website. These
  data will register the student to take the Skills
  Checklist creating a file on the Skills Checklist
  website for March.

24
AA-MAS

• Definition A modified achievement standard is an
  expectation of performance that is challenging
  for eligible students, but may be less difficult
  than a grade-level academic achievement standard
• Emerging research results
• RI EAG
• GA data

25
RI EAG - Reaching Students in the Gap Reaching
Students in the Gap
Who are the students in the gap? Of all students
who are not proficient, how can states identify
those who are in the assessment gap? What are the
attributes of students in the gap and how do
these students perform? What issues in the
assessments themselves contribute to the
gaps? Are there specific aspects of multiple
choice items used in state assessments that
contribute to the assessment gap?
26
RI EAG - Reaching Students in the Gap Who are the
students?
Gap 1 students appear to be proficient in class
but not proficient on test Gap 2 students far
below grade level in class and very low scores on
assessment Could not distinguish between
assessment gap and instruction gap, nor the
effects of teacher expectations and content
coverage. Full results at www.necompact.org
27
RI EAG - Reaching Students in the Gap Gap 1

• Perform at a proficient level in the classroom
  but fall below proficient on eighth grade
  mathematics test
• Majority are in general education
• Teachers puzzled by test results
• Most do not receive any accommodations on the
  test
• Generally taught at grade level
• Students with IEPs were under-represented in gap
  1 and over-represented in non-gap 1

28
RI EAG - Reaching Students in the Gap Gap 2
(perform well below grade level in the classroom
and on the test)

• Majority are SwDs, ELLs, and SwD/ELLs
• Test scores fall into the lowest achievement
  level
• Teachers not surprised by the test results
• Most common accommodations
• -alternative settings, especially in small groups
  with someone other than the general education
  teacher
• -scheduling changes, including extended time and
  breaks
• -presentation/modality adaptations, such as
  reading test and/or directions aloud
• Generally taught below grade level

29
RI EAG - Reaching Students in the GapThe Item
Modification Study (grade 8 mathematics)

• Purpose
• Assess the quality and usefulness of items
  designed to decompose skills/knowledge required
  to solve complex problems.
• Examine the extent to which students who perform
  well on the complex items also perform well on
  the decomposed items.
• Examine the extent to which students in the gap
  are able to succeed on decomposed items while
  struggling with the complex items.

30
RI EAG - Reaching Students in the Gap Item
Modification Results

• Effective
• Using whole numbers
• Using whole numbers removing the context
• Simplifying information in the table (for Gap 1)
• Not effective
• Changing the table format from vertical to
  horizontal
• Removing the context
• Simplifying information in the table (for Gap 2)
• Removing the context changing the numerical
  sequence from decreasing to increasing

31
Georgia EAGAssessing One and All A Partnership
for Success

• Who are the students?
• Persistently low performing
• - lowest performance level for 3 years
• - grade 5 cohort grade 8 cohort
• Approximately 2 consistently low in both reading
  and math

32
Georgia EAGCharacteristics of Persistently
Low-Performing Students

• When compared to baseline (all students)
• More males 50 vs 60-65
• More black students 40 vs 60-65
• More free/reduced lunch 50 vs 75-80
• More students with disabilities 15 vs 40-55
• More students with mild intellectual
  disabilities 10 vs 20-30

33
Georgia EAGMore Demographic Data

• A higher proportion of ELL students are
  persistently low performing in reading (compared
  to the baseline)
• Approximately 2 of students at each grade were
  identified as persistently low performing in both
  content areas (Reading and Mathematics)
• In grade 8, 77 of the students identified in
  Reading were also identified in Mathematics

34
Georgia EAG Next steps

• Are these students receiving instruction on
  grade-level standards?
• Do these students perform differently in specific
  test items? Strands? Content standards?

35
AA-MAS in Connecticut

• Development of New Modified Assessment
• The United States Department of Education has
  provided the opportunity for states to develop a
  new assessment for special education students
  whose disability has precluded them from
  achieving grade-level proficiency and whose
  progress is such that they will not reach
  grade-level proficiency in the same time frame as
  other students. Connecticuts Bureau of Student
  Assessment has been working closely with the
  Bureau of Special Education to develop this new
  alternate assessment. Special education directors
  have been receiving correspondence about our
  progress and will be providing valuable
  information about their districts to help refine
  eligibility criteria and guide us in assessment
  development.
• (Source The Bureau Blog, January 2008)

36
Students Impact on Accountability (SINI)
37
Surprising Results

• Chancellor Klein Salutes English Language
  Learners for Academic Gains
• Date 06/19/2007 Last Modified 6/19/2007
  105334 PM Press ID N-61, 2006-2007
•     Schools Chancellor Joel I. Klein today
  congratulated English Language Learners (ELLs)
  for their progress in reading and math during a
  visit to PS 149 in Queens. ELLs at PS 149 and
  across the City made significant gains on State
  Math and English Language Arts (ELA) exams this
  year, building on progress made since the
  beginning of the Children First school reforms.
  The ELA gains are especially significant because
  a new state regulation, prompted by the federal
  No Child Left Behind Act, required ELLs with more
  than one year in the school system to take the
  test previously, ELLs were exempt until they had
  spent at least three years in the system. As a
  result, more than twice as many ELLs took the
  exam this year citywide, and 16 met standards, a
  5.3 percentage point increase over 2006, and a
  12.1 point increase since 2003. In math, 45.1 of
  ELL students are now proficient, up from 35.8 in
  2006 and a 28.4 point increase since 2003.

38
CT successes and challenges
39
CT successes and challenges
40
CT successes and challenges
41
Data Driven Actions District level

• Align curriculum and instruction
• Increase professional development opportunities
• Provide targeted technical assistance
• Provide curriculum and instructional resources
• Promote and support collaborative teams of
  content and student learning specialists
• Provide student specific resources e.g.,
  communication systems

Source NAAC 2008
42
Data Driven Actions School level

• Insure that IEP teams are assigning students to
  the appropriate assessment
• Align curriculum and Instruction
• Provide professional development opportunities
• Promote collaborative content and student
  specialist teams
• Request technical assistance
• Insure instructional material and environment
  accessibility

Source NAAC 2008
43
Data Driven Actions Student level

• Apply participation criteria appropriately
• Provide appropriate interpretations of results to
  parents
• Use assessment data in conjunction with other
  student data to make decisions
• Insure all students have a way to communicate
• Include academic content on communication systems
• Identify individualized curriculum and
  instructional strategies and supports
• Implement instructional data-based
  decision-making

Source NAAC 2008
44
Our Shared Goal

• to ensure that all children have a fair,
  equal, and significant opportunity to obtain a
  high-quality education and reach, at a minimum,
  proficiency on challenging State academic
  achievement standards and state academic
  assessments
• No Child Left Behind