Standards, Structure,

1
Standards, Structure, Field Testing of ACCESS
for ELLs

• Jessica Motz
• Center for Applied Linguistics
• Washington, DC

January 2005 Illinois 28th Annual Statewide
Conference for Teachers Serving Linguistically
and Culturally Diverse Students
2
Outline

• Background on the WIDA Project
• WIDA Standards for English Language Proficiency
  language vs. content
• Structure of the ACCESS for ELLs Test
• Sample Items
• Pilot Test Results
• Field Testing Update
• Training for ACCESS for ELLs
• Other Issues/Questions/Discussion

3
Origins of the WIDA Consortium
2002 U.S. Dept. of Education Enhancement Grant
Competition 3 Original States
Wisconsin Delaware Arkansas
Early Additions District of Columbia Rhode
Island, Maine, New Hampshire, Vermont
Later Additions Illinois, Alabama

• Ten states, representing approx. 275,000 ELLs

4
Multiple WIDA Components

• Language Proficiency
• English Language Proficiency Standards
• Large-Scale Assessment (ACCESS for ELLs)
• Classroom Instruction Assessment
• Academic Content
• Alternate Assessment (Alternate ACCESS)
• Professional Development
• Standards
• Assessment
• Instruction
• Validation and Research

5
WIDA Partners

• State Leadership
• Tim Boals, Wisconsin DPI, Chair Steering
  Committee
• Lead Developer (Standards and Project PI)
• Margo Gottlieb, Illinois Resource Center
• Item Specification Development
• Fred Davidson, UIUC
• Test Development
• Language Testing Division, CAL
• Professional Development
• Lorraine Valdez-Pierce, George Mason University
• Technical Applications (Database, Desire2Learn)
• University of Wisconsin - Oshkosh

6
Centrality of the ELP Standards
ClassroomAssessmentFramework
Large-ScaleAssessmentFramework
English Language ProficiencyStandards
PerformanceDefinitions
ModelPerformanceIndicatorsClassroom
ModelPerformanceIndicatorsLarge-Scale
7
Overall Organization of Standards
Frameworks for Classroom Large-Scale Assessment
(2)
English Language Proficiency Standards (5)
Language Domains (4)
Grade Level Clusters (4)
Language Proficiency Levels (5)
Model PIs are the lowest level ofexpression of
the standards
Model Performance Indicators
8
The WIDA ELP Standards

• Standard 1SI
• English language learners communicate in English
  for social and instructional purposes in the
  school setting.
• Standard 2 LA
• English language learners communicate
  information, ideas and concepts necessary for
  academic success in the content area of Language
  Arts.
• Standard 3MA
• English language learners communicate
  information, ideas and concepts necessary for
  academic success in the content area of Math.
• Standard 4SC
• English language learners communicate
  information, ideas and concepts necessary for
  academic success in the content area of Science.
• Standard 5 SS
• English language learners communicate
  information, ideas and concepts necessary for
  academic success in the content area of Social
  Studies.

9
The Levels of English Language Proficiency
5
BRIDGING
4
EXPANDING
3
DEVELOPING
2
BEGINNING
1
Formerly LEP
ENTERING
6
Never LEP
7
10
Criteria for Proficiency Level Definitions
1
2
3
4
5
ENTERING
BEGINNING
DEVELOPING
EXPANDING
BRIDGING

• Comprehension and use of the technical language
  of the content areas
• Extent of language (text or discourse) control
• Development of phonological, syntactic, and
  semantic understanding or usage

11
Large-Scale Standards SC Reading
12
Large-Scale Standards SC Reading
Classify living organisms (such as birds and
mammals) by using pictures or icons
13
Large-Scale Standards SC Reading
Interpret data presented in text and tables in
scientific studies
14
Large-Scale Standards SC Reading
15
Assessment Forms

• Non-secure form for initial screening (July 1)
• One for each grade level cluster - with items at
  all 5 proficiency levels
• Kindergarten form - individually administered
• Secure forms for annual testing
• Two (initially 100 and 200) for each grade level
  cluster
• Tier A Proficiency levels 1-3
• Tier B Proficiency levels 2-4
• Tier C Proficiency levels 3-5

16
Tier Alignment with Proficiency Levels
17
WIDA Steering Committee Administration Decisions

• Listening (15) 20-25 minutes, machine scored
• Reading (35) 35-40 minutes, machine scored
• Writing (35) Up to 1 hour, rater scored
• Speaking (15) Up to 15 minutes, administrator
  scored

18
Structure of ACCESS for ELLs

• Grade Leveland Tier

K 1-2 3-5 6-8 9-12
A (adaptive) A B C A B C A B C A B C
Listening group admin, machine scored Reading
group admin, machine scored Speaking
individual admin, adaptive, TA scored Writing
group admin, rater scored
Domains
100 (roll-out Spring 2005) 999 (used to
produce screener) 200 (roll-out Spring 2006)
Forms
19
Item Creation Process (Fall 2003 present)

• Item specifications drafted
• Item writers assembled from nominated ESL
  teachers in consortium states
• Item writers trained using Blackboard
  distance-learning management software
• Item writers submitted items electronically
• Items reviewed by External Reviewers, also
  trained via Blackboard
• Items reviewed and revised internally organized
  into themes

20
Item Creation Process, continued (Fall 2003
present)

• Thematic Folders of items arranged onto pilot
  test forms
• Forms piloted in 5 WIDA districts
• Pilot analysis feedback incorporated items
  created, revised by ESL teachers
• Thematic Folders of items arranged onto field
  test forms
• Forms field tested in 8 WIDA states
• Field test analysis occurring presently
• For more information on item development process
  Jim Baumans session

21
Pilot Testing Results
22
Pilot Test Participation

• April May 2004
• 5 Districts
• Kenosha Milwaukee, WI Chicago Cicero, IL
  Washington, DC
• Approx. 1100 students in grades 1-12

23
Listening Test

• Multiple choice
• 20-25 minutes

24
Sample Items Listening, Science 1-2
25
Science Listening 1-2 P1

• PI identify objects according to chemical or
  physical properties from pictures and oral
  statements
• SCRIPT A seed is small. Find the small seed.

26
Science Listening 1-2 P2

• PI match objects with their chemical or
  physical properties from pictures and oral
  statements
• SCRIPT One day the seed will grow into
  something large, round, and heavy. Find what the
  seed grows into.

27
Science Listening 1-2 P3

• PI identify and group objects according to
  chemical or physical properties from oral
  statements
• SCRIPT Seeds grow into plants. Find something
  else that grows.

28
Percent Correct on Each Item (Grades 1-2 n 173)
29
Percent Correct on Each Item (by Tier)
30
Percent Correct on Each Item (by Grade Level)
31
Average Listening Item Difficulty Across Grade
Clusters (by Proficiency Levels)
32
Average Reading Item Difficulty by Grade Level
Cluster across P-Levels
33
Reading Test

• Multiple choice
• 35-40 minutes

34
Examining Reading Items Across a Strand for the
Different Tiers
Language Arts, Reading, Grades 3-5
35
Sample Item (Field Test) Reading, Lang. Arts,
Grades 3-5, Tier C
36
Items Tied to Performance Indicators Tier C Items

• 11 To Jessica, he was the best dog in the
  world. This sentence shows Jessicas opinion.
  Which of the following also shows an opinion?PI
  (p3) Identify language associated with stating
  opinions found in fiction or non-fiction text.
• 12 When Jessica saw the sign for the lost dog,
  why did she believe it was Blue?PI (p4)
  Differentiate between statements of fact and
  opinion found in various reading selections.
• 13 Why does the woman say at the end of the
  story, Hes your dog, all right!?PI (p5)
  Identify authors reasons or intent for selecting
  facts or opinions found in fiction or non-fiction
  from grade-level language arts text.

37
Reading Items Adapted for Tier A

• Simpler text and more graphic support
• Items at proficiency levels 1, 2, and 3
• For example
• 1. Which is Blue?PI (p1) Match labels or
  identify facts from pictures and phrases.
• 2. I know he has white spots. Which words in
  this sentence tell you it is a fact?PI (p2)
  Identify language associated with stating facts
  found in short fiction or non-fiction text
  supported by pictures or graphics
• 3. Same item from Tier C To Jessica, he was
  the best dog in the world. This sentence shows
  Jessicas opinion. Which of the following also
  shows an opinion?PI (p3) Identify language
  associated with stating opinions found in fiction
  or non-fiction text.

38
Tier Alignment with Proficiency Levels on Test
Forms (L, R, W)
39
Writing Test

• Up to 1 hour
• 4 tasks per tiered form
• SI
• MA
• SC
• LA/SS
• Model provided to give background and structure
  for the task

40
Sample Item Writing Grades 6-8
Lang. Arts Writing 6-8 P5 PI defend positions
or stances using original ideas with supporting
details
41
(No Transcript)
42
Speaking Test Adaptive Format
43
Sample Item Speaking
44
Task (Proficiency Level) 1 Example Grades 3-5
First lets talk about things people do outside.
This is a picture of people in a park. Im going
to ask you some questions about this picture. Q1
(Point to TREE) What is this? Q2 (Point to BALL)
What is this? Q3 (Point to DOG) What is
this? Q4 (If necessary) What else do you see in
this picture (OR) What other things do you see in
this picture?
SI Speaking 3-5 P1 Respond to WH- questions
45
Task (Proficiency Level) 2 Example
Now listen carefully. Ive just asked you some
questions about this picture. Now I want you to
ask me some questions about it. (OR) Pretend you
are the teacher and want to ask me some questions
about this picture. For example, you could ask
me, Where are the people? OK? Q1 (Point to BOY
ON BIKE) What do you want to know about him? (OR)
Ask me a question about him. Q2 (Point to PICNIC
TABLE) What do you want to know about this? (OR)
Ask me a question about this. Q3 What other
things do you want to know about his picture?
(OR) Whats another question you can ask me about
(anything in) this picture?(Answer students
question.)
SI Speaking 35 P2 Ask and respond to questions
46
Task (Proficiency Level) 3 Example
Now let me tell you something about these
children. (Point to CHILDREN PLAYING CATCH) Their
names are Alex and Leticia. They like to play
catch. Q1 Do you like to play catch? Q2 (If
Yes) What else do you like to do? Q3 (If No)
What do you like to do? Q4 What do you like
about __________? (OR) Tell me something about
___________. Q5 (If necessary) Tell me more.
SI Speaking 35 P3 Exchange personal information
47
Field Testing Overview
48
Field Test Participation

• Request
• Group-administered sections, 600 per form
  (minimum 400 per form)
• Individually-administered sections, 225 per form
• Includes 8 WIDA states
• Proportional representation of states (approx.
  5.5)
• Approx. 8700 students total (3500 from Illinois)

49
Initial Field Test Analyses

• Concurrent calibration of MC items (Rasch)
• Separate scale construction for each domain
• Raw score to scale score on the screener
• NOTE Steering Committee determined the following
  weights when a single level designation is
  needed
• Writing 35 (e.g 2)
• Reading 35 (e.g. 2)
• Listening 15 (e.g. 5)
• Speaking 15 (e.g. 4)
• Composite 2.75 (not yet 3)

50
Sample Feedback received to date

• Input from test administrators and coordinators
• Test Length The test is taking longer to
  administer than anticipated.
• Test Difficulty The test is more difficult than
  anticipated.
• Grades 1-2 Test Too challenging for fall first
  graders these students should take K test.
• Grades 9-12 Test There is not enough authentic
  literature on the Reading Test.
• CAL and DPI responded to these concerns via
  e-mail and the D2L (online training) discussion
  board.

51
Timeline for Development and Implementation

• Field Testing (incl. English speakers)
  September - January 2005
• Analysis of Field Test Data November 2004 -
  January 2005
• Setting Cut Scores January 2005
• Spring Roll-Out of Form 100 (AL, ME, VT)
  February - May 2005
• Practice Items for Fall Roll-out Available March
  15, 2005
• Screener Available July 2005
• Fall Roll-Out of Form 100 (IL, DC) Fall 2005

52
Training for ACCESS for ELLs
53
Training for ACCESS

• August 2004 SEA WebEx conference on logistics
  and data needed for Field Test
• September 2004 LEA Coordinators trained online
  via Desire2Learn at UW Oshkosh
• September November 2004 Field Test
  Administrators trained online via D2L
• December 2004 January 2005 Speaking Field Test
  Administrators trained in-person (DC, WI)
• March May 2005 Operational Test Administrators
  giving spring roll-out test trained

54
WIDA Field Test Administrator Workshop
55
Operational ACCESS Test Administrator Training
Possible Scenarios

• Beginning March 2005
• 2-hour total online training for all sections of
  the test
• Additional, as-needed training on scoring the
  Speaking test, with online or live
  (state/district) trained facilitators.
• For more information and updates www.wida.us

56
Other Issues/Questions/Discussion