Design Experiment on Reading Acquisition

1
Design Experiment on Reading Acquisition

• Calfee, Trainin, Norman
• Presented at the American Educational Research
  Association Annual Meeting Seattle 2001

2
WordWork Program Features

• Lessons fit a thin "slice of time
• Instruction focuses on full engagement through
  reflecting, and explaining.
• Connects metacognition to language learning
• A blend of inquiry and explicit instruction
• Review lessons build connections across
  interruptions

3
Design Experiments

• The rigor of systematic experimentation combine
  with the realities of the classroom
• Key Components
• Realistic Environment
• Systematic Variations
• Systematic Observations
• Assessment
• Flexibility

4
Realistic Environment Year 1

• Participants
• 260 First Grade Students
• Large Urban School District
• First wave of accountability
• 90 of students should perform above the 50th
  percentile
• Four Schools 2 year round schedules, 2
  traditional
• volunteer teachers
• 17 classrooms
• 2 mixed age k-1, 1 mixed age 1st 2nd, 1 ELL, 1
  k-3 SDC
• Disrtict wide adoption of new curriculum
  (Scholastic), one school just abandoned OPEN
  COURT
• Ethnicity 45 Caucasian, 42 Hispanic, 8
  African American, 5 other 51 males

5
Schools and SES, do they impact scores at the
beginning of first grade?

• Tile test
• Range 0-140
• Includes
• Decoding
• Spelling of synthetic words
• Real word reading
• Sentence reading
• Sentence construction

6
Design Experiments
Systematic variations

• Key Components
• Realistic Environment
• Systematic Variations
• Systematic Observations
• Assessment
• Flexibility

• Contrast between and within classrooms
• Differences large enough to show above natural
  variation and measurement noise
• Turning teachers into a planning element
• Reformulating questions between years

7
Designed Variations 1999
8
WordWork 2000 design
9
WordWork 2001 Design Transforming Teacher
Practice

• 4 Kindergartens
• 2 first grades
• Within class variation
• Fostering independence and cooperative learning
• homogenous / heterogeneous groups
• Integration with overall curriculum
• Applying assessment to curriculum design
• Extending to parents, faculty and administration
• Best practice curriculum

10
Design Experiments
Systematic Observations

• The rigor of systematic experimentation combine
  with the realities of the classroom
• Key Components
• Realistic Environment
• Systematic Variations
• Systematic Observations
• Assessment
• Flexibility

• Weekly observations in classrooms
• Structured observations (RAMOS-R)
• Video visits
• Biweekly/ monthly site meeting
• Peer observations
• Teaching logs and reflection

11
WordWork in One Teachers Schedule

• WordWork lesson 40 minutes a day
• In conjunction with Zoo Phonics
• Scholastic 1st grade program
• Repeated reading
• Interactive Writing
• Assessment Schedule
• SAT9
• District Writning
• District Reading
• District Running Records
• Practice and adminstration 3-4 weeks each time

12
Natural Variation Example

• Jane four years teaching experience, felt that
  she had a particularly tough class to teach
• Dropped out of the program by block 2 and then
  decided to come back
• Planned curriculum variations (5
  blocks)Articulation, Articulation, Augmented
  Word Building, Long Vowel Simultaneous, Ending
  System Simultaneous

Metaphonics
Planned Time
Articulation
Actual Time
13
WordWork 2000 Results
D C B A
14
Extending WordWork into second grade

• Two second grade classrooms
• Teacher A followed the program, while teacher B
  omitted significant portions
• Range is 0-5 on each. Students in second grade
  still have a problem decoding complex words

Decoding hin, flass, vute, lodded, wembick Score
range 0-5 Spelling leb, plat, mape, pridder,
radmin Score range 0-5
Decoding
Spelling
15
Design Experiments

• The rigor of systematic experimentation combine
  with the realities of the classroom
• Key Components
• Realistic Environment
• Systematic Variations
• Systematic Observations
• Assessment
• Flexibility

Assessment
16
Assessment

• Multi measures on pre and post assessment
• Decoding
• spelling
• reading fluency and accuracy
• sentence reading
• Research independent measures (state and
  district)
• Reading Running Records
• Writing
• Reading
• SAT9

• Block assessment
• Decoding, spelling, sentence reading,
  articulation and metaphonics
• designed to enable longitudinal tracking,block by
  block information,
• fast administration and turn over to teachers and
  researchers

17
Word Work 1999 results IRAS- sentence reading
N109 not all students were available for both
pre and post tests
8 - Slowly the women ascended the steep and icy
mountain. 6 - About three miles from the harbor
Rays boat was caught in an unexpected
current. 4 - The kitten was scared and climbed
up the tree. The girl tried to reach it but she
had no luck. 2 - Ann wants Mom to bake a cake.
Mom cannot do it, she has to go to work.
Post
Pre
18
WordWork 2000 Results Writing
District Writing Rubric
Mean4.60
Mean4.25
19
Sat9 Reading by Classroom
District Mean
Class Means
School Means
Program Fidelity 3high 1low
School A
School D
School C
School B
20
Design Experiments
Flexibility

• The rigor of systematic experimentation combine
  with the realities of the classroom
• Key Components
• Realistic Environment
• Systematic Variations
• Systematic Observations
• Assessment
• Flexibility

21
Lost sheep Where do they come from?

• Tile test
• Range 0-120
• Includes
• Decoding
• Spelling of synthetic words
• Real word reading
• Sentence reading
• Sentence construction

60 of the low achievers joined the normative
group
Lost Sheep n 18 40 of Low achievers at the
beginning
22
The Lost SheepFlexibility and Real-time
Exploration in Design Experiments

• 18 students
• Small group tutoring
• WordWork
• Read-Write Connection
• 10 weeks, twice weekly, 30 min. sessions

23
Lost Sheep Individual Results
8 ? Slowly the women ascended the steep and icy
mountain. 6 ? About three miles from the harbor
Rays boat was caught in an unexpected
current. 4 ? The kitten was scared and climbed
up the tree. The girl tried to reach it but she
had no luck. 2 ? Ann wants Mom to bake a cake.
Mom cannot do it, she has to go to work.
24
Students talk about themselves before tutoring
after

• I know the letters.

• I can only read one or two of the words.
• I dont know any of those words.
• I have trouble reading the words because Im not
  smart.
• When I see a word I dont know, I just dont try
  to figure it out.
• When I cant read a word, its because no one
  helped me to read it.

• Im learning.

• Ill sound them out.

• Ill keep working until Im done.

• Im going to say the word in my head.

25
Traditional View of ScalabilityData is only an
illustration
Ready for practitioner consumption
Larger scale replication
Highly monitored case study
Numbers reflect of students
26
Reverse Scalability in the WordWork Design
Experiment
Numbers reflect of students
27
Reverse Scalability in the WordWork Design
Experiment
Numbers reflect of students
28
What have we learned?

• It helps to have a design
• Monitor natural variations or risk getting the
  story wrong
• Identify what works as an interaction between
  curriculum, student ability teacher practice
• Flexibility increases what can be learned