Advances in Monitoring Progress in Writing

1
Advances in Monitoring Progress in Writing

• Kristen L. McMaster Heather Campbell
• University of Minnesota
• Presented at the
• Progress Monitoring Conference
• Minneapolis, MN
• May 5, 2006

2
Work Described Today is Supported By

• Research Institute on Progress Monitoring (RIPM)
  (Grant No. H324H030003) awarded by the Office of
  Special Education Programs (OSEP) to the
  Institute on Community Integration in
  collaboration with the Department of Educational
  Psychology, College of Education and Human
  Development, University of Minnesota

3
Special Thanks To

• Stan Deno
• Chris Espin
• Teri Wallace
• Sue Rose
• Sally Du
• Kathy Gustafson
• Anna-Lind Petursdottir
• Amanda Anderson

4
Organizer

• Background Progress Monitoring Research in
  Written Expression
• Recent Work
• How to Implement Progress Monitoring in Written
  Expression

5
Criteria for Curriculum-Based Measures

• General Standard Decision Validity
• Can the measure be used for the decision to be
  made?

6
Criteria for Curriculum-Based Measures

• Specific standards
• Technical adequacy
• Reliable
• Valid
• Practicality
• Easy to use
• Brief
• Easy to understand
• Easy to teach
• Repeatable
• Durability
• Appropriate for range of age/skill levels

7
Background CBM Research in Written Expression

• 1970s-80s
• University of Minnesota Institute for Research on
  Learning Disabilities (IRLD)
• Elementary
• 1990s to present
• Extensions of IRLD work
• Elementary, Middle, Secondary

8
IRLD What Measures?

• Tasks
• Story starters (narrative prompts)
• Scoring Procedures
• Words written (WW)
• Words spelled correctly (WSC)
• Correct word sequences (CWS)
• Duration
• 3 to 5 min

Deno et al., 1980 Deno, Marston, Mirkin, Lowry
et al., 1982 Fuchs et al., 1982 Marston Deno,
1981 Marston et al., 1981 Shinn et al., 1982
Tindal, Marston, Deno, 1983 Tindal, Germann,
Deno, 1983 Videen et al., 1982
9
Extensions of IRLD Work

• Elementary level
• Narrative prompts, WW or WSC in 3 min
• Weaker reliability (rs
• Weaker validity (rs
• Other scoring procedures
• Punctuation, words in complete sentences, simple
  sentences (rs
• Production independent measures (WSC, CWS)

Gansle, Noell, VanDerHeyden, Naquin, Slider,
(2002) Gansle, Noell, VanDerHeyden, Slider,
Hoffpauir, Whitmarsh, et al. (2004) Tindal and
Parker (1991) Parker, Tindal, Hasbrouk (1991)
10
Extensions of IRLD Work

• Secondary Level
• Correct Minus Incorrect Word Sequences
• More viable measure for detecting growth
• Works well for narrative and expository text
• Reliability/validity increase with duration of
  sample (up to 7 min)

Espin, Scierka, Skare, Halverson (1999) Espin,
Shin, Deno, Skare, Robinson, Benner (2000)
Espin and Wallace et al. (2005) Malecki Jewell
(2003) Jewell Malecki (2005) Tindal Parker
(1989)
11
Criteria for Curriculum-Based Measures

• Specific standards
• Technical adequacy
• Reliable
• Valid
• Practicality
• Easy to use
• Brief
• Easy to understand
• Easy to teach
• Repeatable
• Durability
• Appropriate for range of age/skill levels

12
Research Institute on Progress Monitoring

• Year 1 Technical Adequacy Studies in Reading,
  Math, and Written Expression
• Across Age and Skill Levels

13
Advances in Writing ResearchPurpose

• Replicate and extend research in written
  expression both within and across elementary and
  secondary levels
• Task
• Scoring procedures
• Duration

14
Advances in Writing ResearchImportant
Considerations

• Task
• Narrative vs. expository matters?
• No direct comparison has been made of these
  measures across grade levels
• Different expectations at different grade levels
• Many low-performing writers struggle to adjust
  their writing to these two different approaches
• Compare narrative vs. expository

(e.g., Deshler, Ellis, Lenz, 1996 Englert,
Raphael, Fear, Anderson, 1989 Englert,
Stewart, Hiebert, 1988 Miller
Lignugaris-Kraft, 2002 Taylor, 1987 Wong, Wong,
Blenkinsop, 1989)
15
Advances in Writing Research Important
Considerations

• Task
• Prompts relevant to todays students background
  knowledge and experience?
• Tell a story about the night you were camping in
  the woods and you heard strange noises
• It was the night before Halloween, and the
  students planned to
• I stepped into a time machine and
• Make prompts more universal

16
Advances in Writing Research Important
Considerations

• Task
• Are there other ways to obtain simple, efficient
  measures of written expression ?
• Transcription skills are strongly related to
  overall writing skill
• Copying tasks have been shown to be sensitive to
  differences in student handwriting performance
  across 1st through 9th grades
• Include passage copying measure

(e.g., Berninger, Yates, Cartwritght, Rutberg,
Remy, Abbott, 1992 Graham, 1990 Graham,
Berninger, Abbott, Abbott, Whitaker, 1997
Graham, Berninger, Weintraub, Schafer, 1998
Graham, Weintraub, Berninger, 1998 Jones
Christenson, 1999
17
Advances in Writing ResearchImportant
Considerations

• Task
• Prompts extendable to different student
  populations (e.g., ELL, D/HH, students with
  significant cognitive disabilities)?
• Make prompts more universal
• Include passage copying measure
• Include picture prompts

18
Advances in Writing ResearchResearch Questions

• Which measures of writing performance
  successfully discriminate among students at
  different grade levels?
• Which measures yield sufficient alternate form
  reliability coefficients for students at grades
  3, 5, 7?
• Which measures yield sufficient criterion
  validity coefficients for students at grades 3,
  5, 7?
• Which measures of writing performance show growth
  from fall to spring?

19
Setting and Participants

• Urban K-8 school in Minneapolis
• 28 third-graders
• 52 fifth-graders
• 64 seventh-graders

20
Independent Variables

• Tasks
• Passage Copying
• Writing Prompts
• Picture
• Narrative
• Expository

• Scoring Procedures
• (a variety)
• Time
• 3, 5, and 7 minutes

21
Independent Variables

• Tasks
• Passage copying (easy vs. hard)

One wall was of marigolds, one of daffodils, one
of pink roses, and one of cherry blossoms. The
ceiling was made of sunflowers, and the floor was
a carpet of daisies. They had water lilies in
their bathtub, and buttercups in their
refrigerator. It was a rather fragile house and
it swayed in the wind, but it was very beautiful.
Next day the big bad pig came prowling down the
road and saw the house of flowers that the three
little wolves had built. He rang the bluebell at
the door.
Johns car was low on fuel. He stopped his car
and asked a pedestrian for directions to the
nearest gas station. Here are the directions he
received Make a right at Main Street. Go past
three traffic lights to the corner of First
Street and Main Street, where there is a fire
station on the left. Make a right immediately
after the fire station. Go one block until you
reach a stop sign. In the next block, on your
right, you will see a gas station that is open 24
hours a day.
22
(No Transcript)
23
Independent Variables

• Tasks
• Writing prompts (picture vs. narrative vs.
  expository)
• As free as possible of cultural bias
• Simple vocabulary and sentence structure
• Appropriate a wide range of ages and skill levels
• Control for background knowledge

24
Independent Variables
Tasks Writing Prompts Set 1

• Picture

• Narrative
• On my way home from school, a very exciting thing
  happened.
• Expository
• Write about a trip you would like to take with
  the students in your class.

25
Independent Variables
Tasks Writing Prompts Set 2

• Picture

• Narrative
• One day, we were playing outside the school and.
  
• Expository
• Write about a game that you would like to play.

26
Sample Picture Prompt
27
Sample Narrative Prompt
28
Sample Expository Prompt
29
Independent Variables

• Scoring Procedures
• Words written (WW)
• Legible words (LW)
• Correct words (CW)
• Correct word sequences (CWS)
• Correct minus incorrect word sequences (CIWS)
• Percent correct words (CW)
• Percent correct word sequences (CWS)

30
Independent Variables

• Time
• 3 min
• 5 min
• 7 min (7th-graders only)

31
Criterion Variables

• Test of Written Language-3 (TOWL-3)
• Minnesota Test of Basic Skills Writing Subtest
  (Grade 5 only)

32
Procedures

• Data collection
• Alternate forms (2)
• 2 consecutive weeks in Fall and Spring
• Criterion measures administered by research team
  (TOWL) and collected from district scores (MCA)
• Analyses
• Grade level differences
• Alternate-form reliability
• Criterion-related validity
• Growth from fall to spring

33
Results Research Question 1

• Which measures of writing performance
  successfully discriminate among students at
  different grade levels?

34
Easy Passage Copying Grade Level Means
35
Hard Passage Copying Grade Level Means
36
Picture (5 min) Grade Level Means
37
Narrative (3 min) Grade Level Means
38
Narrative (5 min) Grade Level Means
39
Expository (3 min) Grade Level Means
40
Expository (5 min) Grade Level Means
41
RQ1. Which measures of writing performance
successfully discriminate among students at
different grade levels?

• Reliable differences across grade levels
• Grade 3
• Same patterns observed for
• Task (copying, picture, narrative, expository)
• Scoring Procedure (WW, CW, CWS, CIWS)
• Duration (3 min, 5 min)

42
Results Research Question 2

• Which measures yield sufficient alternate form
  reliability coefficients for students at grades
  3, 5, 7?

43
Alternate Form Reliabilityr .70 r .80
44
RQ2. Which measures yield sufficient alternate
form reliability coefficients for students at
grades 3, 5, and 7?

• Third grade
• Easy and hard passage copying
• Picture prompt - 5 minutes
• Narrative prompts- 3 5 minutes
• Fifth grade
• Easy and hard passage copying
• Picture prompt - 3 5 minutes
• Narrative expository prompts - 5 minutes
• Seventh grade
• Picture prompt - 7 minutes
• Narrative prompt - 7 minutes
• Expository prompt - 5 7minutes

45
Results Research Question 3

• Of the measures that yield sufficiently reliable
  data, which measures yield sufficient criterion
  validity coefficients for students at grades 3,
  5, 7?

46
Predictive Validity (3rd-grade) TOWL 3
Spontaneous Sample
47
Predictive Validity (5th-grade) TOWL 3
Spontaneous Sample
48
Predictive Validity (5th-grade) State Standards
Test in Writing Percentile
49
Predictive Validity (7th-grade) TOWL 3
Spontaneous Sample
50
RQ3. Of the measures that yield sufficiently
reliable data, which measures yield sufficient
criterion validity coefficients for students at
grades 3, 5, 7?

• Third grade
• CWS, CIWS
• Hard passage copying
• Picture prompt - 5 minutes
• Narrative prompts- 3 5 minutes
• Fifth grade
• CIWS, CW, CWS
• Narrative expository prompts - 5 minutes
• Seventh grade
• CIWS
• Picture prompt - 7 minutes
• Narrative prompt - 7 minutes
• Expository prompt - 5 7minutes

51
Results Research Question 4

• Of the measures that yield sufficiently reliable
  and valid data, which measures of writing
  performance show growth from fall to spring?

52
Fall to Spring Growth (3rd Grade) Picture (CIWS
in 5 min)
53
Fall to Spring Growth Narrative (3rd Grade
Only CWS CIWS in 3 min)
54
Fall to Spring Growth Narrative (3rd Grade
Only, CWS in 5 min)
55
Fall to Spring Growth Narrative (3rd 5th
Grades, CIWS in 5 min)
56
Fall to Spring Growth Narrative (7th Grade
Only, CWS CIWS in 7 min)
57
Fall to Spring Growth Expository CIWS(5th
7th Grade, CIWS in 5 min)
58
RQ4. Of the measures that yield sufficiently
reliable and valid data, which measures of
writing performance show growth from fall to
spring?

• Third grade
• Picture prompt CIWS in 5 minutes
• Narrative prompts - CWS, CIWS in 3 5 minutes
• Fifth grade
• Narrative expository prompts - CIWS in 5
  minutes
• Seventh grade
• Narrative prompt - CIWS in 7 minutes
• Expository prompt - CIWS in 5 7minutes

59
Conclusions

• Across grade levels
• Task Narrative
• Scoring procedure CIWS
• Duration 5-7 min
• Within grade levels
• Task Genre may change with grade level
• Scoring Complexity should increase with grade
  level
• Duration Time should increase with grade level

60
Criteria for Curriculum-Based Measures

• Specific standards
• Technical adequacy
• Reliable
• Valid
• Practicality
• Easy to use
• Brief
• Easy to understand
• Easy to teach
• Repeatable
• Durability
• Appropriate for range of age/skill levels

61
Conclusions

• Questions still remain
• What is the best measure?
• Complexity of writing
• Some promising findings for estimating
  performance level/screening
• Not yet ready for high-stakes decision-making

62
Next Steps

• Can technical adequacy be improved?
• Aggregating scores?
• New scoring procedures?
• New measures?
• Can measures be used to monitor progress over
  time?
• Standard error of estimate around slopes?
• Can measures be used for
• Instructional decision-making
• Placement programmatic decision-making?

63
How to Implement Progress Monitoring in Written
Expression
64
Materials

• Material selection
• Select story starters to which students are
  likely to be able to produce 5-7 min of writing
• End story starter in mid-sentence
• Probe development
• Write story starter on paper or present on board
  or overhead

65
Sample Story Starters
66
Administration Procedures

• Read story starter to students
• Give students 30 seconds to think
• Reread story starter
• Give students 5 to 7 minutes to write (choose 5
  or 7 and stay consistent all year)

67
Scoring Procedures

• Words written
• Count number of word units
• Ignore spelling and usage
• Ignore capitalization and punctuation
• Words spelled correctly
• Scores as a computer would
• Count any correctly spelled English word
• Ignore usage
• Ignore capitalization and punctuation

68
Scoring Practice Words Written and Words
Spelled Correctly

• The cave was very dark. I try to close my eyes,
  so I couldnt see anything, but that didnt help.
  Than I hear some one breathing. I try to
  stream, but nother came out. The breathing
  became close and close to me, and the worst Part
  was that I couldnt see athing. At first I
  thought meself that I an Just emaging stuff.

Words spelled correctly 54
Words written 58
69
Scoring Procedures Correct Word Sequences and
Correct Minus Incorrect Word Sequences

• Read entire sample before beginning to score.
• Place a vertical line at the place where a
  sentence should end.
• You may have to judge where the sentence should
  end.
• Underline incorrect words
• (spelled incorrectly, are grammatically incorrect
  or used incorrectly).

70
Scoring Practice
The cave was very dark. I try to close my
eyes, so I couldnt see anything, but that didnt
help. Than I hear some one breathing. I try to
stream, but nother came out. The breathing
became close and close to me, and the worst Part
was that I couldnt see athing. At first I
thought meself that I an Just emaging stuff.
71
Scoring Procedures CWS and CIWS

• Score the passage for correct and incorrect word
  sequences
• A correct word sequence is one that contains any
  two adjacent, correctly spelled words that are
  acceptable within the context of the sample to a
  native speaker of the English language.
• The term acceptable means that a native speaker
  would judge the word sequences as syntactically
  and semantically correct.

72
Scoring Procedures CWS and CIWS

• Use the carat method for scoring.
• Place a carat above two words if it represents a
  correct word sequence and
• Place a carat below the words if it represents an
  incorrect sequence.
• Score incorrect sequences first using a red
  pencil below the line. Count incorrect
  sequences.
• Then score correct sequences with a blue pencil
  above the line. Count correct sequences.

73
Score for correct/incorrect word sequence

• Correct word sequence Two adjacent correctly
  spelled words acceptable within the context of
  the sample
• Joe went to the store.
• Joe went to the store .

74
Scoring Score for correct/incorrect sequences

• The cave was very dark. I? try ?toclosemy
  eyes,soIcouldntseeanything,butthat
  didnthelp. ?Than?I?hear?some?one?
  breathing.I?try?to?stream?,but?nother?came
• out.Thebreathing?became?close?and?
  close?tome,andtheworstPartwasthat
  Icouldntsee?athing?. AtfirstIthought?
  meself? thatI?an?Just?emaging?stuff.

Correct 36
Incorrect 27
Correct minus incorrect 9
75
Scoring Procedures CWS and CIWS

• Count correct sequences.
• Subtract incorrect sequences from correct
  sequences.
• Record and graph score.

76
Graph Data Baseline
77
Graph Set Goal
78
Graph Data Intervention
79
(No Transcript)
80
(No Transcript)
81
For More Information

• Literature Review, Technical Reports, and Scoring
  Guidelines available
• Research Institute for Progress Monitoring
• http//www.progressmonitoring.org/
• mcmas004_at_umn.edu