PROGRESS MONITORING TRAINING

1
PROGRESS MONITORING TRAINING

• Series 2 Staff
• October 22, 2008
• Presenters L. Williams and K. Todd

2

• Training Schedule
• Admin/Scoring of R-CBM and MAZE
• Progress Monitoring (PM) Overview
• PM Software Overview

3
(No Transcript)
4
Administration and Scoring of READING-CURRICULUM
-BASED MEASUREMENT (R-CBM) for Use in General
Outcome Measurement

• Mark R. Shinn, Ph.D.
• Michelle M. Shinn, Ph.D.

5
Things you Need Before Testing

• 1. Standard Reading Assessment Passage Student
  Copy
• No numbers
• Between 250-300 words (exception 1st grade)
• An informative first sentence
• Same font style and size
• Text without pictures
• Obtain from your LAM

6
Things you Need Before Testing

• 2. Standard Reading Assessment Passage Examiner
  Copy
• Pre-numbered so they can be scored quickly and
  immediately.
• Obtain from your LAM.

7
Things You Need to do While Testing

• Follow the standardized directions
• R-CBM is a standardized test
• Administer the assessment with consistency
• Remember its about testing, not teaching
• Dont teach or correct
• Dont practice reading the passages
• Remember best, not fastest reading
• Sit across from, not beside student

8
R-CBM Standard Directions for 1 Minute
Administration

• Place the unnumbered copy in front of the
  student.
• Place the numbered copy in front of you, but
  shielded so the student cannot see what you
  record.
• Say
• When I say Begin, start reading aloud at the
  top of this page. Read across the page
  (DEMONSTRATE BY POINTING). Try to read each word.
  If you come to a word you dont know, I will
  tell it to you. Be sure to do your best reading.
  Are there any questions? (PAUSE)
• Say Begin and start your stopwatch when the
  student says the first word. If the student
  fails to say the first word of the passage after
  3 seconds, tell them the word, mark it as
  incorrect, then start your stopwatch.
• Follow along on your copy. Put a slash ( / )
  through words read incorrectly.
• At the end of 1 minute, place a bracket ( )
  after the last word and say, Stop.
• Score and summarize by writing WRC/Errors

9
Familiar Shortened Directions
When students are assessed frequently and know
the directions. Say When I say Begin, start
reading aloud at the top of this page.
10
Items to Remember
Emphasize Words Read Correctly (WRC). Get an
accurate count. 3-Second Rule. No Other
Corrections. Discontinue Rule. Be Polite. Best,
not fastest. Interruptions.
11
Things to do After Testing
Score immediately! Determine WRC. Put a slash (/)
through incorrect words. If doing multiple
samples, organize your impressions of qualitative
features.
12
What is a Word Read Correctly?
Correctly pronounced words within
context. Self-corrected incorrect words within 3
seconds.
13
What is an Error?
Mispronunciation of the word Substitutions
Omissions 3-Second pauses or struggles
(examiner provides correct word)
14
What is not Incorrect? (Neither a WRC or an Error)
Repetitions Dialect differences Insertions
(consider them qualitative errors)
15
Example
Juan finished reading after 1 minute at the 145th
word, so he read 145 words total. Juan also made
3 errors. Therefore his WRC was 142 with 3
errors. Reported as 142/3.
16
Calculating and R-CBM Scores
Record total number of words read. Subtract the
number of errors. Report in standard format of
WRC/Errors (72/3).
17
R-CBM Scoring Rules and Examples
A complete list of scoring rules can be found in
the Appendix ofyour workbook. Please review and
become familiar with the more unusual errors.
18

• Benchmark or SLA Data Obtain MEDIAN score
  for students 3 passages
• 67 / 2 85 / 8 74
  / 9

1 min. 1 min.
1 min.
Why use Median vs. Average? Averages are
susceptible to outliers when dealing with small
number sets.Median Score is a statistically more
reliable number than average for R-CBM.
19
The Data Obtain MEDIAN score for 3
passages 67 / 2 85 / 8
74 / 9
1 min. 1 min.
1 min.
1. Throw out the HIGH and LOW scores for
Words Read Correct (WRC)
20

• The Data Obtain MEDIAN score for 3 passages
  67 / 2 85 / 8 74 /
  9

1 min. 1 min.
1 min.
2. Throw out the HIGH and LOW scores for the
Errors. Remaining scores MEDIAN. 3.
Report this score in your AIMSweb account.

74/8
21
Determining Inter-Rater Agreement
Example Dave 2 examiners observed Daves
reading 1 scored Dave as 100 WRC 1 scored Dave
as 98 WRC They agreed that Dave read 98 of
the words correct. They disagreed on 2 words
correct. Inter-rater Agreement
FormulaAgreements/(Agreements Disagreements)
x 100IRA (98)/ (98 2) 98/100 .98 .98 x
100 98 Inter-rater Agreement for Dave is
98. (Goal is 95 or better.)
See R-CBM Workbook Page 14
22
Prepare to Practice
23
Practice Session

• Watch and score Practice Exercises 3 7
• Note Refer to (blue) Handout

24
Administration and Scoring of READING-MAZE
(R-MAZE) for Use in General Outcome Measurement

• Power Point Authored by
• Jillyan Kennedy
• Based on Administration and Scoring of Reading
  R-MAZE for Use
• with AIMSweb Training Workbook
• By
• Michelle M. Shinn, Ph.D.
• Mark R. Shinn, Ph.D

25
Curriculum Based Measurement Reading R-MAZE
CBM R-MAZE is designed to provide educators a
more complete picture of students reading
skills, especially when comprehension problems
are suspected.
26
Curriculum Based Measurement Reading R-MAZE

• R-MAZE is a multiple-choice cloze task that
  students complete while reading silently.
• The students are presented with 150-400 word
  passages.
• The first sentence is left intact.
• After the first sentence, every 7th word is
  replaced with three word choices inside a
  parenthesis.
• The three choices consist of

• Near Distracter
• Exact Match
• Far Distracter

27
Sample Grade 4 R-MAZE Passage
28
Examples of R-MAZE
R-MAZE Workbook Page 9
29
Administration and Scoring of CBM R-MAZE

• What examiners need to do . . .

• Before testing students
• While testing students
• After testing students

30
Items Students Need Before Testing

• What the students need for testing
• CBM R-MAZE practice test
• Appropriate CBM R-MAZE passages
• Pencils

31
Items Administrators Need Before Testing

• What the tester uses for testing
• Stopwatch
• Appropriate CBM R-MAZE answer key
• Appropriate standardized directions
• List of students to be tested.

32
Additional Assessment Aids

• A List of students to be tested
• Stopwatch (requireddigital preferred)

33
Things You Need to do While Testing
Follow the standardized directions

• Attach a cover sheet that includes the practice
  test so that students do not begin the test
  right away.
• Do a simple practice test with younger
  students.
• Monitor to ensure students are circling answers
  instead of writing them.
• Be prepared to Prorate for students who may
  finish early.
• Try to avoid answering student questions.
• Adhere to the end of timing.

34
CBM R-MAZE Standard Directions

• Pass R-MAZE tasks out to students. Have students
  write their names on the cover sheet, so they do
  not start early. Make sure they do not turn the
  page until you tell them to.
• Say this to the student (s)
• When I say Begin I want you to silently read a
  story. You will have 3 minutes to read the story
  and complete the task. Listen carefully to the
  directions. Some of the words in the story are
  replaced with a group of 3 words. Your job is to
  circle the 1 word that makes the most sense in
  the story. Only 1 word is correct.
• Decide if a practice test is needed. Say . . .
• Lets practice one together. Look at your first
  page. Read the first sentence silently while I
  read it out loud The dog, apple, broke, ran
  after the cat. The three choices are apple,
  broke, ran. The dog apple after the cat. That
  sentence does not make sense. The dog broke
  after the cat. That sentence does not make
  sense. The dog ran after the cat. That
  sentence does make sense, so circle the word ran.
  (Make sure the students circle the word ran.)

35
CBM R-MAZE Standard Directions (Continued)
Lets go to the next sentence. Read it
silently while I read it out loud. The
cat ran fast, green, for up the hill. The three
choices are fast, green, for up the
hill. Which word is the correct word
for the sentence? (The students answer
fast) Yes, The cat ran fast up the
hill is correct, so circle the correct
word fast. (Make sure students circle fast)
Silently read the next sentence and raise
your hand when you think you know the
answer. (Make sure students know the correct
word. Read the sentence with the correct
answer) Thats right. The dog barked
at the cat is correct. Now what do
you do when you choose the correct
word? (Students answer Circle it.
Make sure the students understand the task)
Thats correct, you circle it. I think youre
ready to work on a story on your own.
36
CBM R-MAZE Standard Directions (Continued)

• Start the testing by saying . . .
• When I say Begin turn to the first story and
  start reading silently. When you come to a group
  of three words, circle the 1 word that makes the
  most sense. Work as quickly as you can without
  making mistakes. If you finish a/ the page/first
  side, turn the page and keep working until I say
  Stop or you are all done. Do you have any
  questions?
• Then say, Begin. Start your stopwatch.
• Monitor students to make sure they understand
  that they are to circle only 1 word.
• If a student finished before the time limit,
  collect the students R-MAZE task and record the
  time on the students test booklet.
• At the end of 3 minutes say Stop. Put your
  pencils down. Please close your booklet.
• Collect the R-MAZE tasks.

37
CBM R-MAZE Familiar Directions

• After the students have put their names on the
  cover sheer, start the testing by saying . . .
• When I say Begin turn to the first story and
  start reading silently. When you come to a group
  of three words, circle the 1 word that makes the
  most sense. Work as quickly as you can without
  making mistakes. If you finish a/ the page/first
  side, turn the page and keep working until I say
  Stop or you are all done. Do you have any
  questions?
• Then say, Begin. Start your stopwatch.
• Monitor students to make sure they understand
  that they are to circle only 1 word.
• If a student finished before the time limit,
  collect the students R-MAZE task and record the
  time on the students test booklet.
• At the end of 3 minutes say Stop. Put your
  pencils down. Please close your booklet.
• Collect the R-MAZE tasks.

38
Things to Do After Testing

• Score immediately to ensure accurate results!
• Determine the number of words (items) correct.
• Use the answer key and put a slash (/) through
  incorrect words.

39
CBM R-MAZE Scoring
What is correct? The students circles the word
that matches the correct word on the scoring
template.

• What is incorrect?
• An answer is considered an error if the student
• Circles an incorrect word
• Omits word selections other than those the
  student
• was unable to complete before the 3 minutes
• expired

40
Making Scoring Efficient

• Count the total number of items up to the last
  circled word.
• Compare the student answers to the correct
  answers on the scoring template. Mark a slash
  / through incorrect responses.
• Subtract the number of incorrect answers from the
  total number of items attempted.
• Record the total number of correct answers on the
  cover sheet followed by the total number of
  errors (e.g., 35/2).

41
CBM R-MAZE Prorating
If a student finishes all the items before 3
minutes, the score may be prorated.

• When the student finished, the time must be
  recorded and the number of correct answers
  counted. For example, the student may have
  finished in 2 minutes and correctly answered 40
  items.
• Convert the time taken in seconds. (2 minutes
  120 seconds)
• Divide the number of seconds by the number
  correct. (120/40 3)
• Calculate the number of seconds in the full 3
  minutes. (3 minutes 180 seconds)
• Divide the number of full seconds by the
  calculated value from step 3. (180/3 60)

42
MAZE

• Score MAZE Practice Exercise
• (Refer to handout)

43
Progress Monitoring

• Strategies for Writing Individual Goals in
  General Curriculum and More Frequent Formative
  Evaluation
• Mark Shinn, Ph.D.Lisa A. Langell, M.A., S.Psy.S.

44
Big Ideas About Frequent Formative Evaluation
Using General Outcome Measures and the Progress
Monitoring Program

• One of the most powerful interventions that
  schools can
• use is systematic and frequent formative
  evaluation.
• Benchmark Assessment is not enough for some
  students because they may be in ineffective
  programs too long. (3 mos )
• The solution is to write individualized goals and
  determine a feasible progress monitoring
  schedule.
• The core of frequent progress monitoring is
• Survey-Level Assessment
• Goal setting using logical educational practices
• Analysis of student need and resources for
  determining progress monitoring frequency.

45
Formative Assessment

• Formative Assessment Process of assessing
  student achievement during instruction to
  determine whether an instructional program is
  effective for individual students.
• When students are progressing, keep using your
  instructional programs.
• When tests show that students are not
  progressing, you can change your instructional
  programs in meaningful ways.
• Has been linked to important gains in student
  achievement (L. Fuchs, 1986) with effect sizes
  of .7 and greater.

46
Systematic formative evaluation requires the use
of

• Standard assessment tools
• That are the same difficulty
• That are Given the same way each time.

47
More Severe Achievement Problems and/or More
Resource Intensive Programs Require More Frequent
Formative Evaluation
Benchmark Testing (3 - 4 x Per Year) is not
enough for some students.
48
With Very Low Performers, Not Satisfactory to
Wait This Long!
49
Programs That are More Resource Intensive

• Title I, English Language Learning, Special
  Education
• Should monitor student outcomes more frequently
  than the Benchmark Testing schedule.

50
Thinking About A Students Data

• Sample Student
• Melissa Smart
• 3rd grade student
• Progress Monitor

51
8
Melissa Smart
110
92
77
50
34
52
Formative EvaluationIs simply data enough?
53
Formative Evaluation Is data and a goal enough?
54
Formative Evaluation Are data, goals trends
enough?
55
Formative Evaluation is Impossible without all
dataGoals Make Progress Decisions Easier
56
Current Goal Setting Practices Are Unsatisfying!
Do you like these IEPs? I do not like these
IEPs I do not like them Jeeze Louise We test, we
check We plan, we meet But nothing ever seems
complete. Would you, could you Like the form? I
do not like the form I see Not page 1, not 2, not
3 Another change A brand new box I think we
all Have lost our rocks!
57
Need Shift to Few But Important Goals

• Often Ineffective Goal Smorgasbord!
• Student will perform spelling skills at a high
  3rd grade level.
• Student will alphabetize words by the second
  letter with 80 accuracy.
• Student will read words from the Dolch Word List
  with 80 accuracy.
• Student will increase reading skills by
  progressing through Scribner with 90 accuracy as
  determined by teacher-made fluency and
  comprehension probes by October 2006.
• To increase reading ability by 6 months to 1 year
  as measured by the Woodcock Johnson.
• Student will be a better reader.
• Student will read aloud with 80 accuracy and 80
  comprehension.
• Student will make one year's gain in general
  reading from K-3.
• Students will read 1 story per week.

58
Improving the Process of Setting Goals for
Formative Evaluation
Set a few, but important goals. Ensure goals are
measurable and linked to validated formative
evaluation practices. Base goal setting on
logical educational practices.
59
Reduce the Number of Goals to a Few Critical
Indicators
Reading In () weeks (Student name) will read ()
Words Correctly in 1 minute from randomly
selected Grade () passages. Spelling In ()
weeks (Student name) will write () Correct
Letter Sequences and () Correct Words in 2
minutes from randomly selected Grade () spelling
lists. Math Computation In () weeks (Student
name) will write () Correct Digits in 2 minutes
from randomly selected Grade () math
problems. Written Expression In () weeks
(Student name) will write () Total Words and ()
Correct Writing Sequences when presented with
randomly selected Grade () story starters.
60
Conducting a Survey Level Assessment
Students are tested in successive levels of
general curriculum, beginning with their current
expected grade placement, until a level at which
they are successful is determined.
61
When to use Survey Level Assessment (SLA)
To determine grade level equivalent for reading.
To determine grade level probe to use for
progress monitoring. Note If you have
supplemental data (i.e. QRI, DRA2, SRA testing,
or READ 180 lexile score) that indicates current
instructional level in reading, survey level
assessment is not necessary.
62
John5th grader5th grade passage 26/12
John4th grade passage49/7
John3rd grade passage62/4
Conducting a Survey Level Assessment
63
Base Goal Setting on Logical Educational Practices
Example of PLEP statement John currently reads
about 26 words correctly from Grade 5 Standard
Reading Assessment Passages. He reads Grade 3
reading passages successfully 62 correct words
per minute with 4 errors, which is how well
beginning 3rd grade students read this material.
64
Setting the Time Frame, Goal Level Material, and
Criterion
Time Frame End of Year (At Risk or Grade-Level
Expectations) In 18 Weeks Annual IEP Goals
(Special Education) In 1 year (or) In 32 Weeks
65
When Grade-Level Expectations Are Not Appropriate

• Consider the Severity of the Discrepancy
• Consider the Intensity of the Program

66
Determining the Criterion for Success Options
to use

• Local Benchmark (school-based) Standards.
• National Benchmark Standards.
• Normative Growth Rates (see Progress Monitoring
  Training Workbook- found on AIMSweb downloads).

67
Normative Growth Rates
Criterion for Success Score on SLA (Grade
Growth Rate times of Weeks) Score on SLA (30)
(Ambitious Grade Growth Rate (2.0) times of
Weeks (32) Or 30 (2.0 32) or 30 64
Annual goal of 94 WRC
68
How Frequently to Assess?
Balancing IDEAL with FEASIBLE twice a week to
once a month
69

• Making Data-Based Decisions With Progress Monitor
  
• Need at LEAST 4-7 data points before making
  programming decisionand you may sometimes
  want to collect more if you are uncertain.
• Err on the side of caution
• Criteria To Consider
• Trendline meets Aimline for ultimate
  goal Consider return to LRE.
• Trendline and AIMline will intersect in
  relatively near future?
• Keep with current intervention until goal is
  reached.
• Trendline exceeds AIMline?
• Consider increasing goal or difficulty level.
• Trendline not going to intersect AIMlinemoves in
  opposite direction Consider adding additional
  intervention, changing variable, and/or
  insensifying program changes (LRE).

70
(No Transcript)
71
(No Transcript)
72
(No Transcript)
73
(No Transcript)
74
(No Transcript)
75

• Contact us
• leslie.williams_at_d300.org
• kara.todd_at_d300.org
• Good luck and may progress be in your future