OK, Ive Collected Data Now What

1
OK, Ive Collected Data Now What?

• Rebecca Fogarty, SSP, NCSP
• Eastern Illinois University
• Email rfogarty_at_eiu.edu

2
Agenda

• RTI review
• DIBELS/Aimsweb overview
• Looking at the data
• Converting to informative graphs
• Deciding what the data tells us
• Where do we go from there?

3
Foundational Concepts, Vocabulary, and Tools

• Response-to-Intervention and Problem-Solving are
  about
• Thinking Differently
• About Problems, Causes, and Solutions (Concepts)
• Talking Differently
• About Problems, Causes, and Solutions
  (Vocabulary)
• Doing Some Things Differently (Tools and
  Behaviors)

4
What is Response to Intervention
(RtI)?(Batsche, Elliott, Graden, Grimes,
Kovaleski, Prasse, Reschly, Scharg, Tilley, 2005)

• Identifying and providing high quality
  instruction and research-based interventions
  matched to students needs
• Measuring rate of improvement (ROI) over time to
  make important educational decisions
• Educators use ongoing student performance data to
  determine if an intervention is working. If it is
  not, it is time to do something different.

5
What RTI Is and Is Not

• Is
• RtI is an overall integrated system of service
  delivery- a systems change.
• Is Not
• RtI is not just an eligibility systema way of
  finding students eligible and/or reducing the
  numbers of students placed into special
  education. This is incidential to an RtI
  system.

6
The VISION To Provide Effective Interventions
to Meet the Needs of ALL Students Through Early
and Scientifically Based Interventions Through
Careful Systems Planning
STUDENTS
7
Problem Solving Method
8
What is CBM?

• CBM is a form of Curriculum-Based Assessment
  (CBA).
• Curriculum-Based Measurement (CBM) is the method
  of screening and monitoring student progress
  through direct, continuous assessment of basic
  skills.
• CBM (General Outcome Measures-GOM) is used to
  assess skills such as reading fluency,
  comprehension, spelling, mathematics, and written
  expression.
• CBM probes last from 1 to 4 minutes depending on
  the skill being measured and student performance
  is scored for speed and accuracy to determine
  proficiency. Because CBM probes are quick to
  administer and simple to score, they can be given
  frequently to provide continuous progress data.
  The results are charted and provide for timely
  evaluation based on hard data.

9
Things to Always Remember About CBM
Designed to serve as indicators of general
reading achievement CBM probes dont measure
everything, but measure the important
things. Standardized tests to be given, scored,
and interpreted in a standard way Researched
with respect to psychometric properties to ensure
accurate measures of learning
10
Items to Remember (continued)
Are sensitive to improvement in brief intervals
of time Designed to be as short as possible to
ensure its do ability Are linked to decision
making for promoting positive achievement and
Problem-Solving
11
Use Scientifically Based Problem Identification
Progress Monitoring Tools
NATIONAL CENTER ON STUDENTP PROGRESS MONITORING
www.studentprogress.org
12
Not All Assessment Tools Schools Use Meet
Accepted Psychometric Standards Members of the
CBM Family Do
www.studentprogress.org/tools
13
What is a Universal Screening?

• Given to everyone
• Measures Critical Skills
• Brief
• Repeatable
• Cheap and easy to administer and score
• Tells us who needs intervention

14
What do the Universal Screening measures look
like? What do they tell us?
15
Clarification What is the difference between
DIBELS and Aimsweb?

• DIBELS (Dynamic Indicators of Basic Early
  Literacy Skills) are early literacy and oral
  reading fluency measures developed by the
  University of Oregon.
• AIMSWeb also has a set of Early Literacy and Oral
  Reading Fluency Measures. They are almost
  identical to the DIBELS measures. The
  differences are not significant.

16
(No Transcript)
17
1. Aimsweb/DIBELS Letter Naming Fluency (Measures
the number of letters a student can name in one
minute.)
Here are some letters. Begin here (point to the
first letter) and tell me the names of as many
letters as you can. If you come to a letter you
dont know, Ill tell it to you. Are there any
questions? Put your finger under the first
letter. Ready, begin. In general, does the
student have automaticity/fluency of naming?
g N E Y R l V d H ZN d x S C n j H s SE n G h c
i h B b OY F p D L i q c D QR v F J Z M P o p
ul G A f V B P k m IV M e r y z a L U Ad y q v
w u T w N UH j K e r X T z Y XZ x f m W W s J I
kl E R K g N E Y R l
18
2. Aimsweb Letter Sound Fluency (Measures the
number of letter sounds a student can name in one
minute.)
Here are some letters. Begin here and tell me
the sounds (with emphasis) of as many letters as
you can. If you come to a sound you dont know,
Ill tell it to you. Are there any questions?
Put your finger under the first letter. Ready,
begin. In general, can the student efficiently
convert the visual symbol into an auditory
one..with automaticity?
g N E Y R l V d H ZN d x S C n j H s SE n G h c
i h B b OY F p D L i q c D QR v F J Z M P o p
ul G A f V B P k m IV M e r y z a L U Ad y q v
w u T w N UH j K e r X T z Y XZ x f m W W s J I
kl E R K g N E Y R l
19

• 2. DIBELS Initial Sound Fluency
• Student must recognize and produce the initial
  sound of a given picture
• Say these specific directions to the student
  This is mouse, flowers, pillow, letters (point to
  each picture while saying its name). Mouse (point
  to mouse) begins with the sound /m/.
• Listen, /m/, mouse. Which one begins with the
  sounds /fl/?

20
3. Aimsweb/DIBELS Phoneme Segmentation
Fl (Measures the number of phonemes students can
segment in 1 minute.)
I am going to say a word. After I say it, you
tell me all the sounds in the word. So, if I
say, sam, you would say /s/ /a/ /m/. Lets
try one. (one second pause). Tell me the sounds
in mop Ok. Here is your first word. In
general, does the student understand that words
can be broken into its component phoneme/parts?-
a basis for phonics acquisition
21
4. Aimsweb/DIBELS Nonsense Word Fluency (Measures
the number of phonemes students can read in 1
minute.)
Here are some more make-believe words (point to
the student probe). Start here (point to the
first word) and go across the page (point across
the page). When I say, begin, read the words
the best you can. Point to each letter and tell
me the sound or read the whole word. Read the
words the best you can. Put your finger on the
first word. Ready, begin. In general, does the
student have automaticity with mapping/recalling
the sound-letter relationships?Can they CRACK
THE CODE?
22
Why THESE Early Literacy Measures?

• J. Torgesen says that Measures of letter
  knowledge continue to be the best single
  predictor of reading difficulties.
• Marilyn J. Adams, in her article, The Elusive
  Phoneme, says that a childs level of phonemic
  awareness on entering school is widely held to be
  the strongest single determinant of the success
  that he or she will experience in learning to
  read.

23
5. Aimsweb/DIBELS CBM Oral Reading
Fluency (Measures students ability to read grade
level passages accurately and fluently.)
Please read this (point) out loud. If you get
stuck, I will tell you the word so you can keep
reading. When I say, stop I may ask you to
tell me about what you read, so do your best
reading. Start here (point to the first word
of the passage). Begin. In general, has the
student developed automatic phonemic awareness,
phonics skills, and word recognition skills to be
a fluent reader?
24
IN GENERAL, ORAL READING FLUENCY MEASURES
PROVIDES QUALITATIVE INFORMATION ABOUT 3 BROAD
COMPETENCIES

• 1. RATE Words read correctly
• -Above 75th- consider differentiating
  instruction
• -Below 25- consider need for Tier 2
  interventions
• -Below 10- further assess, do problem analysis,
  and consider need for Tier 2 and/or 3
  interventions
• 2. ACCURACY Error rates
• -0-5 error rate acceptable accuracy (skilled
  readers are 95 or better accurate)..
• -5-10 error rate accuracy in question
• -gt10 error rateunacceptable accuracy
• 3. COMPREHENSION
• -ADEQUATE FLUENCY AND RATE CORRELATE STRONGLY
  WITH ADEQUATE COMPREHENSION

25
BIG IDEA Integrated Assessment Systems
This is what weve had.
Assessment
Instruction
This is what we want..
Instruction
Assessment
Aligning Assessment and Instruction
26
JOLLY PHONICS
ELEMENTS OF READ-VOCAB.
COLLABORATIVE STRAT. READ.
27
BIG IDEA
Use assessment data to determine student need and
link that to research-based interventions
that match the need.
DATA
INFORMS NEED
INTERVENTION
28
CHANGING FOCUS
29
How does the data collection and measuring RtI
all fit together with Intervention Systems and
Problem Solving?
30
With screening and progress monitoring data, you
then begin making informed Data-Based Decisions.

• District or school level decisions
• Classroom or Group Decisions
• Individual Student Decisions

31
Using Universal Screening/Benchmarking Dataand
Progress Monitoring Data to drive Instructional
and Intervention DecisionsDIBELS Kindergarten
Class Fall 2007
32
Using Universal Screening/Benchmarking Dataand
Progress Monitoring Data to drive Instructional
and Intervention DecisionsKindergarten Class
Fall 2007

• DIBELS cut offs are
• Low risk ISFgt8
• Some risk 4ltISFlt8
• At- Risk ISFlt4

33
Old System of Problem Solving
Special Education
General Education
Amount of Resources Needed To Benefit
General Education with Support
Severity of Educational Need or Problem
34
Why hasnt this old system of problem solving
been very effective?

• Because weve been trying to solve students
  problems one student at a time.
• This has been impractical and too time intensive
  to be effective.

35
Bridging the Gap
Core Intensive
Core Supplemental
Weekly
Core
Amount of Resources Needed To Benefit
Weekly-Monthly
3x/year
Severity of Educational Need or Problem
36
ALWAYS THINK ABOUT STUDENT NEEDS IN THIS
FRAMEWORK
INDIVIDUAL NEEDS
TIER III.

TIER II.

• CLASS NEEDS
• SMALL GROUP NEEDS

• DISTRICT NEEDS
• SCHOOL NEEDS
• GRADE LEVEL NEEDS

TIER I.
37
Using Data to Problem Solve at Tier 1.
38
Data Based Decision Making
We want these percentages or better
39
Problem Solving Approach
Amount of Resources Needed To Solve Problem
Special Education
Universal Interventions

• DEFINE THE PROBLEM
• Is there a problem? What is it?
• ANALYZE
• Why is it happening?
• DEVELOP A PLAN
• What shall we do about it?
• EVALUATE
• Did our plan work?

General Education
Intensity of Problem
40
Example of Tier 1 Problem Solving Screening
Detects a Problem with Core Curriculum

• Classwide Intervention
• Large Group
• Most students will respond
• Typically this is a general education
  responsibility
• Joe Witt, www.isteep.com/compcenters

41
How Do you KNOW if Core Instruction is Working
Screen-Many students not Learning at Tier 1
Grade Level Standard
Now does she look like a problem?
Kalisha in Red Seems to be a Problem
42
Go Back to the Triangle- Are 80 of students
making it?
We want these percentages or better
43
Universal Screening can Identify School Wide
Reading Deficits
These 29 students are the only ones at meets or
exceeds
Mastery
Instructional
At Risk
Reading data- 1st grade
44
After Grade Wide Intervention--No Systemic Problem
First Grade
Now these 9 are the only ones below instructional
level
45
Data Based Decision Making
We want these percentages
Tier 1. 50 or better on Aimsweb norms. Tier 3.
25 or lower on Aimsweb norms. Tier 2 Everyone
in between.
46
Second Grade Fall Data ORF
47
Second Grade Aimsweb Norms
ORF scores 28 or lower
3
What we have
We want these percentages
Tier 1. 50 or better on Aimsweb norms. Tier 3.
25 or lower on Aimsweb norms. Tier 2 Everyone
in between.
71
ORF scores 55 or better
48
Aimsweb Growth Table
Tier 1- 50 or better Tier 3- 25 or lower
49
Second Grade Aimsweb Norms
ORF scores 28 or lower
3
What we have
We want these percentages
Tier 1. 50 or better on Aimsweb norms. Tier 3.
25 or lower on Aimsweb norms. Tier 2 Everyone
in between.
71
ORF scores 55 or better
50
Actual scores in Triangles vs National Scores
At what tier should we problem solve?
51
If there is a Tier 1 concern

• There are generally 3 ways to address this
  concern
• Implement your Tier 1 curriculum with higher
  integrity
• Explore and adopt a new Core curriculum
• Supplement your Core curriculum with robust,
  research-based interventions

52
Examining Treatment Integrity

• Teacher self-report/implementation logs
• Teacher may be interviewed regarding steps
  followed during intervention or keep a log of the
  steps implemented
• Ratings scales
• Written step-by-step intervention plan can be
  used as a checklist implementer would complete
  checklist
• Direct Observation
• Of teacher behavior could be conducted
  periodically during intervention (use of IPF)
• Permanent Products
• Teacher/student created products that would
  demonstrate the intervention components were
  implemented

53
UNIVERSAL TIER 1 Benchmark/Core
Programs 1. Trophies (Harcourt School
Publishers, 2003) 2. The Nations Choice
(Houghton Mifflin, 2003) 3. Macmillan/McGraw-Hill
Reading (2003) 4. Open Court (SRA/McGraw-Hill,
2002) 5. Reading Mastery Plus (SRA/ McGraw-Hill,
2002) 6. Scott Foresman Reading (2004) 7. Success
For All (1998-2003) Reviewed by Oregon Reading
First Comprehensive Addressed all 5 areas and
included at least grades K-3
5
15
80 of Students
54
EXAMPLE FRAMEWORK for READING INTERVENTIONS
Wilson Reading Mastery Corrective
Reading Language! Reasoning and
Writing Horizons Great Leaps, REWARDS Selected
Interventions from Trophies Vocabulary through
Morphemes Heggerty Phonemic Awareness
Curriculum The Six Minute Solution Collaborative
Strategic Reading
TIER III. Highly at-risk students Intensive
interventions
TIER II. At-risk students- Supplemental
interventions
Selected Interventions from Trophies
Series KPALS, 1st Gr. PALS Vocabulary through
Morphemes Heggerety Phonemic Awareness
Curriculum The Six Minute Solution Collaborative
Strategic Reading
TIER I. Harcourt Trophies- Increase integrity Of
implementation
55
D105s FRAMEWORK for READING INTERVENTIONS
SRA DI PROGRAMS- READING MASTERY
HORIZONS CORRECTIVE READING SOAR TO SUCCESS GREAT
LEAPS /SLANT REWARDS, LIPS 6 MIN. SOLUTIONS
TIER III. Highly at-risk students Intensive
interventions
K PALS, 1st Gr. PALS JOLLY PHONICS/GRAMMAR M.
HEGGERTY EAROBICS GREAT LEAPS/ SLANT REWARDS,
QUICK READS 6 MINUTE SOLUTIONS BRINGING WORDS TO
LIFE ELEMENTS OF READING-VOCABULARY METACOGNITIVE
STRAT.- COLLABORATIVE STRATEGIC READING SOAR TO
SUCCESS
TIER II. At-risk students- Supplemental
interventions
K PALS, 1st Gr. PALS, JOLLY PHONICS, JOLLY
GRAMMAR M. HEGGERTY PROGRAM 6 MINUTE SOLUTIONS,
QUICK READS, BRINGING WORDS TO LIFE ELEMENTS OF
READING-VOCABULARY, SOAR TO SUCCESS METACOGNITIVE
STRAT., COLLABORATIVE STRATEGIC READING,
TIER I.
56
(No Transcript)
57
(No Transcript)
58
7th Grade
8th Grade
59
Examples of what can occurWith effective Tier 1
Problem Solving.
60
School-Wide Reading Improvement in a School
Using Problem-Solving
Courtesy of Christine Martin, Indian Prairie
School District, IL
61
(No Transcript)
62
With MHeggerty
With MHeggerty Jolly Phonics
63
Fall 1 of 15 is above 50th percentile Spring 9 of
15 are above 50th Jump from 6 to 60
With MHeggerty
64
SKILL BUILDER
Calcuating triangles for Risk Categories
65
Step 1. Determine your Approach/Cut Score
1. Norm-Based Approaches Percentile Rank Cut
Scores 2. Standards-Based Approaches Illinois
AIMSweb Standards (Cut Scores for ISAT and
Minnesota State Test) Oregon DIBELS Standards
(Cut Scores for Oregon State Test)
66
1. Norm-Based Approaches
Percentile Rank Cut Scores Percentile rank
scores are derived scores that indicate the
percentage of people in the norming sample that
scored at or below a given raw score. Percentile
rank scores for at risk students typically are
derived from local norms, but Aimsweb national
norms can be used.
67
Steps for determining percentages of risk
categories

• Low risk Count the number of students scoring at
  the 50ile or higher. (6/1540)
• Determine percentage. Is it 80 or higher?
• High risk Count the number of students scoring
  at the 25ile or lower. (5/1533.5)
• Determine percentage. Is it 5 or higher?
• Some risk Count the number of students between
  the 50th and 25thile. Determine percentage. Is
  it 15 or higher? (4/1526.5)

68
(No Transcript)
69
2. Standard-Based Approaches

• Illinois AIMSweb Standards Tied to ISAT and
  Minnesota State
• Oregon DIBELS Standards
• With a Standards Based Approach, Use Linkages to
  High Stakes Tests
• The desired outcome is to have the student meet
  standards on High Stakes Tests.

70
Illinois AIMSweb Standards (Cut Scores for ISAT)
71
Standards-Based Approaches and Universal Screening
Red Highly Unlikely to Pass the State
TestGreen Highly Like to PassYellow
Uncertain to Pass
72
Setting Standard-Based criteria for Tier 1School
GOAL Students should have LNF scores gt 40 by
Spring of Kindergarten.
73
Standard-Based Criteria
Total number of students 47 Total meeting
criteria (66) (gt40 LNF) 31 Total in
Emerging range (28) (29 lt LNF lt40) 13
Total in At-Risk range (6) (LNF lt 29) 3
Sample of students in Low Risk criteria
Sample of students in Some-Risk criteria
Students in At-Risk range
74
Steps for determining percentages of risk
categoriesUsing a Standards-Based approach

• Low risk Count the number of students scoring at
  the proficient no. or higher on ISAT correlates
• Determine percentage. Is it 80 or higher?
• High risk Count the number of students scoring
  at the Below Basic on ISAT correlates.
• Determine percentage. Is it 5 or higher?
• Some risk Count the number of students between
  the Proficient and Below Basic on ISAT
  correlates. Determine percentage. Is it 15 or
  higher?

75
Example Standard-Based Criteria Discrepancy for
Tier 1
At Tier 1, 66 of students are in the Low Risk
range (40 LS/min), whereas it is expected that
80 of students are in the Low Risk range.
76
When using either approach, compare cut scores to
student scores to determine risk categories.
77
(No Transcript)
78
Keep In Mind

• Advantages and disadvantages of cut off scores

79
Watching Individual Students Tier 2 and 3
Core Intensive
Core Supplemental
Amount of Resources Needed To Solve Problem
Special Education
Specific Interventions
Weekly

• DEFINE THE PROBLEM
• Is there a problem? What is it?
• ANALYZE
• Why is it happening?
• DEVELOP A PLAN
• What shall we do about it?
• EVALUATE
• Did our plan work?

Weekly-Monthly
Intensity of Problem
80
Watching Students at Tier 2 and 3
Core Intensive
Core Supplemental
Weekly
Amount of Resources Needed To Benefit
Weekly-Monthly
Severity of Educational Need or Problem
81
Benchmark Data Collection vs. Progress Monitoring
82
Progress monitoring is designed to answer 3
fundamental questions

• Is the student benefiting from the intervention?
• Is the students rate of improvement sufficient?
• Does the intervention need to be modified?

83
Jill Grade 2Core and SupplementaryData
Collection biweekly
84
Tier 2 and Tier 3 Students

• Even though you are charting data for individual
  students.
• you still want to look at the groups as a whole

85
Watching Individual Students Tier 2 and 3
Special Education
Specific Interventions
Weekly
Weekly-Monthly
Are these percentages changing and getting
smaller????
86
Tool Problem-Solving Self Study Instruments
87
TIER 1, 2, or 3Instructional Planning Form
10/03 Adapted from the U of Oregon
88
Making the triangles in excel

• See attached triangle template as reference. See
  data-template
• Enter percentages into cells
• Select all- including National and Grade cells
• Go to chart wizard
• For Chart Type, scroll to bottom and select
  pyramid
• Select the 3-tier pyramid- top right option
• NEXTgt

89
Making triangles in excel (cont.)

• Series In Select Rows
• NEXTgt
• Title graph as you want
• Go to data labels. Select Show Value
• Select New Sheet
• You can change color of tiers to clicking on each
  tier and selecting new color

90
www.fcrr.orgFlorida Center for Reading
Research

• www.fcrr.org

91
Questions???COMMENTS?