Understanding by Design

1
Debunking the Five Biggest Myths of Data-Based
Decision Making
Dr. Ronald S. Thomas Center for Leadership in
Education at Towson University rathomas_at_towson.ed
u Douglas Elmendorf Assistant Principal Dundalk
Elementary School delmendorf_at_bcps.org
2
Show Us the Data!
Data are observations, facts, or numbers that,
when collected, organized, and analyzed, become
information and, when used productively in
context, become knowledge.

3
Show Us the Data!
Data are observations, facts, or numbers that,
when collected, organized, and analyzed, become
information and, when used productively in
context, become knowledge. Data
Information Knowledge A
LEARNING ORGANIZATION

4
The DRIP Syndrome
DATA RICH
INFORMATION POOR
5
Myth 1
Data analysis for instructional decision making
is mostly numbers crunching using sophisticated
statistical formulas such as for standard
deviation, regression, and measures of variance.
6
Reality 1
Data analysis for instructional decision making
is identifying patterns analyzing trends
and solving curriculum, instructional, and
assessment problems.
7
Data analysis is a collaborative social process
of making meaning and taking action.

• This means
• Clarity and transparency
• of learning targets for
• teachers and students.

8
Data analysis is a collaborative social process
of making meaning and taking action.

• This means
• Instruction with a
• laser-like focus
• on the target.

9
Data analysis is a collaborative social process
of making meaning and taking action.

• This means
• Assessments that are the
• best fit to the target so that
• we can make valid
• inferences from the
• results.

10
Data analysis is a collaborative social process
of making meaning and taking action.

• This means
• Acting on the results of the assessments
• Enrichment
• Interventions
• Instructional change

11
Myth 1
Data analysis for instructional decision making
is mostly numbers crunching using sophisticated
statistical formulas such as for standard
deviation, regression, and measures
of variance. Reality 1 Data analysis for
instructional decision making is
identifying patterns analyzing trends and
solving curriculum, instructional, and assessment
problems. Talk to a few people around you about
some of the specific actions you would take as
a leader to replace the myth with this reality
in your school. You may take notes on page 3.
12
Myth 2
Data are data. There are very few differences
among data sources.
13
Reality 2
There are significant differences among data
sources. There are accountability data (data to
prove) and instructional improvement data (data
to improve). While both types of data are
important, they are different data.
14
Data for Two Major Purposes

15
Examples of Data To Prove that
Education Is Doing its Job
MSA/HSA percent passing Stanford 10 median
percentiles SAT total scores AYP data Other
examples of scores the public sees in the
newspaper
16
Examples of Data To Improve the Job
Education Is Doing
Good MSA content standard scale scores Better
Common unit test sub-scores Best Rubric results
from classroom assessments Classroom observations
and every pupil responses during authentic
tasks
17

• Data To Prove Asks Mostly
• How Many Questions
• How many students were dual CTE and USM
  completers?
• How many schools in the system met AYP?
• How many students performed on the bubble in
  MSA/HSA?
• 
  

18

• Data to Improve Asks Mostly What, How, and
  Why Questions
• What knowledge and skills are the strengths of
  this years third graders in mathematics?
• What knowledge and skills are third graders the
  weakest in?
• How can we align the curriculum more completely
  with the skills demonstrated by successful
  employees in the world of work?
• Why do our students consistently do most poorly
  on the algebra/data analysis component of MSA?
• 
  

19
Data for Two Major Purposes
Accountability Data (to prove) MUCH OF WHAT MSDE
AND SCHOOL SYSTEMS COLLECT Instructional
Decision Making Data (to improve) MUCH OF WHAT
SCHOOLS NEED IN ORDER TO CHANGE
20
A Major Premise
If schools focus on the Instructional Decision
Making Data (data to improve), the Accountability
Data (data to prove) will take care of
itself.
21
Think about it . . .
Do you have a school improvement plan?
Or a school accountability plan?
Or a SAP?
A SIP ?
Have a three minute conversation with someone
sitting near you about what you think most
schools currently have and how they could change.
You may take notes on page 6.
22
Myth 3
The School Improvement Team or Data Committee are
the best groups to analyze data because they see
the big picture.
23
Myth 4
State assessment data, like those available from
MSA and HSA, are the most useful type of data
to make ongoing curriculum decisions, such as for
curriculum focus, pacing, and interventions.
24
The Old
Model
The School Improvement Team, a Data Committee,
or one person analyzes data, using primarily
state and national test data. These data are
mined for every possible nuance.
25
The Old Model
The data are presented at a faculty, SIT, or
department meeting, and faculty members
brainstorm ideas for what to do to increase
student performance.
26
The Old Model
Faculty or team members average opinions and
put forth the solutions that are acceptable to
the largest majority of people.
27
The Old
Model
This results in schoolwide or departmentwide
initiatives that may or may not be done. Data
expert Mike Schmoker has estimated that about 10
of what is planned in SIPs actually is
implemented at a high level of quality.
28
Why is the old model not working any more?
29
Why? Wrong Data

• We have been using the wrong data. State and
  national test data are
• Way too general
• Instructionally insensitive not designed for
  instructional improvement

30
Why? Wrong Time

• The data come at the wrong time. State and
  national test data are
• Out of date when they arrive
• For students we no longer have
• The results of the changes that are implemented
  will not be known for a year.

31
Why? Wrong Team

• The SIT, a full department, or a Data Committee
  is the wrong team to do the analysis.
• Membership too diverse (often including
• parents)
• Meet too infrequently
• Not connected to immediate classroom
• needs

32
Why? Wrong Plan

• The initiatives that are put in place are
• Too global to address
• the diversity of students
• Aimed at performance increases
• of groups on average
• Looking for the silver bullet that
• will have a schoolwide impact

33
Results of the Old Model
34
Data Analysis
Too important to be based on one type of data
and to be left to groups that are so diverse and
meet so sporadically.
35
What are the right teams to conduct data
dialogues?

• Content
• Vertical
• Grade level

36
Why? Right Time

• On an ongoing basis
• Integrated into the
• lesson planning cycle

37
When is the right time to conduct data dialogues?

• Devote at least one
• hour to data dialogues
• every two weeks
• According to studies, schools that realized the
  greatest shift to a data culture scheduled
  meetings once a week.

38
Frequency of Data Dialogues
Source Stanford University Center
for Research Education Week, January 24,
2004
39
What are the right data to use in the data
dialogues?

• Three types of data are triangulated
• External Assessment Data
• Coursewide or Districtwide
• Benchmark Assessment Data
• Classroom Assessment Data

40
THE GPS ANALOGY
41
What is the right way to use the results of the
data dialogues?

• Conclusions are used to identify enrichments and
  interventions for the students in the class.
• Conclusions are used to plan upcoming daily
  instruction.

42
Reality 3
Structured data analyses should be completed at
least weekly by content or grade-level teams that
know their students well and can impact
instruction immediately.
43
Reality 4
Periodic common assessment data and ongoing
analyses of student work are the best sources to
use to impact instruction and increase achievement
.
44
We need a new process.

• Real time
• Specific to each grade and subject
• Addresses individual students needs
• Results in instructional improvements that will
  actually
• occur
• Can be re-directed frequently
• Has meaning for teachers (seen by teachers as a
• worthwhile use of their time)
• Use the scales on page 11 to rate the data
  conversations in your team, department, or school.

45
What should the new model look like?
School improvement is most surely and
thoroughly achieved when teachers engage in
frequent, continuous, and increasingly concrete
and precise talk about teaching practice . . .
adequate to the complexities of teaching, capable
of distinguishing one practice and its virtue
from another. --Judith Warren
Little
46
In other words . . .
A Classroom-Focused Improvement Process (CFIP)
47
Education After Standards
48
Goal of the New Process

• Frequent, continuous, and increasingly concrete
  and precise dialogue by school teams, informed by
  data

49
Myth 5
Because of the training faculty members have
received, they are usually highly skilled in the
data analysis process. Left on their own, most
teachers know enough about their students to
enable all of them to reach high standards.
50
Reality 5
School staff often lack training to conduct
effective data analyses. A consistent protocol,
based on true dialogue and the triangulation
of data, offers the best potential for teachers
to enable all students to reach high standards.
51
A Data Protocol
The New
Model
Classroom-focused improvement process
52
The new process needs to be built on

• 1. Dialogue

53
What Is True Dialogue?
In dialogue, a group accesses a larger pool of
common meaning, which cannot be accessed
individually. People are no longer primarily in
opposition, rather they are participating in
generating this pool of common meaning We are
not trying to win in a dialogue. We all win if
we are doing it right.
- Senge, The Fifth
Discipline (1990)
54
The new process needs to be built on

• 1. Dialogue
• 2. Protocols

55
What Is a Protocol?
A protocol consists of agreed-upon guidelines
for dialogue which everyone understands and has
agreed to that permit a certain kind of
conversation to occur, often a kind of
conversation which people are not in the habit of
having. Protocols build the skills and culture
necessary for collaborative work. Protocols
often allow groups to build trust by doing
substantive work together.
56
Why Use a Protocol?
In some educational organizations, protocols
may at first seem foolish an unwarranted
interference in ordinary business. The more
dysfunctional the organization, the stronger the
negative reaction may be.One could argue
thatcommunication precision, faithful
replication, and scripts would prove
counterproductive here. Dont we learn best by
just talking with each other?
McDonald, et al. The Power of Protocols
(2003), pp. 1,4.
57
NO!
Among educators especially, just talking may not
be enough. The kind of talking needed to educate
ourselves cannot rise spontaneously and unaided
from just talking. It needs to be carefully
planned and scaffolded. McDonald, et al. The
Power of Protocols (2003), pp. 4,5.
58
The new process needs to be built on

• 1. Dialogue
• 2. Protocols
• 3. Triangulation of
• Data

59
Triangulation of Data

• Use a variety of sources of data to identify the
  patterns.
• By combining multiple sources, we can overcome
  the weaknesses of individual data points and
  generate powerful insights not available from one
  source.

60
TEAM DATA DIALOGUE PROTOCOL MOVING FROM DATA
TO INCREASED STUDENT LEARNING DATA SOURCE(S)
__________________________________________________
_________ Step 1 Identify the questions to
answer in the data dialogue. Step 2 Build
assessment literacy. Define terms (if
needed). Step 3 Identify the big picture
conclusions from the data. Step 4 Identify the
patterns of class strengths and weaknesses (using
more than one data source, if possible).
Step 5 Drill down in the data to individual
students. Identify and implement needed
enrichments and interventions.
Step 6 Reflect on the reasons for student
performance. Identify and implement needed
instructional changes for the next unit.
61
CFIP Data Dialogue Protocol Formats

• One-page overview of the model, p. 15
• CFIP model with reflection questions, pp. 17-18
• CFIP model worksheets, pp. 19-23
• Reflection Guide to Instructional Changes,
• pp. 25-26
• Steps 4-6 of CFIP model, in another format, pp.
  27-29
• Examples of CFIP model as completed by school
  teams, pp. 30-42
• Take a few minutes to preview
• the next several pages.

62
TEAM DATA DIALOGUE PROTOCOL MOVING FROM DATA TO
INCREASED STUDENT LEARNING DATA SOURCE(S)
__________________________________________________
_________ Step 1 Identify the questions to
answer in the data dialogue. Step 2 Build
assessment literacy. Define terms (if
needed). Step 3 Identify the big picture
conclusions from the data. Step 4 Identify the
patterns of class strengths and weaknesses (using
more than one data source, if possible).
Step 5 Drill down in the data to individual
students. Identify and implement needed
enrichments and interventions.
Step 6 Reflect on the reasons for student
performance. Identify and implement needed
instructional changes for the next unit.
63
STEP 1 When Analyzing Data, Begin with a
Question.
All data analyses should be designed to answer a
question. Unless there is an important question
to answer, there is no need for a data
analysis.
64
Data Dialogues at Middleborough Elementary School
STEP 1 When Analyzing Data, Begin with a
Question.

• What can we learn about our students by looking
  at data, given our beliefs about the value of MSA
  and MSA scores?

65
STEP 2 Understand the Data Source.

• Build ASSESSMENT LITERACY with questions like
  these
• What assessment is being described in this data
  report? What were the characteristics of the
  assessment?
• Who participated in the assessment? Who did not?
  Why?
• Why was the assessment given? When?
• What do the terms in the data report mean?
• Be sure everyone on the team has clear and
  complete answers to these questions before
  proceeding any further.

66
Data Dialogues at Middleborough ES STEP 2 Build
Assessment Literacy
MSA

• Data from standards-based tests compared
• to data from norm-referenced tests

Classroom Assessment Literacy

• On-line Math Benchmarks
• Math Summatives
• HM Theme tests
• Daily Formatives

67
STEP 3 Look for the Big Picture Views in
the Data.

• Identify
• What do we see in the
• data?
• What pops out at us
• from the data?

68
Data Dialogues at Middleborough ES STEP 3
Getting the Big Picture

• Special education population in grades 3 and 4
• Depersonalize data with anonymous data walk.
• Case managers compare with IEP goals.
• Less desirable data displays were thrown out for
  future meetings. Others were modified to make
  more user-friendly.

69
(No Transcript)
70
(No Transcript)
71
STEP 4A Look for Patterns in a Single Data
Source.

• What patterns do you see over and over again in
  the data?
• What are the students strengths? What
  knowledge and skills do students have?
• What are their weaknesses? What knowledge and
  skills
• do students lack?

72
STEP 4B Look for Patterns in the Data from
Multiple Data Sources.

• What patterns do you see over and over again from
  multiple sources of data?
• What are the students strengths? What
  knowledge and skills do students have?
• What are their weaknesses? What
• knowledge and skills do
• students lack?

73
Data Dialogues at Middleborough ES STEP 4 Look
for Patterns

• Are there strengths and areas for growth that
  exist through most or all data sources?

74
STEP 5 Identify the Implications from the
Data Patterns for Students.

• What are the implications for enrichments from
  what you learn from the data?
• Which students need
• them?What should enrichments
• focus on?

75
STEP 5 Identify the Implications from the
Data Patterns for Students.

• What are the implications for interventions from
  what you learn from the data?
• Which students need interventions?What should
  interventions
• focus on?

76
Data Dialogues at Middleborough ES STEP 5
Implement Interventions

• Structural Adjustments
• Personnel (Instructional Assistants)
• Technology

• Specific Intervention Examples
• After school tutorial for specific students.
  (Skill deficits identified in data analysis were
  the targets of this intervention.)
• Teacher websites to address pervasive homework
  concern related to math review and reinforcement.
• Lunch Bunch to address number sense concerns.
• Actual change of groupings of students to better
  meet individual student needs.

77
Data Dialogues at Middleborough ES Effect of
Interventions
78
STEP 6 Identify the Implications from the
Data Patterns for Your Teaching.

• What are the implications from what you learn
  from the data for your instruction?
• Content focus
• Pacing
• Teaching methods
• Assignments

79
The Big Six of Data Analysis 1. Begin with a
question. 2. Understand the data source. 3. Look
for the big picture. 4. Look for patterns in the
data. 5. Identify and act on the
implications of the patterns for your
students. 6. Identify and act on the implications
of the patterns for your instruction.
80
Your Next Steps

• Unless you emerge from
• the data analysis process with a clear plan of
  action for your classroom, you have wasted your
  time.
• Implement your plan of interventions,
  enrichments, and changes in instruction.
• Collect your next set of data.

81
Is It Worth the Effort?

• Take a look at the following results.
• Then you tell us.

82
(No Transcript)
83
(No Transcript)
84
(No Transcript)
85
(No Transcript)
86
Caveats about CFIP

• It is a paradigm shift from the traditional
  lesson planning format.
• It is not easy, especially at first.
• Follow the steps faithfully until they become
  second nature.
• The CFIP is a guide until you make the process
  your own.
• Expect mistakes and imprecision in the data.
• The results are worth the effort.

87
Where does a school go from here in becoming more
data-driven?
DISCUSSION What drivers and barriers would you
see schools facing in implementing data
dialogues, such as the CFIP model (p. 48)?
88
Where does a school go from here in becoming more
data-driven?
DISCUSSION What role could you play in helping
schools overcome the barriers and move forward in
their data dialogue process (p. 49)?
89
Questions and Answers