DATA DRIVEN DECISION MAKING

1
DATA DRIVEN DECISION MAKING

• A Leadership Seminar for
• Pennsylvania Urban Academy

June 23-34, 2004
2
Some Basic Understandings
3
In solving student achievement problems, you
cant blame the kids!

• First of all, whatever the circumstances are, no
  one really cares. They want you to do your job
  no blame and no excuses.
• Second, whatever circumstances they bring are at
  least partially out of their control. Im sure
  they didnt wish those circumstances on
  themselves.
• Finally, our job is to overcome circumstances so
  we can do the job were authorized to do.

4
You cant blame the parents either!

• First of all, no one in public policy cares about
  their role in shaping student achievement. They
  have no public accountability for their childs
  academic learning.
• Second, if they could do better, most of them
  probably would. By and large, at least one parent
  wishes for their child something better than they
  currently have.
• Blaming them for their childs academic failure
  only allows the school to try and deflect its own
  professional responsibility, and that wont work.

5
Nobody has all the answers!

• None of us is genetically encoded to solve the
  almost intractable problems of educating all
  children to high levels of performance.
• However, everybody knows something, and the
  thing one person knows might be exactly what
  another person needs to know.
• The trick is to figure out how to tap into
  other peoples experiences. That can start by
  your encouraging them to tap into your own.

6
Beating a dead horse harder will not bring the
horse to life!

• If something isnt working as it should, spending
  more time and energy doing the same thing is
  ineffective.
• Remember the organizational definition of
  insanity.
• Most good athletic coaches adjust to their
  talent, to their successes, and to their past
  failures. The difference between consistent
  winners and all others is in their passion for
  learning how to do ordinary things differently.

7
There is no safety net in the business of school
improvement.

• If youre trying to navigate the process of
  change as a risk-free enterprise, youre in the
  wrong business.
• Remember the Gretsky principle You will miss
  100 of the shots you never take.

8
There are not enough resources and there never
have been!

• Any thinking that the most possible solution lies
  in more (of anything) will be a problem before
  it ever gets off the ground.
• Most plausible solutions will require
  reallocating existing resources to achieve
  heretofore elusive results.
• The basic question is How can you get the best
  results with the resources you know you can
  control? If you cant control them, you might as
  well not have them.

9
Data change nothing People change everything!

• Information is inert. Its only purpose is to be
  useful. Otherwise, it just sits somewhere,
  occupying space.
• Three kids are sent to you because of their
  directing profanity towards their teacher. Will
  all three of them get the same treatment from
  you? What will differentiate punishments?
• School improvement has less to do with what you
  know than what you do with what you know.

10
Ignorance is no excuse!

• If doing heavy instructional leadership lies
  outside your comfort zone, not doing it is no
  longer an option. Either learn how or delegate
  and empower it to someone else.
• If a teacher teaching in a tested grade level
  lacks the ability to manage curriculum content
  mastery, either help him/her learn how to or find
  this person another less obtrusive assignment.

11
Shift happens!

• Whatever you think current reality is doesnt
  make it tomorrows reality. Decisions others make
  have a way of altering your status quo, sometimes
  in profound ways.
• Consider the effect of any serious demographic
  shift in your schools population on the schools
  overall achievement profile.
• Are there any probable consequences if you had to
  replace a gifted and experienced 5th grade
  teacher with a lateral entry rookie who got into
  teaching because her dot.com company was
  downsized?

12
So, what is this fuss about data all about?
How do you decide whether or not you can afford
to put a whole new roof on the house or simply
shingle over the old ones?
What factors will influence your decision as to
whether or not your 15 year old daughter can go
to a friends Halloween party and spend the night
there?
You see, while the information you have is
important, it is only important to help you make
the best decision possible under whatever
circumstances exist. When circumstances or the
data change, so might your decision.
13
Lets look at some examples of data and you tell
me the relative value of each.
Assume youre the coach of a championship caliber
high school basketball team. As you begin
planning for the next season, you have these
data. Your goal is nothing less than the
Conference Championship.
Team averaged 2 hours of practice per game, 75
on offense Each team member shot about 30 free
throws per practice Teams scoring average was
66.5 points per game they gave up an average of
63.6 points per game The starting guards averaged
about 27 minutes of playing time per game, out of
possible 36 minutes
The team averaged 12.6 turnovers per game
(Conference champion was 8.2) Teams free throw
percentage was 70.3 (Conference champion was
75.2) Team averaged 24 rebounds per game, 6 were
offensive (Conference champion was 27) Team
committed 46 of its fouls in the 4th quarter
Teams record this past season was 16-10 Finished
in 2nd place in the Western Division of the
Conference Record for games on the road was 5-8
(Division average was 6-7) The team lost in the
2nd round of the Division Playoffs
14
Lets look at some other examples of data and you
tell me the relative value of each.
Assume youre the principal of a high school
committed to getting better. You and your school
improvement team have these data from last year.
You want nothing less than to be regarded as the
highest performing high school in the region.
11th Graders scored at the 56th percentile in
reading and 62nd percentile in math on the
PSSA 62 of the students taking Algebra I scored
at proficiency or higher 68 of the 11th grade
Asian students taking the Reading Exam scored at
proficiency or higher The schools overall
proficiency percentage composite was 58.4
The analysis of teacher-assigned grades over the
previous academic year showed 44 of all
students received As and Bs, while 22 received
Ds or Fs. The teacher turnover rate for the
last three years has averaged 22 each year. 25
of last years teachers did not hold
tenure. About 40 of the teachers either are not
familiar or they ignore the Pennsylvanias
Curriculum Standards.

In the reading test, 48 of the students missed 3
or more test items dealing with finding key facts
that support a main idea. 70 of the students not
making proficiency in Algebra I were not
proficient in math in the 8th grade PSSA. Females
outscore males in English by almost 11 The
school has had 3 Merit Scholarship winners in the
last 5 years. It used to average that many every
year.
15
Okay, if using data is so great, why is it done
so little or so poorly?

• EXCUSES OFTEN HEARD
• We know what the problems are were here every
  day.
• Listen, we cant control who gets sent to the
  school. We just take who they send us and do the
  best we can.
• We dont have the proper technology to do the
  curriculum well. If I just had ____ Id be a lot
  more successful.
• Look, Algebra is Algebra. Whats data going to
  tell me that I dont already know?
• If I have to do data analysis on everything I
  teach, when will I have time for planning?
• Its not my job.

• REASONS OFTEN CITED
• The schools culture doesnt value data. The
  principal never shares it.
• Staff lacks training on how to do data analysis.
• It is perceived as too time consuming.
• It is seen as a weapon to exploit vulnerability.
• Often, the tools for analysis do not accompany
  the data themselves. Testing coordinators dont
  share.
• Because it is seen as painting an incomplete
  picture, it is dismissed.
• There are not a lot of good, practical models to
  emulate.

16
Lets summarize the importance of data in
improving schools and elevating the qualities of
the decisions we make

• Data are important because
• They help us identify the most important ways to
  help students.
• They help us get past guesses and hunches by
  making us confront facts.
• They give us a reality check on how we are
  actually doing.
• They allow us to demonstrate our successes
  clearly and without prejudice.
• They help us make better decisions by allowing us
  to focus on priorities and not just preferences.

• Data are used most effectively when
• They are correlated with other sources of data,
  some of which may not be standardized.
• They are aligned to the mission of the school and
  the standards the school is bound to.
• They draw from multiple perspectives and multiple
  sources so a more complete picture of reality can
  emerge.
• They enable school leaders to concentrate
  organizational attention on things that matter
  most.
• They promote systemic change. We are all a part
  of the solution

17
LETS LOOK AT SOME REAL DATA AND THEIR
IMPLICATIONS
What follows are several pages of data on a
particular school. This is a real school, and the
data are as real as the school is. The only thing
that is faked is the name of the school.

18
Central City Elementary Fifth Grade
ReadingPercentage Making Proficiency or Higher
19
Central City Elementary Fifth Grade
MathematicsPercentage Making Proficiency or
Higher
20
Anywhere Elementary (Reading and
Mathematics)Percent Proficient by Grade Level
21
Anywhere Elementary (Reading and
Mathematics)Percent Proficient by Grade Level
22
Anywhere Elementary (Reading and
Mathematics)Percent Proficient by Grade Level
23
Anywhere Elementary (Reading and
Mathematics)Percent Proficient by Grade Level
24
Anywhere Elementary (Reading and
Mathematics)Percent Proficient by Grade Level
25
Looking for other Data Sources to help the School
Improvement Processes

• School Data for Students
• Home Data for Students
• Teacher Preparation Data
• Teacher Performance Data
• District Goals and Priorities
• Community Goals and Needs for School

26
Pulling the D3M Process Together
Step one Check Your Alignment
Step two Formulate questions that needs answers

Step three Gather the data
Step four Answer your questions
Step five Communicate your findings
Step six Plan your school improvement plan
27
Pulling the D3M Process Together
Step one Check your Alignment
Success in any form of accountability depends in
large measure on how well the taught curriculum
is aligned with the tested curriculum. In
Pennsylvania, it is a safe presumption that the
PSSA tests the states curriculum standards. If
that is true, then it makes sense to get a clear
and consistent alignment between these standards
and what teachers actually teach and what kids
are expected to learn. Your job as the schools
instructional leader is to ensure that such an
alignment is in place. In high stakes
accountability systems (code for NCLB), you need
to (1) make sure teachers know the standards and
can articulate their alignment to the standards,
(2) monitor both lesson plans and lesson
execution to extend the power of your
instructional supervision, and, (3) give focused
and constructive feedback on continuous
improvement, using the teachers classroom
assessment data to frame the focus for these
feedback conversations. This process not only
applies to the tested grades, it applies to those
grade levels whose curriculum mastery contributes
to success in the tested grade. Make sure the
alignments in grades K-4 are as consistent as
they are in grade 5, for example.

28
Pulling the D3M Process Together
Step two Formulate questions that needs answers

• Examples of Other Data Sources to Help Generate
  Questions
• School Improvement Plan (e.g.)
• You said you were going to improve math and
  reading instruction so student learning would
  improve in those areas. Did you?
• The SIP specifically indicated that the school
  would integrate thinking maps into the curriculum
  to improve student thinking. Were they? Did
  student thinking improve?
• School History(e.g.)
• The school has struggled for a long time with
  transition difficulties most 9th graders
  experienced. The school established a special 9th
  grade academy to help overcome this problem two
  years ago. What have been the effects of this
  academy? Has there been a positive effect on 11th
  grade achievement scores.
• Recent events (e.g.)
• This past year, the percentage of Hispanic
  students doubled from the previous year, making
  them a bona fide sub-group for NCLB
  consideration. As a defined ethnic group, how did
  they do in achievement? Attendance? Disciplinary
  referrals?

29
Pulling the D3M Process Together
Step three Gather the data
Your primary purpose is to answer your questions.
You might find other interesting stuff along the
way, but you need to stay focused. Remember the
power of multiple measures to get multiple
perspectives.

Gather the data readily at your disposal Student
Data Teacher Data Community Data Leadership Data
Secure the data you dont have but need to answer
your questions (e.g.) Survey students in 9th
grade academy about their experiences Do parent
focus groups to determine their impressions of
school safety Do surveys with teachers on their
comfort and use of thinking maps in their
curriculum
30
Pulling the D3M Process Together
Step four Answer your questions
This may be the most straightforward of the
steps, but it is sneaky. You will have enough
data to answer 200 or more questions, but you
cant work on two hundred separate school
improvement initiatives. Stay focused on the
questions most likely to produce and let others
take your data and mine it for other insights and
possibilities. This is where cooperation with a
university can be helpful.

If you still have unanswered questions, chances
are you dont have the right assortment of
information to answer them. What do you need to
know that you dont have? Can you get it? At what
cost? What will happen if this question is
carried over until you have the data you need to
answer it.
31
Pulling the D3M Process Together
Step four Answer your questions
For example, you wanted to improve instruction in
reading and math to promote student learning.
What did you do to improve instruction? Did you
provide professional development? Was it used?
How do you know? How well was it used? What
evidence do you have to support this conclusion?
Did you assign different teachers to teach in
these areas? Why them? Do you have any evidence
that this strategic move had the desired effect?
Did students improve their learning in these two
areas? What are your indicators to support this
conclusion (student grades, scores on testlets,
improved attendance, fewer referrals, better
attention to homework, quality of student work
handed in, standardized test scores, etc.)?

You also wanted to integrate thinking maps into
the curriculum to improve student thinking. What
did you do to make this happen? Was their
thinking maps staff development? Who received the
training? Did they integrate thinking maps into
their curriculum? How do you know? How
effectively did they do this integration? What
evidence supports your conclusion? Did students
use them? What evidence supports that? Did the
quality of student thinking improve? How do you
know?
32
Pulling the D3M Process Together
Step five Communicate your findings
Internally These findings and conclusions need to
form the basis for a series of high quality
professional conversations with the faculty and
with some particular groupings within the
faculty. The principal and the entire School
Improvement Team should have specific
responsibilities to deliver a series of
data-based reports that will lead to teacher-led
work groups to address one or more of the
concerns raised from the data analysis. These
presentations must be carefully crafted, since
not all members of the faculty will be equally
conversant with the language and concepts of data
driven decision making. You also have another
internal audience. Your superintendent and your
immediate supervisor (if different) need to know
what you now now about your school and its school
improvement efforts. They also need to know what
your likely plans for improvement are. You have
an opportunity to acquire administrative support
and understanding by being proactive with your
knowledge. In fact, the only other significant
way to get additional help is to fail, and then,
the mountain is much higher and harder to climb.

33
Pulling the D3M Process Together
Step five Communicate your findings
Externally Each school has community and business
leaders who are key stakeholders in this business
of school improvement. Many are also probably
pretty conversant in the importance of data, and
will be good customers of the messages of
improvement you can deliver. Often, they rely on
information sources outside the schools control
to learn about their schools. You can change that
by taking what you know to Chamber meetings,
Service Club gatherings, education reporters, and
well-established social clubs who have
educational agendas. If you fail to inform them,
theyll find information elsewhere, and you can
then only react, and thats a position of
weakness, not strength. Remember to craft this
message as jargon-free as possible. You dont
want to lose them. Then, there are the parents.
They too need to know how their childs school is
doing, but you have to craft this message
differently. No jargon, please. The range of
language sophistication is likely to be pretty
wide, and your objective is to get them to
understand complexity in as simple a way as you
can communicate. Make information dissemination
as short as possible and rely on questions to
articulate additional information.

34
Pulling the D3M Process Together
Step six Plan your school improvement plan
Now that you and your School Improvement Team and
your faculty have done all this work, and youve
communicated the results to those who need to
know, the important part now kicks in. The
decision making part of D3M is now front and
center, and the data have helped you identify the
areas of greatest accomplishment (Hooray!!) as
well as the areas of greatest need. What follows
are some guidance for how to manage the
decisioning process. First, keep in mind that you
probably got to this point by working with key
staff, and they need to have a voice in the
decisions. You are still the principal, but their
insights are important too. Second, recognize
that you and the school cant do it all, so you
need to concentrate on those areas most likely to
provide the best growth. Focus is important! Keep
the main things the main things. For example

35
Pulling the D3M Process Together
1. Define your areas of need identified by
your data analysis. You will probably have more
things than you can possibly accomplish in a
single year or even two, but they all have to be
made visible so everyone can see them.
2. Because you and your folks cant do all
these things and do them well, you have to set
priorities. Rank ordering the possibilities is
very helpful, so long as everyone understands
what the basis of the rank ordering is. If you
tell them the critical variable is expense, they
will rank on affordability. If you tell them the
ranking is to be based on their perception of
which will deliver the most student benefit, they
will rank on that variable. Assigning numerical
values to peoples rankings will yield a
prioritized list. However, it is important to get
your folks to affirm that the priorities actually
represent their collective intentions. If they
agree, then pick the top 1-2 for that year. This
doesnt mean that people dont pay attention to
those other things it only means that the focus
for improvement will be on those 1-2 things,
because the potential for greatest improvement in
student benefit lies there.

36
Pulling the D3M Process Together
3. Now that you have your 1-2 priorities, you
need to develop the real planning. What will you
and the school actually do to accomplish these
priorities? What needs to take place (e.g.
training, materials purchase, special approvals,
special consultant, etc.) in order for any of
these strategies to happen successfully? What
timelines are reasonable for each of the
strategies to be in place? What leadership
assignments have to change to make sure this
strategy is on track?

• Once you know the 1-2 things youre going to
  focus on, and what youre going to do to make
  these priorities become real, you have to
  allocate the necessary resources to each priority
  to ensure its success. This an often neglected
  step in the school improvement planning process,
  yet can singly undermine any chance for success.
  Is new training required? How much will it cost?
  When will it be done? Where will the money come
  from? Is that assured? How long will it take to
  do the training? Are there materials costs? How
  much? Where will the money come from? Is that
  assured? Who will be assigned to get the training
  and implement the strategy? Are they the best
  persons to do this? Do they work together well?
  Will this strategy require any additional use of
  space? What kind? Is it available? Are there
  space modifications to be made? How much? Where
  will the money come from? Is that assured?

37
Pulling the D3M Process Together
5. OK! Now, you and your folks need to
define, for each of these priorities, what
Acceptable Progress or Good Progress looks
like. It is not enough to develop a strategy,
create a timeline, and assign resources. Everyone
involved has to know what the expectations are.
How can anyone know whether things are going as
they should be going? What kinds of things should
people be able to determine so that they can feel
theyre on the right track? Perhaps a rubric
would help in an instructional area, or a
benchmark from another school doing this
successfully in an area of professional
development would be useful. People need to know
that they are doing the things they need to be
doing and that what theyre doing is producing
the results they are supposed to produce.
Otherwise, people will expend a lot of energy to
end up frustrated, and thats a poor motivation
for sustained change and improvement.

6. Finally, someone on the staff has to be
accountable for each priority. Yes, the principal
is accountable for all of them, but someone else
should have both the assignment and the authority
to make sure progress is going in the right
direction. Someone has to worry about a
prioritys success, and has to have a sense of
accountability for that success. It could be an
assistant principal, a lead teacher, a department
head, a media specialist, a guidance counselor,
etc. But, someone has to be in charge of making
sure the good things that are supposed to happen
actually happen.
38
So, why is the use of data so important in our
work and in our lives?

• They replace guesses with facts, and allows
  decisions to be based on those facts.
• It allows us to get past symptoms and try to get
  at causes.
• They focus on reality, not hyperbole. Do the data
  support the rhetoric? Data help us answer the
  How do I know? question.
• They help determine if previous decisions are
  accomplishing what theyre supposed to be
  accomplishing.
• They allow you to tell your story without
  sounding like its special pleading.