Data-based Decision Making:

1

• Data-based Decision Making
• What to Know?
• and
• Who Needs to Know It?

Jack States Randy Keyworth Ronnie Detrich
2
Why Systematize Data-based Decision Making?

• To Make Better Decisions
• Provide Accountability
• Guide Quality Improvement Efforts
• Assess the Effectiveness of Teaching

3
Federal Mandates NCLB
Employment Regulations
Unions
School Safety
District Reforms
Special Education Regulations
State Reforms
Parental Expectations
Local School Improvement Plans
analysis paralysis, or the process of being
over-stimulated by data... Joe McKendrick (2003)
4
Why Are We Overloaded?

• Quantity of data
• Lack of alignment across levels
• Dysfunctional organization of systems
• Insufficient prioritizing
• Poor training
• Competing contingencies
• Poor use of technology
• Insufficient resources

5
Organize Decision Making 1
Information
Raw Data
Data-Based Decisions
Data-Based Decision
Informed Action
6
Decision Makers - The People2

• The Same Rules That Apply To Implementation
• Capacity
• Motivation
• Time
• Technical resources

7
Ask the Right Questions3

1. What are you going to do with the data?
2. Can the data answer the stakeholders questions?
3. Are the data an indicator of key elements (input,
   process, and outcomes)?
4. Are the data objective, measurable, and reliable?
5. Who needs to interact with the data?
6. Does the data drive decisions for improving the
   process?
7. Are the data provided timely?
8. Are the proper technologies available for storage
   and delivery?
9. Are the data systems capable of making queries
   (drill-down)

Wayman, J., Stringfield, S. Yakimowski, M.
(2004) Software Enabling School Improvement,
8
Use Data That Drives The System4

• Three Categories of Data
• Outcome Data - Are we achieving our Goals?
• Process Data - Is the system working?
• Satisfaction Data - Are we meeting stakeholders
  expectations?

9
Outcome and Process Data

• Formative
• Data used to adjust instructional practices to
  maximize individual students learning, to gage
  progress, and direct instructional interventions.
• Lesson Quiz
• Chapter Test
• Curriculum Based Measures (CBM)
• Dynamic Indicators of Basic Early Literacy Skills
  (DIBELS)

• Summative
• Data collected after instruction has been
  completed, at the end course, or annually and
  used to make strategic program adjustments,
  curriculum decisions, and improve instructional
  practices.
• SAT, ACT
• Standardized test
• Exit Exams
• Graduation/Drop

• Process
• Data collected on systems that support
  instruction.
• Demographic information
• Finances
• Teacher credentials
• Hiring
• Facility safety
• Treatment integrity
• School improvement plans

10
Understand How Your System Works5
Input
Performance
Outcomes
Rummler Brache, (1990) Improving Performance
11
Federal
State
Alignment the Data Across the System 6
District
School
Classroom
Students
12
Federal
State
District
School
Classroom
Process Outcome Support, Teaching and
Aggregated Student
Student
Student Outcome Formative and Summative
13
What is a Key Indicator? Discrete pieces of
information that let us know how the system is
running.
Key Indicator Report - 7
What It Does What It Does Not Do
Identifies status or trends But, not a diagnostic tool
Indicates system going well or poorly But, does not identify cause
Tells you where to begin to look But, does not prescribe solutions
Thermometer
Dow Jones
14
Create a Key Indicator Report

• Less is more Data must be provided in less than
  two pages - limit items
• Staff do not have time
• Staff do not have the training
• Staff do not have the motivation
• Data can be disaggregated (drill down
  capability)
• Display data in easily understandable
• Provide Meaningful Comparisons (status and trend)
• Identify current performance
• Identify progress over time
• Compare to standards
• Compare to peers

15
When Selecting Indicators Ask One Question!

• Will it Make the Boat Go Faster?
• (Peter Blake - 1995 New Zealand Yachtsman - Won
  the Americas Cup)
• What are the critical Outcomes required of your
  job?
• What is the critical Process required of your job?

16
Indicator System Selection Criteria

• Functional Value
• Confidence
• Timely
• Measurable
• Numerical
• Comparable
• Within available resources
• Align the data across levels
• Informative Value
• Indicative of critical outcomes, process, and
  input
• Be aware - paper or performance compliance
• Predictive of multiple factors
• Data can be disaggregated Drill down
• Communicative Value
• Less is more Gate keep and prune
• Utilize graphic displays
• Display simple to understand convey complex
  issues but simply

17
Key Indicator ReportKennedy Elementary School -
2009
Report Item Sept Oct Nov Dec Jan Feb 2008 YTD Ave.
Outcome Reading - Words/minute to grade level 75 79 81 82 85 87 95
Process Treatment integrity check - items met per observation 60 80 80 50 90 90 95
Input Fully credentialed teachers - teachers current credential 80 80 80 90 90 90 100
18
IndicatorReport(Sample) AchievementGAPStude
nt AttractionEngagementRetention Teacher
RetentionFunding Equity
Celio,B. and Harvey, J. (2005) Buried Treasure
19
The End

• Resources
• Data Based Decisions - The Tyranny of Small
  DecisionsOrigins, Outcomes and Proposed
  Solutions - (2000) Bickel, W., Marsch, L.
• Data Based Decision Model - From Data to Wisdom
  Quality Improvement Strategies Supporting Large
  Scale Implementation of Evidence-Based Services -
  (2005) Daleiden Chorpita
• Data Based Decision Models - Making Sense of
  Data-Driven Decision Making in Education - (2006)
  Marsh, J., Pane J, Hamilton, L.
• Effective Dashboard Designs - http//www.slideshar
  e.net/hursman/effective-dashboard-design-presentat
  ion?fromemailtypeshare_slideshowsubtypeslides
  how
• Key Indicators Systems - Celio,B. and Harvey, J.
  (2005) Buried Treasure
• Performance Improvement - Rummler, G., Brache,
  A., (1990) Improving
• Performance
• Software and Hardware - Wayman, J., Stringfield,
  S. Yakimowski, M. (2004) Software Enabling
  School Improvement Through Analysis of Student
  Data