DataBased Decision Making: How to Decide What Needs to be Done

1
Data-Based Decision Making How to Decide What
Needs to be Done

• Colorado School Counselors Association
• April 20, 2006
• John Carey,
• National Center for School Counseling Outcome
  Research

2
Factors Leading to Need Data Skills in School
Counseling

• Accountability Movement
• Standards-Based Education Movement (NCLB)
• School Reform Movement
• New Models of School Counseling Practice
• Education Trust Transforming School Counseling
  Initiative
• ASCA National Model

3
A Model of Evidence-Based Practice
Practitioners Individual Expertise
Best Evidence
Client Values and Expectations
EBP
Adapted from Shlonsky and Gibbs (2004), Will
the Real Evidence-Based Practice Please Stand Up?
Teaching the Process of Evidence-Based Practice
to the Helping Professions. In Brief Treatment
and Crisis Intervention, 4(2), 137-153.
4
Evidence-Based Practice in School Counseling
Intervention and Program EvaluationCompetence
Outcome Research Competence
Assessment of Intervention Targets and
Goals
EBP
5
Evidence-Based Practice in School Counseling
Intervention and Program EvaluationCompetence
Outcome Research Competence
Assessment of Intervention Targets and
Goals Knowing what is needed
EBP
6
Evidence-Based Practice in School Counseling
Intervention and Program EvaluationCompetence
Outcome Research Competence Knowing what
generally works
Assessment of Intervention Targets and
Goals
EBP
7
Evidence-Based Practice in School Counseling
Outcome Research Competence
Intervention and Program EvaluationCompetence
Knowing how studentschanged
Assessment of Intervention Targets and Goals
EBP
8
Evidence-Based Practice in School Counseling
Intervention and Program EvaluationCompetence
Outcome Research Competence
Assessment of Intervention Targets and
Goals
EBP
9
Terms

• Date-Based Decision Making use of school data
  to determine problems that need to be addressed
• Outcome Research use of the scientific method to
  discover generalizable truth about the
  effectiveness of interventions.
• Evaluation use of the scientific method to
  improve local decision-making by determining
  whether it was likely that an intervention was
  resulted desired changes in behavior

10
How These Fit Together

• Data-Based Decision Making
• 8th graders in Carey Middle school are doing
  poorly on the state test
• Outcome Research
• Research yields strong evidence that Student
  Success Skills can increase test score by
  teaching self-management and enhancing motivation
• Program Evaluation
• Carey Middle School students learn self
  management and improve grades and test scores
  after SSS

11
Seven Steps in Using Data in Advocacy and Systems
Change
1. Describe the Problem
2. Generate Vision Data
3. Commit To Benchmarks
4. Identify Places to Intervene First Order
Change? Second Order Change?
6. Evaluate Implementation
5. Select Interventions
7. Monitor Problem Data
12
Seven Steps in Using Data in Advocacy and Systems
Change
2. Generate Vision Data
3. Commit To Benchmarks
1. Describe the Problem
4. Identify Places to Intervene First Order
Change? Second Order Change?
6. Evaluate Implementation
5. Select Interventions
7. Monitor Problem Data
13
Seven Steps in Using Data in Advocacy and Systems
Change
1. Describe the Problem
2. Generate Vision Data
3. Commit To Benchmarks
4. Identify Places to Intervene First Order
Change? Second Order Change?
6. Evaluate Implementation
5. Select Interventions
7. Monitor Problem Data
14
Seven Steps in Using Data in Advocacy and Systems
Change
1. Describe the Problem
2. Generate Vision Data
3. Commit To Benchmarks
4. Identify Places to Intervene First Order
Change? Second Order Change?
6. Evaluate Implementation
5. Select Interventions
7. Monitor Problem Data
15
Seven Steps in Using Data in Advocacy and Systems
Change
1. Describe the Problem
2. Generate Vision Data
3. Commit To Benchmarks
4. Identify Places to Intervene First Order
Change? Second Order Change?
6. Evaluate Implementation
5. Select Interventions
7. Monitor Problem Data
16
Seven Steps in Using Data in Advocacy and Systems
Change
1. Describe the Problem
2. Generate Vision Data
3. Commit To Benchmarks
4. Identify Places to Intervene First Order
Change? Second Order Change?
6. Evaluate Implementation
5. Select Interventions
7. Monitor Problem Data
17
Seven Steps in Using Data in Advocacy and Systems
Change
1. Describe the Problem
2. Generate Vision Data
3. Commit To Benchmarks
4. Identify Places to Intervene First Order
Change? Second Order Change?
6. Evaluate Implementation
5. Select Interventions
7. Monitor Problem Data
18
Seven Steps in Using Data in Advocacy and Systems
Change
1. Describe the Problem
2. Generate Vision Data
3. Commit To Benchmarks
4. Identify Places to Intervene First Order
Change? Second Order Change?
6. Evaluate Implementation
5. Select Interventions
7. Monitor Problem Data
19
Seven Steps in Using Data in Advocacy and Systems
Change
2. Generate Vision Data
3. Commit To Benchmarks
1. Describe the Problem
4. Identify Places to Intervene First Order
Change? Second Order Change?
6. Evaluate Implementation
5. Select Interventions
7. Monitor Problem Data
20
Step 1 Describe the Problem

• Disaggregation Tools
• Triangulation Tools

21
Disaggregating Data

• Comparing and contrasting the performance of
  different groups of students on some outcome
  measure. (Results Data)
• Comparing and contrasting the rates of
  participation of different groups of students in
  school programs and activities related to
  outcomes. (Process Data)
• Comparing and contrasting the perceptions of
  different groups of students on factors related
  to outcomes. (Perceptual Data)

22
Disaggregation Categories

• Race
• Gender
• Limited English Proficient
• Academic/Vocational Track
• English Language Learners
• Free or Reduced School Lunch
• Mobility
• Special Needs
• Achievement Quartile
• Grade

23
Disaggregating DataAchievement Outcomes2002
10th Grade MCAS English Language Arts
24
Disaggregating DataAchievement Outcomes2002
10th Grade MCAS English Language Arts
25
Disaggregating DataAchievement Outcomes2002
10th Grade MCAS English Language Arts
26
Disaggregating DataAchievement Outcomes2002
10th Grade MCAS Mathematics
27
Disaggregating DataAchievement Outcomes2002
10th Grade MCAS Mathematics
28
Disaggregating School Process Data Attending
Gateway ClassesPartnership District Data
29
Disaggregating Perceptual DataSchool Climate
SurveyMinority Student Achievement Project
I work hard in school because the teacher demands
it.
30
Disaggregating Perceptual DataSchool Climate
SurveyMinority Student Achievement Project
I am happy to be at this school.
31
Disaggregating Perceptual DataSchool Climate
SurveyMinority Student Achievement Project
How many teachers know how capable you are to do
well in school?
32
Triangulate

• Use three different independent sources of data
  to describe the problem.
• Results Data
• School Process Data
• Perceptual Data

33
Triangulating
High percentages of African American Students
Fail 10th grade MCAS.
School Process Data
THE PROBLEM
Perceptual Data
34
Triangulating
High percentages of African American Students
Fail 10th grade MCAS.
Low percentages of African American Students Take
8th Grade Algebra and Algebra 2
THE PROBLEM
Perceptual Data
35
Triangulating
High percentages of African American Students
Fail 10th grade MCAS.
Low percentages of African American Students Take
8th Grade Algebra and Algebra 2
THE PROBLEM
Many African American Students Report Teachers Do
Not Think They Are Able To Go To College.
36
Seven Steps in Using Data in Advocacy and Systems
Change
1. Describe the Problem
2. Generate Vision Data
3. Commit To Benchmarks
4. Identify Places to Intervene First Order
Change? Second Order Change?
6. Evaluate Implementation
5. Select Interventions
7. Monitor Problem Data
37
Step 2 Generate Vision Data

• Successful Vision Data is
• Agreed upon by most members of the system
• Concrete and specific
• Measurable
• Related to student learning outcomes
• Ambitious
• Attainable
• Tied to a deadline
• Related to Values and Passion

38
Vision Data MCAS Pass Rates for History and
Social Studies in 2003
39
Vision Data MCAS Pass Rates for History and
Social Studies in 2008
40
Seven Steps in Using Data in Advocacy and Systems
Change
1. Describe the Problem
2. Generate Vision Data
3. Commit To Benchmarks
4. Identify Places to Intervene First Order
Change? Second Order Change?
6. Evaluate Implementation
5. Select Interventions
7. Monitor Problem Data
41
Step 3 Commit to Benchmarks

• The term benchmarking was originally used by
  land surveyors to mark reference points
  (buildings, rocks, landmarks) measuring the
  distance from a particular spot. Setting a
  benchmark told you how far away you were from a
  certain reference point (Vision Data).

42
Commit to Benchmarks

• What are the yearly markers we will use to
  determine how far away we are from our goals and
  vision? (Expected Results Over Time)
• How will we know that we are getting closer?
  What are our measurements?

43
Commit to Benchmarks
44
Commit to Benchmarks
45
Seven Steps in Using Data in Advocacy and Systems
Change
1. Describe the Problem
2. Generate Vision Data
3. Commit To Benchmarks
4. Identify Places to Intervene First Order
Change? Second Order Change?
6. Evaluate Implementation
5. Select Interventions
7. Monitor Problem Data
46
Step 4 Identify Places to Intervene

• Level
• Student
• Peer
• Teachers
• School Climate and Policies
• Family
• Community

47
Seven Steps in Using Data in Advocacy and Systems
Change
1. Describe the Problem
2. Generate Vision Data
3. Commit To Benchmarks
4. Identify Places to Intervene First Order
Change? Second Order Change?
6. Evaluate Implementation
5. Select Interventions
7. Monitor Problem Data
48
Step 5 Select Interventions

• Identify Possible Interventions
• Individual Interventions
• Systemic Interventions
• Align with Levels (Student thru Community)
• Align with Evidence Base
• Identify Expertise and Learning Needs
• Identify Community Resources
• Identify Possible Roadblocks and Resistance
• Write Action Plan

49
Seven Steps in Using Data in Advocacy and Systems
Change
1. Describe the Problem
2. Generate Vision Data
3. Commit To Benchmarks
4. Identify Places to Intervene First Order
Change? Second Order Change?
6. Evaluate Implementation
5. Select Interventions
7. Monitor Problem Data
50
Step 6 Evaluate Intervention

• Formative EvaluationProcess
• Monitor Implementation
• Correct Problems
• Summative Evaluation
• Perceptual Data
• Results Data

51
Seven Steps in Using Data in Advocacy and Systems
Change
1. Describe the Problem
2. Generate Vision Data
3. Commit To Benchmarks
4. Identify Places to Intervene First Order
Change? Second Order Change?
6. Evaluate Implementation
5. Select Interventions
7. Monitor Problem Data
52
Step 7 Monitor Problem Data

• Evaluate Results Over Time
• Alter Implementation Strategy
• Stay the Course
• Celebrate
• Disseminate
• Publicize

53
Questions and Discussion
54
National Center for School Counseling Outcome
Research

• Thank You

www.cscor.org