Fast Track Program Evaluation Using Assessments Diagnostically

1
Fast Track Program Evaluation Using Assessments
Diagnostically

• Philip A. Streifer, Ph.D.
• Bristol Superintendent of Schools UCONN
  Executive Leadership Program
• Yvel Crevecoeur
• Doctoral Student, Department of Educational
  Psychology, program in Special Education

2
Primary Issues

• Bristol an urban leader in reform
• Next phase of development program evaluation and
  focus on instructional excellence
• Primary reading intervention program
  Fundamentals of Literacy Development (FOLD)
• Rising CT AYP Standard for 2008 Need to track
  student gains over time vs. NCLB/AYP year by year
  analysis

3
Pressures

• 181 First Days of School
• Growing Poverty Rate
• Manage Expectations in City
• Article for upcoming Chamber Views
• Ongoing speaking and publications on challenges
• Focusing work of 1000 people
• Rising AYP Standards and City confidence in
  schools can we get over the bar?

4
Challenges

• Migration
• Economically Disadvantaged
• Rising State NCLB Standards for 2008
• Reading 68 to 79 proficiency
• Math 74 to 82 proficiency
• Special Education Placements and Costs

5
181 First Days of School One Schools Experience

• Transfers In

• Transfers Out

• July-Aug- Sep 53
• October 16
• November 11
• December 4
• January 24
• February 12
• March 11
• April 1
• May 0
• June 0
• TOTAL 132

• July-Aug- Sep 61
• October 8
• November 14
• December 3
• January 8
• February 12
• March 10
• April 2
• May 3
• June 1
• TOTAL 122

GRAND TOTAL for ONE YEAR 254 Children in and
out/School Population /-350
6
(No Transcript)
7
Data Driven Reform and Improvement

• Past, Present and Future

8
The Bristol Accountability Initiative
Data-Driven Decision Making
MAKING STANDARDS WORK
9
CT Mastery Test and CAPT

• CMT new version for 2005-06
• CAPT new version for 2006-07
• Both are tougher tests
• Each year NCLB/AYP standard rises
• NCLB/AYP standard for 2007-08 (spring 2008) is
  rising dramatically

10
(No Transcript)
11
Matched Cohort Data (2006 to 2007) Writing
12
(No Transcript)
13
District CAPT 2007 Results
Areas highlighted in yellow show where Bristol
exceeded State averages.
14
Performance Level Graph District vs. State
Averages
15
Gains Analysis Matched Cohorts

• Grade 3 to 4 (2006 to 2007)
• Math Expected Gain
• Reading Moderate Gain Beyond Expected
• Writing Expected Gain
• Grade 4 to 5 (2006 to 2007)
• Math Expected Gain
• Reading Moderate Loss Below Expected
• Writing Expected Gain

16
Gains Analysis Matched Cohorts

• Grade 5 to 6 (2006 to 2007)
• Math Expected Gain
• Reading Moderate Gain Beyond Expected
• Writing Expected Gain
• Grade 6 to 7 (2006 to 2007)
• Math Expected Gain
• Reading Expected Gain
• Writing Expected Gain

17
Gains Analysis Matched Cohorts

• Grade 7 to 8 (2006 to 2007)
• Math Expected Gain
• Reading Moderate Gain Beyond Expected
• Writing Moderate Bain Beyond Expected
• Grade 8 to 10 (2004 to 2007)
• Math Moderate Gain Beyond Expected
• Reading Expected Gain
• Writing Expected Gain

18
Role of Program Evaluation

• FOLD
• working needs to be expanded
• Read-180 vs. ReadAbout
• Gains Analysis
• ReadAbout Deployed in More Classes 2007-08 with
  Grant Funds

19
Return on Investment

• Bristol Per Pupil Expenditure is 125th out of 169
  (upper end of lower third)
• CMT 2007 at or just above state averages
• CAPT 2007 above state average
• City of Bristol getting a good return on its
  educational investment

20
New Pressing Issue

• CT AYP standard rises dramatically in 2008 in
  reading and math
• Projection of 2008 AYP indicates many schools
  will be cited under NCLB
• Current district performance at state average on
  CMT above state average on CAPT
• Poor AYP performance in 2008 will likely cause
  erosion of public confidence and City support
• Deeper analysis ongoing to determine
• Areas of focus for each school
• Safe Harbor status and why it is important
• Gains analysis to show progress of students over
  time who remain in the district to shift focus
  and opinion and show staff that they are doing a
  good job

21
Key Strategies Going Forward

• AYP and Safe Harbor achieving improvement for
  all children
• Program Evaluation FOLD (first pilot)
• Gains Analysis Cohort Improvement Over Time and
  Across Disparate Tests
• Instructional Excellence

22
Preliminary 2008 AYP Analysis
23
FOLD Evaluation

• Why were Hubbell Elementary Schools reading
  scores so high?
• Development of the Foundations of Literacy
  Development (FOLD) evaluation process
• Identifying Growth Model of Improvement for each
  elementary school
• SWAT Team and focused plans on other
  intervnetions
• Long term objective Maximizing what really works
  for students participating in FOLD.

24
FOLD Evaluation Process

• Semi-structured interview of principal who
  originally implemented FOLD
• Key design features
• Developed pilot survey questions
• Reviewed and refined survey questions
• Administered survey to literacy teachers and
  principals

25
FOLD Evaluation Process

• Preliminary method of analysis
• Identified each schools performance on
  indicators of DRA2, CMT, and history of
  achievement
• Ranked each schools performance to identify
• Differences across high and low performing
  schools
• Differences between respondents (i.e., principals
  and literacy teachers) to identify features of
  implementation that need revision

26
Gains Analysis Logic

• AERA paper presentation 2007
• Data must be matched pairs and equal interval
  scales
• Rescale data if using different tests
• Must know the absolute possible range of each
  dataset
• Apply Modified Effect Size analysis
• Interpret with reconceptualization of Cohens d

27
Gains Analysis Different Scales

• Rescale
• Reset to zero scale
• Ratio of one scale to the other
• Analysis
• Cohens denominator for pooled SD for populations
  (for unequal variances

28
Interpretation
Less Overlap More Difference in Scores /- We
would expect little overlap and a positive
Cohens d
29
Transformation Logic Step 1

• Two Scales
• 0 to 10 range 10
• 20 to 40 range 20
• Step 1 Put scales on same starting point
• 0 to 10 range 10
• 0 to 20 range 20
• Scores are reset to start at a value of zero.
  That is, tests with possible scoring ranges of
  A?B are rescaled such that scores range from
  0?(B-A).

30



R1
0 10 20
30 40 50




R2
31



R1
0 10 20
30 40 50




R2
Ratio of R1 to R2 .5
32
5


R1
0 10 20
30 40 50

10


R2
Step 2 Multiply R2 scores by ratio of the scales
(.5) to reset to R1 scale
33
5


R1
0 10 20
30 40 50

5


R2
34
Rescale

• Range1/Range2 Resets one score to the other
• Example
• R1/R2 5/10 0.5 (orR2/R1 10/5 2.0)

35
Reconceptualizing Cohens d in this Context
36
Gains Analysis FindingsSo Far
All Schools Improving the Same Over Time Poorer
Performing Schools Not Catching Up Need Faster
Growth May be Unreasonable to Achieve
37
Using Data Mining to go Deeper - Variables

• School
• Grade 3 Teacher
• Grade 4 Teacher
• Grade 5 Teacher
• Gender
• Ethnicity
• SPED status
• F-R Eligible
• ELL status

• GR5 CMT Math Scale
• GR5 CMT Read Scale
• GR5 CMT Writing Scale
• GR4 CMT Math Scale
• GR4 CMT Read Scale
• GR4 CMT Write Scale
• GR3 DRA Reading Level

38
(No Transcript)
39
Results So Far

• Poverty not a significant factor in achievement
  all schools performing one year gain (/-) for
  one year instruction
• Teacher is associated with achievement but not
  clear as to how
• Strong teachers are assigned in many cases to
  lower performing students which probably accounts
  for gains analysis
• Need to focus interventions on specific groups of
  students identified through data mining and
  teacher/administrator knowledge

40
Summary

• Past work by district on DDDM and Cultural
  acceptance critical
• Next Steps
• Program evaluation and advanced data analysis
• Focus on instructional excellence
• Marzano Effective Teaching Strategies
• Saphier Gower The Skillful Teacher
• Maintain City confidence get to safe harbor
  for 2008
• Change the Conversation from NCLB/AYP to Growth
  Over Time