Fast Track Program Evaluation Using Assessments Diagnostically

1
Fast Track Program Evaluation Using Assessments
Diagnostically

• Philip A. Streifer, Ph.D.
• Bristol Superintendent of Schools UCONN
  Executive Leadership Program
• Yvel Crevecoeur
• Doctoral Student, Department of Educational
  Psychology, program in Special Education

2
Primary Issues

• Bristol an urban leader in reform
• Next phase of development program evaluation and
  focus on instructional excellence
• Primary reading intervention program
  Fundamentals of Literacy Development (FOLD)
• Rising CT AYP Standard for 2008 Need to track
  student gains over time vs. NCLB/AYP year by year
  analysis

3
Pressures

• 181 First Days of School
• Growing Poverty Rate
• Manage Expectations in City
• Article for upcoming Chamber Views
• Ongoing speaking and publications on challenges
• Focusing work of 1000 people
• Rising AYP Standards and City confidence in
  schools can we get over the bar?

4
Challenges

• Migration
• Economically Disadvantaged
• Rising State NCLB Standards for 2008
• Reading 68 to 79 proficiency
• Math 74 to 82 proficiency
• Special Education Placements and Costs

5
181 First Days of School One Schools Experience

• Transfers In

• Transfers Out

• July-Aug- Sep 53
• October 16
• November 11
• December 4
• January 24
• February 12
• March 11
• April 1
• May 0
• June 0
• TOTAL 132

• July-Aug- Sep 61
• October 8
• November 14
• December 3
• January 8
• February 12
• March 10
• April 2
• May 3
• June 1
• TOTAL 122

GRAND TOTAL for ONE YEAR 254 Children in and
out/School Population /-350
6
(No Transcript)
7
Data Driven Reform and Improvement

• Past, Present and Future

8
The Bristol Accountability Initiative
Data-Driven Decision Making
MAKING STANDARDS WORK
9
CT Mastery Test and CAPT

• CMT new version for 2005-06
• CAPT new version for 2006-07
• Both are tougher tests
• Each year NCLB/AYP standard rises
• NCLB/AYP standard for 2007-08 (spring 2008) is
  rising dramatically

10
Matched Cohort Comparison on the Connecticut Mastery Test Matched Cohort Comparison on the Connecticut Mastery Test Matched Cohort Comparison on the Connecticut Mastery Test Matched Cohort Comparison on the Connecticut Mastery Test Matched Cohort Comparison on the Connecticut Mastery Test
Percent of Students Scoring At or Above Proficiency Percent of Students Scoring At or Above Proficiency Percent of Students Scoring At or Above Proficiency Percent of Students Scoring At or Above Proficiency Percent of Students Scoring At or Above Proficiency
Grade/Cohort Year Math Reading Writing
3 2006 82.7 68.5 85.3
4 2007 85.6 74.5 87.9
  Change 2.9 6.0 2.6
4 2006 86.0 71.1 83.2
5 2007 83.8 74.8 89.2
  Change - 2.2 3.7 6.0
5 2006 80.8 74.2 89.9
6 2007 84.0 75.5 80.9
  Change 3.2 1.3 - 9.0
6 2006 81.7 74.5 79.3
7 2007 83.5 80.2 80.3
  Change 1.8 5.7 1.0
7 2006 79.6 80.0 78.0
8 2007 82.1 79.7 82.0
  Change 2.5 - 0.3 4.0
Percent Increase Percent Increase
11
Matched Cohort Data (2006 to 2007) Writing
Total Matched Students Tested Grade / Cohort Percent Moved From BELOW Proficient to Proficient and Above Number of Students Moved to Proficient and Above
590 Gr 3 - 4 48 41 out of 85
634 Gr 4 - 5 57 61 out of 107
619 Gr 6 -7 34 43 out of 128
679 Gr 7-8 38 56 out of 146
Grade 5 to 6 matched cohort did NOT make gains in Writing. Grade 5 to 6 matched cohort did NOT make gains in Writing. Grade 5 to 6 matched cohort did NOT make gains in Writing. Grade 5 to 6 matched cohort did NOT make gains in Writing. Grade 5 to 6 matched cohort did NOT make gains in Writing. Grade 5 to 6 matched cohort did NOT make gains in Writing. Grade 5 to 6 matched cohort did NOT make gains in Writing.
12
Matched Cohort Comparison on the Connecticut Mastery Test Matched Cohort Comparison on the Connecticut Mastery Test Matched Cohort Comparison on the Connecticut Mastery Test Matched Cohort Comparison on the Connecticut Mastery Test Matched Cohort Comparison on the Connecticut Mastery Test
Percent of Students Scoring At or Above Goal Percent of Students Scoring At or Above Goal Percent of Students Scoring At or Above Goal Percent of Students Scoring At or Above Goal Percent of Students Scoring At or Above Goal
Grade/Cohort Year Math Reading Writing
3 2006 62.7 52.6 62.8
4 2007 66.4 58.4 68.9
  Change 3.7 5.8 6.1
4 2006 68.6 58.6 61.7
5 2007 68.7 61.6 69.7
  Change 0.1 3.0 8.0
5 2006 58.0 58.9 68.1
6 2007 63.7 63.1 59.3
  Change 5.7 4.2 - 8.8
6 2006 60.2 60.5 55.3
7 2007 60.9 69.7 56.3
  Change 0.7 9.2 1.0
7 2006 56.3 67.3 51.7
8 2007 59.9 71.3 62.3
  Change 3.6 4.0 10.6
Percent Increase Percent Increase
13
District CAPT 2007 Results
  Mathematics Mathematics Science Science Reading Across The Disciplines Reading Across The Disciplines Writing Across the Disciplines Writing Across the Disciplines
  At/Above Goal At/Above Proficient At/Above Goal At/Above Proficient At/Above Goal At/Above Proficient At/Above Goal At/Above Proficient
Bristol 51.6 88.1 40.8 88.1 50.2 87.2 50.2 86.8
Male 54.3 87.6 44.9 86.8 45.6 81.3 40.1 81.7
Female 48.5 88.6 36.2 89.6 55.5 94.0 61.6 92.6
Black 28.3 71.7 20.8 77.4 46.2 71.2 36.5 71.2
Hispanic 25.4 69.8 10.8 70.8 26.6 79.7 27.0 73.0
White 57.1 91.7 47.1 91.3 53.4 89.6 54.0 89.7
F/R Meals 27.2 67.6 19.0 73.7 26.3 74.5 23.9 69.4
Full Price 58.2 93.6 46.8 92.0 56.7 90.7 57.2 91.4
Special Ed. 14.3 50.8 11.1 49.2 11.3 38.7 11.5 44.3
Not Special Ed. 55.7 92.2 44.1 92.4 54.4 92.4 54.3 91.3
Not ELL 51.9 88.7 41.2 88.4 50.7 87.5 50.5 87.3

ELL, Asian American and American Indian subgroups are not reported, as there are fewer than 20 students in each subgroup. ELL, Asian American and American Indian subgroups are not reported, as there are fewer than 20 students in each subgroup. ELL, Asian American and American Indian subgroups are not reported, as there are fewer than 20 students in each subgroup. ELL, Asian American and American Indian subgroups are not reported, as there are fewer than 20 students in each subgroup. ELL, Asian American and American Indian subgroups are not reported, as there are fewer than 20 students in each subgroup. ELL, Asian American and American Indian subgroups are not reported, as there are fewer than 20 students in each subgroup. ELL, Asian American and American Indian subgroups are not reported, as there are fewer than 20 students in each subgroup. ELL, Asian American and American Indian subgroups are not reported, as there are fewer than 20 students in each subgroup. ELL, Asian American and American Indian subgroups are not reported, as there are fewer than 20 students in each subgroup.
Areas highlighted in yellow show where Bristol
exceeded State averages.
14
Performance Level Graph District vs. State
Averages
15
Gains Analysis Matched Cohorts

• Grade 3 to 4 (2006 to 2007)
• Math Expected Gain
• Reading Moderate Gain Beyond Expected
• Writing Expected Gain
• Grade 4 to 5 (2006 to 2007)
• Math Expected Gain
• Reading Moderate Loss Below Expected
• Writing Expected Gain

16
Gains Analysis Matched Cohorts

• Grade 5 to 6 (2006 to 2007)
• Math Expected Gain
• Reading Moderate Gain Beyond Expected
• Writing Expected Gain
• Grade 6 to 7 (2006 to 2007)
• Math Expected Gain
• Reading Expected Gain
• Writing Expected Gain

17
Gains Analysis Matched Cohorts

• Grade 7 to 8 (2006 to 2007)
• Math Expected Gain
• Reading Moderate Gain Beyond Expected
• Writing Moderate Bain Beyond Expected
• Grade 8 to 10 (2004 to 2007)
• Math Moderate Gain Beyond Expected
• Reading Expected Gain
• Writing Expected Gain

18
Role of Program Evaluation

• FOLD
• working needs to be expanded
• Read-180 vs. ReadAbout
• Gains Analysis
• ReadAbout Deployed in More Classes 2007-08 with
  Grant Funds

19
Return on Investment

• Bristol Per Pupil Expenditure is 125th out of 169
  (upper end of lower third)
• CMT 2007 at or just above state averages
• CAPT 2007 above state average
• City of Bristol getting a good return on its
  educational investment

20
New Pressing Issue

• CT AYP standard rises dramatically in 2008 in
  reading and math
• Projection of 2008 AYP indicates many schools
  will be cited under NCLB
• Current district performance at state average on
  CMT above state average on CAPT
• Poor AYP performance in 2008 will likely cause
  erosion of public confidence and City support
• Deeper analysis ongoing to determine
• Areas of focus for each school
• Safe Harbor status and why it is important
• Gains analysis to show progress of students over
  time who remain in the district to shift focus
  and opinion and show staff that they are doing a
  good job

21
Key Strategies Going Forward

• AYP and Safe Harbor achieving improvement for
  all children
• Program Evaluation FOLD (first pilot)
• Gains Analysis Cohort Improvement Over Time and
  Across Disparate Tests
• Instructional Excellence

22
Preliminary 2008 AYP Analysis
Schhol AYP Status for 05-06 Based on CMT 06 Scores AYP Targets Math - 74 / Reading - 68 AYP Status for 06-07 Based on CMT 07 Scores AYP Targets Math - 74 / Reading - 68 PREDICTED AYP Status for 07-08 Based on CMT 07 Scores AYP Targets Math - 82 / Reading - 79
A Achieved - all areas Achieved - all areas May Not Achieve - Reading (Econ Disadv)
B Safe Harbor - Reading - Whole School, Rdg (64), Econ Disadv, Rdg (61) Not Achieved -Reading - Whole School, Rdg (70), Econ Disadv, Rdg (61.1), White, Rdg (64.3) Whole School, Math (70), Econ Disadv, Math (68.7) May be Year 1 - Reading (Whole School, White, Econ Disadv) - Whole School, Econ Disadv - Math for first time...Watch Hispanic cell size!
C Not Achieved - Reading - Econ. Disadv, Reading (64) Year 1 - Reading - Econ Disadv, Rdg (61.6) Not Achieved for first time - Econ Disadv, Math (72.3) May be Year 2 - Reading/Math - Whole School Rdg (1st time), Econ Disadv Math (2nd time), Econ Disadv Rdg (3rd time)
D Safe Harbor - Reading - Econ. Disadv, Reading (67) Achieved - all areas May Not Achieve - Math/Reading - (Econ Disadv Math/Reading), Watch sped cell size!
E Achieved - all areas Achieved - all areas May Not Achieve - Reading - Whole School, White (barely making target - 81), Econ Disadv cell will be reported / confidence interval may drop!
F Achieved - all areas Safe Harbor - Reading - Econ Disadv, Rdg (61.6) (first time) May Not Achieve - Reading - (Econ Disadv - 2nd time)
23
FOLD Evaluation

• Why were Hubbell Elementary Schools reading
  scores so high?
• Development of the Foundations of Literacy
  Development (FOLD) evaluation process
• Identifying Growth Model of Improvement for each
  elementary school
• SWAT Team and focused plans on other
  intervnetions
• Long term objective Maximizing what really works
  for students participating in FOLD.

24
FOLD Evaluation Process

• Semi-structured interview of principal who
  originally implemented FOLD
• Key design features
• Developed pilot survey questions
• Reviewed and refined survey questions
• Administered survey to literacy teachers and
  principals

25
FOLD Evaluation Process

• Preliminary method of analysis
• Identified each schools performance on
  indicators of DRA2, CMT, and history of
  achievement
• Ranked each schools performance to identify
• Differences across high and low performing
  schools
• Differences between respondents (i.e., principals
  and literacy teachers) to identify features of
  implementation that need revision

26
Gains Analysis Logic

• AERA paper presentation 2007
• Data must be matched pairs and equal interval
  scales
• Rescale data if using different tests
• Must know the absolute possible range of each
  dataset
• Apply Modified Effect Size analysis
• Interpret with reconceptualization of Cohens d

27
Gains Analysis Different Scales

• Rescale
• Reset to zero scale
• Ratio of one scale to the other
• Analysis
• Cohens denominator for pooled SD for populations
  (for unequal variances

28
Interpretation
Less Overlap More Difference in Scores /- We
would expect little overlap and a positive
Cohens d
29
Transformation Logic Step 1

• Two Scales
• 0 to 10 range 10
• 20 to 40 range 20
• Step 1 Put scales on same starting point
• 0 to 10 range 10
• 0 to 20 range 20
• Scores are reset to start at a value of zero.
  That is, tests with possible scoring ranges of
  A?B are rescaled such that scores range from
  0?(B-A).

30



R1
0 10 20
30 40 50




R2
31



R1
0 10 20
30 40 50




R2
Ratio of R1 to R2 .5
32
5


R1
0 10 20
30 40 50

10


R2
Step 2 Multiply R2 scores by ratio of the scales
(.5) to reset to R1 scale
33
5


R1
0 10 20
30 40 50

5


R2
34
Rescale

• Range1/Range2 Resets one score to the other
• Example
• R1/R2 5/10 0.5 (orR2/R1 10/5 2.0)

35
Reconceptualizing Cohens d in this Context
36
Gains Analysis FindingsSo Far
  Read Mean Mean SD SD Read Mean Mean SD SD Math Mean Mean SD SD Math Mean Mean SD SD
  3 to 4 3 4 3 4 4 to 5 4 5 4 5 3 to 4 3 4 3 4 4 to 5 4 5 4 5
Hi Poverty, Low Performing 0.68 215 237 30 34 -0.24 227 219 40 34 0.26 235 246 42 44 -0.11 238 232 54 55
Hi Poverty, High Performing 0.61 244 263 32 32 -0.21 261 252 43 43 0.42 259 276 43 39 -0.13 284 277 46 47
Low Poverty 0.43 238 258 43 50 -0.21 258 248 52 48 0.13 260 267 52 49 -0.15 277 268 55 63
Low Poverty 0.43 229 245 37 41 -0.30 256 243 43 42 -0.10 261 256 53 39 0.20 265 274 45 47
All Schools Improving the Same Over Time Poorer
Performing Schools Not Catching Up Need Faster
Growth May be Unreasonable to Achieve
37
Using Data Mining to go Deeper - Variables

• School
• Grade 3 Teacher
• Grade 4 Teacher
• Grade 5 Teacher
• Gender
• Ethnicity
• SPED status
• F-R Eligible
• ELL status

• GR5 CMT Math Scale
• GR5 CMT Read Scale
• GR5 CMT Writing Scale
• GR4 CMT Math Scale
• GR4 CMT Read Scale
• GR4 CMT Write Scale
• GR3 DRA Reading Level

38
(No Transcript)
39
Results So Far

• Poverty not a significant factor in achievement
  all schools performing one year gain (/-) for
  one year instruction
• Teacher is associated with achievement but not
  clear as to how
• Strong teachers are assigned in many cases to
  lower performing students which probably accounts
  for gains analysis
• Need to focus interventions on specific groups of
  students identified through data mining and
  teacher/administrator knowledge

40
Summary

• Past work by district on DDDM and Cultural
  acceptance critical
• Next Steps
• Program evaluation and advanced data analysis
• Focus on instructional excellence
• Marzano Effective Teaching Strategies
• Saphier Gower The Skillful Teacher
• Maintain City confidence get to safe harbor
  for 2008
• Change the Conversation from NCLB/AYP to Growth
  Over Time