The Inter-temporal Stability of Teacher Effect Estimates

1
The Inter-temporal Stabilityof Teacher Effect
Estimates
J. R. Lockwood Daniel F. McCaffrey Tim R.
Sass The RAND Corporation The RAND
Corporation Florida State University
National Conference on Value-Added Modeling,
April 2008 This presentation has not been
formally reviewed and should not be cited or
distributed without the authors permission.
2
Introduction

• Several school districts and states have begun
  using measures of teachers contributions to
  student achievement to assess and reward teachers
• Denver, Houston, Florida
• For value added measures to provide correct
  incentives and be acceptable to stakeholders they
  must
• be relatively accurate measures of productivity
  (ie.unbiased)
• be relatively stable over time
• Most observers (implicitly) assume that a given
  teachers productivity doesnt vary much from
  year to year

3
Research Questions

• How stable are estimated teacher effects?
• What factors affect the stability of estimated
  teacher effects?
• Are there methods to enhance the stability of
  estimated teacher effects?

4
Previous Literature

• Ballou (2005)
• Elementary and middle school teachers in a
  moderately large Tennessee school district in
  two consecutive years
• Nearly 50 percent of math teachers in top
  quartile in one year stay in the top quartile the
  next year
• Precision increases with number of student
  observations per teacher
• Aaronson, et al. (2007)
• High school teachers in Chicago over two years
• 57 percent of teachers in the top quartile in one
  year remain there in the next year

5
Previous Literature

• Koedel and Betts (2007)
• Teachers in San Diego
• Within-school estimates of teacher quality
• Models with student and school fixed effects
• 35 percent of teachers ranked in the top quintile
  remain there in the next year
• Omission of student and school fixed effects
  increases stability of estimated teacher effects

6
Models of Teacher Effects

• General Value-added Model

• Student i, classroom j, teacher k, school m
• X time-varying student characteristics
• P time-varying classroom peer characteristics
• T time-varying teacher characteristics
• S time-varying school characteristics

• Teacher Classroom Average Effect

7
Data

• Seven Large Countywide School Districts in
  Florida
• Two among 10 largest in the U.S. (Dade, Broward)
• Remainder among 25 largest in the U.S.
  (Hillsborough, Palm Beach, Orange, Duval and
  Pinellas)
• Testing in Grades 3-10
• FCAT-NRT (Stanford Achievement Test)
• 1999/2000-2004/05 (SAT-9 1999/2000, SAT-10
  2004/05)
• FCAT-SSS (Criterion reference exam)
• 2000/2001-2004/05
• Focus on Middle School Math Teachers
• Teacher effects greater in math
• More students per teacher in middle school

8
How Stable are Estimated Teacher Effects?

• Year-to-Year Correlations
• Proportion of Top-Quintile Teachers Remaining in
  the Top Quintile the Next Year

9
Inter-temporal Correlation in Estimated Teacher
Classroom Average Effects
Varying Teachers Across 2-Year Periods Correlation Between Varying Teachers Across 2-Year Periods Correlation Between Varying Teachers Across 2-Year Periods Correlation Between Varying Teachers Across 2-Year Periods Correlation Between Varying Teachers Across 2-Year Periods Correlation Between
County 2000/01 and 2001/02 2001/02 and 2002/03 2002/03 and 2003/04 2003/04 and 2004/05
Broward 0.48 0.55 0.47 0.35
Dade 0.44 0.42 0.31 0.38
Duval 0.41 0.45 0.34 0.23
Hillsborough 0.35 0.33 0.36 0.23
Orange 0.23 0.18 0.21 0.32
Palm Beach 0.28 0.35 0.36 0.13
Pinellas 0.45 0.38 0.49 0.34
10
Quintile Ranking of Estimated Teacher Classroom
Average Effect in 2001/02 by Quintile Ranking in
2000/01 (in Percent)Broward County Cross-Year
Correlation 0.48
Quintile in 2000/01 Quintile in 2001/02 Quintile in 2001/02 Quintile in 2001/02 Quintile in 2001/02 Quintile in 2001/02
Quintile in 2000/01 1 2 3 4 5
1 41.4 20.7 25.9 8.6 3.5
2 21.1 29.8 28.07 14.04 7.02
3 25.0 17.2 20.3 20.3 17.2
4 20.3 15.9 20.3 26.1 17.4
5 4.7 9.4 15.6 15.6 54.7
Total 22.1 18.3 21.8 17.3 20.5
11
Quintile Ranking of Estimated Teacher Classroom
Average Effect in 2001/02 by Quintile Ranking in
2000/01 (in Percent)Orange County Cross-Year
Correlation 0.23
Quintile in 2000/01 Quintile in 2001/02 Quintile in 2001/02 Quintile in 2001/02 Quintile in 2001/02 Quintile in 2001/02
Quintile in 2000/01 1 2 3 4 5
1 27.6 17.2 27.6 17.2 10.3
2 32.4 11.8 29.4 11.8 14.7
3 13.6 18.2 15.9 25.0 27.3
4 9.4 28.1 15.6 25.0 21.9
5 6.8 18.2 18.2 15.9 40.9
Total 16.9 18.6 20.8 19.1 24.6
12
Percentage of Teachers Who Remain in Top Quintile
from One Year to the Next
County 2000/01 and 2001/02 2001/02 and 2002/03 2002/03 and 2003/04 2003/04 and 2004/05
Broward 54.7 55.2 48.3 46.2
Dade 45.7 40.4 38.9 39.3
Duval 35.9 41.2 34.3 33.3
Hillsborough 39.7 33.3 37.3 30.6
Orange 40.9 40.4 26.0 39.5
Palm Beach 34.0 40.5 30.6 23.1
Pinellas 39.4 41.0 52.5 39.4
13
What Factors Affect the Stability of Estimated
Teacher Effects?

• Changes in the Measurement of Achievement
• Test Scale
• If scaling changes over time, could decrease
  stability
• Norming by grade and year should reduce
  fluctuations due to scaling changes
• Could increase stability if distribution changes
  over time
• Test Content
• If teacher ability varies across content, changes
  in test could contribute to instability in
  measured teacher effectiveness
• Compare FCAT-SSS and FCAT-NRT

14
Inter-Temporal Correlation of Estimated Teacher
Classroom Average Effects Under Alternative Test
Score Measures
Counties Counties Counties Counties Counties Counties Counties Counties Counties Counties
Outcome Student Controls Student Min Broward Dade Duval Hills-borough Orange Palm Beach Pinellas
Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates
Gain on Normed FCAT-NRT Student Fixed Effects 10 per class 0.48 0.44 0.41 0.35 0.23 0.28 0.45
Gain on FCAT-NRT Scale Score Student Fixed Effects 10 per class 0.41 0.47 0.47 0.32 0.22 0.38 0.45
15
Inter-Temporal Correlation of Estimated Teacher
Classroom Average Effects Under Alternative
Achievement Tests
Counties Counties Counties Counties Counties Counties Counties Counties Counties Counties
Outcome Student Controls Student Min Broward Dade Duval Hills-borough Orange Palm Beach Pinellas
Correlation Between 2001/02 and 2002/03 Estimates Correlation Between 2001/02 and 2002/03 Estimates Correlation Between 2001/02 and 2002/03 Estimates Correlation Between 2001/02 and 2002/03 Estimates Correlation Between 2001/02 and 2002/03 Estimates Correlation Between 2001/02 and 2002/03 Estimates Correlation Between 2001/02 and 2002/03 Estimates
Gain on Normed FCAT-NRT Student Fixed Effects 10 per class 0.55 0.42 0.45 0.33 0.18 0.35 0.38
Gain on Normed FCAT-SSS Student Fixed Effects 10 per class 0.54 0.35 0.50 0.25 0.34 0.49 0.56
16
What Factors Affect the Stability of Estimated
Teacher Effects?

• Changes in Reference Point (Stratification)
• Individual Teacher Effectiveness Must be Measured
  Relative to Some Reference Point
• Holdout teacher or Average teacher
• If reference teacher changes, measured
  effectiveness changes
• Comparisons Can Only be Made to Other Teachers
  Who Are Interconnected by Common Students
• Different strata will have different reference
  points
• Within-school vs. between-school measures

17
Number of Teacher-Years in Interconnected Groups
Counties Counties Counties Counties Counties Counties Counties Counties
Group Type Broward Dade Duval Hills-borough Orange Palm Beach Pinellas
No Movers 69 793 354 430 521 455 36
Primary Group 5,939 16,034 6,837 9,583 8,928 8,964 5,020
All Others No. of Groups 134 59 97 46 26 13 61 28 50 25 48 23 59 26
18
What Factors Affect the Stability of Estimated
Teacher Effects?

• Omitted Variable Bias
• If Measured Teacher Effects Reflect Omitted
  Variables, Stability of Measured Teacher Effects
  Will Depend on Stability of Omitted Variables and
  Extent of Selection
• Past educational inputs (persistence)
• Achivement levels vs. achievement gains
• Student heterogeneity
• No controls vs. student covariates vs. student
  fixed effects
• Peer heterogeneity
• No controls vs. controls for peer characteristics

19
Inter-Temporal Correlation of Estimated Teacher
Classroom Average Effects Under Alternative
Persistence Assumptions
Counties Counties Counties Counties Counties Counties Counties Counties Counties Counties
Outcome Student Controls Student Min Broward Dade Duval Hills-borough Orange Palm Beach Pinellas
Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates
Gain on Normed FCAT-NRT Student Fixed Effects 10 per class 0.48 0.44 0.41 0.35 0.23 0.28 0.45
Level of Normed FCAT-NRT Student Fixed Effects 10 per class 0.56 0.55 0.38 0.32 0.50 0.41 0.50
20
Inter-Temporal Correlation of Estimated Teacher
Classroom Average Effects Under Alternative
Controls for Time-Invariant Student Heterogeneity
Counties Counties Counties Counties Counties Counties Counties Counties Counties Counties
Outcome Student Controls Student Min Broward Dade Duval Hills-borough Orange Palm Beach Pinellas
Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates
Gain on Normed FCAT-NRT Student Fixed Effects 10 per class 0.48 0.44 0.41 0.35 0.23 0.28 0.45
Gain on Normed FCAT-NRT Student Co-variates 10 per class 0.54 0.42 0.38 0.39 0.30 0.30 0.53
21
Inter-Temporal Correlation of Estimated Teacher
Classroom Average Effects Under Alternative
Controls for Time-Varying Factors (Baseline Model
Gain on Normed FCAT-NRT, Student Fixed Effects,
Minimum 10 Students per Class)
Controls for Time- Varying Co-variates Counties Counties Counties Counties Counties Counties Counties
Controls for Time- Varying Co-variates Broward Dade Duval Hills-borough Orange Palm Beach Pinellas
Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates
None 0.48 0.44 0.41 0.35 0.23 0.28 0.45
Student, Peer and Teacher 0.49 0.46 0.36 0.37 0.19 0.27 0.44
22
What Factors Affect the Stability of Estimated
Teacher Effects?

• Measurement Error in Student Achievement
• If measurement error is uncorrelated across
  students within a classroom, then precision
  should be higher for teachers with larger classes
• Minimum class size
• Minimum number of movers per teacher
• If measurement error is correlated across
  students within a classroom, but not across
  classrooms, precision should increase with the
  number of classes per teacher
• Using middle school teachers who generally teach
  multiple class per term

23
Inter-Temporal Correlation of Estimated Teacher
Classroom Average Effects Under Alternative Class
Size Restrictions
Counties Counties Counties Counties Counties Counties Counties Counties Counties Counties Counties
Outcome Student Controls Student Min Broward Dade Duval Hills-borough Hills-borough Orange Palm Beach Pinellas
Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates
Gain on Normed FCAT-NRT Student Fixed Effects 10 per class 0.48 0.44 0.41 0.35 0.23 0.23 0.28 0.45
Gain on Normed FCAT-NRT Student Fixed Effects 2 per class 0.38 0.33 0.48 0.29 0.33 0.33 0.25 0.14
Gain on Normed FCAT-NRT Student Fixed Effects 20 per class 0.49 0.52 0.41 0.39 0.31 0.31 0.27 0.58
24
Inter-Temporal Correlation of Estimated Teacher
Classroom Average Effects Under Alternative
Student Mover Restrictions (Minimum 10 students
per class restriction)
Counties Counties Counties Counties Counties Counties Counties Counties Counties Counties Counties
Outcome Student Controls Student Min Broward Dade Duval Hills-borough Hills-borough Orange Palm Beach Pinellas
Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates
Gain on Normed FCAT-NRT Student Fixed Effects 1 mover per teacher 0.48 0.44 0.41 0.35 0.23 0.23 0.28 0.45
Gain on Normed FCAT-NRT Student Fixed Effects 10 movers per teacher 0.48 0.44 0.41 0.35 0.23 0.23 0.27 0.45
Gain on Normed FCAT-NRT Student Fixed Effects 20 movers per teacher 0.46 0.50 0.45 0.36 0.25 0.25 0.29 0.50
25
What Factors Affect the Stability of Estimated
Teacher Effects?

• True Variation in Teacher Quality Over Time
• Instability in Estimated Effects Could Reflect
  Changes in True Teacher Quality Over Time
• Add time-varying teacher covariates to model
• Regress estimated teacher-by-year effects on
  teacher fixed effect and time-varying teacher
  covariates

26
Inter-Temporal Correlation of Estimated Teacher
Classroom Average Effects Under Alternative
Controls for Time-Varying Factors (Baseline Model
Gain on Normed FCAT-NRT, Student Fixed Effects,
Minimum 10 Students per Class)
Controls for Time- Varying Co-variates Counties Counties Counties Counties Counties Counties Counties
Controls for Time- Varying Co-variates Broward Dade Duval Hills-borough Orange Palm Beach Pinellas
Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates Correlation Between 2000/01 and 2001/02 Estimates
None 0.48 0.44 0.41 0.35 0.23 0.28 0.45
Teacher Only 0.46 0.45 0.39 0.35 0.21 0.26 0.44
27
Are There Methods to Enhance the Stability of
Estimated Teacher Effects?

• Are there methods to enhance the stability of
  teacher effect estimates?
• 3-Year Running Averages
• Reduces noise by averaging sampling errors
• Could add bias if true performance is changing
  across years
• Empirical Bayes or Shrinkage Estimators
• Place greater weight on more reliable estimates
  and push less reliable estimates toward
  population mean
• If already have a significant minimum class size
  restriction, simple EB adjustments which
  account for differences in the number of classes
  per teacher or students per teacher not likely to
  yield large improvements to stability
• Accounting for variability at the individual
  teacher level requires computation of standard
  errors on individual teacher effects, which can
  be problematic

28
Problems in Computing Shrinkage Estimators

• Default estimates from most software packages are
  estimating contrasts between every teacher and a
  holdout teacher
• Such estimates support within-year comparisons
  and cross-year correlations
• We cannot shrink these estimates directly and we
  cannot use the resulting standard errors for
  shrinkage
• We cannot average these estimates without
  removing yearly means because changes to the
  holdout create year-to-year fluctuations

29
Summary

• Moderate Stability in Teacher Effects
• Cross-year correlations in range on 0.2-0.5
• About 40-50 percent of teachers in top quintile
  remain in the top quintile the following year
• Stability increases with number of students per
  teacher and when persistence is assumed to equal
  0
• Nothing else has a consistent appreciable effect
  on stability
• Variation across districts appears substantial
• Shrinkage estimators or running averages could
  improve stability of teacher-by-year effects, but
  need to get appropriate estimates and standard
  errors
• Findings suggest caution in using value-added
  measures for high-stakes personnel decisions

30
Next Steps

• Determine sources of year-to-year variability
• Improve estimation techniques to obtain
  comparable estimates across years with accurate
  measures of within-year standard errors
• Current software does not support such estimation
• Separate noise from true short-term variation
• Model sources of short-term variation
• If true year-to-year variations exists more
  efficient estimation than three year averages
  might be possible via smoothing or filtering