What Makes For a Good Teacher and Who Can Tell?

1
What Makes For a Good Teacher and Who Can Tell?

• Douglas N. Harris
  Tim R. Sass
• Dept. of Ed. Policy Studies Dept.
  of Economics
• Univ. of Wisconsin
  Florida State Univ.

IES Research Conference, June 2008
2
Background

• Growing Interest in Linking Teacher Performance
  and Compensation
• Value Added scores based on performance of a
  teachers students have limitations
• Only available for teachers in tested subject
  (reading and math)
• Subject to measurement error and inter-temporal
  instability
• Movement to Grant Principals Greater Autonomy in
  Personnel Decisions
• Charter schools
• School reform in New York City

3
Background

• Still Know Relatively Little About What Teacher
  Characteristics are Associated With Ability to
  Improve Student Performance
• Most recent studies using panel data on
  individual student achievement find that teacher
  experience (in the early years) matters, but
  little else in the way of readily observable
  characteristics seems to affect student
  achievement
• Knowing what characteristics determine teacher
  quality has implications for teacher preparation,
  licensure, hiring practices and professional
  development

4
Research Questions

• Do principals evaluations of teachers contain
  information beyond readily-observable objective
  teacher characteristics?
• What factors determine a principals overall
  evaluation of her teachers?
• What specific teacher characteristics are
  associated with a teachers ability to raise
  student achievement (ie. value added)?
• Can principals do as well (or better) at
  predicting future teacher value added than past
  performance measures?

5
Literature Review

• Early Literature
• Murnane (1975)
• Conditioning on student characteristics and prior
  achievement, a one-standard-deviation increase in
  principals rating is associated with a 0.125
  standard deviation increase in student
  third-grade math scores
• Peterson (1987, 2000)
• Finds weak correlation between principal ratings
  of teachers and parent or student satisfaction
• Milanowski (2004), Kimball, et al. (2004), White
  (2004)
• Comparisons of principals four-point teacher
  evaluation scales with single-year estimates of
  teacher value added
• Generally find weakly positive correlation
  between principal evaluations and estimated
  teacher quality (correlation 0.2 - 0.4)

6
Literature Review

• Recent Literature
• Jacob and Lefgren (2005)
• Consider both principals overall rating and
  evaluations of specific teacher characteristics
• Find principals good at identifying best and
  worse teachers, but less able to distinguish
  teachers in the middle
• Principal evaluations better at predicting future
  student achievement than teacher experience or
  educational attainment, though generally not as
  good as value-added estimates
• Principal evaluations better at predicting
  parental requests for teachers than teacher
  value-added scores

7
Data

• Anonymous mid-sized school district in Florida
• Schools serve a diverse set of student
  populations
• Free/Reduced-Price Lunch 5-90
• Percent Minority 10-90
• Longitudinal student-level achievement data
• Grades 1-10
• 1999/00 through 2004/05
• Math and reading scores on the Stanford
  Achievement Test

8
Data

• Principal Interviews
• In-person interviews conducted in Summer 2006
• Single-blind evaluation of up to 10 teachers (5
  per subject)
• Random sample of teachers at school who taught at
  least three classes in the tested grades and
  subjects during 1999/00-2004/05
• 9-Point Rating Scale
• Not effective (1-3) performance is
  substantially below minimum standards
• Adequate (4-6) meets standards but can improve
  in several areas.
• Exceptional (7-9) teacher is among the best I
  have seen
• Asked to evaluate teachers on
• Overall performance
• Ability to test scores
• Specific characteristics

9
Sample Studentand Teacher Characteristics

• Math Reading
• Total Students 76,308 70,916
• Total Teachers 943 960
• Principals Interviewed 30 30
• Teachers Rated by Principal 234 231

10
Sample Teacher Grade Levelsand Principal Ratings

• Math Reading
• Teachers
• Taught Primarily Elementary School 0.718 0.706
• Taught Primarily Middle School 0.152 0.153
• Taught Primarily High School 0.131 0.140
• Mean Principal Rating of Teacher
• Overall 7.103 7.108
• Ability to Raise Test Scores 7.232 7.164
• Caring 7.397 7.468
• Enthusiastic 7.269 7.385
• Motivated 7.436 7.494
• Strong Teaching Skills 7.560 7.597
• Knows Subject 7.868 7.896
• Communication Skills 7.594 7.710
• Intelligence 7.897 7.922
• Positive Relationship with Parents 7.511 7.59
  6
• Positive Relationship with Students 7.670 7.7
  30

11
Pairwise Correlation of Principals Ratings
ofTeachers With Teacher Characteristic Factors
(Math)
Overall Rating Ability to Raise Test Scores Inter-personal Skills Motivation/ Enthusiasm Works Well With Others Knowledge/ Teaching Skills
Overall Rating 1
Ability to Raise Test Scores 0.745 1
Interpersonal Skills 0.703 0.550 1
Motivation/ Enthusiasm 0.738 0.596 0.734 1
Works Well With Others 0.762 0.598 0.756 0.732 1
Knowledge/ Teaching Skills 0.881 0.752 0.612 0.682 0.644 1
Significant at the 0.05 level
12
Pairwise Correlation of Principals Ratings
ofTeachers With Teacher Characteristic Factors
(Reading)
Overall Rating Ability to Raise Test Scores Inter-personal Skills Motivation/ Enthusiasm Works Well With Others Knowledge/ Teaching Skills
Overall Rating 1
Ability to Raise Test Scores 0.736 1
Interpersonal Skills 0.719 0.626 1
Motivation/ Enthusiasm 0.699 0.569 0.714 1
Works Well With Others 0.725 0.589 0.762 0.677 1
Knowledge/ Teaching Skills 0.861 0.697 0.644 0.682 0.650 1
Significant at the 0.05 level
13
Factor Loadings of Normalized Principal Ratings
(Math)
Teacher Characteristic Rated by Principal Interpersonal Skills Motivation/ Enthusiasm Works Well With Others Knowledge/ Teaching Skills
Intelligent -0.0481 0.0839 0.0606 0.7067
Works Well With Grade Team/Dept. -0.0046 -0.0887 0.9711 0.0399
Works Well With Me (Principal) 0.1743 0.0835 0.7415 -0.0814
Positive Relationship With Parents 0.7231 0.0781 0.0768 0.0742
Positive Relationship With Students 0.9408 0.0103 -0.0131 0.0636
Caring 0.5591 0.1372 0.2422 -0.0185
Enthusiastic 0.1086 0.9721 -0.0707 -0.0035
Motivated 0.0398 0.5224 0.2802 0.1624
Strong Teaching Skills 0.1512 0.0258 -0.0462 0.8471
Knows Subject -0.0088 -0.0551 -0.0036 0.9831
Communication Skills 0.1040 0.1705 0.2734 0.3191
14
Factor Loadings of Normalized Principal Ratings
(Reading)
Teacher Characteristic Rated by Principal Interpersonal Skills Motivation/ Enthusiasm Works Well With Others Knowledge/ Teaching Skills
Intelligent 0.0030 0.0055 0.0332 0.7182
Works Well With Grade Team/Dept. 0.0243 -0.0598 0.8522 0.0838
Works Well With Me (Principal) 0.1390 0.0461 0.8329 -0.0597
Positive Relationship With Parents 0.7554 0.0528 0.0649 0.0757
Positive Relationship With Students 0.9171 0.0271 0.0193 0.0313
Caring 0.6046 0.0983 0.2541 -0.0367
Enthusiastic 0.0775 0.9879 -0.0464 -0.0191
Motivated 0.0247 0.5366 0.2017 0.2445
Strong Teaching Skills 0.2237 0.0174 -0.0802 0.8140
Knows Subject -0.0834 -0.0156 0.0385 0.9819
Communication Skills 0.1650 0.2167 0.1715 0.3290
15
Estimation Procedure

• Estimate Student Achievement Model
• Includes student, teacher and school fixed
  effects
• Estimated teacher effects yield within-school
  measure of teacher effectiveness
• Regress Estimated Teacher Fixed Effects on
  Objective Teacher Characteristics and
  Normalized Principal Ratings
• Objective characteristics include teacher
  experience, possession of an advanced degree and
  certification status
• Weighted least squares estimates with the square
  root of the numbers of students per teacher as
  weights

16
Value-Added Model
Ait achievement level of student i in year t
Xit time-varying student characteristics
P-ijmt time-varying characteristics of peers in classroom j in school m
gi student fixed-effect
dk teacher fixed effect
fm school fixed effect
17
Value-Added Results Used for Estimation
of Teacher Effects (Grades 2 10, 1999/2000
2004/05)

Math
Reading ______________________________________
__________________ Number of Schools
Attended -2.131 -0.977 (2.02) (0.84)
Attended Different School in Prior Year
1.910 0.834 (2.99) (1.15) Class
Size -0.111 -0.038 (2.21) (0.68)
Proportion of Classroom Peers -7.788 -2.48
1 Who are White (4.37) (1.30) ____
__________________________________________________
__ Covariance of Achievement Gain and Student
FE 0.186 0.183 Teacher FE 0.032 0.100
Model (Including School Indicators) 0.191 0.183
Error 0.591 0.534 No. of Observations
76,308
70,916 ___________________________
_____________________________ Note Model
Includes Student, Teacher, School and
Grade-By-Year fixed effects. t-ratios adjusted
for clustering at the classroom level.
18
Weighted Least Squares Estimates of the
Relationship Between Principal Ratings and
Estimated Teacher Effectiveness
Math Reading


1 2 1
2 _____________________________________________
____________ Overall Rating 2.685
1.661 (2.82) (1.76) Ability to Raise
Test Scores 2.570 0.975 (4.52) (1.
48) ______________________________________________
___________ R-squared 0.078
0.149 0.061 0.057 No. of Observations 234
207 231 201 _____________________________
____________________________ Note Model
includes a set of six experience category
indicators and indicators for possession of
advanced degrees and full
certification.
19
WLS Estimates of the Relationship Between
Principal Ratings and Estimated Teacher
Effectiveness -- By Grade Level
Math
Reading
1 2
1 2 ____________________________
____________________________ Overall Rating
3.328 2.483 Elementary
(2.84) (2.15) Overall Rating
1.424 0.029 Middle/High (0.87) (0.02)
Ability to Raise Test Scores 4.071
2.971 Elementary
(3.31) (2.25) Ability to Raise Test
Scores 1.456 -0.554
Middle/High (0.88) (0.32) ________
_______________________________________________ R
-squared 0.082 0.126 0.067
0.071 No. of Observations
234 199 231 201 _______________
________________________________________ Note
Model includes a set of six experience category
indicators and indicators for possession of
advanced degrees and full
certification.
20
WLS Estimates of the Relationship Between Teacher
Subjective Characteristics and a Principals
Overall Rating of Teachers

• Math Reading
• __________________________________________________
  _____
• Interpersonal Skill 0.091
  0.192
• (1.86) (3.26)
• Knowledge/Teaching Skills
  0.592 0.592
• (14.53) (11.49)
• Motivation/Enthusiasm 0.069
  -0.002
• (1.45) (0.04)
• Works Well With Others 0.237
  0.193
• (4.83) (3.30)
• __________________________________________________
  _____
• R-squared 0.853 0.786
• No. of Observations 207 206

21
WLS Estimates of the RelationshipBetween Teacher
Subjective Characteristicsand Estimated Teacher
Effectiveness

• Math
  Reading
• __________________________________________________
  _____
• Interpersonal Skill -0.250 1.656
• (0.15) (0.93)
• Knowledge/Teaching Skills 1.579
  3.044
• (1.10) (1.96)
• Motivation/Enthusiasm -1.030 -1.532
• (0.61) (0.96)
• Works Well With Others 2.523 -1.745
• (1.46) (0.99)
• __________________________________________________
  _____
• R-squared 0.078 0.056
• No. of Observations 207 206

22
WLS Estimates of the RelationshipBetween Teacher
Subjective Characteristicsand Estimated Teacher
Effectiveness (Math)

• 1 2 3 4
• __________________________________________________
  _____
• Interpersonal Skill 1.866
• (1.94)
• Knowledge/Teaching Skills 2.295
• (2.38)
• Motivation/Enthusiasm 1.661
• (1.77)
• Works Well With Others 2.572
• (2.63)
• __________________________________________________
  _____
• R-squared 0.058 0.066 0.055 0.072

23
WLS Estimates of the RelationshipBetween Teacher
Subjective Characteristicsand Estimated Teacher
Effectiveness (Reading)

• 1 2 3
  4
• __________________________________________________
  _____
• Interpersonal Skill 1.167
• (1.15)
• Knowledge/Teaching Skills 1.881
• (1.84)
• Motivation/Enthusiasm 0.559
• (0.56)
• Works Well With Others 0.521
• (0.50)
• __________________________________________________
  _____
• R-squared 0.035 0.045 0.030
  0.030
• No. of Observations 206 206
  206 206

24
Estimates of the Determinants of Student
Achievement Gains(Grades 2 10, 2004/05)

• Math Reading
• 1 2 1
  2
• __________________________________________________
  ____________________
• Overall Rating 2.107
  1.446
• (3.86)
  (2.69)
• Teacher Fixed Effect (from 99/00-03/04)
  0.230 0.106
• (3.85) (1.44)
• __________________________________________________
  ____________________
• R-squared 0.236 0.237
  0.115 0.114
• No. of Observations 5361
  5361 4399 4399
• __________________________________________________
  ____________________
• Note All models include controls for individual
  student mobility, class size, peer
  characteristics, student fixed effects (from
• 1999/00-2003/04), school indicators and
  a constant term.

25
Summary

• Principal ratings of teachers predict teacher
  value-added even after controlling for teacher
  experience, educational attainment and
  certification status
• Principal ratings most strongly influenced by
  knowledge/teaching skill of teachers and the
  teachers ability to work with others
• A teachers ability to raise test scores is also
  most strongly influenced by knowledge/teaching
  skill and ability to work with others
• Relationships imprecise when all factors included
  simultaneously
• Principals overall evaluation does as well as
  past value-added at predicting current teacher
  effectiveness

26
Implications

• Possibly greater role for principals in
  evaluating teachers and in retention decisions
• pay for performance scheme which includes
  principal ratings likely better than current pay
  scale based on teacher experience and education
  alone
• value added scores could be part of the mix as
  well
• Possibly greater emphasis on teaching
  skills/knowledge in teacher preparation and
  professional development