Informing Policy: State Longitudinal Data Systems

1
Informing Policy State Longitudinal Data Systems

• Jane Hannaway, Director
• The Urban Institute
• CALDER
• www.caldercenter.org

2
State of U.S. Education

• ½ of minority students graduate from high school
• 4 grade level gap between white and minority
  students by 12th grade
• 15 of minorities earn BAs w/in 9 years of 9th
  grade

3
The WILL and the WAY

• The WILL
• Left, Right, Center
• Agreement on education crisis
• Strange bedfellows
• The WAY
• Few, but growing, guideposts

4
Finding the WAY with Evidence-A New Day-

• Who has the evidence?
• States have the makings of the evidence
• Where are the makings?
• State administrative data systems
• Why do states have it?
• Important effect of NCLB
• Why important?
• Address questions never before possible

5
Research Background What We Know

• Teachers matter- single most important schooling
  contributor to student outcomes
• Wide variation in teacher effectiveness. Some
  teachers are simply much better than others
• Standard measures of teacher quality not much
  related to effectiveness, but directly related to
  spending.

6
Research BackgroundWhat We Dont Know

• What is it about teachers that matters?

7
3 Research Probes

• Teacher Maldistribution
• Teacher Selection
• Teacher Mobility

8
Teacher Maldistribution 1

• Comparison of VA of teachers in high/ low poverty
  schools
• North Carolina and Florida
• Findings
• Low poverty - higher va, but not much
• High poverty larger variation in school

9
Teacher Value-Added at Percentiles by School
Poverty Levels (North Carolina-Math)
10
Teacher Value-Added at Percentiles by School
Poverty Levels (Florida- Math)
11

• Novice teachers are less effective than
  experienced teachers.
• Returns to experience taper off 3-5 years.

12
Distribution of Value-Added of Elementary Math
Teachers in High Poverty Schools
Solid line Novice teachers Dash line Teachers
with 1-2 years of experience Dotted line
Teachers with 3-5 years of experience
13
Distribution of Value-Added of Elementary Math
Teachers in Lower Poverty Schools
Solid line Novice teachers Dash line Teachers
with 1-2 years of experience Dotted line
Teachers with 3-5 years of experience
14
Teacher Maldistribution 2

• New York City
• Phasing out of emergency certification
• Introduction of alternative route teachers

15
LAST Exam Failure Rate of Elementary Teachers by
Poverty Quartile, 2000-2005
16
LAST Exam Failure Rate of New Teachers by Poverty
Quartile, 2000-2005
17
Predicted Effectiveness For Highest and Lowest
Poverty Schools
Narrows by 25
18
Can change predicted effectiveness by selection
up-front
Mean VA by Quintile (poor schools) Passed Exam Not Certified Math SAT Verbal SAT college competitiveness college competitiveness college competitiveness college competitiveness
Mean VA by Quintile (poor schools) Passed Exam Not Certified Math SAT Verbal SAT Most Some Less Not
-0.068 0.46 0.73 355 440 0.04 0.07 0.55 0.35
-0.032 0.66 0.14 414 467 0.05 0.07 0.54 0.34
-0.010 0.78 0.08 423 462 0.09 0.13 0.44 0.34
0.010 0.85 0.03 450 470 0.16 0.20 0.37 0.27
0.045 0.91 0.01 512 474 0.25 0.25 0.35 0.15

• Meaningful difference based only on attributes,
  though monitoring, development and selective
  retention also needed

19
Teacher Selection

• Teach for America
• North Carolina
• Secondary school
• Mainly math and science

20
TFA Findings high school
Student FE, Math subjects
TFA v. all others TFA v. in-field non-TFA TFA v. traditional track
TFA 0.11 0.10 0.08
Experience
3-5 yrs 0.05 0.06 0.03
6-10 yrs 0.05 0.06 0.02
gt 10 yrs 0.05 0.05 0.03
All TFA coefficients are significant at the .05
level.
21
Teacher Mobility

• Mobility highest at most challenging schools
• The worst teachers are the first to leave
• General tendency to move to more affluent schools

22
Topic of the Day Performance Incentives

• Objective??
• Recruitment/ selection
• Retention/ deselection
• Increase performance thru effort

23
Issues

• How good are the measures?
• Individual vs school rewards?
• Teachers without test scores?

24
VA Measures

• Problems
• Year to year variability
• Measurement error
• Sorting
• How serious?
• Less serious for policy research
• More serious for individual stakes

25
Predicting Performance

• Using first 2 yrs of performance top to top/
  bottom to bottom quintile
• Goldhaber and Hansen (NC) 46/ 44
• Koedel and Betts (SanDiego) 29/ 35
• Sass (Florida) 22-32/ 24-24

26
Policy Implications

• Use VA freely for research
• Use VA carefully for individual teacher judgments
• Important information
• Corrorboration
• More years are better
• Move tenure decision out!

27
Research Questions

• Are teachers in high poverty schools more/ less
  effective (value-added) than teachers in lower
  poverty schools?
• Do school factors affect differences in the
  value-added of high poverty and lower poverty
  teachers?
• Do teacher qualifications affect differences in
  the value-added of high poverty and lower poverty
  teachers?

28
Data

• Florida (2000/01- 2004/05)
• Elementary
• Student achievement FCAT-SSS
• Grades 3-10
• Teacher links
• Assignment, certification, experience, education
• North Carolina (2000/1-2004/5)
• Elementary
• Student achievement
• EOG grades 3-8
• EOC secondary subjects
• Teacher linked through proctor and verification
• Assignment, certification, experience, education

29
Definitions

• High poverty elementary schools (gt70 FRL
  students)
• Lower poverty elementary schools (lt70 FRL
  students)
• Very low poverty schools (lt30 FRL students).

30
NC Student-Teacher Link
EOC student-level records
Aggregate to EOC test classrooms by school,
year, subject, proctor id
Decision Rules Match if teacher and proctor id
identical and  fit statistic lt 1.5.
31
Sample Restrictions

• Exclude charter schools
• Exclude schools that switch high poverty to lower
  poverty status
• Only classrooms w/ 10-40 students
• Only self-contained elementary classrooms

32
Analytic Sample

0-30 FRL 30-70 FRL 70-100 FRL Total
Florida(Elementary School Level) 3, 084 6, 975 5,232 14, 052
North Carolina (Elementary School Level) 2,207 5, 945 2, 316 9,212
Note We focus on elementary schools, grades 3-5
where poverty information is most reliable. We
exclude teachers from charter schools and we
exclude classrooms with lt10 students or gt40
students in our samples.
33
Methodological Challenges

• Non-random sorting of teachers and students
• Distinguishing teacher and school effects
• Precision in Teacher Effects Estimates
• Sources of Teacher Effectiveness Differentials

34
Descriptive FindingsElementary Student
Demographics
35
Descriptive Findings Student Performance
Florida Florida Florida Florida North Carolina North Carolina North Carolina
0-30 FRL 0-30 FRL 30-70 FRL 70-100 FRL 0-30 FRL 30-70 FRL 70-100 FRL
Level Scores Level Scores Level Scores Level Scores Level Scores Level Scores Level Scores Level Scores
Math Math 0.49 0.25 -0.16 0.43 0.15 -0.32
Reading Reading 0.50 0.26 -0.18 0.39 0.14 -0.34
Gain Scores Gain Scores Gain Scores Gain Scores Gain Scores Gain Scores Gain Scores Gain Scores
Math -0.02 -0.02 -0.01 0.02 0.02 0.01 0.02
Reading -0.01 -0.01 -0.01 -0.01 0.02 0.01 0.00
Differences between the given estimate and the
corresponding estimates for schools with 70-100
FRL students significant at 5 and
differences significant at 1.
36
Descriptive FindingsTeacher Experience
37
Descriptive FindingsTeacher Qualifications
38
Distribution of Value-Added of Elementary Reading
Teachers in Lower Poverty Schools
Solid line Novice teachers Dash line Teachers
with 1-2 years of experience Dotted line
Teachers with 3-5 years of experience
39
Distribution of Value-Added of Elementary Reading
Teachers in High Poverty Schools
Solid line Novice teachers Dash line Teachers
with 1-2 years of experience Dotted line
Teachers with 3-5 years of experience
40
Differences in Estimates of Teacher Value-Added
Florida Florida Florida North Carolina North Carolina
Difference Difference Difference Difference Difference Difference
Math Math Math Math
FE Estimates FE Estimates -
Student Covariate Estimates Student Covariate Estimates -
Reading Reading Reading Reading
FE Estimates FE Estimates
Student Covariate Estimates Student Covariate Estimates
41
Magnitude of Differences in Value Added Estimates
42
Differences in Standard Deviations of Value-Added
Florida Florida Florida North Carolina North Carolina
Difference Difference Difference Difference Difference Difference
Math Math Math Math
FE Estimates FE Estimates - -
Student Covariate Estimates Student Covariate Estimates - -
Reading Reading Reading Reading
FE Estimates FE Estimates - -
Student Covariate Estimates Student Covariate Estimates - -
43
Differences between Lower- and High-Poverty by
Percentile of Teacher Value Added
44
Teacher Value-Added at Percentiles by School
Poverty Levels (North Carolina- Reading)
45
Teacher Value-Added at Percentiles by School
Poverty Levels (Florida- Reading)
46
Correlation of Teacher Qualifications and
Value-Added
47
Sources of Difference in Teacher Value-Added
Between High-Poverty and Lower-Poverty Elementary
Schools
48
Sensitivity Analysis

• School Effect
• Empirical Bayes Adjustment

49
Conclusions

• Teachers in high poverty schools, on average, are
  less effective than teachers in lower poverty
  schools.
• Changing schools (high poverty/lower poverty)
  does not affect teacher effectiveness
• There is greater teacher variation within high
  poverty schools than within lower poverty
  schools.

50
Conclusions (cont)

• Differences in teachers in High Poverty and
  Lower Poverty schools
• only weakly related to teacher qualifications
• more strongly related to marginal effect of
  qualifications (experience)
• not explained by school poverty level

51
Study Limitations

• Issues with value-added measures
• separating current teacher contributions from
  other current contributions
• E.g., current family circumstances
• - dynamic sorting
• sorting on time variant characteristics
• Instability of measures
• E.g., measurement error, motivation