School Effectiveness, Classroom Effectiveness, Effective Teachers

1
School Effectiveness, Classroom Effectiveness,
Effective Teachers

• Evaluation and Accountability
• February 2007

2
School Effectiveness
3
Measuring School Effectiveness

• The Commission on Educational Excellence wanted a
  fair way of comparing school achievement in 1991.
  
• RE had to come up with a method of comparing
  school achievement where no school entered with
  an advantage or started at a loss.

4
School Effectiveness Indices

• School Effectiveness Indices resulted.
• SEIs give a fair comparison when compared by
  measurable factors.
• SEIs contain more than achievement, but we will
  concentrate on achievement.

5
Fair How?

• Indices provide fair comparisons with
• Different ethnicities
• Different achievement levels
• Different income/free lunch levels
• Different language proficiency levels
• Different genders
• Different sized schools

6
Fair How?

• SEIs use test data only for students who are
  enrolled for a full year.
• SEIs only use student scores when the student has
  both a pretest and a posttest.
• SEIs compare schools using all students who meet
  these two conditions.

7
How Is It Done?

• This is the first and last statistical slide in
  this presentation.
• SEIs are computed using a two-stage, multi-level,
  fixed-factor, hierarchical linear model analysis.
  Stage 1 controls student conditioning variables
  and the HLM analysis controls school factors and
  prior achievement.

8
Arent You Glad That Parts Done?

• From now on we use no mathematics higher than the
  6th grade TEKS
• Now, in plain English, how do SEIs work?
• For student achievement SEIs work like this

9
How SEIs Work

• An expected score is established for each
  student, based on a students ethnicity, gender,
  block income level, language proficiency, school,
  and prior achievement.
• A school was effective with that student if
  achievement was above expectation.
• A school was ineffective with the student if it
  was below expectation.

10
How SEIs Work

• The effectiveness of a school with all of its
  students is computed and a total effectiveness
  score is reached for a test and grade.
• This is repeated with all tests and grades.
• The results are combined with the weights from
  the Performance Award System

11
How SEIs Work

• The resulting SEIs get an average of 50.
• Schools with SEIs above 50 are more effective.
  They add value because more students were above
  the value that we expected.
• Schools with SEIs below 50 are less effective.
  They had more students below the expected value.

12
The Important Point

• Effective schools get higher levels of
  achievement with more of their students.
• Every student counts. High achieving students
  help. Low achieving students help. LEP students
  help. Any student with test data helps.
• AEIS doesnt use all scores. AYP/NCLB doesnt
  use all scores.

13
Now Classroom Effectiveness Indices

• We will segue to a short presentation that shows
  the way CEIs work.
• This presentation was designed to show the logic
  behind CEIs in an understandable fashion.
• It uses graphs and bar charts, but keep in mind
  that it uses them at the level of the 6th grade
  TEKS or lower.

14
Classroom Effectiveness
15
Objectives

• Without complex formulas and statistics
• Understand the basics of the value-added model
  used to compute CEIs

16
Traditional Analysis of Performance

• Performance status (pass/fail)on standardized
  tests
• Doesnt account for individual differencesor
  growth
• Doesnt challenge proficient students or teachers
  to excel further

17
Solution Value-Added Model
18
Sample Teachers
19
Value-Added Step-by-Step

• For each student, examine
• 2005 score (previous year)
• 2006 score (current year)

20
Teacher As Students
Current Years Score
Previous Years Score
21
Teacher Bs Students
Current Years Score
Previous Years Score
22
Control for Student Variables

• Student characteristics beyond the teachers
  control known to impact outcomes
• Pretest
• Gender
• Ethnicity
• English language proficiency
• Socio-economic variables

23
Control for Student Variables

• From the student population districtwide,identify
  unique student groups to compare only like
  students
• Group students by pretest, gender, ethnicity,
  etc...

24
Districts Predicted Score for One Group
Students in the group share the same grade,
pretest, gender, ethnicity, language proficiency,
and socio-economicstatus.
Current Years Score
25
Districts Predicted Score for Another Group
Students inthe group share the same grade,
pretest, gender, ethnicity, language proficiency,
and socio-economicstatus.
Current Years Score
26
Group Averages for the Sample
A student that scored170 points below the
predictedscore lost that much value compared
to their own peers in the district.
Current Years Score
2405
2235
Previous Years Score
27
Group Averages for the Sample
Students that scored140 points above the
predictedscore gained that much value
compared to their own peers in the district.
Current Years Score
2265
2405
Previous Years Score
28
Teacher As Students
Group Average Actual Score with Teacher A (10)
Current Years Score
10 of the students passed. Most had moderate to
large gains.Whats this teachers CEI?
Previous Years Score
29
Teacher Bs Students
Group Average Actual Score with Teacher B (84)
Current Years Score
84 of the students passed. All had moderate to
large losses.Whats this teachers CEI?
Previous Years Score
30
Classroom Effectiveness Index (CEI)

• A CEI measures the value added to students
  after receiving instruction from a teacher
• A high CEI indicates that a lot of students
  outperformed their peers in the district

31
Consequences Gaining Trends
Predicted Score Actual Score with Teacher A (10)
Current Years Score
If this gaining trend continues, these students
will likely pass this year...
Previous Years Score
32
Consequences Losing Trends
Predicted Score Actual Score with Teacher B (84)
Current Years Score
If this losing trend continues, these students
will likely fail this year.
Previous Years Score
33
Putting It Together

• CEIs measure the amount of academic progress
  students made after a year of instruction from a
  teacher
• A high CEI indicates that students progressed at
  a greater rate thanlike students in the
  district

34
OK. Effective Teachers. So What?

• Do the CEIs measure something that has a real
  effect, or are we just looking at trivial
  differences?
• The next part of the presentation has a lot of
  answers to this question.

35
Effective Teachers (And Ineffective Teachers)
36
The Research

• RE has conducted three major studies of teacher
  effectiveness,
• Four studies of characteristics associated with
  teacher effectiveness
• And, at least ten evaluations examining what
  makes an effective teacher

37
A Sample of National Studies that Corroborate
DISD Investigations

• Sanders Rivers (1996) Found massive and
  longitudinal effects of teachers
• Marzano (2003) Found massive teacher effects on
  students
• Hanushek Rivkin (2004) Found little
  relationship between teacher experience
  achievement
• Rockoff (2004) Found little relationship
  between experience and achievement after the
  first two years
• Goldhaber Brewer (2000) Found no relationship
  between teacher education coursework and
  achievement

38
The DISD Jordan Study

• The Jordan study looked at longitudinal effects
  of teachers for three years
• Teachers in any one year were divided into 5
  groups from least effective (assigned a 1) to
  most effective (assigned a 5)
• Students were followed for three years by the
  level of teacher to whom they were assigned
• Students had a pretest and three years of test
  scores

39
Nomenclature

• Students were grouped by the level of teacher
  they had in each of the 3 years
• A student assigned to a level 1 teacher, then a
  level 3 teacher and then a level 2 teacher was in
  the 132 group
• Similarly, a student assigned all top teachers
  was in group 555 and a student assigned all poor
  teachers was in group 111

40
Extended the Tennessee Study

• The Jordan study extended a study by Sanders and
  Rivers in Tennessee we noted earlier that looked
  at only 5th grade reading and math
• Jordan looked at grades 3 to 8 reading and math
• Jordan and Sanders found almost identical results
  at grade 5
• Thus, first results were for 4 major cities, two
  major tests, different analyses, different ways
  of computing effectiveness

41
DISD Jordan Results

• Students in 555 versus students in 111 scored 40
  to 50 percentile points higher when initially
  equal on the pretest
• Students assigned to a 155 group still lagged
  behind 555 students at the end of 3 years
• When first assigned a level 1 teacher, students
  dropped an average of 15 to 20 percentile points
• It took 2 years of a level 5 teacher to erase
  most of the effects of 1 year of a level 1 teacher

42
DISD Jordan Results

• Reanalysis of the Jordan data shows
• An average 14 NCE point difference in reading
  between 111 and 555 groups
• An average 20 NCE point difference in math
  between 111 and 555 groups
• An average 10 NCE point difference in reading
  between 12 and 45 groups
• An average 14 NCE point difference in math
  between 12 and 45 groups

43
DISD Jordan Results

• In the reanalysis
• In one year, level 1 reading teachers in the 12
  groups had an average net loss of approximately
  10 NCE points
• In one year, level 5 reading teachers in the 45
  groups had an average net gain of approximately 2
  NCE points
• In one year, level 1 math teachers in the 12
  groups had an average net loss of approximately
  10 NCE points
• In one year, level 5 math teachers in the 45
  groups had an average net gain of approximately 8
  NCE points

44
DISD Bembry/Mendro Study

• Bembry/Mendro looked at 4 years of data
• All 3 year results were confirmed
• Students assigned an ineffective teacher in year
  1 followed by 3 effective teachers lagged behind
  those with 4 effective teachers at the end of 4
  years.
• Differences between all effective teachers and
  all ineffective teachers remained in the 50 point
  range (It was hard for the groups to get farther
  apart.)

45
DISD Bembry/Mendro Study

• The bias in assignment of low scoring students to
  ineffective teachers was more pronounced over 4
  years

46
DISD Babu/Mendro Study

• Babu/Mendro examined 3 years of effectiveness
  relative to TAAS
• Results were similar to Jordan and Bembry/Mendro
  except that the low passing level on the TAAS
  made some of the effects with higher scoring
  students difficult to determine

47
DISD Babu/Mendro Study

• The differences between 555 and 155 were lower
  for the 155 groups. Having one level 1 teacher
  results in a lower passing rate after two years
  of effective teachers
• Having all level 1 and 2 teachers versus all
  level 4 and 5 teachers resulted in TAAS passing
  rates that were 20 to 55 lower on average

48
DISD Babu/Mendro Study

• Low achieving student groups assigned to three
  years of level 1 or level 2 teachers when
  compared to those assigned 3 level 4 or level 5
  teachers produced TAAS meltdown
• Passing rate differences ranged from 22 to 76
  with most between 40 to 50
• Despite the danger to accreditation, low
  achieving students were assigned to level 1 and
  level 2 teachers more frequently than to higher
  level teachers

49
DISD Stability

• Teachers in the 1 and 5 levels generally stay at
  level 1 and 2 or level 4 and 5 for 3 years
• Principals have little or no effect on the group
  effectiveness of teachers whether they stay at
  their school or are moved to a new school
• Principal effects are related mostly to the
  effectiveness of teachers that are brought in or
  that leave

50
DISD Effectiveness Distributions

• Beginning teachers have slightly lower
  effectiveness indices for the first 2 or 3 years.
  After this, there is no relation between
  experience and effectiveness
• Alternatively certified teachers are slightly
  lower in effectiveness as a group
• The variance of these groups are almost
  identical, meaning many teachers are effective or
  not when they begin or regardless of
  certification