Measuring School, Grade Level

1
Measuring School, Grade Level Teacher
Performance Using Value-Added Analysis by John
Schacter, Ph.D. President The Teaching
Doctors http//valueadded3.teachingdoctors.com
schacter_at_teachingdoctors.com Tel - 650.323.1378
2
About Me

• My mom is a teacher
• My wife is a teacher (Statistics at Stanford)
• Im a teacher
• First grade
• Elementary science
• Summer School Principal
• San Jose State University
• Former Vice President of Research at the Milken
  Family Foundation
• Senior Fellow at Stanford University
• And most importantly, I have a son, Elijah, who
  wants to join my company, The Teaching Doctors.

3
When I told Elijah I was going to talk about
statistics he said . . . Daddy, I like biscuits,
can I come?
4
Presentation Outline

• Voting Statistics, Not Biscuits
• Single Point in Time Achievement Tests
• Value-added Analysis
• Now What

5
Voting Systems

• 21 friends go out to eat. 13 prefer light and
  healthy. 8 love red meat, potatoes, and butter.
  One of the friends proposes that they vote to
  decide where they should eat.

6
Three Restaurant Choices
Sushi
Steak
Thai
7
Plurality Voting Steak Wins

• Notice how under this voting system our 13 light
  and healthy eaters (the real majority) are stuck
  eating steak.

8
Borda Voting Sushi Wins

• Notice how under this voting system the healthy
  eaters get food more to their liking.

9
Instant Run Off Voting

• Sushi is eliminated because of the fewest votes.

Steak is eliminated in the Run Off.
10
Voting Statistics

• With voting, the majority (healthy eaters) were
  BLINDED by a plurality voting system.
• With current state achievement reporting,
  schools are BLINDED because the results do not
  reflect their true impact on students learning.

11
Voting Statistics

• Single point in time achievement test results
  dont tell you much about your performance as a
  school or teacher.
• To teach, not preach this idea, see the
  following slides for the achievement results from
  a real school called Jefferson Elementary.
• Analyze the single point in time achievement
  data results, then answer the questions at the
  end. You have 5 minutes.

12
Jefferson Achievement Results
3.
2.
1.
4.
4.
3.
13
Jefferson Achievement Results
6.
5.
14
Jefferson Achievement Results
ACHIEVEMENT RESULTS ANSWERS VALUE-ADDED
RESULTS ANSWERS
15
Jefferson Achievement Conclusions

• Language Arts
• ELL students are learning much less in Language
  Arts than Non-ELL students
• Highest performing grade 7 lowest performing
  grade 8
• Highest performing teacher Sisley lowest
  performing teacher Bing
• Math
• Math and all ELL students Math grades 5, 6, 7
  Teachers Bing, Brush Obanon in Math and Bing
  Brush in ELA.

16
How Much of a Students Single Point in Time
Achievement Test Score is Due to Schools?
Only 15 of a students single point in time test
score is due to the school she attends. The rest
is attributable to family wealth and education
(see research by James Coleman (1966), Chris
Jencks (1973) Eric Hanushek (1998).
17
What Researchers Have Said Through the Decades
About Single Point In Time Achievement Test
Scores

• Harvard Professor Christopher Jencks
• A schools achievement output depends almost
  entirely on the characteristics of the entering
  children.
• James Coleman
• The educational resources provided by a child's
  family and neighborhood are far more important
  for his achievement than are the resources
  provided by the school.
• Harvard Professor Howard Gardner stated
• We can accurately project a childs chances of
  completing college and her eventual income by
  knowing only her ZIP code.

18
Single Point in Time Achievement Tests

• Students that live in neighborhoods with wealthy
  educated parents enter school with higher test
  scores than students from middle and lower income
  neighborhoods, and these differences persist.
• Single point in time achievement test scores have
  virtually nothing to do with schools or teachers.

19
Proof With Randomly Selected School Districts In
Northwest Georgia

• I selected the following districts
• Walker 60 Free Lunch Students
• Trion 30 Free Lunch Students
• Chatooga 64 Free Lunch
• Rome 72 Free Lunch Students
• Bremen 25 Free Lunch Students
• Research shows that family and neighborhoods
  account for 85 percent of a students single
  point in time test score.
• If this statistic is true, which districts from
  the list above will have the schools with the
  highest test scores?
• Take a minute, and order the districts from
  highest to lowest, based on what you think their
  achievement test scores were in 2007.

20
Single Point in Time Achievement Tests
4/5 80
Percent of students achieving at the Advanced
performance level
21
Single Point in Time Achievement Tests

• Do these results mean that
• Breman and Trion counties have better
  administrators, schools, and teachers, than
  Walker, Chatooga Rome?
• NO!
• These schools just happen to be located in
  neighborhoods with higher parental income.

22
Single Point in Time Achievement Tests

• These results just dont happen in Northwest
  Georgia.
• They happen in every district in the United
  States.
• Go home and pick 5 schools from high income
  neighborhoods and compare their single point in
  time achievement test scores with 5 schools in
  low and middle income neighborhoods.
• 8-to-9 times out of 10 you will find that schools
  in the high income neighborhoods achieve
  considerably more than the schools in the low and
  middle income neighborhoods.

23
Single Point in Time Achievement Tests

• There are two points I want to make about single
  point in time achievement tests
• They dont measure school or teacher performance.
• Making intervention decisions based on single
  point in time achievement tests will be flawed.

24
Presentation Outline

• Voting Statistics, Not Biscuits
• Single Point in Time Achievement Tests
• Value-added Analysis
• Now What

25
Value-Added Analysis The Right Tool for the Job

• Value-added analysis is a statistical
  methodology that measures the same set of
  students learning gains from one year to the
  next!
• Value-added gains ARE NOT effected by family
  wealth and education, because they measure gains
  in achievement, not the students achievement at
  one point in time.
• Value-added analysis reflects the schools and
  teachers true impact on student learning, not
  the students parents income and educational
  levels.

26
Value Added A Concrete Example

• Ms. Jones is a 4th grade teacher. She teaches in
  a middle income neighborhood school where
  approximately 70 percent of the students are
  proficient in language arts.
• Students who enter her class have a reading test
  mean of 60 NCEs.
• At the end of the year, the class reading test
  mean is 61 NCEs.
• The value Ms. Jones added to her students
  reading achievement from 3rd to 4th grade was 1
  NCE.

27
Ms. Jones Class Reading Gains Versus the
District Reading Gains
Mrs. Jones
School
State
When we compare Ms. Jones students gains to
other teachers in her school, and to the state
her students do not gain as much.
28
The Reference Group

• It turns out the mean state teachers reading
  gain was 10 NCEs.
• Ms. Jones class gained 1 NCE.
• Her students only gained 1/10th of what the
  average fourth grader gained in the state in
  reading.
• Stated another way, Ms. Jones students gained
  90 less than the average 4th grade classroom in
  the state
• (1/10100) - 100 - 90

29
Value-Added Analysis The Right Tool for the Job

• If we only look at Ms. Jones percent of students
  who are proficient (e.g., around 70), we draw
  the wrong conclusion.
• By measuring how much Ms. Jones students gained
  in achievement from one year to the next, we
  uncover a much more accurate estimate of her
  impact.

30
Value-Added Analysis The Right Tool for the Job

• On the following slides are Jefferson Elementary
  Schools value-added results by school, grade,
  and teacher
• With a partner, analyze the data and answer the
  same set of questions you answered for the
  Achievement Results, for these Value-Added
  Results in the Value-Added column.

31
Value Added A Concrete Example

• Remember the Ms. Jones example
• Just like we did with her, we calculated the
  value-added percent gains for Jefferson
  Elementary.
• When you read and interpret our charts, you are
  comparing the school, grade, and teacher level
  results to the district average gain.

32
Jefferson Value-Added Results
33
Jefferson Value-Added Results ELA by Grade
34
Jefferson Value-Added Results Math by Grade
35
Jefferson Value-Added Results ELA by Teacher
36
Jefferson Value-Added Results Math by Teacher
37
Comparing Jefferson Achievement Conclusions to
Value-Added Conclusions
Value-Added
Achievement

• Language Arts
• ELL students are learning much less in Language
  Arts than Non-ELL students
• Highest performing grade 7 lowest performing
  grade 8
• Highest performing teacher Sisley lowest
  performing teacher Bing
• Math
• Math and all ELL students Math grades 5, 6, 7
  Teachers Bing, Brush Obanon in Math and Bing
  Brush in ELA.

• Math
• They are learning more in English and the same in
  Math
• Highest performing grade 8 lowest performing
  grade 7
• Highest performing teacher Flesh lowest
  performing teacher Dinner
• Language Arts
• Professional Development for ELA instruction as
  only 1 teacher in the school achieved positive
  learning gains.

38
Comparing Jefferson Achievement Conclusions to
Value-Added Conclusions

• The conclusions we came to based on the single
  point in time achievement results, are completely
  different from the conclusions we reached based
  on value-added analysis results.
• Which results reflect your school and teacher
  performance?
• The value-added results.
• Let me show you further evidence why this
  statement holds true.

39
The Research On Value-Added Analysis

• Family Income Schools serving high percentages
  of economically disadvantaged students gain just
  as much as schools serving low percentages of
  economically disadvantaged students.
• Minorities Schools with high minority
  populations gain just as much as schools with low
  minority populations.
• Initial Achievement Schools with high initial
  achieving students gain just as much as schools
  with low initial achieving students.
• Other Student Demographics English Learners,
  Gifted, and Special Education students all
  exhibit similar learning gains.
• None of this is true for single point in time
  achievement test results.

40
The Right Tool For the Job Value-Added Analysis
Each dot represents one school Horizontal
line at 0 represents gain equal to national
norm gain
Free Lunch
41
The Right Tool For the Job Value-Added Analysis
Each dot represents one school Horizontal line
at 100 represents gain equal to state gain
Percent Minority
42
Presentation Outline

• Voting Statistics
• Single Point in Time Achievement Tests
• Value-added Analysis
• Now What

43
The Problem of Intervening Based on The Average

• Kevin O'Keefe wrote a book titled, The Average
  American Here are some of the averages he found.
• Who in this room is the average age, 36.6 years
  old?
• Raise your hand if you exercise the average 20
  minutes a day?
• The average is 20 minutes b/c 17 of us exercise
  an hour and a half, while the rest of us do
  nothing!
• Before you came to this meeting, did you spend
  the average 24 minutes grooming yourself?
• Who drove to this meeting in the average
  eight-year-old car?
• In the past year, how many of you bought 264 cans
  of beer?
• Yes, thats the national average.

44
The Problem Of Intervening Based On the Average

• When all we do is look at the average, we realize
  that few of us are average.
• And if thats the case, we need to make decisions
  in our schools based on every individual.
• Value-added can help you do this by aiding you in
  placing the right student with the teacher that
  will teach that student most.
• Heres how.

45
Place the Right Student With The Right Teacher

• Lets look at three 7th grade teachers
  value-added analysis results to see how to place
  the right student with the right teacher.

46
Analyze Each Teachers Average Gain by Subject
Two of the 7th grade teachers (Adelman
Chemarones) produce about a 10 percent gain in
English Language Arts, the other (McNutt)
produces an 18 percent loss.
47
Analyze Each Teachers Average Gain by Subject
Two teachers produce about 80 percent gains
(McNutt Chemarones) in Math, the other, an 8
percent gain (Adelman).
48
How Can Schools Place the Right Students with the
Right Teachers?

• ELA - Two teachers (Adelman Chemarones) produce
  10 percent gains, the other (McNutt) an 18
  percent loss.
• MATH - Two teachers produce 80 percent gains
  (McNutt Chemarones), the other, 8 percent
  (Adelman).
• What do we need to do to place the right students
  with the right teachers? Tell the person sitting
  next to you.
• One possibility, McNutt (80 math, -18 ELA),
  from now on youre teaching Math only, and
  Adelman (8 Math, 12 ELA), youre teaching
  English Language Arts only. Chemarones (76
  Math, 10 ELA) youre teaching both. We are
  going to re-organize student assignments.
• This decision isnt terrible, but its based on
  the average, and we know that none of us are
  average.

49
Going Beyond the Average
Chemarones students gained on average 10 more
than the state in English Language Arts.
We determined that she is an effective English
Language Arts teacher.
50
Chemarones Individual Student Gains
Mark each students pretest and posttest on a
scatter plot
ELA NCE SCORE 2006
ELA NCE SCORE 2005
51
Chemarones Individual Students Gains
Draw the expected gain line based on national,
state or district growth
ELA NCE SCORE 2006
ELA NCE SCORE 2005
52
Chemarones Individual Student Gains
0
86
86
Divide the students into 3 groups

• List the percentage of students in each group
  that achieved gains greater than the expected
  growth.

53
Going Beyond the Average

• Ms. Chemarones did an outstanding job with her
  low and middle achieving students.
• She didnt do well with her high achieving
  students.
• Based on the INDIVIDUAL, not the AVERAGE, wed
  place low and middle achieving ELA students with
  Ms. C., and see if we can find the right teachers
  for the high achieving students.

54
Going Beyond the Average
McNutts students lost 18 in ELA when compared
to other 7th graders in the state.
When we only looked at her class average gain,
our advice was, McNutt, from now on youre
teaching Math only!
55
Mrs. McNutts ELA Gains by Student
100
14
0

• What students should be placed with McNutt?

56
Place the Right Students with the Right Teachers

• When we used the average teacher gain, we didnt
  place the right students with the right teachers.
  
• We needed to look at each individual students
  performance by teacher to create assignments that
  will maximize learning for every pupil.

57
Presentation Outline

• Voting Statistics
• Single Point in Time Achievement Tests
• Value-added Analysis
• Now What

58
Place the Right Students with the Right Teacher

• For more information on how schools and districts
  have used our value-added analyses to improve
  student learning and intervention planning
  contact
• John Schacter, Ph.D.
• Tel (650) 323-1378
• Email schacter_at_teachingdoctors.com
• Web http//valueadded3.teachingdoctors.com

59
Measuring School, Grade Level Teacher
Performance Using Value-Added Analysis by John
Schacter, Ph.D. President The Teaching
Doctors http//valueadded3.teachingdoctors.com
schacter_at_teachingdoctors.com Tel - 650.323.1378