A Comparison of Two ValueAdded Models:

1
A Comparison of Two Value-Added Models

• Some major considerations related to implementing
  a value-added assessment system?
• Value-Added Conference
• Albany, NY
• May 28, 2008

2
The Trend Toward Value-Added .

• Several national organizations, publications, and
  panels support this metric for gauging school
  quality
• Education Commission of the States (2002)
• Education Week (Room for Improvement, 9/22/04)
• Education Trust (2004)
• American Educational Research Association (2004)
• National School Boards Association (2003)
• The Teaching Commission (2004)
• Carnegie Corporation of NY (2003)
• Etc.

3
The Trend Toward Value-Added .

• As of 2002, districts in 21 states had
  value-added projects (Education Week, November
  20, 2002).
• More than 300 of these school districts have
  formed contracted with professionals for VA work.
• Several states now have legislation for
  value-added measurements of school progress
• Arizona, Colorado, Florida, North Carolina, Ohio,
  and Tennessee (House Research Organization, Texas
  House of Rep., 2004).
• Pennsylvania via PA Department of Ed. Regulation.
  

4
The Question is Increasingly not If a .. - but
Which? VA system?
Considerations for Choosing a Value-Added Model
for Calculating and Reporting school effects on
student learning?
5
The Primary Works Guiding this Observation

• Millman, J. (1997). Grading Teachers, Grading
  Schools Is Student Achievement a Valid
  Evaluation Measure? (Corwin Press Thousand
  Oaks, CA).
• McCaffrey, D.F., Lockwood, J.R., Koretz, D.M.,
  Hamilton, L.S. (2003). Evaluating Value-Added
  Models for Teacher Accountability. (RAND Santa
  Monica, CA).
• Stronge, J.H. Tucker, P.D. (2000). Teacher
  Evaluation and Student. (National Education
  Association Washington, DC).
• Value-Added Assessment Special Issue (Spring
  2004). Journal of Educational and Behavioral
  Statistics, volume 29, Number 1.
• 16 other reference materials

6
Two Primary Value-Added Models

• Two models consistently emerged in the
  literature.
• The Multivariate VAM
• The Covariate Adjustment VAM
• These two VAMs are most recognizable by the
  countrys two longest running VA Systems
• Multivariate VAM
• Tennessee Value-Added Assessment System (William
  Sanders)
• The Covariate Adjustment VAM
• Dallas Value-Added Accountability System (William
  J. Webster, Robert L. Mendro )
• These two models share similarities and
  differences with regard to the considerations.

7
Covariate Adjustment Models

• Covariate adjustment models specify the current
  score as a function of the prior score and
  possibly other covariates, using separate models
  for each year, explicitly linking students
  scores to the effects of the current teacher
  only. (McCaffrey et al., 2003, p. 55)

8
Multivariate VAM

• Multivariate models directly specify a joint
  distribution for the entire multivariate vector
  of scores for the student. The models express the
  score means as a function of time, specify the
  variances and correlations between pairs of
  scores for different years, and link students
  scores to teacher effects from multiple years.
  (McCaffrey et al., 2003, p. 57)

9
Differences in Model Specifications for
Calculating a Given Years VA Measures
10
Considerations for VA ModelsWhat are important
VAM characteristics?

• Treatment of school and student characteristics
• Transparency of VA models algorithm
• Treatment of student records with missing data
• Inclusion of other subject test results
• Reporting of VA measures to the public
• Contributions to Research
• Safeguards against results stemming from sampling
  error
• Safeguards against possible distortion of data
• Reporting of VA measures to the public
• Existence in overall state accountability systems
• Price

11
Consideration 1a Treatment of Differing Student
Characteristics
12
Consideration 1b Treatment of Differing School
Characteristics
13
Poor and Minority Students are Disproportionately
found w/Teachers with Temp/Emerg. Cert. Weak
Preparation.

• The research firm SRI found in California more
  than 40,000 classroom teachers were teaching on
  emergency permits or waivers in 1999-2000. Low
  achieving schools were nearly five times as
  likely as high-achieving schools to employ these
  teachers. High-minority schools were nearly seven
  times as likely as low-minority schools to employ
  them.
• In Illinois, a Chicago Sun-Times investigation
  revealed that teachers in schools with the
  highest proportions of poor, minority and
  low-performing students were five times more
  likely to have failed at least one
  teacher-competency test than teachers in the
  state's most affluent schools. Nearly one-fifth
  of Chicago teachers failed at least one teacher
  test, which is 3.5 times the failure rate of
  suburban teachers

Prince, C. (2002). Missing top staff in bottom
schools the challenge of attracting exemplary
teachers to neediest schools. School
Administrator Website (http//www.findarticles.com
/p/articles/mi_m0JSD/is_7_59/ai_89927216 )
Date Retrieved 10/24/04.
14
Relationship Between Advanced Degrees and Student
Poverty/Ethnicity
Watson, S., Foley, E., Tighe, E., Wang A.
(2001). Recruiting and Retaining Teachers Keys
to Improving the Philadelphia Public Schools.
(Consortium for Policy Research in Education
Philadelphia, PA).
15
Low Math Achievers are More Likely to be Assigned
Ineffective Teachers
Carey, K. (2004). The Real Value of Teachers
Using New Information about Teacher Effectiveness
to Close the Achievement Gap. (Washington, DC
The Education Trust).
16
Low Reading Achievers are More Likely to be
Assigned Ineffective Teachers
Carey, K. (2004). The Real Value of Teachers
Using New Information about Teacher Effectiveness
to Close the Achievement Gap. (Washington, DC
The Education Trust).
17
Consideration 2 Transparency of VAMs algorithm
18
Consideration 3 Treatment of student records
with missing data
19
Common Gain Patterns Relative to Student
Achievement
Downward Shed Pattern
Low Achievers
Middle Achievers
High Achievers
Upward Shed Pattern
Patterns of Gain likely relate to the priorities,
resources, and skills of the schools/teachers.
20
The Effects of Not Using All Student Records
(Wright, 2004)
Bias
Variance
Gain Estimate Error
Average Error Magnitudes for 3 VA Models
Assuming Downward Shed Pattern Student
Records Missing Mostly at Non-Random
21
Consideration 4 Inclusion of other subject test
results
22
Consideration 5 Reporting of VA measures to the
public
23
Consideration 6 Contributions to Research
24
Consideration 7 Safeguards against results
stemming from sampling error
25
Consideration 8 Safeguards against possible
distortion of data
26
Consideration 9 Price
27
Consideration 10 Existence in Overall
Accountability System
28
Bibliography

• Ballou, D., Sanders, W.L., Wright, P. (2004).
  Controlling for Student Background in
  Value-Added Assessment of Teachers. Journal of
  Educational and Behavioral Statistics. p. 37
  65.
• Bembry, K.L. Shumacher, R.E. (2003).
  Investigating a Classroom Effectiveness Measure
  over Time. Paper presented at the annual meeting
  of the American Educational Research Association.
  (April 25, 2003 Chicago, IL).
• Carey, K. (2004). The Real Value of Teachers
  Using New Information about Teacher Effectiveness
  to Close the Achievement Gap. (The Education
  Trust Washington, DC).
• Cross, R.W., Rebarber, T., Torres, J. (2004).
  Grading the Systems The Guide to State
  Standards, Tests, and Accountability Policies.
  (The Fordham Foundation Washington, D.C.)
• Fitz-Gibbon, C. Tynne, P. (2002). Technical
  and Ethical Issues in Indicator Systems Doing
  Things Right and Doing Wrong Things. Education
  Policy Analysis Archives, V10 n6. Retrieved
  October 21, 2004 from http//epaa.asu.edu/epaa/v
  10n6/.
• Hobbs, T.D. (2004). Personal Communication
  (September 15, 2004).
• Kain, J.F. (1998). The Impact of Individual
  Teachers and Peers on Individual Student
  Achievement, paper presented at the Association
  for Public Policy Analysis and Management 20th
  Annual Research Conference, New York, October 31.
  
• McCaffrey, D.F., Lockwood, J.R., Koretz, D.M.,
  Hamilton, L.S. (2003). Evaluating Value-Added
  Models for Teacher Accountability. Research
  conducted for the Carnegie Corporation of New
  York. (RAND Education Santa Monica, CA)
• Mendro, R., Jordan, H., Gomez, E., Anderson, M.,
  Bembry, K. (1998). An application of multiple
  linear regression in determining longitudinal
  teacher effectiveness. Paper presented at the
  1998 Annual Meeting of th e AERA, San Diego, CA.
  
• Rivers, J.C. (1999). The Impact of Teacher
  Effect on Student Math Competency Achievement,
  dissertation, University of Tennessee, Knoxville.
  Ann Arbor, MI Univerity Microfilms
  International, 9959317, 2000.

29
Bibliography

• Sanders, W.L. (2004). Personal Communication
  (August 9, 2004)
• Sanders, W.L. (2004). Personal Communication.
  (August 13, 2004)
• Sanders, W.L. Rivers, J. C. (1996). Cumulative
  and residual effects of teachers on future
  student academic achievement. Knoxville, TN
  University of Tennessee Valued-Added Research
  Center.
• Sanders, W.L., Saxton, A.M., Horn, S.P. (1997).
  The Tennessee Value-Added Assessment System A
  Quantitative Outcomes-Based Approach to
  Educational Assessment. In J. Millmans (Ed.),
  Grading Teachers, Grading Schools Is Student
  Achievement a Valid Evaluation Measure? (p.
  137-162). (Corwin Press Thousand Oaks, CA)
• SAS Institute (2004). About the Tennessee
  Value-Added Assessment System (TVAAS).
  (Retrieved August 20, 2004 weblink
  http//tvaas.sasinschool.com/evaas/help/TVAAS/TVAA
  S_help_about.html.
• Stronge, J.H. Tucker, P.D. (2000). Teacher
  Evaluation and Student Achievement. NEA
  Washington, DC.
• von Hippel, P. (2004). Good News on the
  Accountability of Small Schools A comment on
  Kane and Staiger (2002). Journal of Economic
  Perspectives, v18 n3.
• Webster, W.J. (1998). A Comprehensive System for
  the Evaluation of Schools. Paper presented at
  the annual meeting of the American Educational
  Research Association (April 1-17, 1998 San
  Diego, CA).
• Webster, W.J. (2002). Dallas Independent School
  District Accountability System. PowerPoint
  Presentation made to Policy Seminar on No Child
  Left Behind Implications for Michigan Schools
  (Decmber 17, 2002)
• Webster, W.J. Mendro, R.L. (1997). The Dallas
  Value-Added Accountability System. In J.
  Millmans (Ed.), Grading Teachers, Grading
  Schools Is Student Achievement a Valid
  Evaluation Measure? (p. 81-99). (Corwin Press
  Thousand Oaks, CA)