Creating Teaching

1
Creating Teaching Quality Data Systems
Ed Fuller, PhD Senior Research Associate Center
for Teaching Quality Wilmington, Delaware April
29, 2006
2
TOPICS Philosophical Principles One State
Teaching Quality Data System Technical
Principals Specifics of Value-Added Methodology
3
Philosophical Principles in Creating Teaching
Quality Data Systems 1.  Data should not be
collected and analyzed in order to punish
individuals, programs, or agencies.    2.  
Student achievement data from standardized tests
should not be the only measure used to gauge
student success.   3.   Creating a useful TQ
data warehouse and system requires the
participation of state agencies, preparation
programs, and school districts.   4.   States
should invest in experts to ensure that student
achievement examinations possess the proper
psychometric properties necessary for value-added
analyses.   5. Both quantitative and
qualitative data should be collected.
4
Philosophical Principles 1.  Data should not be
collected and analyzed in order to punish
individuals, programs, or agencies.    The data
and the statistical methodologies are simply not
refined enough to make precise judgments about
individual teachers, strategies, or programs.
Thus, punishing people based on imprecise
measures is neither ethical nor good management
practice.
5
Philosophical Principles 2.   Student
achievement data from standardized tests should
not be the only measure used to gauge student
success. As Dr. Eric Hanushek recently
recommended to the Texas Legislature, student
test scores should never be used as a sole
indicator of teacher or teaching effectiveness.
Rather, he and other thoughtful advocates suggest
that such evidence become one piece of a larger
body of evidence upon which districts can reward
some teachers and target professional development
to other teachers. Other such evidence could
include principal evaluations, peer group
evaluations, portfolios, and self-assessment
products such as those used in the National Board
Certified Teacher process.  
6
Philosophical Principles 3.   Creating a useful
TQ data warehouse and system requires the
participation of state agencies, preparation
programs, and school districts. The use of a
value-added model that aims to identify more and
less effective teachers typically requires the
integration of data sets from the state and the
district. The state can typically provide the
student assessment information while the district
typically must match the teachers to
students. Furthermore, additional data from
teacher preparation programs are necessary to
answer such important questions such as, Are
differences in pre-service experiences associated
with differences in teacher effectiveness?
7

• Philosophical Principles
• 4.   States should invest in experts to ensure
  that student achievement examinations possess the
  proper psychometric properties necessary for
  value-added analyses.
• The foundation of any well-designed value-added
  analysis of teacher effectiveness is a
  high-quality test.
• Tests must be administered at least every year.
• Tests must valid and reliable.
• Tests must be properly constructed so that
  students must actually know the content in order
  to correctly answer the question.
• Tests must have a large enough range so that
  there is no ceiling effect.
• Tests must be constructed in such a way that
  scores are comparable across years and grade
  levels.

8

• Philosophical Principles
•  
• Both quantitative and qualitative data should be
  collected.
• Quantitative data alone will not be able to
  answer all of the pertinent questions regarding
  teacher effectiveness. Even some of the most
  ardent supports of pay-for-performance plans
  argue that qualitative data such as principal
  evaluations, peer evaluations, and portfolios
  must be used in conjunction with quantitative
  data in order to accurately identify the most and
  least effective teachers.
• Moreover, only quantitative data can reveal the
  actual teaching practices employed by the most
  effective and least effective teachers. Learning
  such information is the only way in which school
  districts can develop effective professional
  development and instructional leadership.

9
Designing a Statewide Teaching Quality Data System
10
Preparation Program Characteristics
District Responsibility
Teacher SSN
State Education Agency Responsibility

Preparation Program Number
Preparation Program Responsibility
Graduate Characteristics

Linkages
Teacher SSN
Teacher Certification Data
School Demographic Achievement Data
Teacher SSN
Teacher Assignment Data
Campus Number
Student ID
District Demographic Achievement Data
Student Achievement Data
11
Preparation Program Characteristics
District Responsibility
Teacher SSN
State Education Agency Responsibility

Preparation Program Number
Preparation Program Responsibility
Graduate Characteristics

Linkages
Teacher SSN
Teacher Certification Data
School Demographic Achievement Data
Teacher SSN
Foundation file
Teacher Assignment Data
Campus Number
Student ID
District Demographic Achievement Data
Student Achievement Data
12
Preparation Program Characteristics
District Responsibility
Teacher SSN
State Education Agency Responsibility
Preparation Program Number
Preparation Program Responsibility
Graduate Characteristics

Linkages
Teacher SSN
Key linkage
Teacher Certification Data
School Demographic Achievement Data
Teacher SSN
Teacher Assignment Data
Campus Number
Student ID
District Demographic Achievement Data
Student Achievement Data
13
Preparation Program Characteristics
District Responsibility
Teacher SSN

State Education Agency Responsibility
Preparation Program Number
Graduate Characteristics

Preparation Program Responsibility
Teacher SSN
Linkages
Teacher Certification Data
School Demographic Achievement Data
Teacher SSN
Only districts can make this link
Teacher Assignment Data
Campus Number
Student ID
District Demographic Achievement Data
Data everyone needs
Student Achievement Data
14
Technical Principles of Building a Teaching
Quality Data System
15
Unique Identifier Because comprehensive data at
each levelstudent, teacher, school, and
preparation programmust be drawn from a
combination of other databases at the same level,
a unique identifying number should be included in
each data set. For example, all data sets that
include information on students should include a
unique identifying number for each student. 
Because most useful analyses must examine data
longitudinally, the unique identifying numbers
should remain constant over time.
16
Additional Unique Identifiers Because most
useful analyses require data from multiple
levels, data sets at each level also should
include another unique identifying number that
can be linked to another level of data. For
example, teacher data sets should include both a
unique number that identifies the teacher and a
unique number that identifies the school(s) that
employ(s) the teacher. Because most useful
analyses must examine data longitudinally, the
unique identifying numbers should remain constant
over time.
17
Data Accuracy Data should be as complete and as
accurate as possible. At each level, investments
should be made to ensure accurate data submission
through the review and cleaning of data. This
is CRITICAL when it comes to correctly matching
students to teachers and keeping track of
students migration patterns in and out of
classrooms and schools.
18
Privacy Issues are Paramount The privacy of
individual records in the database must be
protected, and any release of records to
researchers or members of the public should be
without Social Security numbers or other personal
identifiers.
19

• Most Important Data Element
•  
• Students matched to teachers
•  
• Must deal with
•  
• team teaching
• students who move from school to school or switch
  classrooms
• pull-out teachers
• teachers who quit or start mid-year
• students who change schedules mid-year

20

• Second Most Important Element
•  
• Student achievement scores
•  
• Must have
•  
• At least two years of scores, preferably three
  years
• A test with vertically aligned scale scores
• A test aligned with the curriculum in all
  classrooms and schools
• A test with a wide score range to avoid a ceiling
  and floor effect
• A test that measures a range of topics
• A test that is valid and reliable
• Good to have two different tests to check the
  accuracy of the effects.

21
Other Variables   Should student background
characteristics be included?     Maybe.
Typically, the omission of any data that has an
effect on test scores will result in biased
(incorrect) estimates of teacher effectiveness.
Thus, the researcher must test to see if
variables such as economically disadvantaged
status, race/ethnicity, gender, age, parental
level of education, and neighborhood
characteristics have an independent effect on
student achievement.
22
Other variables   What about including classroom
and school characteristics?     These should
possibly be included.   Again, the omission of
any data that has an effect on test scores will
result in biased (incorrect) estimates of teacher
effectiveness. Thus, the researcher must test to
see if variables such as class size, the of
economically disadvantaged students, of
students from different racial/ethnic categories,
of students male, and other variables have an
independent effect on student achievement.
23
Missing Achievement Data   Missing data is a very
important and often over-looked problem. When
students move from one class to another or one
school to another, the data can be missing.
Too much missing dataespecially when the missing
data is not random, can have a serious effect on
the estimates of the effects. Data can also be
missing because of special education or LEP
exemptions, students absent from testing, and
other reasons.   This is a huge problem at the
school level. In Texas, less than 50 of
students in urban high schools have valid scores
in both 9th- and 10th-grade.
24
RECOMMENDATIONS   Invest heavily in developing an
excellent testthis is the key component.   Invest
in computer system that can track students,
teachers, test scores.   Collect other evidence
on teacher effectiveness and use the VAM and
other evidence collectively to make judgments of
teacher effectiveness. Create a system to
collect important student, classroom, and school
characteristics.  
25
976 Martin Luther King, Jr Blvd Suite 250 Chapel
Hill, North Carolina 27514 (919)
951-0200 www.teachingquality.org www.teachingdata
.org efuller_at_mail.utexas.edu [phone]