LINKING TEACHER PREPARATION TO P12 STUDENT LEARNING AFTER CANDIDATES GRADUATE

1
LINKING TEACHER PREPARATION TO P-12 STUDENT
LEARNING AFTER CANDIDATES GRADUATE

• ASSESSING CANDIDATE AND COMPLETER IMPACT ON P-12
  STUDENT LEARNING
• FACTE PRE CONFERENCE
• HILDA ROSSELLI, DEAN
• WESTERN OREGON UNIVERSITY
• rossellh_at_wou.edu

2
Identifying the link between teacher education
and student achievement may be one of the most
complex tasks undertaken by the education
community.

• Linda Quinn (2004) Overview and Framework

3
Are there any other comparable scenarios?

• Medicine

Rehabilitation
Fitness

• Religion

4

• Students sail in, but no one knows what happens
  to them after they come out. No on knows which
  students are succeeding as teachers, which are
  struggling, and what training was useful or not.
• Arne Duncan
• The Chronicle, October 9, 2009

5
Race to the Top Theme

• Great Teachers and Leaders
• Knowing which teachers and principals are
  effective, as judged in large part by multiple
  measures of student achievement and growth
• Ensuring that local decision-makers are able to
  use this effectiveness information to inform
  key decisions, such as, evaluation, compensation,
  tenure, promotion and dismissal
• Rewarding excellence and attracting effective
  talent to the school and subject areas where its
  needed most
• Shining a light on which credentialing/preparation
  programs best prepare teachers and principals
  for success

6
Possible Approaches

• Link student performance back to individual
  teachers and their preparation programs.
• Study student progress employing measures other
  than test performance.
• Analyze how preservice preparation affects
  teacher behaviors, instructional practices and
  career decisions, instead of student outcomes.
• Analyze employment evaluation data.

7
Confounding factors

• To what degree were the candidates preservice
  experiences in schools similar to their first
  years of teaching in terms of grade level and
  subject area.
• 2008-09 Oregon data showed that of 644 mentees
• 13 stated they were assigned to a grade level
  for which they were not licensed and
• 19 stated they were assigned to a subject area
  for which they were not licensed.

8
Complications

• Candidates self select their pathway, programs,
  and their employment.
• Programs and their treatments differ greatly.
• Hiring practices often place new teachers in the
  most challenging positions.
• Mentoring varies greatly across districts.
• Establishing causality and factoring in
  mitigating factors is difficult.

9
Complex questions

• What is the relative importance of
  university-based teacher preparation in
  accounting for the academic progress of K-12
  students compared with the relative strength of
  other factors that are known to influence student
  learning such as student factors, school factors
  and community factors?

10
Types of Analyses
Teacher Preparation and Student Achievement
(2008)) D. Boyd, P. Grossman, H. Lankford, S.
Loeb, and J. Wyckoff
11

• Founded in 2005 by 10 organizations (now 14),
  this national effort advocates for the
  development of state data systems to improve
  student achievement.
• The primary focus of the first three years of the
  campaign was on building the political will for
  states to implement 10 essential elements of a
  longitudinal data system.

http//www.dataqualitycampaign.org/
12
Current Managing Partners

• Achieve, Inc
• Alliance for Excellent Education
• Council of Chief State School Officers
• Education Commission Of The States
• Education Trust
• National Association of State Boards of Education
• National Association of System Heads
• National Center for Educational Achievement
• National Center for Higher Education Management
  Systems
• National Conference Of State Legislatures
• National Governors Association Center for Best
  Practices
• Schools Interoperability Framework Association
• State Educational Technology Directors
  Association
• State Higher Education Executive Officers

http//www.dataqualitycampaign.org/
13
Endorsing Partner Agreement

• As an Endorsing Partner, our organizations role
  and responsibilities include the following
• attending Data Quality Campaign quarterly
  meetings (on specific topics)
• identifying an organization liaison with whom DQC
  will communicate
• featuring the DQC at conferences, board meetings
  and other gatherings of our constituencies
• highlighting the DQC efforts in regular
  communications with our constituencies
• promoting a focus on data quality (and the
  messages of the DQC) in ongoing efforts
• using the DQC materials and tools as part of
  on-going efforts
• sending a 50- word company description, web-ready
  logo and organization URL
• adding the DQC logo to the organizational web
  site with a link to the www.DataQualityCampaign.or
  g web site
• providing input and guidance to the campaign
  direction and resources/tools and
• being listed on DQC documents, materials, website
  as an Endorsing Partner, as appropriate.

14
2008 State Survey of 10 Elements

• 1. Statewide Student Identifier (48)
• 2. Student-Level Enrollment Data (49)
• 3. Student-Level Test Data (48)
• 4. Information on Untested Students (48)
• 5. Statewide Teacher Identifier with a
  Teacher-Student Match (21)

15
2008 State Survey of 10 Elements

• 6. Student-Level Course Completion (Transcript)
  Data (17)
• 7. Student-Level SAT, ACT, and Advanced Placement
  Exam Data (29)
• 8. Student-Level Graduation and Dropout Data (50)
• 9. Ability to Match Student-Level P-12 and Higher
  Education Data (26)
• 10. A State Data Audit System (45)

http//www.dataqualitycampaign.org/survey/elements
16

• http//www.dataqualitycampaign.org/survey/compare

http//www.dataqualitycampaign.org/survey/compare
17
5 Teacher Identifier with a Teacher-Student
Match

• Unique teacher ID
• ID stays with teacher across districts
• Unique or SS
• Teacher knows own ID
• No teacher can have more than 1 ID
• No two teachers can have the same ID

18
5 Teacher Identifier with a Teacher-Student
Match

• Procedures to resolve duplicate IDs
• Upon entry to teacher prep program
• When hired as a student teacher
• When first hired as a teacher
• Upon credentialing
• At other times
• When is ID assigned? (New to teaching, returning
  teacher, transfer from another state)

http//www.dataqualitycampaign.org/survey/elements
19
5 Teacher Identifier with a Teacher-Student
Match

• Is teacher mobility tracked across schools?
  Across districts? Across states?
• Does the state collect the reason teachers leave
  the profession?
• At what level is HQT standard determined?
• Does state know if teacher met HQT via HOUSSE?
• Does state know if teacher met HQT via state
  assessment?

http//www.dataqualitycampaign.org/survey/elements
20
5 Teacher Identifier with a Teacher-Student
Match

• Can teacher and student record be matched by
• Elem Course/Subject Data/Elem Assessment Data
• MS Course/Subject Data/MS Assessment Data
• HS Course/Subject Data/HS Assessment Data
• Type of teacher data maintained daily
• Type of program, Year of certification, Other
• Who maintains data
• First year teacher ID collected
• If now is there a plan? Date?

http//www.dataqualitycampaign.org/survey/elements
21
Until recently, most states have used these
student and teacher data systems separately,
but by linking them, much more can be learned
about effective teacher preparation and what
work to improve teaching and learning in
districts, schools and classrooms.

• Bergner, Steiny, and Armstrong (2007)
• Benefits of and Lessons Learned from Linking
  Teacher and Student Data

22
Possible policy uses

• Pipeline
• Productivity
• Employment
• Retention
• Working conditions
• Distribution
• Effectiveness

Barnett Berry (2005) Center for Teaching Quality
23
Early Findings on IDs

• Delaware, Utah, and Washington are moving towards
  assigning ID when candidate enters program
• Seen as providing more accurate data on HQT when
  linked to class rosters
• W Virginia also uses teachers score on PRAXIS to
  determine HQT

24
Potential Evaluation of Teacher Prep

• Which programs train teachers who persist in
  teaching as a career?
• When and how are new teachers exhibiting the
  greatest increases in teaching effectiveness?
• Which training program are especially effective
  at equipping teachers with content skills to
  effectively teach students mathematics, writing
  or reading comprehension?
• Which institutions provide the most effective
  professional development for addressing specific
  weaknesses in in-service teachers?

25
Tennessee

• Furthest along in the use of Value Added Models
• Has kept information confidential between teacher
  and principal
• Provides data back to preparation program
• Allows districts to determine how used in teacher
  assessments
• Chattanooga uses the value-added data to identify
  high performing teachers and recruit them to work
  in high-poverty schools

26
Tenn Bd of Regents Teacher Prep Redesign

• Virtually eliminates traditional university
  classroom seat time for teacher candidates,
  streamlining these experiences into
  participatory, student-directed learning in
  authentic school settings. 
• The model emphasizes constructivist learning,
  intensive induction, and performance-based
  planning and assessment. 
• To accommodate and meet the needs of
  post-baccalaureate candidates, the TBR Teacher
  Education Redesign offers multiple entry points.

http//www.ncate.org/documents/news/NCATE_TN_BD_of
_Regents_May_2009.doc.
27
Utah

• Teachers can look at the scores of their students
  on the Iowa Test of Basic Skills by concept,
  standard, or demographic group.
• Leaves it to districts to do any further analyses
  linking student achievement data to individual
  teachers.
• Identifier is used primarily for licensing and
  NCLB reporting requirements

28
Delaware

• Used its linked teacher and student databases to
  develop a statewide educator evaluation system
• Teachers, specialists and principals can use data
  to set goals for student achievement and includes
  indicators to measure how well the goals are met
• Tracking teacher data across years to identify
  trends
• Used a third party study to make sure the systems
  were built as intended and communicated.

29
Louisiana

• Created a Louisiana Education Accountability
  System (LEADS) with a teacher student data link.
• It is not used for any type of teacher evaluation
• It is used to improve the level of preparedness
  for pre-service teachers.

30
Louisiana Education Accountability System

• Level 1 Effectiveness of Planning (Redesign of
  Teacher Preparation Programs)
• Level 2 Effectiveness of Implementation (NCATE
  PASS-PORT)
• Level 3 Effectiveness of Impact (Teacher
  Preparation Accountability System)
• Level 4 Effectiveness of Growth in Student
  Learning (Value-Added Teacher Preparation Program
  Assessment)

31
Value-Added Teacher PreparationAssessment Model
Process

• Predict student achievement based on
• prior achievement, demographics, and attendance.
• Assess actual student achievement.
• Calculate degree to which students taught by new
  teachers met achievement of similar students
  taught by experienced teachers.
• Act on results.

Noell Burns (2008) Powerpoint presented to the
LA Board of Regents 2007-08 Results Value-Added
Teacher Preparation Assessment Model
32
Effect Estimates for Post-RedesignAlternate
Certification ProgramSocial Studies
SAMPLE REPORT
Noell Burns (2008) Powerpoint presented to the
LA Board of Regents 2007-08 Results Value-Added
Teacher Preparation Assessment Model
33
George Noells Findings

• Varying levels of effectiveness existed within
  teacher preparation programs and across teacher
  preparation programs. 
• Some teacher preparation programs prepared new
  teachers whose teaching effectiveness in specific
  content areas was equivalent to experienced
  teachers. 
• Other programs prepared new teachers whose
  effectiveness was comparable to new teachers or
  below the effectiveness of other new teachers in
  specific content areas. 

34
Other Findings

• ACT scores of new teachers within programs did
  not account for variance in teacher preparation
  program effectiveness.
• ACT mathematical scores of individual new
  teachers across programs was a modest predictor
  of teacher effectiveness in mathematics.
• Teachers who are certified in the content area
  they are teaching are more effective than those
  who are not certified to teach that content.

Noell Burns (2008) Powerpoint presented to the
LA Board of Regents 2007-08 Results Value-Added
Teacher Preparation Assessment Model
35
Reactions from the Press

• the study takes the lead in linking the
  preparation programs for teachers to results in
  tests among their students. That allows two good
  things to happen Programs that need improvement
  can focus on the weaker areas for the teachers of
  the future, and there is an objective seal of
  approval for teacher training programs that
  work.
• The Advocate.com
• January 9, 2009

36
Reactions from the Press

• New teacher makes the grade in Louisianas
  schools
• Novice teachers who were trained by The New
  Teacher Project outperformed their more
  experienced counterparts in getting students to
  improve on test scores, and that's a strong
  argument for continuing to use this approach.
• The Times-Picayune
• January 6, 2009

37
West Virginia

• Chooses not to link teacher accountability to
  student performance

38
The participating states did not report
encountering major political barriers in
implementing their systems.Recommend involving
all stakeholders throughout the process.
Bergner, Steiny, and Armstrong (2007) Benefits of
and Lessons Learned from Linking Teacher and
Student Data
39
Arguments for Using Standardized Student
Achievement Data

• (a) Relying on a common set of statewide learning
  measures enables states to combine evidence from
  diverse communities and regions of the state.
• (b) Often a states measures of learning are
  closely aligned with a K-12 adopted
  standards-based curriculum.
• (c) Use of a states standardized exams can
  enable researchers to take into account each
  students prior level of learning.
• (d) Pupil scores on the states standardized
  tests have relatively strong levels of
  reliability, compensating for the inaccuracies
  that characterize all educational measures.

40
Student versus Class Issue

• In the California University System
• CTQ is using a student-by-student method to
  measure instructional impact, rather than relying
  on evidence of average learning levels by large
  groups of K-12 students.
• If CTQ relied on summaries of learning by all
  students in a district, school, grade or subject,
  the effects of different institutions would be
  co-mingled with each other.
• CTQ (2007) Teacher Preparation Program
  Evaluation Based on
• K-12 Student Learning and Performance
  Assessments by School Principals

41
Promising Practices

• One approach is to assess the gain that each
  student realizes by comparing evidence assembled
  before and after her or his instruction in a
  subject that is tested on multiple occasions.
• CTQ (2007) Teacher Preparation Program Evaluation
  Based on
• K-12 Student Learning and Performance
  Assessments by School Principals

42
Taking this Work to Scale

• Located in distinct regions of the state, in the
  vicinity of twelve CSU campuses, seven school
  districts educate more than one million students,
  employ more than 40,000 teachers, and annually
  hire approximately 3,350 California State
  University graduates as new teachers.
• CTQ (2007) Teacher Preparation Program
  Evaluation Based on
• K-12 Student Learning and Performance Assessments
  by School Principals

43
More than just test scores
44
Six Outcomes used in CSUs

• Intrinsic qualities of each program as reported
  by its graduated when they complete the program.
• Effects of each program on graduates teaching
  (self-reported after 1-3 year)
• Effects of a program on graduates teaching as
  reported by job supervisors.
• Effects on graduates teaching as measured by
  assessments of performance.
• Participation and persistence in the profession
• K-12 student learning outcomes that can be traced
  to teacher education.

CTQ (2007) Teacher Preparation Program Evaluation
Based on K-12 Student Learning and Performance
Assessments by School Principals
45
Question being researched

• Compared with factors associated with students,
  their families and their communities, how much of
  their learning is associated with their teachers
  and the preparation of those teachers in CSU and
  other institutions?
• Requires hierarchical linear modeling

46
Complex Analysis

• Requires use of hierarchical linear modeling to
  estimate
• how much learning was associated with student
  factors when teacher factors were statistically
  held constant
• (b) how much learning was associated with teacher
  factors when student factors were statistically
  held constant and
• (c) how much learning could not be explained by
  this procedure because of the limited numbers of
  student and teacher factors that were measured in
  the evidence sets.

47
CSU Preliminary Findings
CTQ (2007) Teacher Preparation Program Evaluation
Based on K-12 Student Learning and Performance
Assessments by School Principals
48
For three different areas of learning
CTQ (2007) Teacher Preparation Program Evaluation
Based on K-12 Student Learning and Performance
Assessments by School Principals
49
CTQ (2007) Teacher Preparation Program Evaluation
Based on K-12 Student Learning and Performance
Assessments by School Principals
50
Far Reaching Implications

• Does evidence of K-12 student achievement help to
  identify specific programs of professional
  teacher preparation that are particularly
  effective and, if it does, can the effective
  features and characteristics of these programs be
  identified? For university students who want to
  teach, would it be feasible to extend and enlarge
  the most effective programs?

51
Cautions on sole reliance on standardized test
scores

• Average scores for groups of students are
  affected by exclusion and accommodation policies
  (e.g. students with disabilities and English
  learners), retest policies for absentees, the
  timing of testing over the course of the school
  year, and by performance incentives that
  influence test takers effort and motivation.
• Letter to Secretary Arne Duncan
• National Accademy of Sciences (October 5, 2009)

52
Cautions about widespread reliance on VAM

• The considerable majority of experts at the
  workshop cautioned that although VAM approaches
  seem promising, particularly as an additional way
  to evaluate teachers, there is little scientific
  consensus about the many technical issues that
  have been raised about the techniques and their
  use.
• Summary from recent workshop sponsored by
  the Board on Testing and Assessment, National
  Research Council

53
General Issues

• Reliability
• Multiple years of student data to match with a
  teacher in use of Growth model
• Mobility/biasstudents without matches, teachers
  who move
• Need for comparable course data at secondary
  level to compare to teacher group
• Rolf Blank, ASR, CCSSO
• Blog 10/08/09

54
General Issues

• Validity
• Clarity of purpose/need theory of action
• How used by LEAs for teacher evaluation
• Expected growth for studentsnormative vs.
  criterion
• Limitations of single test measure
• Rolf Blank, ASR, CCSSO
• Blog 10/08/09

55
General Issues

• Reporting
• Growth reports for teacher evaluation
• Appropriate comparison group-school, district,
  etc.
• Who receives the report? Personnel, Policy,
  legal
• How to measure characteristics of effective
  teachers
• Student and teacher confidentiality must be
  protected.
• Rolf Blank, ASR, CCSSO
• Blog 10/08/09

56
Specific weaknesses

• Ceiling or floor effects for students whose
  achievements are too high or too low make VAM
  scores problematic.
• VAM scores attributed to a teacher may be
  affected by other factors, e.g. student
  motivation and parental support.
• VAM models cannot be used to evaluate educators
  for untested grades and subjects.

57
Results from New York Calder Report

• The results also suggest that features of
  teacher preparation can make a difference in
  outcomes for studentsTeacher preparation that
  focuses more on the work of the classroom and
  provides opportunities for teachers to study what
  they will be doing produces teachers who are more
  effective during their first year of teaching.
• Teacher Preparation and Student Achievement
  (2008))
• D. Boyd, P. Grossman, H. Lankford, S. Loeb, and
  J. Wyckoff

58
Initial Findings about Effective Programs

• Programs that provide more oversight of student
  teaching experiences or require a capstone
  project supply significantly more effective
  first-year teachers to New York City schools.
• Teachers who have had the opportunity to review
  curriculum used in New York City.

59
Initial Findings about Effective Programs

• Teachers who have had the opportunity in their
  preparation to engage in the actual practices
  involved in teaching (e.g., listening to a child
  read aloud for the purpose of assessment,
  planning a guided reading lesson, or analyzing
  student math work).
• Teacher Preparation and Student Achievement
  (2008))
• D. Boyd, P. Grossman, H. Lankford, S. Loeb, and
  J. Wyckoff

60
Final Word

• We are not sure the extent to which the
  value-added measures of student achievement are
  actually good measures of either the range of
  student learning that we care about or of
  teachers impact on learning.
• Teacher Preparation and Student Achievement
  (2008))
• D. Boyd, P. Grossman, H. Lankford, S. Loeb, and
  J. Wyckoff

61
Is the Jury Still Out on this Work?