Extending the Culture of Evidence: Research

1
Extending the Culture of Evidence Research
Teacher Work Sample Conference Portland, Oregon
July 20, 2006

• Raymond Pecheone Ruth R. Chung
• Stanford University

2
The Research Base for Teacher Licensing Tests

• Weak relationship between traditional licensing
  tests and teacher effectiveness (NRC, 2001)
• Strauss Sawyer (1986)
• Ferguson (1991, 1998)
• Ferguson Ladd (1996)
• Clotfelter, Ladd, Vigdor (forthcoming)
• Goldhaber (2005, 2006)
• Effect sizes quite small in recent value-added
  research (.01-.06)

3
Portfolio Assessments as Evidence in Teacher
Certification Teacher Education

• PACT Performance Assessment for California
  Teachers
• Connecticut BEST program
• National Board for Professional Teaching
  Standards

4
Portfolio Assessments as Evidence - Rationale

• Authenticity - Draw on artifacts of teaching,
  including student work
• Face validity - Stronger theoretical link to
  teaching effectiveness (Cochran-Smith, 2003
  Danielson McGreal, 2000 Darling-Hammond, 2001)
• Reliability and validity meet EEOC and APA/AERA
  guidelines

5
The PACT Assessment System/ Electronic Portfolio
Elements
The Capstone Teaching Event
Assessments Embedded in Local Programs (examples)

• Teaching Event
• Demonstrates
• Planning
• Instruction
• Assessing
• Reflecting
• Academic Language

Child Case Studies
Analyses of Student Learning
Curriculum /Teaching Analyses
Observation/Supervisory Evaluation Feedback
6
The PACT Teaching Event
7
PACT Rubrics

• PLANNING
• Establishing a Balanced Instructional Focus
• Making Content Accessible
• Designing Assessments
• INSTRUCTION
• Engaging Students in Learning
• Monitoring Student Learning During Instruction

• ASSESSMENT
• Analyzing Student Work From an Assessment
• Using Assessment to Inform Teaching
• REFLECTION
• Monitoring Student Progress
• Reflecting on Teaching
• ACADEMIC LANGUAGE
• Understanding Language Demands
• Supporting Academic Language Development

8
PACT Rubrics (one example)
9
PACT ScoresInter-rater Reliability (2004)
Spearman-Brown Reliability Estimate 0.88
10
PACT Validity Studies

• Construct validity
• Factor Analysis
• 2002-03 Pilot Year 1) Reflection Assessment
• 2) Instruction 3) Planning
• 2003-04 Pilot Year 1) Planning Instruction
• 2) Assessment Reflection
• Content validity
• Development teams, Program directors, Program
  faculty, Leadership team
• TPE alignment study
• Survey of piloting candidates
• Concurrent validity
• Evaluation of score validity
• Holistic recommendation for credential
• Bias and fairness review
• Bias and fairness score analyses

11
PACT Scores - Assessment of Student Learning
(2003-2005)
12
PACT Scores by Institution (2004)
13
PACT Candidate Survey (2004) Preparation and
Learning
DV I learned important skills from the process
of constructing the Teaching Event
14
PACT Candidate Survey (2004)Learning and
Performance
DV Total Mean Item Score on Teaching Event
15
PACT Teaching Event as a Tool for Research on
Teaching

• Measurement instrument - Use of Teaching Event
  rubrics in research studies for scoring teaching
  performance
• Teacher practice data - Use of PACT-like
  portfolios as data for research on teaching

16
10 Years of Connecticut BEST / INTASC
1996-2006

• Two-year induction program of support and
  portfolio assessment (third year available, if
  needed)
• Portfolio assessments in elementary, English
  language arts, mathematics, music, physical
  education, science, social studies, special
  education, visual arts, world languages and
  bilingual education
• All portfolios, regardless of the content area,
  require the following
• Daily lesson plans for a 5-8 hour unit of
  instruction with one class
• 2-3 videotapes segments of teaching totaling 40
  min.
• Work samples of two students and
• Reflective commentaries on teaching and learning

17
CT BEST Portfolio AssessmentReader Process
Studies

• Purposes - To understand differences in
  interpretation that are not accessible with
  respect to conventional studies of reader
  reliability
• Data Sources - Transcription of reader dialogues
  and interviews
• Findings
• Pieces of evidence only have coherent meaning as
  part of a larger pattern of evidence.
• Same evidence - different interpretations
• Small differences can cascade into significant
  differences
• Moss, Shutz, Collins (1998)
• Schutz Moss (2004)

18
CT BEST Portfolio AssessmentExpert Reader Studies

• Research Question - How would expert readers who
  work with beginning teachers arrive at licensure
  decisions based on portfolio evidence?
• Data Sources - Portfolio benchmarks, interviews,
  transcriptions
• Findings
• Context matters-where the teacher worked and type
  of support they received.
• Potential considered and valued
• Common evidence - different interpretations
• Union of insufficiencies
• Moss, Schutz and Pecheone (in process)

19
CT BEST Portfolio AssessmentStudies of Portfolio
Generalizability

• Research Question - How likely is it that
  licensure decisions would be reversed or
  substantially altered if a second independent
  assessment of the same kind were made?
• Data Sources - Scored portfolios, case studies
  and interviews
• Findings
• Portfolio and case-based portraits of teacher
  performance were consistent and coherent.
• Some significant differences were found across
  occasions
• Context matters within and across subject matter
• Perceptions of student ability level and student
  needs matter.
• Moss, Sutherland, Pecheone et al. (2004)

20
CT BEST Portfolio AssessmentPolicy Implications

• Standards without standardization
• Multiple measures
• Expert review (Moderation processes)
• Differentiated consequences and support

21
National Board for Professional Teaching
Standards Research

• Teachers Professional Learning Practice
• Athanases (1994)
• Tracz, Sienty, Mata (1994) Tracz et al. (1995)
  
• Sato (2000)
• Lustick Sykes (2005)
• Bond, et al (2000)
• Darling-Hammond, Atkin, Sato, Chung
  (forthcoming)
• NBCTs report improvements in classroom practices,
  including assessment of student learning,
  planning, and instruction. NBCTs show evidence
  of greater teaching expertise in observational
  studies.

22
National Board for Professional Teaching
Standards Research

• Teacher Leadership
• Yankelovich Partners (2001)
• NBPTS (2001)
• Waller Klotz (2001)
• Williams Bearer (2001)
• NBCTs report taking on range of leadership roles
  related to supporting PD of other teachers and
  greater involvement in the district.
  Administrators report NBCTs have active
  leadership roles.

23
National Board for Professional Teaching
Standards Research

• NBPTS Certification Student Achievement
• Cavalluzzo (2004)
• Goldhaber (2004)
• Vandevoort, Beardsley, Berliner (2004)
• Smith (2005)
• Sanders (2005)
• Majority of studies find that students of NBCTs
  have higher student achievement gains than
  students of similar teachers who were not NBCTs.
  The magnitude of effect size varies, but is
  somewhat small in recent value-added studies.

24
Portfolio Assessments are Extending the Culture
of Evidence in

• Teacher Education
• Preservice teachers look for evidence of learning
  in student work
• Portfolio assessments can identify gaps in the
  preparation of new teachers
• Portfolio assessments can evaluate the
  effectiveness of credential programs and promote
  program improvements
• Portfolio assessments can inform the
  accreditation process

25
Portfolio Assessments are Extending the Culture
of Evidence in

• Teacher Licensing
• More validly identify effective teachers
• Induction Professional Development
• Identify areas for improvement and promote
  teacher learning
• Promote teachers use of student work to evaluate
  their teaching

26
Next Steps in Research -Portfolio Assessments as
Evidence

• Study of Beginning Teacher Effectiveness
• Examining correlates of PACT Teaching Event
  scores
• Individual - other tests, grades, supervisory
  scores
• Program features related to performance in
  different areas
• Following candidates into first year of teaching
• Practices
• Student learning outcomes
• Examining the contributions of PACT ESAs to
  judgments about teacher competence