WASH DC BRIEFING ON JP AUG 2000

1
What Was Learned from a Second Year of
Implementation
William Corrin, Senior Research Associate MDRC
IES Research Conference Washington, DC June 8,
2009
2
Presentation Overview

• Project Background
• Do teachers need more than a year to master new
  practices?
• Year 2 Implementation Findings (and Year 1
  Comparison)
• Impact findings
• Exploratory findings
• Two-Year Teachers and Replacement Teachers
• Two-Year Teachers Year 1 vs. Year 2
• Summary

3
Project Background
4
The ERO Study

• The ERO study is an impact evaluation of two
  supplemental literacy programs targeted at 9th
  grade students with limited reading skills.
• Key research question
• What are the impacts of the two interventions,
  together and separately, on ninth-grade students
  reading achievement and reading behaviors?

5
The Interventions

• Two literacy programs targeted at 9th grade
  students with limited literacy skills (2-5 years
  below grade level)
• Reading Apprenticeship Academic Literacy (RAAL)
• Xtreme Reading (XR)
• The programs are supplemental reading classes
• Designed as full-year courses that replace a 9th
  grade elective class (rather than a core academic
  class)
• The two programs share common goals and
  instructional principles

6
Delivery of the Interventions

• Each high school implemented one of the two
  programs for ninth graders for two school years
• 2005-06 (Cohort 1) and 2006-07 (Cohort 2)
• One ERO teacher per high school
• Experienced full-time ELA or social studies
  teacher was trained to teach the ERO reading
  course
• Responsible for 4 sections of 10-15 students each
  
• Training and technical assistance were provided
  to teachers by the developers
• Summer training institutes, school-year off-site
  booster training sessions and on-site coaching

7
Implementation Data Collection

• To answer questions about how well the programs
  were put into place
• Allows us to answer questions about how
  implementation compared across program years Do
  teachers do better with more experience teaching
  these programs? That is, is implementation
  stronger in the second year compared to the
  first?
• Allows us to investigate associations between
  implementation and impacts

8
Measuring Implementation Fidelity

• Two site visits in Year 2 (one per semester only
  one site visit in Year 1)
• Implementation fidelity was assessed on two
  dimensions
• Classroom learning environment
• Comprehension instruction
• Classroom observation ratings were obtained for 6
  constructs common to both programs, and 7
  program-specific constructs
• Ratings are based on a 3-point scale

9
Measuring Implementation Fidelity (cont.)

• Composite ratings for each of the two dimensions
  were calculated, and then used to classify sites
  as
• well aligned (average rating gt 2)
• moderately aligned (average rating 1.5 1.9)
• poorly aligned (average rating lt 1.5)

10
Implementation Findings
11
Implementation Findings

• ERO teachers in Year 2
• Of the 34 ERO teachers in Year 2
• 25 had taught the entire first year of the study
  (13 RAAL, 12 XR)
• 2 had taught part of the first year (both XR)
• 7 were new to the programs (4 RAAL, 3 XR)
• All Year 2 teachers taught the entire year

12
Implementation Findings (cont.)

• Implementation fidelity in Year 2
• In terms of classroom learning environment, 1
  school was rated as poorly aligned with program
  models (vs. 4 schools in Year 1)
• In terms of reading comprehension instruction, 1
  school was rated as poorly aligned with program
  models (vs. 9 schools in Year 1)
• 23 schools were rated as well aligned on both
  dimensions (vs. 16 schools in Year 1)
• Similar pattern of implementation findings at
  RAAL and XR schools

13
Implementation Findings (cont.)

• Implementation fidelity rated higher in Year 2
• In terms of classroom learning environment, the
  average rating at the spring site visit was 2.5
  in Year 2 compared to 2.2 in Year 1
• In terms of reading comprehension instruction,
  the average rating at the spring site visit was
  2.3 in Year 2 compared to 1.9

14
Learning EnvironmentBy Site Visit
15
Comprehension Instructionby Site Visit
16
Impact Findings
17
Random Assignment and Analysis

• Random assignment
• School-Level
• 34 schools randomly assigned within 10 districts
  to RAAL or Xtreme Reading (17 RAAL sites and 17
  Xtreme Reading sites)
• Student-Level
• In each high school, eligible students were
  randomly assigned to either
• Enroll in an ERO class (ERO group) or
• Take one of the regular elective classes (non-ERO
  group)
• Analysis Impacts are estimated by comparing the
  outcomes of students in the ERO and non-ERO group

18
Impact Findings Reading Achievement

• Reading achievement was measured using the GRADE,
  a standardized reading assessment.
• Overall positive impact on reading comprehension
• 0.09 SD for Cohort 1
• 0.08 SD for Cohort 2
• No impact on vocabulary
• Same result for both cohorts

19
Impact Findings Reading Behaviors

• Cohort 1
• Estimated impacts on three reading behaviors were
  not statistically significant (frequency of
  school-related reading, freq. of
  non-school-related reading, use of reading
  strategies)
• Cohort 2
• Estimated impact on students use of the reading
  strategies taught by the programs is 0.09 SD and
  statistically significant

20
Exploratory Analyses
21
Implementation Fidelity by Teacher Experience
and Year

• Implementation ratings by year and by teacher
  experience

22
Implementation Fidelity by Teacher Experience
and Year

• Overall implementation fidelity
• Of the 25 sites with two-year teachers, 19 had
  higher overall implementation ratings in the
  second year compared to the first year
• Of the 9 sites with replacement teachers, 8 sites
  had higher overall implementation ratings with
  the replacement teacher compared to the original
  teacher

23
Composite Fidelity Scores by Site Visit
24
Variation in Impacts

• Impacts on Student Outcomes by Teacher Experience
  Teaching ERO Year 2

25
Variation in Impacts (cont.)

• Impacts on Student Outcomes by Implementation
  Year Schools w/ Two-Year Teachers (N25)

26
Summary

• Supplemental literacy courses for 9th grade
  students can have a positive impact on students
  reading comprehension.
• These programs can be implemented with reasonable
  fidelity in a diverse array of high schools, and
  implementation fidelity can improve over time.
• On average, implementation fidelity was stronger
  in the second year.
• There is suggestive, but not convincing, evidence
  that teachers gained greater mastery of the
  programs having a second year to teach it.
• However, impacts on students reading
  comprehension remained the same. (Impacts on
  students use of reading strategies were greater.)

27
Further questions?

• William Corrin
• william.corrin_at_mdrc.org
• [phone]
• Marie-Andree Somers
• marie-andree.somers_at_mdrc.org
• [phone]
• Paul Strasberg
• paul.strasberg_at_ed.gov
• [phone]