Will That Work for Us? Interpreting Research from The Memphis Striving Readers Project (MSRP)

1
Will That Work for Us? Interpreting Research
from The Memphis Striving Readers Project (MSRP)

• Presented by
• Ric Potts, MCS J. Helen Perkins, U of M
  Elizabeth Heeren, MCS Rorie Harris, MCS and
  Jill Feldman, RBS
• 2008 International Reading Association Research
  Conference
• Atlanta, GA

2
Session Overview

• Introduction to the Striving Readers grant
• Overview of Memphis SR research design
• Year One Impact Analyses
• Collection of implementation fidelity data
• implications for practitioners and researchers
• Planned (Ongoing) Analyses
• Q A /Group Discussion

3
Introduction Memphis Striving Readers Project
(MSRP)

• Ric Potts, PI MSRP
• Memphis City Public Schools

4
Memphis-The City
The City of Memphis has a population of
642,251. 63.1 African American 31.3
Caucasian 4.1 Hispanic
5
And one Elvis
6

• Approximately 70 percent of adolescents
• struggle to read. The young people enrolled in
• middle and high school who lack the broad
• literacy skills to comprehend and learn advanced
• academic subjects will suffer serious social,
• emotional, and economic consequences.
• Reading at Risk The State Response to the Crisis
  in Adolescent Literacy, Oct. 2005

7
Urban Child InstituteThe State of Children in
Memphis and Shelby County2006

• Under-educated children have no future.

8
Urban Child InstituteThe State of Children in
Memphis and Shelby County2006

• by U.S. standards roughly 75 percent of students
  in Tennessee fail to meet national grade
  appropriate standards, and Memphis is at the
  bottom in Tennessee. . . . Memphis is one of
  theleast-educated cities in America.

9
Motivation behindMemphis Striving Readers Project

• Memphis is among the poorest and least-educated
  cities in the US
• 30.1 of all children live in poverty
• 24.3 of adults have less than a HS education
• 36.7 have HS diploma or equivalent
• 30.5 have Assoc. or some college
• 8.5 have at least a BA
• MCS is 21st largest K12 district in US gt116,000
  students
• Over 95 of MCS 196 schools are Title I schools
• 71 of MCS students qualify for free/reduced
  price lunch
• MCS students are 87 AA 9 White 4 other
• In 85 of MCS schools, 33 of students change
  schools during year
• In 2003-04, the system-wide graduation rate was
  61 percent
• 71 of students in grades 6-8 scored below the
  50th percentile on TCAP (Reading/Language Arts)

10
Striving Readers A Federal Response

• In 2005, the Department of Education called for
  proposals for the Striving Readers grant.
• In March, 2006, Memphis was one of eight sites
  awarded the grant.

11
Memphis Striving Reader Program Targeted Schools
School Grade Span Total Enrollment Total Of Non-Special Education Students Scoring In Bottom Quartile In Reading
School 2 6-8 1,021 414
School 1 6-8 1,033 384
School 6 6-8 700 251
School 5 6-8 765 245
School 8 6-8 547 178
School 4 6-8 486 196
School 3 6-8 976 357
School 7 6-8 877 274
12
The Whole School Intervention Memphis Content
Literacy Academy (MCLA)

• Overview presented by
• J. Helen Perkins, SR Co-PI
• University of Memphis

13
A Change Model
14
A Capacity-Building Model for Teacher
Development (Cooter Cooter, 2003)
Expertise Ability to Coach Others
Refined and Expanded Capacity
Practice with Coaching
Deeper Learning with Limited Capacity
First Exposure
No Knowledge
Emphasis Deep Training (180 hours over two
years)
15
Memphis Content Literacy Academy Infusing
Simultaneously Across Core Subject Areas
Scientifically-based Reading Research (SBRR)
Strategies in
Vocabulary Reading Comprehension Reading Fluency
16
Benefits to Teacher Laureates

• Advanced Training (180 hours) on
  scientifically-based reading instruction (SBRR)
  for urban children
• A Master Teacher Coach to Assist (30 hours)
  with Implementing New Strategies (in their own
  classrooms!)
• Twelve (12) Graduate Semester Hours of Credit
  from University of Memphis (FREE) (applicable to
  an advanced degree)
• Can Seek Highly Qualified Endorsement in
  Reading
• Books and Materials (FREE)
• Success in Helping Children Achieve AYP
• Principal Support

17
MCLA Year 1 Selected Strategies

• Fluency
• Choral Reading
• Paired reading
• Guided, repeated,
• oral reading (pairs)

18

• Comprehension
• Question Generation
• Three- Level Retelling
• Oral
• Graphic Organizor
• Written

• Comprehension monitoring
• Expository Text Patterns
• Multiple Strategies

19

• Vocabulary Development
• Pre-instruction of vocabulary
• Repeated, multiple exposures
• Semantic Maps

20
Classroom Organizational Tools Strategies Year
1

• CREDE Standards
• Whole class v. collaborative small group
• Reading Next Elements

• Use of leveled materials
• (e.g., National Geographic)

21
CREDE Formatting of Professional Development
Training
http//crede.berkeley.edu/standards/standards.html
22
Classroom Action Plans (CAPs)Spring
2008Science, Social Studies, ELAYour task is
to develop a series of class lessons where you
teach academic vocabulary in a unit of your
choice.You must have at least one vocabulary
learning strategy/activity that occurs1. BEFORE
students read the assigned text,2. DURING the
reading assignment, and3. AFTER the reading
assignment
23
MCLA Classroom Model

• Gradual release of responsibility
• (teacher modeling, guided practice, independent
  practice, independent use)
• Integration of 12 literacy strategies
• (vocabulary, fluency comprehension)
• Development of Classroom Action Plans (CAPs)
• (content area lesson plans for strategy
  implementation including procedures for
• student assessment)
• On-site support provided by coaches
• Use of Curriculum Resource Center (CRC) materials

24
The Principals Fellowship

• Literacy Leadership Practices
• Real World Problem Solving
• Create Literacy Materials Centers
• Early Identification w/ Intense/Focused
• Remediation
• Research-Informed Decision Making
• Involve Families
• Needs-Based Scheduling
• Matching the Most Successful Teachers
• with Critical Condition Kids

25
READ 180, Our Targeted Intervention
Overview provided by Elizabeth Heeren, SR Grant
Coordinator Memphis City Schools
26
(No Transcript)
27
Program Components
Support materials for differentiated instruction
in small group rotation
Tools for student placement and assessment
Student workbooks for Independent Practice in
small and whole group rotations
28
Key Elements of READ 180

• Fidelity of Implementation
• 90 minute classes
• Certified teachers (LA or Reading)
• District Instructional Support
• District Technological Support
• Scholastic training (site-based and on-line)

29
R180 Correlations to Reading Next Recommendations
for Adolescent Literacy

• Direct, explicit comprehension instruction
• Motivation and self-directed learning
• Strategic tutoring
• Differentiated texts (levels and topics)
• Technology component
• Ongoing formative assessment
• Extended time for literacy
• Professional development (long-term and on-going)

30
Memphis Implementation

• We have 8 schools in the Striving Readers Grant,
  with up to 120 randomly selected R180 students at
  each school.
• Students receive R180 instruction for 2 years.
• Each student placed in R180 falls in the lowest
  quartile of TCAP (Reading score).
• Each student in R180 is paired with a similar
  student from the lowest quartile who does not
  receive the treatment (for impact comparison).

31
MSRP Research Design

• Overview presented by
• Jill Feldman, SR Research Director
• Research for Better Schools

32
Overall MSRP Goals

• To determine
• The effects of MCLA on core subject teachers
  knowledge and use of SBRR
• 2. The separate and combined effects of MCLA and
  Read 180 on students reading achievement levels,
  especially students who are identified as
  struggling readers
• 3. The separate and combined effects of MCLA and
  Read 180 on students achievement in core
  subjects, especially students who are identified
  as struggling readers

33
MCLA Program Logic Model
Outputs
Long-term Outcomes
Shortterm Outcomes
Funding, staff, curriculum resource center,
facilities, incentives, research materials
Principals 45 hours of Principal Fellowship
participation 100 of principals incorporate
plan for using MCLA strategies in SIP 100
attendance of key MCLA events 80 of principals
report actively supporting the program 100 of
MCLA schools have allocated space for the
CRC Teachers 90 of hours of MCLA training/yr
for 2 years (180 hours) Engage in weekly
coaching sessions or as needed to meet teachers
differentiated needs 8 CAP cycles completed
each year for two years 100 of teachers
complete performance measures identifying
supplemental resources available/those necessary
to support content area instruction Students 50
of students attend 4 classes taught daily by
teachers participating in MCLA Students learn to
use 7 of 8 MCLA CAP strategies
Principals Awareness of and interest in staff
implementation of MCLA concepts and strategies
Increased advocacy for school-wide use of MCLA
strategies Teachers Increased knowledge about
MCLA strategies Improved preparedness to use
research-based literacy strategies to teach core
academic content Increased use of direct,
explicit instruction to teach research-based
comprehension, fluency, and vocabulary strategies
in content area classes Integrated use of
multiple MCLA strategies to support ongoing
development of content-related instructional
units Students Increased familiarity with and
use of MCLA strategies when engaging with
text Increased internalization of literacy
strategies Increased confidence engaging with
content related texts Increased interest in
school/learning

• Principals
• Improved school climate
• School-wide plans include focus on content
  literacy
• Improved instructional leadership
• Teachers
• Increased effectiveness supporting students
  content literacy development
• Continued collaboration among community of
  teachers to develop and implement CAPs
• Students
• Improved reading achievement and content
  literacy
• 10 increase in students scoring proficient in
  Reading/LA and other subject areas of TCAP
• mean increase of five NCEs on ITBS

Principals Attend 45-hour sessions/yr (2
yrs) Participate in motivational, recruitment
and celebratory events Discuss MCLA at faculty
meetings Conduct walkthrough observations Provid
e opptys for teacher collab Allocate space for
CRC materials Teachers Attend 30 weekly 3-hour
MCLA training sessions/yr (2 years) Develop and
implement 8 CAPs per year in collab content-area
groups Meet with coaches for feedback to improve
impl of MCLA strategies Learn to use of leveled
texts to support SR content literacy
needs Students Learn to use MCLA strategies to
read/react to content related text (
Higher Quality Teaching

Higher Student Achievement
34
Study Design and Analytic Approach MCLA

• Study Design MCLA
• Evaluate teacher and student outcomes
• experimental design
• randomly assigning schools
• (to treatment and control conditions)
• Teacher outcomes include
• preparedness
• frequency of literacy strategy use

• Analytic Approach MCLA
• Two-level HLM
• spring ITBS and TCAP scores as a function of
  teacher and school variables

35
Analytic Decisions

• Missing Data
• students missing pretest score(s) deleted from
  impact analysis on relevant measure(s)
• teachers missing pretest score deleted from
  impact analysis on measure
• Covariates
• include all student- and school-level covariates
  in the model
• run the model
• eliminate the school covariate with the lowest
  significance level
• (highest p-value) not less than 0.2
• repeat steps 2 and 3 until the remaining
  covariates had p-values less than 0.2
• repeat steps 2-4 for the student covariates

36
MCLA Random Assignment of Schools
37
Demographic Characteristics of Year 1 MCLA
Student Sample
38
Baseline Comparisons of Students in MCLA
Treatment and Control Schools
39
Selected Characteristics of the Year 1 Teacher
Sample for MCLA Impact Analyses
40
All Variables Included in MCLA Impact
Analytical Models for Year 1
41
READ 180 Logic Model
42
R180 Study Design Analytic Approach

• Study Design
• Evaluate student outcomes using RCT based on
  random assignment of students to conditions
  across schools
• Student outcome measures
• reading achievement (ITBS)
• core content areas (TCAP)

• Analytic Approach
• Cross-sectional ITT analyses of reading and core
  content area achievement
• Two-level models using spring ITBS and TCAP
  scores as a function of student and school
  variables

43
READ 180 Enrolled Students
44
Variables Included in READ 180 Impact Analytic
Models (Year One)Dependent and Independent
45
Variables Included in READ 180 Impact Analytic
Models (Year One) Covariates
46
Year One Impact
47
Comparison of Teachers in MCLA Treatment and
Control Schools on Year-End Indices for
Preparedness and Frequency of Use
48
MCLA Impacts on Students (Year One)
49
READ 180 Impacts on Students (Year 1)
50
Collection of Data about Implementation Fidelity
51
Implications for Researchers and Practitioners

• What are our purposes for collecting
  implementation data?
• To provide other districts with information about
  outcomes they might expect when implementing
  similar interventions with their struggling
  readers
• To set the context for understanding student
  outcomes
• Requires MCS to place the needs of the field
  above local concerns

52
Reasons to Collect Double Data

• R180 evaluation is intended to test effects of a
• replicable intervention in the real-world
• Without the support of external evaluators
• In ways that emulate what districts will need to
  do to
• monitor implementation
• obtain process feedback

53
Reasons to Collect Double Data

• Collecting data about MCLA and R180 fidelity
• helps researchers explain patterns of impact
  findings
• can be useful in identifying predictors of
  outcomes

54
What Is the Role of the Researcher?

• RBS collects data about
• Impact (MCLA R180)
• Implementation fidelity
• To better understand impact or lack thereof
• (MCLA R180)
• To support development of MCLA (only)
• Counterfactual
• To compare effects to what would have happened in
  SR schools in the absence of MSRP

55
What is the Role of MCS?

• Implement R180 MCLA
• Monitor the implementation process
• Ensure implementation is on model
• Refine service delivery based on formative data

56
Defining Implementation Fidelity MCLA

• Innovation Configuration Mapping

57
MCLA Implementation Framework

• Developing an Innovation Configuration (IC) Map
• (Hall Hord, 2006)
• Operationally defines levels of implementation
  fidelity among clusters of key active
  ingredients
• Iterative process involving key stakeholders
• Development team (University of Memphis)
• Grantee (Memphis City Public Schools)
• Researchers (Research for Better Schools)

58
MCLA Roles Responsibilities

• MCS Administrators
• Participate in Principals Fellowship
• Support recruitment and retention efforts
• Link MCLA w/School Improvement Plan
• Observe MCLA teachers
• (once/marking period)
• Allocate space for CRC materials
• Protect/respect role of coach
• Developer
• Design MCLA curricula
• (for teachers principals)
• Facilitate writing team activities
• Meet weekly with instructors ( coaches)
• Disseminate research about adolescent SR

59
MCLA Training

• Provided by the Developer
• 3-hour weekly principal meetings
• (fallYear 1)
• 3-hour weekly teacher training sessions per
  content area
• (180 hours over 2 years)
• PD for coaches in
• Mentorship Urban education Adolescent lit
• Provided by MCS (coaches)
• On-site observation of CAPs
• Model/co-teach strategies
• Feedback
• Supplemental resources
• has included coaches since spring 2007

60
MCLA Innovation Configuration Map Framework
61
Instrument Development

• With the IC map guiding development, the
  following
• measures were designed to collect data about MCLA
  
• implementation
• Surveys
• Teacher knowledge about preparedness to use
  MCLA strategies
• Teacher demographic characteristics
• Teachers MCLA Feedback
• Interviews
• Principals, coaches, development team, and MCS
  administrators
• Teacher Focus Group Discussions

62
Operationally defining componentsJob
Definition
63
Aligning the IC Map and Instrument Development
Job Definition Teacher Survey
64
Job Definition - Principal Interviews
65
MCLA Innovation Configuration Map Framework
66
(No Transcript)
67
Where the rubber hits the runway

• MCLA Classroom Implementation

68
Operationally defining components
Implementation of Lesson Plans
69
Implementation of lesson plansCollecting
classroom observation data
70
Implementation of lesson plansCollecting
classroom observation data
71
MCLA Implementation Barriers

• Barriers
• Limited development/planning time
• Need for coaches with disciplinary content
  knowledge
• Challenges in establishing a critical mass of
  enrolled teachers at each school
• CRC materials not received until spring 2007
• Pressure to focus on TCAP test preparation
  (spring)
• Difficulty maintaining principal attendance at
  weekly meetings

72
MCLA Planned Implementation Changes

• Changes
• Adoption of CREDE (UC-Berkeley) JPA instructional
  model
• Reduction in the number of CAPs required of
  teachers
• Shortened class schedule/more intensive work with
  coaches
• Inclusion of special education teachers among
  those eligible to enroll
• Restructured Principal Fellowship
• (includes other school leaders meets monthly)

73
Defining Implementation Fidelity R180

• Rorie Harris
• Memphis City Public Schools

74
Findings Related to Implementation

• Scheduling
• Scheduling 90 minute blocks in schools using the
  Middle School concept is difficult. Teams of core
  content teachers traditionally have 55 minute
  classes.
• Interruptions to the 90 minute block occur.
• Special Education Students
• READ 180 will only suffice as a SPED students
  intervention if the teacher is SPED-certified.

75
Findings Related to Implementation

• Use of Technology
• Technology issues can negatively affect
  instructional time.
• Parents Students
• Some parents do not want their children in
  Reading Intervention classes. They feel like this
  is a label.
• Classroom management issues impact instruction.
• Student mobility affects the scope and sequence
  of reading instruction.

76
Findings Related to Implementation

• School Administration
• Without administrator buy-in to the importance
  of smaller classes and protection of the 90
  minute block, fidelity is not supported.
• Read 180 Teachers
• It is challenging to encourage ALL teachers to
  engage in on-line professional development and/or
  to attend network meetings.
• Teacher turn-over brings out the need for
  repeated initial training and reduces the
  development of teacher leaders.

77
Indicators of Read 180 Implementation

• Scholastic identifies several key program aspects
• Teacher Training/Professional Development
• Computer Hardware/Software Use
• Use of Read 180 Materials
• Group Rotation
• Class Size
• Classroom Environment
• Student Engagement

78
Sources of Implementation Data

• Classroom observations during the school year
  (Fall Spring)
• Read 180 program databases (SAM)
• Enrollment and course-related data from district
  databases
• Surveys administered to students (Fall Spring)
  and teachers (Spring)
• Information collected during professional
  development programs

79
MCS Data Linked to Implementation Indicators
MCS Data Source Key Program Area
Completion of Scholastic RED Course Teacher Training
Attendance at district-wide Read 180 Network Meetings Teacher Training
Fall Spring Classroom Observations Computer Hardware Software Use Group Rotations Class Size Classroom Environment Use of Read 180 Materials
Enrollment Data Class Size
80
MCS Data Linked to Implementation Indicators
MCS Data Source Key Program Area
Student Usage Data from SAM Computer Hardware Software Usage
Student Surveys Classroom Environment Student Engagement Use of Read 180 Materials
Teacher Survey Computer Hardware/Software Use Classroom Environment Group Rotations Use of Read 180 Materials
81
Overview of Year One Conclusions

• Jill Feldman, RBS

82
(Brief) Conclusions Discussion

• READ 180 No significant Year One student impact
• Late startup
• (Most) students will receive two years of
  intervention
• Planned Future Analyses
• Three-level analyses planned to examine whether
  teacher characteristics exert a moderating effect
  on student outcomes
• Exploratory analyses of relationships between
  amount of
• READ 180 instruction and effects on student
  outcomes

83
(Brief) Conclusions Discussion

• MCLA
• Significant (moderate) impact on teachers
  frequency and preparedness to use MCLA
  strategies
• No significant impact on students achievement in
  reading or core content areas
• Discuss
• Subjectivity of measure (Hawthorne Effect)
• Teacher findings support program logic model
• Explore relationship between impact and
  participation in PD

84
Next Steps
85
Planned Exploratory Analyses

• Re-run HLM impact analyses to test effects of
  teacher variables on outcomes
• Preparedness and use of MCLA strategies
• Age
• Experience as teacher ( years at MCS)
• PD in year prior to MCLA

86
Planned/ongoing analyses

• Individual students growth over time
• Rerun HLM with student-level variables
• MCLA teachers
• Students school attendance
• ITS analyses
• Using TCAP Spring 2003 2004 scores
• Correlating R180 data with TCAP ITBS
• for possible use as covariates in HLM

87
Now Its Your Turn

• Ask the panel
• Share your experiences
• Triumphs
• Tribulations

88
Thank you for joining us!

• For additional information contact
• feldman_at_rbs.org