Evaluation of South Carolina

1
Evaluation of South Carolinas Reading First
(SCRF) Initiative

• Achievement and Survey Results from the First
  Three Years
• Presented at the American Educational Research
  Associations Annual Meeting
• March 27, 2008

2
Diane M. Monrad, PhD Sarah J. Gareau,
MEd Jennifer Gay Diana Mindrila Anita Rawls

• Tammiee S. Dickenson, PhD
• Robert L. Johnson, PhD
• Heather L. Bennett, MSW
• Katie A. Sesso-Dahlke, MSW
• Brett Ermer
• Mugdha Galande
• Joanna Gilmore
• Jessalyn Smith

Pam Wills Patricia Branham Becca Driggers Diane
Dunham, EdS
3
Overview

• SCRF evaluation
• Data sources
• Achievement data
• Participant group survey results
• Conclusions

4
SCRF Evaluation
5
Participants

• 2004-2005 served 52 schools in 24 districts
  (approximately 12,000 students)
• 2005-2006 served 51 schools in 23 districts
  (approximately 11,000 students)
• 2006-2007 served 49 schools in 23 districts
  (approximately 10,500 students)

6
Data Sources

• Stanford Reading First assessment
• Palmetto Achievement Challenge Test (PACT)
• Participant group surveys
• School Leadership Team (SLT) surveys
• Professional development workshop surveys

7
Annual SLT Presentation

• Results presented from previous year
• Stanford Reading First assessment results
• Cohort analysis of all students in previous year
• Matched analysis of students in all years
• Individual school reports provided
• Participant group survey results
• Formative feedback for program improvement

8
Stanford Reading First Assessment

• Measures achievement in grades 1 through 3
• Sections
• Multiple choice (components phonemic awareness
  phonics vocabulary development reading fluency
  and reading comprehension strategies)
• Oral fluency (components speaking vocabulary
  and oral reading fluency)

9
Results At Grade Level (AGL) Grade 1
Percentage of SCRF Grade 1 Students Scoring At
Grade Level (AGL) on Stanford Reading First and
Annual Gains
Fall Spring Annual Gain
Year 1 17.0 48.2 31.2
Year 2 18.8 51.2 32.4
Year 3 21.1 54.3 33.2
10
Results At Grade Level (AGL) Grade 2
Percentage of SCRF Grade 2 Students Scoring At
Grade Level (AGL) on Stanford Reading First and
Annual Gains
Fall Spring Annual Gain
Year 1 30.8 35.5 4.7
Year 2 35.7 40.4 4.7
Year 3 38.0 44.6 6.6
11
Results At Grade Level (AGL) Grade 3
Percentage of SCRF Grade 3 Students Scoring At
Grade Level (AGL) on Stanford Reading First and
Annual Gains
Fall Spring Annual Gain
Year 1 21.0 41.5 20.5
Year 2 25.2 46.2 21.0
Year 3 27.6 49.0 21.4
12
Average NCE Scores on Stanford Reading First for
SCRF Students with AGL Reference Line
13
Average Scale Scores on Stanford Reading First
for SCRF Students and the Norm Group
14
Participant Group Surveys

• Respondent groups interventionists, literacy
  coaches, principals, and teachers
• Sections implementation, level of support, roles
  and responsibilities, professional development,
  and program effectiveness
• Comparison across years (2005-2006 compared to
  2006-2007)

15
Results Participant Group Surveys

• Better understanding of program goals
• Reported a high rate of involvement in
  professional development activities and found
  those activities helpful
• Decrease in the need for professional development
  about the program and use of assessments
• Increased need for professional development on
  effective instructional strategies to use for
  students below grade level

16
Implementation Survey Items
17
Implementation Survey Items (Cont.)
18
Describe One Benefit of the SCRF Initiative

• Commonly cited benefits across groups
• Focus on assessment
• Increased awareness of students
  needs/strengths/weaknesses
• Progress monitoring Dominie
• Improved use of assessments use of assessment
  data to guide instruction
• Resources
• Additional books and materials
• Extra funding/money
• Professional development study groups
• Increased collaboration
• Learned new strategies

19
Results Participant Recommendations
20
Results Effectiveness Ratings
83.1 of teachers rated SCRF as effective or very effective. 88.1 of interventionists rated SCRF as effective or very effective. 81.8 of literacy coaches rated SCRF as effective or very effective. 90.9 of principals rated SCRF as effective or very effective.
21
Conclusions Achievement

• Students performed better on the Stanford Reading
  First assessment in the 2006-2007 school year as
  compared with the 2005-2006 school year.
• Students scores on the Stanford Reading First
  assessment improved from fall to spring in all
  three years for all grade levels.
• The largest gains were made by students in grade
  1 and the lowest gains occurred in grade 2 in
  2006-2007, which is also consistent with the last
  two years.

22
Conclusions Participant Groups

• Program participants have a better understanding
  of program goals.
• Participant groups find professional development
  useful.
• Over 80 of each participant group rated the
  initiative as either effective or very effective
  in Years 2 and 3.

23
Recommendations

• More time for sustained reading for students in
  grade 2
• Additional strategies for working with students
  below grade level
• Participant groups suggestions regarding study
  groups

24
Contact Information

• South Carolina Educational Policy Center
• University of South Carolina, College of
  Education
• Wardlaw Suite 010, Columbia, SC 29208
• P 803-777-8244
• F 803-777-0220
• E dmonrad_at_gwm.sc.edu (Diane Monrad)
• Office of Program Evaluation
• University of South Carolina, College of
  Education
• Wardlaw Suite 023, Columbia, SC 29208
• P 803-777-3402
• F 803-777-8838
• E tsdicken_at_gwm.sc.edu (Tammiee Dickenson)