School District of Philadelphia Evaluation of the Partnership for Character Education Grant

1
School District of PhiladelphiaEvaluation of the
Partnership for Character Education Grant

• Shelley H. Billig, Evaluator
• RMC Research Corporation
• Michelle Grimley, Project Director
• School District of Philadelphia
• March 2007

2
Agenda

• Description of the project
• Description of the quasi-experimental design
• Strategies that worked
• Challenges and how we addressed them
• Program insights
• Changes for the next round

3
Project Description

• Establish cadre of Service-Learning Liaisons
  throughout School District of Philadelphia
• Provide professional development activities to
  teacher-liaisons and at least 528 other teachers
• Provide mini-grants for teachers implementing
  service-learning projects in classrooms

4
School District of PhiladelphiaBackground
Information

• Approximately 180,000 K-12 students
• Poverty average of students 74.6
• 2006 PSSA test results for math 41.8
  proficient/advanced 36.5 below basic
• 2006 PSSA test results for reading 38
  proficient/advanced 41.3 below basic
• Over 270 schools organized in 12 regions
• 41 schools (out of 270) are managed by private
  education companies
• Each region is directed by a Regional
  Superintendent
• School District governed by 5 member School
  Reform Commission appointed by the state of PA
  and city of Philadelphia
• The District was taken over by the state in
  December 2001

5
Evaluation Design

• Quasi-experimental design with matched comparison
  groups
• Match is based on grade span, student
  demographics, and student achievement profile.
• Ideal comparison school was selected,
  principals were contacted and asked to
  participate schools were given an incentive
  specific classrooms were surveyed.
• Statistical checks for equivalence were made (no
  adjustments were needed).
• Analysis of mediating variables and test scores.

6
Methodology

• Pre/post student and teacher surveys with
  validated scales linked to the constructs being
  measured.
• Pilot was conducted in the first year to ensure
  validity and reliability.
• Cronbachs alpha was calculated for each subscale
  and all were in the adequate to excellent range.
• Measures included impacts on students and quality
  of processes.
• Professional development quality and
  organizational processes were measured in the
  first year in depth and in smaller ways in
  subsequent years. These measures included
  observations, surveys, and document analysis.

7
Factors that Facilitated Success of the
Evaluation

• Multiple communications about the evaluation by
  the project director and by the evaluators
• Passive parent permission to boost response rates
• Evaluator administration of all surveys
• SWAT team approach with minimal burden to the
  client
• Acknowledgement through thank you notes
• Dissemination of results to partners so the data
  were valued

8
Challenges

• IRB
• Getting achievement data in a timely fashion
• Ensuring the coding of the intervention was right
• Multiple project directors
• Context for program implementation in the district

9
Program Strengths

• Strong grant partners
• Advisory Board-Service Learning Support
  Organization
• Associate Superintendent of Curriculum support of
  grant goals
• Desire to implement character education/service-le
  arning in schools

10
Program Challenges

• School instability
• Number of reform initiatives implemented in
  District schools
• District-wide core curriculum and pacing guides
• Relationship building among key people for new
  director

11
Tips for Program Success

• Build relationships with influential District
  personnel.
• Offer incentives for comparison site
  participation.
• Promote program impact and importance.

12
Changes for the Next Project

• Qualitative study of best practice
• K-8 focus
• Regional focus
• Senior projects
• Stronger connection of Character Education grant
  program to Districts core curriculum