CaMSP Qualitative Findings and Best Practices

1
CaMSP Qualitative Findings and Best Practices

• Results from the Evaluation
• Year 3 Report (2006-07)
• Presented at the CDE CaMSP Network Meeting
• November 3, 2008
• Sacramento
• Patty ODriscoll
• Scott Phelps

2
Who Are We?

• Public Works, Inc. a nonprofit in Pasadena
  dedicated to working with communities,
  government, schools and parents by providing
  services and resources to educate and inform
  children, youth and families. Our work is in
  three areas
• Education Reform
• Workforce Development
• Intervention/Prevention

3
Research Questions

• How have the Partnerships ensured that all
  students have access to, are prepared for, and
  are encouraged to participate and succeed in
  challenging and advanced mathematics and science
  courses?
• How have the Partnerships enhanced the quality of
  the mathematics and science teacher workforce?
• What evidence-based outcomes from the
  Partnerships contribute to our understanding of
  how students effectively learn mathematics and
  science?

4
Statewide Evaluation Approach

• Process measures focus on how a program is
  implemented and the outcome measures focus on the
  results of the program or intervention.
• Public Works, Inc. uses both quantitative and
  qualitative data collection methods
• Purpose find trends across the Partnerships for
  a statewide picture and best practices discovered
  throughout the state

5
Statewide Evaluation Approach

• Website to collect Teacher Data and log
  Partnership Activities
• Partner Survey
• Participating Teacher Survey
• Site Visits (15-17 per year)
• Teacher Outcomes through Web database (teacher
  characteristics, program attendance)
• Student Outcomes (control/treatment)
• Final Report each year..

6
Levels of Evaluation

• Teacher/Student Assessment (Participants)
• District/School Evaluation (Teachers, Principals,
  Administrators)
• Local Evaluation of the Partnership (Evaluator
  and Leadership Team)
• State Evaluation (CDE/Public Works, Inc.)
• Federal Evaluation (USDOE)

7
Components of Qualitative Evaluation

• Partner Survey (involvement, roles and
  responsibilities)
• Teacher Survey (opinions and impact on teaching
  practice)

8
Components of Qualitative Evaluation

• Site Visits
• 2 Days
• Subset of sites (15-20) usually new cohorts in
  first year
• Interviews of multiple partners related to 5 key
  features
• PD Observation
• Telephone interviews
• New this year

9
Key Features, Findings Best Practices

• Features of the programs reviewed
• Partnership-driven
• Teacher quality
• Challenging courses and curricula
• Evidence-based design and outcomes
• Institutional change and sustainability

10
Partnership-Driven Feature

• Represents its target population of districts,
  teachers and institutions of higher education
• Commitment and engagement to the partnership
• Governance and equality of partners in planning
  and logistics
• Target population of the grant is served

11
Partnership-Driven Findings

• Partnerships are becoming more streamlined
• Concrete and committed partners--high quality PD
• Partners involved in previous cohorts helpful to
  recruiting and retaining teachers and can hit the
  ground running
• Strong coordination and strong district personnel

12
Partnership-Driven Best Practices

• Whittier and CSU Dominguez Hills PROMISE embeds
  coaching and continuous improvement

13
Teacher Quality Feature

• Cohort of teachers that are enrolled in all
  aspects of the professional development
• Target number and hours are being met
• Approach to professional development is tied to
  state standards and is focused on both improving
  the content knowledge and pedagogical approach of
  teachers
• Research-based, high quality PD for both
  intensive training and classroom follow-up
  components

14
Teacher Quality Findings

• Recruitment and retention of cohort is
  challenging but improving
• Differentiation of Intensive and Classroom
  Follow-up more clear
• High satisfaction with training
• More explicit integration with standards and
  textbooks needed
• Very little emphasis on teacher college credit

15
Teacher Quality Best Practices

• San Benito Rigorous Coaching Model
• RAAFA Lesson Study
• North San Mateo County Coaching
• Vallejo Observation and Coaching
• Content Lesson Study in Alameda
• San Diego City Intensive Lesson Study and K-12
  Alliance

16
Challenging Curricula and Courses Feature

• Professional development is aligned to standards
  and aimed at transforming and improving
  instruction
• The project is creating new challenging courses,
  lessons and curricula for pre-service or existing
  teachers and/or students
• New courses, curricula, expectations or
  experiences for Students will result from the
  professional development teachers receive in this
  grant

17
Challenging Courses Curricula Findings

• Hoping that program will result in new courses
  for students and teachers
• Some connections being made between districts and
  IHE training
• Exposure to collaborative curriculum development
  at most partnerships
• A few partnerships were offering challenging
  online online teacher courses or supplemental
  courses or assignments

18
Challenging Courses Curricula Best Practices

• San Benito PROJECT Smart use of local resources
• CSSP in Orange County integration of EL
  strategies and coaching
• ExCEL in Fullerton Institute and Summer Lab
  School
• San Diego COE Online PD
• Summer School
• Pacing Guides

19
Institutional Change Sustainability Feature

• The role of the IHE is clear and integral to the
  project. The institutions (IHE and Districts) is
  impacted by the project in terms of a tangible
  result.
• The IHE is involved in the planning, curriculum
  development and delivery.
• Education department and the discipline
  department (math/science) are involved.
• Teachers can receive credit for their
  professional development.
• Sustainability

20
Institutional Change Sustainability Findings

• Involvement of both IHE disciplines strengthens
  the partnership
• Teachers appreciate access to IHE content
  specialties
• CaMSP raising priority of mathematics and science
• Professors improved their practices
• Want to sustain coaching and lesson study but
  still working on how

21
Institutional Change Sustainability Best
Practices

• TAPS/Sustainability Teacher Apprenticeship
  Project placing undergraduate students
• Pre-Services Courses
• IHE Professor and Department Change
• Chula Vista and USD Masters Level Programs

22
Evidence Based Design Outcomes Feature

• There is a research or evidence-based model for
  their professional development
• Evaluation plan makes sense for the project
• The design and measurement system will produce an
  impact on teacher and classroom quality. There is
  a cohort of teachers that make sense to examine
  the impact on student outcomes
• A pre-/post- assessment of teacher knowledge is
  conducted and local student assessment is
  conducted (above and beyond CSTs)

23
Evidence-Based Design Outcomes Findings

• Impact of grant on teachers and students
  continues to be difficult to measure for local
  evaluators
• Teacher feedback is positive but knowing the
  impact on student outcomes is a challenge
• Partnerships are responsive to state evaluation
  requests

24
Evidence-Based Design Outcomes

• Student Assessment Across Districts in CSSP
• Targeting a grade level as groundwork for local
  evaluation
• Video Clip Analysis in Lesson Study
• Local Student Assessment in the multi-district
  North State Mathematics Partnership

25
Year 3 Report Outline

• Introduction
• Findings related to five key features (integrated
  site visit and survey with more focus on Cohort
  3)
• Best practices
• Teacher Outcomes Cohort 3
• Student Outcomes from control/treatment group of
  teachers on statewide measures from Cohort 1 and
  Cohort 2
• Available at www.publicworksinc.org

26
Recommendations

• Continue to help sites refine structure of model
  and encourage consistency in intensive and
  classroom follow-up activities
• Focus on the integration of other statewide
  initiatives in mathematics and science at the
  state and local level and with other policymakers

27
Recommendations

• Encourage involvement of IHEs across departments
  that align to long-term needs of teacher
  workforce in mathematics and science
• Provide support to districts (especially small
  and mid-size) to sustain professional development
  efforts and build professional development
  capacity

28
Recommendations

• 5. Strengthen local research/evaluation projects
• 6. Provide technical assistance in student and
  teacher assessment
• 7. Improve alignment of contract period,
  reporting and research design for Cohorts
• 8. Improve statewide data collection and
  accessibility