Key issues in the evaluation of schoolbased SEL programmes

1
Key issues in the evaluation of school-based SEL
programmes

• Dr. Neil Humphrey
• Senior Lecturer Psychology of Education
• University of Manchester

2
Overview

• Introduction
• Examples of evaluation models
• Primary SEAL small group work
• Secondary SEAL
• Targeted Mental Health in Schools (TaMHS)
• Key Issues
• Questions/Discussion

3
Examples of evaluation models

• Primary SEAL small group work evaluation (2007-8)
• Aims
• To assess the impact of small group work on
  children requiring more support in developing
  their social and emotional skills.
• To gather information on successful
  implementation of small group interventions
• Staff, pupil parent ELAI, staff and parent SDQ
  (N624)
• Lead practice case studies (N6 schools)
  interviews and/or focus groups with small group
  facilitators, class teachers, senior managers,
  pupils, parents observations of small group work
  and other settings document analysis
• Plus additional quantitative measures
• CRPM
• KAI

4
Examples of evaluation models

• Secondary SEAL evaluation (2007-2010)
• Aims
• To assess the impact of SEAL on pupils and whole
  school climate, organisation and processes.
• To explore how SEAL is implemented in secondary
  schools and examine how different models of
  implementation affect the pupils.
• 26 SEAL schools vs. 23 non-SEAL schools (N7500
  Y7 pupils)
• Schools matched on size, attendance, attainment,
  SES, SEN
• Baseline assessment in 2008, followed up in 2009
  2010 using ELAI and SDQ

5
Examples of evaluation models

• Secondary SEAL evaluation (2007-2010) (contd.)
• Longitudinal case-studies (N9 SEAL schools)
• Quantitative measures (2008, 2009, 2010)
• Pupil school climate (SCCP), understanding/invol
  vement in SEAL
• Staff school climate (SCCP), EI skills (RTS),
  SEAL implementation status
• Qualitative methods
• Classroom level observations of a range of
  lessons, interviews with subject teachers and
  teaching assistants, focus groups with pupils
• School level observations of other contexts
  (e.g. lunch-time), interviews with head-teacher
  and/or senior management lead with responsibility
  for SEAL, document analysis (e.g. behaviour
  policy), analysis of attendance and exclusion
  trends
• Local Authority level interviews with LA staff
  with SEAL responsibility and relevant Childrens
  Services staff (e.g. educational psychologists,
  behaviour and attendance consultants)
• Community level interviews with parents and
  staff from relevant voluntary and community
  services (e.g. youth workers).

6
Examples of evaluation models

• TaMHS evaluation (2008-11)
• Aims
• To investigate the comparative efficacy of
  different approaches to TMHS
• To identify the barriers to and facilitators of
  implementation
• To use the above information to develop and
  introduce an experimental intervention introduced
  as part of an RCT to see how far these barriers
  can be successfully addressed.
• Study 1 (2008-11) a natural variation/multi-leve
  l-modelling study in 25 Pathfinder areas linking
  TaMHS models with specified outcomes (e.g.
  improvements in child mental health)
• Study 2 (2009-11) an RCT using an intervention
  developed from the initial findings of Study 1
  TaMHS (10), TaMHS-enhanced (10), no TaMHS (20)
• N 35,000 pupils
• Staff and parental measures for a (much) smaller
  proportion
• Measures
• Area e.g. typology of TaMHS provision
• School e.g. school climate
• Child e.g. SDQ

7
Key Issues

• What constitutes an SEL programme?
• What should be evaluated?
• What constitutes a good outcome?
• Whose perspective should be prioritised?
• Other key issues

8
What constitutes an SEL programme?

• SEL is a process for helping children and even
  adults develop the fundamental skills for life
  effectiveness. SEL teaches the skills we all need
  to handle ourselves, our relationships, and our
  work, effectively and ethically. These skills
  include recognizing and managing our emotions,
  developing caring and concern for others,
  establishing positive relationships, making
  responsible decisions, and handling challenging
  situations constructively and ethically. They are
  the skills that allow children to calm themselves
  when angry, make friends, resolve conflicts
  respectfully, and make ethical and safe choices.
  (CASEL, 2007, p.1)
• Typically underpinned by models of emotional
  intelligence (e.g. Goleman, Salovey Mayer)
• A fairly broad church!
• Some argue that such an all-encompassing
  characterisation leaves SEL bereft of any
  conceptual meaning (Zeidner, Roberts Matthews,
  2002, p.215)
• The emotional content of some SEL programmes is
  sparse
• Some SEL programme evaluations dont include any
  assessment of proximal SEL outcomes

9
What should be evaluated?

• Proximal vs. distal variables
• Proximal
• Pupil social and emotional skills
• Staff social and emotional skills
• School climate
• Trait vs. ability measures of SE skills
• Distal
• Improved behaviour and reductions in exclusions
• Increased attendance
• More effective learning and increases in
  attainment
• Improvements in mental health (e.g. lower levels
  of stress and anxiety)
• Many SEL programme evaluations select only 1 or 2
  of the above

10
What constitutes a good outcome?

• Statistical significance vs. practical
  significance
• Tendency to equate statistical significance with
  success, but NHST is confounded by sample size
• Whole school SEL programme evaluations tend to
  have larger sample sizes increased power
  (sensitivity) of test even small differences
  flagged as statistically significant
• Effect size analysis can be more meaningful, but
  is rare in this field
• e.g. in Adi et als (2007) recent systematic
  review of universal approaches in primary
  education, only 4 of 31 studies reported effect
  size (and the average effect size was small)

11
Whose perspective should be prioritised?

• Range of stakeholders
• e.g. children, staff, parents
• Low concordance rate between stakeholders
• e.g. primary SEAL ELAI measure
• staff-parent correlation 0.33
• staff-pupil correlation 0.37
• pupil-parent correlation 0.21
• Some studies using triangulated outcome
  measurement report positive outcomes from one
  source (e.g. pupils) and not the others (e.g.
  staff, parents)
• School-based SEL programmes (whole school or
  targeted) more likely to impact upon outcomes in
  school than at home
• Generalisation of skills?
• Involving parents?
• Views of children are desirable, but these tend
  to be the least reliable
• Also, involvement in programmes may increase
  sensitivity to change
• e.g. ratings of difficulties may increase as they
  become more aware

12
What should be evaluated? Part 2

• Not everything that can be counted counts, and
  not everything that counts can be counted
  (Einstein)
• Much of the change that occurs during the
  implementation of an SEL programme can be very
  difficult to quantify
• Highlighting process can be just as enlightening
  as reporting outcomes
• We need to know the how and why for that,
  qualitative data can be invaluable
• Context and conditions of implementation can help
  to explain why a programme has an impact in one
  school, but not another
• Enhancing the credibility of qualitative inquiry
  in SEL evaluations
• Triangulation
• Member checks/respondent validation
• Reflexivity
• Comprehensive documentation
• Negative case analysis

13
Other key issues

• Fidelity!
• The tussle between rigidly following guidance and
  adapting to meet specific needs can impact
  greatly on evaluation outcomes
• Efficacy vs. effectiveness
• What else is in the mix?
• The pace of government-led innovation has
  outstripped the capacity of schools to respond
  (Coffield et al, 2007)
• Healthy Schools
• SEAL
• TaMHS
• BIP and BEST
• PSHE
• .etc.
• All of the above create a lot of noise when
  trying to isolate the effects of an SEL programme
• The cart before the horse?
• Educational policy in this area has outpaced the
  science on which it is ostensibly based (Mayer
  Cobb, 2000, p.163)

14
Thats all folks!

• Thank-you for listening
• Any questions?