School Selection and Randomization for a School RCT of a Universal Social-Emotional Learning and Literacy Intervention

1
School Selection and Randomization for a School
RCT of a Universal Social-Emotional Learning and
Literacy Intervention
Joshua L. Brown Fordham UniversityConferen
ce on National and International Perspectives on
Place-Based Randomized Trials in Education
Institute of Human Development and Social Change
New York UniversityOctober 3, 2008
2
NYC Study of Social and Literacy Development
Principal Investigators Joshua L. Brown
Stephanie M. Jones Fordham University
Harvard University J. Lawrence Aber New York
University
3
Acknowledgements
Research TeamGenevieve Okada, Site
CoordinatorSuzanne Elgendy, Vanessa Lyles, Emily
Pressler, RAsWendy Hoglund, Postdoctoral
FellowMaria LaRusso, Postdoctoral
FellowJuliette Berg, Catalina Torrente, GRAs
Program Partners Tom RoderickAudrey
MajorMorningside Center for the Teaching of
Social Responsibility FundersInstitute for
Education Sciences, DOENational Center for
Injury Prevention and Control, CDCWilliam T.
Grant Foundation National Institute of Mental
Health
4
Outline

1. Background/Context of Study
2. Program and Study Design
3. Pre-Randomization Activities
4. Matching and Randomization
5. Implications of Approach
6. Current Preliminary Findings (Y1, Y1-2)
7. Conclusions and Future Challenges

5
Background (1)

• Co-occurrence of social-emotional and behavioral
  problems with low academic achievement.
• Theoretical and initial empirical links between
  self-regulation and reading/math.
• Emphases on standardized testing and
  instructional improvement have crowded out
  attention to social-emotional-character
  development (among other things).

6
Background (2)

• Early efforts at whole school strategies to
  prevent behavior problems, violence, and
  substance use plagued by intervention design and
  implementation fidelity problems.
• Early research on whole school strategies plagued
  by low power, and inappropriate statistical
  analyses.
• Need to rigorously test promising but unproven
  approaches to SEL/SACD.

7
Background (3)

• Birth of Social and Character Development
  Research Network.
• 7 different interventions in 7 different sites.
• 7 Local Evaluations and 1 National Evaluation
  (Mathematica Policy Research).

8
Reading, Writing, Respect and Resolution
(4Rs)Program and Study Design

9
The 4Rs Program

• Universal, school-based intervention in literacy
  development, conflict resolution, and intergroup
  understanding.
• 3 Primary components
• 7-unit literacy-based curriculum in conflict
  resolution and social-emotional learning.
• Each unit organized around grade-appropriate
  book, includes 2 literacy activities, and 3-5 SEL
  lessons (21-35 total lessons).
• Total possible activities per unit 5-7
• Total possible activities per year 35-49
• Training and ongoing coaching of teachers in the
  delivery of the 4Rs curriculum.
• 25 hours introductory training
• Ongoing classroom coaching, minimum 12 contacts
• Learning kit
• Family Connections
• 1 parent-child homework per unit

10
Heuristic Model 4Rs Child-Level Study
Teacher Development
Social-Emotional Skills Behaviors
4Rs Experimental (classroom and parent)
vs. Control
Literacy Skills Academic Achievement
Extended Opportunities Supports
11
Heuristic Model 4Rs Setting-Level Study
School Culture and Climate
The Classroom System Culture and Climate
Teacher Affective Pedagogical Processes
Practices
Classroom Emotional, Instruct. Org. Climate
Child Developmental Outcomes SEL Academic
Achievement
4Rs Instruction, Teacher Training Coaching
Teacher- Child Relationships
Child Behavioral Dispositions Normative
Beliefs
12
Overall Study Design

• 3-year, 6 wave longitudinal experimental design
• 18 NYC elementary schools matched and randomly
  assigned to 4Rs and control group (9 assigned to
  each group)
• Intervention is implemented school-wide, grades
  K-6 for 3 years
• All 3rd grade children in each school followed
  over three years through 5th grade
• Schools represent demographic character of NYC
  public elementary schools
• Racially/ethnically diverse School lunch receipt
  70 Mobility/Stability 18/60 Suspensions
  23

13
Pre-Randomization Activities

14
Identifying Candidate Schools

• Planning Year (January-March, 2004)
• History of practitioners work in NYC led to
  letters of support from Regional Superintendents
• Facilitated direct contact with Local
  Instructional Superintendents who recommended
  schools based on
• no prior history of implementing 4Rs
• willingness to implement program (and research)
  activities
• all Teachers participate in Intro. Training,
  teach curriculum 1 lesson/week, and receive
  ongoing coaching from 4Rs Staff Devs
• Principals attend 2-3 workshops/year and appoint
  4Rs Liaison
• School administration and teachers cooperate with
  data collection
• Resulted in LIS recommendations of 41 schools

15
Assessing Candidate Schools

• Planning Year (March-June, 2004)
• Goal assess/recruit viable schools for
  program implementation in context of research
  study, (i.e., capacity for sustained,
  high-quality implementation, but room to improve
  willing to be randomly assigned)
• Process Meetings and walk-throughs of all 41
  schools
• Individual meeting with Principals to present
  program model and overall research design
• Visits to classrooms
• Organizational Readiness assessment completed
  by practitioners (co-developed with research
  team)

16
Organizational Readiness

• Principal Leadership
• Organizational skills
• Enthusiasm for/compatibility with 4Rs
• Rapport with Students Staff
• Teachers School Leadership Team
• Relationship with Principal
• Enthusiasm for/quality of questioning about 4Rs
• Stress, morale attitudes toward children
• School Environment
• Tone of adult-child interaction
• Engagement and behavior of students
• Physical environment (e.g., use of bulletin
  boards, etc.)

17
Recruiting/Selecting Schools

• 17 schools eliminated from initial pool
• Grade structure other than K-5 (e.g., no 5th
  grade, K-3)
• Lack of Principal and/or teacher interest in
  program
• E.g., cant support school-wide implementation
  requirement and/or balance competing academic
  demands
• Highly chaotic environments (e.g.,
  adult-adult/adult-child yelling, extensive
  behavior problems, frequent crises)
• Unwilling to risk random-assignment to Control
  condition (one preferred to purchase program)
• 24 schools held staff vote, signed letters of
  agreement for random assignment

18
Pairwise Matching of Schools

• Given potential for bad draw with small number
  of schools, IES grantees agreed to pairwise match
  schools to ensure balance on key variables and
  increase precision
• 24 schools pairwise matched and rank-ordered
  based on distance of each school from every
  other eligible school across 20 key school
  characteristics ? 12 pairs
• School Characteristics Include
• Size (total N)
• Race/ethnic and gender composition
• School lunch receipt
• Attendance (Students and Teachers)
• Reading achievement ( of students at or above
  proficiency on ELA test)
• Within year student mobility/two-year stability
• Teacher full licensure and years of experience
• Expenditures
• Organizational Readiness (Overall)

19
Matching ? Random Assignment

• Funding for 18-schools, kept 9 best matching
  pairs, but maintained 3 back-up pairs during
  lead-up to program implementation/data collection
• Random numbers generator used to assign 1 school
  in each pair to intervention and 1 to control
  conditions
• Post-random assignment, 2 schools and their
  respective matches were dropped and 2 back-ups
  engaged
• Principal of Tx school had been previously
  trained in RCCP
• LIS ultimately did not condone RCT design for her
  schools
• Note, pairwise matching can protect the
  experimental design (from selection bias) in case
  of schools dropping out after start of study
  (King et al., 2007)

20
Implications of Matching for Analyses

• Should blocks (e.g., matched pairs) be regarded
  as fixed or random effects?
• Current debate in field, depends on
• Number of units per block (when only 2, need RE
  model)
• Treatment effect heterogeneity (i.e, across
  matched pairs)
• If large, RE model allows heterogeneity of Tx
  effect to contribute to standard errors and tests
  for the average effect of Tx.
• Interest in generalizability
• In FE model, the blocks constitute the population
  or universe of generalization
• In RE model the blocks are seen as representing a
  larger universe of possible blocks (or settings)
  in which Tx might be implemented.
• See Schochet, 2004 Raudenbush, 2004 and Bloom,
  2005 all SACD internal network communications
• We estimate blocks as random effects at the
  school-level most conservative

21
Implications of Approach

• Evidence of effective matching process -- no
  Tx/Control differences in 20 initial matching
  variables, or baseline constructs assessed via
  child, teacher, and parent-reports
• Focus on initial identification and subsequent
  selection of viable schools limits
  generalizability to district- or citywide
  population
• However, features of school organizational
  capacity and support have been clearly linked to
  schools ability for quality program
  implementation (Payne, Gottfredson Gottfredson,
  2006)
• A fully representative sample of NYC elementary
  schools would yield many schools with weak
  engagement and early withdrawal from study
• Although generalizable only to willing
  implementers, we see preliminary evidence of Tx
  impacts

22
Year 1 Child Impacts (Jones et al, under review)

• Main effects of Tx on 2 of 9 child outcomes (2L
  HLM)
• Controlling for baseline levels, children in the
  Tx group had lower mean levels of Hostile
  Attribution Bias and Depression than those in the
  control group at the end of Y1
• Tx by baseline covariate interactions for 5 of 9
  outcomes
• E.g., Tx by baseline Behavioral Risk (elevated
  on teacher-report aggression and/or conduct
  problems at baseline)
• Children with the highest level of baseline
  behavioral risk show the greatest positive
  difference in Aggressive Fantasies,
  Teacher-Report of Academic Skills, Reading Scale
  Score, and Attendance between the intervention
  and control groups

23
Year 1 Classroom-Level Impacts (in Effect Sizes)

• Classrooms in the Tx group had higher mean
  Overall Classroom Quality scores, accounted for
  by higher mean Emotional Support
• and Instructional Support scores, than the
  control group

n.s.
24
Preliminary Year 1-2 Program ImpactsChild-Level

25
Results Child-LevelTX Main Effects

• Local significant impacts for 2 of 6 constructs
• Child Self-Report Hostile Attributional Biases
• Child Self-Report Depression
• Multisite significant impacts for 3 constructs
• Teacher-Report of Aggression
• Teacher-Report Social Competence
• Teacher-Report ADHD Symptoms

26
Y1-Y2 TX Main Effects Summary
Intercept Unstand. Est. (SE) Slope Unstand. Est. (SE)
Hostile Attributional Bias -.034 (.029) -.043 (.023) t
Depression .013 (.034) -.065 (.023)
TR Social Competence (Emotion Regulation Prosocial Behavior) -.009 (.100) .137 (.077)
TR ADHD (Hyperactivity Inattention) .030 (.086) -.084 (.053) t
TR Child Aggression .027 (.031) -.047 (.017)
27
TX on HAB Slope
28
Conclusions and Future Challenges

• Trade-offs between generalizability and design
  feasibility in school selection process for
  school RCTs.
• We opted for selectivity based on
• LIS perspective of school need/capacity
• Pre-randomization assessment by program
  practitioners of school organizational readiness,
  used to identify final sample
• Need rigorous and field efficient assessment
  tools that tap multiple dimensions of school
  organizational capacity
• How might RCT design innovations enable the
  inclusion of disorganized and at-risk schools
  most in need of intervention?