Selecting and Randomizing Student Samples in Randomized Field Trials RFTs in Schools

1
Selecting and Randomizing Student Samples in
Randomized Field Trials (RFTs) in Schools

• April 14, 2009
• Philip Gleason
• Mathematica Policy Research, Inc.
• 2009 American Educational Research Association
  Meeting
• San Diego, CA

2
Many Studies Involve Randomization of Students

• Applicants to charter or magnet school randomly
  assigned to be admitted or denied admission
• Students randomly assigned into or out of an
  after-school program
• Students randomly assigned into classrooms either
  with or without an alternatively certified
  teacher (such as a TFA teacher)

3
Key Questions Involving Student Samples

• What students should be included in the sample?
• How should they be randomly assigned?
• Should informed consent be obtained?
• How should student mobility be handled?

4
Selecting Student Samples

• Determine sample size
• Depends on target minimum detectable effect
• Depends on assumptions about clustering
• Sample should be as representative of population
  of interest as possible
• Sample should include only students who are
  randomized

5
Selecting Student Samples Examples

• Lottery-based charter school study
• Applicants (siblings) exempt from lottery
  excluded
• Applicants from outside priority zone excluded
• TFA study
• Students assigned to a particular teacher with
  certainty (e.g., because of parental influence or
  special needs) excluded

6
Random Assignment of Students

• Random assignment of students has two parts
• Randomization mechanism (the coin flip)
• Actually assigning students to their intervention
  group
• Possible complications during randomization
• Stratification
• Special cases (e.g., students who must be kept
  together)
• Honest mistakes
• Each student must have known probability (between
  0 and 1) of assignment to the treatment

7
Using School Lotteries as Random Assignment

• School lottery is rare case in which random
  assignment is not conducted by study team
• Schools conduct fair lotteries, but with
  complications
• Siblings applying together
• Stratification / admission preferences
• Honest mistakes
• Schools do not typically document lottery
  outcomes as researchers would

8
Informed Consent

• Usually, researchers should obtain consent for
  students who are randomized in studies
• In some situations, FERPA may allow researchers
  to obtain student data without consent
• Challenge Design consent process that
• Maintains adequate sample sizes
• Maintains representativeness of sample
• Preserves integrity of random assignment

9
Obtaining Consent Prior to Lottery
Treatment Group Sample
Consenters
Select Sample
Random Assignment
Obtain Consent
Control Group Sample
Non-Consenters
10
Obtaining Consent After Lottery
Non-Consenters
Obtain Consent
Treatment Group
Treatment Group Sample
Select Sample
Random Assignment
Control Group Sample
Control Group
Obtain Consent
Non-Consenters
11
Student Mobility

• Should students who leave school be followed
  outcome data collected?
• Collecting outcome data for entire student sample
  is only way to ensure random assignment integrity
• Caveats
• No point in following student if no outcome data
  available
• Cost of data collection for movers can be high
• Modest amount of mobility/attrition acceptable if
  not influenced by treatment intervention

12
Conducting Studies Involving Random Assignment
of Students

• Objectives
• Maximize external validity of study sample
• Maximize internal validity of study design

• Concerns
• Minimize study costs
• Minimize burden on study participants

13
Best Practices for Selecting and Randomizing
Student Samples

• To promote internal validity
• Carefully randomly assign students yourself
• Obtain consent prior to random assignment
• Follow all student sample members
• To promote external validity
• Randomly assign all students who want
  intervention
• Maximize consent rate

14
Reasonable Compromises to Reduce Costs or
Respondent Burden

• Allow a few exemptions from random assignment
• Allow schools to conduct random assignment as
  long as study team monitors documents process
• Obtain consent after random assignment if
  treatment status unlikely to influence consent
  rate
• Drop sample members who leave study schools if
  mobility rate is low and not affected by
  intervention