Recruiting Sites and Participants for Random Assignment Studies

1
Recruiting Sites and Participants for Random
Assignment Studies

• IES/NCER Summer Institute
• June 25, 2007
• Fred Doolittle

2
Overview of the Session Topics

• What is the purpose of site and participant
  selection?
• The art of recruiting sites for random assignment
  studies
• Topics to address
• Building the foundation for longer term success
  of the study
• Addressing the tough issues in advance
• Assuring cooperation in study implementation

3
The Goal of a Fair Test of the
Intervention/Program

• Some elements of the fair test
• Sites that serve the intended target group
• Sites that can operate the intervention
  reasonably well
• Sites that can provide the intended service
  contrast
• A research design that can produce credible
  findings on impacts and can be implemented
• A data collection plan that preserves the good
  impact design and documents differences in
  services and outcomes at the various stages of
  the theory of action
• An analysis strategy and report process that
  produces useful findings
• In various ways, all of these involve the sites

4
Topics for a conversation

• The process of recruiting sites
• Embedding a random assignment design and data
  collection into normal program operations
• Asking the tough questions
• Real understanding of the design
• Political issues
• Ethical concerns
• Working to achieve long-term cooperation in study
  implementation
• Lack of crossovers and withdrawals
• Data collection

5
Reading the Site Recruitment Context
6
Factors to Consider

• How many sites do you need?
• What restrictions are there on your choices?
• Geographic- need to be close to you, need to be a
  specific places because of funder, need to test
  in specific kinds of places, need to be
  representative
• Programmatic-
• do sites need to be doing something on which you
  build
• not doing something so get service contrast
• Size need to be big enough to provide the
  desired sample of clusters, teachers and/or kids

7
Factors (cont.)

• Are you offering something new or wanting to
  study existing programs?
• Does what you offer fit into their existing plans
  programmatically and operationally so easy sell?
• After-school example
• Reading Professional Development example
• Is the existing program long established and
  popular?
• Is it oversubscribed so there is scarcity?
• How have selection decisions been made?
• Does it have outcome-based performance standards?
  
• Others????

8
Recruiting Framework

• What is the benefit-cost balance in participating
  from the perspective of sites?
• Possible Benefits- the potential benefits for
  kids because of new services, staff professional
  development, the visibility of being in a study,
  site-level findings, etc
• Possible Costs the need to devote management
  time to the intervention and study, a divergence
  from local priorities, hassle of integrating the
  intervention into local operations, controversy
  of random assignment, operational hassles of
  random assignment, the hassle of data collection,
  etc.

9
Responding to the Site Recruitment Context

• The context affects the sequence of topics
• The more it looks like a tough sell, the more you
  lead with benefits of participating
• The more it looks like an easy sell, the more you
  move quickly through the benefits and into the
  roles and responsibilities
• In either case
• The goal is to get folks interested in
  participating so they can help you solve the
  inevitable problems, find ways to lessen the
  costs, and identify locally-relevant benefits
• Always include the benefits in any material
  because it may be passed around to new folks who
  need to understand why to bother with this.

10
Examples of Strategies

• Building site recruitment into special program
  funding decisions
• Still need engagement process
• Selecting representative sites
• When might this be appropriate?
• What is the pitch?
• Using data bases to identify prospects that fit
  the profile and then making cold call contact
• Data bases may not include key criteria
• With whom to make the contact?
• What is the pitch?
• Working through service provider or funding
  networks to gain access
• Pros and cons
• What is the pitch?
• Substantial outreach to build demand and
  stimulate applications
• What is the balance between benefits and
  obligations in initial outreach?
• How orchestrate the application and selection
  process?

11
What to Expect When Make Contact

• In districts/schools/programs
• Everyone is overburdened and stressed
• Many things are changing simultaneously
• A new intervention is of greater interest than
  the research
• Getting chosen for a study and/or getting
  something free might be regarded as
  significant--but not for long
• Evaluation results either overall or
  site-specific - often do not matter much because
  way off in the future

12
Recruiting Process

• What is the point of entry?
• The pros and cons of starting high in an
  organization
• Recovering from an initial rejection
• First dates (i.e., meetings) are pivotal
• Make a good impression by bringing the right
  info and people
• Get the right people from the site in the room
• Build relationships from the outset
• Be the buyer and the sellerexplain the benefits
  but ask the tough questions relatively early
• Try to understand their perspective
• Be willing to dig in to details

13
Embedding Random Assignment into Normal
Operations
14
Why Does This Matter?

• Your goal is to mesh the research procedures with
  their operations
• If you do this well, it significantly reduces the
  cost to a site of participating
• If you get them interested enough in the possible
  benefits, they will start helping you figure this
  out

15
Understanding Normal Program Operations

• How does the sample usually get to the place
  (program, school, classroom, etc) where you want
  to introduce the intervention and do random
  assignment?
• Ask managers and line staff
• Focus on understanding the usual way and any
  exceptions, alternative routes in, etc.
• Understand the timing of the steps and the
  information that is available at each step
• Understand the frequency of later corrections
  and how done
• Can be because of mobility of students, late
  hires, turnover
• Cannot fix this after random assignment

16
Understanding Normal Program Operations (cont.)

• For the kids
• Do kids apply? If so how and when?
• If not, how are appropriate kids identified? Who
  is involved? Counselors, other teachers, referral
  agencies? Do they expect all they refer to get
  in?
• Who decides which kids are accepted?
• When are these decisions made?
• What data is normally available or collected on
  the kids at various stages?
• What percent of those selected actually
  participate?

17
Understanding Normal Program Operations (cont.)

• For the staff
• How and when are staff assignments made?
• What is the usual skill set of staff?
• What other responsibilities do these staff have
  that might affect your ability to train on
  intervention?
• Are there other constraints, for example union
  rules?
• Matching kids with staff and services
• How and when is this normally done?
• Is there a tradition of special cases parents,
  staff requesting changes?
• Is there a tradition of deciding some kids just
  have to have this staff person, program
  services, etc. and these are handled outside the
  usual process?

18
Normal Program Operations (cont.)

• Ask about information and time available at key
  stages
• What info is available on kids at different
  stages to see if eligible/appropriate for
  intervention or to serve as baseline data?
• Is there time to introduce informed consent and
  baseline data collection prior to random
  assignment?
• When will any research delays be a problem
  because scheduling complicated, because
  teachers/kids/parents need to know what is up,
  etc. ?

19
Examples

• Individual random assignment
• After-school services
• Upward Bound
• School level random assignment
• Elementary school teacher professional
  development in reading through summer institutes
  and coaching
• Middle school teacher professional development in
  math through summer institutes and coaching

20
Focusing on Service Receipt

• Two parts to this are key because the service
  contrast is what drives impacts
• Program group Want strong implementation of the
  intervention to intended kids, with intended
  participation
• Worry about mobility of kids and turnover of
  staff
• Control group Want clear service contrast for
  the control group
• Not necessarily no service
• But clearly different services

21
Picking the Point of Random Assignment

• How close to the start of services do you put the
  lottery?
• Choice affects
• the question you address,
• the sample,
• the baseline data you will have, and
• service participation for program and control
  group
• The tradeoff
• Late means greater participation rate for the
  program group but also more hassle for staff,
  harder to plan, more disappointment if not
  selected, and -usually - more motivated control
  group so more services for controls
• Early means the opposite

22
Examples

• Individual level random assignment
• KIPP middle school evaluation
• Mentoring programs
• Cluster random assignment
• The previous professional development examples

23
Setting Random Assignment Ratios

• This can be part of the site recruitment
  discussion
• A balanced design is best for power, but not
  necessary
• The power drop-off is not major until move past
  21
• An unbalanced design can make a big difference to
  sites and help address ethical concerns
• A very unbalanced design with many sites is an
  option if data collection costs are manageable
• Avoiding empty program slots is important
• Overbook somewhat the program group
• Use a non-research waiting list to fill empty
  slots
• In extreme, can set program group size to fill
  available program slots and the control group is
  the rest

24
Handling the Sensitive Cases

• Much better if do this prior to random assignment
• Tell programs if absolutely necessary they have a
  few chips they can use prior to the lottery for
  cases where they could not live with control
  group designation
• They apply to use a chip for a person or cluster
  before random assignment
• Affects generalizability but not internal
  validity
• In cluster random assignment, can include some
  extra non-study staff from program clusters in
  program services
• Sometimes have to accept a crossover to save a
  study
• Small numbers in a large sample do not matter a
  lot

25
Dont let post-lottery procedures undo the
randomness

• Post-random assignment allocation of kids to
  clusters
• Compare school-level random assignment to
  teacher-level random assignment
• In the latter, important to get commitment of
  kids to clusters or a clear process to do it
  before conduct random assignment

26
Avoid Undoing Randomness

• Data collection problems
• Low rates for sample
• Differential rates across research groups
• Build tracking into design
• Try to mesh data collection with normal
  participation in services
• Create incentives
• Cover the costs at a minimum
• Provide positive incentives- there are OMB limits
• Special incentives for control sample?
• Clusters
• Individuals
• Pay for a local study liaison to coordinate data
  collection

27
Asking the Tough Questions Early
28
Tough Questions to Ask

• Do you really want to do this or is someone
  forcing you?
• Do you really understand random assignment?
• Are you committed to using a lottery to decide
  who gets the intervention and who does not?
• Do you really understand your roles and
  responsibilities related to
• Research procedures
• Data collection
• Implementation of the intervention

29
More Tough Things to Ask

• Who could object to the study procedures and are
  you prepared to confront these objections and
  stay the course?
• unions, parents, governing board, participant
  referral sources
• Can you devote the staff time needed to manage
  the study and the intervention?

30
Even More Tough Questions to Ask

• Are you about to (likely to) do something else
  that will affect our ability to pull off the
  study?
• Change local priorities so not interested?
• Introduce some other similar intervention so not
  a service contrast?
• Lay off staff so cannot staff it?
• Study champion is about to retire or take a new
  job?

31
Still More Tough Questions

• Are there lurking ethical concerns?
• Uncertainty about real scarcity?
• Need to recruit more to have a control group?
• Local certainty the intervention really works so
  should not deny access?
• Local certainty can identify those most in need
  who will benefit?
• Sense of coercion despite informed consent?
• ????

32
Negotiating and Closing the Deal
33
Negotiating a Formal Agreement

• A formal, detailed agreement is important
• It should specify respective roles and
  responsibilities about both program and research
  activities, timelines, costs
• The agreement must be cleared directly with all
  relevant decision makers
• The head person must read and sign
• This takes time but worth the effort in the long
  run

34
Negotiating and setting the stage (cont.)

• Once signed an agreement is gradually forgotten
  by many
• Therefore, one must have at least one properly
  placed internal champion to push forward the
  study
• The champion must be tended and kept engaged via
  ongoing communication
• Other relationships also matter make the rounds
  periodically