Randomisation Bias and PostRandomisation Selection Bias in RCTs:

1
Randomisation Bias and Post-Randomisation
Selection Bias in RCTs
The role of non-experimental methods in the ERA
demonstration

• Barbara Sianesi
• Institute for Fiscal Studies
• September 14, 2006

Randomised Controlled Trials in the Social
Sciences Challenges and Prospects The
University of York
2
Talk Outline

• RCTs are the gold standard in evaluation
• BUT not immune from limitations
• parameter retrieved
• outcomes that can be looked at
• ? Judicious combination with non-experimental
  methods can enhance (under suitable assumptions!)
  what can be learnt from a RCT
• Excellent example to illustrate this ERA

3
What is ERA

• a new package of
• support
• financial incentives (job retention, training)
• to assist ND25 and NDLP customers obtain, retain
  and advance in work
• evaluated via RA (14,000 in 6 districts)

4
Issue 1

• Some eligibles in the ERA Districts
• did not reach the decision stage or
• refused to take part in research scheme
• Experimental contrast ? unbiased estimate of ERA
  impact for those who have reached the RA stage
  have agreed to participate.

5
(No Transcript)
6
ERA impact on ERA participants
7
(No Transcript)
8

• Staff
• discretion. choice of marketing strategy
• Customer
• finding job unlikely
• finding job likely but no desire to stay in
  touch with JCP
• antipathy to government, systems of support,
  mandatory programmes
• resistant to change,

9
(No Transcript)
10

• Staff
• Some may not think customer would benefit / be
  interested.
• Some non-ERA advisers may think customers close
  to job entry may provide quick win.

11
?
12
?
Non-participants
13
Why do non-participants pose a potential issue?

• Would have liked experimental estimate of impact
  of ERA for the full eligible population in the
  ERA Districts.
• Benchmark pilot/control area evaluation
• In ideal scenario
• staff would offer ERA to any eligible (no
  discretion)
• all eligibles would participate (no need for
  consent)

14
Why do non-participants pose a potential issue?

• Would have liked experimental estimate of impact
  of ERA for the full eligible population in the
  ERA Districts.
• But ERA tested only on a subset of ERA eligibles
  in ERA Districts the participants.

15
How to view this

• Interested in impact on eligibles but only get
  impact on participants.
• Randomization Bias occurs when random
  assignment causes the type of persons
  participating in a program to differ from the
  type that would participate in the program as it
  normally operates.
• (Heckman and Smith, 95, p.99)

16
How to view this

• Focus on what the RCT consistently estimates
  (impact on participants) and interested in how
  well it generalizes to wider population (impact
  on eligibles).
• Issue of External Validity.
• How representative of the full eligible
  population?

17
When are non-participants a problem?

• E(impact eligibles)

What we want
18
When are non-participants a problem?

• E(impact eligibles)
• E(impact eligible partic.)

what we want
19
When are non-participants a problem?

• E(impact eligibles)
• E(impact eligible partic.) Prob(eligible
  partic.)

E(impact eligibles) E(impact eligible
partic.) Prob(eligible partic.)
what we want
observed
20
When are non-participants a problem?

• E(impact eligibles)
• E(impact eligible partic.) Prob(eligible
  partic.)
• E(impact eligible non-part.)

what we want
observed
?
21
When are non-participants a problem?

• E(impact eligibles)
• E(impact eligible partic.) Prob(eligible
  partic.)
• E(impact eligible non-part.) Prob(eligible
  non-part.)

what we want
observed
what we get
?
22
When are non-participants a problem?

• ?eligibles ?partic (?nonpartic
  ?partic)?Probnonpart
• bias ??p
• It depends on 
• relative size of eligible non-participant group
• whether it is very different from RA group

23

• Descriptive Analysis
• Extent of non-participation
• How different
• observable characteristics
• outcomes (non-participants vs. controls)
• Non-Experimental Analysis

24
Issue 2

• Does ERA enhance
• hourly wages (productivity)
• wage growth (advancement)
• employment duration (retention)
• Cannot use experimental contrast due to
  post-randomisation selection bias into employment.

?
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
To conclude

• Since experiments can answer only a subset of
  the questions of interest to the evaluator, it
  remains important to build up the stock of basic
  social science knowledge required to successfully
  utilize non-experimental methods, both in
  themselves and as tools for more extensive
  analyses of experimental data.
• Heckman and Smith (1995, p.95)