Sources of bias in RCTs

1
Sources of bias in RCTs

• With focus on blinding and concealment of
  allocation
• David Torgerson

2
Background

• Most reported educational (and health) trials are
  of POOR quality.
• They are too small and do not adequately report
  their methods.

3
All RCTs are NOT the same.

• Although the RCT is rightly regarded as the
  premier research method, by those who know, some
  trials are better than others.

4
Selection Bias - A reminder

• Selection bias is one of the main threats to the
  internal validity of an experiment.
• Selection bias occurs when participants are
  SELECTED for an intervention on the basis of a
  variable that is associated with outcome.
• Randomisation or other similar methods abolishes
  selection bias.

5
After Randomisation

• Once we have randomised participants we eliminate
  selection bias but the validity of the experiment
  can be threatened by other forms of bias, which
  we must guard against.

6
REMEMBER

• Whilst most RCTs are susceptible to a range of
  potential biases other study types (e.g., pre
  test post test, quasi experiments) are also
  susceptible to the same biases.

7
Blinding

• The gold-standard in health care trials is
  supposedly the double blind placebo controlled
  trial.
• It is helpful to blind some people some of the
  time but may not be always appropriate.

8
What does blinding achieve?

• If a participant, their doctor (teacher) and the
  researcher are unaware of treatment received.
  This prevents a number of potential biases
  including ascertainment and performance.

9
Ascertainment Bias

• This occurs when the person reporting the outcome
  can be biased.
• A particular problem when outcomes are not
  objective and there is uncertainty as to
  whether an event has occurred.
• For example, if the person marking an essay knows
  into what group the person belongs this might
  affect their mark.

10
Example.

• A group of students essays were randomly
  assigned photographs purporting to be the
  student. The photos were of people judged to be
  attractive average below average. The
  average mark was significantly HIGHER for the
  average looking student.
• Why? Markers were biased into marking higher for
  students whom they believed were average looking
  (like themselves).

11
Performance Bias

• Blinding, by sham or placebo, can reduce this
  bias BUT is it a bias?
• This is where the student because they are in a
  group they like performs better than in a group
  they dont like. It could be argued that if they
  like the intervention and perform better this is
  part of the intervention. Conversely, if the
  intervention is ineffective despite being liked
  better (e.g., computers) then it really is
  ineffective.

12
Preference effects

• When students (or patients) prefer an
  intervention this may bias an outcome. One way
  of attempting to control for this bias is to ask
  before randomisation the students preference and
  use this in the analysis.

13
Subversion Bias

• Subversion Bias occurs when a researcher or
  clinician manipulates participant recruitment
  such that groups formed at baseline are NOT
  equivalent.
• Anecdotal, or qualitative evidence (I.e gossip),
  suggest that this is a widespread phenomenon.
• Statistically this has been demonstrated as
  having occurred widely.

14
Subversion - qualitative evidence

• Schulz has described, anecdotally, a number of
  incidents of researchers subverting allocation by
  looking at sealed envelopes through x-ray lights.
• Researchers have confessed to breaking open
  filing cabinets to obtain the randomisation code.

Schulz JAMA 19952741456.
15
Subversion in a non health trial

• Boruch describes a trial in USA where incidents
  of domestic violence were being randomised to a
  caution or being seen in the police station.
• Some evidence that police officers made sure
  that offenders they knew got taken to the police
  station.
• This WILL damage the trial.

16
Subversion and Education

• I dont know of an educational trial that has
  been subverted BUT they almost certainly exist.
• Educational researchers are no different from
  health researchers and are just as likely to
  subvert their trials as we do!

17
Quantitative Evidence

• Trials with adequate concealed allocation show
  different effect sizes, which would not happen if
  allocation wasnt being subverted.
• Trials using simple randomisation are too
  equivalent for it to have occurred by chance.
• Often educational trials used paired
  randomisation but have UNEQUAL numbers in each
  group which means they lost one some where!

18
Poor concealment

• Schulz et al. Examined 250 RCTs and classified
  them into having adequate concealment (where
  subversion was difficult), unclear, or inadequate
  where subversion was able to take place.
• They found that badly concealed allocation led to
  increased effect sizes showing CHEATING by
  researchers.

19
Comparison of concealment
Schulz et al. JAMA 1995273408.
20
Recent Health Trials.
Hewitt et al. 2004 submitted
21
Case Study

• Subversion is rarely reported for individual
  studies.
• One study where it has been reported was for a
  large, multicentred surgical trial.
• Participants were being randomised to 5 centres
  using sealed envelopes.

22
Case-study (cont)

• After several hundred participants had been
  allocated the study statistician noticed that
  there was an imbalance in age.
• This age imbalance was occurring in 3 out of the
  5 centres.
• Independently 3 clinical researchers were
  subverting the allocation

23
Mean ages of groups
24
Example of Subversion
25
Using Telephone Allocation
26
Subversion - summary

• Appears to be widespread in health trials almost
  certainly occurs in psychology trials but remains
  undetected.
• Secure allocation usually prevents this form of
  bias.
• Need not be too expensive.
• Essential to prevent cheating.

27
Secure allocation

• Can be achieved using telephone allocation from a
  dedicated unit.
• Can be achieved using independent person to
  undertake allocation.

28
Attrition Bias

• Usually most trials lose participants after
  randomisation. This can cause bias, particularly
  if attrition differs between groups.
• If a treatment has side-effects this may make
  drop outs higher among the less well
  participants, which can make a treatment appear
  to be effective when it is not.

29
Attrition Bias

• We can avoid some of the problems with attrition
  bias by using Intention to Teach (or treat)
  Analysis, where we keep as many of the patients
  in the study as possible even if they are no long
  on treatment.

30
Dilution Bias

• This occurs when the intervention or control
  group get the opposite treatment.
• For example, in a trial of domestic violence the
  judge over-rode the random assignment and put
  3.5 (14) men in the intervention group.

Feder Dugan, Justice Quarterly, 200219343.
31
Resentful Demoralisation

• This can occur when participants are randomised
  to treatment they do not want.
• This may lead to them reporting outcomes badly in
  revenge.
• This can lead to bias.

32
Resentful Demoralisation

• One solution is to use a patient preference
  design where only participants who are
  indifferent to the treatment they receive are
  allocated.
• This should remove its effects.

33
Hawthorne Effect

• This is an effect that occurs by being part of
  the study rather than the treatment.
  Interventions that require more TLC than controls
  could show an effect due to the TLC than the drug
  or surgical procedure.
• TLC should be given to controls as well.

34
Forms of Bias

• Subversion Bias
• Technical Bias
• Attrition Bias
• Consent Bias
• Ascertainment Bias
• Dilution Bias
• Recruitment Bias

35
Bias (cont)

• Resentful demoralisation
• Delay Bias
• Chance Bias
• Hawthorne effect
• Analytical Bias.

36
Conclusions

• There are a range of biases associated with
  POORLY designed RCTs.
• Many, if not all, can be eliminated through
  careful attention to methods.