Introduction to the Design and Analysis of Trials can be found on:

1
Before and After Studies A Reminder

• Introduction to the Design and Analysis of Trials
  can be found on
• http//www-users.york.ac.uk/djt6/

2
Background

• Many researchers (?) use before and after studies
  they are, of course, nearly completely useless.
• Why? This is because of
• Regression to the mean
• Temporal changes

3
Which Researchers (?) use before and after?

• Clinicians, teachers assessing individuals.
• Action researchers.
• Audit.

4
Temporal Change

• Things change, people get better, policy changes
  all of which may make a difference.
• A before and after study CANNOT possibly cope
  with these temporal events.

5
Regression to the Mean

• Is a group phenomenon applies when we measure a
  group of people and re-measure them.
• Those with values below or above the mean will
  tend to regress back towards the mean on
  re-measurement.

6
Before and after treatment for neck pain
Improvement highly significant p lt 0.0001
7
Plot of difference scores

• A symptom of regression to the mean is if you
  plot change scores (baseline follow up) against
  baseline scores. A correlation indicates RTM.
• Thus, those with the lowest baseline improve the
  most and those with the highest improve the least.

8
Scatterplot showing RTM
Correlation of Change Score with baseline values
0.33 p lt 0.0001
9
Some benefit of vaccination is due to regression
to mean
10
Meningitis

• After vaccination new cases of meningitis fell
  from about 240 to 35 an 85 decrease. HOWEVER,
  of the 205 cases that were prevented the
  majority 120 were due to regression to the mean
  effects ONLY 41 were probably due to the
  efficacy of the vaccine.

11
Education intervention

• Wheldall selected 40 pupils whose reading was at
  least 2 years behind their peers.
• Half were exposed to an intervention.

Wheldall Educational Review 20005229.
12
Before and after reading programme
Difference highly statistically significant p lt
0.001
13
Before and after reading programme
Differences between groups NOT statistically
significant
14
RTM misunderstanding

• the mean gain scores translated to impressive
  effect sizes of 0.6.
• It could be argued that it is asking too much of
  any program to demonstrate enhanced efficacy on
  top of such high existing efficacy
• control group gains were largely attributable
  to pre-existing literacy programme..
• Perhaps, BUT much of the gain will be due to RTM.

15
Evaluation of School intervention

• A secondary school routinely offered children who
  scored badly on a reading test an ICT
  intervention.
• This was shown to improve childrens literacy.

16
ICT and Reading
17
Did it work?

• Impossible to tell. Regression to the mean and
  temporal effects does not allow us to find this
  out.
• Fortunately, we are doing a RCT of ICT and
  reading.

18
RTM and Policy Decisions

• Government policy targets 10 worst areas for
  street crime. 1 year later 17 fall in crime
  some or all due to RTM.
• 40 increase in gun crime results in a months
  amnesty for fire arms will probably work
  through RTM.

19
Annual Increase in offences with firearms
Amnesty
20
Exam marking

• In MSc double blind marking. Two markers
  disagree at the extremes of the distribution.
• We might fool ourselves that one marker is hard
  and the other a softie but really it is RTM.

21
RTM and exam scripts
22
Policy Changes

• Regression to the mean is an excellent method of
  proving something works
• Failing schools or hospitals can have an
  expensive management change and there is a good
  chance that regression to the mean will do the
  job.

23
Proving Effective Treatments

• RTM is an excellent phenomenon to prove to
  doubting clinicians the value of a new treatment.
• Choose an outcome measure with a high variance
  (e.g., single BP measure, FEV). Identify
  patients with extreme values (preferably only
  measured once), treat and re-measure. The group
  mean ought to decline (not all patients will
  improve but most will).

24
Dealing with RTM

• Sequential measurements taking an average (e.g.,
  3 BP measurements averaged out) will reduce the
  problem.
• The only way to reliably deal with the problem is
  through randomised trials.
• Which is why before and after data are generally
  regarded as almost USELESS.

25
Ceiling and Floor Effects

• As well as RTM before and after studies are
  blighted by ceiling and floor problems.
• Often measurement instruments have a floor (e.g.,
  0) or a ceiling (e.g., 100), which means if
  someones value is close to either of these
  extremes they cannot change much except towards
  the mean.

26
League Tables

• Classic problem of RTM with ceiling and floor
  effects. For example, schools that get close to
  100 5 GCSEs cannot do any better, whereas
  schools with very low levels can only go upwards.
  This phenomenon is skillfully exploited by
  politicians to show an effect. Similarly with
  hospital league tables.
• Same problem applies to quality of life measures.
  EuroQol for example, has ceiling problems.

27
Summary

• Before and after studies are the weakest
  evaluative method of proving something does or
  does not work.
• To control for temporal changes and regression to
  the mean controlled trials are required.

28
Conclusion

• You can prove virtually any crackpot theory
  using RTM.
• NEED a control group.