Introduction to Statistics

1
Introduction to Statistics

• Chris Mullin, Senior Statistical Consultant
• Integra Clinical Trial Solutions
• October 2005

2
Description and Objectives

• This talk will describe statistics as applied to
  clinical research, including the consequences of
  statistical ideas for non-statisticians
• Understand the problems of variability and bias
• Understand methods that are used to address these
  problems
• 3) Appreciate statistical issues in the analysis
  of clinical data

3
Talk Outline

• Variability
• Bias
• Randomization
• Blinding
• Intent-to-treat
• Statistical vs. clinical significance

4
Familiar Quotations 101

• There are three kinds of lies lies, damn lies,
  and statistics

Mark Twain(?)
Lying is easier without statistics
5
PlatoThe Allegory of The Cave

• Imagine people who have grown up inside a cave
• Only see shadows of the real world reflected on
  the cave walls
• What do they really know about the real world?
• Clinical trials are similar
• Research staff live in the cave of medical
  research
• Interested in what happens in the real world
  (non-research settings)?

6
Fundamental Problems

• Variability
• The spread of the darts relative to each other
• Bias
• The location of the group of darts relative to
  the bulls eye

7
Variability

• Toss a tack into the air
• Does it land point up or down?
• Do this many times, obtain estimate of
  probability of the point landing up or down
• Variability due to complex physics

8
Patient Variability

• Provide a patient a new treatment
• Does patient improve or get worse?
• By how much?
• Variability due to complex interplay between
  patient history, genetics, adherence to
  treatment, etc.

9
Sampling Variability

• Interested in inferences about a large group
  (population)
• Perform experiment on a sample of observations
• Different samples have different properties, may
  lead to different outcomes

10
Variability andScientific Conclusions

• With simple phenomenon and strong effects easy to
  see, results often obvious
• If your experiment requires statistics, youve
  havent done a very good study
• Variability applies to the process of experiments
  and gathering evidence
• Do many experiments, occasionally you may come to
  an incorrect conclusion

11
Generic Experimental Process

• State hypothesis
• Null hypothesis (no effect)
• Alternative hypothesis (some meaningful effect)
• Gather data
• Analyze it
• Make conclusions

12
Errors in Conclusions

• Type I Error Rate
• Probability of rejecting the null when it is true
• Almost always set at 5 (a0.05)
• Type II Error Rate
• Probability of failing to reject a false Null
• Power (1 ß). Power is usually set to 80

13
Hypothesis Testingand Errors
14
Hypothesis Testingand Errors
15
Bias

• The systematically incorrect estimation of the
  treatment effect

16
Bias
Problem Estimate how far this serving machine
can throw a ball on a tennis court by taking the
average distance
17
Bias

• Suppose the arm of the launcher is set too low
  and some of the balls strike the net
• If we include, well underestimate the average
• If exclude, we will overestimate the average
• Increasing the number of balls does not help -
  the system is flawed

18
Bias in Clinical Trials

• Selection bias
• Subjects under study are different than those of
  interest
• Related to sampling variability
• Clinician/patient
• Actions that influence treatment or outcomes
• May be intentional or unintentional

19
Investigator bias

• Investigator thinks drug A is superior to drug B
• Puts less healthy patients on drug A
• If A and B have effect, A will probably look
  worse
• Why? Patients were sicker to begin with!

20
Investigator bias

• Two hypothetical anti-cholesterol drugs
• Each provides 20 decrease in mean total
  cholesterol
• After 6 months, Group A looks better, since they
  started off worse

21
Selection Bias

• Healthy vs. unhealthy patients
• Role of physician treatment preference
• Must be healthy/sick enough to be in trial
• Must be able to comply with treatment, and trial
  assessments (radiology scans, clinic visits)
• Adherence
• Trial participants tend to be more adherent to
  treatments
• How are patients in your trial different from
  those left out? Are you interested in those left
  out?

22
Protections Against Bias

• Randomization
• helps ensure the treatment groups are comparable
  at baseline (selection bias)
• Blinding
• deal with placebo effect
• prevent investigator bias

23
Randomization

• Simple form flip a coin
• Heads Treatment A
• Tails Treatment B
• Tends to produce equal groups
• Balance of all known and unknown risk factors
• Reduce investigator bias
• More complicated stratified/blocked
  randomization schemes, adaptive designs, unequal
  allocation ratios

24
Randomization

• What good are equal groups?
• The only difference is due to randomized
  assignment
• Any difference in outcome not due to chance must
  be due to the treatment
• Superiority over observational studies

25
Randomizationand Confounding

• Randomization balances all confounders
• Confounder a variable associated with outcome
  that might interfere with variable of interest
• Coffee and Cancer - observational study
• High coffee consumption associated with smoking
• Smoking causes cancer
• Looks like high coffee consumption is associated
  with cancer
• (Randomizing to high vs. low/no coffee would give
  equal numbers of smokers in each group)

26
Blinding

• Patient blinding
• Reduce role of adherence issues
• Clinician blinding
• Role of other treatments, dose modification
• Blind evaluation of endpoint status
• Example judgment of borderline cases of
  myocardial infarction

27
Problems with Blinding

• Blinding not always possible
• Especially with many devices
• Creates artificial situation
• Real world, both patients and clinicians know
  treatments
• Logistically difficult when possible
• How to maintain blind?

28
Problems with Blinding

• Blinding not always possible
• Can you make a placebo heart valve?
• Can you make a placebo pacemaker?
• Could used one that is turned off
• Still exposes patients to risk (surgery)

29
Problems with Blinding

• Creates artificial situation
• Placebo not used in real world
• Choice usually between experimental treatment and
  alternative treatment
• Knowing current treatment changes behavior
• Role of adjunct therapies

30
Problems with Blinding

• Logistically difficult when possible
• How to maintain blind?
• Does the placebo appear to be like the
  experimental treatment?
• Drugs even when melted down?
• Sometimes need to be able to quickly break blind
  in emergencies

31
Hypothesis Testing

• Many clinical trials use superiority testing
• Is treatment superior to placebo control?
• Null hypothesis treatment and placebo equivalent
• Alternative hypothesis treatment superior to
  control
• With sufficient evidence, reject null hypothesis
  and claim treatment effective
• Occasionally, placebo not ethical or feasible

32
Statistical Issues

• Non-inferiority trials
• Missing data
• Intent to treat
• Statistical vs. clinical significance

33
Non-inferiority testing

• Placebo control not ethical if already know
  standard treatment (SOC) better than placebo
• Hope to show new treatment not much worse than
  SOC

New Txt
Placebo
SOC
Bad outcome
Good outcome
Non-inferiority margin
34
Issues in Non-inferiority Testing

• How do you choose the non-inferiority margin?
• Smaller margin larger sample size
• Larger margin smaller sample, but you get
  closer to placebo
• FDA wants smaller margin, companies what larger
• Which standard of care do you use?
• What if your original trial (establishing SOC)
  really committed a type I error?

35
Missing data

• Ubiquitous in clinical trials
• Missing data isnt a lemon, its no lemons
• Cant make lemonade
• Missing data puts an upper limit on the quality
  of your inferences
• Lots of statistical methodology exists to deal
  with the problems
• None really solve the problem

36
Result of Losses to follow-up

• A high rate of losses can bias the outcome of a
  trial
• Bias can cause
• misestimation of the true treatment effect
• failure to achieve statistical significance
• finding that an inferior treatment is superior!

37
Effects of LossesOn Interpretability

• For a given calculated required sample size (n)
• Project loss rate of 15
• Inflate n by 15 (throw more darts)
• Actual observed n will hopefully be close to the
  calculated
• Problem
• While inflating sample size maintains power (low
  Type II error rate), it does not help with
  possible bias losses introduce
• You never can really know what bias exists

38
Effects of Losses On Interpretability Example 1

• Suppose interested in estimating 1-year mortality
  rate
• Follow 1000 patients for 12 months
• Suppose 50 patients die and 100 are
  lost-to-follow-up
• Possible candidate death rates

50/1000 5
150/1000 15
50/900 5.5
39
Effects of Losses On Interpretability Example 2

• Suppose interested in comparing two treatments
• Follow 1000 (two groups of 500) patients for 12
  months

40
Effects of Losses On Interpretability Example 2
41
Practical options

• Analyze only available data
• Last-observation carried forward
• Imputation
• Model the missing process
• Methods rely on untestable assumptions
• Some people wont be convinced by anything you do
• Take home message MINIMIZE MISSING DATA

42
Intent-to-Treat

• An analysis which includes all randomized
  patients in the groups to which they were
  randomly assigned, regardless of their compliance
  with entry criteria, the treatment they actually
  received, subsequent withdrawal from treatment,
  or deviation from protocol.
• Fisher et al., 1990, in Peace et al. Statistical
  Issues in Drug Development, Marcel Dekker.

43
Issues With Intent-to-Treat

• Preservation of Randomization
• groups made-up of patients who remain after an
  exclusion process may not be comparable
• Exclusions/drop-outs may be related to treatment
  and outcome
• We cant evaluate this
• Protocol violations during follow-up are included
  in the analysis
• These may be related to the treatment assignment
• A true treatment effect could be diluted or
  exaggerated

44
Alternative Approaches

• As-treated Data is collected and analyzed for
  each patient in accordance with what treatment
  was actually received
• Per-protocol Exclude from the analysis any
  subject who did not follow the study protocol.
  This is sometimes called a per-protocol
  analysis.

45
What to do?

• Careful monitoring to ensure protocol compliance
• Simple study designs
• Analyze the data multiple ways
• Hope for consistent results
• If results are not consistent, further analysis
  is needed to determine why
• Poor compliance is an outcome

46
Importance of Intent-to-Treat

• Compare two diets, Atkins vs. Mullin
• Atkins low carbohydrates
• Mullin sauerkraut and beets
• 10 people on each diet, follow for 1 month

47
Can it work vs.does it work?

• Can it work
• Under ideal conditions, with ideal patients, does
  the treatment work?
• Does it work?
• With real patients, in real clinics, does the
  treatment work?
• Choice has consequences for
• Whether to randomize
• Blinding
• Inclusion/exclusion criteria

48
Statistical Significance

• Is the observed effect due to chance?
• Strength of evidence measured by p-values
• Small p-values (usually lt0.05) are called
  statistically significant
• Larger trials have more power, a better chance of
  claiming an effect is statistically significant
• (Note that your effect may still be biased!
• P-values dont capture this)
• Does the observed effect matter?

49
Statistical Significance

• P-values are calculated from test statistics.
  These are generally in the form of
• (effect estimate) x v(sample size)
• (variability estimate)

• Larger test statistic leads to smaller p-value
• Larger effect or sample size leads to smaller
  p-value
• Larger variability leads to larger p-value

50
Statistical vs. Practical (Clinical) Importance

• Example
• Hypothetical study of new pacemaker
• 1 to 2 change Quality of Life (QoL) score is not
  important
• 10 to 20 change is important for individuals
• Consider six hypothetical trials
• Results are mean change and 95 confidence
  intervals (CI)
• If 0 not in CI, then the change is statistically
  significant
• Does the device change mean QoL score?

51
Statistical vs. Practical (Clinical) Importance
Statistical Sig?
Clinical Sig?
mean effect (95 CI)
Yes
No
Study 1 2 (1,3)
Yes
Yes
Study 2 30 (20,40)
Yes
???
Study 3 30 (2,58)
Study 4 1 (-1,3)
No
No
Study 5 2 (-58,62)
No
???
Study 6 30 (-2,62)
No
???
52
The Cardiac Arrhythmia Suppression Trial

• Early 80's - arrhythmia associated with increased
  risk of cardiac death
• Anti-arrhythmias used for treatment
• Effective at suppressing arrhythmia
• Studies of survival benefit inconclusive
• maybe they were too small? not long enough?
• Question can it be proved that these treatments
  work?

53
The Cardiac Arrhythmia Suppression Trial

• Critical view There is no need to do the study!
• Drugs associated with reduction of arrhythmias
• Arrhythmias associated with death
• Therefore drugs will reduce death rate
• Scientific counterpoint We need the study!
• association does not mean causation
• maybe treatment has other side effects

54
The Cardiac Arrhythmia Suppression Trial

• Study patients with
• asymptomatic/mildly symptomatic ventricular
  arrhythmias
• prior myocardial infarction
• Compare 3 different drugs (and their placebo)
• Look for reduced rate of death due to arrhythmias
• Study was stopped after only two years of
  enrollment (n1500)
• ???

55
The Cardiac Arrhythmia Suppression Trial

• Why?
• Two of the drugs were likely harmful
• 63 on active drug died
• only 26 on placebo died!
• Treatments did not decrease death
• They actually increased it!!
• Untested therapies require proof
• Ref Echt et al, NEJM, Mar 21 1991, Vol 324, 12

56

• Chris Mullin
• Senior Statistical Consultant
• Integra Clinical Trial Solutions
• 763-746-8880
• October 2005