Making Sense of Multiple Studies

1
Making Sense of Multiple Studies

• Causal Analysis of Multiple Studies

2
MULTIPLE TESTING

• Whenever we use multiple hypothesis tests to
  examine the same basic biological or medical
  question, or to search for relationships among
  many variables ("fishing expedition"), the
  results must be interpreted with great caution

3
WHY?

• Because the chance of finding some statistically
  significant relationship increases with the
  number of tests we make, even when nothing
  whatsoever is really going on biologically

4
Cont.

• This chance is sometimes called the "class-wise"
  error rate when applied to a specific class (or
  group) of tests, and the "experiment-wise" error
  rate when applied to all the hypothesis tests
  used in a particular study. The experiment-wise
  error rate may be much, much, much, much higher
  than the "nominal" error rate a of any single one
  of the tests.

5
(No Transcript)
6
EXAMPLE CASE-CONTROL TRAWLING.

• You run a case-control study to try to find out
  the cause of some disease. Not knowing the cause,
  you ask the patients and controls about each of
  100 exposures that might have been involved. If
  none of these exposures really had anything to do
  with generating the disease, how many would you
  expect to look suspicious (i.E. Statistically
  significant exposure odds-ratio)?

7
EXAMPLE OVERDOSING ON HYPOTHESIS TESTS.

• To evaluate the effectiveness of a new
  pharmacologic agent and to find the optimal dose,
  you randomly assign patients to placebo or 100,
  200, 300 or 400 mg of the drug daily. At the end
  of the study, you separately test whether each of
  the doses was followed by a better results than
  the placebo, using a5. If the drug doesn't work
  at all, what's the chance that some dose will
  appear to work?

8
EXAMPLE MULTIPHASIC HEALTH SCREENING.

• Routinely, as part of the office physical you
  give to every new patient and periodically to
  continuing patients, you draw a blood sample and
  send it to the lab for a smac-20. The report
  labels every result outside the middle 95 of
  values of healthy people as "abnormal." How
  likely is a perfectly healthy patient to produce
  a perfectly clean lab report?

9
EXAMPLE MULTIPLE OUTCOMES.

• In a comparative trial of two surgical
  procedures, you define 10 outcome variables to
  describe the results of these procedures. Some
  describe basic critical results, e.G. Life or
  death and major surgical complications, while
  others describe several aspects of quality of
  life after surgery. If the average results of the
  two procedures were identical on all scores,
  what's the chance you'd see a spurious difference
  anyway?

10
EXAMPLE MULTIPLE LOOKS AT CLINICAL TRIALS

• In comparing two therapies over time, you
  compare the patient's current status via an
  hypothesis test every three months for ten years.
  What's the chance you'll see no difference over
  all that time?

11
CRITERIA OF CAUSALITY

• Strength of association
• Consistency
• Temporality
• Dose response
• Reversibility
• Biologic plausibility
• Specificity
• Analogy

12
STRENGTH OF ASSOCIATION

• The larger the relative risk, the stronger the
  evidence for causality however, a small relative
  risk (lt2) does not rule out cause and effect.
• CONSISTENCYRepeatedly observed in different
  studies, different populations, different times
  and places, different study designs

13
TEMPORALITY

• Cause precedes effect problem outcome (at a
  sub-clinical level) may cause change in exposure
  e.g. 1. Heart disease and exercise 2. Peptic
  disease and spicy food 3. Lung cancer and recent
  smoking cessation

14
DOSE-RESPONSE

• Larger exposures associated with higher rates of
  disease. Problems 1. Confounder can explain
  dose-response relationship 2. Threshold
  phenomenon higher level exposure (beyond a
  threshold level) does not increase outcome rate
  any further.

15
REVERSIBILITY

• Reversing the exposure is associated with lower
  rates of disease. (E.G. Smoking cessation). Can a
  confounder explain this one?
• BIOLOGIC PLAUSIBILITY Can a biologic mechanism
  explain how the exposure causes the effect? But
  beware biologic plausibility is always absent
  for new discoveries

16
SPECIFICITY

• One cause leads to one effect
• ANALOGY Cause and effect relationship already
  established for a similar exposure-disease
  combination
• THE BETTER THE STUDY DESIGN THE STRONGER THE
  EVIDENCE

17
HIERARCHY OF RESEARCH DESIGN IN ESTABLISHING CAUSE

• 1. CASE REPORT
• 2. CASE SERIES
• 3. CASE-CONTROL
• 4. COHORT STUDY
• 5. CLINICAL TRIAL

18
METANALYSIS

• CRITERION-BASED
• POOLING
• CRITERION-BASED
• EXAMPLE BCG

19
(No Transcript)
20
(No Transcript)
21
POOLING

• Combining the patients in all trials and
  reanalyzing the data, as if they had come from a
  single large but stratified study

22
EXAMPLES

• Quinidine for atrial fibrillation
• IV streptokinase for acute MI

23
(No Transcript)
24
(No Transcript)
25
Pooling cont.

• Criteria for the study, design
• criteria for patients.
• both included.
• Used mostly for randomized studies.
• Each individual study may have little power, but
  the combined analysis has a much higher power
  (smaller confidence interval).

26
Metanalysis Cont.

• Sometimes the individual studies had little power
  because they were addressing a different (more
  common) end point.
• used to plan sample size for future studies.

27
PROBLEMS WITH METAANALYSIS

• Remember metaanalysis deals only with "chance" by
  increasing power and decreasing confidence
  interval widths. It does not correct for bias and
  confounding.
• Patients, treatment, outcome evaluation may
  differ across studies. What then?

28
Publication bias

• One potential problem with metaanalysis is
  publication bias That studies with positive
  findings are more likely to have been published,
  and hence be included in the metaanalysis than
  negative studies.

29
(No Transcript)
30
Funnel Plot
31
WHY READ THE JOURNALS

• Other sources of clinical guidelines
• Expert opinion
• Textbooks
• National bodies
• Editorials

32
Embargo

• Why?
• How is it done?
• The WHI study

33
SPAF Trial

• Stroke Prevention in Atrial Fibrillation
• 627 patients were randomized to warfarin,
  aspirin, or placebo
• Warfarin was superior.

34
Average Clinicians Conclusion

• Every patient with atrial fibrillation who
  doesnt have any of the exclusion criteria should
  receive warfarin.
• You would reach that conclusion if you read the
  articles abstract and conclusion.

35
My Conclusion

• Not every patient with atrial fibrillation should
  be started on warfarin.
• My reason?
• I read the whole article.
• What did I find?

36
They Started with 18,376 Patients!

• Thats not bad by itself.
• Most were excluded for appropriate criteria
• BUT among the excluded
• 717 refused
• Another 1,084 their doctors refused
• Another 2,262 no reason was recorded
• Another 239 patient or doctor refused
  anticoagulation. (So they were randomized into
  the aspirin versus placebo trial.)

37
Compare These Numbers

• 717 refused
• Another 1,084 their doctors refused
• Another 2,262 no reason was recorded
• Another 239 patient or doctor refused
  anticoagulation. (So they were randomized into
  the aspirin versus placebo trial.)
• To 627 randomized (210 ended on warfarin)

38
(No Transcript)
39
Why not point out that at 12 mcg/L PSA IS 100
specific

• Misses early disease.
• Subjects an occasional patient to endless
  invasive testing.
• You are responsible for all of those.
• When proven wrong gives competitors an automatic
  publication at your expense.

40
HOW TO READ THE JOURNALS

• Read the conclusion
• If valid, would this be useful or interesting
  (common disease, new finding)?

41
Types of articles

• 1. THERAPY Randomization
• 2. DIAGNOSIS Independent blind comparison with
  a gold standard
• 3. CAUSE
• 4. COURSE/PROGNOSIS
• Inception
• Adequate follow up

42
Clinical trial, Look for

• 1. Design randomization
• 2. Hypothesis stated 3. Clinically
  significant end
  points 4.
  Sample size estimation 5. Subject selection,
  description of subjects 6. Exclusions why? How
  many

43
Look for cont.

• 7. Blinding, outcome criteria 8.
  Intervention, specific 9. Interim review,
  guidelines 10. Drop outs, intention to treat
  11. All clinically relevant outcomes 12. Few
  comparisons (data derived hypotheses)

44
Any study that

• Randomizes
• Patients similar to the ones I see
• And looks at all clinically relevant outcomes

45
Clinical Trials Jargon

• Consecutive patients (versus a random sample)
• Baseline characteristics of patients (to see if
  randomization worked)
• Number of subjects and average duration of
  follow-up (versus patient years)
• Interim analysis, problems
• Cumulative incidence (versus incidence density)

46
Jargon cont

• Relative risk (hopefully lt1)is rate of outcome
  in a drug group rate of outcome in a placebo
  group
• Relative risk reduction (similar to attributable
  risk ) (But here it is 1-RR)
• Absolute difference in risk (similar to AR, very
  important, sometimes not reported
• Relative risk reduction versus absolute
  difference in risk
• Number needed to treat

47
Jargon cont.

• Subgroup analyses (looking for effect modifiers)
• Problems with subgroup analyses (multiplicity,
  data-derived hypotheses) other methods of data
  torturing
• "Intention to treat" versus efficacy (hazards of
  non-randomized comparisons)

48
DIAGNOSIS

• SENSITIVITY Spectrum
• SPECIFICITY Similar conditions
• WHO GETS THE GOLD STANDARD?
• Is it dependent on the test?
• Is it blinded?

49
BAD ARTICLES

• Negative with no power
• Test with no evaluation of negatives

50
Review of Bias and Confounding
51
BIASES CASE-CONTROL STUDIES

• 1. SELECTION BIAS CASES Could the way you
  selected your cases be related to exposure?
  Diagnosis referral response availability
  survival

52
Biases cont.

• Controls if this person had the disease, would
  she have been a case?
• 2. Measurement (observation, information) bias
  subjects recall, lying interviewer

53
BIASES COHORT STUDIES

• 1. Selection bias randomization could the
  way you assigned the subjects (or they assigned
  themselves) (to exposed or unexposed) be related
  to outcome? Volunteer non participation
  compliance

54
2. MEASUREMENT BIAS

• Migration loss to follow up
  misclassification random differential

55
CONFOUNDING

• A confounder is 1. Associated with the exposure
  2. Statistically associated with outcome (risk
  factor) independent of the exposure 3. Not
  necessarily a cause of the outcome 4. Not a
  result of the exposure

56
CONTROL OF BIAS AND COUNFOUNDING

• In the design
• 1. Randomization 2. Restriction
  -disadvantages 3. Matching -as a concept
  -matched pairs -disadvantages
  -overmatching

57
In the design cont.

• Additional controls
• Include assessment of known risk factor
• Include data on all risk factors
• Reliability check

58
IN THE ANALYSIS

• 1. Stratification 2. Adjustment 3.
  Multivariate analysis