BIAS: threats to validity and interpretation

1
BIAS threats to validity and interpretation

• Bias is the result of systematic error in the
  design or conduct of a study a tendency toward
  erroneous results
• Systematic error results from flaws in either the
  (1) method of selection of study participants, or
• (2) in the procedures for gathering relevant
  exposure and/or disease information
• Hence - the observed study results will tend to
  be different from the true results

systematic error is different from error due to
random variability (sampling error)
2
Main points to be covered

• What is selection bias?
• in a case-control study
• in a cohort study
• in a clinical trial
• Avoiding and detecting selection bias
• What is information bias?
• Examples of information bias
• recall bias,
• interviewer or observer bias,
• surveillance bias,
• Differential and non-differential
  misclassification
• How to prevent or minimize information bias
• Information bias in a case-control study (recall
  bias)

3
Selection bias

• Selection Bias is present when individuals have
  different probabilities of being included in the
  study according to relevant study
  characteristics namely the exposure and the
  outcome of interest
• Selection bias anytime there is systematic error
  in the selection of study subjects cases or
  controls in a case-control study, exposed or
  unexposed in a cohort study

4
REFERENCE POPULATION
Diseased
-
-
Exposed
STUDY SAMPLE
Figure 4.2
5
Example of bias in a case-control study Coffee
and cancer of the pancreas MacMahon et al (N
Eng J Med 1981 304630-3)
Cases patients with histologic diagnoses of
cancer of the exocrine pancreas who were in any
of 11 large hospitals in the Boston metropolitan
area and Rhode Island between October 1974 and
August 1979 Controls Other patients who were
under the care of the same physician in the same
hospital at the time of an interview with a
patient with pancreatic cancer. Patients with
diseases known to be associated with smoking or
alcohol consumption were excluded with diseases
of the pancreas or hepatobiliary tract,
cardiovascular disease,diabetes mellitus,
respiratory or bladder cancer, and peptic ulcer.
Patients with gastroenterologic conditions were
probably overepresented in relation to a general
hospital population Describe the cases - What
study base gave rise to these CASES? What would
be a good population to get controls from?
6
Coffee and cancer of the pancreas MacMahon et
al, (N Eng J Med 1981 304630-3)
Males Case Control
482 41
Coffee No coffee
216 307
OR 2.7 (1.2-6.5)
7
Selection bias case-control studies

• In case-control studies, selection bias may arise
  if the cases and controls do not represent all
  the individuals in the population being studied,
  ie. the study base.
• Selection bias is particularly likely when
  dealing with a secondary study base ie. when the
  cases are defined first.
• In the previous case, the lower odds of exposure
  (coffee intake) in controls led to a spurious
  positive association between coffee intake and
  pancreatic cancer.

8
Coffee and pancreatic cancer chapter 2 Hsieh et
al, (N Eng J Med 1986 315 587-588)
All Case Control
Coffee No coffee
370 70
170 270
OR 1.5 (0.8-2.6)
9
Selection bias cohort studies

• In cohort studies, the most common form of
  selection bias occurs with loss to follow up.
• When those lost to follow-up are more or less
  likely to include exposed and affected
  individuals - differential loss to follow-up
  probability of the outcome is different in those
  who remain in cohort vs. those who leave (due to
  refusal, migration, jail, ...)
• Eg In a study following adolescents at risk for
  HIV, those who sell drugs may be more likely to
  be incarcerated. If selling drugs is associated
  with a more needle sharing, a biased association
  would result for needle sharing and HIV
  incidence.

10
Selection bias cohort studies

• Selection bias is less likely to occur in cohort
  studies (compared to case-control) since study
  participants (exposed or unexposed) are selected
  (theoretically) before the disease occurs
• Selection bias can occur on the front-end of
  the cohort if diseased individuals are mistakenly
  entered into the cohort
• Eg In a study of physically active men a
  positive association was found for exercise
  exposure and all-cause mortality. If study
  participants were enrolled who had undiagnosed
  cardiovascular disease were more likely to
  exercise less, what would the effect be on the
  measure of association?

11
Selection bias cohort studies

• Results obtained from studies comparing survival
  among HIV infected IDU and homosexual men were
  not consistent. One study showed that
  progression rates in cohorts of IDU were slower
  than those in cohorts of homosexual men after
  adjusting for for potential confounders such as
  age at seroconversion. (Multicohort analysis
  Project Workshop. Part 1. AIDS 1994, 8911-921).
  Others found no such difference.
• A subsequent study found that IDU had a high
  pre-AIDS mortality rate .. ...In contrast,
  pre-AIDS mortality was much higher in IDU than
  homosexual men... ...Withdrawal differed
  between the two risk groups. Therefore the
  inclusion or exclusion of losses to follow-up and
  pre-AIDS deaths in the denominator altered
  results. Pre-AIDs mortality may not be
  independent of progression and non-progression.
  (Prins et al, 1997, AIDS 1997, 11621-631).
• Prins et al, further showed that pre-AIDS
  death from natural causes was found to be related
  with ongoing time since HIV seroconversion and
  immunosuppression (Prins et al, AIDS 1997,
  111747-1756).

12
Selection bias clinical trial

• The potential for selection bias in clinical
  trials is in losses to follow-up
• Consider - a drug that causes a symptomatic side
  effect that frequently results in discontinuation
  of the study medication or drop-out.
• What would happen if you discontinued following
  them?
• What will the effect be on the study findings if
  the side effect is associated with the main
  outcome?

13
Example of possible bias due to losses to
follow-up in a clinical Chestnut CH et al, A
randomized trial of nasal spray salmon calcitonin
in postmenopausal women with established
osteioporosis the PROOF study. Am J. Med 2000
109267-276. Purpose a 5-year, double-blind,
randomized, placebo-controlled study to determine
whether salmon calcitonin (SC) nasal spray
reduced the risk of vertebral fractures.
Subjects and methods 1,255 post-menopausal
women with established osteoporosis were randomly
assigned to receive the SC nasal spray. Vertebral
fracture assessed with lateral radiographs.
Primary efficacy endpoint was the risk of new
vertebral fractures in the 200 IU SC nasal spray
group compared to placebo group. Results During
5 years, 1,108 participants had at least one
follow-up radiograph. A total of 783 women
completed 2 years of treatment, and 511 completed
5 years. The 200 IU SC nasal spray dose
significantly reduced the risk of new vertebral
fractures by 33 compared with placebo 200 IU
51 of 287, placebo 70 of 270, RR0.67, 95 CI,
0.47-0.97, p0.03. In the 817 women with one to
five prevalent vertebral fractures at enrollment,
the risk was reduced by 36 (RR-0.64, 95
CI0.43-0.96, p0.03) Conclusion SC nasal spray
at a dose of 200 IU daily significantly reduces
the risk of new vertebral fractures in
postmenopausal women with osteoporosis.
14
Results 59 of participants withdrew from the
study prematurely. Rates of discontinuation were
similar in all dosage groups. Cummings
Chapurlat What PROOF proves about calcitonin and
clinical trials (Editorial). Am J Med 2000
109330-331. There was a 36 reduction in risk of
vertebral deformities in the group that received
20- IU a day, but no significant effects were
seen with a higher dose. There wa not consistent
reduction in the risk of other types of fractures
across doses. The authors state that women who
were lost from the placebo group were similar in
the most easily measurable respects to those who
were lost form the calcitonin groups. When so
many participants fail to finish a trial, ,
however, readers rightly wonder whether this
really retains the valididy of a randomized
trial. The authors used an intention to treat
analysisBut the participants who were lost were
not followed up, and the investigators have not
information about the treatments of outcomes of
more than half the women who were originally
enrolled It was not known if fractures had
occurred in these participants. Because the
overall number of fractures was small, even a few
fractures in the participants lost to follow-up
could have altered findings of the trial.
15
Avoiding and detecting selection bias

• In case-control studies, Choose controls from the
  same study-base as cases.
• In cohort studies, the rate of loss to follow-up
  indicates the potential for selection bias.
  Comparison of the characteristics of those lost
  to follow-up with those persons remaining under
  follow-up, may indicate the potential
  consequences of any selection bias.

16
Information Bias

• Information Bias results from a systematic error
  in measurement thus leading to misclassification
  (in exposure or outcome category).
• A classic example is recall bias, in which the
  ability to recall past exposure is dependent on
  case or control status. Cases may be more likely
  than controls to overstate past exposure

17
Misclassification of EXPOSURE
REFERENCE POPULATION
Diseased
The direction of the association is a function of
which cell(s) are subjected to a higher or lower
probability
-
-
Exposed
Cases Control
STUDY SAMPLE
Eg...unexposed cases in this example tend to
mistakenly report past exposure to a
greater extent than do controls
18
Misclassification of OUTCOME
REFERENCE POPULATION
Diseased
-
-
Exposed
Cases Control
STUDY SAMPLE
Egcases in this are mistakenly classified as
controls due to low sensitivity on a screening
test
19
Information Bias

• These errors result in misclassification of
  exposure and/or outcome status
• Terms validity, sensitivity, specificity and
  reliability refer to classification of both
  disease and exposure status (and confounders)

20
Definition of Terms Related to Classification

• Validity the extent to which a measurement
  measures what it purports to measure.
• Sensitivity the ability of a test to identify
  correctly those who have the disease (or
  characteristic) of interest.
• Specificity the ability of a test to identify
  correctly those who do not have the disease (or
  characteristic) of interest
• Reliability (repeatability) the extent to which
  the results obtained by a test are replicated in
  the test is repeated.

21
1. Exposure Identification Bias

• Problems in the collection of exposure data
• 2 main examples
• Recall bias
• Interviewer bias

22
1.1 Recall Bias

• Most cited inaccurate recall of past exposure
  (may be due to temporality, social desirability
  or diagnosis).
• If recall differs between cases and controls,
  misclassification is differential
• If the error is of similar magnitude, then it is
  said to be non-differential
• Example study of association between hair color
  and tanning ability and melanoma
• Weinstock et al (Recall (report) bias and
  reliability in the retrospective assessment of
  melanoma risk AJE 1991 133240-245.

23
How to Prevent Recall Bias

• Verification of exposure information from
  participants by review of pre-existing records
• Objective markers of exposure or susceptibility
  (for example- genetic markers).
• A disadvantage of some biologic markers is that
  they assess current, rather than past exposure.
  Eg. Cotinine as a marker of cigarette smoking.
• Nested case-control studies allow evaluation of
  exposures prior to case status

24
1.2 Interviewer Bias

• May occur when interviewers are not blinded to
  disease status.
• They may probe more
• Interviewers may be biased toward the study
  hypothesis (or have other biases).
• They may ignore protocols
• Prevent or assess with reliability/validity
  substudies (phantom studies)
• eg. Doll and Hill study of lung cancer and smoking

25
2. Outcome Identification Bias

• May occur in both case-control or cohort studies
• Problems in the collection of outcome
  measurements.
• Two main examples
• Observer bias
• Respondent bias

26
2.1 Observer Bias

• In a Cohort study decision to classify outcome
  may be affected by knowledge of exposure status.
  Especially soft outcomes such as migraine, or
  psychiatric symptoms
• Eg assignment of diagnosis of hypertensive
  end-stage renal disease (ESRD). Nephrologists
  sent case histories were twice as likely to
  diagnose black patients with ESRD than white
  patients.

27
Preventing Observer Bias

• Mask observers in charge of classifying outcome
  with respect to exposure status
• Multiple observers

28
2.2 Respondent Bias

• Most often examples are in case-control studies
  where biases are associated with identification
  of exposure.
• Eg Parents may be more likely to identify having
  given their children aspirin in a case-control
  study of Reyes syndrome compared to population
  controls

29
2.2 Respondent Bias

• In a Cohort study respondents may respond with
  little consistency to un-standardized questions
  or to subjective questions.
• Eg. Questions about depression may be very
  subjective. A solution is to use a standardized
  instrument.
• Eg. 2 Have you witnessed an overdose?

30
3. The result of information bias
Misclassification

• Nondifferential misclassification
• Differential misclassification

31
3.1 Nondifferential Misclassification

• Occurs when the degree of misclassification of
  exposure is independent of case-control status
  (vice-versa)
• Example misclassification of HCV infection due
  to window period in a study looking at risk
  factors for HCV. Both the exposed and the
  unexposed are equally likely to be misclassified
  as unexposed.

32
Misclassification of OUTCOME
REFERENCE POPULATION
Diseased
-
-
Exposed
Cases Control
STUDY SAMPLE
Egcases in this are mistakenly classified as
controls due to low sensitivity on a screening
test
33
Nondifferential Misclassification

• Example misclassification of exposed subjects
  as unexposed in 30 of cases and 30 of controls

No misclassification Exposure Cases Controls Yes
50 20 No 50 80 OR (50/50)/(20/80) or
(5080)/(5020) 4.0 30 Exposure
misclassification in each group Exposure Cases Co
ntrols Yes 50-1535 20-614 No 501565 80686
OR (3586)/(6514) 3.3 Effect of
non-differential misclassification with 2
exposure categories to bias the OR toward the
null value of 1.0
34
Application of sensitivity/specificity concepts
in misclassification of exposure schematic
representation of true and misclassified relative
odds. Sensitivity of exposure ascertainment
TP ? (TP FN)Specificity of exposure
ascertainment TN ?(TN FP)
Cases Controls True Exp
Unexp Exp Unexp OR
(pos) (neg) (pos) (neg) True
results A C B C
A/C? B/D Study
total study total study
MIS- Results
cases
controls classified OR Exp TP FP
TP FP a TP FP TP
FP b Unexp FN TN FN TN c
FN TN FN TN d a/c?b/d
Figure 4-4
35
Effects of nondifferential misclassification on
the odds ratio--gtSince the sensitivity and
specificity values are the same for cases and
control,the effect is nondifferential
Cases Controls
True exp
unexp exp unexp
OR true distribution (gold
standard) 50 50
20 80
50/50?20/80 4.0 Study distribution
Cases
Controls Exposed 45 10
55 18 16
34 Unexposed 5 40
45 2 64 66
55/45?34/66 2.4 sensitivity (exp) or
specificity (unexp) 0.90 0.80
0.90 0.80
Misclassified OR
Exhibit 4-3
36
Nondifferential misclassification of exposure
effects of sensitivity and specificity of
exposure identification and of exposure
prevalence in controls on a studys OR (true
OR4.0)
Table 4-5
37
3.2 Differential Misclassification

• Occurs when the degree of misclassification of
  exposure (outcome) differs between the groups
  being outcome (exposure) groups
• Effect is bias toward or away from the null

38
Effect of differential misclassification on the
OR, in which, for Sensitivity Cases gt Controls
and, for specificity, Cases Controls
Cases Controls
True exp
unexp exp unexp
OR true distribution (gold
standard) 50 50 20
80 50/50?20/80
4.0 Study distribution Cases
Controls Exposed 48
0 48 14 0 14
Unexposed 2 50 52
6 80 86
48/52?14/86 5.7 sensitivity (exp) or
specificity (unexp) 0.96 1.00
0.70 1.00
Misclassified OR
Exhibit 4-5
39
Effect of differential misclassification on the
OR, in which, for both Sensitivity and
Specificity, CasesgtControls
Cases Controls
True exp
unexp exp unexp
OR true distribution (gold
standard) 50 50 20
80 50/50?20/80
4.0 Study distribution Cases
Controls Exposed 48
0 48 14 16 30
Unexposed 2 50 52
6 64 70
48/52?30/70 2.1 sensitivity (exp) or
specificity (unexp) 0.96 1.00
0.70 0.80
Misclassified OR
Exhibit 4-6
40
Example Weinstock et als study of melanoma
associated with hair color and tanning
ability. Table 4-8

• Nested case-control study allowed data collection
  at baseline and during follow-up
• Participants were compared with regard to their
  report of hair color and tanning ability
• Compared with pre-disease development, the odds
  for hair color use increased slightly only for
  controls when the postmelanoma interview data
  were used Result OR changed very little.
• Differential misclassification of tanning ability
  was severe, leading to a reversal in the
  direction of the association.

41
Cases 143 without a prior history of cancer who
reported a melanoma (June 1976 to June 1984)
responded to mailed questionnaire or telephone
interview. Controls 316 age-matched controls
randomly sampled from the NHS cohort members
w/out history of cancer
42
(No Transcript)