Experimental Design of Observational Studies:The Role of the Epidemiologist

1
Experimental Design of Observational Studies The
Role of the Epidemiologist

• 1 October 2003
• Viktor E. Bovbjerg, PhD MPH
• Department of Health Evaluation Sciences

Life is short, science is long opportunity is
elusive, experiment is dangerous, judgment is
difficultHippocrates
2
real creativity in medicine lies in the hands of
the world community of innovative investigators
who together carry out hundreds of thousands of
small research studies confirmatory large
randomised controlled trials become the task of
burnt-out leaders who are only fit enough for
administration and organization.Lancet
editorial, May 12, 1990
3
Study designs
4
(Some) roles of descriptive, observational studies

• Describing patient, at risk populations, health
  care use
• Documenting natural history of disease, treatment
• Identifying etiological factors in disease
  development, outcomes
• Evaluating therapy in practice, post-marketing
  surveillance
• Setting the stage for intervention trials

5
Descriptive studies

• Used to describe disease patterns and generate
  hypotheses
• extremely limited in ability to shed light on
  causes no attempt to test causal hypotheses
• Often the first step in learning about disease
  etiology, patient outcomes
• Relatively easy, rapid, and inexpensive
• Use diverse, often existing, sources of data

6
National Center for Health Statistics
7
Cohort studies
Crucial comparison outcomes in persons with vs.
without predictor variable (or with different
levels of predictor)

8
Relative merits cohort studies

• Advantages
• Clear temporal relationship
• Least susceptible to some forms of bias
• Can examine multiple predictors of outcome
• Efficient for rare exposures
• Useful when RCT infeasible, unethical

• Disadvantages
• No control over predictor (vs. RCT)
• confounding
• Inefficient for rare or long-latent diseases
• Loss to follow-up threatens validity
• Potential bias in outcome ascertainment
• Relatively resource- and time-intensive

9
Death of a drug, as told by CNN Bayer pulls key
drug (08-Aug-01) Bayer said it recalled
Baycol after it received reports that the drug
caused deterioration in muscle tissue, a
condition called rhabdomyolysis, which is known
to cause severe pain and potential kidney
failure. Baycol is in a family of drugs
called statins, which blocks an enzyme that
creates cholesterol. Other drugs in the family,
Mevacor, Pravachol, Zocor, Lescol, and Lipitor,
remain on the market Bayer faces more pressure
(12-Aug-01) Baycol linked to 52 deaths
(13-Aug-01) Bayer We'll survive
(20-Aug-01) Group wants tougher warning on
statins (20-Aug-01) Baycol pulled in Japan
(23-Aug-01) Roche denies Bayer offer (28-Aug-01)
Groups Statin benefits outweigh risks
(30-Aug-01) Bayer faces Baycol probe (04-Sep-01)
10
Case-control studies
Hypothetical study population
no investigator control
ascertainment
Controls
Cases
classification on outcome
previous exposure of interest
time
Comparison of exposure rates between cases and
controls provides estimate of effect (usually
odds ratio OR)
11
Relative merits case-control studies

• Advantages
• Efficient use of time, resources
• Efficient for rare outcomes
• Efficient for outcomes with long latency
• Can assess multiple exposures
• Best when cohort study infeasible, RCT unethical
  (e.g. harm)

• Disadvantages
• Inefficient for rare exposures
• Difficult to identify appropriate controls,
  identify study base
• Ascertaining previous exposure often difficult
• records, recall
• Potential bias in predictor ascertainment,
  confounding

12
Observational studies

• Goal obtain results that would have occurred if
  RCT had been conducted
• Observational studies must
• minimize random error (precision, statisical
  significance)like RCTs
• control mixing of effects (confounding)
• minimize systematic error in participation and
  data (bias)
• Extent to which confounding and bias are
  controlled determine credibility

13
Observational studies in support of experimental
design
14
In Summary

• Planning an experiment can and should be
  accomplished to control for type I and II errors.
  WE CAN EXPLORE ALL KINDS OF OPTIONS WITHOUT
  COST!!!
• For the purpose of
• Avoiding the waste of human and institutional
  resources (and your time).
• To improve funding opportunities.
• Critically evaluating the literature.

RD Abbott, 17 September 2003
15
Observational studies can

• Provide size of potential participant pool
• Provide description of potential study
  participants
• Provide estimates of effect/outcomes in various
  treatments, usual care

16
Why care?
Preliminary Studies. For new applications, use
this section to provide an account of the
principal investigator/program directors
preliminary studies pertinent to the application
information that will also help to establish the
experience and competence of the investigator to
pursue the proposed project. Peer review
committees generally view preliminary data as an
essential part of a research grant application.
Preliminary data often aid the reviewers in
assessing the likelihood of the success of the
proposed project. U.S. Department of Health and
Human Services Public Health Service Grant
Application (PHS 398), p. 17
17
Identifying participant pool
A search of our institutional tumor registry
indicates that 1340 breast cancers were diagnosed
between 1990 and 1999. We anticipate 180
additional cases will have been identified in
2000. Thus, we anticipate a total of 1520 women
diagnosed with breast cancer between 1990 and
2000. Based on our prior work, we anticipate that
70 of women will be postmenopausal, 40 of these
will be using HRT at the time of diagnosis, and
40 of these will be using both E P Harvey
JA, Santen RJ, Petroni GR, Bovbjerg VE, Williams
MB, 2000
18
Description of study participants
60.4 are women, 20.9 are non-Caucasian,
38.2 have baccalaureate or advanced degrees, and
the mean age is 53.3. years Halfwere diagnosed
with type 2 diabetes prior to 1995. Mean values
for BMI (36.9), weight (235.4 pounds/107 kilos),
waist circumference (46.3 inches115.8 cm), HbA1c
(7.72), total cholesterol (181.2 mg/dL) LDL
(104.2 mg/dL), and triglyceride (179.2 mg/dL)
suggestsomewhat increased risk of poor health
outcomes due to overweight, glycemic control, and
elevated lipids an average of 2 diabetes
medications at baseline, and substantial
proportions of participants were taking
medication for lipid lowering (46) and blood
pressure control (76). Bovbjerg VE, Wolf AM,
and the ICAN investigators, 2002
19
Estimation of participation
There are approximately 163,000 health plan
members in the four study regions 20,000 in
Roanoke, 15,000 in Harrisonburg/Valley, 65,000 in
Charlottesville, and 63,000 (50,000 commercial
and 13,000 Medicaid) in Richmond Based on the
ICAN pilot study, we expect that 489 members will
contact the study, be eligible, and wish to
participate among all four sites, a total of 390
will be enrolled we expect approximately 60 of
participants to be female, and approximately 30
African-American, with a mean age of
approximately 50 years. Approximately 8 will be
covered by Medicaid. Bovbjerg VE, Wolf AM, and
the ICAN investigators, 2002
20
Estimating effect size
The risk of chronic renal failure in our study
was higher among recipients of liver transplants
who were treated with cyclosporin than among
those who were treated with tacrolimus RR1.25
The results of studies comparing
cyclosporin-based immunosuppressive regimens
withtacrolimus-based regimens have been
contradictory. Most comparative evaluationsare
of limited validity Ojo et al. Chronic Renal
Failure after Transplantation of a Nonrenal
Organ. N Engl J Med 2003 349 931-40. Should
you propose a clinical trial to answer the
question?
21
Estimating sample size
Given ?0.05, power0.80 From observational
study incidence of renal failure in
tacrolimus0.16, relative risk of renal failure
in cyclosporin1.25 From study design random
assignment to equal sized groups Sample size
needed c 1500 in each group (Conversely--if one
could enroll only 200 patients, a relative risk
of over 2.5 would be needed to rule out chance.)
22
With an initial enrollment of 195 subjects per
group, and allowing for a 10 annual dropout
rate, the two-sample t-test has 80 power, with a
two-sided significance level of 5, to detect
differences between the treatment groups, when
the true mean difference is equal to 0.62.
sufficient power to detect slightly larger,
clinically significant group differences in HbA1c
(1.0 absolute difference) when stratified by
gender or race, both of which are important
potential effect modifiers. The use of the
two-sample t-test to compare the mean change from
the month 12 HbA1c level at a single point in
time (42 months) is a conservative estimate
does not represent the complexity of the analyses
These analyses will use repeated measures
models and all the HbA1c values collected at
months 12, 18, 24, 30, 36 and 42 to compare the
mean HbA1c profiles between the treatment groups
over time. Bovbjerg VE, Wolf AM, Conaway MR,
and the ICAN investigators, 2002
23
Sources of data
24
Deciding on data sources
Accuracy and practicability of data collection
methods are often inversely correlated. A method
providing more satisfactory information will
often be a more elaborate expensive, or
inconvenient one Accuracy must be balanced
against practical considerations and that method
chosen which will provide the maximal accuracy
within the bounds of the investigators resources
and other practical limitations. Abramson JH.
Survey methods in community medicine. Edinburgh
Churchill Livingstone , 1984.
25
Choices

• Direct measurement
• person (e.g. labs, anthropometry, maxV02)
• environment (e.g. surgical equipment, nurse
  staffing)
• Records
• existing databases (e.g. medical records, CDR,
  lab databases, clinic databases--IRB!)
• Participant provided information
• Questionnaires (e.g. food intake, activity)
• Surveys (e.g. health status SF-36)
• Diaries (e.g. pain diaries)

26
Existing records

• Definition data that have not been collected for
  the specific question at hand
• clinical vs. administrative vs. regulatory
• Diagnosis, treatment of conditions that come to
  attention of clinicians
• Validity of medical records depends on
• comprehensiveness of the record
• data included in record
• Special case electronic databases

27
Anyone who has ever worked with medical records
know that they are an imperfect source of
scientific data. The records contain all the
idiosyncrasies and errors that can occur in the
communication of human observations The recorded
data often consist of anecdotal generalities
rather than precise specifications the
statements often represent opinions or
interpretations rather than descriptions of
observed evidence important information may
sometimes be absent or unobtainable and the
remarks made by one doctor may sometimes
contradict those made by another. Feinstein et
al. Arch Intern Med 1969 123 448-61.
28
Relative merits of data sources
Adapted from Armstrong BK, White E, Saracci R.
Principles of exposure measurement in
epidemiology. New York Oxford University
Press, 1994.
29
Observational studies for

• questions inherently not testable by RCT
  (feasibility, ethics)
• getting answers faster than RCTs
• identifying risk factors for disease, outcomes
• providing support for RCT planning
• identifying outcomes in usual practice
• investigating the potential for harm