Critically Reviewing an Epidemiologic Study

1
Introduction to Research Methods In the Internet
Era
Critically Reviewing an Epidemiologic Study
Thomas Songer, PhD
2

• Learning Objectives
• Describe the approach to reviewing a manuscript
• Identify the research hypothesis of a manuscript
• Identify the quality of the research and the
  validity of the findings of a manuscript
• Describe the factors which may raise concern
  about the truth of a research finding

2
3
Reviewing a Paper is about Asking Questions
? ? ?
3
4
What level of measurement error exists?
Are there unstated confounding factors?
What is the study hypothesis?
What is my overall impression?
Is the study adequately powered?
Were appropriate statistical procedures used?
What population do the study subjects originate
from?
4
5
Key Areas of Focus

• Critique of data collection
• Critique of data analysis
• Critique of data interpretation

5
6
Example Manuscript

• Fitzpatrick AL, Kuller LH, Lopez OL, Diehr P,
  OMeara ES, Longstreth Jr. WT, Luchsinger JA.
  Midlife and Late-Life Obesity and the Risk of
  Dementia. Archives of Neurology 66(3)336-342,
  2009.

6
7
What to Examine When Evaluating Data Collection

• Study Population
• Potential for
• Selection Bias
• Information Bias
• Confounding

• Study Context
• Study Objectives
• Is a hypothesis stated?
• Exposure and Outcome Variables
• Study Design

7
8
Study Context

• Several Issues to consider
• What is the public health significance of this
  study?
• Does this study generate new hypotheses or
  confirm previous results with improved methods?
• Is the study hypothesis biologically plausible?

8
9
Study Context Fitzpatrick, et. al.

• - What is the public health significance of this
  study?
• Rising obesity levels in US
• Dementia increasing in US
• Aging population
• Does this study generate new hypotheses or
  confirm previous results with improved methods?
• Seeks to clarify conflicting results in the
  literature by examining a large study sample
  longitudinally
• Is the study hypothesis biologically plausible?

9
10
Study Objectives

• What do the investigators want to achieve in this
  research?
• What is the hypothesis of this study?
• There may be more then one
• Is the hypothesis specific or too general to
  refute?

10
11
Exposure and Outcome Variables

• Primary exposure
• How was variable defined?
• How was information on exposure collected?
• Best method?
• Sensitivity/specificity of this method?
• Potential for misclassification?

• Primary outcome
• Conceptual vs. operational outcome?
• e.g. breast cancer vs. malignant neoplasm of the
  breast tissue
• How was information on outcome collected?
• Best method?
• Sensitivity/specificity of this method?
• Potential for misclassification?

11
12
Type of Study

• What study design was employed?
• Is it an appropriate design?
• Exposure or outcome rare?
• New hypothesis?
• What are the limitations and strengths of this
  design?

12
13
Type of Study -- Fitzpatrick

• What study design was employed?
• Cohort study (retrospective)
• Is it an appropriate design?
• Exposure or outcome rare? Neither
• New hypothesis? No, but conflicting study
  results
• What are the limitations and strengths of this
  design?
• Strengths longitudinal assessment, incidence of
  dementia, uses previously collected data
• Limitations short period of assessment

13
14
The Study Population

• What was the source of study population?
• How does the study population compare to the
  general population?
• How were subjects selected?
• Could this method introduce selection bias?
• What was the sample size?
• Is the statistical power of the study identified?
• Out of the projected study sample, how many
  persons participated?

14
15
The Study Population -- Fitzpatrick

• What was the source of study population?
• Community dwellers who were Medicare eligible
  over age 65 years - 4 sites
• How were subjects selected? Not stated
• Could this method introduce selection bias?
• What was the sample size? 2798 out of original
  cohort of 5888 adults
• Is the statistical power of the study identified?
  Yes
• Out of the projected study sample, how many
  persons participated? Unknown from original
  study.

15
16
Potential for Bias
SELECTION BIAS

• INFORMATION BIAS

• Could there have been bias in the selection of
  subjects?
• What type of bias would this be?
• e.g. healthy worker bias
• In which direction would this bias affect the
  measure of association?

• Could there have been bias in the collection of
  information?
• What type of bias would this be?
• e.g. interviewer bias
• In which direction would this bias affect the
  measure of association?

16
17
Potential for Confounding

• What factors were potentially confounding the
  study relationship?
• What methods did the authors use to minimize the
  influence of confounding when planning the study?
• E.g. restriction, matching, randomization, etc.
• Is there still residual confounding?

17
18
Potential for Confounding - Fitzpatrick

• What factors were potentially confounding the
  study relationship?
• Table 1, others
• What methods did the authors use to minimize the
  influence of confounding when planning the study?
• E.g. restriction, matching, randomization, etc.
• Is there still residual confounding? likely

18
19
What to Examine When Evaluating Data Analysis

• Confounding
• Measures of Association
• Measures of Statistical Stability

19
20
Data Analysis -- Confounding

• What methods were used to control confounding?
• -Standardization Indirect and Direct. Usually
  used to control for differences in age
  distribution among populations.
• -Stratification Allows you to examine data
  more closely. However, it is difficult to control
  for more than 1 confounder.
• -Matching Done in Case-Control Studies.
• -Multivariate Analysis Linear Regression,
  Logistic Regression, Poisson Regression, Cox
  Proportional Hazards model. Allows you to
  control for multiple confounders simultaneously.

20
21
Data Analysis Measures of Association

• What Measures of Association were reported in the
  study? Was the correct measure used?
• Cohort Study Relative Risk (RR), Odds Ratio
  (OR), Hazard Ratio (HR), Incidence Rate Ratio
  (IRR.
• Case-Control Exposure or Disease OR (if
  nested). Can not use RR. However, the OR is a
  good estimate of the RR when the prevalence of
  the disease in the study population is very low.
• Cross-sectional Study Prevalence Ratio.
• Ecologic Study Correlation coefficient.

21
22
Data Analysis Statistical Stability

• How was the potential for random error accounted
  for in the study?
• Hypothesis Testing Can use p-values or
  confidence Intervals (CI) to test the null
  hypothesis.
• P-value The probability of observing the study
  results given that the null hypothesis is true.
  Plt0.05 is a standard value that investigators use
  to reject the null hypothesis of no association
  and declare that there is a significant
  relationship between 2 variables.

22
23
Data Analysis Statistical Stability

• 95 CI This measure can be used for hypothesis
  testing and interval estimation. Can be defined
  as, if one will repeat the study 100 times the
  true association will lie inside the interval 95
  of the time.
• We fail to reject the null hypothesis when a
  confidence interval contains the null value of 1
  between its lower and upper limits for relative
  measures.

23
24
Data Analysis Statistical Stability

• Large confidence intervals indicate that the
  standard error is high. A high standard error is
  often related to a small sample size.
  Underpowered studies normally have wider
  confidence intervals and thus difficulty in
  rejecting the null hypothesis.
• The problem, therein, lies that it is difficult
  to know if the non-association is real or false.

24
25
What to Examine When Interpreting the Results of
the Study

• Major findings of the research
• Influence (on the results) of
• Bias and confounding
• misclassification

25
26
Major Findings

• The first paragraph of the discussion section in
  a manuscript should summarize the main findings
  of the study.
• Example Sedentary individuals in this study
  have 3 (95 CI1.5-4.9) times the risk of
  developing a Myocardial (MI) compared to active
  individuals after controlling for potential
  confounders.

26
27
Influence of Bias and Confounding

• Reader should be able to recognize information
  bias, selection bias, or confounding in the study
  and assess their magnitude and direction in the
  study.
• Bias or confounding that is large in magnitude
  signals that the findings in this sample may not
  approximate what you would expect to see in the
  population.

27
28
Misclassification

• Misclassification of the exposure or the outcome
  (or both) can influence study results
• Non-Differential Misclassification is similar in
  the exposure or outcome groups. This would bias
  the results to the null making it unlikely for
  investigators to reject the null hypothesis.

28
29

• Differential Misclassification occurs at a
  different rate in exposure or outcome groups.
• Example of differential misclassification, a
  larger number of individuals are classified as
  high stress instead of medium stress than
  individuals classified as medium stress instead
  of high stress. This type of misclassification
  can bias results away or towards the null
  hypothesis.

29
30
Formulating an Overall Impression of the
Manuscript

• What are the strengths and limitations of the
  report?
• How do these balance?
• Can the results be generalized to the whole
  population?

30
31
Strengths and Limitations

• Examine the overall issues related to data
  collection, data analysis, and data
  interpretation.
• What conclusions do you draw from the results
  based upon your interpretation of the strengths
  and limitations of the study?
• Do the strengths outweigh the limitations?
• They are often mentioned in the discussion
  section of a manuscript.

31
32
Generalizability

• Goal is to have a study where the results can be
  used to infer what is going on in the population
• Major problems with the internal validity of the
  study make it difficult to for the results to be
  generalized to any population.
• Example, the study population excluded a certain
  groups, minorities, women, blacks, or low income
  individuals. The results would not be
  generalizable to these groups.

32
33
Conclusions and Justification

• The conclusions are a brief summary of the
  findings.
• Authors tend to include recommendations for
  future studies or policy.
• It is essential that the recommendations do not
  stray far from the study findings.
  Recommendations should be made in the context of
  the findings or the readers may be deceived and
  make incorrect conclusions about the actual
  results of the study.

33
34
The Big Picture of Research Findings

• Publication bias
• John P.A. Ioannidis
• Why Most Published Research Findings are False.
  PLoS Medicine 2(8)e124, 2005.

34
35
Publication bias

• Definition
• Publication bias refers to the greater
  likelihood that studies with positive results
  will be published
• JAMA
  20022872825-2828

Abbasi
35
36
Publication bias may .

• Distort the scientific record
• Hide the truth of association/no association
• Influence doctors decision making
• Mislead policy makers
• Etc.

36
37
Ioannidis Corollaries

• The smaller the studies conducted, the less
  likely the research findings are to be true
• The smaller the effect size, the less likely the
  research findings are to be true
• The greater the financial interest and prejudice,
  the less likely the research findings are to be
  true
• The hotter a topic interest, the less likely the
  research findings are to be true

37