Appraising the literature

1
Appraising the literature

• Mark Stevenson
• EpiCentre, IVABS, Massey University, Palmerston
  North
• M.Stevenson_at_massey.ac.nz

2
Roadmap

• Journals
• Appraising the literature
• description of the evidence
• internal validity
• external validity
• comparison of the results with other evidence

3
Journals

• Text books
• one stop shop for information
• represent the opinion of one or a small number of
  authors
• out of date by the time they are published
• Journals
• articles deal with a single subject (in detail)
• up to date
• peer reviewed

4
Journals

• Text books
• one stop shop for information
• represent the opinion of one or a small number of
  authors
• out of date by the time they are published
• Journals
• articles deal with a single subject (in detail)
• up to date
• peer reviewed

5
(No Transcript)
6
Journals

• A citation
• Bates T, Thurmond M, Hietala S, et al. (2003)
  Surveillance for detection of foot-and-mouth
  disease. Journal of the American Veterinary
  Medical Association 223, 609 - 614.

7
Journals

• A citation
• Bates T, Thurmond M, Hietala S, et al. (2003)
  Surveillance for detection of foot-and-mouth
  disease. Journal of the American Veterinary
  Medical Association 223, 609 - 614.
• who wrote the article
• et al. is latin for et alia which means and
  others

8
Journals

• A citation
• Bates T, Thurmond M, Hietala S, et al. (2003)
  Surveillance for detection of foot-and-mouth
  disease. Journal of the American Veterinary
  Medical Association 223, 609 - 614.
• the year the article was published

9
Journals

• A citation
• Bates T, Thurmond M, Hietala S, et al. (2003)
  Surveillance for detection of foot-and-mouth
  disease. Journal of the American Veterinary
  Medical Association 223, 609 - 614.
• the title of the article

10
Journals

• A citation
• Bates T, Thurmond M, Hietala S, et al. (2003)
  Surveillance for detection of foot-and-mouth
  disease. Journal of the American Veterinary
  Medical Association 223, 609 - 614.
• the journal in which the article was published
• frequently abbreviated (e.g. JAVMA)

11
Journals

• A citation
• Bates T, Thurmond M, Hietala S, et al. (2003)
  Surveillance for detection of foot-and-mouth
  disease. Journal of the American Veterinary
  Medical Association 223, 609 - 614.
• the volume in which the article appeared

12
Journals

• A citation
• Bates T, Thurmond M, Hietala S, et al. (2003)
  Surveillance for detection of foot-and-mouth
  disease. Journal of the American Veterinary
  Medical Association 223, 609 - 614.
• the page numbers

13
Journals

• In the olden days
• travel to the library
• photocopy the articles youre interested in
• These days
• on-line access to journal material
• articles distributed in PDF (portable document
  format)
• on-line library access for Massey graduates?
• University of Sydney VEIN

14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
Journals

• Advantages
• remote access
• searchable
• saves paper
• portable
• Disadvantages
• just not the same as flicking through a journal

20
Journals

• Internet
• facilitated the dissemination of information
• varying quality
• As good scientists, we need to be discerning
  about what we read and (more importantly) what we
  believe
• A systematic method of appraising (or evaluating)
  the literature helps us to do this

21
Roadmap

• Journals
• Appraising the literature
• description of the evidence
• internal validity
• external validity
• comparison of the results with other evidence

22
Roadmap

• Journals
• Appraising the literature
• description of the evidence
• internal validity
• external validity
• comparison of the results with other evidence

23
Description of the evidence

• What hypothesis was being tested?
• What was the study design?
• survey
• clinical trial
• case-control study
• prospective or retrospective cohort study
• cross sectional study
• What were the exposure variable(s) and what was
  the outcome variable?

24
Randomised clinical trial
25
Cross sectional study
26
Description of the evidence

• Definition of the subjects that were studied in
  terms of
• source populations
• time frames
• eligibility criteria
• participation rates of the different groups
  compared
• Summary of the main result
• what is the result of the study in terms of the
  relationship between exposure and outcome?

27
Roadmap

• Journals
• Appraising the literature
• description of the evidence
• internal validity
• external validity
• comparison of the results with other evidence

28
Internal validity

• Internal validity
• describes the truthfulness of inferences about
  the study population (i.e. those that took part
  in the study)
• how well was the study done?
• often, researchers claim that exposure to one
  factor causes a specific outcome
• exposure to A2 milk causes insulin dependent
  diabetes
• we need to evaluate these claims very carefully
• differentiate between causation and association

29
Age adjusted mortality rates as a function of
particulate air matter concentration for 100
capital cities throughout the world.
30
Internal validity

• Two aspects of internal validity
• non-causal explanations
• positive features of causation

31
Internal validity

• Three possible non-causal mechanisms which could
  produce the observed results
• bias errors in measurement
• confounding more next year
• chance variation did they fluke it?

32
Internal validity

• Bias
• bias in ascertainment
• surveillance, diagnosis, referral, selection,
  non-response, length of stay, survival bias
• bias in the estimation of exposure
• recall, interviewer, prevarification, improper
  analysis

33
Internal validity

• Confounding
• refers to the effect of an extraneous variable
  that wholly or partially accounts for the
  apparent effect of the study exposure, or masks
  an underlying true association
• ???
• best explained by an example

34
Internal validity

• Confounding (cont.)
• ? use of oral contraceptives ? risk of myocardial
  infarction (MI)
• ? smoking ? risk of myocardial infarction
• there is evidence to show that women who use oral
  contraceptives are more likely to smoke more than
  women who do not
• if we conduct a study and find a positive
  relationship between oral contraceptive use and
  MI, we need to be careful that we have accounted
  for the confounding influence of smoking, before
  making a claim that a causal relationship between
  oral contraceptives and MI exists

35
Internal validity

• Chance
• was a relationship between exposure and outcome
  identified by chance?
• if we perform 20 statistical tests using an alpha
  level of 0.05, we will make a type I error on one
  occasion
• what is a type I error?
• you reject the null hypothesis when, in reality,
  it is true

36
Internal validity

• The order of these non-causal explanations is
  important
• if there is severe bias, no analytical
  manipulation of the data will overcome the
  problem
• if there is confounding, then appropriate
  analysis will (in most cases) overcome the
  problem
• assessment of chance variation should be made on
  the main result of the study, after considering
  issues of bias and confounding

37
Internal validity

• Three aspects of positive features of causation
  should be considered
• 1. Is there a correct temporal relationship?
• 2. Is the relationship strong?
• 3. Is there a dose-response relationship?

38
Internal validity

• Is there a correct temporal relationship?
• for a relationship to be causal, the exposure
  must act before the outcome occurs
• in a prospective study design this requirement is
  established by ensuring that subjects do not
  already have the outcome of interest when the
  study starts
• the ability to clarify time relationships is
  weaker in retrospective studies, and care is
  required to ensure that possible causal factors
  did in fact occur before the outcome of interest

39
Internal validity

• Is there a correct temporal relationship (cont.)?
• a difficulty in all study designs, but more so in
  retrospective studies, is that the occurrence in
  biological terms of the outcome of interest may
  precede the recognition and documentation of that
  outcome by a long and variable period of time
  (e.g. some cancers)

40
Internal validity

• Is the relationship strong?
• a stronger measure of association is more likely
  to indicate a causal relationship
• the fact that a relationship is strong does not
  protect us against certain non-causal
  relationships, however if the relationship that
  is observed is due to bias, then the bias must be
  large and therefore easily identified
• if a strong relationship is due to confounding,
  either the association of the exposure with the
  confounder must be very close, or the association
  of the confounder with the outcome must be very
  strong

41
Internal validity

• Is there a dose-response relationship?
• in some circumstances the demonstration of a
  smooth dose-response relationship may be a strong
  argument against an identified relationship
  arising as a result of bias
• in general, we should expect uni-directional
  dose-effect relationships and evidence that this
  is not the case should be considered carefully

42
Roadmap

• Journals
• Appraising the literature
• description of the evidence
• internal validity
• external validity
• comparison of the results with other evidence

43
External validity

• External validity ? generalisability of the
  results
• if the internal validity of a study is poor, then
  there is no point in proceeding further
• Three aspects of external validity should be
  considered
• 1. Can the results be applied to the eligible
  population?
• 2. Can the results be applied to the source
  population?
• 3. Can the results be applied to other relevant
  populations?

44
External validity

• Can the results be applied to the eligible
  population?
• the relationship between the study participants
  (those that participated in the study) and the
  population of eligible subjects (those that met
  the study inclusion criteria but did not take
  part) should be well documented
• losses due to non-participation have to be
  considered carefully as they are likely to be
  non-random, and the reasons for the losses may be
  related to the exposure or the outcome

45
External validity

• Can the results be applied to the source
  population?
• the important issue is not whether the subjects
  studied are 'typical', but whether the
  association between outcome and exposure given by
  the study participants is likely to apply to
  other groups

46
External validity

• Can the results be applied to other relevant
  populations?
• in general, the difficulties of applying results
  from one groups of subjects to another will be
  minimal for issues of basic physiology and
  maximal for effects in which cultural and
  psycho-social aspects are dominant

47
Roadmap

• Journals
• Appraising the literature
• description of the evidence
• internal validity
• external validity
• comparison of the results with other evidence

48
Comparison with other evidence

• Clinical questions
• useful to consider a hierarchy of evidence
• given that studies are adequately performed
  within the limitations of the design used, the
  reliability of the information from them can be
  ranked as follows
• randomised trials
• cohort and case-control studies
• other comparative studies
• case series, descriptive studies, clinical
  experience

49
Comparison with other evidence

• Clinical questions
• randomised clinical trials, if properly performed
  on adequate numbers of subjects, provide greatest
  evidence because of the unique advantages in
  overcoming problems of bias and confounding
• Three aspects of comparison should be considered
• 1. Are the results consistent with other
  evidence?
• 2. Are the results plausible biologically?
• 3. Is there coherency with the distribution of
  the exposure and the outcome?

50
Comparison with other evidence

• Are the results consistent with other evidence?
• this is the most important characteristic used in
  the judgement that an association is causal
• to say that the result is consistent requires
  that the association has been observed in a
  number of different studies, each of which
  individually can be interpreted as showing a
  causal explanation, and which have enough
  variation in their methodology and study
  populations to make it unlikely that the same
  biases or confounding factors apply in all the
  studies
• lack of consistency argues against causality

51
Relative risks (and their 95 confidence
interval) from six trials comparing the effect of
CIDR treatment with untreated controls on
submission rate.
Meta-analysis
52
Relative risks (and their 95 confidence
interval) from 12 trials comparing the effect of
post insemination CIDR treatment with untreated
controls on conception rate.
53
Comparison with other evidence

• Are the results plausible biologically?
• plausibility refers to the observed association
  being biologically understandable on the basis of
  current knowledge concerning its likely
  mechanisms
• however, any dramatically new observation may be
  in advance of current biological thinking and its
  lack of plausibility may reflect deficiencies in
  biological knowledge rather than error in
  observation
• John Snow effectively prevented cholera in London
  25 years before the isolation of the cholera
  bacillus and the general acceptance of the
  principle that the disease could be spread by
  water

54
Comparison with other evidence

• Coherency with the distribution of the exposure
  and the outcome?
• an association is regarded as coherent if it fits
  the general features of the distribution of both
  the exposure and the outcome under assessment
• if lung cancer is due to smoking, the frequency
  of lung cancer in different populations and in
  different time periods should relate to the
  frequency of smoking in those populations at
  earlier relevant time periods

55
Summary
What are they on about?

• Description of the evidence
• type of study, outcome measure, population
  investigated, results
• Internal validity

Did they do things properly?

• non-causal explanations
• bias
• confounding
• chance

• positive features of causation
• temporal relationship
• strength of relationship
• dose-response

56
Summary
Can we apply these results to other populations?

• External validity
• can the results be applied to the eligible
  population?
• can the results be applied to the source
  population?
• can the results be applied to other relevant
  populations?
• Comparison of the results with other evidence
• are the results consistent with other evidence?
• are the results plausible biologically?
• is there coherency with the distribution of the
  exposure and the outcome?

Are the findings reported here consistent with
other studies that looked at the same thing?
57
(No Transcript)