Evaluation of Diagnostic Tests

1
Evaluation of Diagnostic Tests

• Gary E Raskob, Ph. D.
• Associate Vice President for Clinical Research
  Dean, College of Public Health
• University of Oklahoma

2
Objectives

• To review the key study design features for
  evaluating diagnostic tests
• To review the measures of performance or
  properties of a diagnostic test
• To illustrate how to use clinical outcome on
  follow-up as a reference standard for evaluating
  a diagnostic test or strategy
• To discuss a current example of bias in
  evaluating a diagnostic test

3
Reference Materials

• Jaeschke R, Guyatt G, Sackett DL. Users guides
  to the Medical Literature III. How to use an
  article about a diagnostic test.
• A. Are the results of the study valid?
  JAMA 1994271 389 391
• B. What are the results and will they help me
  in caring for my patients?
  JAMA 1994 271 703 707

4
Reference Materials contd

• Birdwell B, Raskob G, Whitsett T et al The
  clinical validity of normal compression
  ultrasonography in outpatients suspected of
  having deep-vein thrombosis.
  Ann Intern Med 1998128 1 7
• Punglia R,Damico A, Catalona W et al Effect
  of verification bias on screening for prostate
  cancer by measurement of prostate specific
  antigen.
  N Engl J Med 2003 349 335- 42

5
Key Study Design Features

• Independent, blind comparison with a reference
  standard
• Include appropriate spectrum of patients
  -
  with/without disease of interest -
  with varying severity of disease - with
  co-morbid conditions

6
Key Study Design Features contd

• Test being evaluated and the reference standard
  performed in all patients
  (ie. test result does not influence decision
  to perform reference standard)
• Methods for performing test described in
  sufficient detail to permit replication
• Sufficient patients studied to make valid
  conclusions (95 CI )
  

7
Comparison of test result to reference standard
(gold standard)
8
Performance measures of a diagnostic test

• Sensitivity
• Specificity
• Positive predictive value (PV)
• Negative predictive value (-PV)
• Likelihood ratios

9
Comparison of test result to reference standard
(gold standard)
10
Calculating performance measures of a diagnostic
test

• Sensitivity A/AC
• Specificity D/BD
• Positive predictive value (PV)
• PV A/AB
• Negative predictive value (-PV)
• -PV D/CD

11
Impact of Disease Prevalence

• Predictive values vary with prevalence of disease
  in population
• Sensitivity and specificity do not vary with
  prevalence (over the
  broad spectrum of disease severity)

12
Receiver Operating Characteristic Curve (ROC)

• Graphic representation of the relationship
  between sensitivity and specificity
• Plots sensitivity (y-axis) against
  1-specificity (x-axis)
• Can be used to compare two diagnostic tests

13
Likelihood ratios (LR)

• Useful when a test has multiple levels of results
  rather than negative/positive
• The LR indicates how much a given diagnostic test
  result will raise or lower the pre-test
  probability of disease
• LRgt1 raise and LR lt1 lower the pre-test
  probability
• Likelihood ratios do not vary as a function of
  the prevalence of disease

14
Likelihood ratios (LR) General Rules of
Thumb

• LR gt 10 or lt 0.1 produce large changes in
  pre-test probability
• LR of 5 to 10 or 0.1 to 0.2 produce moderate
  changes
• LR of 1 to 2 or 0.5 to 1 produce small changes in
  pre-test probability

15
Clinical Outcome on Follow-upCan sometimes be
used to

• Establish a test as the reference standard
• Evaluate a new test/strategy when no established
  reference standard exists
• Test the safety of withholding treatment based on
  a negative test result

16
Common Potential Biases in Studies Evaluating
Diagnostic Tests

• Work-up bias (verification
  bias in Punglia et al)
• Interpretation bias
• Incorporation bias
• Sackett DL. Bias in analytic research.
  J Chron Disease 197932 51 - 63