Session 4: Assessing a Document on Diagnosis

1
Session 4 Assessing a Document on Diagnosis
October 29, 2008

• Peter Tarczy-Hornoch MD
• Head and Professor, Division of BHI
• Professor, Division of Neonatology
• Adjunct Professor, Computer Science and
  Engineering
• faculty.washington.edu/pth

2
Using Questionmark Software

• See e-mail MIDM Important - testing your
  Questionmark login id/web browser before MIDM
  final exam for more details
• First get your login id/password from MyGrade
• Second test your login id/password and your
  computers/browsers ability to save and retrieve
  you exam
• https//primula.dme.washington.edu/q4/perception.d
  ll
• 2 question Test exam up until 5P Monday 11/3

3
Assessing a Document on Diagnosis

• Context for Assessing a Diagnosis Document
• Diagnosis Statistics
• Applying to a Scenario

4
Diagnostic vs. Therapeutic Studies
Diagnostic Testing (What is it?) (Session 4)
Patient Data Information
Therapy/Treatment (What do I do for it?) (Session
3)
Case specific decision making
General Information Knowledge
5
Steps to Finding Assessing Information

• Translate your clinical situation into a formal
  framework for a searchable question (Session 1)
• Choose source(s) to search (Session 2)
• Search your source(s) (Session 2)
• Assess the resulting articles (documents)
• Therapy documents (Session 3)
• Diagnosis documents (Session 4)
• Systematic reviews/comparing documents (Session
  5)
• Decide if you have enough information to make a
  decision, repeat 1-4 as needed (ICM, clinical
  rotations, internship, residency)

6
PubMed Finding a Diagnostic Article
7
Assessing a Document
8
Assessing a Document on Diagnosis

• Context for Assessing a Diagnosis Document
• Diagnosis Statistics
• Applying to a Scenario

9
Many Different Kinds of Tests

• Tests predict presence of disease
• Types of tests
• Screening test before symptoms appear look for
  disease
• Example Screening mammograms
• Diagnostic test given symptoms/suggestion of a
  disease help rule in (confirm) or rule out
  (reject) a diagnosis
• Example ultrasound of appendix in face of
  abdominal pain
• Gold Standard a perfect test that
  definitively categorizes a patient as having
  one disease
• Example Surgery to remove appendix and then
  pathologic exam
• Cant always use Gold Standard gt Use diagnostic
  tests
• E.g. high risk/cost, only rules in/out one
  disease vs. multiple, etc.

10
The 2x2 Table Diagnostic Test vs. Gold Standard
Gold Standard Test
Diagnostic Test

• Non-intuitive labels
• Disease Present Disease Positive () Dz()
• Test Positive Test predicting disease present
• From patient/provider point of view neither
  Disease Positive nor Test Positive () are good
  things!

11
Sensitivity (Sn)
Gold Standard Test
Diagnostic Test

• Sensitivity is proportion of all people with
  disease who have a positive test
• Sensitivity TP/(TPFN)
• SnNOut - sensitive test, if negative, rules out
  disease
• Sensitivity useful to pick a test sensitivity
  key for screening test

12
Specificity (Sp)
Gold Standard Test
Diagnostic Test

• Specificity is proportion of all people without
  disease who have negative test
• Specificity TN/(FPTN)
• SpPIn A specific test, if positive, rules in
  disease
• Specificity useful to pick a test specificity
  key for diagnostic test

13
Cut Off Values Impact Sn/SpExample blood
sugar to predict diabetes
Sensitivity key for screening test
Specificity key for diagnostic test
14
Positive Predictive Value (PPV)
Gold Standard Test
Diagnostic Test

• PPV is proportion of all people with a positive
  test who have a disease
• PPVTP/(TPFP)
• PPV is useful to use a test if you have a
  positive result for your patient, what of
  people with positive results actually have the
  disease

15
Negative Predictive Value (NPV)
Gold Standard Test
Diagnostic Test

• NPV is proportion of all people with a negative
  test who dont have a disease
• NPVTN/(FNTN)
• NPV is useful to use a test if you have a
  negative result for your patient, what of
  people with negative results actually dont have
  the disease

16
Prevalence, pre-test post-test probabilities

• Prevalence
• total cases of disease in the population at given
  time
• 2x2 table disease ())/disease () disease
  (-)
• Pre-test probability
• Estimate of probability/likelihood your patient
  has a disease before you order your test
• Often an estimation based on experience or
  prevalence
• Screening test pre-test probability prevalence
• Post-test probability
• The probability/likelihood that your patient has
  a disease, after you get the results of the test
  back

17
PPV/NPV Dependence on Disease PrevalencePPV
Example
Prevalence 50
Prevalence 5
SnTP/(TPFN) 45/(455)90 SpTN/(FPTN)
912/(91238)96 PPVTP/(TPFP)
45/(4538)54.2 of those with T() have Dz()

• SnTP/(TPFN)
• 450/(45050)90
• SpTN/(FPTN)
• 480/(20480)96
• PPVTP/(TPFP)
• 450/(45020)95.7
• of those with T() have Dz()

450
20
45
38
50
480
5
912
Prevalence 50
Prevalence 5
18
Pros/Cons Sn/Sp/PPV/NPV

• Relative Pros
• PPV/NPV useful for diagnosis - probability of
  disease after ( ) or () test
• Sn/Sp useful for choosing a test
  (screening/diagnosis)
• Relative Cons
• PPV/NPV vary with prevalence of disease
• Prevalence of disease in general population may
  not be the same as that of patients you see in
  clinic/ER
• Your estimation of probability of disease
  (pre-test probability) may not match prevalence
  in a population
• Current tendency therefore gt use likelihood
  ratios

Note Sn/Sp/PPV/NPV on boards
19
Bayes Theorem

• Note this slide is here for completeness,
  likelihood ratios better, this slide is thus not
  on the exam
• Bayes Theorem
• How to update or revise beliefs in light of new
  evidence
• http//plato.stanford.edu/entries/bayes-theorem/
• Related to Bayes is an alternate form of PPV/NPV
  as f(Sn, Sp, pre-test) that pulls out pre-test
  probability or prevalence
• P(Dz)probability of disease (e.g. prevalence,
  pre-test)
• PPVSnP(Dz)/SnP(Dz) (1-Sp)(1-P(Dz))
• NPVSp(1-P(Dz))/Sp(1-P(Dz)) (1-Sn)(P(Dz))

20
Likelihood Ratios

• Likelihood Ratio does NOT vary with prevalence
• Likelihood Ratio (LR)
• LR Sn/(1-Sp) likelihood ratio for a positive
  test
• LR- (1-Sn)/Sp likelihood ratio for a
  negative test
• Applying LR given a pre-test disease probability
• PrePre-test probability (can be prevalence)
• PostPost-test probability
• PostPre/(Pre(1-Pre)/LR)
• Same as Bayes PPV/NPV but cleanly separates
  test characteristics (LR) from disease
  prevalence/pre-test probabilities

21
Interpreting Likelihood Ratios (I)

• LR1.0
• Post-test probability the pre-test probability
  (useless)
• LR gt1.0
• Post-test probability gt pre-test probability
  (helps rule in)
• Test result increases the probability of having
  the disorder
• LR lt1.0
• Post-test probability lt pre-test probability
  (helps rules out)
• Test result decreases the probability of having
  the disorder
• LR (Likelihood Ratio for a Positive Test) vs.
  LR- (Likelihood Ratio for a Negative Test) gt See
  Appendicitis Slide

22
Interpreting Likelihood Ratios (II)

• Likelihood ratios gt10 or lt0.1
• Test generates large changes in pre- to post-test
  probability
• Test provides strong evidence to rule in/rule out
  a diagnosis
• Likelihood ratios of 5-10 and 0.1-0.2
• Test generates moderate changes in pre- to
  post-test probability
• Test provides moderate evidence to rule in/rule
  out a diagnosis
• Likelihood ratios of 2-5 and 0.2-0.5
• Test generates small changes in pre- to post-test
  probability
• Test provides minimal evidence to rule in/rule
  out a diagnosis
• Likelihood ratios 0.5-2
• Test generates almost no changes in pre- to
  post-test probability
• Test provides almost no evidence to rule in/rule
  out a diagnosis

23
Interpreting Likelihood Ratios (III)

• From slide on impact of prevalence
• Sn90, Sp96
• LR 0.90/(1-0.96)22.5
• PostPre/(Pre(1-Pre)/LR)
• If prevalence (pre-test) is 50 gt post-test
  95.7
• If prevalence (pre-test) is 5 gt post-test 54.2

LR Nomogram gt
24
Likelihood Ratios for Physical Exam for
Appendicitis
PresentModerate evidence for appendicitis
PresentModerate evidence for appendicitis BUT
95 CI of LR includes lt2 thus includes Minimal
Evidence
LR LR-
PresentAlmost no evidence for appendicitis
25
Assessing a Document on Diagnosis

• Context for Assessing a Diagnosis Document
• Diagnosis Statistics
• Applying to a Scenario

26
Learning to Diagnose Pneumonia

• Medical School
• Preclinical anatomy, histology, pathology,
  microbiology, pharmacology, physiology,
• Clinical medicine, pediatrics, family medicine,
  surgery,.
• Residency Outpatient, inpatient, specialty
  rotations, general rotations, emergency room,.
• Fellowship More of the same
• Result a number of items on history, physical
  exam, laboratory studies that suggest pneumonia
  with chest X-ray as gold standard

27
Literature on Diagnosis of Pneumonia

• Clinical query for pneumonia diagnosis (1478)
• Change to community acquired pneumonia (181)
• Add in likelihood ratio (27)
• Find Derivation of a triage algorithm for chest
  radiography of community-acquired pneumonia
  patients in the emergency department. Acad Emerg
  Med. 2008 Jan15(1)40-4.

28
Paper Background/Objectives

• BACKGROUND Community-acquired pneumonia (CAP)
  accounts for 1.5 million emergency department
  (ED) patient visits in the United States each
  year.
• OBJECTIVES To derive an algorithm for the ED
  triage setting that facilitates rapid and
  accurate ordering of chest radiography (CXR) for
  CAP.

29
Paper Methods

• METHODS The authors conducted an ED-based
  retrospective matched case-control study using
  100 radiographic confirmed CAP cases and 100
  radiographic confirmed influenzalike illness
  (ILI) controls. Sensitivities and specificities
  of characteristics assessed in the triage setting
  were measured to discriminate CAP from ILI. The
  authors then used classification tree analysis to
  derive an algorithm that maximizes sensitivity
  and specificity for detecting patients with CAP
  in the ED triage setting.

30
Paper Results (I)

• RESULTS Temperature greater than 100.4 degrees F
  (likelihood ratio 4.39, 95 confidence interval
  CI 2.04 to 9.45), heart rate greater than 110
  beats/minute (likelihood ratio 3.59, 95 CI
  1.82 to 7.10), and pulse oximetry less than 96
  (likelihood ratio 2.36, 95 CI 1.32 to 4.20)
  were the strongest predictors of CAP. However, no
  single characteristic was adequately sensitive
  and specific to accurately discriminate CAP from
  ILI.
• Evidence
• LRgt10 Strong, LR 5-10 moderate
• 2-5 minimal, 1-2 scant evidence

31
Paper Results (II)

• RESULTS (continued) A three-step algorithm
  (using optimum cut points for elevated
  temperature, tachycardia, and hypoxemia on room
  air pulse oximetry) was derived that is 70.8
  sensitive (95 CI 60.7 to 79.7) and 79.1
  specific (95 CI 69.3 to 86.9).
• LR Sn/(1-Sp)0.708/(1-0.791)3.39 (minimal)
• LR- (1-Sn)/Sp(1-0.708)/0.791 0.37 (minimal)
• PostPre/(Pre(1-Pre)/LR)
• Post if Pre 1/10 0.1/(0.1(1-0.1))/3.39)0.27
• Post if Pre 1/2 0.5/(0.5(1-0.5))/3.39)0.77

32
Paper Conclusions

• CONCLUSIONS No single characteristic adequately
  discriminates CAP from ILI, but a derived
  clinical algorithm may detect most radiographic
  confirmed CAP patients in the triage setting.
  Prospective assessment of this algorithm will be
  needed to determine its effects on the care of ED
  patients with suspected pneumonia.
• Note all of these characteristics are among the
  tried and true findings taught in medical school,
  residency, fellowship but typically not
  quantitatively taught

33
Small Group Monday November 3rd

• Students to complete assignment for Small Group
  Session 5 by Mon 11/3 2-250
• Small group leads to give examples of recent
  clinical situations where they had to evaluate
  one or more documents related to making a
  diagnosis
• Group to review and discuss from assignment
  short examples related to diagnosis focusing on
• Sensitivity, Specificity, Positive Predictive
  Value (PPV), Negative Predictive Value (NPV)
• Likelihood Ratios (Positive/Negative) LR, LR-
• AND/OR group to search for treatment article(s)
  on a topic of interest and assess results

34
QUESTIONS?

• Context for Assessing a Diagnosis Document
• Diagnosis Statistics
• Applying to a Scenario
• Small Group Portion