Prediction Models in Medicine

1
Prediction Models in Medicine

• Clinical Decision Support
• The Road Ahead
• Chapter 10

2
Overview

• Prediction models currently in use in Clinical
  Decision Support
• What models are used (or not used) in the
  community
• How are the models evaluated
• Examples of currently used prediction models

3
Limited adaptation of learning algorithms in
practice

• Most clinical decision support systems do not use
  machine-learning/data mining techniques
• Data is not available or is not structured enough
• Learning techniques are not well disseminated or
  well evaluated enough
• Rules defined by experts are more understandable
  to clinicians

4
Model Preference

• Simple, understandable models are preferred
• Linear and Logistic Regression is by far the most
  popular
• SVMs, Neural Networks, and other sophisticated
  models are not very popular
• Unsupervised Learning is not used at all

5
Model Evaluation

• Discrimination
• How well the model discriminates positive and
  negative cases
• How large is
• P(1Positive Case)
• P(0Negative Case)
• Based on the ROC(Receiver operating
  characteristic) curve

6
Model Evaluation

• Calibration
• How close is the models estimated probability to
  the true underlying probability
• For logistic regression, calibration is assessed
  by Hosmer-Lemeshow goodness-of-fit test

7
Case Study 1 Prognosis of ICU Mortality

• APACHE (Acute Physiology and Chronic Health
  Evaluation) series of models
• Predict the individual patient's risk of hospital
  death, based on a variety of physiological
  variables
• History
• APACHE (1981) Expert-based scoring system
• APACHE II (1985) Logistic Regression on 5,815
  cases from 13 hospitals
• APACHE III (1991) Logistic Regression on 17,440
  cases based on 40 hospitals. Commercial product

8
Case Study 1 Prognosis of ICU Mortality

• Large number of reviews and external evaluations
  show good discrimination, but variable
  calibration
• Other systems (more popular in Europe)
• SAPS-II
• MPM-II
• Multiple studies compare LR to ANN
• Some studies suggest that the models are
  equivalent
• Some suggest that ANN achieves superior
  discrimination

9
Case Study 2 Cardiovascular Disease Risk

• Estimates the risk of developing future heart
  disease
• Based on most recent 10-year heart disease data
  from Framingham cohort (in US)
• Uses Logistic Regression
• External validation shows good discrimination and
  moderate calibration (but limited to similar
  demographic)
• The model is used to determine the risk factors
  for heart disease (used to generate guidelines
  for care)

10
Case Study 3 Pneumonia Severity of Illness Index

• Predicts the risk of death within 30 days for
  adult patients with pneumonia
• Developed by Pneumonia Patient Outcome Research
  Team (PORT) 1997
• Logistic Regression!
• The model was validated over 50,000 patients in
  275 US and Canadian hospitals
• Using this model, 26 to 31 percent of patients
  can be treated safely as outpatients
• Savings of more than 1.2 Billion dollars per year
  in US