Testing Predictive Performance of Ecological Niche Models

1
Testing Predictive Performance of Ecological
Niche Models
A. Townsend Peterson, STOLEN FROM Richard Pearson
2
Niche Model Validation

• Diverse challenges
• Not a single loss function or optimality
  criterion
• Different uses demand different criteria
• In particular, relative weights applied to
  omission and commission errors in evaluating
  models
• Nakamura which way is relevant to adopt is not
  a mathematical question, but rather a question
  for the user
• Asymmetric loss functions

3
(No Transcript)
4
Where do I get testing data????
5
Model calibration and evaluation strategies
resubstitution
Projection
Calibration
Same region Different region Different
time Different resolution
All available data
100
Evaluation
(after Araújo et al. 2005 Gl. Ch. Biol.)
6
Model calibration and evaluation strategies
independent validation
Projection
Same region Different region Different
time Different resolution
All available data
Calibration
100
Evaluation
(after Araújo et al. 2005 Gl. Ch. Biol.)
7
Model calibration and evaluation strategies data
splitting
Projection
Calibration
Calibration data
Same region Different region Different
time Different resolution
70
Test data
30
Evaluation
(after Araújo et al. 2005 Gl. Ch. Biol.)
8
Types of Error
9
The four types of results that are possible when
testing a distribution model
(see Pearson NCEP module 2007)
10
Presence-absence confusion matrix
Recorded (or assumed) absent
Recorded present
Predicted present
a (true positive)
b (false positive)
Predicted absent
c (false negative)
d (true negative)
11
Thresholding
12
(No Transcript)
13
Selecting a decision threshold (p/a data)
(Liu et al. 2005 Ecography 29385-393)
14
Selecting a decision threshold (p/a data)
15
Selecting a decision threshold (p/a data)
16
Selecting a decision threshold (p-o data)
17
Threshold-dependent Tests( loss functions)
18
The four types of results that are possible when
testing a distribution model
(see Pearson NCEP module 2007)
19
Presence-absence test statistics
Recorded (or assumed) absent
Recorded present
Predicted present
a (true positive)
b (false positive)
Predicted absent
c (false negative)
d (true negative)
Proportion () correctly predicted (or
accuracy, or correct classification rate)
(a d)/(a b c d)
20
Presence-absence test statistics
Recorded (or assumed) absent
Recorded present
Predicted present
a (true positive)
b (false positive)
Predicted absent
c (false negative)
d (true negative)
Cohens Kappa
21
Presence-only test statistics
Recorded (or assumed) absent
Recorded present
Predicted present
a (true positive)
b (false positive)
Predicted absent
c (false negative)
d (true negative)
Proportion of observed presences correctly
predicted (or sensitivity, or true positive
fraction) a/(a c)
22
Presence-only test statistics
Recorded (or assumed) absent
Recorded present
Predicted present
a (true positive)
b (false positive)
Predicted absent
c (false negative)
d (true negative)
Proportion of observed presences correctly
predicted (or sensitivity, or true positive
fraction) a/(a c)
Proportion of observed presences incorrectly
predicted (or omission rate, or false negative
fraction) c/(a c)
23
Presence-only test statisticstesting for
statistical significance
U. sikorae
U. sikorae
Success rate 4 from 7 Proportion predicted
present 0.231 Binomial p 0.0546
Success rate 6 from 7 Proportion predicted
present 0.339 Binomial p 0.008
24
Absence-only test statistics
Recorded (or assumed) absent
Recorded present
Predicted present
a (true positive)
b (false positive)
Predicted absent
c (false negative)
d (true negative)
Proportion of observed (or assumed) absences
correctly predicted (or specificity, or true
negative fraction) d/(b d)
25
Absence-only test statistics
Recorded (or assumed) absent
Recorded present
Predicted present
a (true positive)
b (false positive)
Predicted absent
c (false negative)
d (true negative)
Proportion of observed (or assumed) absences
correctly predicted (or specificity, or true
negative fraction) d/(b d)
Proportion of observed (or assumed) absences
incorrectly predicted (or commission rate, or
false positive fraction) b/(b d)
26
AUC a threshold-independent test statistic
(1 omission rate)
(fraction of absences predicted present)
sensitivity a/(ac)
specificity d/(bd)
27
Threshold-independent assessment The Receiver
Operating Characteristic (ROC) Curve
A
B
set of absences
set of presences
1
Frequency
1
0
Predicted probability of occurrence
sensitivity
C
set of absences
set of presences
Frequency
0
1
0
0
1
Predicted probability of occurrence
1 - specificity
(check out http//www.anaesthetist.com/mnm/stats/
roc/Findex.htm)