A Tutorial on ROC Curve and AUC

1
A Tutorial on ROC Curve and AUC
2
Outline

• The Evaluation of Learning Algorithms
• ROC Curve
• AUC and Ranking
• Related Research Topics

3
Inductive Learning Algorithms

• Inductive Learning Algorithms
• Given training examples x1, , xn, where xi(A1,
  , An), and their classes ,, learn a
  classifier f'(x)?, with probability
  
• or
• f(x) underlying true classifier
• P(x) given an example xi, the probability of
  xi belonging to class

4
Evaluation Criteria

• Classification Accuracy
• The percentage of examples correctly classified
• Probability-based Ranking
• How close the result ranking to the underlying
  ranking.
• Probability Estimation
• The difference between P(xi) and

5
Probability of Class Membership

• an estimate of P(xi)
• Most traditional classifiers also produce
  probability of class membership as by-product.
• For a decision tree, it is the fraction of the
  examples of class in the leaf xi falls into.

6
Why Ranking

• Classification assign a class label to an
  example.
• Ranking rank a set of examples according to
  their scores.
• Ranking is more desirable in some applications.
• Direct marketing.

7
Outline

• The Evaluation of Learning Algorithms
• ROC Curve
• AUC and Ranking
• Related Research Topics

8
ROC Curve (1)

• True positive rate TP and false positive rate FP,
  of a classifier are

9
ROC Curve (2)

• Given a ranking
• examples x1 x2 x3 x4 x5
  x6 x7 x8 x9 x10
• 0.1 0.3 0.3 0.4 0.4 0.5
  0.6 0.7 0.8 0.9
• Classes
  
• Try setting different cutoff
• 
  TP5/5 FP4/5
• 
  TP5/5 FP3/5
• 
  TP1/5 FP0/5

10
ROC Curve (3)
11
Read ROC Curves

• A perfect ranking
• A ROC curve A is said to dominate another ROC
  curve B if A is always above and to the left of
  B.
• Loosely, A is better than B.
• No clear dominating relation between two ROC
  curves in many cases.

12
Outline

• The Evaluation of Learning Algorithms
• ROC Curve
• AUC and Ranking
• Related Research Topics

13
AUC (1)

• The area under the ROC curve (AUC).
• summary'' for comparing the two ROC curves.
• One ROC curve dominates the other, its AUC must
  be larger.
• Intuitively, the larger the AUC of a ROC, the
  better.

14
AUC (2)

• Given a classifier G, let
• estimated probability of ith positive example in
  class , i1, , n0
• estimated probability of ith negative example in
  class , i1, , n1

15
AUC (3)

• Rank the set g1, , gn1, f1, , fn0. Let ri
  be the rank of ith positive example.

16
AUC (4)

• AUC of G is given below

17
AUC(5)

• AUC reflects the quality of a ranking.
• For the perfect ranking,
• its AUC is 1.
• For the worst ranking,
• its AUC is 0.

18
AUC (6)

• Comparison between AUC and accuracy
• One example

Classifier1 AUC is 24/25, accuracy is
0.8 Classifier2 AUC is 16/25, accuracy is 0.8
19
AUC (7)

• One counter example

Classifier1 AUC is 0.889, accuracy is
0.667 Classifier2 AUC is 0.694, accuracy is
0.833
20
Outline

• The Evaluation of Learning Algorithms
• ROC Curve
• AUC and Ranking
• Related Research Topics

21
Related Research Topics

• What is the performance of traditional learning
  algorithms, such as decision trees, Bayesian
  networks in terms of AUC ?
• How to develop new algorithms that aim at maximum
  AUC scores?