Example of a Decision Tree

1
(No Transcript)
2
(No Transcript)
3
Example of a Decision Tree
Splitting Attributes
Refund
Yes
No
MarSt
NO
Married
Single, Divorced
TaxInc
NO
lt 80K
gt 80K
YES
NO
Model Decision Tree
Training Data
4
Another Example of Decision Tree
categorical
categorical
continuous
class
Single, Divorced
MarSt
Married
Refund
NO
No
Yes
TaxInc
lt 80K
gt 80K
YES
NO
There could be more than one tree that fits the
same data!
5
Decision Tree Classification Task
Decision Tree
6
(No Transcript)
7
(No Transcript)
8
(No Transcript)
9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
Decision Tree Induction is often based on
Information Theory
14
Information
15
DT Induction

• When all the marbles in the bowl are mixed up,
  little information is given.
• When the marbles in the bowl are all from one
  class and those in the other two classes are on
  either side, more information is given.
• Use this approach with DT Induction !

16
(No Transcript)
17
Information/Entropy

• Given probabilitites p1, p2, .., ps whose sum is
  1, Entropy is defined as
• Entropy measures the amount of randomness or
  surprise or uncertainty.
• Goal in classification
• no surprise
• entropy 0

18
Entropy
19
(No Transcript)
20
ID3

• Creates tree using information theory concepts
  and tries to reduce expected number of
  comparison..
• ID3 chooses split attribute with the highest
  information gain

21
C4.5

• ID3 favors attributes with large number of
  divisions
• Improved version of ID3
• Missing Data
• Continuous Data
• Pruning
• Rules
• GainRatio

22
CART

• Create Binary Tree
• Uses entropy
• Formula to choose split point, s, for node t
• PL,PR probability that a tuple in the training
  set will be on the left or right side of the tree.

23
Measure of Impurity GINI

• Gini Index for a given node t
• (NOTE p( j t) is the relative frequency of
  class j at node t).
• Maximum (1 - 1/nc) when records are equally
  distributed among all classes, implying least
  interesting information
• Minimum (0.0) when all records belong to one
  class, implying most interesting information

24
Examples for computing GINI
P(C1) 0/6 0 P(C2) 6/6 1 Gini 1
P(C1)2 P(C2)2 1 0 1 0
P(C1) 1/6 P(C2) 5/6 Gini 1
(1/6)2 (5/6)2 0.278
P(C1) 2/6 P(C2) 4/6 Gini 1
(2/6)2 (4/6)2 0.444
25
Splitting Based on GINI

• Used in CART, SLIQ, SPRINT.
• When a node p is split into k partitions
  (children), the quality of split is computed as,
• where, ni number of records at child i,
• n number of records at node p.

26
Stopping Criteria for Tree Induction

• Stop expanding a node when all the records belong
  to the same class
• Stop expanding a node when all the records have
  similar attribute values
• Early termination

27
Notes on Overfitting

• Overfitting results in decision trees that are
  more complex than necessary
• Training error no longer provides a good estimate
  of how well the tree will perform on previously
  unseen records
• Need new ways for estimating errors

28
Estimating Generalization Errors

• Re-substitution errors error on training (? e(t)
  )
• Generalization errors error on testing (? e(t))
• Methods for estimating generalization errors
• Optimistic approach e(t) e(t)
• Pessimistic approach
• For each leaf node e(t) (e(t)0.5)
• Total errors e(T) e(T) N ? 0.5 (N number
  of leaf nodes)
• For a tree with 30 leaf nodes and 10 errors on
  training (out of 1000 instances)
  Training error 10/1000 1
• Generalization error (10
  30?0.5)/1000 2.5
• Reduced error pruning (REP)
• uses validation data set to estimate
  generalization error

29
Minimum Description Length (MDL)

• Cost(Model,Data) Cost(DataModel) Cost(Model)
• Cost is the number of bits needed for encoding.
• Search for the least costly model.
• Cost(DataModel) encodes the misclassification
  errors.
• Cost(Model) uses node encoding (number of
  children) plus splitting condition encoding.

30
How to Address Overfitting

• Pre-Pruning (Early Stopping Rule)
• Stop the algorithm before it becomes a
  fully-grown tree
• Typical stopping conditions for a node
• Stop if all instances belong to the same class
• Stop if all the attribute values are the same
• More restrictive conditions
• Stop if number of instances is less than some
  user-specified threshold
• Stop if expanding the current node does not
  improve impurity measures (e.g., Gini or
  information gain)

31
How to Address Overfitting

• Post-pruning
• Grow decision tree to its entirety
• Trim the nodes of the decision tree in a
  bottom-up fashion
• If generalization error improves after trimming,
  replace sub-tree by a leaf node.
• Class label of leaf node is determined from
  majority class of instances in the sub-tree
• Can use MDL for post-pruning

32
Example of Post-Pruning
Training Error (Before splitting)
10/30 Pessimistic error (10 0.5)/30
10.5/30 Training Error (After splitting)
9/30 Pessimistic error (After splitting) (9
4 ? 0.5)/30 11/30 PRUNE!
Class Yes 20
Class No 10
Error 10/30 Error 10/30
Class Yes 8
Class No 4
Class Yes 3
Class No 4
Class Yes 4
Class No 1
Class Yes 5
Class No 1
33
Examples of Post-pruning
Case 1

• Optimistic error?
• Pessimistic error?
• Reduced error pruning?

Dont prune for both cases
Dont prune case 1, prune case 2
Case 2
Depends on validation set
34
(No Transcript)
35
(No Transcript)