Information Gain

1
??????7.??????

• ????????????? ?? ??

2
???????

• ??????
• ???????
• ??????
• Information Gain
• ???
• ????
• ???
• ????
• http//www.jaist.ac.jp/tsuruoka/lectures/

3
??????Chapter 3 of Mitchell, T., Machine
Learning (1997)

• ???(Decision Trees)
• Disjunction of conjunctions
• ??????????
• ?????
• ???????????
• ??
• ?????????????????
• ??????

4
?????

• Concept PlayTennis

Outlook
Sunny
Rain
Overcast
Humidity
Wind
Yes
High
Normal
Strong
Weak
No
Yes
No
Yes
5
??????????

• ??
• ltOutlook Sunny, Temperature Hot, Humidity
  High, Wind Stronggt

Outlook
Sunny
Rain
Overcast
Humidity
Wind
Yes
High
Normal
Strong
Weak
No
Yes
No
Yes
6
Disjunction of conjunctions

• (Outlook Sunny Humidity Normal)
• v (Outlook Overcast)
• v (Outlook Rain Wind Weak)

Outlook
Sunny
Rain
Overcast
Humidity
Wind
Yes
High
Normal
Strong
Weak
No
Yes
No
Yes
7
?????????

• ???????
• ?????????
• Disjunctive ??????????
• ???????????????????
• ?????????????????????

8
??????
Day Outlook Temperature Humidity Wind PlayTennis
D1 Sunny Hot High Weak No
D2 Sunny Hot High Strong No
D3 Overcast Hot High Weak Yes
D4 Rain Mild High Weak Yes
D5 Rain Cool Normal Weak Yes
D6 Rain Cool Normal Strong No
D7 Overcast Cool Normal Strong Yes
D8 Sunny Mild High Weak No
D9 Sunny Cool Normal Weak Yes
D10 Rain Mild Normal Weak Yes
D11 Sunny Mild Normal Strong Yes
D12 Overcast Mild High Strong Yes
D13 Overcast Hot Normal Weak Yes
D14 Rain Mild High Strong No
9
??????????

• ???????????????
• ???????????????????????????????????
• ??????????????????????

10
??????

• ????2????
• ????

11
Information Gain

• ??????????????????????????????????????

12
?
13
Information Gain ????
Humidity
Wind
High
Normal
Weak
Strong
14
??????????

• ????? information gain ????????

15
Outlook ?????????
D1,D2,,D14 9,5-
Outlook
Sunny
Rain
Overcast
D1,D2,D8,D9,D11 2,3-
D3,D7,D12,D13 4,0-
D4,D5,D6,D10,D14 3,2-
Yes
? (??)
?
16
???(Overfitting)

• ??????????
• ???????????????
• ??????????????????(!)
• ???
• ?????????????????????????????????????????
• ??????????????????????

17
Reduced Error Pruning

• ???(Pruning)??????????
• ?????????
• ????????????????
• Validation set ????????????
• Greedy ??????????????????????????????

18
Validation (development) set

• ??????????????????
• ???????????????

Training set
Training set
Validation set
Test set
Test set