Christopher%20M.%20Bishop%20(2006):

1
????????????1???(??)

• Christopher M. Bishop (2006)
• Pattern Recognition and Machine Learning,
  Springer, pp.37-57

2
????????

• 1.5 ????
• 1.5.1 ????????
• 1.5.2 ????????
• 1.5.3 ???????
• 1.5.4 ?????
• 1.5.5 ??????????
• 1.6 ????
• 1.6.1 ??????????????

???
????
????
1????
3
????

• ??????????????,??????????????????????
• ????????????????????????????????????????
• ??????x?????????t
• ???x??????t??????????
• ex)????t?????
• ?????t???????
• ??????t?????
• ??????p(x,t)?????????????????????
• p(x,t)??????????????????

4
????????(1)

• ????????????X??????????????
• ??????x??????????
• ????t????????C1(t0), ???????C2(t1)
• ?????????????p(x, t)??????????,
• ???????????????????
• ??????????????????????,
• ??????????????????????????????
• ???????????????????
• ?????(decision)??????,??????????????
• ????????????????????????????????

5
????????(2)

• ??????X???x????????,???2????????????????????

(1.77)
??????/????(???????(????)??????) x???X????????(
?????????????) Ck????x???X????????(?????????)
6
????????(1)

• (1)??????????????????????
• ???????x?????????????1??????????????
• ??????????????????1???????????(decision
  region)???????Rk????Rk???????????Ck??????.
• ??????????????(?????decision boundary)????????(de
  cision surface)?????
• ?????????????,?????????????????????.

7
????????(2)

• ???
• ????????C1???????????C2???????????????????.???????
  ??

(1.78)
?????????x??? x0???
8
????????

• (2)????????????????????
• ???????????????,????????????????????????????
• ???????(loss function),?????(cost function)???

(1.80)
?
??
?
??
9
???????

• ????????????????????
• ????????????????????,????????(???)????????????????
  ???????
• ????????????

10
??????????(1)

• ????????????????????
• ????????x??????????y(x)?????.???,??L(t,
  y(x))??????.
• ?????

(1.86)

• ????????????????????????????,

(1.87)
11
??????????(2)

• ???EL??????y(x)?????
• ????????

(1.88)
(1.89)
????????? ????(regression function) ?????????????
???????????????????
12
????

• ????x????.??????????????????????,?????????????????
  
• ?????x????????????????
• ?????????????????????,?????????????

ex) ???,????????????????? ???,????????????????????
?????
??????????? ????????
13
???
????h()????????
(1.92)
?????
ex). ??????1?????????????
4
4
x
y
x,y
??p(x)
p(y)
p(x,y)
???h(x)
h(y)
h(x,y)
14
(????)?????? (entropy)

• ???????????????????????, ????????????????(1.92)?
  ??p(x)? ???????????????,????????????

(1.93)
?????????????? ???????(entropy)?
?????????,??????????????? ???????????????????????
??????? ????????????????? ???????????????????????
? ??????????????????????????????
15
??????????(2)

• 8????????a, b, c, d, e, f, g, h
• ???????1/8, 1/8 , 1/8 , 1/8 , 1/8 , 1/8 , 1/8 ,
  1/8
• ???????????

???

• 8????????a, b, c, d, e, f, g, h
• ???????1/2, 1/4 , 1/8 , 1/16 , 1/64 , 1/64 ,
  1/64 , 1/64
• ???????????

???
16
??????????(3)

• ?????????????????????????????????
• ???????????????

N???????????????????????????. ?????????????????N??
??????, ?????N-1??????,N?????????????????N!???? i?
?????ni!????????????,N???????????????
???(multiplicity)
(1.94)
17
?????
???(multiplicity)
(1.94)
ex). N8, ????4????
18
????????????

• ???????????????????????

(1.95)

• ???,ni/N?????????N?8????????,???????????????

(1.96)
19
????????????

• ???????????????????????

• ???,pi?????i??????????????
• ??????????,
• ?????????????
• ?????(?????microstate)
• ?ni/N???????????????
• ?????(?????macrostate)
• ???W?????????(weight)??????

(1.97)
20
??????????????

• ????????X???xi????????,p(Xxi)pi???.???????X?????
  ???

(1.98)

• ????????????????????
• ??,????????????

21
???????????????

• ????????????????
• H??????????????????????

(1.99)

• ????,M?xi?????????

• ?????????(????)

22
??????????

• ????x???p(x)?????
• ??,x????????????
• p(x)????????????,????????

(1.101)

• ???xi???????
• i????????????x??xi????????????????,xi??????????p(x
  i)????
• ????,????????????(1.101)??

(1.102)
23
????????
(1.102)

• ???(1.102)????????????????,??0????????

(1.103)

• ????????????????????
• ????????????????ln ??????
• ??????0???????????
• ????????????????????????????????????

24
?????????????????

• ????????????
• ????????????????????

(1.105)
(1.106)
(1.107)
25
?????????????????

• ??????,?????????0????

(1.108)

• ????

(1.109)

• ?????.??????????

(1.110)

• ????????????????????????????????

26
??????????????

• ???????????????????????????????????
• ?????p(x)???,???????q(x)??????????
• q(x)????x???????????????????????
• ????p(x)?????q(x)????x????????????
  ??(additional)??????????????

(1.113)

• ?????p(x)?q(x)???????????(relative entropy)
• ???????????????????????(Kullback-Leibler
  divergence)???????KL??????????? ??????

27
???

• ???,KL????????? ????,?????????????p(x)q(x)?
  ??,????????????
• ??,???(convex function)????????
• ??f(x)????????????????,??????????????????

(1.114)
28
???????????????????

• ???????????(1.114)?????f(x)???????xi????,
• ??????????.???S?1???.
• (1.115)???????????????????.
• ?i??xi?????????x??????????????
• ????.?????????????????????????????(1.113)?????????
  ?,
• ?????

(1.115)
(1.116)
(1.118)
29
KL?????????????(1)

• ??????????(???????????????)??????????,????????????
  ???????????????
• ??????????????,??????????,?????????????????????2??
  ??????????????????????????????
• ?????????p(x)?????????,????????????
• ??????????????????????q(x?)(???????????)?????????
  ?????
• ?????1?????p(x)?q(x?)????????????????????????????
  ?????????????
• ???,p(x)?????????????????????

30
KL?????????????(2)

• p(x)????????????????xnn1,..,N??????
• p(x)???????????????????????,
• ???.??????????????,???????????????????????????????
  ???.
• ???,KL?????????????????????????

(1.119)
31
?????

• 2??????x?y?????p(x, y)????
• ???????????????????????????? p(x, y)p(x)p(y)???
• ??????????,???????????????,??????????????KL???????
  ??????????,
• ?????x,y????????(mutual information)?????
• ??????y???????????x?????????????????????.
• ????????p(x)?x?????,p(xy)???????y????????????????
  ?.?????,???y??????????,x??????????????????????

(1.120)