Support Vector Regression

1
??? ???????
2
??(1/1)

• ? ?
• ??
• ???????????
• ?????
• ???
• ?????????
• ?????????
• ??????????????
• Support Vector Regression
• ????

3
1 ??(1/13)

• 1 ??
• ??????????????????????,??????????????????,????????
  .
• ??????????????????????,???????????????,???????????
  ?
• ??????,???????????,??????????,??????????????,?????
  ????????????????????????
• ???????????(??)???

4
1 ??(2/13)

• ????????????,???????(???)?????????,???????????????
  ????,??????????????
• ??????,????(??)??,??????????????,?????????????,???
  ??????????????????
• ??,???,???????????????
• ????,?????????????????????,???????????????
• ???(??)??????
• ???????,?ANN
• ??????

5
1 ??(3/13)

• A. ???????(??)??????.
• ???????NN???,???????????????????????.
• ??????????????,??????,???????(????)????,??????????
  ??.
• ???????????,
• ??,???????????,?????????.
• ??,????????????????????????,????????????????.
• ???????????????????.

6
1 ??(4/13)

• ???????,?????????,????????
• ??????????????,???????????????????????,???????????
  ??????????(???????)???
• ?????????,???????????????????,????????????????????
  ???,?????????????
• ??,????????????????ANN??????????????.

7
1 ??(5/13)

• B. ???????,?ANN.
• ?????????????????,??????????????.
• ??,???????????????.
• C. ??????.
• ??????????????????????,?????????????????
• ??,????????????(??????????)????????,??????????????
  ?????
• ???????????????,???????????????,?????????

8
1 ??(6/13)

• ??????(Statistical Learning Theory, SLT)
  ???V.Vapnik?60??????????????????,?70??????????????
  ?,???????????????,????????

• ?90????,?????????????,???NN????????????????,??????
  ??????.

9
1 ??(7/13)

• SLT???????????????????
• ????????????
• ???????VC?,???????????????????????VC?
• SLT????????,?????????????????????,??????????,?????
  ???,??1992??????????(Support Vector Machine, SVM)
  ???????????????????????.
• ?????SLT??????????????????????????????,???????????
  ??SVM????????????????,?????????????20????????????

10
1 ??(8/13)

• SVM????SLT???????????????,??SLT??????,????????.
• ??( Cristianini Shawe-Taylor ,2000 ) Support
  Vector Machines are a system for efficiently
  training linear learning machines in
  kernel-induced feature spaces, while respecting
  the insights of generalisation theory and
  exploiting optimisation theory.
• SVM becomes popular because of its success in
  handwritten digit recognition
• 1.1 test error rate for SVM. This is the same as
  the error rates of a carefully constructed neural
  network, LeNet 4.

11
1 ??(9/13)

• ??,SVM????????????????????????????.
• ??,???????,??????????,????????????????????,SVM????
  ????????????????????.
• SVM is now regarded as an important example of
  kernel methods, arguably the hottest area in
  machine learning.

12
1 ??(10/13)

• SVMs combine three important ideas
• Apply optimization algorithms from Operations
  Reseach
• Linear Programming and Quadratic Programming
• Implicit feature transformation using kernels
  Implicit feature transformation using kernels
• Control of overfitting by maximizing the margin

13
1 ??(11/13)

• ??,?SVM?????????????,??????????.??
• ???????????????,???????????
• ??SVM?????????????(????????????????SVM??????????)
  
• ????????????VC????????????
• SVM???????????????????????????.

14
1 ??(12/13)

• ??,?????????,??SLT?SVM???,???????80????ANN????????
  ????
• ??,?????,SLT????????????????(????ANN?????????),???
  ??????????????????.
• ??,??????,SLT?????????????,???????????????????????
  ????

15
1 ??(13/13)

• ??????SVM,?????
• ???????????
• ?????
• ???
• ?????????
• SVM??????????

16
2 SLT?????(1/4)

• 2 ???????????
• ????????,??????(SLT)??????????????????????.
• ???????????????????????,??????????????????????????
  ,?????????????????????.
• V. Vapnik???????????????????,???????,?????????????
  ,???NN????????????????,SLT????????????.

17
2 SLT?????(2/4)

• SLT??????
• ?????????????????????????????
• ???????????????,??????????????
• ?????????????????????
• ????????,??????????????????
• SVM??,????????????????????

18
2 SLT?????(3/4)

• SLT?????????VC?(VC Dimension)??,??????????????????
  ?????????????,????????????????????????????????????
  ????.
• SLT?????????????????,?????????????????????.
• ?????????????,?????????????????(??NN??????????????
  ?)
• ??,????????????????????--SVM,?????????????????.

19
2 SLT?????(4/4)

• ??????,SLT?SVM?????NN??????????,??????????????????
  ?.
• ????SLT?????,?????
• ????
• ???????????
• ??????????
• VC????
• ??????????

20
2.1 ????(1/3)

• 2.1 ????
• ??????????????????????????????.
• ??????????????y?x???????????,????????????F(x,y).
• ??????????n??????????
• x1, y1, x2, y2, , xn, yn
• ?????f(x,w)?,???????f(x,w0)?????????,??????????

21
2.1 ????(2/3)

• ??w????????
• f(x,w)???????,????????????.
• L(y,f(x,w))????f(x,w)?y??????????,????????,L(y,f(x
  ,w))??????

22
2.1 ????(3/3)

• ????????????????F(x,y),?????????,???????????????,?
  ?????????

• ????????????????,???????,???R(w)???.

23
2.2 ???????????(1/2)

• 2.2 ???????????
• ???????????????,??????????????????????????,???????
  ?????.
• ?? ???????L(y,w)???????F(y),??????????????????,??
  ??????????

24
2.2 ???????????(2/2)

• ???????????????????,?????????????L(y,wl),l1,2,,?
  ????????????????????????.
• ????????????????????,

25
2.3 ??????????(1/3)

• 2.3 ??????????
• SLT????? ??????????????????????????,??????.
• ???,???????????????????????,??????
• ????Remp(w) ?????R(w)?????1-??????????

??n????, h?????VC?.
26
2.3 ??????????(2/3)

• ????,???????????????
• ???????,
• ?????????(VC confidence).
• ??????????????????????,?????????????,???????VC?h??
  ????l??.
• ???????????

???(h,l)??h?????.
27
2.3 ??????????(3/3)

• ??,?????????,??????????,VC???,???????,????????????
  ??????????.
• ??????,??VC???????,?????????.

28
2.4 VC????(1/7)

• 2.4 VC????
• Anthony?VC????
• ?F????n????X???0,1????,?F?VC??X???E??????,??E??
  
• ????S?E,?????fs?F,???x?S?fs(x)1, x?S?x?E?fs(x)
  0.
• VC???????F??????,???????,?????????,???????????????
  ?????F????????????????.
• ??????,???????????,???????VC?????????VC?.

29
2.4 VC????(2/7)

• ????????????VC??????.
• ????(dichotomy)???
• A dichotomy of a set S is a partition of S into
  two disjoint subsets.
• ??(shatter)???
• A set of instances S is shattered by hypothesis
  space H if and only if for every dichotomy of S
  there exists some hypothesis in H consistent with
  this dichotomy.

30
2.4 VC????(3/7)

• VC????
• ????????,??????h????????????????????2h?????,??????
  ??h?????.
• ????VC??????????????h.
• ???????????????????,?????VC??????.
• ??????VC???????????????????????.

31
2.4 VC????(4/7)

• ??,??2???????????
• yw0w1x1w2x2
• ??????????(??)???????3,?????.

• ??,????????VC???3.

32
2.4 VC????(5/7)

• ????,??n???????????
• yw0w1x1wnxn
• ?VC??n1.

33
2.4 VC????(6/7)

• ??,?????,2???????????????????(??)???????4.

• ??,????????VC???4.

34
2.4 VC????(7/7)

• VC????????????.
• ????,VC???????????,????????.
• ??????????????VC??????,????????????????.
• ??
• ?n??????????????????VC??n1,
• ?f(x,a)sin(ax)?VC??????.
• ??????????????VC????SLT?????????,

35
2.5 ??????????(1/4)

• 2.5 ??????????
• ???NN???,????????????????????,?????,??????????????
  ????????.
• ?????,??????????,???????,???????????.
• ???????,??????????????,???????,???????????????????
  ?.
• ???????????????????????.

36
2.5 ??????????(2/4)

• ??????BP?????????????????.
• ??????,?????????????????.
• ???,????????????????????????,??????????????.
• ????BP???????????(???,??,???).
• ??????????,?BP???????,????50?,???????????????????
  ??,??????.
• ???????

37
2.5 ??????????(3/4)

• ???????,??????????,????????,???????5????????????,?
  ????????????????????????????.??????????
• ????(???????)??????????????,???????.
• ???????????(????????),???????????????.

38
2.5 ??????????(4/4)

• ?????????,???????????.
• ??,????????????,?????????????(???????)
• ?????????????????,?????????????????????????????.
• ??,????????,?????????????????????????,????????????
  ????,??????????.

39
3 SVM(1/3)

• 3 ?????
• SVM??????SLT?VC????????????????,??????????
• ??????(?????????????)?
• ????(??????????????)
• ????????,???????????.
• SVM??????????
• ?????????????,?????????????????????????????????

40
3 SVM(2/3)

• ??????????????????,?????,??????????,????NN????????
  ??????
• ????????????????????????,?????????????????????????
  ????.
• ??????????????????????????,?????????????,?????????
  ????.
• ?SVM???,???????????,??????????????????RBF?????????
  ??????????.

41
3 SVM(3/3)

• ??????SVM,?????
• ?????SVM
• ??????SVM

42
3.1 ?????SVM(1/4)

• 3.1 ?????SVM
• ?????????????
• ?n????m?????(x1,y1), (x2,y2) (xm,ym),
• ??xi?n????????
• yi?-1,1????????.???-1??I?,?1??II?.
• ?????n???????????
• f(x)wTx-b0
• ????????

43
3.1 ?????SVM(2/4)

• ???????,????????????,??????????.
• ?????,?????????,???????????????????BP??RBF??,?????
  ??
• ??????????????????????,
• ????????,????????
• ???????,???????????
• ???????,????????
• ???BP??RBF???,????????????

44
3.1 ?????SVM(3/4)
BP
RBF
45
3.1 ?????SVM(4/4)

• ?????????????????,??????????????????.
• ???????.
• ?????,???????????.
• ??,???????,????,?????????????????.
• ????SLT,??????????????????,???SVM.
• ?????SVM
• ??????? SVM

46
3.1 ?????SVM--?????SVM(1/5)

• A. ?????SVM
• ??SLT,??????????????????,???SVM.
• SVM????????????
• ?n????m?????(x1,y1), (x2,y2) (xm,ym),
• ?????n????????
• ??????(????)
• ??????(????)
• ??????
• f(x)wTx-b0

47
3.1 ?????SVM --?????SVM(2/5)

• ?????????,????????,??????????????????????,????????
  ??????????.
• ????????(b)????(a)?(c)??????,?????????????????????
  ??????.

(c)
(a)
(b)
48
3.1 ?????SVM --?????SVM(3/5)

• ?????????????????????.?????,

• ????????????
• r(wTxb)/w
• ??,???????????

??????????????? r1/w
49
3.1 ?????SVM --?????SVM(4/5)

• ??,SVM???????????????????????

????????

• SVM???????????????,????????,????????????,?????????
  ??.

50
3.1 ?????SVM --?????SVM(5/5)

• After learning both RBFN and BP decision surfaces
  might not be at the optimal position.
• For example, as shown in the figure, both
  learning rules will not perform further
  iterations (learning) since the error criterion
  is satisfied

51
3.1 ?????SVM ???????SVM(1/16)

• B. ???????SVM
• ????????,?????????????

• ????.
• ?????????????????,????????????????.
• ????????????,???????????

52
3.1 ?????SVM ???????SVM(2/16)
53
3.1 ?????SVM ???????SVM(3/16)

• ??,??????,?????????????.
• ??,???????????????,??????????????.
• ???,???????????????????????????.
• ???????????????????????????

54
3.1 ?????SVM ???????SVM(4/16)
55
3.1 ?????SVM ???????SVM(5/16)

• ??????????????????????i,??????????

????????
??,SVM???????????????????
56
3.1 ?????SVM ???????SVM(6/16)

• ????

• ??Parameter C is tradeoff parameter between error
  and margin and can be viewed as a way to control
  overfitting.
• C??,???????,??????????????,???????,?????
• ??????????????????????The Soft Margin Hyperplane
  incorporating slack variables?

57
3.1 ?????SVM ???????SVM(6/16)

• ????????????

??????C1?104???????
58
3.1 ?????SVM ???????SVM(7/16)

• ?????????????,???????????.
• ??????????

• ???i??i???????,??
• ?i, ?i?0
• ??????????Kuhn-Tucker??,?

59
3.1 ?????SVM ???????SVM(8/16)
60
3.1 ?????SVM ???????SVM(9/16)

• ??

• ?????i ,????w,b,?i, ?i

61
3.1 ?????SVM ???????SVM(10/16)

• ?????????????L?,?

• ??w,b,?i, ?i????i???,????????i.
• ??,SVM??????????????????

????????
62
3.1 ?????SVM ???????SVM(11/16)

• In practice, solving the optimization problem
  involved computing the inner products xiTxj
  between all training points!
• SVM??????????????????????????????????,?????.
• ????,????????(??????)?i???,???????????.
• ????(?i???)??????(?i??)??????

63
3.1 ?????SVM ???????SVM(12/16)
64
3.1 ?????SVM ???????SVM(13/16)

• ????????????????

• ????????????????.
• b?????,??????????(??(1)????)??,??????????????????
  ?.
• Notice that ???? relies on an inner product
  between the test point x and the support vectors
  (learning sample)xi ?

65
3.1 ?????SVM ???????SVM(14/16)

• Characteristics of the Solution of SVM are
• Many of the ai are zero
• w is a linear combination of a small number of
  data
• Sparse representation
• xi with non-zero ai are called support vectors
  (SV)
• The decision boundary is determined only by the
  SV
• Let tj (j1, ..., s) be the indices of the s
  support vectors. We can write

66
3.1 ?????SVM ???????SVM(15/16)

• For testing with a new data z
• Compute the classifying function

• and classify z as class 1 if the sum is positive,
  and class 2 otherwise
• There are theoretical upper bounds on the error
  on unseen data for SVM
• The larger the margin, the smaller the bound
• The smaller the number of SV, the smaller the
  bound

67
3.1 ?????SVM ???????SVM(16/16)

• Note that in both training and testing, the data
  are referenced only as inner product, xi ? xj
• This is important for generalizing to the
  non-linear case

68
3.2 ??????SVM(1/12)

• 3.2 ??????SVM
• ???????????????????,?????????????????????.
• ???,?????????????????????

• ????????????????.
• ??????2???.

69
3.2 ??????SVM(2/12)

• ????????,?????x???????????H,??H????????.

70
3.2 ??????SVM(3/12)

• ??????????????????????????,?????
• ??(Cover's theorem) A complex pattern-classificat
  ion problem cast in a high-dimensional space
  nonlinearly is more likely to be linearly
  separable than in a low-dimensional space.
• A binary classification is ?-separable if there
  is an m-dimensional function vector ? that cast
  the inputs into a m-dimensional space,
• where the classification is linearly separable by
  the hyperplane wT?(x) 0, where w is the weight
  vector associated to an output neuron.

71
3.2 ??????SVM(4/12)

• This hyperplane is called the separation surface
  of the network.
• A corollary of Cover's theorem
• Corollary In a space of dimensionality of m, the
  expected maximum number of randomly assigned
  vectors that are linearly separable is 2m.

72
3.2 ??????SVM(5/12)

• ????
• ?(x) Rn?H
• ???x???????H??????.
• ??????(x)??????x,??????????????????????????????

???????????????
73
3.2 ??????SVM(6/12)

• ???????,????????????????x??????(x)?????,??????????
  ????(x).
• ??,???,????????H??????????(x)????,????????????????
  ?????,????????????(x)???.
• ??????,???????K(xi,xj)?T(xi)?(xj)??Mercer??,?????
  ???????.
• ??,?????????????????????????????????,???????????.
• ??????????????????????SVM.

74
3.2 ??????SVM(7/12)

• SVM??????????????????????????
• ????????,????????????????,??????????
• ????????????(???????????),??????????.
• ??,????????????????????????.
• SVM????????????NN,???s?????????,??????????????,???
  ??.

75
3.2 ??????SVM(8/12)
76
3.2 ??????SVM(9/12)

• ?????K(xi,xj),?????SVM????????????????????

• ??,??????SVM?????????????.
• In practical use of SVM, only the kernel function
  (and not f(.)) is specified.
• ????????????????????.

77
3.2 ??????SVM(10/12)

• ???????????????.
• ??

??????Lagrange??????

• ??KKT???

78
3.2 ??????SVM(11/12)

• ?i (?i ? C ) 0 ? i
• ??,????????????
• ??i0, ??i?0, ?i0 ? (Fi?bi)yi?0
• ?0lt?iltC, ??i0, ?i0 ? (Fi?bi)yi0
• ??iC, ??i0, ?i?0 ? (Fi?bi)yi?0
• ??KKT??????????????,????????????????????KKT???????
  ????.

79
3.2 ??????SVM(12/12)

• Steps for Classification with SVM
• Prepare the pattern matrix
• Select the kernel function to use
• Select the parameter of the kernel function and
  the value of C
• You can use the values suggested by the SVM
  software, or you can set apart a validation set
  to determine the values of the parameter
• Execute the training algorithm and obtain the ai
• Unseen data can be classified using the ai and
  the support vectors

80
4 ???(1/8)

• 4 ???
• ??????,SVM???????????????????????????????,????????
  ??????,????????????????????.
• K(xi,xj)?T(xi)?(xj)????SVM?????????????
• 1. Must be symmetric
• K(xi,xj)K(xj,xi)
• 2. Must satisfy Cauchy-Schwarz inequality
• K2(xi,xj)?K(xi,xi)K(xj,xj)

81
4 ???(2/8)

• 3. ??Mercers Theorem,?

• must be positive semi-definite,
• ??????,?????????

K(xi,xj)exp-xi-xj2/?2 K(xi,xj)fT(xi)f(xj)
K(xi,xj)K1(xi,xj)K2(xi,xj) K(xi,xj)?K1(xi,xj)
?gt0
82
4 ???(3/8)

• K(xi,xj)expK1(xi,xj)
• K(xi,xj)K1(xi,xj)K2(xi,xj)
• ???????
• ??????
• K(x,xi)xTxi
• ??????
• K(x,xi)(xTxi1)d
• ?????d???????

83
4 ???(4/8)

• ?????
• K(x,xi)exp-x-xi2/?2
• ?????????????

• ??kr(x-xi) ????????????x-xi.
• ?????,?????sigmoid??
• K(x,xi)tanh(b1xTxib2)
• ????????????

84
4 ???(5/8)

• ?????
• ??1999?Amari?Wu?????????????,??????????????????,??
  ?????????.
• ????????,??????????K(xi,xj)???????????(x).
• ???,?????????,????????????.
• ?2???????? K(x,xi)(xTxi1)2 ,?????????????(x)?

85
4 ???(6/8)

• ???,Mercer????????K(xi,xj)???????????(x)????.
• ??( Mercer??). ???L2???????K(x,z)??????

???k(x)????????,?????????????0?L2??f(x),????
86
4 ???(7/8)

• ???K(xi,xj)???????????????????????.
• Kernel function can be thought of as a similarity
  measure between the input objects
• However, not all similarity measure can be used
  as kernel function.
• The kernel function needs to satisfy the Mercer
  function, i.e., the function is
  positive-definite
• This has the consequence that the kernel matrix,
  where the (i,j)-th entry is the K(xi, xj), is
  always positive definite

87
4 ???(8/8)

• Note that xi needs not be vectorial for the
  kernel function to exist.
• This opens up enormous opportunities for
  classification with sequences, graphs, etc., by
  SVM

88
5 SVM????(1/3)

• 5 SVM????
• ??SVM????????????????????????????????,???,????SVM?
  ????,?????????????????,???????.
• ??SVM?????????????????????,????????????????,??
• ????????
• ???????????
• ????????
• ??.

89
5 SVM????(2/3)

• ????????????????????????????????????
• ??,SVM??????????????,?????????,???????.
• ??,????????4000?,???????K(xi,xj)????128???.
• ??,SVM???????????????????,?????,????????????????.

90
5 SVM????(3/3)

• SVM????????????????????,??????????????????????????
  ????.
• ???????????????????
• ????????????,????????,?????????,???????????????.
• ???????????????,???????
• ???
• ??????????
• ????????
• ??????

91
5 SVM????--???(1/2)

• A. ???
• ??????????(chunking algorithm).
• ?????????????,???Lagrange???????????????????.
• ??????????,?????????????,??????????????,??????????
  ?(?Lagrange??)??.
• ???????????,???????????????????????????.

92
5 SVM????--???(2/2)

• ??????
• ??????????????????,??????????,
• ???????????????,????????(??????KKT??)???(???????)?
  ??????????????????????,??????.
• ??????????????.
• ???????????????????,???????????????.
• ??,???????????????,???????????,???????????,???????
  ????.

93
5 SVM????--??????????(1/5)

• B. ??????????
• ???????????????????????
• ???????????????????????,????????????????????????
  ??????????????,
• ???????????????????,????????????,????????????????.

94
5 SVM????--??????????(2/5)

• ?????????????????????
• ???????????????????????,
• ???????????????????????,???????????????,??????????
  ??Lagrange?????????????,????????????0.
• ??????????????????????????(??????????????).
• ??,???????????????????????????????????????????.

95
5 SVM????--??????????(3/5)

• ??????????????Osuna?????,?????????????.??????
• ??????????B?N,??B????????????SVM??,??N??????Lagran
  ge??????.
• ??,?????B???Lagrange???????i(??i
  0,i?B)???N????j(??i0,j?N)??,???????????????(????
  ?????)
• ??,??????????
• (Fi?bi)yi?0
• ?,?????????????.

96
5 SVM????--??????????(4/5)

• ??????????????
• ????B,?????
• ????????i,i?B?b,???j0,j?N
• ??Fj,j?N?????????
• (Fi?bi)yi?0
• ???j,?B????i0???i??,???????.
• Osuna???????????????,?????????????????????????.

97
5 SVM????--??????????(5/5)

• ??????, Osuna??????B???????.
• ????????????????,??????B???,?????????.
• ?????????????B??,????,???????????.
• ????????B????????????,?????????B??????,????????.

98
5 SVM????--??????(1/5)

• C. ??????
• ????,???????????????????????????.
• Joachims???????????????????????????.
• ??John C. Platt????????(Sequential Minimal
  Optimization, SMO)??.
• ????????????????--????.
• ??????????????????????????????Lagrange??????.

99
5 SVM????--??????(2/5)

• ????????,???????????????????????,?????????????????
  ??????????????.
• ??,???????????????????
• ??,Platt??????????????????????????,???????????????
  ????.
• ????????????,SMO?????????????.

100
5 SVM????--??????(3/5)

• ?????????????????,SMO????????????2,???????????????
  ?.
• ??SMO???????????????????,????????????.
• ??,SMO????????????????????,??,SMO???????.

101
5 SVM????--??????(4/5)

• SMO??????????????,??????????.
• SMO??????????b?,?????????????(?????0lt?iltC???,?????
  ?????),??SMO????????????????????
• ??,?????????b????????????????????,???b????????????
  ?,????????????????????KKT?????,???????????.

102
5 SVM????--??????(5/5)

• ????????????????????????.
• ?????????????????????,???????????,??????????????,
• ??,???????????????????????,?????????????????????.
• Freitas???????????????????.

103
5 SVM????--??????(1/9)

• D. ??????
• ????SVM??????????,??????????????????.
• ???????.
• ??????????????????????,???SVM?????????.
• ?SVM????,??????,SVM?????????????,?????????????.

• ??, ???????SVM???????????? ??????
  .

104
5 SVM????--??????(2/9)

• ??????,???C?i??w?????,?????????????????????yi/C,??
  ???????????????,??????????????,???????????????????
  ???.

• ??????? (Successive Over Relaxation, SOR)
  ????????????b2,????????????
  ,??????????????.

• ????????????????????????,??????.
• ??????????,?????Lagrange???????,?????????????,????
  ?????.

105
5 SVM????--??????(3/9)

• ????(????,Least Squares,LS)SVM?LS?????SVM?,??????

?????

• ?????Lagrange??,???KT????,?????????.
• ???????????????.
• ??????????????.

106
5 SVM????--??????(4/9)

• ????(Incremental Learning)??
• ??????????????????,??????????????????????.
• ??,????????????????????????????,?????????????.
• ??SVM??????????????.

107
5 SVM????--??????(5/9)

• Syed????SVM??????.
• ??????????????SVM??,????????????????????????,?????
  ???,???????,?????????????,????????.
• ???????????????.
• ??????????SVM????,Mettera?????????Adatron????SVM??
  ??,?????????????????????????.
• ?Adatron???????Kernel-Adatron??,Frieb?????????????
  ??,???????,???????????.

108
5 SVM????--??????(6/9)

• Cauwenburghs??????????????????????,??????????????L
  agrange????????????????????.
• ??????????????SVM???????-ISVM.
• ????????????????????????,?????????????????.
• Fung???????????SVM.
• ???????????SVM??,?????????.

109
5 SVM????--??????(7/9)

• ????
• ????????????????,?SVM???????????.
• ????? (Nearest Point Algorithm,
  NPA)????????????,?????????????????????????????????
  ??.
• ????,?????????,?????????????????,????????????.
• ???????????????????????.

110
5 SVM????--??????(8/9)

• ??????????????????SVM??????????Hessian?????.
• ???????????,???????,???????????????,???????.
• Fang??????????????,??????????????,?????????SVM????
  ?????.
• ??????SVM??????????,???????????????????????,??????
  ??.

111
5 SVM????--??????(9/9)

• A list of SVM implementation can be found at
• http//www.kernel-machines.org/software.html
• Some implementation (such as LIBSVM) can handle
  multi-class classification
• SVMLight is among one of the earliest
  implementation of SVM
• Several Matlab toolboxes for SVM are also
  available

112
6 SVM????(1/6)

• 6 ?????????
• SVMs Key Ideas, main contributions to learning
  theory, are
• maximize the margin between positive and negative
  examples and optimal classification hyperplane
• the kernel trick and nonlinear classification
• non-linear Kernels map examples into a new,
  hihg-dimension space in which these examples are
  linear discriminable.
• Other contributions are
• Penalize errors in non-separable case
• Only the support vectors contribute to the
  solution

113
6 SVM????(2/6)

• SVM?????
• ??
• SVM?????????????????,???????????(?????),??????????
  ???????????????????????.
• ??????,???????????.
• ??SVM???????????????,????????????????,??????????.
• ??SVM?NN?????????.

114
6 SVM????(3/6)

• Given a kernel and a C, there is one unique
  solution
• Kernels allow very flexible hypotheses
• The kernel trick allows for a varying complexity
  of the classifier
• Training is relatively easy
• No local optimal, unlike in neural networks
• Tradeoff between classifier complexity and error
  can be controlled explicitly
• Non-traditional data like strings and trees can
  be used as input to SVM, instead of feature
  vectors
• variable-sized hypothesis space sized

115
6 SVM????(4/6)

• polynomial-time exact optimization rather than
  approximate methods
• unlike decision trees and neural networks
• The kernel trick allows for especially engineered
  representations for problems
• No strict data model is required (when you can
  assume it, then use it)
• The foundation of the SVC is pretty solid (when
  the slack-variables are not used)
• An error estimate is available using just the
  training data (but it is a pretty loose estimate,
  and cross-validation is still required to
  optimize K or C)

116
6 SVM????(5/6)

• ??
• Need to choose a good kernel function
• ???????????????F,????F???????????????,??????????.
• Very large problems are computationally
  intractable
• The kernel and C have to be optimized
• ?????????,SVM????????????????????????.

117
6 SVM????(6/6)

• ???????????,????????,?????????,???????????????????
  ??,????????????????(although more and more
  specialized optimizers appear).
• ??????O(m3)
• problems with more than 20,000 examples are very
  difficult to solve
• ??????O(m2)
• ???SVM?????. 
• Batch algorithm
• The SVC tends to have problems with highly
  overlapping classes

118
7 SVM??????????(1/7)

• 7 SVM??????????
• SVM?????
• ??????
• ????????SVM??????,???????????????????????????????.
  
• ????????
• SVM??????????,?????????????????.
• ??K(K????)?SVM??????????????????,????SVM??????????
  ??,??????????.

119
7 SVM??????????(2/7)

• SVM????????
• ????SVM?????,????SVM?????.
• ???????
• ???SVM????????????????,?????????????????????,?????
  ??,????????????.
• SVM??????
• ????SVM?????????,????SVM?????,?SVM?QP????????.
• ?SVM??????,????SVM?????????,????SVM?????,?SVM?QP??
  ??????.

120
7 SVM??????????(3/7)

• ????
• ??,SVM??????(Batch Learning),??SVM???????????????.
  
• ??,????,?????????????.
• ???SVM????????????????????????,?SVM????????????.
• ??????SVM,?????????,?????????????,?????????????.
• ???????,????????????????LS-SVM.

121
7 SVM??????????(4/7)

• LS-SVR????????????????????,????????,??????????????
  ???????.
• ??????????????,LS-SVM??????????????,?????,????????
  ?.
• ??????????????????,?????????,????????????.
• ??,??????????????????,?LS-SVM??????????,Suyken????
  ??????,????????.
• ????,???????????,????????????????.

122
7 SVM??????????(5/7)

• ??,????LS-SVM?????--???????(Vector Base
  LearningVBL)?????????SVM???????.
• ?????(Base Vector SetsBVS)?????????,?????????????
  ???????,????????????????LS-SVM?????????????,??????
  ?????????????.
• SVM?????
• ????
• ????????NN???????,??????NN?????,??????????????????
  On-line????,?????????????.

123
7 SVM??????????(6/7)

• ????
• ????SVM?????????????????????????,????????,
• ??????????????interesting??
• 3D object recognition
• ????????
• ??????
• ????
• Stock forecasting
• Intrusion Detection Systems (IDSs)
• Image classification

124
7 SVM??????????(7/7)

• Detecting Steganography in digital images
• Medical applications diagnostics, survival rates
  ...
• Technical Combustion Engine Knock Detection
• Elementary Particle Identification in High Energy
  Physics
• Bioinformatics protein properties, genomics,
  microarrays
• Information retrieval, text categorization
• In the following web, more SVM applications can
  be found
• http//www.clopinet.com/isabelle/Projects/SVM/appl
  ist.html

125
8 Support Vector Regression(1/15)

• 8 Support Vector Regression
• ???????????????????,???????????.
• ?????????????????,????????????????.
• ??????????????????????????,????????????.
• ??????????????????????(??????)??????????????????,?
  ????.
• ??,??????,??????????????????.

126
8 Support Vector Regression(2/15)

• ????????????,??SVM???,??????????????????????????(
  Support Vector Regression,SVR)?.
• SVR??????
• nonlinear regression the with approximation of
  kernel function
• Linear regression in feature space
• Unlike in least square regression, the error
  function is e-insensitive loss function
• Intuitively, mistake less than e is ignored
• This leads to sparsity similar to SVM

127
8 Support Vector Regression(3/15)

• ??LS?????SVR?????????????????.

128
8 Support Vector Regression(4/15)
129
8 Support Vector Regression(5/15)

• ????LS????,?????????

130
8 Support Vector Regression(6/15)

• ????-insensitive???SVR????,?????????

131
8 Support Vector Regression(7/15)

• ????SVR??
• Given a data set x1, ..., xn with target
  values u1, ..., un, we want to do e-SVR
• The optimization problem is

132
8 Support Vector Regression(8/15)

• ??C is a parameter to control the amount of
  influence of the error
• The ½w2 term serves as controlling the
  complexity of the regression function
• This is similar to ridge regression.
• Similar to SVM, this can be solved as a quadratic
  programming problem

133
8 Support Vector Regression(9/15)

• Similar to SVM, the SVR optimization problem can
  be solved by the Lagrange method as follows

• ???i, ?i, ?i??i???????,??
• ?i, ?i, ?i, ?i?0
• ??????????Kuhn-Tucker??,?

134
8 Support Vector Regression(10/15)
135
8 Support Vector Regression(11/15)

• ??,??

• ?????i, ?i,????w,b,?i, ?i, ?i, ?i

136
8 Support Vector Regression(12/15)

• ?????????????L?,?

??w,b,?i, ?i, ?i, ?i????i, ?i???,????????i??i.
137
8 Support Vector Regression(13/15)

• ??,SVM??????????????????????????

????????
138
8 Support Vector Regression(14/15)

• ????????????????

????????????????.
139
8 Support Vector Regression(15/15)

• Similar to SVM, foe the nonlinear regressiove
  peoblem, the regressive function with the kernel
  function is as foloows.

• ??K(x,z)????.
• ???SVR?????????,??????????
• ?????
• ???
• ???

140
????(1/2)

• ????
• 1 B.E. Boser et al. A Training Algorithm for
  Optimal Margin Classifiers. Proceedings of the
  Fifth Annual Workshop on Computational Learning
  Theory 5 144-152, Pittsburgh, 1992.
• 2 L. Bottou et al. Comparison of classifier
  methods a case study in handwritten digit
  recognition. Proceedings of the 12th IAPR
  International Conference on Pattern Recognition,
  vol. 2, pp. 77-82.
• 3 V. Vapnik. The Nature of Statistical Learning
  Theory. 2nd edition, Springer, 1999.
• 4 C. Burges. A tutorial on support vector
  machines for pattern recognition. Data Mining and
  Knowledge Discovery, 2(2)955-974, 1998.
• 5 N. Cristianini and J. Shawe-Taylor. An
  introduction to support vector machines,
  Cambridge University Press. 2000.

141
????(2/2)

• ????(?)
• 6 A. J. Smola and B. Schölkopf. A Tutorial on
  Support Vector Regression. NeuroCOLT Technical
  Report NC-TR-98-030, Royal Holloway College,
  University of London, UK, 1998.
• 7 Feng J., and Williams P. M. (2001) The
  generalization error of the symmetric and scaled
  support vector machine IEEE T. Neural Networks
  Vol.  12, No. 5.  1255-1260
• 8 http//www.kernel-machines.org/
• 9 http//svmlight.joachims.org
• 10 http//www.support-vector.net/
• 11 http//www.clopinet.com/isabelle/Projects/SVM
  /