Suppose we have 5 1D data points

1
Example

• Suppose we have 5 1D data points
• x11, x22, x34, x45, x56, with 1, 2, 6 as
  class 1 and 4, 5 as class 2 ? y11, y21, y3-1,
  y4-1, y51
• We use the polynomial kernel of degree 2
• K(x,y) (xy1)2
• C is set to 100
• We first find ai (i1, , 5) by

2
Example

• By using a QP solver, we get
• a10, a22.5, a30, a47.333, a54.833
• Note that the constraints are indeed satisfied
• The support vectors are x22, x45, x56
• The discriminant function is
• b is recovered by solving f(2)1 or by f(5)-1 or
  by f(6)1, as x2, x4, x5 lie on
  and all give b9

3
Example
Value of discriminant function
class 1
class 1
class 2
1
2
4
5
6
4
Multi-class Classification

• SVM is basically a two-class classifier
• One can change the QP formulation to allow
  multi-class classification
• More commonly, the data set is divided into two
  parts intelligently in different ways and a
  separate SVM is trained for each way of division
• Multi-class classification is done by combining
  the output of all the SVM classifiers
• Majority rule
• Error correcting code
• Directed acyclic graph

5
Software

• A list of SVM implementation can be found at
  http//www.kernel-machines.org/software.html
• Some implementation (such as LIBSVM) can handle
  multi-class classification
• SVMLight is among one of the earliest
  implementation of SVM
• Several Matlab toolboxes for SVM are also
  available

6
Summary Steps for Classification

• Prepare the pattern matrix
• Select the kernel function to use
• Select the parameter of the kernel function and
  the value of C
• You can use the values suggested by the SVM
  software, or you can set apart a validation set
  to determine the values of the parameter
• Execute the training algorithm and obtain the ai
• Unseen data can be classified using the ai and
  the support vectors

7
Strengths and Weaknesses of SVM

• Strengths
• Training is relatively easy
• No local optimal, unlike in neural networks
• It scales relatively well to high dimensional
  data
• Tradeoff between classifier complexity and error
  can be controlled explicitly
• Non-traditional data like strings and trees can
  be used as input to SVM, instead of feature
  vectors
• Weaknesses
• Need a good kernel function

8
Resources

• http//www.kernel-machines.org/
• http//www.support-vector.net/
• http//www.support-vector.net/icml-tutorial.pdf
• http//www.kernel-machines.org/papers/tutorial-nip
  s.ps.gz
• http//www.clopinet.com/isabelle/Projects/SVM/appl
  ist.html