Support Vector Machines

1

• Support Vector Machines
• for Classification

• Jeong Hwan Bang

2

• Overview

• Theory
• Simulations
• Applications

3

• Artificial Neural Networks

• Analytic techniques modeled after the processes
  of learning in the cognitive system and the
  neurological functions of the brain
• Capable of predicting new observations from other
  observations

4
Model of a Neuron

• Information is received at the synapses on its
  dendrites
• An electro-chemical transmission occurs at the
  synapses
• At the cell body, a summation of the electric
  impulses takes place
• If summation meets a particular threshold, the
  neuron will send a signal

5

• Simplified Model of A Neuron

6
Artificial Neuron

• Receives a number of inputs
• Each input has a weight that corresponds to
  synaptic efficacy in a biological neuron
• Weighted sum of the inputs is formed
• Activation if weighted sum meets the threshold
  value

7

• Model of an Artificial Neural Network

8
Support Vector Machines

• Extensively used for classification and
  regression problems
• Does not suffer the curse of dimensionality
• Valuable for solving problems with a large number
  of inputs

9
SVM Theory

• Maximize the margin between the separating
  patterns using a hyperplane
• Margin of separation is the separation between
  the hyperplane and the closest data point
• The goal is to find the optimal hyperplane to
  maximize the margin of separation

10
SVM Theory
11
SVM Theory

• Support vectors are data points that lie directly
  on the decision boundary
• Support vectors serve a very important role in
  the operation of this algorithm

12

• SVM Theory

13
SVM Theory

• Slack variables consider the case of nonseparable
  patterns, and measure the deviation of the data
  points from the boundary of the region of
  separability

14

• SVM Theory

15
Optimization Problem

• Given the training example
• Find the optimum values of the weight vector and
  bias such that they satisfy the constraint

16
Optimization Problem

• Minimize the cost functional
• The parameter C quantifies the trade-off between
  training error and system capacity

17
Inner-Product Kernels

• The inner-product kernel may be used to construct
  the optimal hyperplane in the feature space
  without having to consider the feature space
  itself in explicit form
• The requirement on the construction of the kernel
  is that it satisfies Mercels theorem

18
Inner-Product Kernels

• polynomial learning machine
• radial-basis function network kernel
• two-layer perceptron kernel

19
Simulations

• Linearly Separable Data
• Nonlinearly Separable Data
• Polynomial Mapping
• Classification Example

20

• Linearly Separable Data

21

• 5

• Nonlinearly separable data, C 10

22

• -8

• Nonlinearly separable data, C 10

23

• Polynomial Mapping

24
Classification Example

• The goal is to classify the class of an iris
  given two features - pedal length and pedal width

25

• Separation of the Setosa class

26

• Polynomial SVM of degree 2, C ? Infinity

27

• Polynomial SVM of degree 10, C ? Infinity

28

• Polynomial SVM of degree 10, C 10

29
Applications

• Nonlinear Equalization
• Text Categorization
• 3D Object Recognition

30

• Nonlinear Equalization

• Output of a channel is used as the input of the
  classifier
• Channel output is transformed into a pattern
  space, which is mapped into a higher-dimensional
  feature space
• Classifier matches a delayed version of the
  original signal

31

• Text Categorization

• Utilizing Information Retrieval Theory, word
  stems are used as representation units
• Each distinct word corresponds to a feature, with
  the frequency of occurrence as its value
• To avoid an unnecessarily large number of
  features, only words of a threshold frequency are
  considered, and stop-words (and, or, etc.)
  are ignored

32

• 3D Object Recognition

• Many views of an object are given to the SVM
• Types of features selected
• Shape of the object
• Color of the object
• Shape and the color of the object

33
Conclusion

• Support vector machines method is an efficient,
  robust method for classification
• Advantages
• Small number of adjustable parameters
• Doesnt require prior information or heuristic
  assumptions
• Train with relatively small amounts of data

34

• THANK YOU

• THANK YOU