Kernel Bootstrapping

1
Kernel Bootstrapping

• Vijaya V. Saradhi
• Ph. D., CSE., IIT Kanpur.
• Advisor Prof. Harish Karnick

2
Organization

• Introduction
• Support vector machines
• Limitations
• Re-sampling in input and feature space
• Properties of SVMs bootstrapped SVMs
• Experimental results
• Conclusions

3
Support Vector Machines
?
Optimal Separating Hyper-plane
Margin of Separation
Support Vectors
4
Support Vector Machines
?
Optimal Separating Hyper-plane
Margin of Separation
Support Vectors
5
Objective Function Formulation
Primal Formulation
Subjected To (ST)
Dual Formulation
6
Support Vector Machines Outliers
Outliers
Outliers
Optimal Separating Hyper-plane
Support Vectors
7
Draw-backs/Limitations

• Outliers get picked as potential support vectors
• Margin of separation gets reduced
• Number of support vectors increases hence
  classification time
• Generalization performance decreases

8
Bootstrapping Algorithm Input Space
Weighted Average
9
Bootstrapping Algorithm Feature Space
f(.)
10
Modified Objective Function Formulation
Primal Formulation
Subjected To (ST)
Dual Formulation
Subjected To (ST)
Bootstrap Kernel
11
Effects of Bootstrapping on SVM

• Number of support vectors decreases
• Margin of separation Increases
• Generalization performance retained
• Classification time decreases

• Outliers are pulled towards the sample-mean of
  the corresponding class
• Reduces number of outliers
• Redundancy increases in the data set

Effects
12
Classification
Where
13
Properties of SVMs

• Optimal separating hyper-plane depend upon
• Margin vectors i 0 lt ai lt C
• Error vectors i ai C
• Regularization parameter C

14
Properties of SVMs Contd..

• Lagrangian multipliers dependence on margin
  vectors, error vectors and regularization
  parameter is given by

15
Properties of Bootstrapped SVMs

• Optimal separating hyper-plane depend upon
• Margin Vectors i 0 lt ai lt C
• Error Vectors i ai C
• Regularization parameter C
• Bootstrap parameter r

16
Properties of Bootstrapped SVMs Contd..

• Lagrangian multipliers dependence on margin
  vectors, error vectors, regularization parameter
  and bootstrap parameter r is given by

17
Experimental Results
18
Two Bells Data Set
Two Bells Data Set
19
Noise in Two Bells Data
Two Bells Data Set with 5.0 noise
20
Example1 and Example2
Example1
Example2
21
Iris Data Set
Iris Data Set
22
Results Two Bells Noise
Two Bells
Classification Accuracy
SVs
Margin
Two Bells 5.0 noise
23
Results Example1 Example2
Example1
Classification Accuracy
SVs
Margin
Example2
24
Results Iris
Classification Accuracy
SVs
Kernel Matrix Rank
Margin
25
Results Wine WDBC
Wine
Classification Accuracy
SVs
Margin
WDBC
26
Conclusions

• SVMs are not robust in the presence of outliers
• Re-sampling both in input and feature space is
  introduced to address the robustness issues for
  SVMs
• Several advantages over traditional SVM
  formulation are shown
• Effects of bootstrapping on SVM
• Reduced number of support vectors
• Margin increases as bootstrap parameter r
  increases
• Classification accuracy is retained very close to
  conventional SVMs as r increases