Reduced Data Classifiers via Support Vector Machines SIAM International Conference on Data Mining Chicago April 5-7, 2001

1
Reduced Data Classifiersvia Support Vector
MachinesSIAM International Conference on Data
Mining Chicago April 5-7, 2001

• O. L. Mangasarian Y. J. Lee

Data Mining Institute University of Wisconsin -
Madison
Second Annual Review June 1, 2001
2
Key Objective

3
Outline of Talk

• What is a support vector machine (SVM)?

• What is a smooth support vector machine (SSVM)?

• An SVM solvable without optimization software
  (LP,QP)

• Difficulties with nonlinear SVM classifiers

• Storage Classifier depends on almost entire
  dataset

• Reduced Support Vector Machines (RSVMs)

• Speeds computation reduces storage

• Numerical Results

4
What is a Support Vector Machine?

• An optimally defined surface
• Typically nonlinear in the input space
• Linear in a higher dimensional space
• Implicitly defined by a kernel function

5
What are Support Vector Machines Used For?

• Classification
• Regression Data Fitting
• Supervised Unsupervised Learning

(Will concentrate on classification)
6
Geometry of the Classification Problem2-Category
Linearly Separable Case
A
A-
7
Support Vector MachinesAlgebra of 2-Category
Linearly Separable Case
8
Support Vector MachinesMaximizing the Margin
between Bounding Planes
A
A-
9
Support Vector Machine Formulation
10
SVM as an Unconstrained Minimization Problem
11
Smoothing the Plus Function Integrate the
Sigmoid Function
12
SSVM The Smooth Support Vector Machine

13
Nonlinear Smooth Support Vector Machine
Nonlinear Separating Surface (Instead of Linear
Surface

• Use Newton algorithm to solve the problem

• Nonlinear separating surface depends on entire
  dataset

14
Examples of Kernels
15
Difficulties with Nonlinear SVM for Large
Problems

• Separating surface depends on almost entire
  dataset

• Need to store the entire dataset after solving
  the problem

16
Overcoming Computational Storage
DifficultiesUse a Rectangular Kernel
17
Reduced Support Vector Machine AlgorithmNonlinear
Separating Surface
18
How to Choose in RSVM?
19
A Nonlinear Kernel ApplicationCheckerboard
Training Set 1000 Points in Separate 486
Asterisks from 514 Dots
20
Conventional SVM Result on Checkerboard Using 50
Randomly Selected Points Out of 1000
21
RSVM Result on Checkerboard Using SAME 50 Random
Points Out of 1000
22
RSVM on Moderately Sized Problems(Best Test Set
Correctness , CPU seconds)
23
RSVM on Large UCI Adult DatasetStandard
Deviation over 50 Runs 0.001
24
CPU Times on UCI Adult DatasetRSVM, SMO and
PCGC with a Gaussian Kernel
25
CPU Time Comparison on UCI DatasetRSVM, SMO and
PCGC with a Gaussian Kernel
Time( CPU sec. )
Training Set Size
26
Conclusion

• RSVM An effective classifier for large datasets

• Classifier uses 10 or less of dataset

• Can handle massive datasets

• Much faster than other algorithms

• Test set correctness

• Novel practical idea

• Applicable to all nonlinear kernel problems