Knowledge-Based%20Support%20Vector%20Machine%20Classifiers

1
Knowledge-Based Support Vector Machine
Classifiers
NIPS2002, Vancouver, December 9-14, 2002

• Glenn Fung
• Olvi Mangasarian
• Jude Shavlik

University of Wisconsin-Madison
2
Outline of Talk

• Support Vector Machine (SVM) Classifiers

• Standard QP formulation

• LP formulation1-norm linear SVM classifier

• Polyhedral Knowledge Sets

• Knowledge-Based SVMs

• Incorporating knowledge sets into a classifier

• Numerical Results

• The promoter DNA sequence dataset

• Wisconsin breast cancer prognosis dataset

• Conclusion

3
What is a Support Vector Machine?

• An optimally defined surface
• Typically nonlinear in the input space
• Linear in a higher dimensional space
• Implicitly defined by a kernel function
• Used for
• Regression Data Fitting
• Supervised Unsupervised Learning

4
Geometry of the Classification Problem2-Category
Linearly Separable Case
A
A-
5
Algebra of the Classification Problem 2-Category
Linearly Separable Case

• Given m points in n dimensional space

• Represented by an m-by-n matrix A

• More succinctly

6
Support Vector MachinesMaximizing the Margin
between Bounding Planes
A
A-
7
Support Vector Machines QP Formulation

• Solve the following quadratic program

8
Support Vector MachinesLinear Programming
Formulation

• Use the 1-norm instead of the 2-norm

• This is equivalent to the following linear
  program

9
Conventional Data-Based SVM
10
Knowledge-Based SVM via Polyhedral Knowledge
Sets
11
Incoporating Knowledge Sets Into an SVM
Classifier

• Will show that this implication is equivalent to
  a set of constraints that can be imposed on the
  classification problem.

12
Knowledge Set Equivalence Theorem
13
Proof of Equivalence Theorem( Via Nonhomogeneous
Farkas or LP Duality)
Proof By LP Duality
14
Knowledge-Based SVM Classification
15
Knowledge-Based SVM Classification
16
Parametrized Knowledge-Based LPMinimize Error in
Knowledge Set Constraints
17
Knowledge-Based SVM via Polyhedral Knowledge
Sets
18
Numerical TestingThe Promoter Recognition Dataset

• Promoter Short DNA sequence that precedes a
  gene sequence.
• A promoter consists of 57 consecutive DNA
  nucleotides belonging to A,G,C,T .
• Important to distinguish between promoters and
  nonpromoters
• This distinction identifies starting locations
  of genes in long uncharacterized DNA sequences.

19
The Promoter Recognition DatasetNumerical
Representation

• Feature space mapped from 57-dimensional nominal
  space to a real valued 57 x 4228 dimensional
  space.

57 nominal values
57 x 4 228 binary values
20
Promoter Recognition Dataset Prior Knowledge
Rules

• Prior knowledge consist of the following 64
  rules

21
Promoter Recognition Dataset Sample Rules
22
The Promoter Recognition DatasetComparative
Algorithms

• KBANN Knowledge-based artificial neural network
  Shavlik et al
• BP Standard back propagation for neural
  networks Rumelhart et al
• ONeills Method Empirical method suggested by
  biologist ONeill ONeill
• NN Nearest neighbor with k3 Cost et al
• ID3 Quinlans decision tree builderQuinlan
• SVM1 Standard 1-norm SVM Bradley et al

23
The Promoter Recognition DatasetComparative Test
Results
24
Wisconsin Breast Cancer Prognosis Dataset
Description of the data

• 110 instances corresponding to 41 patients
  whose cancer had recurred and 69 patients whose
  cancer had not recurred
• 32 numerical features
• The domain theory two simple rules used by
  doctors

25
Wisconsin Breast Cancer Prognosis Dataset
Numerical Testing Results

• Doctors rules applicable to only 32 out of 110
  patients.
• Only 22 of 32 patients are classified correctly
  by this rule (20 Correctness).
• KSVM linear classifier applicable to all
  patients with correctness of 66.4.
• Correctness comparable to best available
  results using conventional SVMs.
• KSVM can get classifiers based on knowledge
  without using any data.

26
Conclusion

• Prior knowledge easily incorporated into
  classifiers through polyhedral knowledge sets.
• Resulting problem is a simple LP.
• Knowledge sets can be used with or without
  conventional labeled data.
• In either case, KSVM is better than most
  classifiers tested.

27
Future Research

• Generate classifiers based on prior expert
  knowledge in various fields
• Diagnostic rules for various diseases
• Financial investment rules
• Intrusion detection rules
• Extend knowledge sets to general convex sets
• Nonlinear kernel classifiers. Challenges
• Express prior knowledge nonlinearly
• Extend equivalence theorem

28
Web Pages