MURI Meeting July 2002

1
Convex Optimization in Machine Learning

• MURI MeetingJuly 2002
• Gert Lanckriet (gert_at_eecs.berkeley.edu)
• L. El Ghaoui, M. Jordan, C. Bhattacharrya, N.
  Cristianini, P. Bartlett
• U.C. Berkeley

2
Convex Optimization in Machine Learning
3
Advanced Convex Optimization in Machine Learning
SDP
SOCP
QCQP
QP
LP
4
Advanced Convex Optimization in Machine Learning
5
Linear Programming (LP)
6
Quadratic Programming (QP)
7
Quadratic Constrained Quadratic Programming (QCQP)
8
Second Order Cone Programming (SOCP)
9
Semi-Definite Programming
10
Advanced Convex Optimization in Machine Learning
11
MPM Problem Sketch (1)
aT z b decision hyperplane
12
MPM Problem Sketch (2)
13
MPM Problem Sketch (3)
14
MPM Main Result (1)
?
?
Marshall Olkin / Popescu Bertsimas
15
MPM Main Result (2)
16
MPM Main Result (3)
Lemma
17
MPM Main Result (4)
Probabilistic Constraint
Lemma
Deterministic Constraint
18
MPM Main Result (5)
19
MPM Geometric Interpretation
20
MPM Link with FDA (1)
21
MPM Link with FDA (2)
22
MPM Link with FDA (3)
23
Robustness to Estimation Errors Robust MPM
(R-MPM)
24
Robust MPM (R-MPM)
25
Robust MPM (R-MPM)
26
MPM Convex Optimization to solve the problem
Lemma
Linear Classifier
Convex Optimization Second Order Cone Program
(SOCP)
Kernelizing
Nonlinear Classifier
) competitive with Quadratic Program (QP) SVMs
27
MPM Empirical results
a1b and TSA (test-set accuracy) of the MPM,
compared to BPB (best performance in Breiman's
report (Arcing classifiers, 1996)) and SVMs.
(averages for 50 random partitions into 90
training and 10 test sets)

• Comparable with existing literature, SVMs
• a1-b is indeed smaller than the test-set
  accuracy in all cases (consistent with b as
  worst-case bound on probability of
  misclassification)
• Kernelizing leads to more powerfull decision
  boundaries (alinear decision boundary lt
  anonlinear decision boundary (Gaussian kernel))

28
Conclusions
29
Future directions
30
Advanced Convex Optimization in Machine Learning
31
The idea (1)
32
The idea (2)
33
The idea (3)
34
The idea (4)
35
The idea (5)
36
Hard margin SVM classifiers (1)
37
Hard margin SVM classifiers (2)
38
Hard margin SVM classifiers (3)
39
Hard margin SVM classifiers (4)
40
Hard margin SVM classifiers (5)
SDP !
41
Hard margin SVM classifiers (6)
Optimization
Learning the kernel matrix !
42
Hard margin SVM classifiers (7)
training set (labelled)
test set (unlabelled)
Learning the kernel matrix !
43
Hard margin SVM classifiers (8)
?
44
Hard margin SVM classifiers (9)
45
Hard margin SVM classifiers (9)
46
Hard margin SVM classifiers (9)
47
Hard margin SVM classifiers (10)
48
Hard margin SVM classifiers (11)
Learning Kernel Matrix with SDP !
49
Empirical results hard margin SVMs
50
Conclusions and future directions
51
Conclusions and future directions
52
See also