Convex Programming

1
Convex Programming

• Brookes Vision Reading Group

2
Huh?

• What is convex ???
• What is programming ???
• What is convex programming ???

3
Huh?

• What is convex ???
• What is programming ???
• What is convex programming ???

4
Convex Function
f(t x (1-t) y) lt t f(x) (1-t) f(y)
5
Convex Function
Is a linear function convex ???
6
Convex Set
Region above a convex function is a convex set.
7
Convex Set
Is the set of all positive semidefinite matrices
convex??
8
Huh?

• What is convex ???
• What is programming ???
• What is convex programming ???

9
Programming

• Objective function to be minimized/maximized.
• Constraints to be satisfied.

10
Example
Optimal solution
Vertices
Objective function
Feasible region
11
Huh?

• What is convex ???
• What is programming ???
• What is convex programming ???

12
Convex Programming

• Convex optimization function
• Convex feasible region
• Why is it so important ???

• Global optimum can be found in polynomial time.

• Many practical problems are convex

• Non-convex problems can be relaxed to convex
  ones.

13
Convex Programming

• Convex optimization function
• Convex feasible region
• Examples ???

• Linear Programming
• Refer to Vladimir/Pushmeets reading group

• Second Order Cone Programming
• What ???

• Semidefinite Programming
• All this sounds Greek and Latin !!!!

14
Outline

• Convex Optimization
• Second Order Cone Programming (SOCP)
• Semidefinite Programming (SDP)
• Non-convex optimization
• SDP relaxations
• SOCP relaxations
• Optimization Algorithms
• Interior Point Method for SOCP
• Interior Point Method for SDP

2 out of 3 is not bad !!!
15
Outline

• Convex Optimization
• Second Order Cone Programming (SOCP)
• Semidefinite Programming (SDP)
• Non-convex optimization
• SDP relaxations
• SOCP relaxations
• Optimization Algorithms
• Interior Point Method for SOCP
• Interior Point Method for SDP

16
Second Order Cone

• u lt t
• u - vector of dimension d-1
• t - scalar
• Cone lies in d dimensions
• Second Order Cone defines a convex set

• Example Second Order Cone in 3D

x2 y2 lt z2
17
x2 y2 lt z2
18
Second Order Cone Programming
Minimize fTx Subject to Ai x bi lt
ciT x di
i 1, , L
Constraints are SOC of ni dimensions
Feasible regions are intersections of conic
regions
19
Example
20
Why SOCP ??

• A more general convex problem than LP
• LP ? SOCP
• Fast algorithms for finding global optimum
• LP - O(n3)
• SOCP - O(L1/2) iterations of O(n2?ni)
• Many standard problems are SOCP-able

21
SOCP-able Problems

• Convex quadratically constrained quadratic
  programming
• Sum of norms
• Maximum of norms
• Problems with hyperbolic constraints

22
SOCP-able Problems

• Convex quadratically constrained quadratic
  programming
• Sum of norms
• Maximum of norms
• Problems with hyperbolic constraints

23
QCQP
Minimize xT P0 x 2 q0T x r0 Subject to
xT Pi x 2 qiT x ri
Pi gt 0
P01/2 x P0-1/2 x 2 r0 -q0TP0-1p0
24
QCQP
Minimize xT P0 x 2 q0T x r0 Subject to
xT Pi x 2 qiT x ri
Minimize t Subject to P01/2 x P0-1/2 x
lt t P01/2 x P0-1/2 x lt (r0
-q0TP0-1p0)1/2
25
SOCP-able Problems

• Convex quadratically constrained quadratic
  programming
• Sum of norms
• Maximum of norms
• Problems with hyperbolic constraints

26
Sum of Norms
Minimize ? Fi x gi
Minimize ? ti Subject to Fi x gi lt ti
Special Case L-1 norm minimization
27
SOCP-able Problems

• Convex quadratically constrained quadratic
  programming
• Sum of norms
• Maximum of norms
• Problems with hyperbolic constraints

28
Maximum of Norms
Minimize max Fi x gi
Minimize t Subject to Fi x gi lt t
Special Case L-inf norm minimization
29
You werent expecting a question, were you ??
30
SOCP-able Problems

• Convex quadratically constrained quadratic
  programming
• Sum of norms
• Maximum of norms
• Problems with hyperbolic constraints

31
Hyperbolic Constraints
x gt 0 , y gt 0
w2 lt xy
2w x-y lt xy
32
Lets see if everyone was awake !
33
Outline

• Convex Optimization
• Second Order Cone Programming (SOCP)
• Semidefinite Programming (SDP)
• Non-convex optimization
• SDP relaxations
• SOCP relaxations
• Optimization Algorithms
• Interior Point Method for SOCP
• Interior Point Method for SDP

34
Semidefinite Programming
Minimize C ? X Subject to Ai ? X bi
X gt 0
Linear Programming on Semidefinite Matrices
35
Why SDP ??

• A more general convex problem than SOCP
• LP ? SOCP ? SDP
• Generality comes at a cost though
• SOCP - O(L1/2) iterations of O(n2?ni)
• SDP - O((?ni)1/2) iterations of O(n2?ni2)
• Many standard problems are SDP-able

36
SDP-able Problems

• Minimizing the maximum eigenvalue
• Class separation with ellipsoids

37
SDP-able Problems

• Minimizing the maximum eigenvalue
• Class separation with ellipsoids

38
Minimizing the Maximum Eigenvalue
Matrix M(z) To find vector z such that ?max is
minimized.
Let ?max(M(z)) lt n
?max(M(z)-nI) lt 0
?min(nI - M(z)) gt 0
nI - M(z) gt 0
39
Minimizing the Maximum Eigenvalue
Matrix M(z) To find vector z such that ?max is
minimized.
Max -n nI - M(z) gt 0
40
SDP-able Problems

• Minimizing the maximum eigenvalue
• Class separation with ellipsoids

41
(No Transcript)
42
(No Transcript)
43
Outline

• Convex Optimization
• Second Order Cone Programming (SOCP)
• Semidefinite Programming (SDP)
• Non-convex optimization
• SDP relaxations
• SOCP relaxations
• Optimization Algorithms
• Interior Point Method for SOCP
• Interior Point Method for SDP

44
Non-Convex Problems
Minimize xTQ0x 2q0Tx r0 Subject to xTQix
2qiTx ri lt 0
Qi gt 0 gt Convex
Non-Convex Quadratic Programming Problem !!!
Redefine x in homogenous coordinates. y (1 x)

45
Non-Convex Problems
Minimize xTQ0x 2q0Tx r0 Subject to xTQix
2qiTx ri lt 0
Minimize yTM0y Subject to yTMiy lt 0
Mi ri qiT qi Qi
46
Non-Convex Problems

• Problem is NP-hard.
• Lets relax the problem to make it convex.
• Pray !!!

47
Outline

• Convex Optimization
• Second Order Cone Programming (SOCP)
• Semidefinite Programming (SDP)
• Non-convex optimization
• SDP relaxations
• SOCP relaxations
• Optimization Algorithms
• Interior Point Method for SOCP
• Interior Point Method for SDP

48
SDP Relaxation
Minimize yTM0y Subject to yTMiy lt 0
Minimize M0 ? Y Subject to Mi ? Y lt 0
Y yyT
49
SDP Relaxation
Minimize yTM0y Subject to yTMiy lt 0
Minimize M0 ? Y Subject to Mi ? Y lt 0
Y gt 0
Nothing left to do . but Pray
Note that we have squared the number of variables.
50
Example - Max Cut

• Graph G(V,E)
• Maximum-Cut

51
Example - Max Cut

• Graph G(V,E)
• Maximum-Cut

52
Example - Max Cut

• Graph G(V,E)
• Maximum-Cut

Alright !!! So its an integer programming
problem !!!
Doesnt look like quadratic programming to me !!!
53
Max Cut as an IQP
Naah !! Lets get it into the standard quadratic
form.
54
(No Transcript)
55
Max Cut as an IQP
Naah !! Lets get it into the standard quadratic
form.
56
Solving Max Cut using SDP Relaxations
To the white board. (You didnt think Ill
prepare slides for this, did you??)
57
Outline

• Convex Optimization
• Second Order Cone Programming (SOCP)
• Semidefinite Programming (SDP)
• Non-convex optimization
• SDP relaxations
• SOCP relaxations
• Optimization Algorithms
• Interior Point Method for SOCP
• Interior Point Method for SDP

58
SOCP Relaxation
Minimize yTM0y Subject to yTMiy lt 0
Remember Y 1 xT x X
Minimize M0 ? Y Subject to Mi ? Y lt 0
Y gt 0
59
SOCP Relaxation
Say youre given C C1, C2, Cn such that Cj
gt 0
Cj ? (X - xxT) gt 0
(Ux)T (Ux) lt Cj ? X
Wait .. Isnt this a hyperbolic constraint
Therefore, its SOCP-able.
60
SOCP Relaxation
Minimize yTM0y Subject to yTMiy lt 0
Minimize Q0 ? X 2q0Tx r0 Subject to Qi ? X
2qiTx ri lt 0 Cj ? (X -
xxT) gt 0 Cj ? C
61
SOCP Relaxation
If C is the infinite set of all semidefinite
matrices SOCP Relaxation SDP Relaxation
If C is finite, SOCP relaxation is looser than
SDP relaxation.
Then why SOCP relaxation ???
Efficiency - Accuracy Tradeoff
62
Choice of C
Remember we had squared the number of variables.
Lets try to reduce them with our choice of C.
For a general problem - Kim and Kojima
Using the structure of a specific problem - e.g.
Muramatsu and Suzuki for Max Cut
63
Choice of C
Minimize cT x Subject to Qi ? X 2qiTx ri lt
0
Q ? X 2qTx r lt 0
Q ?n ?i uiuiT
Let ?1 gt ?2 gt . ?k gt 0 gt ?k1 gt
?n
64
Choice of C
Q ?k ?i uiuiT
C
Q ? X 2qTx r lt 0
xT Q x - Q ? X lt 0
xT Q x ?k1 ?i uiuiT ? X 2qTx r lt 0
zi
65
Choice of C
Q ?k ?i uiuiT
C
uiuiT i k1, k2, n
Q ? X 2qTx r lt 0
xT Q x ?k1 ?i zi 2qTx r lt 0
xTuiuiTx - uiuiT ? X lt 0
66
Choice of C
Q ?k ?i uiuiT
C
uiuiT i k1, k2, n
Q ? X 2qTx r lt 0
xT Q x ?k1 ?i zi 2qTx r lt 0
xTuiuiTx - zi lt 0
67
Specific Problem Example - Max Cut
ei 0 0 . 1 0 0
uij ei ej
vij ei - ej
68
Specific Problem Example - Max Cut
Warning Scary equations to follow.
69
Outline

• Convex Optimization
• Second Order Cone Programming (SOCP)
• Semidefinite Programming (SDP)
• Non-convex optimization
• SDP relaxations
• SOCP relaxations
• Optimization Algorithms
• Interior Point Method for SOCP
• Interior Point Method for SDP

70
Outline

• Convex Optimization
• Second Order Cone Programming (SOCP)
• Semidefinite Programming (SDP)
• Non-convex optimization
• SDP relaxations
• SOCP relaxations
• Optimization Algorithms
• Interior Point Method for SOCP
• Interior Point Method for SDP

71
Back to work now !!!