Semidefinite Relaxations for Combinatorial Optimization Problems

1
Semidefinite Relaxations for Combinatorial
Optimization Problems

• Ben Recht
• MIT
• October 6, 2003

2
Hard Problems
effort
problem size
Gershenfeld 1999
3
Hard Problems
phase transition
effort
problem size
Gershenfeld 1999
4
Hard Problems?
effort
problem size
Not if you have global information!
5
Successes of this agenda

• Goemans-Williamson algorithm for MAXCUT
• Global bounds of polynomials
• Shannon coding
• Inference
• Distinguishing separable and entangled states
• Stability of nonlinear systems

6
Approximability Results
http//www.nada.kth.se/viggo/wwwcompendium/
7

• Graph G(V,E)
• Maximum-Cut

8

• Graph G(V,E)
• Maximum-Cut

9

• Graph G(V,E)
• Maximum-Cut

10

• Graph G(V,E)
• Maximum-Cut
• Easy for bipartite graphs. In general, NP-Hard

11
Petersen Graph

• Classic Counterexample
• Maximum-Cut 12

12
Petersen Graph

• Classic Counterexample
• Maximum-Cut 12

13
Algorithm 1 Erdos

• Expected Error is 50

14
Algorithm 1 Erdos

• Expected Error is 50
• Flip a coin for each node

15
Algorithm 1 Erdos

• Expected Error is 50
• Flip a coin for each node
• Probability edge is cut1/2

16
Algorithm 1 Erdos

• Expected Error is 50
• Flip a coin for each node
• Probability edge is cut1/2
• State of the art until 1993

25 error
17
Algorithm 2 Goemans-Williamson

• Idea each vertex is a point on the sphere in RV
• Minimize the Square Energy åE (xixj)2

18
Algorithm 2 Goemans-Williamson

• Idea each vertex is a point on the sphere in RV
• Minimize the Square Energy åE (xixj)2
• Cuts are hyperplanes in RV

19
Algorithm 2 Goemans-Williamson

• Idea each vertex is a point on the sphere in RV
• Minimize the Square Energy åE (xixj)2
• Cuts are hyperplanes in RV

20
Algorithm 2 Goemans-Williamson

• Idea each vertex is a point on the sphere in RV
• Minimize the Square Energy åE (xixj)2
• Cuts are hyperplanes in RV. Expected Error12.

8 error
21
Algorithm 2 Goemans-Williamson

• Expected error is 12
• If you find an algorithm with error lt6 then PNP
  (Håstad 01).
• Can we find polynomial time algorithms that split
  the difference?
• Answer Yes. Generalize G-W with Relaxations

22
Outline

• Relaxations
• Semidefinite Programs as Relaxations
• Applications and Algorithms

23
Relaxations Settling for Sufficiency

• Given a hard problem P, come up with problem Pr
  such that

P
24
Relaxations Settling for Sufficiency

• Given a hard problem P, come up with problem Pr
  such that
• We can efficiently find a solution for Pr

P
Pr
25
Relaxations Settling for Sufficiency

• Given a hard problem P, come up with problem Pr
  such that
• We can efficiently find a solution for Pr
• We can efficiently determine if this solution is
  valid for P

P
Pr
xopt
26
Relaxations Settling for Sufficiency

• Given a hard problem P, come up with problem Pr
  such that
• We can efficiently find a solution for Pr
• We can efficiently determine if this solution is
  valid for P
• If no valid solution exists, we hope to get a
  bound on the complexity of P

P
Pr
xopt
27
Convex Optimization

• A set W ½ Rn is convex if it contains all lines
  between all points.

x1
x2
28
Convex Optimization

• A set W ½ Rn is convex if it contains all lines
  between all points.

x1
x2
convex optimization
29
Linear Programming

• If ai, i1,,M and c are vectors in Rn, bi are
  real numbers, then a linear program is

30
Semidefinite programs

• An m m symmetric matrix A is positive
  semidefinite if for all z2 Rm,
• zgtA z 0
• In this case we write Aº 0.
• If Fi, i1,,N are symmetric m m matrices and
  c2Rn then a semidefinite program is

31
Linear Programming

• Can be thought of as a special case of
  semidefinite programming

32
Application Examples 2 Control

• Solving Lyupanov Equations AX XAgtgt0, Xgt0
• Integral Quadratic Constraints
• H2 and H1 control design

33
Application Examples 3

• Filter design
• Approximation
• Kernel Machines
• Clustering
• And so on

34
Example Max-Cut

• We can phrase max-cut as quadratic assignment

35
Example Max-Cut

• We can phrase max-cut as quadratic assignment

Rewrite this as
36
Example Max-Cut

• We can phrase max-cut as quadratic assignment

Rewrite this as
Where A is the adjacency matrix of G Aij1 iff
(i,j)2 E
37
Example Max-Cut

• This is equivalent to an optimization in
  probability
• This optimization is convex
• BUT p(x) is a 2V dimensional vector

38
Example Max-Cut
39
Example Max-Cut
40
Example Max-Cut
41
Example Max-Cut

• Idea Search over covariance matrices not
  probabilities

42
Example Max-Cut

• Goemans-Williamson 1993
• Gives a lower bound for the minimum (upper-bound
  for MAX-CUT)

43
Example Max-Cut

• Lasserre 2001
• More general
• A hierarchy of better and better relaxations
• min1 min2 minn true minimum

44
Method of Moments

• Lasserre 2001
• More general
• A hierarchy of better and better relaxations
• min1 min2 minn true minimum

45
Method of Moments

• We can also use linear relaxations via Mobius
  inversion
• Sherali-Adams hierarchy
• Local consistency is enforced
• Equivalent to Generalized Belief Propagation at
  zero temperature

46
Petersen Graph Revisited
Exact

• The moment method produces the correct answer by
  using all moments 4.
• Using extraction techniques, we can even find the
  optimal solution

47
Sums of Squares
48
Notation

• Let RL be the set of all polynomials with real
  coefficients and L variables.
• A form is a polynomial f2RL such that
• f(lx)lLf(x)
• Let SL be the cone of sums of squares of real
  forms in L variables.
• Just notation
• will denote the set of integers 1, n.

49
Positivstellensatz

• Let f1,,fn, g1,,gm be polynomials in RL. Let
• Then W if and only if there exist lI(x)2 SL
  and gj(x) 2RL such that

50
Interpretation

• If aigt0 and bj0 and
• with all li positive, then either one of the ai
  is negative or one of the bj is not zero.

51
SOS/SDP Theorem

• One can search for a p(x) with deg(p)ltN to prove
  that W using semidefinite programming (Parillo
  2000).
• Let x be a vector of all monomials in L
  variables of degree less than or equal to N/2.
  Write
• Then if Qº0, p is SOS and W.

52
Back to Max Cut

• Let for i1,,V.
• Let's find the maximum t such that

53
Applying SOS

• dividing by the positive l and grouping into a
  quadratic form gives the SDP

54
Dual Program

• That is, the dual of the first SOS relaxation is
  the Goemans-Williamson relaxation.

55
more sums of squares

• The next in the chain is to have degree 2
  multipliers Let Gi be the sparse matrix with
  Giii-Gim1,m11, T the sparse matrix with
  Tm1,m11. Then we look for
• over Lº0, Gi arbitrary.

56
Applications in inference

• Inference on exponential families
  (Wainwright-Jordan 2003)

57
Applications in inference
4x4 Ising Model
BP Mean Error .3.1 LD Mean Error .05.03
58
Algorithms
59
Polynomial time solutions to convex programs

• where W½Rn is a convex set and c2 Rn.

Solution Find a barrier function, fW for W.
fW
W
60
Barrier functions

• f is convex on W
• D3f(x) af D2f(x)3/2
• Df(x)gtD2f(x)Df(x) Jf
• f tends to 1 as x approaches W

fW
W
61
Interior Point Methods

• Solve
• for the minimizer z(h).
• FACT if we can find any x2 W there is a
  deterministic schedule for h such that the convex
  program converges to an e-approximate minimizer
  using Newton's method. The number of Newton
  iterations is bounded by

62
Interior Point Methods for semidefinite programs

• Barrier function fSDP(x) log det F(x)-1
• Jf m
• O(m1/2 log(1/e)) Newton iterations (worst case!)
• least squares problem in n variables and m(m1)/2
  equations per Newton step (worst case!)

63
Try this at home

• Matlab
• SeDuMi
• http//fewcal.kub.nl/sturm/software/sedumi.html
• Yalmip
• http//www.control.isy.liu.se/johanl/yalmip.html
• SOSTOOLS
• http//www.cds.caltech.edu/sostools/
• Gloptipoly
• http//www.laas.fr/henrion/software/gloptipoly/