Message Passing on Planted Models: What do we know? Why do we care?

1
Message Passing on Planted ModelsWhat do we
know?Why do we care?

• Elchanan Mossel
• Joint works with
• 1. Uri Feige and Danny Vilenchik
• 2. Amin Coja-Oghlan and Danny Vilenchik

2
Planted 3SAT model

• Generates a random satisfiable formula, with
  probability proportional to the number of
  satisfying assignments that it has.
• Generate m clauses at random.
• Fix an assignment a at random.
• Remove those clauses not satisfied by a. (Roughly
  7m/8 clauses remain.)

3
Planted 3-coloring model

• Take a random 3-partite (2d)-regular graph.
• Take a partition into 3 sets of size n/3.
• Each node connected to d random nodes in each of
  the other two parts.
• Color each part with a different color.

4
Why study these models?

• Clearly satisfiable by construction.
• Clearly Replica Symmetric
• Become easier at higher densities.
• So why study?

5
Why study these models?

• At high constant density, the model is close to
  the standard SAT model conditioned on SAT
  (Coja-Oghlan, Krivelevich,Vilenchik).
• Feige An efficient alg. for distinguishing
  (almost) SAT dense formulas from typical
  formulas ) Hardness of Approximation results.

6
Why study these models?

• Crypto
• Planted models a good one way function if they
  are hard to solve.

7
Why study these models?

• Real Life Models
• In real life models, constraints are not random.
• Constraints are correlated via nature.
• More constraints make problems easier.
• Planted models may be a better model than random
  SAT.

8
Random-SAT model

• Random 3CNF formula with n variables and m
  clauses.
• Conjecture
• When m lt 4.2n almost all formulas are
  satisfiable.
• When m gt 4.3n almost all formulas not
  satisfiable.
• Theorem Finding satisfying assignments is more
  difficult near the threshold. (Proof add random
  clauses.)

9
Planted-SAT model

• Random 3CNF formula with n variables and m
  clauses.
• Fix an assignment a
• Remove clauses not consistent with the
  assignment.
• Observation Many clauses make the problem
  easier.
• Pf E.G. gt n log n clauses For each variable
  i, set
• ai 1 if appears more positive than negative.

10
Our contribution

• Feige-M-Vilenchik A proof that warning
  propagation finds satisfying assignments for most
  3CNF formulas with planted assignments and large
  enough constant density.
• Coja-Oghlan-M-Vilenchik Same for BP with density
  O(log n)
• Coja-Oghlan-M-Vilenchik A proof that BP find
  satisfying assignment for most planted 3-coloring
  problems with planted assignments and large
  enough constant density.
• First rigorous analysis of message passing
  algorithms for satisfiability problems
• Convergence for all variables (not just for a
  typical one)

11
Overview of message passing algorithms

• The factor graph
• A bipartite graph, variables on one side and
  clauses on the other.
• Edges connect variables to the clauses that
  contain them.
• Nodes send messages to their neighbors, and
  iteratively update the messages that they send
  based on messages received.

12
Factor Graph

• X1 X2 X3 X4 X5 X6 X7
• C1 C2 C3
  C4

13
Cavity principle

• Message from clause C to variable x
• Based on the messages received by C from other
  variables, an estimate of how badly C needs x to
  satisfy it.
• Message from variable x to clause C
• Based on the messages received by x from other
  clauses, an estimate of how likely x is to
  satisfy C.

14
Different message passing algorithms

• Warning propagation (WP) 0/1 messages.
• Belief propagation (BP) fractional messages.
• Survey propagation (SP) fractional messages, but
  with a more sophisticated update rule.
• In all variants, if the algorithm converges, a
  decimation stage follows that fixes the value of
  some variables (those that do not receive
  conflicting messages) and simplifies the formula.

15
Previous analysis of message passing algorithms
(typically, BP)

• Converges correctly when the factor graph is a
  tree, or contains at most one cycle Weiss 1997.
• W.h.p., corrects most (but not necessarily all)
  errors in random Low Density Parity Check (LDPC)
  codes.
• W.h.p., corrects all errors for specially
  designed LDPC codes Luby, Mitzenmacher,
  Shokrollahi, Spielman.

16
Known algorithmic results for planted model

• When m gtgt n log n, majority vote finds planted
  assignment.
• When m gt cn (for sufficiently large constant c),
  a complicated algorithm finds a satisfying
  assignment Flaxman, and so does a certain local
  search algorithm Feige and Vilenchik.
• Analysis based on principles developed in Alon
  and Kahale, SICOMP 1997.

17
Our results WP for planted SAT

• For most formulas with large enough constant
  clause density, w.h.p. (over initialization of
  random messages) WP converges after O(log n)
  iterations.
• Moreover, with probability close to 1, the
  formula that results after decimation is pure. WP
  makes no further progress on it, but this formula
  can be satisfied by a trivial greedy algorithm.

18
Structural properties

• Sufficiently dense random formulas with a planted
  solution are known to w.h.p. have the following
  structure
• There is a large subset of the clauses which are
  jointly satisfied only by the induced planted
  solution. This is called a core.
• Decimation for all core variables makes the
  resulting formula simple (the factor graph
  decomposes into small components).

19
Algorithmic consequences

• Algorithms for the planted model first try to get
  the core variables right (for some core).
  Thereafter, the rest of the formula can be
  handled efficiently because of its simple
  structure.
• The majority vote algorithm followed by
  unassigning variables that do not support many
  clauses produces a core.

20
WP and core

• For WP, the core is not an intrinsic property of
  the input formula. It also depends on the random
  initialization of messages in WP. We show
• W.h.p., after one iteration of WP a core forms
  a large set of variables for which all subsequent
  messages agree with the planted assignment.
• After O(log n) additional iterations, all noncore
  variables converge (either to their planted value
  or to dont care).
• Decimation of all variables that do care results
  in a pure formula.

21
Results for BP for Planted SAT

• Similar results but analysis harder and degrees
  logarithmic due to unbounded messages (analysis
  should extend to the case of bounded degrees)

22
Results for BP for Planted Coloring

• Note Inherit symmetry of the coloring problem.
• Advantage Initial messages could be chosen
  close to (1/3,1/3,1/3).
• Disadvantage How does the algorithm breaks the
  symmetry?

23
Results for BP for Planted Coloring

• Techniques Use linearization Spectral
  techniques to show symmetry is broken.
• Potential extensions To random SAT problems with
  many solutions.

24
Positive conclusions

• Rigorous analysis proving that a message passing
  algorithm finds satisfying assignments in
  nontrivial SAT problems.

25
Open questions

• Still far from explaining the empirical success
  of survey propagation on random 3CNF formulas
  near the density threshold.

26
Random Graphical Games on (Random) GraphsJoint
work with Costas Daskalakis and Alex Dimakis
27
Motivation

• What is the effect of selfishness in networks
  such as
• Recommendation networks?
• Network design?
• Load balancing?
• A crucial question
• Do networks have Nash equilibrium?
• What is the effect of network topology?
• Study random games.
• Joint work with Costis Daskalakis and Alex
  Dimakis

28
Chicken game
Paul plays
John plays
29
Pure Nash Equilibrium

• A player is in best response if there is no
  incentive to change strategy, if others do not.
• A strategy profile is a Pure Nash equilibrium, if
  all players are in best response. (DC or CD)
• But doesnt always exist

30
Matching pennies
same
different
Paul plays
John plays
There exists no Pure Nash equilibrium
31
Best response tables
Chicken game
Matching pennies game
32
Random Binary Games
Player 2

• What is the chance that a 2 player, 2 strategy
  random game has a PNE?

1/2?11/2?3/4
33
N strategies, 2 players? (1pt)

• What is the chance that a 2 player, N strategy
  random game has a PNE?

N
34
Previous work
For two players, many strategies, we know exactly
the distribution of the number of PNE. (Dresher
1970, Powers 1990, Stanford 1996) For many
players? For complete graph Rinott et al,
(2000) There exist PNE with constant
probability. Further obtain asymptotic
distribution on their number. (Poisson)
35
Main Question
What if the effect of connectivity? Model Study
G(n,p).
36
Constant degrees

• Claim 1 For an (undirected) random graphical
  game with constant degrees, whp at least a
  constant fraction of players are always unhappy.

37
A witness for failure indifferent
matching pennies
A
never happy at the same time
D
B
C

38
Indifferent matching pennies shows up a lot
39
Logarithmic Connectivity Suffcies

• Claim 2 for directed graphical games, on
  GD(N,p), there exists a PNE with at least a
  constant probability.
• (for all p gt 2logN/N)
• In fact, number of matching Pennies is
  asymptotically Poisson(1) with high probability.

40
Expected of pure Nash

• For each strategy profile, a assign an indicator
  Xa 1 if a is a PNE. Define the RV S to be the
  sum of Xa

1
41
Poisson limit behavior
In order to establish Poisson limit
behavior Suffices to find for each strategy
profile a a set B(a) of strategies such that Xa
is independent of all strategies out not in B(a)
S1 ?a ?b 2 B(a) EXa Xb o(1) S2 ?a ?b
2 B(a) EXa EXb o(1) This follows from
Steins method (Arratia et. Al).
42
Poisson on random graphs

• Given the graph G, and a0,
• B(0) x 9 i 8 j with (i,j) 2 G x(j) 0
• Claim X0 is independent from (Xj j 2 B0).
• Pf No information about responses outside B(0)
• In order to prove Poisson behavior, suffices to
  show that ES1, ES2 -gt 0.

S
43
Poisson on random graphs
Lemma 1 If a is of weight s then EGEXa EX0
1(a in B(0)) 2-2n PGa 2 B(0) 2-2n min(1,n
(1-p)s-1) Lemma 2 If a is of weight s
then EGEXa X0 1(a in B(0)) 2-2n (1
(1-ps))n-s (1-(1-ps))n-s) After some
calculation this gives the desired result.
44
Future research

• Find exact threshold.
• Prove Poisson behavior.
• Other models of random graphs?
• Efficiently finding equilibrium?