Expectation Propagation for Graphical Models

1
Expectation Propagation for Graphical Models

• Yuan (Alan) Qi
• Joint work with Tom Minka

2
Motivation

• Graphical models are widely used in real-world
  applications, such as wireless communications and
  bioinformatics.
• Inference techniques on graphical models often
  sacrifice efficiency for accuracy or sacrifice
  accuracy for efficiency.
• Need a new method that better balances the
  trade-off between accuracy and efficiency.

3
Motivation
Accuracy
Current Techniques
Efficiency
4
Outline

• Background
• Expectation Propagation (EP) on dynamic systems
• Poisson tracking
• Signal detection for wireless communications
• Tree-structured EP on loopy graphs
• Conclusions and future work

5
Outline

• Background
• Expectation Propagation (EP) on dynamic systems
• Poisson tracking
• Signal detection for wireless communications
• Tree-structured EP on loopy graphs
• Conclusions

6
Graphical Models
Directed Undirected
Generative Bayesian networks Boltzman machines
Conditional (Discriminative) Maximum entropy Markov models Conditional random fields
7
Inference on Graphical Models

• Bayesian inference techniques
• Belief propagation(BP) Kalman filtering
  /smoothing, forward-backward algorithm
• Monte Carlo Particle filter/smoothers, MCMC
• Loopy BP typically efficient, but not accurate
• Monte Carlo accurate, but often not efficient

8
Efficiency vs. Accuracy
MC
EP ?
Accuracy
BP
Efficiency
9
Expectation Propagation in a Nutshell

• Approximate a probability distribution by
  simpler parametric terms
• Each approximation term lives in an
  exponential family (e.g. Gaussian)

10
Update Term Approximation

• Iterate the fixed-point equation by moment
  matching

Where the leave-one-out approximation is
11
Outline

• Background
• Expectation Propagation (EP) on dynamic systems
• Poisson tracking
• Signal detection for wireless communications
• Tree-structured EP on loopy graphs
• Conclusions

12
EP on Dynamic Systems
Directed Undirected
Generative Bayesian networks Boltzman machines
Conditional (Discriminative) Maximum entropy Markov models Conditional random fields
13
Object Tracking
Guess the position of an object given noisy
observations
Object
14
Bayesian Network
e.g.
(random walk)
want distribution of xs given ys
15
Approximation
(proportional)
Factorized and Gaussian in x
16
Message Interpretation
(forward msg)(observation)(backward msg)
Forward Message
Backward Message
Observation Message
17
EP on Dynamic Systems

• Filtering t 1, , T
• Incorporate forward message
• Initialize observation message
• Smoothing t T, , 1
• Incorporate the backward message
• Compute the leave-one-out approximation by
  dividing out the old observation messages
• Re-approximate the new observation messages
• Re-filtering t 1, , T
• Incorporate forward and observation messages

18
Extension of EP

• Instead of matching moments, use any method for
  approximate filtering.
• Examples Extended Kalman filter, statistical
  linearization, unscented filter
• All methods can be interpreted as finding
  linear/Gaussian approximations to original terms

19
Example Poisson Tracking

• is an integer valued Poisson variate with
  mean

20
Poisson Tracking Model
21
Approximate Observation Message

• is not Gaussian
• Moments of x not analytic
• Two approaches
• Gauss-Hermite quadrature for moments
• Statistical linearization instead of
  moment-matching
• Both work well

22
EP Accuracy Improves Significantly in only a few
Iterations
23
Approximate vs. Exact Posterior
24
EP vs. Monte Carlo Accuracy
Mean
Variance
25
Accuracy/Efficiency Tradeoff
26
EP for Digital Wireless Communication

• Signal detection problem
• Transmitted signal st
• vary to encode each symbol
• Complex representation

Im
Re
27
Binary Symbols, Gaussian Noise

• Symbols are 1 and 1 (in complex plane)
• Received signal yt
• Optimal detection is easy

28
Fading Channel

• Channel systematically changes amplitude and
  phase
• changes over time

29
Benchmark Differential Detection

• Classical technique
• Use previous observation to estimate state
• Binary symbols only

30
Bayesian network for Signal Detection
31
On-line EP Joint Signal Detector and Channel
Estimation

• Iterate over the last observations
• Observations before act as prior for the
  current estimation

32
Computational Complexity

• Expectation propagation O(nLd2)
• Stochastic mixture of Kalman filters O(LMd2)
• Rao-blackwised paricle smoothers O(LMNd2)
• n Number of EP iterations (Typically, 4 or 5)
• d Dimension of the parameter vector
• L Smooth window length
• M Number of samples in filtering
• N Number of samples in smoothing

33
Experimental Results
(Chen, Wang, Liu 2000)
EP outperforms particle smoothers in efficiency
with comparable accuracy.
34
Bayesian Networks for Adaptive Decoding
The information bits et are coded by a
convolutional error-correcting encoder.
35
EP Outperforms Viterbi Decoding
36
Outline

• Background
• Expectation Propagation (EP) on dynamic systems
• Poisson tracking
• Signal detection for wireless communications
• Tree-structured EP on loopy graphs
• Conclusions

37
EP on Boltzman machines
Directed Undirected
Generative Bayesian networks Boltzman machines
Conditional (Discriminative) Maximum entropy Markov models Conditional random fields
38
Inference on Grids
Problem estimate marginal distributions of the
variables indexed by the nodes in a loopy graph,
e.g., p(xi), i 1, . . . , 16.
39
Boltzmann Machines
Joint distribution is product of pair potentials
Want to approximate by a simpler distribution
40
BP vs. EP
EP
BP
41
Junction Tree Representation

• p(x) q(x)
  Junction tree

42
Approximating an Edge by a Tree
Each potential f a in p is projected onto the
tree-structure of q
Correlations are not lost, but projected onto the
tree
43
Moment Matching

• Match single and pairwise marginals of
• Reduces to exact inference on single loops
• Use cutset conditioning

and
44
Local Propagation

• Original EP globally propagate evidence to the
  whole tree
• Problem Computationally expensive
• Exploit the junction tree representation only
  locally propagate evidence within the minimal
  subtree that is directly connected to the
  off-tree edge.
• Reduce computational complexity
• Save memory

45
Global propagation
Local propagation
46
4-node Graph

• TreeEP the proposed method, BP loopy
  belief propagation, GBP generalized belief
  propagation on triangles, MF mean-field, TreeVB
  variational tree.

47
Fully-connected graphs

• Results are averaged over 10 graphs with randomly
  generated potentials
• TreeEP performs the same or better than all other
  methods in both accuracy and efficiency!

48
8x8 grids, 10 trials
Method FLOPS Error
Exact 30,000 0
TreeEP 300,000 0.149
BP/double-loop 15,500,000 0.358
GBP 17,500,000 0.003
49
TreeEP versus BP and GBP

• TreeEP is always more accurate than BP and is
  often faster
• TreeEP is much more efficient than GBP and more
  accurate on some problems
• TreeEP converges more often than BP and GBP

50
Outline

• Background
• Expectation Propagation (EP) on dynamic systems
• Poisson tracking
• Signal detection for wireless communications
• Tree-structured EP on loopy graphs
• Conclusions

51
Conclusions
Accuracy
Efficiency

• EP algorithms outperform state-of-art inference
  methods on graphical models in the trade-off
  between accuracy and efficiency

52
Future Work

• EP is applicable to a wide range of applications
• EP is sensitive to choice of approximation
• How to choose an approximation family (e.g. tree
  structure)
• More flexible approximation mixture of EP?
• Error bound?

53
Future Work
Directed Undirected
Generative Bayesian networks Boltzman machines
Conditional (Discriminative) Maximum entropy Markov models Conditional random fields
54
End
55
EP versus BP

• EP approximation is in a restricted family, e.g.
  Gaussian
• EP approximation does not have to be factorized
• EP applies to many more problems
• e.g. mixture of discrete/continuous variables

56
EP versus Monte Carlo

• Monte Carlo is general but expensive
• EP exploits underlying simplicity of the problem
  if it exists
• Monte Carlo is still needed for complex problems
  (e.g. large isolated peaks)
• Trick is to know what problem you have

57
(Loopy) Belief propagation

• Specialize to factorized approximations
• Minimize KL-divergence match marginals of
  (partially factorized) and
  (fully factorized)
• send messages

messages
58
Limitation of BP

• If the dynamics or measurements are not linear
  and Gaussian, the complexity of the posterior
  increases with the number of measurements
• I.e. BP equations are not closed
• Beliefs need not stay within a given family

or any other exponential family
59
Approximate filtering

• Compute a Gaussian belief which approximates the
  true posterior
• E.g. Extended Kalman filter, statistical
  linearization, unscented filter, assumed-density
  filter

60
EP perspective

• Approximate filtering is equivalent to replacing
  true measurement/dynamics equations with
  linear/Gaussian equations

Gaussian
implies
Gaussian
61
EP perspective

• EKF, UKF, ADF are all algorithms for

Linear, Gaussian
Nonlinear, Non-Gaussian
62
Terminology

• Filtering p(xty1t )
• Smoothing p(xty1tL ) where Lgt0
• On-line old data is discarded (fixed memory)
• Off-line old data is re-used (unbounded memory)

63
Kalman filtering / Belief propagation

• Prediction
• Measurement
• Smoothing