Distant Kin in the EM Family

1
Distant Kin in the EM Family

• David A. van Dyk
• Department of Statistics
• University of California, Irvine
• .
• (Joint work with Xiao Li Meng and Taeyong Park.)

2
Outline

• The EM Family Tree
• The Stochastic Cousins
• Some Odd Relations
• Nested EM and The Partially Blocked Gibbs Sampler
• A Newly Found Kinsman
• A Stochastic ECME/AECM Sampler
• The Partially Collapsed Gibbs Sampler

3
The EM Family Tree
1977 (Dempster, Laird, Rubin)
1987 (Tanner Wong)
1990 (Wie Tanner)
1998 (Liu, Rubin, Wu)
1995 (van Dyk, Meng, Rubin)
1991 (Meng Rubin)
1993 (Meng Rubin)
2000 (van Dyk)
1994 (Liu Rubin)
1997 (Meng van Dyk)
EM Algorithm
Stochastic Simulation
Variance Calculations
Gauss- Seidel
Monte Carlo Integration
Efficient DA
ECM
SEM
DA sampler
MCEM
ECME
Efficient DA EM
NEM
SECM
Algorithms
Methods
PXEM
AECM
4
Stochastic Cousins
EM Algorithm
DA Sampler
Gauss- Seidel
Monte Carlo Integration
Gibbs Sampler
Efficient DA
ECM
MCEM
Partially Blocked Gibbs Sampler
ECME
Efficient DA EM
NEM
???
Marginal DA PX-DA
PXEM
AECM
5
The EM and DA Algorithms
DA Sampler
EM Algorithm
p(M?)
p(M?)
Expectation Step
Maximization Step
p(?M)
p(?M)
Random Draw
BACK
6
An NEM Algorithmwith a Monte Carlo E-step
NEM Algorithm
The Stochastic Version
p(M1M2,?)
p(M2M1,?)
p(M1M2,?)
p(M2M1,?)
p(M1?)
p(M?)
p(M2M1,?)
p(?M)
p(M2M1,?)
p(?M)
E-Step
M-Step
p(?M1)
p(?, M2M1)
Draw
7
A Partially-Blocked Sampler
NEM Algorithm
Partially-Blocked Sampler
p(M1M2,?)
p(M2M1,?)
p(M1?, M2)
p(M1?)
p(M2M1,?)
p(?M)
p(M2M1,?)
p(?M)
E-Step
M-Step
p(?M1)
p(?, M2M1)
Draw
BACK
8
Ordering CM-steps in ECME
ECME Algorithm
ECME Algorithm
p(M?)
p(M?)
Reducing conditioning Speed up convergence!
But BE CAREFUL! Step order Matters!
p(?2?1)
p(?1M,?2)
p(?1M, ?2)
p(?2?1)
E-Step
M-Step
Monotone Convergence
NO Monotone Convergence
Draw
9
A Stochastic Version of ECME
ECME Algorithm
What is the stationary distribution of this
chain??
p(M?)
p(M?)
p(?2?1)
p(?1M,?2)
p(?2?1)
p(?1M, ?2)
E-Step
M-Step
Incompatible Conditional Distributions
Draw
10
AECM and Partially Collapsed Samplers
p(M?)
p(M?)
p(M,?2?1)
Blocked Sampler
ECME
Partially Collapsed
p(M,?2?1)
p(?1M,?2)
p(?1M,?2)
p(?2?1)
p(?1M,?2)
p(M?)
p(M?)
Incompatible draws
AECM
Partially Collapsed
p(?1M,?2)
p(?2M1,?1)
p(?2M1,?1)
p(?1M,?2)
Stationary distribution must be verified!
E
M
D
11
Completely Collapsed Samplers
p(M?)
p(M?)
ECME
Collapsed Sampler
Complete Collapse
p(?2?1)
p(?1?2)
p(M,?2?1)
p(M,?1?2)
p(?1M,?2)
p(?2?1)

• Blocking (ECME) is a special case of Partial
  Collapse (AECM).
• We expect Collapsed Samplers (CM) to perform
  better than Partially Collapsed Samplers (AECM).
• And we expect Collapsing (CM) to perform better
  than Blocking (ECME).
• Many of these relationships are known, I
  emphasize the connections between EM-type DA-type
  algorithms.

E
M
D
12
Reducing Conditioning in Gibbs The Simplest
Example

• Consider a two-step Gibbs Sampler

The Markov Chain has stationary distn With
target margins but Without the correlation of the
target distribution AND converges quickly!

Iteration t
Iteration t1/2
We regain the target distribution with a one-step
shifted chain.
13
Heads Up!!

• Reducing the conditioning within Gibbs involves
  new challenges
• The order of the draws may effect the stationary
  distribution of the chain.
• The conditional distributions may no be
  compatible with any joint distribution.
• The steps sometimes can be blocked to form an
  ordinary Gibbs sampler with fewer steps.

14
An Example from Astronomy
Spectral Model for Photon Counts

• Parameterized Latent Poisson Process
• Underlying Poisson intensity is a mixture of a
  broad feature and several narrow features.
• The line location and mixture indicator are
  highly correlated.

Emission Line
Photon energy
Line location
15
An Example from Astronomy
Spectral Model for Photon Counts

• Standard sampler simulates

which may converge very slowly or not at all.
Emission Line
Photon energy
Line location
16
An Incompatible Gibbs Sampler
17
Computational Gains
18
Verifying the Stationary Distribution of Sampler 2
19
The General Strategy
BACK