The EM algorithm'

1
The EM algorithm.

• Niall Rea.
• 23/5/2003.

2
Tutorial Outline.

• Introduction
• Why
• Origins.
• EM from fundamentals.
• EM for line fitting.

3
Introduction

• Estimate model parameters from incomplete /
  missing data.
• Optimisation technique.
• Iterative 2 step algorithm
• Expectation Expected value of complete data log
  likelihood.
• Maximization Maximise the expectation
  expression.
• Applications
• Motion segmentation, colour segmentation.

4
Introduction Origins 1/2

• MLE technique.
• Density function
• Data set
• Resulting density
• MLE problem

Incomplete data likelihood expression
5
Introduction Origins 2/2

• If simple distribution Easy
• Set
• Solve for and
• But if distribution more difficult - EM

6
EM from fundamentals 1/7

• General method for MLE when data set contains
  missing variables / incomplete data.
• Term in data present / missing
• Simplify likelihood function. e.g. Hidden
  variables in HMM or mixture densities.
• Assume observed data generated by
  distribution.
• Introduce incomplete data
• Define a new joint density function

7
EM from fundamentals 2/7

• New likelihood function
• Justification Consider finding ML Mixture
  density parameters. Digression
• Define probabilistic model
• M component densities with M mixing coefficients.

8
EM from fundamentals 3/7

• Incomplete data LL expression for density from
  data

Log of sum. Difficult to optimise. Introduce
hidden variable whose value indicates which
component density generated which data item
i.e.
9
EM from fundamentals 4/7

• For each ,
• If is known, complete LL given by
• Problem unknown, can assume r.v. since
  unknown, random governed by underlying
  distribution

10
EM from fundamentals 5/7

• First step - Expectation Define the auxiliary
  function
• Find E-value of complete-data L.L. w.r.t
• given and current parameter set.

Current parameter set.
Random variable governed by
Parameter set to be optimised.
11
EM from fundamentals 7/7

• Maximisation step
• Maximisation of expectation
• Iterate process until convergence.
• Will always converge to local optimum
• c.f. M.L from incomplete data via the EM
  algorithm Dempster, Laird, Rubin.

12
EM for line fitting 1/12

• Mixture estimation problem.
• Observation model
• Assume normally distributed errors
• Estimate line parameters and assignment of each
  datapoint to the process that generated it.
• EM intuition - Need to know one to estimate the
  other.
• EM structure
• Random parameter assignment for line models
• Iterate until convergence
• E-step assign points to model that results in
  best fit
• M step update parameters of model using points
  assigned to it.

13
EM for line fitting 2/12

• E-step
• Define , the squared residual difference
  between observation at point and predictions of
  each model.
• Compute the posterior
• Probability of assignment of each point to
  particular model.

14
EM for line fitting 3/12

• M-Step For a given line , minimize

15
EM for line fitting 4/12

• Solving for for each line

16
EM for line fitting 5/12
17
EM for line fitting 6/12
18
EM for line fitting 7/12
19
EM for line fitting 8/12
20
EM for line fitting 9/12
21
EM for line fitting 10/12
22
EM for line fitting 11/12
23
EM for line fitting 12/12
24
Questions?