Newton Method for the ICA Mixture Model

1
Newton Method for theICA Mixture Model

• Jason A. Palmer1 Scott Makeig1Ken
  Kreutz-Delgado2 Bhaskar D. Rao2
• 1 Swartz Center for Computational Neuroscience2
  Dept of Electrical and Computer
  EngineeringUniversity of California San Diego,
  La Jolla, CA

2
Introduction

• Want to model sensor array data with multiple
  independent sources ICA
• Non-stationary source activity mixture model
• Want the adaptation to be computationally
  efficient Newton method

3
Outline

• ICA mixture model
• Basic Newton method
• Positive definiteness of Hessian when model
  source densities are true source densities
• Newton for ICA mixture model
• Example applications to analysis of EEG

4
ICA Mixture Modeltoy example

• 3 models in two dimensions, 500 points per model
• Newton method converges lt 200 iterations, natural
  gradient fails to converge, has difficulty on
  poorly conditioned models

5
ICA Mixture Model

• Want to model observations x(t), t 1,,N,
  different models active at different times
• Bayesian linear mixture model, h 1, . . . , M
• Conditionally linear given the model,
  
• Samples are modeled as independent in time

6
Source Density Mixture Model

• Each source density mixture component has unknown
  location, scale, and shape
• Generalizes Gaussian mixture model,
  more peaked, heavier tails

7
ICA Mixture ModelInvariances

• The complete set of parameters to be estimated
  is
• h 1, . . ., M, i 1, . . ., n, j 1, . .
  ., m
• Invariances W row norm/source density scale and
  model centers/source density locations

8
Basic ICA Newton Method

• Transform gradient (1st derivative) of cost
  function using inverse Hessian (2nd derivative)
• Cost function is data log likelihood
• Gradient
• Natural gradient (positive definite transform)

9
Newton Method Hessian

• Take derivative of (i,j)th element of gradient
  with respect to (k,l)th element of W
• This defines a linear transform
• In matrix form, this is

10
Newton Method Hessian

• To invert rewrite the Hessian transformation
  in terms of the source estimates
• Define , ,
• Want to solve linear equation

11
Newton Method Hessian

• The Hessian transformation can be simplified
  using source independence and zero mean
• This leads to 2x2 block diagonal form

12
Newton Direction

• Invert Hessian transformation, evaluate at
  gradient
• Leads to the following equations
• Calculate the Newton direction

13
Positive Definiteness of Hessian

• Conditions for positive definiteness
• Always true for true when model source densities
  match true densities
• 1)
• 2)
• 3)

14
Newton for ICA Mixture Model

• Similar derivation applies to ICA mixture model

15
Convergence Rates

• Convergence is really much faster than natural
  gradient. Works with step size 1!
• Need correct source density model

log likelihood
iteration
iteration
16
Segmentation of EEG experiment trials
3 models
4 models

trial
trial
time
time
log likelihood
log likelihood
iteration
iteration
17
Applications to EEGEpilepsy
1 model
5 models

log likelihood
time
time
log likelihood difference from single model
time
18
Conclusion

• We applied method of Amari, Cardoso and Laheld,
  to formulate a Newton method for the ICA mixture
  model
• Arbitrary source densities modeled with
  non-gaussian source mixture model
• Non-stationarity modeled with ICA mixture model
  (multiple mixing matrices learned)
• It works! Newton method is substantially faster
  (superlinear). Also Newton can converge when
  Natural Gradient fails

19
Code

• There is Matlab code available!!
• Generate toy mixture model data for testing
• Full method implemented mixture sources, mixture
  ICA, Newton
• Extended version of paper in preparation, with
  derivation of mixture model Newton updates
• Download from
• http//sccn.ucsd.edu/jason

20
Acknowledgements

• Thanks to Scott Makeig, Howard Poizner, Julie
  Onton, Ruey-Song Hwang, Rey Ramirez, Diane
  Whitmer, and Allen Gruber for collecting and
  consulting on EEG data
• Thanks to Jerry Swartz for founding and providing
  ongoing support the Swartz Center for
  Computational Neuroscience
• Thanks for your attention!

21
Newton for ICA Mixture Model