Independent Component Analysis ICA

1
Independent Component Analysis (ICA)

• Jaymie Strecker
• CMSC828D
• May 10, 2006

2
Outline

• ICA a data transformation
• Blind source separation
• Single-channel
• Multi-channel/spatial
• Other applications
• Identifiability of ICA model
• ICA model estimation
• FastICA
• History

3
Data transformation

• Suitable representation
• Suitable transformation
• Preprocessing for analysis
• Simple example speech recognition

4
Transformation problem

• Linear mixture
• Random variables xobserved, ssources
• Linear transform
• Solve for a and S

5
Selecting a transformation

• Optimize
• Dimension reduction
• Statistical interestingness
• Simplicity of W
• Application-oriented criteria

ICA
6
Statistical interestingness

• Find W that makes components si as statistically
  independent as possible
• Maximize some F(s1,,sm) that measures
  independence
• Objective (contrast) function deferred
• 2nd-order vs. higher-order measures of
  interestingness

7
2nd-order linear trans.

• Use only the info in the covariance matrix of x
• Completely determined if x is Gaussian
• Minimize mean-square error
• Reduce data dimension
• Examples
• Principal component analysis (PCA)
• Factor analysis

8
Higher-order linear trans.

• Use info beyond covariance matrix of x
• x non- Gaussian
• Clustering
• Independence of components
• Examples
• Projection pursuit
• Redundancy reduction
• Blind deconvolution

9
2nd-order vs. higher-order
10
ICA and related methods
11
Blind source separation (BSS)

• Unknown aij (mixing weights), si
• Assume W exists and s non-Gaussian
• Solution with statistically independent
  components is unique

12
Special cases of BSS

• Cocktail party problem
• n simultaneous speakers
• At least n listeners
• Denoising
• Brain signals (EEG and MEG)
• 23D and 122D signals
• Economic time series

13
Cocktail party demos

• http//www.cis.hut.fi/projects/ica/cocktail/cockta
  il_en.cgi
• ICA method
• http//www.cnl.salk.edu/tewon/Blind/blind_audio.h
  tml
• TDD-ICA hybrid method

14
Spatial audio and the cocktail party problem

• Certain factors can enhance ability to monitor
  multiple speech sources
• Number of talkers
• Sex of talker
• Spatial separation
• Temporal asynchrony
• Virtual spatial audio displays

(Nelson et al, 1999)
15
BSS and beamforming

• Both recover mixed signals
• BSS uses time and frequency info
• Adaptive beamforming uses spatial info
• Must know direction to aim

(Pan and Aboulnasr, 2005)
16
BSS with time delays

• Optimize to find weights and delays

(Torkkola, 1996)
17
Identifiability of ICA model

• Assume sources statistically independent
• Guaranteed when
• No more than one component of s is Gaussian
• m gt n
• m number of observed linear mixtures
• n number of independent components
• A has full column rank

18
Independence of components

• What does it mean for the components to be
  independent?
• Statistical independence
• f(y1,...,ym)f1(y1)f2(y2)...fm(ym)
• Uncorrelatedness (weaker)

19
ICA method

• ICA method Objective function Optimization
  algorithm
• Algorithm minimizes/maximizes function

20
Objective (contrast) function

• Multi-unit (all components at once)
• Likelihood
• Entropy
• Mutual information
• One-unit (one component at a time projection
  pursuit)
• Negentropy
• Kurtosis (a measure of non-Gaussianity)

21
Optimization algorithm

• Maximum likelihood or infomax
• Gradient ascent (stochastic or not)
• Non-linear PCA
• Iterative extract principal component
• Neural algorithms
• FastICA
• Can converge faster than stochastic gradient
  ascent and neural algorithms

22
FastICA

• Fixed-point iteration
• Data has been sphered (covariance matrix equals
  unity)
• Comparison to neural algorithms
• Parallel and distributed
• Not adaptive uses sample averages computed over
  sufficiently large data samples

23
FastICA Demo

• FastICA package for Matlab
• http//www.cis.hut.fi/projects/ica/fastica/
• Quick guided tour
• http//www.cis.hut.fi/projects/ica/icademo/
• Another implementation
• http//www.nic.uoregon.edu/hipersat/index.php

24
History

• Early 1980s Jerault, Jutten, Ans
• Neurophysiological problem (muscle contraction)
• 1990s wider use
• Infomax algorithm (Bell, Sejnowski)
• Connection to maximum likelihood (Amari et al)
• FastICA algorithm (Hyvarinen, Karhunen, Oja
• 1999 first international workshop
• 2006 6th International Conference on
  Independent Component Analysis and Blind Source
  Separation

25
References

• A. Hyvrainen, J. Karhunen, E. Oja. Independent
  Component Analysis. 2001. (http//www.cis.hut.fi/p
  rojects/ica/book/)
• A. Hyvarinen. Survey on Independent Component
  Analysis. 1999. (http//www.cis.hut.fi/aapo/paper
  s/NCS99web/)
• A. Hyvarinen, E. Oja. Independent Component
  Analysis A Tutorial. 1999. (http//www.cis.hut.fi
  /aapo/papers/IJCNN99_tutorialweb/)
• W. T. Nelson, R. S. Bolia, M. A. Ericson, R. L.
  McKinley. Spatial audio displays for speech
  communications A comparison of free field and
  virtual acoustic environments. 1999.
• Q. Pan, T. Aboulnasr. Combined spatial/beamforming
  and time/frequency processing for blind source
  separation. 2005.
• K. Torkkola. Blind separation of delayed sources
  based on information maximization. 1996.
• S. Ukai, T. Takatani, T. Nishikawa, H.
  Saruwatari. Blind source separation combining
  SIMO-model-based ICA and adaptive beamforming.
  2005.