PCA, pPCA, ICA

1
Lecture 14

• PCA, pPCA, ICA

2
Principal Components Analysis

• PCA is a data analysis technique to find the
  subspace of input space that carries most of the
  variance of the data.
• It is therefore useful as a tool to reduce the
  dimensionality of input space.
• The solution is found by an eigen-value
  decomposition of the sample covariance matrix.
• PPCA is a probabilistic model that has ML
  solution equal to the PCA solution. It is a
  special case of FA with isotropic variance.
• Therefore, the EM algorithm for FA is applicable
  for learning.

3
Independent Component Analysis

• FA, PPCA have Gaussian prior models. In ICA we
  use non-Gaussian prior models (i.e. heavy tailed
  or bi-modal).
• We also do not insist on dimensionality
  reduction, although that is also possible, but
  not necessary.
• The canonical example is 2 speakers producing
  different mixtures of sound in 2 microphones that
  we wish to unmix.
• The source distributions are non-Gaussian but
  independent, the noise model is typically
  Gaussian.
• The simplest ICA model is square and has no
  noise. We can use a change of variable to go from
  sources to inputs.
• Learning is through gradient descend with the
  natural gradient.