Multivariate Regression

1
Multivariate Regression- Techniques and
ToolsHeikki Hyötyniemi
2
LESSON 7

• Towards the Structure

3
New Interpretation

• First, determine the system structure
  Then, use this structure for model
  construction

4
About Structure

• Now, underlying variables searched
  Visible measurements are linear combinations
  of these original sources
• When the hidden basis is solved, familiar
  regression methods can be applied
• Remember that causalities and real structures
  cannot be determined from data alone

5
Factor Analysis

• Familiar setting Decompose x so that
• but additionally

6

• Goal Uncorrelated errors
• Algorithms are based on iteration, PCA can be
  used as the initial guess
• Results are not unique, more or less heuristic
  rotations can be applied
• Could one define a more restrictive criterion?

7
Independence

• Independent variables give no information about
  each other
• Independence is more than uncorrelatedness

8
Uncorrelated Variables ...
9
... vs. Independent Ones
10

• Independence originality
• Combinations of different variables typically
  have more or less Gaussian distribution
• Inverse idea If the variable distribution is
  non-Gaussian, it cannot be composite!

11
Characterizing Distributions

• Moments characterize a
  distribution
• Cumulants are semantic functions of moments
  First is mean, second is variance, third is
  skewness, fourth is peakedness, etc.
• Gaussian distribution is characterized by the two
  first cumulants alone, mean and variance

12
Role of Kurtosis

• Non-Gaussianity Any of the higher cumulants of
  the distribution differs from zero
• Typically emphasis on the fourth cumulant
  kurtosis
• If data normalized

13
Extra Preprocessing Needed

• Typically, low-order data properties dominate
  when kurtosis should be made visible
• cumulants eliminated when data is centered
• cumulants neutralized when data is whitened
• cumulants eliminated when data symmetricity is
  assumed.

14
ICA

• Typically studied in algorithmic, iterative form
  (Hyvärinen, Oja, Pajunen, ...)
• Now we want to present ICA in the powerful,
  homogeneous eigenproblem framework
• The data has to be additionally preprocessed so
  that the fourth-order properties can be analyzed
  using the second-order methodologies

15
FOBI

• If one defines
• there holds

16
See Any Structure Here?
17
Extracted Sources!
18
Failure of Independence Idea
19
Overcomplete Basis Sets

• N gt n

20
Sparse Coding

• General framework for different kinds of mixture
  models