Multi Dimensional signal proc' Lecture 3 5

1
Multi Dimensional signal proc. Lecture 35

• Dimensionality reduction
• Correlation between variables
• Correlation vs. covariance
• Methods to reduce the dimensionality
• Multi-dimensional scaling
• Isomap
• Local linear embedding (LLE)
• Presentation of students compression algorithms

2
Dimensionality reduction

• Why?
• The curse of dimensionality
• Visualization
• Remove noise (10 correlated variables 1
  uncorrelated)
• Save processing time
• Understand the data
• How?
• Variables/features are often correlated gt
  redundancy
• How to measure correlation covariance matrix
• Correlation vs. covariance

3
Covariance
4
Methods to reduce the dimensionality

• Principal component analysis (PCA)
• Multi-dimensional scaling (MDS)
• Isomap
• Local linear embedding (LLE)

5
Linear methods

• Principal component analysis (PCA)
• Find directions with large variance
• Multi-dimensional scaling (MDS)
• Keep interpoint distances

mapping
(note 12)
6
Multi-dimensional scaling (MDS)
7
Linear methods

• What about data that cannot be described by a
  linear combination of variables?
• Ex swiss roll, s-curve, the earth
• In general manifolds
• Linear methods fail
• In the end, linear methods do nothing more than
  rotate/translate/scale data

PCA
8
Non-linear methods

• Estimate the intrinsic structure
• The manifold

9
Intuition how does your brain store these
pictures?
10
Computer Representation
11
Brain Representation

• Every pixel?
• Or perceptually meaningful structure?
• Up-down pose
• Left-right pose
• Lighting direction
• So, your brain successfully reduced the
  high-dimensional inputs to an intrinsically
  3-dimensional manifold!

12
Non-linear methods

• Estimate the intrinsic structure (manifold)
• Methods
• Isomap
• Local linear embedding (LLE)

13
Isomap

• Isometric feature mapping
• Same principle as MDS, but use geodesic distance
  instead of Euclidean distance
• ( shortest distance along a manifold, eg. the
  Earth )
• Algorithm
• Find geodesic distance between all points
• Use MDS to map to lower dimensional space

14
Isomap Algorithm

• Find geodesic distance between all points
• 1) Construct neighborhood graph
• For each point Find K neighbors and measure
  Euclidean distance
• Graph Each point a node. Each edge is a distance
  (weight)
• 2) Compute shortest path between any two points
• Graph theory. Dijkstras algorithm
• 3) Use MDS to map to lower dimensional space

(note 3)
15
Isomap Examples
16
Isomap Example

• 64X64 Input Images form
• 4096-dimensional vectors
• Intrinsically, three dimensions is enough for
  presentations Two pose parameters and azimuthal
  lighting angle

17
Data for Hands
18
Local linear embedding (LLE)
19
Local linear embedding (LLE)

• We expect each data point and its neighbors to
  lie on (or close to) a locally linear patch of
  the manifold
• This local linearity should also exist after the
  mapping

20
LLE Algorithm

• Find neighbors for each point
• Represent each point as a weighted combination of
  its neighbors
• The weights are the same in the low dimensional
  space. Use this fact to do
• the mapping

(note 4)
21
LLE examples
22
(No Transcript)
23
LLE examples
PCA
LLE
24
What to remember

• Dimensionality reduction
• The curse of dimensionality
• Visualization
• Remove noise (10 correlated variables 1
  uncorrelated)
• Save processing time
• Understand the data the intrinsic structure
• How?
• Variables/features are often correlated gt
  redundancy
• Linear vs non-linear methods
• Linear PCA, MDS,
• Non-linear Isomap, LLE,
• Isomap vs LLE
• Different arguments
• In general Isomap is applied
• (slightly) more often
• The end!

25
Xtras
26
Isomap Data for Handwritten 2
27
LLE Examples
28
Principal Component Analysis (PCA)

• Metode til at kombinere features til nye og færre
• PCA er meget brugt, specielt ved mange
  dimensioner
• Grund tanke Features med stor varians adskiller
  klasserne bedst
• Begge features har
• stor varians hvad så ?
• Transformere feature-rummet,
• så vi opnår størst mulig varians
• og ingen korrelation !
• Varians Information !

29
PCA Transformationen

• Ignorere y2 uden at miste information
• ifbm. klassificering
• y1 og y2 er de principale komponenter

30
PCA Hvordan gør man ?

• Data (x)
• Udregn covarians matricen Cx
• Matlab Cx cov(x)
• Løs Eigen-værdi problemet gt A og Cy
• Matlab Evec,Eval eig(Cx)
• Transformere x gt y y A (x-m)
• Analyse (PCA)
• M-metoden
• Variabilitets mål fra SEPCOR
• J-mål

31
Hvad skal I huske ?

• Feature reduktion hvor vi ikke bruger klasse
  info
• Unsupervised
• Hierachical dimensionality reduction
• Correlation matrix
• Merge features eller smid nogle væk
• Principal Component Analysis (PCA)
• Kombineres til nye og færre features
• VARIANCE INFORMATION
• Transformere featurerummet udfra Eigen-vektorerne
  af covarians matricen gt ukorrelerede features !
• Analyse
• M-metoden (ingen klasse info.)
• Bruge klasse info.
• Variabilitets mål fra SEPCOR
• J-mål