Local Fisher Discriminant Analysis for Supervised Dimensionality Reduction

1
Local Fisher Discriminant Analysis for Supervised
Dimensionality Reduction

Masashi Sugiyama

• Presented by Xianwang Wang

2
Dimensionality Reduction

• Goal
• Embed high-dimensional data to low-dimensional
  space
• Preserve intrinsic information
• Example

High dimension
3-dimension
3
Categories

• Nonlinear
• ISOMAP
• Locally Linear Embedding (LLE)
• Laplacian Eigenmap (LE)
• Linear
• Principal Components Analysis (PCA)
• Locality-Preserving Projection (LPP)
• Fisher Discriminant Analysis (FDA)
• Unsupervised
• S-ISOMAP, S-LLE, PCA
• Supervised
• LPP, FDA

4
Formulation

• Number of samples
• d-dimensional samples
• Class labels
• Number of samples in the class
• Data matrix
• Embedded samples

5
Goal for linear dimensionality Reduction

• Find a transformation matrix
• Use Iris data for demos (http//archive.ics.uci.ed
  u/ml/machine-learning-databases/iris/iris.data)
• Attribute Information
• sepal length in cm
• sepal width in cm
• petal length in cm
• petal width in cm
• class
• Iris Setosa Iris Versicolour Iris Virginica

6
FDA(1)

• Mean of samples in the class
• Mean of all samples
• Within-class scatter matrix
• Between-class scatter matrix

7
FDA(2)

• Maximize the following objective
• Maximize the following constrained optimization
  problem equivalently
• Use the lagrangian,
• Apply KKT conditions
• Demo

8
LPP

• Minimize
• Equivalently
• We can get
• Demo

9
Local Fisher Discriminant Analysis(LFDA)

• FDA can perform poorly if samples in some class
  form several separate clusters
• LPP can make samples of different classes
  overlapped if they are close in the original high
  dimensional space
• LFDA combines the idea of FDA and LPP

10
LFDA(1)

• Reformulating FDA

11
LFDA(2)

• Definition of LFDA

12
LFDA(3)

• Maximize the following objective
• Equivalently,
• Similarly, we can get
• Demo

13
Conclusion

• LFDA provided more separate embedding than FDA
  and LPP
• FDA (globally), while LFDA(locally)
• More discussion about efficiently computing of
  LFDA transformation matrix and Kernel LFDA in the
  paper

14

• Questions?