Robust Fisher Discriminant Analysis

1
Robust Fisher Discriminant Analysis

• Article presented at NIPS 2005
• By Seung-Jean Kim, Alessandro Magnani, Stephen P.
  Boyd
• Presenter Erick Delage
• February 14, 2006

2
Outline

• Background on Fisher Linear Discriminant Analysis
• Making the approach robust to small sample sets
  while maintaining computation efficiency
• Experimental results

3
Fisher Discriminant Analysis

• Given two Random Variables X,Y ??n, find the
  linear discriminant ? ??n that maximizes Fishers
  discriminant ratio

• Unique solution
• Easy to compute
• Probabilistic interpretation
• Kernelizable
• Naturally extends to k-class problems

4
Probabilistic Interpretation
5
Using Kernels

• When discriminating in feature space ?(x).
• We can use kernels
• And show that ? is of the form
• A projection along ? given by
• And find ? by solving

6
Robust Fisher Discriminant Analysis

• Uncertainty in (?x, ?x) (?y, ?y)
• FDA is sensitive to estimation errors of these
  parameters.
• Can we make it more robust using general convex
  uncertainty models on the problem data?
• Is it still a computationally feasible technique?

7
Max Worst-case Fisher Discriminant ratio
8
Max Worst-case Fisher Discriminant ratio
9
Max Worst-case Fisher Discriminant ratio
Given ,
10
Experimental Results

• Two benchmark problems from the machine learning
  repository
• Sonar 208 points, n 60
• Ionosphere 351 points, n 34
• Uncertainty models

11
Experimental Results
12
References

• S.-J. Kim, A. Magnani, and S. P. Boyd Robust
  Fisher Discrminant Analysis. In T. Leen, T.
  Dietterich and V. Tresp, editors, Advances in
  Neural Information Processing Systems, 18, pp
  659-666 ,MIT Press, 2006.
• S. Mika, G. Rätsch, and K.-R. Müller A
  Mathematical Programming Approach to the Kernel
  Fisher Algorithm. In T. Leen, T. Dietterich and
  V. Tresp, editors, Advances in Neural Information
  Processing Systems, pp 591-597,MIT Press, 2000.