1. Robust Fisher Discriminant Analysis
Article presented at NIPS 2005
By Seung-Jean Kim, Alessandro Magnani, Stephen P. Boyd
Presenter: Erick Delage
February 14, 2006

2. Outline Background on Fisher Linear Discriminant Analysis
Making the approach robust to small sample sets while maintaining computation efficiency
Experimental results

3. Fisher Discriminant Analysis
Given two Random Variables X,Y n, find the linear discriminant  n that maximizes Fisher’s discriminant ratio:
Unique solution :
Easy to compute
Probabilistic interpretation
Kernelizable
Naturally extends to k-class problems

4. Probabilistic Interpretation
The Fisher discriminant is the Bayes optimal classifier for two normal distributions with equal covariance.
Fisher discriminant analysis can be shown to:

5. Using Kernels When discriminating in feature space (x).
We can use kernels:
And show that  is of the form:
A projection along  given by:
And find  by solving:
K_i,j = k(x_i,x_j) where x_I, x_j are data samples I and j.

6. Robust Fisher Discriminant Analysis
Uncertainty in (x, x) & (y, y)
FDA is sensitive to estimation errors of these parameters.
Can we make it more robust using general convex uncertainty models on the problem data?
Is it still a computationally feasible technique?

7. Max Worst-case Fisher Discriminant ratio
Assuming
,where U is a convex compact subset.
We can try optimizing:
From basic min-max theory, we know (1)  (2)

8. Max Worst-case Fisher Discriminant ratio

9. Max Worst-case Fisher Discriminant ratio
Given ,
Because , (1) is equivalent to (2) which is convex and can be solved efficiently using a tractable general methods (e.g. interior point methods).

10. Experimental Results
Two benchmark problems from the machine learning repository
Sonar: 208 points, n = 60
Ionosphere: 351 points, n = 34
Uncertainty models:

11. Experimental Results

12. References
S.-J. Kim, A. Magnani, and S. P. Boyd: Robust Fisher Discrminant Analysis. In T. Leen, T. Dietterich and V. Tresp, editors, Advances in Neural Information Processing Systems, 18, pp ,MIT Press, 2006.
S. Mika, G. Rätsch, and K.-R. Müller: A Mathematical Programming Approach to the Kernel Fisher Algorithm. In T. Leen, T. Dietterich and V. Tresp, editors, Advances in Neural Information Processing Systems, pp ,MIT Press, 2000.