1. A Hybrid PCA-LDA Model for Dimension Reduction Nan Zhao1, Washington Mio2 and Xiuwen Liu1 1Department of Computer Science, 2Department of Mathematics Florida State University, Tallahassee, FL, 32306
Introduction:
Even though Linear Discriminant Analysis (LDA) is preferred in many application of dimension reduction, it does not always outperform Principle Component Analysis (PCA) [1].
The goal of this paper is to develop a discriminative dimension reduction model for the choice of a 1-dimensional subspace that yields an optimal balance between generalization and class discrimination to new datat
The Hybrid PCA-LDA Model:
A 1-dimensional subspace will be represented by a unit vector e.
<u,v> is the usual product of vectors u and v belonging to Rm
The cost functions associated with PCA and regularized LDA are
respectively, where є > 0 is a small number that prevents the occurrence of vanishing denominators.
For each t, 0 ≤ t ≤ 1, we propose the linear interpolation of the cost functions for PCA and LDA:
The goal is to find a unit vector et belonging to Rm that maximize the cost function Ft(e):
The main computational task is to calculate et. Once et is known, we choose t so that the classification performance is optimized under projection to the subspace spanned by et.
Experiment 2 – A Comparison between the Hybrid Model and the Two-stage Model
The average performance over all runs using the hybrid model (left figure)
The average performance over all runs using the two-stage model (right figure)
Conclusion
Neither pure LDA (t=1) nor PCA (t=0) outperform the hybrid model.
The hybrid model is superior to the two-stage model concerning the classification performance.
Analysis:
A combination between PCA and LDA is reasonable:
Performance
PCA followed by LDA > pure LDA [2][3]
Pros
PCA reserve geometry and clustering patterns for generalization
LDA reserve discriminative information for classification
Cons
PCA may lose potential discriminative information in the ignored principle components
PCA may involve useless information for classification in the first few components
Conclusion
A more intrinsic way is needed rather than combining PCA and LDA in two separate stages
Future work and conclusion
The model and the methodology will be extended to subspace higher than 1-dimension
Note that after the low dimensional representation is learned, we can use any classifier, including SVM and neural networks for classification and comparison.
PCA can be replaced by other generative models and LDA by other discriminative algorithms
Reference
[1] A.-M. Martinez and A.-C. Kak, “PCA versus LDA,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 2, pp , 2001.
[2] J. Yang and J.-Y. Yang, “Why can LDA be performed in PCA transformed space?” Pattern Recognition, vol. 36, pp , 2003.
[3] D.-L. Swets and J. Weng, “Using discriminant eigenfeatures for image retrieval,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 18, no. 8, pp , 1996.
Experiment 1 – The Choice of ε
Observation:
The optimal choice of t, based on
cross-validation data, is not sensitive
to the choice of є, so long as є is small.
Acknowledgements
This research was supported in part by NSF grand DMS and NIH Roadmap for Medical Research grant U54 RR021813