The Chinese University of Hong Kong Learning Larger Margin Machine Locally and Globally Dept. of Computer Science and Engineering The Chinese University of Hong Kong Shatin, NT. Hong Kong Kaizhu Huang February 9, 2004

The Chinese University of Hong Kong Contributions Theory: A unified model of Support Vector Machine (SVM), Minimax Probability Machine (MPM), and Linear Discriminant Analysis (LDA). Practice: A sequential Conic Programming Problem.

The Chinese University of Hong Kong Outline Background And Motivation Maxi-Min Margin Machine(M 4 ) –Model Definition –Geometrical Interpretation –Solving Methods –Connections With Other Models M 4 : Non-separable case Experimental Results Future Work Conclusion

The Chinese University of Hong Kong Background: Classifier

The Chinese University of Hong Kong Background: SVM A more reasonable decision plane SVM Support Vectors

The Chinese University of Hong Kong Maxi-Min Margin Machine(M 4 )

The Chinese University of Hong Kong M4:Geometrical Interpretation

The Chinese University of Hong Kong M4:Solving Method Basic Technique: Divide and Conquer –If we fix to a specific, the problem changes to check whether this satisfies the following constraints: –If yes, we increase ; otherwise, we decrease it. Second Order Cone Programming Problem!!!

The Chinese University of Hong Kong M4:Solving Method (Continue) Iterate the following two steps to solve M 4 :

The Chinese University of Hong Kong M4:Solving Method (Continue) can it satisfy the constraints? Yes No

The Chinese University of Hong Kong Connection with MPM Span all the data points and add them together + Exactly MPM Optimization Problem!!!

The Chinese University of Hong Kong Connection with MPM Remarks: –The procedure is not reversible: MPM is a special case of M 4 –MPM focuses on building decision boundary GLOBALLY, i.e., it exclusively depends on the means and covariances. However, means and covariances may not be accurately estimated. MPM M 4

The Chinese University of Hong Kong Connection With SVM The magnitude of w can scale up without influencing the optimization SVM with a further assumption:

The Chinese University of Hong Kong Connection With SVM M 4 SVM M 4 SVM

The Chinese University of Hong Kong Connection With SVM SVM assumes

The Chinese University of Hong Kong Links With LDA Perform the similar procedure as in MPM LDA

The Chinese University of Hong Kong Link With LDA

The Chinese University of Hong Kong Non-separable Case

The Chinese University of Hong Kong Experimental Results-Synthetic Toy example

The Chinese University of Hong Kong Experimental Results-Benchmark Datasets

The Chinese University of Hong Kong Future Work Kernelization? –Nonlinear extension of M4 Speed-up algorithms? –Is critical in large-scale applications Generation error bound? –SVM and MPM have both error bounds. Multi-way classification extension?

The Chinese University of Hong Kong Conclusion Propose a unified model of MPM and SVM Propose feasible solving methods based on sequential Second Order Cone Programming.