Proximal Plane Classification KDD 2001 San Francisco August 26-29, 2001 Glenn Fung & Olvi Mangasarian Second Annual Review June 1, 2001 Data Mining Institute University of Wisconsin - Madison

Key Contributions  Fast new support vector machine classifier  An order of magnitude faster than standard classifiers  Extremely simple to implement  4 lines of MATLAB code  NO optimization packages (LP,QP) needed

Outline of Talk  (Standard) Support vector machine (SVM) classifiers  Proximal support vector machines (PSVM) classifiers  Geometric motivation  Linear PSVM classifier  Nonlinear PSVM classifier  Full and reduced kernels  Numerical results  Correctness comparable to standard SVM  Much faster classification!  2-million points in 10-space in 21 seconds  Compared to over 10 minutes for standard SVM

Support Vector Machines Maximizing the Margin between Bounding Planes A+ A-

Proximal Vector Machines Fitting the Data using two parallel Bounding Planes A+ A-

SVM as an Unconstrained Minimization Problem At the solution of (QP) : where, Hence (QP) is equivalent to : min s. t. (QP) Changing to 2-norm and measuring margin in ( ) space:

PSVM Formulation We have from the QP SVM formulation: (QP) min s. t. This simple, but critical modification, changes the nature of the optimization problem tremendously!! Solving for in terms of and gives: min

Advantages of New Formulation  Objective function remains strongly convex  An explicit exact solution can be written in terms of the problem data  PSVM classifier is obtained by solving a single system of linear equations in the usually small dimensional input space  Exact leave-one-out-correctness can be obtained in terms of problem data

Linear PSVM We want to solve: min  Setting the gradient equal to zero, gives a nonsingular system of linear equations.  Solution of the system gives the desired PSVM classifier

Linear PSVM Solution Here,  The linear system to solve depends on: which is of the size  is usually much smaller than

Linear Proximal SVM Algorithm Classifier: Input Define Solve Calculate

Nonlinear PSVM Formulation By QP “duality”,. Maximizing the margin in the “dual space”, gives: min  Replace by a nonlinear kernel : min  Linear PSVM: (Linear separating surface: ) (QP) min s. t.

The Nonlinear Classifier  Gaussian (Radial Basis) Kernel :  Polynomial Kernel :  The nonlinear classifier:  Where K is a nonlinear kernel, e.g.:

Nonlinear PSVM Defining slightly different:  Similar to the linear case, setting the gradient equal to zero, we obtain: However, reduced kernels techniques can be used (RSVM) to reduce dimensionality.  Here, the linear system to solve is of the size

Linear Proximal SVM Algorithm Input Solve Calculate Non Define Classifier:

PSVM MATLAB Code function [w, gamma] = psvm(A,d,nu) % PSVM: linear and nonlinear classification % INPUT: A, d=diag(D), nu. OUTPUT: w, gamma % [w, gamma] = pvm(A,d,nu); [m,n]=size(A);e=ones(m,1);H=[A -e]; v=(d’*H)’ %v=H’*D*e; r=(speye(n+1)/nu+H’*H)\v % solve (I/nu+H’*H)r=v w=r(1:n);gamma=r(n+1); % getting w,gamma from r

Linear PSVM Comparisons with Other SVMs Much Faster, Comparable Correctness Data Set m x n PSVM Ten-fold test % Time (sec.) SSVM Ten-fold test % Time (sec.) SVM Ten-fold test % Time (sec.) WPBC (60 mo.) 110 x Ionosphere 351 x Cleveland Heart 297 x Pima Indians 768 x BUPA Liver 345 x Galaxy Dim 4192 x

Linear PSVM Comparisons on Larger Adult Dataset Much Faster & Comparable Correctness Dataset SizeTesting correctness % Running time Sec. (Best in Red) (Train,Test) Attributes=123 PSVMLSVMSSVMSORSMOSVM (11221,21341) (16101,16461) (22697,9865) (32562,16282)

Linear PSVM vs LSVM 2-Million Dataset Over 30 Times Faster DatasetMethodTraining Correctness % Testing Correctness % Time Sec. NDC “Easy” LSVM PSVM NDC “Hard” LSVM PSVM

Nonlinear PSVM: Spiral Dataset 94 Red Dots & 94 White Dots

Nonlinear PSVM Comparisons Data Set m x n PSVM Ten-fold test % Time (sec.) SSVM Ten-fold test % Time (sec.) LSVM Ten-fold test % Time (sec.) Ionosphere 351 x BUPA Liver 345 x Tic-Tac-Toe 958 x Mushroom * 8124 x * A rectangular kernel was used of size 8124 x 215

Conclusion  PSVM is an extremely simple procedure for generating linear and nonlinear classifiers  PSVM classifier is obtained by solving a single system of linear equations in the usually small dimensional input space for a linear classifier  Comparable test set correctness to standard SVM  Much faster than standard SVMs : typically an order of magnitude less.

Future Work  Extension of PSVM to multicategory classification  Massive data classification using an incremental PSVM  Parallel extension and implementation of PSVM