October 2-4, 2000M20001 Support Vector Machines: Hype or Hallelujah? Kristin Bennett Math Sciences Dept Rensselaer Polytechnic Inst.

October 2-4, 2000M20002 Outline zSupport Vector Machines for Classification yLinear Discrimination yNonlinear Discrimination zExtensions zApplication in Drug Design zHallelujah zHype

October 2-4, 2000M20003 Support Vector Machines (SVM) Key Ideas: z“Maximize Margins” z“Do the Dual” z“Construct Kernels” A methodology for inference based on Vapnik’s Statistical Learning Theory.

October 2-4, 2000M20004 Best Linear Separator?

October 2-4, 2000M20005 Best Linear Separator?

October 2-4, 2000M20006 Best Linear Separator?

October 2-4, 2000M20007 Best Linear Separator?

October 2-4, 2000M20008 Best Linear Separator?

October 2-4, 2000M20009 Find Closest Points in Convex Hulls c d

October 2-4, 2000M Plane Bisect Closest Points d c

October 2-4, 2000M Find using quadratic program Many existing and new solvers.

October 2-4, 2000M Best Linear Separator: Supporting Plane Method Maximize distance Between two parallel supporting planes Distance = “Margin” =

October 2-4, 2000M Maximize margin using quadratic program

October 2-4, 2000M Dual of Closest Points Method is Support Plane Method Solution only depends on support vectors:

October 2-4, 2000M Statistical Learning Theory zMisclassification error and the function complexity bound generalization error. zMaximizing margins minimizes complexity. z“Eliminates” overfitting. zSolution depends only on Support Vectors not number of attributes.

October 2-4, 2000M Margins and Complexity Skinny margin is more flexible thus more complex.

October 2-4, 2000M Margins and Complexity Fat margin is less complex.

October 2-4, 2000M Linearly Inseparable Case Convex Hulls Intersect! Same argument won’t work.

October 2-4, 2000M Reduced Convex Hulls Don’t Intersect Reduce by adding upper bound D

October 2-4, 2000M Find Closest Points Then Bisect No change except for D. D determines number of Support Vectors.

October 2-4, 2000M Linearly Inseparable Case: Supporting Plane Method Just add non-negative error vector z.

October 2-4, 2000M Dual of Closest Points Method is Support Plane Method Solution only depends on support vectors:

October 2-4, 2000M Nonlinear Classification

October 2-4, 2000M Nonlinear Classification: Map to higher dimensional space IDEA: Map each point to higher dimensional feature space and construct linear discriminant in the higher dimensional space. Dual SVM becomes:

October 2-4, 2000M Generalized Inner Product By Hilbert-Schmidt Kernels (Courant and Hilbert 1953) for certain  and K, e.g.

October 2-4, 2000M Final Classification via Kernels The Dual SVM becomes:

October 2-4, 2000M200027

October 2-4, 2000M zSolve Dual SVM QP zRecover primal variable b zClassify new x Final SVM Algorithm Solution only depends on support vectors :

October 2-4, 2000M Support Vector Machines (SVM) zKey Formulation Ideas: y“Maximize Margins” y“Do the Dual” y“Construct Kernels” zGeneralization Error Bounds zPractical Algorithms

October 2-4, 2000M SVM Extensions zRegression zVariable Selection zBoosting zDensity Estimation zUnsupervised Learning yNovelty/Outlier Detection yFeature Detection yClustering

October 2-4, 2000M Example in Drug Design zGoal to predict bio-reactivity of molecules to decrease drug development time. zTarget is to predict the logarithm of inhibition concentration for site "A" on the Cholecystokinin (CCK) molecule. zConstructs quantitative structure activity relationship (QSAR) model.

October 2-4, 2000M SVM Regression:  -insensitive loss function ++ --

October 2-4, 2000M SVM Minimizes Underestimate+Overestimate

October 2-4, 2000M LCCKA Problem zTraining data – 66 molecules z323 original attributes are wavelet coefficients of TAE Descriptors. z39 subset of attributes selected by linear 1-norm SVM (with no kernels). zFor details see DDASSL project link off of zTesting set results reported.

October 2-4, 2000M LCCK Prediction Q2=.25

October 2-4, 2000M Many Other Applications zSpeech Recognition zData Base Marketing zQuark Flavors in High Energy Physics zDynamic Object Recognition zKnock Detection in Engines zProtein Sequence Problem zText Categorization zBreast Cancer Diagnosis zSee: SVM/applist.html

October 2-4, 2000M Hallelujah! zGeneralization theory and practice meet zGeneral methodology for many types of problems zSame Program + New Kernel = New method zNo problems with local minima zFew model parameters. Selects capacity. zRobust optimization methods. zSuccessful Applications BUT…

October 2-4, 2000M HYPE? zWill SVMs beat my best hand-tuned method Z for X? zDo SVM scale to massive datasets? zHow to chose C and Kernel? zWhat is the effect of attribute scaling? zHow to handle categorical variables? zHow to incorporate domain knowledge? zHow to interpret results?

October 2-4, 2000M Support Vector Machine Resources zhttp:// zhttp:// zLinks off my web page: