MMLD1 Support Vector Machines: Hype or Hallelujah? Kristin Bennett Math Sciences Dept Rensselaer Polytechnic Inst.

MMLD2 Outline zSupport Vector Machines for Classification yLinear Discrimination yNonlinear Discrimination zExtensions zHallelujah zHype

MMLD3 Binary Classification zExample – Medical Diagnosis Is it benign or malignant?

MMLD4 Linear Classification Model zGiven training data zLinear model - find zSuch that

MMLD5 Best Linear Separator?

MMLD6 Best Linear Separator?

MMLD7 Best Linear Separator?

MMLD8 Best Linear Separator?

MMLD9 Best Linear Separator?

MMLD10 Find Closest Points in Convex Hulls c d

MMLD11 Plane Bisect Closest Points d c

MMLD12 Find using quadratic program Many existing and new solvers.

MMLD13 Best Linear Separator: Supporting Plane Method Maximize distance Between two parallel supporting planes Distance = “Margin” =

MMLD14 Maximize margin using quadratic program

MMLD15 Dual of Closest Points Method is Support Plane Method Solution only depends on support vectors:

MMLD16 Support Vector Machines (SVM) Key Ideas: z“Maximize Margins” z“Do the Dual” z“Construct Kernels” A methodology for inference based on Vapnik’s Statistical Learning Theory.

MMLD17 Statistical Learning Theory zMisclassification error and the function complexity bound generalization error. zMaximizing margins minimizes complexity. z“Eliminates” overfitting. zSolution depends only on Support Vectors not number of attributes.

MMLD18 Margins and Complexity Skinny margin is more flexible thus more complex.

MMLD19 Margins and Complexity Fat margin is less complex.

MMLD20 Linearly Inseparable Case Convex Hulls Intersect! Same argument won’t work.

MMLD21 Reduced Convex Hulls Don’t Intersect Reduce by adding upper bound D

MMLD22 Find Closest Points Then Bisect No change except for D. D determines number of Support Vectors.

MMLD23 Linearly Inseparable Case: Supporting Plane Method Just add non-negative error vector z.

MMLD24 Dual of Closest Points Method is Support Plane Method Solution only depends on support vectors:

MMLD25 Nonlinear Classification

MMLD26 Nonlinear Classification: Map to higher dimensional space IDEA: Map each point to higher dimensional feature space and construct linear discriminant in the higher dimensional space. Dual SVM becomes:

MMLD27 Generalized Inner Product By Hilbert-Schmidt Kernels (Courant and Hilbert 1953) for certain  and K, e.g.

MMLD28 Final Classification via Kernels The Dual SVM becomes:

MMLD29

MMLD30 zSolve Dual SVM QP zRecover primal variable b zClassify new x Final SVM Algorithm Solution only depends on support vectors :

MMLD31 Support Vector Machines (SVM) zKey Formulation Ideas: y“Maximize Margins” y“Do the Dual” y“Construct Kernels” zGeneralization Error Bounds zPractical Algorithms

MMLD32 Hallelujah! zGeneralization theory and practice meet zGeneral methodology for many types of problems zSame Program + New Kernel = New method zNo problems with local minima zFew model parameters. Selects capacity. zRobust optimization methods. zSuccessful Applications BUT…

MMLD33 HYPE? zWill SVMs beat my best hand-tuned method Z for X? zDo SVM scale to massive datasets? zHow to chose C and Kernel? zWhat is the effect of attribute scaling? zHow to handle categorical variables? zHow to incorporate domain knowledge? zHow to interpret results?

MMLD34 Support Vector Machine Resources zhttp:// zhttp:// zLinks off my web page: