MMLD1 Support Vector Machines: Hype or Hallelujah? Kristin Bennett Math Sciences Dept Rensselaer Polytechnic Inst.

MMLD2 Outline zSupport Vector Machines for Classification yLinear Discrimination via geometry yNonlinear Discrimination zNitty Gritty Details zResults from Cortes and Vapnik zHallelujah zHype

MMLD3 Binary Classification zExample – Medical Diagnosis Is it benign or malignant?

MMLD4 Linear Classification Model zGiven training data zLinear model - find zSuch that

MMLD5 Best Linear Separator?

MMLD6 Best Linear Separator?

MMLD7 Best Linear Separator?

MMLD8 Best Linear Separator?

MMLD9 Best Linear Separator?

MMLD10 Find Closest Points in Convex Hulls c d

MMLD11 Plane Bisect Closest Points d c

MMLD12 Find using quadratic program Many existing and new solvers.

MMLD13 Best Linear Separator: Supporting Plane Method Maximize distance Between two parallel supporting planes Distance = “Margin” =

MMLD14 Maximize margin using quadratic program

MMLD15 Dual of Closest Points Method is Support Plane Method Solution only depends on support vectors:

MMLD16 Support Vector Machines (SVM) Key Ideas: z“Maximize Margins” z“Do the Dual” z“Construct Kernels” A methodology for inference based on Vapnik’s Statistical Learning Theory.

MMLD17 Statistical Learning Theory zMisclassification error and the function complexity bound generalization error. zMaximizing margins minimizes complexity. z“Eliminates” overfitting. zSolution depends only on Support Vectors not number of attributes.

MMLD18 Margins and Complexity Skinny margin is more flexible thus more complex.

MMLD19 Margins and Complexity Fat margin is less complex.

MMLD20 Linearly Inseparable Case Convex Hulls Intersect! Same argument won’t work.

MMLD21 Reduced Convex Hulls Don’t Intersect Reduce by adding upper bound D

MMLD22 Find Closest Points Then Bisect No change except for D. D determines number of Support Vectors.

MMLD23 Linearly Inseparable Case: Supporting Plane Method Just add non-negative error vector z.

MMLD24 Closest Points equivalent to Support Plane Method Solution only depends on support vectors:

MMLD25 Nonlinear Classification

MMLD26 Nonlinear Classification: Map to higher dimensional space IDEA: Map each point to higher dimensional feature space and construct linear discriminant in the higher dimensional space. Dual SVM becomes:

MMLD27 Generalized Inner Product By Hilbert-Schmidt Kernels (Courant and Hilbert 1953) for certain  and K, e.g.

MMLD28 Final Classification via Kernels The Dual SVM becomes:

MMLD29

MMLD30 zSolve Dual SVM QP zRecover primal variable b zClassify new x Final SVM Algorithm Solution only depends on support vectors :

MMLD31 Support Vector Machines (SVM) zKey Formulation Ideas: y“Maximize Margins” y“Do the Dual” y“Construct Kernels” zGeneralization Error Bounds zPractical Algorithms

MMLD32 Nitty Gritty zNeed Dual of

MMLD33 Wolfe Dual Problem with Inequalities zPrimal zDual

MMLD34 Lagrangian Function zPrimal zLagrangian

MMLD35 Wolfe Dual Eliminate

MMLD36 Wolfe Dual Use grad b to simplify objective

MMLD37 Wolfe Dual Eliminate w

MMLD38 Wolfe Dual Simplify inner products

MMLD39 Final Wolfe Dual Usually convert to minimization at this point

MMLD40 Cortes and Vapnik: Figure 1: degree 2 polynomials SV =circles errors =crosses

MMLD41 Fig 6: US postal data 7.3K train 2K test (16 by 16 resolution)

MMLD42 Results on US postal service: Gaussian Kernel

MMLD43 Errors on US postal data

MMLD44 NIST data 60K train 10K test 28X28 resolution 4 degree polynomial misclassified examples below 1 false negatives, others false positives

MMLD45 NIST results

MMLD46 Hallelujah! zGeneralization theory and practice meet zGeneral methodology for many types of problems zSame Program + New Kernel = New method zNo problems with local minima zFew model parameters. Selects capacity. zRobust optimization methods. zSuccessful Applications BUT…

MMLD47 HYPE? zWill SVMs beat my best hand-tuned method Z for X? zDo SVM scale to massive datasets? zHow to chose C and Kernel? zWhat is the effect of attribute scaling? zHow to handle categorical variables? zHow to incorporate domain knowledge? zHow to interpret results?

MMLD48 Support Vector Machine Resources zhttp:// zhttp:// zLinks off my web page: