1 C.A.L. Bailer-Jones. Machine Learning. Support vector machines Machine learning, pattern recognition and statistical data modelling Lecture 9. Support vector machines Coryn Bailer-Jones

2 C.A.L. Bailer-Jones. Machine Learning. Support vector machines Separating hyperplanes

3 C.A.L. Bailer-Jones. Machine Learning. Support vector machines Separable problem

4 C.A.L. Bailer-Jones. Machine Learning. Support vector machines Separating hyperplanes

5 C.A.L. Bailer-Jones. Machine Learning. Support vector machines Lagrangian formulation

6 C.A.L. Bailer-Jones. Machine Learning. Support vector machines Lagrangian dual

7 C.A.L. Bailer-Jones. Machine Learning. Support vector machines Applying the classifier

8 C.A.L. Bailer-Jones. Machine Learning. Support vector machines Non-separable case Lagrangian for separable case now has no solution L D grows arbitrarily large

9 C.A.L. Bailer-Jones. Machine Learning. Support vector machines Non-separable case

10 C.A.L. Bailer-Jones. Machine Learning. Support vector machines Non-separable case

11 C.A.L. Bailer-Jones. Machine Learning. Support vector machines Non-separable case

12 C.A.L. Bailer-Jones. Machine Learning. Support vector machines Nonlinear functions via kernels

13 C.A.L. Bailer-Jones. Machine Learning. Support vector machines Nonlinear functions via kernels

14 C.A.L. Bailer-Jones. Machine Learning. Support vector machines Support Vector Machines ● for nonlinear, multidimensional two class problems ● implicit projection into a higher (possibly infinite) dim. space – where we hope to get closer to linear separation ● free parameters – problem is a convex linear programming problem – so the solution is unique (still needs numerical methods) ● solution is a combination of the support vectors (only) – solution is sparse in these, so suitable for high-dim. problems with limited training data (does not have scaling problem like an MLP) ● two control parameters – C controls regularization – Generally: larger C => error term dominates => less regularization – 1/ is a correlation/relevance length scale in the data space

15 C.A.L. Bailer-Jones. Machine Learning. Support vector machines The margin, cost and no. of SVs in the non-separable case We are still trying to maximize the margin width, even though the margin is no longer defined as the region void of data points (as in the separable case). Rather, the margin is defined as the region which contains all points relevant to (i.e. included in) the solution, that is, the support vectors, even if they are on the correct side of the separating hyperplane. However, support vectors only contribute to the errors (i.e. have >1) if they are on the wrong side of the separating hyperplane. We are maximizing the margin width with a cost penalty. If we increase the cost, C, then errors incur a higher penalty and so tend to make the margin smaller. If the separating hyperplane were fixed this would generally reduce the number of support vectors. But of course it is not fixed (we change C in order to find a different solution!), so the support vectors, and their number, will change, and the new margin is not necessarily narrower. Thus the number of support vectors and size of the margin in the solution do not have a monotonic dependence on C (although they might for some problems). Generally, a larger cost produces a more flexible solution (as we are encouraging fewer error) and thus a smaller error on the training data, and we would normally associate a more complex solution with more support vectors. But this is depends very much on the specific problem, i.e. how well the data can be separated. Moreover, there is no guarantee that a larger cost produces a better solution.

16 C.A.L. Bailer-Jones. Machine Learning. Support vector machines SVM application: star-galaxy separation with PS1 blue: stellar library red: galaxy library ● 4 noise-free simulated colours ● package libsvm – C++, Java – R import in svm{e1071} ● see R scripts for example and syntax

17 C.A.L. Bailer-Jones. Machine Learning. Support vector machines SVM application For i-z = 0.2 z-y = 0.2

18 C.A.L. Bailer-Jones. Machine Learning. Support vector machines Multiclass problems ● with K classes, either solve – K models, each of type 1 class vs. combined K-1 classes, or – K(K-1)/2 models, each with 1 class vs. 1 class (“pairwise coupling”)

19 C.A.L. Bailer-Jones. Machine Learning. Support vector machines Summary ● SVMs operate on principle of separating hyperplanes – maximize margin – only data points near margin are relevant (the support vectors) ● Nonlinearity via kernels – possible because training data appear only as dot products – project data into a higher dimensional space where more separation is possible – projection is only implicit (big computational saving) ● Unique solution (convex objective function) – optimization via the Lagrangain dual ● Various extensions – multiclass case. No natural probabilistic interpretation, but we can remap outputs – regression