Support Vector Machines Piyush Kumar

Perceptrons revisited Class 1 : (+1) Class 2 : (-1) Is this unique?

Which one is the best? Perceptron outputs :

Perceptrons: What went wrong? Slow convergence Can overfit Cant do complicated functions easily Theoretical guarantees are not as strong

Perceptron: The first NN  Proposed by Frank Rosenblatt in 1956  Neural net researchers accuse Rosenblatt of promising ‘too much’  Numerous variants  Also helps to study LP  One of the simplest Neural Network.

The new concept Car From Perceptrons to SVMs Margins Linearization Kernels

Support Vector Machines Margin

Classification Margin Distance from example to the separator is Examples closest to the hyperplane are support vectors. Margin ρ of the separator is the width of separation between classes. r ρ

Support Vector Machines Maximizing the margin is good according to intuition and PAC theory. Implies that only support vectors are important; other training examples are ignorable. Leads to Simple classifiers and hence better? (Simple = large margin)

Let’s start some math… N samples : Can we find a hyperplane that separates the two classes? (labeled by y) i.e. : For all j such that y = +1 : For all j such that y = -1 Where y = +/- 1 are labels for the data.

Further assumption 1 Which we will relax later! Lets assume that the hyperplane that we are looking for passes thru the origin

Further assumption 2 Lets assume that we are looking for a halfspace that contains a set of points Relax now!!

Lets Relax FA 1 now “Homogenize” the coordinates by adding a new coordinate to the input. Think of it as moving the whole red and blue points in one higher dimension From 2D to 3D it is just the x-y plane shifted to z = 1. This takes care of the “bias” or our assumption that the halfspace can pass thru the origin.

Further Assumption 3 Assume all points on a unit sphere! If they are not after applying transformations for FA 1 and FA 2, make them so. Relax now!

What did we want? Maximize the margin. What does it mean in the new space?

What’s the new optimization problem? Max |ρ| subject to –x i.w >= ρ (Note that we have gotten rid of the y’s by mirroring around the origin). Here w is a unit vector. ||w|| = 1.

Same Problem Min 1/ρ subject to x i.((1/ρ)w) >= 1 Let v = (1/ρ) w Then the constraint becomes x i.v >= 1. Objective = Min 1/ρ = Min || (1/ρ) w || = Min ||v|| is the same as Min ||v|| 2

New formulation Min ||v|| 2 Subject to : v.x i >= 1 Using matlab, this is a piece of cake to solve. Decision boundary sign(w.x i ) Only for support vectors v.x i = 1.

Support Vector Machines Linear Learning Machines like perceptrons. Map non-linearly to higher dimension to overcome the linearity constraint. Select between hyperplanes, Use margin as a test (This is what perceptrons don’t do) From learning theory, maximum margin is good

Another Reformulation Unlike Perceptrons SVMs have a unique solution but are harder to solve.

Support Vector Machines There are very simple algorithms to solve SVMs ( as simple as perceptrons ) If you are interested in learning those, come and talk to me. (Out of reach for this course)

Another twist : Linearization If the data is separable with say a sphere, how would you use a svm to separate it? (Ellipsoids?)

Linearization a.k.a Feature Expansion Delaunay!?? Lift the points to a paraboloid in one higher dimension, For instance if the data is in 2D, (x,y) -> (x,y,x 2 +y 2 )

Linearization Note that replacing x by  (x) the decision boundary changes from w.x = 0 to w.  (x) = 0 This helps us get non-linear separators compared to linear separators when  is non- linear (as in the last example). Another feature expansion example: –(x,y) -> (x^2, xy, y^2, x, y) –What kind of separators are there?

Linearization The more features, the more power. There is a danger of overfitting. When there are lot of features (sometimes even infinite), we can use the “kernel trick” to solve the optimization problem faster. Lets look back at optimization for a moment again…

Lagrange Multipliers

Lagrangian function

At optimum

More precisely

The optimization Problem Revisited

Removing v

Support Vectors v is a linear combination of ‘some examples’ or support vectors. More than likely if we see too many support vectors, we are overfitting. Simple and Short classifiers are preferable.

Substitution

Gram Matrix

The decision surface Recovered

What is Gram Matrix reduction good for? The Kernel Trick Even if the number of features is infinite, G might still be small and hence the optimization problem solvable. We could compute G without computing X, at least sometimes (by redefining the dot product in the feature space).

Recall

The kernel Matrix The trick that ML community uses for Linearization is to use a function that redefines distances between points. Example : The optimization problem no longer needs  to be explicitly evaluated. As long as we can figure out the distance between two mapped points, its enough.

Example Kernels

The decision Surface?

A demo using libsvm Some implementations of SVM –libsvm –svmlight –svmtorch

Checkerboard Dataset

k-Nearest Neighbor Algorithm

LSVM on Checkerboard

Conclusions SVM is an step towards improving perceptrons They use large margin for good genralization In order to make large feature expansions, we can use the gram matrix formulation of the optimization problem (or use kernels). SVMs are popular classifiers because they achieve good accuracy on real world data.