1. Support Vector Machines
Piyush Kumar

2. Perceptrons revisited
Class 2 : (-1)
Class 1 : (+1)
Is this unique?

3. Which one is the best?
Perceptron outputs :

4. Perceptrons: What went wrong?
Slow convergence
Can overfit
Cant do complicated functions easily
Theoretical guarantees are not as strong

5. Perceptron: The first NN
Proposed by Frank Rosenblatt in 1956
Neural net researchers accuse Rosenblatt of promising ‘too much’ 
Numerous variants
Also helps to study LP 
One of the simplest Neural Network.

6. From Perceptrons to SVMs
Margins
Linearization
Kernels
Core-Sets and support vectors
Solvers: As simple as perceptrons!

7. Support Vector Machines
Margin

8. Classification Margin
Distance from example to the separator is
Examples closest to the hyperplane are support vectors.
Margin ρ of the separator is the width of separation between classes. ρ r

9. Support Vector Machines
Maximizing the margin is good according to intuition and PAC theory.

10. Support Vector Machines
Implies that only support vectors are important; other training examples are ignorable.
Leads to Simple classifiers and hence better? (Simple = large margin)

11. Let’s start some math… N samples :
Where y = +/- 1 are labels for the data.
Can we find a hyperplane that separates the two classes?
(labeled by y) i.e.
: For all j such that y = +1
: For all j such that y = -1

12. Which we will relax later!
Further assumption 1
Lets assume that the hyperplane that we are looking for
passes thru the origin

13. Relax now!! 
Further assumption 2
Lets assume that we are looking for a halfspace that contains a set of points

14. Lets Relax FA 1 now
“Homogenize” the coordinates by adding a new coordinate to the input.
Think of it as moving the whole red and blue points in one higher dimension
From 2D to 3D it is just the x-y plane shifted to z = 1. This takes care of the “bias” or our assumption that the halfspace can pass thru the origin.

15. Further Assumption 3 Assume all points on a unit sphere!
Relax now! 
Further Assumption 3
Assume all points on a unit sphere!
If they are not after applying transformations for FA 1 and FA 2 , make them so.

16. What did we want? Maximize the margin.
What does it mean in the new space?

17. What’s the new optimization problem?
Max |ρ|
subject to
xi.w >= ρ
(Note that we have gotten rid of the y’s by mirroring around the origin).
Here w is a unit vector. ||w|| = 1.

18. Same Problem Min 1/ρ subject to xi.((1/ρ)w) >= 1 Let v = (1/ρ) w
Then the constraint becomes xi.v >= 1.
Objective = Min 1/ρ = Min || (1/ρ) w || = Min ||v|| is the same as Min ||v||2

19. New formulation Min ||v||2 Subject to : v.xi >= 1
Using matlab, this is a piece of cake to solve.
Decision boundary sign(w.xi)
Only for support vectors v.xi = 1.

20. Support Vector Machines
Linear Learning Machines like
perceptrons.
Map non-linearly to higher dimension to
overcome the linearity constraint.
Select between hyperplanes, Use margin
as a test
(This is what perceptrons don’t do)
From learning theory, maximum margin is good

21. Another Reformulation
We will revisit this soon…
Another Reformulation
Unlike Perceptrons SVMs
have a unique solution
but are harder to solve.
<QP>

22. Support Vector Machines
There are very simple algorithms to solve SVMs ( as simple as perceptrons )
If you are interested in learning those, come and talk to me.

23. Another twist : Linearization
If the data is separable with say a sphere, how would you use a svm to separate it? (Ellipsoids?)

24. Linearization a.k.a Feature Expansion
Delaunay!??
Linearization a.k.a Feature Expansion
Lift the points to a paraboloid in one higher dimension,
For instance if the data is in 2D,
(x,y) -> (x,y,x2+y2)

25. Linearization
Note that replacing x by (x) the decision boundary changes from w.x = 0 to w.(x) = 0
This helps us get non-linear separators compared to linear separators when  is non-linear (as in the last example).
Another feature expansion example:
(x,y) -> (x^2, xy, y^2, x, y)
What kind of separators are there?

26. Linearization The more features, the more power.
There is a danger of overfitting.
When there are lot of features (sometimes even infinite), we can use the “kernel trick” to solve the optimization problem faster.
Lets look back at optimization for a moment again…

27. Lagrange Multipliers

28. Lagrangian function

30. At optimum

31. More precisely

32. The optimization Problem Revisited

33. Removing v

34. Support Vectors
v is a linear combination of ‘some examples’ or support vectors.
More than likely if we see too many support vectors, we are
overfitting. Simple and Short classifiers are preferable.

35. Substitution

36. Gram Matrix

37. The decision surface Recovered

38. What is Gram Matrix reduction good for?
The Kernel Trick
Even if the number of features is infinite, G might still be small and hence the optimization problem solvable.
We could compute G without computing X, at least sometimes (by redefining the dot product in the feature space).

39. Recall

40. The kernel Matrix
The trick that ML community uses for Linearization is to use a function that redefines distances between points.
Example :
The optimization problem no longer needs  to be explicitly evaluated. As long as we can figure out the distance between two mapped points, its enough.

41. Example Kernels

42. The decision Surface?

44. A demo using libsvm Some implementations of SVM libsvm svmlight
svmtorch

45. Checkerboard Dataset

46. k-Nearest Neighbor Algorithm

47. LSVM on Checkerboard

48. Conclusions SVM is an step towards improving perceptrons
They use large margin for good genralization
In order to make large feature expansions, we can use the gram matrix formulation of the optimization problem (or use kernels).
SVMs are popular classifiers because they achieve good accuracy on real world data.

49. Geometric Solvers for SVM
Frank Wolfe Algorithm
A.k.a. (Gilbert’s algorithm) d

50. Recall Minkowski sum Defined as:
The sweeping of one convex object with another
Defined as:

51. Minkowski difference Minkowski difference, defined as:
Minkowski Difference (A,B) = { a - b | a  A, b  B }
= Minkowski Sum (A, -B)
Can write distance between two objects as:
Distance (A,B) = min{ || a – b || : a  A, b  B }
= min { || c || : c  A - B }
A and B intersecting iff A–B contains the origin!
Distance between A and B given by point of minimum norm in A–B!

52. Again! A and B intersecting iff A–B contains the origin! Distance:
Distance between A and B given by point of minimum norm in A–B!
Distance:
distance(A,B) = min a  A, b B || a - b ||2
distance(A,B) = min c  Minkowski-Difference(A,B) || c ||2
if A and B disjoint, c is point on boundary of Minkowski difference

53. Minkowski Difference ExamplesB B A
Picture by Eric Larsen

54. Frank Wolfe Algorithm a.k.a Gilbert’s Algorithm
1. Start with p in P=A-B. 2. Find the closest point q in P, in the direction op. 3. Let r on pq be the closest point to the origin. 4. If r is a good approximation stop. Else, p = r. Goto 1.

55. Frank Wolfe
At each step the objective function is linearized and then a step is taken in a direction that reduces the objective while maintaining feasibility. The algorithm can be seen as a generalization of the primal simplex algorithm for LP.

56. Frank Wolfe

57. Gilbert’s Algorithm On the chalk board. Gives:
Linear running time in size, 1/ 
Optimal Core-Sets for SVM.
With away steps  Linear Convergence.