1. Support Vector Machines
More information can be found in

2. Linear Classifiers f a yest x f(x,w,b) = sign(w. x - b) denotes +1
How would you classify this data?

3. Linear Classifiers f a yest x f(x,w,b) = sign(w. x - b) denotes +1
How would you classify this data?

4. Linear Classifiers f a yest x f(x,w,b) = sign(w. x - b) denotes +1
How would you classify this data?

5. Linear Classifiers f a yest x f(x,w,b) = sign(w. x - b) denotes +1
How would you classify this data?

6. Linear Classifiers f a yest x f(x,w,b) = sign(w. x - b) denotes +1
Any of these would be fine..
..but which is best?

7. Classifier Margin f a yest x
f(x,w,b) = sign(w. x - b)
denotes +1
denotes -1
Define the margin of a linear classifier as the width that the boundary could be increased by before hitting a datapoint.

8. Maximum Margin f a yest x
f(x,w,b) = sign(w. x - b)
denotes +1
denotes -1
The maximum margin linear classifier is the linear classifier with the, um, maximum margin.
This is the simplest kind of SVM (Called an LSVM)
Linear SVM

9. Maximum Margin f a yest x
f(x,w,b) = sign(w. x - b)
denotes +1
denotes -1
The maximum margin linear classifier is the linear classifier with the, um, maximum margin.
This is the simplest kind of SVM (Called an LSVM)
Support Vectors are those datapoints that the margin pushes up against
Linear SVM

10. Why Maximum Margin?
Intuitively this feels safest.
If we’ve made a small error in the location of the boundary (it’s been jolted in its perpendicular direction) this gives us least chance of causing a misclassification.
LOOCV is easy since the model is immune to removal of any non-support-vector datapoints.
There’s some theory (using VC dimension) that is related to (but not the same as) the proposition that this is a good thing.
Empirically it works very very well.
f(x,w,b) = sign(w. x - b)
denotes +1
denotes -1
The maximum margin linear classifier is the linear classifier with the, um, maximum margin.
This is the simplest kind of SVM (Called an LSVM)
Support Vectors are those datapoints that the margin pushes up against

11. Estimate the Margin wx +b = 0
denotes +1
denotes -1
wx +b = 0 x
What is the distance expression for a point x to a line wx+b= 0?

12. Estimate the Margin wx +b = 0 What is the expression for margin?
denotes +1
denotes -1
wx +b = 0
Margin
What is the expression for margin?

13. Maximize Margin
denotes +1
denotes -1
wx +b = 0
Margin

14. Maximize Margin wx +b = 0 Min-max problem  game problem Margin
denotes +1
denotes -1
wx +b = 0
Margin
Min-max problem  game problem

15. Maximize Margin
denotes +1
denotes -1
wx +b = 0
Margin
Strategy:

16. Maximum Margin Linear Classifier
How to solve it?

17. Learning via Quadratic Programming
QP is a well-studied class of optimization algorithms to maximize a quadratic function of some real-valued variables subject to linear constraints.

18. Quadratic Programming
Quadratic criterion
Find
Subject to
n additional linear inequality constraints
And subject to
e additional linear equality constraints

19. Quadratic Programming
Quadratic criterion
Find
There exist algorithms for finding such constrained quadratic optima much more efficiently and reliably than gradient ascent.
(But they are very fiddly…you probably don’t want to write one yourself)
Subject to
n additional linear inequality constraints
And subject to
e additional linear equality constraints

20. Uh-oh! This is going to be a problem! What should we do? denotes +1

21. Uh-oh! This is going to be a problem! What should we do? Idea 1:
Find minimum w.w, while minimizing number of training set errors.
Problemette: Two things to minimize makes for an ill-defined optimization
Uh-oh!
denotes +1
denotes -1

22. Uh-oh! This is going to be a problem! What should we do? Idea 1.1:
Minimize
w.w + C (#train errors)
There’s a serious practical problem that’s about to make us reject this approach. Can you guess what it is?
Uh-oh!
denotes +1
denotes -1
Tradeoff parameter

23. Uh-oh! This is going to be a problem! What should we do? Idea 1.1:
Minimize
w.w + C (#train errors)
There’s a serious practical problem that’s about to make us reject this approach. Can you guess what it is?
Uh-oh!
denotes +1
denotes -1
Tradeoff parameter
Can’t be expressed as a Quadratic Programming problem.
Solving it may be too slow.
(Also, doesn’t distinguish between disastrous errors and near misses)
So… any other ideas?

24. Uh-oh! This is going to be a problem! What should we do? Idea 2.0:
Minimize
w.w + C (distance of error
points to their
correct place)
Uh-oh!
denotes +1
denotes -1

25. Support Vector Machine (SVM) for Noisy Data
denotes +1
denotes -1
Any problem with the above formulism?

26. Support Vector Machine (SVM) for Noisy Data
denotes +1
denotes -1
Balance the trade off between margin and classification errors

27. Support Vector Machine for Noisy Data
How do we determine the appropriate value for c ?

28. An Equivalent QP: Determine b
Fix w
A linear programming problem !

29. Suppose we’re in 1-dimension
What would SVMs do with this data?
x=0

30. Suppose we’re in 1-dimension
Not a big surprise
x=0
Positive “plane”
Negative “plane”

31. Harder 1-dimensional dataset
That’s wiped the smirk off SVM’s face.
What can be done about this?
x=0

32. Harder 1-dimensional dataset
Remember how permitting non-linear basis functions made linear regression so much nicer?
Let’s permit them here too
x=0

33. Harder 1-dimensional dataset
Remember how permitting non-linear basis functions made linear regression so much nicer?
Let’s permit them here too
x=0