1. Lecture 07: Hard-margin SVM
CS480/680: Intro to ML
Lecture 07: Hard-margin SVM
11/01/2019
Yao-Liang Yu

2. Outline Maximum Margin Lagrangian Dual Alternative View 11/01/2019
Yao-Liang Yu

3. Perceptron revisited Two classes: y = 1 or y = -1
Assuming linear separable
exist w and b such that for all i,
yi(wTxi + b) > 0
Find any such w and b
feasibility problem
11/01/2019
Yao-Liang Yu

4. Margin Take any linear separating hyperplane H
for all i, yi(wTxi + b) > 0
Move H until it touches some positive point, H1
increase b say
Move H until it touches some negative point, H-1
decrease b say
margin = dist(H1,H) ∧ dist(H-1,H)
11/01/2019
Yao-Liang Yu

5. Put into formula H : wTx + b = 0 H1 : wTx + b = t H-1 : wTx + b = -s
What is the distance between
H1 and H ?
equality is attained
11/01/2019
Yao-Liang Yu

6. Put into formula H : wTx + b = 0 H1 : wTx + b = t H-1 : wTx + b = -t
What is the distance between
H1 and H ?
11/01/2019
Yao-Liang Yu

7. Put into formula H : wTx + b = 0 H1 : wTx + b = 1 H-1 : wTx + b = -1
What is the distance between
H1 and H ?
11/01/2019
Yao-Liang Yu

8. Maximum Margin Important facts. For any f, For positive f,
For s. monotone g,
11/01/2019
Yao-Liang Yu

9. Hard-margin Support Vector Machines
Linear programming
Quadratic programming
margin
11/01/2019
Yao-Liang Yu

10. Support Vectors Those touch the parallel hyperplanes H1 and H-1
Usually only a handful
Entirely determine the hyperplanes!
11/01/2019
Yao-Liang Yu

11. Existence and uniqueness
Always exists a minimizer w and b (if linearly separable)
The minimizer w is unique (strong convexity of )
The minimizer b is also unique (why?)
11/01/2019
Yao-Liang Yu

12. Outline Maximum Margin Lagrangian Dual Alternative View 11/01/2019
Yao-Liang Yu

13. Lagrangian Primal Lagrangian Lagrangian multiplier [dual variable]
[primal variable]
11/01/2019
Yao-Liang Yu

14. Joseph-Louis Lagrange (1736-1813)
11/01/2019
Yao-Liang Yu

15. Optimization detour Fermat’s Theorem. Necessarily
(Fréchet) Derivative at x.
Example. f(x) = xTAx + xTb + c
Df(x) = (A+AT)x + b
11/01/2019
Yao-Liang Yu

16. Just in case
11/01/2019
Yao-Liang Yu

17. Deriving the dual
11/01/2019
Yao-Liang Yu

18. The dual problem Dual Rn Only need dot product in the dual !
11/01/2019
Yao-Liang Yu

19. Support Vectors
11/01/2019
Yao-Liang Yu

20. Outline
Maximum Margin
Dual
Alternative View
11/01/2019
Yao-Liang Yu

21. An dual view
11/01/2019
Yao-Liang Yu

22. Convex sets and Convex hull
Convex set. A point set is convex if the line segment [x,y] connecting any two points x and y in C lies entirely in C. Convex hull. Smallest convex set containing C.
11/01/2019
Yao-Liang Yu

23. Regularization vs. Constraint
Computationally
appealing
Always true
Mild conditions
Theoretically
appealing
can use equality if
from top to bottom
11/01/2019
Yao-Liang Yu

24. From regularization to constraint
11/01/2019
Yao-Liang Yu

25. Homogeneity
11/01/2019
Yao-Liang Yu

26. Split
11/01/2019
Yao-Liang Yu

27. NOW this
11/01/2019
Yao-Liang Yu

28. Questions?
11/01/2019
Yao-Liang Yu