1. Classification & Regression Part II
COSC 526 Class 8
Classification & Regression Part II
Arvind Ramanathan
Computational Science & Engineering Division
Oak Ridge National Laboratory, Oak Ridge
Ph:
Start with ebola…

2. Last Class
We saw different techniques for handling large- scale classification:
Decision trees
K Nearest Neighbors (k-NN)
Modified Decision trees to handle streaming datasets
For k-NN, we discussed efficient data structures to handle large-scale datasets

3. This class… More classification… Support vector machines:
Basic outline of the algorithm
Modifications for large-scale datasets
Online approaches

4. Class Logistics

5. 10% Grade… 40% – assignments (2 instead of 3) 50% - class project
What do we do with the 5%?
Class participation
Select papers updated on the class website
Present your take on it (Critique)
Per class 2 presentations – 15 minutes each
Starting Tue (Feb 17)

6. Critique Structure 2-3 slides: what is the paper all about?
2-4 slides: key results presented
2-3 slides: key limitations/shortcomings?
2-3 slides: what could have been done better?
More discussion oriented presentation rather than an in-depth view of the paper
Peer review of critiques due every week…

7. Support Vector Machines (SVM)

8. Support Vector Machines (SVM)
Easiest explanation:
Find the “best linear separator” for a dataset
Training examples:
{(x1, y1), (x2, y2), …, (xn, yn)}
Each data point xi = (xi(1), xi(2) … xi(d))
yi = {-1, +1}
In higher dimensional datasets we want to find the “best hyperplane” α x
f(x,w) y

9. Classifier Margin…
Margin: width of the boundary that could be increased by before hitting a data point
Interested in the maximum margin:
Simplest is the maximum margin SVM classifier
Support Vectors
support vectors

10. Why are we interested in Max Margin SVM?
Intuitively this feels right:
we need the maximum margin to fit the data correctly
Robust to location of the boundary:
even if we have made a small error in the location of the boundary, not much effect on classification
Model obtained is tolerant to removal of any non- support vector data points
Validation works well!
Works very well in practice

11. Why is the maximum margin a good thing?
theoretical convenience and existence of generalization error bounds that depend on the value of margin

12. Why maximizing 𝜸 a good idea?
We all know what the dot product means
𝑨 𝒄𝒐𝒔𝜽
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,

13. Why maximizing 𝜸 a good idea?
Dot product
What is , ?
So, roughly corresponds to the margin
Bigger bigger the separation
w  x + b = 0 𝒘 + x2 x1 + x1 x2 𝒘 + x2
In this case
𝜸 𝟐 ≈ 𝟐 𝒘 𝟐 x1 𝒘
In this case 𝜸 𝟏 ≈ 𝒘 𝟐

14. + What is the margin? Distance from a point to a line A (xA(1), xA(2))
Note we assume 𝒘 𝟐 =𝟏 L
Let:
Line L: w∙x+b = w(1)x(1)+w(2)x(2)+b=0
w = (w(1), w(2))
Point A = (xA(1), xA(2))
Point M on a line = (xM(1), xM(2)) w
A (xA(1), xA(2))
w ∙ x + b = 0 +
d(A, L) H
(0,0)
d(A, L) = |AH|
= |(A-M) ∙ w|
= |(xA(1) – xM(1)) w(1) + (xA(2) – xM(2)) w(2)|
= xA(1) w(1) + xA(2) w(2) + b
= w ∙ A + b
M (x1, x2)
Remember xM(1)w(1) + xM(2)w(2) = - b
since M belongs to line L
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,

15. + + + + - + - + - + - - - - Largest Margin For ith data point:𝒘 + + + +
w  x + b = 0
For ith data point:
Want to solve: - + - + - + - - - -
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,

16. Support Vector Machine
Maximize the margin:
Good according to intuition, theory (VC dimension) & practice
𝜸 is margin … distance from the separating hyperplane + +
wx+b=0 + + + -  +  + - - - +  - - -
Maximizing the margin
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,

17. How do we derive the margin?
Separating hyperplane is defined by the support vectors
Points on +/- planes from the solution
If you knew these points, you could ignore the rest
Generally, d+1 support vectors (for d dim. data)
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,

18. Canonical Hyperplane: Problem
wx+b=0
wx+b=+1
wx+b=-1
Let
Now,
Scaling w increases margin
Solution
Work with normalized w
Also require support vectors to be in the plane defined by:
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets,

19. Canonical Hyperplane: Solution
Want to maximize margin γ
What is the relation between x1 and x2?
We also know:
wx+b=-1
wx+b=0
wx+b=+1 2 -1
Note:

20. Maximizing the Margin 2 We started with
But w can be arbitrarily large!
We normalized and...
Then:
wx+b=-1
wx+b=0
wx+b=+1 2
This is called SVM with “hard” constraints

21. Non-linearly Separable Data
If data is not separable introduce penalty:
Minimize ǁwǁ2 plus the number of training mistakes
Set C using cross validation
How to penalize mistakes?
All mistakes are not equally bad! + +
wx+b=0 + - + + - + - - + - + - - - -

22. Support Vector Machines
Introduce slack variables i
If point xi is on the wrong side of the margin then get penalty i + +
wx+b=0 + - + i + + - j - + + - -
ksi -
For each data point:
If margin  1, don’t care
If margin < 1, pay linear penalty

23. + + + + + + - - + + - - - - What is the role of slack penalty C:
C=: Only want to w, b that separate the data
C=0: Can set i to anything, then w=0 (basically ignores the data)
small C +
“good” C + + + +
big C + - - + + - - -
(0,0) -

24. Support Vector Machines
SVM in the “natural” form
SVM uses “Hinge Loss”:
Margin
Empirical loss L (how well we fit training data)
Regularization parameter
0/1 loss
penalty
Hinge loss: max{0, 1-z}

25. SVM: How to estimate w? Want to estimate and ! Use a quadratic solver:
Standard way: Use a solver!
Solver: software for finding solutions to “common” optimization problems
Use a quadratic solver:
Minimize quadratic function
Subject to linear constraints
Problem: Solvers are inefficient for big data!

26. SVM: How to estimate w? Want to estimate w, b! Alternative approach:
Want to minimize f(w,b):
Side note:
How to minimize convex functions ?
Use gradient descent: minz g(z)
Iterate: zt+1  zt –  g(zt)
g(z) z

27. SVM: How to estimate w? Want to minimize f(w,b):
Compute the gradient (j) w.r.t. w(j)
Empirical loss 𝑳( 𝒙 𝒊 𝒚 𝒊 )

28. SVM: How to estimate w? Iterate until convergence: For j = 1 … d
Gradient descent:
Problem:
Computing f(j) takes O(n) time!
n … size of the training dataset
Iterate until convergence:
For j = 1 … d
Evaluate:
Update: w(j)  w(j) - f(j)
…learning rate parameter C… regularization parameter

29. SVM: How to estimate w? Stochastic Gradient Descent
We just had:
Stochastic Gradient Descent
Instead of evaluating gradient over all examples evaluate it for each individual training example
Stochastic gradient descent:
Notice: no summation over i anymore
Iterate until convergence:
For i = 1 … n
For j = 1 … d
Compute: f(j)(xi)
Update: w(j)  w(j) -  f(j)(xi)

30. Making SVMs work with Big Data

31. Optimization problem set up
kernel function
This is good optimization problem for computers to solve called quadratic programming

32. Problems with SVMs on Big Data
SVM complexity: quadratic programming (QP)
Time: O(m3)
Space: O(m2)
m: size of training data
Two types of approaches:
Modify SVM algorithm to work with large datasets
Select representative training data to use normal SVM

33. Reducing the Training Set
How do we reduce the size of the training set so that we can reduce the time?
Combine (a large number of) ‘small’ SVMs together to obtain the final SVM
Reduced SVM: use random rectangular subset of the kernel matrix
All techniques only find an approximation to the optimal solution by an iterative approach
why not exploit this?

34. Problematic datasets: even in 1D!
How will a SVM handle this dataset?

35. The Kernel Trick: Using higher dimensions to separate data
Project data into a higher dimension
Now find the separation in this space
This is a common trick in ML for many algorithms

36. Commonly used SVM basis functions
zk = (Polynomial terms of xk of degree 1-q)
zk = (Radial basis functions of xk)
zk = (sigmoid functions of xk)
These and many other functions are valid

37. How to tackle training set sizes?
We need:
Efficient and effective method for selecting a “working” set
“shrink” the optimization problem:
Much less support vectors than training examples
many support vectors have an αi at the upper bound C
Computational improvements including caching and incremental updates of the gradient

38. How does this algorithm look like?
In each iteration of our optimization, we will split αi into two categories:
set B of free variables: updated in the current iteration
set N of fixed variables: temporarily fixed in the current iteration
Then divide and solve the optimization
While the optimality constraints are violated:
Select q variables for the working set B.
Remaining l-q variables are part of the fixed set N.
Solve the QP-sub-problem with W(α) on B
Joachims, T., Making large-scale SVM practical, Large-scale Machine Learning

39. Now, how do we select a good “working” set?
Select the set of variables such that the current iteration will make progress towards the minimum of W(α)
Use first order approximation, i.e., steepest direction d of descent which has only q non-zero elements
Convergence:
terminate only when the optimal solution is found
If not, take a step towards the optimum…

40. Other techniques to identify appropriate (and reduced) training sets
Minimum enclosing ball (MEB) for selecting “core” sets
after core sets are selected, solve the same optimization problem…
Complexity:
Time: O(m/ε2 + 1/ε4)
Space: O(1/ε8) εR R
Tsang, I.W., Kwok, J.T., and Cheung, P.-M., JMLR (6): (2005).

41. Incremental (and Decremental) SVM Learning
Solving this QP formulation, we find this:

42. Example of an SVM working with Big Datasets
Example by Leon Bottou:
Reuters RCV1 document corpus
Predict a category of a document
One vs. the rest classification
m = 781,000 training examples (documents)
23,000 test examples
d = 50,000 features
One feature per word
Remove stop-words
Remove low frequency words

43. Text categorization Questions:
(1) Is SGD successful at minimizing f(w,b)?
(2) How quickly does SGD find the min of f(w,b)?
(3) What is the error on a test set?
Training time Value of f(w,b) Test error
Standard SVM “Fast SVM”
SGD SVM
(1) SGD-SVM is successful at minimizing the value of f(w,b)
(2) SGD-SVM is super fast
(3) SGD-SVM test set error is comparable

44. Optimization “Accuracy”
SGD SVM
Conventional SVM
Optimization quality: | f(w,b) – f (wopt,bopt) |
For optimizing f(w,b) within reasonable quality SGD-SVM is super fast

45. SGD vs. Batch Conjugate Gradient
SGD on full dataset vs. Conjugate Gradient on a sample of n training examples
Theory says: Gradient descent converges in linear time 𝒌. Conjugate gradient converges in 𝒌 .
Bottom line: Doing a simple (but fast) SGD update many times is better than doing a complicated (but slow) CG update a few times
𝒌… condition number

46. Practical Considerations
Need to choose learning rate  and t0
Leon suggests:
Choose t0 so that the expected initial updates are comparable with the expected size of the weights
Choose :
Select a small subsample
Try various rates  (e.g., 10, 1, 0.1, 0.01, …)
Pick the one that most reduces the cost
Use  for next 100k iterations on the full dataset

47. Advanced Topics…

48. Sparse Linear SVMs Feature vector xi is sparse (contains many zeros)
Do not do: xi = [0,0,0,1,0,0,0,0,5,0,0,0,0,0,0,…]
But represent xi as a sparse vector xi=[(4,1), (9,5), …]
Can we do the SGD update more efficiently?
Approximated in 2 steps:
cheap: xi is sparse and so few coordinates j of w will be updated
expensive: w is not sparse, all coordinates need to be updated
The two-step update is not equivalent!
Why does it work?
Since eta is close to 0 you can do this. You get one more eta^2 term that is low order

49. Sparse Linear SVMs: Practical Considerations
Solution 1:
Represent vector w as the product of scalar s and vector v
Then the update procedure is:
(1)
(2)
Solution 2:
Perform only step (1) for each training example
Perform step (2) with lower frequency and higher 
Two step update procedure:
(1)
(2)

50. Practical Considerations
Stopping criteria:
How many iterations of SGD?
Early stopping with cross validation
Create a validation set
Monitor cost function on the validation set
Stop when loss stops decreasing
Early stopping
Extract two disjoint subsamples A and B of training data
Train on A, stop by validating on B
Number of epochs is an estimate of k
Train for k epochs on the full dataset

51. What about multiple classes?
Idea 1: One against all Learn 3 classifiers
+ vs. {o, -}
- vs. {o, +}
o vs. {+, -}
Obtain:
w+ b+, w- b-, wo bo
How to classify?
Return class c arg maxc wc x + bc

52. Learn 1 classifier: Multiclass SVM
Idea 2: Learn 3 sets of weights simoultaneously!
For each class c estimate wc, bc
Want the correct class to have highest margin:
wyi xi + byi  1 + wc xi + bc c  yi , i
(xi, yi)

53. Multiclass SVM Optimization problem:
To obtain parameters wc , bc (for each class c) we can use similar techniques as for 2 class SVM

54. SVM : what you must know? One of the most successful ML algorithms
Modifications for handling big datasets:
Reduce the training set (“core set”)
Modify SVM training algorithm
Incremental algorithm
Multi-class modifications are more complex