1. Support Vector Machines
CSC 576: Data Mining

2. Today…
Support Vector Machines
Multiclass Problems
Feature Selection

3. Support Vector Machines (SVM)
What are they?
Developed in the 1990s
Computer Science community
Very popular
Performance:
Often considered one of the best “out of the box” classifiers
Applications: handwritten digit recognition, text categorization

4. Support Vector Machines (SVM)
Comparing to other statistical learning methods:
SVMs work well with high-dimensional data
Unique:
Represents “decision boundary” using a subset of training examples

5. Terminology Maximal Margin Classifier Support Vector Classifier
Support Vector Machine
Often all three are referred to as “Support Vector Machine”

6. The Path Ahead Maximal Margin Classifier Support Vector Classifier
Generalization of Maximal Margin Classifier
Support Vector Machine
Generalization of Support Vector Classifier

7. Maximal Margin Classifier
First, need to define a hyperplane
What is a hyperplane?
Hyperplane has p-1 dimensions in a p dimensional space
Example: in 2 dimension space, a hyperplane has 1 dimension (and thus, is a line)

8. Hyperplane Mathematical Definition
For two dimensions, hyperplane defined as:
B0, B1, B2 are parameters.
X1, X2 are variables.
Note that this equation is a line:
Hyperplane is in one-dimension

9. Hyperplane Mathematical Definition
We’re going to “find” values for B0, B1, B2.
Then, for any values X1 and X2:
if B0 + B1X1 + B2X2 = 0
Point is on the line.

10. Hyperplane Mathematical Definition
We’re going to “find” values for B0, B1, B2.
Then, for any values X1 and X2:
if B0 + B1X1 + B2X2 > 0
Point is not on the line. On one side of the line.
if B0 + B1X1 + B2X2 < 0
Point is on the other side of the line.

11. Hyperplane
… is dividing 2-dimesional space into two halves by a line.

12. Separating Hyperplane
Note: a separating hyperplane means zero training errors.
Dataset with two classes:
Squares
circles
Can find a separating hyperplane with all squares on one side and all circles on the other.
Infinitely many such hyperplanes possible.

13. Classification Using a Separating Hyperplane
For a new test instance, which side of the line is it on?
B0 + B1X1 + B2X2 > 0
B0 + B1X1 + B2X2 < 0

14. Classification Using a Separating Hyperplane
Standard SVM approach:
Label class data as either +1 or -1, depending on which class an instance belongs to.
Prediction:

15. Classification Using a Separating Hyperplane
For a new test instance, which side of the line is it on?
B0 + B1X1 + B2X2 > 0
B0 + B1X1 + B2X2 < 0
Can also look at the magnitude.
How far from zero?
Greater magnitude means more confident prediction.

16. Some Concerns with this Approach:
Datasets with more than 2 target classes
What if a “seperating hyperplane” can’t be formed
Data is more than two dimensions
Regression instead of classification
SVMs can deal with each of these.

17. What if Data is more than 2-Dimensions?
Mathematical definition of hyperplane generalizes to n-dimensions:

18. Maximum Margin Hyperplane
What’s the best separating hyperplane?
Intuition: the one that is farthest from the training observations.
Called the maximum margin hyperplane.

19. The Margin B1 and B2 are each separating hyperplanes
B1 is better
Margin: the smallest distance from the hyperplane to the training data

20. Maximal Margin Hyperplane
Represents the mid-line of the widest “slab” that can be inserted between the two classes.
Maximal Margin Hyperplane
We want the hyperplane that has the greatest margin.
That is, B1 instead of B2 or any of the other infinitely many separating hyperplanes

21. Maximal Margin Hyperplane
Support Vectors: the points in the data, that if moved, the maximal margin hyperplane would move as well.
Moving any of the other data points would not affect the model.

22. Figuring Out the Maximal Margin Classifier
Don’t worry, data mining toolkits do it automatically.
Optimization problem.
Involves calculus.

23. Support Vector Classifier
Maximum Margin Classifier is natural way to perform classification if a separating hyperplane exists.
Perfect segmentation between two classes
In many cases, no separating hyperplane will exist
Find a hyperplane that almost perfectly segments the classes
This generalization is called: support vector classifier

24. Support Vector Classifier
Maximal Margin Classifier: no training errors allowed
Support Margin Classifier: tolerate training errors
Approach: Soft margin
Will allow construction of linear decision boundary even when classes are not linearly separable.

25. Support Vector Classifier
Additional motivation:
New data point added.
Dramatic shift in maximal margin hyperplane.
Model has high variance when trying to maintain perfect segmentation.
Maximum margin classifier.
Perfectly segments training data.

26. Support Vector Classifier
So, interested in:
Greater robustness to individual data instances
Better classification of most of the training data
Some misclassifications permitted:
“Soft” margin: because margin can be violated by some of the instances

27. Red Instances:
on correct side of margin
2 is on the margin
1 is on the wrong side of the margin
Red Instances:
on correct side of margin
2 is on the margin
1 is on the wrong side of the margin
11 is on the wrong side of the hyperplane

28. Using Support Vector Classifier for Classification
Same as before.
Which side of the line is the test instance on?

29. Constructing the Support Vector Classifier
More interesting.
How much “softness” (misclassifications) in the soft margin is ideal?
Complicated math, but python figures it out.
Specification of nonnegative tuning parameter C
Generally chosen by analyst following cross-validation
Large C: wider margin; more instances violate margin
Small C: narrower margin; less tolerance for instances that violate margin

30. Same data points.
Larger C to Smaller C
Lower variance.
Higher variance.

31. Support Vector Machines
What if a non-linear decision boundary is needed?
Poor performance using this decision boundary.

32. Support Vector Machines
Idea: transform the data from its original coordinate space in X into a new space Φ(X) so that a linear decision boundary can separate the two classes
Φ: nonlinear transformation
Huh?
Instead of fitting a support vector classifier using n features:
X1, X2, …, Xn
… use 2n features:
X1, X12, X2, X22, …, Xn, Xn2

33. Support Vector Machines
Enlarged “feature space” compared to original “feature space”
Can even extend to higher-order polynomial terms.
Downside: can easily end up with huge number of features
overfitting

34. Attribute Transformation

35. Learning a Nonlinear SVM Model
Once again:
Complicated, but python does it for us.

36. Other extensions to SVMs:
Regression instead of classification
Categorical variables instead of continuous
Multiclass problems instead of binary
Radial “kernals”
Circle instead of hyperplane

37. Today…
Support Vector Machines
Multiclass Problems
Feature Selection

38. Multiclass Problems Scenario: target class is more than 2 categories
Ideas?
Scenario: target class is more than 2 categories
Motivation: some machine learning algorithms are designed for binary classification
Example: Support Vector Machines (SVM)
How to extend binary classifiers to handle multiclass problems?

39. #1 - Multiclass: One-Against-Rest
Assume multiclass dataset with K target classes
Decompose into K binary problems
Idea: For each target class yi create a single binary problem, with classifier Ci:
Class yi: positive
All other classes: negative
Training:
use all instances; each instance used in training each of the Ci classifiers
Testing:
run testing instance through each classifier
record votes for each yi class (negative prediction is a vote for all other classes)
class with most votes is the predicted class

40. #2 - Multiclass: One-Against-One
Assume multiclass dataset with K target classes
Train K(K-1)/2 binary classifiers (many more than One-Against-Rest)
Idea: Each classifier distinguishes between pair of classes (yi, yj)
Classifier ignores records that don’t belong to yi or yj
Training: use all instances; each instance only used in training “relevant classifiers” (K of them)
Testing:
run testing instance through each classifier
record votes for each yi class
class with most votes is the predicted class

41. Today… Support Vector Machines Multiclass Problems Feature Selection
High Dimensionality
Feature Subset Selection
Adjusted R2 Statistic

42. High Dimensionality … can be bad
Datasets can have a large number of features
Example: big data
Example: stock prices (time series)
Each stock is individual instance
Features/variables are closing price on given day
Imagine 30 years worth of closing prices (30 x 365)

43. Why is it a Problem?
BAD: p > n,
p = # of features
n = # of instances
Often data mining algorithms work better if there are not an overwhelming number of attributes
The “dimensionality” is lower
“The Curse of Dimensionality”
As dimensionality increases (more features), the data becomes increasingly sparse in the “feature space” that it occupies.
Not enough data objects for the number of features that are present
Reduced classification model accuracy

44. Other Benefits to Dimensionality Reduction
More understandable models
Learned model may involve fewer attributes
Better visualizations
Fewer attributes = less variables to plot
Computational time
Fewer attributes = quicker model learning?
Elimination of irrelevant features

45. Techniques for Dimensionality Reduction
Advanced: Linear Algebra Techniques
Automatic approaches
Project data from high-dimensional space into a lower- dimensional space
Principal Components Analysis (PCA)
Singular Value Decomposition (SVD)
Not necessarily interested in “losing information”; rather eliminate some of the sparsity

46. Techniques for Dimensionality Reduction
Feature Construction
Example: combining two separate features (# of full baths, # of half baths) into one feature (“total baths”)
Example: combining features (mass) and (volume) into one feature (density), where density = mass / volume

47. Techniques for Dimensionality Reduction
Feature Subset Selection
Reducing number of features by only using a subset of features
How many should be in the subset?
Losing information if we only consider a subset of features?
Redundant features
Example: (1) purchase price and (2) sales tax
Irrelevant features
Example: student id numbers
By eliminating unnecessary features, we hope for a better model.

48. Eliminating Redundant and Irrelevant Features
Manually via Data Analyst
Intuition about problem domain
Systematic Approach
Try all possible combinations of feature subsets?
See which combination results in best model
For n features, there are 2n possible combinations of subsets
Infeasible to try each of them

49. Three Systematic Approaches
Embedded Approaches
Filter Approaches
Wrapper Approaches

50. Embedded Approaches Algorithm specific
Occurs naturally as part of the data mining algorithm
Example: present in decision tree induction
Only certain subset of features are used in final decision tree
Example: not present in linear regression
Fitted model contained coefficient for each predictor variable

51. Filter Approaches
Features are selected before the data mining algorithm is run
Filter approach is independent of the data mining task
Example: (trying to eliminate redundant features)
Look at pairwise correlation between variables
Pick subset of variables that each have low pairwise correlation
Then use that only that subset in Linear Regression model.

52. Wrapper Approaches
Data mining algorithm is a “black box” for finding best subset of features
Tries different combinations of subsets
Typically will never enumerate all 2n possible combinations
Will search a feature space that is much smaller
Final model uses the specific subset that evaluates the best

53. Top-Down Wrapper Assuming n number of features…
Start with no attributes
Train classifier n times, each time with a different feature
Each classifier only has a single predictor
See which of the n classifiers performs the best
Add to the best classifier. Recursively use remaining attributes to find which attribute that improves performance the most
Keep including best attribute
Stopping criterion: Stop if no improvement to classifier performance, or increase in classifier performance is less than some threshold

54. Bottom-Up Wrapper Assuming n number of features…
Start with all n attributes in model
Create n models, each with a different predictor omitted.
Each classifier has n-1 predictors
See which of the n classifiers affects performance the least
Throw that attribute out
Recursively find the attribute that affects performance the least
Stopping criterion: Stop if classifier performance begins to degrade

55. Other Wrappers Bi-Directional Greedy Search with Backtracking …
Combining Top-Down and Bottom-Up
Greedy Search with Backtracking
(if you’re familiar with AI) …

56. Adjusted R2 Statistic
Always increases as more variables are added to the model.
Recall the R2 statistic that we saw in Linear Regression:
Measured the proportion of variance explained by the model
Always a value between 0 and 1
Higher is better

57. Adjusted R2 Statistic
In contrast to R2, Adjusted R2 penalizes for unnecessary variables in the model.
d = number of predictors
n = number of instances
𝐴𝑑𝑗𝑢𝑠𝑡𝑒𝑑 𝑅 2 =1− 𝑅𝑆𝑆 (𝑛−𝑑−1) 𝑇𝑆𝑆 (𝑛−1)

58. References Data Science from Scratch, 1st Edition, Grus
Introduction to Data Mining, 1st edition, Tan et al.
Data Mining and Business Analytics with R, 1st edition, Ledolter