University of Texas at Austin Machine Learning Group Department of Computer Sciences University of Texas at Austin Support Vector Machines

2 University of Texas at Austin Machine Learning Group Perceptron Revisited: Linear Separators Binary classification can be viewed as the task of separating classes in feature space: w T x + b = 0 w T x + b < 0 w T x + b > 0 f(x) = sign(w T x + b)

3 University of Texas at Austin Machine Learning Group Linear Separators Which of the linear separators is optimal?

4 University of Texas at Austin Machine Learning Group Classification Margin Distance from example to the separator is Examples closest to the hyperplane are support vectors. Margin ρ of the separator is the width of separation between classes. r ρ

5 University of Texas at Austin Machine Learning Group Maximum Margin Classification Maximizing the margin is good according to intuition and PAC theory. Implies that only support vectors are important; other training examples are ignorable.

6 University of Texas at Austin Machine Learning Group Linear SVM Mathematically Assuming all data is at distance 1 from the hyperplane, the following two constraints follow for a training set { ( x i,y i )} For support vectors, the inequality becomes an equality; then, since each example’s distance from the hyperplane is the margin is: w T x i + b ≥ 1 if y i = 1 w T x i + b ≤ -1 if y i = -1

7 University of Texas at Austin Machine Learning Group Linear SVMs Mathematically (cont.) Then we can formulate the quadratic optimization problem: A better formulation: Find w and b such that is maximized and for all { ( x i,y i )} w T x i + b ≥ 1 if y i =1; w T x i + b ≤ -1 if y i = -1 Find w and b such that Φ(w) =½ w T w is minimized and for all { ( x i,y i )} y i (w T x i + b) ≥ 1

8 University of Texas at Austin Machine Learning Group Solving the Optimization Problem Need to optimize a quadratic function subject to linear constraints. Quadratic optimization problems are a well-known class of mathematical programming problems, and many (rather intricate) algorithms exist for solving them. The solution involves constructing a dual problem where a Lagrange multiplier α i is associated with every constraint in the primary problem: Find w and b such that Φ(w) =½ w T w is minimized and for all { ( x i,y i )} y i (w T x i + b) ≥ 1 Find α 1 …α N such that Q( α ) = Σ α i - ½ ΣΣ α i α j y i y j x i T x j is maximized and (1) Σ α i y i = 0 (2) α i ≥ 0 for all α i

9 University of Texas at Austin Machine Learning Group The Optimization Problem Solution The solution has the form: Each non-zero α i indicates that corresponding x i is a support vector. Then the classifying function will have the form: Notice that it relies on an inner product between the test point x and the support vectors x i – we will return to this later! Also keep in mind that solving the optimization problem involved computing the inner products x i T x j between all training points! w = Σ α i y i x i b= y k - w T x k for any x k such that α k  0 f(x) = Σ α i y i x i T x + b

10 University of Texas at Austin Machine Learning Group Soft Margin Classification What if the training set is not linearly separable? Slack variables ξ i can be added to allow misclassification of difficult or noisy examples. ξiξi ξiξi

11 University of Texas at Austin Machine Learning Group Soft Margin Classification Mathematically The old formulation: The new formulation incorporating slack variables: Parameter C can be viewed as a way to control overfitting. Find w and b such that Φ(w) =½ w T w is minimized and for all { ( x i,y i )} y i (w T x i + b) ≥ 1 Find w and b such that Φ(w) =½ w T w + C Σ ξ i is minimized and for all { ( x i,y i )} y i (w T x i + b) ≥ 1- ξ i and ξ i ≥ 0 for all i

12 University of Texas at Austin Machine Learning Group Soft Margin Classification – Solution The dual problem for soft margin classification: Neither slack variables ξ i nor their Lagrange multipliers appear in the dual problem! Again, x i with non-zero α i will be support vectors. Solution to the dual problem is: Find α 1 …α N such that Q( α ) = Σ α i - ½ ΣΣ α i α j y i y j x i T x j is maximized and (1) Σ α i y i = 0 (2) 0 ≤ α i ≤ C for all α i w = Σ α i y i x i b= y k (1- ξ k ) - w T x k where k = argmax α k k f(x) = Σ α i y i x i T x + b But neither w nor b are needed explicitly for classification!

13 University of Texas at Austin Machine Learning Group Theoretical Justification for Maximum Margins Vapnik has proved the following: The class of optimal linear separators has VC dimension h bounded from above as where ρ is the margin, D is the diameter of the smallest sphere that can enclose all of the training examples, and m 0 is the dimensionality. Intuitively, this implies that regardless of dimensionality m 0 we can minimize the VC dimension by maximizing the margin ρ. Thus, complexity of the classifier is kept small regardless of dimensionality.

14 University of Texas at Austin Machine Learning Group Linear SVMs: Overview The classifier is a separating hyperplane. Most “important” training points are support vectors; they define the hyperplane. Quadratic optimization algorithms can identify which training points x i are support vectors with non-zero Lagrangian multipliers α i. Both in the dual formulation of the problem and in the solution training points appear only inside inner products: Find α 1 …α N such that Q( α ) = Σ α i - ½ ΣΣ α i α j y i y j x i T x j is maximized and (1) Σ α i y i = 0 (2) 0 ≤ α i ≤ C for all α i f(x) = Σ α i y i x i T x + b

15 University of Texas at Austin Machine Learning Group Non-linear SVMs Datasets that are linearly separable with some noise work out great: But what are we going to do if the dataset is just too hard? How about… mapping data to a higher-dimensional space: 0 x2x2 x 0 x 0 x

16 University of Texas at Austin Machine Learning Group Non-linear SVMs: Feature spaces General idea: the original feature space can always be mapped to some higher-dimensional feature space where the training set is separable: Φ: x → φ(x)

17 University of Texas at Austin Machine Learning Group The “Kernel Trick” The linear classifier relies on inner product between vectors K(x i,x j )=x i T x j If every datapoint is mapped into high-dimensional space via some transformation Φ: x → φ(x), the inner product becomes: K(x i,x j )= φ(x i ) T φ(x j ) A kernel function is some function that corresponds to an inner product into some feature space. Example: 2-dimensional vectors x=[x 1 x 2 ]; let K(x i,x j )=(1 + x i T x j ) 2, Need to show that K(x i,x j )= φ(x i ) T φ(x j ): K(x i,x j )=(1 + x i T x j ) 2, = 1+ x i1 2 x j x i1 x j1 x i2 x j2 + x i2 2 x j x i1 x j1 + 2x i2 x j2 = = [1 x i1 2 √2 x i1 x i2 x i2 2 √2x i1 √2x i2 ] T [1 x j1 2 √2 x j1 x j2 x j2 2 √2x j1 √2x j2 ] = = φ(x i ) T φ(x j ), where φ(x) = [1 x 1 2 √2 x 1 x 2 x 2 2 √2x 1 √2x 2 ]

18 University of Texas at Austin Machine Learning Group What Functions are Kernels? For some functions K(x i,x j ) checking that K(x i,x j )= φ(x i ) T φ(x j ) can be cumbersome. Mercer’s theorem: Every semi-positive definite symmetric function is a kernel Semi-positive definite symmetric functions correspond to a semi-positive definite symmetric Gram matrix: K(x1,x1)K(x1,x1)K(x1,x2)K(x1,x2)K(x1,x3)K(x1,x3)…K(x1,xN)K(x1,xN) K(x2,x1)K(x2,x1)K(x2,x2)K(x2,x2)K(x2,x3)K(x2,x3)K(x2,xN)K(x2,xN) …………… K(xN,x1)K(xN,x1)K(xN,x2)K(xN,x2)K(xN,x3)K(xN,x3)…K(xN,xN)K(xN,xN) K=

19 University of Texas at Austin Machine Learning Group Examples of Kernel Functions Linear: K(x i,x j )= x i T x j Polynomial of power p: K(x i,x j )= (1+ x i T x j ) p Gaussian (radial-basis function network): K(x i,x j )= Two-layer perceptron: K(x i,x j )= tanh(β 0 x i T x j + β 1 )

20 University of Texas at Austin Machine Learning Group Non-linear SVMs Mathematically Dual problem formulation: The solution is: Optimization techniques for finding α i ’s remain the same! Find α 1 …α N such that Q( α ) = Σ α i - ½ ΣΣ α i α j y i y j K(x i, x j ) is maximized and (1) Σ α i y i = 0 (2) α i ≥ 0 for all α i f(x) = Σ α i y i K(x i, x j )+ b

21 University of Texas at Austin Machine Learning Group SVM applications SVMs were originally proposed by Boser, Guyon and Vapnik in 1992 and gained increasing popularity in late 1990s. SVMs are currently among the best performers for a number of classification tasks ranging from text to genomic data. SVM techniques have been extended to a number of tasks such as regression [Vapnik et al. ’97], principal component analysis [Schölkopf et al. ’99], etc. Most popular optimization algorithms for SVMs are SMO [Platt ’99] and SVM light [Joachims’ 99], both use decomposition to hill-climb over a subset of α i ’s at a time. Tuning SVMs remains a black art: selecting a specific kernel and parameters is usually done in a try-and-see manner.