The value of kernel function represents the inner product of two training points in feature space Kernel functions merge two steps 1. map input data from input space to feature space (might be infinite dim.) 2. do inner product in the feature space Kernel Technique Based on Mercer ’ s Condition (1909)

A Simple Example of Kernel Polynomial Kernel of Degree 2: Let and the nonlinear map defined by. Then.  There are many other nonlinear maps,, that satisfy the relation:

Power of the Kernel Technique Consider a nonlinear mapthat consists of distinct features of all the monomials of degree d. Then. For example:  Is it necessary? We only need to know !  This can be achieved

More Examples of Kernel is an integer:  Polynomial Kernel : ) (Linear Kernel :  Gaussian (Radial Basis) Kernel :  The -entry of represents the “similarity” of data pointsand

Nonlinear SVM Motivation  Linear SVM: (Linear separator: ) min s. t. (QP) By QP “duality”,. Maximizing the margin in the “dual space” gives: min  Dual SSVM with separator: min s. t.

Nonlinear Smooth SVM  Replace by a nonlinear kernel : min  Use Newton-Armijo algorithm to solve the problem  Each iteration solves m+1 linear equations in m+1 variables  Nonlinear classifier depends on entire dataset : Nonlinear Classifier:

Difficulties with Nonlinear SVM for Large Problems  The nonlinear kernel is fully dense  Computational complexity depends on # of example  Separating surface depends on almost entire dataset  Complexity of nonlinear SVM  Runs out of memory while storing the kernel matrix  Long CPU time to compute the dense kernel matrix  Need to generate and store entries  Need to store the entire dataset even after solving the problem

Solving the SVM with Massive Dataset  Limit the SVM to dataset of a few thousand points  Solution I: SMO (Sequential Minimal Optimization)  Standard optimization techniques require that the the data are held in memory  Solve the sub-optimization problem defined by the working set (size =2)  Increase the objective function iteratively  Solution II: RSVM (Reduced Support Vector Machine)

Reduced Support Vector Machine (ii) Solve the following problem by the Newton’s method min (iii) The nonlinear classifier is defined by the optimal solution in step (ii): Using gives lousy results! (i) Choose a random subset matrixof entire data matrix Nonlinear Classifier:

A Nonlinear Kernel Application Checkerboard Training Set: 1000 Points in Separate 486 Asterisks from 514 Dots

Conventional SVM Result on Checkerboard Using 50 Randomly Selected Points Out of 1000

RSVM Result on Checkerboard Using SAME 50 Random Points Out of 1000

RSVM on Moderate Sized Problems (Best Test Set Correctness %, CPU seconds) Cleveland Heart 297 x 13, BUPA Liver 345 x 6, Ionosphere 351 x 34, Pima Indians 768 x 8, Tic-Tac-Toe 958 x 9, Mushroom 8124 x 22, N/A

RSVM on Large UCI Adult Dataset Standard Deviation over 50 Runs = Average Correctness % & Standard Deviation, 50 Runs (6414, 26148) % (11221, 21341) % (16101, 16461) % (22697, 9865) % (32562, 16282) %

Reduced Set: Plays the Most Important Role in RSVM  It is natural to raise two questions:  Is there a way to choose the reduced set other than random selection so that RSVM will have a better performance?  Is there a mechanism to determine the size of reduced set automatically or dynamically?

Reduced Set Selection According to the Data Scatter in Input Space  Expected these points to be representative sample  Choose reduced set randomly but only keep the points in the reduced set that are more than a certain minimal distance apart

Data Scatter in Input Space is NOT Good Enough  An example is given as following: Training data analogous to XOR problem

Mapping to Feature Space  Map the input data via nonlinear mapping ：  Equivalent to polynomial kernel with degree 2:

Data Points in the Feature Space

The Polynomial Kernel Matrix

Experiment Result

Express the Classifier as Linear Combination of Kernel Functions is a linear combination of a set of kernel functions  In SSVM, the nonlinear separating surface is:  In RSVM, the nonlinear separating surface is: is a linear combination of a set of kernel functions

Motivation of IRSVM The Strength of Weak Ties  The strength of weak ties  Mark S. Granovetter, The American Journal of Sociology, Vol. 78, No , May, 1973  If the kernel functions are very similar, the space spanned by these kernel functions will be very limited.

Incremental Reduced SVMs  Start with a very small reduced set, then add a new data point only when the kernel vector is dissimilar to the current function set  This point contributes the most extra information for generating the separating surface  Repeat until several successive points cannot be added

How to measure the dissimilarity? the kernel vector to the column space of is greater than a threshold  Add a point into the reduced set if the distance from

 This distance can be determined by solving a least squares problem Solving Least Squares Problems  The LSP has a unique solution if and

IRSVM Algorithm pseudo-code (sequential version) 1 Randomly choose two data from the training data as the initial reduced set 2 Compute the reduced kernel matrix 3 For each data point not in the reduced set 4 Computes its kernel vector 5 Computes the distance from the kernel vector 6 to the column space of the current reduced kernel matrix 7 If its distance exceed a certain threshold 8 Add this point into the reduced set and form the new reduced kernel matrix 9 Until several successive failures happened in line 7 10 Solve the QP problem of nonlinear SVMs with the obtained reduced kernel 11 A new data point is classified by the separating surface

Speed up IRSVM  The main cost depends on but not on  Take this advantage this, we examine a batch data points at the same  We have to solve the LSP many times and the complexity is

IRSVM Algorithm pseudo-code (Batch version) 1 Randomly choose two data from the training data as the initial reduced set 2 Compute the reduced kernel matrix 3 For a batch data point not in the reduced set 4 Computes their kernel vectors 5 Computes the corresponding distances from these kernel vector 6 to the column space of the current reduced kernel matrix 7 For those points’ distance exceed a certain threshold 8 Add those point into the reduced set and form the new reduced kernel matrix 9 Until no data points in a batch were added in line 7,8 10 Solve the QP problem of nonlinear SVMs with the obtained reduced kernel 11 A new data point is classified by the separating surface

IRSVM on Four Public Datasets

IRSVM on UCI Adult datasets

Time comparison on Adult datasets

IRSVM 10 Runs Average on 6414 Points Adult Training Set