Kernel Methods A B M Shawkat Ali 1

2 Data Mining ¤ DM or KDD (Knowledge Discovery in Databases) Extracting previously unknown, valid, and actionable information  crucial decisions ¤ Approach Model Train Data crucial decisions Test Data

History of SVM The original optimal hyperplane algorithm proposed by Vladimir Vapnik in 1963 was a linear classifier. However, in 1992, Bernhard Boser, Isabelle Guyon and Vapnik suggested a way to create non-linear classifiers by applying the kernel trick (originally proposed by Aizerman et al.) to maximum- margin hyperplanes. The resulting algorithm is formally similar, except that every dot product is replaced by a non-linear kernel function. This allows the algorithm to fit the maximum-margin hyperplane in a transformed feature space. The transformation may be non-linear and the transformed space high dimensional; thus though the classifier is a hyperplane in the high-dimensional feature space, it may be non-linear in the original input space.

4 Property of the SVM ¤ Relatively new approach ¤ Lot of interest recently:  Many successes, e.g., text classification ¤ Important concepts:  Transformation into high dimensional space  Finding a "maximal margin" separation  Structural risk minimization rather than Empirical risk minimization

5 Support Vector Machine (SVM) ¤ Classification  Grouping of similar data. ¤ Regression  Prediction by historical knowledge. ¤ Novelty Detection  To detect abnormal instances from a dataset. ¤ Clustering, Feature Selection

6 SVM Block Diagram               Training Data Domain Non linear Mapping by Kernel                                    To Choose Optimal Hyperplane Linear Feature Space of SVM

7 SVM Block Diagram Constructed Model through Feature knowledge     Class I        Class II Test Data Domain Kernel Mapping           

8 SVM Formulation

9

10 SVM Formulation Mercer’s Condition

11 Types of Kernels Common kernels for SVM ¤ Linear ¤ Polynomial ¤ Radial Basis Function New kernels (not used in SVM) ¤ Laplace ¤ Multiquadratic

12 SVM kernel Polynomial Gaussian (Radial Basis Function) Linear

13 Laplace kernel Introduced by Pavel Paclik et. al. in Pattern Recognition letters 21 (2000) Laplace Kernel based on Laplace Probability Density Smoothing Parameter (Sp)

14 Linear Kernel

The reality of data separation

16 RBF kernel

XOR solved by SVM Input data xOutput class y (-1,-1) (-1,+1)+1 (+1,-1)+1 (+1,+1) Table 5.3. Boolean XOR Problem

First, we transform the dataset by polynomial kernel as: Here,

Therefore the kernel matrix is: We can write the maximization term following SVM implementation given in Figure 5.20 as: subject to:,

By solving these above equations we can write the solution to this optimisation problem as:. Therefore, the decision function in the inner product representation is:

The 2 nd degree polynomial kernel function: Now we can write the 2nd degree polynomial transformation function as:

= Therefore the optimal hyperplane function for this XOR problem is:

Conclusions Research Issues – How to select a kernel automatically – How to select optimal parameter values for kernel