Learning in Feature Space (Could Simplify the Classification Task)  Learning in a high dimensional space could degrade generalization performance  This phenomenon is called curse of dimensionality  By using a kernel function, that represents the inner product of training example in feature space, we never need to explicitly know what the nonlinear map is.  Even do not know the dimensionality of feature space  There is no free lunch  Deal with a huge and dense kernel matrix Reduced kernel can avoid this difficulty

Linear Machine in Feature Space Let be a nonlinear map from the input space to some feature space The classifier will be in the form ( Primal ): Make it in the dual form:

The Perceptron Algorithm (Dual Form) Given a linearly separable training setand Repeat: until no mistakes made within the for loop return:

Kernel: Represent Inner Product in Feature Space The classifier will become: Definition: A kernel is a function such that where

A Simple Example of Kernel Polynomial Kernel of Degree 2: Let and the nonlinear map defined by. Then.  There are many other nonlinear maps,, that satisfy the relation:

Power of the Kernel Technique Consider a nonlinear mapthat consists of distinct features of all the monomials of degree d. Then. For example:  Is it necessary? We only need to know !  This can be achieved

Basic Properties of Kernel Function  Symmetric (inherit from inner product)  Cauchy-Schwarz inequality  These conditions are not sufficient to guarantee the existence of a feature space

Characterization of Kernels Motivation in Finite Input Space Consider a finite space and is a symmetric function on. Letbe a matrix defined as following: There is an orthogonal matrix such that:

Characterization of Kernels Assume: Let Be Positive Semi-definite where

Mercer’s Conditions: Guarantee the Existence of Feature Space and is a symmetric function on. be a finite space Let Then is a kernel function if and only if is positive semi-definite.  What if is infinite (but compact)? Mercer’s conditions: Any finite subset of the corresponding matrix is positive semi-definite.

Making Kernels Kernels Satisfy a Number of Closure Properties Let Then the following functions are kernels: be kernels over be a kernel over and be a symmetric positive semi-definite.

Translation Invariant Kernels two inputs is unchanged if both are translated by the same vector.  The inner product (in the feature space) of  The kernels are in the form:  Some examples:  Gaussian RBF:  Multiquadric:  Fourier: see Example 3.9 on p. 37

A Negative Definite Kernel  Generalized Support Vector Machine  The kernelis negative definite  Does not satisfy Mercer ’ s conditions  Oliv L. Mangansarian used this kernel to solve XOR classification problem