Kernel Regression Prof. Bennett Math Model of Learning and Discovery 1/28/05 Based on Chapter 2 of Shawe-Taylor and Cristianini

Outline Review Ridge Regression LS-SVM=KRR Dual Derivation Bias Issue Summary

Ridge Regression Review Use least norm solution for fixed Regularized problem Optimality Condition: Requires 0(n 3 ) operations

Dual Representation Inverse always exists for any Alternative representation: Solving l  l equation is 0(l 3 )

Dual Ridge Regression To predict new point: Note need only compute G, the Gram Matrix Ridge Regression requires only inner products between data points

Linear Regression in Feature Space Key Idea: Map data to higher dimensional space (feature space) and perform linear regression in embedded space. Embedding Map:

Kernel Function A kernel is a function K such that There are many possible kernels. Simplest is linear kernel.

Ridge Regression in Feature Space To predict new point: To compute the Gram Matrix Use kernel to compute inner product

Alternative Dual Derivation Original math model Equivalent math model Construct dual using Wolfe Duality

Lagrangian Function Consider the problem Lagrangian function is

Wolfe Dual Problem Primal Dual

Lagrangian Function Primal Lagrangian

Wolfe Dual Problem Construct Wolfe Dual Simplify by eliminating z= 

Simplified Problem Get rid of z Simplify by eliminating w=X’ 

Simplified Problem Get rid of w

Optimal solution Problem in matrix notation with G=XX’ Solution satisfies

What about Bias If we limit regression function to f(x)=w’x means that solution must pass through origin. Many models may require a bias or constant factor f(x)=w’x+b

Eliminate Bias One way to eliminate bias is to “center” the response Make response have mean of 0

Center y Y now has sample mean of 0 Frequently good to make y have standard length:

Centering X may be good idea Mean X Center X

Scaling X may be a good idea Compute Standard Deviation Scale columns/variables

You Try Consider data matrix with 3 points in 4 dimensions Computer the centered X by hand and with the following formula, then scale

Center  (X) in Feature Space We cannot center  (X) directly in feature space. Center G = XX’ Works in feature space too for G in kernel space

Centering Kernel Practical Computation:

Ridge Regression in Feature Space Original way Predicted normalized y Predicted original y

Worksheet Normalized Y Invert to get unnormalized y

Centering Test Data Calculate test data just like training data: Prediction of test data becomes:

Alternate Approach Directly add bias to the model: Optimization problem becomes:

Lagrangian Function Consider the problem Lagrangian function is

Lagrangian Function Primal

Wolfe Dual Problem Simplify by eliminating z=  and using e’  =0

Simplified Problem Simplify by eliminating w=X’ 

Simplified Problem Get rid of w

New Problem to be solved Problem in matrix notation with G=XX’ This is a constrained optimization problem. Solution is also system of equations, but not as simple.

Kernel Ridge Regression Centered algorithm just requires centering of the kernel and solving one equation. Can also add bias directly. + Lots of fast equation solvers. + Theory supports generalization - requires full training kernel to compute  - requires full training kernel to predict future points