1. Class 19, spring 2001 CBCl/AI MIT Review by Evgeniou, Pontil and Poggio Advances in Computational Mathematics, 2000 The “b” problem We said that the solution to with K positive definite, is

2. Class 19, spring 2001 CBCl/AI MIT The “b” problem For RN when K is a positive definite kernel the solution is For SVM, since Vapnik introduced them, even when K is a positive definite kernel, the solution is given as This is puzzling since we have seen that RN and SVM are the same (just with a different loss function which is independent of K). What is the solution of this small puzzle?

3. Class 19, spring 2001 CBCl/AI MIT We know that the function f that satisfies has the form where the kernel K is conditionally positive definite of order k and p(x) is a polynomial of degree (k-1) Wahba 1990; Poggio and Girosi, 1989; Smale and Cucker, 2001 The “b” problem: Regularization Networks

4. Class 19, spring 2001 CBCl/AI MIT Definition: Positive definite kernels A real-valued kernel K(x,y) is positive definite if and only if K is symmetric and for any n distinct points x in the bounded domain X The “b” problem

5. Class 19, spring 2001 CBCl/AI MIT Definition: Conditionally positive definite kernels of order 1 (-K is called negative definite) A real-valued kernel K(x,y) is conditionally positive definite of order 1 if and only if K is symmetric and for any n distinct points x in the domain X and scalars c such that the quadratic form is nonnegative, that is The “b” problem

6. Class 19, spring 2001 CBCl/AI MIT Notice that positive definite kernels are conditionally positive definite of order 1 (from the definition). Thus in RN with a positive definite K we can always look for a solution but also (considering K to be conditionally positive) The “b” problem What does it mean?

7. Class 19, spring 2001 CBCl/AI MIT Theorem: If K is positive definite then is conditionally positive definite of order 1. The proof is easy from Corollary 2.1 b of Micchelli, 1986. Notice that this corresponds to subtracting out the constant term in the expansion of K in the eigenfunctions, which in turn means that the constant term in the expansion of f is not penalized in the RKHS norm.. The “b” problem

8. Class 19, spring 2001 CBCl/AI MIT Reproducing Kernel Hilbert Spaces the space is a RKHS with K as the reproducing kernel. Given a positive function Given a positive definite function The RKHS norm is with on a bounded domain

9. Class 19, spring 2001 CBCl/AI MIT Suppose in RN we start with K positive definite and then we consider K’ =K –const which is conditionally positive definite. If I regularize with K’, that is I look for The solution is The “b” problem because of the conditions on the coefficients.

10. Class 19, spring 2001 CBCl/AI MIT In this case the coefficients are given by solving These equations correspond to the minimization problem with K’. The “b” problem

11. Class 19, spring 2001 CBCl/AI MIT If I use K positive definite and I look for The solution is The “b” problem with

12. Class 19, spring 2001 CBCl/AI MIT Thus I have two different approximations for f, corresponding to whether constant shifts in f are penalized or not. The two representations are The “b” problem In the case of RN (quadratic loss function)

13. Class 19, spring 2001 CBCl/AI MIT The same (but not the last equation) is true for SVMs. Thus Vapnik’s solution corresponds to using a conditionally positive K’ instead of K in the RKHS norm, thereby NOT penalizing constants in f. This choice depends on the problem but it is often reasonable especially in classification problems. On the other hand, our argument shows that we can use directly a positive definite stabilizer in SVM. This implies that the QP problem can be solved WITHOUT the constraint which makes the QP optimization an easier problem… The “b” problem

14. Class 19, spring 2001 CBCl/AI MIT Given a positive definite K the are two possible choices. The first is to find the function solving The “b” problem: summary The other is to consider K’=K-const and minimize which is which gives