Learning Kernel Classifiers Chap. 3.3 Relevance Vector Machine Chap. 3.4 Bayes Point Machines Summarized by Sang Kyun Lee 13 th May, 2005

3.3 Relevance Vector Machine ● [M.Tipping, JMLR 2001] ● Modification to Gaussian process – GP ● Prior ● Likelihood ● Posterior – RVM ● Prior ● Likelihood same as GP ● Posterior

● Reasons – To get sparce representation of – Expected risk of classifier, ● Thus, we favor weight vectors with a small number of non- zero coeffs. – One way to achieve this is to modify prior: – Consider ● Then wi=0 is only possible ● Computation of is easier than before

● Prediction funcion – GP – RVM

● How can we learn the sparce vector – To find the best, employ evidence maximizaion – The evidence is given explicitly by, – Derived update rules (App'x B.8):

● Evidence Maximization – Interestingly, many of the decrease quickly toward zero which lead to a high sparsity in – For faster convergence, delete ith column from whenever < pre-def threshold – After termination, set the corresponding = 0 for which < thres. The remaining are set equal to corresponing values in

● Application to Classification – Consider latent target variables – Training objects: – Test object: – Compute the predictive distribution of at the new object, ● by applying a latent weight vector to all the m+1 objects ● and marginalizing over all, we get

– Note – As in the case of GP, we cannot solve this analytically because is no longer Gaussian – Laplace approximaion: approx. this density by a Gaussian density w/ mean and cov.

● Kernel trick – Think about a RKHS generated by – Then ith component of training objects is represented as – Now, think about regression. The concept of becomes the expansion coeff. of the desired hyperplane, such that – In this sense, all the training objects which have non-zero are termed relevance vectors

3.4 Bayes Point Machines ● [R. Herbrich, JMLR 2000] ● In GP and RVMs, we tried to solve classification problem via regression estimation ● Before we assumed prior dist. and used logit transformations to model the likelihood distribution, ● Now we try to model it directly

● Prior – For classification, only the spatial direction of. Note that – Thus we consider only the vectors on unit sphere – Then assume a uniform prior over this ball-shaped hypothesis space

● Likelihood – Use PAC likelihood (0-1 loss) ● Posterior – Remark: using PAC likelihood,

● Predictive distribution – In two class case, the Bayesian decision can be written as: ● That is, the Bayes classification strategy performs majority voting involving all version space classifiers ● However, the expectation is hard to solve ● Hence we approximate it by a single classifier

– That is, BP is the optimal projection of the Bayes classifiers to a single classifier w.r.t. generalization error – However this also is intractable because we need to know input distribution and posterior – Another reasonable approximation:

● Now the Bayes classification of new object equals to the classification w.r.t. the single weight vector ● Estimate by MCMC sampling (‘kernel billiard algorithm’)