Dept. Computer Science & Engineering, Shanghai Jiao Tong University Kernel Methods Dept. Computer Science & Engineering, Shanghai Jiao Tong University

Outline One-Dimensional Kernel Smoothers Local Regression Local Likelihood Kernel Density estimation Naive Bayes Radial Basis Functions Mixture Models and EM 2018/7/4 Kernel Methods

One-Dimensional Kernel Smoothers k-NN: 30-NN curve is bumpy, since is discontinuous in x. The average changes in a discrete way, leading to a discontinuous . 2018/7/4 Kernel Methods

One-Dimensional Kernel Smoothers Nadaraya-Watson Kernel weighted average: Epanechnikov quadratic kernel: 2018/7/4 Kernel Methods

One-Dimensional Kernel Smoothers More general kernel: : width function that determines the width of the neighborhood at x0. For quadratic kernel For k-NN kernel Variance constant The Epanechnikov kernel has compact support 2018/7/4 Kernel Methods

One-Dimensional Kernel Smoothers Three popular kernel for local smoothing: Epanechnikov kernel and tri-cube kernel are compact but tri-cube has two continuous derivatives Gaussian kernel is infinite support 2018/7/4 Kernel Methods

Local Linear Regression Boundary issue Badly biased on the boundaries because of the asymmetry of the kernel in the region. Linear fitting remove the bias to first order 2018/7/4 Kernel Methods

Local Linear Regression Locally weighted linear regression make a first-order correction Separate weighted least squares at each target point x0: The estimate: b(x)T=(1,x); B: Nx2 regression matrix with i-th row b(x)T; 2018/7/4 Kernel Methods

Local Linear Regression The weights combine the weighting kernel and the least squares operations——Equivalent Kernel 2018/7/4 Kernel Methods

Local Linear Regression The expansion for , using the linearity of local regression and a series expansion of the true function f around x0 For local regression The bias depends only on quadratic and higher-order terms in the expansion of . 2018/7/4 Kernel Methods

Local Polynomial Regression Fit local polynomial fits of any degree d 2018/7/4 Kernel Methods

Local Polynomial Regression Bias only have components of degree d+1 and higher. The reduction for bias costs the increased variance. 2018/7/4 Kernel Methods

选择核的宽度 核 中， 是参数，控制核宽度： 窗口宽度导致偏倚-方差权衡： 对于有紧支集的核， 取其支集区域的半径 对于高斯核， 取其方差 核 中， 是参数，控制核宽度： 对于有紧支集的核， 取其支集区域的半径 对于高斯核， 取其方差 对k-对近邻域法， 取 k/N 窗口宽度导致偏倚-方差权衡： 窗口较窄，方差误差大，均值误差偏倚小 窗口较宽，方差误差小，均值误差偏倚大 2018/7/4 Kernel Methods

Structured Local Regression Structured kernels Introduce structure by imposing appropriate restrictions on A Structured regression function Introduce structure by eliminating some of the higher-order terms 2018/7/4 Kernel Methods

Local Likelihood & Other Models Any parametric model can be made local: Parameter associated with : Log-likelihood: Model likelihood local to : A varying coefficient model 2018/7/4 Kernel Methods

Local Likelihood & Other Models Logistic Regression Local log-likelihood for the J class model Center the local regressions at 2018/7/4 Kernel Methods

Kernel Density Estimation A natural local estimate The smooth Parzen estimate For Gaussian kernel The estimate become 2018/7/4 Kernel Methods

Kernel Density Estimation A kernel density estimate for systolic blood pressure. The density estimate at each point is the average contribution from each of the kernels at that point. 2018/7/4 Kernel Methods

Kernel Density Classification Bayes’ theorem: The estimate for CHD uses the tri-cube kernel with k-NN bandwidth. 2018/7/4 Kernel Methods

Kernel Density Classification The population class densities and the posterior probabilities 2018/7/4 Kernel Methods

Naïve Bayes Naïve Bayes model assume that given a class G=j, the features Xk are independent: is kernel density estimate, or Gaussian, for coordinate Xk in class j. If Xk is categorical, use Histogram. 2018/7/4 Kernel Methods

Radial Basis Function & Kernel Radial basis function combine the local and flexibility of kernel methods. Each basis element is indexed by a location or prototype parameter and a scale parameter , a pop choice is the standard Gaussian density function. 2018/7/4 Kernel Methods

Radial Basis Function & Kernel For simplicity, focus on least squares methods for regression, and use the Gaussian kernel. RBF network model: Estimate the separately from the . A undesirable side effect of creating holes——regions of IRp where none of the kernels has appreciable support. 2018/7/4 Kernel Methods

Radial Basis Function & Kernel Renormalized radial basis functions. The expansion in renormalized RBF Gaussian radial basis function with fixed width can leave holes. Renormalized Gaussian radial basis function produce basis functions similar in some respects to B-splines. 2018/7/4 Kernel Methods

Mixture Models & EM Gaussian Mixture Model EM algorithm for mixtures are mixture proportions, EM algorithm for mixtures Given log-likelihood: Suppose we observe Latent Binary Bad Good 2018/7/4 Kernel Methods

Mixture Models & EM Given ,compute In Example 2018/7/4 Kernel Methods

Mixture Models & EM Application of mixtures to the heart disease risk factor study. 2018/7/4 Kernel Methods

Mixture Models & EM Mixture model used for classification of the simulated data 2018/7/4 Kernel Methods

2018/7/4 Kernel Methods