Statistical Learning Dong Liu Dept. EEIS, USTC

Chapter 5. Non-Parametric Supervised Learning Parzen window k-nearest-neighbor (k-NN) Sparse coding 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Non-parametric learning Most of statistical learning methods assume a model For example, linear regression assumes and linear classification assumes Learning is converted to a problem of solving/estimating model parameters In this chapter, we consider learning without explicit modeling Non-parametric learning is sometimes equivalent to instance/memory-based learning 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Chap 5. Non-Parametric Supervised Learning Parzen window Consider the problem of (probabilistic) density estimation We want to estimate an unknown distribution p(x) from a given set of samples Parametric learning: we assume a Non-parametric learning: we do not presume Parzen window, also known as kernel density estimation Kernel function shall satisfy: non-negative, integrate to 1, e.g. 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Comparison between histogram and Parzen window 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Parzen window: Window size Introduce a hyper-parameter to control window size Then Gray: true density Red: h=0.05 Black: h=0.337 Green: h=2 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Chapter 5. Non-Parametric Supervised Learning Parzen window k-nearest-neighbor (k-NN) Sparse coding 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Chap 5. Non-Parametric Supervised Learning Nearest-neighbor If it walks like a duck, quacks like a duck, then it is probably a duck. Compare & find the “closest” sample New sample Samples in memory 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Chap 5. Non-Parametric Supervised Learning Illustration of 1-NN Reproduced from ESL It is sensitive to noise/outlier in dataset. How to improve? 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Chap 5. Non-Parametric Supervised Learning k-nearest-neighbor Compare and find several “closet” samples, and make decision based on them all k-NN classification k-NN regression 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Chap 5. Non-Parametric Supervised Learning Illustration of k-NN Reproduced from ESL Better than 1-NN in eliminating noise effect 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Chap 5. Non-Parametric Supervised Learning Variants of k-NN How to define neighborhood? Fixed k (then how to set?) Adaptive k (e.g. distance thresholding) How to define distance? Euclidean distance Lp distance Mahalanobis distance Cosine similarity Pearson correlation coefficient Distance weighting Equal weight Less distance, more weight 2018/11/17 Chap 5. Non-Parametric Supervised Learning

How to set k: Bias-variance tradeoff Consider k-NN regression Assume the data are generated by Then we have 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Example: Rating prediction for making recommendations 1/5 Items Search Recommendations Advertisements Products, news, movies, music, … 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Example: Rating prediction for making recommendations 2/5 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Example: Rating prediction for making recommendations 3/5 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Example: Rating prediction for making recommendations 4/5 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Example: Rating prediction for making recommendations 5/5 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Chapter 5. Non-Parametric Supervised Learning Parzen window k-nearest-neighbor (k-NN) Sparse coding 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Chap 5. Non-Parametric Supervised Learning Sparse coding Instead of finding “closest” samples, we want to find “correlated” samples 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Chap 5. Non-Parametric Supervised Learning Sparse coding Sparse coding aims to solve Sparse coding can be relaxed to deal with noise or corruption in data, for example To deal with (Gaussian) noise To deal with (sparse) corruption 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Example: Robust face recognition 1/2 Wright, J., Yang, A. Y., Ganesh, A., Sastry, S. S., & Ma, Y. (2009). Robust face recognition via sparse representation. IEEE transactions on pattern analysis and machine intelligence, 31(2), 210-227. 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Example: Robust face recognition 2/2 Able to identify valid/invalid input 2018/11/17 Chap 5. Non-Parametric Supervised Learning

Chap 5. Non-Parametric Supervised Learning Chapter summary Dictionary Toolbox Instance-based learning Memory-based learning k-NN Parzen window Sparse coding 2018/11/17 Chap 5. Non-Parametric Supervised Learning