1. Nonparametric Density Estimation Riu Baring CIS 8526 Machine Learning Temple University Fall 2007 Christopher M. Bishop, Pattern Recognition and Machine Learning, Chapter 2.5 Some slides from http://courses.cs.tamu.edu/rgutier/cpsc689_f07/

2. Overview Density Estimation  Given: a finite set x 1,…,x N  Task: to model the probability distribution p(x) Parametric Distribution  Governed by adaptive parameters Mean and variance – Gaussian Distribution  Need procedure to determine suitable values for the parameters  Discrete rv – binomial and multinomial distributions  Continuous rv – Gaussian distributions

3. Nonparametric Method Attempt to estimate the density directly from the data without making any parametric assumptions about the underlying distribution. Nonparametric Density Estimation

4. Histogram Divide the sample space into a number of bins and approximate the density at the center of each bin by the fraction of points in the training data that fall into the corresponding bin.

5. Histogram Parameter: bin width.

6. Histogram - Drawbacks The discontinuities of the estimate are not due to the underlying density, they are only an artifact of the chosen bin locations  These discontinuities make it very difficult (to the naïve analyst) to grasp the structure of the data A much more serious problem is the curse of dimensionality, since the number of bins grows exponentially with the number of dimensions  In high dimensions we would require a very large number of examples or else most of the bins would be empty

7. Nonparametric DE

10. Kernel Density Estimator

12. k-nearest-neighbors To estimate p(x):  Consider small sphere centered on the point x  Allow the radius of the sphere to grow until it contains k data points

13. k-nearest-neighbors Data set comprising N k points in class C k, so that Suppose the sphere has volume, V, and contains k k points from class C k Density Estimate Unconditional density Class Prior Posterior probability of class membership.

14. k-nearest-neighbors To classify new point x  Identify K nearest neighbors from training data  Assign to the class having the largest number of representatives Parameter, K.

15. My thoughts KDE and KNN require the entire training data set to be stored  Leads to expensive computation Tweak “parameters”  KDE: bandwidth, h  KNN: K