Introduction and Motivation Approaches for DE: Known model → parametric approach: p(x;θ) (Gaussian, Laplace,…) Unknown model → nonparametric approach Assumes only smoothness Lets the data speak for itself Only for Continuous random variables in this presentation.

Histogram Estimator Simplest definition: Uniform intervals: [x o +mh,x o +(m+1)h) Estimator:

Histogram Estimator Drawbacks Choice of the origin greatly affects the estimate. Produces a Discontinuous estimate. Not very accurate. One reason: Asymmetric influence (unnatural). Let’s solve this! Observation Point Samples

Naïve Estimator Uses intervals centered at the observation points: Is it a density? x w(x) Yes, since it can be written as:

Naïve Estimator No need to choose an origin, only a width h. Better, but still not continuous. Reason: Influence of the points decays abruptly: x w(x) x K(x) This is more natural!

Kernel Estimator Definition: or x K(x) Properties of the Kernel K (typically): - it’s a pdf - smooth (inherited by the estimate) - centered at the origin and decaying fast - even (alternative interpretation) 2h opt h opt h opt /2

Maximum Likelihood (ML) Sample Criterion Model Score Likelihood is the probability of the Sample (S) given a model (f ): Maximizing Likelihood is equivalent to Minimizing Entropy. if the samples are independent. or taking logs: this is the Empirical Entropy

Kernel Estimator: Choosing h Maximum Likelihood (ML): Problem when h → 0 Reason: same sample for training and validation. Solution: have different samples (or split the current sample). Intelligent way to do it: Leave-One-Out Cross-validation

Kernel Estimator: one example