1. Outline
Parameter estimation – continued
Non-parametric methods

2. Maximum-Likelihood Estimation
Assumptions
We separate a collection of samples according to class
D1, D2, ....., Dc
Samples in Dj are drawn independently according to the probability p(x|wj)
We assume that p(x|wj) has a known parametric form and is uniquely determined by the value of a parameter vector j
To simplify further, we assume that samples in Di give no information about j if i  j
11/16/2018
Visual Perception Modeling

3. Maximum-Likelihood Estimation – cont.
Suppose that D contains n samples
x1, ....., xn
By assumption that samples were drawn independently, we have
The maximum-likelihood estimate of  is the value of * that maximizes p(D| )
11/16/2018
Visual Perception Modeling

4. Visual Perception Modeling
Bayesian Estimation
Assumptions
The form of the density p(x|q) is assumed to be known, but the value of the parameter vector q is not known exactly
Our initial knowledge about q is assumed to be contained in a known prior density p(q)
The rest of our knowledge about q is contained in a set D of n samples x1, ....., xn drawn independently according to the unknown probability density p(x)
11/16/2018
Visual Perception Modeling

5. Bayesian Estimation – cont.
General theory
The basic problem is to compute the posterior density p(q|D)
By Bayes formula we have
By the independence assumption
11/16/2018
Visual Perception Modeling

6. Bayesian Estimation – cont.
Gaussian case
The univariate case p(m|D)
11/16/2018
Visual Perception Modeling

7. Bayesian Estimation – cont.
Gaussian case – continued
The univariate case p(x|D)
The multivariate case
11/16/2018
Visual Perception Modeling

8. Non-parametric Methods
In maximum-likelihood and Bayesian estimation
The forms of the probability densities are assumed to be known
However, the assumed forms rarely fit the densities in practice
In particular, all of the classical parametric densities are uni-modal
11/16/2018
Visual Perception Modeling

9. Visual Perception Modeling
A Multimodal Density
11/16/2018
Visual Perception Modeling

10. Visual Perception Modeling
Solutions
More complicated parametric models
Mixture of Gaussians
More general, some basis functions to describe a probability density
Learning is intrinsically more difficult when we have more parameters
Non-parametric methods
11/16/2018
Visual Perception Modeling

11. Non-parametric Methods
Most of the non-parametric density estimation methods are based on the following fact
The probability P that a vector x will fall in a region R is given by
11/16/2018
Visual Perception Modeling

12. Non-parametric Methods – cont.
For n smaples x1, ....., xn that are drawn independently according to p(x), the probability that k of n will be in R is given by
V is the volume of R
11/16/2018
Visual Perception Modeling

13. Non-parametric Methods – cont.
11/16/2018
Visual Perception Modeling

14. Non-parametric Methods – cont.
Problems to be addressed
If we fix the volume V and have more samples, the ratio k/n will converge as desired
Averaged version of p(x)
How to estimate p(x)?
Let V approach zero?
11/16/2018
Visual Perception Modeling

15. Visual Perception Modeling
Parzen Windows
Parzen windows
We use a window function for interpolation, each sample contributing to the estimate in accordance with its distance from x
Here hn is a parameter
11/16/2018
Visual Perception Modeling

16. Visual Perception Modeling
Parzen Windows - cont.
Choice of hn
Too large, the spatial resolution is low
Too small, the estimate will have a large variance
11/16/2018
Visual Perception Modeling

17. Visual Perception Modeling
Parzen Windows - cont.
Properties
Convergence of mean
As n approaches infinity, the estimate will also approach p(x) if p(x) is continuous
Smaller Vn is better
Convergence of variance
A smaller variance needs a larger Vn
11/16/2018
Visual Perception Modeling

18. Visual Perception Modeling
Parzen Windows - cont.
11/16/2018
Visual Perception Modeling

19. Visual Perception Modeling
Parzen Windows - cont.
11/16/2018
Visual Perception Modeling

20. Visual Perception Modeling
Parzen Windows - cont.
11/16/2018
Visual Perception Modeling

21. Visual Perception Modeling
Parzen Windows - cont.
11/16/2018
Visual Perception Modeling

22. Kn-Nearest-Neighbor Estimation
Let the cell volume be a function of the training data
To estimate p(x) from n samples, we can center a cell about x and let it grow until it captures kn samples
11/16/2018
Visual Perception Modeling

23. Kn-Nearest-Neighbor Estimation – cont.
11/16/2018
Visual Perception Modeling

24. Kn-Nearest-Neighbor Estimation – cont.
11/16/2018
Visual Perception Modeling

25. Kn-Nearest-Neighbor Estimation – cont.
11/16/2018
Visual Perception Modeling

26. The Nearest-Neighbor Rule
Let Dn={x1, ...., xn} denote a set of n labeled prototypes
Suppose that x' be the prototype nearest to a test point x
We classify x to the class associated with x'
11/16/2018
Visual Perception Modeling

27. The Nearest-Neighbor Rule – cont.
11/16/2018
Visual Perception Modeling