Parameter Estimation 主講人:虞台文
Contents Introduction Maximum-Likelihood Estimation Bayesian Estimation
Parameter Estimation Introduction
Bayesian Rule We want to estimate the parameters of class-conditional densities if its parametric form is known, e.g.,
Methods The Method of Moments Maximum-Likelihood Estimation Not discussed in this course Maximum-Likelihood Estimation Assume parameters are fixed but unknown Bayesian Estimation Assume parameters are random variables Sufficient Statistics
Maximum-Likelihood Estimation Parameter Estimation Maximum-Likelihood Estimation
Samples 1 2 D1 D2 The samples in Dj are drawn independently according to the probability law p(x|j). D3 Assume that p(x|j) has a known parametric form with parameter vector j. 3 e.g., j
Goal D1 D2 D3 1 2 Use Dj to estimate the unknown parameter vector j The estimated version will be denoted by Goal 1 2 D1 D2 D3 Use Dj to estimate the unknown parameter vector j 3
Problem Formulation D Now the problem is: Because each class is consider individually, the subscript used before will be dropped. Now the problem is: D Given a sample set D, whose elements are drawn independently from a population possessing a known parameter form, say p(x|), we want to choose a that will make D to occur most likely.
Criterion of ML MLE By the independence assumption, we have Likelihood function: MLE
Criterion of ML Often, we resort to maximize the log-likelihood function How? MLE
Criterion of ML Example: How?
Differential Approach if Possible Find the extreme values using the method in differential calculus. Let f() be a continuous function, where =(1, 2,…, n)T. Gradient Operator Find the extreme values by solving
Preliminary Let
Preliminary Let (xf )T
The Gaussian Population Two cases: Unknown Unknown and
The Gaussian Population: Unknown
The Gaussian Population: Unknown Set Sample Mean
The Gaussian Population: Unknown and Consider univariate normal case
The Gaussian Population: Unknown and Consider univariate normal case unbiased Set biased
The Gaussian Population: Unknown and For multivariate normal case The MLE of and are: unbiased biased
Unbiasedness Unbiased Estimator Consistent Estimator (Absolutely unbiase) Consistent Estimator (asymptotically unbiased)
MLE for Normal Population Sample Mean Sample Covariance Matrix
Parameter Estimation Bayesian Estimation
Comparison MLE (Maximum-Likelihood Estimation) Bayesian Estimation to find the fixed but unknown parameters of a population. Bayesian Estimation Consider the parameters of a population to be random variables.
Heart of Bayesian Classification Ultimate Goal: Evaluate What can we do if prior probabilities and class-conditional densities are unknown?
Helpful Knowledge Functional form for unknown densities e.g., Normal, exponential, … Ranges for the values of unknown parameters e.g., uniform distributed over a range Training Samples Sampling according to the states of nature.
Posterior Probabilities from Sample
Posterior Probabilities from Sample Each class can be considered independently
Problem Formulation D This the central problem of Bayesian Learning. Let D be a set of samples drawn independently according to the fixed but known distribution p(x). We want to determine D This the central problem of Bayesian Learning.
Parameter Distribution Assume p(x) is unknown but knowing it has a fixed form with parameter vector . is complete known Assume is a random vector, and p() is a known a priori.
Class-Conditional Density Estimation
Class-Conditional Density Estimation The posterior density we want to estimate The form of distribution is assumed known
Class-Conditional Density Estimation If p(|D) has a sharp peak at
Class-Conditional Density Estimation
The Univariate Gaussian: Unknown distribution form is known assume is normal distributed
The Univariate Gaussian: Unknown
The Univariate Gaussian: Unknown Comparison
The Univariate Gaussian: Unknown
The Univariate Gaussian: Unknown
The Univariate Gaussian: p(x|D)
The Univariate Gaussian: p(x|D)
The Univariate Gaussian: p(x|D) =?
The Multivariate Gaussian: Unknown distribution form is known assume is normal distributed
The Multivariate Gaussian: Unknown
The Multivariate Gaussian: Unknown
General Theory 1. the form of class-conditional density is known. 2. knowledge about the parameter distribution is available. samples are randomly drawn according to the unknown probability density p(x). 3.
General Theory 1. the form of class-conditional density is known. 2. knowledge about the parameter distribution is available. samples are randomly drawn according to the unknown probability density p(x). 3.
Incremental Learning Recursive
1. Example 2. 3.
1. Example 2. 3. 4 2 4 6 8 10 p(|Dn) 3 2 1