Part 2b Parameter Estimation CSE717, FALL 2008 CUBS, Univ at Buffalo

Parametric Distribution 2-Class Problem Given class labels {c 1,c 2 } and random variable X, the posterior probability of class C is Parametric Representation of Distribution The conditional p.d.f is usually represented by a math expression of x with various parameters θ 1, θ 2, …

Parametric Representation of Distributions The conditional p.d.f is usually represented by a math expression of x with various parameters θ 1, θ 2, … Parameter Estimation Estimate parameters from samples of X Example: Normal Distribution p.d.f Parameters:,

Maximum Likelihood Estimation Given X, p.d.f. p X (x;θ), n values x 1,…,x n obtained by independent samplings X 1,…,X n : the Maximum Likelihood Estimation of θ is given by

Maximum Likelihood Estimation (Cont.) By independence assumption

Normal Distribution with Unknown and Let

Normal Distribution with Unknown and (Cont.) Let and

Bias of Estimator An estimator of is unbiased if ; is biased otherwise. is an unbiased estimator of Proof

Bias of Estimator (Cont.) is a biased estimator of Proof

Bias of Estimator (Cont.) is an unbiased estimator of Proof

Bias of Estimator (Cont.) is an asymptotically unbiased estimator of if is an asymptotically unbiased estimator of Proof

Variance of Estimator The variance of of

Mean Square Error Mean Square Error of Estimator Relation between MSE, Bias and Variance

Bias/Variance Dilemma Bias: the quality of the estimator Variance: the consistency of the estimator at different groups of selected samples MSE: overall quality of the estimator Low bias sometimes leads to high variance and high MSE Overfitting/overtraining problem

Biased vs. Unbiased Estimators  : biased; : unbiased; Unbiased estimators are NOT always desirable