1. Part 2b Parameter Estimation CSE717, FALL 2008 CUBS, Univ at Buffalo

2. Parametric Distribution 2-Class Problem Given class labels {c 1,c 2 } and random variable X, the posterior probability of class C is Parametric Representation of Distribution The conditional p.d.f is usually represented by a math expression of x with various parameters θ 1, θ 2, …

3. Parametric Representation of Distributions The conditional p.d.f is usually represented by a math expression of x with various parameters θ 1, θ 2, … Parameter Estimation Estimate parameters from samples of X Example: Normal Distribution p.d.f Parameters:,

4. Maximum Likelihood Estimation Given X, p.d.f. p X (x;θ), n values x 1,…,x n obtained by independent samplings X 1,…,X n : the Maximum Likelihood Estimation of θ is given by

5. Maximum Likelihood Estimation (Cont.) By independence assumption

6. Normal Distribution with Unknown and Let

7. Normal Distribution with Unknown and (Cont.) Let and

8. Bias of Estimator An estimator of is unbiased if ; is biased otherwise. is an unbiased estimator of Proof

9. Bias of Estimator (Cont.) is a biased estimator of Proof

10. Bias of Estimator (Cont.) is an unbiased estimator of Proof

11. Bias of Estimator (Cont.) is an asymptotically unbiased estimator of if is an asymptotically unbiased estimator of Proof

12. Variance of Estimator The variance of of

13. Mean Square Error Mean Square Error of Estimator Relation between MSE, Bias and Variance

14. Bias/Variance Dilemma Bias: the quality of the estimator Variance: the consistency of the estimator at different groups of selected samples MSE: overall quality of the estimator Low bias sometimes leads to high variance and high MSE Overfitting/overtraining problem

15. Biased vs. Unbiased Estimators  : biased; : unbiased; Unbiased estimators are NOT always desirable