Point Estimation Notes of STAT 6205 by Dr. Fan

Overview Section 6.1 Point estimation Maximum likelihood estimation Methods of moments Sufficient statistics Definition Exponential family Mean square error (how to choose an estimator) 6205-Ch6

Big Picture Goal: To study the unknown distribution of a population Method: Get a representative/random sample and use the information obtained in the sample to make statistical inference on the unknown features of the distribution Statistical Inference has two parts: Estimation (of parameters) Hypothesis testing Estimation: Point estimation: use a single value to estimate a parameter Interval estimation: find an interval covering the unknown parameter 6205-Ch6

Point Estimator Biased/unbiased: an estimator is called unbiased if its mean is equal to the parameter of estimate; otherwise, it is biased Example: X_bar is unbiased for estimating mu 6205-Ch6

Maximum Likelihood Estimation Given a random sample X1, X2, …, Xn from a distribution f(x; q) where q is a (unknown) value in the parameter space W. Likelihood function vs. joint pdf Maximum Likelihood Estimator (m.l.e.) of q, denoted as is the value q which maximizes the likelihood function, given the sample X1, X2, …, Xn. 6205-Ch6

Examples/Exercises Problem 1: To estimate p, the true probability of heads up for a given coin. Problem 2: Let X1, X2, …, Xn be a random sample from a Exp(mu) distribution. Find the m.l.e. of mu. Problem 3: Let X1, X2, …, Xn be a random sample from a Weibull(a=3,b) distribution. Find the m.l.e. of b. Problem 4: Let X1, X2, …, Xn be a random sample from a N(m,s^2) distribution. Find the m.l.e. of m and s. Problem 5: Let X1, X2, …, Xn be a random sample from a Weibull(a,b) distribution. Find the m.l.e. of a and b. 6205-Ch6

Method of Moments Idea: Set population moments = sample moments and solve for parameters Formula: When the parameter q is r-dimensional, solve the following equations for q: 6205-Ch6

Examples/Exercises Given a random sample from a population Problem 1: Find the m.m.e. of m for a Exp(m) population. Exercise 1: Find the m.m.e. of m and s for a N(m,s^2) population. 6205-Ch6

Sufficient Statistics Idea: The “sufficient” statistic contains all information about the unknown parameter; no other statistic can provide additional information as to the unknown parameter. If for any event A, P[A|Y=y] does not depend on the unknown parameter, then the statistic Y is called “sufficient” (for the unknown parameter). Any one-to-one mapping of a sufficient statistic Y is also sufficient. Sufficient statistics do not need to be estimators of the parameter. 6205-Ch6

Sufficient Statistics 6205-Ch6

Examples/Exercises Let X1, X2, …, Xn be a random sample from f(x) Problem: Let f be Poisson(a). Prove that X-bar is sufficient for the parameter a The m.l.e. of a is a function of the sufficient statistic Exercise: Let f be Bin(n, p). Prove that X-bar is sufficient for p (n is known). Hint: find a sufficient statistic Y for p and then show that X-bar is a function of Y 6205-Ch6

Exponential Family 6205-Ch6

Examples/Exercises Example 1: Find a sufficient statistic for p for Bin(n, p) Example 2: Find a sufficient statistic for a for Poisson(a) Exercise: Find a sufficient statistic for m for Exp(m) 6205-Ch6

Joint Sufficient Statistics Example: Prove that X-bar and S^2 are joint sufficient statistics for m and s of N(m, s^2) 6205-Ch6

Application of Sufficience 6205-Ch6

Example Consider a Weibull distribution with parameter(a=2, b) Find a sufficient statistic for b Find an unbiased estimator which is a function of the sufficient statistic found in 1) 6205-Ch6

Good Estimator? Criterion: mean square error 6205-Ch6

Example Which of the following two estimator of variance is better?