1. Point Estimation
Notes of STAT 6205 by Dr. Fan

2. Overview Section 6.1 Point estimation Maximum likelihood estimation
Methods of moments
Sufficient statistics
Definition
Exponential family
Mean square error (how to choose an estimator)
6205-Ch6

3. Big Picture Goal: To study the unknown distribution of a population
Method: Get a representative/random sample and use the information obtained in the sample to make statistical inference on the unknown features of the distribution
Statistical Inference has two parts:
Estimation (of parameters)
Hypothesis testing
Estimation:
Point estimation: use a single value to estimate a parameter
Interval estimation: find an interval covering the unknown parameter
6205-Ch6

4. Point Estimator
Biased/unbiased: an estimator is called unbiased if its mean is equal to the parameter of estimate; otherwise, it is biased
Example: X_bar is unbiased for estimating mu
6205-Ch6

5. Maximum Likelihood Estimation
Given a random sample X1, X2, …, Xn from a distribution f(x; q) where q is a (unknown) value in the parameter space W.
Likelihood function vs. joint pdf
Maximum Likelihood Estimator (m.l.e.) of q, denoted as is the value q which maximizes the likelihood function, given the sample X1, X2, …, Xn.
6205-Ch6

6. Examples/Exercises
Problem 1: To estimate p, the true probability of heads up for a given coin.
Problem 2: Let X1, X2, …, Xn be a random sample from a Exp(mu) distribution. Find the m.l.e. of mu.
Problem 3: Let X1, X2, …, Xn be a random sample from a Weibull(a=3,b) distribution. Find the m.l.e. of b.
Problem 4: Let X1, X2, …, Xn be a random sample from a N(m,s^2) distribution. Find the m.l.e. of m and s.
Problem 5: Let X1, X2, …, Xn be a random sample from a Weibull(a,b) distribution. Find the m.l.e. of a and b.
6205-Ch6

7. Method of Moments
Idea: Set population moments = sample moments and solve for parameters
Formula: When the parameter q is r-dimensional, solve the following equations for q:
6205-Ch6

8. Examples/Exercises Given a random sample from a population
Problem 1: Find the m.m.e. of m for a Exp(m) population.
Exercise 1: Find the m.m.e. of m and s for a N(m,s^2) population.
6205-Ch6

9. Sufficient Statistics
Idea: The “sufficient” statistic contains all information about the unknown parameter; no other statistic can provide additional information as to the unknown parameter.
If for any event A, P[A|Y=y] does not depend on the unknown parameter, then the statistic Y is called “sufficient” (for the unknown parameter).
Any one-to-one mapping of a sufficient statistic Y is also sufficient.
Sufficient statistics do not need to be estimators of the parameter.
6205-Ch6

10. Sufficient Statistics
6205-Ch6

11. Examples/Exercises Let X1, X2, …, Xn be a random sample from f(x)
Problem: Let f be Poisson(a). Prove that
X-bar is sufficient for the parameter a
The m.l.e. of a is a function of the sufficient statistic
Exercise: Let f be Bin(n, p). Prove that X-bar is sufficient for p (n is known). Hint: find a sufficient statistic Y for p and then show that X-bar is a function of Y
6205-Ch6

12. Exponential Family
6205-Ch6

13. Examples/Exercises
Example 1: Find a sufficient statistic for p for Bin(n, p) Example 2: Find a sufficient statistic for a for Poisson(a) Exercise: Find a sufficient statistic for m for Exp(m)
6205-Ch6

14. Joint Sufficient Statistics
Example: Prove that X-bar and S^2 are joint sufficient statistics for m and s of N(m, s^2)
6205-Ch6

15. Application of Sufficience
6205-Ch6

16. Example Consider a Weibull distribution with parameter(a=2, b)
Find a sufficient statistic for b
Find an unbiased estimator which is a function of the sufficient statistic found in 1)
6205-Ch6

17. Good Estimator?
Criterion: mean square error
6205-Ch6

18. Example Which of the following two estimator of variance is better?