1. Stat Lab 6
Parameter Estimation

2. ? The form of f is known, but θ is unknown. Distribution: f(x; θ) x12

3. ? The form of f is known, but θ is unknown. Distribution: f(x; θ) θ

4. How to estimate θ ? Methods of finding estimators
Method of Moment estimation
Maximum Likelihood estimation

5. Method of Moment Estimation
θ = g ( E(X), E(X2), E(X3), …)
g must be known and the required moments must exist.
E(Xk)
g may not be unique.
= g ( m, m2, m3, …)

6. Maximum Likelihood Estimation
Likelihood function is a joint density function of observing the data we have.
L(θ) = f(x1,…,xn; θ) =
(if Xi are indept and fi is the density of xi)
(same distribution, say f1 ) =

7. Maximum Likelihood Estimation=
L(θ)
Assume that x1…xn is a random sample of size n from the distribution with density
In most situations, it is easy to find the MLE by maximizing the log likelihood function, i.e. ln L(θ)
Maximum likelihood estimator is an estimator maximizing the likelihood function L(θ)

8. Examples: 1) Assume that X1,…Xn is a random sample from Ga(α,β)
Density of Gamma distribution :
for X>0 and both α and β > 0

9. E(X) = α/β β= α0/E(X) β= α0/m ^
1) Assume that X1,…Xn is a random sample from Ga(α,β)
(a) Method of moment estimator :
Case 1) If β is unknown but α is known, say α0
E(X) = α/β
β= α0/E(X)
The method of moment estimator of β given α=α0 is
β= α0/m ^

10. E(X) = α/β α= E(X) β0 ^ α= m β0
1) Assume that X1,…Xn is a random sample from Ga(α,β)
(a) Method of moment estimator :
E(X) = α/β
Case 2) If α is unknown but β is known, say β0
α= E(X) β0
The method of moment estimator of α given β=β0 is
α= m β0 ^

11. β=E(X) / [E(X2) – {E(X)}2] m / [m2 - m2]
1) Assume that X1,…Xn is a random sample from Ga(α,β)
(a) Method of moment estimator :
E(X) = α/β
= K0 (known)
Case 3) Both α and β are unknown
E(X2)=α(α-1)/β2
After some simple algebra, K0 K0
β=E(X) / [E(X2) – {E(X)}2]
m / [m2 - m2] K0 K0
α={E(X)}2 / [E(X2) – {E(X)}2] K0
m2 / [m2 - m2] K0 K0 K0

12. β= α0/m Non-uniqueness ^
1) Assume that X1,…Xn is a random sample from Ga(α,β)
Non-uniqueness
(a) Method of moment estimator :
Case 1) If β is unknown but α is known, say α0
The method of moment estimator of β given α=α0 is
β= α0/m ^
by E(X) = α/β
The method of moment estimator of β given α=α0 is

13. βn= α0/m ^ 1) Assume that X1,…Xn is a random sample from Ga(α,β)
(b) Maximum Likelihood estimator :
Case 1) If β is unknown but α is known, say α0
βn= α0/m ^
Check if it is inside the parameter space of β i.e. is it >0 ?

14. 1) Assume that X1,…Xn is a random sample from Ga(α,β)
(b) Maximum Likelihood estimator :
Case 2) If α is unknown but β is known, say β0
There is no explicit form of the solution, so we have to find it by some numerical methods.
optim(par, fn, …)
par: Initial values for the parameters to be optimized over.
fn: A function to be minimized.

15. √ 2) Assume that X1,…Xn is a random sample from N(μ,1)
(a) Method of moment estimator of μ:
(b) Maximum Likelihood estimator of μ:
What is the parameter space of μ?
If μ can be any real number, then √
However, if μ is restricted to be non-negative, then is still the MLE of μ?

16. 2) Assume that X1,…Xn is a random sample from N(μ,1)
(b) Maximum Likelihood estimator of μ:
However, if μ is restricted to be non-negative, then is still the MLE of μ?
μ≥0
Log-likelihood function:
>0 when
<0 when

17. μ≥0 If >0 when <0 when
2) Assume that X1,…Xn is a random sample from N(μ,1)
μ≥0
(b) Maximum Likelihood estimator of μ:
>0 when
<0 when μ If μ

18. μ≥0 If If 2) Assume that X1,…Xn is a random sample from N(μ,1)
(b) Maximum Likelihood estimator of μ: If If

19. Exercises for students:
1) Assume that X1,…Xn is a random sample from Bin(1,p)
a) Find the method of moment estimator of p.
b) Find the MLE of p when (i) 0< p < 1 and (ii) p ≥ ½.
2) Assume that X1,…Xn is a random sample from the uniform distribution over [0,θ], where θ is positive and finite.
a) Find the method of moment estimator of θ.
b) Find the MLE of θ.
c) What is the MLE of θ if [0,θ] is changed to be [θ-1/2, θ+1/2], where θ is any finite real number?