1. SOME GENERAL PROBLEMS

2. Problem
A certain lion has three possible states of activity each night; they are ‘very active’ (denoted by θ1), ‘moderately active’ (denoted by θ2), and ‘lethargic (lacking energy)’ (denoted by θ3). Also, each night this lion eats people; it eats i people with probability p(i|θ), θ ϵ Θ={θ1, θ2, θ3} . Of course, the probability distribution of the number of people eaten depends on the lion’s activity state θ ϵ Θ. The numeric values are given in the following table.

3. Problem i 1 2 3 4
p(i|θ1)
0.05
0.8
0.1
p(i|θ2)
p(i|θ3)
0.9
0.08
0.02
If we are told X=x0 people were eaten last night, how should we estimate the lion’s activity state (θ1, θ2 or θ3)?

4. Solution
One reasonable method is to estimate θ as that in Θ for which p(x0|θ) is largest. In other words, the θ ϵ Θ that provides the largest probability of observing what we did observe.
: the MLE of θ based on X
(Taken from “Dudewicz and Mishra, 1988, Modern Mathematical Statistics, Wiley”)

5. Problem
Consider the Laplace distribution centered at the origin and with the shape parameter β, which for all x has the p.d.f.
Find MME and MLE of β.

6. Problem
Let X1,…,Xn be independent r.v.s each with lognormal distribution, ln N(,2). Find the MMEs of ,2

7. STATISTICAL INFERENCE PART III
BETTER OR BEST ESTIMATORS, FISHER INFORMATION, CRAMER-RAO LOWER BOUND (CRLB)

8. RECALL: EXPONENTIAL CLASS OF PDFS
If the pdf can be written in the following form
then, the pdf is a member of exponential class of pdfs. (Here, k is the number of parameters)

9. EXPONENTIAL CLASS and CSS
Random Sample from Regular Exponential Class
is a css for .

10. RAO-BLACKWELL THEOREM
Let X1, X2,…,Xn have joint pdf or pmf f(x1,x2,…,xn;) and let S=(S1,S2,…,Sk) be a vector of jss for . If T is an UE of () and (S)=E(TS), then
(S) is an UE of () .
(S) is a fn of S, so it is free of .
Var((S) ) Var(T) for all .
(S) is a better unbiased estimator of () .

11. RAO-BLACKWELL THEOREM
Notes:
(S)=E(TS) is at least as good as T.
For finding the best UE, it is enough to consider UEs that are functions of a ss, because all such estimators are at least as good as the rest of the UEs.

12. Example Hogg & Craig, Exercise 10.10 X1,X2~Exp(θ)
Find joint p.d.f. of ss Y1=X1+X2 for θ and Y2=X2.
Show that Y2 is UE of θ with variance θ².
Find φ(y1)=E(Y2|Y1) and variance of φ(Y1).

13. THE MINIMUM VARIANCE UNBIASED ESTIMATOR
Rao-Blackwell Theorem: If T is an unbiased estimator of , and S is a ss for , then (S)=E(TS) is
an UE of , i.e.,E[(S)]=E[E(TS)]= and
with a smaller variance than Var(T).

14. LEHMANN-SCHEFFE THEOREM
Let Y be a css for . If there is a function Y which is an UE of , then the function is the unique Minimum Variance Unbiased Estimator (UMVUE) of .
Y css for .
T(y)=fn(y) and E[T(Y)]=.
T(Y) is the UMVUE of .
So, it is the best unbiased estimator of .

15. THE MINIMUM VARIANCE UNBIASED ESTIMATOR
Let Y be a css for . Since Y is complete, there could be only a unique function of Y which is an UE of .
Let U1(Y) and U2(Y) be two function of Y. Since they are UE’s, E(U1(Y)U2(Y))=0 imply W(Y)=U1(Y)U2(Y)=0 for all possible values of Y. Therefore, U1(Y)=U2(Y) for all Y.

16. Example Let X1,X2,…,Xn ~Poi(μ). Find UMVUE of μ. Solution steps:
Show that is css for μ.
Find a statistics (such as S*) that is UE of μ and a function of S.
Then, S* is UMVUE of μ by Lehmann-Scheffe Thm.

17. Note
The estimator found by Rao-Blackwell Thm may not be unique. But, the estimator found by Lehmann-Scheffe Thm is unique.

18. RECALL: EXPONENTIAL CLASS OF PDFS
If the pdf can be written in the following form
then, the pdf is a member of exponential class of pdfs. (Here, k is the number of parameters)

19. EXPONENTIAL CLASS and CSS
Random Sample from Regular Exponential Class
is a css for .
If Y is an UE of , Y is the UMVUE of .

20. EXAMPLES Let X1,X2,…~Bin(1,p), i.e., Ber(p).
This family is a member of exponential family of distributions.
is a CSS for p.
is UE of p and a function of CSS.
is UMVUE of p.

21. EXAMPLES
X~N(,2) where both  and 2 is unknown. Find a css for  and 2 .

22. FISHER INFORMATION AND INFORMATION CRITERIA
X, f(x;), , xA (not depend on ).
Definitions and notations:

23. FISHER INFORMATION AND INFORMATION CRITERIA
The Fisher Information in a random variable X:
The Fisher Information in the random sample:
Let’s prove the equalities above.

24. FISHER INFORMATION AND INFORMATION CRITERIA

25. FISHER INFORMATION AND INFORMATION CRITERIA

26. FISHER INFORMATION AND INFORMATION CRITERIA
The Fisher Information in a random variable X:
The Fisher Information in the random sample:
Proof of the last equality is available on Casella & Berger (1990), pg

27. CRAMER-RAO LOWER BOUND (CRLB)
Let X1,X2,…,Xn be sample random variables.
Range of X does not depend on .
Y=U(X1,X2,…,Xn): a statistic; does’nt contain .
Let E(Y)=m().
Let prove this!

28. CRAMER-RAO LOWER BOUND (CRLB)
-1Corr(Y,Z)1
0 Corr(Y,Z)21  
Take Z=′(x1,x2,…,xn;)
Then, E(Z)=0 and V(Z)=In() (from previous slides).

29. CRAMER-RAO LOWER BOUND (CRLB)
Cov(Y,Z)=E(YZ)-E(Y)E(Z)=E(YZ)

30. CRAMER-RAO LOWER BOUND (CRLB)
E(Y.Z)=mʹ(), Cov(Y,Z)=mʹ(), V(Z)=In()
The Cramer-Rao Inequality
(Information Inequality)

31. CRAMER-RAO LOWER BOUND (CRLB)
CRLB is the lower bound for the variance of an unbiased estimator of m().
When V(Y)=CRLB, Y is the MVUE of m().
For a r.s., remember that In()=n I(), so,

32. ASYMPTOTIC DISTRIBUTION OF MLEs
: MLE of 
X1,X2,…,Xn is a random sample.

33. EFFICIENT ESTIMATOR T is an efficient estimator (EE) of  if
T is UE of , and,
Var(T)=CRLB
T is an efficient estimator (EE) of its expectation, m(), if its variance reaches the CRLB.
An EE of m() may not exist.
The EE of m(), if exists, is unique.
The EE of m() is the unique MVUE of m().

34. ASYMPTOTIC EFFICIENT ESTIMATOR
Y is an asymptotic EE of m() if

35. EXAMPLES A r.s. of size n from X~Poi(θ). Find CRLB for any UE of θ.
Find UMVUE of θ.
Find an EE for θ.
Find CRLB for any UE of exp{-2θ}. Assume n=1, and show that is UMVUE of exp{-2θ}. Is this a reasonable estimator?

36. EXAMPLE
A r.s. of size n from X~Exp(). Find UMVUE of , if exists.

37. Summary
We covered 3 methods for finding good estimators (possibly UMVUE):
Rao-Blackwell Theorem (Use a ss T, an UE U, and create a new statistic by E(U|T))
Lehmann-Scheffe Theorem (Use a css T which is also UE)
Cramer-Rao Lower Bound (Find an UE with variance=CRLB)

38. Problems
Let be a random sample from gamma distribution, Xi~Gamma(2,θ). The p.d.f. of X1 is given by:
a) Find a complete and sufficient statistic for θ.
b) Find a minimal sufficient statistic for θ.
c) Find CRLB for the variance of an unbiased estimator of θ.
d) Find a UMVUE of θ.

39. Problems Suppose X1,…,Xn are independent with density for θ>0
a) Find a complete sufficient statistic.
b) Find the CRLB for the variance of unbiased estimators of 1/θ.
c) Find the UMVUE of 1/θ if there is one.