1. STATISTICAL INFERENCE PART I POINT ESTIMATION

2. STATISTICAL INFERENCE
Determining certain unknown properties of a probability distribution on the basis of a sample (usually, a r.s.) obtained from that distribution 
Point Estimation:
Interval Estimation: ( ) 
Hypothesis Testing:

3. STATISTICAL INFERENCE
Parameter Space ( or ): The set of all possible values of an unknown parameter, ; .
A pdf with unknown parameter: f(x; ), .
Estimation: Where in ,  is likely to be?
{ f(x; ),  }
The family of pdfs

4. STATISTICAL INFERENCE
Statistic: A function of rvs (usually a sample rvs in an estimation) which does not contain any unknown parameters.
Estimator of an unknown parameter : A statistic used for estimating .
An observed value

5. POINT ESTIMATION θ: a parameter of interest; unknown
Goal: Find good estimator(s) for θ or its function g(θ).

6. Method of Moments Estimation, Maximum Likelihood Estimation
METHODS OF ESTIMATION
Method of Moments Estimation, Maximum Likelihood Estimation

7. METHOD OF MOMENTS ESTIMATION (MME)
Let X1, X2,…,Xn be a r.s. from a population with pmf or pdf f(x;1, 2,…, k). The MMEs are found by equating the first k population moments to corresponding sample moments and solving the resulting system of equations.
Population Moments
Sample Moments

8. METHOD OF MOMENTS ESTIMATION (MME)
so on…
Continue this until there are enough equations to solve for the unknown parameters.

9. EXAMPLES Let X~Exp(). For a r.s of size n, find the MME of .
For the following sample (assuming it is from Exp()), find the estimate of :
11.37, 3, 0.15, 4.27, 2.56, 0.59.

10. EXAMPLES
Let X~N(μ,σ²). For a r.s of size n, find the MMEs of μ and σ².
For the following sample (assuming it is from N(μ,σ²)), find the estimates of μ and σ²:
4.93, 6.82, 3.12, 7.57, 3.04, 4.98, 4.62, 4.84, 2.95, 4.22

11. DRAWBACKS OF MMES
Although sometimes parameters are positive valued, MMEs can be negative.
If moments does not exist, we cannot find MMEs.

12. LIKELIHOOD FUNCTION
The likelihood function plays an important role in statistical inference.
A likelihood function is a function of the parameters of a statistical model, given specific observed data. 
Let X1, X2, ..., Xn have a joint density function

13. LIKELIHOOD FUNCTION
Given X1 = x1, X2 = x2, ..., Xn = xn is observed, the function of θ defined by:
is the likelihood function.
• The likelihood function is not a probability density function.
• It measures the support provided by the data for each possible value of the parameter. If we compare the likelihood function at two parameter points and find that L(θ1|x) > L(θ2|x) then the sample we actually observed is more likely to have occurred if θ = θ1 than if θ = θ2. This can be interpreted as θ1 is a more plausible value for θ than θ2.

14. American or Canadian M&M’s?
(Discrete parameter): M&M’s sold in the United States have 50% red candies compared to 30% in those sold in Canada. In an experimental study, a sample of 5 candies were drawn from an unlabelled bag and 2 red candies were observed. Is it more plausible that this bag was from the United States or from Canada? The likelihood function is: L(p|x) ∝ p2 (1 − p)3 , p=0.3 or 0.5. L(0.3|x) = < = L(0.5|x), suggesting that it is more plausible that the bag used in the experiment was from the United States

15. LIKELIHOOD PRINCIPLE
If x and y are two sample points such that L(θ|x) ∝ L(θ|y) ∀ θ then the conclusions drawn from x and y should be identical. Thus the likelihood principle implies that likelihood function can be used to compare the plausibility of various parameter values.
For example, if L(θ2|x) = 2L(θ1|x) and L(θ|x) ∝ L(θ|y) ∀ θ, then L(θ2|y) = 2L(θ1|y). Therefore, whether we observed x or y we would come to the conclusion that θ2 is twice as plausible as θ1.

16. MAXIMUM LIKELIHOOD ESTIMATION (MLE)
Let X1, X2,…,Xn be a r.s. from a population with pmf or pdf f(x;1, 2,…, k), the likelihood function is defined by

17. MAXIMUM LIKELIHOOD ESTIMATION (MLE)
For each sample point (x1,…,xn), let
be a parameter value at which
L(1,…, k| x1,…,xn) attains its maximum as a function of (1,…, k), with (x1,…,xn) held fixed. A maximum likelihood estimator (MLE) of parameters (1,…, k) based on a sample (X1,…,Xn) is
The MLE is the parameter point for which the observed sample is most likely.

18. EXAMPLES
Let X~Bin(n,p), where both n and p are unknown. One observation on X is available, and it is known that n is either 2 or 3 and p=1/2 or 1/3. Our objective is to estimate the pair (n,p). x
(2,1/2)
(2,1/3)
(3,1/2)
(3,1/3)
Max. Prob.
1/4
4/9
1/8
8/27 1
1/2
3/8
12/27 2
1/9
6/27 3
1/27
P(X=0|n=2,p=1/2)=1/4 …

19. MAXIMUM LIKELIHOOD ESTIMATION (MLE)
It is usually convenient to work with the logarithm of the likelihood function.
Suppose that f(x;1, 2,…, k) is a positive, differentiable function of 1, 2,…, k. If a supremum exists, it must satisfy the likelihood equations
MLE occurring at boundary of  cannot be obtained by differentiation. So, use inspection.

20. MLE
Moreover, you need to check that you are in fact maximizing the log-likelihood (or likelihood) by checking that the second derivative is negative.

21. EXAMPLES
1. X~Exp(), >0. For a r.s of size n, find the MLE of .

22. EXAMPLES
2. X~N(,2). For a r.s. of size n, find the MLEs of  and 2.

23. EXAMPLES
3. X~Uniform(0,), >0. For a r.s of size n, find the MLE of .

24. INVARIANCE PROPERTY OF THE MLE
If is the MLE of , then for any function (), the MLE of () is
Example: X~N(,2). For a r.s. of size n, the MLE of  is By the invariance property of MLE, the MLE of 2 is

25. ADVANTAGES OF MLE
Often yields good estimates, especially for large sample size.
Invariance property of MLEs
Asymptotic distribution of MLE is Normal.
Most widely used estimation technique.
Usually they are consistent estimators. [will define consistency later]

26. DISADVANTAGES OF MLE
Requires that the pdf or pmf is known except the value of parameters.
MLE may not exist or may not be unique.
MLE may not be obtained explicitly (numerical or search methods may be required.). It is sensitive to the choice of starting values when using numerical estimation. 
MLEs can be heavily biased for small samples.