1. Maximum-Likelihood estimation Consider as usual a random sample x = x 1, …, x n from a distribution with p.d.f. f (x;  ) (and c.d.f. F(x;  ) ) The maximum likelihood point estimator of  is the value of  that maximizes L(  ; x ) or equivalently maximizes l(  ; x ) Useful notation: With a k-dimensional parameter:

2. Complete sample case: If all sample values are explicitly known, then Censored data case: If some ( say n c ) of the sample values are censored, e.g. x i k 2, then where

3. When the sample comes from a continuous distribution the censored data case can be written In the case the distribution is discrete the use of F is also possible: If k 1 and k 2 are values that can be attained by the random variables then we may write where

4. Example:

5. Solution must be numerically found

6. For the exponential family of distributions: Use the canonical form (natural parameterization): Let Then the maximum likelihood estimators (MLEs) of  1, …,  k are found by solving the system of equations

7. Example:

9. Computational aspects When the MLEs can be found by evaluating numerical routines for solving the generic equation g(  ) = 0 can be used. Newton-Raphson method Fisher’s method of scoring (makes use of the fact that under regularity conditions: ) This is the multidimensional analogue of Lemma 2.1 ( see page 17)

10. When the MLEs cannot be found the above way other numerical routines must be used: Simplex method EM-algorithm For description of the numerical routines see textbook. Maximum Likelihood estimation comes into natural use not for handling the standard case, i.e. a complete random sample from a distribution within the exponential family, but for finding estimators in more non-standard and complex situations.

11. Example:

12. Properties of MLEs Invariance: Consistency: Under some weak regularity conditions all MLEs are consistent Efficiency: Under the usual regularity conditions: (Asymptotically efficient and normally distributed)

13. Sufficiency: Example:

15. Invariance property 

18. i.e. the two MLEs are asymptotically uncorrelated (and by the normal distribution independent)

19. Modifications and extensions Ancillarity and conditional sufficiency:

20. Profile likelihood: This concept has its main use in cases where  1 contains the parameters of “interest” and  2 contains nuisance parameters. The same ML point estimator for  1 is obtained by maximizing the profile likelihood as by maximizing the full likelihood function

21. Marginal and conditional likelihood: Again, these concepts have their main use in cases where  1 contains the parameters of “interest” and  2 contains nuisance parameters.

22. Penalized likelihood: MLEs can be derived subjected to some criteria od smoothness. In particulare this is applicable when the parameter is no longer a single value (one- or multidimensional), but a function such as an unknown density function or a regression curve. The penalized log-likelihood function is written

23. Method of moments estimation (MM )

24. The method of moments point estimator of  = (  1, …,  k ) is obtained by solving for  1, …,  k the systems of equations

25. Example:

27. Method of Least Squares (LS) First principles: Assume a sample x where the random variable X i can be written The least-squares estimator of  is the value of  that minimizes i.e.

28. A more general approach: Assume the sample can be written (x, z ) where x i represents the random variable of interest (endogenous variable) and z i represent either an auxiliary random variable (exogenous) or a given constant for sample point i The least squares estimator of  is then

29. Special cases: The ordinary linear regression model: The heteroscedastic regression model:

30. The first-order auto-regressive model: The conditional least-squares estimator of  (given  ) is