1. Prof. Dr. S. K. Bhattacharjee Department of Statistics University of Rajshahi

2. Statistical Inference Statistical inference is the process of making judgment about an unknown population based on sample. An important aspect of statistical inference is using estimates to approximate the value of an unknown population parameter. Another type of inference involve choosing between two opposing views or statements about the population; this process is called hypothesis testing.

3. Statistical Estimation An estimator is a statistical parameter that provides an estimation of a population parameter. Point Estimation Interval Estimation

4. Point Estimation A point estimator is a single numerical estimate of a population parameter. The sample mean, is a point estimator of the population mean, μ. The sample proportion, p is a point estimator of the population proportion, π.,.

5. Properties of a good Estimator Principles of Parameter Estimation Unbiased – The expected value of the estimate is equal to population parameter Consistent – As n (sample size) approaches N (population size), estimator converges to the population parameter Efficient – With the smallest variance. Minimum Mean-Squared Error -Variance of estimator be as low as possible. Sufficient – Contains all information about the parameter through a sample of size n

6. Unbiased Estimator An unbiased estimator is a statistics that has an expected value equal to the population parameter being estimated. An unbiased estimator is a statistics that has an expected value equal to the population parameter being estimated. E[θ] n = θ 0 for any n Examples: The sample mean, is an unbiased estimator of the population mean, μ. The sample mean, is an unbiased estimator of the population mean, μ. The sample variance is an unbiased estimator of the population variance,

7. Consistent Estimators A statistics is a consistent estimator of a parameter if its probability that it will be close to the parameter's true value approaches 1 with increasing sample size. The standard error of a consistent estimator becomes smaller as the sample size gets larger. The sample mean and sample proportions are consistent estimators, since from their formulas as n gets bigger, the standard errors become smaller.

8. Consistent Estimators

9. Relative Efficiency A parameter may have several unbiased estimators. For example, given a symmetrical continuous distribution, both : * The sample mean and * The sample median are unbiased estimators of the distribution mean (when it exists). Which one should we choose ? Certainly we should choose the estimator that generates estimates that are closer (in a probabilistic sense) to the true value θ 0 than estimates generated by the other one. One way to do that is to select the estimator with the lower variance. This leads to the definition of the relative efficiency of two unbiased estimators. Given two unbiased estimators θ * 1 and θ * 2 of the same parameter θ, one defines the efficiency of θ * 2 with respect to θ * 1 (for a given sample size n) as the ratio of their variances : Relative efficiency (θ * 2 with respect to θ * 1 ) n = Var(θ * 1 ) n / Var(θ * 2 ) n

10. Efficient Estimator The estimator has a low variance, usually relative to other estimators, which is called relative efficiency. Otherwise, the variance of the estimator is minimized. An efficient estimator consider the reliability of the estimator in terms of its tendency to have a smaller standard error for the same sample size when compared each other The median is an unbiased estimator of μ when the sample distribution is normally distributed; but is standard error is 1.25 greater than that of the sample mean, so the sample mean is a more efficient estimator than the median. The Maximum Likelihood Estimator is the most efficient estimator among all the unbiased ones.

11. Minimum Mean-Squared Error Estimator

13. Sufficient Estimator We have shown that and are unbiased estimators of μ and. Are we loosing any information about our target parameters relying on these statistics? The statistics, that summarizes all the information about target parameters are said to have the property of sufficiency, or they are called sufficient statistics. “Good” estimators are (or can be made to be) functions of any sufficient statistic.

14. Sufficient Estimator

15. Example : Sufficient Estimator

17. Methods of Point Estimation Classical Approach. Bayesian Approach. Classical Approach: Method of Moment Method of Maximum Likelihood Method of Least Square

18. Method of Moments

19. Example

20. Method of Maximum Likelihood The likelihood and log-likelihood functions are the basis for deriving estimators for parameters, given data. While the shapes of these two functions are different, they have their maximum point at the same value. In fact, the value of parameter that corresponds to this maximum point is defined as the Maximum Likelihood Estimate (MLE). This is the value that is “mostly likely" relative to the other values. The maximum likelihood estimate of the unknown parameter in the model is that value that maximizes the log-likelihood, given the data.

21. Method of Maximum Likelihood Using calculus one could take the first partial derivative of the likelihood or log-likelihood function with respect to the parameter(s), set it to zero and solve for parameter(s). This solution will give the MLE(s).

22. Method of Maximum Likelihood

23. Properties of Maximum Likelihood Estimators For “large" samples (“asymptotically"), MLEs are optimal. 1. MLEs are asymptotically normally distributed. 2. MLEs are asymptotically “minimum variance." 3. MLEs are asymptotically unbiased (MLEs are often biased, but the bias→ 0 as n → ∞. MLE is consistent The Maximum Likelihood Estimator is the most efficient estimator among all the unbiased ones. Maximum likelihood estimation represents the backbone of statistical estimation.

24. Example

26. Method of Least Squares A statistical technique to determine the line of best fit for a model. The least squares method is specified by an equation with certain parameters to observed data. This method is extensively used in regression analysis and estimation. Ordinary least squares - a straight line is sought to be fitted through a number of points to minimize the sum of the squares of the distances (hence the name "least squares") from the points to this line of best fit.

27. Method of Least Squares

29. Example: Method of Least Squares

30. Interval Estimation

31. Interval estimationInterval estimation, Credible interval, and Prediction intervalCredible intervalPrediction interval Confidence intervals are one method of interval estimation, and the most widely used in Classical statistics. An analogous concept in Bayesian statistics is credible intervals, while an alternative Classical and Bayesian both methods is that of prediction intervals which, rather than estimating parameters, estimate the outcome of future samples.interval estimationClassicalBayesian statisticscredible intervals prediction intervals An interval estimator of the sample mean can be expressed as the probability that the mean between two values. Interval estimation, “Confidence Interval” – use a range of numbers within which the parameter is believed to fall (lower bound, upper bound) – e.g. (10, 20)

32. Interval Estimation for the mean of a Normal Distribution

33. Confidence Interval

34. Exponential Distribution

35. Bayesian Estimation

36. Prior and Posterior Distribution

39. Properties of Posterior Mean The Bayes estimate of a parameter is the posterior mean. Usually the posterior distribution will have some common distributional form (such as Gamma, Normal, Beta, etc.). Some things to remember about the posterior mean The data only enter the equation for the posterior in terms of the likelihood function. Therefore, the parameters of the posterior distribution, and hence the posterior mean, are functions of the sufficient statistics. Often the posterior mean has lower MSE than the MLE for portions of the parameter space, so its a worthwhile estimator to consider and compare to the MLE. The posterior mean is consistent, asymptotically unbiased (meaning the bias tends to 0 as the sample size increases), and the asymptotic efficiency of the MLE compared to the posterior mean is 1. Actually, for large n the MLE and posterior mean are very similar estimators, as we will see in the examples.

40. Example: Geometric

41. Example : Binomial

42. Example: Poisson

44. Example: Normal

45. Bayesian Interval Estimation

46. Prediction

47. Predictive Distribution : Binomial-Beta

48. Predictive Density : Normal-Normal

49. Predictive Distribution : Binomial-Beta