1 Lecture 16: Point Estimation Concepts and Methods Devore, Ch

Topics I.Point Estimation Concepts i.Biased Vs. Unbiased Estimators ii.Minimum Variance Estimators iii.Standard Error of a Point Estimate II.Point Estimation Methods A.Method of Moments (MOM) B.Methods of Least Squares C.Methods of Maximum Likelihood (MLE)

I. Point Estimates Objective – obtain an “educated guess” of a population parameter from a sample. Applications: –Parameter Estimation –Hypothesis Testing

Applications Examples: Parameter Estimation: –Suppose the failure times on an 300-hour accelerated life test are 10, 50, 110, 150, 200, 220, 250 (3 have not failed). Estimate the parameter (lambda) if lifetimes follow an exponential distribution. Hypothesis Testing: –Suppose you have two bottle filling operations and wish to test if one machine yields lower variance. –Methodology – obtain point estimate of variance for each of machine 1 and 2. Then, based on point estimates, test for statistically significant difference.

Parameter Estimation Treat the data as constants and the parameters as rv’s –Let X = variable under study –Let  = estimate of a parameter –Note: predicted value of  is  -hat, or –Example: predict the mean,  as  -hat, or

Hypothesis Testing Given point estimator(s) from samples, we may wish to infer about the reproducibility of results, or if any statistical differences exist. Examples: suppose you measure two samples –Common Question: Is it reasonable to conclude that no statistically significant difference exists?

Parameter Estimation Examples Suppose you wish to estimate the population mean, ,. Some possible estimators include: –Mean, Median, Trimmed Mean Recall example from descriptive statistics, which of the following is the best estimator? –Mean = Median = Trim Mean = 971.4

Parameter Estimates - Fit In practice, we would prefer to have one estimator (  -hat) that is always the best. However,  -hat is a function of the observed Xi’s So, Thus, we may identify the “best estimator” as the one with: –Least bias (unbiased) –Minimum variance of estimation error (increase the likelihood that the observed parameter estimate represents the true parameter)

Biased Vs. Unbiased Estimator Bias - difference between the expected value of the statistic  - hat and the parameter  Unbiased Estimator: –Example: X-bar is an unbiased estimated of  (Bias = 0) –Suppose X 1, X 2,.. X n are iid rv’s, with E(X i ) = 

Unbiased Estimator of Variance Which of the following is an unbiased estimator of variance? (n-1) is unbiased (see page 259 for proof) Logic argument: will Xi’s be closer to X-bar or  ? –Thus, will dividing by n tend to overestimate or underestimate the true variance? –What happens to the bias effect as n becomes large?

Is S unbiased of  ? E(S)=s=  ? Simulation Experiment: –Suppose you have a true variance = 1 –Simulate k replications of size n and compare expected values. –A Sample Result from k=5000, n=5 –Variance using n divisor also has negative bias (underestimates) –Note: S has a negative bias (underestimate s) – there are other reasons to use S, we’ll see that later…

Minimum Variance Estimators Several unbiased estimators may exist for a parameter. –Example: Mean, median, trimmed mean Minimum Variance Unbiased Estimator (MVUE) –among all unbiased estimators, MVUE represents the one with minimum variance.

MVUEs Given the following pdf’s for  1 and  2, which is the MVUE of  ?  pdf of  1 pdf of  2

MVUE Example Suppose Xi is N( ,  2 ) Both X-bar and Xi are unbiased estimators of  –Note: variance of Xi is  2 However, if n>=2, –then X-bar is better estimator of  because it has less variance of X-bar (  2 X-bar =  2 / n )

Which is best the estimator? The best estimator often depends on the underlying distribution or data pattern. Consider advantages and disadvantages of Xbar, median or trimmed mean. –If normal, X-bar is the minimum variance estimator. –If bell shaped, but with heavy tails (e.g., Cauchy Distribution), then median is better because outliers are likely. –Trimmed mean (10%) is not better for either, but is robust to both  “robust estimator”

Standard Error ~ Point Estimate Standard error of an estimatoris –standard deviation of the estimator,   Standard error of X-bar

Point Estimate Intervals If the estimator follows a normal distribution (very common), then we may be reasonably confident that the true parameter falls within +/- 2 standard errors of the estimator. –~ 94-96% confident Thus, standard errors may be used to identify an interval estimate of which the true parameter value likely falls within.

Example: Interval Estimate Suppose you have 10 measurements of thermal conductivity , 41.48, 42.34, 41.95, , 41.72, 42.26, 41.81, X-bar = ; S = Calculate an interval +/- 2 standard errors for the true mean conductivity. How precise is the std error of the mean?

II. Methods of Point Estimation Goal: obtain the “best estimator” Conditions: –Least bias (unbiased) –Minimum variance of estimation error –Recognize that we may need to tradeoff these conditions! Applications: –Estimate coefficients (Y =   +   X ) / –Estimate distribution parameters e.g., ,  2 We now discuss three “general” techniques to obtain point estimators. –Moments, maximum likelihood, least squares

A. Method of Moments (MOM) Find first k moments of p.d.f. and equate to first k sample moments. Solve system of simultaneous equations. –The kth moment of population distribution f(x) is E(X k ) –The kth sample moment Let k=1,2,3,.. Example, if k=1, E(X) =  X i / n

# Equations Estimate 1 parameter, use 1 moment Estimate m parameters, need m moments –Suppose you have 2 parameters to estimate f(x;  1,  2 ) E(X 1 ) E(X 2 )

Example: Find Point Estimator based on sample of data Consider a random sample, X 1.. X n. Suppose you wish to find estimator  given the pdf f(x) =  x) 0 <= x <= 1 Exercises: –Obtain an estimate of  based on MOM Hint: 1 parameter, so you only need 1 moment equation

Distribution Applications - Parameter Estimates from Data Bernoulli Distribution (n samples of size 1) –Sample Mean: –Population Mean: E(x) = p –Parameter Estimate:

Moments - Multiple Estimates Exponential Distribution –Sample Mean –Population Mean E(x) = 1/ –Parameter Estimate: Poisson Distribution –Sample Mean –Population Mean: E(x) = –Parameter Estimate:

Aren’t there other estimators? Exponential –Sample Variance: –Population Variance: –Parameter Estimate: So, Which is preferred? OR

Estimating by Method of Moments (MoM) Advantages: –Simple to generate –Unbiased –Asymptotically Normal (tends to normal when n is large) Disadvantages: –Inconsistent results (more than one estimator equation) –May not have desirable statistical properties or may produce flawed results. See Example 6.13 in textbook

B. Method of Least Squares Based on prediction error. Attempts to minimize prediction error. –Error: x i =  + e i or e i = x i -  –Sum of Squared Error: –Estimate parameter(s) by minimizing Sum of Squared Error with respect to the parameter. –Note: this is the basis for regression. Y= mX + b

C. Maximum Likelihood Estimation (MLE) Developed by Sir R.A. Fisher (1920s) Preferred method by statisticians particularly if n is sufficiently large, because the MLE (maximum likelihood estimator) approximates the MVUE. Maximum likelihood estimation ~ maximizes the likelihood that the observed sample is a function of the possible parameter values.

Maximum Likelihood Estimation - Single Parameter Given a sample of size n: x 1, x 2,.., x n from f(x) Likelihood Function: –Continuous: L(  ) = –Discrete: L(  ) = –To obtain MLE: Maximize L or L* = ln(L) usually by setting dL* / d(parameter of interest) = 0 and solving the resulting equations. –Note: if more than 1 parameter, you will have a system of simultaneous equations f(x 1, x 2,.. X n ;  1,..  m )

Example: Exponential distribution f(x) = e - x with x > 0, and > 0 Find MLE for  Note: MLE is the same as MOM (though it is not an unbiased estimator (why? see Devore, p. 273, top)

Invariance Principle Given: Then, the MLE of any function, h(  1,  2,..  n ) of these parameters is the same function, replacing the thetas with their estimators.

MLE Vs. MoM MLEs are usually preferred to MoM since they are: –Consistent –Asymptotically Normal –Asymptotically Efficient –Invariant –May not be unbiased. Disadvantages of MLE - may be complicated to solve Using derivative calculus to maximize L() may not result in a logical answer.

Mean Squared Error (MSE) Sometimes we choose to use a biased estimator. MSE represents the squared difference between the estimator and bias. –If unbiased estimator: MSE (  -hat)= Var(  -hat) –If multiple estimators exist, it may be preferred to induce a small amount of bias to reduce variance of the estimator.