STATISTICS POINT ESTIMATION Professor Ke-Sheng Cheng Department of Bioenvironmental Systems Engineering National Taiwan University

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Is it originated from a normal (or gamma) distribution? 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Statistical inference(統計推論) Given a random sample from the distribution of a population, we often are interested in making inferences about the population. Two important statistical inferences are Estimation (推估) Test of hypotheses (假設檢定). Normal distribution? 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Parameter estimation Assume that some characteristics of the elements in a population can be represented by a RV X with pdf fX(．;θ), where the “form” of the density is assumed known except that it contains some unknown parameters θ. We want to estimate an unknown parameter θ or some function of the unknown parameter, τ(θ), using observed values of a random sample, x1,x2,…,xn. 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Point estimation Let the value of some statistic, say , represent the unknown parameter θ or τ(θ); such a statistic is called a point estimator. For example, and respectively are point estimator of  and . 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Two statistics and , where , so that Interval estimation Two statistics and , where , so that ( , ) constitutes an interval for which the probability can be determined that it contains the unknown parameter θ or τ(θ). 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Point Estimation of Distribution Parameters What parameters are to be estimated? Parameters are variables that characterize distributions. For example mean and standard deviation are parameters. How can we estimate the parameters? methods of finding estimators desired properties of point estimators 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

To estimate parameters we need to have a random sample To estimate parameters we need to have a random sample. For example, a random sample (x1,x2,…,xn) of f(．;θ) is collected in order to estimate the parameter θ. Therefore, There can be many (or infinite) ways of estimation and we need to establish some kind of criteria in order to have an adequate estimator. 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Estimators and Estimates Estimator: Any “statistic” whose values are used to estimate τ(θ), where τ(．) is some function of the parameter θ, is defined to be an estimator of . Note that an estimator is a random variable. Estimate: The estimated value given by an estimator. 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Methods of finding estimator Assume that x1,x2,…,xn is a random sample from a density f(．;θ), where the form of the density is known but the parameter θ is unknown. Further assume that θ is a vector of real numbers, say θ=(θ1,θ2,…,θk). We often let , called the parameter space, denote the set of possible values that the parameter θ can assume. Our objective is to find statistics to be used as estimators of certain functions, say , of θ= (θ1,…,θk). 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Method of moments Let f(．;θ1,θ2,…,θk) be the density of random variable X which has k parameters . In general will be a known function of the k parameters , i.e. Let x1, x2,…, xn be a random sample from the density f(．; ). From the k equations, we have 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

We can solve a solution of expressed in terms of x1, x2,…, xn We can solve a solution of expressed in terms of x1, x2,…, xn. The solution, i.e. , is called an estimator of . 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Given a random sample of a normal distribution with mean μ and variance σ2. Estimate the parameters μ and σ by the method of moments. 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Note that the method of moment estimator of σ2 is NOT the sample variance. 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Let x1, x2,…, xn be a random sample from a Poisson distribution with parameter λ. Estimate using MOM. 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Let be a random sample from a uniform distribution on Let be a random sample from a uniform distribution on . What are the method of moments estimator of and ? 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Maximum Likelihood Method Definition of the likelihood function 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Rationale of using likelihood function for parameter estimation 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Definition of the maximum likelihood estimator 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Example 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Example 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Example 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Example 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Parameter estimators of various distributions 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Properties of point estimators Among different estimators, we want to know whether one estimator is “better” than others or what properties an estimator may or may not possess. Unbiasedness Efficiency Mean squared error Consistency 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Unbiased estimator An estimator ( ) of is said to be unbiased if An unbiased estimator is said to be more efficient than any other unbiased estimator of , if for all . 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Mean squared error of an estimator 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Naturally, we would prefer to find an estimator that has the smallest mean-squared error, however, such estimators rarely exist. In general, the mean-squared error of an estimator depends on . What should we look for if a uniformly minimum MSE estimator rarely exists? 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Uniformly minimum MSE estimator 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Let’s define = bias of T . [Note: is an estimator of .] Then, 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Bias 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

We look for an estimator that has a uniformly minimum MSE within the class of unbiased estimators. Such an estimator is called a uniformly minimum-variance unbiased estimator (UMVUE). 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Example 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

The MSE of is 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Standard Error The “standard error” of a statistic is the standard deviation of its sampling distribution. If the standard error involves unknown parameters whose values can be estimated, substitution of those estimates into the standard error results in an “estimated standard error”. 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Mean-squared-error consistency We discuss the mean-squared error of an estimator derived from a random sample of “fixed” size n. Properties of point estimators that are defined for a “fixed” sample size are referred to as “small-sample” properties, whereas properties that are defined for increasing sample size are referred to as “large-sample” properties. 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Properties of the maximum likelihood estimators Asymptotic properties of the MLEs 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Invariance property of the MLEs The invariance property does not hold for unbiasedness. 5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

5/31/2018 Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.