1. Chapter 19: Unbiased estimators
CIS 3033

2. 19.1 Estimators
A dataset is often modeled as a realization of a random sample from a probability distribution determined by one or more model parameters.
Let t = h(x1, , xn) be an estimate of a parameter based on the dataset x1, , xn only. Then t is a realization of the random variable 
T = h(X1, . . .,Xn), which is called an estimator.
Some examples are in Table 17.2.

3. 19.1 Estimators
Sometimes there can be more than one estimators, based on different analysis of the process.
Example: Given a dataset x1, x2, ..., xn on the number of packages in a minute received at a network server, what is the probability for the server  to receive 0 package within a minute?
Approach 1: as the relative frequency of the event
Approach 2: as p(0) of a Poisson process

4. 19.2 The behavior of an estimator
Which of S and T is a better estimator for the probability of zero arrivals in a Pois(μ) distribution?
When n = 30, neither S nor T can give the exact value of p0. To compare them, assume a parameter value (μ = ln 10, so that p0 = e−μ = 0.1), draw 30 numbers, and get S and T from them. Repeat the process 500 times, and get histograms of S and T.

5. 19.2 The behavior of an estimator

6. 19.3 Unbiasedness
Let T = h(X1, , Xn) be an estimator. The probability distribution of T is called the sampling distribution of T.
An estimator T is called an unbiased estimator for the parameter θ, if E[T] = θ irrespective of the value of θ. Otherwise T is biased, with E[T] − θ as its bias.
For the previous example, S is unbiased, but T has a positive bias, which converges to 0 when n goes to infinity.

7. 19.4 Unbiased estimators

8. 19.4 Unbiased estimators
When calculating sample variance, if the divisor n − 1 is replaced by n, the result will have a negative bias when used as estimator of the "true" variance of the distribution.
If T is an unbiased estimator for a parameter θ, then g(T) does not have to be an unbiased estimator for g(θ). For example, sample standard derivation has a negative bias when used as an estimator of standard derivation.   
An exception is if g(T) = aT + b.