Estimating parameters from data Gil McVean, Department of Statistics Tuesday 3 rd November 2009

Questions to ask… How can I estimate model parameters from data? What should I worry about when choosing between estimators? Is there some optimal way of estimating parameters from data? How can I compare different parameter values? How should I make statements about certainty regarding estimates and hypotheses?

Motivating example I I conduct an experiment where I measure the weight of 100 mice that were exposed to a normal diet and 50 mice exposed to a high-energy diet I want to estimate the expected gain in weight due to the change in diet NormalHigh-calorie

Motivating example II I observe the co-segregation of two traits (e.g. a visible trait and a genetic marker) in a cross I want to estimate the recombination rate between the two markers Bateson and Punnett experiment Phenotype and genotype Observed Expected from 9:3:3:1 ratio Purple, long (P_L_) Purple, round (P_ll)2172 Red, long (ppL_)2172 Red, round (ppll)5524

Parameter estimation We can formulate most questions in statistics in terms of making statements about underlying parameters We want to devise a framework for estimating those parameters and making statements about our certainty In this lecture we will look at several different approaches to making such statements –Moment estimators –Likelihood –Bayesian estimation

Moment estimation You have already come across one way of estimating parameter values – moment methods In such techniques parameter values are found that match sample moments (mean, variance, etc.) to those expected E.g. for random variables X 1, X 2, etc. sampled from a N( ,  2 ) distribution

Example: fitting a gamma distribution The gamma distribution is parameterised by a shape parameter, , and a scale parameter,  The mean of the distribution is  /  and the variance is  /  2 We can fit a gamma distribution by looking at the first two sample moments Alkaline phosphatase measurements in 2019 mice  

Bias Although the moment method looks sensible, it can lead to biased estimators In the previous example, estimates of both parameters are upwardly biased Bias is measured by the difference between the expected estimate and the truth However, bias is not the only thing to worry about –For example, the value of the first observation is an unbiased estimator of the mean for a Normal distribution. However it is a rubbish estimator We also need to worry about the variance of an estimator

Example: estimating the population mutation rate In population genetics, a parameter of interest is the population-scaled mutation rate There are two common estimators for this parameter –The average number of differences between two sequences –The total number of polymorphic sites in the sample divided by a constant that is approximately the log of the sample size Which is better? The first estimator has larger variance than the second – suggesting that it is an inferior estimator It is actually worse than this – it is not even guaranteed to converge on the truth as the sample size gets infinitely large –A property called consistency

The bias-variance trade off Some estimators may be biased Some estimators may have large variance Which is better? A simple way of combining both metrics is to consider the mean-squared error of an estimator

Example Consider two ways of estimating the variance of a Normal distribution from the sample variance The second estimator is unbiased, but the first estimator has lower MSE Actually, there is a third estimator, which is even more biased than the first, but which has even lower MSE

Problems with moment estimation It is not always possible to exactly match sample moments with their expectation It is not clear when using moment methods how much of the information in the data about the parameters is being used –Often not much.. Why should MSE be the best way of measuring the value of an estimator?

Is there an optimal way to estimate parameters? For any model the maximum information about model parameters is obtained by considering the likelihood function The likelihood function is proportional to the probability of observing the data given a specified parameter value One natural choice for point estimation of parameters is the maximum likelihood estimate, the parameter values that maximise the probability of observing the data The maximum likelihood estimate (mle) has some useful properties (though is not always optimal in every sense )

An intuitive view on likelihood

An example Suppose we have data generated from a Poisson distribution. We want to estimate the parameter of the distribution The probability of observing a particular random variable is If we have observed a series of iid Poisson RVs we obtain the joint likelihood by multiplying the individual probabilities together

Comments Note in the likelihood function the factorials have disappeared. This is because they provide a constant that does not influence the relative likelihood of different values of the parameter It is usual to work with the log likelihood rather than the likelihood. Note that maximising the log likelihood is equivalent to maximising the likelihood We can find the mle of the parameter analytically Note that here the mle is the same as the moment estimator Find where the derivative of the log likelihood is zero Take the natural log of the likelihood function

Sufficient statistics In this example we could write the likelihood as a function of a simple summary of the data – the mean This is an example of a sufficient statistic. These are statistics that contain all information about the parameter(s) under the specified model For example, support we have a series of iid normal RVs Mean square Mean

Properties of the maximum likelihood estimate The maximum likelihood estimate can be found either analytically or by numerical maximisation The mle is consistent in that it converges to the truth as the sample size gets infinitely large The mle is asymptotically efficient in that it achieves the minimum possible variance (the Cramér-Rao Lower Bound) as n→∞ However, the mle is often biased for finite sample sizes –For example, the mle for the variance parameter in a normal distribution is the sample variance

Comparing parameter estimates Obtaining a point estimate of a parameter is just one problem in statistical inference We might also like to ask how good different parameter values are One way of comparing parameters is through relative likelihood For example, suppose we observe counts of 12, 22, 14 and 8 from a Poisson process The maximum likelihood estimate is 14. The relative likelihood is given by

Using relative likelihood The relative likelihood and log likelihood surfaces are shown below

Interval estimation In most cases the chance that the point estimate you obtain for a parameter is actually the correct one is zero We can generalise the idea of point estimation to interval estimation Here, rather than estimating a single value of a parameter we estimate a region of parameter space –We make the inference that the parameter of interest lies within the defined region The coverage of an interval estimator is the fraction of times the parameter actually lies within the interval The idea of interval estimation is intimately linked to the notion of confidence intervals

Example Suppose I’m interested in estimating the mean of a normal distribution with known variance of 1 from a sample of 10 observations I construct an interval estimator The chart below shows how the coverage properties of this estimator vary with a If I choose a to be 0.62 I would have coverage of 95%

Confidence intervals It is a short step from here to the notion of confidence intervals We find an interval estimator of the parameter that, for any value of the parameter that might be possible, has the desired coverage properties We then apply this interval estimator to our observed data to get a confidence interval We can guarantee that among repeat performances of the same experiment the true value of the parameter would be in this interval 95% of the time We cannot say ”There is a 95% chance of the true parameter being in this interval”

Example – confidence intervals for normal distribution Creating confidence intervals for the mean of normal distributions is relatively easy because the coverage properties of interval estimators do not depend on the mean (for a fixed variance) For example, the interval estimator below has 95% coverage properties for any mean As you’ll see later, there is an intimate link between confidence intervals and hypothesis testing

Example: confidence intervals for exponential distribution For most distributions, the coverage properties of an estimator will depend on the true underlying parameter However, we can make use of the CLT to make confidence intervals for means For example, for the exponential distribution with different means, the graph shows the coverage properties for the interval estimator (n=100)

Confidence intervals and likelihood Thanks to the CLT there is another useful result that allows us to define confidence intervals from the log-likelihood surface Specifically, the set of parameter values for which the log-likelihood is not more than 1.92 less than the maximum likelihood will define a 95% confidence interval –In the limit of large sample size the LRT is approximately chi-squared distributed under the null This is a very useful result, but shouldn’t be assumed to hold –i.e. Check with simulation

Bayesian estimators As you may notice, the notion of a confidence interval is very hard to grasp and has remarkably little connection to the data that you have collected It seems much more natural to attempt to make statements about which parameter values are likely given the data you have collected To put this on a rigorous probabilistic footing we want to make statements about the probability (density) of any particular parameter value given our data We use Bayes theorem Prior Likelihood Posterior Normalising constant

Bayes estimators The single most important conceptual difference between Bayesian statistics and frequentist statistics is the notion that the parameters you are interested in are themselves random variables This notion is encapsulated in the use of a subjective prior for your parameters Remember that to construct a confidence interval we have to define the set of possible parameter values A prior does the same thing, but also gives a weight to different values

Example: coin tossing I toss a coin twice and observe two heads I want to perform inference about the probability of obtaining a head on a single throw for the coin in question The point estimate/MLE for the probability is 1.0 – yet I have a very strong prior belief that the answer is 0.5 Bayesian statistics forces the researcher to be explicit about prior beliefs but, in return, can be very specific about what information has been gained by performing the experiment

The posterior Bayesian inference about parameters is contained in the posterior distribution The posterior can be summarised in various ways Prior Posterior Posterior mean Credible Interval

Bayesian inference and the notion of shrinkage The notion of shrinkage is that you can obtained better estimates by assuming a certain degree of similarity among the things you want to estimate Practically, this means two things –Borrowing information across observations –Penalising inferences that are very different from anything else The notion of shrinkage is implicit in the use of priors in Bayesian statistics There are also forms of frequentist inference where shrinkage is used –But NOT MLE