1. The Method of Likelihood Hal Whitehead BIOL4062/5062

2. What is likelihood Maximum likelihood Maximum likelihood estimation Likelihood ratio tests Likelihood profile confidence intervals Model selection: –Likelihood ratio tests –Akaike Information Criterion (AIC) Likelihood and least-squares Calculating likelihood

3. The Method of Likelihood Observations: Y = {y 1,y 2,y 3,...} e.g. Weights of 30 crabs of known age and sex Model specified by: μ 1, μ 2, μ 3,… e.g. y = μ 1 + μ 2 ·√Age + μ 3 ·Sex(0:1) + μ 4 ·e where e ~ N(0, 1) The LIKELIHOOD of Y is: L = Probability ( Y | Model & μ 1, μ 2, μ 3,... )

4. Likelihood The LIKELIHOOD of Y is: L = Probability ( Y | Model & μ 1, μ 2, μ 3,... ) The LIKELIHOOD that Z became a criminal: Probability Z became a criminal given what we what we know of Z’s characteristics and how those characteristics translate into the probability of being a criminal

5. The LIKELIHOOD of Y is: L = Probability ( Y | Model & μ 1, μ 2, μ 3,…) We can work this out if we know μ 1, μ 2, μ 3,… Weights of 30 crabs of known age and sex y = μ 1 + μ 2 ·√Age + μ 3 ·Sex(0:1) + μ 4 ·e e.g Prob. of these 30 weights is 0.04 if: female wt at age 0, μ 1 = 30.0 growth parameter, μ 2 = 0.7 excess male weight, μ 3 = 5.0 residual s.d., μ 4 = 6.3 L(μ 1 =30,μ 2 =0.7,μ 3 =5.0, μ 4 =6.3)=0.04

6. If we do not know μ 1, μ 2, μ 3,... MAXIMUM LIKELIHOOD of Y is: L(μ 1,μ 2,μ 3,...) = Max{Prob.( Y | μ 1, μ 2, μ 3,... )} μ 1,μ 2,… e.g Max prob. of 30 weights is 0.12 when: female wt at age 0, μ 1 = 28.4 growth parameter, μ 2 = 0.31 excess male weight, μ 3 = 1.7 residual s.d., μ 4 = 3.9 Maximum Likelihood Estimators

7. Maximum Likelihood μ1μ1 Likelihood Maximum likelihood Maximum likelihood estimator of μ 1

8. Maximum Likelihood μ1μ1 Likelihood Precise estimate Imprecise estimate

9. Likelihood Ratio Tests If: μ 1,μ 2,μ 3,…,μ t is true model μ 1,μ 2,μ 3,…,μ t,...,μ g is more general model then: G = 2∙Log[L(μ 1,μ 2,μ 3,…,μ g )/L(μ 1,μ 2,μ 3,…,μ t )] (twice the log of the ratios of the maximum likelihoods) is distributed as χ² with g-t degrees of freedom for large sample sizes (asymptotically) If G is unexpectedly large then data are unlikely to be from model μ 1,μ 2,μ 3,…,μ t

10. Likelihood Ratio Tests G = 2·Log[L(μ 1,μ 2,μ 3,…,μ g )/L(μ 1,μ 2,μ 3,…,μ t )] This is the "G-test for goodness-of-fit": null hypothesis: μ 1,μ 2,μ 3,…,μ t alternative hypothesis: μ 1,μ 2,μ 3,…,μ t,...,μ g

11. Likelihood: an example ExpectFind Wild Type 75% 80 Mutants 25% 10 Total 100% 90

12. Null hypothesis: Binomial Distribution with q = 0.75 ExpectFind Wild Type 75% 80 Mutants 25% 10 Total 100% 90 Likelihood(q=0.75) = 90 C 10 · 0.75 80 · 0.25 10 =.000551

13. Alternative hypothesis: Binomial Distribution with q = ? ExpectFind Wild Type 75% 80 Mutants 25% 10 Total 100% 90 Likelihood(q) = 90 C 10 · q 80 ·(1-q) 10 This has a maximum value when q = 80/90 = 0.89 Max Likelihood(q) = 90 C 10 ·(0.89) 80 ·(1-0.89) 10 = 0.1236 Maximum Likelihood Estimator

14. Likelihood Ratio Test ExpectFind Wild Type 75% 80 Mutants 25% 10 Total 100% 90 G = 2 · Log { Max Likelihood (q) } Likelihood (q = 0.75) = 2 · Log(0.1236/ 0.000551) = 10.96 is distributed as χ² with 1 d.f. if q=0.75 significantly large (P<0.01) in χ²(1) so: reject null hypothesis.

15. Profile Likelihood Confidence Intervals μ1μ1 Likelihood

16. Profile Likelihood Confidence Intervals μ1μ1 Log- Likelihood 2 Maximum likelihood Maximum likelihood estimator of μ 1 95% c.i.

17. Profile Likelihood Confidence Intervals Log-Likelihood Contours (relative to maximum likelihood) μ1μ1 μ2μ2 MLE(0) -2 95% Confidence region

18. Model Selection Using Likelihood-Ratio Tests Weights of 30 crabs of known age and sex: M(0): y = μ 1 + μ 4 · e M(1): y = μ 1 + μ 2 · √ Age + μ 4 · e M(2): y = μ 1 + μ 2 · √ Age + μ 3 · Sex(0:1) + μ 4 · e

19. Model Selection Using Likelihood-Ratio Tests Weights of 30 crabs of known age and sex: M(0): y = μ 1 + μ 4 · e Log(L)= -23.04 M(1): y = μ 1 + μ 2 · √Age + μ 4 · eLog(L)= -20.34 M(2): y = μ 1 + μ 2 · √Age + μ 3 · Sex(0:1) + μ 4 · e Log(L)= -19.84

20. Model Selection Using Likelihood-Ratio Tests Weights of 30 crabs of known age and sex: M(0): y = μ 1 + μ 4 · e Log(L)= -23.04 M(1): y = μ 1 + μ 2 · √Age + μ 4 · eLog(L)= -20.34 M(2): y = μ 1 + μ 2 · √Age + μ 3 ·Sex(0:1) + μ 4 · e Log(L)= -19.84 G(M(0)vs.M(1)) = 2x(-20.34 - (-23.04)) = 5.40 G(M(1)vs.M(2)) = 2x(-19.84 - (-20.34)) = 1.00 G(M(0)vs.M(2)) = 2x(-19.84 - (-23.04)) = 6.40

21. Model Selection Using Likelihood-Ratio Tests Weights of 30 crabs of known age and sex: M(0): y = μ 1 + μ 4 · e Log(L)= -23.04 M(1): y = μ 1 + μ 2 · √ Age + μ 4 · e Log(L)= -20.34 M(2): y = μ 1 + μ 2 · √Age + μ 3 ·Sex(0:1) + μ 4 · e Log(L)= -19.84 G(M(0)vs.M(1)) = 2x(-20.34 - (-23.04)) = 5.40P(χ²(1))<0.05 G(M(1)vs.M(2)) = 2x(-19.84 - (-20.34)) = 1.00P(χ²(1))>0.10 G(M(0)vs.M(2)) = 2x(-19.84 - (-23.04)) = 6.40 P(χ²(2))<0.05

22. Model Selection Using Likelihood-Ratio Tests Weights of 30 crabs of known age and sex: M(0): y = μ 1 + μ 4 · e Log(L)= -23.04 M(1): y = μ 1 + μ 2 ·√Age + μ 4 · e Log(L)= -20.34 M(2): y = μ 1 + μ 2 · √Age + μ 3 · Sex(0:1) + μ 4 · e Log(L)= -19.84 G(M(0)vs.M(1)) = 2x(-20.34 - (-23.04)) = 5.40P(χ²(1))<0.05 G(M(1)vs.M(2)) = 2x(-19.84 - (-20.34)) = 1.00P(χ²(1))>0.10 G(M(0)vs.M(2)) = 2x(-19.84 - (-23.04)) = 6.40 P(χ²(2))<0.05

23. Model Selection Using Likelihood-Ratio Tests Weights of 30 crabs of known age and sex: M(0): y = μ 1 + μ 4 · e Log(L)= -23.04 M(1): y = μ 1 + μ 2 · √Age + μ 4 · e Log(L)= -20.34 M(2): y = μ 1 + μ 2 · √Age + μ 3 · Sex(0:1) + μ 4 · e Log(L)= -19.84 G(M(0)vs.M(1)) = 2x(-20.34 - (-23.04)) = 5.40P(χ²(1))<0.05 G(M(1)vs.M(2)) = 2x(-19.84 - (-20.34)) = 1.00P(χ²(1))>0.10 G(M(0)vs.M(2)) = 2x(-19.84 - (-23.04)) = 6.40 P(χ²(2))<0.05 But: What is critical p-value?

24. Model Selection Using Likelihood-Ratio Tests Weights of 30 crabs of known age and sex: M(1): y = μ 1 + μ 2 ·√Age + μ 4 ·e M(3): y = μ 1 + μ 3 ·Sex(0:1) + μ 4 ·e But: Cannot compare M(1) and M(3) using likelihood-ratio tests

25. Model Selection Using Likelihood-Ratio Tests What is critical p-value? Cannot compare models which are not subsets of one another using likelihood-ratio tests So: Akaike Information Criteria (AIC)

26. Akaike Information Criteria (AIC) Kullback-Leibler Information (KLI): –“information lost when model M(0) is used to approximate model M(1)” –“distance from M(0) to M(1)” AIC(M) = - 2xLog(Likelihood(M)) + 2xK(M) –K(M) is number of estimable parameters of model M AIC is an estimate of the expected relative distance (KLI) between a fitted model, M, and the unknown true mechanism that generated the data

27. Akaike Information Criteria (AIC) AIC(M) = - 2xLog(Likelihood(M)) + 2xK(M) –K(M) is number of estimable parameters In model selection: choose model with smallest AIC –least expected relative distance between M, and the unknown true mechanism that generated the data

28. Model Selection Using AIC Weights of 30 crabs of known age and sex: M(0): y = μ 1 + μ 4 · e M(1): y = μ 1 + μ 2 · √Age + μ 4 · e M(2): y = μ 1 + μ 2 · √Age + μ 3 · Sex(0:1) + μ 4 · e M(3): y = μ 1 + μ 3 · Sex(0:1) + μ 4 · e

29. Model Selection Using AIC Weights of 30 crabs of known age and sex: M(0): y = μ 1 + μ 4 · e AIC=50.08 M(1): y = μ 1 + μ 2 · √Age + μ 4 · eAIC=46.68 M(2): y = μ 1 + μ 2 · √Age + μ 3 · Sex(0:1) + μ 4 · e AIC=47.68 M(3): y = μ 1 + μ 2 · Sex(0:1) + μ 4 · eAIC=49.95

30. Model Selection Using AIC Weights of 30 crabs of known age and sex: M(0): y = μ 1 + μ 4 · e AIC=50.08 M(1): y = μ 1 + μ 2 · √Age + μ 4 · eAIC=46.68 M(2): y = μ 1 + μ 2 · √Age + μ 3 · Sex(0:1) + μ 4 · e AIC=47.68 M(3): y = μ 1 + μ 3 · Sex(0:1) + μ 4 · eAIC=49.95

31. Model Selection Using AIC Differences in AIC between models: ΔAIC Support for less favoured model –ΔAIC: 0-2Substantial –ΔAIC: 4-7Considerably less –ΔAIC: >10Essentially none

32. Model Selection Using AIC Weights of 30 crabs of known age and sex: M(0): y = μ 1 + μ 4 · e AIC=50.08 Unlikely M(1): y = μ 1 + μ2 · √Age + μ 4 · e AIC=46.68BEST M(2): y = μ 1 + μ 2 ·√Age + μ 3 ·Sex(0:1) + μ 4 ·e AIC=47.68 Good M(3): y = μ 1 + μ 3 · Sex(0:1) + μ 4 · e AIC=49.95 Unlikely

33. Modifications to AIC AIC for small sample sizes: AIC C = - 2x(Log-Likelihood) + 2xKxn/(n-K-1) n is sample size AIC for overdispersed count data: QAIC = - 2xLog-Likelihood/c + 2xK c is “variance inflation factor” (c=χ²/df)

34. Burnham, K. P., and D. R. Anderson 2002 Model selection and multimodel inference: a practical information-theoretic approach, 2nd ed. New York: Springer-Verlag

35. Likelihood and Least-Squares If errors are normally distributed –least squares and maximum-likelihood estimates of parameters are the same –but not σ 2 estimators Likelihood is a more powerful and theoretically-based technique

36. AIC and Least-Squares If all models assume normal errors with constant variance: AIC = n.Log(σ 2 ) + 2.K –σ 2 = Σe i 2 /n (the MLE of σ 2 ) –K is total no of estimated regression parameters, including the intercept and σ 2

37. Calculating Likelihoods Analytical formulae Compute by multiplying probabilities Estimate by simulation –number of times data are obtained in 1,000 simulations given model and parameters

38. The Method of Likelihood Probability of data given model Estimate parameters using maximum likelihood Estimate confidence intervals using likelihood profiles Compare models using –likelihood ratio tests –Akaike Information Criterion (AIC)