1. 458 Quantifying Uncertainty using Classical Methods (Likelihood Profile, Bootstrapping) Fish 458, Lecture 12

2. 458 Quantifying Uncertainty (an overview) Uncertainty comes in several forms: Process uncertainty (e.g. recruitment variability, natural mortality variability, birth-death processes). Observation uncertainty (e.g. CVs for abundance estimates). Model uncertainty (is the model we chose correct; how many alternative models fit the data adequately?) Estimation uncertainty – given a model and some data, how well do the data determine the parameters (and predictions) of the model. Implementation uncertainty – given a management decision, it be enforced?

3. 458 Quantifying Uncertainty (an overview-II) The various types of uncertainties can be distinguished by: Can they be reduced by additional research or are they inherent to the system. Can we quantify them using classical statistical methods. Today we address “estimation uncertainty”. We defer the other types of uncertainties to future lectures.

4. 458 Estimation Uncertainty We are going to quantify uncertainty about the estimates of the model parameters (and its predictions of state variables) under the assumption that the model (and likelihood) are correct. Typical ways to quantify estimation uncertainty include computing standard errors and confidence intervals.

5. 458 Hint: Generic Solver Macro-I Sub ApplySolver(Minpars, TheSheet, TheFunctionValue) Worksheets(TheSheet).Activate SolverReset ' Small precision and automatic scaling SolverOptions Precision:=0.00001, Scaling:=True ' Specify the cell to minimize (MaxMilVal=2) and the parameters to change SolverOK SetCell:=Range(TheFunctionValue), MaxMinVal:=2, ByChange:=Range(Minpars) ‘ Add a constraint (in this case the cells must be positive) SolverAdd CellRef:=Range(Minpars), Relation:=3, FormulaText:=0 ' Don't ask anything at the end of the call SolverSolve UserFinish:=True End Sub

6. 458 Hint: Generic Solver Macro-II Notes: Many EXCEL versions do not have all the Visual Basic libraries needed to call SOLVER from a macro loaded. Within the Visual Basic editor you will need to click “Tools” - “References” - “Solver.xls” to make this library accessible.

7. 458 Likelihood Profile (one parameter) 1. Fit the model to find the ML parameter estimates and the corresponding negative log-likelihood. 2. Select a set of fixed values for the parameter of interest. 3. Minimize the negative log-likelihood fixing the parameter to each value in turn. 4. Plot the difference between the negative log-likelihood from step 1 and those from step 3.

8. 458 A First Example-I The problem: We are fitting a growth curve to some age and length data. We want to compute the likelihood profile for  Note: t 0 =0 and  is assumed known

9. 458 A First Example-II Approximate 95% CI We can compute confidence intervals from likelihood profiles

10. 458 Likelihood profiles and confidence intervals An 100-x% confidence interval for p parameters is determined by finding the values for the parameter(s) for which: is the negative log-likelihood corresponding to the maximum likelihood estimates.

11. 458 Likelihood Profile (multiple parameters) 1. Fit the model to find the ML parameter estimates and the corresponding negative log- likelihood. 2. Select a set of fixed parameter combinations for the set of parameters of interest. 3. Minimize the negative log-likelihood fixing the values for the set of parameters of interest to each set of values in turn. 4. Plot the difference between the negative log- likelihood from step 1 and those from step 3 (this creates a surface).

12. 458 What about State Variables-I Model outputs often include population size, harvest rate, etc. We are usually more interested in these quantities than about the parameters themselves. However, the state variables are seldom parameters of the model (and cannot be made to be parameters of the model). This makes computing a likelihood profile for them difficult.

13. 458 What about State Variables-II For each (target) value of the State Variable: Add a penalty to the negative log-likelihood that increases as the difference between the target value and the model estimate is larger. It is often a good idea to change the size of the penalty, w, as we get closer to the target (i.e. apply Solver several times, each time increasing w)

14. 458 A Likelihood Profile for Current Biomass

15. 458 Bootstrapping-I Likelihood profile has some major disadvantages: Dealing with derived quantities (e.g. state variables) can be difficult. Dealing with multiple parameters simultaneously is very computationally intensive. It is impossible to compute likelihood profiles for all of the state variables simultaneously. These problems can be overcome through bootstrapping.

16. 458 Bootstrapping-II Bootstrapping deserves a course of its own but, in (very) simple terms, it involves: Generate a large number of pseudo data sets, each based on the original data set. Fit the model to each such data set. Compute summary statistics of interest (standard deviations, confidence intervals, etc.) from the results for each model fit.

17. 458 Bootstrapping III The most common form of bootstrapping involves developing the pseudo data set by resampling the residuals (with replacement) and adding these to the model predictions : is the index for year y in pseudo data set U, is the model prediction of the index for year y, and y* is selected at random from 1:n.

18. 458 Bootstrapping – IV (the biomass trajectory for Cape Hake)

19. 458 Readings Hilborn and Mangel, Chapter 7 Haddon, Chapter 3