AN HONEST APPRAISAL OF MANAGEMENT STRATEGY EVALUATION (MSE) Doug S Butterworth MARAM (Marine Resource Assessment and Management Group) Department of Mathematics and Applied Mathematics University of Cape Town, Rondebosch 7701, South Africa

OUTLINE I. DEFINITIONS AND BACKGROUND II. HOW HAVE MANAGEMENT PROCEDURES (MPs) PERFORMED ? Feedback control TAC change constraints Saving time Stakeholder involvement Testing and selection Summary III. MPs vs MSE IV. THE FUTURE V. DISCUSSION POINTS

I. DEFINITIONS AND BACKGROUND MANAGEMENT PROCEDURE (MP): The combination of pre-defined data, together with an algorithm to which such data are input to provide a value for a TAC or effort control measure; this combination has been demonstrated, through simulation trials, to show robust performance in the presence of uncertainties TWO TYPES: Empirical (direct use of data) Model-based (Estimator + Harvest Control Rule) MANAGEMENT STRATEGY EVALUATION (MSE): Usually synonymous with the MP approach; also used to describe the process of testing generic MPs or harvest strategies Definitions from Glossary of “MP Terminology” in Rademeyer et al. (2007) ICES Journal of Marine Science, 64: 618–625.

I. DEFINITIONS AND BACKGROUND MANAGEMENT PROCEDURES (MPs) A very specific approach to fisheries management developed in the International Whaling Commission’s (IWC’s) Scientific Committee in the late 1980s to address problems with the conventional “best assessment + harvest control rule (HCR)” approach for providing Total Allowable Catch (TAC) advice MANAGEMENT STRATEGY EVALUATION (MSE) A similar but more generic approach developed around the same period in Australia

IWC NEW MANAGEMENT PROCEDURE (NMP) 1976 MSY 0.9 MSY C 0.54K 0.6K K Harvest Control Rule: C = 0 for P < 0.54 K C = 0.9 MSY for P > 0.60 K Input required to calculate C: P: current abundance K: pristine abundance MSY

1980s: FAILURE OF THE NMP IWC: LESSONS LEARNT AND SOLUTION How to calculate P, K and MSY? How to take uncertainties into account? IWC: LESSONS LEARNT AND SOLUTION An HCR alone is not enough The Estimator (assessment method) must also be pre- specified Furthermore the data to be input to that estimator must also be pre-specified i.e. Move to a “Management Procedure Approach”

APPROACH FOR THIS TALK EVALUATE THE PERFORMANCE OF THE APPLICATION OF MPs “sensu stricto” i.e. Ones following the IWC approach where the estimator that is applied in practice is the same as the one simulation tested, and on which selection was based “MSE COMPARISONS” DISCUSSED BRIEFLY LATER e.g. Cases where the HCR is selected on the basis of simulation tests, but a current “best assessment” may be used with that HCR rather than the specific estimator tested NOTE THAT GLOBALLY THE TERM “MSE” IS USED WITH A WIDER VARIETY OF MEANINGS THAN IS THE CASE FOR “MP”

II. HOW HAVE MPs PERFORMED ? KEY CLAIMS MADE FOR MPs COMPARED TO THE “BEST ASSESSMENT” APPROACH TO TAC PROVISION, MPs: Utilise Feedback to address uncertainties effectively Provide a defensible basis to limit inter-annual TAC changes Save analysis time Provide a more effective basis to involve stakeholders in the decision making process

HOW EFFECTIVE IS FEEDBACK CONTROL? EXAMPLE FISHERY 30 years of catch + abundance index (CV=20%) data Dynamics governed by Schaefer model B(y+1) = B(y) + r B(y) [1 – B(y)/B0] – C(y)

FEEDBACK: EXAMPLE FISHERY Manage from year 31 under FMSY = r̂ /2: C(y) = 0.5 r̂ B̂(y) with estimation using Schaefer model

FEEDBACK: EXAMPLE FISHERY How well does feedback correct for unreported overcatch of a fixed percentage from year 31?

FEEDBACK: EXAMPLE FISHERY How well does feedback correct for unreported overcatch of a fixed percentage from year 31?

FEEDBACK: EXAMPLE FISHERY How well does feedback correct for unreported overcatch of a fixed percentage from year 31?

FEEDBACK: EXAMPLE FISHERY How well does feedback correct for unreported overcatch of a fixed percentage from year 31?

FEEDBACK: EXAMPLE FISHERY How well does feedback correct for unreported overcatch of a fixed percentage from year 31?

FEEDBACK: EXAMPLE FISHERY How well does feedback correct for unreported overcatch of a fixed percentage from year 31?

FEEDBACK: EXAMPLE FISHERY So how robust was this MP to catch under-reporting? For small extent, feedback corrected well and quickly BUT For large extent, correction was limited and slow FEEDBACK HELPS, BUT NOT ALWAYS PERFECTLY

II. HOW HAVE MPs PERFORMED ? KEY CLAIMS MADE FOR MPs Provide a defensible basis to limit inter-annual TAC changes Save analysis time Provide a more effective basis to involve stakeholders in the decision making process ADDRESS THESE THROUGH A SURVEY OF “IMPLEMENTERS” (usually the developer scientists) Note that this does introduce a potential bias – “developers” may stray towards “over-promoting” the successes of their endeavours

II. HOW HAVE MPs PERFORMED ? LENGTHY IMPLEMENTATIONS ( > 10 years ) Resource Adviser(s) Years applied Empirical/ (No stocks) Model-based SA hake Rademeyer 26 Emp SA sardine/anchovy de Moor 26 Emp Norway minke whale Walloe 24 Mod NZ rock lobster Breen/Starr 21 (8) Emp SA west coast lobster Johnston 19 (5) Emp IWC BCB bowheads* Donovan 15 Mod IWC ENP gray* Donovan 15 Mod * Aboriginal whaling: Satisfies need rather than maximises catch

II. HOW HAVE MPs PERFORMED ? MEDIUM IMPLEMENTATIONS ( 5-10 years ) Resource Adviser(s) Years applied Empirical/ (No stocks) Model-based SA south coast lobster Johnston 8 Emp Australia SE fisheries* Haddon/Smith 8 (7) Emp NAFO G-land halibut Chapman 6 Emp Canada BC sablefish Cox 6 Mod CCSBT bluefin tuna Parma 5 Mod East Canada pollock Chapman 5 Emp * Data-limited cases (Tier 4)

II. HOW HAVE MPs PERFORMED ? STARTING IMPLEMENTATIONS ( < 5 years ) Resource Adviser Years applied Empirical/ (No stocks) Model-based Tristan rock lobster Johnston 3 (3) Emp US – Lake Erie walleye Jones 3 Mod Canada Atlantic halibut Cox 2 Emp IWC W G-land humpbacks* Donovan 2 Emp IWC W G-land bowheads* Donovan 1 Emp * Aboriginal whaling: Satisfies need rather than maximises catch

II. HOW HAVE MPs PERFORMED ? DISCONTINUED IMPLEMENTATIONS Resource Reason NZ rock lobster (1 stock) Unstable CPUE Namibian hake Lack of capacity Namibian seals Lack of capacity DRAWING CONCLUSIONS For some of the questions posed, only some of these cases (the “lengthy implementations”) have been in place sufficiently long for conclusions to be drawn

II. HOW HAVE MPs PERFORMED ? PROVIDING A DEFENSIBLE BASIS TO LIMIT INTER-ANNUAL TAC CHANGES Such restrictions are included in virtually all cases (Except the whale procedures - though these incorporate implicit elements of this, e.g. block quotas) In a number of cases considerations of the associated trade-offs is a key element of interactions with stakeholders

II. HOW HAVE MPs PERFORMED ? SAVING ANALYSIS (AND DEBATE) TIME INTERVALS BETWEEN SCHEDULED REVIEWS 4-8 years – median 5 FRACTION OCCASIONS RESULTS MODIFIED Whales – 10% Lobster – 10% (species); 2% (stock) Others – 8% TIME SAVED South Africa – intermediate Others – substantial Two starting implementations – limited (but too early to judge?)

II. HOW HAVE MPs PERFORMED ? MORE EFFECTIVE BASIS FOR STAKEHOLDER INVOLVEMENT? EXTENT OF INVOLVEMENT Intermediate to Very high ACCEPTABILITY OF OUTPUTS High to Very high (SA west coast lobster: Intermediate) Buy-in effect? PRESSURES FOR CHANGES TO OUTPUTS Whales – Very low; Balance – Low (Exception - Tristan lobster) Basis – Desire for wriggle room PROVIDED IMPROVED RECOMMENDATION High to Very high

II. HOW HAVE MPs PERFORMED ? TESTING AND SELECTION EXTENT OF ROBUSTNESS TESTING Highly variable (very little to extensive) Tended to be more extensive in international fora PLAUSIBILITY WEIGHTING OF SCENARIOS Variable and generally poorly developed Best is IWC’s High/Medium/Low system with lower bars for conservation performance for lesser plausibility SELECTION BASIS (Risk) Virtually exclusively satisfying threshold for low percentile for spawning biomass (i.e. form of biomass LRP)

II. HOW HAVE MPs PERFORMED ? SUMMARY Feedback ability to correct for errors has limitations Tendency towards favouring Empirical over Model-based MPs (important when decisions are needed quickly for some short-lived species; more readily understood by stakeholders) TAC change limitations are important (and offset the need for optimal assessment model based smoothing of abundance indices) Regular reviews (~ 5-yearly) offset the need for the MP to provide a basis to enhance parameter estimate precision MP outputs-overturn frequency < 10% Time saving effect varies: intermediate to very high Stakeholders: involvement good and acceptance of outputs high Selection basis: stay above abundance LRP at high probability Testing for robustness to uncertainties is inconsistent

III. MPs (sensu stricto) vs MSE IWC SC PERSPECTIVE “Sensu stricto” is the only feasible approach Avoids the contested assessment impasse Forces away from inappropriate deterministic thinking (that “best assessment” = “truth”) Changing estimator from that tested can bias performance compared to what MP was designed to achieve SO HAVE ASSESSMENTS, OR EVEN MSE, REALLY ANYTHING TO OFFER?

III. MPs (sensu stricto) vs MSE DIFFICULTIES WITH “sensu stricto” Constraints from existing legislation (USA) Managers want wriggle room (Despite lower catches for the same perceived risk? – flexibility has its costs) Practicality – not possible to undertake case-specific MP testing for every low value resource MSE BENEFITS Checking appropriateness of (eg) HCR, survey frequency Some fora are able to achieve consensus on assessments NEVERTHELESS Are generic MSEs for data limited resources worth the effort to develop? Does adequate robustness imply unacceptably conservative?

IV. THE FUTURE CONTINUED EXPANSION (particularly tuna RFMOs) ICCAT: North Atlantic albacore and bluefin IOTC: Albacore, bigeye, skipjack and yellowfin US New England: Herring Australia: Torres Strait rock lobster IWC: West Greenland fin and minke whale SLAs OR WILL SOME OF THESE END UP AS MSEs RATHER THAN TRUE MPs? THE ECHEBASTAR ADJUDICATION Re MSC certification requirements: The absence of an HCR being in place (formally adopted) is a “fundamental, irremediable and fatal” flaw to certification.

V. DISCUSSION POINTS MSEs vs MPs sensu stricto vs Return to assessments Can results from generic MSEs provide a basis for tactical management advice, or does this require that the MP has been tested against operating models conditioned on data for the resource in question? (Mark Maunder) . The use of MSE depends on the nature of the management objectives. If they are simply to ensure that the stock fluctuates around a management target and stays away from an arbitrary limit reference point, then the operating model must be conditioned on the data otherwise the results are just a function of the arbitrary chosen operating models. However, if the management goals are more related to current status (i.e. stay where it is or always increase) or some arbitrary rebuilding target then generic rules may suffice.

V. DISCUSSION POINTS DROP F < FMSY This forces unnecessary TAC variability without adding any resource benefits, given that the MP will be tuned to achieve biomass targets FORGET REFERENCE POINTS APART FROM Blim Apart from this risk-related criterion, MPs should be selected based on trade-offs amongst performance statistics for conflicting objectives (eg average TAC maximisation vs TAC inter-annual change minimisation), not reference points Thus there is no need, eg, to force HCRs to relate to conventional reference points

Thank you for your attention With thanks for assistance/inputs to: Candysse Vrancken (feedback analyses) Those listed earlier who provided inputs on specific MPs For discussion/comments: Jim Bence; Dorothy Dankel; Campbell Davies; Jon Deroba; Helena Geromont; Laurie Kell; Dale Kolody; Mark Maunder; Eva Plaganyi; and Andre Punt