1. AN HONEST APPRAISAL OF MANAGEMENT STRATEGY EVALUATION (MSE)
Doug S Butterworth
MARAM (Marine Resource Assessment and Management Group)
Department of Mathematics and Applied Mathematics
University of Cape Town, Rondebosch 7701, South Africa

2. OUTLINE I. DEFINITIONS AND BACKGROUND II. HOW HAVE MANAGEMENT
PROCEDURES (MPs) PERFORMED ?
Feedback control
TAC change constraints
Saving time
Stakeholder involvement
Testing and selection
Summary
III. MPs vs MSE
IV. THE FUTURE
V. DISCUSSION POINTS

3. I. DEFINITIONS AND BACKGROUND
MANAGEMENT PROCEDURE (MP):
The combination of pre-defined data, together with an algorithm to which such data are input to provide a value for a TAC or effort control measure; this combination has been demonstrated, through simulation trials, to show robust performance in the presence of uncertainties
TWO TYPES: Empirical (direct use of data)
Model-based (Estimator + Harvest Control Rule)
MANAGEMENT STRATEGY EVALUATION (MSE):
Usually synonymous with the MP approach; also used to describe the process of testing generic MPs or harvest strategies
Definitions from Glossary of “MP Terminology” in Rademeyer et al. (2007) ICES Journal of Marine Science, 64: 618–625.

4. I. DEFINITIONS AND BACKGROUND
MANAGEMENT PROCEDURES (MPs)
A very specific approach to fisheries management developed in the International Whaling Commission’s (IWC’s) Scientific Committee in the late 1980s to address problems with the conventional “best assessment + harvest control rule (HCR)” approach for providing Total Allowable Catch (TAC) advice
MANAGEMENT STRATEGY EVALUATION (MSE)
A similar but more generic approach developed around the same period in Australia

5. IWC NEW MANAGEMENT PROCEDURE (NMP) 1976
MSY
0.9 MSY C
0.54K
0.6K K
Harvest Control Rule:
C = 0 for P < 0.54 K
C = 0.9 MSY for P > 0.60 K
Input required to calculate C:
P: current abundance
K: pristine abundance
MSY

6. 1980s: FAILURE OF THE NMP IWC: LESSONS LEARNT AND SOLUTION
How to calculate P, K and MSY?
How to take uncertainties into account?
IWC: LESSONS LEARNT AND SOLUTION
An HCR alone is not enough
The Estimator (assessment method) must also be pre- specified
Furthermore the data to be input to that estimator must also be pre-specified
i.e. Move to a “Management Procedure Approach”

7. APPROACH FOR THIS TALK
EVALUATE THE PERFORMANCE OF THE APPLICATION OF MPs “sensu stricto”
i.e. Ones following the IWC approach where the estimator that is applied in practice is the same as the one simulation tested, and on which selection was based
“MSE COMPARISONS” DISCUSSED BRIEFLY LATER
e.g. Cases where the HCR is selected on the basis of simulation tests, but a current “best assessment” may be used with that HCR rather than the specific estimator tested
NOTE THAT GLOBALLY THE TERM “MSE” IS USED WITH A WIDER VARIETY OF MEANINGS THAN IS THE CASE FOR “MP”

8. II. HOW HAVE MPs PERFORMED ?
KEY CLAIMS MADE FOR MPs
COMPARED TO THE “BEST ASSESSMENT” APPROACH TO TAC PROVISION, MPs:
Utilise Feedback to address uncertainties effectively
Provide a defensible basis to limit inter-annual TAC changes
Save analysis time
Provide a more effective basis to involve stakeholders in the decision making process

9. HOW EFFECTIVE IS FEEDBACK CONTROL?
EXAMPLE FISHERY
30 years of catch + abundance index (CV=20%) data
Dynamics governed by Schaefer model
B(y+1) = B(y) + r B(y) [1 – B(y)/B0] – C(y)

10. FEEDBACK: EXAMPLE FISHERY
Manage from year 31 under FMSY = r̂ /2:
C(y) = 0.5 r̂ B̂(y) with estimation using Schaefer model

11. FEEDBACK: EXAMPLE FISHERY
How well does feedback correct for unreported overcatch of a fixed percentage from year 31?

12. FEEDBACK: EXAMPLE FISHERY
How well does feedback correct for unreported overcatch of a fixed percentage from year 31?

13. FEEDBACK: EXAMPLE FISHERY
How well does feedback correct for unreported overcatch of a fixed percentage from year 31?

14. FEEDBACK: EXAMPLE FISHERY
How well does feedback correct for unreported overcatch of a fixed percentage from year 31?

15. FEEDBACK: EXAMPLE FISHERY
How well does feedback correct for unreported overcatch of a fixed percentage from year 31?

16. FEEDBACK: EXAMPLE FISHERY
How well does feedback correct for unreported overcatch of a fixed percentage from year 31?

17. FEEDBACK: EXAMPLE FISHERY
So how robust was this MP to catch under-reporting?
For small extent, feedback corrected well and quickly
BUT
For large extent, correction was limited and slow
FEEDBACK HELPS, BUT NOT ALWAYS PERFECTLY

18. II. HOW HAVE MPs PERFORMED ?
KEY CLAIMS MADE FOR MPs
Provide a defensible basis to limit inter-annual TAC changes
Save analysis time
Provide a more effective basis to involve stakeholders in the decision making process
ADDRESS THESE THROUGH A SURVEY OF “IMPLEMENTERS” (usually the developer scientists)
Note that this does introduce a potential bias – “developers” may stray towards “over-promoting” the successes of their endeavours

19. II. HOW HAVE MPs PERFORMED ?
LENGTHY IMPLEMENTATIONS ( > 10 years )
Resource Adviser(s) Years applied Empirical/
(No stocks) Model-based
SA hake Rademeyer Emp
SA sardine/anchovy de Moor Emp
Norway minke whale Walloe Mod
NZ rock lobster Breen/Starr (8) Emp
SA west coast lobster Johnston (5) Emp
IWC BCB bowheads* Donovan Mod
IWC ENP gray* Donovan Mod
* Aboriginal whaling: Satisfies need rather than maximises catch

20. II. HOW HAVE MPs PERFORMED ?
MEDIUM IMPLEMENTATIONS ( 5-10 years )
Resource Adviser(s) Years applied Empirical/
(No stocks) Model-based
SA south coast lobster Johnston Emp
Australia SE fisheries* Haddon/Smith (7) Emp
NAFO G-land halibut Chapman Emp
Canada BC sablefish Cox Mod
CCSBT bluefin tuna Parma Mod
East Canada pollock Chapman Emp
* Data-limited cases (Tier 4)

21. II. HOW HAVE MPs PERFORMED ?
STARTING IMPLEMENTATIONS ( < 5 years )
Resource Adviser Years applied Empirical/
(No stocks) Model-based
Tristan rock lobster Johnston (3) Emp
US – Lake Erie walleye Jones Mod
Canada Atlantic halibut Cox Emp
IWC W G-land humpbacks* Donovan Emp
IWC W G-land bowheads* Donovan Emp
* Aboriginal whaling: Satisfies need rather than maximises catch

22. II. HOW HAVE MPs PERFORMED ?
DISCONTINUED IMPLEMENTATIONS
Resource Reason
NZ rock lobster (1 stock) Unstable CPUE
Namibian hake Lack of capacity
Namibian seals Lack of capacity
DRAWING CONCLUSIONS
For some of the questions posed, only some of these cases (the “lengthy implementations”) have been in place sufficiently long for conclusions to be drawn

23. II. HOW HAVE MPs PERFORMED ?
PROVIDING A DEFENSIBLE BASIS TO LIMIT INTER-ANNUAL TAC CHANGES
Such restrictions are included in virtually all cases (Except the whale procedures - though these incorporate implicit elements of this, e.g. block quotas)
In a number of cases considerations of the associated trade-offs is a key element of interactions with stakeholders

24. II. HOW HAVE MPs PERFORMED ?
SAVING ANALYSIS (AND DEBATE) TIME
INTERVALS BETWEEN SCHEDULED REVIEWS
4-8 years – median 5
FRACTION OCCASIONS RESULTS MODIFIED
Whales – 10%
Lobster – 10% (species); 2% (stock)
Others – 8%
TIME SAVED
South Africa – intermediate
Others – substantial
Two starting implementations – limited (but too early to judge?)

25. II. HOW HAVE MPs PERFORMED ?
MORE EFFECTIVE BASIS FOR STAKEHOLDER INVOLVEMENT?
EXTENT OF INVOLVEMENT
Intermediate to Very high
ACCEPTABILITY OF OUTPUTS
High to Very high (SA west coast lobster: Intermediate)
Buy-in effect?
PRESSURES FOR CHANGES TO OUTPUTS
Whales – Very low; Balance – Low (Exception - Tristan lobster)
Basis – Desire for wriggle room
PROVIDED IMPROVED RECOMMENDATION
High to Very high

26. II. HOW HAVE MPs PERFORMED ?
TESTING AND SELECTION
EXTENT OF ROBUSTNESS TESTING
Highly variable (very little to extensive)
Tended to be more extensive in international fora
PLAUSIBILITY WEIGHTING OF SCENARIOS
Variable and generally poorly developed
Best is IWC’s High/Medium/Low system with lower bars for
conservation performance for lesser plausibility
SELECTION BASIS (Risk)
Virtually exclusively satisfying threshold for low percentile for
spawning biomass (i.e. form of biomass LRP)

27. II. HOW HAVE MPs PERFORMED ?
SUMMARY
Feedback ability to correct for errors has limitations
Tendency towards favouring Empirical over Model-based MPs (important when decisions are needed quickly for some short-lived species; more readily understood by stakeholders)
TAC change limitations are important (and offset the need for optimal assessment model based smoothing of abundance indices)
Regular reviews (~ 5-yearly) offset the need for the MP to provide a basis to enhance parameter estimate precision
MP outputs-overturn frequency < 10%
Time saving effect varies: intermediate to very high
Stakeholders: involvement good and acceptance of outputs high
Selection basis: stay above abundance LRP at high probability
Testing for robustness to uncertainties is inconsistent

28. III. MPs (sensu stricto) vs MSE
IWC SC PERSPECTIVE
“Sensu stricto” is the only feasible approach
Avoids the contested assessment impasse
Forces away from inappropriate deterministic thinking (that “best assessment” = “truth”)
Changing estimator from that tested can bias performance compared to what MP was designed to achieve
SO HAVE ASSESSMENTS, OR EVEN MSE, REALLY ANYTHING TO OFFER?

29. III. MPs (sensu stricto) vs MSE
DIFFICULTIES WITH “sensu stricto”
Constraints from existing legislation (USA)
Managers want wriggle room
(Despite lower catches for the same perceived risk? – flexibility has its costs)
Practicality – not possible to undertake case-specific MP testing for every low value resource
MSE BENEFITS
Checking appropriateness of (eg) HCR, survey frequency
Some fora are able to achieve consensus on assessments
NEVERTHELESS
Are generic MSEs for data limited resources worth the effort to develop? Does adequate robustness imply unacceptably conservative?

30. IV. THE FUTURE CONTINUED EXPANSION (particularly tuna RFMOs)
ICCAT: North Atlantic albacore and bluefin
IOTC: Albacore, bigeye, skipjack and yellowfin
US New England: Herring
Australia: Torres Strait rock lobster
IWC: West Greenland fin and minke whale SLAs
OR WILL SOME OF THESE END UP AS MSEs RATHER THAN TRUE MPs?
THE ECHEBASTAR ADJUDICATION
Re MSC certification requirements:
The absence of an HCR being in place (formally adopted) is a “fundamental, irremediable and fatal” flaw to certification.

31. V. DISCUSSION POINTS
MSEs vs MPs sensu stricto vs Return to assessments
Can results from generic MSEs provide a basis for tactical management advice, or does this require that the MP has been tested against operating models conditioned on data for the resource in question? (Mark Maunder)
. The use of MSE depends on the nature of the management objectives. If they are simply to ensure that the stock fluctuates around a management target and stays away from an arbitrary limit reference point, then the operating model must be conditioned on the data otherwise the results are just a function of the arbitrary chosen operating models. However, if the management goals are more related to current status (i.e. stay where it is or always increase) or some arbitrary rebuilding target then generic rules may suffice.

32. V. DISCUSSION POINTS DROP F < FMSY
This forces unnecessary TAC variability without adding any resource benefits, given that the MP will be tuned to achieve biomass targets
FORGET REFERENCE POINTS APART FROM Blim
Apart from this risk-related criterion, MPs should be selected based on trade-offs amongst performance statistics for conflicting objectives (eg average TAC maximisation vs TAC inter-annual change minimisation), not reference points
Thus there is no need, eg, to force HCRs to relate to conventional reference points

33. Thank you for your attention
With thanks for assistance/inputs to:
Candysse Vrancken (feedback analyses)
Those listed earlier who provided inputs on specific MPs
For discussion/comments:
Jim Bence; Dorothy Dankel; Campbell Davies;
Jon Deroba; Helena Geromont; Laurie Kell;
Dale Kolody; Mark Maunder; Eva Plaganyi; and
Andre Punt