1. © Milliman, Inc. 2007. All Rights Reserved. Estimating Reserve Variability: When Numbers Get Serious! Mark R. Shapland, FCAS, ASA, MAAA 2007 Casualty Loss Reserve Seminar San Diego, California September 10-11, 2006

2. © Milliman, Inc. 2007. All Rights Reserved. Terminology Ranges vs. Distributions Methods vs. Models Types of Methods/Models What is “Reasonable?” Overview

3. © Milliman, Inc. 2007. All Rights Reserved. Reserve – an amount carried in the liability section of a risk-bearing entity’s balance sheet for claims incurred prior to a given accounting date. Liability – the actual amount that is owed and will ultimately be paid by a risk-bearing entity for claims incurred prior to a given accounting date. Loss Liability – the expected value of all estimated future claim payments. Risk (from the “risk-bearers” point of view) – the uncertainty (deviations from expected) in both timing and amount of the future claim payment stream. Terminology

4. © Milliman, Inc. 2007. All Rights Reserved. Process Risk – the randomness of future outcomes given a known distribution of possible outcomes. Parameter Risk – the potential error in the estimated parameters used to describe the distribution of possible outcomes, assuming the process generating the outcomes is known. Model Risk – the chance that the model (“process”) used to estimate the distribution of possible outcomes is incorrect or incomplete. Terminology

5. © Milliman, Inc. 2007. All Rights Reserved. Risk – unknown outcomes with quantifiable probabilities. Uncertainty – unknown outcomes that cannot be estimated or quantified. All entrepreneurship involves both risk (which can be transferred) and uncertainty (which cannot be transferred). Terminology Source: Knight, Frank H. 1921. “Risk, Uncertainty, and Profit.” Houghton Mifflin.

6. © Milliman, Inc. 2007. All Rights Reserved. Variance, standard deviation, kurtosis, average absolute deviation, Value at Risk, Tail Value at Risk, etc. which are measures of dispersion. Other measures useful in determining “reasonableness” could include: mean, mode, median, pain function, etc. The choice for measure of risk will also be important when considering the “reasonableness” and “materiality” of the reserves in relation to the capital position. Terminology Measures of Risk from Statistics:

7. © Milliman, Inc. 2007. All Rights Reserved. A “Range” is not the same as a “Distribution” A Range of Reasonable Estimates is a range of estimates that could be produced by appropriate actuarial methods or alternative sets of assumptions that the actuary judges to be reasonable. A Distribution is a statistical function that attempts to quantify probabilities of all possible outcomes. Ranges vs. Distributions

8. © Milliman, Inc. 2007. All Rights Reserved. Ranges vs. Distributions A Range, by itself, creates problems: A range can be misleading to the layperson – it can give the impression that any number in that range is equally likely. A range can give the impression that as long as the carried reserve is “within the range” anything is reasonable.

9. © Milliman, Inc. 2007. All Rights Reserved. Ranges vs. Distributions A Range, by itself, creates problems: There is currently no specific guidance on how to consistently determine a range within the actuarial community (e.g., +/- X%, +/- $X, using various estimates, etc.). A range, in and of itself, needs some other context to help define it (e.g., how to you calculate a risk margin?)

10. © Milliman, Inc. 2007. All Rights Reserved. Ranges vs. Distributions A Distribution provides: Information about “all” possible outcomes. Context for defining a variety of other measures (e.g., risk margin, materiality, risk based capital, etc.)

11. © Milliman, Inc. 2007. All Rights Reserved. Ranges vs. Distributions Should we use the same: criterion for judging the quality of a range vs. a distribution? basis for determining materiality? risk margins? selection process for which numbers are “reasonable” to chose from?

12. © Milliman, Inc. 2007. All Rights Reserved. Ranges vs. Distributions A Distribution can be used for: Risk-Based / Economic Capital – Reserve Risk – Pricing Risk – ALM Risk Pricing / ROE Reinsurance Analysis – Quota Share – Aggregate Excess – Stop Loss – Loss Portfolio Transfer Dynamic Risk Modeling (DFA) Strategic Planning / ERM Allocated Capital Risk Transfer

13. © Milliman, Inc. 2007. All Rights Reserved. Equally Likely to be High / Low? What Reserve to Book? Mode Median Expected Value 50 th Percentile

14. © Milliman, Inc. 2007. All Rights Reserved. Equally Likely to be High / Low? What Reserve to Book? Mode Median Expected Value 50 th Percentile

15. © Milliman, Inc. 2007. All Rights Reserved. Enough On Average? What Reserve to Book? Mode Median Expected Value 50 th Percentile

16. © Milliman, Inc. 2007. All Rights Reserved. Which Risks are Included? What Reserve to Book? Mode Median Expected Value 50 th Percentile

17. © Milliman, Inc. 2007. All Rights Reserved. A Method is an algorithm or recipe – a series of steps that are followed to give an estimate of future payments. The well known chain ladder (CL) and Bornhuetter-Ferguson (BF) methods are examples Methods vs. Models

18. © Milliman, Inc. 2007. All Rights Reserved. A Model specifies statistical assumptions about the loss process, usually leaving some parameters to be estimated. Then estimating the parameters gives an estimate of the ultimate losses and some statistical properties of that estimate. Methods vs. Models

19. © Milliman, Inc. 2007. All Rights Reserved. Methods vs. Models Many good probability models have been built using “Collective Risk Theory” Each of these models make assumptions about the processes that are driving claims and their settlement values None of them can ever completely eliminate “model risk” “All models are wrong. Some models are useful.”

20. © Milliman, Inc. 2007. All Rights Reserved. Types of Models Individual Claim Models Triangle Based Models vs. Conditional ModelsUnconditional Models vs. Single Triangle ModelsMultiple Triangle Models vs. Parametric ModelsNon-Parametric Models vs. Diagonal TermNo Diagonal Term vs. Fixed ParametersVariable Parameters vs.

21. © Milliman, Inc. 2007. All Rights Reserved. Methods vs. Models Processes used to calculate liability ranges can be grouped into four general categories: 1) Multiple Projection Methods, 2) Statistics from Link Ratio Models, 3) Incremental Models, and 4) Simulation Models

22. © Milliman, Inc. 2007. All Rights Reserved. Multiple Projection Methods Description: Uses multiple methods, data, assumptions Assume various estimates are a good proxy for the variation of the expected outcomes Primary Advantages: Better than no range at all Better than +/- X%

23. © Milliman, Inc. 2007. All Rights Reserved. Multiple Projection Methods Problems: It does not provide a measure of the density of the distribution for the purpose of producing a probability function The “distribution” of the estimates is a distribution of the methods and assumptions used, not a distribution of the expected future claim payments. Link ratio methods only produce a single point estimate and there is no statistical process for determining if this point estimate is close to the expected value of the distribution of possible outcomes or not.

24. © Milliman, Inc. 2007. All Rights Reserved. Multiple Projection Methods Problems: Since there are no statistical measures for these models, any overall distribution for all lines of business combined will be based on the addition of the individual ranges by line of business with judgmental adjustments for covariance, if any.

25. © Milliman, Inc. 2007. All Rights Reserved. Multiple Projection Methods Uses: Data limitations may prevent the use of more advanced models. A strict interpretation of the guidelines in ASOP No. 36 seems to imply the use of this “method” to create a “reasonable” range

26. © Milliman, Inc. 2007. All Rights Reserved. Statistics from Link Ratio Models Description: Calculate standard error for link ratios to calculate distribution of outcomes / range Typically assume normality and use logs to get a skewed distribution Examples: Mack, Murphy, Bootstrapping and others Primary Advantages: Significant improvement over multiple projections Focused on a distribution of possible outcomes

27. © Milliman, Inc. 2007. All Rights Reserved. Statistics from Link Ratio Models Problems: The expected value often based on multiple methods Often assume link ratio errors are normally distributed and constant by (accident) year – this violates three criterion Provides a process for calculating an overall probability distribution for all lines of business combined, still requires assumptions about the covariances between lines

28. © Milliman, Inc. 2007. All Rights Reserved. Statistics from Link Ratio Models Uses: If data limitations prevent the use of more sophisticated models Caveats: Need to make sure statistical tests are satisfied. ASOP No. 36 still applies to the expected value portion of the calculations

29. © Milliman, Inc. 2007. All Rights Reserved. Incremental Models Description: Directly model distribution of incremental claims Typically assume lognormal or other skewed distribution Examples: Bootstrapping, Finger, Hachmeister, Zehnwirth, England, Verrall and others Primary Advantages: Overcome the “limitations” of using cumulative values Modeling of calendar year inflation (along the diagonal)

30. © Milliman, Inc. 2007. All Rights Reserved. Incremental Models Problems: Actual distribution of incremental payments may not be lognormal, but other skewed distributions generally add complexity to the formulations Correlations between lines will need to be considered when they are combined (but can usually be directly estimated) Main limitation to these models seems to be only when some data issues are present

31. © Milliman, Inc. 2007. All Rights Reserved. Incremental Models Uses: Usually, they allow the actuary to tailor the model parameters to fit the characteristics of the data. An added bonus is that some of these models allow the actuary to thoroughly test the model parameters and assumptions to see if they are supported by the data. They also allow the actuary to compare various goodness of fit statistics to evaluate the reasonableness of different models and/or different model parameters.

32. © Milliman, Inc. 2007. All Rights Reserved. Simulation Models Description: Dynamic risk model of the complex interactions between claims, reinsurance, surplus, etc., Models from other groups can be used to create such a risk model Primary Advantage: Can generate a robust estimate of the distribution of possible outcomes

33. © Milliman, Inc. 2007. All Rights Reserved. Simulation Models Problems: Models based on link ratios often exhibit statistical properties not found in the real data being modeled. Usually overcome with models based on incremental values or with ground-up simulations using separate parameters for claim frequency, severity, closure rates, etc. As with any model, the key is to make sure the model and model parameters are a close reflection of reality.

34. © Milliman, Inc. 2007. All Rights Reserved. 150100901151009750 th to 75 th Percentile 1201008012010080Expected +/- 20% HighEVLowHighEVLowMethod More Volatile LOBRelatively Stable LOB What is “Reasonable”? Comparison of “Reasonable” Reserve Ranges by Method

35. © Milliman, Inc. 2007. All Rights Reserved. What is “Reasonable”? Comparison of “Normal” vs. “Skewed” Liability Distributions

36. © Milliman, Inc. 2007. All Rights Reserved. What is “Reasonable”? LOB “A” LOB “B” LOB “C” Aggregate Distribution with 100% Correlation (Added) Aggregate Distribution with 0% Correlation (Independent) Aggregate Distributions

37. © Milliman, Inc. 2007. All Rights Reserved. What is “Reasonable”? Aggregate Distribution with 100% Correlation (Added) Aggregate Distribution with 0% Correlation (Independent) Expected Value 99 th Percentile Capital = 1,000M Capital = 600M Aggregate Distributions

38. © Milliman, Inc. 2007. All Rights Reserved. What is “Reasonable”? $11M $16M

39. © Milliman, Inc. 2007. All Rights Reserved. Mode Median Expected Value 50 th Percentile 75 th Percentile What is “Reasonable”? $11M $16M

40. © Milliman, Inc. 2007. All Rights Reserved. What is “Reasonable”? $11M $16M Mode Median Expected Value 50 th Percentile 75 th Percentile

41. © Milliman, Inc. 2007. All Rights Reserved. What is “Reasonable”? Reasonable Range Reasonable Distribution

42. © Milliman, Inc. 2007. All Rights Reserved. Model Selection and Evaluation Actuaries Have Built Many Sophisticated Models Based on Collective Risk Theory All Models Make Simplifying Assumptions How do we Evaluate Them?

43. © Milliman, Inc. 2007. All Rights Reserved. Liability Estimates Probability How Do We “Evaluate”?

44. © Milliman, Inc. 2007. All Rights Reserved. Fundamental Questions How Well Does the Model Measure and Reflect the Uncertainty Inherent in the Data? Does the Model do a Good Job of Capturing and Replicating the Statistical Features Found in the Data?

45. © Milliman, Inc. 2007. All Rights Reserved. Modeling Goals Is the Goal to Minimize the Range (or Uncertainty) that Results from the Model? Goal of Modeling is NOT to Minimize Process Uncertainty! Goal is to Find the Best Statistical Model, While Minimizing Parameter and Model Uncertainty.

46. © Milliman, Inc. 2007. All Rights Reserved. Model Selection & Evaluation Criteria Model Selection Criteria Model Reasonability Checks Goodness-of-Fit & Prediction Errors

47. © Milliman, Inc. 2007. All Rights Reserved. Model Selection Criteria Criterion 1: Aims of the Analysis – Will the Procedure Achieve the Aims of the Analysis? Criterion 2: Data Availability – Access to the Required Data Elements? – Unit Record-Level Data or Summarized “Triangle” Data?

48. © Milliman, Inc. 2007. All Rights Reserved. Model Selection Criteria Criterion 3: Non-Data Specific Modeling Technique Evaluation – Has Procedure been Validated Against Historical Data? – Verified to Perform Well Against Dataset with Similar Features? – Assumptions of the Model Plausible Given What is Known About the Process Generating this Data?

49. © Milliman, Inc. 2007. All Rights Reserved. Model Selection Criteria Criterion 4: Cost/Benefit Considerations – Can Analysis be Performed Using Widely Available Software? – Analyst Time vs. Computer Time? – How Difficult to Describe to Junior Staff, Senior Management, Regulators, Auditors, etc.?

50. © Milliman, Inc. 2007. All Rights Reserved. Model Reasonability Checks Criterion 5: Coefficient of Variation by Year – Should be Largest for Oldest (Earliest) Year Criterion 6: Standard Error by Year – Should be Smallest for Oldest (Earliest) Year (on a Dollar Scale)

51. © Milliman, Inc. 2007. All Rights Reserved. Model Reasonability Checks Criterion 7: Overall Coefficient of Variation – Should be Smaller for All Years Combined than any Individual Year Criterion 8: Overall Standard Error – Should be Larger for All Years Combined than any Individual Year

52. © Milliman, Inc. 2007. All Rights Reserved. Model Reasonability Checks

53. © Milliman, Inc. 2007. All Rights Reserved. Model Reasonability Checks

54. © Milliman, Inc. 2007. All Rights Reserved. Model Reasonability Checks Criterion 9: Correlated Standard Error & Coefficient of Variation – Should Both be Smaller for All LOBs Combined than the Sum of Individual LOBs Criterion 10: Reasonability of Model Parameters and Development Patterns – Is Loss Development Pattern Implied by Model Reasonable?

55. © Milliman, Inc. 2007. All Rights Reserved. Model Reasonability Checks Criterion 11: Consistency of Simulated Data with Actual Data – Can you Distinguish Simulated Data from Real Data? Criterion 12: Model Completeness and Consistency – Is it Possible Other Data Elements or Knowledge Could be Integrated for a More Accurate Prediction?

56. © Milliman, Inc. 2007. All Rights Reserved. Goodness-of-Fit & Prediction Errors Criterion 13: Validity of Link Ratios – Link Ratios are a Form of Regression and Can be Tested Statistically Criterion 14: Standardization of Residuals – Standardized Residuals Should be Checked for Normality, Outliers, Heteroscedasticity, etc.

57. © Milliman, Inc. 2007. All Rights Reserved. Goodness-of-Fit & Prediction Errors

58. © Milliman, Inc. 2007. All Rights Reserved. Goodness-of-Fit & Prediction Errors

59. © Milliman, Inc. 2007. All Rights Reserved. Goodness-of-Fit & Prediction Errors

60. © Milliman, Inc. 2007. All Rights Reserved. Standardized Residuals

61. © Milliman, Inc. 2007. All Rights Reserved. Standardized Residuals

62. © Milliman, Inc. 2007. All Rights Reserved. Goodness-of-Fit & Prediction Errors Criterion 15: Analysis of Residual Patterns – Check Against Accident, Development and Calendar Periods Criterion 16: Prediction Error and Out-of-Sample Data – Test the Accuracy of Predictions on Data that was Not Used to Fit the Model

63. © Milliman, Inc. 2007. All Rights Reserved. Standardized Residuals

64. © Milliman, Inc. 2007. All Rights Reserved. Standardized Residuals

65. © Milliman, Inc. 2007. All Rights Reserved. Goodness-of-Fit & Prediction Errors Criterion 17: Goodness-of-Fit Measures – Quantitative Measures that Enable One to Find Optimal Tradeoff Between Minimizing Model Bias and Predictive Variance Adjusted Sum of Squared Errors (SSE) Akaike Information Criterion (AIC) Bayesian Information Criterion (BIC)

66. © Milliman, Inc. 2007. All Rights Reserved. Goodness-of-Fit & Prediction Errors Criterion 18: Ockham’s Razor and the Principle of Parsimony – All Else Being Equal, the Simpler Model is Preferable Criterion 19: Model Validation – Systematically Remove Last Several Diagonals and Make Same Forecast of Ultimate Values Without the Excluded Data

67. © Milliman, Inc. 2007. All Rights Reserved. Questions?