1. © 2007 Towers Perrin June 17, 2008 Loss Reserving: Performance Testing and the Control Cycle Casualty Actuarial Society Pierre Laurin

2. © 2007 Towers Perrin 2 Today’s agenda Defining the problem Performance testing — in general and in the context of reserves The reserving actuarial control cycle Case study — real-world results This presentation is based on the paper “Loss Reserving: Performance Testing and the Control Cycle”, authored by Yi Jing, Joseph Lebens, and Stephen Lowe, that has recently been submitted for publication in Variance

3. © 2007 Towers Perrin 3 Questions for the reserving actuary How do you know that the methods you are currently using are the “best”? —What evidence supports your selection of methods? —What are the right weights for combining the results of the methods? —How do you decide when to change methods? —What is the confidence range around estimates? —How do you evaluate the cost/benefit of developing new data sources or implementing more complex methods? —How do you instill hubris in junior actuaries? THE PROBLEM

4. © 2007 Towers Perrin 4 The results of our research illustrate the prevalence of actuarial overconfidence The Quiz Objective: To test respondents understanding of the limits of their knowledge Respondents were asked to answer ten questions related to their general knowledge of the global property/casualty industry For each answer, respondents were asked to provide a range that offered a 90% confidence interval that they would answer correctly Ideally (i.e., if “well calibrated”), respondents should have gotten nine out of ten questions correct Number of Respondents Raw Scores of Respondents Note: based on 374 respondents as of 4/5/04. Profile of respondents: 86% work in P/C industry; 73% are actuaries. Tillinghast Confidence Quiz THE PROBLEM

5. © 2007 Towers Perrin 5 Reserves are forecasts! An actuarial method is used to develop a forecast of future claim payments An actuarial method consists of An algorithm A data set A set of intervention points The actuary must 1. Choose a finite set of methods from the universe M 2. Choose a set of weights to combine the results of each method together Performance testing, via a formal control cycle, can help the actuary make these choices THE PROBLEM

6. © 2007 Towers Perrin 6 Performance testing is a formal analysis of prediction errors Test a particular method by running the method on historical data – comparing estimates from the method with actual run-off Performance testing is a formalized process, not just a numerical exercise PERFORMANCE TESTING

7. © 2007 Towers Perrin 7 The general framework for performance testing is cross-validation Cross-validation in the context of a regression model PERFORMANCE TESTING

8. © 2007 Towers Perrin 8 Criteria for assessing the performance of a reserving method: BLURS-ICE Least-Square Error The method should produce estimates that have minimum expected error, in the least squares sense. UnbiasedIdeally, the method should produce estimates that are unbiased. ResponsiveThe method should respond quickly to changes in underlying trends or operational changes. Given several methods with similar expected errors, we would favor the method that responds to changes in trends or operations most quickly. StableMethods that work well in some circumstances, but “blow up” in other circumstances are less desirable. IndependentThe method should not generate prediction errors that are highly correlated with those of other selected methods. (This criteria is discussed more fully in subsequent sections.) ComprehensiveThe method should use as much data as possible, in an integrated way, so that the maximum available information value is extracted. PERFORMANCE TESTING

9. © 2007 Towers Perrin 9 Performance testing of reserving methods can be part of an institutionalized control cycle The Actuarial Control Cycle for the Reserving Process THE CONTROL CYCLE

10. © 2007 Towers Perrin 10 Case Study: U.S. Insurer Commercial Auto BI data 1972 to 1998 accident years – June 30 th valuations Paid and incurred counts and amounts Estimates of claim liabilities from 1979 to 1998 – twenty years December 31 st valuation used as “actual ultimate” Environmental influences during the period add difficulty to estimation Economic and social inflation Operational changes in claim department Changes in underwriting posture CASE STUDY

11. © 2007 Towers Perrin 11 Skill can be measured by comparing actual to predicted unpaid loss ratios We could just use a paid B-F with a constant ELR and payment pattern every year. This would give us the dotted line as our estimate. We are looking for methods that do better than a crude, constant assumption CASE STUDY

12. © 2007 Towers Perrin 12 Formal measurement of skill The skill of a method is measured by: Where mse = mean squared error msa = mean squared anomaly Skill is the proportion of variance “explained” by the method CASE STUDY

13. © 2007 Towers Perrin 13 Actuarial methods subjected to performance testing Actuarial Projection Method Skill for Accident Year @ 42 Months Overall Skill – for Latest Ten Accident Years Paid Chain-Ladder23%13% Incurred Chain-Ladder52%32% Case Reserve Development60%22% Reported Count Chain-Ladder99% Case Adequacy Adjusted Incurred Chain-Ladder 52% CASE STUDY

14. © 2007 Towers Perrin 14 Skill can be measured by maturity Note that skill can be negative (e.g., paid loss projection method at 6 months), implying that it induces volatility rather than explaining it CASE STUDY

15. © 2007 Towers Perrin 15 The minimum-variance weighting of methods depends on their variances and their correlation For a given correlation, the optimal weights are those with the minimum combined variance Minimum starts at the very right, when correlation is 100% Minimum gradually shifts leftward as correlation decreases CASE STUDY

16. © 2007 Towers Perrin 16 Key conclusions from case study Accuracy, or skill, can be formally measured, using cross-validation approach Performance testing can guide the actuary in choosing methods, and selecting weights between methods; correlation is a relevant criteria in selecting methods and weights Chain-ladder methods are seriously degraded by changing claim policies and procedures; using claim counts to adjust improves skill Experimentation with non-traditional methods is worthwhile; case reserve development has higher skill at later maturities CASE STUDY

17. © 2007 Towers Perrin 17 Good reasons to do performance testing 1. Opportunity to improve accuracy of estimates 2. Formal rationale for selected actuarial methods 3. Empirical validation of stochastic reserve risk models 4. Input to development of reserve ranges 5. Manage actuarial overconfidence 6. Cost / benefit of enhancements to data and systems CONCLUSION