1. CANE 2007 Spring Meeting Visualizing Predictive Modeling Results Chuck Boucek (312) 879-3859

2. 1 Agenda Data Validation Hypothesis Building Model Building Model Testing Monitoring Visualization as a Diagnostic Tool Data Validation Hypothesis Building Model Building Model Testing Monitoring Visualization as a Diagnostic Tool

3. 2 Data Validation Goals –Validate reasonableness of data –Understand key patterns in data –Understand changes in data and underlying business through time Goals –Validate reasonableness of data –Understand key patterns in data –Understand changes in data and underlying business through time

4. 3 Data Validation Histogram is a simple tool to for reasonability testing of modeling database

5. 4 Data Validation Mosaic Plot shows the distribution of predictors in two dimensions

6. 5 Data Validation Missing Data plot shows the relationship of missing data elements

7. 6 Data Validation Time series plots identify consistency of data over time Claims Match to Exposure 19931994199519961997199819992000200120022003 2004 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Company 1 Company 2 Company 3

8. 7 Hypothesis Building Goals –Perform initial analysis of potential predictor variables –Limit the list of predictor variables to be employed in subsequent phases of model building –Further reasonability testing of data Goals –Perform initial analysis of potential predictor variables –Limit the list of predictor variables to be employed in subsequent phases of model building –Further reasonability testing of data

9. 8 Frequency 0.0 0.375 0.750 1.125 1.500 0 0.010.02 0.3 Severity 10000 17500 25000 32500 40000 0 0.010.02 0.3 Pure Premium 5000 7500 10000 12500 15000 0 0.001 0.010.02 0.3 Loss Ratio 0.250 0.375 0.500 0.625 0.750 0 0.010.02 0.3 0 2500 5000 7500 10000 Exposure ($MM) 0 0.010.02 0.3 0 50 100 150 200 Premium ($MM) 0 0.010.02 0.3 Demographic Variable 1 0.001

10. 9 Hypothesis Building Quantile-Quantile plots help identify needed transformations of data

11. 10 Hypothesis Building Correlation Web concisely summarizes a correlation matrix

12. 11 Model Building Model building is an iterative process Understanding patterns and relationships throughout this process is critical Model building is an iterative process Understanding patterns and relationships throughout this process is critical

13. 12 Model Building Partial Plots are a key tool to visualize predictor variables throughout the model building process What is a “Partial Plot?” Linear Predictor = k +  1 X 1 +  2 X 2 +  3 X 3 +  4 X 4 Predicted value = (e k ) x (e  1 X 1 ) x (e  2 X 2 ) x (e  3 X 3 ) x (e  4 X 4 ) Partial Plot demonstrates an individual predictor variables contribution to final prediction Partial Plots are a key tool to visualize predictor variables throughout the model building process What is a “Partial Plot?” Linear Predictor = k +  1 X 1 +  2 X 2 +  3 X 3 +  4 X 4 Predicted value = (e k ) x (e  1 X 1 ) x (e  2 X 2 ) x (e  3 X 3 ) x (e  4 X 4 ) Partial Plot demonstrates an individual predictor variables contribution to final prediction

14. 13 Model Building Partial Plot demonstrates an individual predictor variables contribution to final prediction

15. 14 Model Building Partial Plot with modified scatter plot of variable

16. 15 Model Building Time Consistency plot is a critical tool for numeric predictors

17. 16 Model Building Partial Plot for a factor variable 0.70 0.80 0.90 1.00 1.10 1.20 1.30 Credit Level 1 Credit Level 2

18. 17 Frequency 0.0 0.375 0.750 1.125 1.500 Severity 10000 17500 25000 32500 40000 No Yes Pure Premium 5000 7500 10000 12500 15000 Loss Ratio 0.250 0.375 0.500 0.625 0.750 0 2500 5000 7500 10000 Exposure (Pred. Count) 0 50 100 150 200 Premium ($MM) Credit Variable 1 No Yes No Yes No Yes No Yes No Yes

19. 18 Model Testing Likely the most critical visualizations in predictive modeling work –Management’s perception of a project’s success will likely depend on these visualizations Holdout tests Cross validation tests Likely the most critical visualizations in predictive modeling work –Management’s perception of a project’s success will likely depend on these visualizations Holdout tests Cross validation tests

20. 19 Model Testing Lift Chart shows overall model performance 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.4 0.5 0.6 0.7 0.8 0.7 1.0 Predicted Actual Loss Ratio Lift Chart - Holdout Sample

21. 20 Model Testing ROC Curve shows overall model performance Null, 0 Perfect, 1 prem, 0.51 pred.loss, 0.56 Holdout Sample ROC Curve

22. 21 Model Testing Classical Cross Validation exhibit Number of variables in final model Out of Sample Error In Sample Error Number of Predictors Prediction Error 510152025

23. 22 Monitoring Model Results The work does not end when the lift chart looks good Monitoring tools –Decile management –Exception analysis –Model vs. Actual Results The work does not end when the lift chart looks good Monitoring tools –Decile management –Exception analysis –Model vs. Actual Results

24. 23 Monitoring Model Results Decile Management –Retention –Loss Ratio –Rate Action –Tier/Schedule Mod Decile Management –Retention –Loss Ratio –Rate Action –Tier/Schedule Mod

25. 24 Monitoring Model Results Average score over time

26. 25 Monitoring Model Results Loss ratio of model exceptions

27. 26 Visualization as Diagnostic Tool Frequency and severity models have been developed Model is underperforming in predicting loss ratio Likely cause of underperformance is severity model Frequency and severity models have been developed Model is underperforming in predicting loss ratio Likely cause of underperformance is severity model

28. 27 Visualization as Diagnostic Tool

29. 28 Visualization as Diagnostic Tool

30. 29 Visualization as Diagnostic Tool

31. 30 Visualization as Diagnostic Tool Two different visualizations of the same model tell a very different story!