Download presentation
Presentation is loading. Please wait.
Published byLorena Sullivan Modified over 9 years ago
1
CANE 2007 Spring Meeting Visualizing Predictive Modeling Results Chuck Boucek (312) 879-3859
2
1 Agenda Data Validation Hypothesis Building Model Building Model Testing Monitoring Visualization as a Diagnostic Tool Data Validation Hypothesis Building Model Building Model Testing Monitoring Visualization as a Diagnostic Tool
3
2 Data Validation Goals –Validate reasonableness of data –Understand key patterns in data –Understand changes in data and underlying business through time Goals –Validate reasonableness of data –Understand key patterns in data –Understand changes in data and underlying business through time
4
3 Data Validation Histogram is a simple tool to for reasonability testing of modeling database
5
4 Data Validation Mosaic Plot shows the distribution of predictors in two dimensions
6
5 Data Validation Missing Data plot shows the relationship of missing data elements
7
6 Data Validation Time series plots identify consistency of data over time Claims Match to Exposure 19931994199519961997199819992000200120022003 2004 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Company 1 Company 2 Company 3
8
7 Hypothesis Building Goals –Perform initial analysis of potential predictor variables –Limit the list of predictor variables to be employed in subsequent phases of model building –Further reasonability testing of data Goals –Perform initial analysis of potential predictor variables –Limit the list of predictor variables to be employed in subsequent phases of model building –Further reasonability testing of data
9
8 Frequency 0.0 0.375 0.750 1.125 1.500 0 0.010.02 0.3 Severity 10000 17500 25000 32500 40000 0 0.010.02 0.3 Pure Premium 5000 7500 10000 12500 15000 0 0.001 0.010.02 0.3 Loss Ratio 0.250 0.375 0.500 0.625 0.750 0 0.010.02 0.3 0 2500 5000 7500 10000 Exposure ($MM) 0 0.010.02 0.3 0 50 100 150 200 Premium ($MM) 0 0.010.02 0.3 Demographic Variable 1 0.001
10
9 Hypothesis Building Quantile-Quantile plots help identify needed transformations of data
11
10 Hypothesis Building Correlation Web concisely summarizes a correlation matrix
12
11 Model Building Model building is an iterative process Understanding patterns and relationships throughout this process is critical Model building is an iterative process Understanding patterns and relationships throughout this process is critical
13
12 Model Building Partial Plots are a key tool to visualize predictor variables throughout the model building process What is a “Partial Plot?” Linear Predictor = k + 1 X 1 + 2 X 2 + 3 X 3 + 4 X 4 Predicted value = (e k ) x (e 1 X 1 ) x (e 2 X 2 ) x (e 3 X 3 ) x (e 4 X 4 ) Partial Plot demonstrates an individual predictor variables contribution to final prediction Partial Plots are a key tool to visualize predictor variables throughout the model building process What is a “Partial Plot?” Linear Predictor = k + 1 X 1 + 2 X 2 + 3 X 3 + 4 X 4 Predicted value = (e k ) x (e 1 X 1 ) x (e 2 X 2 ) x (e 3 X 3 ) x (e 4 X 4 ) Partial Plot demonstrates an individual predictor variables contribution to final prediction
14
13 Model Building Partial Plot demonstrates an individual predictor variables contribution to final prediction
15
14 Model Building Partial Plot with modified scatter plot of variable
16
15 Model Building Time Consistency plot is a critical tool for numeric predictors
17
16 Model Building Partial Plot for a factor variable 0.70 0.80 0.90 1.00 1.10 1.20 1.30 Credit Level 1 Credit Level 2
18
17 Frequency 0.0 0.375 0.750 1.125 1.500 Severity 10000 17500 25000 32500 40000 No Yes Pure Premium 5000 7500 10000 12500 15000 Loss Ratio 0.250 0.375 0.500 0.625 0.750 0 2500 5000 7500 10000 Exposure (Pred. Count) 0 50 100 150 200 Premium ($MM) Credit Variable 1 No Yes No Yes No Yes No Yes No Yes
19
18 Model Testing Likely the most critical visualizations in predictive modeling work –Management’s perception of a project’s success will likely depend on these visualizations Holdout tests Cross validation tests Likely the most critical visualizations in predictive modeling work –Management’s perception of a project’s success will likely depend on these visualizations Holdout tests Cross validation tests
20
19 Model Testing Lift Chart shows overall model performance 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.4 0.5 0.6 0.7 0.8 0.7 1.0 Predicted Actual Loss Ratio Lift Chart - Holdout Sample
21
20 Model Testing ROC Curve shows overall model performance Null, 0 Perfect, 1 prem, 0.51 pred.loss, 0.56 Holdout Sample ROC Curve
22
21 Model Testing Classical Cross Validation exhibit Number of variables in final model Out of Sample Error In Sample Error Number of Predictors Prediction Error 510152025
23
22 Monitoring Model Results The work does not end when the lift chart looks good Monitoring tools –Decile management –Exception analysis –Model vs. Actual Results The work does not end when the lift chart looks good Monitoring tools –Decile management –Exception analysis –Model vs. Actual Results
24
23 Monitoring Model Results Decile Management –Retention –Loss Ratio –Rate Action –Tier/Schedule Mod Decile Management –Retention –Loss Ratio –Rate Action –Tier/Schedule Mod
25
24 Monitoring Model Results Average score over time
26
25 Monitoring Model Results Loss ratio of model exceptions
27
26 Visualization as Diagnostic Tool Frequency and severity models have been developed Model is underperforming in predicting loss ratio Likely cause of underperformance is severity model Frequency and severity models have been developed Model is underperforming in predicting loss ratio Likely cause of underperformance is severity model
28
27 Visualization as Diagnostic Tool
29
28 Visualization as Diagnostic Tool
30
29 Visualization as Diagnostic Tool
31
30 Visualization as Diagnostic Tool Two different visualizations of the same model tell a very different story!
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.