1. EFFECTIVE PREDICTIVE MODELING- DATA,ANALYTICS AND PRACTICE MANAGEMENT Richard A. Derrig Ph.D. OPAL Consulting LLC Karthik Balakrishnan Ph.D. ISO Innovative Analytics CANE Meeting Southbridge, MA September 26, 2008

2. Topics in Predictive Modeling Introduction Explanatory vs. Predictive Modeling Data Quality Data augmentation Data adjustments Predictive Problems/software issues Analytic Methods of Interest

3. Introduction Research by IBM indicates only 1% of data collected by organizations is used for analysis Predictive Modeling and Data Mining widely embraced by leading businesses –2002 Strategic Decision Making survey by Hackett Best Practices determined that world class companies adopted predictive modeling technologies at twice the rate of other companies –Important commercial application is Customer retention: 5% increase in retention  95% increase in profit –It costs 5 to 10 times more to acquire new business Another study of 24 leading companies found that they needed to go beyond simple data collection

4. Models Explanatory vs Predictive Explanatory Model The association of some target or decision variable with explanatory variables through mathematical formulations in which it is assumed that all variables are known with certainty. Example: Underwriting Model, decision to accept or reject is target

5. Models Explanatory vs Predictive Predictive Model The association of some target or decision variable with predictor variables through mathematical formulations in which it is assumed that all predictor variables are known with possibly varying probabilities. Example: Claim Investigation Model, decision to investigate is target

6. Data Mining Data Mining, also known as Knowledge-Discovery in Databases (KDD), is the process of automatically searching large volumes of data for patterns. In order to achieve this, data mining uses computational techniques from statistics, machine learning and pattern recognition. www.wikipedia.org

7. Successful Implementation of Predictive Modeling Data availability and quality are essential Insights from different areas are needed Multidisciplinary effort –Quantitative experts –IT –Business experts –Managers –Upper management buy in –Not just for Actuarials

8. DATA

9. Manage the Human Side of Data and Analytics Data Collection: Design and Reality Understand and convey business benefits Belief, model understanding, model complexity ‘Tribal’ Knowledge as model attributes Behavioral change and transparency Disruption in ‘standard’ processes Threat of obsolescence (automation) Don’t over rely on the technology; recognize the disruptive role you play

10. DATA

11. CRISP-DM Cross Industry Standard Process for Data Mining Standardized approach to data mining www.crisp-dm.org

13. Phases of CRISP-DM

14. DATA

15. Data Quality Scope of problem How it is addressed New educational resources for actuaries

16. Survey of Actuaries Data quality issues have a significant impact on the work of actuaries –About a quarter of their time is spent on such issues –About a third of projects are adversely affected –See “Dirty Data on Both Sides of the Pond” – 2008 CAS Winter Forum –Data quality issues consume significantly more time on large predictive modelling Projects

17. Statistical Data Editing Process of Checking data for errors and correcting them Uses subject matter experts Uses statistical analysis of data May include using methods to “fill in” missing values Final result of SDE is clean data as well as summary of underlying causes of errors

18. EDA: Overview Typically first step in analyzing data Purpose: –Explore structure of the data –Find outliers and errors Uses simple statistics and graphical techniques Examples include histograms, descriptive statistics and frequency tables Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Step 1 Data Collection Step 0 Data Requirements Final Step Decisions

19. EDA: Histograms Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Step 1 Data Collection Step 0 Data Requirements Final Step Decisions

20. Data Educational Materials Working Party Formation The closest thing to data quality on the CAS syllabus are introductions to statistical plans The CAS Data Management and Information Committee realized that SOX and Predictive Modeling have increased the need for quality data So they formed the CAS Data Management Educational Materials working party to find and gather materials to educate actuaries

21. CAS Data Management Educational Materials Working Party Publications Book reviews of data management and data quality texts in the CAS Actuarial Review starting with the August 2006 edition These reviews are combined and compared in “Survey of Data Management and Data Quality Texts,” CAS Forum, Winter 2007, www.casact.org “Actuarial IQ (Information Quality)” published in the Winter 2008 edition of the CAS Forum: http://www.casact.org/pubs/forum/08wforum/

22. Data Flow Step 2 Transformations Aggregations Step 3 Analysis Step 4 Presentation of Results Step 1 Data Collection Step 0 Data Requirements Final Step Decisions Information Quality involves all steps: Data Requirements Data Collection Transformations & Aggregations Actuarial Analysis Presentation of Results To improve Final Step: Making Decisions

23. Data Augmentation Add information from Internal data Add information from external data For overview of inexpensive sources of data see: “Free and Cheap Sources of Data”, 2007 Predictive modeling seminar and “External Data Sources” at 2008 Ratemaking Seminar

24. Data Augmentation – Internal Data Create aggregated statistics from internal data sources –Number of lawyers per zip –Claim frequency rate per zip –Frequency of back claims per state Use unstructured data –Text Mining

25. Data Augmentation – External Data Census data: Household type per zip Industry data: California DOI data, IRC Marketing data: Third party vendors IRC Data: Auto Injury data ISO data:??

26. Data Adjustments Trend –Adjust all records to common cost level –Use model to estimate trend Development –Adjust all losses to ultimate –Adjust all losses to a common age –Use model to estimate future development

27. Analytic Model Development Rule # 1: Use appropriate tool Rule #2: There are no other rules

28. Shameless Advertisement Derrig, R.A. and Francis, L.A. 2008 Distinguishing the Forest from the Trees, Next VARIANCE and CAS Seattle

29. Computers advance