1. Disability Pricing and Dental Fraud Detection: Supervised and Unsupervised Learning October 11, 2007 Jonathan Polon FSA Claim Analytics Inc. www.claimanalytics.com

2. Definitions Dental Fraud Disability Pricing Agenda

3. Definitions

4. Supervised Learning Known outcome associated with each record in training dataset Eg, Sports betting: predict winners based on observations from past games Objective: build a model to accurately estimate outcomes from predictor variables for each record Commonly referred to as predictive modeling

5. Unsupervised Learning No known outcome associated with any record in the training dataset Eg, Grocery stores: define types of shoppers and their preferences Learning objective: self-organization or clustering; finding structure in the data

6. Dental Fraud Detection Using Unsupervised Learning

7. Dental Fraud Project Overview Explore use of pattern detection tools in detecting dental claim anomalies 2004 data 1,600 Ontario GPs (non-specialists) 200,000 claims Many examples of anomalous activity discovered

8. Traditional Tools Rule-based Strong at identifying claims that match known types of fraudulent activity Limited to identifying what is known Typically analyze at the level of a single claim, in isolation

9. Pattern Detection Tools Analyze millions of claims to reveal patterns –Technology learns what is normal and what is atypical – not limited by what we already know Categorize each claim. Reveal dentists with high percentage of atypical claims –Immediately highlight new questionable behaviors –Find new large-dollar schemes –Also identify frequent repetition of small-dollar abuses

10. Methodology – 3 Perspectives 1. By Claim e.g. Joe Green’s semi-annual check-up 2. By Tooth e.g. all work done in 2004 on Joe’s bicuspid by Dr. Brown 3. By General Work e.g. all general work (exams, fluoride, radiographs, scaling and polishing) done on Joe in 2004 by Dr. Brown

11. Methodology – 2 Techniques 1. Principal components analysis 2. Clustering

12. PCACluster Claim by claim Tooth by tooth General work Methodology – Summary

13. Principal Components Analysis Visualization technique Allows reduction of multi-dimensional data to lower dimensions while maximizing amount of information preserved Powerful approach for identifying outliers

14. PCA: Simple Example Lots of variance between points around both axes

15. PCA: Simple Example Create new axes, X’ and Y’: rotate original axes 45º

16. PCA: Simple Example In the new axes, very little variance around Y’ Y’ contains little “information”

17. PCA: Simple Example Can set all Y’ values to 0 – ie, ignore Y’ axis Result: reduce to 1 dimension with little loss of info ♦ Original ● Revised

18. PCA: Identifying Atypical Dentists

19. 1.Begin at the individual transaction level 2.Determine the average transaction for each dentist 3.Graph quickly isolates dentists that are outliers

20. Dentist Average 90 th percentile PCA: Identifying Atypical Dentists

21. PCA: Summary 1.Visualization allows for easy and intuitive identification of dentists that are atypical 2.Manual investigation required to understand why a dentist or claim is atypical

22. Clustering Techniques Categorization tools Organize claims into several groups of similar claims Applicable for profiling dentists by looking at the percentage of claims in each cluster We apply two different clustering techniques –K-Means –Expectation-Maximization

23. Clustering Example

25. Example: Clusters found – by Claim 1.Major dental problems 2.Moderate dental problems 3.Age > 28, minor work performed 4.Work includes unbundled procedures 5.Minor work and expensive technologies 6.Age < 28, minor work performed

26. Clusters: Identifying Atypical Dentists 1. Begin at the individual claim level 2. Calculate the proportion of claims in each cluster – in total, and for each dentist 3. Isolate dentists with large deviations from average

27. Clusters: Identifying Atypical Dentists

28. Clustering Techniques: Summary 1.Clustering provides an easy to understand method to profile dentists 2.Unlike PCA, clustering tells why a given dentist is considered atypical 3.Effectively identifies atypical activity at the dentist level

29. Pilot Project Results PCA identified 68 dentists that are atypical Clusters identified 182 dentists that are atypical 36 dentists are identified as atypical by both PCA and Clusters In total, 214 of 1,644 dentists are identified as atypical (13%) Billings by atypical dentists were $2.5 MM out of $16.1 MM billed by all dentists (15%)

30. ExamplesOf Atypical Dentists

31. Analysis: Dentist A is far beyond norms in clusters 6 and 8; both indicate high charges in ‘general dentistry’ What we discovered: Each of Dentist A’s patients is being billed for at least 30 minutes of polishing and 45 minutes of scaling Dentist A

32. Dentist B Analysis : Dentist B is far beyond norms in cluster 7 What we discovered: Frequent extractions and anesthesia  Dentist B looks like an oral surgeon, yet is a general practitioner

33. Dentist C Analysis: Very high proportion in Cluster 1, suggesting many high- ticket visits What we discovered: First, Dentist C performs an inordinate amount of scaling. Second, Dentist C has emergency examinations with atypically high frequency

34. Dentist M Analysis: Dentist M has a disproportionate amount of work beyond the 90 th percentile of work by all dentists on all teeth What we discovered: Dentist M appears to utilize lab work very heavily

35. More Atypical Dentists Frequent use of panoramic x-rays Frequent use of nitrous oxide – including with procedures rarely associated with anesthesia Large number of extractions, often using multiple types of sedation Very high proportion of claims for crowns and endodontic work Individual instruction on oral hygiene provided with very atypical frequency

36. Dental Fraud Summary Highly effective in identifying dentists with claim portfolios significantly different from the norm Enables experts to quickly identify and focus on those dentists with atypical claims activity Pattern detection :

37. Disability Pricing With Predictive Modeling

38. Protects against loss of income due to illness or injury Annual incidence rates ≈ 3 per 1,000 Claim duration ranges from a few days to a few decades About Group LTD Insurance

39. Base rates: tabular, depend on age, gender, EP and max benefit duration Several loading factors Demographic info (eg, occupation, area, salary) Group characteristics (eg, group size, industry) Plan features (eg, benefit %, e’e contributions) Typical Rate Structure

40. Uncover and quantify complex relationships between pricing factors and claim experience Maintain wholeness of data Improved accuracy vs traditional methods Why use PM for pricing?

41. The Challenge Predictive models can be black boxes, difficult to interpret The Objective A process to make rate structure explainable to: Predictive Modeling + Pricing Management Sales Force Regulators Customers

42. The 5 Steps - A PM Pricing Project

43. The Five Steps 1.Set objective 2.Data 3.Select technique and predictors 4.Train the model 5.Validate

44. Develop a predictive model to: Predict claim incidence rates Predict claim severity, conditional upon claim being made for each member of census data 1. Set objective

45. Required: Accurate census data for many groups Plan features for these groups (EP, Ben%) Characteristics of these groups (region, industry) Claim data that often can be linked to census 2a. Pull Data

46. Split Data Into Three Subsets Training Testing Validation 2b. Sample Data

47. Begin exploring with simpler tools Learn key predictors, relationships Migrate to more complex tools Incorporate exploration learning into model design Use test sample to evaluate models 3. Select technique

48. 4. Train the Model Pricing factors Challenge: Understanding the Model Need to extract pricing factors from “black box”

49. Approach: Build several models Each using same modeling technique Each using same data Each excluding different predictor Compare actual to predicted claim cost across excluded predictor Understanding the Model Training the Model

50. Looking into the Black Box Training the Model

51. Example: model includes all predictors, consider historic claims by industry Demonstrates accuracy of model, but provides no information about claim drivers IndustryActualPredictedAct/Pred Finance$1,600 100% Health$1,200 100% Training the Model

52. Example: model includes all predictors except industry, consider historic claims by industry Isolates impact of industry from all other factors IndustryActualPredictedAct/Pred Finance$1,600$1,80089% Health$1,200$1,000120% Training the Model

53. Example: exclude two predictors: region and occ class RegionWhite CollarBlue Collar North East115%95% Mid West75%105% South105%115% Isolates impact by region and occupation class from all other factors Training the Model

54. Benefits of Extracting Factors 1.Ability to test importance of predictors, singly or in combinations Test predictors that are typically not priced for Quantify impact of each predictor or pair of predictors 2.Facilitates use of the traditional Base Rate Approach Training the Model

55. Base Rate: Function of age, gender, EP, max duration Loadings: For other demographic info, group characteristics, plan features Determine by looking inside black box Company can adjust final factors Base Rate Approach Training the Model

56. Base Rate:  Base rate = 1.20 Loadings: Base Rate Example Ben%RegionIndustrySalaryContribFact X 60%NEHealth$60KY10.2 +5%-5%+15%-5%+10%-3% AgeGenderEPMax Dur 45Male180 daysAge 65 Training the Model

57. Base Rate = 1.20 Loadings = {+5%, -5%, +15%, -5%, +10%, - 3%} Rate = 1.20 *1.05 *0.95 *1.15 *0.95 *1.10 *0.97 Final Rate = 1.395 Base Rate Example Training the Model

58. Use excluded data for validation of model Compare actual vs predicted claim costs Ideally, compare predictive modeling approach to traditional approach 5. Pricing Validation

59. Improved accuracy vs traditional methods Capture effects of interaction between several predictors Able to isolate impact of any predictor or pair of predictors Able to test new predictors for impact on claim cost Can maintain existing Base Rate structure Advantages of PM: Pricing

60. Questions?