Fraud Detection Experiments Chase Credit Card –500,000 records spanning one year –Evenly distributed –20% fraud, 80% non fraud First Union Credit Card –500,000 records spanning one year –Unevenly distributed –15% fraud, 85% non fraud

Intra-bank experiments Classifier Selection Algorithm: Coverage/TP-FP Let V be the validation set Until no other examples in V be covered –select the classifier with highest TP-FP rate on V –Remove covered examples from V Setting 12 subsets 5 algorithms (Bayes, C4.5, Cart, ID3, Ripper) 6-fold cross validation

TP-FP vs number of classifiers Input base classifiers: Chase Test data set: Chase Best meta classifier: Naïve Bayes with base classifiers.

TP-FP vs number of classifiers Input base classifiers: First Union Test data set: First Union Best meta classifier: Naïve Bayes with base classifiers.

Accuracy vs number of classifiers Input base classifiers: Chase Test data set: Chase Best meta classifier: Ripper with 50 base classifiers. Comparable performance is attained with classifiers

Accuracy vs number of classifiers Input base classifiers: First Union Test data set: First Union Best meta classifier: Ripper with 13 base classifiers.

Intra-bank experiments Coverage, cost model combined metric algorithm: –Let V be the validation set –Until no examples can be covered from V select classifier C j that achieves the highest savings on V Remove covered examples from V

Savings vs number of classifiers Input base classifiers: Chase Test data set: Chase Best Meta Classifiers: Single naïve bayesian base classifier (~ $820K)

Savings of base classifiers Input base classifiers: Chase Test data set: Chase Conclusion: Learning algorithms focus on binary classification problem. If base classifiers fail to detect expensive fraud, meta learning cannot improve savings.

Savings vs number of classifiers Input base classifiers: First Union Test data set: First Union Best Meta Classifiers: Naïve bayes with 22 base classifiers (~ $945K)

Savings of base classifiers Input base classifiers: First Union Test data set: First Union Conclusion: The majority of base classifiers are able to detect transactions that both fraudulent and expensive. Meta learning saves an additional $100K.

Different distributions experiments –Number of Datasites: 6 –Training sets: 50-50% Fraud/Non-Fraud –Testing sets: 20-80% Fraud/Non-Fraud –Base classifiers: ID3, CART –Meta classifiers: ID3, CART, Bayes, Ripper –Base classifiers: 81% TP, 29% FP –Meta-classifiers: 86% TP, 25% FP

Inter-bank experiments Chase includes 2 attributes not present in First Union data –Add two fictitious fields –Classifier agents support unknown values Chase and First Union define an attribute with different semantics –Project Chase values on First Union semantics

Inter-bank experiments: Input base classifiers: Chase Test data set: Chase and First Union Task: Compare TP and FP rates of a classifier on different test sets. Conclusion: Chase classifiers CAN be applied to First Union data, but not without penalty.

Inter-bank experiments: Input base classifiers: First Union Test data set: First Union and Chase Task: Compare TP and FP rates of a classifier on different test sets. Conclusion: First Union classifiers CAN be applied to Chase data, but not without penalty.

TP-FP vs number of classifiers: Input base classifiers: First Union and Chase Test data set: Chase Result: –Ripper, CART: comparable –Naïve bayes: slightly superior –C4.5, ID3: inferior

Accuracy vs number of classifiers: Input base classifiers: First Union and Chase Test data set: Chase Result: –CART, Ripper: comparable –Naïve Bayes, C4.5, ID3: inferior

TP-FP vs number of classifiers: Input base classifiers: First Union and Chase Test data set: First Union Result: –Naïve Bayes, C4.5, CART: comparable only when using all classifiers –Ripper: superior only when using all classifiers –ID3: inferior

Accuracy vs number of classifiers: Input base classifiers: First Union and Chase Test data set: First Union Result: –Naïve Bayes, C4.5, CART, Ripper: comparable only when using all classifiers –ID3: inferior

CHASE max fraud loss: $1,470K Overhead: $75

FU max fraud loss: $ 1,085K Overhead: $75

Aggregate Cost Model $X overhead to challenge a fraud

Experiment Set-up Training data set: 10/ /1996 Testing data set: 9/1996 Each data point is the average of the 10 classifiers (Oct to July 1996) Training set size: 6,400 transactions (to allow 90% of frauds)

Average Aggregate Cost (C4.5)

Accuracy (C4.5)

Average Aggregate Cost (CART)

Accuracy (CART)

Average Aggregate Cost (RIPPER)

Accuracy (RIPPER)

Average Aggregate Cost (BAYES)

Accuracy (BAYES)

Amount Saved: Overhead = $100 Fraud in training data: 30.00% Fraud in training data: 23.14% Maximum saving: $ 1337K Losses/transaction if no detection: 40.81

Do patterns change over time? Entire Chase credit card data set Original fraud rate (20% - 80%) Due to billing cycle and fraud investigation delays, training data are 2 months older than testing data Two experiments were conducted with different training data sets Test data set: 9/1996 (last month)

Training data sets Back in time experiment: –July 1996 –June + July 1996 –... –October July 1996 Forward in time experiment: –October 1995 –October November 1995 –… –October July 1996

Patterns don’t change: Accuracy

Patterns don’t change: Savings

Divide and Conquer Conflict Resolving Conflicts: Base level data with different class labels yet same predicted classifications.

Class-combiner meta-level training data

Prevalence of Conflicts in Meta-level Training Data Note: True Label: ID3:CART:RIPPER 1: fraud, 0: non-fraud

Divide and Conquer Conflict Resolving (cont’d) We divide the training sets into subsets of training data according to each conflict pattern For each subset, recursively apply divide- conquer until stopping criteria is met We use a rote table to learn meta-level training data

Experiment Set-up A full year’s Chase credit card data. Natural fraud percentage (20%). Fields not available at authorization were removed Each month from Oct to July 1996 was used as a training set Testing set was chosen from month that is 2 months older. In real world, it takes 2 month for billing and fraud investigation Result was averages of 10 runs

Results Without Conflict Resolving Technique but Uses Rote Table to learn Meta-level Data: –Overall Accuracy: 88.8% –True Positive: 59.8% –False Positive: 3.81% With Conflict Resolving Technique: –Overall Accuracy: 89.1% (increase of 0.3%) –True Positive: 61.2% (increase of 1.4% ) –False Positive: 3.88% (increase of 0.07%)

Achievable Maximum Accuracy Using nearest neighbor approach to estimate a loose upper bound of the maximum accuracy we can achieve The algorithm calculates the percentage of noise in training data Approximately: 91.0%...so the we are 1.9% close to the maximum accuracy

Accuracy Result