2. Adaptive Fraud Detection Paper By: Tom Fawcett And Foster Provost Presentation By: Qijun Chen

3. Outline Problem Description Detector Construction The Data Experiments and Evaluation Conclusion

4. The Problem

5. Cellular Fraud – Cloning A kind of Superimposition fraud – Fraudulent usage is superimposed (added to) the legitimate usage of an account – Other Examples: Credit card fraud, Calling card fraud, some types of computer intrusion Mobile Identification Number (MIN) and Electronic Serial Number (ESN) – Identify a specific account – Periodically transmitted unencrypted whenever phone is on

6. Cloning Fraud Cont. Detrimental in several ways: – Fraudulent usage congests cell sites – Cellular carriers must pay costs to other carriers for usage outside home territory (roaming fees) – Crediting process is costly to carrier and inconvenient to the customer These costs are ultimately passed on to the customer

7. Strategies for Fraud Detection Pre-call Methods (Validate the phone or its user when a call is placed ) Post-call Methods (Periodically analyze call data on each account to determine whether cloning fraud has occurred. )

8. Dealing With Cloning Fraud – Pre-call Methods Require Personal Identification Number (PIN) – Although more difficult, can be intercepted like MIN- ESN pairs RF Fingerprinting and Authentication – Method of identifying phones by their unique transmission characteristics – Requires changes in hardware on both customer and carriers sides – An estimated 30 million non-equipped phones are in use in the US alone

9. Dealing With Cloning Fraud – Post-call Methods Collision Detection – Analyze call data for temporally overlapping calls – Usefulness depends on moderate level of legitimate usage Velocity Checking – Analyze locations and times of consecutive calls – Also depend on moderate legitimate usage User Profiling ( DC1 is a kind of User Profiling method )- Analyze calling behavior to detect usage anomalies – Often works well with low-usage customers

10. Sample Frauded Account DateTimeDayDurationOriginDestinationFraud 1/01/9510:05:01Mon13 minutesBrooklyn, NYStamford, CT 1/05/9514:53:27Fri 5 minutesBrooklyn, NYGreenwich, CT 1/08/9509:42:01Mon 3 minutesBronx, NYManhattan, NY 1/08/9515:01:24Mon 9 minutesBrooklyn, NY 1/09/9515:06:09Tue 5 minutesManhattan, NYStamford, CT 1/09/9516:28:50Tue53 secondsBrooklyn, NY 1/10/9501:45:36Wed35 secondsBoston, MAChelsea, MABandit 1/10/9501:46:29Wed34 secondsBoston, MAYonkers, NYBandit 1/10/9501:50:54Wed39 secondsBoston, MAChelsea, MABandit 1/10/9511:23:28Wed24 secondsBrooklyn, NYCongers, NY 1/11/9522:00:28Thu37 secondsBoston, MA Bandit 1/11/9522:04:01Thu37 secondsBoston, MA Bandit

11. The Need to be Adaptive Patterns of fraud are dynamic – bandits constantly change their strategies in response to new detection techniques Levels of fraud can change dramatically from month-to-month Cost of missing fraud and dealing with false alarms change with inter-carrier contracts

12. Automatic Construction of Profiling Fraud Detectors

13. Our Approach Build a fraud detection system to classify calls being fraudulent or legitimate ( but we won’t use current classification techniques) ?

14. Why Not Use Existing Classification Techniques? Context – A call that would be unusual for one customer may be typical for another customer (For example. A call placed from Brooklyn is not unusual for a subscriber who lives there, but might be very strange for a Boston subscriber. ) Granularity – Variation in calling behavior too large at individual call level

15. The Learning Problem 1. Which call features are important? 2. How should profiles be created? 3. When should alarms be raised?

16. Detector Constructor Framework

18. Rule Learning Rule Generation Rule Selection

19. DC-1 uses the RL program to generate rules with certainty factors above user-defined threshold For each Account, RL generates a “local” set of rules describing the fraud on that account. Example: (Time-of-Day = Night) AND (Location = Bronx)  FRAUD Certainty Factor = 0.89 Rule Generation

20. Rule Selection Rule generation step typically yields tens of thousands of rules If a rule “covers” many accounts then it is probably a useful rule Selection algorithm identifies a small set of rules which cover the accounts the best Resulting set of rules is used to construct specific monitors

21. Profiling Monitors Profiling step: – Monitor is applied to an account’s non-fraud usage to measure account’s normal usage, statistics saved Use step: – Monitor process a single account-day and generates a numeric value describing how abnormal the current account-day is through referencing the normalcy measure calculated in profiling.

22. Monitor Template Threshold Standard Deviation

23. Threshold Monitors

24. Standard Deviation Monitors

25. Example for Standard Deviation Rule -(TIME­OF­DAY = NIGHT) AND (LOCATION = BRONX)  FRAUD Template - Standard Deviation. Profiling Step - the subscriber called from the Bronx an average of five minutes per night with a standard deviation of two minutes. At the end of the Profiling step, the monitor would store the values (5,2) with that account. Use step - if the monitor processed a day containing three minutes of airtime from the Bronx at night, the monitor would emit a zero; if the monitor saw 15 minutes, it would emit (15 - 5)/2 = 5. This value denotes that the account is five standard deviations above its average (profiled) usage level.

26. Combining Monitor Outputs Weigh the monitor outputs and learn a threshold on the sum to produce high confidence alarms Train a classifier with attributes(monitor outputs) and class label (fraud or not) DC-1 uses Linear Threshold Unit (LTU) – Simple and fast Feature selection - Sequential forward selection process used to keep only the most useful monitors

27. The Data

28. General Data Information 4 months of call records from the New York City area Each call described by 31 original attributes Other ‘derived’ attributes also added – Time-Of-Day attribute – To-Payphone attribute In training data each call also given class label

29. Data Cleaning Days with only1-4 minutes of fraudulent usage were discarded Eliminated credited calls made to numbers also called by the legitimate user Call times were normalized to Greenwich Mean Time for chronological sorting

30. Data Selection Rule learning and selection – 879 accounts – Over 500,000 calls Profiling, training and testing – 3600 accounts – 96,000 account-days Overall ratio: 20% fraud, 80% non-fraud

31. Experiments and Evaluation

32. Experiment Outputs Rule learning: 3630 rules – Each covering at least two accounts Rule selection: 99 rules 2 monitor templates yielding 198 monitors Final feature selection: 11 monitors

33. The Importance Of Error Cost Different types of errors have different costs Classification accuracy is not sufficient Estimated Costs: – False positive(false alarm): $5 – False negative(let a frauded account-day go undetected): $0.40 per minute of fraudulent air-time Factoring in error costs requires second training pass by LTU

34. Alternatives for Fraud Detection Collisions + Velocities – Errors almost entirely due to false negatives High Usage – detect sudden large jump in account usage Best Individual DC-1 Monitor – (Time-of-day = Evening) ==> Fraud SOTA - State Of The Art – Incorporates 13 hand-crafted profiling methods – Best detectors identified in a previous study

35. DC-1 Vs. Alternatives DetectorAccuracy(%)Cost ($)Accuracy at Cost Alarm on all 202000020 Alarm on none 8018111 +/- 96180 Collisions + Velocities 82 +/- 0.317578 +/- 74982 +/- 0.4 High Usage 88+/- 0.76938 +/- 47085 +/- 1.7 Best DC-1 monitor 89 +/- 0.57940 +/- 31385 +/- 0.8 State of the art 90 +/- 0.46557 +/- 54188 +/- 0.9 DC-1 detector 92 +/- 0.55403 +/- 50791 +/- 0.8 SOTA plus DC-1 92 +/- 0.45078 +/- 31991 +/- 0.8

36. Testify the Importance of Context (Call Classifier Detector) Design two Classifiers ( CC1054, CC1861) which lose context information, but have the advantage of profiling and monitoring

37. DC-1 Vs. Global Classifiers DetectorAccuracy(%)Cost ($)Accuracy at Cost CC 105488 +/- 0.48611 +/- 53188 +/- 0.6 CC 186188 +/-0.58686 +/- 80488 +/- 0.6 DC-192 +/- 0.55403 +/- 50791 +/- 0.8

38. Testify the Importance of Adaptability (Shifting Fraud Distributions) One DC-1 detector trained at 80% non-fraud and tested over range of 75-99% non-fraud Another DC-1 was allowed to adapt (re-train its LTU threshold) for each fraud distribution Second detector consistently most cost effective than the first

40. Conclusion

41. Experiment Conclusions High Usage Detector – Profiles with respect to undifferentiated account usage – Comparison with DC-1 demonstrates the benefit of using rule learning Best Individual DC-1 Monitor – Demonstrates the benefit of combining evidence from multiple monitors

42. Experiment Conclusions(2) Call Classifier Detector – Represent rule learning without the benefit of account context – Demonstrates value of DC-1’s rule generation step, which preserves account context Shifting Fraud Distributions – Shows benefit of making evidence combination sensitive to fraud distribution

43. Summary DC1 use a rule­learning program to uncover indicators of fraudulent behavior from a large database of customer transactions. Then the indicators are used to create a set of monitors, which profile legitimate customer behavior and indicate anomalies. Finally, the outputs of the monitors are used as features in a system that learns to combine evidence to generate high­confidence alarms.

44. Question 1 Of the two major categories of fraud detection methods under which does the DC-1 adaptive profiler fall ? Post Call Method.

45. Question 2 Why is "Context" important in successfully detecting fraud? A call would be unusual for one customer or a group of customers would be typical for another customer or a group of customers. So context keep the unique characteristics of an account or a group of accounts (For example. A call placed from Brooklyn is not unusual for a subscriber who lives there, but might be very strange for a Boston subscriber. )

46. Question 3 Profiling monitors have two distinct stages associated with them. Describe them. Profiling step: – Monitor is applied to an account’s non-fraud usage to measure account’s normal usage Use step: – Monitor process a single account-day and generates a value describing how abnormal the current account- day is