1. Making Data Mining Models Useful to Model Non-paying Customers of Exchange Carriers Wei Fan, IBM T.J.Watson Janek Mathuria, and Chang-tien Lu Virginia Tech, Northern Virginia Center

2. Our Selling Points A real practical problem for an actual CLEC company. A whole process: Start with a great goal. Reality taught us a lesson Settle down with a realistic solution A new set of algorithms to calibrate probability outputs (as distinguished from Zadrozny and Elkans calibration methods)

3. Challenging Problem Differentiate between Late and Default: Late: 1 month past due Default: two month past due. Default Percentage: 20%. Designed feature set: Details in Paper Calling summary. Billing summary. Obvious ones. Other ones out there? Maybe.

4. Failure Failure of Commonly Used Methods: Nearly predicting every customer as paying on time and still has 80% What this means: Our feature set not complete? Probably. Problem itself is just stochastic in nature. Natural next step: cost-sensitive learning? Impossible to define precisely due to complexity.

5. A Compromised Solution Predict a reliable probability score. A customer is uniquely distinguished by its feature vector. If the model predict that a customer has 20% chance to default Indeed the customer has 20% chance to default The predicted score is considered reliable

6. Previously Proposed Calibration Methods Existing approaches that output scores are not reliable (Zadrozny and Elkan) Decision trees. Naïve Bayes SVM Logistic Regression Use function mapping to calibrate unreliable score to reliable ones. Assumption: original unreliable score need to be monotonous. Otherwise, it is not applicable.

7. A Good Calibration

8. A Bad Calibration

9. Random Decision Trees Amazingly Simple and Counter-intuitive: Do not use any purity check function. Pick a feature randomly. Continuous feature, pick a random splitting point. Discrete feature can be picked only once in one decision path. Continuous feature can be picked multiple times. Tree depth up to the number of features. Original feature set. No bootstrap! Each tree computes probability at the leaf node. 10 fraud and 90 normal transaction, p(fraud|x) = 0.1 Multiple trees, 10 min and 30 enough, average probability.

10. Random Forest + Marriage between Random Decision Tree and Random Forest Pick a feature subset randomly. Compute info gain for each feature. Choose the one with highest info gain. Original dataset. Not bootstrap. Leaf node computes probability. 10 to 30 trees.

11. Availability. Software available upon request.