1. SCORE AS YOU LIFT (SAYL) A Statistical Relational Learning Approach to Uplift Modeling Houssam Nassif 1, Finn Kuusisto 1, Elizabeth S. Burnside 1, David Page 1, Jude Shavlik 1, and Vítor Santos Costa 2 1 University of Wisconsin, Madison, USA 2 CRACS-INESC TEC and DCC-FCUP, University of Porto, Portugal

2. The Task Identify patients with breast cancer who may be good candidates for watchful waiting Produce interpretable classifiers, for insight on the problem 9/24/2013Score As You Lift

3. Breast Cancer Stages In Situ Earlier stage Cancer is localized Invasive Later stage Cancer has invaded surrounding tissue 9/24/2013Score As You Lift

4. Breast Cancer Age Difference Older Cancer tends to progress less aggressively Patient has less time for cancer progression Younger Cancer tends to progress more aggressively Patient has more time for cancer progression 9/24/2013Score As You Lift

5. Overtreatment Problem Who is treated? Everyone Can we reduce overtreatment in older patients with in situ cancer? 9/24/2013Score As You Lift

6. Watchful Waiting Candidates Older In Situ Sufficiently different from that of younger patients 9/24/2013Score As You Lift

7. Problem Definition Given: a Multi-Relational DB Labeling Find: Watchful Waiting Candidates Interpretable Explanations 9/24/2013Score As You Lift Subject Group Control Group Positive Class Negative Class OlderYoungerIn SituInvasive

8. Differential Prediction New Problem? Psychology: results of a test on two populations Medicine: ADR Marketing: Uplift Modeling 9/24/2013Score As You Lift

9. Model Filtering (MF) 1. Use ILP to learn theories 2. Learn a theory T O for older 3. T = { Rules in T O that do badly on younger } 9/24/2013Score As You Lift

10. Differentially Predictive Search (DPS) Use ILP as before But change search: Change node scoring to incorporate case/controls + on cases - on controls 9/24/2013Score As You Lift

11. Uplift Modeling How to evaluate different approaches? Key Work from Marketing Idea: Performance on Cases – Performance on Controls Robust Measure of Performance: Area under Lift Curve 9/24/2013Score As You Lift

12. Lift Curve Lift Curve: Fraction E% of Examples to correctly classify P positive examples Lift(0) = 0 Lift(1) = Pos Similar to ROC curve 9/24/2013Score As You Lift

13. AUL and Uplift 9/24/2013Score As You Lift Uplift SC (x) = Lift S (x) – Lift C (x) Uplift SC (0) = 0 Uplift SC (1) = Pos S - Pos C AU_UPLIFT(S, C) = AUL(S) – AUL(C)

14. Lift and Uplift 9/24/2013Score As You Lift

15. Maximise UPLIFT: SAYL Can we do better than DPS? SAYU was designed to max an external theory score Can we use SAYU for a differential score? 9/24/2013Score As You Lift

16. SAYU SAYU learns a classifier from ILP rules Classifiers are NBAYES, TAN It learns incrementally, by adding rules Rule in if it improves score on tune test 9/24/2013Score As You Lift

17. SAYL Separates cases (subjects) and controls Learns one classifier for each Scores on uplift 9/24/2013Score As You Lift

18. SAYL Algorithm 9/24/2013Score As You Lift

19. Evaluation How does it perform? Do rules interact? Are they useful? 9/24/2013Score As You Lift

20. Dataset OlderYounger In SituInvasiveIn SituInvasive 132401110264 InterventionSubject Group Control Group Positive Class Negative Class TimeOlderYoungerIn SituInvasive 9/24/2013Score As You Lift

21. Performance – Uplift Curves 9/24/2013Score As You Lift

22. Performance - Significance AlgorithmUplift AUCLift (Older) AUC Lift (Younger) AUC Rules Avg # DPS p-value SAYL58.1097.2439.159.30.0020 DPS27.83101.0173.1737.1- MF20.90100.8980.9919.90.0039 Baseline11.0066.0055.00-0.0020 9/24/2013Score As You Lift

23. Example Older TAN Model 9/24/2013Score As You Lift

24. Example Younger TAN Model 9/24/2013Score As You Lift

25. Example Rules 1. Current study combined BI-RADS increased up to 3 points over previous mammogram. 2. Patient had previous in situ biopsy at same location. 3. Breast BI-RADS score = 4. 9/24/2013Score As You Lift

26. Conclusions SAYL maximizes area under the uplift curve. SAYL significantly outperforms previous ILP methods on breast cancer application. Output rules are clinically interpretable. 9/24/2013Score As You Lift

27. Future Work Investigate clinical relevance further. Investigate uplift maximization using different metrics. Investigate use of initial TAN model structure. 9/24/2013Score As You Lift

28. Questions?

29. Uplift and AUC AUL is proportional to AUC AUL = P ( skew/2 + (1-skew)AUC) 0 <= skew <= linear and monotonic UPLIFT(s, c) = C 1 + C 2 (AUC s – AUC c ) 9/24/2013Score As You Lift

30. Marketing Customer Groups Persuadables Customers who will respond only when targeted. Sure Things Customers who will respond even when not targeted. Lost Causes Customers who will not respond, regardless of whether they were targeted or not. Sleeping Dogs Customers who will not respond as a result of being targeted. 9/24/2013Score As You Lift

31. Marketing Ideal Ranking 9/24/2013Score As You Lift

32. Marketing Dataset 9/24/2013Score As You Lift TargetControl ResponseNo ResponseResponseNo Response Persuadables Sure Things Sleeping Dogs Lost Causes Sleeping Dogs Sure Things Persuadables Lost Causes

33. In Situ vs. Invasive Dataset 9/24/2013Score As You Lift OlderYounger In SituInvasiveIn SituInvasive Indolent In Situ Aggressive In Situ Always ExciseAggressive In SituAlways Excise