SCORE AS YOU LIFT (SAYL) A Statistical Relational Learning Approach to Uplift Modeling Houssam Nassif 1, Finn Kuusisto 1, Elizabeth S. Burnside 1, David Page 1, Jude Shavlik 1, and Vítor Santos Costa 2 1 University of Wisconsin, Madison, USA 2 CRACS-INESC TEC and DCC-FCUP, University of Porto, Portugal

The Task Identify patients with breast cancer who may be good candidates for watchful waiting Produce interpretable classifiers, for insight on the problem 9/24/2013Score As You Lift

Breast Cancer Stages In Situ Earlier stage Cancer is localized Invasive Later stage Cancer has invaded surrounding tissue 9/24/2013Score As You Lift

Breast Cancer Age Difference Older Cancer tends to progress less aggressively Patient has less time for cancer progression Younger Cancer tends to progress more aggressively Patient has more time for cancer progression 9/24/2013Score As You Lift

Overtreatment Problem Who is treated? Everyone Can we reduce overtreatment in older patients with in situ cancer? 9/24/2013Score As You Lift

Watchful Waiting Candidates Older In Situ Sufficiently different from that of younger patients 9/24/2013Score As You Lift

Problem Definition Given: a Multi-Relational DB Labeling Find: Watchful Waiting Candidates Interpretable Explanations 9/24/2013Score As You Lift Subject Group Control Group Positive Class Negative Class OlderYoungerIn SituInvasive

Differential Prediction New Problem? Psychology: results of a test on two populations Medicine: ADR Marketing: Uplift Modeling 9/24/2013Score As You Lift

Model Filtering (MF) 1. Use ILP to learn theories 2. Learn a theory T O for older 3. T = { Rules in T O that do badly on younger } 9/24/2013Score As You Lift

Differentially Predictive Search (DPS) Use ILP as before But change search: Change node scoring to incorporate case/controls + on cases - on controls 9/24/2013Score As You Lift

Uplift Modeling How to evaluate different approaches? Key Work from Marketing Idea: Performance on Cases – Performance on Controls Robust Measure of Performance: Area under Lift Curve 9/24/2013Score As You Lift

Lift Curve Lift Curve: Fraction E% of Examples to correctly classify P positive examples Lift(0) = 0 Lift(1) = Pos Similar to ROC curve 9/24/2013Score As You Lift

AUL and Uplift 9/24/2013Score As You Lift Uplift SC (x) = Lift S (x) – Lift C (x) Uplift SC (0) = 0 Uplift SC (1) = Pos S - Pos C AU_UPLIFT(S, C) = AUL(S) – AUL(C)

Lift and Uplift 9/24/2013Score As You Lift

Maximise UPLIFT: SAYL Can we do better than DPS? SAYU was designed to max an external theory score Can we use SAYU for a differential score? 9/24/2013Score As You Lift

SAYU SAYU learns a classifier from ILP rules Classifiers are NBAYES, TAN It learns incrementally, by adding rules Rule in if it improves score on tune test 9/24/2013Score As You Lift

SAYL Separates cases (subjects) and controls Learns one classifier for each Scores on uplift 9/24/2013Score As You Lift

SAYL Algorithm 9/24/2013Score As You Lift

Evaluation How does it perform? Do rules interact? Are they useful? 9/24/2013Score As You Lift

Dataset OlderYounger In SituInvasiveIn SituInvasive InterventionSubject Group Control Group Positive Class Negative Class TimeOlderYoungerIn SituInvasive 9/24/2013Score As You Lift

Performance – Uplift Curves 9/24/2013Score As You Lift

Performance - Significance AlgorithmUplift AUCLift (Older) AUC Lift (Younger) AUC Rules Avg # DPS p-value SAYL DPS MF Baseline /24/2013Score As You Lift

Example Older TAN Model 9/24/2013Score As You Lift

Example Younger TAN Model 9/24/2013Score As You Lift

Example Rules 1. Current study combined BI-RADS increased up to 3 points over previous mammogram. 2. Patient had previous in situ biopsy at same location. 3. Breast BI-RADS score = 4. 9/24/2013Score As You Lift

Conclusions SAYL maximizes area under the uplift curve. SAYL significantly outperforms previous ILP methods on breast cancer application. Output rules are clinically interpretable. 9/24/2013Score As You Lift

Future Work Investigate clinical relevance further. Investigate uplift maximization using different metrics. Investigate use of initial TAN model structure. 9/24/2013Score As You Lift

Questions?

Uplift and AUC AUL is proportional to AUC AUL = P ( skew/2 + (1-skew)AUC) 0 <= skew <= linear and monotonic UPLIFT(s, c) = C 1 + C 2 (AUC s – AUC c ) 9/24/2013Score As You Lift

Marketing Customer Groups Persuadables Customers who will respond only when targeted. Sure Things Customers who will respond even when not targeted. Lost Causes Customers who will not respond, regardless of whether they were targeted or not. Sleeping Dogs Customers who will not respond as a result of being targeted. 9/24/2013Score As You Lift

Marketing Ideal Ranking 9/24/2013Score As You Lift

Marketing Dataset 9/24/2013Score As You Lift TargetControl ResponseNo ResponseResponseNo Response Persuadables Sure Things Sleeping Dogs Lost Causes Sleeping Dogs Sure Things Persuadables Lost Causes

In Situ vs. Invasive Dataset 9/24/2013Score As You Lift OlderYounger In SituInvasiveIn SituInvasive Indolent In Situ Aggressive In Situ Always ExciseAggressive In SituAlways Excise