1. Uplift Analysis with the Quadstone System Monday, 7 th January 2005 7.30am PST / 10.30am EST / 3.30pm GMT / 16.30 CET Any trouble getting into the conference call: contact support@quadstone.com.support@quadstone.com

2. © 2005 Quadstone How to ask questions Return to Meeting Manager Use Chat

3. Uplift Analysis Nicholas J. Radcliffe Chief Technology Officer Agenda MOTIVATION Demo 1: Up-sell example (binary outcome) When is Uplift Modelling important? Demo 2: Deep-sell example (continuous outcome) TECHNICAL CONSIDERATIONS Practical considerations and guidelines Small population issues and extensions The quality measure: Qini TRIAL How to get a trial copy & datasets

4. “We have to find a way of making the important measurable, instead of making the measurable important” — Robert McNamara “I know half the money I spend on advertising is wasted, but I can never find out which half” — John Wanamaker

5. Demo 1: Up-sell example Binary outcome SCENARIO Mobile phone company 3G MMS Video phone promotion Some mass advertising - non-targeted customers can purchase Direct calling campaign to drive further sales Random 250k chosen from 10m base for trial c. 75k actually targeted; c. 175k as control

6. When is Uplift Modelling Important?

7. © 2005 Quadstone Two Separate Benefits Not targeting people who are little affected Reponse: Don’t spend money targeting or offer discounts to people who will buy anyway Attrition: Don’t spend money trying to save people who will go anyway Targeting people with low probability but high responsiveness Response: Do spend money on people who aren’t very likely to buy if you do, but are very responsive to offers/contact Attrition: Do spend money trying to save people who aren’t at huge risk of attrition, but can be made much more likely to stay

8. © 2005 Quadstone When would a conventional model be misled? High pre-existing purchase rate Pre-existing knowledge of product coupon Many influences

9. © 2005 Quadstone When are negative effects likely? Sometimes, our actions actually drive customers away, especially when: dissatisfied / angry customers risqué / offensive communications attrition risk intrusive contact mechanisms forgotten standing charges

10. Demo 2: Deep-sell example Continuous outcome SCENARIO Grocery retailer Direct mail campaign to increase spend Weekly Spend measured in 12-week “pre”-period (AWS) Also in 6-week post period (AWSPostCampaign) Objective is difference: (PostMinusPreAWS) Random 250k chosen from 10m base for trial c. 75k actually targeted; c. 175k as control

11. © 2005 Quadstone Control group Must be representative: technique will give misleading results otherwise In practice, this means randomly select controls from target group There must be enough of them Control Group Structure All possible recipients Targets

12. © 2005 Quadstone Population Size Population size “Rule of 500”: to detect a x% difference (uplift), x% of the smaller population (usually controls) should ideally be at least 500 people So if looking for 1% difference, control group needs to have at least 50,000 people So consider longitudinal controls – contact half now, half later

13. © 2005 Quadstone Pruning and Validation Pruning Autopruning is implemented, based on qini variance In practice, fairly unaggressive, so recommend manual pruning Validation Ordinary test-training fine if there is enough data If not, consider k-way crossvalidation

14. © 2005 Quadstone Small Population Extensions Bagging (oversampling method) and k-way cross- validation Analysis candidate selection useful if there are “too many” analysis candidates Stronger pruning (variance-based) Stratification Not part of product, but potentially available as an extension if purchased

15. © 2005 Quadstone Return on Investment Key thing is that Campaign ROI depends on the net effect (i.e. uplift) of action, not apparent response ( reduction in churn) × (value of saved people) – (cost of action) (increase in purchase rate) × (value of purchase) – cost (increase in spend) – (cost of action) etc. Quadstone System has many suitable ROI FDL functions ( f x ) built in (even without uplift license)

16. So how do you measure what’s important?

17. © 2005 Quadstone Quality Measure Considerations Can only estimate uplift by segment This is what we are used to with control groups One person does not have a (knowable, measurable) uplift Generalizing measures like classification error/accuracy or R 2 doesn’t look promising Rank statistics do seem more promising because they can sometimes be computed on a segmented basis

18. © 2005 Quadstone Can we use/modify the Gini for Uplift? % of customers targeted 0% x% 100% uplift Overall uplift: x% Possibility of negative effects x% Positively affected by action Negatively affected by action

19. © 2005 Quadstone Summary: When to use Uplift Uplift modelling is just a better way of modelling the true effect of an action Particularly relevant to: Retention (where it’s the number/value of people you save that’s important Up-sell, cross-sell, deep-sell (where it’s the incremental revenue or profit that’s important) Risk management actions (where it’s the reduction in risk achieved that’s important)

20. © 2005 Quadstone Where to find out more www.quadstone.com/system/uplift/ For more in-depth training: our Uplift Analysis course. Contact support@quadstone.com support@quadstone.com

21. © 2005 Quadstone Questions and answers

22. © 2005 Quadstone After the webinar These slides, the data and a four-week trial license are available via www.quadstone.com/training/webinars/ www.quadstone.com/training/webinars/ Any problems or questions, contact support@quadstone.com support@quadstone.com

23. © 2005 Quadstone Uplift: Quick Reference Building uplift models Ensure random control group exists Set partition field with P interpretation (1 for treated, 0 for control) Set objective (binary, continuous/discrete) Hit go Pruning Switch to test dataset Hit Autoprune Creating results field Use “Uplift as difference” Using difference viewers Crossdistribution Viewer places partition field on “  axis” automatically For view shown, drag count to depth, duplicate mean (ObjectiveField) and drag on to height Can configure which population is viewed by right-clicking on functions Using ROI Functions These are available under f x in Table Viewer when deriving new field.

24. © 2005 Quadstone Upcoming webinars If there’s a webinar topic you’d like to see, please let us know via support@quadstone.com.support@quadstone.com www.quadstone.com/training/webinars/ Thursday, 17 th February 2005 Data Preparation in the Quadstone System Version 5 7.30am PST / 10.30am EST / 3.30pm GMT / 16.30 CET

25. © 2005 Quadstone Your feedback Suggestions or feedback? Please enter them in the feedback form or send them to support@quadstone.com support@quadstone.com

26. © 2005 Quadstone Modifying the Gini for Uplift? % of customers targeted 0% x% 100% uplift x% Positively affected by action Negatively affected by action Unaffected by action

27. © 2005 Quadstone The Shape of the Qini Curve % of customers targeted 0% x% 100% uplift x% +ve –ve neutral Can’t do better than 100% sales: if 90% of control group purchases then maximum uplift = 10% 10% Can’t do worse than 0% sales: if 5% of control group purchases then maximum negative uplift = 5% ? Why is this flat?