Why, and How, your Analytics Project will Fail Peter McCallum Director, CBI

Agenda Introduction Introduction Pyle’s 9 Rules for Analytics Project Failure Pyle’s 9 Rules for Analytics Project Failure Why navigating Pyle’s 9 Rules still doesn’t guarantee success Why navigating Pyle’s 9 Rules still doesn’t guarantee success Incorporating the analytical model into the business process Incorporating the analytical model into the business process Summary Summary

Introduction Who am I? Who am I?  20 years experience in the IT industry  The last 12 years working exclusively delivering Business Intelligence & Analytical solutions  Have experienced the frustration of seeing a data mining project fail to deliver the quick wins promised

Agenda Introduction Introduction Pyle’s 9 Rules for Analytics Project Failure Pyle’s 9 Rules for Analytics Project Failure Why navigating Pyle’s 9 Rules still doesn’t guarantee success Why navigating Pyle’s 9 Rules still doesn’t guarantee success Incorporating the analytical model into the business process Incorporating the analytical model into the business process Summary Summary

Pyle’s 9 Rules Who is Dorian Pyle? Who is Dorian Pyle? What are his rules? What are his rules? Why are they still relevant? Why are they still relevant?

Pyle’s Rule #1 # 1. Jump Right In # 1. Jump Right In  Ignore the business  Use whatever data is on hand  Use whatever tools you’re most comfortable with  And don’t worry about how (or whether) your results can actually be applied

Pyle’s Rule #2 # 2. Frame the problem in terms of the data # 2. Frame the problem in terms of the data  You’ve been given data – mine it!  Don’t stop to ask whether there might be other methods of solving the problem  Don’t think outside of the current data set – simply ignore any environmental or organisational factors  Restate the objective based on “whatever the data can be persuaded to reveal”

Pyle’s Rule #3 # 3. Focus only on the most obvious way to frame the problem # 3. Focus only on the most obvious way to frame the problem  Don’t waste your time exploring the data  Concentrate on the technical merits of the model to the exclusion of all else  Aim for the highest degree of technical perfection

Pyle’s Rule #4 # 4. Rely on your own judgment # 4. Rely on your own judgment  The data miner knows best  The data contains all the required information – focus on revealing the nuggets within  Input from others, especially the business, is unnecessary & should be ignored  Remember – the miner knows best

Pyle’s Rule #5 # 5. Find the best algorithms # 5. Find the best algorithms  For any set of data one particular algorithm will produce the best model  So focus on finding the best algorithm  It’s what data mining is all about

Pyle’s Rule #6 # 6. Rely on memory # 6. Rely on memory  Don’t waste your time documenting  Press on with the data investigation…. As fast as possible  Should you ever need to duplicate the investigation you’ll remember exactly what you did and why  Should anyone ever dare ask you to justify or explain your results, you will remember

Pyle’s Rule #7 # 7. Intuition is more important than standard practice # 7. Intuition is more important than standard practice  Data mining is an art, not a science  Standards are really only intended for “newbies”  All data sets are different, so simply rely on your instincts

Pyle’s Rule #8 # 8. Minimize interaction between miners and business managers # 8. Minimize interaction between miners and business managers  Stay away from the business  Rely exclusively on what the data tells you, irrespective of what the business might try to tell you  After all, mining is primarily about letting the tools do the talking

Pyle’s Rule #9 # 9. Minimize data preparation # 9. Minimize data preparation  Creating the models themselves is the most interesting part of data mining  Data preparation is dull, tedious & time consuming  Let the tools look after the data preparation for you  Do as little preparation as possible and cut straight to the modeling

Agenda Introduction Introduction Pyle’s 9 Rules for Analytics Project Failure Pyle’s 9 Rules for Analytics Project Failure Why navigating Pyle’s 9 Rules still doesn’t guarantee success Why navigating Pyle’s 9 Rules still doesn’t guarantee success Incorporating the analytical model into the business process Incorporating the analytical model into the business process Summary Summary

The Bigger Picture “Data mining is part, and a very small part, of a much larger business process. It may be an essential part of a data mining project, but incorporating the results of mining with all the related parts of the corporate project is equally, if not more, important for ultimate success” Dorian Pyle

Virtuous Cycle of Data Mining Identify business problem Transform Data Measure the results Act on the Information Berry & Linoff

Realising Business Value “The heart of data mining is transforming data into actionable results” Berry & Linoff

Where’s the payback? Large multi-national Large multi-national Undertook a review of their churn management process Undertook a review of their churn management process Led by an international consulting firm Led by an international consulting firm Executive management sponsorship Executive management sponsorship Chasing millions in potential benefits Chasing millions in potential benefits

What went right Everything! Everything!  Fully engaged with the business  Invested time in data exploration & preparation  Focused on the business issue rather than the technicalities  Every step documented  Project uncovered some excellent insights  Models developed showed lift of 3X or more All we had to do was deploy the models All we had to do was deploy the models

What went wrong Deploying the models Deploying the models

Agenda Introduction Introduction Pyle’s 9 Rules for Analytics Project Failure Pyle’s 9 Rules for Analytics Project Failure Why navigating Pyle’s 9 Rules still doesn’t guarantee success Why navigating Pyle’s 9 Rules still doesn’t guarantee success Incorporating the analytical model into the business process Incorporating the analytical model into the business process Summary Summary

The Starting Point Data Warehouse Manual Data Extracts Mining Tool Campaign Management System Churn Lists Outbound Call Lists Customer Management System

The Issues Poor Integration Poor Integration Huge degree of manual effort Huge degree of manual effort Large amount of latency Large amount of latency Non existent feedback loop Non existent feedback loop

The Impacts Introduced a high degree of risk every time the model was refreshed Introduced a high degree of risk every time the model was refreshed Restricted how often the churn propensity models could be run Restricted how often the churn propensity models could be run Drastically reduced the value in running the models Drastically reduced the value in running the models Made it extremely difficult to measure the performance of retention efforts Made it extremely difficult to measure the performance of retention efforts

The Goal To overcome the issues with the existing process To overcome the issues with the existing process To make the churn propensity scores more widely available To make the churn propensity scores more widely available

The Goal (cont’d) Data Warehouse Mining Tool Campaign Management System Customer Management System Direct Connect Contact List Automated Update Churn ScoresDirect Connect Outbound Call Lists

Challenge #1 Data Warehouse Mining Tool Campaign Management System Customer Management System Direct Connect Contact List Automated Update Churn ScoresDirect Connect Outbound Call Lists The Data Mining platform & licenses had to be completely upgraded The Data Mining platform & licenses had to be completely upgraded

Challenge #2 Data Warehouse Mining Tool Campaign Management System Customer Management System Direct Connect Contact List Automated Update Churn ScoresDirect Connect Outbound Call Lists The Data Warehouse was re-platformed mid project The Data Warehouse was re-platformed mid project

Challenge #3 Data Warehouse Mining Tool Campaign Management System Customer Management System Direct Connect Contact List Automated Update Churn ScoresDirect Connect Outbound Call Lists The Campaign Management System was replaced mid project The Campaign Management System was replaced mid project

Challenge #4 Data Warehouse Mining Tool Campaign Management System Customer Management System Direct Connect Contact List Automated Update Churn ScoresDirect Connect Outbound Call Lists The automated process to update the churn scores in the CRM just did not work The automated process to update the churn scores in the CRM just did not work

Finally Data Warehouse Mining Tool Campaign Management System Customer Management System Direct Connect Contact List Automated Update Churn ScoresDirect Connect Outbound Call Lists

The Long Awaited Benefits The time required to refresh the model was slashed by a factor of 10 The time required to refresh the model was slashed by a factor of 10 Churn propensity scores could be refreshed across the entire customer base on a monthly basis Churn propensity scores could be refreshed across the entire customer base on a monthly basis It became possible to accurately measure the success of the retention efforts It became possible to accurately measure the success of the retention efforts The Customer Services Representatives could finally recognize at risk customers during inbound calls. The Customer Services Representatives could finally recognize at risk customers during inbound calls.

Incorporating the model into the business “The more that the use of the analytical solution can be embedded into the business process being supported, the more likely it is that benefits will be realised” “The more that the use of the analytical solution can be embedded into the business process being supported, the more likely it is that benefits will be realised”

Incorporating the model into the business (cont’d) “The key to successful data mining is to incorporate the models into the business” “The key to successful data mining is to incorporate the models into the business” Berry & Linoff

Agenda Introduction Introduction Pyle’s 9 Rules for Analytics Project Failure Pyle’s 9 Rules for Analytics Project Failure Why navigating Pyle’s 9 Rules still doesn’t guarantee success Why navigating Pyle’s 9 Rules still doesn’t guarantee success Incorporating the analytical model into the business process Incorporating the analytical model into the business process Summary Summary

Summary Remember Pyle’s 9 Rules Remember Pyle’s 9 Rules BUT more importantly… Remember The Bigger Picture BUT more importantly… Remember The Bigger Picture

The Bigger Picture “Data mining is part, and a very small part, of a much larger business process. It may be an essential part of a data mining project, but incorporating the results of mining with all the related parts of the corporate project is equally, if not more, important for ultimate success” Dorian Pyle