1. Partnership between research and industry for developing innovative data mining applications Bhavani Raskutti Analytic CRM Westpac

2. Presentation Title & Date 2 Tenet - Need strong and equal partnership between industry and research to develop and implement innovative data mining solutions to solve real world problems Research: data mining solution provider - Research arm of the business - External research provider  university, consultancy, software vendor Industry: data mining solution user - Business unit using analytics solutions - University departments  biology departments using analytics solutions

3. Presentation Title & Date 3 Argument Pathway Two “successful” projects - Text mining - CRM for business customers What does “success” mean? - Innovative solution - Implementation in business Discussion around - How the project started - The evolution of the project - Key success factors Role of industry & research for successful collaborations

4. Text Mining September 2000 - August 2005

5. Presentation Title & Date 5 Conception Text mining A file of in-bound SMS (from customers to Telstra) as a result of an out-bound SMS from Telstra - Over 700 entries in one day - Lots of abbreviations, acronyms and phonetic spelling - Use of space and punctuation was haphazard Why research laboratories? - Failure of standard techniques such as keyword matching in Excel - Ongoing relationship for customer retention modelling - Analysis was not high priority, so did not want to spend money to perform external manual analysis Open ended brief to identify “actionable” themes - No timeframes - No directions regarding what is meant by actionable themes

6. Presentation Title & Date 6 Evolution Demonstrated feasibility with output created by adapting in-house text clustering and summarisation tools to deal with SMS data and some manual processing in Excel - 30 themes including a catch-all containing around ~50 entries Feedback from client after seeing the output - Clusters need to refer to actionable themes - Want to do this cluster creation once-only Demonstrated prototype of text analysis tool on unix with all text analysis executables and a cluster-editor in Java - Positive feedback from business Research group invested in development of a stand-alone PC tool Text mining

7. Presentation Title & Date 7 Evolution (Cont’d) Text mining

8. Presentation Title & Date 8 Evolution (Cont’d) Text mining

9. Presentation Title & Date 9 Evolution (Cont’d) Business decided to restrict SMS for compliance messages only - No need for automated inbound SMS analysis tool Exploration of other applications for the tool - Identification of major issues from customer complaints data - Analysis of verbatim comments in customer satisfaction surveys - Analysis of verbatim comments in employee opinion surveys - Use of text classification part in KDD cup win - Part of demonstration package showcasing research credentials of Telstra Final application implemented in business in August 2005 for customer relationship management for SME customers - Business invested time is usability analysis of tool, and in training their business analysts - Research invested in changes identified by the usability test Text mining intellectual property (patents, code, etc.) licensed to an external company for commercialisation Text mining

10. Presentation Title & Date 10 Summary Text mining Open Ended Brief Feasibility / Vision FeedbackPrototypeFeedbackSystemDeploymentHand-over Find Another Partner Usability Analysis Updated System Ad-hoc analysis of customer complaints Ad-hoc analysis of verbatim comments in surveys Business analysts using the tool for CRM IP licensed externally Implementation Feature extraction to counter “dirty” text Automatic selection of number of clusters based on cohesion Use of patented support vector machine for text classification Interactive editing to fine-tune automatically generated groups Innovation

11. Presentation Title & Date 11 Success Factors Both parties wanted the project to happen - Industry: Needed to process the text data quickly - Research: An interesting problem to apply text mining techniques Collaboration process - Open-ended brief - Regular sharing of knowledge and ideas - Time for research ideas to be developed, piloted and implemented Multi-disciplinary team with researchers, programmers, psychologists and business users Contributions from both parties - Industry:  Data and interesting problem  Implementation support - Research:  Innovative solution  Investment  Selling idea into business  Continuity Text mining

12. CRM for business customers March 2003 - October 2004

13. Presentation Title & Date 13 Conception CRM for business customers Strong competitive pressures in the corporate customer segment - Large drops in margins for the first time - Needed to look for innovative methods to change the trend - Belief in the utility of customer analytics Why research laboratories? - Failure of traditional analysis approaches - Availability of internationally recognised expertise - Ability to provide objective solution - Willingness to work with analysts in business Open ended brief to increase overall value across the corporate customer base by using data mining - Aggressive targets - Short timeframes

14. Presentation Title & Date 14 Evolution CRM for business customers Discussions with stakeholders to determine: - Upsell, win-back or customer retention?  Upsell - What data should be used for mining  Quarterly revenue for 100 products - The segments to focus on  Medium & large businesses - The specific products to focus on Brief was to create upsell models using revenue data -- Straight- forward CRM problem? - High imbalance in class sizes - Average number of take-ups for any product in a quarter is small - Raw accuracy is not most important – need to identify high value take- ups even at the cost of missing many low value take-ups

15. Presentation Title & Date 15 Evolution (Cont’d) CRM for business customers A prototype to produce 5 prioritised customer lists for each segment - Generalised so it could be used to model any product - Models could be re-built every quarter - Models tested on time-independent hold-out set before releasing - Testing of methodology and algorithms on over 30 products and 2 segments Validation of models by business analysts - Comparison of sales opportunities identified manually vs by models - Pilot for medium businesses -- 2 non-metro regions in different states  Region 1: Predictions identified opportunities that were already being processed by sales consultants  Region 2: Predictions for just 5 products generated 9 new opportunities with an increase in revenue of ~400K A$ Predictive modelling spreads the techniques of good sales teams across the whole organisation

16. Presentation Title & Date 16 Evolution (Cont’d) CRM for business customers Business expanded the scope of application to include - More segments  Rural corporate customer segment  All other corporate customers in 3 segments: Large, Medium & Small - More products  Chosen by business analytics group  Different for different segments - Rebuild of models every quarter to avoid staleness Mechanism of delivery of outputs to sales consultants - Automated delivery of leads directly into front-end CRM systems along with supporting data to facilitate the sales - Delivery of prioritised customer lists to business analysts who then superimpose other business rules and create a set of leads

17. Presentation Title & Date 17 Evolution (Cont’d) CRM for business customers Initial implementation used for 4 quarters - 4 segments covering all corporate customers - ~50 products for each segment - Quarterly re-build of models and generation of scores by research - Prioritised customer lists one per product per segment delivered to business analysts - Model explanation provided through weighted rules - Sales consultants receive a list of leads for products Final implementation in business - System handed over to business analysts for model maintenance and scoring - No model explanation generated - Customer-centric lists suggesting a list of products per customer

18. Presentation Title & Date 18 Summary CRM for business customers Open Ended Brief PrototypeFeedbackPilotValidation Score Generation Deployment Hand-over To BA Techniques to boost the number of rare events for modelling Use of support vector machines for learning from unbalanced data Techniques to boost influence of high take-ups in training Use of value-weighted metrics to choose correct algorithm Innovation Research solution implemented in much wider context Scope change from 4 to 50 products Scope change from 2 to 4 segments Models updated quarterly, so no stale models in production Customer-centric lists was beyond the original brief Implementation Note: Published in “Predicting Product Purchase Patterns for Corporate Customers” by Bhavani Raskutti & Alan Herschtal in Proceedings of KDD’05, Chicago, Illinois, USA

19. Presentation Title & Date 19 Success Factors Both parties wanted the project to happen - Industry: Increase in sales - Research: Funding and relationship with that part of business Collaboration process: - Open-ended brief - Regular sharing of knowledge and ideas Multi-disciplinary team with researchers and business analysts - Willingness of stake-holders to try non-standard solutions and instigate change in process Contributions from both parties - Industry:  Problem & Access to data  Implementation support  Investment  Selling idea into business  Continuity - Research:  Innovative solution CRM for business customers Text Mining - Industry:  Data and interesting problem  Implementation support - Research:  Innovative solution  Investment  Selling idea into business  Continuity

20. What makes successful collaborations?

21. Presentation Title & Date 21 Summary Approach research only if problem is not amenable to traditional solutions Support research group with all necessary resources Essential: Data, Feedback, Usability testing, … Provide a free hand, however, be involved at all times Business Both parties should want the project Strong collaboration: respect and sharing Dynamic multi-disciplinary team Begin collaboration only if you need more than just money from industry Set up a collaborative process to ensure time commitment from business Set your own research agenda, however, keep communication lines open Think beyond the current project and build relationships Research Implementation SupportInnovative Solution Data and Problem ContinuityInvestmentSelling Idea into Business

22. Presentation Title & Date 22