Data Mining With SQL Server Data Tools Mining Data Using Tools You Already Have

Introductions  Annelies Beaty, Manager Enterprise Data Strategy at US Xpress  Played many roles at US Xpress over the years  My current role is to architect how Enterprise level data is managed and presented to the organization as a whole.  Development Practices and Guidelines  Tool evaluation  Step in and get my hands ‘dirty’ whenever possible, needed.

Data Mining (or Data Science)  What do we mean by Data Mining  Process By Which Large Sets of Data can be Analyzed for Actionable Information  Look for Answers to Questions over data sets so large you can get lost in it.  Find relationships that are too complex to be seen.  Types of Data Mining Scenarios  Forecasting – Predict future outcomes based on past experience  Risk and Probability – Based on Past results, what factors lead to the results we want  Recommendations – Based on ‘experience’, what else do we think goes with this set?  Finding Sequences – What are the frequent paths or steps taken through a system of possible steps.  Grouping – Separating the dataset into clusters of ‘like’ objects; determining affinity

High Level Data Management New Business Question is asked Answer is Delivered THIS TAKES TIME AND RESOURCES

Challenge with ‘Best Practice’ EDW  EDW development is methodical. Designed to answer a specific related set of questions around a business process.  Time to Deliver Results  Sometimes an ‘Overkill’ solution.  Sometimes an Incomplete solution.  Interesting Fact from a TDWI conference I attended about 2 years ago on Operational Intelligence: “50% of traditional data warehouses are not used in daily decision making”  What if the lifespan of the current ‘question of the day’ is very short? OR - what if you don’t even know the question?

So A Real Challenge Data Mining over (potentially) incomplete data fast enough to get the results needed by the business yesterday to make a strategic business decision -And Can we do it without significant investment in new tools

Gartner Magic Quadrant - Microsoft  Business Intelligence  Leader Quadrant for Completeness of Vision and Ability To Execute  Sql Server/SSIS/SSRS is a complete solution.  Product Quality, availability of skills, low implementation costs, alignment with existing infrastructure  Lacking a true Metadata Management solution and visualizations are not as good as some other vendors (but improving)  Advanced Analytics  Perhaps not so much – still a Niche Player  Product Quality, availability of skills, low implementation costs, alignment with existing infrastructure  Availability of analytics gives MS great reach into organizations that can serve as a springboard for future development  SSAS still lacks in depth and breadth, and usability, when compared to the leaders.  However – MS is expected to put a lot of energy into this space and has the means to do so. Source: Gartner Research *** Early/mid 2014

Demo 1 – Setting up a project  Availability of Skills  Don’t need to wait for SSAS  Easily works with existing infrastructure  Cost – If you have a SQL Server installation, you can do this now.  Set up and cursorily explore a Decision Tree model. Show the new objects on the backend SSAS server, single predictive query.

Predictive Algorithms  Decision Tree:  Presents the data as a series of ‘decisions’ used to reach the conclusion. A new branch is added when a significant correlation is found between the input and predicted variables.  Clustering:  Presents the input data as groups of entities with a high correlation of common attributes. ** Can be used to simply profile the data. Prediction is optional **  Naïve Bayes:  Quick method to analyze relationships between input and predictable columns; Less intense, but also less accurate. However, can be used to help define inputs for more accurate, but costly, solutions.  Neural Network:  Complex algorithm that evaluates every possible combination of inputs and outcome(s).

Results – Singleton Query RESULTS Decision TreeCluster Naïve BayesNeural Network Buyer% ChanceSupport Buyer% ChanceSupport Buyer% ChanceSupport Buyer% ChanceSupport Query Inputs

Demo  Exploration of 4 predictive data mining models.

Strengths of the predictive models  Naïve Bayes – Simplest computationally. May use up front to start the analysis since it processes faster. Use the results to refine the criteria for additional analysis with more complex tools. ** Cannot use continuous data as an input  Decision Tree – Used to predict outcomes based on past data, both discrete and continuous.  Clustering – Used to segment the dataset. Use of a predictable outcome is not required. Makes it useful to detect anomalies in the data.  Neural Network – most complex – can detect rules and relationships other methods can’t. Good use cases include those with large number of inputs and relatively few output: Text mining, (Stock) market analysis, manufacturing processes.

Other Data Mining Algorithms  Time Series: Allows us to use historical data to extrapolate a likely value at some point in the future.  Example: Predict Expected Sales By Region  Association Algorithm: Used to detect associations between items or events – the more frequently items or events occur together, the higher the correlation and the probability that if one occurs, the other will too.  Example – customers who bought this also bought this (Amazon)  Sequence Algorithm: Clusters sequences of events; Similar to cluster algorithm.  Example: Common paths through a website, or application.  Linear Regression Algorithm: allows us to explore the linear relationship between variables. Variation on Decision Trees.  Compute a trend line over sales and marketing data.  Logistic Regression: Variation of Neural Network used to model binary outcomes  Use demographics to determine likelihood of a predicted outcome, such as disease.

Resources  MSDN has substantial documentation and tutorials to bring you up to speed on each algorithm  Sql Server Central (a red gate community site) has a step by step Data Mining series of Articles that take you all the way through the MSDN tutorials on basic Data Mining and then how to leverage them…  SSIS packages to build them, exploration via Excel data mining tools, Power BI suite.