Amer Kanj Data Mining For Business Professionals

Contents Data Mining Overview Types of Data Mining Why use Data Mining How do we Mine Data Models of Data Mining

Data Mining Overview Data Mining deals with large volumes of data stored in DBMS It is the process of analyzing large databases to find useful patterns Data Mining is the process of automating information discovery It automates the process of discovering useful trends and patterns

Data Mining Overview (Cont) The fundamental assumption of Data Mining is that large data may contain recurring hidden patterns A Data Mining tool does not require any assumptions It tries to discover relationships and hidden patterns that may not always be obvious

Types of Data Mining Business professionals look for Data Mining approaches that meet their needs. They requires Data Mining to:  Be understandable  Have good performance  Be accurate They define three fundamental approaches to Data Mining :  Classification Studies  Clustering Studies  Visualization Studies

Classification Studies Classification studies = Supervised learning Very common in business world. A telecommunication company’s analyst wants to:  Understand why some customers remain loyal while others leave  Predict which customers likely to lose to competitors

Classification Studies (cont) So he can:  Construct a model derived from historical data of loyal customers versus customers who have left  A good model enables him to better understanding his customers and to predict which customer will stay and which will leave A study will identify an overall goal and the data to be used

Classification Rules Classification rules help assign new objects to a set of classes  Given a new automobile insurance applicant, should he/she be classified as low risk, medium risk or high risk? Classification rules for above example could use a variety of knowledge, such as educational level of applicant, salary of applicant, age of applicant, etc…   person p, p.degree = masters & p.income > 75,000  p.credit = excellent   person p, p.degree = bachelors and (p.income >= 25,000 and p.income <= 75,000)  p.credit = good

Classification rules can compactly shown as a decision tree

Clustering Studies Clustering Studies = Unsupervised Learning A method of grouping rows of data that share similar trends and patterns We have no dependent variable Clustering can also be based on historical patterns, but the outcome (loyal or lost) is not supplied with the training data Clustering techniques try to look for similarities within a data set and group similar rows together into clusters or segments

Customers are clustered into four segments Cluster 1 Income: High Children: 1 Car: Luxery Income: high Children: 0 Car: Compact Cluster 2 Income: Medium Children: 2 Car: Sedan and Car: Track Income: Medium Children: 3 Cluster 4 Cluster 3

Visualization It is simply the graphical presentation of data Microsoft Excel has graphing and mapping capabilities in its product Representing data graphically often brings out points that you would not normally see

Why use Data Mining Direct Marking Trend Analysis Fraud Detection Forecasting in Financial Markets

Direct Marketing The ability to predict who is most likely or most desirable to buy certain product can save companies immense amounts in marketing expenditures

Trend Analysis Understanding trends in the marketplace is a strategic advantage, because it is useful in reducing costs and timeliness to market

Fraud Detection data Mining techniques can model which insurance claims, cellular phone calls, or credit card purchases are likely to be fraudulent

Forecasting in Financial Markets The use of data mining to model financial markets is used extensively

How Do We Mine Data There are five steps to Data Mining:  Data Manipulating  Defining a study  Reading the data and building a model  Understanding the model  Prediction

Data Preparation Data preparation is considered as the heart of the Data Mining process Data usually accumulates in transactional database where actual records of transactions are stored Data preparation requires that the data from distributed databases be pooled together, cleansed from redundant, inconsistent, incomplete, irrelevant, and otherwise inappropriate data

Data Preparation (Cont) Data Cleaning:  A column containing a list of soft drinks may have the values “Pepsi”, “Pepsi Cola”, and “Cola”.  The values refer to the same drink, but are not known to the computer as the same. Missing Values:  Some Data Mining approaches require rows of data to be complete in order to mine the data  If too many values are missing in a data set, it becomes hard to gather any useful information from this data or to make predictions from it

Data Preparation (Cont) Data Derivation:  If I have column called maximum$-2002 and maximum$-2003 to describe the dollars spent in 2002 and 2003  Then an interesting derivation is $-difference, which is the change in amount of money spent between 2002 and 2003 Merging Data:  Data usually stored in the form of tables  Merging data in a relational system can be achieved in a number of ways: 1. Merging tables through a view (Query Tools) 2. An SQL statement, or 3. An export of data into a flat file

Defining a Study Differs from Supervised (Classification) versus Unsupervised (Clustering) learning For Supervised learning:  Involves articulating a goal  Specifying the data fields that are used in the study For Unsupervised learning:  The goal is to group similar types of data, usually used in many activities, or  To identify exceptions in a data set, which is useful in discovering fraudulent or incorrect data

Read the data and build a Model A data mining product reads a data set and constructs a model A model will summarize large amounts of data by accumulating indicators such Indicators:  Frequencies: Show how often a certain value occurs  Weight: or impacts, indicate how well some inputs indicate the occurrence of an output  Conjunctions: Sometimes inputs have more weight together than apart  Differentiation: Indicates how much more important an input criterion is to one outcome than another

Understanding the Model Model understand takes different forms based on the type of model used to represent the data We will discuss Data Mining Models later…

Prediction Prediction is the process of choosing the best possible outcomes based on historical data Predictive data mining methods fall into three broad categories:  Mathematical methods  Logic methods  Distance methods

Prediction (Cont) Mathematical method:  Linear math solution  Non-linear math solution Logic methods:  Quite different from what math methods produce  Logical methods often produce tree-like solutions  Best known logical solutions are decision trees, and decision rules.

Prediction (Cont) Distance methods:  A representative sample of cases is kept on file  These cases will be used as a benchmark for classifying new cases  Features of the new case are measured against features of the benchmark cases for proximity

Prediction (Cont) Here are a few interesting predictive capabilities :  Understanding why a prediction is made: some models will provide the reasons why a prediction is made  Margin of victory: if the best case prediction has a score of 100 and the challenger prediction has a score of 50, then the margin of victory is 50%. If the prediction has a score of 100 and the challenger has 99, then the margin of victory would be 1%. Generally, the higher the margin of victory, the more likely the prediction is to be true

Prediction (Cont)  Scenario playing: Some prediction models have the ability to change parameters to see how predictions change  Understanding prediction affinities: Is to set two variables constant and see what the other predictions would look like

Data Mining Models Decision Trees Genetic Algorithms Neural Nets Agent Network Technology Hybrid Models Statistics

Data mining Models (Cont) Decision Trees:  Creating a tree-like structure to describe a data set  The greatest benefit to decision tree approaches is their understandability Genetic Algorithms:  Are a method of combinatorial optimization based on processes in biological evolution

Data Mining Models (Cont) Neural Nets:  Are used extensively in the business world as predictive models  Neural Nets are widely used in the financial market to model fraud in credit cards and monetary transactions Agent Network Technology:  This method of model treats all data elements as agents that are connected to each other in a significant way

Data Mining Models (Cont) Hybrid Models:  Vendor Tools that make use of more than one approach are referred to as hybrid systems  Being a hybrid system does not always imply that the tool uses a hybrid algorithm  For example, Thinking Machines, with their Darwin product, makes use of several different mining algorithm. While the algorithm themselves are not hybrid, the product uses the algorithms in combination

Data Mining Models (Cont) Statistics:  Used to create a model of data sets  Uses probability, data analysis, and statistical inference

Thank You For Listening Q s …Q S … Q s