Data Mining Dr. Chang Liu

What is Data Mining Data mining has been known by many different terms Data mining has been known by many different terms Knowledge Discovery in Database (KDD)Knowledge Discovery in Database (KDD) Predictive AnalyticsPredictive Analytics Machine LearningMachine Learning Business AnalyticsBusiness Analytics It is the process of finding hidden patterns in data It is the process of finding hidden patterns in data For example, what is the profile of people who buy from us?For example, what is the profile of people who buy from us? Usage of data mining has become widespread recently for various reasons Usage of data mining has become widespread recently for various reasons Typically, businesses find huge increases in profitability as a result of applying data mining Typically, businesses find huge increases in profitability as a result of applying data mining

Some Common Problems Growing business by cross-selling Growing business by cross-selling A retailer can use buying patterns of customers to generate recommendations for new customersA retailer can use buying patterns of customers to generate recommendations for new customers Determine risk of giving a loan to a particular customer Determine risk of giving a loan to a particular customer Profiles of customers who have defaulted in the past are learned and used with new customersProfiles of customers who have defaulted in the past are learned and used with new customers Forecast the likely unemployment level based on its past trend Forecast the likely unemployment level based on its past trend Is a credit card transaction likely to be a fraudulent? Is a credit card transaction likely to be a fraudulent? Is this tumor in a patient’s breast likely malignant? Is this tumor in a patient’s breast likely malignant?

Data Mining Tasks Data mining problems are solved by performing a specific task: Data mining problems are solved by performing a specific task: Given a problem, an analyst should first determine the data mining task that should be performed.Given a problem, an analyst should first determine the data mining task that should be performed. I need to determine whether a customer is likely to default a loan. I can solve this by performing a classification task I need to determine whether a customer is likely to default a loan. I can solve this by performing a classification task There are a number of tasks: There are a number of tasks: ClassificationClassification Association or market basket analysisAssociation or market basket analysis ForecastingForecasting Deviation AnalysisDeviation Analysis Clustering or segmentationClustering or segmentation Sequence analysisSequence analysis RegressionRegression

Data Mining Tasks (cont.) Classification is used to predict which of a few known outcomes a case is likely to be Classification is used to predict which of a few known outcomes a case is likely to be Is this customer likely to default? Has two known outcome “Yes” or “No”Is this customer likely to default? Has two known outcome “Yes” or “No” Association is used to analyze transaction tables and determine which items in the transaction table tend to go together. Example? Association is used to analyze transaction tables and determine which items in the transaction table tend to go together. Example? Forecasting is used to generate new data points in a time series. Example? Forecasting is used to generate new data points in a time series. Example? Deviation analysis is used to determine anomalous data points or outliers Deviation analysis is used to determine anomalous data points or outliers Used by security experts to detect network intrusion attacksUsed by security experts to detect network intrusion attacks Used by insurance companies and credit card companies to detect fraudUsed by insurance companies and credit card companies to detect fraud

Data Mining Tasks (cont.) Clustering or segmentation is used to discover natural grouping in data Clustering or segmentation is used to discover natural grouping in data Sequence analysis discovers sequence patterns in events Sequence analysis discovers sequence patterns in events E.g., purchase of a computer is followed by purchase of a printer, then webcam …E.g., purchase of a computer is followed by purchase of a printer, then webcam … Used by marketing folks to understand and exploit buying habitsUsed by marketing folks to understand and exploit buying habits Used to analyze web clickstream dataUsed to analyze web clickstream data Regression is used to predict numerical values Regression is used to predict numerical values

Data Ming Algorithms Microsoft SSAS provides the following data mining algorithms: Microsoft SSAS provides the following data mining algorithms: Microsoft Decision TreesMicrosoft Decision Trees Microsoft Neural NetworkMicrosoft Neural Network Microsoft Naïve BayesMicrosoft Naïve Bayes Microsoft Association RulesMicrosoft Association Rules Microsoft Time SeriesMicrosoft Time Series Microsoft ClusteringMicrosoft Clustering Microsoft Sequence ClusteringMicrosoft Sequence Clustering Microsoft Linear RegressionMicrosoft Linear Regression Microsoft Logistic RegressionMicrosoft Logistic Regression

Case The thing you are mining or asking questions about is called a case The thing you are mining or asking questions about is called a case The case is often a row in a table; e.g., when studying which customers are likely to default on a loan, each row in the customer table is a caseThe case is often a row in a table; e.g., when studying which customers are likely to default on a loan, each row in the customer table is a case Transaction tables are an example of nested casesTransaction tables are an example of nested cases

Attributes / Case Key Attributes are the variables that are used in the data mining analysis. Attributes are the variables that are used in the data mining analysis. Attributes are often columns in the case tableAttributes are often columns in the case table An attribute can be input or an output An attribute can be input or an output At modeling time, both input and output attributes are providedAt modeling time, both input and output attributes are provided At the prediction time, input attributes are used to predict output attributesAt the prediction time, input attributes are used to predict output attributes Case Key indicates the identity of the case Case Key indicates the identity of the case This is often the primary key or a row indexThis is often the primary key or a row index

Mining Structure / Mining Model A mining structure is a table that contains the columns to be analyzed. It also contains data mining models used to analyze the data. A mining structure is a table that contains the columns to be analyzed. It also contains data mining models used to analyze the data. Mining model defines how the problem is to be modeled. Mining model defines how the problem is to be modeled. Specify which columns to be included in the modelSpecify which columns to be included in the model Specify the algorithm to be usedSpecify the algorithm to be used Define which columns are input and which are outputDefine which columns are input and which are output

Training Models Many data mining algorithms requires historical data to learn patterns from Many data mining algorithms requires historical data to learn patterns from Training the model is also known as processing the model Training the model is also known as processing the model Typically, not all available historical data is used to train the model Typically, not all available historical data is used to train the model A percentage is left for testing purpose. This set is called the testing setA percentage is left for testing purpose. This set is called the testing set The data is used to train the model is called the training setThe data is used to train the model is called the training set

Class Activity_1 High school student historical data – CollegePlan table from DB661 High school student historical data – CollegePlan table from DB661 You are asked to find out what factors influence a high school student to go to college (or not) You are asked to find out what factors influence a high school student to go to college (or not) What data mining task would you perform? What data mining task would you perform? What is the case in this case? What is the case in this case? What is the case key? What is the case key? What algorithm(s) is/are applicable for this task? What algorithm(s) is/are applicable for this task? Which attribute(s) is/are input? Which attribute(s) is/are input? Which attribute(s) is/are output? Which attribute(s) is/are output?

Class Activity_2 Explore vmMSFTYear2008 data in DB661 Explore vmMSFTYear2008 data in DB661 Predict Microsoft stock values in the first week of 2009 (The real data is available at vmMSFTFirstWeek2009) Predict Microsoft stock values in the first week of 2009 (The real data is available at vmMSFTFirstWeek2009) Can you make money from MSFT based on your data mining knowledge? Can you make money from MSFT based on your data mining knowledge?

QUESTIONS??

With a new student table

Results