Data Mining: Concepts and Techniques Classification November 11, 2018 Data Mining: Concepts and Techniques

Chapter 7. Classification and Prediction What is classification? What is prediction? Issues regarding classification and prediction Classification by decision tree induction Bayesian Classification Other Classification Methods Prediction Classification accuracy Summary November 11, 2018 Data Mining: Concepts and Techniques

Classification vs. Prediction predicts categorical class labels classifies data (constructs a model) based on the training set and the values (class labels) in a classifying attribute and uses it in classifying new data Prediction: models continuous-valued functions, i.e., predicts unknown or missing values Typical Applications credit approval target marketing medical diagnosis treatment effectiveness analysis November 11, 2018 Data Mining: Concepts and Techniques

Classification—A Two-Step Process Model construction: describing a set of predetermined classes Each tuple/sample is assumed to belong to a predefined class, as determined by the class label attribute The set of tuples used for model construction: training set The model is represented as classification rules, decision trees, or mathematical formulae Model usage: for classifying future or unknown objects Estimate accuracy of the model The known label of test sample is compared with the classified result from the model Accuracy rate is the percentage of test set samples that are correctly classified by the model Test set is independent of training set, otherwise over-fitting will occur November 11, 2018 Data Mining: Concepts and Techniques

Classification Process (1): Model Construction Algorithms Training Data Classifier (Model) IF rank = ‘professor’ OR years > 6 THEN tenured = ‘yes’ November 11, 2018 Data Mining: Concepts and Techniques

Classification Process (2): Use the Model in Prediction Classifier Testing Data Unseen Data (Jeff, Professor, 4) Tenured? November 11, 2018 Data Mining: Concepts and Techniques

Supervised vs. Unsupervised Learning Supervised learning (classification) Supervision: The training data (observations, measurements, etc.) are accompanied by labels indicating the class of the observations New data is classified based on the training set Unsupervised learning (clustering) The class labels of training data is unknown Given a set of measurements, observations, etc. with the aim of establishing the existence of classes or clusters in the data November 11, 2018 Data Mining: Concepts and Techniques

Chapter 7. Classification and Prediction What is classification? What is prediction? Issues regarding classification and prediction Classification by decision tree induction Bayesian Classification Other Classification Methods Prediction Classification accuracy Summary November 11, 2018 Data Mining: Concepts and Techniques

Issues regarding classification and prediction (1): Data Preparation Data cleaning Preprocess data in order to reduce noise and handle missing values Relevance analysis (feature selection) Remove the irrelevant or redundant attributes Data transformation Generalize and/or normalize data November 11, 2018 Data Mining: Concepts and Techniques

Data Mining: Concepts and Techniques Issues regarding classification and prediction (2): Evaluating Classification Methods Predictive accuracy Speed time to construct the model time to use the model Robustness handling noise and missing values Scalability efficiency in disk-resident databases Interpretability: understanding and insight provided by the model Goodness of rules decision tree size compactness of classification rules November 11, 2018 Data Mining: Concepts and Techniques

Chapter 7. Classification and Prediction What is classification? What is prediction? Issues regarding classification and prediction Classification by decision tree induction Bayesian Classification Other Classification Methods Prediction Classification accuracy Summary November 11, 2018 Data Mining: Concepts and Techniques

Classification by Decision Tree Induction A flow-chart-like tree structure Internal node denotes a test on an attribute Branch represents an outcome of the test Leaf nodes represent class labels or class distribution Decision tree generation consists of two phases Tree construction At start, all the training examples are at the root Partition examples recursively based on selected attributes Tree pruning Identify and remove branches that reflect noise or outliers Use of decision tree: Classifying an unknown sample Test the attribute values of the sample against the decision tree November 11, 2018 Data Mining: Concepts and Techniques

Data Mining: Concepts and Techniques Training Dataset This follows an example from Quinlan’s ID3 November 11, 2018 Data Mining: Concepts and Techniques

Output: A Decision Tree for “buys_computer” age? <=30 overcast 30..40 >40 student? yes credit rating? no yes excellent fair no yes no yes November 11, 2018 Data Mining: Concepts and Techniques

Algorithm for Decision Tree Induction Basic algorithm (a greedy algorithm) Tree is constructed in a top-down recursive divide-and-conquer manner At start, all the training examples are at the root Attributes are categorical (if continuous-valued, they are discretized in advance) Examples are partitioned recursively based on selected attributes Test attributes are selected on the basis of a heuristic or statistical measure (e.g., information gain) Conditions for stopping partitioning All samples for a given node belong to the same class There are no remaining attributes for further partitioning – majority voting is employed for classifying the leaf There are no samples left for the branch attribute=ai November 11, 2018 Data Mining: Concepts and Techniques

Attribute Selection Measure Information gain (ID3/C4.5) All attributes are assumed to be categorical Can be modified for continuous-valued attributes Gini index (IBM IntelligentMiner) All attributes are assumed continuous-valued Assume there exist several possible split values for each attribute May need other tools, such as clustering, to get the possible split values Can be modified for categorical attributes November 11, 2018 Data Mining: Concepts and Techniques

Data Mining: Concepts and Techniques Information Gain Main idea: information conveyed by a message depends on its probability and can be measured in bits as: Example: if there are 8 equally probable messages, the information conveyed by one of them is: November 11, 2018 Data Mining: Concepts and Techniques

Data Mining: Concepts and Techniques Information Gain Assume that there are two classes N and P. Let the set of examples S contain p elements of class P and n elements of class N. Imagine selecting one case at random from a set S and announcing that it belongs to class P. This message has the probability: The information it conveys is: The amount of information, needed to decide if an arbitrary example in S belongs to P or N is defined as November 11, 2018 Data Mining: Concepts and Techniques

Information Gain in Decision Tree Induction Assume that by using attribute A set S will be partitioned into sets {S1, S2 , …, Sv} If Si contains pi examples of P and ni examples of N, the entropy, or the expected information needed to classify objects in all subtrees Si is The encoding information that would be gained by branching on A November 11, 2018 Data Mining: Concepts and Techniques

Attribute Selection by Information Gain Computation Hence Similarly Class P: buys_computer = “yes” Class N: buys_computer = “no” I(p, n) = I(9, 5) =0.940 Compute the entropy for age: November 11, 2018 Data Mining: Concepts and Techniques

Gini Index (IBM IntelligentMiner) If a data set T contains examples from n classes, gini index, gini(T) is defined as where pj is the relative frequency of class j in T. If a data set T is split into two subsets T1 and T2 with sizes N1 and N2 respectively, the gini index of the split data contains examples from n classes, the gini index gini(T) is defined as The attribute provides the smallest ginisplit(T) is chosen to split the node (need to enumerate all possible splitting points for each attribute). November 11, 2018 Data Mining: Concepts and Techniques

Extracting Classification Rules from Trees Represent the knowledge in the form of IF-THEN rules One rule is created for each path from the root to a leaf Each attribute-value pair along a path forms a conjunction The leaf node holds the class prediction Rules are easier for humans to understand Example IF age = “<=30” AND student = “no” THEN buys_computer = “no” IF age = “<=30” AND student = “yes” THEN buys_computer = “yes” IF age = “31…40” THEN buys_computer = “yes” IF age = “>40” AND credit_rating = “excellent” THEN buys_computer = “yes” IF age = “>40” AND credit_rating = “fair” THEN buys_computer = “no” November 11, 2018 Data Mining: Concepts and Techniques

Avoid Overfitting in Classification The generated tree may overfit the training data Too many branches, some may reflect anomalies due to noise or outliers Result is in poor accuracy for unseen samples Two approaches to avoid overfitting Prepruning: Halt tree construction early—do not split a node if this would result in the goodness measure falling below a threshold Difficult to choose an appropriate threshold Postpruning: Remove branches from a “fully grown” tree—get a sequence of progressively pruned trees Use a set of data different from the training data to decide which is the “best pruned tree” November 11, 2018 Data Mining: Concepts and Techniques

Enhancements to basic decision tree induction Allow for continuous-valued attributes Dynamically define new discrete-valued attributes that partition the continuous attribute value into a discrete set of intervals Handle missing attribute values Assign the most common value of the attribute Assign probability to each of the possible values Attribute construction Create new attributes based on existing ones that are sparsely represented This reduces fragmentation, repetition, and replication November 11, 2018 Data Mining: Concepts and Techniques

Classification in Large Databases Classification—a classical problem extensively studied by statisticians and machine learning researchers Scalability: Classifying data sets with millions of examples and hundreds of attributes with reasonable speed Why decision tree induction in data mining? relatively faster learning speed (than other classification methods) convertible to simple and easy to understand classification rules can use SQL queries for accessing databases comparable classification accuracy with other methods November 11, 2018 Data Mining: Concepts and Techniques

Presentation of Classification Results November 11, 2018 Data Mining: Concepts and Techniques

Scalable Decision Tree Induction Methods in Data Mining Studies SLIQ (EDBT’96 — Mehta et al.) builds an index for each attribute and only class list and the current attribute list reside in memory SPRINT (VLDB’96 — J. Shafer et al.) constructs an attribute list data structure PUBLIC (VLDB’98 — Rastogi & Shim) integrates tree splitting and tree pruning: stop growing the tree earlier RainForest (VLDB’98 — Gehrke, Ramakrishnan & Ganti) separates the scalability aspects from the criteria that determine the quality of the tree builds an AVC-list (attribute, value, class label) November 11, 2018 Data Mining: Concepts and Techniques

SLIQ: A Fast Scalable Classifier Shortcoming of previous classification algorithms ONLY for memory-resident data Purpose of SLIQ scale for LARGE data sets November 11, 2018 Data Mining: Concepts and Techniques

Decision-tree Classification Tree Building – repeatedly partitioning the training data evaluation and selection of the best splitting point creation of partitions using the best split Tree Pruning -- Minimum Description Length November 11, 2018 Data Mining: Concepts and Techniques

Find Best Splitting Point Splitting Index ( gini index )  Minimize it Splits for numeric attributes Sort on the values of this attribute (v1, v2, ..., vn) (n -1) possible splits (vi – vi+1) Splits for categorical attributes (A) November 11, 2018 Data Mining: Concepts and Techniques

Data Mining: Concepts and Techniques Improvement in SLIQ Sorting in EVERY node (previous algorithms) Pre-Sorting Technique (sort only ONCE) Breadth-First Growth A fast subsetting algorithm for splitting categorical attributes A new tree-pruning algorithm November 11, 2018 Data Mining: Concepts and Techniques

Data Structures in SLIQ Attribute Lists < attribute value, index into the class list> Class List < a class label, reference to a leaf node> Class Histogram ( to compute frequency for gini splitting index ) numeric attribute: <class, frequency> categorical attribute: <attribute value, class, frequency> November 11, 2018 Data Mining: Concepts and Techniques

Evaluation of Splitting Points(1) Splits for all the leaves are simultaneously evaluated . Evaluate Splits() for each attribute A do traverse attribute list of A for each value v in attribute list find the corresponding entry in the class list, and hence the corresponding class and the leaf node (l) update class histogram in l if A is a numeric attribute compute splitting index for test (A <= v) for leaf l if A is a categorical attribute for each leaf of the tree do find subset of A with best split November 11, 2018 Data Mining: Concepts and Techniques

Evaluation of Splitting Points (Numeric) (age <= 35) November 11, 2018 Data Mining: Concepts and Techniques

Evaluation of Splitting Points (Categorical) Hybrid approach for categorical attributes: cardinality of S < threshold, all the subsets of S are evaluated cardinality of S >= threshold a greedy algorithm is used (starting with an empty subset, and add one element into it, which gives the best split) November 11, 2018 Data Mining: Concepts and Techniques

Updating the Class List November 11, 2018 Data Mining: Concepts and Techniques

Data Mining: Concepts and Techniques SLIQ Tree Pruning MDL(Minimum Description Length) principle cost ( M, D) = cost ( D | M ) + cost ( M ) M: the model encoding the data D: the data in terms of the model in decision tree: M: the result tree after pruning D: the training set November 11, 2018 Data Mining: Concepts and Techniques

Performance Evaluation (Small Dataset) Classification Accuracy November 11, 2018 Data Mining: Concepts and Techniques

Performance Evaluation (Small Dataset) Pruned-tree Size November 11, 2018 Data Mining: Concepts and Techniques

Performance Evaluation (Small Dataset) Execution Times November 11, 2018 Data Mining: Concepts and Techniques

Performance Evaluation (Scalability) November 11, 2018 Data Mining: Concepts and Techniques

Chapter 7. Classification and Prediction What is classification? What is prediction? Issues regarding classification and prediction Classification by decision tree induction Bayesian Classification Other Classification Methods Prediction Classification accuracy Summary November 11, 2018 Data Mining: Concepts and Techniques

Bayesian Classification: Why? Probabilistic learning: Calculate explicit probabilities for hypothesis, among the most practical approaches to certain types of learning problems Incremental: Each training example can incrementally increase/decrease the probability that a hypothesis is correct. Prior knowledge can be combined with observed data. Probabilistic prediction: Predict multiple hypotheses, weighted by their probabilities Standard: Even when Bayesian methods are computationally intractable, they can provide a standard of optimal decision making against which other methods can be measured November 11, 2018 Data Mining: Concepts and Techniques

Data Mining: Concepts and Techniques Bayesian Theorem Given training data D, posteriori probability of a hypothesis h, P(h|D) follows the Bayes theorem Practical difficulty: require initial knowledge of many probabilities, significant computational cost November 11, 2018 Data Mining: Concepts and Techniques

Bayesian classification The classification problem may be formalized using a-posteriori probabilities: P(C|X) = prob. that the sample tuple X=<x1,…,xk> is of class C. E.g. P(class=N | outlook=sunny,windy=true,…) Idea: assign to sample X the class label C such that P(C|X) is maximal November 11, 2018 Data Mining: Concepts and Techniques

Estimating a-posteriori probabilities Bayes theorem: P(C|X) = P(X|C)·P(C) / P(X) P(X) is constant for all classes P(C) = relative freq of class C samples C such that P(C|X) is maximum = C such that P(X|C)·P(C) is maximum Problem: computing P(X|C) is unfeasible! November 11, 2018 Data Mining: Concepts and Techniques

Naïve Bayesian Classification Naïve assumption: attribute independence P(x1,…,xk|C) = P(x1|C)·…·P(xk|C) If i-th attribute is categorical: P(xi|C) is estimated as the relative freq of samples having value xi as i-th attribute in class C If i-th attribute is continuous: P(xi|C) is estimated thru a Gaussian density function Computationally easy in both cases November 11, 2018 Data Mining: Concepts and Techniques

Play-tennis example: estimating P(xi|C) outlook P(sunny|p) = 2/9 P(sunny|n) = 3/5 P(overcast|p) = 4/9 P(overcast|n) = 0 P(rain|p) = 3/9 P(rain|n) = 2/5 temperature P(hot|p) = 2/9 P(hot|n) = 2/5 P(mild|p) = 4/9 P(mild|n) = 2/5 P(cool|p) = 3/9 P(cool|n) = 1/5 humidity P(high|p) = 3/9 P(high|n) = 4/5 P(normal|p) = 6/9 P(normal|n) = 2/5 windy P(true|p) = 3/9 P(true|n) = 3/5 P(false|p) = 6/9 P(false|n) = 2/5 P(p) = 9/14 P(n) = 5/14 November 11, 2018 Data Mining: Concepts and Techniques

Play-tennis example: classifying X An unseen sample X = <rain, hot, high, false> P(X|p)·P(p) = P(rain|p)·P(hot|p)·P(high|p)·P(false|p)·P(p) = 3/9·2/9·3/9·6/9·9/14 = 0.010582 P(X|n)·P(n) = P(rain|n)·P(hot|n)·P(high|n)·P(false|n)·P(n) = 2/5·2/5·4/5·2/5·5/14 = 0.018286 Sample X is classified in class n (don’t play) November 11, 2018 Data Mining: Concepts and Techniques

The independence hypothesis… … makes computation possible … yields optimal classifiers when satisfied … but is seldom satisfied in practice, as attributes (variables) are often correlated. Attempts to overcome this limitation: Bayesian networks, that combine Bayesian reasoning with causal relationships between attributes Decision trees, that reason on one attribute at the time, considering most important attributes first November 11, 2018 Data Mining: Concepts and Techniques

Bayesian Belief Networks (I) Family History Smoker (FH, S) (FH, ~S) (~FH, S) (~FH, ~S) LC 0.8 0.5 0.7 0.1 LungCancer Emphysema ~LC 0.2 0.5 0.3 0.9 The conditional probability table for the variable LungCancer PositiveXRay Dyspnea Bayesian Belief Networks November 11, 2018 Data Mining: Concepts and Techniques

Bayesian Belief Networks (II) Joint probability of a tuple is computed as: Bayesian belief network allows a subset of the variables conditionally independent A graphical model of causal relationships November 11, 2018 Data Mining: Concepts and Techniques

Chapter 7. Classification and Prediction What is classification? What is prediction? Issues regarding classification and prediction Classification by decision tree induction Bayesian Classification Other Classification Methods Prediction Classification accuracy Summary November 11, 2018 Data Mining: Concepts and Techniques

Other Classification Methods k-nearest neighbor classifier Case-based reasoning Genetic algorithm Rough set approach Fuzzy set approaches November 11, 2018 Data Mining: Concepts and Techniques

Chapter 7. Classification and Prediction What is classification? What is prediction? Issues regarding classification and prediction Classification by decision tree induction Bayesian Classification Other Classification Methods Prediction Classification accuracy Summary November 11, 2018 Data Mining: Concepts and Techniques

Data Mining: Concepts and Techniques What Is Prediction? Prediction is similar to classification First, construct a model Second, use model to predict unknown value Major method for prediction is regression Linear and multiple regression Non-linear regression Prediction is different from classification Classification refers to predict categorical class label Prediction models continuous-valued functions November 11, 2018 Data Mining: Concepts and Techniques

Data Mining: Concepts and Techniques Regress Analysis Linear regression: Y =  +  X Two parameters ,  and  specify the line and are to be estimated by using the data at hand. using the least squares criterion to the known values of Y1, Y2, …, X1, X2, …. Multiple regression: Y = b0 + b1 X1 + b2 X2. Many nonlinear functions can be transformed into the above. November 11, 2018 Data Mining: Concepts and Techniques

Data Mining: Concepts and Techniques A simple example If the university authorities want to predict a student's grade on a freshman college calculus midterm based on his/her SAT score, then they may apply linear regression. November 11, 2018 Data Mining: Concepts and Techniques

Linear Regression result: Y = 65.4 + 0.02 X November 11, 2018 Data Mining: Concepts and Techniques

NBA Salary Prediction Example Use player’s game statistics data to determine if their salaries are due to performance or just popularity. Consider variables that affect salaries positively: Minutes per game Points per game Assists per game Rebounds per game. Try to find the most significant variables that can predict the player’s salary. We felt that minutes per game was important in our analysis because we felt the more minutes the athlete played, the more he should earn. Points per game was included as a variable because we thought it to be a measure of the worth of that player to the team Rebounds and assists are also important in judging the athlete's worth to the team. November 11, 2018 Data Mining: Concepts and Techniques

Data Mining: Concepts and Techniques NBA Statistics Data November 11, 2018 Data Mining: Concepts and Techniques

Variable Selection and Statistical Significance Linear Regression: Y = -.716-.425A-.0756M+.742P+0.0536 Only the variable points-per-game is significant. Adjusted model: Y = -3.42 + .742 P The usual interpretation of R2 is as the relative amount of variance of the dependent variable y explained or accounted for by the explanatory variables x1, x2,..., xq. For example, if R2 = 0.762 we say that the explanatory variables "explains" 76.2 % of the variance of y. ), the squared multiple correlation also called the coefficient of determination is defined as  R2 = 1 - Var(e)/Var(y). E is the error. Y dependent variable. November 11, 2018 Data Mining: Concepts and Techniques

Data Mining: Concepts and Techniques Model analysis In the previous NBA salary example, Y = -3.42 + .742 P Statistics R2 = 0.458. So, only 45.8 % of the salary is caused by Points-Per-Game. For further analysis, we may examine the player's popularity, the impact of their agent, the perceived added income they bring to the team, as well as other, qualitative, variables. November 11, 2018 Data Mining: Concepts and Techniques

Issues of regression analysis in large data sets Regression analysis handles a data set as a whole. Not realistic to assume a single model can fit a large data set. Stringent model assumptions fail in the real world. November 11, 2018 Data Mining: Concepts and Techniques

Data Mining: Concepts and Techniques A new method Regression-class mixture decomposition method ( RCMD) View a complicated data set as a mixture of a finite number of regression models. Goal: Identification of these models. November 11, 2018 Data Mining: Concepts and Techniques

Chapter 7. Classification and Prediction What is classification? What is prediction? Issues regarding classification and prediction Classification by decision tree induction Bayesian Classification Classification by backpropagation Classification based on concepts from association rule mining Other Classification Methods Prediction Classification accuracy Summary November 11, 2018 Data Mining: Concepts and Techniques

Classification Accuracy: Estimating Error Rates Partition: Training-and-testing use two independent data sets, e.g., training set (2/3), test set(1/3) used for data set with large number of samples Cross-validation divide the data set into k subsamples use k-1 subsamples as training data and one sub-sample as test data --- k-fold cross-validation iterate k times November 11, 2018 Data Mining: Concepts and Techniques

Data Mining: Concepts and Techniques Boosting Boosting increases classification accuracy Applicable to decision trees or Bayesian classifier Learn a series of classifiers, where each classifier in the series pays more attention to the examples misclassified by its predecessor Boosting requires only linear time and constant space November 11, 2018 Data Mining: Concepts and Techniques

Chapter 7. Classification and Prediction What is classification? What is prediction? Issues regarding classification and prediction Classification by decision tree induction Bayesian Classification Classification by backpropagation Classification based on concepts from association rule mining Other Classification Methods Prediction Classification accuracy Summary November 11, 2018 Data Mining: Concepts and Techniques

Data Mining: Concepts and Techniques Summary Classification is an extensively studied problem (mainly in statistics, machine learning & neural networks) Classification is probably one of the most widely used data mining techniques with a lot of extensions Scalability is still an important issue for database applications: thus combining classification with database techniques should be a promising topic Research directions: classification of non-relational data, e.g., text, spatial, multimedia, etc.. November 11, 2018 Data Mining: Concepts and Techniques