A task of induction to find patterns Classification A task of induction to find patterns 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

CSE 572, CBS598: Data Mining by H. Liu Outline Data and its format Problem of Classification Learning a classifier Different approaches Key issues 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

CSE 572, CBS598: Data Mining by H. Liu Data and its format Data attribute-value pairs with/without class Data type continuous/discrete nominal Data format flat 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

CSE 572, CBS598: Data Mining by H. Liu Sample data 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

Induction from databases Inferring knowledge from data The task of deduction infer information that is a logical consequence of querying a database Who conducted this class before? Which courses are attended by Mary? Deductive databases: extending the RDBMS RDBMS - relational database management systems RDBMS offer simple operators for the deduction of information, such as join 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

CSE 572, CBS598: Data Mining by H. Liu Classification It is one type of induction data with class labels Examples - If weather is rainy then no golf If Induction is different from deduction and DBMS does not not support induction; The result of induction is higher-level information or knowledge: general statements about data There are many approaches. Refer to the lecture notes for CS3244 available at the Co-Op. We focus on three approaches here, other examples: Other approaches Instance-based learning other neural networks Concept learning (Version space, Focus, Aq11, …) Genetic algorithms Reinforcement learning 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

CSE 572, CBS598: Data Mining by H. Liu Different approaches There exist many techniques Decision trees Neural networks K-nearest neighbors Naïve Bayesian classifiers Support Vector Machines Ensemble methods Co-training and many more ... 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

CSE 572, CBS598: Data Mining by H. Liu A decision tree Outlook Humidity Wind sunny overcast rain YES high normal strong weak NO Issues How to build such a tree from the data? What are the criteria for performance measurement correctness conciseness What are the key components? test stopping criterion 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

Inducing a decision tree There are many possible trees let’s try it on the golfing data How to find the most compact one that is consistent with the data? Why the most compact? Occam’s razor principle Issue of efficiency w.r.t. optimality 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

CSE 572, CBS598: Data Mining by H. Liu Information gain and Entropy - Information gain - the difference between the node before and after splitting 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

Building a compact tree The key to building a decision tree - which attribute to choose in order to branch. The heuristic is to choose the attribute with the maximum IG. Another explanation is to reduce uncertainty as much as possible. 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

CSE 572, CBS598: Data Mining by H. Liu Learn a decision tree Outlook sunny overcast rain Humidity Wind YES high normal strong weak NO YES NO YES 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

Issues of Decision Trees Number of values of an attribute Your solution? When to stop Data fragmentation problem Any solution? Mixed data types Scalability 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

CSE 572, CBS598: Data Mining by H. Liu Rules and Tree stumps Generating rules from decision trees One path is a rule We can do better. Why? Tree stumps and 1R For each attribute value, determine a default class (#of values = # of rules) Calculate the # of errors for each rule Find # of errors for that attribute’s rule set Choose one rule set that has the least # of errors 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

CSE 572, CBS598: Data Mining by H. Liu K-Nearest Neighbor One of the most intuitive classification algorithm An unseen instance’s class is determined by its nearest neighbor The problem is it is sensitive to noise Instead of using one neighbor, we can use k neighbors 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

CSE 572, CBS598: Data Mining by H. Liu K-NN New problems How large should k be lazy learning – does it learn? large storage A toy example (noise, majority) How good is k-NN? How to compare Speed Accuracy 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

Naïve Bayes Classifier This is a direct application of Bayes’ rule P(C|X) = P(X|C)P(C)/P(X) X - a vector of x1,x2,…,xn That’s the best classifier we can build But, there are problems There are only a limited number of instances How to estimate P(x|C) 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

CSE 572, CBS598: Data Mining by H. Liu NBC (2) Assume conditional independence between xi’s We have P(C|x) ≈ P(x1|C) P(xi|C) (xn|C)P(C) An example How good is it in reality? 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

CSE 572, CBS598: Data Mining by H. Liu No Free Lunch If the goal is to obtain good generalization performance, there are no context-independent or usage-independent reasons to favor one learning or classification method over another What does it indicate? Source: Pattern Classification, 2nd Edition 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

CSE 572, CBS598: Data Mining by H. Liu Ensemble Methods Model generation Bagging (Bootstrap Aggregating) Boosting Model combination Majority voting Meta learning Stacking (using different types of classifiers) 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

Classification via Neural Networks Squash  A perceptron 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

What can a perceptron do? Neuron as a computing device To separate a linearly separable points Nice things about a perceptron distributed representation local learning weight adjusting 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

CSE 572, CBS598: Data Mining by H. Liu Linear threshold unit Basic concepts: projection, thresholding W vectors evoke 1 W = [.11 .6] L= [.7 .7] .5 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

E.g. 1: solution region for AND problem Find a weight vector that satisfies all the constraints AND problem 0 0 0 0 1 0 1 0 0 1 1 1 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

E.g. 2: Solution region for XOR problem? 0 0 0 0 1 1 1 0 1 1 1 0 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

Learning by error reduction Perceptron learning algorithm If the activation level of the output unit is 1 when it should be 0, reduce the weight on the link to the ith input unit by r*Li, where Li is the ith input value and r a learning rate If the activation level of the output unit is 0 when it should be 1, increase the weight on the link to the ith input unit by r*Li Otherwise, do nothing 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu

Multi-layer perceptrons Using the chain rule, we can back-propagate the errors for a multi-layer perceptrons. Output layer Hidden layer Differences between DT and NN Speed Accuracy Comprehensibility Which one to use Many successful applications of both approaches Input layer 5/25/2019 CSE 572, CBS598: Data Mining by H. Liu