Naïve Bayes Classifier Ke Chen Modified and extended by Longin Jan Latecki

Slides:

Advertisements

Similar presentations

Based on slides by Pierre Dönnes and Ron Meir Modified by Longin Jan Latecki, Temple University Ch. 5: Support Vector Machines Stephen Marsland, Machine.

Advertisements

Classification Techniques: Decision Tree Learning

What we will cover here What is a classifier

Naïve Bayes Classifier

Naïve Bayes Classifier

On Discriminative vs. Generative classifiers: Naïve Bayes

Algorithms: The basic methods. Inferring rudimentary rules Simplicity first Simple algorithms often work surprisingly well Many different kinds of simple.

Assuming normally distributed data! Naïve Bayes Classifier.

Data Mining with Naïve Bayesian Methods

Introduction to Bayesian Learning Bob Durrant School of Computer Science University of Birmingham (Slides: Dr Ata Kabán)

Review. 2 Statistical modeling  “Opposite” of 1R: use all the attributes  Two assumptions: Attributes are  equally important  statistically independent.

SEG Tutorial 1 – Classification Decision tree, Naïve Bayes & k-NN CHANG Lijun.

Introduction to Bayesian Learning Ata Kaban School of Computer Science University of Birmingham.

Algorithms for Classification: The Basic Methods.

Revision (Part II) Ke Chen COMP24111 Machine Learning Revision slides are going to summarise all you have learnt from Part II, which should be helpful.

Thanks to Nir Friedman, HU

Review Rong Jin. Comparison of Different Classification Models  The goal of all classifiers Predicating class label y for an input x Estimate p(y|x)

SEEM Tutorial 2 Classification: Decision tree, Naïve Bayes & k-NN

Jeff Howbert Introduction to Machine Learning Winter Classification Bayesian Classifiers.

Crash Course on Machine Learning

Naïve Bayes Classifier Ke Chen Extended by Longin Jan Latecki COMP20411 Machine Learning.

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

NAÏVE BAYES CLASSIFIER 1 ACM Student Chapter, Heritage Institute of Technology 10 th February, 2012 SIGKDD Presentation by Anirban Ghose Parami Roy Sourav.

Bayesian Networks. Male brain wiring Female brain wiring.

DATA MINING LECTURE 10 Classification k-nearest neighbor classifier Naïve Bayes Logistic Regression Support Vector Machines.

Last lecture summary Naïve Bayes Classifier. Bayes Rule Normalization Constant LikelihoodPrior Posterior Prior and likelihood must be learnt (i.e. estimated.

Naive Bayes Classifier

Bayesian Learning CS446 -FALL ‘14 f:X  V, finite set of values Instances x  X can be described as a collection of features x = (x 1, x 2, … x n ) x i.

1 CS598: Machine Learning and Natural Language Lecture 7: Probabilistic Classification Oct. 19, Dan Roth University of Illinois, Urbana-Champaign.

Bayesian Classification. Bayesian Classification: Why? A statistical classifier: performs probabilistic prediction, i.e., predicts class membership probabilities.

Classification Techniques: Bayesian Classification

Naïve Bayes Classifier Ke Chen Modified and extended by Longin Jan Latecki

Last lecture summary (SVM). Support Vector Machine Supervised algorithm Works both as – classifier (binary) – regressor De facto better linear classification.

Data Mining Practical Machine Learning Tools and Techniques Chapter 4: Algorithms: The Basic Methods Section 4.2 Statistical Modeling Rodney Nielsen Many.

Algorithms for Classification: The Basic Methods.

CHAPTER 6 Naive Bayes Models for Classification. QUESTION????

Mehdi Ghayoumi MSB rm 132 Ofc hr: Thur, a Machine Learning.

A C B. Will I play tennis today? Features – Outlook: {Sun, Overcast, Rain} – Temperature:{Hot, Mild, Cool} – Humidity:{High, Normal, Low} – Wind:{Strong,

DATA MINING LECTURE 10b Classification k-nearest neighbor classifier

COMP24111 Machine Learning Naïve Bayes Classifier Ke Chen.

Bayesian Learning. Bayes Classifier A probabilistic framework for solving classification problems Conditional Probability: Bayes theorem:

BAYESIAN LEARNING. 2 Bayesian Classifiers Bayesian classifiers are statistical classifiers, and are based on Bayes theorem They can calculate the probability.

Knowledge-based systems Sanaullah Manzoor CS&IT, Lahore Leads University

Data Mining Chapter 4 Algorithms: The Basic Methods Reporter: Yuen-Kuei Hsueh.

Naive Bayes Classifier. REVIEW: Bayesian Methods Our focus this lecture: – Learning and classification methods based on probability theory. Bayes theorem.

Review of Decision Tree Learning Bamshad Mobasher DePaul University Bamshad Mobasher DePaul University.

COMP24111 Machine Learning Naïve Bayes Classifier Ke Chen.

Last lecture summary Naïve Bayes Classifier. Bayes Rule Normalization Constant LikelihoodPrior Posterior Prior and likelihood must be learnt (i.e. estimated.

Bayesian Learning Reading: Tom Mitchell, “Generative and discriminative classifiers: Naive Bayes and logistic regression”, Sections 1-2. (Linked from.

Classification: Naïve Bayes Classifier

Naïve Bayes Classifier

Naive Bayes Classifier

CSE543: Machine Learning Lecture 2: August 6, 2014

text processing And naïve bayes

Data Science Algorithms: The Basic Methods

Decision Trees: Another Example

Naïve Bayes Classifier

Naïve Bayes Classifier

Data Mining Lecture 11.

Classification Techniques: Bayesian Classification

Revision (Part II) Ke Chen

Revision (Part II) Ke Chen

Naïve Bayes Classifier

Generative Models and Naïve Bayes

Artificial Intelligence 9. Perceptron

Generative Models and Naïve Bayes

A task of induction to find patterns

NAÏVE BAYES CLASSIFICATION

Naïve Bayes Classifier

Naïve Bayes Classifier

Presentation transcript:

Naïve Bayes Classifier Ke Chen Modified and extended by Longin Jan Latecki

2 Outline Background Probability Basics Probabilistic Classification Naïve Bayes Example: Play Tennis Relevant Issues Conclusions

3 Background There are three methods to establish a classifier a) Model a classification rule directly Examples: k-NN, decision trees, perceptron, SVM b) Model the probability of class memberships given input data Example: multi-layered perceptron with the cross-entropy cost c) Make a probabilistic model of data within each class Examples: naive Bayes, model based classifiers a) and b) are examples of discriminative classification c) is an example of generative classification b) and c) are both examples of probabilistic classification

4 Probability Basics Prior, conditional and joint probability –Prior probability: –Conditional probability: –Joint probability: –Relationship: –Independence: Bayesian Rule

Example by Dieter Fox

8 Probabilistic Classification Establishing a probabilistic model for classification –Discriminative model –Generative model MAP classification rule –MAP: Maximum A Posterior –Assign x to c* if Generative classification with the MAP rule –Apply Bayesian rule to convert:

Feature Histograms x C1C1 C2C2 P(x) Slide by Stephen Marsland

Posterior Probability x P(C|x) 1 0 Slide by Stephen Marsland

11 Naïve Bayes Bayes classification Difficulty: learning the joint probability Naïve Bayes classification –Making the assumption that all input attributes are independent –MAP classification rule

12 Naïve Bayes Naïve Bayes Algorithm (for discrete input attributes) –Learning Phase: Given a training set S, Output: conditional probability tables; for elements –Test Phase: Given an unknown instance, Look up tables to assign the label c* to X’ if

13 Example Example: Play Tennis

14 Learning Phase OutlookPlay=YesPlay=No Sunny 2/93/5 Overcast 4/90/5 Rain 3/92/5 TemperaturePlay=YesPlay=No Hot 2/92/5 Mild 4/92/5 Cool 3/91/5 HumidityPlay=YesPlay=No High 3/94/5 Normal 6/91/5 WindPlay=YesPlay=No Strong 3/93/5 Weak 6/92/5 P(Play=Yes) = 9/14 P(Play=No) = 5/14 P(Outlook=o|Play=b)P(Temperature=t|Play=b) P(Humidity=h|Play=b)P(Wind=w|Play=b)

15 Example Test Phase –Given a new instance, x ’=(Outlook=Sunny, Temperature=Cool, Humidity=High, Wind=Strong) –Look up tables –MAP rule P(Outlook=Sunny|Play=No) = 3/5 P(Temperature=Cool|Play==No) = 1/5 P(Huminity=High|Play=No) = 4/5 P(Wind=Strong|Play=No) = 3/5 P(Play=No) = 5/14 P(Outlook=Sunny|Play=Yes) = 2/9 P(Temperature=Cool|Play=Yes) = 3/9 P(Huminity=High|Play=Yes) = 3/9 P(Wind=Strong|Play=Yes) = 3/9 P(Play=Yes) = 9/14 P(Yes| x ’): [P(Sunny|Yes)P(Cool|Yes)P(High|Yes)P(Strong|Yes)]P(Play=Yes) = P(No| x ’): [P(Sunny|No) P(Cool|No)P(High|No)P(Strong|No)]P(Play=No) = Given the fact P(Yes| x ’) < P(No| x ’), we label x ’ to be “No”.

16 Relevant Issues Violation of Independence Assumption –For many real world tasks, –Nevertheless, naïve Bayes works surprisingly well anyway! Zero conditional probability Problem –If no example contains the attribute value –In this circumstance, during test –For a remedy, conditional probabilities estimated with Laplace smoothing:

Homework Redo the test on Slide 15 using the formula on Slide 16 with m=1. Compute P(Play=Yes| x ’) and P(Play=No| x ’) with m=0 and with m=1 for x’=(Outlook=Overcast, Temperature=Cool, Humidity=High, Wind=Strong) Does the result change? 17

18 Relevant Issues Continuous-valued Input Attributes –Numberless values for an attribute –Conditional probability modeled with the normal distribution –Learning Phase: Output: normal distributions and –Test Phase: Calculate conditional probabilities with all the normal distributions Apply the MAP rule to make a decision

19 Conclusions Naïve Bayes based on the independence assumption –Training is very easy and fast; just requiring considering each attribute in each class separately –Test is straightforward; just looking up tables or calculating conditional probabilities with normal distributions A popular generative model –Performance competitive to most of state-of-the-art classifiers even in presence of violating independence assumption –Many successful applications, e.g., spam mail filtering –Apart from classification, naïve Bayes can do more…