Optimal Bayes Classification

Slides:

Advertisements

Similar presentations

Applications of one-class classification

Advertisements

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) ETHEM ALPAYDIN © The MIT Press, 2010

Random Variables ECE460 Spring, 2012.

ECE 8443 – Pattern Recognition LECTURE 05: MAXIMUM LIKELIHOOD ESTIMATION Objectives: Discrete Features Maximum Likelihood Resources: D.H.S: Chapter 3 (Part.

2 – In previous chapters: – We could design an optimal classifier if we knew the prior probabilities P(wi) and the class- conditional probabilities P(x|wi)

LECTURE 11: BAYESIAN PARAMETER ESTIMATION

Pattern Classification. Chapter 2 (Part 1): Bayesian Decision Theory (Sections ) Introduction Bayesian Decision Theory–Continuous Features.

What is Statistical Modeling

Assuming normally distributed data! Naïve Bayes Classifier.

Bayesian Decision Theory Chapter 2 (Duda et al.) – Sections

Parameter Estimation: Maximum Likelihood Estimation Chapter 3 (Duda et al.) – Sections CS479/679 Pattern Recognition Dr. George Bebis.

Classification and risk prediction

0 Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John.

Chapter 2: Bayesian Decision Theory (Part 1) Introduction Bayesian Decision Theory–Continuous Features All materials used in this course were taken from.

Pattern Classification, Chapter 1 1 Basic Probability.

Generative Models Rong Jin. Statistical Inference Training ExamplesLearning a Statistical Model  Prediction p(x;  ) Female: Gaussian distribution N(

1 lBayesian Estimation (BE) l Bayesian Parameter Estimation: Gaussian Case l Bayesian Parameter Estimation: General Estimation l Problems of Dimensionality.

Visual Recognition Tutorial

Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John Wiley.

METU Informatics Institute Min 720 Pattern Classification with Bio-Medical Applications PART 2: Statistical Pattern Classification: Optimal Classification.

EE513 Audio Signals and Systems Statistical Pattern Classification Kevin D. Donohue Electrical and Computer Engineering University of Kentucky.

Principles of Pattern Recognition

Machine Learning1 Machine Learning: Summary Greg Grudic CSCI-4830.

Lecture 2: Bayesian Decision Theory 1. Diagram and formulation

IBS-09-SL RM 501 – Ranjit Goswami 1 Basic Probability.

ECE 8443 – Pattern Recognition ECE 8527 – Introduction to Machine Learning and Pattern Recognition LECTURE 02: BAYESIAN DECISION THEORY Objectives: Bayes.

Overview of Supervised Learning Overview of Supervised Learning2 Outline Linear Regression and Nearest Neighbors method Statistical Decision.

Computational Intelligence: Methods and Applications Lecture 12 Bayesian decisions: foundation of learning Włodzisław Duch Dept. of Informatics, UMK Google:

CS 782 – Machine Learning Lecture 4 Linear Models for Classification  Probabilistic generative models  Probabilistic discriminative models.

Learning Theory Reza Shadmehr Linear and quadratic decision boundaries Kernel estimates of density Missing data.

Jun-Won Suh Intelligent Electronic Systems Human and Systems Engineering Department of Electrical and Computer Engineering Speaker Verification System.

Chapter 3: Maximum-Likelihood Parameter Estimation l Introduction l Maximum-Likelihood Estimation l Multivariate Case: unknown , known  l Univariate.

1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 8 Sept 23, 2005 Nanjing University of Science & Technology.

Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John Wiley.

IE 300, Fall 2012 Richard Sowers IESE. 8/30/2012 Goals: Rules of Probability Counting Equally likely Some examples.

Covariance matrices for all of the classes are identical, But covariance matrices are arbitrary.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

ECE 8443 – Pattern Recognition ECE 8527 – Introduction to Machine Learning and Pattern Recognition Objectives: Occam’s Razor No Free Lunch Theorem Minimum.

Goal of Learning Algorithms  The early learning algorithms were designed to find such an accurate fit to the data.  A classifier is said to be consistent.

Machine Learning 5. Parametric Methods.

Lecture 3: MLE, Bayes Learning, and Maximum Entropy

Intro. ANN & Fuzzy Systems Lecture 15. Pattern Classification (I): Statistical Formulation.

METU Informatics Institute Min720 Pattern Classification with Bio-Medical Applications Part 9: Review.

1 Review of Probability and Random Processes. 2 Importance of Random Processes Random variables and processes talk about quantities and signals which.

Giansalvo EXIN Cirrincione unit #4 Single-layer networks They directly compute linear discriminant functions using the TS without need of determining.

Generalization Error of pac Model  Let be a set of training examples chosen i.i.d. according to  Treat the generalization error as a r.v. depending on.

Linear Classifier Team teaching.

Computer Vision Lecture 7 Classifiers. Computer Vision, Lecture 6 Oleh Tretiak © 2005Slide 1 This Lecture Bayesian decision theory (22.1, 22.2) –General.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Objectives: Loss Functions Risk Min. Error Rate Class. Resources: DHS – Chap. 2 (Part 1) DHS – Chap. 2 (Part 2) RGO - Intro to PR MCE for Speech MCE for.

Introduction to Machine Learning Nir Ailon Lecture 11: Probabilistic Models.

Lecture 2. Bayesian Decision Theory

Lecture 1.31 Criteria for optimal reception of radio signals.

CS479/679 Pattern Recognition Dr. George Bebis

Chapter 3: Maximum-Likelihood Parameter Estimation

Probability Theory and Parameter Estimation I

Comp328 tutorial 3 Kai Zhang

Special Topics In Scientific Computing

Overview of Supervised Learning

Chapter 3: Maximum-Likelihood and Bayesian Parameter Estimation (part 2)

Classification Discriminant Analysis

Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John.

EE513 Audio Signals and Systems

Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John.

Parametric Methods Berlin Chen, 2005 References:

Multivariate Methods Berlin Chen

Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John.

Multivariate Methods Berlin Chen, 2005 References:

Chapter 3: Maximum-Likelihood and Bayesian Parameter Estimation (part 2)

Bayesian Decision Theory

Presentation transcript:

Optimal Bayes Classification Tutorial 1: Optimal Bayes Classification

Theory Review We assume all variables are random variables, with known distributions Notation: - A finite set of classes (categories), - Input space (patterns), - A classifier, maps to

Basic Assumption The following distributions are known: - Prior probabilities for each class. - Conditional probability of the input, given that the class is If is a continuous space, denotes the probability density.

Reminder: Fish Classification Two classes Prior probability can be estimated from relative frequency Class conditional probability can be estimated by a frequency histogram

Optimal Bayes Classifier Bayes Rule: Optimal Bayes Classifier: This classifier minimizes the conditional error probability and the average error probability.

Exercise 1 It is given that and The prior probability is uniform: And the class conditional probability is Gaussian: Where What is the Bayes optimal decision rule? What are the decision boundaries in the plane? Does the decision boundary for the Gaussian case always have the same form? What it depends on?

Exercise 2 It is given that the input space is binary vectors of length d, meaning that . The output space is with general prior . We define the per-coordinate class conditional probabilities: In addition, each coordinate is statistically independent of all other coordinates What is the optimal decision rule? What Happens if for some i, ?

Exercise 3 Two classes are given, with uniform prior. In addition it is given that: What is the Bayes optimal decision rule?