Quadratic Classifiers (QC)

Slides:

Advertisements

Similar presentations

University of Joensuu Dept. of Computer Science P.O. Box 111 FIN Joensuu Tel fax Gaussian Mixture.

Advertisements

Notes Sample vs distribution “m” vs “µ” and “s” vs “σ” Bias/Variance Bias: Measures how much the learnt model is wrong disregarding noise Variance: Measures.

Probabilistic Generative Models Rong Jin. Probabilistic Generative Model Classify instance x into one of K classes Class prior Density function for class.

Assuming normally distributed data! Naïve Bayes Classifier.

Classification and risk prediction

Slide 1 EE3J2 Data Mining EE3J2 Data Mining Lecture 10 Statistical Modelling Martin Russell.

Lecture Notes for CMPUT 466/551 Nilanjan Ray

Linear Methods for Classification

Today Today: Chapter 9 Assignment: 9.2, 9.4, 9.42 (Geo(p)=“geometric distribution”), 9-R9(a,b) Recommended Questions: 9.1, 9.8, 9.20, 9.23, 9.25.

Generative Models Rong Jin. Statistical Inference Training ExamplesLearning a Statistical Model  Prediction p(x;  ) Female: Gaussian distribution N(

Expectation Maximization for GMM Comp344 Tutorial Kai Zhang.

PatReco: Bayes Classifier and Discriminant Functions Alexandros Potamianos Dept of ECE, Tech. Univ. of Crete Fall

MACHINE LEARNING 6. Multivariate Methods 1. Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 Motivating Example  Loan.

PatReco: Discriminant Functions for Gaussians Alexandros Potamianos Dept of ECE, Tech. Univ. of Crete Fall

Jeff Howbert Introduction to Machine Learning Winter Classification Bayesian Classifiers.

1 Linear Methods for Classification Lecture Notes for CMPUT 466/551 Nilanjan Ray.

PCA & LDA for Face Recognition

Principal Component Analysis (PCA)

Speech Recognition Pattern Classification. 22 September 2015Veton Këpuska2 Pattern Classification  Introduction  Parametric classifiers  Semi-parametric.

Decision Trees Jyh-Shing Roger Jang ( 張智星 ) CSIE Dept, National Taiwan University.

Remote Sensing Supervised Image Classification. Supervised Image Classification ► An image classification procedure that requires interaction with the.

Learning Theory Reza Shadmehr Linear and quadratic decision boundaries Kernel estimates of density Missing data.

Ch 4. Linear Models for Classification (1/2) Pattern Recognition and Machine Learning, C. M. Bishop, Summarized and revised by Hee-Woong Lim.

Quadratic Classifiers (QC) J.-S. Roger Jang ( 張智星 ) CS Dept., National Taiwan Univ Scientific Computing.

Sep 10th, 2001Copyright © 2001, Andrew W. Moore Learning Gaussian Bayes Classifiers Andrew W. Moore Associate Professor School of Computer Science Carnegie.

Introduction to Machine Learning Multivariate Methods 姓名 : 李政軒.

M.Sc. in Economics Econometrics Module I Topic 4: Maximum Likelihood Estimation Carol Newman.

Generative classifiers: The Gaussian classifier Ata Kaban School of Computer Science University of Birmingham.

Linear Classifiers Dept. Computer Science & Engineering, Shanghai Jiao Tong University.

Maximum Likelihood Estimate Jyh-Shing Roger Jang ( 張智星 ) CSIE Dept, National Taiwan University.

Linear Classifier Team teaching.

Simulation of Stock Trading J.-S. Roger Jang ( 張智星 ) MIR Lab, CSIE Dept. National Taiwan University.

Linear Classifiers (LC) J.-S. Roger Jang ( 張智星 ) MIR Lab, CSIE Dept. National Taiwan University.

Unsupervised Learning Part 2. Topics How to determine the K in K-means? Hierarchical clustering Soft clustering with Gaussian mixture models Expectation-Maximization.

Applied statistics Usman Roshan.

CSIE Dept., National Taiwan Univ., Taiwan

Usman Roshan CS 675 Machine Learning

Maximum Likelihood Estimate

Probability Theory and Parameter Estimation I

Intro to Machine Learning

National Taiwan University

CH 5: Multivariate Methods

Computer vision: models, learning and inference

The point is class B via 3NNC.

Classification Discriminant Analysis

REMOTE SENSING Multispectral Image Classification

REMOTE SENSING Multispectral Image Classification

دانشگاه صنعتی امیرکبیر Instructor : Saeed Shiry

More about Posterior Distributions

Probabilistic Models with Latent Variables

Intro to Machine Learning

Where did we stop? The Bayes decision rule guarantees an optimal classification… … But it requires the knowledge of P(ci|x) (or p(x|ci) and P(ci)) We.

Machine Learning Math Essentials Part 2

Unsupervised Learning II: Soft Clustering with Gaussian Mixture Models

EE513 Audio Signals and Systems

The basics of Bayes decision theory

Pattern Recognition and Machine Learning

Generally Discriminant Analysis

Parametric Methods Berlin Chen, 2005 References:

Multivariate Methods Berlin Chen

Mathematical Foundations of BME

Multivariate Methods Berlin Chen, 2005 References:

Scientific Computing: Closing 科學計算：結語

Naive Bayes Classifiers (NBC)

Test #1 Thursday September 20th

Game Trees and Minimax Algorithm

EM Algorithm and its Applications

Logistic Regression [Many of the slides were originally created by Prof. Dan Jurafsky from Stanford.]

Hairong Qi, Gonzalez Family Professor

Pre and Post-Processing for Pitch Tracking

Presentation transcript:

Quadratic Classifiers (QC) J.-S. Roger Jang (張智星) jang@mirlab.org http://mirlab.org/jang MIR Lab, CSIE Dept. National Taiwan University

Review: PDF Modeling Goal: Steps: Commonly used PDFs: Find a PDF (probability density function) that can best describe a given dataset Steps: Select a class of parameterized PDF Identify the parameters via MLE (maximum likelihood estimate) based on a given set of sample data Commonly used PDFs: Multi-dimensional Gaussian PDF Gaussian mixture models (GMM)

PDF Modeling for Classification Procedure for classification based on PDF Training stage: PDF modeling of each class based on the training dataset Test stage: For each entry in the test dataset, pick the class with the max. PDF Commonly used classifiers: Quadratic classifiers, with n-dim. Gaussian PDF Gaussian-mixture-model classifier, with GMM PDF

1D Gaussian PDF Modeling MLE of m (mean) and s2 (variance)

D-dim. Gaussian PDF Modeling MLE of m (mean) and S (covariance matrix)

2D Gaussian PDF Bivariate normal distribution: PDF Contours

Training and Test Stages of QC Quiz! Training stage Identify the Gaussian PDF of each class via MLE Test stage Assign a sample point to the class C by taking class prior into consideration:

Characteristics of QC If each class is modeled by an Gaussian PDF, the decision boundary between any two classes is a quadratic function. That is why it is called quadratic classifier. How to prove it? Quiz!

QC on Gender Dataset (1/2) Scatter plot of Gender dataset

QC on Gender Dataset (2/2) PDF for each class Decision boundary

QC on Iris Dataset (1/3) Scatter plot of Iris dataset (with only the last two dim.)

QC on Iris Dataset (2/3) PDF for each class:

QC on Iris Dataset (3/3) Decision boundaries among classes:

Strength and Weakness of QC Quiz! Strength Easy computation when the dimension d is small Efficient way to compute leave-one-out cross validation Weakness The covariance matrix (d by d) is big when the dimension d is median large The inverse of the covariance matrix may not exist Cannot handle bi-modal data correctly

Modified Versions of QC Quiz! How to modify QC such that it won’t deal with a big covariance matrix? Make the covariance matrix diagonal  Equivalent to naïve Bayes classifiers (proof?) Make the covariance matrix a constant times an identity matrix