Hairong Qi, Gonzalez Family Professor

Slides:

Advertisements

Similar presentations

Component Analysis (Review)

Advertisements

Principal Component Analysis Based on L1-Norm Maximization Nojun Kwak IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008.

Machine Learning Lecture 8 Data Processing and Representation

Supervised and Unsupervised learning and application to Neuroscience Cours CA6b-4.

Classification and risk prediction

Principal Component Analysis

Pattern Recognition Topic 1: Principle Component Analysis Shapiro chap

Summarized by Soo-Jin Kim

Principle Component Analysis Presented by: Sabbir Ahmed Roll: FH-227.

Dimensionality Reduction: Principal Components Analysis Optional Reading: Smith, A Tutorial on Principal Components Analysis (linked to class webpage)

Principles of Pattern Recognition

N– variate Gaussian. Some important characteristics: 1)The pdf of n jointly Gaussian R.V.’s is completely described by means, variances and covariances.

ECE 8443 – Pattern Recognition LECTURE 08: DIMENSIONALITY, PRINCIPAL COMPONENTS ANALYSIS Objectives: Data Considerations Computational Complexity Overfitting.

Principal Component Analysis Machine Learning. Last Time Expectation Maximization in Graphical Models – Baum Welch.

ECE 471/571 – Lecture 2 Bayesian Decision Theory 08/25/15.

ECE 471/571 – Lecture 6 Dimensionality Reduction – Fisher’s Linear Discriminant 09/08/15.

Discriminant Analysis

ECE 471/571 - Lecture 19 Review 11/12/15. A Roadmap 2 Pattern Classification Statistical ApproachNon-Statistical Approach SupervisedUnsupervised Basic.

ECE 471/571 – Lecture 21 Syntactic Pattern Recognition 11/19/15.

ECE 8443 – Pattern Recognition ECE 8527 – Introduction to Machine Learning and Pattern Recognition LECTURE 10: PRINCIPAL COMPONENTS ANALYSIS Objectives:

ECE 471/571 – Lecture 3 Discriminant Function and Normal Density 08/27/15.

SUPERVISED AND UNSUPERVISED LEARNING Presentation by Ege Saygıner CENG 784.

Part 3: Estimation of Parameters. Estimation of Parameters Most of the time, we have random samples but not the densities given. If the parametric form.

Machine Learning Supervised Learning Classification and Regression K-Nearest Neighbor Classification Fisher’s Criteria & Linear Discriminant Analysis Perceptron:

1 C.A.L. Bailer-Jones. Machine Learning. Data exploration and dimensionality reduction Machine learning, pattern recognition and statistical data modelling.

Use Neural Networks for Pattern Recognition 03/29/17

Principal Component Analysis (PCA)

Syntactic Pattern Recognition 04/28/17

ECE 471/571 - Lecture 19 Review 02/24/17.

Nonparametric Density Estimation – k-nearest neighbor (kNN) 02/20/17

Background on Classification

LECTURE 09: BAYESIAN ESTIMATION (Cont.)

Performance Evaluation 02/15/17

LECTURE 10: DISCRIMINANT ANALYSIS

ECE 471/571 – Lecture 18 Classifier Fusion 04/12/17.

Unsupervised Learning - Clustering 04/03/17

Unsupervised Learning - Clustering

ECE 599/692 – Deep Learning Lecture 6 – CNN: The Variants

Principal Component Analysis

Pattern Recognition CS479/679 Pattern Recognition Dr. George Bebis

REMOTE SENSING Multispectral Image Classification

REMOTE SENSING Multispectral Image Classification

ECE 599/692 – Deep Learning Lecture 1 - Introduction

ECE 471/571 – Lecture 12 Perceptron.

ECE 599/692 – Deep Learning Lecture 9 – Autoencoder (AE)

PCA is “an orthogonal linear transformation that transfers the data to a new coordinate system such that the greatest variance by any projection of the.

Hairong Qi, Gonzalez Family Professor

ECE 471/571 – Review 1.

CS 478 Homework CS Homework.

Parametric Estimation

Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John.

Dimensionality Reduction

Feature space tansformation methods

Generally Discriminant Analysis

LECTURE 09: DISCRIMINANT ANALYSIS

ECE – Pattern Recognition Lecture 16: NN – Back Propagation

ECE – Pattern Recognition Lecture 15: NN - Perceptron

ECE – Pattern Recognition Lecture 13 – Decision Tree

Feature Selection Methods

Hairong Qi, Gonzalez Family Professor

Hairong Qi, Gonzalez Family Professor

ECE – Lecture 1 Introduction.

ECE – Pattern Recognition Lecture 10 – Nonparametric Density Estimation – k-nearest-neighbor (kNN) Hairong Qi, Gonzalez Family Professor Electrical.

Bayesian Decision Theory

Hairong Qi, Gonzalez Family Professor

ECE – Pattern Recognition Lecture 8 – Performance Evaluation

ECE – Pattern Recognition Lecture 4 – Parametric Estimation

Hairong Qi, Gonzalez Family Professor

ECE – Pattern Recognition Lecture 14 – Gradient Descent

ECE – Pattern Recognition Midterm Review

Presentation transcript:

ECE471-571 – Pattern Recognition Lecture 7 – Dimensionality Reduction – Principal Component Analysis Hairong Qi, Gonzalez Family Professor Electrical Engineering and Computer Science University of Tennessee, Knoxville http://www.eecs.utk.edu/faculty/qi Email: hqi@utk.edu

Pattern Classification Statistical Approach Non-Statistical Approach Supervised Unsupervised Decision-tree Basic concepts: Baysian decision rule (MPP, LR, Discri.) Basic concepts: Distance Agglomerative method Syntactic approach Parameter estimate (ML, BL) k-means Non-Parametric learning (kNN) Winner-takes-all LDF (Perceptron) Kohonen maps NN (BP) Mean-shift Support Vector Machine Deep Learning (DL) Dimensionality Reduction FLD, PCA Performance Evaluation ROC curve (TP, TN, FN, FP) cross validation Stochastic Methods local opt (GD) global opt (SA, GA) Classifier Fusion majority voting NB, BKS

Principal Component Analysis or K-L Transform How to find a new feature space (m-dimensional) that is adequate to describe the original feature space (d-dimensional). Suppose m<d x2 y1 y2 x1

K-L Transform (1) Describe vector x in terms of a set of basis vectors bi. The basis vectors (bi) should be linearly independent and orthonormal, that is,

K-L Transform (2) Suppose we wish to ignore all but m (m<d) components of y and still represent x, although with some error. We will thus calculate the first m elements of y and replace the others with constants Error:

K-L Transform (3) Use mean-square error to quantify the error

K-L Transform (4) Find the optimal ai to minimize e2 Therefore, the error is now equal to

K-L Transform (5) The optimal choice of basis vectors is the eigenvectors of Sx The expansion of a random vector in terms of the eigenvectors of the covariance matrix is referred to as the Karhunen-Loeve expansion, or the “K-L expansion” Without loss of generality, we will sort the eigenvectors bi in terms of their eigenvalues. That is l1 >= l2 >= … >= ld. Then we refer to b1, corresponding to l1, as the “major eigenvector”, or “principal component”

Summary Raw data  covariance matrix  eigenvalue  eigenvector  principal component How to use error rate?