1. ECE471-571 – Pattern Recognition Lecture 4 – Parametric Estimation
Hairong Qi, Gonzalez Family Professor
Electrical Engineering and Computer Science
University of Tennessee, Knoxville

2. Pattern Classification
Statistical Approach
Non-Statistical Approach
Supervised
Unsupervised
Decision-tree
Basic concepts:
Baysian decision rule
(MPP, LR, Discri.)
Basic concepts:
Distance
Agglomerative method
Syntactic approach
Parameter estimate (ML, BL)
k-means
Non-Parametric learning (kNN)
Winner-takes-all
LDF (Perceptron)
Kohonen maps
NN (BP)
Mean-shift
Support Vector Machine
Deep Learning (DL)
Dimensionality
Reduction
FLD, PCA
Performance Evaluation
ROC curve (TP, TN, FN, FP)
cross validation
Stochastic Methods
local opt (GD)
global opt (SA, GA)
Classifier Fusion
majority voting
NB, BKS

3. Bayes Decision Rule Maximum Posterior Probability Likelihood
Ratio
If , then x belongs to class 1, otherwise, 2.
Discriminant
Function
Case 1: Minimum Euclidean Distance (Linear Machine), Si=s2I
Case 2: Minimum Mahalanobis Distance (Linear Machine), Si = S
Case 3: Quadratic classifier , Si = arbitrary

4. Multivariate Normal Density

5. Estimating Normal Densities
Calculate m, S

6. Covariance

7. *Why? – Maximum Likelihood Estimation
Compare “likelihood”

8. *Derivation

9. * Derivative of a Quadratic Form

10. **Why? – Baysian Estimation
Maximum likelihood estimation
The parameters are fixed
Find value for q that best agrees with or supports the actually observed training samples – likelihood of q w.r.t. the set of samples
Baysian estimation
Treat parameters as random variable themselves

11. ** The pdf of the parameter (m) is Gaussian

12. ** Derivation

13. ** mn and sn Behavior of Bayesian learning
Our best guess for m after observing n samples
Measures our uncertainty about this guess
Behavior of Bayesian learning
The larger the n, the smaller the sn – each additional observation decreases our uncertainty about the true value of m
As n approaches infinity, p(m|D) becomes more and more sharply peaked, approaching a Dirac delta function.
mn is a linear combination between the sample mean and m0