1. Parametric Estimation
ECE 471/571 – Lecture 4
Parametric Estimation

2. Pattern Classification
Statistical Approach
Non-Statistical Approach
Supervised
Unsupervised
Decision-tree
Basic concepts:
Baysian decision rule
(MPP, LR, Discri.)
Basic concepts:
Distance
Agglomerative method
Syntactic approach
Parameter estimate (ML, BL)
k-means
Non-Parametric learning (kNN)
Winner-takes-all
LDF (Perceptron)
Kohonen maps
NN (BP)
Mean-shift
Support Vector Machine
Deep Learning (DL)
Dimensionality
Reduction
FLD, PCA
Performance Evaluation
ROC curve (TP, TN, FN, FP)
cross validation
Stochastic Methods
local opt (GD)
global opt (SA, GA)
Classifier Fusion
majority voting
NB, BKS

3. Bayes Decision Rule Maximum Posterior Probability Likelihood Ratio
Discriminant
Function

4. Normal/Gaussian Density
The
rule

5. Multivariate Normal Density

6. Estimating Normal Densities
Calculate m, S

7. Covariance

8. *Why? – Maximum Likelihood Estimation
Compare “likelihood”

9. *Derivation

10. * Derivative of a Quadratic Form

11. **Why? – Baysian Estimation
Maximum likelihood estimation
The parameters are fixed
Find value for q that best agrees with or supports the actually observed training samples – likelihood of q w.r.t. the set of samples
Baysian estimation
Treat parameters as random variable themselves

12. ** The pdf of the parameter (m) is Gaussian

13. ** Derivation

14. ** mn and sn Behavior of Bayesian learning
Our best guess for m after observing n samples
Measures our uncertainty about this guess
Behavior of Bayesian learning
The larger the n, the smaller the sn – each additional observation decreases our uncertainty about the true value of m
As n approaches infinity, p(m|D) becomes more and more sharply peaked, approaching a Dirac delta function.
mn is a linear combination between the sample mean and m0