Non-Parameter Estimation 主講人：虞台文

Contents Introduction Parzen Windows k n -Nearest-Neighbor Estimation Classification Techiques – The Nearest-Neighbor rule (1-NN) – The k -Nearest-Neighbor rule ( k -NN) Distance Metrics

Non-Parameter Estimation Introduction

Facts Classical parametric densities are unimodal. Many practical problems involve multimodal densities. Common parametric forms rarely fit the densities actually encountered in practice.

Goals Estimate class-conditional densities Estimate posterior probabilities

R Density Estimation n samples Assume p(x) is continuous & R is small + Randomly take n samples, let K denote the number of samples inside R.

Density Estimation R n samples + Assume p(x) is continuous & R is small Let k R denote the number of samples in R.

R Density Estimation n samples + What items can be controlled? How? Use subscript n to take sample size into account. To this, we should have We hope

Two Approaches Parzen Windows – Control V n k n -Nearest-Neighbor – Control k n R n samples What items can be controlled? How?

Two Approaches Parzen Windows k n -Nearest-Neighbor

Non-Parameter Estimation Parzen Windows

Window Function 1 1 1

hnhn hnhn hnhn

hnhn hnhn hnhn

Parzen-Window Estimation hnhn hnhn hnhn k n : # samples inside hypercube centered at x.

Generalization Requirement Set x/h n =u. The window is not necessary a hypercube. h n is a important parameter. It depends on sample size.

Interpolation Parameter h n  0  n (x) is a Dirac delta function.

Example Parzen-window estimations for five samples

Convergence Conditions To assure convergence, i.e., and we have the following additional constraints:

Illustrations One dimension case:

Illustrations One dimension case:

Illustrations Two dimension case:

Classification Example Smaller windowLarger window

Choosing the Window Function V n must approach zero when n , but at a rate slower than 1/n, e.g., The value of initial volume V 1 is important. In some cases, a cell volume is proper for one region but unsuitable in a different region.

PNN (Probabilistic Neural Network)

Irrelevant for discriminant analysis 2  k (bias) Irrelevant for discriminant analysis

PNN (Probabilistic Neural Network) a k () wk1wk1 wk2wk2 w kd x1x1 x2x2 xdxd kk wkwk x a k (net k )...

PNN (Probabilistic Neural Network)  … 22 o2o2  … 11 o1o1 x1x1 x2x2 xdxd...  … cc ococ

PNN (Probabilistic Neural Network)  … 22 o2o2  … 11 o1o1 x1x1 x2x2 xdxd...  … cc ococ Assign patterns to the class with maximum output values. 1. Pros and cons of the approach? 2. How to deal with prior probabilities?

Non-Parameter Estimation k n -Nearest-Neighbor Estimation

Basic Concept Let the cell volume depends on the training data. To estimate p(x), we can center a cell about x and let it grow until it captures k n samples, where is some specified function of n, e.g.,

Example k n =5

Example

Estimation of A Posteriori Probabilities x P n (  i |x)=?

Estimation of A Posteriori Probabilities x P n (  i |x)=?

Estimation of A Posteriori Probabilities The value of V n or k n can be determined base on Parzen window or k n -nearest-neighbor technique.

Non-Parameter Estimation Classification Techniques The Nearest-Neighbor Rule The k-Nearest-Neighbor Rule

The Nearest-Neighbor Rule x  A set of labeled prototypes x’x’ Classify as

The Nearest-Neighbor Rule Voronoi Tessellation

Error Rate x Baysian (optimum): Optimum:

Error Rate x x’x’ 1-NN Suppose the true class for x is  Optimum:

Error Rate x x’x’ 1-NN ? As n , Optimum:

Error Rate x x’x’ 1-NN Optimum:

Error Bounds 1-NN Bayesian Consider the most complex classification case:

Error Bounds 1-NN Bayesian Consider the opposite case: Maximized this term to find the upper bound This term is minimum when all elements have the same value i.e., Minimized this term

Error Bounds 1-NN Bayesian Consider the opposite case:

Error Bounds 1-NN Bayesian Consider the opposite case: The nearest-neighbor rule is a suboptimal procedure. The error rate is never worse than twice the Bayes rate.

Error Bounds

The k -Nearest-Neighbor Rule k = 5 Assign pattern to the class wins the majority.

Error Bounds

Computation Complexity The computation complexity of the nearest-neighbor algorithm (both in time and space) has received a great deal of analysis. Require O(dn) space to store n prototypes in a training set. – Editing, pruning or condensing To search the nearest neighbor for a d-dimensional test point x, the time complexity is O(dn). – Partial distance – Search tree

Partial Distance Using the following fact to early throw far-away prototypes

Editing Nearest Neighbor Given a set of points, a Voronoi diagram is a partition of space into regions, within which all points are closer to some particular node than to any other node.

Delaunay Triangulation If two Voronoi regions share a boundary, the nodes of these regions are connected with an edge. Such nodes are called the Voronoi neighbors (or Delaunay neighbors).

The Decision Boundary The circled prototypes are redundant.

The Edited Training Set

Editing: The Voronoi Diagram Approach Compute the Delaunay triangulation for the training set. Visit each node, marking it if all its Delaunay neighbors are of the same class as the current node. Delete all marked nodes, exiting with the remaining ones as the edited training set. Demo

Editing: Other Approaches The Gabriel Graph Approach The Relative Neighbour Graph Approach References: – Binay K. Bhattacharya, Ronald S. Poulsen, Godfried T. Toussaint, "Application of Proximity Graphs to Editing Nearest Neighbor Decision Rule", International Symposium on Information Theory, Santa Monica, 1981."Application of Proximity Graphs to Editing Nearest Neighbor Decision Rule", – T.M. Cover, P.E. Hart, "Nearest Neighbor Pattern Classification", IEEE Transactions on Information Theory, vol. IT-13, No.1, 1967, pp – V. Klee, On the complexity of d-dimensional Voronoi diagrams", Arch. Math., vol. 34, 1980, pp – Godfried T. Toussaint, "The Relative Neighborhood Graph of a Finite Planar Set", Pattern Recognition, vol.12, No.4, 1980, pp

Non-Parameter Estimation Distance Metrics

Nearest-Neighbor Classifier Distance Measurement is an importance factor for nearest-neighbor classifier, e.g., – To achieve invariant pattern recognition The effect of change units

Nearest-Neighbor Classifier Distance Measurement is an importance factor for nearest-neighbor classifier, e.g., – To achieve invariant pattern recognition The effect of translation

Properties of a Distance Metric Nonnegativity Reflexivity Symmetry Triangle Inequality

Minkowski Metric ( L p Norm) 1. L 1 norm Manhattan or city block distance 2. L 2 norm Euclidean distance 3. L  norm Chessboard distance

Minkowski Metric ( L p Norm)