1. Outline (I) Introduction to pattern recognition Approaches to PR:
2018/11/30
Outline (I)
Introduction to pattern recognition
Definition
Applications
Approaches to PR:
Histogram analysis
Kernel and window estimators
K-nearest neighbor rule
Adaptive decision boundaries
Adaptive discriminant functions
Density function parameters via sample data
Adaptive networks (MLPs and RBFNs)
Fuzzy classifiers with random search
2018/11/30

2. Outline (II) Data reduction Data clustering Questions and Answers
2018/11/30
Outline (II)
Data reduction
Editing and condensing
Principal component analysis
Discriminant projection
Data clustering
Hierarchical clustering
Partitional clustering
Questions and Answers
2018/11/30

3. Pattern Recognition Also known as: Definition: Example:
2018/11/30
Pattern Recognition
Also known as:
Data classification
Pattern classification
Definition:
Automatic classification of objects or events
根據以往的資料，由外在的特性來推測出內在的本質
Example:
視其所以，觀其所由，察其所安，人焉廋哉？人焉廋哉？
2018/11/30

4. Abalone Age Prediction
2018/11/30
Abalone Age Prediction
Source
Dept. of Primary Industry and FIsheries, Tasmania, Australia
Goal
Predict the age of abalone (鮑魚) from physical measurements
Problem sizes
4177 instances, 29 classes
8 attributes (features): sex, length, diameter, height, whole weight, shucked weight, viscera weight, shell weight
1 output: rings (+1.5 gives the age in years)
2018/11/30

5. Wine Recognition Source Goal Problem size
2018/11/30
Wine Recognition
Source
Institute of Pharmaceutical and Food Analysis and Technologies, Via Brigata Salerno, Genoa, Italy.
Goal
Using 13 chemical constituents to determine the origin of wines
Problem size
178 instances, 3 classes, 13 attributes
2018/11/30

6. Mushroom classification
2018/11/30
Mushroom classification
Source
Mushroom records drawn from The Audubon Society Field Guide to North American Mushrooms (1981)
Goal
To determine a mushroom is poisonous or edible
Problem size
8124 instances, 2 classes, 22 attributes
2018/11/30

7. Liver Disorder Classification
2018/11/30
Liver Disorder Classification
Source
BUPA Medical Research Ltd.
Goal
Use variables from blood tests and alcohol consumption to see if liver disorder exists
Problem size
345 instances, 2 classes, 6 attributes (the first five are results from blood tests, the last one is alcohol consumption per day)
2018/11/30

8. Credit Screening Source Goal Problem size
2018/11/30
Credit Screening
Source
Chiharu Sano,
Goal
Determine people who are granted credit
Problem size
125 instances, 2 classes, 15 attributes
2018/11/30

9. House Price Prediction
2018/11/30
House Price Prediction
Source
CMU StatLib Library
Goal
Predict house price near Boston
Problem Size
506 instances, 13 attributes
2018/11/30

10. Optical Character Recognition (OCR)
2018/11/30
Optical Character Recognition (OCR)
Handwritten Digit Recognition
Handwritten Character Recognition
Segmented
Unsegmented
2018/11/30

11. Face Detection & Recognition
2018/11/30
Face Detection & Recognition
Test image:
Problems:
1. Is there a face?
2. If yes, how many?
3. How many are female?
4. Is Jones there? If yes, which one?
2018/11/30

12. Face Recognition Tasks: Face detection Local feature extraction
2018/11/30
Face Recognition
Image databases:
Test image: A: B:
Who is this guy? C: D:
Tasks:
Face detection
Local feature extraction
Global feature extraction
Data classification E: F: G:
2018/11/30

13. Pattern Recognition Applications
2018/11/30
Pattern Recognition Applications
Biometric ID: Identification of people from physical characteristics
Speech/speaker recognition
Classification of seismic signals
Medical waveform classification (EEG, ECG)
Wave-based nuclear reactor diagnosis
License plate recognition
Decision making in stock trading
User modeling in WWW environments
Satellite picture analyses
Medical image analyses (CAT scan, MRI)
2018/11/30

14. Biometric Identification
2018/11/30
Biometric Identification
Identification of people from their physical characteristics, such as
faces
voices
fingerprints
palm prints
hand vein distributions
hand shapes and sizes
retinal scans
2018/11/30

15. Who’s Doing Pattern Recognition?
2018/11/30
Who’s Doing Pattern Recognition?
Computer science (machine learning, data mining and knowledge discovery)
Nearest neighbor rule
ID3, C4.5
Neural networks
Fuzzy logic
Statistics/Mathematics (multivariate analysis)
Discriminant analysis
CART: Classification and Regression Trees
MARS
Jackknife procedure
Projection pursuit
2018/11/30

16. Example: Particle Classification
2018/11/30
Example: Particle Classification
Particles on an air filter
P1:
P2:
P3:
2018/11/30

17. Histogram Analysis P1: P2: P3: P1 Number P2 Number P2 P3 Area Area
2018/11/30
Histogram Analysis
P1:
P2:
P3: P1
Number P2
Number P2 P3
Area
Area
2018/11/30

18. Histogram Analysis P1: P2: P3: P1 P2 P3 Number Perimeter P3 P2
2018/11/30
Histogram Analysis
P1:
P2:
P3: P1 P2 P3
Number
Perimeter P3 P2
Perimeter
Area
2018/11/30

19. Sample Data Sets Sample data is divided into two disjoint sets:
2018/11/30
Sample Data Sets
Sample data is divided into two disjoint sets:
Design set (or training set) is used for designing a classifier
Test set (or cross-validation set) is used for evaluating the obtained classifier
Sample data is usually represented by an m by (n+1) matrix, where m is the number of sample data entries and n is the number of features.
2018/11/30

20. Sample data set Design set: Test set: Odd-indexed entries
2018/11/30
Sample data set
Features
Class
Design set:
Odd-indexed entries
Test set:
Even-indexed entries
2018/11/30

21. Flowchart for Histogram Analysis
2018/11/30
Flowchart for Histogram Analysis
General flowchart:
Particle example:
Feature
extraction
From image to features
Data
reduction
None
Probability
estimate
Histogram analysis
2018/11/30

22. Histogram Analysis Number of intervals vs. number of data points
2018/11/30
Histogram Analysis
Number of intervals vs. number of data points
50 samples
from a
Gaussian
distribution
3 bins
10 bins
25 bins
2018/11/30

23. Histogram Analysis Properties:
2018/11/30
Histogram Analysis
Properties:
One of those nonparametric techniques which do not require explicit use of density functions
Dilemma between no. of intervals vs. no. of points
Rule of thumb: no. of intervals is equal to the square root of no. of points
Intervals may be unequally spaced
To convert to density functions, the total area must be unity
Can be used in any number of features, but subjected to curse of dimensionality
2018/11/30

24. Kernel and Window Estimators
2018/11/30
Kernel and Window Estimators
s = 0.1
s = 0.3
2018/11/30

25. Kernel and Window Estimators
2018/11/30
Kernel and Window Estimators
Properties:
Also known as Parzen estimator
Its computation is similar to convolution
Can be used in multi-features estimation
Width is found by trial and error
2018/11/30

26. Example: Gender Classification
2018/11/30
Example: Gender Classification
Goal
Determine a person’s gender from his/her profile data
Features collected
Birthday
Blood type
Height and weight
Density
Three measures
Hair length
Voice pitch …
Chromosome
2018/11/30

27. Input Space Partitioning
2018/11/30
Input Space Partitioning
P1:
P2:
P3: P1 P3 P1 P3
Perimeter
Perimeter P2 P2
Area
Area
2018/11/30

28. Flowchart for Nearest Neighbor
2018/11/30
Flowchart for Nearest Neighbor
General flowchart:
Particle example:
Feature
extraction
From image to features
Data
reduction
None
Distance
measure
Distance Computation
2018/11/30

29. K-Nearest Neighbor Rule (K-NNR)
2018/11/30
K-Nearest Neighbor Rule (K-NNR)
Steps:
1. Find the first k nearest neighbors of a given point.
2. Determine the class of the given point by a voting mechanism among these k nearest neighbors.
: class-A point
: class-B point
: point with unknown class
Circle of 3-nearest neighbors
The point is class B via 3-NNR.
Feature 2
2018/11/30
Feature 1

30. Decision Boundary for 1-NNR
2018/11/30
Decision Boundary for 1-NNR
Voronoi diagram: piecewise linear boundary
2018/11/30

31. Distance Metrics L-p norms (aka Minkowski distance):
2018/11/30
Distance Metrics
L-p norms (aka Minkowski distance):
p = 1: City block distance, Manhattan metric, taxicab distance
p = 2: Euclidean distance
p = inf: maximum distance metric
2018/11/30

32. KNNR Other considerations in KNNR: Variants:
2018/11/30
KNNR
Other considerations in KNNR:
Data rescaling to have zero mean and unity variance along each feature
Variants:
k+k-nearest neighbor
Weighted votes
Nearest prototype classification
Edited nearest neighbor classification
2018/11/30

33. Adaptive Decision Boundaries
2018/11/30
Adaptive Decision Boundaries
Network (perceptron) architecture x1 w1 w0
d = S wi xi + w0 w2 x2
y = sgn(d) w3
-1 if d < 0 x3 =
1 if d > 0
Dwi = k t xi
Learning rule
2018/11/30

34. Adaptive Decision Boundaries
2018/11/30
Adaptive Decision Boundaries
Example: Gender classification
Network Arch.
Training data
x1 (hair length)
x2 (voice freq.) x1 w0 w1 y w2 x2
y = sgn(x1w1+x2w2+w0)
-1 if female
1 if male =
2018/11/30

35. Adaptive Decision Boundaries
2018/11/30
Adaptive Decision Boundaries
Properties:
Guaranteed to converge to a set of weights that will perfectly classify all the data if such a solution exists
Data rescaling is necessary to speed up convergence of the algorithm
Stops whenever a solution with zero error rate is found
Nonlinear decision boundaries can also be found by the adaptive technique
For a k-class problem, it needs k(k-1)/2 decision boundaries to do complete classification
2018/11/30

36. Adaptive Discriminant Functions
2018/11/30
Adaptive Discriminant Functions
To have a discriminant function for each class:
A sample is classified as class i if di is maximal among all discriminant functions.
Update weights for class i and j if a sample of class i is misclassified as class j
d1 = w10+w11 x1 + w12 x2+...+w1nxn
d2 = w20+w21 x1 + w22 x2+...+w2nxn
...
dm = wm0+wm1 x1 + wm2 x2+...+wmnxn
2018/11/30

37. Adaptive Discriminant Functions
2018/11/30
Adaptive Discriminant Functions
Properties:
Guaranteed to converge to a set of weights that will perfectly classify all the data if such a solution exists
Data rescaling is necessary to speed up convergence of the algorithm
Stops whenever a solution with zero error rate is found
Nonlinear discriminant functions can also be used
Unable to classify some data sets that can be classified perfectly by the adaptive decision boundaries technique
2018/11/30

38. Adaptive Discriminant Functions
2018/11/30
Adaptive Discriminant Functions
Nine-class problem that cannot by solved by the adaptive discriminant function technique
Feature 2
2018/11/30
Feature 1

39. Density Function Params. via Sample
2018/11/30
Density Function Params. via Sample
Gaussian density function:
where m and s are estimated from sample (via maximum likelihood estimate):
2018/11/30

40. Density Function Params. via Sample
2018/11/30
Density Function Params. via Sample
Normal dist.
estimated by
normal dist.
Uniform dist.
estimated by
normal dist.
2018/11/30

41. Density Function Params. via Sample
2018/11/30
Density Function Params. via Sample
Multivariate Normal Distribution N(m, S)
Likelihood of x in class j:
Log likelihood:
2018/11/30

42. Density Function Params. via Sample
2018/11/30
Density Function Params. via Sample
Bivariate normal density:
mx = 0, my = 0, sx = 3, sy = 2, rxy = 0.5
Density function
Contours
2018/11/30

43. Data Reduction Purpose: Techniques:
2018/11/30
Data Reduction
Purpose:
Reduce classifier’s computation load
Increase data consistency
Techniques:
To reduce data size:
Editing: To eliminate noisy (boundary) data
Condensing: To eliminate redundant (deeply embedded) data
Vector quantization: To find representative data
To reduce data dimensions:
Principal component projection: To reduce the dimensions of the feature sets
Discriminant projection: To find the best set of vectors which best separates the patterns
2018/11/30

44. 2018/11/30
Data Editing
To remove noisy (boundary) data
2018/11/30

45. Data Condensing To remove redundant (deeply embedded) data 2018/11/30

46. VQ: Fuzzy C-Means Clustering
2018/11/30
VQ: Fuzzy C-Means Clustering
A point can belong to various clusters with
various degrees.
2018/11/30

47. A fuzzy classifier is equivalent to a 1-NNR if all MFs have the
2018/11/30
Fuzzy Classifier
Rule base:
if x is close to (A1 or A2 or A3), then class =
if x is close to (B1 or B2 or B3), then class = A3
A fuzzy classifier is equivalent
to a 1-NNR if all MFs have the
same width. v A2 B1 v B2 A1 v
2018/11/30 B3

48. Fuzzy Classifier Adaptive network representation: x1 + y - x2
2018/11/30
Fuzzy Classifier
Adaptive network representation: A1
max A2 x1 A3 + y S x2 B1 -
max B2 B2
multidimensional MFs
x = [x1 x2] belongs to class if y > 0
class if y < 0
2018/11/30

49. Refining Fuzzy Classifier
2018/11/30
Refining Fuzzy Classifier
MFs with
the same width v v v
MFs’ widths
refined via
random search
2018/11/30

50. Principal Component Projection
2018/11/30
Principal Component Projection
Projection onto eigenvectors that correspond to the first
few largest eigenvalues of the covariance matrix.
Ideal situation
Adversary situation
2018/11/30

51. Principal Component Projection
2018/11/30
Principal Component Projection
Eigenvalues of covariance matrix: l1 > l2 > l3 > ... > ld
Projection on v1 & v Projection on v3 & v4
2018/11/30

52. Discriminant Projection
2018/11/30
Discriminant Projection
Projection onto directions that can best separate data
of different classes.
Adversary situation
for PCP
Ideal situation
for DP
2018/11/30

53. Discriminant Projection
2018/11/30
Discriminant Projection
Best discriminant vectors : v1, v2, ... , vd
Projection on v1 & v Projection on v3 & v4
2018/11/30

54. Data Clustering Hierarchical clustering Partitional clustering
2018/11/30
Data Clustering
Hierarchical clustering
Single-linkage algorithm (minimum method)
Complete-linkage algorithm (maximum method)
Average linkage algorithm
Minimum-variance method (Ward’s method)
Partitional clustering
K-means algorithm
Fuzzy c-means algorithm
Isodata algorithm
2018/11/30

55. Hierarchical Clustering
2018/11/30
Hierarchical Clustering
Agglomerative clustering (bottom up)
1. Begin with n clusters; each containing one sample
2. Merge the most similar two clusters into one.
3. Repeat the previous step until done
Divisive clustering (top down)
2018/11/30

56. Hierarchical Clustering
2018/11/30
Hierarchical Clustering
Single-linkage algorithm (minimum method)
Complete-linkage algorithm (maximum method)
Average-linkage algorithm (average method)
2018/11/30

57. Hierarchical Clustering
2018/11/30
Hierarchical Clustering
Ward’s method (minimum-variance method)
d: the number of features
m: data number of Ci plus Cj
2018/11/30

58. Partitional Clustering: K-means
2018/11/30
Partitional Clustering: K-means
K-means algorithm:
1. Begin with k cluster centers
2. For each sample, find the cluster center nearest to it. Put the sample in the cluster represented by the just-found cluster center.
3. If no samples changed clusters, stop.
4. Recompute cluster centers of altered clusters and go back to step 2.
Properties:
The number of cluster k must be given in advance.
The goal is to minimize the square error, but it could end up in a local minimum.
2018/11/30

59. Fuzzy C-means Clustering
2018/11/30
Fuzzy C-means Clustering
Properties:
Similar to k-means, but each sample can belong to various clusters with degrees from 0 to 1
For a given sample, the degrees of membership to all clusters sum to 1.
Computationally more extensive than k-means, but usually reach a better result
2018/11/30

60. Partitional Clustering: Isodata
2018/11/30
Partitional Clustering: Isodata
Similar to k-means with some enhancements:
Clusters with too few elements are discarded.
Clusters are merged if the number of clusters grows too large or if clusters are too close together.
A cluster is split if the number of clusters is too few or if the cluster contains very dissimilar samples
Properties:
The number of clusters k is not given exactly in advance.
The algorithm may requires extensive computation.
2018/11/30

61. Current Projects Video on Demand (VOD): Internet: 盲人有聲圖書館：
2018/11/30
Current Projects
Video on Demand (VOD):
以歌選歌 --> Contents-based retrieval of audio DBs
Internet:
Document classification
盲人有聲圖書館：
Text-speech alignment
Machine translation
Biometric Identification:
Speaker recognition
2018/11/30

62. 2018/11/30
Questions and Answers
Now it’s your turn!
2018/11/30