1. ENN: Extended Nearest Neighbor Method for Pattern Recognition
This lecture notes is based on the following paper:
B. Tang and H. He, "ENN: Extended Nearest Neighbor Method for Pattern Recognition," IEEE Computational Intelligence Magazine, vol.10, no.3, pp , Aug. 2015
ENN: Extended Nearest Neighbor Method for Pattern Recognition
Prof. Haibo He
Electrical Engineering
University of Rhode Island, Kingston, RI 02881
Computational Intelligence and Self-Adaptive Systems (CISA) Laboratory

2. Extended Nearest Neighbor for Pattern Recognition
Limitations of K-Nearest Neighbors (KNN)
“Two-way communication”: Extended Nearest Neighbors (ENN)
Experimental Analysis
Conclusion

3. Pattern Recognition Parametric Classifier
Class-wise density estimation, including naive Bayes, mixture Gaussian, etc.
Non-Parametric Classifier
Nearest Neighbors
Neural Network
Support Vector Machine
Nonparametric nature
Easy implementation
Powerfulness
Robustness
Consistency

4. Limitations of traditional KNN
Scale-Sensitive Problem:
The class 1 samples dominate their near neighborhood with higher density (i.e., more concentrated distribution). The class 2 samples are distributed in regions with lower density (i.e., more spread out distribution).
Those class 2 samples which are close to the region of class 1 may be easily misclassified.

5. ENN: A New Approach
Define generalized class-wise statistic for each class:
Si denotes the samples in class i, and NNr(x, S) denotes the r-th nearest neighbor of x in S.
Ti measures the coherence of data from the same class. 0 ≤ Ti ≤ 1 with Ti = 1 when all the nearest neighbors of class i data are also from the same class i, and with Ti = 0 when all the nearest neighbors are from other classes.

6. Intra-class coherence:
Given an unknown sample Z to be classified, we iteratively assign it to class 1 and class 2, respectively, to obtain two new generalized class-wise statistics Tij, where j=1,2. Then, the sample Z is classified according to:
ENN Classification Rule: Maximum Gain of Intra-class Coherence.
For N-class classification:

7. To avoid the recalculation of generalized class-wise statistics in testing stage, an Equivalent Version of ENN is proposed:

8. The equivalent version has the same result as the original one, but avoids the recalculation of Tij

9. How this simple rule works better than KNN
The ENN method makes a prediction in a “two-way communication” style: it considers not only who are the nearest neighbors of the test sample, but also who consider the test sample as their nearest neighbors.

10. Experimental Results and Analysis

11. Sampling methods Synthetic Data Set:
A 3-dimensional Gaussian data with 3 classes:
Considering the following four models, their error rates are:
Model 2
Class 1
Class 2
Class 3
KNN
ENN
k = 3 32
31.9
39.3
34.4
31.4
30.5
k = 5
31.2
29.7
40.5
33.7
28.6
26.7
k = 7
28.5
28.3
40.8
33.6 25
24.3
Model 3
33.2 31 27
26.8
38.8
30.3
27.3 24
23.2
40.2
33.5
25.1
20.8
40.6 33

12. Sampling methods Real-life Data Sets:
MNIST Handwritten Digit Recognition
Data Examples

13. Sampling methods Real-life Data Sets:
20 data sets from UCI Machine Learning Repository
t-test shows that ENN can significantly improve the classification performance in 17 out of 20 datasets, in comparison with KNN.

14. ENN Summary: Three versions of ENN
B. Tang and H. He, "ENN: Extended Nearest Neighbor Method for Pattern Recognition," IEEE Computational Intelligence Magazine,
vol.10, no.3, pp , Aug. 2015

15. ENN.V1 Summary: Three versions of ENN
B. Tang and H. He, "ENN: Extended Nearest Neighbor Method for Pattern Recognition," IEEE Computational Intelligence Magazine,
vol.10, no.3, pp , Aug. 2015

16. ENN.V2 Summary: Three versions of ENN
B. Tang and H. He, "ENN: Extended Nearest Neighbor Method for Pattern Recognition," IEEE Computational Intelligence Magazine,
vol.10, no.3, pp , Aug. 2015

17. Online Resources Summary: Three versions of ENN
Supplementary materials and Matlab source code implementation available at:
B. Tang and H. He, "ENN: Extended Nearest Neighbor Method for Pattern Recognition," IEEE Computational Intelligence Magazine,
vol.10, no.3, pp , Aug. 2015

18. Conclusion
A new ENN classification methodology based on the maximum gain of intra-class coherence.
“Two-way communication”: ENN considers not only who are the nearest neighbors of the test sample, but also who consider the test sample as their nearest neighbors.
Important and useful for many other machine learning and data mining problems, such as density estimation, clustering, regression, among others.
B. Tang and H. He, "ENN: Extended Nearest Neighbor Method for Pattern Recognition," IEEE Computational Intelligence Magazine,
vol.10, no.3, pp , Aug. 2015