1. Feb 2007Alon Slapak 1 of 1 Classification A practical approach Classification Methods Training Set Classifier Example Definition Bibliography

2. Feb 2007Alon Slapak 2 of 2 What is Classification? “Statistical classification is a statistical procedure in which individual items are placed into groups based on quantitative information on one or more characteristics inherent in the items and based on a training set of previously labeled items.” (WIKIPEDIA) Example: We can measure the shoe size of a student and classify his gender according to an a-priory (already known) distribution of male’s and females’ shoe size. Classification Classification Methods Training Set Classifier Example Definition Bibliography

3. Feb 2007Alon Slapak 3 of 3 What is a Training set? Firstly, we need a database that gives us information on the distribution of students’ shoe size (feature). This database is called a training set. Example: We may measure the shoe size of 21 male students and 26 female students and sketch the histogram of the measurements. The histogram may be regarded (under several assumptions) as an estimation to the shoe size distribution. Classification Methods Training Set Classifier Example Definition Bibliography

4. Feb 2007Alon Slapak 4 of 4 What is a Training set? Based on the assumption that the training set represents the group of objects to be classified, one can determine the shoe size 6.5 as the discriminant boundary between male and female students. Classification Methods Training Set Classifier Example Definition Bibliography But what if for some reason, one of the female student in the training set uses a size 9.5 shoe? No. Another discriminant boundary (more than size 9.25 is a female student) does not make sense.

5. Feb 2007Alon Slapak 5 of 5 “Perfect” Vs. “Simple” classifier 0 1 2 3 4 5 6 x 1 x 2 0 1 2 3 4 5 6 Classification Methods Training Set Classifier Example Definition Bibliography “Perfect” classifier “Simple” classifier Which is better? Check on a test-set (cross validation).

6. Feb 2007Alon Slapak 6 of 6 Cross validation of the “Perfect” classifier 0 1 2 3 4 5 6 x 1 x 2 0 1 2 3 4 5 6 Classification Methods Training Set Classifier Example Definition Bibliography Training set: 0% misclassification Test set: 8/25 = 24% misclassification TrainTest

7. Feb 2007Alon Slapak 7 of 7 Cross validation of the “simple” classifier 0 1 2 3 4 5 6 x 1 x 2 0 1 2 3 4 5 6 TrainTest Classification Methods Training Set Classifier Example Definition Bibliography Training set: 6/29 = 20% misclassification Test set: 2/25 = 8% misclassification

8. Feb 2007Alon Slapak 8 of 8 Training set Vs. Test set Overtraining is said to occur when the decision boundaries are fit too specifically to the training set. To avoid overtraining, the classification algorithm should be tested on a different set of patterns. Attention! A common mistake is to test the classification algorithm on the training set. The statistical distribution of the training set and the test set should be the same. Overtraining is a result of a too small training set. Classification Methods Training Set Classifier Example Definition Bibliography

9. Feb 2007Alon Slapak 9 of 9 Occam’s razor Given a training set of data, we want to use this data to choose a classifier (or decision rule), which will classify future elements based on their features. Based on a particular training set, which of the infinitely many possible classifiers is most likely to be accurate for future data? As seen in the previous example, Occam's Razor is an important factor in making this choice. Classification Methods Training Set Classifier Example Definition Bibliography “Plurality should not be posited without necessity “ Or in pattern recognition language: “simpler classifiers should be preferred over complex ones.”

10. Feb 2007Alon Slapak 10 of 10 What is a Classifier ? “A classifier is a mapping from a (discrete or continuous) feature space X to a discrete set of labels Y.” (W IKIPEDI A) g:  Classification Methods Training Set Classifier Example Definition Bibliography

11. Feb 2007Alon Slapak 11 of 11 Notations for classifiers A commonly used notation for a class is  i where  stands for “class” and i stands for the label (or index) of the class. =  1 =  2 = x 1 = x 2 = x 3 And for the a pattern (a vector in the feature space) we would prefer to use the notation x i Classification Methods Training Set Classifier Example Definition Bibliography

12. Feb 2007Alon Slapak 12 of 12 Crisp classifier Vs. Fuzzy classifier I’m hesitating. 74 percent it’s a male student. 26 percent it’s a female. ? The crisp classifierThe fuzzy classifier 100 percent it’s a male student. Classification Methods Training Set Classifier Example Definition Bibliography

13. Feb 2007Alon Slapak 13 of 13 Crisp classifier Vs. Fuzzy classifier A crisp classifier is a single-valued mapping from a feature space X to a discrete set of labels Y A fuzzy classifier is a multi-valued mapping from a feature space X to a discrete set of labels Y X Y x1x1 11 22 33 x2x2 x3x3 x4x4 X Y x1x1 11 22 33 x2x2 x3x3 x4x4 0.6 0.4 0.1 0.9 1.0 0.23 0.77 Classification Methods Training Set Classifier Example Definition Bibliography

14. Feb 2007Alon Slapak 14 of 14 Example – separate classes Let  1 stands for Female students class,  2 stands for Male students class, and let x stands for the feature which is the students shoe size. It is easy to see that the following crisp classifier definitely classify the students gender: Classification Methods Training Set Classifier Example Definition Bibliography

15. Feb 2007Alon Slapak 15 of 15 Example –non-separate classes A crisp classifier may be: Classification Methods Training Set Classifier Example Definition Bibliography Classification error

16. Feb 2007Alon Slapak 16 of 16 Example –non-separate classes A Fuzzy classifier may be: Classification Methods Training Set Classifier Example Definition Bibliography

17. Feb 2007Alon Slapak 17 of 17 Example –non-separate classes A Fuzzy classifier to crisp classifier : Classification error Classification Methods Training Set Classifier Example Definition Bibliography

18. Feb 2007Alon Slapak 18 of 18 Example –non-separate classes In several books and papers, a Fuzzy classifier may be written as: Classification Methods Training Set Classifier Example Definition Bibliography

19. Feb 2007Alon Slapak 19 of 19 Crisp classifier Vs. Fuzzy classifier From: L. I. Kuncheva, J. C. Bezdek amd R. P.W. Duin, “Decision Templates for Multiple Classier Fusion: An Experimental Comparison”, Pattern Recognition, 34, (2), pp. 299-314, 2001. Classification Methods Training Set Classifier Example Definition Bibliography

20. Feb 2007Alon Slapak 20 of 20 Classification Methods (part of them) Bayes Distance Adaptive filters Neural networks Hidden Markov Model (HMM) Clustering K-Nearest-Neighbors (KNN) Support Vector Machine (SVM) Classification Methods Training Set Classifier Example Definition Bibliography

21. Feb 2007Alon Slapak 21 of 21 Bibliography 1.K. Fukunaga, Introduction to Statistical Pattern Recognition, 2 nd ed., Academic Press, San Diego, 1990. 2.L. I. Kuncheva, J. C. Bezdek amd R. P.W. Duin, “Decision Templates for Multiple Classier Fusion: An Experimental Comparison”, Pattern Recognition, 34, (2), pp. 299-314, 2001. 3.C. Kiesling, MPI for Physics, Munich - ACAT03 Workshop, KEK, Japan, Dec. 2003. Classification Methods Training Set Classifier Example Definition Bibliography