Information Organization: Evaluation of Classification Performance

Classification: Evaluation Measures  Sensitivity = A / (A+C)  Probability that classifier result is correct given that an item belongs to T  Similar to Recall  Specificity = D / (B+D)  Probability that classifier result is correct given that an item belongs to F  Positive Predictive Value = A / (A+B)  Probability that classifier result is correct given that it is T  Similar to Precision  Negative Predictive Value = D / (C+D)  Probability that classifier result is correct given that it is F  Accuracy = (A+D) / (A+B+C+D)  Probability that classifier result is correct  F-measure F-measure Search Engine2 Actual Predicted TF TAB FCD A = True Positive B = False Postive C = False Negative D = True Negative

Classifier Evaluation: Examples  Senario 1  A ball sorter (classifier) with 99% sensitivity and 98% specificity puts 1000 green and 1000 white tennis balls into T-bin (w/ color) and F-bin (w/o color). A= 990 green balls in T-bin (true positive) C= 10 green balls in F-bin (false negative) D= 980 white balls in F-bin (true negative) B= 20 white balls in T-bin (false positive) Positive Predictive Value = 990 / (990+20) = 90% Negative Predictive Value = 980 / (10+980) = 98.99% Accuracy = ( ) / 2000 = 98.5% F-score = (2*990)/( ) = 94.29%  Scenario 2:  A ball sorter (classifier) with 99% sensitivity and 98% specificity puts 100 green and 1900 white tennis balls into T-bin (w/ color) and F-bin (w/o color). A= 99 green balls in T-bin (true positive) C= 1 green balls in F-bin (false negative) D= 1862 white balls in F-bin (true negative) B= 38 white balls in T-bin (false positive) Positive Predictive Value = 99 / (99+38) = 72.26% Negative Predictive Value = 1862 / (1+1862) = 99.95% Accuracy = ( ) / 2000 = 98.05% F-score = (2*99)/( ) = 83.54% Search Engine3 Actual Predicted TF TAB FCD Sensitivity = A / (A+C) Specificity = D / (B+D) PPV = A / (A+B) NPV = D / (C+D) Accuracy = (A+D) / (A+B+C+D) F-score = 2A / (A+C+A+B)

Classifier Evaluation: Multiple Classes  Multiple classes  Macro-Averaging compute performance for each class, then average over all classes gives equal weight to each class – can be unduly influenced by small categories e.g., accuracy = ( ) / 2 = 0.58  Micro-Averaging one confusion matrix for all classes gives equal weights to each item (e.g. document) – can be dominated by large categories e.g., accuracy = 200/600 = 0.33 Search Engine4 Actual Predicted TF T353 F260 Class 1: Accuracy = 95/100 = 0.95 Actual Predicted TF T65360 F3540 Class 2: Accuracy = 105/500 = 0.21 Actual Predicted TF T F37100 Micro-average Table: Accuracy = 200/600 = 0.33

n-Fold Cross Validation  Cross Validation  A technique for estimating classifier performance To reduce variability for accurate prediction of classifier performance  n-fold cross validation construct n sets of training/test data, where – training portion = (n-1)/n of the whole training data – test portion = 1/n of the whole training data (e.g. 3-fold cross validation) compute the classification rate on each set for each classifier, – assess the consistency of each classification rate » if not consistent, problem w/ data or classifier – compute the average of n classification rate to compare w/ other classifiers Search Engine5

Harmonic Mean  Harmonic Mean Harmonic Mean  A type of average e.g., average of rates  Inverse of the mean of the inverse  Count of numbers divided by sum of their reciprocals  F-measure  Harmonic mean of Precision & Recall 6 Example: roundtrip speed A to B: x miles 50 miles/hr from A to B : x/50 hrs 20 miles/hr from B to A: x/20 hrs avg. speed =