1. Information Organization: Evaluation of Classification Performance

2. Classification: Evaluation Measures
Sensitivity = A / (A+C)
Probability that classifier result is correct given that an item belongs to T
Similar to Recall
Specificity = D / (B+D)
Probability that classifier result is correct given that an item belongs to F
Positive Predictive Value = A / (A+B)
Probability that classifier result is correct given that it is T
Similar to Precision
Negative Predictive Value = D / (C+D)
Probability that classifier result is correct given that it is F
Accuracy = (A+D) / (A+B+C+D)
Probability that classifier result is correct
F-measure
Actual
Predicted T F A B C D
A = True Positive B = False Postive C = False Negative D = True Negative
Search Engine

3. Classifier Evaluation: Examples
Senario 1
A ball sorter (classifier) with 99% sensitivity and 98% specificity puts 1000 green and 1000 white tennis balls into T-bin (w/ color) and F-bin (w/o color).
A= 990 green balls in T-bin (true positive)
C= 10 green balls in F-bin (false negative)
D= 980 white balls in F-bin (true negative)
B= 20 white balls in T-bin (false positive)
Positive Predictive Value = 990 / (990+20) = 90%
Negative Predictive Value = 980 / (10+980) = 98.99%
Accuracy = ( ) / 2000 = 98.5%
F-score = (2*990)/( ) = 94.29%
Scenario 2:
A ball sorter (classifier) with 99% sensitivity and 98% specificity puts 100 green and 1900 white tennis balls into T-bin (w/ color) and F-bin (w/o color).
A= 99 green balls in T-bin (true positive)
C= 1 green balls in F-bin (false negative)
D= 1862 white balls in F-bin (true negative)
B= 38 white balls in T-bin (false positive)
Positive Predictive Value = 99 / (99+38) = 72.26%
Negative Predictive Value = 1862 / (1+1862) = 99.95%
Accuracy = ( ) / 2000 = 98.05%
F-score = (2*99)/( ) = 83.54%
Actual
Predicted T F A B C D
Sensitivity = A / (A+C) Specificity = D / (B+D) PPV = A / (A+B) NPV = D / (C+D) Accuracy = (A+D) / (A+B+C+D) F-score = 2A / (A+C+A+B)
Search Engine

4. Classifier Evaluation: Multiple Classes
Actual
Predicted T F 35 3 2 60
Actual
Predicted T F 65
360 35 40
Actual
Predicted T F
100
363 37
Class 1: Accuracy = 95/100 = 0.95
Class 2: Accuracy = 105/500 = 0.21
Micro-average Table: Accuracy = 200/600 = 0.33
Multiple classes
Macro-Averaging
compute performance for each class, then average over all classes
gives equal weight to each class
can be unduly influenced by small categories
e.g., accuracy = ( ) / 2 = 0.58
Micro-Averaging
one confusion matrix for all classes
gives equal weights to each item (e.g. document)
can be dominated by large categories
e.g., accuracy = 200/600 = 0.33
Search Engine

5. n-Fold Cross Validation
A technique for estimating classifier performance
To reduce variability for accurate prediction of classifier performance
n-fold cross validation
Construct n sets of training/test data, where
training portion = (n-1)/n of the whole training data
test portion = 1/n of the whole training data (e.g. 3-fold cross validation)
Compute the classification rate on each set
For each classifier,
assess the consistency of each classification rate
if not consistent, problem w/ data or classifier
compute the average of n classification rate to compare w/ other classifiers
Search Engine

6. Harmonic Mean Harmonic Mean F-measure A type of average
e.g., average of rates
Inverse of the mean of the inverse
Count of numbers divided by sum of their reciprocals
F-measure
Harmonic mean of Precision & Recall
Example: roundtrip speed
A to B: x miles 50 miles/hr from A to B : x/50 hrs 20 miles/hr from B to A: x/20 hrs
avg. speed =