1. Cost-Sensitive Classifier Evaluation Robert Holte Computing Science Dept. University of Alberta Co-author Chris Drummond IIT, National Research Council, Ottawa

2. Classifiers A classifier assigns an object to one of a predefined set of categories or classes. Examples: –A metal detector either sounds an alarm or stays quiet when someone walks through. –A credit card application is either approved or denied. –A medical test’s outcome is either positive or negative. This talk: only two classes, “positive” and “negative”.

3. Two Types of Error False negative (“miss”), FN alarm doesn’t sound but person is carrying metal False positive (“false alarm”), FP alarm sounds but person is not carrying metal

4. 2-class Confusion Matrix Reduce the 4 numbers to two rates true positive rate = TP = (#TP)/(#P) false positive rate = FP = (#FP)/(#N) Rates are independent of class ratio* True class Predicted class positivenegative positive (#P)#TP#P - #TP negative (#N)#FP#N - #FP * subject to certain conditions

5. Example: 3 classifiers True Predicted posneg pos6040 neg2080 True Predicted posneg pos7030 neg50 True Predicted posneg pos4060 neg3070 Classifier 1 TP = 0.4 FP = 0.3 Classifier 2 TP = 0.7 FP = 0.5 Classifier 3 TP = 0.6 FP = 0.2

6. Assumptions Standard Cost Model –correct classification costs 0 –cost of misclassification depends only on the class, not on the individual example –over a set of examples costs are additive Costs or Class Distributions: –are not known precisely at evaluation time –may vary with time –may depend on where the classifier is deployed True FP and TP do not vary with time or location, and are accurately estimated.

7. How to Evaluate Performance ? Scalar Measures –Accuracy –Expected cost –Area under the ROC curve Visualization Techniques –ROC curves –Cost Curves

8. What’s Wrong with Scalars ? A scalar does not tell the whole story. –There are fundamentally two numbers of interest (FP and TP), a single number invariably loses some information. –How are errors distributed across the classes ? –How will each classifier perform in different testing conditions (costs or class ratios other than those measured in the experiment) ? A scalar imposes a linear ordering on classifiers. –what we want is to identify the conditions under which each is better.

9. What’s Wrong with Scalars ? A table of scalars is just a mass of numbers. –No immediate impact –Poor way to present results in a paper –Equally poor way for an experimenter to analyze results Some scalars (accuracy, expected cost) require precise knowledge of costs and class distributions. –Often these are not known precisely and might vary with time or location of deployment.

10. Why visualize performance ? Shape of curves more informative than a single number Curve informs about –all possible misclassification costs* –all possible class ratios* –under what conditions C1 outperforms C2 Immediate impact (if done well) * subject to certain conditions

11. Example: 3 classifiers True Predicted posneg pos6040 neg2080 True Predicted posneg pos7030 neg50 True Predicted posneg pos4060 neg3070 Classifier 1 TP = 0.4 FP = 0.3 Classifier 2 TP = 0.7 FP = 0.5 Classifier 3 TP = 0.6 FP = 0.2

12. Ideal classifier chance always negative always positive ROC plot for the 3 Classifiers

13. Dominance

14. Operating Range Slope indicates the class distributions and misclassification costs for which the classifier is better than always-negative ditto for always-positive

15. Convex Hull Slope indicates the class distributions and misclassification costs for which the red classifier is the same as the blue one.

16. Creating an ROC Curve A classifier produces a single ROC point. If the classifier has a “sensitivity” parameter, varying it produces a series of ROC points (confusion matrices). Alternatively, if the classifier is produced by a learning algorithm, a series of ROC points can be generated by varying the class ratio in the training set.

17. ROC Curve

18. What’s Wrong with ROC Curves ?

19. When to switch from C4.5 to IB1 ? What is the performance difference ? When to use the default classifiers ? ROC curves for two classifiers. How to tell if two ROC curves’ difference is statistically significant ?

20. How to average them? How to compute a confidence interval for the average ROC curve ? ROC curves from two cross-validation runs.

21. And we would like be able to answer all these questions by visual inspection …

22. Cost Curves

23. Cost Curves (1) Error Rate Probability of Positive P(+) 0.81.00.00.20.40.6 0.0 0.2 0.4 0.6 0.8 1.0 Classifier 1 TP = 0.4 FP = 0.3 Classifier 2 TP = 0.7 FP = 0.5 Classifier 3 TP = 0.6 FP = 0.2 FP FN = 1-TP

24. Cost Curves (2) Error Rate Probability of Positive P(+) 0.81.00.00.20.40.6 0.0 0.2 0.4 0.6 0.8 1.0 “always negative”“always positive” Operating Range

25. Lower Envelope Error Rate Probability of Positive P(+) 0.81.00.00.20.40.6 0.0 0.2 0.4 0.6 0.8 1.0

26. Cost Curves Error Rate Probability of Positive P(+) 0.81.00.00.20.40.6 0.0 0.2 0.4 0.6 0.8 1.0 “always negative”“always positive”

27. Taking Costs Into Account Y = FN X + FP (1-X) So far, X = p(+), making Y = error rate Y = expected cost normalized to [0,1] X = p(+) C(-|+) p(+) C(-|+) + (1-p(+)) C(+|-)

28. Comparing Cost Curves

29. Averaging ROC Curves

30. Averaging Cost Curves

31. Cost Curve Avg. in ROC Space

32. Confidence Intervals True Predicted posneg pos7822 neg4060 Original TP = 0.78 FP = 0.4 Predicted negpos True 6238neg 1783pos Resample #2 TP = 0.83 FP = 0.38 Resample confusion matrix 10000 times and take 95% envelope Resample #1 TP = 0.75 FP = 0.45 Predicted negpos True 5545neg 2575pos

33. Confidence Interval Example

34. Paired Resampling to Test Statistical Significance Predicted by Classifier1 Predicted by Classifier2 posneg pos3010 neg060 For the 100 test examples in the negative class: FP for classifier1: (30+10)/100 = 0.40 FP for classifier2: (30+0)/100 = 0.30 FP2 – FP1 = -0.10 Resample this matrix 10000 times to get (FP2-FP1) values. Do the same for the matrix based on positive test examples. Plot and take 95% envelope as before.

35. Paired Resampling to Test Statistical Significance classifier1 classifier2 FN2-FN1 FP2-FP1

36. Correlation between Classifiers Predicted by Classifier1 Predicted by Classifier2 posneg pos3010 neg060 High Correlation Low Correlation (same FP1 and FP2 as above) Predicted by Classifier1 Predicted by Classifier2 posneg pos040 neg30

37. Low correlation = Low significance classifier1 classifier2 FN2-FN1 FP2-FP1

38. Limited Range of Significance

39. Better Data Analysis

40. ROC, C4.5 Splitting Criteria

41. Cost Curve, C4.5 Splitting Criteria

42. ROC, Selection procedure Suppose this classifier was produced by a training set with a class ratio of 10:1, and was used whenever the deployment situation had a 10:1 class ratio.

43. Cost Curves, Selection Procedure

44. ROC, Many Points

45. Cost Curves, Many Lines

46. Conclusions Scalar performance measures should not be used if costs and class distributions are not exactly known or might vary with time or location. Cost curves enable easy visualization of –Average performance (expected cost) –operating range –confidence intervals on performance –difference in performance and its significance

47. Fin Cost curve software is available. Contact: holte@cs.ualberta.ca Thanks to Alberta Ingenuity Centre for Machine Learning (www.aicml.ca)