1. Evaluating Classifiers Reading: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website)An introduction to ROC analysis

2. Evaluating Classifiers What we want: classifier that best predicts unseen data Assumption: Data is “iid” (independently and identically distributed)

3. Accuracy and Error Rate Accuracy = fraction of correct classifications on unseen data (test set) Error rate = 1 − Accuracy

4. How to use available data to best measure accuracy? Split data into training and test sets. But how to split? Too little training data: Don’t get optimal classifier Too little test data: Measured accuracy is not correct

5. One solution: “k-fold cross validation” Each example is used both as a training instance and as a test instance. Split data into k disjoint parts: S 1, S 2,..., S k. For i = 1 to k Select S i to be the test set. Train on the remaining data, test on S i, to obtain accuracy A i. Report as the final accuracy.

6. “Confusion matrix” for a given class c Actual Predicted (or “classified”) PositiveNegative (in class c)(not in class c) Positive (in class c) TruePositiveFalseNegative Negative (not in class c)FalsePositiveTrueNegative Evaluating classification algorithms

7. Example: “A” vs. “B” Results from Perceptron: Instance Class Perception Output 1 “A”−1 2 “A”+1 3 “A”+1 4 “A”−1 5 “B”+1 6 “B”−1 7 “B”−1 1 “B”−1 Accuracy: ActualPredicted PositiveNegative Positive Negative Assume “A” is positive class Confusion Matrix

8. Evaluating classification algorithms “Confusion matrix” for a given class c Actual Predicted (or “classified”) PositiveNegative (in class c)(not in class c) Positive (in class c) TruePositiveFalseNegative Negative (not in class c)FalsePositiveTrueNegative Type 2 error Type 1 error

9. Precision and Recall All instances in test set Positive Instances Negative Instances

10. Precision and Recall All instances in test set Instances classified as positive Positive Instances Negative Instances

11. Instances classified as negative Precision and Recall All instances in test set Instances classified as positive Positive Instances Negative Instances Recall = Sensitivity = True Positive Rate

12. Example: “A” vs. “B” Results from Perceptron: Instance Class Perception Output 1 “A”−1 2 “A”+1 3 “A”+1 4 “A”−1 5 “B”+1 6 “B”−1 7 “B”−1 1 “B”−1 Assume “A” is positive class

13. Results of classifier ThresholdAccuracyPrecisionRecall.9.8.7.6.5.4.3.2.1 -∞ Creating a Precision vs. Recall Curve

14. 14 (“sensitivity”) (1  “specificity”)

15. 15

16. Results of classifier ThresholdAccuracyTPRFPR.9.8.7.6.5.4.3.2.1 -∞ Creating a ROC Curve

17. Precision/Recall versus ROC curves 17 http://blog.crowdflower.com/2008/06/aggregate-turker-judgments-threshold-calibration/

18. 18

19. 19

20. 20 There are various algorithms for computing AOC (often built into statistical software) – we won’t cover this in this class.

21. How to create a ROC curve for a perceptron Run your perceptron on each instance in the test data, without doing the sgn step: Get the range [min, max] of your scores Divide the range into about 200 thresholds, including −∞ and max For each threshold, calculate TPR and FPR Plot TPR (y-axis) vs. FPR (x-axis)