INTRODUCTION TO Machine Learning Lecture Slides for INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, 2004 alpaydin@boun.edu.tr http://www.cmpe.boun.edu.tr/~ethem/i2ml

CHAPTER 14: Assessing and Comparing Classification Algorithms

Introduction Questions: Assessment of the expected error of a learning algorithm: Is the error rate of 1-NN less than 2%? Comparing the expected errors of two algorithms: Is k-NN more accurate than MLP ? Training/validation/test sets Resampling methods: K-fold cross-validation Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Algorithm Preference Criteria (Application-dependent): Misclassification error, or risk (loss functions) Training time/space complexity Testing time/space complexity Interpretability Easy programmability Cost-sensitive learning Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Resampling and K-Fold Cross-Validation The need for multiple training/validation sets {Xi,Vi}i: Training/validation sets of fold i K-fold cross-validation: Divide X into k, Xi,i=1,...,K Ti share K-2 parts Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

5×2 Cross-Validation 5 times 2 fold cross-validation (Dietterich, 1998) Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Bootstrapping Draw instances from a dataset with replacement Prob that we do not pick an instance after N draws that is, only 36.8% is new! Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Measuring Error Error rate = # of errors / # of instances = (FN+FP) / N Recall = # of found positives / # of positives = TP / (TP+FN) = sensitivity = hit rate Precision = # of found positives / # of found = TP / (TP+FP) Specificity = TN / (TN+FP) False alarm rate = FP / (FP+TN) = 1 - Specificity Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

ROC Curve Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Interval Estimation X = { xt }t where xt ~ N ( μ, σ2) m ~ N ( μ, σ2/N) 100(1- α) percent confidence interval Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

When σ2 is not known: Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Hypothesis Testing Reject a null hypothesis if not supported by the sample with enough confidence X = { xt }t where xt ~ N ( μ, σ2) H0: μ = μ0 vs. H1: μ ≠ μ0 Accept H0 with level of significance α if μ0 is in the 100(1- α) confidence interval Two-sided test Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

One-sided test: H0: μ ≤ μ0 vs. H1: μ > μ0 Accept if Variance unknown: Use t, instead of z Accept H0: μ = μ0 if Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Assessing Error: H0: p ≤ p0 vs. H1: p > p0 Single training/validation set: Binomial Test If error prob is p0, prob that there are e errors or less in N validation trials is Accept if this prob is less than 1- α 1- α N=100, e=20 Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Normal Approximation to the Binomial Number of errors X is approx N with mean Np0 and var Np0(1-p0) Accept if this prob for X = e is less than z1-α 1- α Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Paired t Test Multiple training/validation sets xti = 1 if instance t misclassified on fold i Error rate of fold i: With m and s2 average and var of pi we accept p0 or less error if is less than tα,K-1 Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Comparing Classifiers: H0: μ0 = μ1 vs. H1: μ0 ≠ μ1 Single training/validation set: McNemar’s Test Under H0, we expect e01= e10=(e01+ e10)/2 Accept if < X2α,1 Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

K-Fold CV Paired t Test Use K-fold cv to get K training/validation folds pi1, pi2: Errors of classifiers 1 and 2 on fold i pi = pi1 – pi2 : Paired difference on fold i The null hypothesis is whether pi has mean 0 Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

5×2 cv Paired t Test Use 5×2 cv to get 2 folds of 5 tra/val replications (Dietterich, 1998) pi(j) : difference btw errors of 1 and 2 on fold j=1, 2 of replication i=1,...,5 Two-sided test: Accept H0: μ0 = μ1 if in (-tα/2,5,tα/2,5) One-sided test: Accept H0: μ0 ≤ μ1 if < tα,5 Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

5×2 cv Paired F Test Two-sided test: Accept H0: μ0 = μ1 if < Fα,10,5 Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Comparing L>2 Algorithms: Analysis of Variance (Anova) Errors of L algorithms on K folds We construct two estimators to σ2 . One is valid if H0 is true, the other is always valid. We reject H0 if the two estimators disagree. Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Other Tests Range test (Newman-Keuls): Nonparametric tests (Sign test, Kruskal-Wallis) Contrasts: Check if 1 and 2 differ from 3,4, and 5 Multiple comparisons require Bonferroni correction If there are m tests, to have an overall significance of α, each test should have a significance of α/m. Regression: CLT states that the sum of iid variables from any distribution is approximately normal and the preceding methods can be used. Other loss functions ? Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)