1. 1 Formal Evaluation Techniques Chapter 7

2. 2 test set error rates, confusion matrices, lift charts Focusing on formal evaluation methods for supervised learning and unsupervised clustering

3. 3 7.1 What Should Be Evaluated? 1.Supervised Model 2.Training Data 3.Attributes 4.Model Builder 5.Parameters 6.Test Set Evaluation

4. 4

5. 5 Single-Valued Summary Statistics Mean Variance Standard deviation 7.2 Tools for Evaluation

6. 6 The Normal Distribution

7. 7 Normal Distributions and Sample Means A distribution of means taken from random sets of independent samples of equal size are distributed normally. Any sample mean will vary less than two standard errors from the population mean 95% of the time.

8. 8 Computing the Standard Error The population variance is estimated by dividing the sample variance by the sample size. The standard error is computed by taking the square root of the estimated population variance.

9. 9

10. 10 A Classical Model for Hypothesis Testing Hypothesis: educated guess about the outcome of some event Experimental group, Control group Null hypothesis –There is no significant difference in the mean increase or decrease of total allergic reactions per day between patients in the group receiving treatment X and patients in the group receiving the placebo.

11. 11 A Classical Model for Hypothesis Testing To be 95% confident, P must >= 2

12. 12

13. 13 7.3 Computing Test Set Confidence Intervals

14. 14 Computing 95% Confidence Intervals 1.Given a test set sample S of size n and error rate E 2.Compute sample variance as V= E(1-E) 3.Compute the standard error (SE) as the square root of V divided by n. 4.Calculate an upper bound error as E + 2(SE) 5.Calculate a lower bound error as E - 2(SE)

15. 15 Three general comments The rest data has been randomly chosen from the pool of all possible test set instances Test, training, and validation data must represent disjoint sets The instances in each class should be distributed in the training, validation, and test data as they are seen in the entire dataset

16. 16 7.4 Comparing Supervised Learner Models

17. 17 Comparing Models with Independent Test Data where E 1 = The error rate for model M 1 E 2 = The error rate for model M 2 q = (E 1 + E 2 )/2 n 1 = the number of instances in test set A n 2 = the number of instances in test set B

18. 18 7.5 Attribute Evaluation

19. 19 Locating Redundant Attributes with Excel Correlation Coefficient Positive Correlation Negative Correlation Curvilinear Relationship (curve line) –Two attributes having a low r value may still have a curvilinear

20. 20

21. 21

22. 22

23. 23 Creating a Scatterplot Diagram with MS Excel

24. 24

25. 25 Hypothesis Testing for Numerical Attribute Significance

26. 26

27. 27 7.6 Unsupervised Evaluation Techniques Unsupervised Clustering for Supervised Evaluation – If the instances cluster into the predefined classes contained in the training data, a supervised learner model built with the training data is likely to perform well. Supervised Evaluation for Unsupervised Clustering –Designate each formed cluster as a class –Build a supervised learner model by choosing a random sampling of instances from each class –Test the supervised learner model with the remaining instances Additional Methods

28. 28 Additional Methods Designate all instances as training data Apply an alternative technique’s measure of cluster quality Create your own measure of cluster quality Perform a between-cluster attribute-value comparison.

29. 29 7.7 Evaluating Supervised Models with Numeric Output

30. 30 Mean Squared Error where for the i th instance, a i = actual output value c i = computed output value

31. 31 Mean Absolute Error where for the i th instance, a i = actual output value c i = computed output value

32. 32