1. CogNova Technologies 1 Evaluating Induced Models Evaluating Induced Models with Daniel L. Silver Daniel L. Silver Copyright (c), 2004 All Rights Reserved

2. CogNova Technologies 2 Agenda  Interpretation and Evaluation Phase  Model accuracy (fitness) and confidence  Testing the difference between two models  Testing the difference between two DM methods (e.g. IDT versus ANN)

3. CogNova Technologies 3 The KDD Process Selection and Preprocessing Data Mining Interpretation and Evaluation Data Consolidation Knowledge p(x)=0.02 Data Warehouse Data Sources Patterns & Models Prepared Data Consolidated Data

4. CogNova Technologies 4 Inductive Modeling = Data Mining Basic Framework for Inductive Learning Inductive Learning System Environment Training Examples Testing Examples Induced Model of Classifier Output Classification (x, f(x)) (x, h(x)) h(x) = f(x)? Focus is on developing models that can accurately classify new examples. ~

5. CogNova Technologies 5 Model Accuracy and Confidence  to judge fitness or accuracy  Preferably a separate verification set is used to judge fitness or accuracy  Statistical confidence in the accuracy of a model can be expressed as an interval Mean Error or Error Rate h1

6. CogNova Technologies 6 The Normal Curve and Confidence Intervals  Consider a class of 30 persons  True mean (average) mark of 75%  How can we estimate this from the marks of only 10 sample persons?  Let’s do an example using Excel

7. CogNova Technologies 7 Model Accuracy and Confidence Available Examples Training Set Verify Set Approach #1: Large Sample When the amount of available data is large... 70% 30% Used to develop one model Compute Test error Divide randomly Generalization = test/verify fit Test Set

8. CogNova Technologies 8 Model Accuracy and Confidence  Generalization statistic (fit, error or accuracy) is provided by the learning system  Confidence interval must be computed: Continuous target variable - Compute mean error over n examples and confidence interval using Excel (evaluate_models.xls)Continuous target variable - Compute mean error over n examples and confidence interval using Excel (evaluate_models.xls)evaluate_models.xls Nominal (binary) target variable - Given an error rate of P from a sample of n examples, then the 95%conf. interval = 1.96 sqrt( P(1-P) / n ) = 1.96 stdevNominal (binary) target variable - Given an error rate of P from a sample of n examples, then the 95%conf. interval = 1.96 sqrt( P(1-P) / n ) = 1.96 stdev o P = number incorrect / n Strictly speaking this is for n >= 30 Strictly speaking this is for n >= 30

9. CogNova Technologies 9 Testing the Difference Between Two Models  Which of the following two hypotheses is the better? … h1 or h2 ? h2 h3 Fitness or Error Rate h1h2

10. CogNova Technologies 10 Testing the Difference Between Two Models  Assumption: If some measurable characteristic of the models is statistically different then we will consider the models different  We will focus on the characteristics: mean error, and error rate (proportion incorrect) which can be computed from the test results

11. CogNova Technologies 11 Testing the Difference Between Two Models  Continuous target variable Use a Difference of Means TestUse a Difference of Means Test  Nominal (binary) target variable Use a Difference of Proportions TestUse a Difference of Proportions Test  For 95% confidence in a difference then p-value statistic must be <= 0.05 (see Excel spreadsheet example)

12. CogNova Technologies 12 Testing the Difference Between Two DM Methods  Cross-Validation must be performed  Requires generating several models with different train, test and verify sets  With WEKA use the accuracy or error rate on the test sets

13. CogNova Technologies 13 Network Training Available Examples Training Set Ver. Set Approach #2: Cross-validation Provides a sense of confidence in model... 10% 90% Repeat 10 times Used to develop 10 different models Accumulate test errors Generalization determined by mean test fit and stddev Test Set

14. CogNova Technologies 14 Testing the Difference Between Two DM Methods  A Difference of Means T-test can be used to determine a p-value statistic  For 95% confidence in a difference then p-value statistic must be <= 0.05 (see Excel spreadsheet example)

15. CogNova Technologies 15 Example: Using Census Data  Problem: To identify males given census data  Performance measure: Accuracy = Goodness of fitAccuracy = Goodness of fit  Model generation: IDT and ANN

16. CogNova Technologies 16 Example: Using Census Data  Record results: Goodness of fit stats on test set for 10 different models Mean fitness: ANN= 26.6, IDT = 31.8Mean fitness: ANN= 26.6, IDT = 31.8  Test difference between models : Use a difference of means T-test (see evaluate_models.xls) evaluate_models.xls p-value = 0.00124p-value = 0.00124 Since p-value < 0.05, the two models are significantly differentSince p-value < 0.05, the two models are significantly different

17. CogNova Technologies 17 THE END danny.silver@acadiau.ca