1. chapter5: Evaluating Hypothesis

2. 개요 개요 Evaluating the accuracy of hypotheses is fundamental to ML. - to decide whether to use this hypothesis - integral component of many learning system Difficulty from limited set of data - Bias in the estimate - Variance in the estimate

3. 1. Contents Methods for evaluating learned hypotheses Methods for comparing the accuracy of two hypotheses Methods for comparing the accuracy of two learning algorithms when limited set of data is available

4. 2. Estimating Hypothesis Accuracy Two Interests 1. Given a hypothesis h and a data sample, what is the best estimate of the accuracy of h over unseen data? 2. What is probable error in accuracy estimate?

5. 2. Evaluating… (Cont’d) Two Definitions of Error 1. Sample Error with respect to target function f and data sample S, 2. True Error with respect to target function f and distribution D, How good an estimate of error D (h) is provided by error S (h)?

6. 2. Evaluating… (Cont’d) Problems Causing Estimating Error 1. Bias : if S is training set, error S (h) is optimistically biased estimation bias = E[error S (h)] - error D (h) For unbiased estimate, h and S must be chosen independently 2. Variance : Even with unbiased S, error S (h) may vary from error D (h)

7. 2. Evaluating… (Cont’d) Estimators Experiment : 1. Choose sample S of size n according to distribution D 2. Measure error S (h) error S (h) is a random variable error S (h) is an unbiased estimator for error D (h) Given observed error S (h) what can we conclude about error D (h) ?

8. 2. Evaluating… (Cont’d) Confidence Interval if 1. S contains n examples, drawn independently of h and each other 2. n >= 30 then with approximately N% probability, errorD(h) lies in interval

9. 2. Evaluating… (Cont’d) Normal Distribution Approximates Binomial Distribution error S (h) follows a Binomial distribution, with Approximate this by a Normal distribution with

10. 2. Evaluating… (Cont’d) More Correct Confidence Interval if 1. S contains N examples, drawn independently of h and each other 2. N>= 30 then with approximately 95% probability, error S (h) lies in interval equivalently, error S (h) lies in interval which is approximately

11. 2. Evaluating… (Cont’d) Two-sided and One-sided bounds 1. Two-sided What is the probability that error D (h) is between L and U? 2. One-sided What is the probability that error D (h) is at most U? 100(1-a)% confidence interval in Two-sided implies 100(1-a/2)% in One-sided.

12. 3. General Confidence Interval Consider a set of independent, identically distributed random variables Y 1 …Y n, all governed by an arbitrary probability distribution with mean  and variance  2. Define sample mean, Central Limit Theorem As n , the distribution governing approaches a Normal distribution, with mean  and variance  2 /n.

13. 3. General Confidence Interval (Cont’d) 1. Pick parameter p to estimate error D (h) 2. Choose an estimator error S (h) 3. Determine probability distribution that governs estimator error S (h) governed by Binomial distribution, approximated by Normal distribution when n>=30 4. Find interval (L, U) such that N% of probability mass falls in the interval

14. 4. Difference in Error of Two Hypothesis Assumption - two hypothesis h1, h2. - h1 is tested on sample S1 containing n1 random examples. h2 is tested on sample S2 containing n2 ramdom examples. Object - get difference between two true errors. where, d = error D (h1) - error D (h2)

15. 4. Difference in Error of Two Hypothesis(Cont’d) Procedure 1. Choose an estimator for d 2. Determine probability distribution that governs estimator 3. Find interval (L, U) such that N% of probability mass falls in the interval

16. 4. Difference in Error of Two Hypothesis(Cont’d) Hypothesis Test Ex) size of S 1, S 2 is 100 error s1 (h 1 )=0.30, error s2 (h 2 ) = 0.20 What is the probability that error D (h 1 ) > error D (h 2 )?

17. 4. Difference in Error of Two Hypothesis(Cont’d) Solution 1. The problem is equivalent to getting the probability of the following 2. From former expression, 3. Table of Normal distribution shows that associated confidence level for two-sided interval is 90%, so for one-sided interval, it is 95%

18. 5. Comparing Two Learning Algorithms What we’d like to estimate: where L(S) is the hypothesis output by learner L using training set S But, given limited data D 0, what is a good estimator?  Could partition D 0 into training set S and test set T 0, and measure error T0 (L A (S 0 )) - error T0 (LB(S 0 ))  Even better, repeat this many times and average the results

19. 5. Comparing Two Learning Algorithms(Cont’d) 1. Partition data D 0 into k disjoint test sets T 1, T 2, …, T k of equal size, where this size if at least 30. 2. For 1 <= i <=k, do use T i for the test set, and the remaining data for training set S i S i = {D 0 - T i }, h A = LA(S i ), h B = L B (S i ) 3. Return the value  i, where

20. 5. Comparing Two Learning Algorithms(Cont’d) 4. Now, use paired t test on to obtain a confidence interval The result is… N% confidence interval estimate for  :