Evaluating Hypotheses CS 9633 Machine Learning Evaluating Hypotheses Adapted from notes by Tom Mitchell http://www-2.cs.cmu.edu/~tom/mlbook-chapter-slides.html Computer Science Department CS 9633 Machine Learning

Computer Science Department Three big questions Given the observed accuracy of a hypothesis over a limited sample of data, how well does this estimate its accuracy over additional data? Given that one hypothesis outperforms another over some sample of data, how probable is it that this hypothesis is more accurate in general? When data is limited, what is the best way to use this data to both learn a hypothesis and estimate its accuracy? Computer Science Department CS 9633 Machine Learning

Difficulties in estimating hypotheses with small data sets Bias in the estimate Performance over training data provides an over-optimistic estimate of performance More likely with rich hypothesis space Typically use test set to estimate accuracy Variance in the estimate Measured accuracy over test set can still differ from true accuracy The smaller the test sample, the greater the expected variance Computer Science Department CS 9633 Machine Learning

Estimating Hypothesis Accuracy We want to know the accuracy of the hypothesis in classifying future instances We also want to know the probably error of the accuracy estimate Computer Science Department CS 9633 Machine Learning

Two Questions More Precisely Given a hypothesis h and a data sample containing n examples drawn at random according to the distribution D, what is the best estimate of the accuracy of h over future instances drawn from the same distribution? What is the probably error of the accuracy estimate? Computer Science Department CS 9633 Machine Learning

Sample Error and True Error Sample error: error rate of the hypothesis over the sample of data that is available True Error: the error rate of the hypothesis over the entire unknown distribution D of examples Computer Science Department CS 9633 Machine Learning

Computer Science Department Sample Error Definition: The sample error (denoted errors(h)) of hypothesis h with respect to target function f and data sample S is Where n is the number of examples in S, and the quantity d(f(x),h(x)) is 1 if f(x)h(x) and 0 otherwise. Computer Science Department CS 9633 Machine Learning

Computer Science Department True Error Definition: The true error (denoted errorD(h)) of hypothesis h with respect to target function f and distribution D , is the probability that h will misclassify an instance drawn at random according to D. Computer Science Department CS 9633 Machine Learning

Computer Science Department The problem We want to know errorD(h) but we can only obtain errors(h) So we must determine how well the sample error estimates the true error. Computer Science Department CS 9633 Machine Learning

Confidence Intervals for Discrete-Valued Hypotheses We wish to estimate the true error for some discrete-valued hypothesis h, based on its observed sample error over a sample S where The sample S contains n examples drawn independent of one another, and independent of h, according to the probability distribution D n 30 Hypothesis h commits r errors over these n examples Errors(h)  r/n Computer Science Department CS 9633 Machine Learning

Computer Science Department Important Result Given no other information, the most probable value of errorD(h) is errors(h) With approximately 95% probability, the true errorD(h) lies in the interval Computer Science Department CS 9633 Machine Learning

Computer Science Department More General Result Confidence Level N% 50% 68% 80% 90% 95% 98% 99% Constant zN 0.67 1.00 1.28 1.64 1.96 2.33 2.58 Computer Science Department CS 9633 Machine Learning

Where the Binomial Distribution Applies Notation Let n be the number of trials Let r be the number of successes Let p be the probability of success Conditions for Application Two possible outcomes (success and failure) The probability of success is the same on each trial There are n trials, where n is constant The n trials are independent Computer Science Department CS 9633 Machine Learning

The Binomial Distribution Computer Science Department CS 9633 Machine Learning

Mean (Expected Value) and Variance Computer Science Department CS 9633 Machine Learning

Difference in sampling error and true error errors(h) is an estimator of errorD(h) Is it an unbiased estimator? The estimation bias of an estimator Y for an arbitrary parameter p is E[Y]-p If the estimation bias is zero, Y is an unbiased estimator of P Computer Science Department CS 9633 Machine Learning

Answer to previous question What is expected value of r? E(r) = What is expected value of r/n? E(r/n) = Is r/n unbiased estimate of p? Computer Science Department CS 9633 Machine Learning

Important distinction Estimation bias is a number Inductive bias is a set of assertions Computer Science Department CS 9633 Machine Learning

Variance of an estimator Problem: r = 15 n = 50 What is variance of errors(h)? Computer Science Department CS 9633 Machine Learning

General Expression for Variance Computer Science Department CS 9633 Machine Learning

Computer Science Department Confidence Intervals Definition: An N% confidence interval for some parameter p is an interval that is expected with probability N% to contain p. Computer Science Department CS 9633 Machine Learning

Finding Confidence Intervals Question: For a given value of N how can we find the size of the interval that contains N% of the probability mass? Answer: Approximate binomial distribution with normal distribution Rule of thumb: ok for n>30 Computer Science Department CS 9633 Machine Learning

Computer Science Department Results If a random variable Y obeys a normal distribution with mean  and standard deviation , then the measured value y of Y will fall in the following interval N% of the time   zN Equivalently, the mean  will fall in the following interval N% of the time y  zN Computer Science Department CS 9633 Machine Learning

N% confidence interval 2 approximations When estimating standard deviation used errors(h) as approximation of errorD(h) Used normal distribution to approximate binomial Computer Science Department CS 9633 Machine Learning