10701 / 15781 Machine Learning Today: - Cross validation, - maximum likelihood estimation Next time: Mixtures models, EM

Linear regression A simple linear regression function is given by: We can estimate the parameters w by minimizing the empirical (or training) error The hope here is that the resulting parameters/linear function has a low “generalization error”, i.e., error on the new examples

Beyond linear regression Linear regression for input vectors: Can also generalize these classes of functions to be non-linear functions of the inputs x but still linear in the parameters w.

Polynomial regression examples

Over fitting With too few training examples our polynomial regression model may achieve zero training error but nevertheless has a large generalization error When the training error no longer bears any relation to the generalization error we say that the function overfits the (training) data

Cross validation Cross-validation allows us to estimate the generalization error based on training examples alone. For example, the leave-one-out cross-validation error is given by where are the least squares estimates of the parameters w computed without the tth training example.

Cross validation: Example

Statistical view of linear regression In a statistical regression model we model both the function and noise where ~N(0,2) Whatever we cannot capture with our chosen family of functions will be interpreted as noise

Statistical view of linear regression Our function f(x; w) here is trying to capture the mean of the observations y given the input where E{ y | x, model} is the conditional expectation (mean) of y given x, evaluated according to the model.

Conditional probability According to our statistical model the outputs y given x are normally distributed with mean f(x; w) and variance 2 thus,

Conditional probability According to our statistical model the outputs y given x are normally distributed with mean f(x; w) and variance 2 thus, As a result we can also measure the uncertainty in the predictions, not just the mean Loss function? Estimation?

Maximum likelihood estimation Given observations Dn = {(x1,y1),...,(xn,yn)}, we find the parameters w that maximize the likelihood of the outputs where:

Log likelihood estimation Likelihood of data given model: It is often easier (but equivalent) to try to maximize the log-likelihood:

Log likelihood estimation Likelihood of data given model: It is often easier (but equivalent) to try to maximize the log-likelihood:

Maximum likelihood vs. loss Log likelihood: Our model of the noise in the outputs and the resulting (effective) loss-function in maximum likelihood estimation are intricately related

Maximum likelihood estimation The likelihood of observations is a generic fitting criterion. We can also fit the noise variance 2 by maximizing the log-likelihood l(D; w,2) with respect to 2 if are the maximum likelihood parameters for f(x;w), then the optimal choice for 2 is i.e., mean squared prediction error.

Solving for w When the noise is assumed to be zero mean Gaussian, maximum likelihood setting of the linear regression parameters w =[w0,w1]T reduces to least squares fitting: where

Properties of w We can study the estimator w further if we assume that the linear model is indeed correct, i.e., the outputs were generated according to with some unknown parameters

Bias and variance of estimator Major assumption: where: We keep the training inputs or X fixed and study how varies if we resample the corresponding outputs Bias: whether deviates from w* on average Variance (covariance): how much varies around its mean

Computing the bias We use the fact that y=Xw*+e to represent in terms of w*:

Computing the bias We use the fact that y=Xw*+e to represent in terms of w*:

Computing the bias We use the fact that y=Xw*+e to represent in terms of w*:

Computing the bias We use the fact that y=Xw*+e to represent in terms of w*: The parameter estimate based on the sampled data is therefore the correct parameter plus estimate based purely on noise.

Bias (cont) Since the noise is zero mean by assumption, our parameter estimate is:

Bias (cont) Since the noise is zero mean by assumption, our parameter estimate is unbiased:

Bias (cont) Since the noise is zero mean by assumption, our parameter estimate is unbiased:

Bias (cont) Since the noise is zero mean by assumption, our parameter estimate is unbiased: where the conditional expectation is over the noisy outputs while keeping the inputs x1,...,xn, or X, fixed.

Estimator variance We can also evaluate the (conditional) covariance of the parameters, i.e., how the individual parameters co-vary due to the noise in the outputs:

Estimator variance We can also evaluate the (conditional) covariance of the parameters, i.e., how the individual parameters co-vary due to the noise in the outputs:

Estimator variance We can also evaluate the (conditional) covariance of the parameters, i.e., how the individual parameters co-vary due to the noise in the outputs:

Estimator variance We can also evaluate the (conditional) covariance of the parameters, i.e., how the individual parameters co-vary due to the noise in the outputs:

Estimator variance We can also evaluate the (conditional) covariance of the parameters, i.e., how the individual parameters co-vary due to the noise in the outputs:

ML summary When the assumptions in the linear model are correct, the ML estimator follows a simple Gaussian distribution given by The above Gaussian distribution summarizes the uncertainty that we have about the parameters based on a training set Dn = {(x1,y1),...,(xn,yn)},

Acknowledgment These slides are based in part on slides from previous machine learning classes taught by Andrew Moore at CMU and Tommi Jaakkola at MIT. I thank Andrew and Tommi for letting me use their slides.