Least-Squares Regression Chapter 17 Least-Squares Regression Lecture Notes Dr. Rakhmad Arief Siregar Universiti Malaysia Perlis Applied Numerical Method for Engineers

Curve Fitting

Curve Fitting

Curve Fitting

Simple Statistics

Regression Polynomial fit Experimental data Least-squares fit

Linear Regression The simplest example of a least-squares approximation is fitting a straight line a0 and a1 are coefficients representing the intercept and the slope e is the error or residual between the model and the observations

Linear Regression By rearranging: e is the error or residual, the discrepancy between the true value of y a0+a1x is the approximate value

Criteria for a “Best” Fit By minimizing the sum of the residual error n is total number of points

Criteria for a “Best” Fit By minimizing the sum of absolute residual error n is total number of points

Criteria for a “Best” Fit By minimizing the sum of the squares of the residuals between the measured y and the y calculated with the linear mode

Best fit Minimizes the sum of the residuals Minimizes the sum of the absolute value of residuals Minimizes the maximum error of any individual point

Least-Squares Fit of a Straight Line Differentials:

Least-Squares Fit of a Straight Line After several mathematical steps, a0 and a1 will yields: Where y and x are the means of y and x, respectively

Ex. 17.1 Fit a straight line to the x and y values in the first two columns of Table below.

Ex. 17.1 The following quantities can be computed

Ex. 17.1 a1 and a0 can be computed:

Ex. 17.1 The least-squares fit is:

Problem 17.4 Use least-squares regression to fit a straight line to:

Problem 17.4

Quantification of Error of Linear Regression Squared of residual error:

Quantification of Error of Linear Regression If those criteria are met, a “standard deviation” for regression line can be determined as: where: Sy/x is called standard error of estimate. Subscript y/x means the error is for a predicted value of y corresponding to a particular value of x

Quantification of Error of Linear Regression The spread of the data around the mean The spread of the data around best fit line

Quantification of Error of Linear Regression Small residual errors Large residual errors

Quantification of Error of Linear Regression The difference between the two quantities, St –Sr, quantifies the improvement or error reduction due to describing the data in terms of a straight line. The difference is normalized to St to yield: r2 : coefficient of determination r : correlation coefficient

Ex. 17.2 Compute the total standard deviation, the standard error of the estimate and the correlation coefficient for the data in Ex. 17.1

Ex. 17.2 Solution Standard deviation: Standard error of estimate: The extent of the improvement is qualified because sy/x < sy the linear regression model has merit

Ex. 17.2 Solution The correlation coefficient: These results indicate 86.8 percent of the original uncertainty has been explained by the linear model

Linearization of Nonlinear Relationships Linear regression provides a powerful technique for fitting a best line to data. How about data shown below?

Linearization of Nonlinear Relationships Exponential equation Linearization of Nonlinear Relationships Transformations can be used to express the data in form that is compatible with linear regression A straight line with a slope 1 and intercept of ln 1 By natural logarithm

Linearization of Nonlinear Relationships Power equation A straight line with a slope 2 and intercept of log 2 By base-10 logarithm

Linearization of Nonlinear Relationships The saturation-growth-rate equation A straight line with a slope 3 / 3 and intercept of 1/3 By inverting

Ex. 17.4 Fit Eq. below to the data in table 17.3 using a logarithmic transformation of the data.

Ex. 17.4 Intercept of log 2 Slope of 1 Intercept data

Polynomial Regression

Polynomial Regression This method can utilize the least-squares procedure to fit the data to a higher-order polynomial.

Polynomial Regression Derivation with respect to each unknown coefficients of polynomial as in

Polynomial Regression Derivations can be set equal to zero and rearranged as: How to solve it?

Polynomial Regression In matrix form What method can be used?

Polynomial Regression The two dimensional case can be easily extended to an m-th order polynomial as: The standard error for this case is formulated as

Ex. 17.5 Fit a second-order polynomial to the data in table below:

Ex. 17.5 Solution: m=2, n=6

Ex. 17.5 Solution: The simultaneous linear equation are:

Ex. 17.5 Solution: By using gauss elimination it will yield: a0=2.47857, a1=2.35929 and a2=1.86071 The least-square quadratic equation:

Ex. 17.5 The standard error:

Ex. 17.5 The coefficient of determination:

Ex. 17.5 99.851% of the original uncertainty has been explain by the model

Assignment 3 Do Problems 17.5, 17.6, 17.7, 17.10 and 17.12 Submit next week

Multiple Linear Regression For this section, two-dimensional case, regression line become a plane

Multiple Linear Regression This method can utilize the least-squares procedure to fit the data to a higher-order polynomial.

Multiple Linear Regression Derivation with respect to each unknown coefficients of polynomial as in

Multiple Linear Regression Derivations can be set equal to zero and rearranged as in matrix form

Multiple Linear Regression The two dimensional case can be easily extended to an m-th order polynomial as: The standard error for this case is formulated as

Ex. 17.6 The following data was calculated from equation: y=5+4x1-3x2 Use multiple linear regression to fit this data

Ex. 17.6

Ex. 17.6 Solution

Ex. 17.6 solution a0=5, a1=4 and a2=-3

Problems 17.17 Use multiple linear regression to fit. Compute the coefficients, the standard error of estimate and the correlation coefficient

Problems 17.17

Problems 17.17 Solution

Nonlinear Regression The Gauss-Newton method is one algorithm for minimizing the sum of the squares of the residuals between data and nonlinear equation. For convenience

Nonlinear Regression The nonlinear model can be expanded in a Tailor series around the parameter values and curtailed after the first derivative Ex. For a two-parameter case:

Nonlinear Regression It needs to be linearized by substituting into It will yields

Nonlinear Regression In matrix form

Nonlinear Regression By applying least-square theory to It will yield in normal equation: By using ave Eq. we can compute values for:

Ex. 17.9