Chapter 13. General Least-Squares and Nonlinear Regression Gab-Byung Chae

13.1 Polynomial Regression -> poorly represented by a straight line. As discussed in Chap. 12, one method to accomplish this objective is to use transformations. Another alternative is to fit polynomials to the data using polynomial regression.

Least Squares Regression  Minimize some measure of the difference between the approximating function and the given data points.  In least squares method, the error is measured as :  The minimum of E occurs when the partial derivatives of E with respect to each of the variables are 0.

Linear Least Squares Regression  f(x) is in a linear form : f(x)=ax+b  the error :  Is minimized when : 

Quadratic Least Squares Approximation  f(x) is in a quadratic form : f(x)=ax 2 +bx+c  the error :  Is minimized when :

Cubic Least Squares Approximation  f(x) is in a cubic form : f(x)=ax 3 +bx 2 +cx+d  the error :  Is minimized when :  This case can be easily extended to an mth-order polynomial.

 Determining the coefficients of an mth-order polynomial is equivalent to solving a system of m+1 simutaneous linear equations.  The standard error is formulated as (m+1) data-drived coefficients- a 0, a 1, … a m, - were used to compute S r.

Example 13.1 Fit a second-order polynomial to the data in the first two columns of Table 13.1

 Sol>

>> N = [ ; ; ]; >> r = [ ] >> a = N\r a = a =

The standard error The coefficient of determination The correlation coefficient Sum of the squares of the residuals between the data points(yi) and the mean Sum of the squares of the residuals between the data points(y i ) and regression curve

Fit of a second-order polynomial. Figure 13.2

13.2 Multiple Linear Regression  An extension of linear regression : y is a linear function of two or more independent variables.  For this two-dimensional case, the regression line becomes a plane(Fig. 13.3).  The sum of the squares of the residuals:

Graphical depiction of multiple linear regression where y is a linear function of x 1 and x 2. Figure 13.3

Example 13.2 Multiple Linear Regression  Use multiple linear regression to fit this data.

Which gives us

Extension to m dimensions  Power equations of the form Standard error

13.3 General Linear Least Squares When We have simple or multiple linear regression. When We have polynomial regression.

 The functions can be highly nonlinear.  For example:  Or

 Equation (13.7) can be expressed in matrix notation as where m is the number of variables in the model and n is the number of data points. where m is the number of variables in the model and n is the number of data points.

 Because n>m, mostly Z is not a square matrix.

 By taking its partial derivative with respect to each of the coefficients and setting the resulting equation equal to zero.

Example 13.3 Polynomial Regression with MATLAB Repeat example 13.1 >> x=[ ]'; >> y=[ ]'; >> Z=[ones(size(x)) x x.^2] Z =

>> Z'*Z ans = ans = >> a=(Z'*Z)\(Z'*y) a = a =

>> Sr = sum((y-Z*a).^2) Sr = Sr = >> r2=1-Sr/sum((y-mean(y)).^2) r2 = r2 = >> Syx= sqrt(Sr/(length(x)-length(a))) Syx = Syx =

13.4 QR factorization and the backslash operator  QR factorization and singular value decomposition : beyond the scope of this book but we can use it in MATLAB which is implemented as polyfit and backslash {y} = [Z]{a} : general model Eq.(13.8) {y} = [Z]{a} : general model Eq.(13.8) >> x = [ ]' ; >> y=[ ]‘; >> z=[ones(size(x)) x x.^2]; >> a=polyfit(x,y,2) >> a =z\y

13.5 Nonlinear regression  Ex :  The sum of the square  Find a 0 and a 1 that minimize the function f  Matlab’s fminsearch function can be used for this purpose. [x, fval] =fminsearch(fun, x0, options, p1, p2,…) [x, fval] =fminsearch(fun, x0, options, p1, p2,…)

Example 13.4 Nonlinear Regression with MATLAB  Recall example 12.4 with the table 12.1: we have  This time use nonlinear regression. Employ initial guesses of 1 for the coefficient.

 Sol  M-file function f = fSSR(a, xm, ym) yp = a(1)*xm.^a(2); f =sum((ym-yp).^2); >> x=[ ]; >> y = [ ]; >> [1,1],[], x,y) ans =

Comparison of transformed and untransformed model fits for force versus velocity data from Table Figure 13.4