Numerical Analysis – Data Fitting Hanyang University Jong-Il Park

Department of Computer Science and Engineering, Hanyang University Fitting Exact fit  Interpolation  Extrapolation Approximate fit  Allows some errors  Optimality depends on noise model x x xx x x x x x Approximate fitting to a line e

Department of Computer Science and Engineering, Hanyang University Eg. Interpolation vs. Data Fitting

Department of Computer Science and Engineering, Hanyang University Regression errors(I) Zero-order model1st-order model

Department of Computer Science and Engineering, Hanyang University Regression errors(I) Small errors Large errors

Department of Computer Science and Engineering, Hanyang University Least-Square Data Fitting Problem statement Given N data points (x i, y i ), i=1,...,N a model that has M adjustable parameters, find that minimizes ※ Maximum Likelihood Estimation ML ≡ Least-square if e i is independently distributed Gaussian

Department of Computer Science and Engineering, Hanyang University Fitting data to a straight line Model error Sum of Errors At the minimum error S a b (a*,b*)

Department of Computer Science and Engineering, Hanyang University Data fitting to a polynomial(I) Model At the minimum error (n+1) simultaneous eg. (linear) (n+1) unknowns

Department of Computer Science and Engineering, Hanyang University Data fitting to a polynomial(II) Rewriting the eq.’s into a matrix form where Linear equation

Department of Computer Science and Engineering, Hanyang University General linear least-square(I) error Sum of Errors At the minimum error Model

Department of Computer Science and Engineering, Hanyang University General linear least-square(II) Matrix form where Since We obtain

Department of Computer Science and Engineering, Hanyang University Chi-Square Fitting If each data point (x i, y i ) has its own, known standard deviation σ i, we can formulate a weighted least-square problem : Called the “chi-square” small large Large : less weighted Small : more weighted y x

Department of Computer Science and Engineering, Hanyang University Eg. Chi-square fitting Fitting to a line At the minimum error Define Then we obtain

Department of Computer Science and Engineering, Hanyang University Multi-Dimensional Fit Model error Similarly to the derivation for the 1D fits, we get The only difference is that the dimension of x is generalized to an arbitrary dimension

Department of Computer Science and Engineering, Hanyang University Nonlinear Models Model Eg.) Problem Given find the least-square solution nonlinear fcn. of a parameters

Department of Computer Science and Engineering, Hanyang University Nonlinear fitting: easy case ; linear ◦ Some Nonlinear functions can be transformed into a linear form. Simple BUT Not an optimum fitting!

Department of Computer Science and Engineering, Hanyang University Nonlinear fitting by linearization

Department of Computer Science and Engineering, Hanyang University Nonlinear Least-Square Fitting Model Problem Minimize → where Cost function

Department of Computer Science and Engineering, Hanyang University Levenberg-Marquardt Method(I) Some insight Hessian matrix update term gradient Inverse-Hessian method Steepest Descent method : Near the minimum : Far from the minimum

Department of Computer Science and Engineering, Hanyang University Levenberg-Marquardt Method(II) Idea -Start : steepest descent method -End : Inverse-Hessian method gradually change : Inverse Hessian Steepest descent

Department of Computer Science and Engineering, Hanyang University Levenberg-Marquardt Method(III) Algorithm ⅰ ) Guess ⅱ ) compute ⅲ ) pick a modest, say ⅳ ) solve where ⅴ ) if increase by a factor of 10 and go to ⅳ ) else decrease by a factor of 10 and update and go to ⅳ ) ①

Department of Computer Science and Engineering, Hanyang University Levenberg-Marquardt Method(IV) Inverse Hessian decrease Steepest descent ※ For large, in eq. ①, the is diagonal dominant eq. ① is close to steepest descent!

Department of Computer Science and Engineering, Hanyang University Calculation of the gradient Calculation of the gradient and the Hessian of ◦ Gradient

Department of Computer Science and Engineering, Hanyang University Calculation of the Hessian ignore Sensitive to noise ; symmetric

Department of Computer Science and Engineering, Hanyang University Homework #8: Programming [Due: Dec. 3] Data files are given in the course homepage (fitdata1.dat, fitdata2.dat, fitdata3.dat).