1. Numerical Analysis – Data Fitting Hanyang University Jong-Il Park

2. Department of Computer Science and Engineering, Hanyang University Fitting Exact fit  Interpolation  Extrapolation Approximate fit  Allows some errors  Optimality depends on noise model x x xx x x x x x Approximate fitting to a line e

3. Department of Computer Science and Engineering, Hanyang University Eg. Interpolation vs. Data Fitting

4. Department of Computer Science and Engineering, Hanyang University Regression errors(I) Zero-order model1st-order model

5. Department of Computer Science and Engineering, Hanyang University Regression errors(I) Small errors Large errors

6. Department of Computer Science and Engineering, Hanyang University Least-Square Data Fitting Problem statement Given N data points (x i, y i ), i=1,...,N a model that has M adjustable parameters, find that minimizes ※ Maximum Likelihood Estimation ML ≡ Least-square if e i is independently distributed Gaussian

7. Department of Computer Science and Engineering, Hanyang University Fitting data to a straight line Model error Sum of Errors At the minimum error S a b (a*,b*)

8. Department of Computer Science and Engineering, Hanyang University Data fitting to a polynomial(I) Model At the minimum error (n+1) simultaneous eg. (linear) (n+1) unknowns

9. Department of Computer Science and Engineering, Hanyang University Data fitting to a polynomial(II) Rewriting the eq.’s into a matrix form where Linear equation

10. Department of Computer Science and Engineering, Hanyang University General linear least-square(I) error Sum of Errors At the minimum error Model

11. Department of Computer Science and Engineering, Hanyang University General linear least-square(II) Matrix form where Since We obtain

12. Department of Computer Science and Engineering, Hanyang University Chi-Square Fitting If each data point (x i, y i ) has its own, known standard deviation σ i, we can formulate a weighted least-square problem : Called the “chi-square” small large Large : less weighted Small : more weighted y x

13. Department of Computer Science and Engineering, Hanyang University Eg. Chi-square fitting Fitting to a line At the minimum error Define Then we obtain

14. Department of Computer Science and Engineering, Hanyang University Multi-Dimensional Fit Model error Similarly to the derivation for the 1D fits, we get The only difference is that the dimension of x is generalized to an arbitrary dimension

15. Department of Computer Science and Engineering, Hanyang University Nonlinear Models Model Eg.) Problem Given find the least-square solution nonlinear fcn. of a parameters

16. Department of Computer Science and Engineering, Hanyang University Nonlinear fitting: easy case ; linear ◦ Some Nonlinear functions can be transformed into a linear form. Simple BUT Not an optimum fitting!

17. Department of Computer Science and Engineering, Hanyang University Nonlinear fitting by linearization

18. Department of Computer Science and Engineering, Hanyang University Nonlinear Least-Square Fitting Model Problem Minimize → where Cost function

19. Department of Computer Science and Engineering, Hanyang University Levenberg-Marquardt Method(I) Some insight Hessian matrix update term gradient Inverse-Hessian method Steepest Descent method : Near the minimum : Far from the minimum

20. Department of Computer Science and Engineering, Hanyang University Levenberg-Marquardt Method(II) Idea -Start : steepest descent method -End : Inverse-Hessian method gradually change : Inverse Hessian Steepest descent

21. Department of Computer Science and Engineering, Hanyang University Levenberg-Marquardt Method(III) Algorithm ⅰ ) Guess ⅱ ) compute ⅲ ) pick a modest, say ⅳ ) solve where ⅴ ) if increase by a factor of 10 and go to ⅳ ) else decrease by a factor of 10 and update and go to ⅳ ) ①

22. Department of Computer Science and Engineering, Hanyang University Levenberg-Marquardt Method(IV) Inverse Hessian decrease Steepest descent ※ For large, in eq. ①, the is diagonal dominant eq. ① is close to steepest descent!

23. Department of Computer Science and Engineering, Hanyang University Calculation of the gradient Calculation of the gradient and the Hessian of ◦ Gradient

24. Department of Computer Science and Engineering, Hanyang University Calculation of the Hessian ignore Sensitive to noise ; symmetric

25. Department of Computer Science and Engineering, Hanyang University Homework #8: Programming [Due: Dec. 3] Data files are given in the course homepage (fitdata1.dat, fitdata2.dat, fitdata3.dat).