1. A system of nonlinear equations
Newton’s method
Steepest gradient descent method
Advanced Numerical Computation 2008, AM NDHU

2. Advanced Numerical Computation 2008, AM NDHU
Outline
Matlab toolbox
Newton’s method for nonlinear system solving
Updating rule
Matlab implementation
Partial derivative
Jacobi and Evaluation of Jacobi
Gradient descent method
Natural gradient decent method
Advanced Numerical Computation 2008, AM NDHU

3. Advanced Numerical Computation 2008, AM NDHU
Nonlinear system
Advanced Numerical Computation 2008, AM NDHU

4. Advanced Numerical Computation 2008, AM NDHU
Nonlinear equations
Advanced Numerical Computation 2008, AM NDHU

5. Advanced Numerical Computation 2008, AM NDHU
x=linspace(-2,2);
plot(x,x); hold on;
plot(x,sqrt(4-x.^2))
plot(x,-sqrt(4-x.^2))
Advanced Numerical Computation 2008, AM NDHU

6. Advanced Numerical Computation 2008, AM NDHU
function F = myfun(x)
F(1) = x(1).^2 + x(2)^2-4;
F(2) = x(1) - x(2);
return
Advanced Numerical Computation 2008, AM NDHU

7. Advanced Numerical Computation 2008, AM NDHU
x=linspace(-3.5,3.5);
plot(x,sqrt(24-2*x.^2),'r');hold on;
plot(x,-sqrt(24-2*x.^2),'r')
x=linspace(-5,5);
plot(x,sqrt(12+x.^2),'b')
plot(x,-sqrt(12+x.^2),'b')
Advanced Numerical Computation 2008, AM NDHU

8. Advanced Numerical Computation 2008, AM NDHU
Demo_lsq_2c
source code
Advanced Numerical Computation 2008, AM NDHU

9. Advanced Numerical Computation 2008, AM NDHU

10. Advanced Numerical Computation 2008, AM NDHU
x=linspace(-3,3);
x=x(find(abs(x) > 0.1));
plot(x,sqrt((x.^2-2)/2),'r');hold on;
plot(x,-sqrt((x.^2-2)/2),'r')
x=linspace(0.5,3);
plot(x,2./x,'b');
x=linspace(-0.5,-3);
plot(x,2./x,'b')
Advanced Numerical Computation 2008, AM NDHU

11. Advanced Numerical Computation 2008, AM NDHU

12. Advanced Numerical Computation 2008, AM NDHU
Nonlinear systems
A system of nonlinear equations
Advanced Numerical Computation 2008, AM NDHU

13. Advanced Numerical Computation 2008, AM NDHU
Example
Advanced Numerical Computation 2008, AM NDHU

14. Advanced Numerical Computation 2008, AM NDHU
myfun
function F = myfun(x)
F(1) = 3*x(1)-cos(x(2)*x(3))-1/2;
F(2) = x(1).^2 -81*(x(2)+0.1).^2+sin(x(3))+1.06;
F(3) = exp(-x(1)*x(2))+20*x(3)+1/3*(10*pi-3);
return
Advanced Numerical Computation 2008, AM NDHU

15. Advanced Numerical Computation 2008, AM NDHU

16. Advanced Numerical Computation 2008, AM NDHU
lsqnonlin
x0= rand(1,3)-0.5;
x =
y=myfun(x);
sum(y.^2);
Advanced Numerical Computation 2008, AM NDHU

17. Advanced Numerical Computation 2008, AM NDHU
Demo_ex1
demo_ex1.m
Advanced Numerical Computation 2008, AM NDHU

18. Advanced Numerical Computation 2008, AM NDHU
Example
Advanced Numerical Computation 2008, AM NDHU

19. Advanced Numerical Computation 2008, AM NDHU
Neural dynamics
Advanced Numerical Computation 2008, AM NDHU

20. Advanced Numerical Computation 2008, AM NDHU
Newton’s method
Single valued function f
Find x s.t. f(x)=0
A system of nonlinear functions F
Find vector x s.t. F(x)=0
Advanced Numerical Computation 2008, AM NDHU

21. Newton’s method -Tangent line
y=f(x)
yn=f(xn) x
xn+1 xn
Advanced Numerical Computation 2008, AM NDHU

22. Advanced Numerical Computation 2008, AM NDHU
Updating rule
x : scalar
x : vector
Advanced Numerical Computation 2008, AM NDHU

23. Advanced Numerical Computation 2008, AM NDHU
Taylor series
Second order expansion at x=
Hessian matrix
Jacobi matrix
Advanced Numerical Computation 2008, AM NDHU

24. Advanced Numerical Computation 2008, AM NDHU
Hessian Matrix
Advanced Numerical Computation 2008, AM NDHU

25. Advanced Numerical Computation 2008, AM NDHU
Jacobi matrix
Advanced Numerical Computation 2008, AM NDHU

26. Advanced Numerical Computation 2008, AM NDHU
First order expansion
Advanced Numerical Computation 2008, AM NDHU

27. Advanced Numerical Computation 2008, AM NDHU
Newton’s method
Advanced Numerical Computation 2008, AM NDHU

28. Advanced Numerical Computation 2008, AM NDHU
Flow Chart
n=0; guess x0
~ halting
condition
Advanced Numerical Computation 2008, AM NDHU

29. MATLAB implementation
Newton method
Partial derivatives
Jacobi
Evaluation of an inline function that is with multiple dependent variables
Advanced Numerical Computation 2008, AM NDHU

30. Advanced Numerical Computation 2008, AM NDHU
Partial derivative
ss='3*x1-cos(x2*x3)-1/2';
s=sym(ss)
fx=inline(s);
x1=sym('x1');
x2=sym('x2');
x3=sym('x3');
diff(s,x1)
diff(s,x2)
diff(s,x3)
Advanced Numerical Computation 2008, AM NDHU

31. Advanced Numerical Computation 2008, AM NDHU
Partial derivative
demo_pdiff.m
Find partial derivatives with respect
to each of three dependent variables
for a given function,
f(x1,x2,x3)
Advanced Numerical Computation 2008, AM NDHU

32. Advanced Numerical Computation 2008, AM NDHU
Example
s='3*x1-cos(x2*x3)-1/2';
[f1 f2 f3]=demo_pdiff(s)
f1 =
Inline function:
fx1(x) = 3
f2 =
fx2(x2,x3) = sin(x2.*x3).*x3
f3 =
fx3(x2,x3) = sin(x2.*x3).*x2
Advanced Numerical Computation 2008, AM NDHU

33. Advanced Numerical Computation 2008, AM NDHU
Form Jacobi
s1='3*x1-cos(x2*x3)-1/2';
s2='x1.^2 -81*(x2+0.1).^2+sin(x3)+1.06';
s3='exp(-x1*x2)+20*x3+1/3*(10*pi-3)';
[f1 f2 f3]=demo_pdiff(s1);
[f4 f5 f6]=demo_pdiff(s2);
[f7 f8 f9]=demo_pdiff(s3);
Advanced Numerical Computation 2008, AM NDHU

34. Advanced Numerical Computation 2008, AM NDHU

35. Evaluation of an inline function
feva.m
function v=feva(f,x)
s=symvar(f);
xx=[];
for i=1:size(s,1)
ss=char(s(i));
switch ss
case 'x'
xx=[xx 0];
case 'x1'
xx=[xx x(1)];
case 'x2'
xx=[xx x(2)];
case 'x3'
xx=[xx x(3)];
end
v=f(xx);
return
Prepare an input vector, xx
Use symvar to get
dependent variables
Add instances of dependent
variables to xx
Substitute xx to f
Advanced Numerical Computation 2008, AM NDHU

36. Advanced Numerical Computation 2008, AM NDHU
feva
>> f1
f1 =
Inline function:
f1(x) = 3
>> v=feva(f1,[1 1 1])
v = 3
x=[1 pi 2*pi]
f2(x(2),x(3))
ans =
4.8811
>> feva(f2,x)
>> f3
f3 =
Inline function:
f3(x2,x3) = sin(x2.*x3).*x2
>> f3(x(2),x(3))
ans =
2.4406
>> feva(f3,x)
Advanced Numerical Computation 2008, AM NDHU

37. Advanced Numerical Computation 2008, AM NDHU
>> f4
f4 =
Inline function:
f4(x1) = 2.*x1
>> f4(x(1))
ans = 2
>> feva(f4,x)
>> f5
f5 =
Inline function:
f5(x2) = -162.*x2-81./5
>> f5(x(2))
ans =
>> feva(f5,x)
>> f6
f6 =
Inline function:
f6(x3) = cos(x3)
>> f6(x(3))
ans = 1
>> feva(f6,x)
Advanced Numerical Computation 2008, AM NDHU

38. Advanced Numerical Computation 2008, AM NDHU
>> f7
f7 =
Inline function:
f7(x1,x2) = -x2.*exp(-x1.*x2)
>> f7(x(1),x(2))
ans =
>> feva(f7,x)
>> f8
f8 =
Inline function:
f8(x1,x2) = -x1.*exp(-x1.*x2)
>> f8(x(1),x(2))
ans =
>> feva(f8,x)
Advanced Numerical Computation 2008, AM NDHU

39. Flow Chart Function x=Newton(s1,s2,s3,x0) Create 3 inlines functions,
~ halting
condition
Create 3 inlines functions,
F1,F2,F3 according to s1,s2,s3
Create f1,…,f9 that represent
partial derivatives of F1,F2 and F3
Substitute x to f1-f9 to form J
Substitute x to F1-F3
Refine x by
x=x0;n=0;
Advanced Numerical Computation 2008, AM NDHU

40. Advanced Numerical Computation 2008, AM NDHU
Example
s1='(x1).^2+x2.^2+x3.^2-4';
s2='2*x1-x2+x3-1';
s3='x1+3*x2-x3-3';
x_ini=rand(3,1)*2-1;
x=Newton(s1,s2,s3,x_ini) '
F1=inline(s1);F2=inline(s2);F3=inline(s3);
er=F1(x(1),x(2),x(3)).^2+F2(x(1),x(2),x(3)).^2+F3(x(1),x(2),x(3)).^2;
fprintf('%f %f %f mse = %f\n',x',er);
for i=1:5
Exit
Advanced Numerical Computation 2008, AM NDHU

41. Advanced Numerical Computation 2008, AM NDHU
mse =
mse =
Advanced Numerical Computation 2008, AM NDHU

42. Function x=Newton(s1,s2,s3,x0)
% s1, s2 and s3 are strings, each representing a nonlinear function of x1, x2 and x3
% x0 denotes an initial guess
Use s1, s2 and s3 to create three inline functions, F1-F3
Create nine inline functions, f1-f9, that represent all partial derivatives of F1-F3
Set x to x0
While ~ (halting condition)
Substitute x to f1-f9 to create a 3x3 matrix, J
Substitute x to F1-F3
Refine x by the updating rule of the Newton’s method
Advanced Numerical Computation 2008, AM NDHU

43. Advanced Numerical Computation 2008, AM NDHU
Halting condition
The sum of abs(F1(x)), abs(F2(x)) and abs(F3(x)) is less than a pre-determined threshold
Advanced Numerical Computation 2008, AM NDHU

44. Advanced Numerical Computation 2008, AM NDHU
Exercise due to 27 Oct
Implement the Newton’s method for solving a three-variable nonlinear system
Test your matlab codes with the following nonlinear system
Advanced Numerical Computation 2008, AM NDHU

45. Unconstrained optimization
A target function, y = g(x), where x  Rd
A sample from the surface of f collects paired data
S={(xi ,yi)}i
Let G(x; ) denote an approximating function to g
Minimizing the mean square approximating error induces an unconstrained optimization
Advanced Numerical Computation 2008, AM NDHU

46. Mean square approximating error
Advanced Numerical Computation 2008, AM NDHU

47. Advanced Numerical Computation 2008, AM NDHU

48. Advanced Numerical Computation 2008, AM NDHU
Minima
dE/d = 0
Local minima : unsatisfied approximation
Global minima : reliable and effective approximation
Advanced Numerical Computation 2008, AM NDHU

49. Gradient-based approaches
The gradient descent method
The NG (Newton-Gauss) method
The Levenberg-Marquardt method
Advanced Numerical Computation 2008, AM NDHU

50. Unconstrained Optimization
Translate solving nonlinear systems to a task of unconstrained optimization
Advanced Numerical Computation 2008, AM NDHU

51. Advanced Numerical Computation 2008, AM NDHU
Square errors
is translated to minimize
Advanced Numerical Computation 2008, AM NDHU

52. Unconstrained Optimization
Advanced Numerical Computation 2008, AM NDHU

53. Advanced Numerical Computation 2008, AM NDHU
Iterative approaches
Gradient method
Natural Gradient method
Steepest gradient method
Conjugate gradient method
Advanced Numerical Computation 2008, AM NDHU

54. Advanced Numerical Computation 2008, AM NDHU
Gradient
Advanced Numerical Computation 2008, AM NDHU

55. Advanced Numerical Computation 2008, AM NDHU
An Iterative approach
An iterative approach based on the gradient descent method
Advanced Numerical Computation 2008, AM NDHU

56. Advanced Numerical Computation 2008, AM NDHU
Flow Chart
n=0; guess x0
~ halting
condition
Advanced Numerical Computation 2008, AM NDHU

57. Advanced Numerical Computation 2008, AM NDHU
Flow Chart
n=0; guess x0
~ halting
condition
Advanced Numerical Computation 2008, AM NDHU

58. Advanced Numerical Computation 2008, AM NDHU
Equivalent Flow Chart
n=0; guess x0
~ halting
condition T
Advanced Numerical Computation 2008, AM NDHU

59. Advanced Numerical Computation 2008, AM NDHU

60. Advanced Numerical Computation 2008, AM NDHU
Comparison
Newton’s method
Gradient descent method
Advanced Numerical Computation 2008, AM NDHU

61. Natural Gradient Descent method
Amari 1998 (Neural Computation)
Wu, Chen & Lin
Natural Gradient Method
Advanced Numerical Computation 2008, AM NDHU

62. Steepest gradient descent (SGD)
Choose to minimize the cost of the upcoming state
Advanced Numerical Computation 2008, AM NDHU

63. Advanced Numerical Computation 2008, AM NDHU
Flow Chart
n=0; guess x0
~ halting
condition
EXIT T
Advanced Numerical Computation 2008, AM NDHU

64. Advanced Numerical Computation 2008, AM NDHU
Quadratic polynomial
Original nonlinear function of
Approximate h by a quadratic polynomial determined by
Advanced Numerical Computation 2008, AM NDHU

65. Advanced Numerical Computation 2008, AM NDHU
Quadratic polynomial
Advanced Numerical Computation 2008, AM NDHU

66. Quadratic Polynomial fitting
The polynomial that passes (0,f(0)), (0.5,f(0.5)), (1,f(1))
can be determined by
f=inline('3*x.^2+2*x-1');
x=[ ];
y(1)=f(x(1));y(2)=f(x(2));y(3)=f(x(3));
p=polyfit(x,y,2)
x_min=-p(2)/(2*p(1))
Advanced Numerical Computation 2008, AM NDHU

67. Advanced Numerical Computation 2008, AM NDHU

68. Advanced Numerical Computation 2008, AM NDHU

69. Flow Chart Function x=SGD(s1,s2,s3,x0) Create 3 inlines functions,
~ halting
condition
Create 3 inlines functions,
F1,F2,F3 according to s1,s2,s3
Create f1,…,f9 that represent
partial derivatives of F1,F2 and F3
Substitute x to f1-f9 to form J
Substitute x to F1-F3
Find *
x=x0;n=0;
Advanced Numerical Computation 2008, AM NDHU

70. Advanced Numerical Computation 2008, AM NDHU
Exercise
Implement the gradient descent method for solving a three-variable nonlinear system
Test your matlab codes with the following nonlinear system
Advanced Numerical Computation 2008, AM NDHU

71. Advanced Numerical Computation 2008, AM NDHU
Exercise
Implement the natural gradient descent method for solving a three-variable nonlinear system
Test your matlab codes with the following nonlinear system
Advanced Numerical Computation 2008, AM NDHU

72. Advanced Numerical Computation 2008, AM NDHU
Exercise
Implement the steepest gradient descent method for solving a three-variable nonlinear system
Test your matlab codes with the following nonlinear system
Advanced Numerical Computation 2008, AM NDHU