1. CS B553: Algorithms for Optimization and Learning
Root finding

2. g(x) x
Roots of g

3. Key Ideas Newton’s method Secant method Superlinear convergence rates
Initialization and termination
Approximate differentiation
Numerical considerations

4. Newton’s method Figure 10 g(x) x0 x
In a neighborhood of a root, the line tangent to the graph crosses the x axis near the root

5. Newton’s method Figure 10 g(x) x1 x
In a neighborhood of a root, the line tangent to the graph crosses the x axis near the root… iterate!

6. Newton’s method Figure 10 g(x) x2 x
In a neighborhood of a root, the line tangent to the graph crosses the x axis near the root… iterate!

7. Figure 11
Divergence x1 x
g(x)

8. Figure 11
Divergence x1 x2 x
g(x)

9. Figure 11
Divergence x3 x1 x2 x
g(x)

10. Figure 11
Divergence x3 x1 x2 x4 x
g(x)

11. Figure 11
Divergence x5 x3 x1 x2 x4 x
g(x)

12. Figure 12
Oscillation x

13. Figure 12
Oscillation x

14. Figure 12
Oscillation x

15. Figure 12
Oscillation x

16. Secant method Figure 13 g(x) x0 x1 x
Idea: Use line through two points on graph as approximation of the derivative

17. Secant method Figure 13 g(x) x0 x1 x2 x
Idea: Use line through two points on graph as approximation of the derivative

18. Secant method Figure 13 g(x) x3 x0 x1 x2 x
Idea: Use line through two points on graph as approximation of the derivative

19. Secant method Figure 13 g(x) x3 x0 x1 x2 x
Idea: Use line through two points on graph as approximation of the derivative

20. Orders of convergence Bisection: linear Newton’s method: quadratic
Secant method: order   1.6
Only bisection has guaranteed convergence (given appropriate initial interval)
Newton’s method needs derivatives
Most “out of the box” subroutines take a hybrid approach

21. Basins of attraction in complex plane: x5-1=0
Figure 14
Basins of attraction in complex plane: x5-1=0