1. University of Colorado Boulder APPM 4380 October 10th, 2016
Tomographic Reconstruction – Optimization Toolbox and Global Optimization
University of Colorado Boulder
APPM 4380
October 10th, 2016

2. Outline MATLAB’s Optimization Toolbox Global Optimization
Fsolve Implementation
Improving Code
Global Optimization
Genetic Algorithm
Simulated Annealing
Direct Search
Particle Swarm

3. MATLAB’s Optimization Toolbox
Objective Type
Solvers
Function Type
Scalar
fmincon, fminunc, fminbd, fminsearch, fseminf, fzero
Scalar Objective Function
Nonlinear Least Squares
lsqcurvefit, lsqnonlin
Vector/Matrix Objective Function
Multivariable Equation Solving
fsolve
Multi-objective
fgoalattain, fminimax
Linear Programming
linprog
Linear/Quadratic Problems
Mixed-integer Linear Programming
intlinprog
Linear Least Squares
lsqlin, lsqnonneg
Quadratic Programming
quadprog

4. When to Use MATLAB’s Optimizers
Objective Type
Linear
Quadratic
Least Squares
Smooth nonlinear
Non-smooth
None
n/a
quadprog
mldivide, lsqcurvefit, lsqnonlin
fminsearch, fminunc
fminsearch
Bound
linprog
lsqcurvefit, lsqlin, lsqnonlin, lsqnonneg
fminbnd, fmincon, fseminf
fminbnd
lsqlin
fmincon, fseminf
General smooth
fmincon
Discrete
intlinprog
Constraint Type

5. MATLAB’s fsolve Function
Nonlinear system solver
Problem specified by 𝐹 𝑥 =0
𝑥 can be scalar, vector, or matrix
Generally – minimize the sum of the squares of the components
Three main algorithms:
Trust region
Trust region dogleg
Levenberg-Marquardt
Trust region - min 𝑠 𝑇 𝐻𝑠+ 𝑠 𝑇 𝑔 𝑠𝑢𝑐ℎ 𝑡ℎ𝑎𝑡 𝐷𝑠 ≤Δ
Trust region dogleg - 𝐽 𝑥 𝑘 𝑑 𝑘 =−𝐹 𝑥 𝑘 , 𝑥 𝑘+1 = 𝑥 𝑘 + 𝑑 𝑘
Levenberg-Marquardt - 𝐽 𝑥 𝑘 𝑇 𝐽 𝑥 𝑘 + 𝜆 𝑘 𝐼 𝑑 𝑘 =−𝐽 𝑥 𝑘 𝑇 𝐹 𝑥 𝑘

6. Implementation Scan any guess object and create scan-data
Take difference between scan-data and true scan-data
Minimize the difference by changing the guess object
Several developments of the code
Problem setup: abs (~5.7 s) versus norm (~43.4 s)
Saving the true scan-data as a variable (7.891 s)
Making the true scan-data a global variable (5.816 s)
Using nested functions to keep true-scan data in memory (5.698 s)

7. Original Image 21x21 matrix C = value of 1.0 U = value of 0.5
White = 0

8. Improving the Code – Save Variable
21x21 Image
Guess: zeros
# of rays: 20
# of angles: 23
Loading variable took 1/3 the time as the scan function

9. Improving the Code – Global Variable
21x21 Image
Guess: zeros
# of rays: 20
# of angles: 23
Global variable is barely noticeable – 0.03 secs

10. Improving the Code – Nested Function
21x21 Image
Guess: zeros
# of rays: 20
# of angles: 23
Calls to ndgrid and meshgrid go down

11. Changing the Scan Size Reconstruction # of rays: 20 # of angles: 20
Max error:
Iterations: 4
Reconstruction
# of rays: 20
# of angles: 23
Max error:
Iterations: 6
Reconstruction
# of rays: 20
# of angles: 25
Max error:
Iterations: 16

12. Changing the Scan Size (Error Plots)
# of rays: 20
# of angles: 20
Max error:
# of rays: 20
# of angles: 23
Max error:
# of rays: 20
# of angles: 25
Max error:

13. Changing the Initial Guess Object
Guess: rand
# of rays: 20
# of angles: 23
Max error:
Iterations: 6
Guess: ones
# of rays: 20
# of angles: 23
Max error:
Iterations: 8
Guess: zeros
# of rays: 20
# of angles: 23
Max error:
Iterations: 6

14. 63x63 Original Image Original Image 63x63 matrix Reconstruction
# of rays: 62
# of angles: 67
Run time: 51 mins
Error Plot
Max Error:
Iterations: 22

15. Global Optimization: Direct Search
Function call: Pattern Search
This Method “Polls” the values around the current point and determines the direction that will either further maximize or minimize the objective function
Pattern Search terminology:
A pattern vector is a set of vectors that the direct search method uses to generate new values to probe in the space around it. The set of pattern vectors is defined to be twice the number of free variables in the objective function. The set of these pattern vectors is called a mesh.
Basic Idea:
For each Possible Direction, create all linear combination of the the current position, and each pattern multiplied by the size of the mesh to obtain a new mesh. Each of these new patterns is then polled using the objective function until a more optimal position then the current is found. The current position is then replaced by the new one, and the process is repeated until convergence.
Expanding and Contracting:
After the Polling process is completed, the mesh size is then is either expanded, or contracted depending on whether or not the poll was successful.

16. Benefits of Pattern Search
Robustness of Pattern Search:
Because pattern search does not really on derivative information to traverse the space defined by the objective function.; it tolerates function evaluations resulting in NaN, Inf, or complex values.
Global optimization:
When the objective function at the initial point x0 is a real, finite value, pattern search treats polls point failures as if the objective function values are large, and ignores them and contracts. Whenever the pattern search finds a point that is ”better” then any previously seen, it will automatically count the element as a successful poll and expand. This behavior allows for ”intelligent” traversal of the space, and a large class of algorithms has been developed around this basic idea.
Relatively Quick:
While it will take larger amount of function evaluations then a combined gradient decent / quasi newton method, it will search more troughs and achieve better results while still having drastically evaluations then other optimization techniques.

17. A Comparison:
Consider the functions: y1 = | cos(x(1)^2)+1*x(1)*x(2) +9|
y2 = | cos(x(2)^2)+1*x(1)*x(2) +9|

18. Global Optimization: Simulated Annealing
Basic Idea:
In regular optimization procedures, the algorithm will choose the best possible direction, or set of directions to move in. In Simulated annealing, to ensure that a comparable amount of the space is traversed, the algorithm will with some non-trivial probability pick a “worse” direction.
Annealing:
Like the annealing of metal and other structures, as time goes on, energy dissipates from the system. In this algorithm, the probability of taking a non optimal step is (generally) described by the Boltzmann distribution, which over time decreases as time passes in the simulation.

19. Another Comparison: De Jong's fifth function

20. Visualizations:

21. Tomographic reconstruction:
The Importance of a multi-variable objective functions in global optimization:
Unfortunately both pattern search and simulated annealing do not have multi objective optimization options currently! By using the objective functions 2 norm, we may navigate this problem for smaller images, however any image larger then 3x3 is no longer guaranteed convergence.
Original
Reconstructed

22. Further Extensions: Algorithm Combinations to increase robustness:
GODLIKE (Global Optimum Determination by Linking and Interchanging Kindred Evaluators); combines components of genetic algorithms, particle swarm and adaptive simulated annealing to increase the likely hood of finding a global minimum.
Neldar-Mead Algorithm: (also called an amoeba method)
A linear programming method, where the use of a of a derivative free approximation in the calculation of movement follows very closely the convergence of pattern search methods. Extensions of this simple linear programming method have been widely used since its inception in 1965.
Momentum Accelerated Stochastic Gradient Decent:
This method is often used to search a space and find and area near the global minimum. After the area is found, most adaptive methods will switch to a quasi newton iteration to speed up convergence within the trough.

23. Nelder-Mead amoeba method
SGD (with momentum)

24. Genetic Algorithm Essentially, randomized brute-force search
More of a heuristic than an algorithm in it of itself

25. Sample run

26. Iteration 1

28. Intuition behind “information”
Our error for our guess image is:
vs.

29. Intuition behind “information”
The error for our guess image is:
542.00
vs. ga
gamultiobj

30. … forever time later

31. Particle swarm

33. Pros & Cons
For convex optimization problems, it’s useless. Use least squares instead.
For nonconvex problems, more likely to find a global maximum