1. Goal
We present a hybrid optimization approach for solving global optimization problems, in particular automated parameter estimation models.
The hybrid approach is based on the coupling of the Simultaneous Perturbation Stochastic Approximation (SPSA) and a Newton-Krylov Interior-Point method (NKIP) via a surrogate model.

2. Problem Formulation
We consider the global optimization problem in the form:
where the global solution x* is such that
We are interested in problem (1) that have many local minima.

3. Types of local minima
local minima
global minimum x

4. Hybrid Optimization Framework
Global Method: SPSA
Surrogate Model
Local Method: NKIP

5. Global Method: SPSA Stochastic steepest descent direction algorithm
(James Spall, 1998)
Advantage
SPSA gives region(s) where the function value is low, and this allows to conjecture in which region(s) is the global solution.
Disadvantages
Slow method
Do not take into account equality/inequality constraints
It can be clasifieda as …
Given and initial point a ssdd is calculated

6. Global Search: SPSA
SPSA performs random simultaneous perturbations of all model parameters to generate a descent direction at each iteration
This process may be performed by starting with different initial guesses (multistart).
Multistart increases the chances for finding a global solution, and yields to find a vast sampling of the parameter space.

7. Hybrid Optimization Scheme (1)
Explore Parameter space
Multistart
(x0)1
(x0)2 .
(x0)k
Global Search
Via SPSA .

8. SPSA: Pseudocode gama = .1/6 alfa=1 one typically finds that in a
high-noise setting (Le., poor quality measurements
of L(8)) it is necessary to pick a smaller a and
larger c than in a low-noise setting.

9. Observations
SPSA gives region(s) where the function value is low, and this allows to conjecture in which region(s) is the global solution.
This give us a motivation to apply a local method in the region(s) found by SPSA.
SPSA gives a good approximation to the global minimum but further refinement can be achieved by a local method.
We obtain by SPSA information about the behavior of the function on a wider range.
Solutions may not satisfied general inequalities constraints.
This give us a motivation to the next step. Apply local optimization in the region(s) found by SPSA.

10. Local Method Advantages Disadvantage Solution
Fast Method: Newton Type Methods
Interior-Point Methods allow to add equality/inequality constraints
Disadvantage
Needs first/second order information
Solution
Construct a Surrogate Model using the SPSA function values inside the conjecture region(s)
Then from such surrogate model we will able to evaluate first a second order inform…

11. Surrogate Model
A surrogate model is created by using an interpolation method with the data, , provided by SPSA.
This can be performed in different ways, e.g., radial basis functions, kriging, regression analysis, or using artificial neural networks.

12. Why Surrogate Models?
Most real problems require thousands or millions of objective and constraint function evaluations, and often the associated high cost and time requirements render this infeasible.
Most problems require experiments and/or simulations to evaluate objective and constraint functions as in terms of design variables.
Frequently, optimization strategies requires thousands or even millions of evaluations, and often the associated high cost and time requirements render this infeasible.

13. Radial Basis Function (RBF)
RBF is typically parameterized by two sets of parameters: the center c which defines its position, and shape r that determines its width or form
An RBF interpolation algorithm (Orr,1996) characterizes the uncertainty parameters:

14. Surrogate Model Our goal is to optimize the surrogate function
Where the radial basis functions can be defined as:
that are the multiquadric and the Gaussian basis functions, respectively

15. Explore Parameter space
Hybrid Optimization Scheme (2)
Explore Parameter space
Multistart
(x0)1
(x0)2 .
(x0)k
Global Search
Via SPSA .
Target Region
Filtering +
Sampling
Then from such surrogate model we will able to evaluate first a second order inform…
Surrogate Model

16. Surrogate Model
We plot the original model function and the surrogate function:

17. Local Search: NKIP
NKIP is a globalized and derivative dependent optimization method based on the global strategy introduced by Miguel Argaéz and Richard Tapia in 2002.
This method calculates the directions using the conjugate gradient algorithm, and a linesearch is implemented to guarantee a sufficient decrease of the objective function.

18. Local Search: NKIP We consider the optimization problem in the form:
where a and b are determined by the sampled points given by SPSA.

19. Hybrid Optimization Scheme (3)