1. The New Faces of Nonlinear Programming
Jorge Nocedal
Optimization Technology Center
Argonne-Northwestern
.I wil provide a perspective of the whole development of the algorithm, the things we have learned, the software package and of the major developments that are about to occur. Th work started backward with a convegence analysis that would guarantee that the iterates could not crash at the boundary, that the new Cauchy point concepts worked, etc The paper of 2000 with Gilbert provided the foundation, the paper of 1999 with Hribar gives the IP algo and 1997 gives a superlinear analysis which was not tired until last week. The first release with in 2000 after the
MP meeting and Knitro 2.0 is the same algorithm – just a good software package. Improvements are going to
Be made very often in the next few months.

2. New Problems, New Algorithms, New Software
Traditional Applications: solve larger problems, more robustness
New classes of applications
Advances in modeling languages: AMPL, …
Automatic differentiation
Interior Methods
Test problems (CUTE, COPS)
New packages: LOQO, KNITRO,.
Internet Optimization: NEOS server
A confluence of factors have contributed to the rapid progress in nonlinear programming
9/22/2018
McMaster

3. NEOS Server Argonne Northwestern MINOS. SNOPT, FILTER, LANCELOT
USER
MINOS. SNOPT, FILTER, LANCELOT
LOQO, KNITRO
9/22/2018
McMaster

4. Semi-infinite Optimization Mixed Integer Nonlinearly Constrained Optimization Mixed Integer Linear Programming Nonlinearly Constrained Optimization Semidefinite & Second Order Cone Programming Linear Programming Unconstrained Optimization Linear Network Optimization Complementarity Problems Nondifferentiable Optimization Stochastic Linear Programming Global Optimization Application-specific Optimization
9/22/2018
McMaster

5. Part I: New Classes of Problems
Instead of new algorithms/software/ adapt existing techniques
Equilibrium constraints: (T. Luo)
Bi-level programming
Complementarity constraints
Semi-definite programming (??)
PDE-constrained optimization
Differential algebraic systems
9/22/2018
McMaster

6. Nonlinear Optimization Formulation
Theme: constraints involve a difficult computation/simulation.
Limitations of this formulation?
Logic constraints
9/22/2018
McMaster

7. Equilibrium Constraints
Structurally difficult
No strictly feasible direction
Algorithmically… y x
9/22/2018
McMaster

8. Confirmed by experimental evidence (??)
Optimization problem with equilibrium constraints not stable  cannot apply NLP algorithms to it
Confirmed by experimental evidence (??)
Reality: software not capable of dealing with degeneracy, not sufficiently robust
Theoretical mistake: lack of stability does not imply practical problems. Structural degeneracy.
Active Set SQP (Leyffer et al)
Interior Methods: solve perturbed problem
9/22/2018
McMaster

9. Traffic Assignment
For each origin-destination pair (o,d) we have:
qod: demand (in terms of flow) between o and d
K : index set of paths from o to d
fk : flow along path k, for each k in K
ck(f): cost of travel along path k (usually time), for each path k in K
λ = λ(qod) : minimum possible travel cost between o and d
origin
destination
Vector x of link flows,
Efficiency parameters (capacity, speed limit) given at link level
The path flows and costs are aggregated (based on x) through adjacency matrix A
Need constraints for demand satisfaction and conservation of flow
Many origin-destination pairs may exists in the network
9/22/2018
McMaster

10. Network Design (Continuous Equilibrium)
origin
destination
Improvements to network: Traffic Network Design
Discrete: add lanes, links
Continuous: link capacity expansion
Boyce (1979) ed,
Continuous capacity?
Complex interaction between System Optimal and User Equilibrium is recognized -> bilevel programming (e.g. Abdulaal-LeBlanc)
One important problem in transportation is the determination of improvements to an existing network. The problem is known as the Traffic Network Design
9/22/2018
McMaster

11. Example of Continuous Equilibirium Network Design
Given network: G=(N,A)
Find additions yi to capacities ci of links i in A
So that:Cost of improvement and efficiency of network is minimized
origin
destination
An equlibrium flow
One important problem in transportation is the determination of improvements to an existing network. The problem is known as the Traffic Network Design
9/22/2018
McMaster

12. KKT Conditions path k is a minimum cost path (ck = λ)
Together with demand satisfaction and conservation of flow,
we need to demand EQUILIBRIUM, which in this case looks like:
λ = λ(qod) : minimum possible travel cost between o and d
If there is flow on path k (fk > 0):
path k is a minimum cost path (ck = λ)
If path k is relatively expensive (ck > λ):
no one uses this path (fk = 0)
9/22/2018
McMaster

13. Partial Differential Equations and Optimization (Tsai, Byrd,N)
Navier-Stokes equations
Desired
flow
Mems flaps
Determine position of mems flaps to optimize
Quality of exhaust flow
Phase II: boundary control
9/22/2018
McMaster

14. Partial Differential Equations (PDEs)
Systems that evolve in space (several dimensions) and time are described by PDEs
Solution: function u(x,t) – infinite-dimen prob
More space dimen.: great computational and storage cost
9/22/2018
McMaster

15. Success of PDE simulations
3D Large Eddy Simulation around an airfoil
9/22/2018
McMaster

16. Solution of nonlinear PDEs
Fluid flow described by Navier-Stokes
Solution of nonlinear PDEs
Newton-Krylov
Sequence of meshes, Krylov (FGMRES)-(full)
multigrid (Krylov smoother).
Parallel computing to obtain high resolution
9/22/2018
McMaster

17. Now optimize!
Robustness of PDE solvers: millions of variables, hundreds of processors, multiple physical interactions
Introduce free parameters
Finite-dimensional formulation
system of PDEs
9/22/2018
McMaster

18. State-of-the-art algorithms
KKT system Newton-Lagrange
Active set: SNOPT, FilterSQP factor subset of A, reduced Hessian
Interior: LOQO, KNITRO factor
Algorithms must accept iterative solution of constraint linearization. Av A’v
9/22/2018
McMaster

19. Unconstrained reformulation
Linearize constraints
Eliminate state variables xs (basic)
Minimize w.r.t. controls xd (non-bas)
New problem
Modern optimization SQP:
9/22/2018
McMaster

20. Weather Forecasting - Oceanography
= state of atmosphere,
Observations: Time windows i:
length = a few time steps
Short integration: from initial condition
Problem: unknown
Given x_0 compute x1, x2
9/22/2018
McMaster

21. Nonlinear Least Squares Problem
Background field
Observations
Background covar
Obs covar
Time
Constraints eliminated, no bounds, inequalities
3 Spaces: grid point, spectral, observation
X-b is typically the results of the previous analysis or a short term integration
9/22/2018
McMaster

22. Part III: New Algorithms
New applications
New methods
New software
New tools (modeling languages, automatic differentaition)
9/22/2018
McMaster

23. Part III Advances in NLP Algorithms:Active Set SQP
Before 1998:
Active Set SQP software: highly complex
Many dense, substandard versions
Quasi-Newton (SNOPT, MINOS)
Present:
Filter, Second derivatives (FilterSQP)
SNOPT second derivatives in progress
Can SQP compete with Interior Methods?
Future:
Linear Programming Based (Dundee, Northwestern.)
9/22/2018
McMaster

24. Interior Methods Terlaky
Newton’s method to KKT conditions of equal-problem:
Reformulate to avoid rational functions: primal-dual
Backtrack (difficulties!)
Update barrier parameter
Initial point strategy-failures
Our interior method is infeasible and makes use of slacks. Slacks have proved to be very useful: they have allowed to us to introduce a feasible version in the trust region approach by making just a minor modification; they have proved to be very useful when solving mpecs. Rather than directly solving the KKT conditions of the barrier problem we model it by a QP using trust regions. Note the curious scaling for the step in the slacks. The method is essentially primal in that the multipliers are computed at a function of x. We will return to the effect of doing this later on.
9/22/2018
McMaster

25. Nonlinear Interior Methods
Approach I: LOQO,OPINEL,BOEING,IPOPT,…
Modify W,
Merit Function/Filter
Approach II: (KNITRO)
Our interior method is infeasible and makes use of slacks. Slacks have proved to be very useful: they have allowed to us to introduce a feasible version in the trust region approach by making just a minor modification; they have proved to be very useful when solving mpecs. Rather than directly solving the KKT conditions of the barrier problem we model it by a QP using trust regions. Note the curious scaling for the step in the slacks. The method is essentially primal in that the multipliers are computed at a function of x. We will return to the effect of doing this later on.
Eliminate constraints-null space approach
Approximate solution
Iterative solution by conjugate gradients
Merit function
enforcement
9/22/2018
McMaster

26. The Future (summer 2003) KNITRO 3.0 Active Interior Iterative Direct
The next major release of Knitro will provide both active-set and interior point options. The two
Algorithms will be closely integrated. At present the iterative and direct versions are considered on an
Equal footing.
Iterative
Direct
9/22/2018
McMaster

27. LP-EQP based on a Penalty Approach
Working set W
Equality constrained quadratic program
L-1 penalty
The active set algorithm is built on the slp-eqp idea of Fletcher and Sainz de la Maza. See also
Chin and Fletcher. The algorithm has two phases. First a linear program is solved to generate and estimate of the optimal active set. In the second phase we attempt to achieve optimality on that active set by minimizing a quadratic model over that active set. Is it a quadratic model of the Lagrangian? No, it is a model of the L-1 penalty function. Therefore contrary to the current trend we use a penalty function (not a filter) not only to determine if the step is acceptable – but for generating the actual search directions!
EQP
9/22/2018
McMaster

28. Step computation dc: min quadratic dLP deqp d dc xk model of merit
function: Cauchy point
Dogleg approach
EQP by projected CG
dLP
deqp d dc
The second innovation is to exploit the information provided by the two phases as well as possible. A dogleg approach explores the set of directions generated in the LP and EQP phases xk
9/22/2018
McMaster

29. Rationale for Integration
Active set approach: LP + EQP (Fletcher)
shares EQP solution with Interior Algor.
preconditioning
Final active set identification
Warm starts
Share interfaces, stop tests, testing, object c.
Both trust region methods
New algorithms…
The integration will also allow us to develop new algorithms for nlp that lie between interior and active set
Methods. These algorithms are still in the drawing phase.
9/22/2018
McMaster

30. Remarks on: New Software—Interior Methods
Change of barrier parameter
Guiding principles (increase/decrease)
Scale invariance, initial point
Global convergence
When to attempt superlinear convergence
Seemingly superior to active set SQP codes
Only choice for very large (reduced space)
Versatile
1st derivs only; do/not factor Hessian
Iterative vs direct solvers
Feasible/infeasible modes
LOQO, KNITRO
9/22/2018
McMaster

31. Tests Sets: CUTE (850 problems) COPS
9/22/2018
McMaster

32. On many practical applications…
   
   
   

33. Final Remarks
Extension of core NLP algorithms/sofware instead of special-purpose methods
Investigation of limitations: degeneracies, multilevels
Robustness (fundamental algorithmic)
More iterative options
How far can NLP methods scale up?
9/22/2018
McMaster

34. L1 Merit Function L1 linear program and penalty parameter selection
Try to find minimum s.t. residuals = zero
W= constraints with zero residuals
Can achieve robustness and efficiency
One of the innovations is an automatic procedure for selecting the penalty parameter. This is done during the
Linear programming phase. First of all to ensure that the constraints are compatible we use an L-1 relaxation of the LP.
9/22/2018
McMaster

35. Interior Method- Direct
Solve primal-dual system
Retain trust region
Revert to Interior Iterative
Step is too long
Inertia is not correct
Step is rejected by merit function
Adaptive barrier parameter (Morales, Orban) dn
I will not address here the direct solution of the augmented system or a condensed form (normal equations). Roland Freund has spoken about this. A question about this approach is how to take advantage of predictor corrector strategies. Instead we will consider the reduced approach. In one implementation this gives rise to a hybrid direct-iterative method. An advantage of this approach is that second derivatives need not be computed explicitly
Slack bound
9/22/2018
McMaster

36. Interior Method (Iterative, CG)
Infeasible
slacks
Projected CG:
= barrier function
Our interior method is infeasible and makes use of slacks. Slacks have proved to be very useful: they have allowed to us to introduce a feasible version in the trust region approach by making just a minor modification; they have proved to be very useful when solving mpecs. Rather than directly solving the KKT conditions of the barrier problem we model it by a QP using trust regions. Note the curious scaling for the step in the slacks. The method is essentially primal in that the multipliers are computed at a function of x. We will return to the effect of doing this later on.
Primal method, no multipliers
9/22/2018
McMaster