1. A comparison between PROC NLP and PROC OPTMODEL Optimization Algorithm Chin Hwa Tan December 3, 2008

2. Overview: NLP Procedure The NLP procedure (NonLinear Programming) offers a set of optimization techniques for minimizing or maximizing a continuous nonlinear function f(x) of n decision variables, x = (x 1, …, x n ) T with lower and upper bound, linear and nonlinear, equality and inequality constraints. This can be expressed as solving where f is the objective function, the c i 's are the nonlinear functions, and the l i 's and u i 's are the lower and upper bounds. Problems of this type are found in many settings ranging from optimal control to maximum likelihood estimation. The NLP procedure provides a number of algorithms for solving this problem that take advantage of special structure on the objective function and constraints. Two example are the quadratic programming problem and the least-squares problem.

3. Overview (cont.) The following problems are handled by PROC NLP: – quadratic programming with an option for sparse problems – unconstrained minimization/maximization – constrained minimization/maximization – linear complementarity problem The following optimization techniques are supported in PROC NLP: – Quadratic Active Set Technique – Trust Region Method – Newton-Raphson Method with Line Search – Newton-Raphson Method with Ridging – Quasi-Newton Methods – Double Dogleg Method – Conjugate Gradient Methods – Nelder-Mead Simplex Method – Levenberg-Marquardt Method – Hybrid Quasi-Newton Methods

4. Choosing an Optimization Algorithm The factors that go into choosing a particular optimizer for a particular problem are complex and may involve trial and error. Several things must be taken into account. – The structure of the problem has to be considered: Is it quadratic? least-squares? Does it have linear or nonlinear constraints? – It is important to consider the type of derivatives of the objective function and the constraints that are needed and whether these are analytically tractable or not.

5. Choosing an Optimization Algorithm (cont.) The following provides some guidance for matching an algorithm to a particular problem. – Quadratic Programming QUADAS LICOMP – General Nonlinear Optimization Nonlinear Constraints – Small Problems: NMSIMP Not suitable for highly nonlinear problems or for problems with n > 20. – Medium Problems: QUANEW Only Linear Constraints – Small Problems: TRUREG (NEWRAP, NRRIDG) ( n ≤ 40 ) where the Hessian matrix is not expensive to compute. Sometimes NRRIDG can be faster than TRUREG, but TRUREG can be more stable. NRRIDG needs only one matrix with n(n+1)/2 double words; TRUREG and NEWRAP need two such matrices. – Medium Problems: QUANEW (DBLDOG) ( n ≤ 200 ) where the objective function and the gradient are much faster to evaluate than the Hessian. QUANEW and DBLDOG in general need more iterations than TRUREG, NRRIDG, and NEWRAP, but each iteration can be much faster. QUANEW and DBLDOG need only the gradient to update an approximate Hessian. QUANEW and DBLDOG need slightly less memory than TRUREG or NEWRAP (essentially one matrix with n(n+1)/2 double words). – Large Problems: CONGRA ( n > 200 ) where the objective function and the gradient can be computed much faster than the Hessian and where too much memory is needed to store the (approximate) Hessian. CONGRA in general needs more iterations than QUANEW or DBLDOG, but each iteration can be much faster. Since CONGRA needs only a factor of n double-word memory, many large applications of PROC NLP can be solved only by CONGRA. – No Derivatives: NMSIMP ( n ≤ 20 ) where derivatives are not continuous or are very difficult to compute. – Least-Squares Minimization Small Problems: LEVMAR (HYQUAN) ( n ≤ 60 ) where the crossproduct Jacobian matrix is inexpensive to compute. In general, LEVMAR is more reliable, but there are problems with high residuals where HYQUAN can be faster than LEVMAR. Medium Problems: QUANEW (DBLDOG) ( n ≤ 200 ) where the objective function and the gradient are much faster to evaluate than the crossproduct Jacobian. QUANEW and DBLDOG in general need more iterations than LEVMAR or HYQUAN, but each iteration can be much faster. Large Problems: CONGRA No Derivatives: NMSIMP

6. Overview: OPTMODEL Procedure The OPTMODEL procedure comprises the powerful OPTMODEL modeling language and state-of-the-art solvers for several classes of mathematical programming problems. The problems and their solvers are listed in the following table. ProblemSolver Linear ProgrammingLP Mixed Integer Linear ProgrammingMILP Quadratic ProgrammingQP (experimental) Nonlinear Programming, UnconstrainedNLPU General Nonlinear ProgrammingNLPC General Nonlinear ProgrammingSQP General Nonlinear ProgrammingIPNLP (experimental)

7. Overview: The NLPC Nonlinear Optimization Solver In nonlinear optimization, we try to minimize or maximize an objective function that can be subject to a set of constraints. The objective function is typically nonlinear in terms of the decision variables. If the problem is constrained, it can be subject to bound, linear, or nonlinear constraints. In general, we can classify nonlinear optimization (minimization or maximization) problems into the following four categories: – unconstrained – bound constrained – linearly constrained – nonlinearly constrained These different problem classes typically call for different types of algorithms to solve them. The algorithm(s) devised specifically to solve a particular class of problem might not be suitable for solving problems in a different class. For instance, there are algorithms that specifically solve unconstrained and bound constrained problems. For linearly constrained problems, the fact that the Jacobian of the constraints is constant enables us to design algorithms that are more efficient for that class.

8. Optimization Techniques and Types of Problems Solved for NLPC Nonlinear Optimization Solver The algorithms in the NLPC solver take advantage of the problem characteristics and automatically select an appropriate variant of an algorithm for a problem. Each of the optimization techniques implemented in the NLPC solver can handle unconstrained, bound constrained, linearly constrained, and nonlinearly constrained problems without your explicitly requesting which variant of the algorithm should be used. The NLPC solver is also designed for backward compatibility with PROC NLP, enabling you to migrate from PROC NLP to the more versatile PROC OPTMODEL modeling language. The NLPC solver implements the following optimization techniques: – conjugate gradient method – Newton-type method with line search – trust region method – quasi-Newton method (experimental) These techniques assume the objective and constraint functions to be twice continuously differentiable. The derivatives of the objective and constraint functions, which are provided to the solver by using the PROC OPTMODEL modeling language, are computed using one of the following two methods: – automatic differentiation – finite-difference approximation

9. Using The NLPC Nonlinear Optimization Solver The NLPC solver solves unconstrained nonlinear optimization problems and problems with a nonlinear objective function subject to bound, linear, or nonlinear constraints. It provides several optimization techniques that effectively handle these classes of problems. To solve a particular problem with a supported optimization technique in the NLPC solver, you need to specify the problem by using the PROC OPTMODEL modeling language. The problem specification typically includes the MIN/MAX statement for the objective, the CON statement for the constraints, and the VAR statement for declaring the decision variables and defining bounds. Hence familiarity with the PROC OPTMODEL modeling language is assumed for requesting a particular optimization technique in the NLPC solver to solve the optimization problem. After you have specified the nonlinear optimization problem by using the PROC OPTMODEL modeling language, you can specify the NLPC solver by using the SOLVE statement as follows: SOLVE WITH NLPC [ / OPTIONS ]; where OPTIONS can specify the optimization technique, termination criteria, and/or whether to display the iteration log.

10. Optimization Algorithms for NLPC Nonlinear Optimization Solver There are four optimization algorithms available in the NLPC solver. A particular algorithm can be selected by using the TECH= option in the SOLVE statement. AlgorithmTECH= Newton-type method with line searchNEWTYP Trust region methodTRUREG Conjugate gradient methodCONGRA Quasi-Newton method (experimental)QUANEW

11. Choosing an Optimization Algorithm Several factors play a role in choosing an optimization technique for a particular problem. – The structure of the problem has to be considered: Is it unconstrained, bound constrained, or linearly constrained? – It is important to consider the type of derivatives of the objective function and the constraints that are needed, and whether these are analytically tractable or not. The NLPC solver automatically identifies the structure and chooses an appropriate variant of the algorithm for the problem. For an optimization problem, computing the gradient takes more computer time than computing the function value, and computing the Hessian matrix sometimes takes much more computer time and memory than computing the gradient, especially when there are many decision variables. Optimization techniques that do not use the Hessian usually require more iterations than techniques that do use the Hessian. The former tend to be slower and less reliable. However, the techniques that use the Hessian can be prohibitively slow for larger problems.

12. The following guidelines can be used in choosing an algorithm for a particular problem Without nonlinear constraints: – Smaller Problems: TRUREG or NEWTYP if n ≤ 1000 and the Hessian matrix is not expensive to compute. Sometimes NEWTYP can be faster than TRUREG, but TRUREG is generally more stable. – Larger Problems: CONGRA if n > 1000, the objective function and the gradient can be computed much more quickly than the Hessian, and too much memory is needed to store the Hessian. CONGRA in general needs more iterations than NEWTYP or TRUREG, but each iteration tends to be much faster. Since CONGRA needs less memory, many larger problems can be solved more efficiently by CONGRA. With nonlinear constraints: – QUANEW (experimental) QUANEW is the optimization method in the NLPC solver that solves problems with nonlinear constraints.