1. Improving Performance of The Interior Point Method by Preconditioning Project Proposal by: Ken Ryals For: AMSC 663-664 Fall 2007-Spring 2008

2. 2 Background The IPM method solves a sequence of constrained optimization problems such that the sequence of solutions approaches the true solution from within the “valid” region. As the constraints are “relaxed” and the problem re-solved, the numerical properties of the problem often become more “interesting”. μ0μ0

3. 3 Application Why is the IPM method of interest? – It applies to a wide range of problem types: Linear Constrained Optimization Semidefinite Problems Second Order Cone Problems –Once in the “good region” of a solution to the set of problems in the solution path (of µ ‘s): Convergence properties are great (“quadratic”). It keeps the iterates in the “valid” region. Specific Research Problem: –Optimization of Distributed Command and Control

4. 4 Optimization Problem The linear optimization problem can be formulated follows: inf{ c T x | Ax = b}. The search direction is implicitly defined by the system: Δx + π Δz = r A Δx = 0 A T Δy + Δz = 0. For this, the Reduced Equation is: A π A T Δy = −Ar (= b)  From Δy we can get Δx = r − π ( −A T Δy ). Note: The Nesterov-Todd direction corresponds to π = D⊗D, where: π z = x. i.e., D is the metric geometric mean of X and Z −1  D = X ½ (X ½ ZX ½ ) −½ X ½ x is the unknown y is the “dual” of x z is the “slack”

5. 5 The Problem From these three equations, the Reduced Equations for Δy are: A π A T Δy = −Ar (= b) The optimization problem is “reduced” to solving a system of linear equations to generate the next solution estimate. If π cannot be evaluated with sufficient accuracy, solving these equations becomes pointless due to error propagation. Aside: Numerically, evaluating ‘r − π Δz’ can also be a challenge. Namely, if the iterates converge, then Δx (= r − π Δz) approaches zero, but r does not; hence the accuracy in Δx can suffer from problems, too.

6. 6 Example: A Poorly-Conditioned Problem Consider a simple problem: Let’s change A 1,1 to make it ill-conditioned: Constraint Matrix 

7. 7 Observations The AD 2 A T condition exhibits an interesting dip around iteration 4-5, when the solution enters the region of the answer. How can we exploit whatever caused the dip? –The standard approach is to use factorization to improve numerical performance. The Cholesky factorization is: U T U = A π A T –Factoring a matrix into two components often trades one matrix with a condition of “M” for two matrices with conditions of ≈“√M”. –My conjecture is that AA T and D 2 interacted to lessen the impact of the ill conditioning in A ⇒Can we precondition with AA T somehow?

8. 8 Conjecture - Math We are solving: A π A T Δy = −Ar A is not square, so it isn’t invertible; but AA T is… What if we pre-multiplied by it? (AA T ) -1 A π A T Δy = − (AA T ) -1 Ar Note: Conceptually, we have: = Since, this looks like a similarity transform, it might have “nice” properties… − (AA T ) -1 Ar (AA T ) -1 A π A T Δy − (AA T ) -1 b (A T ) -1 π A T Δy

9. 9 Conjecture - Numbers Revisit the ill-conditioned simple problem, Condition of A π A T ( π is D 2 ) = 4.2e+014 Condition of (AA T ) = 4.0e+014 Condition of (AA T ) -1 A D 2 A T = 63.053 (which is a little less than 10 14 ) How much would it cost? AA T is m by m (m = # constraints) neither AA T or(AA T ) -1 is likely to be sparse… Let’s try it anyway… (If it behaves nicely, it might be worth figuring out how to do it efficiently)

10. 10 Experiment - Results It does work ☺ The condition number stays low (<1000) instead of hovering in the 10 14 range. It costs more ☹ Need inverse of AA T once. (AA T ) -1 gets used every iteration. The inverse is needed later, rather than early in the process; thus, it could be iteratively developed during the iteration process… Solution enters region of “Newton” convergence

11. 11 Project Proposal 1.Develop a system to for preconditioned IPM. Create Matlab version to define structure of system. Permits validation against SeDuMi and SPDT3 Use C++ transition from Matlab to pure C++. Create MEX modules from C++ code to use in Matlab 2.Apply the technique to the Distributed C 2 problem 3.Modify the system to develop (A*A T ) -1 iteratively or to solve the system of equations iteratively Can we use something like the Sherman-Morrison-Woodbury Formula? (A - ZV T ) -1 = A -1 + A -1 Z(I - V T A -1 Z) -1 V T A -1 Can the system can be solved using the Preconditioned Conjugate Gradient Method? 4.Time permitting, “parallel-ize” the system –Inverse generation branch, and –Iteration branch using current version of Inverse. Testing: Many test optimization problems can be found online “AFIRO” is a good start (used in AMSC 607 – Advanced Numerical Optimization)