A Newton Method for Linear Programming Olvi L. Mangasarian University of California at San Diego

Outline  Fast Newton method for class of linear programs  Large number of constraints (millions)  Moderate number of variables (hundreds)  Method is based on an overlooked fact  Dual of asymptotic exterior penalty problem of primal LP  Provides an exact least 2-norm solution of the dual LP  Use a finite value of the penalty parameter  Exact least 2-norm dual solution generates:  Highly accurate primal solution when it is unique  Globally convergent Newton method established  Eleven lines of MATLAB code  Method solves LP with 2-million constraints & 100 variables  14-place accuracy in 17.4 minutes on 400Mhz machine  CPLEX 6.5 ran out of memory on same problem  Dual method for million-variable LP

Primal-Dual LPs Primal Exterior Penalty & Dual Least 2-Norm Problem Dual LP: Exact Dual Least 2-Norm: Primal LP: Asymptotic Primal Exterior Penalty:

The Plus Function

Dual Least 2-Norm: Karush-Kuhn-Tucker Optimality Conditions for Least 2-Norm Dual Lagrangian: KKT Conditions: Equivalently:

Karush-Kuhn-Tucker Optimality Conditions for Least 2-Norm Dual (Continued) Thus : Hence: Thus: solves the exterior penalty problem for the primal LP We have:

Equivalence of Exact Least 2-Norm Dual Solution to Finite-Parameter Primal Penalty Minimization

Primal Exterior Penalty Primal Exterior Penalty(  exact least 2-norm dual solution) : Gradient: Generalized Hessian: where:

LP Newton Algorithm (LPN)

LPN Convergence

Remarks on LPN

lpgen: Random Solvable LP Generator %lpgen: Generate random solvable lp: min c'x s.t. Ax =< b; A:m-by-n %Input: m,n,d(ensity); Output: A,b,c; (x,u): primal-dual solution pl=inline('(abs(x)+x)/2');%pl(us) function tic;A=sprand(m,n,d);A=100*(A-0.5*spones(A)); u=sparse(10*pl(rand(m,1)-(m-3*n)/m)); x=10*spdiags((sign(pl(rand(n,1)-rand(n,1)))),0,n,n)*(rand(n,1)-rand(n,1)); c=-A'*u;b=A*x+spdiags((ones(m,1)-sign(pl(u))),0,m,m)*10*ones(m,1);toc0=toc; format short e;[m n d toc0]  Elements of A uniformly distributed between –50 and +50  Primal random solution x with elements in [-10,10], approximately half of which are zero  Dual random solution u in [0,10], approximately 3n of which are positive

lpnewt1: MATLAB LPN Algorithm without Armijo %lpnewt1: Solve primal LP: min c'x s.t. Ax= =0 %Input: c,A,b,epsi,delta,tol,itmax;Output:v(l2norm dual sol),z primal sol epsi=1e-3;tol=1e-12;delta=1e-4;itmax=100;%default inputs pl=inline('(abs(x)+x)/2');%pl(us) function tic;i=0;z=0;y=((A(1:n,:))'*A(1:n,:)+epsi*eye(n))\(A(1:n,:))'*b(1:n);%y0 while (i tol & toc<1800) df=A'*pl((A*y-b))+epsi*c; d2f=A'*spdiags(sign(pl(A*y-b)),0,m,m)*A+delta*speye(n); z=y;y=y-d2f\df; i=i+1; end toc1=toc;v=pl(A*y-b)/epsi;t=find(v);z=A(t,:)\b(t);toc2=toc; format short e;[epsi delta tol i-1 toc1 toc2 norm(x-y,inf) norm(x-z,inf)]

LPN & CPLEX 6.5 Comparison (LPN without Armijo vs. Dual Simplex) (400 Mhz Pentium II 2Gig) (oom=out of memory)

What About the Case of: n>> m ?  LPN is not appropriate  LPN needs to invert a very large n-by-n matrix  Use instead DLPN (Dual LPN):  Find least 2-norm primal solution  By solving dual exterior penalty problem with finite penalty parameter

Equivalence of Exact Least 2-Norm Primal Probelm and Asymptotic Dual Penalty Problem

Difficulty: Nonnegativity Constrained Dual Penalty  Use an exterior penalty to handle nonnegativity constraint  By:  Replace:

D LPN & CPLEX 7.5 Comparison (DLPN without Armijo vs. Primal Simplex ) (1.8 MHz Pentium 4 1Gig)(oom=out of memory;error 1001) (infs/unbd=presolve determines infeasibility/unboundedness;error 1101 )

Conclusion  LPN & DLPN: Fast new methods for solving linear programs  LPN capable of solving problems with millions of constraints and hundreds of variables  DLPN capable of solving problems with hundreds of constraints and millions of variables  Eleven lines of MATLAB code for each of LPN & DLPN  Competitive with state-of-the-art CPLEX  Very suitable for classification problems  Armijo stepsize not needed in many applications  Typical termination in 3 to 30 steps

Future Work  Investigate & establish finite termination for LPN  Extend to convex quadratic programming  Establish convergence/termination under no assumptions for LPN  Generate self-contained codes for classification and data- mining problems

Paper & MATLAB Code Available