1. Chapter 4 Sensitivity Analysis, Duality and Interior Point Methods

2. Simplex methods reach the optimal solution by traversing a series of basic solutions. The path followed by the algorithm moves around the boundary of the feasible region. This can be cripplingly inefficient given that the number of extreme points grows exponentially with n for m fixed. The running time of an algorithm as a function of the problem size is known as its computational complexity. In practice, the simplex method works surprisingly well, exhibiting linear complexity—i.e., its running time is proportional to m + n. Researchers, however, have long tried to develop solution algorithms whose worst-case running times are polynomial functions of the problem size. Interior Point Methods

3. First Success: Russian Mathematician Khachian proposed the ellipsoid method with complexity O(n 6 ). It is efficient only theoretically, no one ever realize an implementation that matched the performance of current simplex codes. Big Success: Karmarkar(1984) IPMs have produced solutions to many industrial problems that previously were intractable. Interior Point Methods (IPMs)

4. Three major types of IPMs: (1)Potential Reduction algorithm: most closely embodies the constructs of Karmarkar; (2)Affine Scaling algorithm: the simplest to implement; (3)Path Following algorithms: combine excellent behavior in theory and practice. We discuss a member of the third category known as the primal-dual path following algorithm. The algorithm operates simultaneously on the primal and dual linear programming problems.

5. Example: Primal model Dual model Max Z P = 2X 1 + 3 X 2 Min Z D = 8π 1 + 6π 2 s.t. 2 X 1 + X 2 + X 3 =8 s.t. 2π 1 +π 2 - Z 1 =2 X 1 + 2X 2 + X 4 = 6 π 1 + 2π 2 - Z 2 =3 X j ≧ 0, j=l,.,.,4 π 1 -Z 3 =0 π 2 -Z 4 =0 Z j ≧ 0, j=1,...,4 Note: Z j ’s denote the slack variables of the dual model.

6. The algorithm iteratively modifies the primal and dual solutions until the optimality conditions of primal feasibility, dual feasibility, and complementary slackness are satisfied to a sufficient degree. This event is signaled when the duality gap— the difference between the primal and dual objective functions—is sufficiently small. For the purposes of the discussion, "interior of the feasible region" is taken to mean that the values of the primal and dual variables are always strictly greater than zero.

8. The Primal and Dual Problems

9. Basic Ideas: The application of the Lagrange multiplier method to the transformation of an equality constrained optimization problem into an unconstrained one. The transformation of an inequality constrained optimization problem into a sequence of unconstrained problems by incorporating the constraints in a logarithmic barrier function that imposes a growing penalty as the boundary { X j = 0, and Z j = 0 for all j) is approached. The solution of a set of nonlinear equations using Newton's method, thereby arriving at a solution of the unconstrained optimization problem.

10. The Lagrangian Approach Consider the general problem Maximize f(x) subject to g i (x) = 0, i= 1,...,m where f(x) and g i (x) are scalar functions of the n- dimensional vector x. The Lagrangian for this problem is L(x,π)=f(x)- where the variables π = ( π 1, π 2,..., π m ) are the Lagrange multipliers

11. Necessary conditions for a stationary point (maximum or minimum): For linear constraints (a i x –b i = 0), the conditions are sufficient for a maximum if the function f(x) is concave and sufficient for a minimum if f(x) is convex.

12. The Barrier Approach Considering the non-negativity conditions, the idea of the barrier approach, as developed by Fiacco and McCormick [1968], is to start from a point in the strict interior of the inequalities (x j > 0, and z j > 0 for all j) and construct a barrier that prevents any variable from reaching a boundary (e.g., x j = 0). Adding log(x j ) to the objective function of the primal problem, for example, will cause the objective to decrease without bound as X j approaches 0.

13. The difficulty is that if the constrained optimal solution is on the boundary (that is, one or more X j *= 0, which is always the case in linear programming), then the barrier will prevent us from reaching it. To get around this difficulty, we use a barrier parameter μ that balances the contribution of the true objective function with that of the barrier term.

14. The parameter μis required to be positive and controls the magnitude of the barrier term. Because function log(x) takes on very large negative values as x approaches zero from above, as long as the problem variables x and z remain positive the optimal solutions to the barrier problems will be interior to the nonnegative orthants (X j > 0 and Z j > 0 for all j). The barrier term is added to the objective function for a maximization problem and subtracted for a minimization problem. The new formulations have nonlinear objective functions with linear equality constraints, and can be solved with the Lagrangian technique for μ > 0 fixed. The solutions to these problems will approach the solution to the original problem as μ approaches zero.

16. Note: These conditions are also sufficient, because the primal Lagrangian is concave and the dual Lagrangian is convex. The optimal conditions are nothing more than primal feasibility, dual feasibility, and complementary slackness satisfied to within a tolerance of μ. To facilitate the presentation, we define two n × n diagonal matrices containing the components of x and z, respectively—i.e., X = diag{ x 1, x 2,.., x n } Z = diag{ z 1, z 2,..., z n }

17. Let e = (1,1,..., 1) T be a column vector of size n. The necessary and sufficient conditions can be written as Primal feasibility: Ax - b = 0 (m linear equations) Dual feasibility: A T π T - z - c T = 0 (n linear equations) μ-Complementary slackness: XZe - μe = 0 (n nonlinear equations) We must now solve this set of nonlinear equations for the variables (x, π, z).

18. Stationary Solutions Using Newton's Method Consider the single variable problem of finding y to satisfy the nonlinear equation f(y)=0 where f is once continuously differentiable. Let y* be the unknown solution. Taking the first-order Taylor series expansion of f(y) around y k yields where △ = y k+1 – y k. Setting the approximation f(y k+1 ) equal to zero and solving for △ yields

19. The point y k+1 = y k + △ is an approximate solution of the equation. It can be shown that if one starts at a point y° sufficiently close to y*, then y k →y* as k→∞.

20. Newton's method can be extended to multidimensional functions. Consider the general problem of finding the r-dimensional vector y that solves the set of r equations f i (y)=0, i=l,...,r or f(y)=0

21. The Jacobian at y k is All the partial derivatives are evaluated at y k.

22. Taking the first-order Taylor series expansion around the point y k and setting it equal to zero yields f(y k )+ J (y k )d=0 where d = y k+1 - y k is an r-dimensional vector whose components represent the change in position for the (k+l)th iteration. Solving for d leads to d= -J(y k ) -1 f(y k ) The point y k+1 = y k + d is an approximation of the solution of the set of equations.

23. Newton's Method for Solving Barrier Problems: We now use Newton's method to solve the optimality conditions for the barrier problems for a fixed value of μ. For y = (x, π, z) and r = 2n + m, the corresponding equations and Jacobian are where e is an n-dimensional column vector of 1's. Assume that we have a starting point (x°, π°, z°) satisfying x° > 0 and z° > 0, and denote by δ p =b-Ax° δ D = c T –A T (π°) T + z°

24. The optimality conditions can be written as J(y)d = -f(y) where the (2n + m)-dimensional vector d = (d X, d π, d z ) T is called the Newton direction. We must now solve for the individual components of d. In explicit form, the preceding system is

25. The first step is to find d π. With some algebraic manipulation, we get or Note that Z -l = diag{ 1/z 1, l/z 2,.., 1/z n } and is trivial to compute. Further multiplications and substitutions lead to d z = - δ D +A T d π and d x = Z -l (μe-XZe-Xd z ) From these results, we can see in part why it is necessary to remain in the interior of the feasible region. In particular, if either Z -l or X -l does not exist, the procedure breaks down.

26. We move from the current point (x°, π°, z°) to a new point (x 1, π 1, z 1 ) by taking a step in the direction (d x, d π, d z ). The step sizes α p and α D are chosen to preserve x > 0 and π > 0. This requires a ratio test similar to that performed in the simplex algorithm. The simplest approach is to use where γ is the step size factor that keeps us from actually touching the boundary. Typically, γ= 0.995. The notation (d x ) j refers to the jth component of the vector d x

27. The new point is which completes one iteration. Ordinarily, we would now resolve using (x 1, π 1, z 1 ) to find a new Newton direction and hence a new point. Rather than iterating in this manner until the system converges for the current value of μ, it is much more efficient to reduce μ at every iteration.

28. It is straightforward to show that the Newton step reduces the duality gap—the difference between the dual and primal objective values at the current point. Assume that x° is primal feasible and that ( π 0, z°) is dual feasible. Let "gap(0)" denote the current duality gap. gap(0) =π 0 b - cx° =π 0 (Ax 0 ) - (π 0 A - z 0 ) T x° (primal and dual feasibility) = (z 0 ) T x° If we let α = min{ α p, α D }, then

29. It can be shown that gap(α) < gap(0) as long as In our computations, we have used which indicates that the value of μ k is proportional to the duality gap. Termination occurs when the gap is sufficiently small—say, less than 10 -8.

30. Primal-Dual Interior Point Algorithm Inputs: 1. The data of the problem (A, b, c), where the m x n matrix A has full row rank 2. Initial primal and dual feasible solutions x° > 0, z° > 0, π° 3. The optimality tolerance ε > 0 and the step size parameter γ (0,1)

31. Step 1: (Initialization) Start with some feasible point x° > 0, z° > 0, π° and set the iteration counter k = 0. Step 2: (Optimality Test) If (z k ) T x k < ε, stop; otherwise, go to Step 3. Step 3: (Compute Newton Directions) Let

32. Solve the following linear system to get Note that δ P =0 and δ D =0 as a result of the feasibility of the initial point.

33. Step 4: (Find Step Lengths) Let Step 5: (Update Solution) Take a step in the Newton direction to get Put k← k + 1 and go to Step 2.

34. Example: We have n = 4 and m = 2, with data and variable definitions For an initial interior solution, we take x° = (1,1, 5, 3), z° = (4, 3, 2, 2), and π° = (2, 2).

35. The primal residual vectorδ P and the dual residual vector δ D are both zero, since x° and z° are interior and feasible to their respective problems. The column labeled "Comp. slack" is the μ-complementary slackness vector computed as XZe-μe Since this vector is not zero, the solution is not optimal.