2. MIT and James Orlin © 2003 1 Chapter 3. The simplex algorithm Putting Linear Programs into standard form Introduction to Simplex Algorithm file Simplex2_AMII_05a_gr Rev. 3.1 by M. Miccio on December 17, 2014 from a presentation at the Fuqua School of Business MIT and James Orlin © 2003

3. The Simplex Algorithm 2 Background George Dantzig born 8.11.1914, Portland invented "Simplex Method of Optimisation" in 1947 this grew out of his work with the USAF

4. The Simplex Algorithm 3 Background originates from military planning tasks: –plans or schedules for training –logistical supply –deployment of men has usually a “polynomial” computational cost

5. MIT and James Orlin © 2003 4 Overview of Lecture Getting an LP into standard form Getting an LP into canonical form Optimality conditions Improving a solution A simplex pivoting

6. MIT and James Orlin © 2003 5 Overview of Lecture Getting an LP into standard form

7. MIT and James Orlin © 2003 6 Linear Programs in Standard Form We say that a linear program is in standard form if the following are all true: 1. Non-negativity constraints for all variables. 2. All remaining constraints are expressed as equality constraints. 3. The right hand side vector, b, is non-negative. EXAMPLE 1: maximize 3x 1 + 2x 2 - x 3 + x 4 x 1 + 2x 2 + x 3 - x 4  5; not equality -2x 1 - 4x 2 + x 3 + x 4  -1; not equality x 1  0, x 2  0, x 4  0 x 3 may be negative  The EXAMPLE 1 is a LP problem in its original form, but not in standard form

8. MIT and James Orlin © 2003 7 s 1 is called a slack variable, which measures the amount of “unused resource” Note that s 1 = 5 - x 1 - 2x 2 - x 3 + x 4 Converting “  ” constraints  To convert a “  ” constraint to an equality, add a slack variable. After (augmented form) x 1 + 2x 2 + x 3 - x 4 + s 1 = 5 s 1  0 Before (original form) x 1 + 2x 2 + x 3 - x 4  5

9. MIT and James Orlin © 2003 8 Converting Inequalities into Equalities plus Non-negatives Resources Consider the 2nd inequality: -2x 1 - 4x 2 + x 3 + x 4  -1 (original form) Step 1. Eliminate the negative RHS 2x 1 + 4x 2 - x 3 - x 4  1 (standard form)

10. MIT and James Orlin © 2003 9 Converting “  ” constraints 2x 1 + 4x 2 - x 3 - x 4  1 (standard form) Convert to an equality: 2x 1 + 4x 2 - x 3 - x 4 – s 2 = 1 (augmented form) The variable added s 2  0 is called a “surplus variable”, which measures the amount of “resource in excess of minimum”   To convert a “  ” constraint to an equality, subtract a surplus variable.

11. MIT and James Orlin © 2003 10 More Transformations How can one convert a maximization problem to a minimization problem? Example: Maximize 3W + 2P Subject to “constraints” has the same optimum solution(s) as Minimize -3W -2P Subject to “constraints”

12. MIT and James Orlin © 2003 11 The Last Transformations (for now) Transforming variables that may take on negative values. EXAMPLE 2: maximize 3x 1 + 4x 2 + 5 x 3 subject to 2x 1 - 5x 2 + 2x 3 = 7 other constraints: x 1  0 x 2 is unconstrained in sign x 3  0

13. MIT and James Orlin © 2003 12 Transforming variables that may take on negative values. Transforming x 1 : replace x 1 by y 1 = -x 1 → y 1  0 max -3 y 1 + 4x 2 + 5 x 3 -2 y 1 -5 x 2 +2 x 3 = 7 y 1   0, x 2 is unconstrained in sign, x 3  0 One can recover x 1 from y 1 e.g., y 1 = 1, x 2 = -1, x 3 = 2 is feasible thenx 1 =-1, x 2 = -1, x 3 = 2 is a solution maximize 3x 1 + 4x 2 + 5 x 3 subject to 2x 1 - 5x 2 + 2x 3 = 7 other constraints: x 1  0, x 2 is unconstrained in sign, x 3  0

14. MIT and James Orlin © 2003 13 max -3 y 1 + 4(y 3 - y 2 ) + 5 x 3 -2 y 1 -5 y 3 + 5 y 2 +2 x 3 = 7 all vars  0 Transforming variables that may take on negative values. Transforming x 2 : replace x 2 by x 2 = y 3 - y 2 ; with y 2  0, y 3  0 One can recover x 2 from y 2, y 3 e.g., y 1 = 1, y 2 = 0, y 3 = 1, x 3 = 2 is feasible theny 1 = 1, x 2 = 1, x 3 = 2 is a solution maximize 3x 1 + 4x 2 + 5 x 3 subject to 2x 1 - 5x 2 + 2x 3 = 7 other constraints x 1  0, x 2 is unconstrained in sign, x 3  0

15. Standard form in vector-matrix representation MIT and James Orlin © 2003 14 x is the n-size column vector of variables (decision + slack + surplus) m is the number of constraints c T is the k-size row vector (with k < n) of cost coefficients A is an mn matrix of constraint coefficients  A has to be a full rank matrix, i.e., r = min(m,n) b is the m-size column vector of resources Ax = b represents a system of m linear eqs. in n unknows

16. MIT and James Orlin © 2003 15 Standard form with Tableau representation x1x1 x2x2 x4x4 x3x3 -z = -331600 2001 = 02-42010 = TABLEAU  A linear program in standard form can be represented in a tabular arrangement known as tableau Example 3:maximizez =-3x 1 + 2x 2 +0  x 3 +0  x 4 (2D case)subject to -3x 1 + 3x 2 + x 3 = 6 -4x 1 + 2x 2 + x 4 = 2 x 1, x 2, x 3, x 4  0 b

17. MIT and James Orlin © 2003 16 Another Example (for home) Exercise 4 (also 2D): transform the following to a standard form for maximization: Minimize x 1 + 3x 2 Subject to 2x 1 + 5x 2   12 x 1 + x 2  1 x 1  0

18. MIT and James Orlin © 2003 17 Overview of Lecture Getting an LP into standard form Getting an LP into canonical form

19. Solutions for Ax = b MIT and James Orlin © 2003 18 How many solutions for Ax = b ? Rouchè-Capelli theorem : A full-rank system, consisting of m linear equations in n variables, has ∞ n-m solutions

20. Special solutions for Ax = b MIT and James Orlin © 2003 19 basic solution A solution to a linear program model, consisting of m equations in n variables, obtained by solving for m variables (basic variables) in terms of the remaining (n - m) variables and setting the (n - m) variables equal to zero (non-basic variables) feasible basic solution A basic solution for which the m variables that are different from zero and form the basis are also non-negative degenerate feasible basic solution Basic feasible solutions where at least one of the basic variables is zero are called degenerate

21. generic solutions Map of solutions for Ax = b MIT and James Orlin © 2003 20 feasible solutions feasible basic solutions degenerate feasible basic solutions basic solutions

22. MIT and James Orlin © 2003 21 -32001 310 -4201 0 0 -3310 -4201 0 0 -32001 LP Canonical Form = LP Standard Form + Jordan Canonical Form ==== 2 6 x1x1 x2x2 x4x4 x3x3 -z = 0  The simplex method starts with an LP in canonical form (or it creates canonical form at a preprocess step.) z is not a decision variable 10 01 00 x4x4 x3x3 If the columns of A can be rearranged so that it contains the identity matrix of order m (the number of rows in A) then the tableau is said to be in canonical form EXAMPLE 3

23. MIT and James Orlin © 2003 22 Overview of Lecture Getting an LP into standard form Getting an LP into canonical form The Simplex algoritm with ≤ constraints

24. MIT and James Orlin © 2003 23 -32001 310 -4201 0 0 -3310 -4201 0 0 -32001 LP Canonical Form with the first feasible basic solution ==== 2 6 x1x1 x2x2 x4x4 x3x3 -z = 0 The 1st feasible basic solution is x 1 = 0, x 2 = 0, x 3 = 6, x 4 = 2 (set the non-basic variables to 0, and then solve) The basic variables are x 3 and x 4 The non-basic variables are x 1 and x 2 10 01 00 x4x4 x3x3 EXAMPLE 3

25. MIT and James Orlin © 2003 24 -3310 -4201 For each constraint there is a basic variable ==== 2 6 x2x2 x4x4 x3x3 -3200 -z 0 0 1 = 0 Constraint 1: basic variable is x 3 Constraint 1 Constraint 2: basic variable is x 4 Constraint 2 The basis consists of variables x 3 and x 4 x1x1 -331060-420120 EXAMPLE 3

26. The underlying Theorems The corners or vertices of the feasible region are referred to as the extreme points. Theorem of the Corner Principle : In any LP problem with a nonempty bounded feasible region, the optimal value of the objective function, if exists, will be achieved on an extreme point. –Restatement of the previous Theorem : If an LP problem has an optimal solution, then it has a basic optimal solution. Theorem of Solutions: The No. of extreme points is equal to the N b, i.e., the No. of feasible basic solutions. When looking for the optimal solution, you do not have to evaluate all feasible solution points. You have to consider only the extreme points of the feasible region. 25

27. 87654321 1 2 3 4 5 6 7 8 9 10 Extreme Points in 2D x1x1x1x1 FeasibleRegion 12 3 4 5 x 2 x 2 © 2006 Thomson South-Western. All Rights Reserved.

28. The Simplex Algorithm 27 The Simplex algorithm strategy in a qualitative way: 1.start from a first basic feasible solution 2.look for an "adjacent" basic feasible solved form whose basic feasible solution improves the value of the objective function  "adjacent" basic feasible solution means that (m-1) variables remain the same in the set of basic variables, whereas 1 basic variable is replaced by a non-basic variable 3.if there is no such adjacent basic feasible solved form, then the optimum has been found ( ideally !)

29. The Simplex algorithm and Extreme points MIT and James Orlin © 2003 28 A system of linear inequalities defines a polytope as a feasible region in the hyperspace. The simplex algorithm begins at a starting vertex (1 st feasible basic solution) and moves along the edges of the polytope until it reaches the vertex of the optimum solution. The algorithm always terminates because the number of vertices in the polytope is finite

30. The Simplex algorithm result Any LP problem falls in one of three categories: 1.has a optimal solution (unique or ”alternate” optimal solutions) 2.has an objective function that can be increased without bound 3.is infeasible (e.g., non-convex feasible region) However, an unbounded feasible region does not imply an unbounded objective function Yet there may be an optimal solution –This is common in minimization problems and is possible in maximization problems.

31. MIT and James Orlin © 2003 30 Overview of Lecture Getting an LP into standard form Getting an LP into canonical form Optimality conditions

32. MIT and James Orlin © 2003 31 x1x1 -3310 -4201 Optimality Conditions Preview ==== 2 6 x2x2 x4x4 x3x3 -3200 -z 0 0 1 = 0 Obvious Fact: If one can improve the current basic feasible solution x, then x is not optimal. Idea: assign a small value  to just one of the non- basic variables, and then adjust the basic variables. z = -3x 1 + 2x 2 max!  current value of -z EXAMPLE 3

33. MIT and James Orlin © 2003 32 x1x1 -3310 -4201 The current basic feasible solution (bfs) is not optimal! ==== 2 6 x2x2 x4x4 x3x3 -3200 -z 0 0 1 = 0 Increase x 2 from 0 to  > 0. Let x 1 stay at 0. What happens to x 3, x 4 and z? x 3 = 6 - 3 . x 4 = 2 - 2 . z = 2 . x1x1 -3 -4 -3 If there is a positive coefficient in the z row, the basis is not optimal** z = -3x 1 + 2x 2 max! EXAMPLE 3

34. MIT and James Orlin © 2003 33 -3310 -4201 Optimality Conditions ==== 2 6 x2x2 x4x4 x3x3 -2-400 -z 0 0 1 = -8 x1x1  If there is no positive coefficient in the z row, the basic feasible solution is optimal! NB: EXAMPLE 3 with different cost coeff. z = -2x 1 - 4x 2 + 8 max! z  8 for all other feasible solutions. But z = 8 in the current basic feasible solution This basic feasible solution is optimal!

35. MIT and James Orlin © 2003 34 x 1 = 0 x 2 =  x 3 = 6 - 3 . x 4 = 2 - 2 . z = 2 . x 1 = 0 x 2 = 1 x 3 = 3 x 4 = 0 z = 2 x1x1 -3310 -4201 Let x 2 = . How large can  be? What is the solution after changing x 2 ? ==== 2 6 x2x2 x4x4 x3x3 -3200 -z 0 0 1 = 0 What is the value of  that maximizes z, but leaves a feasible solution? x1x1 -3 -4 -3  = 1.   The resulting solution is a new basic feasible solution with a different basis. EXAMPLE 3

36. Optimality criterion It relies on a check of the (reduced) cost coefficients c j ’ Search for the maximum: Search for the minimum: 35

37. MIT and James Orlin © 2003 36 Overview of Lecture Getting an LP into standard form Getting an LP into canonical form Optimality conditions Improving a solution: a pivot

38. 11 November, 2002 The Simplex Algorithm 37 Improving a solution, a pivot pivoting  the algebraic manipulation of the Tableau result of pivoting  move one variable out of basic variables (  exit variable) and another one in (  entry variable) adjacent solution  the new basic feasible solution obtained just after one single pivoting operation on Tableau

39. Text description of Pivoting MIT and James Orlin © 2003 38 The geometrical operation of moving from a basic feasible solution to an adjacent basic feasible solution is implemented as a pivot operation. First, a nonzero pivot element is selected in a non-basic column. The row containing this element is multiplied by its reciprocal to change this element to 1, and then multiples of the row are added to the other rows to change the other entries in the column to 0. The result is that, if the pivot element is in row r, then the column becomes the r-th column of the identity matrix. The variable for this column is now a basic variable, replacing the variable which corresponded to the r-th column of the identity matrix before the operation. The variable corresponding to the pivot column enters the set of basic variables and is called the entering (entry) variable The variable being replaced leaves the set of basic variables and is called the leaving (exit or departing) variable. The tableau is still in canonical form, but with the set of basic variables changed by one element.

40. Optimal choice of Pivot Column MIT and James Orlin © 2003 39 The column with j=c becomes pivot. The corresponding variable x c is the entry variable. max !min !

41. Choice of Pivot Row MIT and James Orlin © 2003 40 The row with i=r becomes pivot. The variable x * for which the coefficient is unity in the pivot row is selected as the leaving (exit) variable. Minimum ratio rule

42. Pivoting formulas MIT and James Orlin © 2003 41 r is the pivot row c is the pivot column a rc is the pivot element  j=1,n a rj  a rj /a rc b r  b r /a rc  i=1,m with m≠r and  j=1,n a ij  a ij - a rj a ic b i  b i - a rj a ic c j  c j - a rj a ic

43. MIT and James Orlin © 2003 42 2 x1x1 -3310 -401 Pivoting to obtain a better solution (Tableau animation) ==== 2 6 2 x2x2 x4x4 x3x3 -3200 -z 0 0 1 = 0 If we pivot on the coefficient 2, we obtain the new basic feasible solution. -210.51 100-2 301-1.53 x 1 = 0 x 2 = 1 x 3 = 3 x 4 = 0 z = 2 New Solution: Basic variables are x 2 and x 3. Nonbasics: x 1 and x 4. EXAMPLE 3 z = -3x 1 + 2x 2 max!

44. MIT and James Orlin © 2003 43 x1x1 OK. Let’s iterate again. ==== x2x2 x4x4 x3x3 -z 0 0 1 = The cost coefficient of x 1 is positive. Set x 1 =  and x 4 = 0. -210.5 100 301-1.5 1 -2 3 x 1 =  x 2 = 1 + 2  x 3 = 3 - 3 . x 4 = 0 z = 2 +  How large can  be? z = x 1 – x 4 + 2 max!.5 -1.5 EXAMPLE 3

45. MIT and James Orlin © 2003 44 1 -2 3 10.5 100 301-1.5 3 -3 1 012/3-1/2 00-1/3-1/2 01/3-1/2 x1x1 Perform another pivot ==== x2x2 x4x4 x3x3 -z 0 0 1 = What is the largest value of  ? x 1 =  x 2 = 1 + 2  x 3 = 3 - 3 . x 4 = 0 z = 2 +   x 1 =  x 2 = 3 x 3 = 0 x 4 = 0 z = 3 Pivot on the coefficient with the number 3. 31 Variable x 1 becomes basis, x 3 becomes nonbasic. So, x 1 becomes the basic variable for constraint 1.

46. MIT and James Orlin © 2003 45 3 -3 1 012/31/2 00-1/3-1/2 101/3-1/2 x1x1 Check for optimality ==== x2x2 x4x4 x3x3 -z 0 0 1 = There is no positive coefficient in the z-row. x 1 =  x 2 = 1 + 2  x 3 = 3 - 3 . x 4 = 0 z = 2 +  The current basic feasible solution is optimal! x 1 =  x 2 = 3 x 3 = 0 x 4 = 0 z = 3 z = -x 3 /3 – x 4 /2 + 3 max!

47. MIT and James Orlin © 2003 46 Summary of Simplex Algorithm (  constraints) Start in canonical form with the 1 st basic feasible solution 1. Check for optimality conditions  for max: is there a positive coefficient in the cost row? 2. If not optimal, determine a non-basic variable that should be made positive  for max: choose a variable with a positive coeff. in the cost row 3. Increase that non-basic variable and perform a pivot, obtaining a new bfs (or unboundedness) 4. Continue until optimal (or unbounded).