1. Algorithms CISC 4080/CISC 5825
Linear Programming, Simplex Method and Duality Theory Dr Kamesam Graduate School Of Business Fordham University

2. Outline Elementary Row Operations Graphical Approach to solving LP
Solution of Maximization Problems with Simplex Method
Special Situations in the Simplex method.
Simplex method with artificial variables.
Solving LP with Excel Solver.
Duality
You may find the following free chapter helpful

3. Background Elementary Row Operations
Given a system of linear equations, Elementary row operations lead to an equivalent system of linear equations, in the sense that the solution set of both systems of equations are the same.
Elementary Row Operations we will use
Multiply both sides of an equation with a nonzero constant
Add a multiple of one equation to another equation

5. Standard Maximization Problems in Standard Form
A linear programming problem is said to be a standard maximization problem in standard form if its mathematical model is of the following form: Maximize the objective function Subject to problem constraints of the form With non-negative constraints

6. Slack Variables
“A mathematical representation of surplus resources.” In real life problems, it’s unlikely that all resources will be used completely, so there usually are unused resources. Slack variables represent the unused resources between the left-hand side and right-hand side of each inequality.

7. Solve the following LP Graphically
We shall next, learn how the Simplex method works on this problem

8. LP Characteristics
Feasible Region: is a convex set with extreme points. An extreme point is a point which cannot be expressed as a convex combination of other points in the set.
Corner Point Property: An optimal solution must lie at one or more corner points (extreme points)
Optimal Solution: The corner point with the best objective function value is optimal

9. The Simplex Method
The geometric method of solving linear programming problems was presented before. The graphical method is useful only for problems involving two decision variables and relatively few problem constraints.
What happens when we need more decision variables and more problem constraints?
We use an algebraic method called the simplex method, which was developed by George B. DANTZIG ( ) in 1947 while on assignment with the U.S. Department of the air force.

11. Algorithm for LP
The graphical method can only be used for problems in 2 variables. However, it gives us insight
Practical applications of LPs can be very large (tens of thousands of variables). So we need a robust and efficient algorithm, that can also take advantage of sparsity in the data
Answer: Simplex Method

12. Basic and Nonbasic Variables
Basic variables are selected arbitrarily with the restriction that there be as many basic variables as there are equations. The remaining variables are non-basic variables. This system has two equations, we can select any two of the four variables as basic variables. The remaining two variables are then non-basic variables. A solution found by setting the two non-basic variables equal to 0 and solving for the two basic variables is a basic solution. If a basic solution has no negative values, it is a basic feasible solution.

14. Extreme Points and BFS
The Feasible Region has Extreme points (Corner Points)
Note that there is a one to one correspondence between Corner Points and BFS.
So, a BFS is an algebraic Characterization of the geometric concept, extreme point.
In order to develop effective algorithms, geometric insight is helpful, but an algebraic characterization is a must.

15. Enumerating all BFS to find Optimal Solution
So, can we turn this into an algorithm ?
Unfortunately, NO
For a system with n variables and m constraints, there are nCm possible Basic solutions.
Even for a very small Linear Program with n = 100 and m = 40, there are E+28 Basic Solutions.
Although the Simplex Algorithm does move from BFS to BFS, it does so cleverly, ensuring that the objective function keeps increasing.

16. SIMPLEX METHOD Simplex algorithm for standard maximization problems
Step-1
Write the standard maximization problem in standard form, introduce slack variables to form the initial system, and write the initial tableau.
Step-3
Select the pivot column
Step-5
Select the pivot element and perform the pivot operation
Step 4
Are there any positive elements in the pivot column above the dashed line?
Step 2
Are there any negative indicators in the bottom row?
STOP
The optimal solution has been found.
STOP
The linear programming problem has no optimal solution. UNBOUNDED
Simplex algorithm for standard maximization problems

17. To solve a linear programming problem in standard form, use the following steps.
1- Convert each inequality in the set of constraints to an equation by adding slack variables. 2- Create the initial simplex tableau. 3- Select the pivot column. ( The column with the “most negative value” element in the last row.) 4- Select the pivot row. (The row with the smallest non-negative result when the last element in the row is divided by the corresponding in the pivot column.) 5-Use elementary row operations to calculate new values for the pivot row so that the pivot is 1 (Divide every number in the row by the pivot number.) 6- Use elementary row operations to make all numbers in the pivot column equal to 0 except for the pivot number. If all entries in the bottom row are zero or positive, this is the Optimal Tableau. If not, go back to step If you obtain a Optimal tableau, then the linear programming problem has a maximum solution, which is given by the entry in the lower-right corner of the tableau.

18. Pivot
Pivot Column: The column of the tableau representing the variable to be entered into the solution mix. Pivot Row: The row of the tableau representing the variable to be replaced in the solution mix. Pivot Number: The element in both the pivot column and the pivot row.

19. Simplex Tableau
Most real-world problems are too complex to solve graphically. They have too many corners to evaluate, and the algebraic solutions are lengthy. A simplex tableau is a way to systematically evaluate variable mixes in order to find the best one.
P.S: Commercial software for LPs do not use Tableaus. They implement another variation called “Revised Simplex”, where by it is possible to take advantage of the sparsity, make the algorithm numerically stable. They also transition from BFS to BFS in the most computationally efficient way, making use of Matrix algebra.

20. Initial Simplex Tableau
All variables
Solution
Basic variables
coefficients

22. EXAMPLE 2
The Cannon Hill furniture Company produces tables and chairs. Each table takes four hours of labor from the carpentry department and two hours of labor from the finishing department. Each chair requires three hours of carpentry and one hour of finishing. During the current week, 240 hours of carpentry time are available and 100 hours of finishing time. Each table produced gives a profit of $70 and each chair a profit of $50. How many chairs and tables should be made?

23. All information about example
STEP 1
All information about example
Resource
Table s ( )
Chairs ( )
Constraints
Carpentry (hr) 4 3
240
Finishing (hr) 2 1
100
Unit Profit
$70
$50
Objective Function
Carpentry Constraint
Finishing Constraint
Non-negativity conditions

24. The first step of the simplex method requires that each inequality be converted into an equation. ”less than or equal to” inequalities are converted to equations by including slack variables. Suppose carpentry hours and finishing hours remain unused in a week. The constraints become; or As unused hours result in no profit, the slack variables can be included in the objective function with zero coefficients:

25. The problem can now be considered as solving a system of 3 linear equations involving the 5 variables in such a way that P has the maximum value; Now, the system of linear equations can be written in matrix form or as a 3x6 augmented matrix. The initial tableau is;

26. STEP 2 Basic Variables x1 x2 S1 S2 P Right Hand Side 4 3 1 240 2 100
240 2
100
-70
-50
The tableau represents the initial solution;
The slack variables S1 and S2 form the initial solution mix. The initial solution assumes that all avaliable hours are unused. i.e. The slack variables take the largest possible values.

27. Variables in the solution mix are called basic variables
Variables in the solution mix are called basic variables. Each basic variables has a column consisting of all 0’s except for a single 1. all variables not in the solution mix take the value 0. The simplex process, a basic variable in the solution mix is replaced by another variable previously not in the solution mix. The value of the replaced variable is set to 0.

28. STEP 3
Select the pivot column (determine which variable to enter into the solution mix). Choose the column with the “most negative” element in the objective function row.
Basic Variables x1 x2 S1 S2 P
Right hand side 4 3 1
240 2
100
-70
-50
Pivot column
x1 should enter into the solution mix because each unit of x1 (a table) contributes a profit of $70 compared with only $50 for each unit of x1 (a chair)

29. Step 4
Yes, There are positive elements in the pivot column above the last row. We can go on step 5. We next calculate, ϴ , the step size.

30. STEP 5 Basic Variables x1 x2 S1 S2 P Right hand side 4 3 1 240 2 100
Select the pivot row (determine which variable to replace in the solution mix). Divide the last element in each row by the corresponding element in the pivot column. The pivot row is the row with the smallest non-negative result.
Enter
Basic Variables x1 x2 S1 S2 P
Right hand side 4 3 1
240 2
100
-70
-50
Exit
Pivot row
Pivot column
Pivot number

31. Basic Variables x1 x2 S1 S2 P Right hand side
Should be replaced by x1 in the solution mix. 60 tables can be made with 240 unused carpentry hours but only 50 tables can be made with 100 finishing hours. Therefore we decide to make 50 tables. Now calculate new values for the pivot row. Divide every number in the row by the pivot number.
Basic Variables x1 x2 S1 S2 P
Right hand side 4 3 1
240
1/2 50
-70
-50

32. Basic Variables x1 x2 S1 S2 P Right hand side
Use elementary row operations to make all numbers in the pivot column equal to 0 except for the pivot number which remains as 1.
Basic Variables x1 x2 S1 S2 P
Right hand side 1 -2 40
1/2 50
-15 35
3500
If 50 tables are made, then the unused carpentry hours are reduced by 200 hours (4 h/table multiplied by 50 tables); the value changes from 240 hours to 40 hours. Making 50 tables results in the profit being increased by $3500; the value changes from $0 to $3500.

33. Basic Variables x1 x2 S1 S2 P Right hand side
In this case, Now repeat the steps until there are no negative numbers in the last row. Select the new pivot column. x2 should enter into the solution mix. Select the new pivot row. S1 should be replaced by x2 in the solution mix.
Enter
Basic Variables x1 x2 S1 S2 P
Right hand side 1 -2 40
1/2 50
-15 35
3500
Exit
New pivot row
New pivot column

34. Basic Variables x1 x2 S1 S2 P Right hand side
Calculate new values for the pivot row. As the pivot number is already 1, there is no need to calculate new values for the pivot row.
Use row operations to make all numbers in the pivot column equal to except for the pivot number.
Basic Variables x1 x2 S1 S2 P
Right hand side 1 -2 40
-1/2
3/2 30 15 5
4100

35. If 40 chairs are made, then the number of tables are reduced by 20 tables (1/2 table/chair multiplied by 40 chairs); the value changes from 50 tables to 30 tables. The replacement of 20 tables by 40 chairs results in the profit being increased by $600; the value changes from $3500 to $4100. As the last row contains no negative numbers, this solution gives the maximum value of P.

36. Result
This simplex tableau represents the optimal solution to the LP problem and is interpreted as: and profit P=$4100 The optimal solution (maximum profit to be made) is to make 30 tables and 40 chairs for a profit of $4100.

37. Starting BFS for the Simplex algorithm
In the Maximization problem, subject to <= constraints, we conveniently used the slack variables as the starting BFS, since their Columns form the identity matrix. (in other words, origin is a feasible solution.)
In many problems this may not be possible, since the origin may not be part of the feasible set. We then start the LP with an artificial BFS, and eventually eliminate these artificial variables.

39. EX: An LP whose feasible region does not contain the Origin.
So we need an artificial BFS to get started. C
1000
600
450
Feasible
Region T

40. Possible outcomes of Solving an LP
Given any LP, there are 3 possibilities.
The LP is Infeasible (feasible region is empty)
The LP is Feasible, but has NO optimal solution because the objective is Unbounded.
The LP has an Optimal Solution
Optimal solution is Unique.
Has multiple optimal solutions.

41. Special Situations in LP
Infeasibility – when no feasible solution exists (there is no feasible region) Example: x < 10 x > 15 Detecting Infeasibility: When you solve Phase 1 of an LP (with artificial variables), if the optimal solution contains artificial variables, then the problem is Infeasible.

42. Special Situations in LP
Unbounded Solutions – when nothing prevents the solution from becoming infinitely large
Max 2T + 2C
Subject to:
2T + 3C > 6
T, C > 0 C 2 1
Direction of solution
Real World Problems can never have unbounded solutions. If you get this condition, it is usually because the problem formulation is incorrect. T

43. Detecting Unboundedness
In order to decide which variable leaves the basic solution, we compute θ, the step size.
If in the pivot column, there are no positive entries to compute θ, then it shows the LP is unbounded.
the pivot row must be selected so that all the other basic variables remain positive. A calculation shows that this occurs when the resulting value of the entering variable is at a minimum. In other words, if the pivot column is c, then the pivot row r is chosen so that br /arc is the minimum over all r , where arc > 0. If there no positive entries in the pivot column, then the LP is unbounded.

44. Special Situation in LP
Alternate Optimal Solutions – when there is more than one optimal solution C 10 6
Max 2T + 2C
Subject to:
T + C < 10
T < 5
C < 6
T, C > 0
2T + 2C = 20
All points on Red segment are optimal T

45. Degeneracy and Cycling in Simplex Method
Sometimes, it may be possible that an extreme point has more than one algebraic representation. In n space, if more than n hyperplanes intersect at that point, then the vertex is called “degenerate, which could in theory lead to Simplex method Cycling infinitely.
In practice, Cycling does not seem to occur, so the interest in degenerate vertices is mostly for theoretical reasons.
There is a variation of the simplex method, known as Lexico- Simplex method, which ensures that Cycling can never occur.

46. Software for Solving LPs
There is a wide choice of software packages for solving LPs, including many open source packages.
High-end packages (ex: CPLEX from IBM ) can solve problems with tens of thousands of variables
Microsoft Excel has a LP solver built in, which is very easy to use and we will learn how to use it.

47. Example-2
A farmer owns a 100 acre farm and plans to plant at most three crops. The seed for crops A,B, and C costs $40, $20, and $30 per acre, respectively. A maximum of $3200 can be spent on seed. Crops A,B, and C require 1,2, and 1 workdays per acre, respectively, and there are maximum of 160 workdays available. If the farmer can make a profit of $100 per acre on crop A, $300 per acre on crop B, and $200 per acre on crop C, how many acres of each crop should be planted to maximize profit? Formulate the above problem as a LP. Solve it using Solver in Excel. Then interpret the solution. What should the former do?

48. Ex3: Media Selection for Advertising
Win Big gambling club promotes gambling junkets from a large Midwestern city to casinos in the Bahamas. The club has budgeted up to $8000 per week for local advertising. The money is to be allocated among 4 promotional media. TV spots, newspaper ads and two types of radio advertisements. Win- Big’s goal is to reach the largest possible high-potential audience through the various media. The table below shows the number of potential gamblers reached by one advertisement in each of the four media. The cost per advertisement is also shown in the table. Win Big requires that at least 5 radio spots be placed each week. To ensure a broad campaign, management also insists that no more than $1800 is spent on radio ads per week.

49. Problem Formulation

50. Duality in Linear Programming
Linear programming problems exist in pairs. That is in linear programming problem, every maximization problem is associated with a minimization problem. Conversely, associated with every minimization problem is a maximization problem. Once we have a problem with its objective function as maximization, we can write by using duality relationship of linear programming problems, its minimization version. The original linear programming problem is known as primal problem, and the derived problem is known as dual problem.

51. Primal dual problems
Linear programming problem specified with equalities
Linear programming in standard form
Dual problem
Primal problem y x
Primal problem
Dual problem y x

53. Duality theorems
It is easy to show that we can move from one pair of primal-dual problems to the other.
It is also easy to show that the dual of the dual problem is the primal problem.
Thus we are showing the duality theorems using the pair where the primal primal is in the standard form:
primal
Dual

54. Duality theorems Weak duality theorem
If (i.e., x is feasible for the primal problem) and
if (i.e., y is feasible for the dual problem), then
Proof Indeed,

55. Duality theorems Corollary If and , and if
, then x* and y* are optimal solutions for the primal and dual problems, respectively..
Proof It follows from the weak duality theorem that for any feasible solution x of the primal problem
Consequently x* is an optimal solution of the primal problem.
We can show the optimality of y* for the dual problem using a similar
proof.

56. Duality theorems
Strong duality theorem If one of the two primal or dual problem has a finite value optimal solution, then the other problem has the same property, and the optimal values of the two problems are equal. If one of the two problems is unbounded, then the feasible domain of the other problem is empty.

57. Complementary slackness theory
We now introduce new necessary and sufficient conditions for a pair of feasible solutions of the primal and of the dual to be optimal for each of these problems.
Consider first the following pair of primal-dual problems.
primal
Dual x

58. Complementary slackness theory
Complementary slackness theorem 1
Let x and y be feasible solution for the primal and the dual, respectively. Then x and y are optimal solutions for these problems if and only if for all
j = 1,2,…,n

59. Complementary slackness theory
Now consider the other pair of primal-dual problems
Complementary slackness theorem 2
Let x and y be feasible solution for the primal and dual problems, recpectively. Then x and y are opyimal solutions of these problems
if and only if
for all j = 1,2,…,n for all i=1,2,…,m y x

60. Summary
Primal
Dual
(a) Maximize.
Minimize
(b) Objective Function.
Right hand side.
(c) Right hand side.
Objective function.
(d) i th row of input-output coefficients.
i th column of input output coefficients.
(e) j th column of input-output coefficients.
j the row of input-output coefficients.