Algorithms CISC 4080/CISC 5825 Linear Programming, Simplex Method and Duality Theory Dr Kamesam Graduate School Of Business Fordham University

Outline Elementary Row Operations Graphical Approach to solving LP Solution of Maximization Problems with Simplex Method Special Situations in the Simplex method. Simplex method with artificial variables. Solving LP with Excel Solver. Duality You may find the following free chapter helpful http://college.cengage.com/mathematics/larson/elementary_linear/4e/shared/downloads/c09s3.pdf

Background Elementary Row Operations Given a system of linear equations, Elementary row operations lead to an equivalent system of linear equations, in the sense that the solution set of both systems of equations are the same. Elementary Row Operations we will use Multiply both sides of an equation with a nonzero constant Add a multiple of one equation to another equation

Standard Maximization Problems in Standard Form A linear programming problem is said to be a standard maximization problem in standard form if its mathematical model is of the following form: Maximize the objective function Subject to problem constraints of the form With non-negative constraints

Slack Variables “A mathematical representation of surplus resources.” In real life problems, it’s unlikely that all resources will be used completely, so there usually are unused resources. Slack variables represent the unused resources between the left-hand side and right-hand side of each inequality.

Solve the following LP Graphically We shall next, learn how the Simplex method works on this problem

LP Characteristics Feasible Region: is a convex set with extreme points. An extreme point is a point which cannot be expressed as a convex combination of other points in the set. Corner Point Property: An optimal solution must lie at one or more corner points (extreme points) Optimal Solution: The corner point with the best objective function value is optimal

The Simplex Method The geometric method of solving linear programming problems was presented before. The graphical method is useful only for problems involving two decision variables and relatively few problem constraints. What happens when we need more decision variables and more problem constraints? We use an algebraic method called the simplex method, which was developed by George B. DANTZIG (1914-2005) in 1947 while on assignment with the U.S. Department of the air force.

Algorithm for LP The graphical method can only be used for problems in 2 variables. However, it gives us insight Practical applications of LPs can be very large (tens of thousands of variables). So we need a robust and efficient algorithm, that can also take advantage of sparsity in the data Answer: Simplex Method

Basic and Nonbasic Variables Basic variables are selected arbitrarily with the restriction that there be as many basic variables as there are equations. The remaining variables are non-basic variables. This system has two equations, we can select any two of the four variables as basic variables. The remaining two variables are then non-basic variables. A solution found by setting the two non-basic variables equal to 0 and solving for the two basic variables is a basic solution. If a basic solution has no negative values, it is a basic feasible solution.

Extreme Points and BFS The Feasible Region has Extreme points (Corner Points) Note that there is a one to one correspondence between Corner Points and BFS. So, a BFS is an algebraic Characterization of the geometric concept, extreme point. In order to develop effective algorithms, geometric insight is helpful, but an algebraic characterization is a must.

Enumerating all BFS to find Optimal Solution So, can we turn this into an algorithm ? Unfortunately, NO For a system with n variables and m constraints, there are nCm possible Basic solutions. Even for a very small Linear Program with n = 100 and m = 40, there are 1.37798E+28 Basic Solutions. Although the Simplex Algorithm does move from BFS to BFS, it does so cleverly, ensuring that the objective function keeps increasing.

SIMPLEX METHOD Simplex algorithm for standard maximization problems Step-1 Write the standard maximization problem in standard form, introduce slack variables to form the initial system, and write the initial tableau. Step-3 Select the pivot column Step-5 Select the pivot element and perform the pivot operation Step 4 Are there any positive elements in the pivot column above the dashed line? Step 2 Are there any negative indicators in the bottom row? STOP The optimal solution has been found. STOP The linear programming problem has no optimal solution. UNBOUNDED Simplex algorithm for standard maximization problems

To solve a linear programming problem in standard form, use the following steps. 1- Convert each inequality in the set of constraints to an equation by adding slack variables. 2- Create the initial simplex tableau. 3- Select the pivot column. ( The column with the “most negative value” element in the last row.) 4- Select the pivot row. (The row with the smallest non-negative result when the last element in the row is divided by the corresponding in the pivot column.) 5-Use elementary row operations to calculate new values for the pivot row so that the pivot is 1 (Divide every number in the row by the pivot number.) 6- Use elementary row operations to make all numbers in the pivot column equal to 0 except for the pivot number. If all entries in the bottom row are zero or positive, this is the Optimal Tableau. If not, go back to step 3. 7- If you obtain a Optimal tableau, then the linear programming problem has a maximum solution, which is given by the entry in the lower-right corner of the tableau.

Pivot Pivot Column: The column of the tableau representing the variable to be entered into the solution mix. Pivot Row: The row of the tableau representing the variable to be replaced in the solution mix. Pivot Number: The element in both the pivot column and the pivot row.

Simplex Tableau Most real-world problems are too complex to solve graphically. They have too many corners to evaluate, and the algebraic solutions are lengthy. A simplex tableau is a way to systematically evaluate variable mixes in order to find the best one. P.S: Commercial software for LPs do not use Tableaus. They implement another variation called “Revised Simplex”, where by it is possible to take advantage of the sparsity, make the algorithm numerically stable. They also transition from BFS to BFS in the most computationally efficient way, making use of Matrix algebra.

Initial Simplex Tableau All variables Solution Basic variables coefficients

EXAMPLE 2 The Cannon Hill furniture Company produces tables and chairs. Each table takes four hours of labor from the carpentry department and two hours of labor from the finishing department. Each chair requires three hours of carpentry and one hour of finishing. During the current week, 240 hours of carpentry time are available and 100 hours of finishing time. Each table produced gives a profit of $70 and each chair a profit of $50. How many chairs and tables should be made?

All information about example STEP 1 All information about example Resource Table s ( ) Chairs ( ) Constraints Carpentry (hr) 4 3 240 Finishing (hr) 2 1 100 Unit Profit $70 $50 Objective Function Carpentry Constraint Finishing Constraint Non-negativity conditions

The first step of the simplex method requires that each inequality be converted into an equation. ”less than or equal to” inequalities are converted to equations by including slack variables. Suppose carpentry hours and finishing hours remain unused in a week. The constraints become; or As unused hours result in no profit, the slack variables can be included in the objective function with zero coefficients:

The problem can now be considered as solving a system of 3 linear equations involving the 5 variables in such a way that P has the maximum value; Now, the system of linear equations can be written in matrix form or as a 3x6 augmented matrix. The initial tableau is;

STEP 2 Basic Variables x1 x2 S1 S2 P Right Hand Side 4 3 1 240 2 100 240 2 100 -70 -50 The tableau represents the initial solution; The slack variables S1 and S2 form the initial solution mix. The initial solution assumes that all avaliable hours are unused. i.e. The slack variables take the largest possible values.

Variables in the solution mix are called basic variables Variables in the solution mix are called basic variables. Each basic variables has a column consisting of all 0’s except for a single 1. all variables not in the solution mix take the value 0. The simplex process, a basic variable in the solution mix is replaced by another variable previously not in the solution mix. The value of the replaced variable is set to 0.

STEP 3 Select the pivot column (determine which variable to enter into the solution mix). Choose the column with the “most negative” element in the objective function row. Basic Variables x1 x2 S1 S2 P Right hand side 4 3 1 240 2 100 -70 -50 Pivot column x1 should enter into the solution mix because each unit of x1 (a table) contributes a profit of $70 compared with only $50 for each unit of x1 (a chair)

Step 4 Yes, There are positive elements in the pivot column above the last row. We can go on step 5. We next calculate, ϴ , the step size.

STEP 5 Basic Variables x1 x2 S1 S2 P Right hand side 4 3 1 240 2 100 Select the pivot row (determine which variable to replace in the solution mix). Divide the last element in each row by the corresponding element in the pivot column. The pivot row is the row with the smallest non-negative result. Enter Basic Variables x1 x2 S1 S2 P Right hand side 4 3 1 240 2 100 -70 -50 Exit Pivot row Pivot column Pivot number

Basic Variables x1 x2 S1 S2 P Right hand side Should be replaced by x1 in the solution mix. 60 tables can be made with 240 unused carpentry hours but only 50 tables can be made with 100 finishing hours. Therefore we decide to make 50 tables. Now calculate new values for the pivot row. Divide every number in the row by the pivot number. Basic Variables x1 x2 S1 S2 P Right hand side 4 3 1 240 1/2 50 -70 -50

Basic Variables x1 x2 S1 S2 P Right hand side Use elementary row operations to make all numbers in the pivot column equal to 0 except for the pivot number which remains as 1. Basic Variables x1 x2 S1 S2 P Right hand side 1 -2 40 1/2 50 -15 35 3500 If 50 tables are made, then the unused carpentry hours are reduced by 200 hours (4 h/table multiplied by 50 tables); the value changes from 240 hours to 40 hours. Making 50 tables results in the profit being increased by $3500; the value changes from $0 to $3500.

Basic Variables x1 x2 S1 S2 P Right hand side In this case, Now repeat the steps until there are no negative numbers in the last row. Select the new pivot column. x2 should enter into the solution mix. Select the new pivot row. S1 should be replaced by x2 in the solution mix. Enter Basic Variables x1 x2 S1 S2 P Right hand side 1 -2 40 1/2 50 -15 35 3500 Exit New pivot row New pivot column

Basic Variables x1 x2 S1 S2 P Right hand side Calculate new values for the pivot row. As the pivot number is already 1, there is no need to calculate new values for the pivot row. Use row operations to make all numbers in the pivot column equal to except for the pivot number. Basic Variables x1 x2 S1 S2 P Right hand side 1 -2 40 -1/2 3/2 30 15 5 4100

If 40 chairs are made, then the number of tables are reduced by 20 tables (1/2 table/chair multiplied by 40 chairs); the value changes from 50 tables to 30 tables. The replacement of 20 tables by 40 chairs results in the profit being increased by $600; the value changes from $3500 to $4100. As the last row contains no negative numbers, this solution gives the maximum value of P.

Result This simplex tableau represents the optimal solution to the LP problem and is interpreted as: and profit P=$4100 The optimal solution (maximum profit to be made) is to make 30 tables and 40 chairs for a profit of $4100.

Starting BFS for the Simplex algorithm In the Maximization problem, subject to <= constraints, we conveniently used the slack variables as the starting BFS, since their Columns form the identity matrix. (in other words, origin is a feasible solution.) In many problems this may not be possible, since the origin may not be part of the feasible set. We then start the LP with an artificial BFS, and eventually eliminate these artificial variables.

EX: An LP whose feasible region does not contain the Origin. So we need an artificial BFS to get started. C 1000 600 450 Feasible Region 0 100 500 800 T

Possible outcomes of Solving an LP Given any LP, there are 3 possibilities. The LP is Infeasible (feasible region is empty) The LP is Feasible, but has NO optimal solution because the objective is Unbounded. The LP has an Optimal Solution Optimal solution is Unique. Has multiple optimal solutions.

Special Situations in LP Infeasibility – when no feasible solution exists (there is no feasible region) Example: x < 10 x > 15 Detecting Infeasibility: When you solve Phase 1 of an LP (with artificial variables), if the optimal solution contains artificial variables, then the problem is Infeasible.

Special Situations in LP Unbounded Solutions – when nothing prevents the solution from becoming infinitely large Max 2T + 2C Subject to: 2T + 3C > 6 T, C > 0 C 2 1 Direction of solution Real World Problems can never have unbounded solutions. If you get this condition, it is usually because the problem formulation is incorrect. 0 1 2 3 T

Detecting Unboundedness In order to decide which variable leaves the basic solution, we compute θ, the step size. If in the pivot column, there are no positive entries to compute θ, then it shows the LP is unbounded. the pivot row must be selected so that all the other basic variables remain positive. A calculation shows that this occurs when the resulting value of the entering variable is at a minimum. In other words, if the pivot column is c, then the pivot row r is chosen so that br /arc is the minimum over all r , where arc > 0. If there no positive entries in the pivot column, then the LP is unbounded.

Special Situation in LP Alternate Optimal Solutions – when there is more than one optimal solution C 10 6 Max 2T + 2C Subject to: T + C < 10 T < 5 C < 6 T, C > 0 2T + 2C = 20 All points on Red segment are optimal 0 5 10 T

Degeneracy and Cycling in Simplex Method Sometimes, it may be possible that an extreme point has more than one algebraic representation. In n space, if more than n hyperplanes intersect at that point, then the vertex is called “degenerate, which could in theory lead to Simplex method Cycling infinitely. In practice, Cycling does not seem to occur, so the interest in degenerate vertices is mostly for theoretical reasons. There is a variation of the simplex method, known as Lexico- Simplex method, which ensures that Cycling can never occur.

Software for Solving LPs There is a wide choice of software packages for solving LPs, including many open source packages. High-end packages (ex: CPLEX from IBM ) can solve problems with tens of thousands of variables Microsoft Excel has a LP solver built in, which is very easy to use and we will learn how to use it.

Example-2 A farmer owns a 100 acre farm and plans to plant at most three crops. The seed for crops A,B, and C costs $40, $20, and $30 per acre, respectively. A maximum of $3200 can be spent on seed. Crops A,B, and C require 1,2, and 1 workdays per acre, respectively, and there are maximum of 160 workdays available. If the farmer can make a profit of $100 per acre on crop A, $300 per acre on crop B, and $200 per acre on crop C, how many acres of each crop should be planted to maximize profit? Formulate the above problem as a LP. Solve it using Solver in Excel. Then interpret the solution. What should the former do?

Ex3: Media Selection for Advertising Win Big gambling club promotes gambling junkets from a large Midwestern city to casinos in the Bahamas. The club has budgeted up to $8000 per week for local advertising. The money is to be allocated among 4 promotional media. TV spots, newspaper ads and two types of radio advertisements. Win- Big’s goal is to reach the largest possible high-potential audience through the various media. The table below shows the number of potential gamblers reached by one advertisement in each of the four media. The cost per advertisement is also shown in the table. Win Big requires that at least 5 radio spots be placed each week. To ensure a broad campaign, management also insists that no more than $1800 is spent on radio ads per week.

Problem Formulation

Duality in Linear Programming Linear programming problems exist in pairs. That is in linear programming problem, every maximization problem is associated with a minimization problem. Conversely, associated with every minimization problem is a maximization problem. Once we have a problem with its objective function as maximization, we can write by using duality relationship of linear programming problems, its minimization version. The original linear programming problem is known as primal problem, and the derived problem is known as dual problem.

Primal dual problems Linear programming problem specified with equalities Linear programming in standard form Dual problem Primal problem y x Primal problem Dual problem y x

Duality theorems It is easy to show that we can move from one pair of primal-dual problems to the other. It is also easy to show that the dual of the dual problem is the primal problem. Thus we are showing the duality theorems using the pair where the primal primal is in the standard form: primal Dual

Duality theorems Weak duality theorem If (i.e., x is feasible for the primal problem) and if (i.e., y is feasible for the dual problem), then Proof Indeed,

Duality theorems Corollary If and , and if , then x* and y* are optimal solutions for the primal and dual problems, respectively.. Proof It follows from the weak duality theorem that for any feasible solution x of the primal problem Consequently x* is an optimal solution of the primal problem. We can show the optimality of y* for the dual problem using a similar proof.

Duality theorems Strong duality theorem If one of the two primal or dual problem has a finite value optimal solution, then the other problem has the same property, and the optimal values of the two problems are equal. If one of the two problems is unbounded, then the feasible domain of the other problem is empty.

Complementary slackness theory We now introduce new necessary and sufficient conditions for a pair of feasible solutions of the primal and of the dual to be optimal for each of these problems. Consider first the following pair of primal-dual problems. primal Dual x

Complementary slackness theory Complementary slackness theorem 1 Let x and y be feasible solution for the primal and the dual, respectively. Then x and y are optimal solutions for these problems if and only if for all j = 1,2,…,n

Complementary slackness theory Now consider the other pair of primal-dual problems Complementary slackness theorem 2 Let x and y be feasible solution for the primal and dual problems, recpectively. Then x and y are opyimal solutions of these problems if and only if for all j = 1,2,…,n for all i=1,2,…,m y x

Summary Primal Dual (a) Maximize. Minimize (b) Objective Function. Right hand side. (c) Right hand side. Objective function. (d) i th row of input-output coefficients. i th column of input output coefficients. (e) j th column of input-output coefficients. j the row of input-output coefficients.