Linear Programming (Linear optimization) Reference: Linear Programming by Chvatal, The Allocation of Resources by Linear Programming by Bob Bland in Scientific American Some of the slides are adapted from lecture notes of Kevin Wayne

Linear Programming Significance Essential tool for optimal allocation of scarce resources, among a number of competing activities. Powerful model generalizes many classic problems: shortest path, max flow, multicommodity flow, MST, matching Ranked among most important scientific advances of 20th century. accounts for a major proportion of all scientific computation Helps find "good" solutions to NP-hard optimization problems. Ex: LP accounts for 10% of Exxon's computing load. Schedule drilling operations. Blend gasoline optimally. Allocate crude oil among refineries. Distribute products. Plan business strategy.

Brewery Problem Small brewery produces ale and beer. Production limited by scarce resources: corn, hops, barley malt. Recipes for ale & beer require different proportions of resources. How can brewer maximize profits??? Devote all resources to beer: 32 barrels of beer  Devote all resources to ale: 34 barrels of ale  7½ barrels of ale, 29½ barrels of beer  12 barrels of ale, 28 barrels of beer  Beverage Corn (pounds) Malt (pounds) Hops (ounces) Beer 15 20 4 Ale 5 35 Profit ($) 23 13 Quantity 480 1190 160 Ask the students to work on the problem $736 $442. $776. $800.

Brewery Problem Objective function Constrains

Brewery Problem

Brewery Problem: Feasible Region Hops 4A + 4B  160 Malt 35A + 20B  1190 B Corn 5A + 15B  480 (34, 0) (0, 32) (12, 28) (26, 14) (0, 0) Beer Ale A

Brewery Problem: Objective Function 13A + 23B = 1600 Profit 13A + 23B = 800 13A + 23B = 442 (0, 32) (12, 28) (26, 14) Beer (0, 0) Ale (34, 0)

Brewery Problem: Geometry Brewery problem observation. Regardless of coefficients of linear objective function, there exists an optimal solution that is an extreme point. Ale Beer (34, 0) (0, 32) (12, 28) (26, 14) (0, 0) Extreme points

Linear Programming LP "standard" form. Input data: rational numbers cj, bi, aij . Maximize linear objective function. Subject to linear inequalities. Solving simulataneous linear equations (Ax = b) can be accomplished using Gaussian elimination. LP is a generalization with inequalities instead of equalities.

LP: Geometry Geometry. Forms an n-dimensional polyhedron. Convex: if y and z are feasible solutions, then so is ½y + ½z. Extreme point: feasible solution x that can't be written as ½y + ½z for any two distinct feasible solutions y and z. z y Convex Not convex Extreme points

LP: Geometry Extreme Point Theorem. If there exists an optimal solution to standard form LP (P), then there exists one that is an extreme point. Only need to consider finitely many possible solutions. ? Note: it is a common misperception that the extreme point theorem is true for LPs in non-standard form, as some polyhedra have no vertices (min x : x >= 0)

Brewery Problem How can brewer maximize profits??? Beverage Corn (pounds) Malt (pounds) Hops (ounces) Beer 15 20 4 Ale 5 35 Profit ($) 23 13 Quantity 480 1190 160 How can brewer maximize profits??? Ale Beer (34, 0) (0, 32) (12, 28) (26, 14) (0, 0) Extreme points Ask the students to work on the problem

LP: Algorithms Simplex (George Dantzig 1947) Developed shortly after WWII in response to logistical problems used for 1948 Berlin airlift. One of the top 10 algorithms in last century. Finite (exponential) complexity, but no polynomial implementation known. Practical solution method that moves from one extreme point to a neighboring extreme point. As a graduate student at the University of California Berkeley in 1939, Dantzig arrived late to class one day and copied two problems from a blackboard. After struggling with what he thought was a difficult homework assignment, he submitted his work to the eminent statistician Jerzy Neyman. Six weeks later on a Sunday at 8 AM, Neyman excitedly awoke Dantzig to say he had written an introduction to Dantzig's paper. It turned out that Dantzig had found solutions to two famous, previously unsolved statistical problems. Moral of the Story: if you come in late to class, you must solve a previously unsolved problem.

Slack form (Normal form) of LP LP in Standard form: LP in slack form

Brewery Problem Standard form: Slack form (normal form):

(on the left side of the eqs) Brewery Problem Standard form: Slack form (normal form): nonbasic variables basic variables (on the left side of the eqs)

Brewery Problem Rewrite it as: Step1: Find a basic feasible solution Ale Beer (34, 0) (0, 32) (12, 28) (26, 14) (0, 0) Extreme points Rewrite it as: Step1: Find a basic feasible solution (a feasible solution in which at least nm of xj (nonbasic var are zeros) n: # variables, m: # of constraints

Brewery Problem Step1: Find a basic feasible solution Step2: Test for optimality (if the coeffs of Z are all negative, Z can not be increased.) Step3: Choose a pivot to find a better BFS (try to increase Z) Pick B, b/c each unit increase in B increases obj values by $23. Pick 1st equ, b/c minimum ratio rule (to ensure the basic solution remains feasible):

Brewery Problem Step3: Choose a pivot for a better BFS Solve the 1st equ for B: Rewrite LP: BFS:

Brewery Problem Step4: Repeat step 2,3,4 BFS: Z=800 optimal solution Test for optimality Pick pivot A Pick 2nd equ (min ratio rule) Solve the equ for A: Rewrite LP: Ale Beer (34, 0) (0, 32) (12, 28) (26, 14) (0, 0) BFS: Z=800 optimal solution

Applications

LP Application: Maximum Flow

LP Application: Min-cost Flow Given directed graph G=(V,E), s, t V Capacity c(u,v)0, (u,v)E Cost a(u,v)0, (u,v)E Problem: Given a flow target d, from s to t. Minimize the total cost:

LP Application: Multicommodity Flow I Multicommodity flow. Given a network G = (V, E) with edge capacities c(e)  0, set of commodities K, and each is specified by (si, ti, di). Problem. ? Find if there is a flow that satisfies all of the demand.

LP Application: Multicommodity Flow II Multicommodity flow. Given a network G = (V, E) with edge capacities ck(u,v)  0, edge costs ak(u,v)  0, set of commodities K, and demand dk(v) for commodity k at node v, find a minimum cost flow that satisfies all of the demand. Applications. Transportation networks. Communication networks. VLSI design.

LP Application: Weighted Bipartite Matching Matching problem. Given a complete bipartite network Kn,n and edge weights cij, find a perfect matching of minimum weight. Birkhoff-von Neumann theorem (1946, 1953). All extreme points of the above polyhedron are {0-1}-valued. Corollary. Can solve assignment problem using LP techniques since LP algorithms return optimal solution that is an extreme point.

Transshipment Problem

Brewery Problem How can brewer maximize profits??? Beverage Corn (pounds) Malt (pounds) Hops (ounces) Beer 15 20 4 Ale 5 35 Profit ($) 23 13 Quantity 480 1190 160 How can brewer maximize profits??? Ale Beer (34, 0) (0, 32) (12, 28) (26, 14) (0, 0) Extreme points Ask the students to work on the problem

LP Example Maximize: P = 10x + 12y subject to the constraints: x + y ≤ 40 (i) x + 2y ≤ 75 (ii) x ≥ 0, y ≥ 0. Step 1: Introduce slack variables : (i): x + y + s = 40 s ≥ 0 (ii): x + 2y + t = 75 t ≥ 0

Tableau Format Last column Pivotal column Shaded cells ≥ 0 The aim is to solve the equations by combining rows together. The solution is reached when all entries in the first row (except possibly the value in the last column) are non-negative. Most negative entry in the objective function : -12 in column y

Tableau Format Pivotal column pivot

Tableau Format P x y s t l Equation Ratio 1 - 4 6 450 ½ 2.5 (5) = (2) pivot P x y s t l Equation Ratio 1 - 4 6 450 (4)=(1)+12×(6) 450/ 4= 112.5 ½ 2.5 (5) = (2) – (6) 2.5/0.5 = 5 37.5 (6) = (3)/2 37.5/0.5 = 75 Now, get 0 entries elsewhere in the pivotal column.

Tableau Format The optimal solution: x = 5, y = 35, P = 470.

Questions to ask Remarkable property of Simplex method: But, Typically it requires less than 2(m+n) pivots to attain optimality. But, How do we know LP is feasible? What if the initial basic solution is not feasible? Is it possible that Simplex method never terminates? How do we know Simplex method returns the optimal solution when it does terminate?

Initial Basic Feasible Solution Example: Basic solution: x3 = 2  2x1+x2 x4 =  4  x1+5x2 Basic solution is not feasible! Does LP have a feasible solution?

Auxiliary LP Find a slack form for which the basic solution is feasible. P is feasible if and only if the optimal objective value of Paux = 0. A feasible solution to Paux will be a feasible solution to P. proof: …

Example Try BS: x0= x1= x2=0 → x4<0!, ? One step pivoting: swap the roles of x0 and x4. Swap x0 with most negative variable (x4). Only one step pivot is needed. Solve 2nd eq. for x0: x0= 4 + x1 5x2+ x4 Rewrite LP: not feasible Solve it

Example x0 = 0

Questions to ask How do we know LP is feasible? What if the initial basic solution is not feasible? Is it possible that Simplex method never terminates? How do we know Simplex method returns the optimal solution when it does terminate?

Complexity Upper Bound How many possible slack forms are there? Ale Beer (34, 0) (0, 32) (12, 28) (26, 14) (0, 0) Extreme points Step1: Find a basic feasible solution Step2: … Rewrite it as:

The solution is unbounded.

Complexity Upper Bound How many possible slack forms are there? If the steps are not repeated, Simplex method terminates in at most n+m choose m steps. See sec 29.3 for more rigorous proof.

Complexity Upper Bound How many possible slack forms are there? If the steps are not repeated, Simplex method terminates in at most n+m choose m steps. Lemma 29.6 Bland’s rule: ties in Simplex method are always broken by choosing the variable with the smallest index, then Simplex must terminate.

LP Duality Primal problem. Find a lower bound on optimal value. (x1, x2, x3, x4) = (0, 0, 1, 0)  z*  5. (x1, x2, x3, x4) = (2, 1, 1, 1/3)  z*  15. (x1, x2, x3, x4) = (3, 0, 2, 0)  z*  22. (x1, x2, x3, x4) = (0, 14, 0, 5)  z*  29.

LP Duality Primal problem. Find an upper bound on optimal value. Multiply 2nd inequality by 2: 10x1 + 2x2 + 6x3 + 16x4  110. z* = 4x1 + x2 + 5x3 + 3x4  10x1 + 2x2 + 6x3 + 16x4  110. Adding 2nd and 3rd inequalities: 4x1 + 3x2 + 6x3 + 3x4  58.  z* = 4x1 + x2 + 5x3 + 3x4  4x1 + 3x2 + 6x3 + 3x4  58.

LP Duality Primal problem. Find an upper bound on optimal value. Adding 11 times 1st inequality to 6 times 3rd inequality:  z* = 4x1 + x2 + 5x3 + 3x4  5x1 + x2 + 7x3 + 3x4  29. Recall. (x1, x2, x3, x4) = (0, 14, 0, 5)  z*  29.

What have we done? Find a linear combination of the constrains Try to minimize the right hand side Relate the objective function to the linear combination

LP Duality Primal problem. General idea: add linear combination (y1, y2, y3) of the constraints. Dual problem. Y1*eq1 +y2*eq2+… <= obj to minimize Coefficient of x1 has to be >= coefficient of x1 in the original obj

LP Duality Primal and dual linear programs: given real numbers aij, bi, cj, find values xi, yj that optimize (P) & (D). Duality Theorem (Gale-Kuhn-Tucker 1951, Dantzig-von Neumann 1947). If (P) and (D) are nonempty then max = min. Dual solution provides certificate of optimality Special case: max-flow min-cut theorem. (D) ? Duality theorem first written down by Gale-Kuhn-Tucker, but inspired from conversation between Dantzig and von-Neumann to show in NP and co-NP also need to use the fact that vertex solutions are of polynomial size (e.g., via Kramer's rule)

LP Duality Theorem LP Duality Theorem. For Amn , bm , cn, if (P) and (D) are nonempty then max = min. Reference: notes by Oktay Gunluk and Paul Seymour

LP Weak Duality Proof: Corollary. If cTx = yTb, x and y are optimal. LP Weak Duality. For A  mn , b  m , c  n, if (P) and (D) are nonempty, then max  min. Proof: Suppose x  m is feasible for (P) and y  n is feasible for (D). x  0, yTA  c  yTAx  cTx. y  0, Ax  b  yTAx  yTb combining two inequalities: cTx  yTAx  yTb Corollary. If cTx = yTb, x and y are optimal.

LP Duality Theorem LP Duality Theorem. For Amn , bm , cn, if (P) and (D) are nonempty then max = min. LP Weak Duality max  min  LP Strong Duality min  max (read book) Reference: notes by Oktay Gunluk and Paul Seymour

LP Duality: Economic Interpretation Brewer's problem: find optimal mix of beer and ale to maximize profits. Entrepreneur's problem: Buy resources from brewer at minimum cost. C, H, M = unit price for corn, hops, malt. Brewer won't agree to sell resources if 5C + 4H + 35M < 13. A* = 12 B* = 28 OPT = 800 (D) C* = 1 H* = 2 M* = 0 OPT = 800

Optimal Dual Solution Theorem. Suppose that Simplex returns values on the primal LP problem (A, b, c): Let N: set of nonbasic variable (all going to be zero) B: set of basic variables. Consider the final slack form: Then the optimal dual solution ў is given by Proof: text P882-885.

Questions to ask How do we know LP is feasible? What if the initial basic solution is not feasible? Is it possible that Simplex method never terminates? How do we know Simplex method returns the optimal solution when it does terminate?