1. Linear Programming (Linear optimization)
Reference: Linear Programming by Chvatal, The Allocation of Resources by Linear Programming by Bob Bland in Scientific American
Some of the slides are adapted from lecture notes of Kevin Wayne

2. Linear Programming Significance
Essential tool for optimal allocation of scarce resources, among a number of competing activities.
Powerful model generalizes many classic problems:
shortest path, max flow, multicommodity flow, MST, matching
Ranked among most important scientific advances of 20th century.
accounts for a major proportion of all scientific computation
Helps find "good" solutions to NP-hard optimization problems.
Ex: LP accounts for 10% of Exxon's computing load.
Schedule drilling operations. Blend gasoline optimally.
Allocate crude oil among refineries. Distribute products.
Plan business strategy.

3. Brewery Problem Small brewery produces ale and beer.
Production limited by scarce resources: corn, hops, barley malt.
Recipes for ale & beer require different proportions of resources.
How can brewer maximize profits???
Devote all resources to beer: 32 barrels of beer 
Devote all resources to ale: 34 barrels of ale 
7½ barrels of ale, 29½ barrels of beer 
12 barrels of ale, 28 barrels of beer 
Beverage
Corn (pounds)
Malt (pounds)
Hops (ounces)
Beer 15 20 4
Ale 5 35
Profit ($) 23 13
Quantity
480
1190
160
Ask the students to work on the problem
$736
$442.
$776.
$800.

4. Brewery Problem
Objective function
Constrains

5. Brewery Problem

6. Brewery Problem: Feasible Region
Hops 4A + 4B  160
Malt 35A + 20B  1190 B
Corn 5A + 15B  480
(34, 0)
(0, 32)
(12, 28)
(26, 14)
(0, 0)
Beer
Ale A

7. Brewery Problem: Objective Function
13A + 23B = 1600
Profit
13A + 23B = 800
13A + 23B = 442
(0, 32)
(12, 28)
(26, 14)
Beer
(0, 0)
Ale
(34, 0)

8. Brewery Problem: Geometry
Brewery problem observation. Regardless of coefficients of linear objective function, there exists an optimal solution that is an extreme point.
Ale
Beer
(34, 0)
(0, 32)
(12, 28)
(26, 14)
(0, 0)
Extreme points

9. Linear Programming LP "standard" form.
Input data: rational numbers cj, bi, aij .
Maximize linear objective function.
Subject to linear inequalities.
Solving simulataneous linear equations (Ax = b) can be accomplished using Gaussian elimination. LP is a generalization with inequalities instead of equalities.

10. LP: Geometry Geometry. Forms an n-dimensional polyhedron.
Convex: if y and z are feasible solutions, then so is ½y + ½z.
Extreme point: feasible solution x that can't be written as ½y + ½z for any two distinct feasible solutions y and z. z y
Convex
Not convex
Extreme points

11. LP: Geometry
Extreme Point Theorem. If there exists an optimal solution to standard form LP (P), then there exists one that is an extreme point.
Only need to consider finitely many possible solutions. ?
Note: it is a common misperception that the extreme point theorem is true for LPs in non-standard form, as some polyhedra have no vertices (min x : x >= 0)

12. Brewery Problem How can brewer maximize profits??? Beverage
Corn (pounds)
Malt (pounds)
Hops (ounces)
Beer 15 20 4
Ale 5 35
Profit ($) 23 13
Quantity
480
1190
160
How can brewer maximize profits???
Ale
Beer
(34, 0)
(0, 32)
(12, 28)
(26, 14)
(0, 0)
Extreme points
Ask the students to work on the problem

13. LP: Algorithms Simplex (George Dantzig 1947)
Developed shortly after WWII in response to logistical problems used for 1948 Berlin airlift.
One of the top 10 algorithms in last century.
Finite (exponential) complexity, but no polynomial implementation known.
Practical solution method that moves from one extreme point to a neighboring extreme point.
As a graduate student at the University of California Berkeley in 1939, Dantzig arrived late to class one day and copied two problems from a blackboard. After struggling with what he thought was a difficult homework assignment, he submitted his work to the eminent statistician Jerzy Neyman. Six weeks later on a Sunday at 8 AM, Neyman excitedly awoke Dantzig to say he had written an introduction to Dantzig's paper. It turned out that Dantzig had found solutions to two famous, previously unsolved statistical problems.
Moral of the Story: if you come in late to class, you must solve a previously unsolved problem.

14. Slack form (Normal form) of LP
LP in Standard form:
LP in slack form

15. Brewery Problem
Standard form:
Slack form (normal form):

16. (on the left side of the eqs)
Brewery Problem
Standard form:
Slack form (normal form):
nonbasic variables
basic variables
(on the left side of the eqs)

17. Brewery Problem Rewrite it as: Step1: Find a basic feasible solution
Ale
Beer
(34, 0)
(0, 32)
(12, 28)
(26, 14)
(0, 0)
Extreme points
Rewrite it as:
Step1: Find a basic feasible solution
(a feasible solution in which at least nm of xj (nonbasic var are zeros)
n: # variables, m: # of constraints

18. Brewery Problem Step1: Find a basic feasible solution
Step2: Test for optimality
(if the coeffs of Z are all negative, Z can not be increased.)
Step3: Choose a pivot to find a better BFS
(try to increase Z)
Pick B, b/c each unit increase in B increases obj values by $23.
Pick 1st equ, b/c minimum ratio rule (to ensure the basic solution
remains feasible):

19. Brewery Problem Step3: Choose a pivot for a better BFS
Solve the 1st equ for B:
Rewrite LP:
BFS:

20. Brewery Problem Step4: Repeat step 2,3,4 BFS: Z=800 optimal solution
Test for optimality
Pick pivot A
Pick 2nd equ (min ratio rule)
Solve the equ for A:
Rewrite LP:
Ale
Beer
(34, 0)
(0, 32)
(12, 28)
(26, 14)
(0, 0)
BFS:
Z=800
optimal solution

21. Applications

22. LP Application: Maximum Flow

23. LP Application: Min-cost Flow
Given directed graph G=(V,E), s, t V
Capacity c(u,v)0, (u,v)E
Cost a(u,v)0, (u,v)E
Problem: Given a flow target d, from s to t. Minimize the total cost:

24. LP Application: Multicommodity Flow I
Multicommodity flow. Given a network G = (V, E) with edge capacities c(e)  0, set of commodities K, and each is specified by (si, ti, di).
Problem. ?
Find if there is a flow that satisfies all of the demand.

25. LP Application: Multicommodity Flow II
Multicommodity flow. Given a network G = (V, E) with edge capacities ck(u,v)  0, edge costs ak(u,v)  0, set of commodities K, and demand dk(v) for commodity k at node v, find a minimum cost flow that satisfies all of the demand.
Applications.
Transportation networks.
Communication networks.
VLSI design.

26. LP Application: Weighted Bipartite Matching
Matching problem. Given a complete bipartite network Kn,n and edge weights cij, find a perfect matching of minimum weight.
Birkhoff-von Neumann theorem (1946, 1953). All extreme points of the above polyhedron are {0-1}-valued.
Corollary. Can solve assignment problem using LP techniques since LP algorithms return optimal solution that is an extreme point.

27. Transshipment Problem

28. Brewery Problem How can brewer maximize profits??? Beverage
Corn (pounds)
Malt (pounds)
Hops (ounces)
Beer 15 20 4
Ale 5 35
Profit ($) 23 13
Quantity
480
1190
160
How can brewer maximize profits???
Ale
Beer
(34, 0)
(0, 32)
(12, 28)
(26, 14)
(0, 0)
Extreme points
Ask the students to work on the problem

29. LP Example
Maximize: P = 10x + 12y subject to the constraints: x + y ≤ 40 (i) x + 2y ≤ 75 (ii) x ≥ 0, y ≥ 0. Step 1: Introduce slack variables : (i): x + y + s = 40 s ≥ 0 (ii): x + 2y + t = 75 t ≥ 0

30. Tableau Format Last column Pivotal column
Shaded cells ≥ 0
The aim is to solve the equations by combining rows together. The solution is reached when all entries in the first row (except possibly the value in the last column) are non-negative.
Most negative entry in the objective function : -12 in column y

31. Tableau Format
Pivotal column
pivot

32. Tableau Format P x y s t l Equation Ratio 1 - 4 6 450 ½ 2.5 (5) = (2)
pivot P x y s t l
Equation
Ratio 1 - 4 6
450
(4)=(1)+12×(6)
450/ 4=
112.5
2.5
(5) = (2) –
(6)
2.5/0.5 = 5
37.5
(6) = (3)/2
37.5/0.5 = 75
Now, get 0 entries elsewhere in the pivotal column.

33. Tableau Format
The optimal solution: x = 5, y = 35, P = 470.

34. Questions to ask Remarkable property of Simplex method: But,
Typically it requires less than 2(m+n) pivots to attain optimality.
But,
How do we know LP is feasible?
What if the initial basic solution is not feasible?
Is it possible that Simplex method never terminates?
How do we know Simplex method returns the optimal solution when it does terminate?

35. Initial Basic Feasible Solution
Example:
Basic solution:
x3 = 2  2x1+x2
x4 =  4  x1+5x2
Basic solution is not feasible!
Does LP have a feasible solution?

36. Auxiliary LP
Find a slack form for which the basic solution is feasible.
P is feasible if and only if the optimal objective value of Paux = 0. A feasible solution to Paux will be a feasible solution to P.
proof: …

37. Example Try BS: x0= x1= x2=0 → x4<0!, ?
One step pivoting: swap the roles of x0 and x4.
Swap x0 with most negative variable (x4).
Only one step pivot is needed.
Solve 2nd eq. for x0: x0= 4 + x1 5x2+ x4
Rewrite LP:
not feasible
Solve it

38. Example
x0 = 0

39. Questions to ask How do we know LP is feasible?
What if the initial basic solution is not feasible?
Is it possible that Simplex method never terminates?
How do we know Simplex method returns the optimal solution when it does terminate?

40. Complexity Upper Bound
How many possible slack forms are there?
Ale
Beer
(34, 0)
(0, 32)
(12, 28)
(26, 14)
(0, 0)
Extreme points
Step1: Find a basic feasible solution
Step2: …
Rewrite it as:

41. The solution is unbounded.

42. Complexity Upper Bound
How many possible slack forms are there?
If the steps are not repeated, Simplex method terminates in at most n+m choose m steps.
See sec 29.3 for more rigorous proof.

43. Complexity Upper Bound
How many possible slack forms are there?
If the steps are not repeated, Simplex method terminates in at most n+m choose m steps.
Lemma 29.6
Bland’s rule: ties in Simplex method are always broken by choosing the variable with the smallest index, then Simplex must terminate.

44. LP Duality Primal problem. Find a lower bound on optimal value.
(x1, x2, x3, x4) = (0, 0, 1, 0)  z*  5.
(x1, x2, x3, x4) = (2, 1, 1, 1/3)  z*  15.
(x1, x2, x3, x4) = (3, 0, 2, 0)  z*  22.
(x1, x2, x3, x4) = (0, 14, 0, 5)  z*  29.

45. LP Duality Primal problem. Find an upper bound on optimal value.
Multiply 2nd inequality by 2: 10x1 + 2x2 + 6x3 + 16x4  110.
z* = 4x1 + x2 + 5x3 + 3x4  10x1 + 2x2 + 6x3 + 16x4  110.
Adding 2nd and 3rd inequalities: 4x1 + 3x2 + 6x3 + 3x4   z* = 4x1 + x2 + 5x3 + 3x4  4x1 + 3x2 + 6x3 + 3x4  58.

46. LP Duality Primal problem. Find an upper bound on optimal value.
Adding 11 times 1st inequality to 6 times 3rd inequality:  z* = 4x1 + x2 + 5x3 + 3x4  5x1 + x2 + 7x3 + 3x4  29.
Recall.
(x1, x2, x3, x4) = (0, 14, 0, 5)  z*  29.

47. What have we done? Find a linear combination of the constrains
Try to minimize the right hand side
Relate the objective function to the linear combination

48. LP Duality Primal problem.
General idea: add linear combination (y1, y2, y3) of the constraints.
Dual problem.
Y1*eq1 +y2*eq2+… <= obj to minimize
Coefficient of x1 has to be >= coefficient of x1 in the original obj

49. LP Duality
Primal and dual linear programs: given real numbers aij, bi, cj, find values xi, yj that optimize (P) & (D).
Duality Theorem (Gale-Kuhn-Tucker 1951, Dantzig-von Neumann 1947). If (P) and (D) are nonempty then max = min.
Dual solution provides certificate of optimality
Special case: max-flow min-cut theorem.
(D) ?
Duality theorem first written down by Gale-Kuhn-Tucker, but inspired from conversation between Dantzig and von-Neumann
to show in NP and co-NP also need to use the fact that vertex solutions are of polynomial size (e.g., via Kramer's rule)

50. LP Duality Theorem
LP Duality Theorem. For Amn , bm , cn, if (P) and (D) are nonempty then max = min.
Reference: notes by Oktay Gunluk and Paul Seymour

51. LP Weak Duality Proof: Corollary. If cTx = yTb, x and y are optimal.
LP Weak Duality. For A  mn , b  m , c  n, if (P) and (D) are nonempty, then max  min.
Proof:
Suppose x  m is feasible for (P) and y  n is feasible for (D).
x  0, yTA  c  yTAx  cTx.
y  0, Ax  b  yTAx  yTb
combining two inequalities: cTx  yTAx  yTb
Corollary. If cTx = yTb, x and y are optimal.

52. LP Duality Theorem
LP Duality Theorem. For Amn , bm , cn, if (P) and (D) are nonempty then max = min.
LP Weak Duality max  min 
LP Strong Duality min  max (read book)
Reference: notes by Oktay Gunluk and Paul Seymour

53. LP Duality: Economic Interpretation
Brewer's problem: find optimal mix of beer and ale to maximize profits.
Entrepreneur's problem: Buy resources from brewer at minimum cost.
C, H, M = unit price for corn, hops, malt.
Brewer won't agree to sell resources if 5C + 4H + 35M < 13.
A* = 12 B* = 28 OPT = 800
(D)
C* = 1 H* = 2 M* = 0 OPT = 800

54. Optimal Dual Solution
Theorem. Suppose that Simplex returns values on the primal LP problem (A, b, c):
Let N: set of nonbasic variable (all going to be zero)
B: set of basic variables.
Consider the final slack form:
Then the optimal dual solution ў is given by
Proof: text P

55. Questions to ask How do we know LP is feasible?
What if the initial basic solution is not feasible?
Is it possible that Simplex method never terminates?
How do we know Simplex method returns the optimal solution when it does terminate?