1. UMass Lowell Computer Science 91.503 Analysis of Algorithms Prof. Karen Daniels Spring, 2010 Lecture 9 Tuesday, 4/20/10 Linear Programming 1

2. Overview Motivation & Basics Standard & Slack Forms Formulating Problems as Linear Programs Simplex Algorithm Example High-Level Algorithm Correctness Roadmap Key Concepts Initial Basic Feasible Solution Duality More examples… 2

3. Motivation & Basics 3

4. Motivation: A Political Problem 4 source: 91.503 textbook Cormen et al. Goal: Win election by winning majority of votes in each region. Subgoal: Win majority of votes in each region while minimizing advertising cost. 100,000 voters 200,000 voters 50,000 voters Thousands of voters who could be won with $1,000 of ads related to a policy

5. Motivation: A Political Problem (continued) 5 Thousands of voters representing majority. urban suburban rural source: 91.503 textbook Cormen et al. This search space is 4-dimensional.

6. General Linear Programs 6 real numbers real-valued variables Linear function Linear inequalities Linear constraints source: 91.503 textbook Cormen et al. =

7. Overview of Linear Programming 7 Convex feasible region: intersection of half-planes Objective function Objective value source: 91.503 textbook Cormen et al. Different from voter example: this example has 2-dimensional search space

8. Overview of Linear Programming More examples (on board) of 2-dimensional search space: -Infeasible example in which constraints conflict -Unbounded example

9. Standard & Slack Forms 9

10. Standard Form 10 objective function constraints source: 91.503 textbook Cormen et al.

11. Standard Form (compact) 11 mxn matrix m-dimensional vector n-dimensional vectors source: 91.503 textbook Cormen et al. Can specify linear program in standard form by (A,b,c).

12. Converting to Standard Form 12 source: 91.503 textbook Cormen et al.

13. Converting to Standard Form (continued) 13 source: 91.503 textbook Cormen et al. Negate coefficients Transforming minimization to maximization

14. Converting to Standard Form (continued) 14 source: 91.503 textbook Cormen et al. If x j has no non-negativity constraint, replace each occurrence of x j with x j ’ – x j ”. Giving each variable a non-negativity constraint New non-negativity constraints - 3(x 2 ’ – x 2 ”) x 2 has no non-negativity constraint

15. Converting to Standard Form (continued) 15 source: 91.503 textbook Cormen et al. Transforming equality constraints to inequality constraints

16. Converting to Standard Form (continued) 16 source: 91.503 textbook Cormen et al. Changing sense of an inequality constraint Rationale: (Rename variables for notational consistency.)

17. Converting Linear Programs into Slack Form 17 source: 91.503 textbook Cormen et al. for algorithmic ease, transform all constraints except non-negativity ones into equalities for inequality constraint: define slack slack variable instead of s basic variables non-basic variables

18. Converting Linear Programs into Slack Form (continued) 18 source: 91.503 textbook Cormen et al. objective function happens to be empty in this case

19. Converting Linear Programs into Slack Form (continued) 19 source: 91.503 textbook Cormen et al. Compact Form: (N, B, A, b, c, v) set of indices of non-basic variables set of indices of basic variables Slack Form Example Compact Form negative of slack form coefficients basic variables

20. Formulating Problems as Linear Programs 20

21. Shortest Paths 21 source: 91.503 textbook Cormen et al., 2 nd edition Single-pair shortest path: minimize “distance” from source s to sink t. Use Bellman-Ford idea: translate relaxation into a constraint. Can we replace maximize with minimize here? Why or why not? where  is shortest path weight Recall:

22. Maximum Flow 22 source: 91.503 textbook Cormen et al., 3 rd edition Flow conservation subject to Capacity constraints

23. Minimum Cost Flow 23 Flow target d is prespecified. cost = 4 cost = 7 cost = 3 cost = 10 cost = 3 Total weighted flow cost = 27 cost = 4 cost = 7 cost = 3 cost = 10 Total weighted flow cost = 27 Flow conservation subject to Capacity constraints Flow target d is prespecified here as d=4. source: 91.503 textbook Cormen et al., 3 rd edition

24. Multicommodity Flow 24 source: 91.503 textbook Cormen et al., 3 rd edition s i is source for commodity i t i is sink for commodity i d i is target flow value for commodity i subject to Capacity constraints Flow conservation Flow target d i is prespecified for each i. “The only known polynomial-time algorithm for this problem expresses it as a linear program and then solves it with a polynomial-time linear-programming algorithm.” k is number of commodities

25. Simplex Method 25

26. Solving a Linear Program Simplex algorithm – Geometric interpretation Visit some vertices on the boundary of the simplex representing the convex feasible region – Transforms set of inequalities using process similar to Gaussian elimination – Run-time not polynomial in worst-case – 1972 exponential time result often very fast in practice – sparse matrix techniques provide speed-up Ellipsoid method – Iterative approach creating sequence of ellipsoids: holds current point as center of ellipsoid that holds a feasible point, if one exists. Cut off part of ellipsoid violated by some constraint. – Run-time polynomial slow in practice Interior-Point methods – Run-time polynomial for large inputs, performance can be competitive with simplex method – Moves through interior of feasible region 26 source: 91.503 textbook Cormen et al.

27. Simplex Algorithm: Example Basic Solution 27 source: 91.503 textbook Cormen et al. Standard Form Slack Form Basic Solution: set each nonbasic variable to 0. Basic Solution:

28. Simplex Algorithm: Example Reformulating the LP Model 28 source: 91.503 textbook Cormen et al. Main Idea: In each iteration, reformulate the LP model so basic solution has larger objective value (since we’re maximizing). Select a nonbasic variable whose objective coefficient is positive (since we’re maximizing): x 1 Increase its value as much as possible. Identify tightest constraint on increase. For basic variable x 6 of that constraint, swap role with x 1. Rewrite other equations with x 6 on RHS. PIVOT leaving variable entering variable new objective value

29. Simplex Algorithm: Example Reformulating the LP Model 29 source: 91.503 textbook Cormen et al. Next Iteration: select x 3 as entering variable. PIVOT leaving variable entering variable New Basic Solution: new objective value

30. Simplex Algorithm: Example Reformulating the LP Model 30 source: 91.503 textbook Cormen et al. Next Iteration: select x 2 as entering variable. PIVOT leaving variable entering variable New Basic Solution: new objective value Stop since all coefficients are negative.

31. Simplex Algorithm: Pivoting 31 source: 91.503 textbook Cormen et al. leaving variable entering variable Rewrite the equation that has x l on LHS to have x e on LHS Update remaining equations by substituting RHS of new equation for each occurrence of x e. Do the same for objective function. Update sets of nonbasic, basic variables.

32. Simplex Algorithm: Pseudocode 32 source: 91.503 textbook Cormen et al. to be defined later (detects infeasibility) initial basic solution optimal solution detects unboundedness

33. Correctness: Roadmap (Key Pieces) 33 Theorem 29.13: Fundamental Theorem of Linear Programming For LP model in standard form, either: 1. exists optimal solution with finite objective function value & SIMPLEX returns one, or 2. infeasible & SIMPLEX returns INFEASIBLE, or 3. unbounded & SIMPLEX returns UNBOUNDED (Lemmas 29.2 & 29.7) Lemma 29.2: Basic solution feasible -> if SIMPLEX finds solution it is feasible; if reports unbounded, then model is unbounded Lemma 29.3: Algebraic lemma Theorem 29.10: LP duality: SIMPLEX primal result is optimal & dual is optimal * analogous to max flow/ min cut idea Lemma 29.5: Iteration bound, cycling Lemma 29.4: Slack form uniqueness Lemma 29.6: Tie-breaking Lemma 29.8: Weak LP duality Corollary 29.9: Conditions for which feasible solutions for primal, (max) and dual (min) programs are optimal Lemma 29.1: Pivot results Lemma 29.12: Infeasibility detection Lemma 29.11: L aux Lemma 29.7: Basic solution feasible -> SIMPLEX either reports unbounded or finds feasible solution in ≤ iterations # variables # basic variables

34. Proof of Correctness Key Concepts Initial Basic Feasible Solution 34

35. Finding an Initial Solution 35 source: 91.503 textbook Cormen et al. An LP model whose initial basic solution is not feasible Board work for picture, slack form and initial basic solution.

36. Finding an Initial Solution (continued) 36 source: 91.503 textbook Cormen et al. Auxiliary LP model L aux : 0 L aux L feasible -> 0 optimal for L aux L feasible <- 0 optimal for L aux in standard but not slack form max -x 0

37. Finding an Initial Solution (continued) 37 source: 91.503 textbook Cormen et al. 3 rd edition most negative see p. 888 for details (also example in next slides)

38. Finding an Initial Solution (continued) 38 source: 91.503 textbook Cormen et al. Original LP model L aux L aux in slack form L aux Basic (infeasible) Solution:

39. Finding an Initial Solution (continued) 39 source: 91.503 textbook Cormen et al. PIVOT entering leaving most negative (restrictive) Feasible Basic Solution: (0, 4/5, 14/5, 0) Entering Leaving (most negative, restrictive) Entering L aux Basic (feasible) Solution: Why is this feasible for L aux ? Leaving (most restrictive) Why can we stop here?

40. Finding an Initial Solution (continued) 40 source: 91.503 textbook Cormen et al. See textbook for proof (p. 890).

41. Duality 41 Proof of Correctness Key Concepts

42. Linear Programming Duality 42 source: 91.503 textbook Cormen et al. max becomes min RHS coefficients swap places with objective function coefficients sense changes x variables go away y variables appear

43. Duality Example 43 source: 91.503 textbook Cormen et al.

44. Weak Linear Programming Duality 44 source: 91.503 textbook Cormen et al. Any feasible solution to primal LP has value no greater than that of any feasible solution to the dual LP. standard form 29.84 29.18 29.83-29.85

45. Weak Linear Programming Duality (continued) 45 source: 91.503 textbook Cormen et al. minimized maximized

46. Finding a Dual Solution 46 source: 91.503 textbook Cormen et al. 3 rd edition Finding a dual solution whose value is equal to that of an optimal primal solution… Suppose the last slack form of the primal is: Then, to produce an optimal dual solution, we set:

47. Optimality 47 source: 91.503 textbook Cormen et al. Theorem 29.10 ( Linear-Programming Duality ). Suppose that S IMPLEX returns values for the primal linear program (A, b, c). Let N and B denote the non-basic and basic variables for the final slack form, let c’ denote the coefficient in the final slack form, and let be defined by (29.91). Then is an optimal solution the primal program, is an optimal solution to the dual program and Proof: see text (p. 882-885).

48. Correctness: Roadmap (Key Pieces) 48 Theorem 29.13: Fundamental Theorem of Linear Programming For LP model in standard form, either: 1. exists optimal solution with finite objective function value & SIMPLEX returns one, or 2. infeasible & SIMPLEX returns INFEASIBLE, or 3. unbounded & SIMPLEX returns UNBOUNDED (Lemmas 29.2 & 29.7) Lemma 29.2: Basic solution feasible -> if SIMPLEX finds solution it is feasible; if reports unbounded, then model is unbounded Lemma 29.3: Algebraic lemma Theorem 29.10: LP duality: SIMPLEX primal result is optimal & dual is optimal * analogous to max flow/ min cut idea Lemma 29.5: Iteration bound, cycling Lemma 29.4: Slack form uniqueness Lemma 29.6: Tie-breaking Lemma 29.8: Weak LP duality Corollary 29.9: Conditions for which feasible solutions for primal, (max) and dual (min) programs are optimal Lemma 29.1: Pivot results Lemma 29.12: Infeasibility detection Lemma 29.11: L aux Lemma 29.7: Basic solution feasible -> SIMPLEX either reports unbounded or finds feasible solution in ≤ iterations # variables # basic variables

49. Beyond Chapter 29… 49

50. Some Variations on Linear Programming Integer Programming: Linear constraints and objective Variables are restricted to integer values. Feasibility is NP-complete. Mixed Integer Programming: Linear constraints and objective Some variable(s) are restricted to integer values. Some variables are not restricted to integers. Quadratic Programming: Linear constraints Quadratic objective function Real-valued variables Convex Programming Generalization of Linear Programming Minimize a convex function, subject to convex constraints Interior point methods can be applied if the functions satisfy additional properties. Reference: “A Tutorial on Convex Optimization” by Hindi Increasingly being used. 50

51. A Hierarchy of Related Types of Mathematical Programming 51 http://en.wikipedia.org/wiki/Convex_optimization

52. 52 See docs portion of web site for additional information on Karp’s work. (courtesy of Nate)

53. My Applied Mathematical Programming Research Lagrangian Relaxation – Apparel trim placement (Grinde & Daniels 2000) – Polygon covering (Daniels et al. 2003) – Box covering (England, Daniels 2008) Mixed Integer Programming – Dynamic Channel assignment (Liu, Daniels, Chandra 2001, 2004) Quadratic Programming – Support vector clustering enhancements (Lee, Daniels 2005, 2006) 53 See http://www.cs.uml.edu/~kdaniels/papers/papers.html

54. Linear Programming Resource Elementary Linear Programming with Applications (2 nd edition), by Kolman and Beck, Academic Press, 1980. 54