1. 15-853Page1 15-853:Algorithms in the Real World Linear and Integer Programming III – Integer Programming Applications Algorithms

2. 15-853Page2 Integer (linear) Programming Related Problems –Mixed Integer Programming (MIP) –Zero-one programming –Integer quadratic programming –Integer nonlinear programming minimize:cTxcTx subject to:Ax · b x ¸ 0 x 2 Z n

3. 15-853Page3 History Introduced in 1951 (Dantzig) TSP as special case in 1954 (Dantzig) First convergent algorithm in 1958 (Gomory) General branch-and-bound technique 1960 (Land and Doig) Frequently used to prove bounds on approximation algorithms (late 90s)

4. 15-853Page4 Current Status Has become “dominant” over linear programming in past decade Saves industry Billions of Dollars/year Can solve 10,000+ city TSP problems 1 million variable LP approximations Branch-and-bound, Cutting Plane, and Separation all used in practice General purpose packages do not tend to work as well as with linear programming --- knowledge of the domain is critical.

5. 15-853Page5 Subproblems/Applications Facility location Locating warehouses or franchises (e.g. a Burger King) Set covering and partitioning Scheduling airline crews Multicomodity distribution Distributing auto parts Traveling salesman and extensions Routing deliveries Capital budgeting Other Applications VLSI layout, clustering

6. 15-853Page6 Knapsack Problem where: b = maximum weight c i = utility of item i a i = weight of item i x i = 1 if item i is selected, or 0 otherwise The problem is NP-hard. Integer (zero-one) Program: maximizecTxcTx subject to:ax · b x binary

7. 15-853Page7 Traveling Salesman Problem Find shortest tours that visit all of n cities. courtesy: Applegate, Bixby, Chvátal, and Cook Applegate BixbyChvátalCook

8. 15-853Page8 Traveling Salesman Problem c ij = c ji = distance from city i to city j (assuming symmetric version) x ij if tour goes from i to j or j to i, and 0 otherwise Anything missing? (path enters and leaves) minimize: subject to:

9. 15-853Page9 Traveling Salesman Problem c ij = distance from city i to city j x ij = 1 if tour visits i then j, and 0 otherwise (binary) t i = arbitrary real numbers we need to solve for (out degrees = 1) (in degrees = 1) (??) minimize: subject to:

10. 15-853Page10 Traveling Salesman Problem c ij = distance from city i to city j x ij = 1 if tour visits i then j, and 0 otherwise (binary) t i = arbitrary real numbers we need to solve for minimize: subject to: (out degrees = 1) (in degrees = 1) (??)

11. 15-853Page11 Traveling Salesman Problem The last set of constraints: prevents “subtours”: Consider a cycle that goes from some node 4 to 5, t 4 – t 5 + nx 4,5 · n-1 ) t 5 ¸ t 4 + 1 Similarly t has to increase by 1 along each edge of the cycle that does not include vertex 1. Therefore, for a tour of length m that does not go through vertex 1, t 4 ¸ t 4 + m, a contradiction. Every cycle must go through vertex 1. Together with other constraints, it forces one cycle.

12. 15-853Page12 Traveling Salesman Problem Many “Real World” applications based on the TSP.applications –They typically involve more involved constraints –Not just routing type problems. Consider a drug company with k drugs they can make at a lab. They can only make the drugs one at a time. The cost of converting the equipment from making drug i to drug j is c ij Current best solutions are based on IP –Applegate, Bixby, et. al., have solutions for more than 15K cities in Germany > 150,000 CPU hours (more info)more info –Involves “branch-and-bound” and “cutting planes”

13. 15-853Page13 Set Covering Problem Find cheapest sets that cover all elements Courtesy: Darmstadt University of Technology

14. 15-853Page14 Set Covering and Partitioning Given m sets and n items: Columns = sets Rows = items Set covering: minimize:cTxcTx subject to:Ax ¸ 1, x binary Set partitioning: minimize:cTxcTx subject to:Ax = 1, x binary

15. 15-853Page15 Set Covering and Partitioning Best cover: s 2, s 4, s 5 =.5 Best partition: s 4, s 6 =.7 setmemberscost s1s1 {a,c,d}.5 s2s2 {b,c}.2 s3s3 {b,e}.3 s4s4 {a,d}.1 s5s5 {c,e}.2 s6s6 {b,c,e}.6 s7s7 {c,d}.2

16. 15-853Page16 Set Covering and Partitioning Applications: Facility location. Each set is a facility (e.g. warehouse, fire station, emergency response center). Each item is an area that needs to be covered Crew scheduling. Each set is a route for a particular crew member (e.g. NYC->Pit->Atlanta->NYC). Each item is a flight that needs to be covered.

17. 15-853Page17 Constraints Expressible with IP Many constraints are expressible with integer programming: –logical constraints (e.g. x implies not y) –k out of n –piecewise linear functions

18. 15-853Page18 Constraints Expressible with IP Logical constraints (x 1, x 2 binary): Either x 1 or x 2 ) x 1 + x 2 = 1 If x 1 then x 2 ) x 1 – x 2 · 0 Either a 1 x · b 1 or a 2 x · b 2 ) a 1 x - My · b 1 a 2 x - M(1-y) · b 2 Combining constraints: y is a binary variable, M needs to be “large”, a 1, a 2, and x can be vectors

19. 15-853Page19 Algorithms 1.Use a linear program –round to integer solution (what if not feasible?) 2.Search –Branch and bound (integer ranges) –Implicit (0-1 variables) 3.Cutting planes –Many variants

20. 15-853Page20 Important Properties LP solution is an upper bound on IP solution (assuming maximization) If LP is infeasible then IP is infeasible If LP solution is integral (all variables have integer values), then it is the IP solution.

21. 15-853Page21 Linear Programming Solution 1.Some LP problems will always have integer solutions transportation problem assignment problem min-cost network flow These are problems with a unimodular matrix A. (unimodular matrices have det(A) = 1). 2.Solve as linear program and round. Can violate constraints, and be non-optimal. Works OK if integer variables take on large values accuracy of constraints is questionable

22. 15-853Page22 Branch and Bound Lets first consider 0-1 programs. Exponential solution: try all {0,1} n Branch-and-bound solution: x 2 = 0 x 2 = 1 x 1 = 0 x 1 = 1 x 3 = 0 x 3 = 1 Traverse tree keeping current best solution. If it can be shown that a subtree never improves on the current solution, or is infeasable, prune it.

23. 15-853Page23 Zero-One Branch and Bound minimize: z = c T x, subject to: Ax · b, x ¸ 0, x 2 {0,1} n Assume all elements of c are non-negative function ZO r (A, b, c, x f, z*) // x f : a fixed setting for a subset of the variables // z* is the cost of current best solution x = x f + 0 // set unconstrained variables to zero if (cx ¸ z*) or (no feasible completion of x f ) return z* if (Ax · b) then return cx pick an unconstrained variable x i from x z 0 * = ZO r (A, b, x f [ {x i = 0}, c, z*) z 1 * = ZO r (A, b, x f [ {x i = 1}, c, z 0 *) return z 1 * function ZO(A, b, c) = ZO r (A, b, c, , 1 )

24. 15-853Page24 Zero-One Branch and Bound Checking for feasible completions: check each constraint and find if minimum of left is greater than right. Example: x f = {x 1 = 1, x 3 = 0} and one of the constraints is 3x 1 + 2x 2 – x 3 + x 4 · 2 then 3 + 2x 2 - 0 + x 4 · 2 2x 2 + x 4 · -1 which is impossible.

25. 15-853Page25 Integer Branch and Bound The zero-one version is sometimes called “implicit enumeration” since it might enumerate all possibilities. An integer version cannot branch on all possible integer values for a variable. Even if the integer range is bounded, it is not practical. Will “bound” by adding inequalities to split the two branches. Since solutions are integral, each split can remove a strip of width 1

26. 15-853Page26 Integer Branch and Bound maximize: z = c T x, subject to: Ax · b, x ¸ 0, x 2 Z n function IP r (A e, b e, c, z*) // A e, b e are A and b with additional constraints // z* is the cost of current best solution z, x, f = LP(A,b,c) // f indicates whether feasible if not(f) or (z < z*) return z* if (integer(x)) return z pick a non-integer variable x i ’ from x z l * = IP(extend A e,b e with x i · b x i ’ c, c, z*) z g * = IP(extend A e,b e with –x i · - d x i ’ e, c, z l *) return z g * function IP(A, b, c) = IP r (A, b, c, - 1 )

27. 15-853Page27 Example z l u Find optimal solution. Cut along y axis, and make two recursive calls feasible

28. 15-853Page28 Example Find optimal solution. Solution is integral, so return it as current best z* z l u

29. 15-853Page29 Example Find optimal solution. It is better than z*. Cut along x axis, and make two recursive calls lu z = z*

30. 15-853Page30 Example lu z Infeasible, Return.

31. 15-853Page31 Example z lu l u Find optimal solution. It is better than z*. Cut along y axis, and make two recursive calls = z*

32. 15-853Page32 Example l u Find optimal solution. Solution is integral and better than z*. Return as new z*. = z* z

33. 15-853Page33 Example l u Find optimal solution. Not as good as z*, return. = z* z

34. 15-853Page34 Cutting Plane The idea is to start with a “relaxation” R of the problem and then add constraints on the fly to find an actual feasible solution in S. relaxation S = S = R Example 1 Example 2 A “linear” relaxation new constraint

35. 15-853Page35 Cutting Plane: general algorithm minimize: z = c T x, subject to x 2 S function CP(R, c) // R a relaxed set of constraints Ax · b s.t. S ½ polytope(Ax · b) repeat: x = LP(R,c) if x 2 S return x find an inequality r satisfied by S, but violated by x (r separates x from S) R = R  {r} Can add multiple inequalities on each iteration

36. 15-853Page36 Cutting Plane Note that we are removing a corner, and no integer solutions are being excluded. z feasible New plane

37. 15-853Page37 Picking the Plane Method 1: Gomory cuts (1958) –Cuts are generated from the LP Tableau Each row defines a potential cut –Guaranteed to converge on solution –General purpose, but inefficient in practice Method 2: problem specific cuts (templates) –Consider the problem at hand and generate cuts based on its structure –A template is a problem specific set of cuts (probably of exponential size) which S satisfies. Each round picks a cut from this set.

38. 15-853Page38 Templates for the TSP problem We consider some example templates used in solutions of the Traveling Salesman Problem. Recall that x ij indicates the edge from v i to v j Assume the symmetric TSP: x ij = x ji Consider subsets of vertices U ½ V. define:  x (U) =  x ij, v i 2 U, v j 2 V-U (i.e. the number of times path crosses into/outof U) Degree Constraints:  x ({v i }) = 2, 1 · i · n Subtour Constraints:  x (U) ¸ 2, U ½ V A template There are an exponential number of these

39. 15-853Page39 Templates for the TSP problem A set of contraints (a template) is facet-defining for S if each constraint is on a facet of the convex hull of S. We would like templates which are facet defining since, intuitively, they will more quickly constrain us to the boundary of S. The subtour template is facet defining. In practice the subtour inequalities are not enough to contrain the solution to integral solutions. Are there other sets of facet defining constraints?

40. 15-853Page40 Templates for the TSP problem Blossom inequalities (Edmonds 1965): Defined by H (handle) and T 1, …, T k (teeth) satisfying: k ¸ 3 and odd, |T i | = 2 T i Å T j =  |H Å T i | = 1, |T i \ H| = 1 T1T1 T2T2 T3T3 H  x (H) +  i=1 k  x (T i ) ¸ 3k + 1

41. 15-853Page41 Templates for the TSP problem Comb inequalities (Grotschel 1977) Just generalizes T i to be any size. At least one element of each T has to be in and out of H. T1T1 T2T2 T3T3 H  x (H) +  i=1 k  x (T i ) ¸ 3k + 1

42. 15-853Page42 The art of Templates Picking the right set of templates, and applying them in the right way is the art of solving NP-hard problems with integer programming. Different problems have different templates. One needs to find good algorithms for selecting a member of a template that separates x from S (can be quite complicated on its own). Cutting planes often used in conjunction with branch and bound. Can interleave template cuts with Gomory cuts (e.g. use Gomory cuts when the set of template cuts “dries out”).

43. 15-853Page43 Practical Developments Good formulations, heuristics and theory Goal: to get LP solution as close as possible to IP solution Disaggregation, adding constraints (cuts) Preprocessing Automatic methods for reformulation Some interesting graph theory is involved Cut generation (branch-and-cut) Add cuts during the branch-and-bound Column generation Improve formulation by introducing an exponential number of variables.