1. Explorations in Artificial Intelligence Prof. Carla P. Gomes gomes@cs.cornell.edu Module 7 Part 3 Integer Programming

2. Divisibility Decision variables in an LP model are allowed to have any values, including noninteger values, that satisfy the functional and nonnegativity constraints. i.e., activities can be run at fractional levels. What to do when divisibility assumption violated: realm of integer programming!!!

3. Revisiting the TBA Airlines Problem An Example where Integrality Matters

4. The TBA Airlines Problem TBA Airlines is a small regional company that specializes in short flights in small airplanes. The company has been doing well and has decided to expand its operations. The basic issue facing management is whether to purchase more small airplanes to add some new short flights, or start moving into the national market by purchasing some large airplanes, or both. Question: How many airplanes of each type should be purchased to maximize their total net annual profit?

5. Data for the TBA Airlines Problem Small Airplane Large Airplane Capital Available Net annual profit per airplane$1 million$5 million Purchase cost per airplane5 million50 million$100 million Maximum purchase quantity2—

6. Linear Programming Formulation LetS = Number of small airplanes to purchase L = Number of large airplanes to purchase Maximize Profit = S + 5L ($millions) subject to Capital Available:5S + 50L ≤ 100 ($millions) Max Small Planes:S ≤ 2 and S ≥ 0, L ≥ 0.

7. Graphical Method for Linear Programming

8. Violates Divisibility Assumption of LP Divisibility Assumption of Linear Programming: Decision variables in a linear programming model are allowed to have any values, including fractional values, that satisfy the functional and nonnegativity constraints. Thus, these variables are not restricted to just integer values. Since the number of airplanes purchased by TBA must have an integer value, the divisibility assumption is violated.

9. Integer Programming Formulation LetS = Number of small airplanes to purchase L = Number of large airplanes to purchase Maximize Profit = S + 5L ($millions) subject to Capital Available:5S + 50L ≤ 100 ($millions) Max Small Planes:S ≤ 2 and S ≥ 0, L ≥ 0 S, L are integers.

10. Graphical Method for Integer Programming

11. When an integer programming problem has just two decision variables, its optimal solution can be found by applying the graphical method for linear programming with just one change at the end. We begin as usual by graphing the feasible region for the LP relaxation, determining the slope of the objective function lines, and moving a straight edge with this slope through this feasible region in the direction of improving values of the objective function. However, rather than stopping at the last instant the straight edge passes through this feasible region, we now stop at the last instant the straight edge passes through an integer point that lies within this feasible region. This integer point is the optimal solution.

12. Why integer programs? Advantages of restricting variables to take on integer values –More realistic –More flexibility Disadvantages –More difficult to model –Can be much more difficult to solve

13. Integer Programming When are “non-integer” solutions okay? –Solution is naturally divisible e.g., $, pounds, hours –Solution represents a rate e.g., units per week –Solution only for planning purposes When is rounding okay? –When numbers are large e.g., rounding 114.286 to 114 is probably okay. When is rounding not okay? –When numbers are small e.g., rounding 2.6 to 2 or 3 may be a problem. –Binary variables yes-or-no decisions

14. Types of Integer Programming Problems Pure integer programming problems are those where all the decision variables must be integers. Mixed integer programming problems only require some of the variables (the “integer variables”) to have integer values so the divisibility assumption holds for the rest (the “continuous variables”). Binary variables are variables whose only possible values are 0 and 1. Binary integer programming (BIP) problems are those where all the decision variables restricted to integer values are further restricted to be binary variables. –Such problems can be further characterized as either pure BIP problems or mixed BIP problems, depending on whether all the decision variables or only some of them are binary variables.

15. Examples of Applications of Binary Variables Making “yes-or-no” type decisions –Build a factory? –Manufacture a product? –Do a project? –Assign a person to a task? Logical constraints –Alternative constraints –Conditional constraints Representing non-linear functions –Fixed Charge Problem If a product is produced, must incur a fixed setup cost. If a warehouse is operated, must incur a fixed cost. –Piecewise linear representation –Diseconomies of scale –Approximation of nonlinear functions Set-covering, and set partitioning –Make a set of assignments that “cover” a set of requirements. –Partition a set into subsets meeting given requirements

16. StockCompany Example Capital Budgeting Allocation Problem StockCompany is considering 6 investments. The cash required from each investment as well as the NPV of the investment is given next. The cash available for the investments is $14,000. Stockco wants to maximize its NPV. What is the optimal strategy? An investment can be selected or not. One cannot select a fraction of an investment.

17. Data for the StockCompany Problem Investment budget = $14,000

18. Integer Programming Formulation Max 16x 1 + 22x 2 + 12x 3 + 8x 4 + 11x 5 + 19x 6 5x 1 + 7x 2 + 4x 3 + 3x 4 + 4x 5 + 6x 6  14 x j  {0,1} for each j = 1 to 6 What are the decision variables? Objective and Constraints?

19. Capital Budgeting Allocation Problem (one resource) Knapsack Problem Why is a problem with the characteristics of the previous problem called the Knapsack Problem? It is an abstraction, considering the simple problem: A hiker trying to fill her knapsack to maximum total value. Each item she considers taking with her has a certain value and a certain weight. An overall weight limitation gives the single constraint. Practical applications: Project selection and capital budgeting allocation problems Storing a warehouse to maximum value given the indivisibility of goods and space limitations Sub-problem of other problems e.g., generation of columns for a given model in the course of optimization – cutting stock problem (beyond the scope of this course)

20. The previous constraints represent “economic indivisibilities”, either a project is selected, or it is not. There is no selecting of a fraction of a project. Similarly, integer variables can model logical requirements (e.g., if stock 2 is selected, then so is stock 1.)

21. How to model “logical” constraints Exactly 3 stocks are selected. If stock 2 is selected, then so is stock 1. If stock 1 is selected, then stock 3 is not selected. Either stock 4 is selected or stock 5 is selected, but not both.

22. Formulating Constraints Exactly 3 stocks are selected x 1 + x 2 + x 3 + x 4 + x 5 + x 6 =3

23. If stock 2 is selected then so is stock 1 A 2-dimensional representation Stock 2  Stock 1 The integer programming constraint: x 1  x 2

24. If stock 1 is selected then stock 3 is not selected A 2-dimensional representation Stock 3  Stock 1 The integer programming constraint: x 1 + x 3  1

25. Either stock 4 is selected or stock 5 is selected, but not both. A 2-dimensional representation stock 5  stock 4 The integer programming constraint: x 4 + x 5 = 1

26. California Manufacturing Company The California Manufacturing Company is a diversified company with several factories and warehouses throughout California, but none yet in Los Angeles or San Francisco. A basic issue is whether to build a new factory in Los Angeles or San Francisco, or perhaps even both. Management is also considering building at most one new warehouse, but will restrict the choice to a city where a new factory is being built. Question: Should the California Manufacturing Company expand with factories and/or warehouses in Los Angeles and/or San Francisco?

27. Data for California Manufacturing Decision Number Yes-or-No Question Decision Variable Net Present Value (Millions) Capital Required (Millions) 1Build a factory in Los Angeles?x1x1 $8$6 2Build a factory in San Francisco?x2x2 53 3Build a warehouse in Los Angeles?x3x3 65 4Build a warehouse in San Francisco?x4x4 42 Capital Available: $10 million

28. Binary Decision Variables Decision Number Decision Variable Possible Value Interpretation of a Value of 1 Interpretation of a Value of 0 1x1x1 0 or 1 Build a factory in Los Angeles Do not build this factory 2x2x2 0 or 1 Build a factory in San Francisco Do not build this factory 3x3x3 0 or 1 Build a warehouse in Los Angeles Do not build this warehouse 4x4x4 0 or 1 Build a warehouse in San Francisco Do not build this warehouse

29. Algebraic Formulation Letx 1 = 1 if build a factory in L.A.; 0 otherwise x 2 = 1 if build a factory in S.F.; 0 otherwise x 3 = 1 if build a warehouse in Los Angeles; 0 otherwise x 4 = 1 if build a warehouse in San Francisco; 0 otherwise Maximize NPV = 8x 1 + 5x 2 + 6x 3 + 4x 4 ($millions) subject to Capital Spent:6x 1 + 3x 2 + 5x 3 + 2x 4 ≤ 10 ($millions) Max 1 Warehouse:x 3 + x 4 ≤ 1 Warehouse only if Factory:x 3 ≤ x 1 x 4 ≤ x 2 and x 1, x 2, x 3, x 4 are binary variables. Contingent decisions Mutually exclusive decisions Resource Availability

30. Using Excel Solver to Solve Integer Programs Add the integrality constraints (or add that a variable is binary) Set the Solver Tolerance. (The tolerance is the percentage deviation from optimality allowed by solver in solving Integer Programs.) –The default is 5% –The default is way to high –It often finds the optimum for small problems

31. Spreadsheet Model

32. Mpl Model Max 8x1 + 5x2 + 6x3 + 4x4; subject to 6x1 + 3x2 + 5x3 + 2x4 <= 10; x3 + x4 <= 1; x3 <= x1; x4 <= x2; BINARY x1; x2; x3; x4;

33. Sensitivity Analysis with Solver Table

34. Management’s Conclusion Management’s initial tentative decision had been to make $10 million of capital available. With this much capital, the best plan would be to build a factory in both Los Angeles and San Francisco, but no warehouses. An advantage of this plan is that it only uses $9 million of this capital, which frees up $1 million for other projects. A heavy penalty (a reduction of $4 million in total net present value) would be paid if the capital made available were to be reduced below $9 million. Increasing the capital made available by $1 million (to $11 million) would enable a substantial ($4 million) increase in the total net present value. Management decides to do this. With this much capital available, the best plan is to build a factory in both cities and a warehouse in San Francisco.

35. Some Other Applications Investment Analysis –Should we make a certain fixed investment? –Examples: Turkish Petroleum Refineries (1990), South African National Defense Force (1997), Grantham, Mayo, Van Otterloo and Company (1999) Site Selection –Should a certain site be selected for the location of a new facility? –Example: AT&T (1990) Designing a Production and Distribution Network –Should a certain plant remain open? Should a certain site be selected for a new plant? Should a distribution center remain open? Should a certain site be selected for a new distribution center? Should a certain distribution center be assigned to serve a certain market area? –Examples: Ault Foods (1994), Digital Equipment Corporation (1995)

36. Some Other Applications Dispatching Shipments –Should a certain route be selected for a truck? Should a certain size truck be used? Should a certain time period for departure be used? –Examples: Quality Stores (1987), Air Products and Chemicals, Inc. (1983), Reynolds Metals Co. (1991), Sears, Roebuck and Company (1999) Scheduling Interrelated Activities –Should a certain activity begin in a certain time period? –Examples: Texas Stadium (1983), China (1995) Scheduling Asset Divestitures –Should a certain asset be sold in a certain time period? –Example: Homart Development (1987) Airline Applications: –Should a certain type of airplane be assigned to a certain flight leg? Should a certain sequence of flight legs be assigned to a crew? –Examples: American Airlines (1989, 1991), Air New Zealand (2001)

37. Modeling Fixed Charge Problems If a product is produced, must incur a fixed setup cost. If a warehouse is operated, must incur a fixed cost.  The problem is non-linear. x – quantity of product to be manufactured x = 0  cost =0; x > 0  cost = C 1 x + C 2  How to model it? Using an indicator variable y y = 1  x is produced; y = 0  x is not produced Objective function becomes  C 1 x + C 2 y Additional Constraint  x ≤ My

38. Wyndor with Setup Costs (Variation 1) Suppose that two changes are made to the original Wyndor problem: 1.For each product, producing any units requires a substantial one-time setup cost for setting up the production facilities. 2.The production runs for these products will be ended after one week, so D and W in the original model now represent the total number of doors and windows produced, respectively, rather than production rates. Therefore, these two variables need to be restricted to integer values.

39. Graphical Solution to Original Wyndor Problem

40. Net Profit for Wyndor Problem with Setup Costs Net Profit ($) Number of Units ProducedDoorsWindows 00(300) – 0 = 00 (500) – 0 = 0 11(300) – 700 = –4001(500) – 1,300 = –800 22(300) – 700 = –1002(500) – 1,300 = –300 33(300) – 700 = 2003(500) – 1,300 = 200 44(300) – 700 = 5004(500) – 1,300 = 700 5Not feasible5(500) – 1,300 = 1,200 6Not feasible6(500) – 1,300 = 1,700

41. Feasible Solutions for Wyndor with Setup Costs Optimal solution

42. Algebraic Formulation LetD = Number of doors to produce, W = Number of windows to produce, y 1 = 1 if perform setup to produce doors; 0 otherwise, y 2 = 1 if perform setup to produce windows; 0 otherwise. Maximize P = 300D + 500W – 700y 1 – 1,300y 2 subject to Original Constraints: Plant 1:D ≤ 4 Plant 2:2W ≤ 12 Plant 3:3D + 2W ≤ 18 Produce only if Setup: Doors:D ≤ My 1 Windows:W ≤ My 2 and D ≥ 0, W ≥ 0, y 1 and y 2 are binary.

43. Spreadsheet Model

44. Wyndor with Mutually Exclusive Products (Variation 2) Suppose that now the only change from the original Wyndor problem is: The two potential new products (doors and windows) would compete for the same customers. Therefore, management has decided not to produce both of them together. –At most one can be chosen for production, so either D = 0 or W = 0, or both.

45. Feasible Solution for Wyndor with Mutually Exclusive Products (for non-binary variables)

46. Algebraic Formulation LetD = Number of doors to produce, W = Number of windows to produce, y 1 = 1 if produce doors; 0 otherwise, y 2 = 1 if produce windows; 0 otherwise. Maximize P = 300D + 500W subject to Original Constraints: Plant 1:D ≤ 4 Plant 2:2W ≤ 12 Plant 3:3D + 2W ≤ 18 Auxiliary variables must =1 if produce any: Doors:D ≤ My 1 Windows:W ≤ My 2 Mutually Exclusive:y 1 + y 2 ≤ 1 and D ≥ 0, W ≥ 0, y 1 and y 2 are binary.

47. Spreadsheet Model

48. Wyndor with Either-Or Constraints (Variation 3) Suppose that now the only change from the original Wyndor problem is: The company has just opened a new plant (plant 4) that is similar to plant 3, so the new plant can perform the same operations as plant 3 to help produce the two new products (doors and windows). However, management wants just one of the plants to be chosen to work on these new products. The plant chosen should be the one that provides the most profitable product mix.

49. Data for Wyndor with Either-Or Constraints (Variation 3) Production Time Used for Each Unit Produced (Hours) Production Time Available per Week (Hours) PlantDoorsWindows 1104 20212 33218 42428 Unit Profit$300$500

50. Graphical Solution with Plant 3 or Plant 4

51. Algebraic Formulation LetD = Number of doors to produce, W = Number of windows to produce, y = 1 if plant 4 is used; 0 if plant 3 is used Maximize P = 300D + 500W subject to Plant 1:D ≤ 4 Plant 2:2W ≤ 12 Plant 3:3D + 2W ≤ 18 + My Plant 4:2D + 4W ≤ 28 + M(1 – y) and D ≥ 0, W ≥ 0, y is binary.

52. Spreadsheet Model

53. Applications of Binary Variables Making “yes-or-no” type decisions –Build a factory? –Manufacture a product? –Do a project? –Assign a person to a task? Fixed costs –If a product is produced, must incur a fixed setup cost. –If a warehouse is operated, must incur a fixed cost. Either-or constraints –Production must either be 0 or ≥ 100. Subset of constraints –meet 3 out of 4 constraints.

54. Special Kinds of Integer Programming Models Knapsack Problem Set Covering Problem Set Partitioning Problem Set Packing Problem The Traveling Salesman Problem The Quadratic Assignment Problem

55. Set Covering Problem We are given a set of objects S = {1, 2, 3, …, n}. We are also given a set of subsets of S, S. Each subset has a cost associated with it. Problem: –to “cover” all the members of S at the minimum cost using members of S. Properties: –The problem is a minimization and all constraints are >=; –All RHS coefficients are 1; –All other matrix coefficients are 0 or 1.

56. Fire Station Problem Set Covering Problem 123 4 56 7 89 11 10 12 1415 13 16 Locate fire stations so that each district has a fire station in it, or next to it. Minimize the number of fire stations needed.

57. Representation as Set Covering Problem 123 4 56 7 89 11 10 12 1415 13 16 SetCovers 11, 2, 4, 5 21, 2, 3, 5, 6 32, 3, 6, 7 1613, 15, 16

58. Representation as Graph Cover Problem A node covers itself and its neighbors. Thus, node 16 covers nodes 13, 15, 16. 123 4 56 7 89 11 10 12 1415 13 16 What is the minimum size of a subset of nodes that covers all of the nodes? Replace each district with a node. Two nodes are adjacent if their districts are adjacent 16

59. Representation as Integer program 123 4 56 7 89 11 10 12 1415 13 16 x j = 1 if node j is selected x j = 0 otherwise Minimize x 1 + x 2 + … + x 16 s.t. x 1 + x 2 + x 4 + x 5  1 x 1 + x 2 + x 3 + x 5 + x 6  1 x 13 + x 15 + x 16  1 x j  {0, 1} for each j. 1 7 11 15

60. Representation as Integer program 123 4 56 7 89 11 10 12 1415 13 16 x j = 1 if node j is selected x j = 0 otherwise Minimize x 1 + x 2 + … + x 16 s.t. x 1 + x 2 + x 4 + x 5  1 x 1 + x 2 + x 3 + x 5 + x 6  1 x 13 + x 15 + x 16  1 x j  {0, 1} for each j. 1 7 11 15

61. Southwestern Airways Crew Scheduling Southwestern Airways needs to assign crews to cover all its upcoming flights. We will focus on assigning 3 crews based in San Francisco (SFO) to 11 flights. Question: How should the 3 crews be assigned 3 sequences of flights so that every one of the 11 flights is covered?

62. Southwestern Airways Flights

63. Data for the Southwestern Airways Problem Feasible Sequence of Flights (pairings) Flights123456789101112 1. SFO–LAX1111 2. SFO–DEN1111 3. SFO–SEA1111 4. LAX–ORD22323 5. LAX–SFO2355 6. ORD–DEN334 7. ORD–SEA33334 8. DEN–SFO2445 9. DEN–ORD222 10. SEA–SFO2445 11. SEA–LAX22442 Cost, $1,000s234675789989

64. Algebraic Formulation Letx j = 1 if flight sequence (paring) j is assigned to a crew; 0 otherwise. (j = 1, 2, …, 12). Minimize Cost = 2x 1 + 3x 2 + 4x 3 + 6x 4 + 7x 5 + 5x 6 + 7x 7 + 8x 8 + 9x 9 + 9x 10 + 8x 11 + 9x 12 (in $thousands) subject to Flight 1 covered:x 1 + x 4 + x 7 + x 10 ≥ 1 Flight 2 covered:x 2 + x 5 + x 8 + x 11 ≥ 1 :: Flight 11 covered:x 6 + x 9 + x 10 + x 11 + x 12 ≥ 1 Three Crews:x 1 + x 2 + x 3 + x 4 + x 5 + x 6 + x 7 + x 8 + x 9 + x 10 + x 11 + x 12 ≤ 3 and x j are binary (j = 1, 2, …, 12). pairings

65. Spreadsheet Model

66. Set Covering Problem We are given a set of objects S = {1, 2, 3, …, n}. We are also given S, a set of subsets of S. Each subset has a cost associated with it. Problem: –to “cover” all the members of S at the minimum cost using members of S. Properties: –The problem is a minimization and all constraints are >=; –All RHS coefficients are 1; –All other matrix coefficients are 0 or 1.

67. Some Comments on IP models There are often multiple ways of modeling the same integer program. Solvers for integer programs are extremely sensitive to the formulation. (not true for LPs)

68. Example constraint A: 2x 1 + 2x 2 + … + 2x 50  51 constraint B: x 1 + x 2 + … + x 50  25 assume that x is binary constraints C: x 1  y, x 2  y, …, x 50  y (where y is binary) constraint D: x 1 + … + x 50  50 y B dominates A, C dominates D It is not obvious why, until you see the algorithms.

69. Summary on Integer Programming Dramatically improves the modeling capability –Economic indivisibilities –Logical constraints –Modeling nonlinearities (e.g., fixed cost) –classical problems in capital budgeting and in supply chain management –Lots of other applications and models Not as easy to model Not as easy to solve.

70. Solving LP

71. The Challenges of Rounding Rounded Solution may not be feasible. Rounded solution may not be close to optimal. There can be many rounded solutions. –Example: Consider a problem with 30 variables that are non- integer in the LP-solution. How many possible rounded solutions are there?

72. Overview of Techniques for Solving Integer Programs Enumeration Techniques –Complete Enumeration list all “solutions” and choose the best –Branch and Bound Implicitly search all solutions, but cleverly eliminate the vast majority before they are even searched –Implicit Enumeration Branch and Bound applied to binary variables Cutting Plane Techniques –Use LP to solve integer programs by adding constraints to eliminate the fractional solutions.

73. How Integer Programs are Solved

75. Branch and Bound –It is the starting point for all solution techniques for integer programming –Lots of research has been carried out over the past 40 years to make it more and more efficient –But, it is an art form to make it efficient. (We shall get a sense why.) –Integer programming is intrinsically difficult.

76. Capital Budgeting Example Investment budget = $14,000 maximize 16x 1 + 22x 2 + 12x 3 + 8x 4 +11x 5 + 19x 6 subject to 5x 1 + 7x 2 + 4x 3 + 3x 4 +4x 5 + 6x 6  14 x j binary for j = 1 to 6

77. Complete Enumeration Systematically considers all possible values of the decision variables. –If there are n binary variables, there are 2 n different ways. Usual idea: iteratively break the problem in two. At the first iteration, we consider separately the case that x 1 = 0 and x 1 = 1.

78. An Enumeration Tree x 1 = 0x 1 = 1 x 2 = 0 x 2 = 1 x 2 = 0 x 2 = 1 x 3 = 0x 3 = 1x 3 = 0x 3 = 1x 3 = 0x 3 = 1x 3 = 0x 3 = 1 Original problem

79. On complete enumeration Suppose that we could evaluate 1 billion solutions per second. Let n = number of binary variables Solutions times (approx.) –n = 30, 1 second –n = 40, 18 minutes –n = 50 13 days –n = 60 31 years

80. On complete enumeration Suppose that we could evaluate 1 trillion solutions per second, and instantaneously eliminate 99.9999999% of all solutions as not worth considering Let n = number of binary variables Solutions times –n = 70, 1 second –n = 80, 17 minutes –n = 90 11.6 days –n = 100 31 years

81. Branch and Bound The essential idea: search the enumeration tree, but at each node 1.Solve the linear program at the node 2.Eliminate the subtree (fathom it) if 1.The solution is integer (there is no need to go further- why?) or 2.The best solution in the subtree cannot be as good as the best available solution (the incumbent- how does that happen?) or 3.There is no feasible solution

82. Branch and Bound 1 44 3/7 Solution at node 1: x 1 =1 x 2 = 3/7 x 3 = x 4 = x 5 = 0 x 6 =1 z = 44 3/7 Node 1 is the original LP Relaxation maximize 16x 1 + 22x 2 + 12x 3 + 8x 4 +11x 5 + 19x 6 subject to 5x 1 + 7x 2 + 4x 3 + 3x 4 +4x 5 + 6x 6  14 0  x j  1 for j = 1 to 6 The IP cannot have value higher than 44 3/7.

83. Branch and Bound 1 2 x 1 = 0 44 3/7 44 Solution at node 2: x 1 = 0 x 2 = 1 x 3 = 1/4 x 4 = x 5 = 0 x 6 = 1 z = 44 Node 2 is the original LP Relaxation plus the constraint x 1 = 0. maximize 16x 1 + 22x 2 + 12x 3 + 8x 4 +11x 5 + 19x 6 subject to 5x 1 + 7x 2 + 4x 3 + 3x 4 +4x 5 + 6x 6  14 0  x j  1 for j = 1 to 6, x 1 = 0

84. Branch and Bound 1 2 x 1 = 0 44 3/7 44 Node 3 is the original LP Relaxation plus the constraint x 1 = 1. 3 x 1 = 1 The solution at node 1 was x 1 =1 x 2 = 3/7 x 3 = x 4 = x 5 = 0 x 6 =1 z = 44 3/7 Note: it was the best solution with no constraint on x 1. So, it is also the solution for node 3. (If you add a constraint, and the old optimal solution is feasible, then it is still optimal.) 44 3/7

85. Branch and Bound 1 23 x 1 = 0x 1 = 1 44 3/7 44 44 3/7 Solution at node 4: 0 0 1 0 1 1 z = 42 4 x 2 = 0 42 Our first incumbent solution! No further searching from node 4 because there cannot be a better integer solution. Node 4 is the original LP Relaxation plus the constraints x 1 = 0, x 2 = 0. 4 No solution in the subtree can have a value better than 42. 1

86. Branch and Bound 1 23 x 1 = 0x 1 = 1 44 3/7 44 44 3/7 We next solved the LP’s associated with nodes 5, 6, and 7 4 x 2 = 0 42 No new integer solutions were found. 5 x 2 = 1 6 x 2 = 0 7 x 2 = 1 44 44 1/3 The incumbent solution has value 42 4 We would eliminate (fatham) a subtree if we were guaranteed that no solution in the subtree were better than the incumbent. The incumbent is the best solution on hand. 1

87. Branch and Bound 1 23 x 1 = 0x 1 = 1 44 3/7 44 44 3/7 We next solved the LP’s associated with nodes 8 -13 4 x 2 = 0 42 5 x 2 = 1 6 x 2 = 0 7 x 2 = 1 44 44 1/3 The incumbent solution has value 42 8 x 3 = 0 9 x 3 = 1 10 x 3 = 0 11 x 3 = 1 12 x 3 = 0 13 x 3 = 1 43.7543.543.2543.844.3 -- 4 13

88. Summary so far We have solved 13 different linear programs so far. –One integer solution found –One subtree fathomed (pruned) because the solution was integer (node 4) –One subtree fathomed because the solution was infeasible (node 13) –No subtrees fathomed because of the bound

89. Branch and Bound 1 23 x 1 = 0x 1 = 1 44 3/7 44 44 3/7 We next solved the LP’s associated with the next nodes. 4 x 2 = 0 42 5 x 2 = 1 6 x 2 = 0 7 x 2 = 1 44 44 1/3 The incumbent solution has value 42 8 x 3 = 0 9 x 3 = 1 10 x 3 = 0 11 x 3 = 1 12 x 3 = 0 13 x 3 = 1 43.7543.543.2543.844.3 -  43.7542.66 We can fathom the node with z = 42.66. Why?

90. Getting a better bound The bound at each node is obtained by solving an LP. But all costs are integer, and so the objective value of each integer solution is integer. So, the best integer solution has an integer objective value. If the best integer valued solution for a node is at most 42.66, then we know the best bound is at most 42. Other bounds can also be rounded down.

91. Branch and Bound 1 23 x 1 = 0x 1 = 1 44 3/7 44 44 3/7 4 x 2 = 0 42 5 x 2 = 1 6 x 2 = 0 7 x 2 = 1 44 44 1/3 The incumbent solution has value 42 8 x 3 = 0 9 x 3 = 1 10 x 3 = 0 11 x 3 = 1 12 x 3 = 0 13 x 3 = 1 43.7543.543.2543.844.3 -.-. 43.7542.66

92. Branch and Bound 1 23 x 1 = 0x 1 = 1 44 4 x 2 = 0 42 5 x 2 = 1 6 x 2 = 0 7 x 2 = 1 44 The incumbent solution has value 42 8 x 3 = 0 9 x 3 = 1 10 x 3 = 0 11 x 3 = 1 12 x 3 = 0 13 x 3 = 1 43 44 -- 4342

93. Branch and Bound 1 23 x 1 = 0x 1 = 1 44 4 x 2 = 0 42 5 x 2 = 1 6 x 2 = 0 7 x 2 = 1 44 The incumbent solution has value 42 8 x 3 = 0 9 x 3 = 1 10 x 3 = 0 11 x 3 = 1 12 x 3 = 0 13 x 3 = 1 43 44 -- 4342434243 We found a new incumbent solution! x 1 = 1, x 2 = x 3 = 0, x 4 = 1, x 5 = 0, x 6 = 1 z = 43

94. Branch and Bound 1 23 x 1 = 0x 1 = 1 44 4 x 2 = 0 42 5 x 2 = 1 6 x 2 = 0 7 x 2 = 1 44 The new incumbent solution has value 43 8 x 3 = 0 9 x 3 = 1 10 x 3 = 0 11 x 3 = 1 12 x 3 = 0 13 x 3 = 1 43 44 -- 4342434243 We found a new incumbent solution! x 1 = 1, x 2 = x 3 = 0, x 4 = 1, x 5 = 0, x 6 = 1 z = 43 891011

95. Branch and Bound 1 23 x 1 = 0x 1 = 1 44 4 x 2 = 0 42 5 x 2 = 1 6 x 2 = 0 7 x 2 = 1 44 The new incumbent solution has value 43 8 x 3 = 0 9 x 3 = 1 10 x 3 = 0 11 x 3 = 1 12 x 3 = 0 13 x 3 = 1 43 44 -- If we had found this incumbent earlier, we could have saved some searching. 891011

96. Finishing Up 1 23 x 1 = 0x 1 = 1 44 4 x 2 = 0 42 5 x 2 = 1 6 x 2 = 0 7 x 2 = 1 44 The new incumbent solution has value 43 8 x 3 = 0 9 x 3 = 1 10 x 3 = 0 11 x 3 = 1 12 x 3 = 0 13 x 3 = 1 43 44 -- 891011 -- -- 1415 1617 44 1819 -- 38

97. Lessons Learned Branch and Bound can speed up the search Only 25 nodes (linear programs) were evaluated Other nodes were fathomed Obtaining a good incumbent earlier can be valuable only 19 nodes would have been evaluated. Solve linear programs faster, because we start with an excellent or optimal solution uses a technique called the dual simplex method Obtaining better bounds can be valuable. We sometimes use properties that are obvious to us, such as the fact that integer solutions have integer solution values

98. Branch and Bound Notation: –z* = optimal integer solution value –Subdivision: a node of the B&B Tree –Incumbent: the best solution on hand –z I : value of the incumbent –z LP : value of the LP relaxation of the current node –Children of a node: the two problems created for a node, e.g., by saying x j = 1 or x j = 0. –LIST: the collection of active (not fathomed) nodes, with no active children. NOTE: z I  z*

99. Illustrating the definitions 1 23 x 1 = 0x 1 = 1 44 3/7 44 44 3/7 4 x 2 = 0 42 5 x 2 = 1 6 x 2 = 0 7 x 2 = 1 44 44 1/3 8 x 3 = 0 9 x 3 = 1 10 x 3 = 0 11 x 3 = 1 12 x 3 = 0 13 x 3 = 1 43.75 43.5 43.25 43.844.3 -- 4 13 z* = 43 = optimal integer solution value. (We found it later in the search) Incumbent is 0 0 1 0 1 1 z I = 42. It is the optimal solution for the subdivision 4. The incumbent solution has value 42 The z LP values for each subdivision are next to the nodes.

100. Illustrating the definitions 1 23 x 1 = 0x 1 = 1 44 3/7 44 44 3/7 4 x 2 = 0 42 5 x 2 = 1 6 x 2 = 0 7 x 2 = 1 44 44 1/3 8 x 3 = 0 9 x 3 = 1 10 x 3 = 0 11 x 3 = 1 12 x 3 = 0 13 x 3 = 1 43.75 43.5 43.25 43.844.3 -- 4 13 The children of node (subdivision) 1 are nodes 2 and 3. The children of node 3 are nodes 6 and 7. LIST = { 8, 9, 10, 11, 12 } = unfathomed nodes with no active children The incumbent solution has value 42

101. Branch and Bound Algorithm INITIALIZE LIST = {original problem} Incumbent: =  z I = -  SELECT: If LIST = , then the Incumbent is optimal if it exists, and the problem is infeasible if no incumbent exists; else, let S be a node (subdivision) from LIST. Let x LP be the optimal solution to S Let z LP = its objective value CASE 1. z LP = -  (the LP is infeasible) Remove S from LIST (fathom it) Return to SELECT 1 44 3/7 e.g., S = {1} 13 -- e.g., S = {13} 13

102. Branch and Bound Algorithm INITIALIZE SELECT: If LIST = , then the Incumbent is optimal (if it exists), and the problem is infeasible if no incumbent exists; else, let S be a node from LIST. Let x LP be the optimal solution to S Let z LP = its objective value CASE 2. -  < z LP  z I. That is, the LP is dominated by the incumbent. Then remove S from LIST (fathom it) Return to SELECT 8 43 14 43 8 42 e.g., the incumbent has value 43, and node 14 is selected. z LP = 43. 14

103. Branch and Bound Algorithm INITIALIZE SELECT: If LIST = , then the Incumbent is optimal (if it exists), and the problem is infeasible if no incumbent exists; else, let S be a subdivision from LIST. Let x LP be the optimal solution to S Let z LP = its objective value CASE 3. z I < z LP and x LP is integral. That is, the LP solution is integral and dominates the incumbent. Then Incumbent := x LP ; z I := z LP Remove S from LIST (fathomed by integrality) Return to SELECT 42 4 4 e.g., node 4 was selected, and the solution to the LP was integer-valued.

104. Branch and Bound Algorithm INITIALIZE SELECT: If LIST = , then the Incumbent is optimal (if it exists), and the problem is infeasible if no incumbent exists; else, let S be a subdivision from LIST. Let x LP be the optimal solution to S Let z LP = its objective value CASE 4. z I < z LP and x LP is not integral. There is not enough information to fathom S Remove S from LIST Add the children of S to LIST Return to SELECT 44 3/7 6 x 2 = 0 7 x 2 = 1 3 x 1 = 1 e.g., select node 3. List := List – 3 + {6,7}

105. Different Selection Rules are Possible Rule of Thumb 1: Don’t let LIST get too big (the solutions must be stored). So, prefer nodes that are further down in the tree. Rule of Thumb 2: Pick a node of LIST that is likely to lead to an improved incumbent. Sometimes special heuristics are used to come up with a good incumbent.

106. X X X X XX X X X X X 2 1 1 2 3 3 3 3 3 3 3 = 0 = 1 = 0 = 1 = 0 = 1 = 0 = 1 XX 8 8 = 0 = 1................................................ Branching One does not have to have the B&B tree be symmetric, and one does not select subtrees by considering variables in order. Choosing how to branch so as to reduce running time is largely “art” and based on experience. X 3

107. Different Branching Rules are Possible Branching: determining children for a node. There are many choices. Rule of thumb 1: if it appears clear that x j = 1 in an optimal solution, it is often good to branch on x j = 0 vs x j = 1. –The hope is that a subdivision with x j = 0 can be pruned. Rule of thumb 2: branching on important variables is worthwhile

108. Different Bounding Techniques are Possible We use the bound obtained by dropping the integrality constraints (LP relaxation). There are other choices. Key tradeoff for bounds: time to obtain a bound vs quality of the bound. If one can obtain a bound much quicker, sometimes we would be willing to get a bound that is worse It usually is worthwhile to get a bound that is better, so long as it doesn’t take too long.

109. What if the variables are general integer variables? One can choose children as follows: –child 1: x 1  3 (or x j  k) –child 2 x 1  4 (or x j  k+1) How would one choose the variable j and the value k –A common choice would be to take a fractional value from x LP. e.g., if x 7 = 5.62, then we may branch on x 7  5 and x 7  6. –Other choices are also possible.

110. Branch and Bound is the standard way of solving IPs to optimality. There is art to making it work well in practice. Much of the art is built into state-of-the-art solvers such as CPLEX. New exciting area  combining LP based branch-and-bound based techniques with constraint programming techniques - forward checking; arc-consistency and other forms of consistency checking and propagation!

111. A* - well-known AI algorithm that is a generalization of branch-and-bound with LP relaxations – notion of admissible heuristic that overestimates (underestimates) the objective function for a maximization (minimization) problem (analogously to what a relaxation does) Used in e.g. MapQuest and Darpa Challenge

112. It was fun to teach INFO 372! Hope you had fun too! THE END !!!