Integer Programming 3 Brief Review of Branch and Bound Cutting plane techniques for getting improved bounds MIT and James Orlin © 2003

Review from Last Lecture Investment budget = $14,000 maximize 16x1 + 22x2 + 12x3 + 8x4 +11x5 + 19x6 subject to 5x1 + 7x2 + 4x3 + 3x4 +4x5 + 6x6  14 xj binary for j = 1 to 6 MIT and James Orlin © 2003

Branch and Bound The incumbent solution has value 43 1 x1 = 0 x1 = 1 2 44 x1 = 0 x1 = 1 2 3 44 44 x2 = 0 5 x2 = 1 6 x2 = 0 7 x2 = 1 42 4 44 44 44 x3 = 0 x3 = 1 x3 = 0 x3 = 1 x3 = 0 x3 = 1 8 9 10 11 43 8 43 9 43 10 43 11 44 12 - 13 - 14 15 16 17 44 18 19 38 MIT and James Orlin © 2003

On Bounds from Linear Programming We found an incumbent integer solution with a value of 43. The best LP solution value is between 44 and 45. Is there a way that we could have directly established an upper bound between 43 and 44? Perhaps we could form a “better linear program.” The closer the LP is to the Integer Program, the better. Note: not all LP formulations are the same in terms of “closeness” MIT and James Orlin © 2003

Overview of cutting planes Try to get better bounds so that more nodes of the branch and bound tree can be fathomed, and the fathoming can be done earlier. Recall: we eliminate a node j if the LP bound for j is no more than the value of the incumbent. The lower the bound (for a maximization problem) the better This lecture will provide several approaches for obtaining bounds. MIT and James Orlin © 2003

Goals of this lecture Illustrate how valuable obtaining better LP’s can be Illustrate techniques for obtaining better bounds set packing capital budgeting traveling salesman problem general integer programs MIT and James Orlin © 2003

What is the maximum number of diamonds that can be packed on a Chinese checkerboard. Special Thanks to Professor Andreas Schulz for this example. MIT and James Orlin © 2003

The diamonds are not permitted to overlap, or even to share a single circle. What is the best packing you can find? MIT and James Orlin © 2003

Is this the best possible? MIT and James Orlin © 2003

Set Packing Problem Let D be the collection of diamonds Let O be the pairs of diamonds that overlap. (d, d’)  O, implies that diamonds d and d’ have a point in common Decision variables: xd for d  D xd = 1 if diamond d is selected xd = 0 if diamond d is not selected. MIT and James Orlin © 2003

How many decision variables are there? For each black circle, one can hang a yellow diamond from it. There are 88 black circles. And there are 88 green and red diamonds. So, there are 88 possible yellow diamonds. for a total of 264 diamonds. MIT and James Orlin © 2003

The First Integer Programming Formulation This problem is known as the set packing problem. The optimal objective value for the LP relaxation is 132. The optimal objective value for this IP is 27. MIT and James Orlin © 2003

For many dots, there are 12 diamonds that go through the dot. Heading towards an improved IP formulation. 2 5 6 8 7 1 3 4 MIT and James Orlin © 2003

Getting an improved LP We say that S is an overlapping set of diamonds, if every two diamonds in S overlap. x1 + x2  1 x1 + x3  1 x1 + x4  1 x2 + x3  1 x2 + x4  1 x3 + x4  1 x1, x2, x3, x4  {0,1} Conclusion: x1 + x2 + x3 + x4  1 for each overlapping set S. MIT and James Orlin © 2003

An improved integer program The optimal objective value for this IP is 27 Next: a proof that the at most 27 diamonds can be packed. The optimal objective value for the LP relaxation is 27.5 MIT and James Orlin © 2003

The weight of a diamond is the sum of its node weights. 1 1/2 1/3 1/6 The weight of a diamond is the sum of its node weights. MIT and James Orlin © 2003

The weight of a diamond is the sum of its node weights. 1 1/2 1/3 1/6 The weight of a diamond is the sum of its node weights. Each diamond has weight at least 1. MIT and James Orlin © 2003

The weight of 27 independent diamonds is at least 27. Node Weight 1 1/2 1/3 1/6 18 1 24 6 The weight of 27 independent diamonds is at least 27. The weight of 28 independent diamonds would be at least 28. But the total weight of all nodes is 27.5 MIT and James Orlin © 2003

No solution has more than 27 diamonds. Node Weight 1 1/2 1/3 1/6 No solution has more than 27 diamonds. The weights were the shadow prices for the LP MIT and James Orlin © 2003

The Optimal Solution Again MIT and James Orlin © 2003

Set Packing Problem Putting items into storage Manufacturing How many car doors pack into a metal sheet? How many rolls of paper can be cut from a giant roll? Closely related to fire station problem: set covering. MIT and James Orlin © 2003

Review There may be different ways of formulating an IP. Each way gives the same IP The LPs may be very different Some LPs have different bounds NEXT: The geometry of Integer Programs and Linear Programs MIT and James Orlin © 2003

Using Cutting Planes y P x Optimum (integer) solution P Optimum fractional (i.e. infeasible) solution x Example. Minimize x + 10y subject to x, y are in P x, y integer

Using Cutting Planes y P x Idea: add constraints that eliminate fractional solutions to the LP without eliminating any integer solutions. y P add “y 1” add “y  x –1” x Example. Minimize x + 10y subject to x, y are in P x, y integer These constraints were obtained by inspection. We will develop techniques later.

Using Cutting Planes y P x If we add exactly the right inequalities, then every corner point of the LP will be integer, and the IP can be solved by solving the LP y Optimum (integer) solution We call this minimal LP, the convex hull of the IP solutions. P x Example. Minimize x + 10y subject to x, y are in P x, y integer For large problems, these constraints are hard to find.

More on adding constraints The tightest possible constraints are very useful, and are called facets. Suppose that we are maximizing, and zLP is the opt for the LP relaxation, and zIP is the opt for the IP. Then zIP  zLP Ideally, we want zIP to be close to zLP. This is GREAT for branch and bound. Adding lots of valid inequalities can be very helpful. It has no effect on zIP. It can reduce zLP significantly. MIT and James Orlin © 2003

“Pure” Cutting Plane Technique Instead of partitioning the feasible region, the (pure) cutting plane technique works with a single LP It adds cutting planes (valid linear programming inequalities) to this LP iteratively. At each iteration the feasible region is successively reduced until an integer optimal is found by solving the LP. In practice, it is also used as part of branch and bound. The essential idea is finding valid “cuts” or inequalities. MIT and James Orlin © 2003

Where do these cuts come from? Two approaches Problem specific Set Packing Capital Budgeting (knapsack) Traveling Salesman Problem LP-based approach, that works for general integer programs Gomory cutting planes MIT and James Orlin © 2003

Problem Specific The capital budgeting (knapsack) problem maximize 16x1 + 22x2 + 12x3 + 8x4 +11x5 + 19x6 subject to 5x1 + 7x2 + 4x3 + 3x4 +4x5 + 6x6  14 xj binary for j = 1 to 6 MIT and James Orlin © 2003

The LP Relaxation The capital budgeting (knapsack) problem maximize 16x1 + 22x2 + 12x3 + 8x4 +11x5 + 19x6 subject to 5x1 + 7x2 + 4x3 + 3x4 +4x5 + 6x6  14 0  xj  1 for j = 1 to 6 The optimal solution: x1 = 1, x2 = 3/7, x3 = 0 x4 = 0, x5 = 0, x6 = 1 We say that a subset S is a cover if the sum of its weights is more than 14 (more than the budget) What are some covers of this capital budgeting problem? MIT and James Orlin © 2003

Getting constraints from covers 5x1 + 7x2 + 4x3 + 3x4 +4x5 + 6x6  14 Consider the cover S = {1, 2, 3} We have the constraint 5x1 + 7x2 + 4x3  14 We can obtain the constraint x1 + x2 + x3  2. This is a cover constraint This constraint is “stronger” It does not eliminate any IP solutions, but it cuts off fractional LP solutions. In general, for each cover S,we obtain the constraint SjS xj  |S| - 1 MIT and James Orlin © 2003

The LP Relaxation The capital budgeting (knapsack) problem maximize 16x1 + 22x2 + 12x3 + 8x4 +11x5 + 19x6 subject to 5x1 + 7x2 + 4x3 + 3x4 +4x5 + 6x6  14 0  xj  1 for j = 1 to 6 The optimal solution: x1 = 1, x2 = 3/7, x3 = 0 x4 = 0, x5 = 0, x6 = 1 z = 44 3/7 5x1 + 7x2 + 6x6  14  x1 + x2 + x6  2 Our approach: Given the linear solution, try to find a violated cover constraint. A violated cover is {1, 2, 6} Excel MIT and James Orlin © 2003

After one cut maximize 16x1 + 22x2 + 12x3 + 8x4 +11x5 + 19x6 subject to 5x1 + 7x2 + 4x3 + 3x4 +4x5 + 6x6  14 x1 + x2 + x6  2 0  xj  1 for j = 1 to 6 The optimal solution: x1 = 0, x2 = 1, x3 = 1/4 x4 = 0, x5 = 0, x6 = 1 z = 44 Exercise: find a violated cover. Excel MIT and James Orlin © 2003

After two cuts maximize 16x1 + 22x2 + 12x3 + 8x4 +11x5 + 19x6 subject to 5x1 + 7x2 + 4x3 + 3x4 +4x5 + 6x6  14 x1 + x2 + x6  2 x2 + x3 + x6  2 0  xj  1 for j = 1 to 6 The optimal solution: x1 = 1/3, x2 = 1, x3 = 1/3 x4 = 0, x5 = 0, x6 = 1 z = 44 Exercise: Find a violated cover: Excel MIT and James Orlin © 2003

Obtaining a stronger constraint 5x1 + 7x2 + 4x3 + 6x6  14 At most 2 of x1, x2, x3, x6 can be 1. So x1 + x2 + x3 + x6  2. MIT and James Orlin © 2003

After “three” cuts maximize 16x1 + 22x2 + 12x3 + 8x4 +11x5 + 19x6 subject to 5x1 + 7x2 + 4x3 + 3x4 +4x5 + 6x6  14 x1 + x2 + x6  2 x2 + x3 + x6  2 x1 + x2 + x3 + x6  2 0  xj  1 for j = 1 to 6 Note: the new cuts dominates the other cuts. Excel MIT and James Orlin © 2003

We eliminate the redundant constraints maximize 16x1 + 22x2 + 12x3 + 8x4 +11x5 + 19x6 subject to 5x1 + 7x2 + 4x3 + 3x4 +4x5 + 6x6  14 x1 + x2 + x3 + x6  2 0  xj  1 for j = 1 to 6 The optimal solution: x1 = 0, x2 = 1, x3 = 0, x4 = 0, x5 = 1/4, x6 = 1 z = 43 3/4 So, z*  43 Excel MIT and James Orlin © 2003

Summary for knapsack problem We could find some simple valid inequalities that showed that z*  43. This is the optimal objective value. It took 3 cuts Had we been smarter it would have taken 1 cut We had a simple approach for finding cuts. This does not find all of the cuts. Recall, it took 25 nodes of a branch and bound tree In fact, researchers have found cutting plane techniques to be necessary to solve large integer programs (usually as a way of getting better bounds.) MIT and James Orlin © 2003

Traveling Salesman Problem (TSP) What is a minimum length tour that visits each point? MIT and James Orlin © 2003

Comments on the TSP Very well studied problem It’s often the problem for testing out new algorithmic ideas NP-complete (it is intrinsically difficult in some technical sense) Large instances have been solved optimally (5000 cities and larger) Very large instances have been solved approximately (10 million cities to within a couple of percent of optimum.) We will formulate it by adding constraints that look like cuts MIT and James Orlin © 2003

The TSP as an IP, almost Are these constraints enough? Let xe = 1 if arc e is in the tour xe = 0 otherwise Let A(i) = arcs incident to node i Minimize Se ce xe subject to SeA(i) xe = 2 xe is binary Are these constraints enough? MIT and James Orlin © 2003

Subtour: a cycle that passes through a strict subset of cities Fact: Any integer solution with exactly two arcs incident to every node is the union of cycles. Why? 3 2 4 7 9 Improved IP/LP: for each possible subtour, add a constraints that makes the subtour infeasible for the IP. These are called subtour breaking constraints. MIT and James Orlin © 2003

A subtour breaking constraint Let S be any proper subset of nodes, e.g., S = {2, 3, 4, 7, 9}. 7 2 9 3 4 Observations: A subtour that includes all nodes of S has |S| arcs 2. A tour for the entire network has at most |S| - 1 arcs with two endpoints in S. This ensures that the set S will not have a subtour going through all five nodes. MIT and James Orlin © 2003

A traveling salesman tour 7 2 9 3 4 There are at most 4 arcs of any tour incident to 2, 3, 4, 7, 9 MIT and James Orlin © 2003

Formulation of the TSP as an IP Minimize Se ce xe subject to Se inc to i xe = 2 Se in S xe  |S| - 1 (subtour breaking constraints) xe is binary Exponentially many constraints, too many to include in an IP or an LP In practice: Include only some of the constraints. Solve the LP. If a subtour appears as part of the LP solution, add a new subtour elimination constraint to the LP, and solve again. MIT and James Orlin © 2003

More on the TSP The IP formulation is a very good formulation good in the following sense: for practical problem the LP bound is usually 1% to 2% from the optimal TSP tour length. (The LP bound is close.) One can add even more complex constraints to get a better LP formulation, and people do. (and it helps) MIT and James Orlin © 2003

Gomory Cuts: a method of generating cuts using LP tableaus Consider the following constraint in an integer program x1 x2 x4 x3 1 3/5 4 1/5 2/5 3 = 9 4/5 Focus on the fractions.  3/5 x1 + 1/5 x2 +2/5 x4 = Integer + 4/5  3/5 x1 + 1/5 x2 + 2/5 x4  4/5  3 x1 + x2 + 2 x4  4. MIT and James Orlin © 2003

Gomory Cuts What did we rely on to obtain the Gomory cut? x1 x2 x3 x4 1 3/5 4 1/5 3 2/5 9 4/5 = A single constraint with a fractional right hand side All coefficients of the constraint were positive. All variables must be integral MIT and James Orlin © 2003

What to do if coefficients are negative x1 x2 x4 x3 1 3/5 -4 3/5 -2/5 -3 = -1 1/5 Rewrite so that the coefficients of the fractional parts are positive. 1 3/5 -5 +2/5 -3 -1 +3/5 -2 +4/5 =  3/5 x1 + 2/5 x2 +3/5 x4 = integer + 4/5 .  3/5 x1 + 2/5 x2 +3/5 x4  4/5 .  3 x1 + 2 x2 + 3 x4  4 . MIT and James Orlin © 2003

In general   Let fr(a) be the positive fractional part of a ; fr(a) = a - a fr(2 3/5) = 2 3/5 – 2 = 3/5 fr(-2 3/5) = - 2 3/5 – (-3) = 2/5 x1 x2 x3 x4 1 3/5 -4 3/5 -3 -2/5 -1 1/5 = 3/5 2/5 +3/5  +4/5 x1 x2 x3 x4 ai1 ai2 ai3 ai4 bi = fr(ai1) fr(ai2) fr(ai3) fr(ai4)  fr(bi) MIT and James Orlin © 2003

How to generate cutting planes? In general After pivoting, find a basic variable that is fractional. Write a Gomory cut. (It is an inequality constraint, leading to a new slack variable). Note: the only coefficients that are fractional correspond to non-basic variables (why?) The Gomory cut makes the previous basic feasible solution infeasible. (why?) Resolve the LP with the new constraint, and iterate. MIT and James Orlin © 2003

Integer Programming Summary Dramatically improves the modeling capability Economic indivisibilities Logical constraints Modeling non-linearities Not as easy to model. Not as easy to solve. MIT and James Orlin © 2003

IP Solution Techniques Summary Branch and Bound very general and adaptive used in practice (e.g. Excel Solver) Implicit Enumeration branch and bound technique for binary IPs Cutting planes clever way of improving bounding active area for research, theoretical and applied MIT and James Orlin © 2003

Example: Fire company location. Consider locating fire companies in different districts. Objective: place fire companies so that each district either has a fire company in it, or one that is adjacent, and so as to minimize cost. MIT and James Orlin © 2003

Example for the Fire Station Problem Let xj = 1 if a fire station is placed in district j. xj = 0 otherwise. 1 2 3 5 6 7 Let cj = cost of putting a fire station is district j. 4 8 9 With partners, formulate the fire station problem. 11 12 13 10 14 15 16 MIT and James Orlin © 2003

Set covering problem set S= {1, …, m} of items to be covered districts that need to have a fire station or be next to a district with one n subsets of S For each possible location j for a fire station, the subset is district j plus the list of districts that are adjacent to district j. aij = 1 if district i is adjacent to district j or if i = j. Minimize subject to MIT and James Orlin © 2003

Covering constraints are common Fleet routing for airlines: Assigning airplanes to flight legs. Each flight must be included Assigning crews to airplanes Each plane must be assigned a crew Warehouse location Each retailer needs to be served by some warehouse MIT and James Orlin © 2003

Independent Set (set packing) What is the maximum number of districts no two of which have a common border? 1 2 3 5 6 7 4 With your partner, formulate the independent set problem. List all constraints containing x10. 8 9 11 12 13 10 14 15 16 MIT and James Orlin © 2003

Set Packing constraints arise often in manufacturing and logistics Cutting shapes out of sheet metal Manufacturing many items at once which share resources (how much work can be done in parallel?) MIT and James Orlin © 2003