1. Instructors: Fei Fang (This Lecture) and Dave Touretzky
Artificial Intelligence: Representation and Problem Solving Optimization (3): (Mixed) Integer Linear Programming
/ 681
Instructors: Fei Fang (This Lecture) and Dave Touretzky
Wean Hall 4126
Don’t expect them to remember, just use as resource
12/8/2018

2. Nonconvex but continuous optimization Linear program
Recap
Convex optimization
𝑓 is a convex function and ℱ is a convex set
Gradient descent leads to global optimum
Nonconvex but continuous optimization
Gradient descent leads to local optima
Linear program
𝑓 is a linear function and ℱ is a convex polytope
This section: 𝑓 is a linear function and ℱ is defined by a set of linear constraints + integer constraints
min 𝑥 𝑓(𝑥)
s.t. 𝑥∈ℱ
Properties
Finite options (brute force when the problem scale is small)
(Often) NP-Complete
Solution approach
Brute force when the problem scale is small
Fei Fang
12/8/2018

3. (Mixed) Integer Linear Program How to solve a MILP
Outline
(Mixed) Integer Linear Program
How to solve a MILP
Formulate/Convert to MILP
Fei Fang
12/8/2018

4. (Mixed) Integer Linear Program: Definition
A special case of non-convex optimization problem
An optimization problem whose optimization objective is a linear function and feasible region is defined by a set of linear constraints + integer constraints
𝑐∈ ℝ 𝑛
𝐺∈ ℝ 𝑚×𝑛 ,ℎ∈ ℝ 𝑚
0< 𝐽 𝑧 ≤𝑛
Integer Linear Program (ILP): 𝐽 𝑧 =𝑛
max 𝑥 𝑐 𝑇 𝑥
s.t. 𝐺𝑥≤ℎ
𝑥 𝑖 ∈ℤ,𝑖∈ 𝐽 𝑧
Can be minimization
Can be ≥ℎ or =ℎ
Efficient solvers / code packages available!
Fei Fang
12/8/2018

5. (Mixed) Integer Linear Program: Example
Example: Maximize Profit in Manufacturing
max 𝑥,𝑦 30𝑥+30𝑦
s.t.
0.2𝑥+0.5𝑦≤90 4𝑥+2𝑦≤800 𝑥≥0,𝑦≥0
𝑥,𝑦∈ℤ
Original product can be: cloth, rope etc
Price
Labor
Machine
Product 1
$30
0.2 hour
4 hour
Product 2
0.5 hour
2 hour
Total
<=90
<=800
Earphone
Charger
Fei Fang
12/8/2018

6. Binary Linear Program: Definition
Binary Linear Program (BIP):
All variables are restricted to take value 0 or 1
max 𝑥 𝑐 𝑇 𝑥
s.t. 𝐺𝑥≤ℎ
𝑥 𝑖 ∈ 0,1 ,∀𝑖
Fei Fang
12/8/2018

7. Binary Linear Program: Example
0-1 Knapsack
Maximum weight =10
How to select items to maximize total value?
Items 1 2 3 4 5
Weight 6 7
Value 9
You have a knapsack, and the maximum weight you can take is 10
Fei Fang
12/8/2018

8. Binary Linear Program: Example
0-1 Knapsack
𝑛 indivisible items. Item 𝑖 has weight 𝑤 𝑖 , value 𝑣 𝑖 .
Maximum weight is 𝑊 (𝑊≤ 𝑖 𝑤 𝑖 )
How to pick the items to maximize total value?
You have a knapsack, and the maximum weight you can take is 10
Fei Fang
12/8/2018

9. (Mixed) Integer Linear Program How to solve a MILP
Outline
(Mixed) Integer Linear Program
How to solve a MILP
Formulate/Convert to MILP
Fei Fang
12/8/2018

10. LP Relaxation
LP relaxation of an MILP or BLP is the LP with the same linear constraints
LP Relaxation
MILP
max 𝑥 𝑐 𝑇 𝑥
s.t. 𝐺𝑥≤ℎ
𝑥 𝑖 ∈ℤ,𝑖∈ 𝐽 𝑧
max 𝑥 𝑐 𝑇 𝑥
s.t. 𝐺𝑥≤ℎ
BLP
LP Relaxation
max 𝑥 𝑐 𝑇 𝑥
s.t. 𝐺𝑥≤ℎ
𝑥 𝑖 ∈ 0,1 ,∀𝑖
max 𝑥 𝑐 𝑇 𝑥
s.t. 𝐺𝑥≤ℎ
𝑥 𝑖 ∈[0,1]
Fei Fang
12/8/2018

11. Quiz 1: LP Relaxation
Let 𝑥 ∗ , 𝑓 ∗ be the optimal solution and the optimal value of a MILP. Let 𝑥 ∗ , 𝑓 ∗ be the optimal solution and the optimal value of the LP relaxation. Which of the following are true?
A: 𝑥 ∗ = 𝑥 ∗
B: 𝑓 ∗ ≤ 𝑓 ∗ if it is a maximization problem
C: 𝑓 ∗ ≤ 𝑓 ∗ if it is a minimization problem
D: 𝑓 ∗ ≥ 𝑓 ∗ if it is a maximization problem
E: 𝑓 ∗ ≥ 𝑓 ∗ if it is a minimization problem
B,E
MILP
LP Relaxation
max 𝑥 𝑐 𝑇 𝑥
s.t. 𝐺𝑥≤ℎ
𝑥 𝑖 ∈ℤ,𝑖∈ 𝐽 𝑧
max 𝑥 𝑐 𝑇 𝑥
s.t. 𝐺𝑥≤ℎ
Fei Fang
12/8/2018

12. Quiz 2: (Mixed) Integer Linear Program
Which are optimal solutions?
(137.5,125)
(138,125)
137,125
138,124
None of above
max 𝑥,𝑦 30𝑥+30𝑦
s.t.
0.2𝑥+0.5𝑦≤90 4𝑥+2𝑦≤800 𝑥≥0,𝑦≥0
𝑥,𝑦∈ℤ
0.2𝑥+0.5𝑦=90
4𝑥+2𝑦=800
Fei Fang
12/8/2018

13. How to solve
Is it sufficient to consider integer points around the corresponding LP solution? No!
Solution to the MILP can be arbitrarily far from the solution of the corresponding LP (relaxing all the integer constraints)
Sometimes even hard to find a feasible integer point
Fei Fang
12/8/2018

14. For Binary Linear Program (BLP)
How to solve
For Binary Linear Program (BLP)
Discrete optimization problem
Naïve approach: Enumerate all the possibilities
Can apply uninformed or informed search
Natural tree structure
Items 1 2 3 4 5
Weight 6 7
Value 9
max 4𝑥 1 +3 𝑥 2 +6 𝑥 3 +9 𝑥 4 +5 𝑥 5
s.t. 5𝑥 1 +4 𝑥 2 +2 𝑥 3 +6 𝑥 4 +7 𝑥 5 ≤10
𝑥 𝑖 ∈{0,1}
Fei Fang
12/8/2018

15. Depth-First Search for BLP
max 4𝑥 1 +3 𝑥 2 +6 𝑥 3 +9 𝑥 4 +5 𝑥 5
s.t. 5𝑥 1 +4 𝑥 2 +2 𝑥 3 +6 𝑥 4 +7 𝑥 5 ≤10
𝑥 𝑖 ∈{0,1}
Fei Fang
12/8/2018

16. Recall: Informed Search
function BEST-FIRST-SEARCH (problem, EVAL-FN) returns a solution sequence
inputs: problem, a problem
EVAL-FN, an evaluation function
Queuing-Fn ← a function that orders nodes by EVAL-FN
return GENERAL-SEARCH (problem, Queuing-Fn)
function GREEDY-SEARCH (problem) returns a solution or failure
return BEST-FIRST-SEARCH (problem, h)
𝐴 ∗ search: Best-first-search with EVAL-FN 𝑓(𝑧)=𝑔(𝑧)+ ℎ(𝑧) where ℎ(𝑧) is an admissible heuristic for node 𝑧
𝑔 𝑧 =path cost up to node 𝑧, ℎ(𝑧)=heuristic estimate of distance to go
In a minimization problem, ℎ(𝑧) never overestimates the distance to go
In a maximization problem, ℎ(𝑧) never underestimates the value of the remaining path
Optimal for tree search
Fei Fang
12/8/2018

17. For Binary Linear Program (BLP)
How to solve
For Binary Linear Program (BLP)
Discrete optimization problem
Naïve approach: Enumerate all the possibilities
Can apply uninformed or informed search
Natural tree structure
Vanilla branch and bound (BnB)=Best-first-search with EVAL-FN 𝑓(𝑧)= optimal value of LP relaxation at node 𝑧
Fei Fang
12/8/2018

18. Evaluation Function in Branch and Bound
LP relaxation at node 𝑏 𝑎
max 4𝑥 1 +3 𝑥 2 +6 𝑥 3 +9 𝑥 4 +5 𝑥 5
s.t. 5𝑥 1 +4 𝑥 2 +2 𝑥 3 +6 𝑥 4 +7 𝑥 5 ≤10
𝑥 𝑖 ∈[0,1],𝑖=1..5
𝑥 1 =0
𝑥 1 =0
𝑥 1 =1 𝑏 𝑐
𝑓 𝑧 = optimal value of LP relaxation at node 𝑧
LP relaxation provides a upper bound of the objective value of any solution with a path through 𝑧 if it is a maximization problem (lower bound if minimization)
Never underestimates the real value of the whole path given the path goes through 𝑧 for maximization problem (never overestimate the total cost for minimization)
𝑓 𝑧 −(path value up to 𝑧) is an admissible heuristic
Fei Fang
12/8/2018

19. For Binary Linear Program (BLP)
How to solve
For Binary Linear Program (BLP)
Discrete optimization problem
Naïve approach: Enumerate all the possibilities
Can apply uninformed or informed search
Natural tree structure
Vanilla branch and bound (BnB)=Best-first-search with EVAL-FN 𝑓(𝑧)= optimal value of LP relaxation at node 𝑧
Optimality guaranteed
Fei Fang
12/8/2018

20. For Binary Linear Program (BLP)
How to solve
For Binary Linear Program (BLP)
Discrete optimization problem
Naïve approach: Enumerate all the possibilities
Can apply uninformed or informed search
Natural tree structure
Vanilla branch and bound (BnB)=Best-first-search with EVAL-FN 𝑓(𝑧)= optimal value of LP relaxation at node 𝑧
Optimality guaranteed
BnB with upper bound and lower bound (not required)
Terminate early if we know the remaining nodes are not as good as a binary solution we have already found
Fei Fang
12/8/2018

21. For MILP (not required)
How to solve
For MILP (not required)
BnB: For each integer variable, branching a node by considering 𝑥 𝑖 ≤ 𝑥 𝑖 and 𝑥 𝑖 ≥ 𝑥 𝑖 where 𝑥 𝑖 is a non- integer value
Branch and cut: use cutting planes (which are hyperplanes) to separate current non-integer solution and integer solutions
Fei Fang
12/8/2018

22. How to solve
Practically efficient solvers: Cplex, Gurobi, intlinprog (MATLAB), SCIP solver
Fei Fang
12/8/2018

23. (Mixed) Integer Linear Program How to solve a MILP
Outline
(Mixed) Integer Linear Program
How to solve a MILP
Formulate/Convert to MILP
Fei Fang
12/8/2018

24. Constraint satisfaction problem
TWO+TWO=FOUR
Fei Fang
12/8/2018

25. Constraint satisfaction problem
TWO+TWO=FOUR
Variables: 𝑥 𝑇 , 𝑥 𝑊 , 𝑥 𝑂 , 𝑥 𝐹 , 𝑥 𝑈 , 𝑥 𝑅 , 𝑐 1 , 𝑐 2 , 𝑐 3
min x,c 1
s.t. 𝑥 𝑂 + 𝑥 𝑂 = 𝑥 𝑅 +10 𝑐 1
𝑥 𝑊 + 𝑥 𝑊 + 𝑐 1 = 𝑥 𝑈 +10 𝑐 2
𝑥 𝑇 + 𝑥 𝑇 + 𝑐 2 = 𝑥 𝑂 +10 𝑐 3
0+ 𝑐 3 = 𝑥 𝐹
𝑥 𝑇 , 𝑥 𝐹 ∈{1,..,9}
𝑥 𝑊 , 𝑥 𝑂 , 𝑥 𝑈 , 𝑥 𝑅 ∈{0,..,9}
𝑐 1 , 𝑐 2 , 𝑐 3 ∈{0,1}
Fei Fang
12/8/2018

26. Formulate a Problem as a MILP
Sudoku
Fill a 9×9 grid with digits so that each column, each row, and each of the nine 3×3 subgrids that compose the grid (also called "boxes", "blocks", or "regions") contains all of the digits from 1 to 9
Fei Fang
12/8/2018

27. Formulate a Problem as a MILP
Sudoku
Variables 𝑥 𝑖𝑗 𝑘 indicate if we place number 𝑘 in row 𝑖, col 𝑗
𝑆 𝑡 is the set of (row,col) index pairs of the 𝑡 𝑡ℎ groups (rows, columns, and subgrids)
For example, 𝑆 1 ={ 1,1 , 1,2 ,…(1,9)}
min 𝑥 1
s.t. 𝑥 𝑖𝑗 𝑘 ∈ 0,1 ,∀𝑖,𝑗,𝑘∈{1..9}
𝑖,𝑗 ∈ 𝑆 𝑡 𝑥 𝑖𝑗 𝑘 =1 ,∀𝑘∈ 1..9 ,∀𝑡∈{1..27}
𝑘 𝑥 𝑖𝑗 𝑘 =1 ,∀𝑖,𝑗∈{1..9}
Fei Fang
12/8/2018

28. Formulate a Problem as a MILP
Kidney Exchange
Fei Fang
12/8/2018

29. Formulate a Problem as a MILP
Kidney Exchange
Given directed graph 𝐺=(𝑉,𝐸), where each node represent a patient-donor pair, and an edge 〈𝑢,𝑣〉 means donor of node 𝑢 can give one kidney to patient of node 𝑣
Find a set of disjoint cycles so as to maximize the number of nodes covered 6 2 1 4 5 3 7 8
Fei Fang
12/8/2018

30. Quiz 3: Kidney Exchange
Given the graph below, what is the maximum number of patients that can get a kidney through kidney exchange assuming the length of each cycle should be less than or equal to 3?
A: 3
B: 6
C: 7
D: 8 6 2 1 4 5 3 7 8
Fei Fang
12/8/2018

31. Formulate a Problem as a MILP
Kidney Exchange
Given directed graph 𝐺=(𝑉,𝐸), where each node represent a patient-donor pair, and an edge 〈𝑢,𝑣〉 means donor of node 𝑢 can give one kidney to patient of node 𝑣
Find a set of disjoint cycles so as to maximize the number of nodes covered
Hint: enumerate all the cycles 6 2 1 4 5 3 7 8
Fei Fang
12/8/2018

32. No queen in the same row, column and diagnal
8-Queens Problem
No queen in the same row, column and diagnal
𝑥 𝑖𝑗 : whether there is a queen on row 𝑖 col 𝑗
max x 𝑖 𝑗 𝑥 𝑖𝑗
s.t. 𝑥 𝑖𝑗 ∈{0,1}
𝑗 𝑥 𝑖𝑗 ≤1,∀𝑖∈{1..8}
𝑖 𝑥 𝑖𝑗 ≤1, ∀𝑗∈{1..8}
𝑖,𝑗:𝑖−𝑗=𝑘 𝑥 𝑖𝑗 =1,∀𝑘∈ −6,−5,…,5,6
𝑖,𝑗:𝑖+𝑗=𝑘 𝑥 𝑖𝑗 =1,∀𝑘∈ 2,3,…,14
Fei Fang
12/8/2018

33. Recall in the first lecture about optimization, we had
8-Queens Problem
Recall in the first lecture about optimization, we had
Can we rewrite it using only linear constraints + integer constraints?
Variables 𝑥 𝑖 : row index of the queen in 𝑖 𝑡ℎ column
min x 1
s.t. 𝑥 𝑖 ∈{1…8}
𝑥 𝑖 ≠ 𝑥 𝑗 ,∀𝑖,𝑗∈ 1..8 ,𝑖≠𝑗
𝑥 𝑖 − 𝑥 𝑗 ≠ 𝑖−𝑗 ,∀𝑖,𝑗∈ 1..8 ,𝑖≠𝑗
Fei Fang
12/8/2018

34. 8-Queens Problem min x 1 s.t. 𝑥 𝑖 ∈{1…8} 𝑥 𝑖 ≠ 𝑥 𝑗 ,∀𝑖,𝑗∈ 1..8 ,𝑖≠𝑗
𝑥 𝑖 − 𝑥 𝑗 ≠ 𝑖−𝑗 ,∀𝑖,𝑗∈ 1..8 ,𝑖≠𝑗
Convert constraints
(We omit the range of index below)
min x 1
s.t. 𝑥 𝑖 ∈{1…8}
𝑥 𝑖 ≥ 𝑥 𝑗 +1 or 𝑥 𝑖 ≤ 𝑥 𝑗 −1,
| 𝑥 𝑖 − 𝑥 𝑗 |≥|𝑖−𝑗|+1 or | 𝑥 𝑖 − 𝑥 𝑗 |≤|𝑖−𝑗|−1
Fei Fang
12/8/2018

35. Formulate a Problem as a MILP
8-Queens Problem
Introduce variables 𝑦 𝑖𝑗 + , 𝑦 𝑖𝑗 − , set 𝐵=100
min x,𝑦,𝑧 1
s.t. 𝑥 𝑖 ∈{1…8}
𝑥 𝑖 −( 𝑥 𝑗 +1)+𝐵 𝑦 𝑖𝑗 + ≥0
𝑥 𝑖 −( 𝑥 𝑗 −1)−𝐵 𝑦 𝑖𝑗 − ≤0
𝑦 𝑖𝑗 + + 𝑦 𝑖𝑗 − ≤1
𝑦 𝑖𝑗 + , 𝑦 𝑖𝑗 − ∈{0,1}
| 𝑥 𝑖 − 𝑥 𝑗 |≥|𝑖−𝑗|+1 or
| 𝑥 𝑖 − 𝑥 𝑗 |≤|𝑖−𝑗|−1
min x 1
s.t. 𝑥 𝑖 ∈{1…8}
𝑥 𝑖 ≥ 𝑥 𝑗 +1 or
𝑥 𝑖 ≤ 𝑥 𝑗 −1
| 𝑥 𝑖 − 𝑥 𝑗 |≥|𝑖−𝑗|+1 or
| 𝑥 𝑖 − 𝑥 𝑗 |≤|𝑖−𝑗|−1
Fei Fang
12/8/2018

36. General techniques for conversion
At-most-one constraint: At most one of the options can be chosen
Add variable 𝑥 𝑖 ∈{0,1} to represent whether or not option 𝑖 is chosen
Add constraint 𝑖 𝑥 𝑖 =1
Fei Fang
12/8/2018

37. General techniques for conversion
Constraint feasibility constraint: If 𝑦=0, then 𝑓 𝑥 1 , 𝑥 2 ,…, 𝑥 𝑛 ≤𝑏
Choose a large enough constant 𝐵 such that 𝑓 𝑥 1 , 𝑥 2 ,…, 𝑥 𝑛 −𝐵≤𝑏 is always satisfied (imagine 𝐵= +∞)
Add constraint 𝑓 𝑥 1 , 𝑥 2 ,…, 𝑥 𝑛 −𝐵𝑦≤𝑏
Fei Fang
12/8/2018

38. General techniques for conversion
Alternative Constraints: 𝑓 1 𝑥 1 , 𝑥 2 ,…, 𝑥 𝑛 ≤ 𝑏 1 or 𝑓 2 𝑥 1 , 𝑥 2 ,…, 𝑥 𝑛 ≤ 𝑏 2
Introduce some large enough constants 𝐵 1 , 𝐵 2 such that 𝑓 𝑘 𝑥 1 , 𝑥 2 ,…, 𝑥 𝑛 − 𝐵 𝑘 ≤ 𝑏 𝑘 always hold
Introduce binary variables 𝑦 1 , 𝑦 2
Add constraints
𝑓 1 𝑥 1 , 𝑥 2 ,…, 𝑥 𝑛 − 𝐵 1 𝑦 1 ≤ 𝑏 1
𝑓 2 𝑥 1 , 𝑥 2 ,…, 𝑥 𝑛 − 𝐵 2 𝑦 2 ≤ 𝑏 2
𝑦 1 + 𝑦 2 ≤1
𝑦 1 , 𝑦 2 ∈{0,1}
Fei Fang
12/8/2018

39. General techniques for conversion
Conditional Constraints
If 𝑓 1 𝑥 1 , 𝑥 2 ,…, 𝑥 𝑛 > 𝑏 1 , then 𝑓 2 𝑥 1 , 𝑥 2 ,…, 𝑥 𝑛 ≤ 𝑏 2
Equivalent to 𝑓 1 𝑥 1 , 𝑥 2 ,…, 𝑥 𝑛 ≤ 𝑏 1 or 𝑓 2 𝑥 1 , 𝑥 2 ,…, 𝑥 𝑛 ≤ 𝑏 2
Apply techniques for alternative constraints
Fei Fang
12/8/2018

40. General techniques for conversion
Bilinear term 𝑥𝑦 where 𝑥∈{0,1} and 𝑦∈ [ 𝑦 𝑚𝑖𝑛 , 𝑦 𝑚𝑎𝑥 ]
Introduce new variable 𝑧
Add constraints 𝑦 𝑚𝑖𝑛 𝑥≤𝑧≤ 𝑦 𝑚𝑎𝑥 𝑥
Fei Fang
12/8/2018

41. Summary Optimization Problems Convex Programs Mixed Integer
Linear Programs
Gradient Descent
Linear Programs
Branch and Bound
Simplex
Fei Fang
12/8/2018

42. (Mixed) Integer Linear Program: Additional Resources
Textbook
Applied Mathematical Programming, Chapter 9
By Bradley, Hax, and Magnanti (Addison-Wesley, 1977) 
Online course
integer-programming-and-combinatorial-optimization-fall- 2009/index.htm
Fei Fang
12/8/2018

43. Acknowledgment
Some slides are borrowed from previous slides made by J. Zico Kolter and Ariel Procaccia
Fei Fang
12/8/2018