1. More on Intractability Knapsack Problem Wednesday, August 5 th 1

2. Outline For Today 1.Knapsack Pseudo-Poly-time DP 1 2.Knapsack Pseudo-Poly-time DP 2 3.Knapsack Greedy ½-Approximation Alg 4.Knapsack Fully Poly-time Approx. Scheme (FPTAS) 2

3. Recap: Classes P, NP 3  Given a computational problem C  P: C is ∈ P (polynomial-time solvable) if ∃ an algorithm solving C with O(n k ) run-time, for some constant k.  where n is the input length in bits  NP (or brute-force solvable): C ∈ NP if: 1.Correct solutions have polynomial length. 2.Claimed solutions are verifiable in poly-time.

4. Recap: NP-completeness 4 NP C1C1 C1C1 C*: NP- complete C3C3 C3C3 C4C4 C4C4 C5C5 C5C5 C6C6 C6C6 CkCk CkCk C2C2 C2C2 C* is as hard as any NP problem!

5. Recap: History of NP-completeness 5 NP C2C2 C2C2 SAT C3C3 C3C3 C4C4 C4C4 C5C5 C5C5 CkCk CkCk C6C6 C6C6 C1C1 C1C1 K1K1 K1K1 K2K2 K2K2 … … K 20 K 21 1971: Cook-Levin1972: Karp

6. Recap: History of NP-completeness 6 NP C2C2 C2C2 SAT C3C3 C3C3 C4C4 C4C4 C5C5 C5C5 CkCk CkCk C6C6 C6C6 C1C1 C1C1 K1K1 K1K1 K2K2 K2K2 … … K 20 K 21 NP-complete Since 1972

7. Recap: Showing C** is NP-complete 7 NP C2C2 C2C2 C*: NP- complete C3C3 C3C3 C4C4 C4C4 C5C5 C5C5 CkCk CkCk C6C6 C6C6 C ** If we can solve C** efficiently => we solve C* efficiently => we solve all NP problems efficiently C** is NP-complete!

8. Recap: Two Very Important Skills 8 1.Recognizing NP-complete problems. 2.Learning The Right Methods of Approaching NP- complete Problems

9. Recap: Approaching NP-complete Problems 9  Option 1: Focus to special-case inputs.  Option 2: Find an approximate answer.  Option 3: Be exponential time but better than brute- force search.  Option 4: Heuristics: fast algorithms that are not always correct (or even approximate)  Option 5: Mix some of these options

10. Outline For Today 1.Knapsack Pseudo-Poly-time DP 1 2.Knapsack Pseudo-Poly-time DP 2 3.Knapsack Greedy ½-Approximation Alg 4.Knapsack Fully Poly-time Approx. Scheme (FPTAS) 10

11. Knapsack (Sec 6.4, 11.8)  Input:  n items  values for items v 1, …, v n ≥ 0  sizes for items w 1, …, w n ≥ 0  knapsack capacity W ≥ 0  Output: subset S ⊆ {1, 2, …, n} items s.t. 11 Fact: Knapsack is NP- complete. 3SAT ≤ p KNAPSACK SUBSET-SUM ≤ p KNAPSACK

12. Knapsack Example 12 v 1 = 2.2 w 1 = 1.5 v 1 = 2.2 w 1 = 1.5 v 2 = 4 w 2 = 3 v 2 = 4 w 2 = 3 v 4 = 3 w 4 = 4.6 v 4 = 3 w 4 = 4.6 W=7.8 v 3 = 2 w 3 = 3 v 3 = 2 w 3 = 3

13. Knapsack Example 13 v 1 = 2.2 w 1 = 1.5 v 1 = 2.2 w 1 = 1.5 v 2 = 4 w 2 = 3 v 2 = 4 w 2 = 3 v 4 = 3 w 4 = 4.6 v 4 = 3 w 4 = 4.6 W=7.8 v 3 = 2 w 3 = 3 v 3 = 2 w 3 = 3 v 2 = 4 w 2 = 3 v 2 = 4 w 2 = 3 v 3 = 2 w 3 = 3 v 3 = 2 w 3 = 3 v 1 = 2.2 w 1 = 1.5 v 1 = 2.2 w 1 = 1.5 OPT = 4+2+2.2=8.2

14. Restrict w i and W to be Integers 14  Input:  n items  values for items v 1, …, v n ≥ 0  sizes for items w 1, …, w n ≥ 0 & w i are **INTEGERS**  knapsack capacity W ≥ 0 & W is an **INTEGER**  Output: subset S ⊆ {1, 2, …, n} items s.t.

15. Recap: Recipe of a DP Algorithms 15 1.Identify small # of subproblems 2.Quickly + correctly solve “larger” subproblems given solutions to smaller ones 3.After solving all subproblems, can quickly compute final solution

16. Knapsack DP Algorithm 1 16  Order the n items in arbitrary order: {1, 2, …, n}.  Consider the optimal solution S* A Claim that Doesn’t Require A Proof: (1) n ∉ S* or (2) n ∈ S*

17. Case 1: n ∉ S* 17 Q: What can we assert about S* for items {1, …, n-1}? A: S* is opt. for items {1, …, n-1} and capacity W. Proof: Assume ∃ better S** w/ cap. W for {1, …, n-1}  S** is feasible for {1, …, n} and better than S*  Which would contradict S*’s optimality Q.E.D.

18. Case 2: n ∈ S* 18 Q: What can we assert about S*-{n} for items {1, …, n-1}? A: S*-{n} is opt. for items {1, …, n-1} and cap. W-w n. Pf: Assume ∃ better S** w/ cap ≤ W-w n for {1, …, n-1}  S** ∪ {n} has capacity ≤ W  S** ∪ {n} is feasible for {1, …, n} and better than S*  Which would contradict S*’s optimality Q.E.D.

19. What Are The Subproblems? 19 K (i, c) : opt. knapsack for the first i items and cap c. Q: How many subproblems are there? A: n*W K (i, c) = max K (i-1, c) K (i-1, c - w_i) + v i

20. Knapsack DP Algorithm 1 Pseudocode procedure DP-Knapsack-1(n, W): Base Cases: A[0][i] = 0 for i = 1,2,…,n: for c = 1, …, W: A[i][c] = max{A[i-1][c], A[i-1][c-w i ]+v i } return A[n][W] 20 K (i, c) = max K (i-1, c) K (i-1, c - w_i)

21. Run-time 21 Runtime: O(nW) Brute-Force Search: Ω(2 n ) Observation: This is polynomial in n and W. Q: Why does this not prove P=NP? A: B/c we’re still exponential in input size.

22. Input 22 Input size: # bits (key strokes) to represent the problem n weights, values (n * (log of max weight and value)) capacity => log(W) bits. Note: W is exponential in log(W) w1w1 v1v1 w2w2 v2v2 ……wnwn vnvn W

23. Pseudo-polynomial Time Algorithm 23 An algorithm that’s polynomial in the numeric values of the inputs but not the # bits to represent it. Ex: O(nW) is pseudo-polynomial Interpretation: If we fix W to an integer value => Knapsack is tractable Called “Fixed-Parameter Tractable” Problem

24. Summary of Knapsack DP Alg 1 24 1.Took Knapsack, which is NP-complete. 2.Restricted to inputs with integer w i and W 3.Got a pseudo-poly-time algorithm DP algorithm (exponential but better than brute-force search) 4.Further fixing W yields a full poly-time algorithm

25. Recap: Approaching NP-complete Problems 25  Option 1: Focus to special-case inputs.  Option 2: Find an approximate answer.  Option 3: Be exponential time but better than brute- force search.  Option 4: Heuristics: fast algorithms that are not always correct (or even approximate)  Option 5: Mix some of these options

26. Important Note To Keep In Mind 26 When attacking NP-complete problems, if you stick with always correct algorithms (i.e. you don’t pick option 2 and try to approximate) you necessarily have to be exponential time. However can get non-trivial speed-ups if some aspect of the problem is small.

27. Outline For Today 1.Knapsack Pseudo-Poly-time DP 1 2.Knapsack Pseudo-Poly-time DP 2 3.Knapsack Greedy ½-Approximation Alg 4.Knapsack Fully Poly-time Approx. Scheme (FPTAS) 27

28. Now Restrict v i to be Integers 28  Input:  n items  values for items v 1, …, v n ≥ 0 & v i are **INTEGERS** let v * = max i { v i }, and V = v 1 + v 2 + … + v n ≤ nv*  sizes for items w 1, …, w n ≥ 0  knapsack capacity W ≥ 0  Output: subset S ⊆ {1, 2, …, n} items s.t.

29. A Different DP Algorithm 29  DP Algorithm 1 asked: What is the maximum value we can pack into at most X capacity given the first k items?  We can also ask: What is the minimum capacity needed to pack at least Y value into the knapsack given the first k items?

30. Subproblems For Approach 2 30 M (i, v) : min capacity needed to pack value v from the first i items. Q: How many subproblems are there? A: n*V≤n(nv*)=n 2 v* Runtime is O(n 2 v*) M (i, v) = min M (i-1, v) M (i-1, v – v_i) + w i

31. Knapsack DP Algorithm 2 Pseudocode procedure DP-Knapsack-2(n, V = v 1 + … + v n ): Base Cases: A[0][0] = 0, A[0][j] = +∞ for i = 1,2,…,n: for v = 1, …, V: A[i][v] = min{A[i-1][v], A[i-1][v-v i ]+w i } return max v s.t. A[n][v] ≤ W 31 M (i, v) = min M (i-1, v) M (i-1, v - v_i) + w i

32. Knapsack DP Algorithm Output 32 Min capacity needed to pack a value of 1

33. Knapsack DP Algorithm Output 33 Min capacity needed to pack a value of 2

34. Knapsack DP Algorithm Output 34 Min capacity needed to pack a value of 45

35. Knapsack DP Algorithm Output 35 Min capacity needed to pack a value of 46

36. Knapsack DP Algorithm Output 36 Min capacity needed to pack a value of V

37. 3 Exp-time Correct Knapsack Algorithms 37 Takeaway 1: There is good & bad exponential times Takeaway 2: We can get non-trivial speed-ups if some aspect of the problem is small.

38. Outline For Today 1.Knapsack Pseudo-Poly-time DP 1 2.Knapsack Pseudo-Poly-time DP 2 3.Knapsack Greedy ½-Approximation Alg 4.Knapsack Fully Poly-time Approx. Scheme (FPTAS) 38

39. Back To Original Knapsack  Input:  n items  values for items v 1, …, v n ≥ 0  sizes for items w 1, …, w n ≥ 0  knapsack capacity W ≥ 0  Output: subset S ⊆ {1, 2, …, n} items s.t. 39 Option 2: Now we’ll give up correctness.

40. Recall Scheduling Problem  Input: Each job i has length l i AND weight w i l 1, w 1 Job 1 l 2, w 2 Job 2 l n,w n Job n …  Output: A schedule of the jobs on a processor s.t: is minimum over all possible n! schedules. weighted completion time of job i

41. A Possible Greedy Approach  Similar to Greedy Weighted Scheduling (Lecture 5)  If weights are the same, put higher value items first  If values are the same, put lighter items first  Greedy Algorithm: 1.Combine v i and w i into a single score v i / w i: 2.Sort items in increasing combined score Assume w.l.o.g.: v 1 /w 1 ≥ v 2 /w 2 ≥ … ≥ v n /w n 3.Pack until can’t pack anymore 41

42. Example 1 42 v 1 = 5 w 1 = 1 v 1 = 5 w 1 = 1 v 2 = 4 w 2 = 2 v 2 = 4 w 2 = 2 v 3 = 3 w 3 = 3 v 3 = 3 w 3 = 3 W=5

43. Example 1 43 v 1 = 5 w 1 = 1 v 1 = 5 w 1 = 1 v 2 = 4 w 2 = 2 v 2 = 4 w 2 = 2 v 3 = 3 w 3 = 3 v 3 = 3 w 3 = 3 W=5

44. Example 1 44 v 1 = 5 w 1 = 1 v 1 = 5 w 1 = 1 v 2 = 4 w 2 = 2 v 2 = 4 w 2 = 2 v 3 = 3 w 3 = 3 v 3 = 3 w 3 = 3 3 rd item does not fit Output: 4 + 5 = 9 Found the optimal Q: What can go wrong? W=5

45. Bad Example 45 v 1 = 2 w 1 = 1 v 1 = 2 w 1 = 1 v 2 =100 w 2 =100 v 2 =100 w 2 =100 W=100

46. Bad Example 46 v 1 = 2 w 1 = 1 v 1 = 2 w 1 = 1 v 2 =100 w 2 =100 v 2 =100 w 2 =100 Output: 2 Optimal is 100 Can be arbitrarily far from optimal. W=100

47. Simple Fix 47  Fixed-Greedy Algorithm: 1.Combine v i and w i into a single score v i / w i: 2.Sort items in increasing combined score Assume w.l.o.g.: v 1 /w 1 ≥ v 2 /w 2 ≥ … ≥ v n /w n 3.Pack until can’t pack anymore 4.Return Y = max {step 3, max-value item} (assume all items have weight ≤ W) Claim: Y is a ½-approximation to OPT Runtime: O(nlogn)

48. Thought Experiment 48 Assume greedy packed first k items And only x% of item k+1 fits into our knapsack Suppose we can slice x% of k Fill in the rest of out knapsack And get x% of v k.

49. Example of Slicing 49 v 1 = 2 w 1 = 1 v 1 = 2 w 1 = 1 v 2 =100 w 2 =100 v 2 =100 w 2 =100 W=100 Slice 99% of item 2

50. Example of Slicing 50 v 1 = 2 w 1 = 1 v 1 = 2 w 1 = 1 v 2 =1 w 2 =1 v 2 =1 w 2 =1 W=100 v 2 =99 w 2 =99 v 2 =99 w 2 =99 Claim: This “Fractional” Solution is better than OPT.

51. Proof Sketch (Fill in the Gaps As Exercise) 51 Same Take arbitrary solution S Assume they differ in l units Greedy picks those l units from highest value per weight (plus might have extra stuff)  value(l units of greedy) ≥ value(l units of S)  Greedy ≥ S Greedy Fractional Any Solution S Same Different

52. Proof of ½-Approximation 52 Assume greedy packs first k items and gets stuck. Fixed-Greedy returns: max{A=first k, or max-value item} ≥ max{A=first k, B=value of k+1 st item} Fractional-Greedy: A + fraction of B max{A, B} >= ½ (A+B) Fixed-Greedy ≥ ½ Fractional-Greedy ≥ ½ OPT Q.E.D

53. Outline For Today 1.Knapsack Pseudo-Poly-time DP 1 2.Knapsack Pseudo-Poly-time DP 2 3.Knapsack Greedy ½-Approximation Alg 4.Knapsack Fully Poly-time Approx. Scheme (FPTAS) 53

54. Back To Integer Values  Input:  n items  values for items v 1, …, v n ≥ 0 (**INTEGERS**) let v * = max i { v i }, and V = v 1 + v 2 + … + v n ≤ nv*  sizes for items w 1, …, w n ≥ 0  knapsack capacity W ≥ 0  Output: subset S ⊆ {1, 2, …, n} items s.t. 54

55. Very Ambitious Goal 55 Knapsack Algorithm Knapsack Algorithm I: (w i, v i, W) accuracy ε (say 0.01) ≥ (1-ε)*OPT (1-ε)-approx **Catch: Run-time will grow as ε decreases** Upshot Instead of I solve incorrect input I’(w i, ṽ i, W) But solve I’ optimally with DP2 Argue opt for I’ is (1-ε) close to opt for I

56. Recall DP2 56 Pseudo poly-time DP Algorithm Run-time: O(n 2 v*) Performs badly only when values are very large. Idea: Remove Low Order Bits.

57. Rounding Algorithm 57 Q1: How small should m be to satisfy ε accuracy? Q2: What is the running time of the algorithm once we pick the the appropriate m? procedure Knapsack-FPTAS(w i, v i, W, ε): round each v i down to nearest mult. of m=f(ε) let ṽ i = rounded(v i )/m, return DP2(w i, ṽ i, W)

58. Accuracy Analysis (1) 58 vivi x i =m ṽ i ≤ v i x i =m ṽ i ≤ v i ≤ y i =m( ṽ i + 1) (1) (2) Let S be our solution, and S* is optimal solution Hint: Weights were NOT perturbed. Goal: Goal is to say we’re not that far from OPT.

59. Accuracy Analysis (2) 59 Goal : by (1) by (2) x i ’s y i ’s

60. Accuracy Analysis (3) 60 error term This inequality is true for any m. Q: What should m be so that we’re ε off? A: Pick m s.t.

61. Accuracy Analysis (4) 61 Pickingsatisfies the inequality.

62. Run-time Analysis 62 ∀ elements i, we have: DP2’s run-time was: O(n 2 v*) When inputs are ṽ i each ṽ i ≤ n/ε So run-time is: O(n 3 /ε)

63. Key Takeaway 63 We took an NP-complete problem. We are approximating it to arbitrary precision. And in poly-time for each precision rate!

64. 64 Next Week More Algorithms For Intractability & What’s Beyond CS 161