1. Introduction to Intractability P, NP, NP-completeness How To Approach Intractable Problems Monday, August 3 rd 1

2. Outline For Today 1.Intractability: P, NP, NP-completeness 2.How to Approach Intractable Problems? 2

3. Outline For Today 1.Intractability: P, NP, NP-completeness 2.How to Approach Intractable Problems? 3

4. Focus of CS161 4  Practical Algorithms to Fundamental Problems  Almost-linear time super-fast algorithms  Sorting: O(nlogn)  Dijkstra: O(mlog(n))  MST: O(mlog(n))  Superlinear but still practical  Strassen’s Algorithm: O(n 2.83 )  Bellman-Ford: O(mn)  Floyd-Warshall: O(n 3 )  Karger, etc…

5. Sad Fact 5 Many other important problems seem impossible to solve very efficiently. Ex: TSP, Max-Cut A critical skill that we all have to acquire is to recognize such problems.

6. Formalizing Tractability: Class P 6  Given a computational problem C  P: C is ∈ P (polynomial-time solvable) if ∃ an algorithm solving C with O(n k ) run-time, for some constant k.  where n is the input length in bits  k is some constant, say, 1, 2, 5, 1M, etc.  Ex poly-times: O(n), O(nlog(n)), O(n 3 ), O(n 100000 )  Ex: Almost all problems we have seen so far except TSP, Independent Set, Max-Cut

7. Interpretation of P and Its Caveats 7 A rough test for **tractability**. Caveat 1: n 1000 is not tractable in practice Caveat 2: Some intractable problems can actually be solved in some restricted form efficiently But generally speaking, has been **extremely successful** in classifying problems.

8. Example of Intractable Problem: TSP 8 TSP: Traveling Salesman Problem Input: Complete graph G(V, E) with non-negative edge costs Output: Min-cost tour [cycle visiting each vertex once] D C B A 2 1 4 3 5 6

9. Example of Intractable Problem: TSP 9 TSP: Traveling Salesman Problem Input: Complete graph G(V, E) with non-negative edge costs Output: Min-cost tour [cycle visiting each vertex once] D C B A 2 1 4 3 5 6 Answer: A->B->D->C->A = 13

10. Conjecture from 1965 (Jack Edmonds) 10  We have been looking for a fast algorithm for TSP for > 80 years, and no one has succeeded.  After 30 years or so in 1965, Jack Edmonds made the following conjecture: There is no poly-time algorithm for TSP. (equivalent to conjecture: P != NP) To this day, no one has been able to prove/disprove this conjecture.

11. Making a Case For TSP’s Intractability? 11  (So far) We have not been able to argue for TSP’s intractability in an absolute sense.  Instead accumulate evidence of intractability Idea From Early 1970s: Instead show “relative” intractability (i.e. show TSP is as hard as bunch of other problems.)

12. Reductions 12 What does it mean for problem C 2 to be as hard as C 1 ? Definition: C 1 reduces to C 2, (C 1 ≤ p C 2 ), if given a poly- time algorithm for C 2, we can solve C 1 in poly-time. If C 1 reduces to C 2 => C 2 is “as hard as” C 1 Poly-time Alg for C 2 Π 1 ∈ C 1 Poly-time C 1 -> C 2 Converter Poly-time C 1 -> C 2 Converter Π 2 ∈ C 2 Solution to Π 2 Poly-time C 2 Solution -> C 1 Solution Converter Poly-time C 2 Solution -> C 1 Solution Converter Solution to Π 1

13. High-level Idea of Reduction 13 Efficient Alg A for C 2 Π 1 ∈ C 1 Efficient Π 2 ∈ C 2 Solution to Π 2 Efficient C 2 Solution -> C 1 Solution Converter Efficient C 2 Solution -> C 1 Solution Converter Solution to Π 1 If we can solve C 2 efficiently, we can also solve C 1 efficiently  C 2 as hard as C 1 Effectively an efficient algorithm for C 1

14. Reductions are Transitive 14 Claim: If C 1 ≤ p C 2, and C 2 ≤ p C 3, then C 1 ≤ p C 3 Proof: We have to argue if we have a poly-time algorithm for C 3, we can solve C 1 efficiently. Or C 3 is as hard as C 1. If we had a poly-time algorithm for C 3  we could solve C 2 in poly-time (since C 2 ≤ C 3 )  we could solve C 1 in poly-time (since C 1 ≤ C 2 ) Q.E.D.

15. Contrapositive 15 Efficient Alg A for C 2 Π 1 ∈ C 1 Efficient Π 2 ∈ C 2 Solution to Π 2 Efficient C 2 Solution -> C 1 Solution Converter Efficient C 2 Solution -> C 1 Solution Converter Solution to Π 1 Then if C 1 ≤ p C 2, there cannot be an efficient algorithm A for C 2 Assume we knew NO efficient algorithm for C 1 exists X

16. Making a Case for TSP’s Intractability 16 Efficient Alg A for TSP Π 1 ∈ C 1 Efficient Π 2 ∈ TSP Solution to Π 2 Efficient TSP Solution -> C 1 Solution Converter Efficient TSP Solution -> C 1 Solution Converter Solution to Π 1 If we can solve TSP efficiently, we can also solve C 1 efficiently  TSP is as hard as C 1 Goal: Argue TSP is as hard as a large C of problems

17. Definition: Completeness 17 Let C be a set of problems. Ci ∈ CCi ∈ C If ∀ C k ∈ C, C k ≤ p C i, then C i is C-complete. (i.e. C i is as hard as every other problem in C ) (i.e. C i is the hardest problem in C ) Goal: Argue TSP is C-complete for as large a set C as possible. The larger C, the more evidence we accumulate for TSP’s intractability.

18. Arguing TSP is C -complete for a large C 18 C C1C1 C1C1 TSP C3C3 C3C3 C4C4 C4C4 C5C5 C5C5 C6C6 C6C6 CkCk CkCk C2C2 C2C2

19. Picking an Appropriate C Option 1: Let C be any problem Q: Could we hope to prove that TSP is as hard as any other computational problem? A: No, there are problems, such as the Halting Problem, which we know are simply not solvable! (i. e, do not have any algorithms solving them, let alone an efficient one). But TSP is solvable: if nothing, by brute-force search. Exponential-time but solvable.

20. Option 2: C = NP: Brute-force Solvable Problems Option 2: Let C be all brute-force solvable problems Definition (NP or brute-force solvable problems): A problem C ∈ NP if: 1.Correct solutions have polynomial length. 2.Claimed solutions are verifiable in poly-time. ***All that is required to be in NP is that we can verify the solution to the problem efficiently.***

21. Example NP problems  Every problem we have seen in CS 161.  Every problem in P.  (Most-likely) Every problem you will ever see in practice.  Ex: Decision version of TSP: ∃ T of size ≤ k? 1.Each tour T is a cycle of size n, so poly-length. 2.We can verify length of T in poly-time: just sum each edge in T.

22. What is an NP-complete Problem? 22 NP C1C1 C1C1 C*: NP- complete C3C3 C3C3 C4C4 C4C4 C5C5 C5C5 C6C6 C6C6 CkCk CkCk C2C2 C2C2 C* is as hard as any NP problem!

23. Solving an NP-complete Problem 23 NP C1C1 C1C1 C*: NP- complete C3C3 C3C3 C4C4 C4C4 C5C5 C5C5 C6C6 C6C6 CkCk CkCk C2C2 C2C2 If we can solve C* efficiently, we can solve every single NP problem (basically all problems in practice)!!!

24. Do NP-complete Problems Exist? 24 YES! They’re Everywhere!

25. History of NP-completeness 25  < 1970: lots of unsolved problems  1971: Cook-Levin Thm: SAT is NP-complete (just from the definition of NP)  1972: Karp: By showing SAT reduces to 21 other problems, showed the existence of 21 other NP- complete problems. (why?)  Since 1972: 1000s of problems are NP-complete (including TSP).

26. History of NP-completeness 26 NP C2C2 C2C2 SAT C3C3 C3C3 C4C4 C4C4 C5C5 C5C5 CkCk CkCk C6C6 C6C6 C1C1 C1C1 K1K1 K1K1 K2K2 K2K2 … … K 20 K 21 1971: Cook-Levin1972: Karp

27. History of NP-completeness 27 NP C2C2 C2C2 SAT C3C3 C3C3 C4C4 C4C4 C5C5 C5C5 CkCk CkCk C6C6 C6C6 C1C1 C1C1 K1K1 K1K1 K2K2 K2K2 … … K 20 K 21 NP-complete Since 1972

28. Reducing C* to C** => C** is NP-complete 28 NP C2C2 C2C2 C*: NP- complete C3C3 C3C3 C4C4 C4C4 C5C5 C5C5 CkCk CkCk C6C6 C6C6 C ** If we can solve C** efficiently => we solve C* efficiently => we solve all NP problems efficiently C** is NP-complete!

29. Overall “Intractability” Argument for TSP 29 Can’t prove TSP is hard in an absolute sense. Argue that TSP is “hard” in a relative sense. Show that TSP is NP-complete => i.e. TSP is as hard as any other NP problem This is our evidence that TSP is intractable.

30. Why is TSP NP-complete? 30 HAM-CYCLE Problem: Input: Undirected Graph G(V, E) Output: YES if G contain a Hamiltonian cycle/NO o.w Dfn: A Ham. Cycle is a simple cycle that contain each vertex of the graph once. Fact: Hamiltonian Cycle is NP-complete. SAT≤ p 3-SAT≤ p CLIQUE≤ p VERTEX-COVER≤ p HAM-CYCLE

31. Hamiltonian Cycle Example (1) 31 F CB A D E G

32. Hamiltonian Cycle Example 32 F CB A D E G Answer: YES.

33. Hamiltonian Cycle Example 33 F C B A E

34. Hamiltonian Cycle Example 34 F C B A E Answer: NO. Any complete cycle has to visit A twice

35. HAM-CYCLE ≤ p TSP 35 Let G(V, E) be the input to HAM-CYCLE Let G*(V, E*) be a complete graph with edge weights: w((u, v)) = 0 if (u, v) ∈ E w((u,v)) = 1 if (u, v) ∉ E Claim: ∃ a Hamiltonian Cycle in G ⇔ ∃ a TSP Tour with weight 0

36. Proof of Claim 36 => If ∃ a Hamiltonian Cycle C in G, then since each edge of C has weight 0 in E*, then C is a tour in G* with weight 0 <= If ∃ a tour T with weight 0 in G*, then all of its edges must be of weight 0, and hence from E, so T is a hamiltonian cycle in G Q.E.D Therefore, if we can solve TSP efficiently, we can solve HAM-CYCLE efficiently,

37. Completing TSP’s NP-completeness Proof 37 If we can solve TSP efficiently, we can solve HAM- CYCLE efficiently (by just transforming the input G to G* in poly-time) since HAM-CYCLE is NP-complete  we can solve every single NP problem efficiently  TSP is NP-complete Q.E.D

38. Does P = NP? 38 We know P ⊆ NP. Two possibilities: NP P P=NP Is every problem, whose solutions are efficiently verifiable, is also efficiently solvable? We don’t know. But most people believe P ≠ NP.

39. Why Do People Believe P ≠ NP? 39 1.There are 1000s of NP-complete problems and for none of them is there a poly-time algorithm. 2.Counter to General Human Experience 3.Also some weird mathematical consequences, such as polynomial hierarchy collapsing, and others… But We Simply Don’t Know (Yet)!

40. Outline For Today 1.Intractability: P, NP, NP-completeness 2.How to Approach Intractable Problems? 40

41. Your Problem is NP-complete. Now What? 41  Option 1: Focus to special-case inputs.  Ex: Independent Set is NP-complete.  Focusing on line graphs, had a O(n) DP alg.  Option 2: Find an approximate answer.  Ex: Max-cut is NP-complete  We had O(n)-time expected ½-approximation  Option 3: Be exponential time but better than brute- force search.  Option 4: Heuristics: fast algorithms that are not always correct (or even approximate)  Option 5: Mix some of these options

42. Vertex Cover (Ex of Options 1 & 3) 42  Input: Undirected Graph G(V, E)  Output: Minimum Vertex Cover of G  Vertex Cover: S ⊆ V, s.t. for each (u, v) ∈ E: either u ∈ S, or v ∈ S.  Fact: Vertex Cover is NP-complete:  3-SAT≤ p CLIQUE≤ p VERTEX-COVER

43. Vertex Cover Example (1) 43

44. Vertex Cover Example (1) 44

45. Vertex Cover Example (2) 45 B D C A E F

46. Vertex Cover Example (2) 46 B D C A E F

47. Vertex Cover Example (2) 47 B D C A F Min Vertex Cover: {A, B} F

48. Let’s Hope that There is a VC That’s Small 48  Suppose we know that ∃ VC of size ≤ k, with k small  So, we’re focusing on special case inputs  Bad Brute-force Search:  Look at all choose(n, k) possible VCs = Ω(n k )  So now we have a poly-time algorithm  Problem 1: we didn’t understand the problem at all  Problem 2: even with k=5, we’re n 5, not practical

49. Smarter Brute-Force Search (1) 49  Consider an edge (u, v). Let S* be a VC with |S*|≤ k  Q: What’s a claim about VCs that doesn’t need a proof?  A: S* contains either (1) u or (2) v (or both)  Idea: Guess which case we’re in and recurse

50. Smarter Brute-Force Search (2) 50 u G` v G`` Case 1 Case 2 Q: Consider case 1. What can you assert about S*-{u}? A: S*-{u} is a VC in G` with size ≤ k -1.

51. Smarter Brute-Force Search (2) 51 u G` v G`` Case 1 Case 2 Q: Consider case 2. What can you assert about S*-{v}? A: S*-{v} is a VC in G`` with size ≤ k -1.

52. Smarter Brute-Force Search (3) 52 G has a VC of size ≤ k ⇔ G` or G`` has a VC of size ≤ k-1

53. Smarter Brute-Force Search Pseudocode 53 procedure Smarter-BruteFS (G(V,E), k): if k == 0: if (G is empty): return empty-set else return null pick an arbitrary edge (u, v) Let G` = (V – {u}, E- all (u, w)) Let G`` = (V – {v}, E- all (v, w)) Let T 1 = Smarter-BruteFS(G`, k-1) Let T 2 = Smarter-BruteFS (G``, k-1) if (both T 1 T 2 are null) return null else return T 1 ∪ u or T 2 ∪ v

54. Runtime: 54  T(k) ≤ 2T(k-1) + m  Branching factor 2, depth k: O(2 k m)  Compared to the first brute-force search alg  k=5 => n 5 vs 2 5 m  k=8 => n 8 vs 2 8 m  k=15 => n 15 vs 2 15 m Could be much better than the first brute-fs!

55. TSP with Triangle Inequality (Options 1& 2) 55 Input: Complete graph G(V, E) non-negative edge costs, where triangle inequality holds Output: Min-cost tour [cycle visiting each vertex once] Triangle Inequality: ∀ u, v, w: d(u, v) ≤ d(u, w) + d(w, v) Upshot: We will get at 2-approximation to OPT Tour C* Option 1 b/c we’re restricting to a subclass of all inputs Option 2 b/c we only approximate the exact solution

56. TSP with Triangle Inequality Example 56 D C B A 2 1 1 3 2 2 Triangle Inequality: ∀ u, v, w: d(u, v) ≤ d(u, w) + d(w, v)

57. TSP with Triangle Inequality Example 57 D C B A 2 1 1 3 2 2 Triangle Inequality: ∀ u, v, w: d(u, v) ≤ d(u, w) + d(w, v)

58. TSP with Triangle Inequality Example 58 D C B A 2 1 1 3 2 2 Triangle Inequality: ∀ u, v, w: d(u, v) ≤ d(u, w) + d(w, v) d(B, D) = 3 ≤ d(B, A) + d(A, D)=3

59. TSP with Triangle Inequality Example 59 D C B A 2 1 1 3 2 2 Triangle Inequality: ∀ u, v, w: d(u, v) ≤ d(u, w) + d(w, v) D(B, D) = 3 ≤ d(B, C) + d(C, D)=4 Similarly holds for all other edges

60. TSP with Triangle Inequality Example 60 D C B A 2 1 1 3 2 2 Triangle Inequality: ∀ u, v, w: d(u, v) ≤ d(u, w) + d(w, v) OPT C*= 1+2+2+2 = 7

61. “Taking Shortcuts” Lemma 61 Assume the triangle inequality holds in a complete graph. Let P=u->x 1 -> … -> x k ->v be any path in G, then: Claim: d(u, v) ≤ d(u, x 1 ) + d(x 1, x 2 ) + … d(x k-1, x k )+d(x k, v) uv xkxk x1x1 P (u,v) Claim: d(u, v) ≤ cost(P)

62. Proof by Induction on path lengths k 62 Base case: k=1 => This is true by the triangle inequality IH: Holds for path lengths k-1 d(u, v) ≤ d(u, x k ) + d(x k, v) (by triangle inequality) uv xkxk x1x1 d(u, x k ) ≤ d(u, x 1 ) + d(x 1, x 2 ) + … d(x k-1, x k ) (by IH) d(u, v) ≤ d(u, x 1 ) + d(x 1, x 2 ) + … d(x k-1, x k ) + d(x k, v)

63. Removing any Edge of TSP Tours 63 D C B A 2 1 1 3 2 2 Any tour C is a simple cycle in the graph Consider removing any edge of C: C-(u, v) Ex: Consider C*, and remove (A, D) (any edge in general)

64. Removing any Edge of TSP Tours 64 D C B A 2 1 1 3 2 2 Any tour C is a simple cycle in the graph Consider removing any edge of C: C-(u, v) Ex: Consider C*, and remove (A, D) (any edge in general) Q: What other graph structure is C` = C –(A, D)? A: C` is a spanning tree B/c it has n-1 edges in it’s connected

65. Cost(MST) ≤ Cost(C* - (u,v)) ≤ Cost(C*) 65 D C B A 2 1 1 3 2 2 MST T* = 1 + 1 + 2 =4

66. Let’s Run a DFS On MST T* 66 D C B A 2 1 1 3 2 2

67. Let’s Run a DFS On MST T* 67 D C B A 2 1 1 3 2 2 D

68. Let’s Run a DFS On MST T* 68 D C B A 2 1 1 3 2 2 D -> A

69. Let’s Run a DFS On MST T* 69 D C B A 2 1 1 3 2 2 D -> A -> C

70. Let’s Run a DFS On MST T* 70 D C B A 2 1 1 3 2 2 D -> A -> C

71. Let’s Run a DFS On MST T* 71 D B A 2 1 1 3 2 2 D -> A -> C -> A C

72. Let’s Run a DFS On MST T* 72 D B A 2 1 1 3 2 2 D -> A -> C -> A -> B C

73. Let’s Run a DFS On MST T* 73 D B A 2 1 1 3 2 2 D -> A -> C -> A -> B C

74. Let’s Run a DFS On MST T* 74 D B A 2 1 1 3 2 2 D -> A -> C -> A -> B -> A C

75. Let’s Run a DFS On MST T* 75 D B A 2 1 1 3 2 2 D -> A -> C -> A -> B -> A C

76. Let’s Run a DFS On MST T* 76 D B A 2 1 1 3 2 2 D -> A -> C -> A -> B -> A -> D C

77. Let’s Run a DFS On MST T* 77 D C B A 2 1 1 3 2 2 D -> A -> C -> A -> B -> A -> D

78. Cost of X=(D -> A -> C -> A -> B -> A -> D)? 78 D C B A 2 1 1 3 2 2 A: 2*cost(T*) But it’s not a valid tour, because it visits some vertices twice.

79. Trimming X=(D -> A -> C -> A -> B -> A -> D) 79 D -> A -> C -> A -> B -> A -> D Trim X by taking “shortcuts”. D

80. Trimming X=(D -> A -> C -> A -> B -> A -> D) 80 D -> A -> C -> A -> B -> A -> D Trim X by taking “shortcuts”. D

81. Trimming X=(D -> A -> C -> A -> B -> A -> D) 81 D -> A -> C -> A -> B -> A -> D Trim X by taking “shortcuts”. D -> A

82. Trimming X=(D -> A -> C -> A -> B -> A -> D) 82 D -> A -> C -> A -> B -> A -> D Trim X by taking “shortcuts”. D -> A

83. Trimming X=(D -> A -> C -> A -> B -> A -> D) 83 D -> A -> C -> A -> B -> A -> D Trim X by taking “shortcuts”. D -> A -> C

84. Trimming X=(D -> A -> C -> A -> B -> A -> D) 84 D -> A -> C -> A -> B -> A -> D Trim X by taking “shortcuts”. D -> A -> C

85. Trimming X=(D -> A -> C -> A -> B -> A -> D) 85 D -> A -> C -> A -> B -> A -> D Trim X by taking “shortcuts”. D -> A -> C

86. Trimming X=(D -> A -> C -> A -> B -> A -> D) 86 D -> A -> C -> A -> B -> A -> D Trim X by taking “shortcuts”. D -> A -> C -> B

87. Trimming X=(D -> A -> C -> A -> B -> A -> D) 87 D -> A -> C -> A -> B -> A -> D Trim X by taking “shortcuts”. D -> A -> C -> B

88. Trimming X=(D -> A -> C -> A -> B -> A -> D) 88 D -> A -> C -> A -> B -> A -> D Trim X by taking “shortcuts”. 1.Trimmed path Y is now a valid tour. 2.Shortcuts can only decrease the cost => cost(Y) ≤ 2cost(T*) ≤ 2C* D -> A -> C -> B -> D

89. 2-Approx TSP with Triangle Inequality 89 procedure 2-ApproxTSP(G(V,E), edge weights): Let T* = MST(G(V, E)) // Kruskal/Prim Run DFS(T*) and get a non-simple tour X Y = trim(X) return Y Correctness (already argued) Run-time: O(mlog(n))

90. Summary 90 1.Took TSP, an NP-complete problem (don’t hope to solve exactly in general form) 2.Restricted inputs to those with triangle ineq. 3.Got a 2-approximation in O(mlog(n)) time

91. Attacking NP-complete Problems 91  Option 1: Focus to special-case inputs.  Ex: Independent Set is NP-complete.  Focusing on line graphs, had a O(n) DP alg.  Option 2: Find an approximate answer.  Ex: Max-cut is NP-complete  We had O(n)-time expected ½-approximation  Option 3: Be exponential time but better than brute- force search.  Option 4: Heuristics: fast algorithms that are not always correct (or even approximate)  Option 5: Mix some of these options

92. Attacking Vertex Cover 92  Option 1: Focus to special-case inputs.  Ex: Independent Set is NP-complete.  Focusing on line graphs, had a O(n) DP alg.  Option 2: Find an approximate answer.  Ex: Max-cut is NP-complete  We had O(n)-time expected ½-approximation  Option 3: Be exponential time but better than brute- force search.  Option 4: Heuristics: fast algorithms that are not always correct (or even approximate)  Option 5: Mix some of these options

93. Attacking TSP 93  Option 1: Focus to special-case inputs.  Ex: Independent Set is NP-complete.  Focusing on line graphs, had a O(n) DP alg.  Option 2: Find an approximate answer.  Ex: Max-cut is NP-complete  We had O(n)-time expected ½-approximation  Option 3: Be exponential time but better than brute- force search.  Option 4: Heuristics: fast algorithms that are not always correct (or even approximate)  Option 5: Mix some of these options

94. 94 More on Approaching NP-complete problems on Wednesday (Knapsack)