Computational Complexity Dr. Colin Campbell Course: EMAT20531

Order Notation for Functions The order notation is used to give a bound on the limiting behaviour of a function. We write f(x) is O(g(x)) iff

Order example f=100x3+50x2+10x+100 g=x3 f-mg m=110 m=120 m=130 m=140

The big O Notation for Algorithms We can represent the complexity of an algorithm A by a function fA(n)=number of steps (or time) it takes for A to complete given an input of size n. The size of input may be measured in a number of different ways.

Big O Notation f(n)=2n+1 O(f(n))=n A is linear in n A= Do i=1,..,n Print “hello world” Clear Repeat Print “End” f(n)=2n+1 O(f(n))=n A is linear in n

Big O Notation: 2 f(n)=2n2+1 O(f(n) is n2 A= Do i=1,…,n Do j=1,…,n Print “Hello World” Clear Repeat Print “End” f(n)=2n2+1 O(f(n) is n2

The Quicksort Algorithm Input: a list L=(a1,…,an) of numbers Choose an number p at random from L Determine L1 all numbers ai less than p and L2 all numbers ai greater than or equal to p Recursively sort L1 and L2 Solution=concatenate L1 and L2

Quicksort Example Sort the list (1,5,2,7,6,4) L=(1,5,2,7,6,4) sort p=4 [(1,2)(4,5,6,7)=(1,2,4,5,6,7)] L1=(1,2) L2=(5,7,6,4) sort L=(5,7,6,4) p=6 [(4,5)(6,7)=(4,5,6,7)] L1=(5,4), L2=(7,6) sort L=(5,4) [(4,5)] sort L=(7,6) [(6,7)]

Quicksort: The Worst Case Given p partitioning into L1 and L2 is O(n) The worst possible case would be that every time we partition the list we have L1 length n-1 and L2 length 1 For the i’th partition we have a computation O(n-i) Now (n-i)=n2. Hence the entire computation is O(n2).

Quicksort: On Average On average the split between L1 and L2 will result in two lists of n/2. This is based on the assumption that the elements of the list are selected from an interval [x,y] according to a uniform distribution Consider P(pai), the expected number of elements in L2 is nP(pai)

Quicksort: On Average (2) p is a uniformly distributed random variable taking values in [x,y] Therefore ai is also a uniformly distributed random variable from [x,y] with expected value (x+y)/2. Therefore, the expected value of P(pai) is ½ and the expected number of elements in L2 is n/2

Quicksort: On Average (3) Consider, the number of such partitions before we end up with strings of length 1. i.e. find k for which n/2k=1 Log(n/2k)=0 log(n)=log(2k)=k Hence, there are on average log(n) partitions each requiring O(n) computations This gives O(nlog(n))

Polynomial Time An algorithm runs in polynomial time if fA(n) is O(nk) for some positive integer k. E.g. Quicksort always runs in polynomial time even in the worst case when it is O(n2) The class P refers to those algorithms which run in polynomial time on a Turing Machine as described in the earlier section.

Non-deterministic Turing Machines A Non-deterministic Turing machine is a Turing machine where for some state qi and symbol S there is more than one instruction beginning qiS. 1:R q2 q1 0:R q3

Computation Paths A deterministic Turing machine follows a computation path. A Non-Deterministic generates a tree of computations. We can either think of a NDTM as following all paths simultaneously until it finds an accepting state Or being a very lucky guesser between when choosing between states

Accepting States and Paths For NDTM we usually think of binary yes no responses from a computation. We define a set of accepting state AQ. An accepting path is a path which terminates in an accepting state. A NDTM accepts an input x if there is an accepting path for that input.

NDTM Example Let TMP be a deterministic Turing machine that given a string of 1s and 0s terminates in state q+ if there are an even number of 1 and q- otherwise. AQ={q+} Further suppose that the start state of TMP is q3 Given a string of 1s we mean substrings to be a string of the same length where some or all of the 1s have been switched to 0. Eg 111 has substrings 111,110,101,100, 011,010,001,000

NDTM Example: 2 TMP 1111 0:R 1:0 1111 0211 1111 1021 0111 1:R 1113 q1 q2 1111 1021 0111 1:R TMP q3 1113 1103 1011 0111 0011 q- q+ 1013 1003 0113 0103 0013 0003 Several deterministic steps q+ q- q+ q- q- q+

Non-Deterministic Polynomial Intuitively, we see that a NDTM is more efficient than a DTM since it can explore many computational paths simultaneously. NP refer to the class of algorithms which run in polynomial time on a NDTM. Clearly PNP but does P=NP?

NP Complete NP-Complete are a collection of decision problems for which there is an algorithm in NP but no known algorithm in P. Formally, a problem in NP is NP-Complete if any other problem in NP can be reduced to it. Problem C is reducible to NP-Complete problem D if there is an algorithm A which solves C with a subroutine A+ that solves D, such if A+ ran in polynomial time so would A.

SAT: An NP Complete Problem SAT: Given a sentence of Propositional Logic S in CNF, is there an allocation of truth values for which S is true? There is no known algorithm in P which solves SAT This is basically because all (known) approaches require us (in the worst case) to check all possible truth value allocations. If S has n propositional variables there are 2n possible allocations of truth values.

A NDTP algorithm for SAT Simplify things by assuming that S is in CNF and each clause has exactly k literals for k>2. (k-SAT) Notice that for any given allocation of truth values we can check if satisfies S in O(nk) – n is the number of variables in S. There are 2n(2n-1)..(2n-k+1)/3! Different clauses. Assume we can check if each is satisfied in 1 step then the computation is O(nk)

SAT Algorithm: 2 S=(PR Q)(PRQ) Algorithm For each variable guess either t or f. [O(n) on NDTM] Check if allocation of truth values satisfies S [O(nk)] P t f Q Q t f t f R R R R t f t f f t f t accept accept accept accept accept accept reject reject

Proving NP-Completeness Usually a problem D is shown to be NP-complete by proving that a known NP-complete problem C reduces to it. All NP problems reduce All NP problems reduce NP complete C reduce D D

CLIQUE In graph theory a CLIQUE is a set of vertices (nodes) where each is connected to the others. 5 6 1 4 5 2 1 Clique of size 3 3 2

The CLIQUE Problem For a given graph decide whether or not there is a clique of size k. This has application in chip design where for example nodes are wires on a chip and edges signify that two wires can overlap. CLIQUE tells the designer something about how much space is required on the chip. We shall show that SAT reduces to CLIQUE So CLIQUE is NP-Complete.

CLIQUE and SAT Let S be a sentence in CNF with k clauses. S=C1C2…Ck We construct a graph G where Vertices={Li:LiCi} i.e. every literal from every clause Edges={<Li,Lj> LiLj and ij} i.e. edges between literals from different clauses provides they do not conflict. S is satisfiable if and only if G has a clique of size k.

CLIQUE and SAT: 2 Example S=(PQ)(PQ)(PR) L1=P, L2=Q, L3=P, L4=Q, L5=P, L6=R Graph G L1 L5 L4 L2 L6 L3

CLIQUE and SAT: 3 (If) Suppose there is a clique of size k then there are k non-conflicting literals, one from each clause in S. Since they are non-conflicting there is a truth allocation that makes each true and hence each clause in S true. (Only If) Suppose there is an allocation of truth values for which S is true. At least one literal from each clause must be true These cannot be conflicting, hence they must form a clique of size k in G

CLIQUE and SAT: 4 L1 L5 L4 Graph G L1 L4 L5 L4 L2 P=t,Q=t,R={t,f} L6 P={t,f},Q=t,R=t P=t,Q=t,R={t,f} P=f,Q=t,R=t

Other NP-Complete Problems: The Knapsack Problem Weight: wi Utility: ui Quantity: xi 500g 100 300g 400 Maximize 1kg 200 subject to 750g 80 Decision problem: is there a solution for which and ?

Other NP-Complete Problems: The Travelling Salesman Problem 10 Starting at A find the shortest route which visits all towns and then returns to A. A B 8 7 18 4 C D F 12 9 16 25 E