1 Complexity Lecture Ref. Handout p

2 Measuring Program Performance How can one algorithm be ‘better’ than another? Time taken to execute ? Size of code ? Space used when running ? Scaling properties ?

3 Scaling Amount of data Time

4 Time and Counting initialise for (i = 1; i<=n; i++) { do something } finish Time needed: t1 set up loop t2 each iteration t4 t5 The total ‘cost’ is t1+t2+n ∗ t4+t5 Instead of actual time, t i can be treated as how many instructions needed

5 The computation cost can be estimated as a function f(x) In the above example, cost is f(x) = t1+t2+x ∗ t4+t5 = c + m ∗ x ( c = t1+t2+t5, m = t4 ) f(x) =c+mx x (size of data) c m = slope a linear function

6 The computation cost can be estimated as a function f(x) – more examples cost functions can be any type depending on the algorithm, e.g. f(x) = c ( c is a constant ), h(x) = log(x), or g(x) = x 2 f(x) =c x (size of data) c cost stays same h(x) = log(x) cost grows very slow g(x) =x 2 quadratic growth

7 The Time Complexity A cost function for an algorithm indicates how computation time grows when the amount of data increases A special term for it – TIME COMPLEXITY complexity = time complexity/space complexity Space complexity studies the memory usage (but mostly we pay our attention to time complexity) “Algorithm A’s time complexity is linear” = A’s computation time is proportional to the size of A’s input data

8 The Big-O Notation How to compare (evaluate) two cost functions Is n+100*n 2 better than 1000*n 2 It has been agreed to use a standard measurement – an order of magnitude notation – the Big-O notation

9 The Big-O Notation - definition A function f(x) is said to be the big-O of a function g(x) if there is an integer N and a constant C such that f(x) ≤ C ∗ g(x) for all x ≥ N We write f(x) = O(g(x))

10 The Big-O Notation - Examples f(x) = 3 ∗x g(x) = x C = 3, N =0 f(x) ≤ 3 ∗g(x) f(x) = 4+7 ∗x g(x) = x C = 8, N =4 f(x) ≤ 8 ∗g(x) g(x) = x f(x) = 3x = 3g(x) f(x) = 4+7x g(x) = x 8*g(x) 3 ∗ x = O(x)4+7 ∗ x = O(x) N

11 The Big-O Notation - Examples f(x) = x+100 ∗x 2 g(x) = x 2 C = 101, N = 1 f(x) ≤ 101 ∗g(x) f(x) = 20+x+5 ∗x 2 +8 ∗x 3 g(x) = x 3 C = 15, N =20 f(x) ≤ 9 ∗g(x) when x > 1 x+100 ∗x 2 < x ∗x 2 x ∗x 2 = 101*g(x) x+100 ∗x 2 = O(x 2 ) when x > x+5 ∗x 2 +8 ∗x 3 < x 3 + x 3 +5 ∗x 3 +8 x 3 = 15 ∗x 3 = 15*g(x) 20+x+5 ∗x 2 +8 ∗x 3 = O(x 3 )

12 Power of n A smaller power is always better eventually. for larger enough n The biggest power counts most. for larger enough n * n < n 2 10 *n *n *n *n + 77 < 11*n 4

13 Using Big-O Notation for classifying algorithms 23*n *n *n or even 100*n 2 + n All cost functions above is O(n 3 ) Performance is no worse than C*n 3 eventually

14 Working Out Big-O Ignore constants in multiplication - 10*n is O(n) Leave products – n*log(n) is O( n*log(n) ) Use worst case in sums - n *n + 2 is O(n 2 )

15 Memo for In-class test 17 [ /5] questionsmy answers correct answers comments

16 Some Typical Algorithms - Search Search an array of size n for a particular item Sequential search Time taken is O( ) Binary search Time taken is O( )

17 Some Typical Algorithms - Sorting Insertion sort Use 2 lists – unsorted and sorted insertionSort(input_list) { unsorted = input_list (assume size = n) sorted = an empty list loop from i =0 to n-1 do { insert (unsorted[i], sorted) } return sorted } Time taken is O( )

18 Some Typical Algorithms - Sorting Bubble sort Swap if x i < x i-1 for all pairs Do this n times (n = length of the list) x 0 3 x 1 5 x 2 1 x 3 2 x 4 6 Time taken is O( )

19 Some Typical Algorithms - Sorting Quick sort quickSort(list) { If List has zero or one number return list Randomly pick a pivot x i to create two partitions larger than x i and smaller than x i return quickSort(SmallerList) + x i + quickSort(LargerList) } // ‘+’ means appending two lists into one Time taken is O( )

20 Worst Cases, Best Cases, and Average Cases Complexity: performance on the average cases Worst/Best cases: The worst (best) performance which an algorithm can get E.g. 1 - bubble sort and insertion sort can get O(n) in the best case (when the list is already ordered) E.g. 2 – quick sort can get O(n 2 ) in the worst case (when partition doesn’t work well)

21 Sorting a sorted list

22 A Summary on Typical Algorithms Sequential search O(n) Binary search (sorted list) O(log(n)) Bubble sort O(n 2 ) best O(n) worst O(n 2 ) Insertion sort O(n 2 ) best O(n) worst O(n 2 ) Quick sort O(log(n)*n) best O(log(n)*n) worst O(n 2 )

23 Can you order these functions ? smaller comes first log(n) log(n)*n n 4/3 √n n 2 n (log(n)) 2 n 2

24 Memo for In-class test 18 [ /5] questionsmy answers correct answers comments

25 Speeding up the Machine can solve problem of size N in one day machine speed up Complexity X 1000 X O( log(n) ) N 1000 N O( n ) 1000 * N * N O( n 2 ) 32 * N 1000 * N O( n 3 ) 10 * N 100 * N O( 2 n ) N + 10 N + 20 size

26 Code Breaking A number is the product of two primes How to work out prime factors? i.e.35 = 7* = 59* = * Complexity is O(10 n ) where n is the number of digits ( or O(s) where s is the number) Typically n is about 150

27 Cost of Speeding Up For each additional digit multiply time by 10 ( as the complexity is O(10 n ) ) For 10 extra digits increase time from 1 second to 317 years For 18 extra digit increase time from 1 second to twice the age of the universe

28 Where Different Formulas Appear O(log(n)) find an item in an ordered collection O( n ) look at every item O( n 2 ) for every item look at every other item O( 2 n ) look at every subset of the items

29 Do Big-O Times Matter? Is n 3 worse than *n ? Algorithm A is O(n 2 ) for % of cases and O(2 n ) for 0.001% cases Algorithm B is always O(n 10 ) Which one is better?

30 Degree of Difficulty log(n) n*log(n) n n, n (2^n) n, n 2, n 3, n, 3 n, polynomial time ‘tractable’ exponential time ‘intractable’

31 A Simple O(2 n ) Problem Given n objects of different weights, split them into two equally heavy piles (or say if this isn’t possible)

32 Hard Problem Make Easy Magic Machine (for problem A) Possible Data for problem A Correct Answer? Answer checker

33 NP Problems A problem is in the class NP (Non-deterministic Polynomial Time) if the checking requires polynomial time i.e. O(n c ) where C is a fixed number) Note: problems not algorithms

34 Example Given n objects of different weights, split them into two equally heavy piles {1,2,3,5,7,8} Algorithm = 8+5 Checking ok ✔ O(2 n ) O(n)

35 In Previous Example... We know there are algorithms takes O(2 n ) But How do we know there isn’t a better algorithm which only takes polynomial time anyway?

36 Some Classes of Problems P NP problems which can be solved in polynomial time using a magic box problems which can be solved in polynomial time

37 Problems in NP but not in P NPP P ?=?= is a subset of but Most problems for which the best known algorithm is believed to be O(2 n ) are in NP P NP

38 The Hardest NP Problems If a polynomial time algorithm is found for any problem in NP-Complete then every problem in NP can be solved in polynomial time. i,e, P = NP P NP NP-Complete

39 Examples of NP-Complete Problems The travelling salesman problem Finding the shortest common superstring Checking whether two finite automata accept the same language Given three positive integers a, b, and c, do there exist positive integers such that a*x 2 + b*y 2 = c

40 How to be Famous Find a polynomial time algorithm for this problem Given n objects of different weights, split them into two equally heavy piles (or say if this isn’t possible)

41 Memo for In-class test 19 [ /5] questionsmy answers correct answers comments