Algorithms and their Applications CS2004 ( ) Dr Stephen Swift 4.1 Time Complexity and Asymptotic Notation

The Laboratories – Part 1 Blackboard The list of laboratories and what grade they cover is on Blackboard Note we are using threshold based grading With the Mathematics Test: E You only need to take one test (both parts) to get an E grade You get up to five attempts however… The purpose of the test is for you to assess your own Mathematics ability ASK Sign up to ASK workshops if there is anything that you need to refresh yourself on Time Complexity and Asymptotic NotationSlide 2

The Laboratories – Part 2 Most of the worksheets go towards a specific grade The worksheets must be completed in order When you feel you have completed a worksheet, ask a GTA to assess it Keep a log book of any feedback and comments If the worksheet is deemed to have been completed successfully the GTA will “sign off” the worksheet must We will keep a note and you must get the GTA to date and sign a print out of the worksheet coversheet Time Complexity and Asymptotic NotationSlide 3

Previously On CS2004… So far we have looked at: Concepts of Computation and Algorithms Comparing algorithms Some mathematical foundation Time Complexity and Asymptotic NotationSlide 4

Time Complexity and Asymptotic Notation Within this lecture we will discuss: The Selection sort algorithm The Binary search algorithm Big-Oh notation Big-Omega and Big-Theta notations Primitive Operations This builds upon the core topic of counting Primitive Operations we covered two weeks ago... Time Complexity and Asymptotic NotationSlide 5

Why Sorting? Sorting is one of the most common tasks in data analysis Examples: Print out a collection of employees sorted by salary Print out a list of names in alphabetical order There are many sorting algorithms Selection Sort The Selection Sort algorithm repeatedly finds the smallest element in the unsorted tail region of the array and moves it to the front Question: How do you sort 11,9,17,5,12? Time Complexity and Asymptotic NotationSlide 6

Selection Sort algorithm – Part 1 Sorting an Array of Integers Array in original order Find the smallest and swap it with the first element Time Complexity and Asymptotic NotationSlide 7

Find the next smallest It is already in the correct place Find the next smallest and swap it with the first element of unsorted portion Selection Sort algorithm – Part Time Complexity and Asymptotic NotationSlide 8

Repeat When the unsorted portion is of length 1, we are done Selection Sort algorithm – Part Time Complexity and Asymptotic NotationSlide 9

How Fast is the Algorithm? In an array of size n, count how many primitive operations are needed To find the smallest, visit n elements + 2 visits for the swap To find the next smallest, visit ( n -1) elements + 2 visits for the swap The last term is 2 elements visited to find the smallest + 2 visits for the swap The number of visits: (n + 2) + [(n-1) + 2] +[(n-2) +2] (1+ 2) + 2 This can be simplified to n 2 /2 + 5n/ n/2 +2 is small compared to n 2 /2 - so let's ignore it Also ignore the 1/2 - use the simplest expression of the class The number of visits is of the order n 2 Time Complexity and Asymptotic NotationSlide 10

The Big-Oh notation The number of visits is of the order n 2 Using Big-Oh notation: The number of visits is O(n 2 ) If you multiply the number of elements in an array by 2, the processing time will be multiplied by 4! So what is the formal definition of Big-Oh notation? So what is the formal definition of Big-Oh notation? Time Complexity and Asymptotic NotationSlide 11

Big-Oh Notation – Part 1 Given functions f(n) and g(n), we say that f(n) is O(g(n)) if there are positive constants c and n 0 such that: f(n)  cg(n) for n  n 0 Time Complexity and Asymptotic NotationSlide 12 Example: Is 2n + 10  O(n) ? Can we find c and n 0 ? 2n + 10  cn (c-2)n  10 n  10/(c-2) Pick c = 3 and n 0 = 10

f(n) is O(g(n)) iff f(n)  cg(n) for n  n 0 Time Complexity and Asymptotic NotationSlide 13

Big-Oh Notation – Part 2 Example: the function n 2 is not O(n)? Why? n 2  cn n  c The above inequality cannot be satisfied since c must be a constant Time Complexity and Asymptotic NotationSlide 14

Big-Oh and Growth Rate The Big-Oh notation gives an upper bound on the growth rate of a function The statement “ f(n) is O(g(n)) ” means that the growth rate of f(n) is no more than the growth rate of g(n) The statement “ f(n) is O(g(n)) ” means that the growth rate of f(n) is no more than the growth rate of g(n) f(n) grows no faster than g(n) f(n) grows no faster than g(n) We can use the Big-Oh notation to rank functions according to their growth rate Time Complexity and Asymptotic NotationSlide 15

Example: Binary Search Task: search for a key in a sorted list First check the middle list element If the key matches the middle element, we are done If the key is less than the middle element, the key must be in the first half If the key is larger than the middle element, the key must be in the second half Examples? Spell checkers, phone books, internet browser cache… Time Complexity and Asymptotic NotationSlide 16

Time Complexity and Asymptotic Notation Binary Search Example 3 3 33 Binary search is an O( log 2 (n) ) algorithm The Binary search algorithm is much faster that the linear search algorithm of order O(n) But it only works on ordered data....

Questions T(n) will be used to denote the time an algorithm takes to execute Is T(n) = 9n n  O(n 4 )? Is T(n) = 9n n  O(n 3 )? Is T(n) = 9n n  O(n 27 )? T(n) = n n  O(?) T(n) = 3n + 32n n 2  O(?) Time Complexity and Asymptotic NotationSlide 18

Big-Oh Rules If f(n) is a polynomial of degree d, then f(n) is O(n d ), i.e., Drop lower-order terms Drop constant factors Use the smallest possible class of functions Say “ 2n is O(n) ” instead of “ 2n is O(n 2 ) ” Use the simplest expression of the class Say “ 3n  5 is O(n) ” instead of “ 3n  5 is O(3n) ” Time Complexity and Asymptotic NotationSlide 19

Asymptotic Algorithm Analysis The asymptotic analysis of an algorithm determines the running time in Big-Oh notation asymptotic The word asymptotic refers to long term (large input) algorithmic trends To perform the asymptotic analysis We find the worst-case number of primitive operations executed as a function of the input size, T(n) We express this function with Big-Oh notation Example: We have already known that the algorithm ArrayMax executes at most T(n) = 8n  5 primitive operations We say that algorithm ArrayMax “runs in O(n) time” Since constant factors and lower-order terms are eventually dropped, we can disregard them when counting primitive operations This is why we do not have to be 100% accurate as long as we get the powers of n correct I.e. we do not miscount n 3 as n 2 etc… Confusing 7n for 5n will not matter…… Time Complexity and Asymptotic NotationSlide 20

Rank From Fast to Slow… Before… T(n) = n 4 T(n) = n log n T(n) = n 2 T(n) = n 2 log n T(n) = n T(n) = 2 n T(n) = log n T(n) = n + 2n Time Complexity and Asymptotic NotationSlide 21 After… 1. T(n) =log n  O(log n) 2. T(n) = n  O(n) 2. T(n) = n + 2n  O(n) 4. T(n) = n log n  O(n log n) 5. T(n) = n 2  O(n 2 ) 6. T(n) = n 2 log n  O(n 2 log n) 7. T(n) = n 4  O(n 4 ) 8. T(n) = 2 n  O(2 n )

Relatives of Big-Oh  (g): functions that grow at least as fast as g  (g): functions that grow at the same rate as g  (g): functions that grow no faster than g Big-Omega Big-Theta Big-Oh Time Complexity and Asymptotic NotationSlide 22

Asymptotic Notation  (g): functions that grow at least as fast as g  (g): functions that grow at the same rate as g  (g): functions that grow no faster than g y=g(x) Time Complexity and Asymptotic NotationSlide 23

Big-Oh is the interesting one Because we are interested in efficiency,  ( g ) will not be of much interest to us because  ( n 2 ) includes all functions that grow faster than n 2, for example, n 3 and 2 n For a similar reason, we are not much interested in  ( g ), the class of functions that grow at the same rate as the function g Big-Oh is the class of functions that will be of the greatest interest to us Considering two algorithms, we will want to know if the first is in Big-Oh of the second If yes, we know that the second algorithm does not do better than the first in solving the problem Time Complexity and Asymptotic NotationSlide 24

25 But Can We Do Better? There are algorithms that have a computational complexity of O(2 n ) (this is very poor: 2 50  ) But isn’t How do we know there isn’t a better algorithm which only takes polynomial time anyway? Time Complexity and Asymptotic Notation

26 Some Classes of Problems P NP Problems which can be solved in polynomial time using a “magic box” (Non-deterministic) Problems which can be solved in polynomial time Time Complexity and Asymptotic Notation

27 Problems in NP but not in P NPP P ?=?= is a subset of but Most problems for which the best known algorithm is believed to be O( 2 n ) are in NP P NP Time Complexity and Asymptotic Notation

28 The Hardest NP Problems any every If a polynomial time algorithm is found for any problem in NP-Complete then every problem in NP can be solved in polynomial time i,e, P = NP P NP NP-Complete Time Complexity and Asymptotic Notation The “hardest” problems in NP

29 Examples of NP-Complete Problems Time Complexity and Asymptotic Notation The travelling salesman problem Finding the shortest common superstring Checking whether two finite automata accept the same language Given three positive integers a, b, and c, do there exist positive integers ( x and y ) such that ax 2 + by 2 = c

30 How to be Very Famous! Find a polynomial time algorithm for this problem: Given n objects of different weights, split them into two equally heavy piles (or say if this isn’t possible) Time Complexity and Asymptotic Notation

31 Consequences of P=NP Time Complexity and Asymptotic Notation If this was found to be true, the news would make world headlines! All passwords could be cracked in polynomial time (very, very fast) End of the World The End of the World??!!!! You would not be able to secure any computer or device on the internet, wireless or mobile networks..... Algorithms that currently take years might take minutes! Luckily in the last year a researcher (Vinay Deolalikar at HP Labs, Palo Altois) has claimed that they have proved P  NP (phew!)

This Weeks Laboratory Time Complexity and Asymptotic NotationSlide 32 D This laboratory contributes towards the D grade You will be implementing and comparing two algorithms Computing T(n) Computing O(n) Running some experiments

Next Lecture Time Complexity and Asymptotic NotationSlide 33 We will look at data structures and their applications