2IL50 Data Structures Spring 2016 Lecture 2: Analysis of Algorithms

Please read … your … the slides … the course web- page Homework assignments & OnCourse  Available via OnCourse on Monday (first assignment also on webpage) You will be automatically enrolled this week. Homework assignments & LaTeX  Assignments have to be typeset in English. No handwritten solutions will be accepted. You should seriously consider to use LaTex to typeset your assignments, here is an example Latex file.  Due Sundays at 23:55 as.pdf using the OnCourse submission form. ➨ Typeset and.pdf mandatory, LaTeX not

Please read … your … the slides … the course web- page Tutorial locations Subject line: [2IL50] Tutorial Group 3 in AUD 1 tomorrow, Wedneday Feb 03, hours 1+2  Last minute changes of tutorial locations might happen  Always to AUD 1

Analysis of algorithms the formal way …

Analysis of algorithms  Can we say something about the running time of an algorithm without implementing and testing it? InsertionSort(A) 1. initialize: sort A[1] 2. for j = 2 to A.length 3. do key = A[j] 4. i = j while i > 0 and A[i] > key 6. do A[i+1] = A[i] 7. i = i A[i +1] = key

Analysis of algorithms  Analyze the running time as a function of n (# of input elements) best case average case worst case elementary operations add, subtract, multiply, divide, load, store, copy, conditional and unconditional branch, return … An algorithm has worst case running time T(n) if for any input of size n the maximal number of elementary operations executed is T(n).

Linear Search Input: increasing sequence of n numbers A = ‹a 1, a 2, …, a n › and value v Output: an index i such that A[i] = v or NIL if v not in A LinearSearch(A, v) 1. for i = 1 to n 2. do if A[i] = v 3. then return i 4. return NIL Running time best case: 1 average case: n/2 (if successful) worst case: n v = 7

Binary Search Input: increasing sequence of n numbers A = ‹a 1, a 2, …, a n › and value v Output: an index i such that A[i] = v or NIL if v not in A BinarySearch(A, v) 1. x = 0 2. y = n while x + 1 < y 4. do h = (x + y)/2 5. do if A[h] ≤ v then x = h 6. else y = h 7. if A[x] = v then return x else return NIL Running time best case: log n average case: log n worst case: log n v = 7

Analysis of algorithms: example InsertionSort: 15 n 2 + 7n – 2 MergeSort: 300 n lg n + 50 n n=10 n=100 n= x x 10 6 InsertionSort 6 x faster InsertionSort 1.35 x faster MergeSort 5 x faster n = 1,000,000 InsertionSort 1.5 x MergeSort 6 x x faster ! The rate of growth of the running time as a function of the input is essential!

Θ-notation Let g(n) : N ↦ N be a function. Then we have Θ(g(n)) = { f(n) : there exist positive constants c 1, c 2, and n 0 such that 0 ≤ c 1 g(n) ≤ f(n) ≤ c 2 g(n) for all n ≥ n 0 } “Θ(g(n)) is the set of functions that grow as fast as g(n)”

Θ-notation Let g(n) : N ↦ N be a function. Then we have Θ(g(n)) = { f(n) : there exist positive constants c 1, c 2, and n 0 such that 0 ≤ c 1 g(n) ≤ f(n) ≤ c 2 g(n) for all n ≥ n 0 } n n0n0 c 1 g(n) c 2 g(n) f(n) 0 Notation: f(n) = Θ(g(n))

Θ-notation Let g(n) : N ↦ N be a function. Then we have Θ(g(n)) = { f(n) : there exist positive constants c 1, c 2, and n 0 such that 0 ≤ c 1 g(n) ≤ f(n) ≤ c 2 g(n) for all n ≥ n 0 } Claim:19n n 2 - 3n = Θ(n 3 ) Proof:Choose c 1 = 19, c 2 = 36 and n 0 = 1. Then we have for all n ≥ n 0 : 0 ≤ c 1 n 3 = 19n 3 (trivial) ≤ 19n n 2 - 3n (since 17n 2 > 3n for n ≥ 1) ≤ 19n n 3 (since 17n 2 ≤ 17n 3 for n ≥1) = c 2 n 3 ■

Θ-notation Let g(n) : N ↦ N be a function. Then we have Θ(g(n)) = { f(n) : there exist positive constants c 1, c 2, and n 0 such that 0 ≤ c 1 g(n) ≤ f(n) ≤ c 2 g(n) for all n ≥ n 0 } Claim:19n n 2 - 3n ≠ Θ(n 2 ) Proof:Assume that there are positive constants c 1, c 2, and n 0 such that for all n ≥ n 0 0 ≤ c 1 n 2 ≤ 19n n 2 - 3n ≤ c 2 n 2 Since 19n n 2 - 3n ≤ c 2 n 2 implies 19n 3 ≤ c 2 n 2 + 3n – 17n 2 ≤ c 2 n 2 (3n – 17n 2 ≤ 0) we would have for all n ≥ n 0 19n ≤ c 2.

O-notation Let g(n) : N ↦ N be a function. Then we have O(g(n)) = { f(n) : there exist positive constants c and n 0 such that 0 ≤ f(n) ≤ cg(n) for all n ≥ n 0 } “O(g(n)) is the set of functions that grow at most as fast as g(n)”

O-notation Let g(n) : N ↦ N be a function. Then we have O(g(n)) = { f(n) : there exist positive constants c and n 0 such that 0 ≤ f(n) ≤ cg(n) for all n ≥ n 0 } n n0n0 cg(n) f(n) 0 Notation: f(n) = O(g(n))

Ω-notation Let g(n) : N ↦ N be a function. Then we have Ω(g(n)) = { f(n) : there exist positive constants c and n 0 such that 0 ≤ cg(n) ≤ f(n) for all n ≥ n 0 } “Ω(g(n)) is the set of functions that grow at least as fast as g(n)”

Ω-notation Let g(n) : N ↦ N be a function. Then we have Ω(g(n)) = { f(n) : there exist positive constants c and n 0 such that 0 ≤ cg(n) ≤ f(n) for all n ≥ n 0 } n n0n0 cg(n) f(n) 0 Notation: f(n) = Ω(g(n))

Asymptotic notation Θ(…) is an asymptotically tight bound O(…) is an asymptotic upper bound Ω(…) is an asymptotic lower bound  other asymptotic notation o(…) → “grows strictly slower than” ω(…) → “grows strictly faster than” “ asymptotically equal” “asymptotically smaller or equal” “asymptotically greater or equal”

More notation … f(n) = n 3 + Θ(n 2 )means f(n) = means O(1) or Θ(1) means 2n 2 + O(n) = Θ(n 2 )means there is a function g(n) such that f(n) = n 3 + g(n) and g(n) = Θ(n 2 ) there is one function g(i) such that f(n) = and g(i) = O(i) a constant for each function g(n) with g(n)=O(n) we have 2n 2 + g(n) = Θ(n 2 )

Quiz 1.O(1) + O(1) = O(1) 2.O(1) + … + O(1) = O(1) 3. = 4.O(n 2 ) ⊆ O(n 3 ) 5.O(n 3 ) ⊆ O(n 2 ) 6.Θ(n 2 ) ⊆ O(n 3 ) 7.An algorithm with worst case running time O(n log n) is always slower than an algorithm with worst case running time O(n) if n is sufficiently large. true false true false true false

Quiz 8.n log 2 n = Θ(n log n) 9.n log 2 n = Ω(n log n) 10. n log 2 n = O(n 4/3 ) 11. O(2 n ) ⊆ O(3 n ) 12. O(2 n ) ⊆ Θ(3 n ) false true false

Analysis of algorithms

Analysis of InsertionSort InsertionSort(A) 1. initialize: sort A[1] 2. for j = 2 to A.length 3. do key = A[j] 4. i = j while i > 0 and A[i] > key 6. do A[i+1] = A[i] 7. i = i A[i +1] = key  Get as tight a bound as possible on the worst case running time. ➨ lower and upper bound for worst case running time Upper bound: Analyze worst case number of elementary operations Lower bound: Give “bad” input example

Analysis of InsertionSort InsertionSort(A) 1. initialize: sort A[1] 2. for j = 2 to A.length 3. do key = A[j] 4. i = j while i > 0 and A[i] > key 6. do A[i+1] = A[i] 7. i = i A[i +1] = key Upper bound: Let T(n) be the worst case running time of InsertionSort on an array of length n. We have T(n) = Lower bound: O(1) worst case: (j-1) ∙ O(1) O(1) + { O(1) + (j-1)∙O(1) + O(1) } = O(j) = O(n 2 ) Array sorted in de-creasing order ➨ Ω(n 2 ) The worst case running time of InsertionSort is Θ(n 2 ).

MergeSort(A) // divide-and-conquer algorithm that sorts array A[1..n] 1. if A.length = 1 2. then skip 3. else 4. n = A.length ; n 1 = floor(n/2); n 2 = ceil(n/2); 5. copy A[1.. n 1 ] to auxiliary array A 1 [1.. n 1 ] 6. copy A[n n] to auxiliary array A 2 [1.. n 2 ] 7. MergeSort(A 1 ); MergeSort(A 2 ) 8. Merge(A, A 1, A 2 ) Analysis of MergeSort O(1) O(n) ?? T( n/2 ) + T( n/2 ) MergeSort is a recursive algorithm ➨ running time analysis leads to recursion

Analysis of MergeSort  Let T(n) be the worst case running time of MergeSort on an array of length n. We have O(1) if n = 1 T(n) = T( n/2 ) + T( n/2 ) + Θ(n) if n > 1 frequently omitted since it (nearly) always holds often written as 2T(n/2)

Solving recurrences

 Easiest: Master theorem caveat: not always applicable  Alternatively: Guess the solution and use the substitution method to prove that your guess it is correct.  How to guess: 1. expand the recursion 2. draw a recursion tree

Let a and b be constants, let f(n) be a function, and let T(n) be defined on the nonnegative integers by the recurrence T(n) = aT(n/b) + f(n) Then we have: 1.If f(n) = O(n log a – ε ) for some constant ε > 0, then T(n) = Θ(n log a ). 2.If f(n) = Θ(n log a ), then T(n) = Θ(n log a log n) 3.If f(n) = Ω(n log a + ε ) for some constant ε > 0, and if af(n/b) ≤ cf(n) for some constant c < 1 and all sufficiently large n, then T(n) = Θ(f(n)) The master theorem b b b b can be rounded up or down note: log b a - ε b

The master theorem: Example T(n) = 4T(n/2) + Θ(n 3 )  Master theorem with a = 4, b = 2, and f(n) = n 3 log b a = log 2 4 = 2 ➨ n 3 = f(n) = Ω(n log a + ε ) = Ω(n 2 + ε ) with, for example, ε = 1  Case 3 of the master theorem gives T(n) = Θ(n 3 ), if the regularity condition holds. choose c = ½ and n 0 = 1 ➨ af(n/b) = 4(n/2) 3 = n 3 /2 ≤ cf(n) for n ≥ n 0 ➨ T(n) = Θ(n 3 ) b

The substitution method  The Master theorem does not always apply  In those cases, use the substitution method: 1.Guess the form of the solution. 2.Use induction to find the constants and show that the solution works Use expansion or a recursion-tree to guess a good solution.

Recursion-trees T(n) = 2T(n/2) + n n n/2 n/4 n/4 n/2 i n/2 i … n/2 i Θ(1) Θ(1) … Θ(1)

Recursion-trees T(n) = 2T(n/2) + n n 2 ∙ (n/2) = n 4 ∙ (n/4) = n 2 i ∙ (n/2 i ) = n n ∙ Θ(1) = Θ(n) log n + Θ(n log n) n n/2 n/4 n/4 n/2 i n/2 i … n/2 i Θ(1) Θ(1) … Θ(1)

Recursion-trees T(n) = 2T(n/2) + n 2 n2n2 (n/2) 2 (n/4) 2 (n/4) 2 (n/2 i ) 2 (n/2 i ) 2 … (n/2 i ) 2 Θ(1) Θ(1) … Θ(1)

Recursion-trees T(n) = 2T(n/2) + n 2 n2n2 (n/2) 2 (n/4) 2 (n/4) 2 (n/2 i ) 2 (n/2 i ) 2 … (n/2 i ) 2 Θ(1) Θ(1) … Θ(1) n2n2 2 ∙ (n/2) 2 = n 2 /2 4 ∙ (n/4) 2 = n 2 /4 2 i ∙ (n/2 i ) 2 = n 2 /2 i n ∙ Θ(1) = Θ(n) + Θ(n 2 )

Recursion-trees T(n) = 4T(n/2) + n n n/2 n/2 … n/4 n/4 n/4 n/4 … Θ(1) Θ(1) … Θ(1)

Recursion-trees T(n) = 4T(n/2) + n n n/2 n/2 … n/4 n/4 n/4 n/4 … Θ(1) Θ(1) … Θ(1) n 4 ∙ (n/2) = 2n 16 ∙ (n/4) = 4n n 2 ∙ Θ(1) = Θ(n 2 ) + Θ(n 2 )

The substitution method Claim: T(n) = O(n log n) Proof: by induction on n to show: there are constants c and n 0 such that T(n) ≤ c n log n for all n ≥ n 0 n = 1 ➨ T(1) = 2 ≤ c 1 log 1  n = n 0 = 2 is a base case  Need more base cases? Base cases: n = 2: T(2) = 2T(1) + 2 = 2∙2 + 2 = 6 = c 2 log 2 for c = 3 n = 3: T(3) = 2T(1) + 3 = 2∙2 + 3 = 7 ≤ c 3 log 3 T(n) = 2 if n = 1 2T( n/2 ) + nif n > 1 3/2 = 1, 4/2 = 2 ➨ 3 must also be base case ➨ n 0 = 2

The substitution method Claim: T(n) = O(n log n) Proof: by induction on n to show: there are constants c and n 0 such that T(n) ≤ c n log n for all n ≥ n 0  choose c = 3 and n 0 = 2 Inductive step: n > 3 T(n) = 2T( n/2 ) + n ≤ 2 c n/2 log n/2 + n (ind. hyp.) ≤ c n ((log n) - 1) + n ≤ c n log n ■ T(n) = 2 if n = 1 2T( n/2 ) + nif n > 1

The substitution method Claim: T(n) = O(n) Proof: by induction on n Base case: n = n 0 T(2) = 2T(1) + 2 = 2c + 2 = O(2) Inductive step: n > n 0 T(n) = 2T( n/2 ) + n = 2O( n/2 ) + n (ind. hyp.) = O(n) ■ T(n) = Θ(1) if n = 1 2T( n/2 ) + nif n > 1 Never use O, Θ, or Ω in a proof by induction!

Tips  Analysis of recursive algorithms: find the recursion and solve with master theorem if possible  Analysis of loops: summations  Some standard recurrences and sums: T(n) = 2T(n/2) + Θ(n) ➨ ½ n(n+1) = Θ(n 2 ) Θ(n 3 ) T(n) = Θ(n log n)

Announcements  If you are not in a group yet, talk to me immediately!  Office hours on both Thursday and  Monday evening lecture for senior students