1. David Luebke 1 7/2/2015 Merge Sort Solving Recurrences The Master Theorem

2. David Luebke 2 7/2/2015 Review: Asymptotic Notation l Upper Bound Notation: n f(n) is O(g(n)) if there exist positive constants c and n 0 such that f(n)  c  g(n) for all n  n 0 n Formally, O(g(n)) = { f(n):  positive constants c and n 0 such that f(n)  c  g(n)  n  n 0 l Big O fact: n A polynomial of degree k is O(n k )

3. David Luebke 3 7/2/2015 Review: Asymptotic Notation l Asymptotic lower bound: n f(n) is  (g(n)) if  positive constants c and n 0 such that 0  c  g(n)  f(n)  n  n 0 l Asymptotic tight bound: n f(n) is  (g(n)) if  positive constants c 1, c 2, and n 0 such that c 1 g(n)  f(n)  c 2 g(n)  n  n 0 n f(n) =  (g(n)) if and only if f(n) = O(g(n)) AND f(n) =  (g(n))

4. David Luebke 4 7/2/2015 Other Asymptotic Notations l A function f(n) is o(g(n)) if  positive constants c and n 0 such that f(n) < c g(n)  n  n 0 l A function f(n) is  (g(n)) if  positive constants c and n 0 such that c g(n) < f(n)  n  n 0 l Intuitively, n o() is like < n O() is like  n  () is like > n  () is like  n  () is like =

5. David Luebke 5 7/2/2015 Merge Sort MergeSort(A, left, right) { if (left < right) { mid = floor((left + right) / 2); MergeSort(A, left, mid); MergeSort(A, mid+1, right); Merge(A, left, mid, right); } // Merge() takes two sorted subarrays of A and // merges them into a single sorted subarray of A // (how long should this take?)

6. David Luebke 6 7/2/2015 Merge Sort: Example Show MergeSort() running on the array A = {10, 5, 7, 6, 1, 4, 8, 3, 2, 9};

7. David Luebke 7 7/2/2015 Analysis of Merge Sort StatementEffort l So T(n) =  (1) when n = 1, and 2T(n/2) +  (n) when n > 1 l So what (more succinctly) is T(n)? MergeSort(A, left, right) {T(n) if (left < right) {  (1) mid = floor((left + right) / 2);  (1) MergeSort(A, left, mid); T(n/2) MergeSort(A, mid+1, right); T(n/2) Merge(A, left, mid, right);  (n) }

8. David Luebke 8 7/2/2015 Recurrences l The expression: is a recurrence. n Recurrence: an equation that describes a function in terms of its value on smaller functions

9. David Luebke 9 7/2/2015 Solving Recurrences l The substitution method (CLR 4.1) n A.k.a. the “making a good guess method” n Guess the form of the answer, then use induction to find the constants and show that solution works n Examples: u T(n) = 2T(n/2) +  (n)  T(n) =  (n lg n) u T(n) = 2T(  n/2  ) + n  ???

10. David Luebke 10 7/2/2015 Solving Recurrences l The substitution method (CLR 4.1) n A.k.a. the “making a good guess method” n Guess the form of the answer, then use induction to find the constants and show that solution works n Examples: u T(n) = 2T(n/2) +  (n)  T(n) =  (n lg n) u T(n) = 2T(  n/2  ) + n  T(n) =  (n lg n) u T(n) = 2T(  n/2  )+ 17) + n  ???

11. David Luebke 11 7/2/2015 Solving Recurrences l The substitution method (CLR 4.1) n A.k.a. the “making a good guess method” n Guess the form of the answer, then use induction to find the constants and show that solution works n Examples: u T(n) = 2T(n/2) +  (n)  T(n) =  (n lg n) u T(n) = 2T(  n/2  ) + n  T(n) =  (n lg n) u T(n) = 2T(  n/2  + 17) + n   (n lg n)

12. David Luebke 12 7/2/2015 l s(n) =n + s(n-1) =n + n-1 + s(n-2) =n + n-1 + n-2 + s(n-3) =n + n-1 + n-2 + n-3 + s(n-4) =…=… = n + n-1 + n-2 + n-3 + … + n-(k-1) + s(n-k)

13. David Luebke 13 7/2/2015 l s(n) =n + s(n-1) =n + n-1 + s(n-2) =n + n-1 + n-2 + s(n-3) =n + n-1 + n-2 + n-3 + s(n-4) =… = n + n-1 + n-2 + n-3 + … + n-(k-1) + s(n-k) =

14. David Luebke 14 7/2/2015 l So far for n >= k we have

15. David Luebke 15 7/2/2015 l So far for n >= k we have l What if k = n?

16. David Luebke 16 7/2/2015 l So far for n >= k we have l What if k = n?

17. David Luebke 17 7/2/2015 l So far for n >= k we have l What if k = n? l Thus in general

18. David Luebke 18 7/2/2015 l T(n) = 2T(n/2) + c 2(2T(n/2/2) + c) + c 2 2 T(n/2 2 ) + 2c + c 2 2 (2T(n/2 2 /2) + c) + 3c 2 3 T(n/2 3 ) + 4c + 3c 2 3 T(n/2 3 ) + 7c 2 3 (2T(n/2 3 /2) + c) + 7c 2 4 T(n/2 4 ) + 15c … 2 k T(n/2 k ) + (2 k - 1)c

19. David Luebke 19 7/2/2015 l So far for n > 2k we have n T(n) = 2 k T(n/2 k ) + (2 k - 1)c l What if k = lg n? n T(n) = 2 lg n T(n/2 lg n ) + (2 lg n - 1)c = n T(n/n) + (n - 1)c = n T(1) + (n-1)c = nc + (n-1)c = (2n - 1)c

20. David Luebke 20 7/2/2015 Sorting Revisited l So far we’ve talked about two algorithms to sort an array of numbers n What is the advantage of merge sort? n What is the advantage of insertion sort? l Next on the agenda: Heapsort n Combines advantages of both previous algorithms

21. David Luebke 21 7/2/2015 l A heap can be seen as a complete binary tree: n What makes a binary tree complete? n Is the example above complete? Heaps 16 1410 8793 241

22. David Luebke 22 7/2/2015 l A heap can be seen as a complete binary tree: n The book calls them “nearly complete” binary trees; can think of unfilled slots as null pointers Heaps 16 1410 8793 241 1 1 1 11

23. David Luebke 23 7/2/2015 Heaps l In practice, heaps are usually implemented as arrays: 16 1410 8793 241 1614108793241 A ==

24. David Luebke 24 7/2/2015 Heaps l To represent a complete binary tree as an array: n The root node is A[1] n Node i is A[i] n The parent of node i is A[i/2] (note: integer divide) n The left child of node i is A[2i] n The right child of node i is A[2i + 1] 16 1410 8793 241 1614108793241 A ==

25. David Luebke 25 7/2/2015 Referencing Heap Elements l So… Parent(i) { return  i/2  ; } Left(i) { return 2*i; } right(i) { return 2*i + 1; } l An aside: How would you implement this most efficiently? l Another aside: Really?

26. David Luebke 26 7/2/2015 The Heap Property l Heaps also satisfy the heap property: A[Parent(i)]  A[i]for all nodes i > 1 n In other words, the value of a node is at most the value of its parent n Where is the largest element in a heap stored? l Definitions: n The height of a node in the tree = the number of edges on the longest downward path to a leaf n The height of a tree = the height of its root

27. David Luebke 27 7/2/2015 Heap Height l What is the height of an n-element heap? Why? l This is nice: basic heap operations take at most time proportional to the height of the heap

28. David Luebke 28 7/2/2015 Heap Operations: Heapify() Heapify() : maintain the heap property n Given: a node i in the heap with children l and r n Given: two subtrees rooted at l and r, assumed to be heaps n Problem: The subtree rooted at i may violate the heap property (How?) n Action: let the value of the parent node “float down” so subtree at i satisfies the heap property u What do you suppose will be the basic operation between i, l, and r?

29. David Luebke 29 7/2/2015 Heap Operations: Heapify() Heapify(A, i) { l = Left(i); r = Right(i); if (l A[i]) largest = l; else largest = i; if (r A[largest]) largest = r; if (largest != i) Swap(A, i, largest); Heapify(A, largest); }

30. David Luebke 30 7/2/2015 Heapify() Example 16 410 14793 281 1641014793281 A =

31. David Luebke 31 7/2/2015 Heapify() Example 16 410 14793 281 161014793281 A = 4

32. David Luebke 32 7/2/2015 Heapify() Example 16 410 14793 281 1610793281 A = 414

33. David Luebke 33 7/2/2015 Heapify() Example 16 1410 4793 281 1614104793281 A =

34. David Luebke 34 7/2/2015 Heapify() Example 16 1410 4793 281 161410793281 A = 4

35. David Luebke 35 7/2/2015 Heapify() Example 16 1410 4793 281 16141079321 A = 48

36. David Luebke 36 7/2/2015 Heapify() Example 16 1410 8793 241 1614108793241 A =

37. David Luebke 37 7/2/2015 Heapify() Example 16 1410 8793 241 161410879321 A = 4

38. David Luebke 38 7/2/2015 Heapify() Example 16 1410 8793 241 1614108793241 A =

39. David Luebke 39 7/2/2015 Analyzing Heapify(): Informal Aside from the recursive call, what is the running time of Heapify() ? How many times can Heapify() recursively call itself? What is the worst-case running time of Heapify() on a heap of size n?

40. David Luebke 40 7/2/2015 Analyzing Heapify(): Formal l Fixing up relationships between i, l, and r takes  (1) time l If the heap at i has n elements, how many elements can the subtrees at l or r have? n Draw it l Answer: 2n/3 (worst case: bottom row 1/2 full) So time taken by Heapify() is given by T(n)  T(2n/3) +  (1)

41. David Luebke 41 7/2/2015 Analyzing Heapify(): Formal l So we have T(n)  T(2n/3) +  (1) l By case 2 of the Master Theorem, T(n) = O(lg n) Thus, Heapify() takes linear time

42. David Luebke 42 7/2/2015 Heap Operations: BuildHeap() We can build a heap in a bottom-up manner by running Heapify() on successive subarrays n Fact: for array of length n, all elements in range A[  n/2  + 1.. n] are heaps (Why?) n So:  Walk backwards through the array from n/2 to 1, calling Heapify() on each node. u Order of processing guarantees that the children of node i are heaps when i is processed

43. David Luebke 43 7/2/2015 BuildHeap() // given an unsorted array A, make A a heap BuildHeap(A) { heap_size(A) = length(A); for (i =  length[A]/2  downto 1) Heapify(A, i); }

44. David Luebke 44 7/2/2015 BuildHeap() Example l Work through example A = {4, 1, 3, 2, 16, 9, 10, 14, 8, 7} 4 13 216910 1487

45. David Luebke 45 7/2/2015 Analyzing BuildHeap() Each call to Heapify() takes O(lg n) time l There are O(n) such calls (specifically,  n/2  ) l Thus the running time is O(n lg n) n Is this a correct asymptotic upper bound? n Is this an asymptotically tight bound? l A tighter bound is O(n) n How can this be? Is there a flaw in the above reasoning?

46. David Luebke 46 7/2/2015 Analyzing BuildHeap(): Tight To Heapify() a subtree takes O(h) time where h is the height of the subtree n h = O(lg m), m = # nodes in subtree n The height of most subtrees is small l Fact: an n-element heap has at most  n/2 h+1  nodes of height h CLR 7.3 uses this fact to prove that BuildHeap() takes O(n) time

47. David Luebke 47 7/2/2015 Heapsort Given BuildHeap(), an in-place sorting algorithm is easily constructed: n Maximum element is at A[1] n Discard by swapping with element at A[n] u Decrement heap_size[A] u A[n] now contains correct value Restore heap property at A[1] by calling Heapify() n Repeat, always swapping A[1] for A[heap_size(A)]

48. David Luebke 48 7/2/2015 Heapsort Heapsort(A) { BuildHeap(A); for (i = length(A) downto 2) { Swap(A[1], A[i]); heap_size(A) -= 1; Heapify(A, 1); }

49. David Luebke 49 7/2/2015 Analyzing Heapsort The call to BuildHeap() takes O(n) time Each of the n - 1 calls to Heapify() takes O(lg n) time Thus the total time taken by HeapSort() = O(n) + (n - 1) O(lg n) = O(n) + O(n lg n) = O(n lg n)

50. David Luebke 50 7/2/2015 Priority Queues l Heapsort is a nice algorithm, but in practice Quicksort (coming up) usually wins l But the heap data structure is incredibly useful for implementing priority queues n A data structure for maintaining a set S of elements, each with an associated value or key Supports the operations Insert(), Maximum(), and ExtractMax() n What might a priority queue be useful for?

51. David Luebke 51 7/2/2015 Priority Queue Operations l Insert(S, x) inserts the element x into set S l Maximum(S) returns the element of S with the maximum key l ExtractMax(S) removes and returns the element of S with the maximum key l How could we implement these operations using a heap?