1. Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu

2. CS 477/6772 Counting Sort Assumption: –The elements to be sorted are integers in the range 0 to k Idea: –Determine for each input element x, the number of elements smaller than x –Place element x into its correct position in the output array 30320352 12345678 A 77422 12345 C 8 0 53332200 12345678 B

3. CS 477/6773 Radix Sort Considers keys as numbers in a base-R number –A d -digit number will occupy a field of d columns Sorting looks at one column at a time –For a d digit number, sort the least significant digit first –Continue sorting on the next least significant digit, until all digits have been sorted –Requires only d passes through the list

4. CS 477/6774 Bucket Sort Assumption: –the input is generated by a random process that distributes elements uniformly over [0, 1) Idea: –Divide [0, 1) into n equal-sized buckets –Distribute the n input values into the buckets –Sort each bucket –Go through the buckets in order, listing elements in each one Input: A[1.. n], where 0 ≤ A[i] < 1 for all i Output: elements a i sorted Auxiliary array: B[0.. n - 1] of linked lists, each list initially empty

5. CS 477/6775 BUCKET-SORT Alg.: BUCKET-SORT(A, n) for i ← 1 to n do insert A[i] into list B[  nA[i]  ] for i ← 0 to n - 1 do sort list B[i] with insertion sort concatenate lists B[0], B[1],..., B[n -1] together in order return the concatenated lists

6. CS 477/6776 Example - Bucket Sort.78.17.39.26.72.94.21.12.23.68 0 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 10.21.12 /.72 /.23 /.78.94 /.68 /.39 /.26.17 / / / /

7. CS 477/6777 Example - Bucket Sort 0 1 2 3 4 5 6 7 8 9.23.17 /.78 /.26 /.72.94 /.68 /.39 /.21.12 / / / /.17.12.23.26.21.39.68.78.72.94 / Concatenate the lists from 0 to n – 1 together, in order

8. CS 477/6778 Correctness of Bucket Sort Consider two elements A[i], A[ j] Assume without loss of generality that A[i] ≤ A[j] Then  nA[i]  ≤  nA[j]  –A[i] belongs to the same group as A[j] or to a group with a lower index than that of A[j] If A[i], A[j] belong to the same bucket: –insertion sort puts them in the proper order If A[i], A[j] are put in different buckets: –concatenation of the lists puts them in the proper order

9. CS 477/6779 Analysis of Bucket Sort Alg.: BUCKET-SORT(A, n) for i ← 1 to n do insert A[i] into list B[  nA[i]  ] for i ← 0 to n - 1 do sort list B[i] with insertion sort concatenate lists B[0], B[1],..., B[n -1] together in order return the concatenated lists O(n)  (n) O(n)  (n)

10. CS 477/67710 Conclusion Any comparison sort will take at least nlgn to sort an array of n numbers We can achieve a better running time for sorting if we can make certain assumptions on the input data: –Counting sort: each of the n input elements is an integer in the range 0 to k –Radix sort: the elements in the input are integers represented with d digits –Bucket sort: the numbers in the input are uniformly distributed over the interval [0, 1)

11. CS 477/67711 A Job Scheduling Application Job scheduling –The key is the priority of the jobs in the queue –The job with the highest priority needs to be executed next Operations –Insert, remove maximum Data structures –Priority queues –Ordered array/list, unordered array/list

12. CS 477/67712 Example

13. CS 477/67713 PQ Implementations & Cost Worst-case asymptotic costs for a PQ with N items InsertRemove max ordered array ordered list unordered array unordered list N N N N 1 1 1 1 Can we implement both operations efficiently?

14. CS 477/67714 Background on Trees Def: Binary tree = structure composed of a finite set of nodes that either: –Contains no nodes, or –Is composed of three disjoint sets of nodes: a root node, a left subtree and a right subtree 2 148 1 16 4 3 910 root Right subtree Left subtree

15. CS 477/67715 Special Types of Trees Def: Full binary tree = a binary tree in which each node is either a leaf or has degree exactly 2. Def: Complete binary tree = a binary tree in which all leaves have the same depth and all internal nodes have degree 2. Full binary tree 2 148 1 16 7 4 3 910 12 Complete binary tree 2 1 16 4 3 910

16. CS 477/67716 The Heap Data Structure Def: A heap is a nearly complete binary tree with the following two properties: –Structural property: all levels are full, except possibly the last one, which is filled from left to right –Order (heap) property: for any node x Parent(x) ≥ x Heap 5 7 8 4 It doesn’t matter that 4 in level 1 is smaller than 5 in level 2 2

17. CS 477/67717 Definitions Height of a node = the number of edges on a longest simple path from the node down to a leaf Depth of a node = the length of a path from the root to the node Height of tree = height of root node =  lgn , for a heap of n elements 2 148 1 16 4 3 910 Height of root = 3 Height of (2)= 1 Depth of (10)= 2

18. CS 477/67718 Array Representation of Heaps A heap can be stored as an array A. –Root of tree is A[1] –Left child of A[i] = A[2i] –Right child of A[i] = A[2i + 1] –Parent of A[i] = A[  i/2  ] –Heapsize[A] ≤ length[A] The elements in the subarray A[(  n/2  +1).. n] are leaves The root is the maximum element of the heap A heap is a binary tree that is filled in order

19. CS 477/67719 Heap Types Max-heaps (largest element at root), have the max-heap property: –for all nodes i, excluding the root: A[PARENT(i)] ≥ A[i] Min-heaps (smallest element at root), have the min-heap property: –for all nodes i, excluding the root: A[PARENT(i)] ≤ A[i]

20. CS 477/67720 Operations on Heaps Maintain the max-heap property –MAX-HEAPIFY Create a max-heap from an unordered array –BUILD-MAX-HEAP Sort an array in place –HEAPSORT Priority queue operations

21. CS 477/67721 Operations on Priority Queues Max-priority queues support the following operations: –INSERT (S, x) : inserts element x into set S –EXTRACT-MAX (S) : removes and returns element of S with largest key –MAXIMUM (S) : returns element of S with largest key –INCREASE-KEY (S, x, k) : increases value of element x ’s key to k (Assume k ≥ x ’s current key value)

22. CS 477/67722 Maintaining the Heap Property Suppose a node is smaller than a child –Left and Right subtrees of i are max-heaps Invariant: –the heap condition is violated only at that node To eliminate the violation: –Exchange with larger child –Move down the tree –Continue until node is not smaller than children

23. CS 477/67723 Maintaining the Heap Property Assumptions: –Left and Right subtrees of i are max-heaps –A[i] may be smaller than its children Alg: MAX-HEAPIFY( A, i, n ) 1.l ← LEFT( i ) 2.r ← RIGHT( i ) 3.if l ≤ n and A[l] > A[i] 4. then largest ← l 5. else largest ← i 6.if r ≤ n and A[r] > A[largest] 7. then largest ← r 8.if largest  i 9. then exchange A[i] ↔ A[largest] 10. MAX-HEAPIFY( A, largest, n )

24. CS 477/67724 Example MAX-HEAPIFY(A, 2, 10) A[2] violates the heap property A[2]  A[4] A[4] violates the heap property A[4]  A[9] Heap property restored

25. CS 477/67725 MAX-HEAPIFY Running Time Intuitively: –A heap is an almost complete binary tree  must process O(lgn) levels, with constant work at each level Running time of MAX-HEAPIFY is O(lgn) Can be written in terms of the height of the heap, as being O(h) –Since the height of the heap is  lgn 

26. CS 477/67726 Building a Heap Alg: BUILD-MAX-HEAP (A) 1.n = length[A] 2. for i ←  n/2  downto 1 3. do MAX-HEAPIFY (A, i, n) Convert an array A[1 … n] into a max-heap ( n = length[A] ) The elements in the subarray A[(  n/2  +1).. n] are leaves Apply MAX-HEAPIFY on elements between 1 and  n/2  2 148 1 16 7 4 3 910 1 23 4567 89 4132169101487 A:

27. CS 477/67727 Example: A 4132169101487 2 8 1 16 7 4 3 910 1 23 4567 89 14 28 1 16 7 4 10 93 1 23 4567 89 2 148 1 16 7 4 3 910 1 23 4567 89 14 28 1 16 7 4 3 910 1 23 4567 89 14 28 16 7 1 4 10 93 1 23 4567 89 8 24 14 7 1 16 10 93 1 23 4567 89 i = 5 i = 4 i = 3 i = 2 i = 1

28. CS 477/67728 Correctness of BUILD-MAX-HEAP Loop invariant: –At the start of each iteration of the for loop, each node i + 1, i + 2,…, n is the root of a max-heap Initialization: –i =  n/2  : Nodes  n/2  +1,  n/2  +2, …, n are leaves  they are the root of trivial max-heaps 2 148 1 16 7 4 3 910 1 23 4567 89

29. CS 477/67729 Correctness of BUILD-MAX-HEAP Maintenance: –MAX-HEAPIFY makes node i a max- heap root and preserves the property that nodes i + 1, i + 2, …, n are roots of max-heaps –Decrementing i in the for loop reestablishes the loop invariant Termination: –i = 0  each node 1, 2, …, n is the root of a max-heap (by the loop invariant) 2 148 1 16 7 4 3 910 1 23 4567 89

30. CS 477/67730 Running Time of BUILD MAX HEAP  Running time: O(nlgn) This is not an asymptotically tight upper bound Alg: BUILD-MAX-HEAP (A) 1.n = length[A] 2. for i ←  n/2  downto 1 3. do MAX-HEAPIFY (A, i, n) O(lgn) O(n)

31. CS 477/67731 Running Time of BUILD MAX HEAP HEAPIFY takes O(h)  the cost of HEAPIFY on a node i is proportional to the height of the node i in the tree HeightLevel h 0 = 3 (  lgn  ) h 1 = 2 h 2 = 1 h 3 = 0 i = 0 i = 1 i = 2 i = 3 (  lgn  ) No. of nodes 2020 2121 2 2323 h i = h – i height of the heap rooted at level i n i = 2 i number of nodes at level i

32. CS 477/67732 Running Time of BUILD MAX HEAP Cost of HEAPIFY at level i  number of nodes at that level Replace the values of n i and h i computed before Multiply by 2 h both at the nominator and denominator and write 2 i as Change variables: k = h - i The sum above is smaller than the sum of all elements to  and h = lgn The sum above is smaller than 2 Running time of BUILD-MAX-HEAP: T(n) = O(n)

33. CS 477/67733 Readings Chapter 6 Chapter 8