CPSC 311 Section 502 Analysis of Algorithm Fall 2002 Department of Computer Science Texas A&M University

CPSC 311 O. Burchan Bayazit – Fall 02 Announcements There will be a quiz on next Tuesday (Sep 17) A new homework will be put to the web site today ( due date next Thursday (Sep 19)) Grades for the previous quiz is available

CPSC 311 O. Burchan Bayazit – Fall 02 Solving Recurrences: Master Method The master method provides a 'cookbook' method for solving recurrences of a certain form. Master Theorem: Let a  1 and b > 1 be constants, let f(n) be a function, and let T(n) be defined on nonnegative integers as: T(n) = aT(n/b) + f(n) Then, T(n) can be bounded asymptotically as follows: 1.T(n) =  (n log b a )if f(n) = O(n log b a-  ) for some constant  > 0 2.T(n) =  (n log b a logn) if f(n) =  (n log b a ) 3.T(n) =  ( f(n))if f(n) =  (n log b a+  ) for some constant  > 0 and if af(n/b)  cf(n) for some constant c < 1 and all sufficiently large n.

CPSC 311 O. Burchan Bayazit – Fall 02 Case 3 T(n) =  ( f(n)), if f(n) =  (n log b a+  ) for some constant  > 0 and if af(n/b)  cf(n) for some constant c < 1 and all sufficiently large n. T(n) = aT(n/b) + f(n) T(n) =3T(n/4)+nlgn a=3,b=4,f(n)=nlgn { PART I: Is f(n) =  (n log b a+  ) for some  >0? n log b a+  = n log 4 3+  =n  f(n)=nlgn can we find  > 0 such that nlgn=  (n  ) If we choose  = the lower bound becomes  (n ) And nlgn has a lower bound n i.e., f(nlgn)=  (n ) So the first part holds.

CPSC 311 O. Burchan Bayazit – Fall 02 Case 3 T(n) =  ( f(n)), if f(n) =  (n log b a+  ) for some constant  > 0 and if af(n/b)  cf(n) for some constant c < 1 and all sufficiently large n. T(n) = aT(n/b) + f(n) T(n) =3T(n/4)+nlgn a=3,b=4,f(n)=nlgn { Part II: is there a constant <1 that satisfies af(n/b)  cf(n) ? af(n/b) = 3f(n/4) So we should find a c such 3f(n/4) 1 3f(n/4)= 3(n/4)lg(n/4) and f(n)=nlgn (3/4)n lg(n)-(3/4)lg4 <=c.nlgn If we select c=3/4 this inequality holds so T(n)=  (nlgn)

CPSC 311 O. Burchan Bayazit – Fall 02 Sorting Algorithms Ch Slightly modified definition of the sorting problem: input: A collection of n data items where each data item has a key drawn from a linearly ordered set (e.g., ints, chars) output: A permutation (reordering) of the input sequence such that a' 1  a' 2 ...  a' n In practice, one usually sorts 'records' according to their key (the non-key data is called satellite data.) If the records are large, we may sort an array of pointers.

CPSC 311 O. Burchan Bayazit – Fall 02 Sorting Algorithms A sorting algorithm is in place if only a constant number of elements of the input array are ever stored outside the array. A sorting algorithm is comparison based if the only operation we can perform on keys is to compare two keys.

CPSC 311 O. Burchan Bayazit – Fall 02 Merge Sort A Divide-and-Conquer algorithm Divide: the n-element array to be sorted into two n/2-element subarrays Conquer: Sort the subarrays recursively using merge sort Combine: Merge the two sorted subarrays to find sorted array

CPSC 311 O. Burchan Bayazit – Fall 02 Merge Sort Unsorted original array

CPSC 311 O. Burchan Bayazit – Fall 02 Merge Sort Divide the problem to two subproblems Call merge sort on subarrays

CPSC 311 O. Burchan Bayazit – Fall 02 Merge Sort Further divide subarrays

CPSC 311 O. Burchan Bayazit – Fall 02 Merge Sort Until the smallest array is reached

CPSC 311 O. Burchan Bayazit – Fall 02 Merge Sort Merge results of two subarrays

CPSC 311 O. Burchan Bayazit – Fall 02 Merge Sort Further merge

CPSC 311 O. Burchan Bayazit – Fall 02 Merge Sort Further merge

CPSC 311 O. Burchan Bayazit – Fall 02 Merge Sort Further merge

CPSC 311 O. Burchan Bayazit – Fall 02 Merge Sort Final merge

CPSC 311 O. Burchan Bayazit – Fall 02 Merge Sort Result

CPSC 311 O. Burchan Bayazit – Fall 02 Merge Sort - Algorithm Merge-Sort(A,start,end) 1. if start=end then { 2. return 3. } 4. half =  (start+end)/2  5. Merge-Sort (A,start,half) 6. Merge-Sort (A,half+1,end) 7. Merge (A,start,half,end)

CPSC 311 O. Burchan Bayazit – Fall 02 Merging

CPSC 311 O. Burchan Bayazit – Fall 02 Merging

CPSC 311 O. Burchan Bayazit – Fall 02 Merging

CPSC 311 O. Burchan Bayazit – Fall 02 Merging

CPSC 311 O. Burchan Bayazit – Fall 02 Merging

CPSC 311 O. Burchan Bayazit – Fall 02 Merging

CPSC 311 O. Burchan Bayazit – Fall 02 Merging

CPSC 311 O. Burchan Bayazit – Fall 02 Merging

CPSC 311 O. Burchan Bayazit – Fall 02 Merging

CPSC 311 O. Burchan Bayazit – Fall 02 Merge Algorithm Merge(A,start,half,end) 1. n 1 = half-start+1; n 2  end-half; 2. for i= 1 to n 1 { 3. Left[i]= A[start+i-1] // copy left subarray 4. } 5. for i= 1 to n 2 { 6. R[i]  A[half+i] // copy right subarray 7. } 8. Left[n 1 +1] = Right[n 2 +1] =  9. i= j= for k = start to end { 11. if Left[i]  Right[j] then { 12. A[k]=Left[i] 13. i= i+1 } 14. else { A[k]= Right[j] 15. j= j+1 } 16. } Running Time =  (n)

CPSC 311 O. Burchan Bayazit – Fall 02 Merge Sort - Analysis Merge-Sort(A,start,end) 1. if start=end then { 2. return 3. } 4. half =  (start+end)/2  5. Merge-Sort (A,start,half) 6. Merge-Sort (A,half+1,end) 7. Merge (A,start,half,end)  (1) T (n/2)  (n) T (n)=2T(n/2)+  (n)  (nlgn) using master method

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort heap concept complete binary tree, except may be missing some rightmost leaves on the bottom level. each node contains a key values in the nodes satisfy the heap property.

CPSC 311 O. Burchan Bayazit – Fall 02 Heap vs. Array binary tree: an array implementation root is A[1] for element A[i] - left child is in position A[2i] - right child is in position A[2i + 1] - parent is in A[  i/2  ] Left child=2i i=2 Right child=2i+1 Parent =i/2

CPSC 311 O. Burchan Bayazit – Fall 02 Max-Heap Max-heap property: A[Parent(i)]  A[i] In the array representation of a max-heap, the root of the tree is in A[1], and given the index i of a node, Parent(i) LeftChild(i)RightChild(i) return (  i/2  ) return (2i) return (2i + 1) A index keys n = 11 height = 3 (# edges on longest leaf to root path)

CPSC 311 O. Burchan Bayazit – Fall 02 Min-Heap Min-heap property: A[Parent(i)]  A[i] Min-heaps are commonly used for priority queues in event-driven simulators. Parent(i) LeftChild(i)RightChild(i) return (  i/2  ) return (2i) return (2i + 1) A index keys n = 11 height = 3 (# edges on longest leaf to root path)

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Assume we have a max-heap

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works The largest number is the root (max-heap property)

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Take the root away

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Put the last one to the root

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Fix the max-heap

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Fix the max-heap

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Take the root away

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Put the last one to the root

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Fix the max-heap

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Take the root away

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Put the last one to the root

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Fix the max-heap

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Take the root away

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Put the last one to the top

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Take the top away

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Put the last one

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort Basic Functions HEAPIFY: maintain (fix) max-heap property BUILD-HEAP: Create a max-heap from unsorted array (requires HEAPIFY) HEAPSORT: sorts an array (requires BUILD-HEAP and HEAPIFY) How it works? Build a max-heap (the root is the largest number) Take the largest number away, put the last number in the heap to the root position and if max-heap property is not satisfied, fix it. Repeat until all the elements are processed.

CPSC 311 O. Burchan Bayazit – Fall 02 Heapify (fix max-heap) Assume one node i does not satisfy MAX-HEAP property but its children are. Heapify function corrects this by recursively fixing inconsistencies Not satisfy max-heap

CPSC 311 O. Burchan Bayazit – Fall 02 Assume one node i does not satisfy MAX-HEAP property but its children are. Heapify function corrects this by recursively fixing inconsistencies Switch with the largest child Heapify (fix max-heap)

CPSC 311 O. Burchan Bayazit – Fall 02 Assume one node i does not satisfy MAX-HEAP property but its children are. Heapify function corrects this by recursively fixing inconsistencies Not satisfy max-heap Heapify (fix max-heap)

CPSC 311 O. Burchan Bayazit – Fall 02 Assume one node i does not satisfy MAX-HEAP property but its children are. Heapify function corrects this by recursively fixing inconsistencies Switch with the largest child Heapify (fix max-heap)

CPSC 311 O. Burchan Bayazit – Fall 02 Heapify (fix max-heap) Find the largest child Switch the node i with its largest child Recursively HEAPIFY for the subtree where node i placed

CPSC 311 O. Burchan Bayazit – Fall 02 Heapify(A,i,n) /* A: Array, I: node to be fixed, n:heap-size */ 1. left  2i; right  2i + 1 /*indices of left & right children of A[i] */ 2. largest  i; 3. if left  n and A[left] > A[i] then { 4.largest  left 5.} 5. if right  n and A[right] > A[largest] then { 6.largest  right 7.} 7. if largest  i then 8. swap(A[i], A[largest]) 9. Heapify(A, largest,n) T (n)=T(2n/3)+  (1) O (lgn) using master method Heapify (fix max-heap)

CPSC 311 O. Burchan Bayazit – Fall 02 Build-Heap Given unsorted array Starting from largest non-leaf node down to the first node, call Heapify for each node

CPSC 311 O. Burchan Bayazit – Fall 02 Build-Heap Build-Max-Heap(A,n) 1. for i   n/2  downto 1 { 2.Heapify(A, i,n) 3.}

CPSC 311 O. Burchan Bayazit – Fall 02 Build-Heap Build-Max-Heap(A,n) 1. for i   n/2  downto 1 { 2.Heapify(A, i,n) 3.}

CPSC 311 O. Burchan Bayazit – Fall 02 Build-Heap Build-Max-Heap(A,n) 1. for i   n/2  downto 1 { 2.Heapify(A, i,n) 3.}

CPSC 311 O. Burchan Bayazit – Fall 02 Build-Heap Build-Max-Heap(A,n) 1. for i   n/2  downto 1 { 2.Heapify(A, i,n) 3.}

CPSC 311 O. Burchan Bayazit – Fall 02 Build-Heap Build-Max-Heap(A,n) 1. for i   n/2  downto 1 { 2.Heapify(A, i,n) 3.}

CPSC 311 O. Burchan Bayazit – Fall 02 Build-Heap Build-Max-Heap(A,n) 1. for i   n/2  downto 1 { 2.Heapify(A, i,n) 3.}

CPSC 311 O. Burchan Bayazit – Fall 02 Build-Heap Build-Max-Heap(A,n) 1. for i   n/2  downto 1 { 2.Heapify(A, i,n) 3.}

CPSC 311 O. Burchan Bayazit – Fall 02 Build-Heap Build-Max-Heap(A,n) 1. for i   n/2  downto 1 { 2.Heapify(A, i,n) 3.}

CPSC 311 O. Burchan Bayazit – Fall 02 Build-Heap Build-Max-Heap(A,n) 1. for i   n/2  downto 1 { 2.Heapify(A, i,n) 3.}

CPSC 311 O. Burchan Bayazit – Fall 02 Build-Heap Build-Max-Heap(A,n) 1. for i   n/2  downto 1 { 2.Heapify(A, i,n) 3.}

CPSC 311 O. Burchan Bayazit – Fall 02 Build-Heap Build-Max-Heap(A,n) 1. for i   n/2  downto 1 { 2.Heapify(A, i,n) 3.}

CPSC 311 O. Burchan Bayazit – Fall 02 Build-Heap Build-Max-Heap(A,n) 1. for i   n/2  downto 1 { 2.Heapify(A, i,n) 3.}

CPSC 311 O. Burchan Bayazit – Fall 02 Build-Heap Build-Max-Heap(A,n) 1. for i   n/2  downto 1 { 2.Heapify(A, i,n) 3.}

CPSC 311 O. Burchan Bayazit – Fall 02 Build-Heap Build-Max-Heap(A,n) 1. for i   n/2  downto 1 { 2.Heapify(A, i,n) 3.}

CPSC 311 O. Burchan Bayazit – Fall 02 Build-Heap Build-Max-Heap(A,n) 1. for i   n/2  downto 1 { 2.Heapify(A, i,n) 3.} O(lgn) repeated (n/2 times) T(n)=O(nlgn) Can we do better? Yes T(n)=O(n)

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort Algorithm HeapSort(A,n) 1. Build-Heap(A,n) /* put all elements in heap */ 2. for i = n downto 2 { 3. swap A[1]  A[i] /* puts max in ith array position */ 4.Heapify(A,1,i-1) /* restore heap property */ 5.} Input: An n-element array A (unsorted). Output: An n-element array A in sorted order, smallest to largest. Running time of HeapSort 1 call to Build-Heap()  O(n) time n-1 calls to Heapify() each takes O(lgn) time  O(nlgn) time

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Assume we have a max-heap

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works The largest number is the root (max-heap property)

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Take the root away

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Put the last one to the root

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Fix the max-heap

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Fix the max-heap

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Take the root away

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Put the last one to the root

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Fix the max-heap

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Take the root away

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Put the last one to the root

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Fix the max-heap

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Take the root away

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Put the last one to the top

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Take the top away

CPSC 311 O. Burchan Bayazit – Fall 02 Heap Sort – How it works Put the last one

CPSC 311 O. Burchan Bayazit – Fall 02 Priority Queues: Application for Heaps An application of max-priority queues is to schedule jobs on a shared processor. Need to be able to get the highest priority job Heap-Maximum(A,n) remove job from the queue Heap-Extract-Max(A,n) insert new jobs into queue Max-Heap-Insert(A, key,n) increase priority of jobs Heap-Increase-Key(A,i,key,n)

CPSC 311 O. Burchan Bayazit – Fall 02 Heap-Extract-Max Heap-Extract-Max(A,n) 1. max=A[1] 2. A[1]=A[n] 3. n=n-1 /* puts max in ith array position */ 4. Heapify(A,1,n) /* restore heap property */ 5. return max T(n)=O(lgn)

CPSC 311 O. Burchan Bayazit – Fall 02 Heap-Increase-Key Heap-Increase-Key(A,i,key,n) 1. A[i]=key 2. while I>1 and A[Parent[i]]<A[i] { 3. swap A[Parent[i]]  A[i] 4. i=Parent[i] T(n)=O(lgn)

CPSC 311 O. Burchan Bayazit – Fall 02 Max-Heap-Insert Max-Heap-Insert(A,i,key,n) 1. n=n+1 2. A[i]=-  3. Heap-Increase-Key(A,n,key,n) T(n)=O(lgn)