1. 1 Analysis of Algorithms Chapter - 03 Sorting Algorithms

2. 2 This Chapter Contains the following Topics: 1.Simple Sorts i.Bubble Sort ii.Insertion Sort 2.Heap sort i.Heaps ii.The Heap Property iii.Maintaining the Heap Property iv.Building a Heap v.The Heapsort Algorithm vi.Priority Queues

3. 3 Simple Sorts

4. 4  Sorting Problem  Input: A sequence of n numbers (a 1, a 2,…, a n ).  Output: A permutation of n numbers (reordering) (a' 1, a' 2,…, a' n ) of the input sequence such that a' 1 ≤ a' 2 ≤… ≤ a' n.  Bubble Sort  It is a popular sorting algorithm  It swaps adjacent elements that are out of order Algorithm BubbleSort (A, n) { for i:=1 to n-1 do { for j := n downto i+1 do { if ( A[j]<A[j-1]) then exchange A[j] ↔ A[j-1]; }  Illustrate the method.  What is the worst-case complexity of Bubble sort? Bubble Sort

5. 5 Insertion Sort  Insert an element to a sorted array such that the order of the resultant array be not changed. Algorithm InsertionSort (A, n) { for i:=2 to n do { key:=A[i]; // Insert A[i] into the sorted sequence A[1…i-1]. j:=i-1; while ( (j>0) and (A[j]>key) ) do { A[j+1]:=A[j]; j:=j-1; } A[j+1]:=key; }  What is the worst-case complexity of Insertion sort?

6. 6 Analyzing Simple Sorts  Bubble sort -O(n 2 ) - Very simple code  Insertion sort - Slightly better than bubble sort - Fewer comparisons required. - Also O(n 2 )  Bubble Sort and Insertion Sort - Use when n is small, (n ~10); - Simple code compensates for low efficiency.

7. 7 Heap sort

8. 8  A heap can be seen as a complete binary tree:  What makes a binary tree complete?  Is the example above complete? 16 1410 8793 241 Heaps

9. 9  A heap can be seen as a complete binary tree:  They are called “nearly complete” or “full” binary trees.  We can think the unfilled slots as null pointers. 16 1410 8793 241 1 1 1 11 Heaps (Contd.)

10. 10  In practice, heaps are usually implemented as arrays.  To represent a complete binary tree as an array:  The root node is A[1]  Node i is A[i]  The parent of node i is A[i/2] (note: integer divide)  The left child of node i is A[2i]  The right child of node i is A[2i + 1]  For example, the array can be represented as the following heap. 16 1410 8793 241 1614108793241 A = Heaps (Contd.)

11. 11 Referencing Heap Elements  So… Parent(i) { return  i/2  ; } Left(i) { return 2*i; } right(i) { return 2*i + 1; }

12. 12  Heaps also satisfy the max-heap property:  A[Parent(i)]  A[i] for all nodes i > 1  In other words, the value of a node is at most the value of its parent.  In min-heap,  A[Parent(i)] ≤ A[i] for all nodes i > 1  In other words, the value of a parent node is at most the value of its child nodes.  Where are the largest elements in a max-heap and min-heap stored?  Definitions:  The height of a node in a head = the number of edges on the longest simple downward path from the node to a leaf.  The height of the heap = the height of its root.  Since a heap of n-elements is based on a complete binary tree, its height is Θ(lg n)  The basic operations on heaps run in time at most proportional to the height of the tree and thus take O(lg n). The Heap Property

13. 13  MaxHeapify(): maintains the max-heap property.  Given: a node i in the heap with children l (left) and r (right).  Given: two subtrees rooted at l and r, assumed to be heaps.  Problem: The subtree rooted at i may violate the heap property (How?)  Action: let the value of the parent node A[i] “float down” so subtree rooted at index i becomes a max-heap.  What do you suppose will be the basic operation between i, l, and r? Maintaining the Heap Property

14. 14 Algorithm MaxHeapify(A, i) { l := Left(i); r := Right(i); if ((l A[i])) then largest := l; else largest := i; if ((r A[largest])) then largest := r; if (largest ≠ i) then { exchange A[i]↔A[largest]; MaxHeapify(A, largest); } The Algorithm MaxHeapify()

15. 15 16 410 14793 281 1641014793281 A = Illustration by an Example 16 410 14793 281 161014793281 A = 4

16. 16 410 14793 2811610793281 A = 414 Example (Contd.) 16 1410 4793 281 1614104793281 A =

17. 17 16 1410 4793 281161410793281 A = 4 Example (Contd.) 16 1410 4793 28116141079321 A = 48

18. 18 16 1410 8793 241 1614108793241 A = Example (Contd.) 16 1410 8793 241161410879321 A = 4

19. 19 16 1410 8793 241 1614108793241 A = Example (Contd.)

20. 20 Analyzing MaxHeapify()  The running time of the algorithm on a subtree of size n rooted at given node i is the   (1) time to fix up the relationships among the elements A[i], A[l] and A[r],  Plus the time to run MaxHeapify() on a subtree rooted at one of the children of node i.  If the heap at i has n elements, how many elements can the subtrees at l or r have?  Answer: 2n/3  The worst case occurs when the last row of the tree is exactly half full.  So time taken by MaxHeapify() is given by T(n)  T(2n/3) +  (1)  By case 2 of the Master Theorem, T(n) = O(lg n)  Alternatively, we can characterize the running time of MaxHeapify() on a node of height h as O(h).

21. 21  We can build a heap in a bottom-up manner by running MaxHeapify() on successive subarrays:  Fact: for array of length n, all elements in range A[  n/2  + 1.. n] are heaps (Why?)  Walk backwards through the array from n/2 to 1, calling MaxHeapify() on each node.  Order of processing guarantees that the children of node i are heaps when i is processed. Algorithm BuildMaxHeap(A) { heap_size(A) := length(A); for i :=  length[A]/2  downto 1 do MaxHeapify(A, i); }  Work through the example A = {4, 1, 3, 2, 16, 9, 10, 14, 8, 7} Building a Heap

22. 22  Each call to MaxHeapify() takes O(lg n) time.  There are O(n) such calls (specifically,  n/2  )  Thus the running time is O(n lg n).  This is an upper bound, though correct, but is not asymptotically tight.  Our tighter analysis relies on the properties that an n-element heap has height  lg n , and at most |n/2 h+1 | of any height h.  The time required by MaxHeapify() when called on a node of height h is O(h).  So we can express the total cost of BuildMaxHeap() as being bounded from above by the following calculation. Analyzing BuildMaxHeap

23. 23 Analyzing BuildMaxHeap (Contd.)

24. 24  The Heapsort algorithm starts by using BuildMaxHeap() to build a max-heap on the input array A[1..n], where n=length[A].  The maximum element of the array is stored at the root A[1].  Swap A[1] with element at A[n] to put the maximum element into its correct position.  Then discard the node n from the heap, which will decrement the heap_size[A] by one.  The children of the root remain max-heap, but the new root element may violate the max-heap property.  So, restore the max-heap property at A[1] by calling MaxHeapify(A,1).  This will leaves a max-heap in A[1..(n-1)].  Always swap A[1] for A[heap_size(A)], and repeat the process for the heap_size(A) down to 2. Heapsort

25. 25 Algorithm Heapsort(A) { BuildMaxHeap(A); for i := length(A) downto 2 do { exchange A[1] ↔ A[i]; heap_size(A) := heap_size(A) - 1; MaxHeapify(A, 1); }  Work through the example A = {4, 1, 3, 2, 16, 9, 10, 14, 8, 7}  Analyzing the heapsort algorithm  The call to BuildMaxHeap() takes O(n) time.  Each of the n - 1 calls to MaxHeapify() takes O( lg n) time.  Thus the total time taken by Heapsort() = O(n) + (n - 1) O(lg n) = O(n) + O(n lg n) = O(n lg n) The Heapsort Algorithm

26. 26 Priority Queues  One of the most popular application of a heap is an efficient priority queue.  A priority queue is a data structure for maintaining a set S of elements, each with an associated value called a key.  There are two kinds of priority queues:  Max-priority queues and  Min-priority queues.  We will focus how to implement max-priority queues, which are based on max-heaps.  A max-priority queue supports the following operations:  Insert(S, x): Inserts the element x into the set S. This operation could be written as S:= S U {x}.  Maximum(S): Returns the element of S with the largest key.  ExtractMax(S): Removes and returns the element of S with the largest key.  IncreaseKey(S, x, k): Increases the value of element x’s key to the new value k, which is assumed to be at least as large as x’s current key value.

27. 27 Priority Queues (Contd.)  Application of max-priority queues:  Scheduling jobs on a shared computer.  It keeps the track of jobs to be performed and their relative priorities.  When a job is finished or interrupted, the highest priority job is selected from those pending using ExtractMax().  A new job can be added to the queue at any time using Insert().  A min-priority queue supports the operations:  Insert(), Minimum(), ExtractMin(), and DecreaseKey().  Application of min-priority queues:  It can be used as event-driven simulator.  The items in the queue are events to be simulated, each with an associated time of occurrence that serves as its key.  The events must be simulated in order of their time of occurrences, because the simulation of an event can cause other events to be simulated in the future.  The simulation program uses ExtractMin() at each step to choose the next event to simulate.  As new events are produced, they are inserted into the min-priority queue using Insert().

28. 28 Operations  The procedure HeapMaximum() implements the Maximum() operation in Θ(1) time. Algorithm HeapMaximum(A) return A[1];  The procedure HeapExtractMax() implements the ExtractMax() operation as follows: Algorithm HeapExtractMax(A) { if (heap_size(A) < 1) then error “heap underflow”; max := A[1]; A[1] := A[heap_size(A)]; heap_size(A) := heap_size(A) – 1; MaxHeapify(A, 1); return max; }  The running time of HeapExtractMax() is O(lg n), since it performs only a constant amount of work on top of the O(lg n) time for MaxHeapify().

29. 29 Operations (Contd.)  The procedure HeapIncreaseKey() implements the IncreaseKey() operation.  The priority-queue element whose key is to be increased is identified by an index i into the array.  The procedure first updates the key of element A[i] to its new value.  Increasing the key of A[i] may violate the max-heap property.  So, the procedure then traverses a path from this node towards the root to find a proper place for the newly increased key.  During this traversal, it repeatedly compares an element to its parent, exchanging their keys and continuing if the element’s key is larger, and terminating if the element’s key is smaller, since the max-heap property now holds

30. 30 Operations (Contd.) Algorithm HeapIncreaseKey(A, i, key) { if (key < A[i]) then error “new key is smaller than current key”; A[i] := key; while ((i > 1) and (A[Parent(i)] < A[i]) do { exchange A[i] ↔ A[Parent(i)]; i := Parent(A); }  Since the path traced from the node updation statement to the root has length O(lg n).  The running time of HeapIncreaseKey() on an n- element heap is O(lg n).  Illustrate the procedure.

31. 31 Illustration 16 1410 8793 241 (a) The max-heap with a node whose index i is red labeled. (b) This node has its key updated to 15. 16 1410 8793 2151 i i

32. 32 Illustration (Contd.) 16 1410 15793 281 (c) After one iteration of the while loop, the node and its parent have exchanged keys, and the index i moves up to the parent. 16 1510 14793 281 i i (d) The max-heap after one more iteration of the while loop. At this points, A[Parent(i)] ≥ A[i]. The max-heap property now holds and the procedure terminates.

33. 33 MaxHeapInsert()  The procedure MaxHeapInsert() implements the Insert() operation.  It takes as an input the key of new element to be inserted into max-heap.  The procedure first expands the max-heap by adding to the tree a new leaf whose key is -∞.  Then it calls HeapIncreaseKey() to set the key of this new node to its correct value and maintain the max-heap property. Algorithm MaxHeapInsert(A, key) { heap_size(A) := heap_size(A) + 1; A[heap_size(A)] := -∞; HeapIncreaseKey(A, heap_size(A), key); }  The running time of MaxHeapInsert() on an n- element heap is O(lg n).  In summary, a heap can support any priority- queue operation on a set of size n in O(lg n) time.

34. 34 End of Chapter - 03