Copyright © 2011-2016 Aiman Hanna All rights reserved Heaps Dr. Aiman Hanna Department of Computer Science & Software Engineering Concordia University, Montreal, Canada These slides has been extracted, modified and updated from original slides of : Data Structures and Algorithms in Java, 5th edition. John Wiley& Sons, 2010. ISBN 978-0-470-38326-1. Data Structures and the Java Collections Framework by William J. Collins, 3rdedition, ISBN 978-0-470-48267-4. Both books are published by Wiley. Copyright © 2010-2011 Wiley Copyright © 2010 Michael T. Goodrich, Roberto Tamassia Copyright © 2011 William J. Collins Copyright © 2011-2016 Aiman Hanna All rights reserved

Coverage & Warning Section 8.3: Heaps Warning: As an upfront warning, the “heap” data structure discussed here has nothing to do with the memory heap used in the run-time environment. 2 6 5 7 9 Heaps

Recall Priority Queue ADT A priority queue stores a collection of entries Each entry is a pair (key, value) Main methods of the Priority Queue ADT insert(k, x) inserts an entry with key k and value x removeMin() removes and returns the entry with smallest key Additional methods min() returns, but does not remove, an entry with smallest key size(), isEmpty() Applications: Standby flyers Auctions Stock market Heaps

Recall P.Q. Sorting We use a priority queue Algorithm PQ-Sort(S, C) Insert the elements with a series of insert operations Remove the elements in sorted order with a series of removeMin operations The running time depends on the priority queue implementation: Unsorted sequence gives selection-sort: O(n2) time Sorted sequence gives insertion-sort: O(n2) time Can we do better? Algorithm PQ-Sort(S, C) Input sequence S, comparator C for the elements of S Output sequence S sorted in increasing order according to C P  priority queue with comparator C while S.isEmpty () e  S.remove (S. first ()) P.insertItem(e, ) while P.isEmpty() e  P.removeMin().getKey() S.addLast(e) Heaps

Heaps 12/4/2018 5:27 AM Heaps Both insertion-sort and selection-sort of P.Q. achieved running time of O(n2). An efficient realization of a priority queue uses a data structure, called heap. This data structure allows us to perform both insertions and removals in logarithmic time, which is significant improvement over the list-based implementation. Fundamentally, the heap achieves such improvement by abandoning the idea of storing entries in a list; instead it stores the entries in a binary tree. Heaps

Heaps The last node of a heap is the rightmost node of maximum depth 2 12/4/2018 5:27 AM Heaps In other words, a heap is a binary tree storing entries at its nodes. Additionally, a heap satisfies the following properties: Heap-Order: for every internal node v other than the root, key(v)  key(parent(v)) Complete Binary Tree: let h be the height of the heap for i = 0, … , h - 1, there are 2i nodes of depth i at depth h - 1, the internal nodes are to the left of the external nodes there is at most one node with one child (only one node can have a single child) and this child must be a left child The last node of a heap is the rightmost node of maximum depth 2 5 6 9 7 last node Heaps

Heaps 12/4/2018 5:27 AM Heap-order Property Heap-Order Property: This s a relational property. For every internal node v other than the root, key(v)  key(parent(v)). Consequently, the keys encountered on a path from the root to an external node are in non-decreasing order. Additionally, the minimum key (which is the most important one) is hence always stored at the root (or the “top of the heap”, hence the name “heap” of this data structure). Heaps

Complete Binary Tree Property Heaps 12/4/2018 5:27 AM Complete Binary Tree Property Complete Binary Tree Property: This is a structural property. This property is needed to insure that the height of the heap is as small as possible. Heaps

Complete Binary Tree Property Heaps 12/4/2018 5:27 AM Complete Binary Tree Property A heap T with height h is a complete binary tree if each depth i = 0, … , h - 1 has the maximum possible number of entries, and at least one entry at the last depth (that is depth h). Formally, each level i = 0, … , h - 1 must have 2i nodes. In a complete binary tree, a node v is to the left of node w if v and w are at the same level and that v is encountered before w (from left to right, which is also an inorder traversal). Heaps

Complete Binary Tree Property Heaps 12/4/2018 5:27 AM Complete Binary Tree Property For instance, node with entry (15, K) is to the left of node with entry (7, Q). Similarly, node with entry (14, E) is to the left of node with entry (8, w), etc. Another important node is the last node. This is the right-most, deepest external node in the tree. Note: right-most means the one to the right of all other nodes in the level (this may not necessary be a right child). (4, C) (6, Z) (5, A) (9, F) (15, K) (7, Q) (20, B) (25, J) (12, H) (8, W) 16, X) (14, E) (11, S) last node Heaps

Height of a Heap Theorem: A heap storing n keys has height O(log n) Proof: (we apply the complete binary tree property) Let h be the height of a heap storing n keys Since there are 2i keys at depth i = 0, … , h - 1 and at least one key at depth h, we have n  1 + 2 + 4 + … + 2h-1 + 1 = 2h - 1 + 1 Thus, n  2h , i.e., h  log n This theorem is very important since it implies that if we can perform updates on a heap in a time proportional to its height, then those operations will run in logarithmic time. 1 2 2h-1 keys h-1 h depth Heaps

Heaps and Priority Queues We can use a heap to implement a priority queue. We store a (key, element) item at each internal node. We keep track of the position of the last node. (2, Sue) (5, Pat) (6, Mark) (9, Jeff) (7, Anna) Heaps

Insertion into a Heap Method insert(k, x) of the priority queue ADT corresponds to the insertion of a key k to the heap. The insertion algorithm consists of three steps: Find the insertion node z (the new last node) Store k at z Restore the heap-order property (discussed next) 2 5 6 z 9 7 insertion node 2 6 5 7 9 1 z Heaps

Upheap After the insertion of a new key k, the heap-order property may be violated. Algorithm upheap restores the heap-order property by swapping k along an upward path from the insertion node. 2 6 5 7 9 1 z 2 1 5 7 9 6 z 1 5 2 z 9 7 6 Heaps

Upheap Upheap terminates when the key k reaches the root or a node whose parent has a key smaller than or equal to k. Since a heap has height O(log n), upheap runs in O(log n) time. 1 5 2 z 9 7 6 Heaps

Upheap Example: (4, C) (6, Z) (5, A) (9, F) (15, K) (7, Q) (20, B) (25, J) (12, H) (8, W) 16, X) (14, E) (11, S) (4, C) (5, A) (6, Z) (15, K) (9, F) (7, Q) (20, B) 16, X) (25, J) (14, E) (12, H) (11, S) (8, W) (2, T) Heaps

Upheap Example (Continues …): (4, C) (5, A) (6, Z) (15, K) (9, F) (7, Q) (2, T) 16, X) (25, J) (14, E) (12, H) (11, S) (8, W) (20, B) (4, C) (5, A) (2, T) (15, K) (9, F) (7, Q) (6, Z) 16, X) (25, J) (14, E) (12, H) (11, S) (8, W) (20, B) Heaps

Upheap Example (Continues …): (2, T) (5, A) (4, C) (15, K) (9, F) (7, Q) (6, Z) 16, X) (25, J) (14, E) (12, H) (11, S) (8, W) (20, B) Heaps

Removal from a Heap (§ 7.3.3) Method removeMin() of the priority queue ADT corresponds to the removal of the root key from the heap. The removal algorithm consists of four steps: Return the root entry Replace the root key (entry in fact) with the key of the last node w Remove w Restore the heap-order property (discussed next) 2 5 6 w 9 7 last node 7 5 6 w 9 new last node Heaps

Downheap After replacing the root key with the key k of the last node, the heap-order property may be violated. Algorithm downheap restores the heap-order property by swapping key k along a downward path from the root. If T has no right child, then the swapping (if needed) starts at the left child (which is the only one actually) If both left and right children are there, then the swapping occurs with the one that has the smaller key; otherwise only one side of the tree may be corrected, which may force further swap operations. 7 5 5 6 7 6 w w 9 9 Heaps

Downheap Downheap terminates when key k reaches a leaf or a node whose children have keys greater than or equal to k. The downward swapping process is called down-heap bubbling. Since a heap has height O(log n), downheap runs in O(log n) time. 7 5 5 6 7 6 w w 9 9 Heaps

Downheap (4, C) Removal Example: (4, C) (6, Z) (5, A) (9, F) (15, K) (7, Q) (20, B) (25, J) (12, H) (13, W) 16, X) (14, E) (11, S) (4, C) (13, W) (5, A) (6, Z) (15, K) (9, F) (7, Q) (20, B) 16, X) (25, J) (14, E) (12, H) (11, S) Heaps

Downheap Removal Example (Continues …): (13, W) (6, Z) (5, A) (9, F) (15, K) (7, Q) (20, B) (25, J) (12, H) 16, X) (14, E) (11, S) (5, A) (13, W) (6, Z) (15, K) (9, F) (7, Q) (20, B) 16, X) (25, J) (14, E) (12, H) (11, S) Heaps

Downheap Removal Example (Continues …): (5, A) (6, Z) (9, F) (13, W) (15, K) (7, Q) (20, B) (25, J) (12, H) 16, X) (14, E) (11, S) (5, A) (9, F) (6, Z) (15, K) (12, H) (7, Q) (20, B) 16, X) (25, J) (14, E) (13, W) (11, S) Heaps

Updating the Last Node The insertion node can be found by traversing a path of O(log n) nodes For instance, if the current last node is a left child then: Go up to the father and down right to the new last node If the current last node is a right child then: Go up until a left child or the root is reached If a left child is reached, go to the right child Go down left until a leaf is reached Similar algorithm for updating the last node after a removal Heaps

Heap Performance Operation Complexity size(), isEmpty() O(1) min() The performance of a P.Q. realized by means of a heap is as follows: Operation Complexity size(), isEmpty() O(1) min() insert() O(log n) removeMin() Heaps

Heap-Sort Let us recall the PQ-Sort algoirthm Algorithm PQ-Sort(S, C) Input sequence S, comparator C for the elements of S Output sequence S sorted in increasing order according to C P  priority queue with comparator C while S.isEmpty () e  S.removeFirst () P.insert (e, ) while P.isEmpty() e  P.removeMin().getKey() S.addLast(e) Heaps

Heap-Sort Using a heap-based priority queue, the following is true: Each insertion takes O(log n) time In fact the insertion of any entry i takes exactly O(1 + log i) since this is actually the ith insertion where 1 <= i <= n. Each removal takes O(log n) time In fact the removal of any entry i takes exactly O(1 + log(n – i + 1)) since this is actually the ith removal (that is, some elements may have already been removed) where 1 <= i <= n. Consequently, the entire algorithm consumes O(n log n) time to have n removals in phase I (first loop) and O(n log n) time to have n insertions in phase I (first loop), which leads to a final complexity of O(n log n). Heaps

Heap-Sort The resulting algorithm using heap to realize the P.Q. is hence called heap-sort. Heap-sort is much faster than quadratic sorting algorithms, such as insertion-sort and selection-sort. Heaps

Heap-Sort Consider a priority queue with n items implemented by means of a heap the space used is O(n) methods insert and removeMin take O(log n) time methods size, isEmpty, and min take time O(1) time Using a heap-based priority queue, we can sort a sequence of n elements in O(n log n) time The resulting algorithm is called heap-sort Heap-sort is much faster than quadratic sorting algorithms, such as insertion-sort and selection-sort Heaps

Array-based Heap Implementation We can represent a heap with n keys by means of an array of length n + 1 For the node at rank i the left child is at rank(index) 2i the right child is at rank 2i + 1 Links between nodes are not explicitly stored The cell of at rank 0 is not used Operation insert corresponds to inserting at rank n + 1 Operation removeMin corresponds to removing at rank n Yields in-place heap-sort 2 5 6 9 7 2 5 6 9 7 1 3 4 Heaps

Merging Two Heaps We are given two heaps and a key k 3 2 We are given two heaps and a key k We create a new heap with the root node storing k and with the two heaps as subtrees We perform downheap to restore the heap-order property 8 5 4 6 7 3 2 8 5 4 6 2 3 4 8 5 7 6 Heaps

Bottom-up Heap Construction It is possible to construct a heap in o(n log n) time. Simply, perform n successive insertions. Since worst case for any of these insertions is log n, the total time consumption is O(n log n). However, if all the entries (that is all the key-value pairs) are known and given in advance, we can have an alternative approach that results only on a complexity of O(n). Such approach is referred to as the bottom-up construction approach. Heaps

Bottom-up Heap Construction To simplify the description of the algorithm, let us assume that the heap is a complete binary tree; that is all levels are full. Consequently, n = 2h+1 – 1 h = log(n + 1) - 1 The construction of the heap consists of h + 1 steps (which is also log( n + 1) steps). Heaps

Bottom-up Heap Construction Generally, in each step i, pairs of heaps with 2i -1 keys are merged to construct larger heaps with 2i+1-1 keys. Step 1: construct (n + 1) / 21 elementary heaps with one entry each. Step 2: construct (n + 1) / 22 heaps, each storing 3 entries by joining pairs of elementary heaps and adding a new entry. New entry is added at the root, and swapping is performed if needed. 2i -1 2i+1-1 Heaps

Bottom-up Heap Construction : Step h + 1 : In this last step, construct (n + 1) / 2h+1 heap. This is actually the final heap containing all n elements by joining the last two heaps (each storing (n -1)/2 entries) from Step h. As before, the new entry is added at the root, and swapping is performed if needed. Note that all swap operations in each of the steps use down-heap bubbling to preserve the heap-order property. 2i -1 2i+1-1 Heaps

Bottom-up Heap Construction Example 16 15 4 12 6 7 23 20 Step 1 25 5 11 27 16 15 4 12 6 7 23 20 Step 2 Heaps

Example (contd.) 25 5 11 27 16 15 4 12 6 9 23 20 Heap-order property is violated; we need to apply down-heap bubbling 15 4 6 20 16 25 5 12 11 9 23 27 Step 2 is concluded Heaps

Example (contd.) 7 8 15 4 6 20 16 25 5 12 11 9 23 27 Step 3 4 6 15 5 8 20 16 25 7 12 11 9 23 27 Heap-order property was violated; Heap after applying down-heap bubbling Heaps

Final heap after applying down-heap bubbling Example (end) 10 4 6 15 5 8 20 16 25 7 12 11 9 23 27 Step 4 4 5 6 15 7 8 20 16 25 10 12 11 9 23 27 Final heap after applying down-heap bubbling Heaps

Analysis Let us consider any internal node v. The construction of the tree rooted at v (subtree actually, except when v is the root) is proportional to the size of the paths of this tree. Which paths are associated with a node? An associated path, p(v),with a node v is the one that goes to the right child of v then goes left downward until it reaches an external node (note that this path may differ from the actual downheap path). Heaps

Analysis Clearly the size (number of nodes) of p(v) is equal to the: height of the tree rooted at v + 1. Each node in the tree belongs to at most 2 associated paths: the one rooted at v and the one coming down from its parent (note that the root and all the nodes on the left-most root-to-leaf path belong only to one associated path). Heaps

Analysis Therefore, the total sizes of the paths associated with the internal nodes of the tree is at most 2n – 1. Consequently, the construction of the tree (all its paths) using bottom-up heap construction runs in O(n) time.  Bottom-up heap construction speeds up the first phase of heap-sort. http://stackoverflow.com/questions/9755721/build-heap-complexity Can someone help explain how can building a heap be O(n) complexity? Inserting an item into a heap is O(logN), and the insert is repeated n/2 times (the remainder are leaves and can't violate the heap property. So this means the complexity should be O(nLog n) I would think. Put another way, for each item we "heapify", it has the potential to have to filter down once for each level for the heap so far (which is log n levels). What am I missing? algorithm complexity-theory share|improve this question asked Mar 18 '12 at 3:15 Will Den 2,35592853   what precisely do you mean by "building" a heap? –  mfrankli Mar 18 '12 at 3:29   As you would in a heapsort, take an unsorted array and filterdown each of the top half elements until it conforms to the rules of a heap –  Will Den Mar 18 '12 at 3:29   Only thing i could find was this link: The complexity of Buildheap appears to be Θ(n lg n) – n calls to Heapify at a cost of Θ(lg n) per call, but this result can be improved to Θ(n) cs.txstate.edu/~ch04/webtest/teaching/courses/5329/lectures/… –  Will Den Mar 18 '12 at 3:30 7 Answers activeoldestvotes up vote26down voteaccepted Your analysis is corrrect. However, it is not tight. It is not really easy to explain. You should better read it. Great analysis of algorithm can be seen here. However, the main idea is, in Buildheap algorithm the actual Heapify cost is not O(logn) for all elements. When Heapify is called, the running time depends on how far an element might move down in tree before the process terminates. In other words it depends on the height of node. In the worst case the element might go down all the way to the leaf level. Let us count the work done level by level. At the bottommost level there are 2^(h) nodes, but we do not call Heapify on any of these so the work is 0. At the next to level there are 2^(h − 1) nodes, and each might move down by 1 level. At the 3rd level from the bottom there are 2^(h − 2) nodes, and each might move down by 2 levels. As you can see not all heapify operations are O(logn), this is why you are getting O(n). share|improve this answer edited May 9 '12 at 6:11 answered Mar 18 '12 at 3:39 emre nevayeshirazi 4,29211533 5   Ah a converging series... –  Will Den Mar 18 '12 at 3:59 2   This is a great explanation...but why is it then that the heap-sort runs in O(n log n). Why doesn't the same reasoning apply to heap-sort? –  hba Mar 30 at 3:11 Heaps