Cpt S 223 – Advanced Data Structures Priority Queues (Heaps) Nirmalya Roy School of Electrical Engineering and Computer Science Washington State University

Motivation Queues are a standard mechanism for ordering tasks on a first-come, first-served basis However, some tasks may be more important or timely than others (higher priority) Priority queues Store tasks using a partial ordering based on priority Ensure highest priority task at head of queue Heaps are the underlying data structure of priority queues

Priority Queues Main operations insert (i.e., enqueue) Dynamic insert Specification of a priority level (0 (high), 1, 2, …, low) deleteMin (i.e., dequeue) Finds the current minimum element (element with highest priority) in the queue, deletes it from the queue, and returns it Performance goal is for operations to be “fast”

Priority Queue Implementations Unordered list O(1) for insert O(N) for deleteMin Ordered list O(N) for insert O(1) for deleteMin Balanced BST O(log2 N) for insert and deleteMin Can it be done faster?

A priority queue data structure Binary Heap A priority queue data structure

Binary Heap A binary heap is a binary tree with two properties Structure property Heap-order property

Structure Property of Binary Heap A binary heap is a complete binary tree Each level is completely filled Bottom level may be partially filled from left to right Height of a complete binary tree with N elements is log2 N

Binary Heap Example N = 10 Every level (except last) completely filled Array representation of the binary heap above:

Not a Binary Heap A B C D E F H G Why is the above cannot be a binary heap?

Heap-Order Property of Binary Heap Heap-order property (for a “min binary heap”) For every node X, key(parent(X))  key(X) Except root node, which has no parent Thus, minimum key always at the root Alternatively, for a “max binary heap”, always keep the maximum key at the root insert and deleteMin must always maintain heap-order property

Heap-Order Property Example An example of a min binary heap 13 14 16 19 21 68 26 65 31 32 Duplicates are allowed No order implied for elements at the same tree level or among siblings

Not a Min Binary Heap The following is NOT a min binary heap 17 14 16 19 21 68 26 65 11 31 32 Why is the above NOT a min binary heap?

Heap-Order Property Example An example of a max binary heap 30 28 27 20 11 2 15 13 1 9 Duplicates are allowed No order implied for elements at the same tree level or among siblings

Not a Max Binary Heap The following is NOT a max binary heap 30 32 27 20 11 2 15 13 1 12 Why is the above NOT a max binary heap?

Implementing Complete Binary Trees as Arrays Given an element at position i in the array i’s left child is at position 2i i’s right child is at position 2i + 1 i’s parent is at position i / 2 Heap data structures Draw the heap as trees with the implication that an actual implementation will use simple arrays An estimate of the maximum heap size is required in advance

An array (of Comparable objects) and an integer for heap size Fix heap after deleteMin

Heap Insert Insert new element into the heap at the next available slot (“hole”) Needed to maintain a complete binary tree Then, “percolate” (or “trickle”) the element up the heap while min heap-order property is not satisfied This strategy is know as percolate up

Heap Insert: Example 14 vs. 31 Insert 14: 14 vs. 21 14 vs. 13 Heap-order okay; structure okay Percolating up

Heap Insert: Implementation O(log N) time Insert 14

Heap DeleteMin Minimum element is always at the root A hole is created at the root when minimum element is removed Heap decreases by one in number of elements (or size) Last element X should be moved somewhere in the heap If it can be placed in the hole, then we are done Push smaller of the hole’s children into the hole, thus pushing the hole down one level Repeat this step until X can be placed in the hole Percolate down until the heap-order property is not restored

Heap DeleteMin: Example Remove value at the root Move 31 (last element) to the root Is 31 > min(14, 16)? If yes, swap 31 with min(14, 16) If no, leave 31 in hole

Heap DeleteMin: Example (cont’d) Percolating down… Is 31 > min(19, 21)? If yes, swap 31 with min(19, 21) If no, leave 31 in hole Is 31 > min(65, 26)? If yes, swap 31 with min(65, 26) If no, leave 31 in hole

Heap DeleteMin: Example (cont’d) Percolating down… Heap-order property okay; Structure okay; Done.

Heap DeleteMin: Implementation O(log N) time

Heap DeleteMin: Implementation (cont’d) Percolating down… Left child Right child Pick child to swap with

Other Heap Operations decreaseKey(p, v) increaseKey(p, v) remove(p) Lowers value of item p to v Need to percolate up E.g., change job priority increaseKey(p, v) Increases value of item p to v Need to percolate down remove(p) First, decreaseKey(p, -) Then, deleteMin E.g., terminate job Run-times for all 3 functions? Assume: We already know the location of node containing p

Building a Heap What if all N elements are all available upfront to build the heap? To build a heap with N elements Default method takes O(N log N) time. New method will take O(N) time – i.e., optimal

Building a Heap (cont’d) Construct a heap from an initial set of N items Solution 1 Perform N successive inserts O(1) constant time on average and O(log N) worst-case O(N) average case, but O(N log2 N) worst-case Solution 2 Randomly populate initial heap with structure property Perform a percolate-down from each internal node (i.e., H[size/2] to H[1]) To take care of heap-order property Insert 33 or Insert 14

Building a Heap: Example Input: 150, 80, 40, 30, 10, 70, 110, 100, 20, 90, 60, 50, 120, 140, 130 So, let us look at each internal node, and fix if necessary Randomly initialize heap Structure property satisfied Heap-order property NOT satisfied Leaves are all valid heaps (implicit)

Building a Heap: Example (cont’d)

Building a Heap: Example (cont’d) Dotted lines show the path of percolating down

Building a Heap: Example (cont’d) Dotted lines show the path of percolating down

Building a Heap: Example (cont’d) Dotted lines show the path of percolating down

Building a Heap: Analysis Running time = ? O(sum of the heights of all internal nodes) Theorem 6.1 For the perfect binary tree of height h containing 2h+1 – 1 nodes, the sum of heights of the nodes is 2h+1 – 1 – (h + 1) Proof. Tree consists of 1 node at height h, 2 nodes at height (h-1), 22 nodes at height (h-2) and in general 2i nodes at height (h-i) The sum of the heights of all the nodes Since h = log N, then the sum of heights is O(N) Will be slightly better in practice Implication: Each insertion costs O(1) amortized time

Binary Heap Operations Worst-Case Analysis Height of heap is log2 N insert: O(log2 N) 2.607 comparisons on average, i.e., O(1) deleteMin: O(log2 N) decreaseKey: O(log2 N) increaseKey: O(log2 N) remove: O(log2 N) buildHeap: O(N)

Building a Heap: Implementation Start with the lowest, rightmost internal node

Applications of Heaps Operating system scheduling Graph algorithms Process jobs by priority Graph algorithms Find shortest path Event simulation Instead of checking for events at each time click, look up next event to happen

An Application: The Selection Problem Given a list of n elements, find the kth smallest element Solution 1: Read the elements into an array and sort them, returning the appropriate element Running time is O(n2) Solution 2: Using simple sorting algorithm Sort the list (O(n log n)) Pick the kth element A better approach Use a MinHeap

Selection Using a MinHeap Input: n elements Algorithm: buildHeap(n elements) ==> O(n) Perform k deleteMin() operations ==> O(k lg n) Report the kth deleteMin() output Total run-time = O(n + k lg n) If k = O(n / lg n) then the run-time becomes O(n)

Other Types of Heaps d-Heaps Leftist heaps Skew heaps Generalization of binary heaps (i.e., 2-heaps) All nodes have d children Insert in O(logd N) time; deleteMin in O(d logd N) time Leftist heaps Like a binary heap, it has both a structural and heap order property Binary tree, but not perfectly balanced Supports merging of two heaps in O(m + n) time (i.e., sub-linear) Skew heaps Binary tress with heap order but without any structural constraints O(log n) amortized run-time

Mergeable Heaps Heap merge operation Useful for many applications Merge two (or more) heaps into one Identify new minimum element Maintain heap-order property Merge in O(log N) time Still support insert and deleteMin in O(log N) time Insert = merge existing heap with one-element heap d-Heaps require O(N) time to merge

Leftist Heaps Null path length npl(X) of node X Leftist heap property Length of the shortest path from X to a node without two children Leftist heap property For every node X in heap, npl(leftChild(X)) ≥ npl(rightChild(X)) Leftist heaps have deep left subtrees and shallow right subtrees Thus if operations reside in right subtree, they will be faster

Leftist Heaps npl(X) shown in nodes Leftist heap Not a leftist heap

Leftist Heaps Merge heaps H1 and H2 DeleteMin Assume root(H1) > root(H2) Recursively merge H1 with right subheap of H2 If result is not leftist, then swap the left and right subheaps Running time O(log N) DeleteMin Delete root and merge children

Leftist Heaps: Example Merge H2 (larger root) with right sub-heap of H1 (smaller root) Root 6 of H2 > Root 3 of H1 Right subheap

Leftist Heaps: Example Merge H2 (larger root) with right sub-heap of H1 (smaller root)

Leftist Heaps: Example Attach previous heap as H1’s right child. Leftist heap?

Leftist Heaps: Example Swap root’s children to make leftist heap.

Skew Heaps Self-adjusting version of leftist heap The relationship of Skew heaps are analogous to leftist heaps as splay trees are to AVL trees Skew merge same as leftist merge, except we always swap left and right subheaps No need to maintain or test NPL of nodes Worst case is O(N) Amortized cost of M operations is O(M log N)

Binomial Heaps

Binomial Heap A forest of heap-ordered binomial trees Structure property Heap-order property The structural property of each heap-ordered tree in binomial heap is different from binary heap The heap-order property applies to each binomial tree

Definition: A “Binomial Tree” Bk A binomial tree of height k is called Bk It has 2k nodes The number of nodes at depth d is the binomial co-efficient A binomial tree need not to be a binary tree is the form of the coefficients in binomial theorem Depth B3 # of nodes d = 0 d = 1 d = 2 d = 3

Binomial Heap with 31 Nodes We know: Binomial heap should be a forest of binomial trees Each binomial tree has power of 2 elements How many binomial trees do we need? n = 31 = 1 1 1 1 12 B4 B3 B2 B1 B0

Binomial Heap with 31 Nodes (cont’d) Bk == Bk-1 + Bk-1 B4 B2 B2 B3 B3 A binomial tree Bk consists of a root with children B0, B1, B2 ….Bk-1 How many trees are there in this forest if the # of nodes are N?

Binomial Heap Property Lemma: There exists a binomial heap for every positive value of n Proof: Any arbitrary n can be represented in powers of two (binary representation) Have one binomial tree for each power of two with co-efficient of 1 e.g., if n = 10 Binary Representation ==> (1010)2 ==> forest contains {B3, B1}

Binomial Heaps: Heap-Order Property Each binomial tree should contain the minimum element at the root of every subtree Just like binary heap, except that the tree here is a binomial tree structure (and not a complete binary tree) The order across binomial trees is irrelevant

Definition of Binomial Heaps A binomial heap of n nodes is: A forest of binomial trees as dictated by the binary representation of n (this is the structure property) Each binomial tree is a min-heap or a max-heap (this is the heap-order property)

Binomial Heaps: Example

Key Properties Could there be multiple trees of the same height in a binomial heap? No How many trees could there be, in the worst-case, in a binomial heap of n nodes? log n Given n, can we tell (for sure) how many trees will there be of a given height k? Check if bit is 1 in the kth LSB that means Bk exists if and only if the kth least significant bit is 1 in the binary representation of n

Binomial Heap: Operations insert(x) merge(H1, H2) x = deleteMin()

insert(x) in Binomial Heap Goal To insert a new element x into a binomial heap H Observation Element x can be viewed as a single element binomial heap insert(x) == merge(H, {x}) So, if we decide how to do merge we will automatically figure out how to implement both insert() and deleteMin().

merge(H1, H2) Let n1 be the number of nodes in H1 Therefore, the new heap is going to have n1 + n2 nodes Logic: Merge trees of same height, starting from lowest height trees If only one tree of a given height, then just copy that May need carryover (just like adding two binary numbers)

merge(H1, H2): Example carryover + B0 B1 B2 13 14 26 16 ? B2

merge(H1, H2): Example Result Merge takes O(log n) time (i.e. comparisons).

Merging Two Binomial Trees of the Same Height + = Simply compare the roots. Merge is defined only for binomial trees with the same height

Merging Two Binomial Trees of the Same Height (cont’d)

Merging More Than Two Trees Merging three or more binomial trees of the same height could generate carry-overs 12 23 7 21 24 + 51 24 + 10 8 = ? 65 65 11 Merge any two and leave the third as carry-over.

findMin() Goal: Given a binomial heap, H, find the minimum and delete it Observation: The root of each binomial tree in H contains its minimum element Approach: Therefore, return the minimum of all the roots (minimums) Complexity: O(log n) comparisons

findMin() and deleteMin() Example Find the binomial tree with smallest root say Bk and original priority queue let be H Remove Bk from the forest of trees in H, forming the new binomial queue H’ Remove the root of Bk creating binomial trees B0, B1, B2 ….Bk-1 which collectively forms H’’ Merge H’ and H’’ B0 B2 B3 B0’ B1’ B2’ For deleteMin(): After delete, how to adjust the heap? New heap: Merge {B0, B2} and {B0’, B1’, B2’}.

Binomial Queue Run-time Summary insert O(log n) worst-case O(1) amortized time if insertion is done in a sequence deleteMin, findMin merge

An Implementation of a Binomial Heap Array of binomial trees, children of each node are kept in a linked list Trees use first-child, right-sibling representation Analogous to a bit-based representation of a binary number n B7 B6 B5 B4 B3 B2 B1 B0 Maintain a linked list of tree pointers (for the forest) What is the value of n?

Priority Queues in STL Binary Heap is implemented by the class template named priority_queue found in standard header file queue Default is max-heap Methods Push, pop, top, empty, clear Duplicates allowed, only one largest is removed if several #include <queue> #include <iostream> using namespace std; int main(void) { priority_queue<int> Q; for (int i = 0; i < 100; i++) Q.push(i); while (!Q.empty()) { cout << Q.top() << endl; Q.pop(); } Calls deleteMax() For min-heap, declare priority queue as: priority_queue<int, vector<int>, greater<int> > Q;

Run-time Per Operation Cpt S 223 Run-time Per Operation Types of Heap Insert DeleteMin Merge (=H1+H2) Binary Heap O(log n) worst-case O(1) amortized for buildHeap (constant time on average) O(log n) O(n) Leftist Heap Skew Heap Binomial Heap O(log n) worst case O(1) amortized for sequence of n inserts (constant time on average) Washington State University

Summary Priority queues maintain the minimum or maximum element of a set Standard binary heap implementation is elegant for its simplicity and speed It requires no links and only a constant amount of space Leftist heap is a wonderful example of power of recursion Binomial queue is a simple idea to achieve good time bound Support O(log N) operations in worst-case insert, deleteMin, merge Many applications of priority queues ranging from operating systems scheduling to simulation