CSE 326 Heaps and the Priority Queue ADT David Kaplan Dept of Computer Science & Engineering Autumn 2001

Back to Queues Some applications Problems? ordering CPU jobs simulating events picking the next search site Problems? short jobs should go first earliest (simulated time) events should go first most promising sites should be searched first Alright, now we all remember what a tree is. Let’s harken back to queues now. Here are some possible applications of a queue which aren’t quite right. All of these want to favor certain entries in the Queue. How do we handle that? Heaps CSE 326 Autumn 2001 2

Priority Queue ADT Priority Queue operations create destroy insert deleteMin is_empty Priority Queue property: for two elements in the queue, x and y, if x has a lower priority value than y, x will be deleted before y F(7) E(5) D(100) A(4) B(6) insert deleteMin G(9) C(3) We need a new ADT for this. One that returns the best (which we’ll generally define as lowest priority value) node next. I’m often going to represent priority queues as collections of priorities; remember that they are collections of data with priorities. MOREOVER, some priority queues don’t even need priorities, just the ability to compare priorities. (all of the ones we’ll talk about fall in this category). Heaps CSE 326 Autumn 2001 3

Applications of the Priority Q Hold jobs for a printer in order of length Store packets on network routers in order of urgency Simulate events Select symbols for compression Sort numbers Anything greedy There are, of course, gobs of applications for priority queues. Here are just a few. Heaps CSE 326 Autumn 2001 4

Naïve Priority Q Data Structures Unsorted list: insert: deleteMin: Sorted list: OK, let’s look at some ways we might implement priority Qs. Unsorted lists give us fast inserts, but we have to search the whole list to delete. Sorted lists give us fast delete (it’s just the first element!), but we have to either search the whole list (for linked list) or move the whole list (for array) on an insert. Heaps CSE 326 Autumn 2001 5

Binary Search Tree Priority Q Data Structure (aka BST PQD :-) 8 insert: deleteMin: 5 11 2 6 10 12 A binary search tree is a binary tree in which all nodes in the left subtree of a node have lower values than the node. All nodes in the right subtree of a node have higher value than the node. We’ll talk more about these later. For now, let’s get a feel for how well they work as PQ data structures. How long might an insert take here? (log n) How about deleteMin? It’s always the leftmost node in the tree, right? So, it might take log n time to find the node. What about worst case? The book says that these lopsided deletes don’t cause a problem. Ask me about that via e-mail if you’re curious why not. 4 7 9 14 13 Heaps CSE 326 Autumn 2001 6

Binary Heap Priority Q Data Structure Heap-order property parent’s key is less than children’s keys result: minimum is always at the top Structure property complete tree: fringe nodes packed to the left result: depth is always O(log n); next open location always known 13 14 15 9 11 8 10 6 7 5 4 2 Alright, there are two problems with binary search trees as pqs. First, they’re overkill. Why keep everything ordered? We just need to know the least at any given time. Second, they’re not guaranteed to be complete, so we have worst case O(n) times. How do we find the minimum? Heaps CSE 326 Autumn 2001 7

Nifty Storage Trick Calculations 13 14 15 9 11 8 10 6 7 5 4 2 children: parent: root: next free: 2 3 4 5 6 7 8 9 10 11 12 It so happens that we can store a complete tree in an array and still be able to find children and parents easily. We’ll take advantage of this. Child: left = 2*node right = 2*node + 1 parent = floor(node/2) nextfree = length + 1 root = 1 What do we 1 2 3 4 5 6 7 8 9 10 11 12 13 13 2 4 5 7 6 10 8 11 9 15 14 13 Note: Walking the array in index order gives us level-order traversal!!! Heaps CSE 326 Autumn 2001 8

BinaryHeap::DeleteMin 2 pqueue.deleteMin() 13 14 15 9 11 8 10 6 7 5 4 2 ? 4 5 7 6 10 8 OK, to delete, we start by plucking out that value in O(1) time. Now, we have a hole at the top, and a node that isn’t proper for a complete tree at the bottom. So, we fix it by putting the value in the hole, and moving the hole until we restore the heap-order property. 11 9 15 14 13 Heaps CSE 326 Autumn 2001 9

Percolate Down ? 13 4 5 4 5 7 6 10 8 7 6 10 8 11 9 15 14 13 11 9 15 14 4 4 13 5 6 5 7 6 10 8 7 13 10 8 11 9 15 14 Done! 11 9 15 14

DeleteMin Code runtime: Object deleteMin() { assert(!isEmpty()); returnVal = Heap[1]; size--; newPos = percolateDown(1, Heap[size+1]); Heap[newPos] = Heap[size + 1]; return returnVal; } int percolateDown(int hole, Object val) { while (2*hole <= size) { left = 2*hole; right = left + 1; if (right <= size && Heap[right] < Heap[left]) target = right; else target = left; if (Heap[target] < val) { Heap[hole] = Heap[target]; hole = target; } break; return hole; Note that there are three things going on here: find the smaller child if the smaller child is still smaller than the moving value, move the smaller child up otherwise, we’ve found the right spot, and stop. runtime: Heaps CSE 326 Autumn 2001 11

BinaryHeap::Insert pqueue.insert(3) 20 14 12 9 11 8 10 6 7 5 4 2 2 4 5 Inserting works similarly. We tack a hole on at the end of the tree, and move that hole _up_ until it’s in the right place for the new value. 11 9 12 14 20 ? Heaps CSE 326 Autumn 2001 12

Percolate Up 2 2 4 5 4 5 3 7 6 10 8 7 6 ? 8 3 11 9 12 14 20 ? 11 9 12 14 20 10 2 2 3 4 ? 4 3 7 6 5 8 7 6 5 8 11 9 12 14 20 10 11 9 12 14 20 10

Insert Code runtime: void insert(Object o) { assert(!isFull()); size++; newPos = percolateUp(size,o); Heap[newPos] = o; } int percolateUp(int hole, Object val) { while (hole > 1 && val < Heap[hole/2]) Heap[hole] = Heap[hole/2]; hole /= 2; } return hole; Notice that this code is a _lot_ easier. runtime: Heaps CSE 326 Autumn 2001 14

Performance of Binary Heap worst case Binary heap avg case AVL tree worst case AVL tree avg case Insert O(log n) O(1) percolates ~1.6 levels DeleteMin In practice: binary heaps much simpler to code, lower constant factor overhead Heaps CSE 326 Autumn 2001 15

Changing Priorities In many applications the priority of an object in a priority queue may change over time if a job has been sitting in the printer queue for a long time increase its priority unix “renice” Sysadmin may raise priority of a critical task Must have some (separate) way to find the position in the queue of the object to change (e.g. a hash table) No log(N) find (as with BSTs) – why not? Heaps CSE 326 Autumn 2001 16

Other Priority Queue Operations decreaseKey given a pointer to an object in the queue, reduce its priority value increaseKey given a pointer to an object in the queue, increase its priority value remove given a pointer to an object in the queue, remove it buildHeap given a set of items, build a heap Why do I insist on a pointer to each item in decreaseKey, increaseKey, and remove? Because it’s hard to find an item in a heap; it’s easy to delete the minimum, but how would you find some arbitrary element? This is where the Dictionary ADT and its various implementations which we will study soon come up. What about build heap? Is that really a necessary operation? It turns out that the necessity for this is based _purely_ on the fact that we can do it faster than the naïve implementation! Heaps CSE 326 Autumn 2001 17

DecreaseKey, IncreaseKey, Remove void decreaseKey(int obj) { assert(size >= obj); temp = Heap[obj]; newPos = percolateUp(obj, temp); Heap[newPos] = temp; } void remove(int obj) { assert(size >= obj); percolateUp(obj, NEG_INF_VAL); deleteMin(); } Note: changeKey functions assume that key value has already been changed! void increaseKey(int obj) { assert(size >= obj); temp = Heap[obj]; newPos = percolateDown(obj, temp); Heap[newPos] = temp; } The code for these is pretty simple. Notice that these are called just after the key is changed, they do not include the new value (just assume it’s there). This is different from the book’s version. And, it’s not necessarily better. Heaps CSE 326 Autumn 2001 18

BuildHeap (Floyd’s Method) 12 5 11 3 10 6 9 4 8 1 7 2 pretend it’s a heap and fix the heap-order property! 12 5 11 Pull other slide back up and ask how long it takes. O(n log n) worst case, but O(n) average case. Can we get a worst case O(n) bound? Let’s try pretending it’s a heap already and just fixing the heap-order property. The red nodes are the ones that are out of order. Question: which nodes MIGHT be out of order in any heap? 3 10 6 9 Thank you, Floyd! 4 8 1 7 2 Heaps CSE 326 Autumn 2001 19

Build(this)Heap 12 12 5 11 5 11 3 10 2 9 3 1 2 9 4 8 1 7 6 4 8 10 7 6 12 12 5 2 1 2 3 1 6 9 3 5 6 9 4 8 10 7 11 4 8 10 7 11

Finish Build(ing)(this)Heap 1 3 2 4 5 6 9 How long does this take? Well, everything above the fringe might move 1 step. Everything height 2 or greater might move 2 steps. … In other words n * Sum I=0 to n of I/2i But, this is (as n->inf) just 2. See chapter I to see why. So, the runtime is O(n). 12 8 10 7 11 Runtime? Heaps CSE 326 Autumn 2001 21

Complexity of Build Heap Note: size of a perfect binary tree doubles (+1) with each additional layer At most n/4 percolate down 1 level at most n/8 percolate down 2 levels at most n/16 percolate down 3 levels… Comp O(n) Heaps CSE 326 Autumn 2001 22

Thinking about Heaps Observations Realities finding a child/parent index is a multiply/divide by two operations jump widely through the heap each operation looks at only two new nodes inserts are at least as common as deleteMins Realities division and multiplication by powers of two are fast looking at one new piece of data sucks in a cache line with huge data sets, disk accesses dominate Now, let’s look at some facts about heaps and see if we can’t come up with a new priority queue implementation. Heaps CSE 326 Autumn 2001 23

Solution: d-Heaps Each node has d children Still representable by array Good choices for d: optimize performance based on # of inserts/removes d = 2k for efficiency (array index calcs) fit one set of children in a cache line fit one set of children on a memory page/disk block 1 3 7 2 4 8 5 12 11 10 6 9 D-heaps address all these problems. 12 1 3 7 2 4 8 5 12 11 10 6 9 What do d-heaps remind us of??? Heaps CSE 326 Autumn 2001 24

Merging Heaps Given two heaps, merge them into one heap first attempt: insert each element of the smaller heap into the larger. runtime: second attempt: concatenate heaps’ arrays and run buildHeap. What if you had networked printers or CPUs and one of them went down. Now you want to merge the priority queues! How can we do that? Here’s two naïve ways. 1st attempt: If the two trees are the same size: O(n log n) 2nd attempt: O(n) (that’s how long buildHeap takes) How about O(log n) time? Heaps CSE 326 Autumn 2001 25

Solution: Leftist Heaps Idea Localize all maintenance work in one small part of the heap Leftist heap: almost all nodes are on the left all the merging work is on the right Leftist heaps are a solution to that. They put all the work on the right side of the tree and most of the nodes on the left (or at least not on the right) Heaps CSE 326 Autumn 2001 26

Null Path Length The null path length (NPL) of a node is the number of nodes between it and a null in the tree npl(null) = -1 npl(leaf) = 0 npl(single-child node) = 0 2 1 1 Before I make this more concrete, let’s get a totally unrelated random definition that we probably will never use. Note that I’m assuming a binary tree here. Null path length is the number of nodes between a given node and a null. Npl of null is -1 npl of a leaf or a node with just one child is 0 Notice that I’m putting the null path length, NOT the values in the nodes. another way of looking at it: NPL is the height of complete subtree rooted at this node 1 Heaps CSE 326 Autumn 2001 27

Leftist Heap Properties Heap-order property parent’s priority value  childrens’ priority values  minimum element is at the root Leftist property  nodes, NPL(left subtree)  NPL(right subtree)  tree is at least as “heavy” on the left as the right So, here are the formal properties of a leftist heap. Heap-order, of course. And, the leftist property; this means that the npl of the right child is never greater than that of the left child. Are leftist trees necessarily complete or balanced? NO. In fact, a _great_ leftist tree is a string of nodes all on the left! Are leftist trees complete? Balanced? Socialist? Heaps CSE 326 Autumn 2001 28

Leftist Tree Examples NOT leftist leftist leftist 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Here are some training examples to help you learn. The red nodes break the leftist property for the first tree. The other two trees are leftist. every subtree of a leftist tree is leftist, comrade! Heaps CSE 326 Autumn 2001 29

Right Path in a Leftist Tree is Short Theorem: If right path-length is at least r, the tree has at least 2r - 1 nodes Proof by induction: Basis: r = 1. Tree has at least one node: 21 - 1 = 1 Inductive step: Assume true for r’< r. Right subtree has a right path of at least r - 1 nodes, so it has at least 2(r-1) - 1 nodes. Left subtree must also have a right path of at least r - 1 (otherwise, there is a null path of r - 3, less than the right subtree). Again, the left has 2(r-1) - 1 nodes. All told then, there are at least: 2(r-1) - 1 + 2(r-1) - 1 + 1 = 2r - 1  Leftist tree with at least n nodes has right path of at most log n nodes Why do we have this leftist property? Because it guarantees that the right path is really short compared to the number of nodes in the tree without being as stodgy as the heap structure property. Here’s a proof which boils down to the fact that if the left path is r long, the null path length must be at least r (otherwise somewhere along there, there’s a node with a larger right npl than left). Since the tree is complete at least to depth r, there are at least 2r - 1 nodes in the tree. Heaps CSE 326 Autumn 2001 30

Merging Two Leftist Heaps Merge(T1,T2) returns one leftist heap containing all elements of the two (distinct) leftist heaps T1 and T2 merge a T1 a merge a < b L1 R1 L1 R1 How do we merge two leftist heaps? We put the smaller root as the new root, hang its left subtree on the left. Recursively merge its right subtree and the other tree. Are we done? No. Because we haven’t ensured that the leftist property is intact. b T2 b L2 R2 L2 R2 Heaps CSE 326 Autumn 2001 31

Merge Continued a a npl(R’) > npl(L1) L1 R’ R’ L1 runtime: So, we check if the leftist property is intact. If it isn’t we switch the right and left children. How long does this take? How much work do we do at each step? CONSTANT And, we recursively traverse the whole right path of both trees. How long are those paths? At most log n. So how much work? O(log n) Pretty cool, huh? R’ = Merge(R1, T2) runtime: Heaps CSE 326 Autumn 2001 32

Operations on Leftist Heaps merge two trees of total size n: O(log n) insert into heap size n: O(log n) pretend node is a size 1 leftist heap insert by merging original heap with one node heap deleteMin with heap size n: O(log n) remove and return root merge left and right subtrees merge Now that we have that wicked merge op. Let’s just use it to implement all the other ops. Is one node of size one leftist heap? merge Heaps CSE 326 Autumn 2001 33

Merge Example merge 3 5 merge 7 10 12 5 5 14 merge 3 10 12 10 12 7 8 8 ? 1 3 5 merge 7 10 12 1 ? 5 5 1 14 merge 3 10 12 10 12 7 8 8 8 Alright, let’s try an example. I’m unwinding the recursion here. I pulled a fast one merging the last two nodes. I could have reduced that to merging a node with null (which is clearly the original node) but I didn’t for convenience. Don’t _you_ do that on tests! 14 8 12

Sewing Up the Example ? ? 2 3 3 3 7 ? 7 1 7 5 1 5 5 14 14 14 10 8 10 10 8 8 No, we need to put the example back together, fixing the leftist property as we go. Are we done here? NO! The root’s right child has a higher npl than its left child! 12 12 12 Done? Heaps CSE 326 Autumn 2001 35

Finally 2 2 3 3 7 5 1 5 1 7 14 10 8 10 8 14 So, we swap the root’s children. Is the final tree a leftist heap? 12 12 Heaps CSE 326 Autumn 2001 36

Iterative Leftist Merge Downward Pass Merge right paths merge 1 5 2 3 10 12 1 7 5 1 3 It turns out that we can do an iterative version of this. First, we traverse down the right side, picking up the lesser node of each tree as we go. 14 10 8 7 8 12 14 Heaps CSE 326 Autumn 2001 37

Iterative Leftist Merge (part deux) 2 2 2 3 3 3 7 5 1 7 5 1 7 5 1 14 10 8 14 10 8 14 10 8 12 12 12 Then, we go back up and fix the leftist property from bottom to top. What do we need to do this iteratively? A STACK of the nodes on the right path! 2 Upward Pass Fix-up Leftist Heap Property 3 5 1 7 10 8 14 What do we need to do leftist merge iteratively? 12

(One More) Amortized Time To write off an expenditure for (office equipment, for example) by prorating over a certain period. time A nonspatial continuum in which events occur in apparently irreversible succession from the past through the present to the future. am·or·tized time Running time limit resulting from writing off expensive runs of an algorithm over multiple cheap runs of the algorithm, usually resulting in a lower overall running time than indicated by the worst possible case. If M operations take total O(M log N) time, amortized time per operation is O(log N) Heaps CSE 326 Autumn 2001 39

Skew Heaps Problems with leftist heaps Solution: skew heaps extra storage for NPL two pass merge (with stack!) extra complexity/logic to maintain and check NPL Solution: skew heaps blind adjusting version of leftist heaps amortized time for merge, insert, and deleteMin is O(log n) worst case time for all three is O(n) merge always switches children when fixing right path iterative method has only one pass There are some problems with leftist heaps. We’ll use skew heaps to fix those problems. Motivation? Remember that we “often” get heavy right sides when we merge. So, why not just assume the right side is heavy and move it over automatically? Heaps CSE 326 Autumn 2001 40

Merging Two Skew Heaps merge T1 a a merge L1 R1 L1 R1 a < b T2 b b This merge should look very similar but simpler than the previous algorithm. L2 R2 L2 R2 Heaps CSE 326 Autumn 2001 41

Skew Heap Example 3 3 merge merge 7 5 7 5 5 merge 14 10 14 12 10 12 10 8 3 8 3 This happens to end up with a leftist tree and start out with leftist trees. Is that guaranteed? 7 8 5 7 14 8 10 14 12

Skew Heap Code void merge(heap1, heap2) { case { heap1 == NULL: return heap2; heap2 == NULL: return heap1; heap1.findMin() < heap2.findMin(): temp = heap1.right; heap1.right = heap1.left; heap1.left = merge(heap2, temp); return heap1; otherwise: return merge(heap2, heap1); } Here’s the code. I won’t go into it except to say that it is _easy_. The iterative version is a touch more intricate but also fairly easy and requires NO STACK! Heaps CSE 326 Autumn 2001 43

Heaps o’ Heaps Binary Heaps d-Heaps Leftist Heaps Skew Heaps Pros Cons Binary Heaps Simple, fast BuildHeap No easy merge d-Heaps Large data sets More complex Leftist Heaps Merge Extra storage Skew Heaps Simple code Good amortized speed Binomial Queues ? ? (check out Weiss 6.8) OK, I said you were being cheated unless you got a comparison of data structures. Who can tell me some good things about each of these? Binary: simple, memory efficient, fast, can run buildHeap. cannot merge quickly. d: Faster than binary heaps. Possibly much faster in some applications. Slightly more complicated. Otherwise, all binary advantages. Leftist: supports merge quickly. Supports all other operations asymptotically quickly. Practically, extra storage, link traversals and recursion make it bigger and slower. Skew: slightly less storage, somewhat faster and simpler than leftist. NOT guaranteed fast on EVERY operation (but guaranteed amortized speed. Binomial queues? Never heard of ‘em. Read up on them in your book! Heaps CSE 326 Autumn 2001 44