1. CSE373: Data Structures & Algorithms Lecture 8: Priority Queues Aaron Bauer Winter 2014

2. Announcements Midterm next week –Midterm review TA session on Tuesday –Shuo extra office hours 12:30-1:30 Monday Homework 1 feedback out soon Winter 20142CSE373: Data Structures & Algorithms

3. Priority Queue ADT Stores elements with data and comparable priorities –“priority 1” is more important than “priority 4” Operations –insert –deleteMin –is_empty Winter 20143CSE373: Data Structures & Algorithms

4. Applications Like all good ADTs, the priority queue arises often –Sometimes blatant, sometimes less obvious Run multiple programs in the operating system –“critical” before “interactive” before “compute-intensive” –Maybe let users set priority level Treat hospital patients in order of severity (or triage) Select print jobs in order of decreasing length? Forward network packets in order of urgency Select most frequent symbols for data compression (cf. CSE143) Sort (first insert all, then repeatedly deleteMin ) –Much like Homework 1 uses a stack to implement reverse Winter 20144CSE373: Data Structures & Algorithms

5. More applications “Greedy” algorithms –May see an example when we study graphs in a few weeks Discrete event simulation (system simulation, virtual worlds, …) –Each event e happens at some time t, updating system state and generating new events e1, …, en at times t+t1, …, t+tn –Naïve approach: advance “clock” by 1 unit at a time and process any events that happen then –Better: Pending events in a priority queue (priority = event time) Repeatedly: deleteMin and then insert new events Effectively “set clock ahead to next event” Winter 20145CSE373: Data Structures & Algorithms

6. Finding a good data structure Will show an efficient, non-obvious data structure for this ADT –But first let’s analyze some “obvious” ideas for n data items –All times worst-case; assume arrays “have room” data insert algorithm / time deleteMin algorithm / time unsorted array unsorted linked list sorted circular array sorted linked list binary search tree AVL tree Winter 20146CSE373: Data Structures & Algorithms add at end O(1) search O(n) add at front O(1) search O(n) search / shift O(n) move front O(1) put in right place O(n) remove at front O(1) put in right place O(n) leftmost O(n) put in right place O( log n) leftmost O( log n)

7. More on possibilities If priorities are random, binary search tree will likely do better –O( log n) insert and O( log n) deleteMin on average One more idea: if priorities are 0, 1, …, k can use array of lists –insert : add to front of list at arr[priority], O(1) –deleteMin : remove from lowest non-empty list O(k) We are about to see a data structure called a “binary heap” –O( log n) insert and O( log n) deleteMin worst-case Possible because we don’t support unneeded operations; no need to maintain a full sort –Very good constant factors –If items arrive in random order, then insert is O(1) on average Winter 20147CSE373: Data Structures & Algorithms

8. Our data structure A binary min-heap (or just binary heap or just heap) is: Structure property: A complete binary tree Heap property: The priority of every (non-root) node is greater than the priority of its parent –Not a binary search tree Winter 20148CSE373: Data Structures & Algorithms 1530 8020 10 not a heap 996040 8020 10 50700 85 a heap So: Where is the highest-priority item? What is the height of a heap with n items?

9. Operations: basic idea findMin : return root.data deleteMin : 1.answer = root.data 2.Move right-most node in last row to root to restore structure property 3.“Percolate down” to restore heap property insert: 1.Put new node in next position on bottom row to restore structure property 2.“Percolate up” to restore heap property Winter 20149CSE373: Data Structures & Algorithms 996040 80 20 10 50700 85 Overall strategy: Preserve structure property Break and restore heap property

10. 10 DeleteMin 34 9857 106911 1. Delete (and later return) value at root node Winter 2014CSE373: Data Structures & Algorithms

11. 11 2. Restore the Structure Property We now have a “hole” at the root –Need to fill the hole with another value When we are done, the tree will have one less node and must still be complete 34 9857 106911 34 9857 10 6911 Winter 2014CSE373: Data Structures & Algorithms

12. 12 3. Restore the Heap Property Percolate down: Keep comparing with both children Swap with lesser child and go down one level Done if both children are  item or reached a leaf node Why is this correct? What is the run time? 34 9857 10 6911 4 9857 10 6911 3 84 91057 6911 3 ? ? Winter 2014CSE373: Data Structures & Algorithms

13. 13 DeleteMin: Run Time Analysis Run time is O(height of heap) A heap is a complete binary tree Height of a complete binary tree of n nodes? –height =  log 2 (n)  Run time of deleteMin is O( log n) Winter 2014CSE373: Data Structures & Algorithms

14. 14 Insert Add a value to the tree Afterwards, structure and heap properties must still be correct 84 91057 6911 1 2 Winter 2014 CSE373: Data Structures & Algorithms

15. 15 Insert: Maintain the Structure Property There is only one valid tree shape after we add one more node So put our new data there and then focus on restoring the heap property 84 91057 6911 1 2 Winter 2014CSE373: Data Structures & Algorithms

16. 16 Maintain the heap property 2 84 91057 6911 1 Percolate up: Put new data in new location If parent larger, swap with parent, and continue Done if parent  item or reached root Why is this correct? What is the run time? ? 2 5 84 9107 6911 1 ? 2 5 8 91047 6911 1 ? Winter 2014CSE373: Data Structures & Algorithms

17. 17 Insert: Run Time Analysis Like deleteMin, worst-case time proportional to tree height –O( log n) But… deleteMin needs the “last used” complete-tree position and insert needs the “next to use” complete-tree position –If “keep a reference to there” then insert and deleteMin have to adjust that reference: O( log n) in worst case –Could calculate how to find it in O( log n) from the root given the size of the heap But it’s not easy And then insert is always O( log n), promised O(1) on average (assuming random arrival of items) There’s a “trick”: don’t represent complete trees with explicit edges! Winter 2014CSE373: Data Structures & Algorithms

18. Review Priority Queue ADT: insert comparable object, deleteMin Binary heap data structure: Complete binary tree where each node has priority value greater than its parent O(height-of-tree)=O( log n) insert and deleteMin operations –insert : put at new last position in tree and percolate-up –deleteMin : remove root, put last element at root and percolate-down But: tracking the “last position” is painful and we can do better Winter 201418CSE373: Data Structures & Algorithms insert deleteMin 6 2 15 23 12 18 45 3 7 996040 8020 10 70050 85

19. Winter 201419 Array Representation of Binary Trees G ED CB A JKHI F L From node i : left child: i*2 right child: i*2+1 parent: i/2 (wasting index 0 is convenient for the index arithmetic) 7 1 23 45 6 98101112 ABCDEFGHIJKL 012345678910111213 implicit (array) implementation: CSE373: Data Structures & Algorithms

20. Judging the array implementation Plusses: Non-data space: just index 0 and unused space on right –In conventional tree representation, one edge per node (except for root), so n-1 wasted space (like linked lists) –Array would waste more space if tree were not complete Multiplying and dividing by 2 is very fast (shift operations in hardware) Last used position is just index size Minuses: Same might-be-empty or might-get-full problems we saw with stacks and queues (resize by doubling as necessary) Plusses outweigh minuses: “this is how people do it” Winter 201420CSE373: Data Structures & Algorithms

21. Pseudocode: insert Winter 201421CSE373: Data Structures & Algorithms void insert(int val) { if(size==arr.length-1) resize(); size++; i=percolateUp(size,val); arr[i] = val; } int percolateUp(int hole, int val) { while(hole > 1 && val < arr[hole/2]) arr[hole] = arr[hole/2]; hole = hole / 2; } return hole; } 996040 8020 10 70050 85 1020804060859970050 012345678910111213 This pseudocode uses ints. In real use, you will have data nodes with priorities.

22. Pseudocode: deleteMin Winter 201422CSE373: Data Structures & Algorithms int deleteMin() { if(isEmpty()) throw… ans = arr[1]; hole = percolateDown (1,arr[size]); arr[hole] = arr[size]; size--; return ans; } int percolateDown(int hole, int val) { while(2*hole <= size) { left = 2*hole; right = left + 1; if(right > size || arr[left] < arr[right]) target = left; else target = right; if(arr[target] < val) { arr[hole] = arr[target]; hole = target; } else break; } return hole; } 996040 8020 10 70050 85 1020804060859970050 012345678910111213 This pseudocode uses ints. In real use, you will have data nodes with priorities.

23. Example 1.insert: 16, 32, 4, 67, 105, 43, 2 2.deleteMin Winter 201423CSE373: Data Structures & Algorithms 01234567

24. Example 1.insert: 16, 32, 4, 67, 105, 43, 2 2.deleteMin Winter 201424CSE373: Data Structures & Algorithms 16 01234567

25. Example 1.insert: 16, 32, 4, 67, 105, 43, 2 2.deleteMin Winter 201425CSE373: Data Structures & Algorithms 1632 01234567 16 32

26. Example 1.insert: 16, 32, 4, 67, 105, 43, 2 2.deleteMin Winter 201426CSE373: Data Structures & Algorithms 43216 01234567 4 3216

27. Example 1.insert: 16, 32, 4, 67, 105, 43, 2 2.deleteMin Winter 201427CSE373: Data Structures & Algorithms 4321667 01234567 4 3216 67

28. Example 1.insert: 16, 32, 4, 67, 105, 43, 2 2.deleteMin Winter 201428CSE373: Data Structures & Algorithms 4321667105 01234567 4 3216 10567

29. Example 1.insert: 16, 32, 4, 67, 105, 43, 2 2.deleteMin Winter 201429CSE373: Data Structures & Algorithms 2324671054316 01234567 2 32 4 164310567

30. Other operations decreaseKey : given pointer to object in priority queue (e.g., its array index), lower its priority value by p –Change priority and percolate up increaseKey : given pointer to object in priority queue (e.g., its array index), raise its priority value by p –Change priority and percolate down remove : given pointer to object in priority queue (e.g., its array index), remove it from the queue –decreaseKey with p = , then deleteMin Running time for all these operations? Winter 201430CSE373: Data Structures & Algorithms

31. Build Heap Suppose you have n items to put in a new (empty) priority queue –Call this operation buildHeap n insert s works –Only choice if ADT doesn’t provide buildHeap explicitly –O(n log n) Why would an ADT provide this unnecessary operation? –Convenience –Efficiency: an O(n) algorithm called Floyd’s Method –Common issue in ADT design: how many specialized operations Winter 201431CSE373: Data Structures & Algorithms

32. Floyd’s Method 1.Use n items to make any complete tree you want –That is, put them in array indices 1,…,n 2.Treat it as a heap and fix the heap-order property –Bottom-up: leaves are already in heap order, work up toward the root one level at a time Winter 201432CSE373: Data Structures & Algorithms void buildHeap() { for(i = size/2; i>0; i--) { val = arr[i]; hole = percolateDown(i,val); arr[hole] = val; }

33. Example In tree form for readability –Purple for node not less than descendants heap-order problem –Notice no leaves are purple –Check/fix each non-leaf bottom-up (6 steps here) Winter 201433CSE373: Data Structures & Algorithms 6718 92103 115 12 4

34. Example Winter 201434CSE373: Data Structures & Algorithms 6718 92103 115 12 46718 92103 115 12 4 Step 1 Happens to already be less than children (er, child)

35. Example Winter 201435CSE373: Data Structures & Algorithms 6718 92103 115 12 4 Step 2 Percolate down (notice that moves 1 up) 67108 9213 115 12 4

36. Example Winter 201436CSE373: Data Structures & Algorithms Step 3 Another nothing-to-do step 67108 9213 115 12 467108 9213 115 12 4

37. Example Winter 201437CSE373: Data Structures & Algorithms Step 4 Percolate down as necessary (steps 4a and 4b) 117108 9613 25 12 467108 9213 115 12 4

38. Example Winter 201438CSE373: Data Structures & Algorithms Step 5 117108 9653 21 12 4117108 9613 25 12 4

39. Example Winter 201439CSE373: Data Structures & Algorithms Step 6 117108 9654 23 1 12117108 9653 21 12 4

40. But is it right? “Seems to work” –Let’s prove it restores the heap property (correctness) –Then let’s prove its running time (efficiency) Winter 201440CSE373: Data Structures & Algorithms void buildHeap() { for(i = size/2; i>0; i--) { val = arr[i]; hole = percolateDown(i,val); arr[hole] = val; }

41. Correctness Loop Invariant: For all j > i, arr[j] is less than its children True initially: If j > size/2, then j is a leaf –Otherwise its left child would be at position > size True after one more iteration: loop body and percolateDown make arr[i] less than children without breaking the property for any descendants So after the loop finishes, all nodes are less than their children Winter 201441CSE373: Data Structures & Algorithms void buildHeap() { for(i = size/2; i>0; i--) { val = arr[i]; hole = percolateDown(i,val); arr[hole] = val; }

42. Efficiency Easy argument: buildHeap is O(n log n) where n is size size/2 loop iterations Each iteration does one percolateDown, each is O( log n) This is correct, but there is a more precise (“tighter”) analysis of the algorithm… Winter 201442CSE373: Data Structures & Algorithms void buildHeap() { for(i = size/2; i>0; i--) { val = arr[i]; hole = percolateDown(i,val); arr[hole] = val; }

43. Efficiency Better argument: buildHeap is O(n) where n is size size/2 total loop iterations: O(n) 1/2 the loop iterations percolate at most 1 step 1/4 the loop iterations percolate at most 2 steps 1/8 the loop iterations percolate at most 3 steps … ((1/2) + (2/4) + (3/8) + (4/16) + (5/32) + …) < 2 (page 4 of Weiss) –So at most 2(size/2) total percolate steps: O(n) Winter 201443CSE373: Data Structures & Algorithms void buildHeap() { for(i = size/2; i>0; i--) { val = arr[i]; hole = percolateDown(i,val); arr[hole] = val; }

44. Lessons from buildHeap Without buildHeap, our ADT already let clients implement their own in  O(n log n) worst case –Worst case is inserting lower priority values later By providing a specialized operation internal to the data structure (with access to the internal data), we can do O(n) worst case –Intuition: Most data is near a leaf, so better to percolate down Can analyze this algorithm for: –Correctness: Non-trivial inductive proof using loop invariant –Efficiency: First analysis easily proved it was O(n log n) Tighter analysis shows same algorithm is O(n) Winter 201444CSE373: Data Structures & Algorithms

45. Other branching factors d-heaps: have d children instead of 2 –Makes heaps shallower, useful for heaps too big for memory (or cache) Homework: Implement a 3-heap –Just have three children instead of 2 –Still use an array with all positions from 1…heap-size used Winter 201445CSE373: Data Structures & Algorithms IndexChildren Indices 12,3,4 25,6,7 38,9,10 411,12,13 514,15,16 ……

46. What we are skipping merge: given two priority queues, make one priority queue –How might you merge binary heaps: If one heap is much smaller than the other? If both are about the same size? –Different pointer-based data structures for priority queues support logarithmic time merge operation (impossible with binary heaps) Leftist heaps, skew heaps, binomial queues Worse constant factors Trade-offs! Winter 201446CSE373: Data Structures & Algorithms