1. www.monash.edu.au 1 COMMONWEALTH OF AUSTRALIA Copyright Regulations 1969 WARNING This material has been reproduced and communicated to you by or on behalf of Monash University pursuant to Part VB of the Copyright Act 1968 (the Act). The material in this communication may be subject to copyright under the Act. Any further reproduction or communication of this material by you may be the subject of copyright protection under the Act. Do not remove this notice.

2. www.monash.edu.au FIT2004 Algorithms & Data Structures L4: Complexity of Sorting Prepared by: Bernd Meyer February 2007

3. www.monash.edu.au 3 Reminder: Big-O Rationale: –We are interested in the Growth of the function, not its absolute value –we want to understand what happens in the limit…

4. www.monash.edu.au 4 Big-Oh Notation Given functions f(n) and g(n), we say that f(n) is O(g(n)) if there are positive constants c and n 0 such that f(n)  cg(n) for n  n 0 Example: 2n  10 is O(n) –2n  10  cn –(c  2) n  10 –n  10  (c  2) –Pick c  3 and n 0  10

5. www.monash.edu.au 5 Big-Oh Notation Given functions f(n) and g(n), we say that f(n) is O(g(n)) if there are positive constants c and n 0 such that f(n)  cg(n) for n  n 0 Example: 2n  10 is O(n) –2n  10  cn –(c  2) n  10 –n  10  (c  2) –Pick c  3 and n 0  10

6. www.monash.edu.au 6 Seven Important Functions Seven functions that often appear in algorithm analysis: –Constant  1 –Logarithmic  log n –Linear  n –N-Log-N  n log n –Quadratic  n 2 –Cubic  n 3 –Exponential  2 n In a log-log chart, the slope of the line corresponds to the growth rate of the function

7. www.monash.edu.au 7 Relatives of Big-Oh big-Omega –f(n) is  (g(n)) if there is a constant c > 0 and an integer constant n 0  1 such that f(n)  cg(n) for n  n 0 big-Theta –f(n) is  (g(n)) if there are constants c’ > 0 and c’’ > 0 and an integer constant n 0  1 such that c’g(n)  f(n)  c’’g(n) for n  n 0

8. www.monash.edu.au 8 Quicksort, Mergesort Worst CaseAverage Case QuickO(n 2 )O(n log n) MergeO(n log n)O(n log n)

9. www.monash.edu.au 9 Sort Informal Reasoning SortSort SortSortSortSort 2*n/2 n 4*n/4 Depth: log n (this is a perfect binary tree)

10. www.monash.edu.au 10 Formal Reasoning for Quicksort T(N) is the runtime for sorting N elements Define T(N) recursively T(N)=T(I)+T(N-I-1)+cN I elements in the first partition N-I-1 elements in the second partition (+pivot) cN is the time to partition N elements Did this help????

11. www.monash.edu.au 11 Worst Case Quicksort Pivot is always the smallest element T(N)=T(N-1)+cN T(N-1)=T(N-2)+c(N-1) T(N-2)= T(N-3)+c(N-2) … T(2)=T(1)+c*2 T(1)=1 Obviously

12. www.monash.edu.au 12 Best Case Quicksort Pivot is always in the middle T(N)=2*T(N/2)+cN … Not so obviously

13. www.monash.edu.au 13 Best Case Quicksort (version 1)

14. www.monash.edu.au 14 Best Case Quicksort (version 2, explicit substitution)

15. www.monash.edu.au 15 Average Case Quicksort The pivot is equally likely to be any element in order So all sizes of partitions are equally likely WHY????

16. www.monash.edu.au 16 Solving Recurrence Relations Recurrence Relations are functions defined by an expression using the same function. T(N) above is an example of a recurrence relation How do we solve them if the solution is not obvious? –Guess & prove (usually by induction often using “smoothness”) –Explicit substitution (aka “telescoping”) –using the D&C Master equation … details see Tutorial 3

17. www.monash.edu.au 17 D&C Master Equation The Master Equation for Divide-and-Coquer (provided we split into equal parts) is This has three solutions It can be solved by backward substitution. Details see Tutorial 3

18. www.monash.edu.au 18 Full History Recurrence Relations Some recurrence relations depend on all previous values of the function… A simple form of full history recurrence relation is This has an upper bound where H(n) is the harmonic series which can be approximated as thus

19. www.monash.edu.au 19 Quicksort Average Runtime Quicksort’s T(n) is a full history relation

20. www.monash.edu.au 20 Heapsort Analysis Recall: Heapsort can be done by generating a heap and then N times deleteMin() determine run-time: deleteMin determine run-time: heap generation we know that tree height of heap is O(log N) deleteMin time is linear in heap height O(log N)

21. www.monash.edu.au 21 Naïve Heapsort Bound generate heap by N successive inserts insert is O(log N) worst case deleteMin is O(log n) heapsort = insert all + delete all

22. www.monash.edu.au 22 Heap Building (bottom-up) Is heap building itself already O(N log N)? No --- we can do this in linear (!!) time. Recall: To create a heap from N items, we first create an arbitrary complete binary tree and then apply percolateDown() to each non-leaf node bottom-up. This effectively creates small “sub-heaps” that are combined bottom-up.

23. www.monash.edu.au 23 Building a Heap bottom-up To obtain the runtime, we need to count the number of dashes. This is bound on the number of operations in percolateDown() (Each dash stands for two comparisons)

24. www.monash.edu.au 24 Building a Heap bottom-up

25. www.monash.edu.au 25 Building a Heap bottom-up

26. www.monash.edu.au 26 Building a Heap bottom-up In total we have the following percolations to build a heap with N-1 nodes: 1 from height h=(log N)-1 2 from height h-1, 4 from height h-2, … … (N/4) from height 1

27. www.monash.edu.au 27 Heap Building is linear

28. www.monash.edu.au 28 Heap Building is linear

29. www.monash.edu.au 29 Heap Building is linear

30. www.monash.edu.au 30 Heap Building is linear

31. www.monash.edu.au 31 Heap Building is linear

32. www.monash.edu.au 32 Heap Sort worst case Heapsort is dominated by the N deleteMin Operations. Even if we can build the heap bottom-up in O(N) we still need O(N log N) for these. Heapsort is, in a sense, superior to Quick &Mergesort –Unlike Quicksort it is O(N log N) in the worst case –Unlike normal Mergesort it can sort in-situ –However, the proportionality constant is higher than Quicksort’s and so in practice quicksort is usually used.

33. www.monash.edu.au 33 Bottom-up Heapsort Can we improve Heapsort’s runtime further? We try to improve percolateDown() We can observe that 50% of the heap nodes are on the last level, 25% one level higher etc. Thus the vast majority of nodes is at the bottom. On average percolateDown() will thus have to run almost to a leaf node. Note that percolateDown makes two decisions in each iteration (two comparisons): –whether two continue –if so, whether to go left or right

34. www.monash.edu.au 34 Recall: percolateDown()

35. www.monash.edu.au 35 Bottom-up Heapsort Note that percolateDown makes two decisions in each iteration (two comparisons): –whether two continue –if so, whether to go left or right Bottom-up Heapsort separates these decisions: to percolateDown root element e –Step 1: Determine find the full potential target path (top down to a leaf) (only making the “left or right” decision) –Step 2: traverse the target path bottom-up until the final position of e found (only making the “whether to continue” decision) –Step 3: shift all elements on the target path from this position one level up –insert the original root element e into the whole thus created.

36. www.monash.edu.au 36 Why is Bottom-up Heapsort faster? Because the vast majority of the heap nodes is on the bottom levels, the second loop (bottom-up traversal of the target path) terminates most often after 1 or 2 iterations We thus save a significant number of comparisons Studies show that Bottom-up Heapsort is faster than quicksort for large datasets. Step 1 Step 2 Step 3 insert 14

37. www.monash.edu.au 37 Lower Bound for Sorting Can we be any faster than O(N log N)? On what can we base the analysis? All our sorting algorithms are based on comparisons (usually 2-way, but could be n-way) A comparison-based algorithm in general can be represented by a decision tree.

38. www.monash.edu.au 38 Decision Tree to sort three items a, b, c leaves are the complete sorted lists algorithm must follow a path from root to leaf each node requires one comparison

39. www.monash.edu.au 39 Lower Bound for Comparsion-based Sorting There must be n! leaves in the decision tree to represent all possible sorting orders A binary tree with k leaves has has at least height log 2 k The algorithm must follow a path from the root to a leaf Each node requires at least one comparison