1. O() Analysis of Methods and Data Structures Reasonable vs. Unreasonable Algorithms Using O() Analysis in Design

2. Now Available Online! http://www.coursesurvey.gatech.edu

3. O() Analysis of Methods and Data Structures

4. The Scenario We’ve talked about data structures and methods to act on these structures… –Linked lists, arrays, trees –Inserting, Deleting, Searching, Traversal, Sorting, etc. Now that we know about O() notation, let’s discuss how each of these methods perform on these data structures!

5. Recipe for Determining O() Break algorithm down into “known” pieces –We’ll learn the Big-Os in this section Identify relationships between pieces –Sequential is additive –Nested (loop / recursion) is multiplicative Drop constants Keep only dominant factor for each variable

6. Array Size and Complexity How can an array change in size? MAX is 30 ArrayType definesa array[1..MAX] of Num We need to know what N is in advance to declare an array, but for analysis of complexity, we can still use N as a variable for input size.

7. Traversals Traversals involve visiting every element in a collection. Because we must visit every node, a traversal must be O(N) for any data structure. –If we visit less than N elements, then it is not a traversal.

8. Comparing Data Structures and Methods Data StructureTraverse Unsorted L ListN Sorted L ListN Unsorted ArrayN Sorted ArrayN Binary TreeN BSTN F&B BSTN LB

9. Searching for an Element Searching involves determining if an element is a member of the collection. Simple/Linear Search: –If there is no ordering in the data structure –If the ordering is not applicable Binary Search: –If the data is ordered or sorted –Requires non-linear access to the elements

10. Simple Search Worst case: the element to be found is the N th element examined. Simple search must be used for: –Sorted or unsorted linked lists –Unsorted array –Binary tree –Binary Search Tree if it is not full and balanced

11. Comparing Data Structures and Methods Data StructureTraverseSearch Unsorted L ListNN Sorted L ListNN Unsorted ArrayNN Sorted ArrayN Binary TreeNN BSTNN F&B BSTN ? LB

12. Balanced Binary Search Trees If a binary search tree is not full, then in the worst case it takes on the structure and characteristics of a sorted linked list with N/2 elements. 7 11 14 42 58

13. Binary Search Trees If a binary search tree is not full or balanced, then in the worst case it takes on the structure and characteristics of a sorted linked list. 7 11 14 42 58

14. Example: Linked List Let’s determine if the value 83 is in the collection: 519 35 42 \\ 83 Not Found! Head

15. Simple/Linear Search Algorithm cur <- head loop exitif(cur = NIL) OR (cur^.data = target) cur <- cur^.next endloop if(cur <> NIL) then print( “Yes, target is there” ) else print( “No, target isn’t there” ) endif

16. Pre-Order Search Traversal Algorithm As soon as we get to a node, check to see if we have a match Otherwise, look for the element in the left sub-tree Otherwise, look for the element in the right sub-tree 14 Left ??? Right ???

17. 943667 22 1493 Find 9 cur If I have to watch one more of these I think I’m going to die. LB

18. Big-O of Simple Search The algorithm has to examine every element in the collection –To return a false –If the element to be found is the N th element Thus, simple search is O(N).

19. Binary Search We may perform binary search on –Sorted arrays –Full and balanced binary search trees Tosses out ½ the elements at each comparison.

20. Full and Balanced Binary Search Trees Contains approximately the same number of elements in all left and right sub-trees (recursively) and is fully populated. 34 25 2129 45 4152

21. Binary Search Example 71242597186104212 Looking for 89

22. Binary Search Example 71242597186104212 Looking for 89

23. Binary Search Example 71242597186104212 Looking for 89

24. Binary Search Example 71242597186104212 89 not found – 3 comparisons 3 = Log(8)

25. Binary Search Big-O An element can be found by comparing and cutting the work in half. –We cut work in ½ each time –How many times can we cut in half? –Log 2 N Thus binary search is O(Log N).

26. What? N 2 4 8 16 32 64 128 256 512 LB Searches 1 2 3 4 5 6 7 8 9

27. What? log 2 (N) log 2 (2) log 2 (4) log 2 (8) log 2 (16) log 2 (32) log 2 (64) log 2 (128) log 2 (256) log 2 (512) LB Searches 1 2 3 4 5 6 7 8 9 ==================

28. CS Notation lg(N) lg(2) lg(4) lg(8) lg(16) lg(32) lg(64) lg(128) lg(256) lg(512) LB Searches 1 2 3 4 5 6 7 8 9 ==================

29. Recall LB log 2 N = k log 10 N k = 0.30103... So: O(lg N) = O(log N)

30. Comparing Data Structures and Methods Data StructureTraverseSearch Unsorted L ListNN Sorted L ListNN Unsorted ArrayNN Sorted ArrayNLog N Binary TreeNN BSTNN F&B BSTNLog N LB

31. Insertion Inserting an element requires two steps: –Find the right location –Perform the instructions to insert If the data structure in question is unsorted, then it is O(1) –Simply insert to the front –Simply insert to end in the case of an array –There is no work to find the right spot and only constant work to actually insert.

32. Comparing Data Structures and Methods Data StructureTraverseSearchInsert Unsorted L ListNN1 Sorted L ListNN Unsorted ArrayNN1 Sorted ArrayNLog N Binary TreeNN1 BSTNN F&B BSTNLog N LB

33. Insert into a Sorted Linked List Finding the right spot is O(N) –Recurse/iterate until found Performing the insertion is O(1) –4-5 instructions Total work is O(N + 1) = O(N)

34. Inserting into a Sorted Array Finding the right spot is O(Log N) –Binary search on the element to insert Performing the insertion –Shuffle the existing elements to make room for the new item

35. Shuffling Elements 5 12 35 77 101 Note – we must have at least one empty cell Insert 29

36. Big-O of Shuffle 5 12 Worst case: inserting the smallest number 101 77 35 Would require moving N elements… Thus shuffle is O(N)

37. Big-O of Inserting into Sorted Array Finding the right spot is O(Log N) Performing the insertion (shuffle) is O(N) Sequential steps, so add: Total work is O(Log N + N) = O(N)

38. Comparing Data Structures and Methods Data StructureTraverseSearchInsert Unsorted L ListNN1 Sorted L ListNNN Unsorted ArrayNN1 Sorted ArrayNLog NN Binary TreeNN1 BSTNN F&B BSTNLog N LB

39. Inserting into a Full and Balanced BST Always insert when current = NIL. Find the right spot –Binary search on the element to insert Perform the insertion –4-5 instructions to create & add node

40. Full and Balanced BST Insert 12 3 7 41 98 Add 4 352

41. Full and Balanced BST Insert 12 3 7 41 98 352 4

42. Big-O of Full & Balanced BST Insert Find the right spot is O(Log N) Performing the insertion is O(1) Sequential, so add: Total work is O(Log N + 1) = O(Log N)

43. Comparing Data Structures and Methods Data StructureTraverseSearchInsert Unsorted L ListNN1 Sorted L ListNNN Unsorted ArrayNN1 Sorted ArrayNLog NN Binary TreeNN1 BSTNN F&B BSTNLog NLog N LB

44. What about a BST Not full & balanced? LB

45. Comparing Data Structures and Methods Data StructureTraverseSearchInsert Unsorted L ListNN1 Sorted L ListNNN Unsorted ArrayNN1 Sorted ArrayNLog NN Binary TreeNN1 BSTNNN F&B BSTNLog NLog N LB

46. Two Sorting Algorithms Bubblesort –Brute-force method of sorting –Loop inside of a loop Mergesort –Divide and conquer approach –Recursively call, splitting in half –Merge sorted halves together

47. Bubblesort Review Bubblesort works by comparing and swapping values in a list 5 12 3542 77 101 1 2 3 4 5 6

48. Bubblesort Review Bubblesort works by comparing and swapping values in a list 77 1235 42 5 1 2 3 4 5 6 101 Largest value correctly placed

49. procedure Bubblesort(A isoftype in/out Arr_Type) to_do, index isoftype Num to_do <- N – 1 loop exitif(to_do = 0) index <- 1 loop exitif(index > to_do) if(A[index] > A[index + 1]) then Swap(A[index], A[index + 1]) endif index <- index + 1 endloop to_do <- to_do - 1 endloop endprocedure // Bubblesort to_do N-1

50. Analysis of Bubblesort How many comparisons in the inner loop? –to_do goes from N-1 down to 1, thus –(N-1) + (N-2) + (N-3) +... + 2 + 1 –Average: N/2 for each “pass” of the outer loop. How many “passes” of the outer loop? –N – 1

51. Bubblesort Complexity Look at the relationship between the two loops: –Inner is nested inside outer –Inner will be executed for each iteration of outer Therefore the complexity is: O((N-1)*(N/2)) = O(N 2 /2 – N/2) = O(N 2 ) LB

52. Graphically N-1 2N-1 LB

53. O(N 2 ) Runtime Example Assume you are sorting 250,000,000 items N = 250,000,000 N 2 = 6.25 x 10 16 If you can do one operation per nanosecond (10 -9 sec) which is fast! It will take 6.25 x 10 7 seconds So 6.25 x 10 7 60 x 60 x 24 x 365 = 1.98 years

54. 674523146339842 674523146339842 45231498 23984514 6763342 6763342 239845146764233 14234598 6334267 614233342456798 Mergesort Log N

55. Analysis of Mergesort Phase I –Divide the list of N numbers into two lists of N/2 numbers –Divide those lists in half until each list is size 1 Log N steps for this stage. Phase II –Build sorted lists from the decomposed lists –Merge pairs of lists, doubling the size of the sorted lists each time Log N steps for this stage.

56. Analysis of the Merging 239845146763342 23984514 6764233 Merge 14234598 6334267 Merge 614233342456798 Merge

57. Mergesort Complexity Each of the N numerical values is compared or copied during each pass –The total work for each pass is O(N). –There are a total of Log N passes Therefore the complexity is: O(Log N + N * Log N) = O (N * Log N) Break apartMerging

58. O(NLogN) Runtime Example Assume same 250,000,000 items N*Log(N) = 250,000,000 x 8.3 = 2, 099, 485, 002 With the same processor as before 2 seconds

59. Summary You now have the O() for basic methods on varied data structures. You can combine these in more complex situations. Break algorithm down into “known” pieces Identify relationships between pieces –Sequential is additive –Nested (loop/recursion) is multiplicative Drop constants Keep only dominant factor for each variable

60. Questions?

61. Reasonable vs. Unreasonable Algorithms

62. Reasonable vs. Unreasonable Reasonable algorithms have polynomial factors –O (Log N) –O (N) –O (N K ) where K is a constant Unreasonable algorithms have exponential factors –O (2 N ) –O (N!) –O (N N )

63. Algorithmic Performance Thus Far Some examples thus far: –O(1) Insert to front of linked list –O(N) Simple/Linear Search –O(N Log N)MergeSort –O(N 2 )BubbleSort But it could get worse: –O(N 5 ), O(N 2000 ), etc.

64. An O(N 5 ) Example For N = 256 N 5 = 256 5 = 1,100,000,000,000 If we had a computer that could execute a million instructions per second… 1,100,000 seconds = 12.7 days to complete But it could get worse…

65. The Power of Exponents A rich king and a wise peasant…

66. The Wise Peasant’s Pay Day(N)Pieces of Grain 12 24 38 416... 2N2N 63 9,223,000,000,000,000,000 64 18,450,000,000,000,000,000

67. How Bad is 2 N ? Imagine being able to grow a billion (1,000,000,000) pieces of grain a second… It would take –585 years to grow enough grain just for the 64 th day –Over a thousand years to fulfill the peasant’s request! ?

68. So the King cut off the peasant’s head. LB

69. The Towers of Hanoi A B C Goal: Move stack of rings to another peg –Rule 1: May move only 1 ring at a time –Rule 2: May never have larger ring on top of smaller ring

70. Original State Move 1 Move 2 Move 3 Move 4 Move 5 Move 6 Move 7 Towers of Hanoi: Solution

71. Towers of Hanoi - Complexity For 3 rings we have 7 operations. In general, the cost is 2 N – 1 = O(2 N ) Each time we increment N, we double the amount of work. This grows incredibly fast!

72. Towers of Hanoi (2 N ) Runtime For N = 64 2 N = 2 64 = 18,450,000,000,000,000,000 If we had a computer that could execute a million instructions per second… It would take 584,000 years to complete But it could get worse…

73. The Bounded Tile Problem Match up the patterns in the tiles. Can it be done, yes or no?

74. The Bounded Tile Problem Matching tiles

75. Tiling a 5x5 Area 25 available tiles remaining

76. Tiling a 5x5 Area 24 available tiles remaining

77. Tiling a 5x5 Area 23 available tiles remaining

78. Tiling a 5x5 Area 22 available tiles remaining

79. Tiling a 5x5 Area 2 available tiles remaining

80. Analysis of the Bounded Tiling Problem Tile a 5 by 5 area (N = 25 tiles) 1st location: 25 choices 2nd location: 24 choices And so on… Total number of arrangements: –25 * 24 * 23 * 22 * 21 *.... * 3 * 2 * 1 –25! (Factorial) = 15,500,000,000,000,000,000,000,000 Bounded Tiling Problem is O(N!)

81. Tiling (N!) Runtime For N = 25 25! = 15,500,000,000,000,000,000,000,000 If we could “place” a million tiles per second… It would take 470 billion years to complete Why not a faster computer?

82. A Faster Computer If we had a computer that could execute a trillion instructions per second (a million times faster than our MIPS computer)… 5x5 tiling problem would take 470,000 years 64-ring Tower of Hanoi problem would take 213 days Why not an even faster computer!

83. The Fastest Computer Possible? What if: –Instructions took ZERO time to execute –CPU registers could be loaded at the speed of light These algorithms are still unreasonable! The speed of light is only so fast!

84. Where Does this Leave Us? Clearly algorithms have varying runtimes. We’d like a way to categorize them: –Reasonable, so it may be useful –Unreasonable, so why bother running

85. Performance Categories of Algorithms Sub-linear O(Log N) Linear O(N) Nearly linear O(N Log N) Quadratic O(N 2 ) Exponential O(2 N ) O(N!) O(N N ) Polynomial

86. Reasonable vs. Unreasonable Reasonable algorithms have polynomial factors –O (Log N) –O (N) –O (N K ) where K is a constant Unreasonable algorithms have exponential factors –O (2 N ) –O (N!) –O (N N )

87. Reasonable vs. Unreasonable Reasonable algorithms May be usable depending upon the input size Unreasonable algorithms Are impractical and useful to theorists Demonstrate need for approximate solutions Remember we’re dealing with large N (input size)

88. Two Categories of Algorithms 2 4 8 16 32 64 128 256 512 1024 Size of Input (N) 10 35 10 30 10 25 10 20 10 15 trillion billion million 1000 100 10 N N5N5 2N2N N Unreasonable Don’t Care! Reasonable Runtime

89. Summary Reasonable algorithms feature polynomial factors in their O() and may be usable depending upon input size. Unreasonable algorithms feature exponential factors in their O() and have no practical utility.

90. Questions?

91. Using O() Analysis in Design

92. Air Traffic Control Coast, add, delete Conflict Alert

93. Problem Statement What data structure should be used to store the aircraft records for this system? Normal operations conducted are: –Data Entry: adding new aircraft entering the area –Radar Update: input from the antenna –Coast: global traversal to verify that all aircraft have been updated [coast for 5 cycles, then drop] –Query: controller requesting data about a specific aircraft by location –Conflict Analysis: make sure no two aircraft are too close together

94. Air Traffic Control System ProgramAlgorithmFreq 1. Data Entry / ExitInsert15 2. Radar Data UpdateN*Search12 3. Coast / DropTraverse60 4. QuerySearch 1 5. Conflict AnalysisTraverse*Search12 LLULLSAUASBTF/B BST #11N1N1LogN #2N^2 NlogNN^2NlogN #3NNNNNN #4NNNLogNN #5N^2 NlogNN^2NlogN

95. Questions?