1. Midterm Review 15-211 Fundamental Data Structures and Algorithms Margaret Reid-Miller 2 March 2004

2. Midterm Mechanics

3. 3 Midterm mechanics  Worth a total of 125 points  Closed books, closed computers  One sheet of notes allowed  If you have a question, raise your hand and stay in your seat

4. Basic Data Structures

5. 5  Linked List  Stack  Queue  Tree  Height of tree, Depth of node, Level  Perfect, Complete, Full  Min & Max number of nodes

6. 6 Java - Implementations  Interfaces vs Abstract Classes vs Classes  Induction: base case vs inductive case  Recursion  Recurrence Relations  Object oriented classes  Invariants:  Loop  Recursion  Representation  Persistent vs Destructive  Sentinels

7. Recurrences

8. 8 Example: list reversal  The reversal of a list L is:  L, if L is empty  M.append(n), otherwise  where n is the first element of L, and  M is the reversal of the tail of L some constant- time steps

9. 9 Recurrence Relations  E.g., T(n) = T(n-1) + n/2  Solve by repeated substitution  Solve resulting series  Prove by guessing and substitution  Divide & Conquer Theorem  T(N) = aT(N/c) + bN

10. 10 Solving recurrence equations Repeated substitution: t(n) = n + t(n-1) = n + (n-1) + t(n-2) = n + (n-1) + (n-2) + t(n-3) and so on… = n + (n-1) + (n-2) + (n-3) + … + 1

11. 11 Incrementing series  This is an arithmetic series that comes up over and over again, because characterizes many nested loops: for (i=1; i<n; i++) { for (j=1; j<i; j++) { f(); }

12. 12 Visualizing it n n 0123… 1 2 3 … Area: n 2 /2 Area of the leftovers: n/2 So: n 2 /2 + n/2 = (n 2 +n)/2 = n(n+1)/2

13. 13 Doubling summation Like the incrementing summation, sums of powers of 2 are also encountered frequently in computer science. What is the closed-form solution for this sum? Prove your answer by induction.

14. 14 Visualizing it Imagine filling a glass by halves… 2 n-1 2 n-2 2 n-3 2 n-4 2 n-5

15. 15 Divide-and-Conquer Theorem  Theorem: Let a, b, c  0.  The recurrence relation  T(1) = b  T(N) = aT(N/c) + bN  for any N which is a power of c  has upper-bound solutions  T(N) = O(N)if a<c  T(N) = O(Nlog N)if a=c  T(N) = O(N log c a )if a>c a=2, b=1, c=2 for recursive sorting

16. Asymptotics

17. 17 Performance and Scaling n 100n sec7n 2 sec2 n sec 1 100 s7 s2 s 5.5 ms 175 s32 s 101 ms.7 ms1 ms 454.5 ms14 ms1 year 100100 ms7 sec10 16 year 1,0001 sec12 min-- 10,00010 sec20 hr-- 1,000,0001.6 min.22 year--

18. 18 “Big-Oh” notation N c  f(N) T(N) n0n0 running time T(N) = O(f(N)) “T(N) is order f(N)”

19. 19 Upper And Lower Bounds f(n) = O( g(n) )Big-Oh f(n) ≤ c g(n) for some constant c and n > n 0 f(n) =  ( g(n) ) Big-Omega f(n) ≥ c g(n) for some constant c and n > n 0 f(n) =  ( g(n) ) Theta f(n) = O( g(n) ) and f(n) =  ( g(n) )

20. 20 Upper And Lower Bounds f(n) = O( g(n) )Big-Oh Can only be used for upper bounds. f(n) =  ( g(n) ) Big-Omega Can only be used for lower bounds f(n) =  ( g(n) ) Theta Pins down the running time exactly (up to a multiplicative constant).

21. 21 Big-O characteristic  Low-order terms “don’t matter”:  Suppose T(N) = 20n 3 + 10nlog n + 5  Then T(N) = O(n 3 )  Question:  What constants c and n 0 can be used to show that the above is true?  Answer: c=35, n 0 =1

22. 22 Big-O characteristic  The bigger task always dominates eventually.  If T1(N) = O(f(N)) and T2(N) = O(g(N)).  Then T1(N) + T2(N) = max( O(f(N)), O(g(N) ).  Also:  T 1 (N)  T 2 (N) = O( f(N)  g(N) ).

23. 23 Some common functions

24. Dictionaries

25. 25 Dictionary  Operations:  Insert  Delete  Find  Implementations:  Binary Search Tree  AVL Tree  Trie  Hash

26. 26 Binary Search Trees (BST)  “Perfect” BST:  Height? Number of nodes?  Number of nodes as level i?  Operations time?  Worse BST?  Operations time?

27. 27 AVL trees  Definition  Min number of nodes of height H  F H+3 -1, where F n is nth Fibonacci number  Insert - single & double rotations. How many?  Delete - lazy. How bad?

28. 28 Single rotation  For the case of insertion into left subtree of left child: Z Y X ZY X Deepest node of X has depth 2 greater than deepest node of Z. Depth reduced by 1 Depth increased by 1

29. 29 Double rotation  For the case of insertion into the right subtree of the left child. Z X Y1Y1 Y2Y2 ZXY1Y1 Y2Y2

30. 30 Tries  Good for unequal length keys or sequences  Find O(m), m sequence length  But: Few to many children 459 466 588 3 3 I likelove you 5 9 lovely … …

31. 31 Hash Tables  Hash function h: h(key) = index  Desirable properties:  Approximate random distribution  Easy to calculate  E.g., Division: h(k) = k mod m  Perfect hashing  When know all input keys in advance

32. 32 Collisions  Separate chaining  Linked list: ordered vs unordered  Open addressing  Linear probing - clustering very bad with high load factor  *Quadratic probing - secondary clustering, table size must be prime  Double hashing - table size must be prime, too complex

33. 33 Hash Tables  Delete?  Rehash when load factor high - double (amortize cost constant)  Find & insert are near constant time!  But: no min, max, next,… operation  Trade space for time--load factors <75%

34. Priority Queues

35. 35 Priority Queues  Operations:  Insert  FindMin  DeleteMin  Implementations:  Linked list  Search tree  Heap

36. 36  Linked list  deleteMinO(1)O(N)  insert O(N)O(1)  Search trees  All operationsO(log N)  Heaps avg (assume random)worst  deleteMinO(log N) O(log N)  insert2.6O(log N) special case :  buildheapO(N)O(N) i.e., insert*N or Possible priority queue implementations

37. 37  Linked list  deleteMinO(1)O(N)  insert O(N)O(1)  Search trees  All operationsO(log N)  Heaps avg (assume random)worst  deleteMinO(log N) O(log N)  insert2.6O(log N) special case :  buildheapO(N)O(N) i.e., insert*N or Possible priority queue implementations

38. 38  Linked list  deleteMinO(1)O(N)  insert O(N)O(1)  Search trees  All operationsO(log N)  Heaps avg (assume random)worst  deleteMinO(log N) O(log N)  insert2.6O(log N)  buildheapO(N)O(N) N inserts or Possible priority queue implementations

39. 39 Heaps  Properties: 1.Complete binary tree in an array 2.Heap order property  Insert: percolate up  DeleteMin: percolate down  BuildHeap: starting at bottom, percolate down  Heapsort: BuildHeap + DeleteMin

40. 40 Representing complete binary trees  Arrays ( 1-based )  Parent at position i  Children at 2i (and 2i+1). 2 1 98 4 10 576 3 1 2 3 4 5 6 7 8 9 10

41. 41 Insert - Percolation up  Insert leaf to establish complete tree property.  Bubble inserted leaf up the tree until the heap order property is satisfied. 13 2665 24 32 316819 16 14 Not really there... 21

42. 42 DeleteMin - Percolation down  Move last leaf to root to restore complete tree property.  Bubble the transplanted leaf value down the tree until the heap order property is satisfied. 14 31 2665 24 32 216819 16 14 -- 2665 24 32 216819 16 31 12

43. 43 BuildHeap - Percolation down  Start at bottom subtrees.  Bubble subtree root down until the heap order property is satisfied. 24 2365 26 21 311916 68 14 32

44. Data Compression

45. 45 Pi 10000 31415926535897932384626433832795028841971693993751058209749445923078164062862089986280348253421170679821480865132823 06647093844609550582231725359408128481117450284102701938521105559644622948954930381964428810975665933446128475648233 78678316527120190914564856692346034861045432664821339360726024914127372458700660631558817488152092096282925409171536 43678925903600113305305488204665213841469519415116094330572703657595919530921861173819326117931051185480744623799627 49567351885752724891227938183011949129833673362440656643086021394946395224737190702179860943702770539217176293176752 38467481846766940513200056812714526356082778577134275778960917363717872146844090122495343014654958537105079227968925 89235420199561121290219608640344181598136297747713099605187072113499999983729780499510597317328160963185950244594553 46908302642522308253344685035261931188171010003137838752886587533208381420617177669147303598253490428755468731159562 86388235378759375195778185778053217122680661300192787661119590921642019893809525720106548586327886593615338182796823 03019520353018529689957736225994138912497217752834791315155748572424541506959508295331168617278558890750983817546374 64939319255060400927701671139009848824012858361603563707660104710181942955596198946767837449448255379774726847104047 53464620804668425906949129331367702898915210475216205696602405803815019351125338243003558764024749647326391419927260 42699227967823547816360093417216412199245863150302861829745557067498385054945885869269956909272107975093029553211653 44987202755960236480665499119881834797753566369807426542527862551818417574672890977772793800081647060016145249192173 21721477235014144197356854816136115735255213347574184946843852332390739414333454776241686251898356948556209921922218 42725502542568876717904946016534668049886272327917860857843838279679766814541009538837863609506800642251252051173929 84896084128488626945604241965285022210661186306744278622039194945047123713786960956364371917287467764657573962413890 86583264599581339047802759009946576407895126946839835259570982582262052248940772671947826848260147699090264013639443 74553050682034962524517493996514314298091906592509372216964615157098583874105978859597729754989301617539284681382686 83868942774155991855925245953959431049972524680845987273644695848653836736222626099124608051243884390451244136549762 78079771569143599770012961608944169486855584840635342207222582848864815845602850601684273945226746767889525213852254 99546667278239864565961163548862305774564980355936345681743241125150760694794510965960940252288797108931456691368672 28748940560101503308617928680920874760917824938589009714909675985261365549781893129784821682998948722658804857564014 27047755513237964145152374623436454285844479526586782105114135473573952311342716610213596953623144295248493718711014 57654035902799344037420073105785390621983874478084784896833214457138687519435064302184531910484810053706146806749192 78191197939952061419663428754440643745123718192179998391015919561814675142691239748940907186494231961567945208095146 55022523160388193014209376213785595663893778708303906979207734672218256259966150142150306803844773454920260541466592 52014974428507325186660021324340881907104863317346496514539057962685610055081066587969981635747363840525714591028970 64140110971206280439039759515677157700420337869936007230558763176359421873125147120532928191826186125867321579198414 84882916447060957527069572209175671167229109816909152801735067127485832228718352093539657251210835791513698820914442 10067510334671103141267111369908658516398315019701651511685171437657618351556508849099898599823873455283316355076479 18535893226185489632132933089857064204675259070915481416549859461637180270981994309924488957571282890592323326097299 71208443357326548938239119325974636673058360414281388303203824903758985243744170291327656180937734440307074692112019 13020330380197621101100449293215160842444859637669838952286847831235526582131449576857262433441893039686426243410773 22697802807318915441101044682325271620105265227211166039666557309254711055785376346682065310989652691862056476931257 05863566201855810072936065987648611791045334885034611365768675324944166803962657978771855608455296541266540853061434 44318586769751456614068007002378776591344017127494704205622305389945613140711270004078547332699390814546646458807972 70826683063432858785698305235808933065757406795457163775254202114955761581400250126228594130216471550979259230990796 54737612551765675135751782966645477917450112996148903046399471329621073404375189573596145890193897131117904297828564 75032031986915140287080859904801094121472213179476477726224142548545403321571853061422881375850430633217518297986622 37172159160771669254748738986654949450114654062843366393790039769265672146385306736096571209180763832716641627488880 07869256029022847210403172118608204190004229661711963779213375751149595015660496318629472654736425230817703675159067 35023507283540567040386743513622224771589150495309844489333096340878076932599397805419341447377441842631298608099888

46. 46 pitiny.c  This C program is just 143 characters long!  And it “decompresses” into the first 10,000 digits of Pi. long a[35014],b,c=35014,d,e,f=1e4,g,h; main(){for(;b=c-=14;h=printf("%04ld",e+d/f)) for(e=d%=f;g=--b*2;d/=g) d=d*b+f*(h?a[b]:f/5), a[b]=d%--g;}

47. 47 Data Compression  Huffman  Optimal prefix-free codes  Full binary tree - short codes for ?  Priority queue on “tree” frequency  LZW  Dictionary of codes for previously seen patterns  When find pattern increase length by one  trie

48. 48 Huffman  Full: every node  Is a leaf, or  Has exactly 2 children.  Build tree bottom up:  Use priority queue of trees weight - sum of frequencies.  New tree of two lowest weight trees. c a b d 0 0 0 1 1 1 a=1, b=001, c=000, d=01

49. 49 Byte LZW: Compress example baddad Input: ^ a b Dictionary: Output: 1032 cd 10335 4 a 5 d 6 d 7 a

50. 50 Byte LZW: Uncompress example 10335 Input: ^ a b Dictionary: Output: 1032 cd baddad 4 a 5 d 6 d 7 a

51. Sorting

52. 52 Simple sorting algorithms Several simple, quadratic algorithms (worst case and average). - Bubble Sort - Insertion Sort - Shell Sort (sub-quadratic) Only Insertion Sort of practical interest: running time linear in number of inversion of input sequence. Constants small. Stable?

53. 53 Sorting Review Asymptotically optimal O(n log n) algorithms (worst case and average). - Merge Sort - Heap Sort Merge Sort purely sequential and stable. But requires extra memory: 2n + O(log n).

54. 54 Quick Sort Overall fastest. In place. BUT: Worst case quadratic. Average case O(n log n). Not stable. Implementation details messy.

55. 55 Average-case analysis  Consider the quicksort tree: 10547131730222519 517134730222105 19 51730222105 1347 105222

56. 56 Radix Sort Used by old computer-card-sorting machines. Linear time: b passes on b-bit elements b/m passes m bits per pass Each pass must be stable BUT: Uses 2n+2 m space. May only beat Quick Sort for very large arrays.

57. Questions?