1. Introduction to Algorithms: Amortized Analysis

2. Introduction to Algorithms Amortized Analysis Dynamic tables Aggregate method Accounting method Potential method Binary Counter Multi-pop Stack CS 421 - Introduction to Algorithms 2

3. How large should a Hash Table be? Goal: Make the table as small as possible, but large enough so that it won’t overflow. Problem: What if we don’t know the proper size in advance? Solution: Dynamic tables. I DEA : Whenever the table overflows, “grow” it by allocating (via malloc or new ) a new, larger table.Move all items from the old table into the new one, and free the storage for the old table. CS 421 - Introduction to Algorithms 3

4. 1 1.I NSERT 2.I NSERT overflow CS 421 - Introduction to Algorithms 4

5. 1 1 1.I NSERT 2.I NSERT overflow Exampleof a Dynamic Table CS 421 - Introduction to Algorithms 5

6. 1 1 1.I NSERT 2.I NSERT 2 Exampleof a Dynamic Table CS 421 - Introduction to Algorithms 6

7. 1 1 2 2 1.I NSERT 2.I NSERT 3.I NSERT overflow Exampleof a Dynamic Table CS 421 - Introduction to Algorithms 7

8. 1.I NSERT 2.I NSERT 3.I NSERT 1 2 overflow Exampleof a Dynamic Table CS 421 - Introduction to Algorithms 8

9. 1212 1.I NSERT 2.I NSERT 3.I NSERT Exampleof a Dynamic Table CS 421 - Introduction to Algorithms 9

10. 1.I NSERT 2.I NSERT 3.I NSERT 4.I NSERT 12341234 Exampleof a Dynamic Table CS 421 - Introduction to Algorithms 10

11. 1.I NSERT 2.I NSERT 3.I NSERT 4.I NSERT 5.I NSERT 1 2 3 4 overflow Exampleof a Dynamic Table CS 421 - Introduction to Algorithms 11

12. 1.I NSERT 2.I NSERT 3.I NSERT 4.I NSERT 5.I NSERT 1 2 3 4 overflow Exampleof a Dynamic Table CS 421 - Introduction to Algorithms 12

13. 1.I NSERT 2.I NSERT 3.I NSERT 4.I NSERT 5.I NSERT 12341234 Exampleof a Dynamic Table CS 421 - Introduction to Algorithms 13

14. Exampleof a Dynamic Table 12345671234567 1.I NSERT 2.I NSERT 3.I NSERT 4.I NSERT 5.I NSERT 6.I NSERT 7.I NSERT CS 421 - Introduction to Algorithms 14

15. Worst-Case Analysis Consider a sequence of n insertions.The worst-case time to execute one insertion is  (n).Therefore, the worst-case time for n insertions is n ·  (n) =  (n 2 ). CS 421 - Introduction to Algorithms 15

16. Worst-Case Analysis Consider a sequence of n insertions.The worst-case time to execute one insertion is  (n).Therefore, the worst-case time for n insertions is n ·  (n) =  (n 2 ). WRONG!In fact, the worst-case cost for n insertions is only  (n) ≪  (n 2 ). Let’s see why. CS 421 - Introduction to Algorithms 16

17. Tighter Analysis Let c i =the cost of the i th insertion = iif i – 1 is an exact power of 2, 1 otherwise. i12345678910 size i 1244888816 cici 1231511191 CS 421 - Introduction to Algorithms 17

18. Tighter Analysis Let c i =the cost of the i th insertion = if i – 1 is an exact power of 2,i 1otherwise. i12345678910 size i 1244888816 cici 11111 1212 11414 1111818 1 CS 421 - Introduction to Algorithms 18

19. Tighter Analysis (cont'd) 2 j2 j  n  n  n Cost of n insertions   c i i  1  3n   (n). Thus, the average cost of each dynamic table operation is  (n)/n =  (1). lg(n-1)  j  0 CS 421 - Introduction to Algorithms 19

20. Amortized Analysis An amortized analysis is any strategy for analyzing a sequence of operations to show that the average cost per operation is small, even though a single operation within the sequence might be expensive. Even though we’re taking averages, however, probability is not involved! An amortized analysis guarantees the average performance of each operation in the worst case. CS 421 - Introduction to Algorithms 20

21. Types of Amortized Analyses Three common amortization arguments: the aggregate method, the accounting method, the potential method. We’ve just seen an aggregate analysis. The aggregate method, though simple, lacks the precision of the other two methods.In particular, the accounting and potential methods allow a specific amortized cost to be allocated to each operation. CS 421 - Introduction to Algorithms 21

22. Accounting Method Charge i th operation a fictitious amortized cost ĉ i, where $1 pays for 1 unit of work (i.e., time). This fee is consumed to perform the operation. Any amount not immediately consumed is stored in the bank for use by subsequent operations. The bank balance must not go negative! We must ensure that n i1i1i1i1 ci cˆici cˆi for all n. Thus, the total amortized costs provide an upper bound on the total true costs. CS 421 - Introduction to Algorithms 22

23. Accounting Analysis: Dynamic Tables $0 $2 $0 $0 $0 $0 $2 $2 $2 $2 Charge an amortized cost of ĉ i = $3 for the i th insertion. $1 pays for the immediate insertion. $2 is stored for later table doubling. When the table doubles, $1 pays to move a recent item, and $1 pays to move an old item. Example: overflow CS 421 - Introduction to Algorithms 23

24. Accounting Analysis: Dynamic Tables $0 $2 Charge an amortized cost of ĉ i = $3 for the i th insertion. $1 pays for the immediate insertion. $2 is stored for later table doubling. When the table doubles, $1 pays to move a recent item, and $1 pays to move an old item. Example: overflow $0 $0 $0 $0 $2 $2 $2 $2 CS 421 - Introduction to Algorithms 24

25. Accounting Analysis: Dynamic Tables $0 $2 Charge an amortized cost of ĉ i = $3 for the i th insertion. $1 pays for the immediate insertion. $2 is stored for later table doubling. When the table doubles, $1 pays to move a recent item, and $1 pays to move an old item. Example: $0 $0 $0 $0 $0 $0 $0 $0 $2 $2 $2 CS 421 - Introduction to Algorithms 25

26. Accounting Analysis (cont'd) Key invariant: Bank balance never drops below 0. Thus, the sum of the amortized costs provides an upper bound on the sum of the true costs. *Okay, so I lied.The first operation costs only $2, not $3. i12345678910 size i 1244888816 cici 1231511191 ĉiĉi 2*2* 333333333 bank i 1224246824 CS 421 - Introduction to Algorithms 26

27. Potential Method I DEA : View the bank account as the potential energy (à la physics) of the dynamic set. Framework: Start with an initial data structure D 0. Operation i transforms D i–1 to D i. The cost of operation i is c i. Define a potential function  : {D i }  R, such that  (D 0 ) = 0 and  (D i )  0 for all i. The amortized cost ĉ i with respect to  is defined to be ĉ i = c i +  (D i ) –  (D i–1 ). CS 421 - Introduction to Algorithms 27

28. Understanding Potentials ĉ i = c i +  (D i ) –  (D i–1 ) potential difference  i If  i > 0, then ĉ i > c i.Operation i stores work in the data structure for later use. If  i < 0, then ĉ i < c i.The data structure delivers up stored work to help pay for operation i. CS 421 - Introduction to Algorithms 28

29. Amortized Costs Bound True Costs The total amortized cost of n operations is n cˆi ci (Di )  (Di1)cˆi ci (Di )  (Di1) i  1i  1 Summing both sides. CS 421 - Introduction to Algorithms 29

30. Amortized Costs Bound True Costs The total amortized cost of n operations is n cˆi ci (Di )  (Di1)cˆi ci (Di )  (Di1) i1i1n ci (Dn )  (D0 )i1i1i1n ci (Dn )  (D0 )i1 The series telescopes. CS 421 - Introduction to Algorithms 30

31. Amortized Costs Bound True Costs The total amortized cost of n operations is n cˆi ci (Di )  (Di1)cˆi ci (Di )  (Di1) i1i1n ci (Dn )  (D0 )i1i1i1n ci (Dn )  (D0 )i1 n cii1n cii1 since  (D n )  0 and  (D 0 ) = 0. CS 421 - Introduction to Algorithms 31

32. Potential Analysis: Table Doubling Define the potential of the table after the ith insertion by  (D i ) = 2i – 2 ⌈ lg i ⌉ (Assume 2 ⌈ lg 0 ⌉ = 0). Note:  (D 0 ) = 0  (D i )  0 for all i. Example: $0 $2  = 2·6 – 2 3 = 4 accounting method ) ( $0 $0 $0 $0 $2 $2 CS 421 - Introduction to Algorithms 32

33. Calculation of Amortized Costs The amortized cost of the i th insertion is ĉ i = c i +  (D i ) –  (D i–1 ) CS 421 - Introduction to Algorithms 33

34. Calculation of Amortized Costs The amortized cost of the i th insertion is ĉ i = c i +  ( D i ) –  ( D i –1 ) i if i – 1 is an exact power of 2 1 otherwise + ( 2i – 2 ⌈ lg i ⌉ )– ( 2(i –1) – 2 ⌈ lg (i – 1) ⌉ ) = CS 421 - Introduction to Algorithms 34

35. Calculation of Amortized Costs The amortized cost of the i th insertion is ĉ i = c i +  ( D i ) –  ( D i –1 ) i if i – 1 is an exact power of 2 1 otherwise + ( 2i – 2 ⌈ lg i ⌉ )– ( 2(i –1) – 2 ⌈ lg (i – 1) ⌉ ) = i if i – 1 is an exact power of 2 1 otherwise + 2 – 2 ⌈ lg i ⌉ + 2 ⌈ lg (i – 1) ⌉ = CS 421 - Introduction to Algorithms 35

36. Calculation Case 1: i – 1 is an exact power of 2. CS 421 - Introduction to Algorithms 36

37. Calculation Case 1: i – 1 is an exact power of 2. ĉ i = i + 2 – 2 ⌈ lg i ⌉ + 2 ⌈ lg (i – 1) ⌉ CS 421 - Introduction to Algorithms 37

38. Calculation Case 1: i – 1 is an exact power of 2. ĉ i = i + 2 – 2 ⌈ lg i ⌉ + 2 ⌈ lg (i – 1) ⌉ = i + 2 – 2(i – 1) + (i – 1) CS 421 - Introduction to Algorithms 38

39. Calculation Case 1: i – 1 is an exact power of 2. ĉ i = i + 2 – 2 ⌈ lg i ⌉ + 2 ⌈ lg (i – 1) ⌉ = i + 2 – 2(i – 1) + (i – 1) = i + 2 – 2i + 2 + i – 1 CS 421 - Introduction to Algorithms 39

40. Calculation Case 1: i – 1 is an exact power of 2. ĉ i = i + 2 – 2 ⌈ lg i ⌉ + 2 ⌈ lg (i – 1) ⌉ = i + 2 – 2(i – 1) + (i – 1) = i + 2 – 2i + 2 + i – 1 = 3 CS 421 - Introduction to Algorithms 40

41. Calculation Case 1: i – 1 is an exact power of 2. ĉ i = i + 2 – 2 ⌈ lg i ⌉ + 2 ⌈ lg (i – 1) ⌉ = i + 2 – 2(i – 1) + (i – 1) = i + 2 – 2i + 2 + i – 1 = 3 Case 2: i – 1 is not an exact power of 2. CS 421 - Introduction to Algorithms 41

42. Calculation Case 1: i – 1 is an exact power of 2. ĉ i = i + 2 – 2 lg i + 2 ⌈ lg (i – 1) ⌉ = i + 2 – 2(i – 1) + (i – 1) = i + 2 – 2i + 2 + i – 1 = 3 Case 2: i – 1 is not an exact power of 2. ĉ i = 1 + 2 – 2 ⌈ lg i ⌉ + 2 ⌈ lg (i – 1) ⌉ CS 421 - Introduction to Algorithms 42

43. Calculation Case 1: i – 1 is an exact power of 2. ĉ i = i + 2 – 2 lg i + 2 ⌈ lg (i – 1) ⌉ = i + 2 – 2(i – 1) + (i – 1) = i + 2 – 2i + 2 + i – 1 = 3 Case 2: i – 1 is not an exact power of 2. ĉ i = 1 + 2 – 2 ⌈ lg i ⌉ + 2 ⌈ lg (i – 1) ⌉ = 3 (since 2 ⌈ lg i ⌉ = 2 ⌈ lg (i – 1) ⌉ ) CS 421 - Introduction to Algorithms 43

44. Calculation Case 1: i – 1 is an exact power of 2. ĉ i = i + 2 – 2 lg i + 2 ⌈ lg (i – 1) ⌉ = i + 2 – 2(i – 1) + (i – 1) = i + 2 – 2i + 2 + i – 1 = 3 Case 2: i – 1 is not an exact power of 2. Therefore, n insertions cost  (n) in the worst case. ĉ i = 1 + 2 – 2 lg i + 2 ⌈ lg (i – 1) ⌉ = 3 (since 2 lg i = 2 ⌈ lg (i – 1) ⌉ ) CS 421 - Introduction to Algorithms 44

45. Binary counter CS 421 - Introduction to Algorithms 45 Goal. Increment a k-bit binary counter (mod 2 k ). Representation. a j = j th least significant bit of counter. Counter value A[k]A[k-1]…A[2]A[1]A[0] 000…000 100…001 200…010 300…011 400…100 500…101 600…110 700…111 800…000 ………………… 2 k - 111…111

46. Binary counter CS 421 - Introduction to Algorithms 46 The sequence of operations is a sequence of increments of the counter: Toggle the lowest order bit. If that bit switches to a 0, toggle the next higher order bit. Continue until the bit that we toggle switches to a 1. Cost Model.Number of bits flipped. Counter value A[k]A[k-1]…A[2]A[1]A[0] Cost 000…0000 100…0011 200…0102 300…0111 400…1003 500…1011 600…1102 700…1111 800…0004 ………………… 2 k - 111…111k

47. Binary counter CS 421 - Introduction to Algorithms 47 When increment up to n, change is at most 1+floor(log n) bits - number of bits in the binary representation of n. Thus, the cost of counting up to n, a sequence of n increments, is O(n log n). Counter value A[k]A[k-1]…A[2]A[1]A[0] Cost 000…0000 100…0011 200…0102 300…0111 400…1003 500…1011 600…1102 700…1111 800…0004 ………………… 2 k - 111…111k

48. Aggregate Method Aggregate method. Sum up sequence of operations, weighted by their cost. How often we flip each individual bit? Then sum those up to bound the total, rather than individually obtaining a worst case bound for each bit. How often is A[0] toggled in counting up to n? How often is A[1] toggled, A[2] toggled, etc? Every time, every other time, every fourth time,... CS 421 - Introduction to Algorithms 48

49. Aggregate Method CS 421 - Introduction to Algorithms 49

50. Aggregate Method 50 k-1 ∞ j  0 CS 421 - Introduction to Algorithms

51. Accounting Method Recall. Each individual operation pays a certain amount to a central pot ( ĉ i – the amortized cost) the actual cost of some operations is less than the amortized cost (c i < ĉ i ) and leaves a reserve for future operations. some operations cost more than the amortized cost (c i > ĉ i ) and use the reserve in the central pot. The amount that each individual operation pays into the pot is set so that there is just enough to do all the work. CS 421 - Introduction to Algorithms 51

52. Accounting Method Binary Counter. For the binary counter, the only work is toggling a bit, so let cost be $1 (c i ) to toggle a bit. What’s the amortized cost to pay for all the toggles as they occur? By the aggregate method, we found ĉ i = 2, $2 would be a good guess (though not a guarantee). CS 421 - Introduction to Algorithms 52

53. Accounting Method What happens on an increment? some m low-order bits (m ≥ 0) change from 1 to 0 the m-th bit changes from 0 to 1. We pay $2 for flipping the m-th bit from 0 to 1 but only use $1, and leave at the m-th bit for later use. For the m bits changed from 1 to 0, we pay for those with the $1 left at those bits by previous operations. CS 421 - Introduction to Algorithms 53

54. Accounting Method Example. Assume a 6-bit counter initialized to zero: CS 421 - Introduction to Algorithms 54 A[5]A[4]A[3]A[2]A[1]A[0] 000000

55. Accounting Method CS 421 - Introduction to Algorithms 55 A[5]A[4]A[3]A[2]A[1]A[0] 000001 Increment. Pay $1 to flip A[0] and save $1 at that location for future operations.

56. Accounting Method CS 421 - Introduction to Algorithms 56 A[5]A[4]A[3]A[2]A[1]A[0] 000010 Increment. Pay $1 to flip A[1] and save $1 at that location for future operations. Pay $1 from reserve for flipping A[0].

57. Accounting Method CS 421 - Introduction to Algorithms 57 A[5]A[4]A[3]A[2]A[1]A[0] 000011 Increment. Pay $1 to flip A[0] and save $1 at that location for future operations.

58. Accounting Method CS 421 - Introduction to Algorithms 58 A[5]A[4]A[3]A[2]A[1]A[0] 000100 Increment. Pay $1 to flip A[2] and save $1 at that location for future operations. Pay $2 from reserve for flipping A[0] and A[1].

59. Accounting Method CS 421 - Introduction to Algorithms 59 A[5]A[4]A[3]A[2]A[1]A[0] 000101 Increment. Pay $1 to flip A[0] and save $1 at that location for future operations.

60. Accounting Method CS 421 - Introduction to Algorithms 60 A[5]A[4]A[3]A[2]A[1]A[0] 000101 And so on…

61. Accounting Method CS 421 - Introduction to Algorithms 61 Theorem. Starting from the zero counter, a sequence of n increment operations flips O(n) bits. Proof. Because the algorithm maintain the invariant that every bit currently set to 1 has $1 credit that can used to flip back to 0. So the value of the credits at each bit is always ≥ 0. Thus, the amortized cost of the increment is 2, and the overall time is O(n) for n increments..

62. Potential Method CS 421 - Introduction to Algorithms 62 Potential Function. Let the potential Φ(D i ) of the counter after the i-th operation be b i, the number of 1s in the counter after the i-th operation. Φ(D 0 ) = 0 Φ(D i ) ≥ 0 for all D i

63. Potential Method CS 421 - Introduction to Algorithms 63

64. Potential Method CS 421 - Introduction to Algorithms 64

65. Conclusions Amortized costs can provide a clean abstraction of data-structure performance. Any of the analysis methods can be used when an amortized analysis is called for, but each method has some situations where it is arguably the simplest or most precise. Different schemes may work for assigning amortized costs in the accounting method, or potentials in the potential method, sometimes yielding radically different bounds. CS 421 - Introduction to Algorithms 65