UMass Lowell Computer Science Graduate Analysis of Algorithms Prof. Karen Daniels Spring, 2010 Lecture 3 Tuesday, 2/9/10 Amortized Analysis

Overview Amortize: “To pay off a debt, usually by periodic payments” [Websters] Amortize: “To pay off a debt, usually by periodic payments” [Websters] Amortized Analysis: Amortized Analysis: “creative accounting” for operations “creative accounting” for operations can show average cost of an operation is small (when averaged over sequence of operations, not distribution of inputs) even though a single operation in the sequence is expensive can show average cost of an operation is small (when averaged over sequence of operations, not distribution of inputs) even though a single operation in the sequence is expensive result must hold for any sequence of these operations result must hold for any sequence of these operations no probability is involved (unlike average-case analysis) no probability is involved (unlike average-case analysis) guarantee holds in worst-case guarantee holds in worst-case analysis method only; no effect on code operation analysis method only; no effect on code operation

Overview (continued) 3 ways to determine amortized cost of an operation that is part of a sequence of operations: 3 ways to determine amortized cost of an operation that is part of a sequence of operations: Aggregate Method Aggregate Method find upper bound T(n) on total cost of sequence of n operations find upper bound T(n) on total cost of sequence of n operations amortized cost = average cost per operation = T(n)/n amortized cost = average cost per operation = T(n)/n same for all the operations in the sequence same for all the operations in the sequence Accounting Method Accounting Method amortized cost can differ across operations amortized cost can differ across operations overcharge some operations early in sequence overcharge some operations early in sequence store overcharge as “prepaid credit” on specific data structure objects store overcharge as “prepaid credit” on specific data structure objects Potential Method Potential Method amortized cost can differ across operations (as in accounting method) amortized cost can differ across operations (as in accounting method) overcharge some operations early in sequence (as in accounting method) overcharge some operations early in sequence (as in accounting method) store overcharge as “potential energy” of data structure as a whole store overcharge as “potential energy” of data structure as a whole (unlike accounting method) (unlike accounting method)

Aggregate Method: Stack Operations Aggregate Method Aggregate Method find upper bound T(n) on total cost of sequence of n operations find upper bound T(n) on total cost of sequence of n operations amortized cost = average cost per operation = T(n)/n amortized cost = average cost per operation = T(n)/n same for all the operations in the sequence same for all the operations in the sequence Traditional Stack Operations Traditional Stack Operations PUSH(S,x) pushes object x onto stack S PUSH(S,x) pushes object x onto stack S POP(S) pops top of stack S, returns popped object POP(S) pops top of stack S, returns popped object O(1) time: consider cost as 1 for our discussion O(1) time: consider cost as 1 for our discussion Total actual cost of sequence of n PUSH/POP operations = n Total actual cost of sequence of n PUSH/POP operations = n STACK-EMPTY(S) time in  (1) STACK-EMPTY(S) time in  (1)

Aggregate Method: Stack Operations (continued) New Stack Operation New Stack Operation MULTIPOP(S,k) pops top k elements off stack S MULTIPOP(S,k) pops top k elements off stack S pops entire stack if it has < k items pops entire stack if it has < k items MULTIPOP actual cost for stack containing s items: MULTIPOP actual cost for stack containing s items: Use cost =1 for each POP Use cost =1 for each POP Cost = min(s,k) Cost = min(s,k) Worst-case cost in O(s) in O(n) Worst-case cost in O(s) in O(n) MULTIPOP(S,k) 1 while not STACK-EMPTY(S) and k = 0 2do POP(S) 3 k k - 1

Aggregate Method: Stack Operations (continued) Sequence of n PUSH, POP, MULTIPOP ops Sequence of n PUSH, POP, MULTIPOP ops initially empty stack initially empty stack MULTIPOP worst-case O(n) O(n 2 ) for sequence MULTIPOP worst-case O(n) O(n 2 ) for sequence Aggregate method yields tighter upper bound Aggregate method yields tighter upper bound Sequence of n operations has O(n) worst-case cost Sequence of n operations has O(n) worst-case cost Each item can be popped at most once for each push Each item can be popped at most once for each push # POP calls (including ones in MULTIPOP) <= #push calls # POP calls (including ones in MULTIPOP) <= #push calls <= n <= n Average cost of an operation = O(n)/n = O(1) Average cost of an operation = O(n)/n = O(1) = amortized cost of each operation = amortized cost of each operation holds for PUSH, POP and MULTIPOP holds for PUSH, POP and MULTIPOP

Accounting Method Accounting Method Accounting Method amortized cost can differ across operations amortized cost can differ across operations overcharge some operations early in sequence overcharge some operations early in sequence store overcharge as “prepaid credit” on specific data structure objects store overcharge as “prepaid credit” on specific data structure objects Let be actual cost of ith operation Let be actual cost of ith operation Let be amortized cost of ith operation (what we charge) Let be amortized cost of ith operation (what we charge) Total amortized cost of sequence of operations must be upper bound on total actual cost of sequence Total amortized cost of sequence of operations must be upper bound on total actual cost of sequence Total credit in data structure = Total credit in data structure = must be nonnegative for all n must be nonnegative for all n

Accounting Method: Stack Operations Paying for a sequence using amortized cost: Paying for a sequence using amortized cost: start with empty stack start with empty stack PUSH of an item always precedes POP, MULTIPOP PUSH of an item always precedes POP, MULTIPOP pay for PUSH & store 1 unit of credit pay for PUSH & store 1 unit of credit credit for each item pays for actual POP, MULTIPOP cost of that item credit for each item pays for actual POP, MULTIPOP cost of that item credit never “goes negative” credit never “goes negative” total amortized cost of any sequence of n ops is in O(n) total amortized cost of any sequence of n ops is in O(n) amortized cost is upper bound on total actual cost amortized cost is upper bound on total actual cost PUSH12 PUSH12 POP10 POP10 MULTIPOPmin(k,s)0 MULTIPOPmin(k,s)0 Operation Actual Cost Assigned Amortized Cost

Potential Method Potential Method Potential Method amortized cost can differ across operations (as in accounting method) amortized cost can differ across operations (as in accounting method) overcharge some operations early in sequence (as in accounting method) overcharge some operations early in sequence (as in accounting method) store overcharge as “potential energy” of data structure as a whole (unlike accounting method) store overcharge as “potential energy” of data structure as a whole (unlike accounting method) Let c i be actual cost of ith operation Let c i be actual cost of ith operation Let D i be data structure after applying ith operation Let D i be data structure after applying ith operation Let  (D i ) be potential associated with D i Let  (D i ) be potential associated with D i Amortized cost of ith operation: Amortized cost of ith operation: Total amortized cost of n operations: Total amortized cost of n operations: Require: so total amortized cost is upper bound on total actual cost Require: so total amortized cost is upper bound on total actual cost Since n might not be known in advance, guarantee “payment in advance” by requiring Since n might not be known in advance, guarantee “payment in advance” by requiring terms telescope

Potential Method: Stack Operations Potential function value = number of items in stack Potential function value = number of items in stack guarantees nonnegative potential after ith operation guarantees nonnegative potential after ith operation Amortized operation costs (assuming stack has s items) Amortized operation costs (assuming stack has s items) PUSH: PUSH: potential difference= potential difference= amortized cost = amortized cost = MULTIPOP(S,k) pops k’=min(k,s) items off stack MULTIPOP(S,k) pops k’=min(k,s) items off stack potential difference= potential difference= amortized cost = amortized cost = POP amortized cost also = 0 POP amortized cost also = 0 Amortized cost O(1) total amortized cost of sequence of n operations in O(n) Amortized cost O(1) total amortized cost of sequence of n operations in O(n)

Dynamic Tables: Overview Dynamic Table T: Dynamic Table T: array of slots array of slots Ignore implementation choices: stack, heap, hash table... Ignore implementation choices: stack, heap, hash table... if too full, increase size & copy entries to T’ if too full, increase size & copy entries to T’ if too empty, decrease size & copy entries to T’ if too empty, decrease size & copy entries to T’ Analyze dynamic table insert and delete Analyze dynamic table insert and delete Actual expansion or contraction cost is large Actual expansion or contraction cost is large Show amortized cost of insert or delete in O(1) Show amortized cost of insert or delete in O(1) Load factor  (T) = num[T]/size[T] Load factor  (T) = num[T]/size[T] empty table:  (T) = 1 (convention guaranteeing load factor can be lower bounded by a useful constant) empty table:  (T) = 1 (convention guaranteeing load factor can be lower bounded by a useful constant) full table:  (T) = 1 full table:  (T) = 1

Dynamic Tables: Table (Expansion Only) Load factor bounds (double size when T is full): Load factor bounds (double size when T is full): Sequence of n inserts on initially empty table Sequence of n inserts on initially empty table Worst-case cost of insert is in O(n) Worst-case cost of insert is in O(n) Worst-case cost of sequence of n inserts is in O(n 2 ) Worst-case cost of sequence of n inserts is in O(n 2 ) LOOSE “elementary” insertion Double only when table is already full. WHY?

Dynamic Tables: Table Expansion (cont.) Aggregate Method: Aggregate Method: c i = i if i-1 is exact power of 2 c i = i if i-1 is exact power of 2 1 otherwise 1 otherwise total cost of n inserts = total cost of n inserts = Accounting Method: Accounting Method: charge cost = 3 for each element inserted charge cost = 3 for each element inserted intuition for 3 intuition for 3 each item pays for 3 elementary insertions each item pays for 3 elementary insertions inserting itself into current table inserting itself into current table expansion: moving itself expansion: moving itself expansion: moving another item that has already been moved expansion: moving another item that has already been moved Amortized Analysis count only elementary insertions

Dynamic Tables: Table Expansion (cont.) PotentialMethod : PotentialMethod : Value of potential function  Value of potential function  0 right after expansion (then becomes 2) 0 right after expansion (then becomes 2) builds to table size by time table is full builds to table size by time table is full always nonnegative, so sum of amortized costs of n inserts is upper bound on sum of actual costs always nonnegative, so sum of amortized costs of n inserts is upper bound on sum of actual costs Amortized Analysis Amortized cost of ith insert Amortized cost of ith insert  i  = potential after ith operation  i  = potential after ith operation Case 1: insert does not cause expansion Case 1: insert does not cause expansion Case 2: insert causes expansion Case 2: insert causes expansion use these: 3 functions: size i, num i,  i

Dynamic Tables: Table Expansion & Contraction Load factor bounds: Load factor bounds: (double size when T is full) (halve size when T is ¼ full): (double size when T is full) (halve size when T is ¼ full): DELETE pseudocode analogous to INSERT DELETE pseudocode analogous to INSERT PotentialMethod : PotentialMethod : Value of potential function  Value of potential function  = 0 for empty table = 0 for empty table 0 right after expansion or contraction 0 right after expansion or contraction builds as  (T) increases to 1 or decreases to ¼ builds as  (T) increases to 1 or decreases to ¼ always nonnegative, so sum of amortized costs of n inserts is upper bound on sum of actual costs always nonnegative, so sum of amortized costs of n inserts is upper bound on sum of actual costs = 0 when  (T)=1/2 = 0 when  (T)=1/2 = num[T] when  (T)=1 = num[T] when  (T)=1 = num[T] when  (T)=1/4 = num[T] when  (T)=1/4 Amortized Analysis count elementary insertions & deletions same as INSERT

Dynamic Tables: Table Expansion & Contraction Amortized Analysis Potential Method

Dynamic Tables: Table Expansion & Contraction Amortized Analysis Analyze cost of sequence of n inserts and/or deletes Analyze cost of sequence of n inserts and/or deletes Amortized cost of ith operation Amortized cost of ith operation Case 1: INSERT Case 1: INSERT Case 1a:  i-1 >= ½. By previous insert analysis: Case 1a:  i-1 >= ½. By previous insert analysis: Case 1b:  i-1 < ½ and  i < ½ Case 1b:  i-1 < ½ and  i < ½ Case 1c:  i-1 = ½ Case 1c:  i-1 = ½ Potential Method holds whether or not table expands no expansion see derivation in textbook

Dynamic Tables: Table Expansion & Contraction Amortized Analysis Amortized cost of ith operation (continued) Amortized cost of ith operation (continued) Potential Method Case 2: DELETE Case 2: DELETE Case 2a:  i-1 >= ½. Case 2a:  i-1 >= ½. Case 2b:  i-1 < ½ and  i < ½ Case 2b:  i-1 < ½ and  i < ½ Case 2c:  i-1 < ½ and  i < ½ Case 2c:  i-1 < ½ and  i < ½ no contraction contraction Conclusion: amortized cost of each operation is bounded above by a constant, so time for sequence of n operations is O(n). textbook exercise

Example: Dynamic Closest Pair source: “Fast hierarchical clustering and other applications of dynamic closest pairs,” David Eppstein, Journal of Experimental Algorithmics, Vol. 5, August S SS

Example: Dynamic Closest Pair (continued) source: “Fast hierarchical clustering and other applications of dynamic closest pairs,” David Eppstein, Journal of Experimental Algorithmics, Vol. 5, August S SS

Example: Dynamic Closest Pair (continued) Rules source: “Fast hierarchical clustering and other applications of dynamic closest pairs,” David Eppstein, Journal of Experimental Algorithmics, Vol. 5, August Partition dynamic set S into subsets. Partition dynamic set S into subsets. Each subset S i has an associated digraph G i consisting of a set of disjoint, directed paths. Each subset S i has an associated digraph G i consisting of a set of disjoint, directed paths. Total number of edges in all graphs remains linear Total number of edges in all graphs remains linear Combine and rebuild if number of edges reaches 2n. Combine and rebuild if number of edges reaches 2n. Closest pair is always in some G i. Closest pair is always in some G i. Initially all points are in single set S 1. Initially all points are in single set S 1. Operations: Operations: Create G i for a subset S i. Create G i for a subset S i. Insert a point. Insert a point. Delete a point. Delete a point. Merge subsets until. Merge subsets until. We use log base 2.

Example: Dynamic Closest Pair (continued) Rules: Operations source: “Fast hierarchical clustering and other applications of dynamic closest pairs,” David Eppstein, Journal of Experimental Algorithmics, Vol. 5, August Create G i for a subset S i : Create G i for a subset S i : Select starting point (we choose leftmost (or higher one in case of a tie)) Select starting point (we choose leftmost (or higher one in case of a tie)) Iteratively extend the path P, selecting next vertex as: Iteratively extend the path P, selecting next vertex as: Case 1: nearest neighbor in S \ P if last point on path belongs to S i Case 1: nearest neighbor in S \ P if last point on path belongs to S i Case 2: nearest neighbor in S i \ P if last point on path belongs to S \ S i Case 2: nearest neighbor in S i \ P if last point on path belongs to S \ S i Insert a point x: Insert a point x: Create new subset S k+1 ={x}. Create new subset S k+1 ={x}. Merge subsets if necessary. Merge subsets if necessary. Create G i for new or merged subsets. Create G i for new or merged subsets. Delete a point x: Delete a point x: Create new subset S k+1 = all points y such that (y,x) is a directed edge in some G i. Create new subset S k+1 = all points y such that (y,x) is a directed edge in some G i. Remove x and adjacent edges from all G i. (We also remove y from its subset.) Remove x and adjacent edges from all G i. (We also remove y from its subset.) Merge subsets if necessary. Merge subsets if necessary. Create G i for new or merged subsets. Create G i for new or merged subsets. Merge subsets until : Merge subsets until : Choose subsets S i and S j to minimize size ratio | S j |/ | S i |. Choose subsets S i and S j to minimize size ratio | S j |/ | S i |. See handout for example.

Example: Dynamic Closest Pair (continued) Potential Function Potential for a subset S i :  i = n|S i |log|S i |. Potential for a subset S i :  i = n|S i |log|S i |. Total potential  = n 2 logn -   i. Total potential  = n 2 logn -   i. Paper proves this Theorem: Paper proves this Theorem: HW#3 presents variation on these operations and asks you to determine the amortized cost per insertion and deletion. HW#3 presents variation on these operations and asks you to determine the amortized cost per insertion and deletion. source: “Fast hierarchical clustering and other applications of dynamic closest pairs,” David Eppstein, Journal of Experimental Algorithmics, Vol. 5, August Theorem: The data structure maintains the closest pair in S in O( n ) space, amortized time O( n log n ) per insertion, and amortized time O( n log 2 n ) per deletion.