CS 361 – Chapter 1 What is this class about? –Data structures, useful algorithm techniques –Programming needs to address both –You already have done some of this. –Analysis of algorithms to ensure an efficient solution.

Outline Ch.1 – Measuring complexity Ch. 2-7 – Data structures Ch. 8 – Sorting Ch – Algorithm techniques Ch – Applications to graphs, networks, data compression, etc. Special topics, e.g. order statistics, encryption, convex hull, branch & bound, FFT

Data structures Data type vs. data structure What data structures have you used to hold your data? Why so many data structures? –Why not use an array for everything? Basic operations common to many/all data structures

Purpose of d.s. There is no data structure that’s perfect for all occasions. Each has specialized purpose or advantage. For a given problem, we want a d.s. that is especially efficient in certain ways. Ex. Consider insertions and deletions. –an equal number? –far more insertions than deletions? –Expected use of d.s.  priority for which operations should be most efficient.

Array list d.s. Java API has a built-in ArrayList class. –What can it do? –Why do we like it? –Anything wrong? What if we also had a “SortedArrayList” class that guaranteed that at all times the data is sorted…. –Why might this be better than ArrayList? –But, is there a trade-off here?

Algorithm efficiency Suppose a problem has 2 solutions. –Implementation #1 requires n 4 steps –Implementation #2 requires 2 n steps …where n is the size of the problem input. Assume 1 step can be done in 1 ns. Is one solution really “better” than the other? Why?

Execution time Problem size (n)Time if n 4 stepsTime if 2 n steps Complete this table:

Execution time Problem size (n)Time if n 4 stepsTime if 2 n steps 216 ns4 ns 5625 ns32 ns 1010 μs1 μs μs1 ms s10 13 years Approximate times assuming 1 step = 1 ns

Algorithm analysis 2 ways –Exact: “count” exact number of operations –Asymptotic: find asymptotic or upper bound Amortization –Technique to simplify calculation –E.g. consider effect of occasional time penalties

Exact analysis Not done often, only for small parts of code We want # assembly instructions that execute. –Easier to determine in C/C++/Java than Pascal/Ada because code is lower-level, more explicit. Control flow makes calculation interesting. Example for (i = 0; i < 10; ++i) a[i] = i-1; –How many loop iterations? –How many instructions executed per iteration? –How many instructions are executed just once? –What if we replace 10 with n?

Example Try a nested loop for (i = 0; i < 10; ++i) for (j = 0; j < 20; ++j) b[i] += i-j; –How many instructions executed in inner loop? –In outer loop? –Outside the loops? –What if we replace 10 and 20 with n and 2*n ? –What if we were assigning to c[i][j] instead?

Asymptotic analysis Much more often we are just interested in an order of magnitude, expressed as a function of some input size, n. –Essentially, it doesn’t matter much if the number of steps is 5n 2 or 7n 2 ; any n 2 is better than n 3. Exact analysis is often unrealistic. –Usually too tedious. –There could be additional instructions behind the scenes. –Not all instructions take the same amount of time.

Big O notation Provides an upper bound on the general complexity of an algorithm. –Ex. O(n 2 ) means that it runs no worse than a quadratic function of the input size. –O(n 2 ) is a set of all functions at or below the complexity of a degree 2 polynomial. Definition: We say that f(n) is O(g(n)) if  constants c, n 0 > 0 such that f(n)  c g(n) for all n > n 0. “Eventually, g defeats f.”

Gist Usually, single loops are O(n), 2 nested loops are O(n 2 ), etc. If the execution time of an algorithm is a polynomial in n, then we only need to keep the largest degree. We can drop coefficient. Although O(…) is a set, we sometimes write “=“ for “is” or . Let’s assume that our functions are positive.

Example loops LoopComplexity for (i = 1; i <= n; i = i + 1) O(n) for (i = 1; i <= n; i = i * 2) O(log 2 (n)) for (i = 2; i <= n; i = i * i) O(log 2 (log 2 (n))) for (i = 1; i * i <= n; i = i + 1) O(sqrt(n))

Example f(n) = 7n 2 + 8n + 11 and g(n) = n 2 We want to show that f(n) is O(g(n)) How do we show this is true using the definition? –We need to specify values of c and n 0. –We want to show that 7n 2 + 8n + 11 <= c n 2 for sufficiently large n. –Note that 7n 2  7n 2, 8n  8n 2, 11  11n 2. –So, let c = = 26, and let n 0 = 1. –And we observe that for all n  0, 7n 2 + 8n + 11  26 n 2

Limits It turns out that if f(n)/g(n) tends to a finite constant as n goes to , then f(n) is O(g(n)). –Special case: if f(n) / g(n) tends to 0, the asymptotic bound is “too high”, but still ok. This helps us with other functions like log’s. –log(n) is O(n). (Log base doesn’t matter) –n 2 is O(n 3 ) but n 3 is not O(n 2 ). –Can we say the same about 2 n and 3 n ? –How about log(n) versus n 1/2 ?

Beyond O Big O has some close friends. –Big O:f(n)  c g(n)for some c –Big Ω: f(n)  c g(n)for some c –Big Θ:f is O(g) and g is O(f) –Little o:f(n)  c g(n)for all c –Little ω: f(n)  c g(n)for all c What do the little letters mean? –f is o(g) means f has a strictly lower order than g. –Ex. n 2 is o(n 3 ) and o(n 2 log n) but not o(n 2 ).

Final thoughts What would O(–n) mean? How do you show that 7n 3 is not O(n 2 ) ? cos(x) = 1 – x 2 /2 + x 4 /24 – x 6 /720 + … –Does this mean that cos(x) can’t be bounded by a polynomial? In other words, is cos(x) Ω(x 6 ) ? Big O doesn’t have to be used just for time. –Space requirements, height of tree, etc. –Gives rise to unusual orders like O(log(log(n)), etc.

Running average The book presents 2 algorithms to compute running averages: for i = 1 to nsum = 0 sum = 0for i = 1 to n for j = 1 to i sum += a[i] sum += a[j] avg[i] = sum/i avg[i] = sum/i Are they correct? What is the (worst-case) complexity of each? –Note the triangular loop in first algorithm.

Amortized analysis A technique that can help determine a tight big- O bound on ops that occur on a data structure. Motivation: Occasional high costs can be spread out. 2 common approaches –Accounting method: Save up “credit” that can be spent on later operations. –Aggregate method: Find cumulative total of n operations, and pro-rate (divide) among all of them.

Accounting method We differentiate between actual & amortized cost of an operation. Some amortized costs will be “too high” so that others may become subsidized/free. At all times we want amortized cost  actual cost so that we are in fact finding an upper bound. –In other words, early operations like “insert” assigned a high cost so we keep a credit balance  0.

Accounting example Stack with 3 operations –Push: actual cost = 1 –Pop: actual cost = 1 –Multipop: actual cost = min(size, desired) What is the cost of performing n stack ops? –Naïve approach: There are n operations, and in the worst case, a multipop operation must pop up to n elements, so n*n = O(n 2 ). –Better approach: Re-assess ops with amortized costs 2, 0, and 0, respectively. Rationale: Pushing an object will also “pay” for its later popping. Thus: Total  2n = O(n). Average per op  2 = O(1).

Aggregate method Another way of looking at the general problem of finding a tight big-O bound. Find cumulative worst-case total cost of n operations, and call this T(n). Then the average of each operation is just T(n) / n. Example: Using an array as a binary counter. a[4]a[3]a[2]a[1]a[0]

Binary counter a = { 0 } for i = 1 to n // If rightmost bit is 0, set it to 1 if a[0] == 0 a[0] = 1 // Otherwise, working from right to left, // turn off 1’s to 0’s until you reach a 0. // Turn that 0 into a 1. else j = 0 while a[j] == 1 a[j++] = 0 a[j] = 1

Count the ops a[4]A[3]A[2]A[1]A[0]# flipsRun tot

Analysis Let the cost = # bit flips Naïve: In counting from 0 to n, we may have to flip several bits for each increment. And because we have a nested loop bounded by n, the algorithm is O(n 2 ). Better way: Note that –a[0] is flipped on every iteration, –a[1] is flipped on every other iteration –a[2] is flipped on every 4 th iteration, etc. –Total number of flips is  2n, so the total is O(n), and the average number of flips per increment is O(1).

Exercise Suppose we have a data structure in which the cost of performing the i-th operation is –i, if i is a power of 2 –1, if i is not a power of 2 Do either an accounting or aggregate analysis to find a tight big-O bound on: the total cost of n operations, and on the cost of a single operation. Work it out before checking your answer…

Solution Cost(i) = i, if i is a power of 2; 1 if not We want an upper bound on the sum of cost(i) from i = 1 to n Worst case is when n is a power of 2, e.g. 2 k. Then our sum is  … –All the powers of 2 from 1 thru 2 k : 2 * 2 k. – … + 1, 2 k times: 2 k. –Total is  3 * 2 k, in other words 3n Thus, total cost of n operations is O(n), and average of each operation is O(1). –Surprising result?