1. 1 CSE 326 Data Structures: Complexity Lecture 2: Wednesday, Jan 8, 2003

2. 2 Overview of the Quarter Complexity and analysis of algorithms List-like data structures Search trees Priority queues Hash tables Sorting Disjoint sets Graph algorithms Algorithm design Advanced topics Midterm around here

3. 3 Complexity and analysis of algorithms Weiss Chapters 1 and 2 Additional material –Cormen,Leiserson,Rivest – on reserve at Eng. library –Graphical analysis –Amortized analysis

4. 4 Program Analysis Correctness –Testing –Proofs of correctness Efficiency –Asymptotic complexity - how running times scales as function of size of input

5. 5 Proving Programs Correct Often takes the form of an inductive proof Example: summing an array int sum(int v[], int n) { if (n==0) return 0; else return v[n-1]+sum(v,n-1); } int sum(int v[], int n) { if (n==0) return 0; else return v[n-1]+sum(v,n-1); } What are the parts of an inductive proof?

6. 6 Inductive Proof of Correctness int sum(int v[], int n) { if (n==0) return 0; else return v[n-1]+sum(v,n-1); } int sum(int v[], int n) { if (n==0) return 0; else return v[n-1]+sum(v,n-1); } Theorem: sum(v,n) correctly returns sum of 1 st n elements of array v for any n. Basis Step: Program is correct for n=0; returns 0. Inductive Hypothesis (n=k): Assume sum(v,k) returns sum of first k elements of v. Inductive Step (n=k+1): sum(v,k+1) returns v[k]+sum(v,k), which is the same of the first k+1 elements of v.

7. 7 Inductive Proof of Correctness int find(int x, int v[], int n) { int left = -1; int right = n; while (left+1 < right) { int m = (left+right) / 2; if (x == v[m]) return m; if (x < v[m]) right = m; else left = m; } return –1; /* not found */ } int find(int x, int v[], int n) { int left = -1; int right = n; while (left+1 < right) { int m = (left+right) / 2; if (x == v[m]) return m; if (x < v[m]) right = m; else left = m; } return –1; /* not found */ } Binary search in a sorted array: v[0]  v[1] ...  v[n-1] Given x, find m s.t. x=v[m] Exercise 1: proof that it is correct Exercise 2: compute m, k s.t. v[m-1] < x = v[m] = v[m+1] =... = v[k] < v[k+1]

8. 8 Proof by Contradiction Assume negation of goal, show this leads to a contradiction Example: there is no program that solves the “halting problem” –Determines if any other program runs forever or not Alan Turing, 1937

9. 9 Does NonConformist(NonConformist) halt? Yes? That means HALT(NonConformist) = “never halts” No? That means HALT(NonConformist) = “halts” Program NonConformist (Program P) If ( HALT(P) = “never halts” ) Then Halt Else Do While (1 > 0) Print “Hello!” End While End If End Program Contradiction!

10. 10 Defining Efficiency Asymptotic Complexity - how running time scales as function of size of input Two problems: –What is the “input size” ? –How do we express the running time ? (The Big-O notation)

11. 11 Input Size Usually: length (in characters) of input Sometimes: value of input (if it is a number) Which inputs? –Worst case: tells us how good an algorithm works –Best case: tells us how bad an algorithm works –Average case: useful in practice, but there are technical problems here (next)

12. 12 Input Size Average Case Analysis Assume inputs are randomly distributed according to some “realistic” distribution  Compute expected running time Drawbacks –Often hard to define realistic random distributions –Usually hard to perform math

13. 13 Input Size Recall the function: find(x, v, n) Input size: n (the length of the array) T(n) = “running time for size n ” But T(n) needs clarification: –Worst case T(n): it runs in at most T(n) time for any x,v –Best case T(n): it takes at least T(n) time for any x,v –Average case T(n): average time over all v and x

14. 14 Input Size Amortized Analysis Instead of a single input, consider a sequence of inputs: –This is interesting when the running time on some input depends on the result of processing previous inputs Worst case analysis over the sequence of inputs Determine average running time on this sequence Will illustrate in the next lecture

15. 15 Definition of Order Notation Upper bound:T(n) = O(f(n))Big-O Exist constants c and n’ such that T(n)  c f(n)for all n  n’ Lower bound:T(n) =  (g(n))Omega Exist constants c and n’ such that T(n)  c g(n)for all n  n’ Tight bound: T(n) =  (f(n))Theta When both hold: T(n) = O(f(n)) T(n) =  (f(n)) Other notations: o(f),  (f) - see Cormen et al.

16. 16 Example: Upper Bound

17. 17 Using a Different Pair of Constants

18. 18 Example: Lower Bound

19. 19 Conventions of Order Notation

20. 20 Which Function Dominates? f(n) = n 3 + 2n 2 n 0.1 n + 100n 0.1 5n 5 n -15 2 n /100 8 2log n g(n) = 100n 2 + 1000 log n 2n + 10 log n n! 1000n 15 3n 7 + 7n Question to class: is f = O(g) ? Is g = O(f) ?

21. 21 Race I f(n)= n 3 +2n 2 g(n)=100n 2 +1000 vs.

22. 22 Race II n 0.1 log n vs.

23. 23 Race III n + 100n 0.1 2n + 10 log n vs.

24. 24 Race IV 5n 5 n! vs.

25. 25 Race V n -15 2 n /1001000n 15 vs.

26. 26 Race VI 8 2log(n) 3n 7 + 7n vs.

27. 27 Eliminate low order terms Eliminate constant coefficients

28. 28 Common Names constant:O(1) logarithmic:O(log n) linear:O(n) log-linear:O(n log n) quadratic:O(n 2 ) exponential:O(c n ) (c is a constant > 1) hyperexponential: (a tower of n exponentials Other names: superlinear:O(n c )(c is a constant > 1) polynomial:O(n c )(c is a constant > 0) Fastest Growth Slowest Growth

29. 29 Sums and Recurrences Often the function f(n) is not explicit but expressed as: A sum, or A recurrence Need to obtain analytical formula first

30. 30 Sums

31. 31 More Sums Sometimes sums are easiest computed with integrals:

32. 32 Recurrences f(n) = 2f(n-1) + 1, f(0) = T Telescoping  f(n)+1 = 2(f(n-1)+1) f(n-1)+1 = 2(f(n-2)+1)  2 f(n-2)+1 = 2(f(n-3)+1)  2 2..... f(1) + 1 = 2(f(0) + 1)  2 n-1  f(n)+1 = 2 n (f(0)+1) = 2 n (T+1)  f(n) = 2 n (T+1) - 1

33. 33 Recurrences Fibonacci: f(n) = f(n-1)+f(n-2), f(0)=f(1)=1  try f(n) = A c n What is c ? A c n = A c n-1 + A c n-2 c 2 – c – 1 = 0 Constants A, B can be determined from f(0), f(1) – not interesting for us for the Big O notation

34. 34 Recurrences f(n) = f(n/2) + 1, f(1) = T Telescoping: f(n) = f(n/2) + 1 f(n/2) = f(n/4) + 1... f(2) = f(1) + 1 = T + 1  f(n) = T + log n = O(log n)