1. Discrete Structures & Algorithms Functions & Asymptotic Complexity

2. 2 To understand how to perform computations efficiently, we need to know –How to count! –How to sum a sequence of steps. –Reason about the complexity/efficiency of an algorithm. –Prove the correctness of our analysis. Logic Methods of inference Proof strategies Induction … –(and eventually) How to create better algorithms.

3. 3 Functions We can represent the number of steps needed to achieve a goal as a function of the size of the problem. Next: A quick refresher on functions –You should know (most of this) from other math/engineering courses.

4. 4 Functions Definition: A function f : A  B is a subset of AxB where  a  A,  b  B unique and  f. (A function is a mapping of elements from one set to another.)

5. 5 Functions Definition: a function f : A  B is a subset of AxB where  a  A,  b  B unique and  f. A B A B A point!

6. 6 Functions A = {Michael, Tito, Janet, Cindy, Bobby} B = {Katherine Evans, Carol Brady, Mother Teresa} Let f: A  B be defined as f(a) = mother(a). Michael Tito Janet Cindy Bobby Katherine Carol Mother Teresa

7. 7 Image and preimage For any set S  A, image(S) = {b :  a  S, f(a) = b} So, image({Michael, Tito}) = {Katherine Evans} image(A) = B - {Mother Teresa} image(S) = f(S) Michael Tito Janet Cindy Bobby Katherine Evans Carol Brady Mother Teresa

8. 8 Image and preimage For any S  B, preimage(S) = {a:  b  S, f(a) = b} So, preimage({Carol Brady}) = {Cindy, Bobby} preimage(B) = A preimage(S) = f -1 (S) Michael Tito Janet Cindy Bobby Katherine Evans Carol Brady Mother Teresa

9. 9 Image and preimage What is image(preimage(S))? a)S b){ } c)subset of S d)superset of S e)who knows?  S Michael Tito Janet Cindy Bobby Katherine Evans Carol Brady Mother Teresa

10. 10 Functions and their properties You should be familiar with these properties (past classes) or Functions derive many of their properties from sets (they are defined on sets) As you already know much about sets -> should be able to derive these properties

11. 11 Functions and their properties f(A  B)  f(A)  f(B)? f(A  B) = {y :  x  (A  B), f(x) = y} Choose an arbitrary b  f(A  B), and show that it must also be an element of f(A)  f(B). So,  x  (A  B) such that f(x) = y. If x  A (it is), then f(x) = y  f(A). If x  B (it is), then f(x) = y  f(B). y  f(A), and y  f(B), so b  f(A)  f(B).

12. 12 Injections A function f: A  B is one-to-one (injective, an injection) if  a,b,c, (f(a) = b  f(c) = b)  a = c Not one-to-one Every b  B has at most 1 preimage. Michael Tito Janet Cindy Bobby Katherine Evans Carol Brady Mother Teresa

13. 13 Surjections A function f: A  B is surjective if  b  B,  a  A f(a) = b Not surjective Every b  B has at least 1 preimage. Michael Tito Janet Cindy Bobby Katherine Evans Carol Brady Mother Teresa

14. 14 Bijections A function f: A  B is bijective if it is both one-to- one (injective) and surjective. Isaak Bri Lynette Aidan Evan Cinda Dee Deb Katrina Dawn Every b  B has exactly 1 preimage. An important implication of this characteristic: The preimage (f -1 ) is a function! Isaak Bri Lynette Aidan Evan Cinda Dee Deb Katrina Dawn

15. 15 Examples Suppose f: R +  R +, f(x) = x 2. Is f one-to-one? Is f surjective? Is f bijective? yes

16. 16 Examples Suppose f: R  R +, f(x) = x 2. Is f one-to-one? Is f surjective? Is f bijective? noyesno

17. 17 Examples Suppose f: R  R, f(x) = x 2. Is f one-to-one? Is f surjective? Is f bijective? no

18. 18 Composition of functions Let g:X  Y, and f:Y  Z be functions. Then the composition of f and g is: (f o g)(x) = f(g(x)) g f

19. Asymptotic complexity

20. 20 Asymptotic complexity Goal: Estimate properties (e.g., running time) of an algorithm as a function of input size n for large n. Expressed using only the highest-order term in the expression for the exact running time. –Instead of exact running time, say  (n 2 ). Describes behavior of function in the limit.

21. 21 Asymptotic notation , O, , o,  Defined for functions over the natural numbers. –Example: f(n)   (n 2 ). –Describes how f(n) grows in comparison to n 2.  (n 2 ) define a set of functions; in practice used to compare two function sizes. The notations describes the rate-of-growth for a set of functions.

22. 22  -notation (big-Theta)  (g(n)) = { f(n) :  positive constants c 1, c 2, n 0, such that  n  n 0, we have c 1 g(n)  f(n)  c 2 g(n) } For function g(n), we define  (g(n)), big-Theta of n, as the set: g(n) is an asymptotically tight bound for f(n). Intuitively: Set of all functions that have the same rate of growth as g(n).

23. 23  -notation For function g(n), we define  (g(n)), big-Theta of n, as the set: g(n) is an asymptotically tight bound for f(n). Technically, f(n)   (g(n)). Older usage, f(n) =  (g(n)). I’ll accept either…  (g(n)) = { f(n) :  positive constants c 1, c 2, n 0, such that  n  n 0, we have c 1 g(n)  f(n)  c 2 g(n) }

24. 24  -notation For function g(n), we define  (g(n)), big-Theta of n, as the set: If f(n)   (g(n)) Then ? (0, ∞, or a constant )  (g(n)) = { f(n) :  positive constants c 1, c 2, n 0, such that  n  n 0, we have c 1 g(n)  f(n)  c 2 g(n) }

25. 25 Example For f(n) = 10n 2 - 3n   (n 2 ) What constants for n 0, c 1, and c 2 will work? Make c 1 a little smaller than the leading coefficient, and c 2 a little bigger. To compare orders of growth, look at the leading term. Exercise: Prove that n 2 /2-3n   (n 2 )  (g(n)) = { f(n) :  positive constants c 1, c 2, and n 0, such that  n  n 0, c 1 g(n)  f(n)  c 2 g(n) }

26. 26 Example Does n 5   (n 4 )? Does 300n 3   (n 4 )? How about 2 2n   (2 n )? Does lg(n!)   (n lg n)? –Proof using Stirling’s approximation (in the text) for lg(n!).  (g(n)) = { f(n) :  positive constants c 1, c 2, and n 0, such that  n  n 0, c 1 g(n)  f(n)  c 2 g(n) }

27. 27 O-notation O(g(n)) = { f(n):  positive constants c and n 0, such that  n  n 0, we have 0  f(n)  cg(n) } For function g(n), we define O(g(n)), big-O of n, as the set: O(g(n)) is an asymptotic upper bound for f(n). Intuitively: Set of all functions f whose rate of growth is the same as or lower than that of g(n). f(n) =  (g(n))  f(n) = O(g(n)).  (g(n))  O(g(n)).

28. 28 Examples Any linear function an + b is in O(n 2 ). How? Show that 3n 3  O(n 4 ) for appropriate c and n 0. O(g(n)) = {f(n) :  positive constants c and n 0, such that  n  n 0, we have 0  f(n)  cg(n) }

29. 29  -notation (omega)  ( g(n)) is an asymptotic lower bound for f(n). Intuitively: Set of all functions whose rate of growth is the same as or higher than that of g(n). f(n) =  (g(n))  f(n) =  (g(n)).  (g(n))   (g(n)).  (g(n)) = { f(n) :  positive constants c and n 0, such that  n  n 0, we have 0  cg(n)  f(n) } For function g(n), we define  (g(n)), big-Omega of n, as the set:

30. 30 Example  n   ( lg n). Choose c and n 0.  (g(n)) = {f(n) :  positive constants c and n 0, such that  n  n 0, we have 0  cg(n)  f(n)}

31. 31 Relations between , O, 

32. 32 Relations between , , O  (g(n)) = O(g(n))   (g(n)) In practice, asymptotically tight bounds are obtained from asymptotic upper and lower bounds. Theorem: For any two functions g(n) and f(n), f(n)   (g(n)) iff f(n)  O(g(n)) and f(n)   (g(n)).

33. 33 Example Insertion sort takes  (n 2 ) in the worst case, so sorting (as a problem) is O(n 2 ). Why? Any sort algorithm must look at each item, so sorting is  (n). In fact, using (e.g.) merge sort, sorting is  (n lg n) in the worst case. –No comparison sort to do better in the worst case. [We may not see a proof of this result in this course.]

34. 34 Asymptotic notation in equations Can use asymptotic notation in equations to replace expressions containing lower-order terms. For example, 4n 3 + 3n 2 + 2n + 1 = 4n 3 + 3n 2 +  (n) = 4n 3 +  (n 2 ) =  (n 3 ). How do we interpret this? In equations,  (f(n)) always stands for an anonymous function g(n)   (f(n)) –In the example above,  (n 2 ) stands for 3n 2 + 2n + 1.

35. 35 o-notation f(n) becomes insignificant relative to g(n) as n approaches infinity: lim [f(n) / g(n)] = 0. n  g(n) is an upper bound for f(n) that is not asymptotically tight. Observe the difference in this definition from previous ones. Why? o(g(n)) = {f(n):  c > 0,  n 0 > 0 such that  n  n 0, we have 0  f(n) < cg(n)}. For a given function g(n), the set little-o:

36. 36  (g(n)) = {f(n):  c > 0,  n 0 > 0 such that  n  n 0, we have 0  cg(n) < f(n)}.  –notation (little-omega) f(n) becomes arbitrarily large relative to g(n) as n approaches infinity: lim [f(n) / g(n)] = . n  g(n) is a lower bound for f(n) that is not asymptotically tight. For a given function g(n), the set little-omega:

37. 37 Limits lim [f(n) / g(n)] = 0  f(n)   (g(n)) n  lim [f(n) / g(n)] <   f(n)   (g(n)) n  0 < lim [f(n) / g(n)] <   f(n)   (g(n)) n  0 < lim [f(n) / g(n)]  f(n)   (g(n)) n  lim [f(n) / g(n)] =   f(n)   (g(n)) n  lim [f(n) / g(n)] undefined  can not say n 

38. 38 Relationships Subset relations between order-of-growth sets. RRRR  ( f ) O( f )  ( f )  ( f ) o( f ) f

39. 39 Properties Transitivity –f(n) =  (g(n)) & g(n) =  (h(n))  f(n) =  (h(n)) –f(n) = O(g(n)) & g(n) = O(h(n))  f(n) = O(h(n)) –f(n) =  (g(n)) & g(n) =  (h(n))  f(n) =  (h(n)) –f(n) = o (g(n)) & g(n) = o (h(n))  f(n) = o (h(n)) –f(n) =  (g(n)) & g(n) =  (h(n))  f(n) =  (h(n)) Reflexivity –f(n) =  (f(n)) –f(n) = O(f(n)) –f(n) =  (f(n))

40. 40 Properties Symmetry f(n) =  (g(n)) iff g(n) =  (f(n)) Complementarity f(n) = O(g(n)) iff g(n) =  (f(n)) f(n) = o(g(n)) iff g(n) =  ((f(n))

41. Common functions

42. 42 Monotonicity f(n) is –monotonically increasing if m  n  f(m)  f(n). –monotonically decreasing if m  n  f(m)  f(n). –strictly increasing if m < n  f(m) < f(n). –strictly decreasing if m > n  f(m) > f(n).

43. 43 Exponentials Useful (and elementary) Identities: Exponentials and polynomials

44. 44 Logarithms x = log b a is the exponent for a = b x. Natural log: ln a = log e a Binary log: lg a = log 2 a lg 2 a = (lg a) 2 lg lg a = lg (lg a)

45. 45 Logarithms and exponentials – Bases If the base of a logarithm is changed from one constant to another, the value is altered by a constant factor. –Example: log 2 n = log 10 n * log 2 10 –Base of logarithm is not an issue in asymptotic notation. Exponentials with different bases differ by a exponential factor (not a constant factor). –Example: 2 n = (2/3) n *3 n.

46. 46 Exercise Express functions in A in asymptotic notation using functions in B. A B 5n 2 + 100n 3n 2 + 2 A   (n 2 ), n 2   (B)  A   (B) log 3 (n 2 ) log 2 (n 3 ) log b a = log c a / log c b; A = 2lgn / lg3, B = 3lgn, A/B =2/(3lg3) n lg4 3 lg n a log b = b log a ; B =3 lg n =n lg 3 ; A/B =n lg(4/3)   as n  lg 2 n n 1/2 lim ( lg a n / n b ) = 0 (here a = 2 and b = 1/2)  A   (B) n  A   (B) A   (B) A   (B)

47. 47 Wrap-up What are the different asymptotic bounds on functions? How are the asymptotic bounds related? Asymptotic bounds and algorithmic efficiency

48. 48 Procedure mergesort (L=a 1, a 2, ….a n ) if n>1 then m:= n/2 L 1 = a 1, a 2, …. a m L 2 = a m+1, …. a n L = merge (mergesort(L 1 ), mergesort (L 2 )) Recursive proof  Correctness  Complexity