2. 4.1 CompSci 102© Michael Frank Today’s topics AlgorithmsAlgorithms – –Algorithm review – –Orders of Growth Reading: Sections 2.1-3.3Reading: Sections 2.1-3.3 UpcomingUpcoming –Integers

3. 4.2 CompSci 102© Michael Frank §2.1: Algorithms The foundation of computer programming.The foundation of computer programming. Most generally, an algorithm just means a definite procedure for performing some sort of task.Most generally, an algorithm just means a definite procedure for performing some sort of task. A computer program is simply a description of an algorithm, in a language precise enough for a computer to understand, requiring only operations that the computer already knows how to do.A computer program is simply a description of an algorithm, in a language precise enough for a computer to understand, requiring only operations that the computer already knows how to do. We say that a program implements (or “is an implementation of”) its algorithm.We say that a program implements (or “is an implementation of”) its algorithm.

4. 4.3 CompSci 102© Michael Frank Algorithm Characteristics Some important general features of algorithms: Input. Information or data that comes in.Input. Information or data that comes in. Output. Information or data that goes out.Output. Information or data that goes out. Definiteness. Algorithm is precisely defined.Definiteness. Algorithm is precisely defined. Correctness. Outputs correctly relate to inputs.Correctness. Outputs correctly relate to inputs. Finiteness. Won’t take forever to describe or run.Finiteness. Won’t take forever to describe or run. Effectiveness. Individual steps are all do-able.Effectiveness. Individual steps are all do-able. Generality. Works for many possible inputs.Generality. Works for many possible inputs. Efficiency. Takes little time & memory to run.Efficiency. Takes little time & memory to run.

5. 4.4 CompSci 102© Michael Frank Our Pseudocode Language: §A2 procedure name(argument: type) variable := expression informal statement begin statements end {comment} if condition then statement [else statement] for variable := initial value to final value statement while condition statement procname(arguments) Not defined in book: return expression

6. 4.5 CompSci 102© Michael Frank procedure procname(arg: type) Declares that the following text defines a procedure named procname that takes inputs (arguments) named arg which are data objects of the type type.Declares that the following text defines a procedure named procname that takes inputs (arguments) named arg which are data objects of the type type. –Example: procedure maximum(L: list of integers) [statements defining maximum…]

7. 4.6 CompSci 102© Michael Frank variable := expression An assignment statement evaluates the expression expression, then reassigns the variable variable to the value that results.An assignment statement evaluates the expression expression, then reassigns the variable variable to the value that results. –Example assignment statement: v := 3x+7 (If x is 2, changes v to 13.) In pseudocode (but not real code), the expression might be informally stated:In pseudocode (but not real code), the expression might be informally stated: –x := the largest integer in the list L

8. 4.7 CompSci 102© Michael Frank Informal statement Sometimes we may write a statement as an informal English imperative, if the meaning is still clear and precise: e.g.,“swap x and y”Sometimes we may write a statement as an informal English imperative, if the meaning is still clear and precise: e.g.,“swap x and y” Keep in mind that real programming languages never allow this.Keep in mind that real programming languages never allow this. When we ask for an algorithm to do so-and- so, writing “Do so-and-so” isn’t enough!When we ask for an algorithm to do so-and- so, writing “Do so-and-so” isn’t enough! –Break down algorithm into detailed steps.

9. 4.8 CompSci 102© Michael Frank begin statements end Groups a sequence of statements together: begin statement 1 statement 2 … statement n endGroups a sequence of statements together: begin statement 1 statement 2 … statement n end Allows the sequence to be used just like a single statement. Might be used: –After a procedure declaration. –In an if statement after then or else. –In the body of a for or while loop. Curly braces {} are used instead in many languages.

10. 4.9 CompSci 102© Michael Frank {comment} Not executed (does nothing).Not executed (does nothing). Natural-language text explaining some aspect of the procedure to human readers.Natural-language text explaining some aspect of the procedure to human readers. Also called a remark in some real programming languages, e.g. BASIC.Also called a remark in some real programming languages, e.g. BASIC. Example, might appear in a max program:Example, might appear in a max program: –{Note that v is the largest integer seen so far.}

11. 4.10 CompSci 102© Michael Frank if condition then statement Evaluate the propositional expression condition.Evaluate the propositional expression condition. –If the resulting truth value is True, then execute the statement statement; –otherwise, just skip on ahead to the next statement after the if statement. Variant: if cond then stmt1 else stmt2Variant: if cond then stmt1 else stmt2 –Like before, but iff truth value is False, executes stmt2.

12. 4.11 CompSci 102© Michael Frank while condition statement Evaluate the propositional (Boolean) expression condition.Evaluate the propositional (Boolean) expression condition. If the resulting value is True, then execute statement.If the resulting value is True, then execute statement. Continue repeating the above two actions over and over until finally the condition evaluates to False; then proceed to the next statement.Continue repeating the above two actions over and over until finally the condition evaluates to False; then proceed to the next statement.

13. 4.12 CompSci 102© Michael Frank while condition statement Also equivalent to infinite nested ifs, like so: if condition begin statement if condition begin statement …(continue infinite nested if’s) end endAlso equivalent to infinite nested ifs, like so: if condition begin statement if condition begin statement …(continue infinite nested if’s) end end

14. 4.13 CompSci 102© Michael Frank for var := initial to final stmt Initial is an integer expression.Initial is an integer expression. Final is another integer expression.Final is another integer expression. Semantics: Repeatedly execute stmt, first with variable var := initial, then with var := initial+1, then with var := initial+2, etc., then finally with var := final.Semantics: Repeatedly execute stmt, first with variable var := initial, then with var := initial+1, then with var := initial+2, etc., then finally with var := final. Question: What happens if stmt changes the value of var, or the value that initial or final evaluates to?Question: What happens if stmt changes the value of var, or the value that initial or final evaluates to?

15. 4.14 CompSci 102© Michael Frank for var := initial to final stmt For can be exactly defined in terms of while, like so:For can be exactly defined in terms of while, like so: begin var := initial while var  final begin stmt var := var + 1 end end

16. 4.15 CompSci 102© Michael Frank procedure(argument) A procedure call statement invokes the named procedure, giving it as its input the value of the argument expression.A procedure call statement invokes the named procedure, giving it as its input the value of the argument expression. Various real programming languages refer to procedures as functions (since the procedure call notation works similarly to function application f(x)), or as subroutines, subprograms, or methods.Various real programming languages refer to procedures as functions (since the procedure call notation works similarly to function application f(x)), or as subroutines, subprograms, or methods.

17. 4.16 CompSci 102© Michael Frank Max procedure in pseudocode procedure max(a 1, a 2, …, a n : integers) v := a 1 {largest element so far} v := a 1 {largest element so far} for i := 2 to n {go thru rest of elems} for i := 2 to n {go thru rest of elems} if a i > v then v := a i {found bigger?} if a i > v then v := a i {found bigger?} {at this point v’s value is the same as the largest integer in the list} {at this point v’s value is the same as the largest integer in the list} return v return v

18. 4.17 CompSci 102© Michael Frank Inventing an Algorithm Requires a lot of creativity and intuitionRequires a lot of creativity and intuition –Like writing proofs. Unfortunately, we can’t give you an algorithm for inventing algorithms.Unfortunately, we can’t give you an algorithm for inventing algorithms. –Just look at lots of examples… –And practice (preferably, on a computer) –And look at more examples… –And practice some more… etc., etc.

19. 4.18 CompSci 102© Michael Frank Algorithm-Inventing Example Suppose we ask you to write an algorithm to compute the predicate:Suppose we ask you to write an algorithm to compute the predicate: IsPrime:N→{T,F} –Computes whether a given natural number is a prime number. First, start with a correct predicate-logic definition of the desired function:First, start with a correct predicate-logic definition of the desired function:  n: IsPrime(n)  ¬  1<d<n: d|n Means d divides n evenly (without remainder)

20. 4.19 CompSci 102© Michael Frank IsPrime example, cont. Notice that the negated exponential can be rewritten as a universal:Notice that the negated exponential can be rewritten as a universal: ¬  1<d<n: d|n   1<d<n: d | n   2≤ d ≤ n−1: d | n This universal can then be translated directly into a corresponding for loop:This universal can then be translated directly into a corresponding for loop: for d := 2 to n−1 { Try all potential divisors >1 & 1 & <n } if d|n then return F { n has divisor d; not prime } return T { no divisors were found; n must be prime} Means d does not divide n evenly (the remainder is ≠0)

21. 4.20 CompSci 102© Michael Frank Optimizing IsPrime The IsPrime algorithm can be further optimized:The IsPrime algorithm can be further optimized: for d := 2 to  n 1/2  if d|n then return F return T This works because of this theorem: If n has any (integer) divisors, it must have one less than n 1/2.This works because of this theorem: If n has any (integer) divisors, it must have one less than n 1/2. Proof: Suppose n’s smallest divisor >1 is a, and let b :≡ n/a. Then n = ab, but if a > n 1/2 then b > n 1/2 (since a is n’s smallest divisor) and so n = ab > (n 1/2 ) 2 = n, an absurdity. Further optimizations are possible: - E.g., only try divisors that are primes less than n 1/2. Note smaller range of search.

22. 4.21 CompSci 102© Michael Frank Another example task Problem of searching an ordered list.Problem of searching an ordered list. –Given a list L of n elements that are sorted into a definite order (e.g., numeric, alphabetical), –And given a particular element x, –Determine whether x appears in the list, –and if so, return its index (position) in the list. Problem occurs often in many contexts.Problem occurs often in many contexts. Let’s find an efficient algorithm!Let’s find an efficient algorithm!

23. 4.22 CompSci 102© Michael Frank Search alg. #1: Linear Search procedure linear search (x: integer, a 1, a 2, …, a n : distinct integers) i := 1 {start at beginning of list} while (i  n  x  a i ) {not done, not found} i := i + 1 {go to the next position} if i  n then location := i {it was found} else location := 0 {it wasn’t found} return location {index or 0 if not found}

24. 4.23 CompSci 102© Michael Frank Search alg. #2: Binary Search Basic idea: On each step, look at the middle element of the remaining list to eliminate half of it, and quickly zero in on the desired element.Basic idea: On each step, look at the middle element of the remaining list to eliminate half of it, and quickly zero in on the desired element. <x<x>x>x<x<x<x<x

25. 4.24 CompSci 102© Michael Frank Search alg. #2: Binary Search procedure binary search (x:integer, a 1, a 2, …, a n : distinct integers) i := 1 {left endpoint of search interval} j := n {right endpoint of search interval} while i 1 item} m :=  (i+j)/2  {midpoint} if x>a m then i := m+1 else j := m end if x = a i then location := i else location := 0 return location

26. 4.25 CompSci 102© Michael Frank Practice exercises 2.1.3: Devise an algorithm that finds the sum of all the integers in a list. [2 min]2.1.3: Devise an algorithm that finds the sum of all the integers in a list. [2 min]

27. 4.26 CompSci 102© Michael Frank Orders of Growth - Motivation Suppose you are designing a web site to process user data (e.g., financial records).Suppose you are designing a web site to process user data (e.g., financial records). Suppose database program A takes f A (n)=30n+8 microseconds to process any n records, while program B takes f B (n)=n 2 +1 microseconds to process the n records.Suppose database program A takes f A (n)=30n+8 microseconds to process any n records, while program B takes f B (n)=n 2 +1 microseconds to process the n records. Which program do you choose, knowing you’ll want to support millions of users?Which program do you choose, knowing you’ll want to support millions of users?

28. 4.27 CompSci 102© Michael Frank Visualizing Orders of Growth On a graph, as you go to the right, the faster- growing func- tion always eventually becomes the larger one...On a graph, as you go to the right, the faster- growing func- tion always eventually becomes the larger one... f A (n)=30n+8 Increasing n  f B (n)=n 2 +1 Value of function 

29. 4.28 CompSci 102© Michael Frank Concept of order of growth We say f A (n)=30n+8 is (at most) order n, or O(n).We say f A (n)=30n+8 is (at most) order n, or O(n). –It is, at most, roughly proportional to n. f B (n)=n 2 +1 is order n 2, or O(n 2 ).f B (n)=n 2 +1 is order n 2, or O(n 2 ). –It is (at most) roughly proportional to n 2. Any function whose exact (tightest) order is O(n 2 ) is faster-growing than any O(n) function.Any function whose exact (tightest) order is O(n 2 ) is faster-growing than any O(n) function. –Later we will introduce Θ for expressing exact order. For large numbers of user records, the exactly order n 2 function will always take more time.For large numbers of user records, the exactly order n 2 function will always take more time.

30. 4.29 CompSci 102© Michael Frank Definition: O(g), at most order g Let g be any function R  R. Define “at most order g”, written O(g), to be: {f:R  R |  c,k:  x>k: f(x)  cg(x)}.Define “at most order g”, written O(g), to be: {f:R  R |  c,k:  x>k: f(x)  cg(x)}. –“Beyond some point k, function f is at most a constant c times g (i.e., proportional to g).” “f is at most order g”, or “f is O(g)”, or “f=O(g)” all just mean that f  O(g).“f is at most order g”, or “f is O(g)”, or “f=O(g)” all just mean that f  O(g). Often the phrase “at most” is omitted.Often the phrase “at most” is omitted.

31. 4.30 CompSci 102© Michael Frank Points about the definition Note that f is O(g) so long as any values of c and k exist that satisfy the definition.Note that f is O(g) so long as any values of c and k exist that satisfy the definition. But: The particular c, k, values that make the statement true are not unique: Any larger value of c and/or k will also work.But: The particular c, k, values that make the statement true are not unique: Any larger value of c and/or k will also work. You are not required to find the smallest c and k values that work. (Indeed, in some cases, there may be no smallest values!)You are not required to find the smallest c and k values that work. (Indeed, in some cases, there may be no smallest values!) However, you should prove that the values you choose do work.

32. 4.31 CompSci 102© Michael Frank “Big-O” Proof Examples Show that 30n+8 is O(n).Show that 30n+8 is O(n). –Show  c,k:  n>k: 30n+8  cn. Let c=31, k=8. Assume n>k=8. Then cn = 31n = 30n + n > 30n+8, so 30n+8 k=8. Then cn = 31n = 30n + n > 30n+8, so 30n+8 < cn. Show that n 2 +1 is O(n 2 ).Show that n 2 +1 is O(n 2 ). –Show  c,k:  n>k: n 2 +1  cn 2. Let c=2, k=1. Assume n>1. Then cn 2 = 2n 2 = n 2 +n 2 > n 2 +1, or n 2 +1 1. Then cn 2 = 2n 2 = n 2 +n 2 > n 2 +1, or n 2 +1< cn 2.

33. 4.32 CompSci 102© Michael Frank Note 30n+8 isn’t less than n anywhere (n>0).Note 30n+8 isn’t less than n anywhere (n>0). It isn’t even less than 31n everywhere.It isn’t even less than 31n everywhere. But it is less than 31n everywhere to the right of n=8.But it is less than 31n everywhere to the right of n=8. n>k=8  Big-O example, graphically Increasing n  Value of function  n 30n+8 cn = 31n 30n+8  O(n)

34. 4.33 CompSci 102© Michael Frank Useful Facts about Big O Big O, as a relation, is transitive: f  O(g)  g  O(h)  f  O(h)Big O, as a relation, is transitive: f  O(g)  g  O(h)  f  O(h) O with constant multiples, roots, and logs...  f (in  (1)) & constants a,b  R, with b  0, af, f 1-b, and (log b f) a are all O(f).O with constant multiples, roots, and logs...  f (in  (1)) & constants a,b  R, with b  0, af, f 1-b, and (log b f) a are all O(f). Sums of functions: If g  O(f) and h  O(f), then g+h  O(f).Sums of functions: If g  O(f) and h  O(f), then g+h  O(f).

35. 4.34 CompSci 102© Michael Frank More Big-O facts  c>0, O(cf)=O(f+c)=O(f  c)=O(f)  c>0, O(cf)=O(f+c)=O(f  c)=O(f) f 1  O(g 1 )  f 2  O(g 2 ) f 1  O(g 1 )  f 2  O(g 2 )  –f 1 f 2  O(g 1 g 2 ) –f 1 +f 2  O(g 1 +g 2 ) = O(max(g 1,g 2 )) = O(g 1 ) if g 2  O(g 1 ) (Very useful!)

36. 4.35 CompSci 102© Michael Frank Orders of Growth (§1.8) - So Far For any g:R  R, “at most order g”, O(g)  {f:R  R |  c,k  x>k |f(x)|  |cg(x)|}.For any g:R  R, “at most order g”, O(g)  {f:R  R |  c,k  x>k |f(x)|  |cg(x)|}. –Often, one deals only with positive functions and can ignore absolute value symbols. “f  O(g)” often written “f is O(g)” or “f=O(g)”.“f  O(g)” often written “f is O(g)” or “f=O(g)”. –The latter form is an instance of a more general convention...

37. 4.36 CompSci 102© Michael Frank Order-of-Growth Expressions “O(f)” when used as a term in an arithmetic expression means: “some function f such that f  O(f)”.“O(f)” when used as a term in an arithmetic expression means: “some function f such that f  O(f)”. E.g.: “x 2 +O(x)” means “x 2 plus some function that is O(x)”.E.g.: “x 2 +O(x)” means “x 2 plus some function that is O(x)”. Formally, you can think of any such expression as denoting a set of functions: x 2 +O(x) :  {g |  f  O(x): g(x)= x 2 +f(x)}Formally, you can think of any such expression as denoting a set of functions: x 2 +O(x) :  {g |  f  O(x): g(x)= x 2 +f(x)}

38. 4.37 CompSci 102© Michael Frank Order of Growth Equations Suppose E 1 and E 2 are order-of-growth expressions corresponding to the sets of functions S and T, respectively.Suppose E 1 and E 2 are order-of-growth expressions corresponding to the sets of functions S and T, respectively. Then the “equation” E 1 =E 2 really means  f  S,  g  T : f=g or simply S  T.Then the “equation” E 1 =E 2 really means  f  S,  g  T : f=g or simply S  T. Example: x 2 + O(x) = O(x 2 ) means  f  O(x):  g  O(x 2 ): x 2 +f(x)=g(x)Example: x 2 + O(x) = O(x 2 ) means  f  O(x):  g  O(x 2 ): x 2 +f(x)=g(x)

39. 4.38 CompSci 102© Michael Frank Useful Facts about Big O  f,g & constants a,b  R, with b  0,  f,g & constants a,b  R, with b  0, –af = O(f); (e.g. 3x 2 = O(x 2 )) –f+O(f) = O(f); (e.g. x 2 +x = O(x 2 )) Also, if f=  (1) (at least order 1), then:Also, if f=  (1) (at least order 1), then: –|f| 1-b = O(f); (e.g. x  1 = O(x)) –(log b |f|) a = O(f). (e.g. log x = O(x)) –g=O(fg) (e.g. x = O(x log x)) –fg  O(g) (e.g. x log x  O(x)) –a=O(f) (e.g. 3 = O(x))

40. 4.39 CompSci 102© Michael Frank Definition:  (g), exactly order g If f  O(g) and g  O(f), then we say “g and f are of the same order” or “f is (exactly) order g” and write f  (g).If f  O(g) and g  O(f), then we say “g and f are of the same order” or “f is (exactly) order g” and write f  (g). Another, equivalent definition:  (g)  {f:R  R |  c 1 c 2 k>0  x>k: |c 1 g(x)|  |f(x)|  |c 2 g(x)| }Another, equivalent definition:  (g)  {f:R  R |  c 1 c 2 k>0  x>k: |c 1 g(x)|  |f(x)|  |c 2 g(x)| } –“Everywhere beyond some point k, f(x) lies in between two multiples of g(x).”

41. 4.40 CompSci 102© Michael Frank Rules for  Mostly like rules for O( ), except:Mostly like rules for O( ), except:  f,g>0 & constants a,b  R, with b>0, af   (f), but  Same as with O. f   (fg) unless g=  (1)  Unlike O. |f| 1-b   (f), and  Unlike with O. (log b |f|) c   (f).  Unlike with O.  f,g>0 & constants a,b  R, with b>0, af   (f), but  Same as with O. f   (fg) unless g=  (1)  Unlike O. |f| 1-b   (f), and  Unlike with O. (log b |f|) c   (f).  Unlike with O. The functions in the latter two cases we say are strictly of lower order than  (f).The functions in the latter two cases we say are strictly of lower order than  (f).

42. 4.41 CompSci 102© Michael Frank  example Determine whether:Determine whether: Quick solution:Quick solution:

43. 4.42 CompSci 102© Michael Frank Other Order-of-Growth Relations  (g) = {f | g  O(f)} “The functions that are at least order g.”  (g) = {f | g  O(f)} “The functions that are at least order g.” o(g) = {f |  c>0  k  x>k : |f(x)| 0  k  x>k : |f(x)| < |cg(x)|} “The functions that are strictly lower order than g.” o(g)  O(g)   (g).  (g) = {f |  c>0  k  x>k : |cg(x)| 0  k  x>k : |cg(x)| < |f(x)|} “The functions that are strictly higher order than g.”  (g)   (g)   (g).

44. 4.43 CompSci 102© Michael Frank Subset relations between order-of-growth sets.Subset relations between order-of-growth sets. Relations Between the Relations RRRR  ( f ) O( f )  ( f )  ( f ) o( f ) f

45. 4.44 CompSci 102© Michael Frank Why o(f)  O(x)  (x) A function that is O(x), but neither o(x) nor  (x):A function that is O(x), but neither o(x) nor  (x):

46. 4.45 CompSci 102© Michael Frank Strict Ordering of Functions Temporarily let’s write f  g to mean f  o(g), f~g to mean f  (g)Temporarily let’s write f  g to mean f  o(g), f~g to mean f  (g) Note that:Note that: Let k>1. Then the following are true: 1  log log n  log n ~ log k n  log k n  n 1/k  n  n log n  n k  k n  n!  n n …Let k>1. Then the following are true: 1  log log n  log n ~ log k n  log k n  n 1/k  n  n log n  n k  k n  n!  n n …

47. 4.46 CompSci 102© Michael Frank Review: Orders of Growth (§1.8) Definitions of order-of-growth sets,  g:R  R O(g) :  {f |  c>0  k  x>k |f(x)| 0  k  x>k |f(x)| < |cg(x)|} o(g) :  {f |  c>0  k  x>k |f(x)| 0  k  x>k |f(x)| < |cg(x)|}  (g) :  {f | g  O(f)}  (g) :  {f | g  O(f)}  (g) :  {f | g  o(f)}  (g) :  {f | g  o(f)}  (g) :  O(g)   (g)  (g) :  O(g)   (g)