Module #5 - Algorithms 1 Module #5: Algorithms Rosen 5 th ed., §2.1 ~31 slides, ~1 lecture

Module #5 - Algorithms 2 Chapter 2: More Fundamentals §2.1: Algorithms (Formal procedures)§2.1: Algorithms (Formal procedures) §2.2: Complexity of algorithms§2.2: Complexity of algorithms –Analysis using order-of-growth notation. §2.3: The Integers & Division§2.3: The Integers & Division –Some basic number theory. §2.6: Matrices§2.6: Matrices –Some basic linear algebra.

Module #5 - Algorithms 3 §2.1: Algorithms The foundation of computer programming.The foundation of computer programming. Most generally, an algorithm just means a definite procedure for performing some sort of task.Most generally, an algorithm just means a definite procedure for performing some sort of task. A computer program is simply a description of an algorithm in a language precise enough for a computer to understand, requiring only operations the computer already knows how to do.A computer program is simply a description of an algorithm in a language precise enough for a computer to understand, requiring only operations the computer already knows how to do. We say that a program implements (or “is an implementation of”) its algorithm.We say that a program implements (or “is an implementation of”) its algorithm.

Module #5 - Algorithms 4 Algorithms You Already Know Grade school arithmetic algorithms:Grade school arithmetic algorithms: –How to add any two natural numbers written in decimal on paper using carries. –Similar: Subtraction using borrowing. –Multiplication & long division. Your favorite cooking recipe.Your favorite cooking recipe. How to register for classes at UF.How to register for classes at UF.

Module #5 - Algorithms 5 Programming Languages Some common programming languages:Some common programming languages: –Newer: Java, C, C++, Visual Basic, JavaScript, Perl, Tcl, Pascal –Older: Fortran, Cobol, Lisp, Basic –Assembly languages, for low-level coding. In this class we will use an informal, Pascal- like “pseudo-code” language.In this class we will use an informal, Pascal- like “pseudo-code” language. You should know at least 1 real language!You should know at least 1 real language!

Module #5 - Algorithms 6 Algorithm Example (English) Task: Given a sequence {a i }=a 1,…,a n, a i  N, say what its largest element is.Task: Given a sequence {a i }=a 1,…,a n, a i  N, say what its largest element is. Set the value of a temporary variable v (largest element seen so far) to a 1 ’s value.Set the value of a temporary variable v (largest element seen so far) to a 1 ’s value. Look at the next element a i in the sequence.Look at the next element a i in the sequence. If a i >v, then re-assign v to the number a i.If a i >v, then re-assign v to the number a i. Repeat previous 2 steps until there are no more elements in the sequence, & return v.Repeat previous 2 steps until there are no more elements in the sequence, & return v.

Module #5 - Algorithms 7 Executing an Algorithm When you start up a piece of software, we say the program or its algorithm are being run or executed by the computer.When you start up a piece of software, we say the program or its algorithm are being run or executed by the computer. Given a description of an algorithm, you can also execute it by hand, by working through all of its steps on paper.Given a description of an algorithm, you can also execute it by hand, by working through all of its steps on paper. Before ~WWII, “computer” meant a person whose job was to run algorithms!Before ~WWII, “computer” meant a person whose job was to run algorithms!

Module #5 - Algorithms 8 Executing the Max algorithm Let {a i }=7,12,3,15,8. Find its maximum…Let {a i }=7,12,3,15,8. Find its maximum… Set v = a 1 = 7.Set v = a 1 = 7. Look at next element: a 2 = 12.Look at next element: a 2 = 12. Is a 2 >v? Yes, so change v to 12.Is a 2 >v? Yes, so change v to 12. Look at next element: a 2 = 3.Look at next element: a 2 = 3. Is 3>12? No, leave v alone….Is 3>12? No, leave v alone…. Is 15>12? Yes, v=15…Is 15>12? Yes, v=15…

Module #5 - Algorithms 9 Algorithm Characteristics Some important features of algorithms: Input. Information or data that comes in.Input. Information or data that comes in. Output. Information or data that goes out.Output. Information or data that goes out. Definiteness. Precisely defined.Definiteness. Precisely defined. Correctness. Outputs correctly relate to inputs.Correctness. Outputs correctly relate to inputs. Finiteness. Won’t take forever to describe or run.Finiteness. Won’t take forever to describe or run. Effectiveness. Individual steps are all do-able.Effectiveness. Individual steps are all do-able. Generality. Works for many possible inputs.Generality. Works for many possible inputs. Efficiency. Takes little time & memory to run.Efficiency. Takes little time & memory to run.

Module #5 - Algorithms 10 Our Pseudocode Language: §A2 procedure name(argument: type) variable := expression informal statement begin statements end {comment} if condition then statement [else statement] for variable := initial value to final value statement while condition statement procname(arguments) Not defined in book: return expression

Module #5 - Algorithms 11 procedure procname(arg: type) Declares that the following text defines a procedure named procname that takes inputs (arguments) named arg which are data objects of the type type.Declares that the following text defines a procedure named procname that takes inputs (arguments) named arg which are data objects of the type type. –Example: procedure maximum(L: list of integers) [statements defining maximum…]

Module #5 - Algorithms 12 variable := expression An assignment statement evaluates the expression expression, then reassigns the variable variable to the value that results.An assignment statement evaluates the expression expression, then reassigns the variable variable to the value that results. –Example: v := 3x+7 (If x is 2, changes v to 13.) In pseudocode (but not real code), the expression might be informal:In pseudocode (but not real code), the expression might be informal: –x := the largest integer in the list L

Module #5 - Algorithms 13 Informal statement Sometimes we may write a statement as an informal English imperative, if the meaning is still clear and precise: “swap x and y”Sometimes we may write a statement as an informal English imperative, if the meaning is still clear and precise: “swap x and y” Keep in mind that real programming languages never allow this.Keep in mind that real programming languages never allow this. When we ask for an algorithm to do so-and- so, writing “Do so-and-so” isn’t enough!When we ask for an algorithm to do so-and- so, writing “Do so-and-so” isn’t enough! –Break down algorithm into detailed steps.

Module #5 - Algorithms 14 begin statements end Groups a sequence of statements together: begin statement 1 statement 2 … statement n endGroups a sequence of statements together: begin statement 1 statement 2 … statement n end Allows sequence to be used like a single statement. Might be used: –After a procedure declaration. –In an if statement after then or else. –In the body of a for or while loop.

Module #5 - Algorithms 15 {comment} Not executed (does nothing).Not executed (does nothing). Natural-language text explaining some aspect of the procedure to human readers.Natural-language text explaining some aspect of the procedure to human readers. Also called a remark in some real programming languages.Also called a remark in some real programming languages. Example:Example: –{Note that v is the largest integer seen so far.}

Module #5 - Algorithms 16 if condition then statement Evaluate the propositional expression condition.Evaluate the propositional expression condition. If the resulting truth value is true, then execute the statement statement; otherwise, just skip on ahead to the next statement.If the resulting truth value is true, then execute the statement statement; otherwise, just skip on ahead to the next statement. Variant: if cond then stmt1 else stmt2 Like before, but iff truth value is false, executes stmt2.Variant: if cond then stmt1 else stmt2 Like before, but iff truth value is false, executes stmt2.

Module #5 - Algorithms 17 while condition statement Evaluate the propositional expression condition.Evaluate the propositional expression condition. If the resulting value is true, then execute statement.If the resulting value is true, then execute statement. Continue repeating the above two actions over and over until finally the condition evaluates to false; then go on to the next statement.Continue repeating the above two actions over and over until finally the condition evaluates to false; then go on to the next statement.

Module #5 - Algorithms 18 while condition statement Also equivalent to infinite nested ifs, like so: if condition begin statement if condition begin statement …(continue infinite nested if’s) end endAlso equivalent to infinite nested ifs, like so: if condition begin statement if condition begin statement …(continue infinite nested if’s) end end

Module #5 - Algorithms 19 for var := initial to final stmt Initial is an integer expression.Initial is an integer expression. Final is another integer expression.Final is another integer expression. Repeatedly execute stmt, first with variable var := initial, then with var := initial+1, then with var := initial+2, etc., then finally with var := final.Repeatedly execute stmt, first with variable var := initial, then with var := initial+1, then with var := initial+2, etc., then finally with var := final. What happens if stmt changes the value that initial or final evaluates to?What happens if stmt changes the value that initial or final evaluates to?

Module #5 - Algorithms 20 for var := initial to final stmt For can be exactly defined in terms of while, like so:For can be exactly defined in terms of while, like so: begin var := initial while var  final begin stmt var := var + 1 end end

Module #5 - Algorithms 21 procedure(argument) A procedure call statement invokes the named procedure, giving it as its input the value of the argument expression.A procedure call statement invokes the named procedure, giving it as its input the value of the argument expression. Various real programming languages refer to procedures as functions (since the procedure call notation works similarly to function application f(x)), or as subroutines, subprograms, or methods.Various real programming languages refer to procedures as functions (since the procedure call notation works similarly to function application f(x)), or as subroutines, subprograms, or methods.

Module #5 - Algorithms 22 Max procedure in pseudocode procedure max(a 1, a 2, …, a n : integers) v := a 1 {largest element so far} v := a 1 {largest element so far} for i := 2 to n {go thru rest of elems} for i := 2 to n {go thru rest of elems} if a i > v then v := a i {found bigger?} if a i > v then v := a i {found bigger?} {at this point v’s value is the same as the largest integer in the list} {at this point v’s value is the same as the largest integer in the list} return v return v

Module #5 - Algorithms 23 Another example task Problem of searching an ordered list.Problem of searching an ordered list. –Given a list L of n elements that are sorted into a definite order (e.g., numeric, alphabetical), –And given a particular element x, –Determine whether x appears in the list, –and if so, return its index (position) in the list. Problem occurs often in many contexts.Problem occurs often in many contexts. Let’s find an efficient algorithm!Let’s find an efficient algorithm!

Module #5 - Algorithms 24 Search alg. #1: Linear Search procedure linear search (x: integer, a 1, a 2, …, a n : distinct integers) i := 1 while (i  n  x  a i ) i := i + 1 if i  n then location := i else location := 0 return location {index or 0 if not found}

Module #5 - Algorithms 25 Search alg. #2: Binary Search Basic idea: On each step, look at the middle element of the remaining list to eliminate half of it, and quickly zero in on the desired element.Basic idea: On each step, look at the middle element of the remaining list to eliminate half of it, and quickly zero in on the desired element. <x<x>x>x<x<x<x<x

Module #5 - Algorithms 26 Search alg. #2: Binary Search procedure binary search (x:integer, a 1, a 2, …, a n : distinct integers) i := 1 {left endpoint of search interval} j := n {right endpoint of search interval} while i 1 item} m :=  (i+j)/2  {midpoint} if x>a m then i := m+1 else j := m end if x = a i then location := i else location := 0 return location

Module #5 - Algorithms 27 Module #6: Orders of Growth Rosen 5 th ed., §2.2 ~22 slides, ~1 lecture

Module #5 - Algorithms 28 Orders of Growth (§1.8) For functions over numbers, we often need to know a rough measure of how fast a function grows.For functions over numbers, we often need to know a rough measure of how fast a function grows. If f(x) is faster growing than g(x), then f(x) always eventually becomes larger than g(x) in the limit (for large enough values of x).If f(x) is faster growing than g(x), then f(x) always eventually becomes larger than g(x) in the limit (for large enough values of x). Useful in engineering for showing that one design scales better or worse than another.Useful in engineering for showing that one design scales better or worse than another.

Module #5 - Algorithms 29 Orders of Growth - Motivation Suppose you are designing a web site to process user data (e.g., financial records).Suppose you are designing a web site to process user data (e.g., financial records). Suppose database program A takes f A (n)=30n+8 microseconds to process any n records, while program B takes f B (n)=n 2 +1 microseconds to process the n records.Suppose database program A takes f A (n)=30n+8 microseconds to process any n records, while program B takes f B (n)=n 2 +1 microseconds to process the n records. Which program do you choose, knowing you’ll want to support millions of users?Which program do you choose, knowing you’ll want to support millions of users?

Module #5 - Algorithms 30 Visualizing Orders of Growth On a graph, as you go to the right, a faster growing function eventually becomes larger...On a graph, as you go to the right, a faster growing function eventually becomes larger... f A (n)=30n+8 Increasing n  f B (n)=n 2 +1 Value of function 

Module #5 - Algorithms 31 Concept of order of growth We say f A (n)=30n+8 is order n, or O(n). It is, at most, roughly proportional to n.We say f A (n)=30n+8 is order n, or O(n). It is, at most, roughly proportional to n. f B (n)=n 2 +1 is order n 2, or O(n 2 ). It is roughly proportional to n 2.f B (n)=n 2 +1 is order n 2, or O(n 2 ). It is roughly proportional to n 2. Any O(n 2 ) function is faster-growing than any O(n) function.Any O(n 2 ) function is faster-growing than any O(n) function. For large numbers of user records, the O(n 2 ) function will always take more time.For large numbers of user records, the O(n 2 ) function will always take more time.

Module #5 - Algorithms 32 Definition: O(g), at most order g Let g be any function R  R. Define “at most order g”, written O(g), to be: {f:R  R |  c,k:  x>k: f(x)  cg(x)}.Define “at most order g”, written O(g), to be: {f:R  R |  c,k:  x>k: f(x)  cg(x)}. –“Beyond some point k, function f is at most a constant c times g (i.e., proportional to g).” “f is at most order g”, or “f is O(g)”, or “f=O(g)” all just mean that f  O(g).“f is at most order g”, or “f is O(g)”, or “f=O(g)” all just mean that f  O(g). Sometimes the phrase “at most” is omitted.Sometimes the phrase “at most” is omitted.

Module #5 - Algorithms 33 Points about the definition Note that f is O(g) so long as any values of c and k exist that satisfy the definition.Note that f is O(g) so long as any values of c and k exist that satisfy the definition. But: The particular c, k, values that make the statement true are not unique: Any larger value of c and/or k will also work.But: The particular c, k, values that make the statement true are not unique: Any larger value of c and/or k will also work. You are not required to find the smallest c and k values that work. (Indeed, in some cases, there may be no smallest values!)You are not required to find the smallest c and k values that work. (Indeed, in some cases, there may be no smallest values!) However, you should prove that the values you choose do work.

Module #5 - Algorithms 34 Example 1

Module #5 - Algorithms 35

Module #5 - Algorithms 36 “Big-O” Proof Examples Show that 30n+8 is O(n).Show that 30n+8 is O(n). –Show  c,k:  n>k: 30n+8  cn. Let c=31, k=8. Assume n>k=8. Then cn = 31n = 30n + n > 30n+8, so 30n+8 k=8. Then cn = 31n = 30n + n > 30n+8, so 30n+8 < cn. Show that n 2 +1 is O(n 2 ).Show that n 2 +1 is O(n 2 ). –Show  c,k:  n>k: n 2 +1  cn 2. Let c=2, k=1. Assume n>1. Then cn 2 = 2n 2 = n 2 +n 2 > n 2 +1, or n Then cn 2 = 2n 2 = n 2 +n 2 > n 2 +1, or n 2 +1< cn 2.

Module #5 - Algorithms 37 Note 30n+8 isn’t less than n anywhere (n>0).Note 30n+8 isn’t less than n anywhere (n>0). It isn’t even less than 31n everywhere.It isn’t even less than 31n everywhere. But it is less than 31n everywhere to the right of n=8.But it is less than 31n everywhere to the right of n=8. n>k=8  Big-O example, graphically Increasing n  Value of function  n 30n+8 cn = 31n 30n+8  O(n)

Module #5 - Algorithms 38

Module #5 - Algorithms 39

Module #5 - Algorithms 40 Example 2

Module #5 - Algorithms 41 Example 3

Module #5 - Algorithms 42 Theorem 1

Module #5 - Algorithms 43 Example 4

Module #5 - Algorithms 44 Example 5

Module #5 - Algorithms 45 Example 6

Module #5 - Algorithms 46

Module #5 - Algorithms 47 Useful Facts about Big O Big O, as a relation, is transitive: f  O(g)  g  O(h)  f  O(h)Big O, as a relation, is transitive: f  O(g)  g  O(h)  f  O(h) O with constant multiples, roots, and logs...  f (in  (1)) & constants a,b  R, with b  0, af, f 1-b, and (log b f) a are all O(f).O with constant multiples, roots, and logs...  f (in  (1)) & constants a,b  R, with b  0, af, f 1-b, and (log b f) a are all O(f). Sums of functions: If g  O(f) and h  O(f), then g+h  O(f).Sums of functions: If g  O(f) and h  O(f), then g+h  O(f).

Module #5 - Algorithms 48 More Big-O facts  c>0, O(cf)=O(f+c)=O(f  c)=O(f)  c>0, O(cf)=O(f+c)=O(f  c)=O(f) f 1  O(g 1 )  f 2  O(g 2 ) f 1  O(g 1 )  f 2  O(g 2 )  –f 1 f 2  O(g 1 g 2 ) –f 1 +f 2  O(g 1 +g 2 ) = O(max(g 1,g 2 )) = O(g 1 ) if g 2  O(g 1 ) (Very useful!)

Module #5 - Algorithms 49

Module #5 - Algorithms 50 Theorem 3

Module #5 - Algorithms 51 Example 7

Module #5 - Algorithms 52 Example 8

Module #5 - Algorithms 53 Orders of Growth (§1.8) - So Far For any g:R  R, “at most order g”, O(g)  {f:R  R |  c,k  x>k |f(x)|  |cg(x)|}.For any g:R  R, “at most order g”, O(g)  {f:R  R |  c,k  x>k |f(x)|  |cg(x)|}. –Often, one deals only with positive functions and can ignore absolute value symbols. “f  O(g)” often written “f is O(g)” or “f=O(g)”.“f  O(g)” often written “f is O(g)” or “f=O(g)”. –The latter form is an instance of a more general convention...

Module #5 - Algorithms 54 Order-of-Growth Expressions “O(f)” when used as a term in an arithmetic expression means: “some function f such that f  O(f)”.“O(f)” when used as a term in an arithmetic expression means: “some function f such that f  O(f)”. E.g.: “x 2 +O(x)” means “x 2 plus some function that is O(x)”.E.g.: “x 2 +O(x)” means “x 2 plus some function that is O(x)”. Formally, you can think of any such expression as denoting a set of functions: “x 2 +O(x)” :  {g |  f  O(x): g(x)= x 2 +f(x)}Formally, you can think of any such expression as denoting a set of functions: “x 2 +O(x)” :  {g |  f  O(x): g(x)= x 2 +f(x)}

Module #5 - Algorithms 55 Order of Growth Equations Suppose E 1 and E 2 are order-of-growth expressions corresponding to the sets of functions S and T, respectively.Suppose E 1 and E 2 are order-of-growth expressions corresponding to the sets of functions S and T, respectively. Then the “equation” E 1 =E 2 really means  f  S,  g  T : f=g or simply S  T.Then the “equation” E 1 =E 2 really means  f  S,  g  T : f=g or simply S  T. Example: x 2 + O(x) = O(x 2 ) means  f  O(x):  g  O(x 2 ): x 2 +f(x)=g(x)Example: x 2 + O(x) = O(x 2 ) means  f  O(x):  g  O(x 2 ): x 2 +f(x)=g(x)

Module #5 - Algorithms 56 Useful Facts about Big O  f,g & constants a,b  R, with b  0,  f,g & constants a,b  R, with b  0, –af = O(f); (e.g. 3x 2 = O(x 2 )) –f+O(f) = O(f); (e.g. x 2 +x = O(x 2 )) Also, if f=  (1) (at least order 1), then:Also, if f=  (1) (at least order 1), then: –|f| 1-b = O(f); (e.g. x  1 = O(x)) –(log b |f|) a = O(f). (e.g. log x = O(x)) –g=O(fg) (e.g. x = O(x log x)) –fg  O(g) (e.g. x log x  O(x)) –a=O(f) (e.g. 3 = O(x))

Module #5 - Algorithms 57 Definition 2  (g) = {f | g  O(f)} “The functions that are at least order g.”  (g) = {f | g  O(f)} “The functions that are at least order g.”

Module #5 - Algorithms 58 Definition:  (g), exactly order g If f  O(g) and g  O(f) then we say “g and f are of the same order” or “f is (exactly) order g” and write f  (g).If f  O(g) and g  O(f) then we say “g and f are of the same order” or “f is (exactly) order g” and write f  (g). Another equivalent definition:  (g)  {f:R  R |  c 1 c 2 k  x>k: |c 1 g(x)|  |f(x)|  |c 2 g(x)| }Another equivalent definition:  (g)  {f:R  R |  c 1 c 2 k  x>k: |c 1 g(x)|  |f(x)|  |c 2 g(x)| } “Everywhere beyond some point k, f(x) lies in between two multiples of g(x).”“Everywhere beyond some point k, f(x) lies in between two multiples of g(x).”

Module #5 - Algorithms 59 Example 10

Module #5 - Algorithms 60 Example 11

Module #5 - Algorithms 61 Theorem 4

Module #5 - Algorithms 62 Rules for  Mostly like rules for O( ), except:Mostly like rules for O( ), except:  f,g>0 & constants a,b  R, with b>0, af   (f), but  Same as with O. f   (fg) unless g=  (1)  Unlike O. |f| 1-b   (f), and  Unlike with O. (log b |f|) c   (f).  Unlike with O.  f,g>0 & constants a,b  R, with b>0, af   (f), but  Same as with O. f   (fg) unless g=  (1)  Unlike O. |f| 1-b   (f), and  Unlike with O. (log b |f|) c   (f).  Unlike with O. The functions in the latter two cases we say are strictly of lower order than  (f).The functions in the latter two cases we say are strictly of lower order than  (f).

Module #5 - Algorithms 63  example Determine whether:Determine whether: Quick solution:Quick solution:

Module #5 - Algorithms 64 Other Order-of-Growth Relations  (g) = {f | g  O(f)} “The functions that are at least order g.”  (g) = {f | g  O(f)} “The functions that are at least order g.” o(g) = {f |  c>0  k  x>k : |f(x)| 0  k  x>k : |f(x)| < |cg(x)|} “The functions that are strictly lower order than g.” o(g)  O(g)   (g).  (g) = {f |  c>0  k  x>k : |cg(x)| 0  k  x>k : |cg(x)| < |f(x)|} “The functions that are strictly higher order than g.”  (g)   (g)   (g).

Module #5 - Algorithms 65 Subset relations between order-of-growth sets.Subset relations between order-of-growth sets. Relations Between the Relations RRRR  ( f ) O( f )  ( f )  ( f ) o( f ) f

Module #5 - Algorithms 66 Why o(f)  O(x)  (x) A function that is O(x), but neither o(x) nor  (x):A function that is O(x), but neither o(x) nor  (x):

Module #5 - Algorithms 67 Strict Ordering of Functions Temporarily let’s write f  g to mean f  o(g), f~g to mean f  (g)Temporarily let’s write f  g to mean f  o(g), f~g to mean f  (g) Note thatNote that Let k>1. Then the following are true: 1  log log n  log n ~ log k n  log k n  n 1/k  n  n log n  n k  k n  n!  n n …Let k>1. Then the following are true: 1  log log n  log n ~ log k n  log k n  n 1/k  n  n log n  n k  k n  n!  n n …

Module #5 - Algorithms 68 Review: Orders of Growth (§1.8) Definitions of order-of-growth sets,  g:R  R O(g)  {f |  c>0  k  x>k |f(x)| 0  k  x>k |f(x)| < |cg(x)|} o(g)  {f |  c>0  k  x>k |f(x)| 0  k  x>k |f(x)| < |cg(x)|}  (g)  {f | g  O(f)}  (g)  {f | g  O(f)}  (g)  {f | g  o(f)}  (g)  {f | g  o(f)}  (g)  O(g)   (g)  (g)  O(g)   (g)