A Lecture /24/2015 COSC3101A: Design and Analysis of Algorithms Tianying Ji Lecture 1

A Lecture /24/2015 Part 1 The course  General information Introduction to algorithms Algorithm analysis basics Asymptotic notation

A Lecture /24/2015 The Course Purpose: The theoretical study of design and analysis of algorithms  Not a programming course  Not a math course, either Textbook: Introduction to Algorithms, Cormen, Leiserson, Rivest, Stein  Second edition  An excellent reference you should own

A Lecture /24/2015 The Course Instructor: Tianying Ji   Office: CSEB 2022  Office hours: 5:30pm-7:00pm, Tuesday  TA: TBA  TA Office hours and location TBA

A Lecture /24/2015 The Course Grading policy (Probably, TBD):  Assignments: 20%  Mid-term Exam: 30%  Final Exam: 50%

A Lecture /24/2015 The Course Format  One lecture/week  About four Assignments oProblem sets oMaybe occasional programming assignments  Mid-term + final exam

A Lecture /24/2015 The Course Course overview  Mathematical background (asymptotic notation, recurrence relations, bounding sums, induction, etc.) [Ch 1, 2.2, 3, 4]  Proofs of correctness, including loop invariants [Ch 2.1]  Recursive algorithms; divide and conquer technique [Ch 2.3, 4]  Sorting algorithms and lower bounds [Ch 6,7,8]  Selection [Ch 9]  Greedy algorithms [Ch 16]  Dynamic programming [Ch 15]  Graph algorithms (searching, spanning trees, shortest path, etc.) [Ch 22,23,24,25,26]  (If time permits) A brief introduction to NP-completeness. [Ch 34]

A Lecture /24/2015 Part 1 Questions?

A Lecture /24/2015 Part 2 The course ☻ Introduction to algorithms [Ch 1]  What are Algorithms?  Analysis vs. Design  RAM Algorithm analysis basics Asymptotic notation

A Lecture /24/2015 What are algorithms? Webster definition: a procedure for solving a mathematical problem (as of finding the greatest common divisor) in a finite number of steps that frequently involves repetition of an operation; broadly : a step-by-step procedure for solving a problem or accomplishing some end especially by a computer

A Lecture /24/2015 Basics of Algorithms Well-defined computational procedure Takes input values Produces output values Results are always correct for any input event Always terminates

A Lecture /24/2015 Musical analogy ComputingMusical SoftwareMusic HardwareMusical instruments AlgorithmsChord progressions Source codeMusical notation ProgrammingComposing

A Lecture /24/2015 Basic goals for an algorithm Always correct Always terminates This class: performance  Performance often draws the line between what is possible and what is impossible.

A Lecture /24/2015 Analysis vs. Design Analysis: predict the cost of an algorithm in terms of resources and performance Design: design algorithms which minimize the cost

A Lecture /24/2015 Machine model: Generic Random Access Machine (RAM) Executes operations sequentially Set of primitive operations:  Arithmetic. Logical, Comparisons, Function calls Simplifying assumption: all ops cost 1 unit  Eliminates dependence on the speed of our computer, otherwise impossible to verify and to compare

A Lecture /24/2015 Part 2 Questions?

A Lecture /24/2015 Part 3 The course ☻ Introduction to algorithms ☻ Algorithm analysis basics [Ch 2.1, 2.2]  Sorting  Insertion sort Asymptotic notation

A Lecture /24/2015 The problem of sorting Input: sequence  a 1, a 2, …, a n  of numbers. Output: permutation  a' 1, a' 2, …, a' n  such that a' 1  a' 2  …  a' n. Example: Input: Output:

A Lecture /24/2015 Insertion sort I NSERTION -S ORT (A, n) ⊳ A[1.. n] for i ← 2 to n dokey ← A[i] j ← i – 1 while j > 0 and A[j] > key doA[j+1] ← A[j] j ← j – 1 A[j+1] ← key “pseudocode” ji key sorted A:A: 1n

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i =  j =  key =  A[j] =  A[j+1] = 

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 2j = 1key = 10 A[j] = 30 A[j+1] = 10

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 2j = 1key = 10 A[j] = 30 A[j+1] = 30

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 2j = 1key = 10 A[j] = 30 A[j+1] = 30

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 2j = 0key = 10 A[j] =  A[j+1] = 30

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 2j = 0key = 10 A[j] =  A[j+1] = 10

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 3j = 0key = 10 A[j] =  A[j+1] = 10

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 3j = 0key = 40 A[j] =  A[j+1] = 10

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 3j = 0key = 40 A[j] =  A[j+1] = 10

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 3j = 2key = 40 A[j] = 30 A[j+1] = 40

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 3j = 2key = 40 A[j] = 30 A[j+1] = 40

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 3j = 2key = 40 A[j] = 30 A[j+1] = 40

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 4j = 2key = 40 A[j] = 30 A[j+1] = 40

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 4j = 3key = 20 A[j] = 40 A[j+1] = 20

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 4j = 3key = 20 A[j] = 40 A[j+1] = 20

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 4j = 3key = 20 A[j] = 40 A[j+1] = 40

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 4j = 3key = 20 A[j] = 40 A[j+1] = 40

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 4j = 3key = 20 A[j] = 40 A[j+1] = 40

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 4j = 2key = 20 A[j] = 30 A[j+1] = 40

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 4j = 2key = 20 A[j] = 30 A[j+1] = 30

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 4j = 2key = 20 A[j] = 30 A[j+1] = 30

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 4j = 1key = 20 A[j] = 10 A[j+1] = 30

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 4j = 1key = 20 A[j] = 10 A[j+1] = 30

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 4j = 1key = 20 A[j] = 10 A[j+1] = 20

A Lecture /24/2015 An Example: Insertion Sort InsertionSort(A, n) for i ← 2 to n do key ← A[i] j ← i - 1 while j > 0 and A[j] > key do A[j+1] ← A[j] j ← j - 1 A[j+1] ← key i = 4j = 1key = 20 A[j] = 10 A[j+1] = 20 Done!

A Lecture /24/2015 Example of insertion sort done

A Lecture /24/2015 Kinds of analyses Worst-case: (usually) T(n) = maximum time of algorithm on any input of size n. Average-case: (sometimes) T(n) = expected time of algorithm over all inputs of size n. Need assumption of statistical distribution of inputs. Best-case: (NEVER) Cheat with a slow algorithm that works fast on some input.

A Lecture /24/2015 Insertion Sort Statement Cost Times InsertionSort(A, n) for i ← 2 to n c 1 n do key ← A[i] c 2 n-1 j ← i - 1 c 3 n-1 while j > 0 and A[j] > key c 4 T do A[j+1] ← A[j] c 5 T-(n-1) j ← j - 1 c 6 T-(n-1) 0 A[j+1] ← key c 7 n-1 T = t 2 + t 3 + … + t n = where t i is number of while expression evaluations for the i th for loop iteration

A Lecture /24/2015 Analyzing Insertion Sort T(n) = c 1 n + c 2 (n-1) + c 3 (n-1) + c 4 T + c 5 (T - (n-1)) + c 6 (T - (n-1)) + c 7 (n-1) = c 8 T + c 9 n + c 10 What can T be?  Best case -- inner loop body never executed ot i = 1  T(n) is a linear function  Worst case -- inner loop body executed for all previous elements ot i = i  T(n) is a quadratic function  Average case o???

A Lecture /24/2015 Analysis Simplifications  Ignore actual and abstract statement costs  Order of growth is the interesting measure: oHighest-order term is what counts ] Remember, we are doing asymptotic analysis ] As the input size grows larger it is the high order term that dominates

A Lecture /24/2015 Part 3 Questions?

A Lecture /24/2015 Part 4 The course ☻ Introduction to algorithms ☻ Algorithm analysis basics [Ch 2.1, 2.2] ☻ Asymptotic notation[Ch3]

A Lecture /24/2015 Asymptotic Complexity Running time of an algorithm for large input sizes. Expressed using only the highest-order term in the expression for the exact running time.  Instead of exact running time, say  (n 2 ). Describe behavior of functions in the limit. Described using Asymptotic Notation.

A Lecture /24/2015 Asymptotic Notation , O, , o,  Defined in terms of functions whose domain is the set of natural numbers.  Ex: f(n) =  (n 2 ).  Describes how f(n) grows in comparison to n 2. Determine sets of functions, in practice used to compare two function sizes. The notations differ in the rate-of-growth relation they describe between the defining function and the defined set of functions.

A Lecture /24/2015  -notation  (g(n)) = {f(n):  +ve constants c 1, c 2, and n 0 such that 0  c 1 g(n)  f(n)  c 2 g(n),  n  n 0 } For function g(n),  (g(n)) is given by: g(n) is an asymptotically tight bound for f(n). Intuitively: Set of all functions that have the same rate of growth as g(n).

A Lecture /24/2015 O -notation O(g(n)) = {f(n):  +ve constants c and n 0 such that 0  f(n)  cg(n),  n  n 0 } For function g(n), O(g(n)) is given by: g(n) is an asymptotic upper bound for f(n). Intuitively: Set of all functions whose rate of growth is the same as or lower than that of g(n). f(n) =  (g(n))  f(n) = O(g(n)).  (g(n))  O(g(n)).

A Lecture /24/2015  -notation  (g(n)) = {f(n):  +ve constants c and n 0 such that 0  cg(n)  f(n),  n  n 0 } For function g(n),  (g(n)) is given by: g(n) is an asymptotic lower bound for f(n). Intuitively: Set of all functions whose rate of growth is the same as or higher than that of g(n). f(n) =  (g(n))  f(n) =  (g(n)).  (g(n))   (g(n)).

A Lecture /24/2015 Relations Between , O, 

A Lecture /24/2015 Upper Bound Notation We say InsertionSort’s run time is O(n 2 )  Properly we should say run time is in O(n 2 )  Read O as “Big-O” (you’ll also hear it as “order”) In general a function  f(n) is O(g(n)) if there exist positive constants c and n 0 such that f(n)  c  g(n) for all n  n 0 Formally  O(g(n)) = { f(n):  positive constants c and n 0 such that f(n)  c  g(n)  n  n 0 }

A Lecture /24/2015 Insertion Sort Is O(n 2 ) Proof  Suppose runtime is an 2 + bn + c oIf any of a, b, and c are less than 0 replace the constant with its absolute value an 2 + bn + c  (a + b + c)n 2 + (a + b + c)n + (a + b + c)  3(a + b + c)n 2 for n  1 Let c’ = 3(a + b + c) and let n 0 = 1 Question  Is InsertionSort O(n 3 )?  Is InsertionSort O(n)?

A Lecture /24/2015 Big O Fact A polynomial of degree k is O(n k ) Proof:  Suppose f(n) = b k n k + b k-1 n k-1 + … + b 1 n + b 0 oLet a i = | b i |  f(n)  a k n k + a k-1 n k-1 + … + a 1 n + a 0

A Lecture /24/2015 Lower Bound Notation We say InsertionSort’s run time is  (n) In general a function  f(n) is  (g(n)) if  positive constants c and n 0 such that 0  c  g(n)  f(n)  n  n 0 Proof:  Suppose run time is an + b oAssume a and b are positive (what if b is negative?)  an  an + b

A Lecture /24/2015 Asymptotic Tight Bound A function f(n) is  (g(n)) if  positive constants c 1, c 2, and n 0 such that c 1 g(n)  f(n)  c 2 g(n)  n  n 0 Theorem  f(n) is  (g(n)) iff f(n) is both O(g(n)) and  (g(n))

A Lecture /24/2015 Practical Complexity

A Lecture /24/2015 Practical Complexity

A Lecture /24/2015 Practical Complexity

A Lecture /24/2015 Practical Complexity

A Lecture /24/2015 Other Asymptotic Notations A function f(n) is o(g(n)) if  positive constants c and n 0 such that f(n) < c g(n)  n  n 0 A function f(n) is  (g(n)) if  positive constants c and n 0 such that c g(n) < f(n)  n  n 0 Intuitively,  o() is like <  O() is like    () is like >   () is like    () is like =

A Lecture /24/2015 Part 4 Questions?

A Lecture /24/2015 Review: Induction Suppose  S(k) is true for fixed constant k oOften k = 0  S(n)  S(n+1) for all n >= k Then S(n) is true for all n >= k

A Lecture /24/2015 Proof By Induction Claim:S(n) is true for all n >= k Basis:  Show formula is true when n = k Inductive hypothesis:  Assume formula is true for an arbitrary n Step:  Show that formula is then true for n+1

A Lecture /24/2015 Induction Example: Gaussian Closed Form Prove … + n = n(n+1) / 2  Basis: oIf n = 0, then 0 = 0(0+1) / 2  Inductive hypothesis: oAssume … + n = n(n+1) / 2  Step (show true for n+1): … + n + n+1 = ( …+ n) + (n+1) = n(n+1)/2 + n+1 = [n(n+1) + 2(n+1)]/2 = (n+1)(n+2)/2 = (n+1)(n+1 + 1) / 2

A Lecture /24/2015 The End Read your text Ch1, Ch2.1, Ch2.2, Ch3!