Lecture 2 What is a computational problem? What is an instance of a problem? What is an algorithm? How to guarantee that an algorithm is correct? What is a step of an algorithm? Analysis of algorithms

Chapter 1: Algorithm Correctness Chapter 2: Analysis of Algorithms

Algorithm Correctness

Computational Problem Examples: –Find the sum of n given integers –Find the minimum of n elements –Sort n numbers –Search Problem: Does the list L of n items contain the element x? n: Input size

Instance Sort the numbers {3, 2, 67, 54, 57, 23} Find the minimum element in the array {2,4, 67, 89, 23, 45, 67, 890, 2, 1} Does 3 belong to {4, 6, 78, 56, 3, 45, 2,109}?

Specification of a problem Min Problem: Precondition: S is a finite, non-empty set of integers Postcondition: m is a minimum element of S

Algorithm Step-by-step procedure to solve a well- specified problem You are given two pieces of rope. You know each of them burns in exactly one hour. But they do not burn uniformly. Find a way to measure ¼ of an hour. This is an instance of a more general problem. Generalize it. Examples: Find Min Sequential search Binary Search Selection Sort Insertion Sort

Iterative Algorithms Loop Invariant Before iteration i: m is the smallest element among S[1], …, S[i] Find Min Precondition: S a non-empty list of integers Postcondition: m a minimum element Algorithm: m = S[1] for i=2 to n if m > S[i] then m = S[i] endif endfor

Recursive Algorithm Find Min Precondition: S a non-empty list of integers Postcondition: m a minimum element Algorithm: FindMin[S,n] = Min[S,1,n] Min[S,b,e] if e = b return S[b] m = Min[S, b+1, e] if m > S[b] then m = S[b] endif return m Invariant Min[S,i,j] returns a smallest element among S[i], …, S[j]

Correctness Proofs For iterative algorithms: by induction, use loop invariant For recursive algorithms: by induction, use the function invariant (pre/post conditions) Example: prove that the iterative and recursive algorithms for finding the min are correct

Analysis of Algorithms

Step Arithmetic operations Comparisons Etc. Do not count in the analysis: implementation overhead (push parameters on stack, compare with loop variable, etc.)

Analysis of Sequential Search Search(x,L,n) Search x in the list L of size n, return position if x is in S; otherwise 0 for i=1 to n if x = L[i] return I Return 0 for i=1 to n 1 comparison per step Time: = n

Analysis of Binary Search Search(x,L,b,e) Search x in the sorted list L from b to e, return position if b=e and x=L[b] return b m=(b+e)/2 if x < L[m] Search[x,L,b,m] else Search[x,L,m,e] T(n) (n = e-b = nr. of elements) Time to search in a list of size n Worst case time: T(n) = T(n/2) + 1 Recurrence Relation T(n) = log n

Which is better? Sequential Search: n steps Binary Search: log n steps Hidden constants in implementing recursive algorithm: -each recursive call pushes data on a stack -Returning from a recursive call pops data from stack More realistically: Sequential Search: 3n Binary Search: 10 log n

Evaluating Efficiency using Big Oh notation Criterion: if lim a(n)/b(n) is a constant, then we say a(n) = O(b(n)) We mean: a is better or at least as good as b Example for search algorithms: lim (10 log n) / (3 n) = 0 Binary search is at least as good as Sequential Search

Two implementations of Sequential Search A takes 10n B takes 27n-25 lim (10 n) / (27 n - 25) = 10/27 The two implementations are comparable, ignoring constants Running B on a faster computer than A may be faster. On the same computer, same programming language, same programmer, etc. A will probably be faster.

Evaluating Efficiency using small Oh notation Criterion: if lim a(n)/b(n) is the zero constant, then we say a(n) = o(b(n)) We mean: a is much better than b Example for search algorithms: lim (10 log n) / (3 n) = 0 Binary search is much better than Sequential Search

Ignoring small cases A (Binary Search) runs in 100 log n B (Sequential Search) runs in 3n A is asymptotically better lim (100 log n) / (3 n) = 0 But on small values of n, sequential search might be better How to find the break-even point? Testing and profiling a program: compare the running times on small data sets, see when one becomes better.

Homework exercises Small programming assignment: implement sequential search and binary search. Find when binary search becomes better in your implementation Limit computation: a small list of simple functions. Tell when one is Big Oh or small Oh of the other. Posted on the web by tonight, due in one week, before class.

If time allows (if not, read it from the book in preparation for next lecture): Towers of Hanoi: recursive algorithm and time analysis