1. Data Structures and Algorithms Lecture 5 and 6 Instructor: Quratulain Date: 15 th and 18 th September, 2009 Faculty of Computer Science, IBA

2. Algorithms An algorithm is any well-defined procedure that take some value, or set values and produce output. Input: Output: CSE 246 Data Structures and Algorithms Fall2009 Quratulain2

3. Describing Algorithms A complete description of an algorithm consists of three parts: 1.the algorithm (expressed in whatever way is clearest and most concise, can be English and / or pseudocode) 2.a proof of the algorithm’s correctness 3.a derivation of the algorithm’s running time CSE 246 Data Structures and Algorithms Fall2009 Quratulain3

4. Correctness proof Use a loop invariant to understand why an algorithm gives the correct answer. Loop invariant A loop invariant is a relation among program variables that is true when control enters a loop, remains true each time the program executes the body of the loop, and is still true when control exits the loop. Understanding loop invariants can help us analyze programs, check for errors, and derive programs from specifications. CSE 246 Data Structures and Algorithms Fall2009 Quratulain4

5. Correctness proof To proof correctness with a loop invariant we need to show three things: Initialization Invariant is true prior to the first iteration of the loop. Maintenance If the invariant is true before an iteration of the loop, it remains true before the next iteration. Termination When the loop terminates, the invariant (usually along with the reason that the loop terminated) gives us a useful property that helps show that the algorithm is correct. CSE 246 Data Structures and Algorithms Fall2009 Quratulain5

6. Example public static int sum(int a[]) { int s = 0; for (int i = 0; i < a.length; i++) { // s is the sum of the first i array elements // s == a[0] +.. + a[i-1] The comments in the loop body describe the invariant. s = s + a[i]; } return s; } The invariant expresses a relation between the three variables s, i, and a that remains true even when the values of s and i change: s is always the sum of the first i elements in a. When control enters the loop, i is 0 so s is the sum of no array elements. When control exits the loop, i is a.length, so s is the sum of all of the array elements. This is the intended result. CSE 246 Data Structures and Algorithms Fall2009 Quratulain6

7. Analyzing Algorithm In this section, we examine a means of analyzing the performance of an algorithm. Usually we are interested in the amount of memory required by an algorithm – its space complexity – and the time taken for an algorithm to run – its time complexity. In this course we will consider only the time complexity of algorithms. CSE 246 Data Structures and Algorithms Fall2009 Quratulain7

8. Introduction to Time Complexity An important question is: How efficient is an algorithm or piece of code? Efficiency covers lots of resources, including: CPU (time) usage memory usage disk usage network usage CSE 246 Data Structures and Algorithms Fall2009 Quratulain8

9. Introduction to Time Complexity Be careful to differentiate between: ◦ Performance: how much time/memory/disk/... is actually used when a program is run. This depends on the machine, compiler, etc. as well as the code. ◦ Complexity: how do the resource requirements of a program or algorithm scale, i.e., what happens as the size of the problem being solved gets larger. Complexity affects performance but not the other way around. CSE 246 Data Structures and Algorithms Fall2009 Quratulain9

10. Introduction to Time Complexity To do this exactly, we should count the number of CPU cycles it takes to perform each operation in the algorithm. ◦ Needless to say, this would be a bit tedious and is not a very practical approach. Instead we will make a simplification. We will count the number of times simple statements are executed. By simple statement we mean a statement that is executed in an amount of time T. we are usually interested in the worst case: what are the max operations that might be performed for a given problem size (other cases -- best case and average case) CSE 246 Data Structures and Algorithms Fall2009 Quratulain10

11. Analysis of Algorithms Dilemma: you have two (or more) methods to solve problem, how to choose the BEST? One approach: implement each algorithm in C, test how long each takes to run. Problems: ◦ Different implementations may cause an algorithm to run faster/slower ◦ Some algorithms run faster on some computers ◦ Algorithms may perform differently depending on data (e.g., sorting often depends on what is being sorted) CSE 246 Data Structures and Algorithms Fall2009 Quratulain11

12. Analysis of Algorithms Analysis Concept of "best" What to measure Types of "best” best-, average-, worst-case Comparison methods Big-O analysis Examples Sequential search and binary search CSE 246 Data Structures and Algorithms Fall2009 Quratulain12

13. Analysis: A Better Approach Idea: characterize performance in terms of key operation(s) ◦ Sorting:  count number of times two values compared  count number of times two values swapped ◦ Search:  count number of times value being searched for is compared to values in array ◦ Recursive function:  count number of recursive calls CSE 246 Data Structures and Algorithms Fall2009 Quratulain13

14. Analysis in General Want to comment on the “general” performance of the algorithm ◦ Measure for several examples, but what does this tell us in general? Instead, assess performance in an abstract manner Idea: analyze performance as size of problem grows Examples: ◦ Sorting: how many comparisons for array of size N? ◦ Searching: #comparisons for array of size N May be difficult to discover a reasonable formula CSE 246 Data Structures and Algorithms Fall2009 Quratulain14

15. Analysis where Results Vary Types of analyses: ◦ Best-case: what is the fastest an algorithm can run for a problem of size N? ◦ Average-case: on average how fast does an algorithm run for a problem of size N? ◦ Worst-case: what is the longest an algorithm can run for a problem of size N? CSE 246 Data Structures and Algorithms Fall2009 Quratulain15

16. How to Compare Formulas? Which is better: or Answer depends on value of N: N 1 120 37 2 511 71 3 1374 159 4 2895 397 5 5260 1073 6 8655 3051 7 13266 8923 8 19279 26465 9 26880 79005 10 36255 236527 CSE 246 Data Structures and Algorithms Fall2009 Quratulain16

17. What Happened? CSE 246 Data Structures and Algorithms Fall2009 Quratulain17 N 1 37 12 32.4 2 71 36 50.7 3 159 108 67.9 4 397 324 81.6 5 1073 972 90.6 6 3051 2916 95.6 7 8923 8748 98.0 8 26465 26244 99.2 9 79005 78732 99.7 10 236527 236196 99.9 ◦ One term dominated the sum

18. As N Grows, Some Terms Dominate CSE 246 Data Structures and Algorithms Fall2009 Quratulain18

19. Order of Magnitude Analysis CSE 246 Data Structures and Algorithms Fall2009 Quratulain19 Measure speed with respect to the part of the sum that grows quickest Ordering:

20. Order of Magnitude Analysis (cont) CSE 246 Data Structures and Algorithms Fall2009 Quratulain20 Furthermore, simply ignore any constants in front of term and simply report general class of the term: grows proportionally to When comparing algorithms, determine formulas to count operation(s) of interest, then compare dominant terms of formulas

21. Analyzing Running Time CSE 246 Data Structures and Algorithms Fall2009 Quratulain21 1. n = read input from user 2. sum = 0 3. i = 0 4. while i < n 5. number = read input from user 6. sum = sum + number 7. i = i + 1 8. mean = sum / n T(n), or the running time of a particular algorithm on input of size n, is taken to be the number of times the instructions in the algorithm are executed. Pseudo code algorithm illustrates the calculation of the mean (average) of a set of n numbers: StatementNumber of times executed1 21 31 4n+1 5n 6n 7n 81 The computing time for this algorithm in terms on input size n is: T(n) = 4n + 5.

22. Big-O Notation Computer scientists like to categorize algorithms using Big-O notation. If you tell a computer scientist that they can choose between two algorithms: ◦ one whose time complexity is O( N ) ◦ another that is O( N2 ), then the O( N ) algorithm will likely be chosen. Let’s find out why… CSE 246 Data Structures and Algorithms Fall2009 Quratulain22

23. Big-O Notation Let’s consider a function on N, T(N) N is a measure of the size of the problem we try to solve, e.g., Definition: T(N) is O( g(N) ) if: So, if T(N) is O( g(N) ) then T(N) grows no faster than g(N). Example 1 Suppose, for example, that T is the time taken to run an algorithm to process data in an array of length N. We expect that T will be a function of N and hence write: If we now say that T(N) is O(N) we are saying that T(N) grows no faster than: This is good news – it tells us that if we double the size of the array, we can expect that the time taken to run the algorithm will double (roughly speaking). CSE 246 Data Structures and Algorithms Fall2009 Quratulain23

24. Example CSE 246 Data Structures and Algorithms Fall2009 Quratulain24 Suppose f(n) = n 2 + 3n - 1. We want to show that f(n) = O(n 2 ). f(n) = n 2 + 3n - 1 < n 2 + 3n (subtraction makes things smaller so drop it) <= n 2 + 3n 2 (since n <= n 2 for all integers n) = 4n 2 Therefore, if C = 4, we have shown that f(n) = O(n 2 ). Notice that all we are doing is finding a simple function that is an upper bound on the original function. Because of this, we could also say that This would be a much weaker description, but it is still valid. f(n) = O(n 3 ) since (n 3 ) is an upper bound on n 2

25. Example 1 Searching Sorted Array Algorithm 1: Sequential Search int search(int A[], int N, int Num) { int index = 0; < while ((index < N) && (A[index] < Num)) index++; if ((index < N) && (A[index] == Num)) return index; else return -1; } CSE 246 Data Structures and Algorithms Fall2009 Quratulain25

26. Analyzing Search Algorithm 1 Operations to count: how many times Num is compared to member of array One after the loop each time plus... Best-case: find the number we are looking for at the first position in the array (1 + 1 = 2 comparisons) O(1) Average-case: find the number on average half-way down the array (sometimes longer, sometimes shorter) (N/2+1 comparisons) O(N) Worst-case: have to compare Num to very element in the array (N + 1 comparisons) O(N) CSE 246 Data Structures and Algorithms Fall2009 Quratulain26

27. Search Algorithm 2: Binary Search int search(int A[], int N, int Num) { int first = 0; int last = N - 1; int mid = (first + last) / 2; while ((A[mid] != Num) && (first <= last)) { if (A[mid] > Num) last = mid - 1; else first = mid + 1; mid = (first + last) / 2; } if (A[mid] == Num) return mid; else return -1; } CSE 246 Data Structures and Algorithms Fall2009 Quratulain27

28. Analyzing Binary Search One comparison after loop First time through loop, toss half of array (2 comps) Second time, half remainder (1/4 original) 2 comps Third time, half remainder (1/8 original) 2 comps … Loop Iteration Remaining Elements 1 N/2 2 N/4 3 N/8 4 N/16 … ?? 1 How long to get to 1? CSE 246 Data Structures and Algorithms Fall2009 Quratulain28

29. Analyzing Binary Search (cont) CSE 246 Data Structures and Algorithms Fall2009 Quratulain29 Looking at the problem in reverse, how long to double the number 1 until we get to N? and solve for X two comparisons for each iteration, plus one comparison at the end -- binary search takes in the worst case Binary search is worst-case Sequential search is worst-case

30. Big-Oh Operations CSE 246 Data Structures and Algorithms Fall2009 Quratulain30 Summation Rule Suppose T1(n) = O(f1(n)) and T2(n) = O(f2(n)). Further, suppose that f2 grows no faster than f1, i.e., f2(n) = O(f1(n)). Then, we can conclude that T1(n) + T2(n) = O(f1(n)). More generally, the summation rule tells us O(f1(n) + f2(n)) = O(max(f1(n), f2(n))). Proof: Suppose that C and C' are constants such that T1(n) <= C * f1(n) and T2(n) <= C' * f2(n). Let D = the larger of C and C'. Then, T1(n) + T2(n)<= C * f1(n) + C' * f2(n) <= D * f1(n) + D * f2(n) <= D * (f1(n) + f2(n)) <= O(f1(n) + f2(n))

31. Big-Oh Operations CSE 246 Data Structures and Algorithms Fall2009 Quratulain31 Product Rule Suppose T1(n) = O(f1(n)) and T2(n) = O(f2(n)). Then, we can conclude that T1(n) * T2(n) = O(f1(n) * f2(n)). The Product Rule can be proven using a similar strategy as the Summation Rule proof. Analyzing Some Simple Programs (with No Sub-program Calls) General Rules: All basic statements (assignments, reads, writes, conditional testing, library calls) run in constant time: O(1). The time to execute a loop is the sum, over all times around the loop, of the time to execute all the statements in the loop, plus the time to evaluate the condition for termination. Evaluation of basic termination conditions is O(1) in each iteration of the loop. The complexity of an algorithm is determined by the complexity of the most frequently executed statements. If one set of statements have a running time of O(n 3 ) and the rest are O(n), then the complexity of the algorithm is O(n 3 ). This is a result of the Summation Rule.

32. Question Suppose you have two algorithms (Algorithm A and Algorithm B) and that both algorithms are O(N). Does this mean that they both take the same amount of time to run?

33. Example We will now look at various Java code segments and analyze the time complexity of each: int count = 0; int sum = 0; while( count < N ) { sum += count; System.out.println( sum ); count++; } Total number of constant time statements executed...?

34. Example int count = 0; int sum = 0; while( count < N ) { int index = 0; while( index < N ) { sum += index * count; System.out.println( sum ); index++; } count++; } Total number of constant time statements executed ….?

35. Example int count = 0; while( count < N ) { int index = 0; while( index < count + 1 ) { sum++; index++; } count++; } Total number of constant time statements executed…?

36. Example int count = N; int sum = 0; while( count > 0 ) { sum += count; count = count / 2; } Total number of constant time statements executed…?