1. Introduction to complexity

2. 2 Analysis of Algorithms Why do we need to analyze algorithms? –To measure the performance –Comparison of different algorithms –Can we find a better one? Is this the best solution? –Running time analysis –Memory usage

3. 3 Complexity: a measure of the performance of an algorithm An algorithm’s performance depends on internal and external factors External Size of the input to the algorithm Speed of the computer on which it is run Quality of the compiler Internal The algorithm’s efficiency, in terms of: Time required to run Space (memory storage) required to run Complexity measures the internal factors (usually more interested in time than space)

4. 4 Running Time Analysis N Algorithm 1 Running Time T 1 (n) N Algorithm 2 Running Time T 2 (n) 1000101010100011111000110001110101 0101010101010010001010101000100000 000000011110101000111010 Algorithm 1 T 1 (N)=1000N Algorithm 2 T 2 (N)=N 2

5. 5 Running Time Analysis N Running Time T(N) Algorithm 2 Algorithm 1 1000

6. 6 Summary of Running Times NT1T2 1010 -2 sec10 -4 sec 10010 -1 sec10 -2 sec 10001 sec 1000010 sec100 sec 100000100 sec10000 sec For the values of N that are less than 1000 the difference bw. running times of the two algorithms can be ignored

7. 7 Growth rates and big-O notation Growth rates capture the essence of an algorithm’s performance Big-O notation indicates the growth rate. It is the class of mathematical formula that best describes an algorithm’s performance, and is discovered by looking inside the algorithm Big-O is a function with parameter N, where N is usually the size of the input to the algorithm –For example, if an algorithm depending on the value n has performance an 2 + bn + c (for constants a, b, c) then we say the algorithm has performance O(N 2 ) For large N, the N 2 term dominates. Only the dominant term is included in big-O

8. 8 big-O notation – Asympthotic Upper Limit Running time of an algorithm T(N)=O(f(n)) If there are c and n 0 values satisfying T(N)  c f(n) for N  n 0 then T(N)  c f(n) f(n), is an asymptotic upper bound for T(N) and T(N)=O(f(n)). This is not a function, but a notation.

9. 9 big-O notation big-O notation is written in the simplest form –Example 3n 2 +2n+5 = O(n 2 ) –The followings are correct, but not used 3n 2 +2n+5 = O(3n 2 +2n+5) 3n 2 +2n+5 = O(n 2 +n) 3n 2 +2n+5 = O(3n 2 )

10. 10 Common growth rates Time complexityExample O(1) constantAdding to the front of a linked list O(log N) logFinding an entry in a sorted array O(N) linearFinding an entry in an unsorted array O(N log N)n-log-nSorting n items by ‘divide-and-conquer’ O(N 2 ) quadraticShortest path between two nodes in a graph O(N 3 ) cubicSimultaneous linear equations O(2 N ) exponentialThe Towers of Hanoi problem

11. 11 Growth rates Number of Inputs Time O(N 2 ) O(Nlog N) For a short time N 2 is better than NlogN

12. 12 Calculating the actual time taken by a program (example) A program takes 10ms to process one data item (i.e. to do one operation on the data item) How long would the program take to process 1000 data items, if time is proportional to: –log 10 N –N –N log 10 N –N 2 –N 3 (time for 1 item) x (big-O( ) time complexity of N items)

13. 13 How do we calculate big-O? 1Loops 2Nested loops 3Consecutive statements 4If-then-else statements 5Logarithmic complexity Five guidelines for finding out the time complexity of a piece of code

14. 14 Guideline 1: Loops The running time of a loop is, at most, the running time of the statements inside the loop (including tests) multiplied by the number of iterations. for (i=1; i<=n; i++) { m = m + 2; } constant time executed n times Total time = a constant c * n = cn = O(N)

15. 15 Guideline 2: Nested loops Analyse inside out. Total running time is the product of the sizes of all the loops. for (i=1; i<=n; i++) { for (j=1; j<=n; j++) { k = k+1; } constant time outer loop executed n times inner loop executed n times Total time = c * n * n * = cn 2 = O(N 2 )

16. 16 Guideline 3: Consecutive statements Add the time complexities of each statement. Total time = c 0 + c 1 n + c 2 n 2 = O(N 2 ) x = x +1; for (i=1; i<=n; i++) { m = m + 2; } for (i=1; i<=n; i++) { for (j=1; j<=n; j++) { k = k+1; } inner loop executed n times outer loop executed n times constant time executed n times constant time

17. 17 Guideline 4: If-then-else statements Worst-case running time: the test, plus either the then part or the else part (whichever is the larger). if (depth( ) != otherStack.depth( ) ) { return false; } else { for (int n = 0; n < depth( ); n++) { if (!list[n].equals(otherStack.list[n])) return false; } then part: constant else part: (constant + constant) * n test: constant another if : constant + constant (no else part) Total time = c 0 + c 1 + (c 2 + c 3 ) * n = O(N)

18. 18 Guideline 5: Logarithmic complexity An algorithm is O(log N) if it takes a constant time to cut the problem size by a fraction (usually by ½) Example algorithm (binary search): finding a word in a dictionary of n pages Look at the centre point in the dictionary Is word to left or right of centre? Repeat process with left or right part of dictionary until the word is found

19. 19 big-O notation - Example 1 Prove/disprove that 3n 2 +2n+5 = O(n 2 ) 10 n 2 = 3n 2 + 2n 2 + 5n 2  3n 2 + 2n + 5 for n  1 c = 10, n 0 = 1 The number of (n 0,c) pairs is not important. Having only one pair satisfying the condition is enough.

20. 20 big-O notation - Example 2 If T(N) can be expressed as T(N)=O(7n 2 +5n+4), T(N) can be anything from the followings T(N)=n 2 T(N)=4n+7 T(N)=1000n 2 +2n+300 T(N)= O(7n 2 +5n+4) =O(n 2 )

21. 21 big-O notation - Example 3 Write the big-O notations for the follwing running times f1(n) = 10 n + 25 n 2 f2(n) = 20 n log n + 5 n f3(n) = 12 n log n + 0.05 n 2 f4(n) = n 1/2 + 3 n log n O(n 2 ) O(n log n) O(n 2 ) O(n log n)

22. 22 Analysis of Running Times Strategy: Upper and lower bounds Upper bound Running time function Lower bound

23. 23 Analysis of Running Times Determing the running times exactly is a difficult task. Therefore: –Analysis of the best cases –Analysis of the average cases which is quite difficult –Analysis of the worst cases which is easier have to be considered

24. 24 In some cases, it is important to consider the best, worst and/or average (or typical) performance of an algorithm: E.g., when sorting a list into order, if it is already in order then the algorithm may have very little work to do The worst-case analysis gives a bound for all possible input (and may be easier to calculate than the average case) Best, average, worst-case complexity – Worst, O(N) or o(N):  or > true function * – Typical, Θ(N):  true function * – Best, Ω(N):  true function * * These approximations are true only after N has passed some value

25. 25  Notation- Asymptotic Bound f(n)f(n) c g(n)  Reverse of big-O notation  If there exist positive c and n 0 constatnts can be found for T(N)  c f(n) where N  n0 then T(N)=  (f(n))  f(n) is an asymptotic lower bound of T(N) n0n0

26. 26  Notation- Example 1 7n 2 +3n+5 = O(n 4 ) 7n 2 +3n+5 = O(n 3 ) 7n 2 +3n+5 = O(n 2 ) 7n 2 +3n+5 =  (n 2 ) 7n 2 +3n+5 =  (n) 7n 2 +3n+5 =  (1)

27. 27 Running Time 6N+4=O(N) 1 1 int Sum (int N) { int i, PartialSum; PartialSum=0; for(i=1 ;i<=N ; i++) PartialSum+=i*i*i; return PartialSum; } Algorithm 1 N+N+2N 1+(N+1)+N

28. 28 Running Time O(N 2 ) for(i=0; i<N; i++) for(j=1; j<=N; i++) k++; Algorithm 2 for(i=0; i<N; i++) A[i]=0; for(i=0; i<N; i++) for(j=1; j<=N; i++) A[i]+=A[j]+i+j; Running Time O(N 2 )

29. 29 Running Time max_ Running_Time (S1,S2) If( condition ) S1 Else S2 Algorithm 3

30. 30 int binary search(A,key,N) low=0, high=N-1 while(low  high) mid=(low+high)/2 if(A[mid]<key) low=mid+1 if(A[mid]>key) high=mid-1; if(A[mid]=key) return mid Return not found Algorithm 4 Running Time O(logN). Since the number of input is halved in each iteration.

31. 31 int binarysearch(A,key,low,high) if (low>high) Return not found else mid=(low+high)/2 if(A[mid]<key) Return binarysearch(A,key,mid+1,high) if(A[mid]>key) Return binary search(A,key,low,mid-1) if (A[mid]=key) Return mid Algorithm 5 T(N)=T(N/2)+O(1)

32. 32 Running Time O(N) MaxSubsequenceSum(const int A[], int n) ThisSum=MaxSum=0; for(j=0;j<N;j++) ThisSum+=A[j]; if (ThisSum<MaxSum) MaxSum=ThisSum; else if(ThisSum<0) ThisSum=0; Return MaxSum; Algorithm 6

33. 33 Performance isn’t everything! There can be a tradeoff between: –Ease of understanding, writing and debugging –Efficient use of time and space So, maximum performance is not always desirable However, it is still useful to compare the performance of different algorithms, even if the optimal algorithm may not be adopted