1. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 1 http://www.cs.cityu.edu.hk/~helena Analysis of Algorithms

2. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 2 http://www.cs.cityu.edu.hk/~helena Analysis of Algorithms Coming up Asymptotic performance, Insertion Sort A formal introduction to asymptotic notation (Chap 2.1-2.2, Chap 3.1)

3. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 3 http://www.cs.cityu.edu.hk/~helena Asymptotic performance “How does the algorithm behave as the problem size gets very large?” Running time Memory/storage requirements Bandwidth/power requirements/logic gates/etc. In analysis of algorithms, we care most about asymptotic performance

4. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 4 http://www.cs.cityu.edu.hk/~helena Asymptotic performance Assume: an algorithm can solve a problem of size n in f(n) microseconds (10 -6 seconds). f(n)n=20n=40n=60 Log 2 n Sqrt(n) n n log 2 n n2n2 n4n4 2n2n n! 5.32 * 10 -6 sec4.32 * 10 -6 sec5.91 * 10 -6 sec 6.32 * 10 -6 sec4.47 * 10 -6 sec7.75 * 10 -6 sec 40 * 10 -6 sec20 * 10 -6 sec60 * 10 -6 sec 213 * 10 -6 sec86 * 10 -6 sec354 * 10 -6 sec 1600 * 10 -6 sec400 * 10 -6 sec3600 * 10 -6 sec 2.56 sec0.16 sec12.96 sec

5. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 5 http://www.cs.cityu.edu.hk/~helena Asymptotic performance Assume: an algorithm can solve a problem of size n in f(n) microseconds (10 -6 seconds).

6. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 6 http://www.cs.cityu.edu.hk/~helena Input Size Time and space complexity is generally a function of the input size E.g., sorting, multiplication How we characterize input size depends: Sorting: number of input items Multiplication: total number of bits Graph algorithms: number of nodes & edges

7. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 7 http://www.cs.cityu.edu.hk/~helena Insertion Sort

8. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 8 http://www.cs.cityu.edu.hk/~helena Insertion Sort To sort A[1..n] in place: Steps: Pick element A[j] Move A[j-1..1] to the right until proper position for A[j] is found. Currently sorted part Currently unsorted part j1…j+1..n

9. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 9 http://www.cs.cityu.edu.hk/~helena Insertion Sort

10. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 10 http://www.cs.cityu.edu.hk/~helena Correctness of Insertion Sort To prove Insertion Sort is correct, we state the loop invariant: “At start of each iteration of the for loop, A[1..j-1] consists of the elements originally in A[1..j-1] but in sorted order.” We observe 3 properties: Initialization: It is true prior to the first iteration of the loop Maintenance: If it is true before an iteration, it remains true before next iteration. Termination: When the loop terminates, the invariant gives a useful property that helps show the algorithm is correct. Can you state these properties for Insertion Sort?

11. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 11 http://www.cs.cityu.edu.hk/~helena Correctness of Insertion Sort loop invariant “At start of each iteration of the for loop, A[1..j-1] consists of the elements originally in A[1..j-1] but in sorted order.” Initialization Before the first iteration, j=2. => A[1.. j-1] contains only A[1]. => Loop invariant holds prior to the first iteration. Maintenance The outer loop moves A[j-1],A[j-2],A[j-3].. to the right until the proper position for A[j] is founded. Then A[j] is inserted. => if the loop invariant is true before an iteration, it remains true before next iteration. Termination The outer loop ends with j=n+1. Substituting n+1 for j in the loop invariant, we get “A[1..n] consists of the n sorted elements.”

12. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 12 http://www.cs.cityu.edu.hk/~helena Analyzing Insertion Sort times n n-1  j=2..n t j  j=2..n (t j -1) n-1 Cost c 1 c 2 0 c 4 c 5 c 6 c 7 c 8 The running time T(n) = c 1 *n+c 2 *(n-1)+c 4 *(n-1)+c 5 *(  j=2..n t j )+c 6 *(  j=2..n (t j -1))+c 7 *(  j=2..n (t j -1))+c 8 *(n-1) t j = no. of times that line 5 is executed, for each j.

13. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 13 http://www.cs.cityu.edu.hk/~helena Analyzing Insertion Sort Noting :  j=2..n j = __________  j=2..n (j-1) = _______ T(n) = c 1 *n+c 2 *(n-1)+c 4 *(n-1)+c 5 *(  j=2..n t j )+ c 6 *(  j=2..n (t j -1))+c 7 *(  j=2..n (t j -1))+c 8 *(n-1) Worse case: Reverse sorted  inner loop body executed for all previous elements.  t j =j.  T(n) = c 1 *n+c 2 *(n-1)+c 4 *(n-1)+c 5 *(  j=2..n j)+ c 6 *(  j=2..n (j-1))+c 7 *(  j=2..n (j-1))+c 8 *(n-1)  T(n) = An 2 +Bn+C  T(n) = c 1 *n+c 2 *(n-1)+c 4 *(n-1)+c 5 *(  j=2..n j)+ c 6 *(  j=2..n (j-1))+c 7 *(  j=2..n (j-1))+c 8 *(n-1)

14. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 14 http://www.cs.cityu.edu.hk/~helena Analyzing Insertion Sort T(n)=c 1 *n+c 2 *(n-1)+c 4 *(n-1)+c 5 *(  j=2..n t j )+c 6 *(  j=2..n (t j -1))+c 7 *(  j=2..n (t j -1))+c 8 *(n-1) Worst caseReverse sorted => inner loop body executed for all previous elements. So, t j =j. => T(n) is quadratic: T(n)=An 2 +Bn+C Average caseHalf elements in A[1..j-1] are less than A[j]. So, t j = j/2 => T(n) is also quadratic: T(n)=An 2 +Bn+C Best case Already sorted => inner loop body never executed. So, t j =1. => T(n) is linear: T(n)=An+B

15. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 15 http://www.cs.cityu.edu.hk/~helena Kinds of Analysis (Usually) Worst case: T(n) = max time on any input of size n Knowing it gives us a guarantee about the upper bound. In some cases, worst case occurs fairly often Average case is often as bad as worst case. (Sometimes) Average case: T(n) = average time over all inputs of size n (Rarely) Best case: Cheat with slow algorithm that works fast on some input. Good only for showing bad lower bound.

16. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 16 http://www.cs.cityu.edu.hk/~helena Random-Access Machine Analysis is performed with respect to a computational model We usually use a generic uniprocessor random-access machine (RAM) All memory equally expensive to access No concurrent operations All reasonable instructions take unit time (Except, of course, function calls) Constant word size

17. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 17 http://www.cs.cityu.edu.hk/~helena Order of Growth  Ignore machine-dependent constants  Look at growth of T(n) as n ->   Drop low-order terms, Ignore leading constants Eg. worse case of insertion sort, T(n) = An 2 +Bn+C Order of Growth = n 2 “T(n) is in  (n 2 )” For convenience, we usually say “T(n) is  (n 2 )” and “T(n) =  (n 2 )” An algorithm is more efficient if its worst-case running time has a lower order of growth.  (n 2 ) is a set of functions that relates to n 2 in some way. (We’ll define later)

18. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 18 http://www.cs.cityu.edu.hk/~helena Asymptotic Notations 3 major notations for describing algorithm complexities: Asymptotic Tight Bound:  Intuitively like “=” Asymptotic Upper Bound:  Intuitively like “  ” Asymptotic Lower Bound:  Intuitively like “  ” Other notations: o,  Intuitively like “ ” Eg., Insertion Sort’s worse case running time is  (n 2 ). Insertion Sort’s best case running time is  (n). Insertion Sort’s running time is  (n). Insertion Sort’s running time is O(n 2 )... running time is in  (n 2 ).

19. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 19 http://www.cs.cityu.edu.hk/~helena Asymptotic Notations Very often the algorithm complexity can be observed directly from simple algorithms: The levels of nested loops. O(n 2 )  (n)

20. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 20 http://www.cs.cityu.edu.hk/~helena Definitions of O, , and  O(g(n)) = {f(n): there exist positive constants c and n 0 such that 0  f(n)  cg(n) for all n  n 0 } f(n) c 1 g(n) c 2 g(n) n0n0 f(n) is  (g(n)) f(n) cg(n) n0n0 f(n) is O(g(n)) f(n) cg(n) n0n0 f(n) is  (g(n))  (g(n)) = {f(n): there exist positive constants c and n 0 such that 0  cg(n)  f(n) for all n  n 0 }  (g(n)) = {f(n): there exist positive constants c 1, c 2, n 0 such that 0  c 1 g(n)  f(n)  c 2 g(n) for all n  n 0 }  (g(n)) O(g(n)) and  (g(n))

21. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 21 http://www.cs.cityu.edu.hk/~helena Example To do so, we must determine positive constants c 1, c 2, n 0 such that c 1 n 2  0.5n 2 -3n  c 2 n 2, for all n  n 0. Dividing by n 2 yields: c 1  0.5 - 3/n  c 2. c 1  0.5-3/n holds for any value of n  7 by choosing c 1  1/14 0.5-3/n  c 2 holds for any value n  1 by choosing c 2  0.5 Hence by choosing c 1 =1/14, c 2 =0.5, and n 0 =7, we can verify that 0.5n 2 - 3n =  (n 2 ). According to the formal definition of , prove that 0.5n 2 - 3n =  (n 2 )

22. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 22 http://www.cs.cityu.edu.hk/~helena Example (cont’d) We have shown that 0.5n 2 - 3n =  (n 2 ). Similar prove can be applied to show An 2 +Bn+C =  (n 2 ) and An+B =  (n). For insertion sort: Worse case running time: T(n) = An 2 +Bn+C => T(n) =  (n 2 ) Average case running time: T(n) = An 2 +Bn+C => T(n) =  (n 2 ) Best case running time: T(n) = An+B => T(n) =  (n) Recall that, informally, the  () term can be obtained by observing the highest order term and drop the leading constant.

23. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 23 http://www.cs.cityu.edu.hk/~helena Example According to the formal definition of O, prove that 6n 3  O(n 2 ) Prove by contradiction: Suppose c, n 0 exist such that 6n 3  cn 2, for all n  n 0. => n  c 2 /6, which is impossible for arbitrarily large n, since c is a constant. => 6n 3  cn 2 is not correct => 6n 3  O(n 2 )

24. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 24 http://www.cs.cityu.edu.hk/~helena Points to Note We can write 2n 2 +3n+1 = 2n 2 +  (n). f(n) = n 2 + O(n) means f(n) = n 2 + h(n) for some h(n)  O(n) O-notation is an upper-bound notation. It makes no sense to say f(n) is at least O(n). Why? It is also correct to write n 2 =O(n 3 ), and n 2 =  (n). Proof?

25. CS3381 Des & Anal of Alg (2001-2002 SemA) City Univ of HK / Dept of CS / Helena Wong 2. Analysis of Algorithms - 25 http://www.cs.cityu.edu.hk/~helena Analysis of Algorithms Summary Algorithm, input, output, instance, “Correct Algorithm” (recall Introduction) Asymptotic Performance, Input Size Insertion Sort Proof: Insertion Sort is correct (Loop Invariant + 3 properties) Analysis of Insertion Sort, Worse/Average/Best case Order of Growth Asymptotic Notations