DATA STRUCTURES AND ALGORITHMS Lecture Notes 1 Prepared by İnanç TAHRALI

2 ROAD MAP What is an algorithm ? What is a data structure ? Analysis of An Algorithm Asymptotic Notations –Big Oh Notation –Omega Notation –Theta Notation –Little o Notation Rules about Asymptotic Notations

3 What is An Algorithm ? Definition : A finite, clearly specified sequence of instructions to be followed to solve a problem.

4 What is An Algorithm ? int Sum (int N) PartialSum  0 i  1 foreach (i > 0) and (i<=N) PartialSum  PartialSum + (i*i*i) increase i with 1 return value of PartialSum int Sum (int N) { int PartialSum = 0 ; for (int i=1; i<=N; i++) PartialSum += i * i * i; return PartialSum; } Problem : Write a program to calculate

5 Properties of an Algorithm Effectiveness –simple –can be carried out by pen and paper Definiteness –clear –meaning is unique Correctness –give the right answer for all possible cases Finiteness –stop in reasonable time

6 Is function F an algorithm? int F (int m) { n = m; while (n>1) { if (n/2 == 0) n=n/2; else n=3*n+1; } return m; }

7 What is a Data Structure ? Definition : An organization and representation of data –representation data can be stored variously according to their type –signed, unsigned, etc. example : integer representation in memory –organization the way of storing data changes according to the organization –ordered, inordered, tree example : if you have more then one integer ?

8 Properties of a Data Structure ? Efficient utilization of medium Efficient algorithms for –creation –manipulation (insertion/deletion) –data retrieval (Find) A well-designed data structure allows using little –resources –execution time –memory space

9 The Process of Algorithm Development Design –divide&conquer, greedy, dynamic programming Validation –check whether it is correct Analysis –determine the properties of algorithm Implementation Testing –check whether it works for all possible cases

10 Analysis of Algorithm Analysis investigates –What are the properties of the algorithm? in terms of time and space –How good is the algorithm ? according to the properties –How it compares with others? not always exact –Is it the best that can be done? difficult !

11 Mathematical Background Assume the functions for running times of two algorthms are found ! For input size N Running time of Algorithm A = T A (N) = 1000 N Running time of Algorithm B = T B (N) = N 2 Which one is faster ?

12 NTATA TBTB sec10 -4 sec sec10 -2 sec sec sec100 sec sec10000 sec If the unit of running time of algorithms A and B is µsec Mathematical Background So which algorithm is faster ?

13 Mathematical Background If N T B (N) o/wT B (N) > T A (N) Compare their relative growth ?

14 Mathematical Background Is it always possible to have definite results? NO ! The running times of algorithms can change because of the platform, the properties of the computer, etc. We use asymptotic notations (O, Ω, θ, o) compare relative growth compare only algorithms

15 Big Oh Notation (O) Provides an “upper bound” for the function f Definition : T(N) = O (f(N)) if there are positive constants c and n 0 such that T(N) ≤ cf(N) when N ≥ n 0 –T(N) grows no faster than f(N) –growth rate of T(N) is less than or equal to growth rate of f(N) for large N –f(N) is an upper bound on T(N) not fully correct !

16 Big Oh Notation (O) Analysis of Algorithm A 1000 N ≤ cN if c= 2000 and n 0 = 1 for all N is right

17 Examples 7n+5 = O(n) for c=8 and n 0 =5 7n+5 ≤ 8nn>5 = n 0 7n+5 = O(n 2 ) for c=7 and n 0 =2 7n+5 ≤ 7n 2 n≥n 0 7n 2 +3n = O(n) ?

18 Advantages of O Notation It is possible to compare of two algorithms with running times Constants can be ignored. –Units are not important O(7n 2 ) = O(n 2 ) Lower order terms are ignored –O(n 3 +7n 2 +3) = O(n 3 )

19 Running Times of Algorithm A and B T A (N) = 1000 N = O(N) T B (N) = N 2 = O(N 2 ) A is asymptotically faster than B !

20 Omega Notation (Ω) Definition : T(N) = Ω (f(N)) if there are positive constants c and n 0 such that T(N) ≥ c f(N) when N≥ n 0 –T(N) grows no slower than f(N) –growth rate of T(N) is greater than or equal to growth rate of f(N) for large N –f(N) is a lower bound on T(N) not fully correct !

21 Omega Notation Example: n 1/2 =  (lg n). for c = 1 and n 0 = 16 Let n > 16 c*(lg n) ≤ n 1/2

22 Omega Notation Theorem: f(N) = O(g(n)) g(n) = Ω(f(N)) Proof: f(N) ≤ c 1 g(n) g(n) ≥ c 2 f(N) divide the left side with c 1 1/c 1 f(N) ≤ g(n) g(n) ≥ c 2 f(N) if we choose c 2 as 1/c 1 then theorem is right.

23 Omega Notation 7n 2 + 3n + 5 = O(n 4 ) 7n 2 + 3n + 5 = O(n 3 ) 7n 2 + 3n + 5 = O(n 2 ) 7n 2 + 3n + 5 = Ω(n 2 ) 7n 2 + 3n + 5 = Ω(n) 7n 2 + 3n + 5 = Ω(1) n 2 and 7n 2 + 3n + 5 grows at the same rate 7n 2 + 3n + 5 = O(n 2 ) = Ω(n 2 ) = θ (n 2 )

24 Theta Notation (θ) Definition : T(N) = θ (h(N)) if and only if T(N) = O(h(N)) and T(N) = Ω(h(N)) –T(N) grows as fast as h(N) –growth rate of T(N) and h(N) are equal for large N –h(N) is a tight bound on T(N) not fully correct !

25 Theta Notation (θ) Example : T(N) = 3N 2 T(N) = O(N 4 ) T(N) = O(N 3 ) T(N) = θ(N 2 )  best

26 Little O Notation (o) Definition : T(N) = o(p(N)) if T(N) = O(p(N)) and T(N)≠θ(p(N)) –f(N) grows strictly faster than T(N) –growth rate of T(N) is less than the growth rate of f(N) for large N –f(N) is an upperbound on T(N) (but not tight) not fully correct !

27 Little O Notation (o) Example : T(N) = 3N 2 T(N) = o(N 4 ) T(N) = o(N 3 ) T(N) = θ(N 2 )

28 RULES RULE 1: if T 1 (N) = O(f(N)) and T 2 (N) = O(g(N)) then a) T 1 (N) + T 2 (N) = max (O(f(N)), O(g(N))) b) T 1 (N) * T 2 (N) = O(f(N) * g(N)) You can prove these ? Is it true for θ notation ? What about Ω notation?

29 RULES RULE 2: if T(N) is a polynomial of degree k T(N) = a k N k + a k-1 N k-1 + … + a 1 N + a 0 then T(N) = θ(N k )

30 RULES RULE 3: log k N = o(N) for any constant k logarithm grows very slowly !

31 Some Common Functions c = o (log N) => c=O(log N) but c≠Ω(logN) log N = o(log 2 N) log 2 N = o(N) N = o(N log N) N = o (N 2 ) N 2 = o (N 3 ) N 3 = o (2 N )

32 Example : T(N) = 4N 2 T(N) = O(2N 2 ) correct but bad style T(N) = O(N 2 ) drop the constants T(N) = O(N 2 +N) correct but bad style T(N) = O(N 2 ) ignore low order terms

33 Another Way to Compute Growth Rates

34 Example : f(N) = 7N 2 g(N) = N 2 + N

35 Example : f(N) = N logNg(N) = N 1.5 compare logN with N 0.5 compare log 2 N with N compare log 2 N with o(N) N logN = o(N 1.5 )