Analysis of Algorithms CS 477/677 Midterm Exam Review Instructor: George Bebis

2 General Advice for Study Understand how the algorithms are working –Work through the examples we did in class –“Narrate” for yourselves the main steps of the algorithms in a few sentences Know when or for what problems the algorithms are applicable Do not memorize algorithms

3 Asymptotic notations A way to describe behavior of functions in the limit –Abstracts away low-order terms and constant factors –How we indicate running times of algorithms –Describe the running time of an algorithm as n goes to  O notation: asymptotic “less than”: f(n) “≤” g(n)  notation: asymptotic “greater than”: f(n) “≥” g(n)  notation: asymptotic “equality”: f(n) “=” g(n)

4 Exercise Order the following 6 functions in increasing order of their growth rates: –nlog n, log 2 n, n 2, 2 n,, n. log 2 n n nlogn n2n2 2n2n

5 Running Time Analysis Algorithm Loop1(n) p=1 for i = 1 to 2n p = p*i Algorithm Loop2(n) p=1 for i = 1 to n 2 p = p*i O(n) O(n 2 )

6 Running Time Analysis Algorithm Loop3(n) s=0 for i = 1 to n for j = 1 to i s = s + I O(n 2 )

7 Recurrences Def.: Recurrence = an equation or inequality that describes a function in terms of its value on smaller inputs, and one or more base cases Recurrences arise when an algorithm contains recursive calls to itself Methods for solving recurrences –Substitution method –Iteration method –Recursion tree method –Master method Unless explicitly stated choose the simplest method for solving recurrences

8 Analyzing Divide and Conquer Algorithms The recurrence is based on the three steps of the paradigm: –T(n) – running time on a problem of size n –Divide the problem into a subproblems, each of size n/b: –Conquer (solve) the subproblems –Combine the solutions  (1) if n ≤ c T(n) = takes D(n) aT(n/b) C(n) aT(n/b) + D(n) + C(n) otherwise

9 Master’s method Used for solving recurrences of the form: where, a ≥ 1, b > 1, and f(n) > 0 Compare f(n) with n log b a : Case 1: if f(n) = O( n log b a -  ) for some  > 0, then: T(n) =  ( n log b a ) Case 2: if f(n) =  ( n log b a ), then: T(n) =  ( n log b a lgn ) Case 3: if f(n) =  ( n log b a +  ) for some  > 0, and if af(n/b) ≤ cf(n) for some c < 1 and all sufficiently large n, then: T(n) =  (f(n)) regularity condition

10 Sorting Insertion sort –Design approach: –Sorts in place: –Best case: –Worst case: Bubble Sort –Design approach: –Sorts in place: –Running time: Yes  (n)  (n 2 ) incremental Yes  (n 2 ) incremental

11 Analysis of Insertion Sort INSERTION-SORT(A) for j ← 2 to n do key ← A[ j ] Insert A[ j ] into the sorted sequence A[1.. j -1] i ← j - 1 while i > 0 and A[i] > key do A[i + 1] ← A[i] i ← i – 1 A[i + 1] ← key cost times c 1 n c 2 n-1 0 n-1 c 4 n-1 c 5 c 6 c 7 c 8 n-1  n 2 /2 comparisons  n 2 /2 exchanges

12 Analysis of Bubble-Sort T(n) =  (n 2 ) Alg.: BUBBLESORT(A) for i  1 to length[A] do for j  length[A] downto i + 1 do if A[j] < A[j -1] then exchange A[j]  A[j-1] T(n) =c 1 (n+1) +c2c2 c3c3 c4c4 =  (n) + (c 2 + c 2 + c 4 ) Comparisons:  n 2 /2 Exchanges:  n 2 /2

13 Sorting Selection sort –Design approach: –Sorts in place: –Running time: Merge Sort –Design approach: –Sorts in place: –Running time: Yes  (n 2 ) incremental No  (nlgn) divide and conquer

14  n 2 /2 comparisons Analysis of Selection Sort Alg.: SELECTION-SORT(A) n ← length[A] for j ← 1 to n - 1 do smallest ← j for i ← j + 1 to n do if A[i] < A[smallest] then smallest ← i exchange A[j] ↔ A[smallest] cost times c 1 1 c 2 n c 3 n-1 c 4 c 5 c 6 c 7 n-1  n exchanges

15 MERGE-SORT Running Time Divide: –compute q as the average of p and r: D(n) =  (1) Conquer: –recursively solve 2 subproblems, each of size n/2  2T (n/2) Combine: –MERGE on an n -element subarray takes  (n) time  C(n) =  (n)  (1) if n =1 T(n) = 2T(n/2) +  (n) if n > 1

16 Quicksort –Idea: –Design approach: –Sorts in place: –Best case: –Worst case: Partition –Running time Randomized Quicksort Yes  (nlgn)  (n 2 ) Divide and conquer  (n) Partition the array A into 2 subarrays A[p..q] and A[q+1..r], such that each element of A[p..q] is smaller than or equal to each element in A[q+1..r]. Then sort the subarrays recursively.  (nlgn) – on average  (n 2 ) – in the worst case

17 Analysis of Quicksort

18 The Heap Data Structure Def: A heap is a nearly complete binary tree with the following two properties: –Structural property: all levels are full, except possibly the last one, which is filled from left to right –Order (heap) property: for any node x Parent(x) ≥ x Heap

19 Array Representation of Heaps A heap can be stored as an array A. –Root of tree is A[1] –Parent of A[i] = A[  i/2  ] –Left child of A[i] = A[2i] –Right child of A[i] = A[2i + 1] –Heapsize[A] ≤ length[A] The elements in the subarray A[(  n/2  +1).. n] are leaves The root is the max/min element of the heap A heap is a binary tree that is filled in order

20 Operations on Heaps (useful for sorting and priority queues) –MAX-HEAPIFY O(lgn) –BUILD-MAX-HEAP O(n) –HEAP-SORT O(nlgn) –MAX-HEAP-INSERT O(lgn) –HEAP-EXTRACT-MAX O(lgn) –HEAP-INCREASE-KEY O(lgn) –HEAP-MAXIMUM O(1) –You should be able to show how these algorithms perform on a given heap, and tell their running time

21 Lower Bound for Comparison Sorts Theorem: Any comparison sort algorithm requires  (nlgn) comparisons in the worst case. Proof: How many leaves does the tree have? –At least n! (each of the n! permutations if the input appears as some leaf)  n! –At most 2 h leaves  n! ≤ 2 h  h ≥ lg(n!) =  (nlgn) h leaves

22 Linear Time Sorting We can achieve a better running time for sorting if we can make certain assumptions on the input data: –Counting sort: Each of the n input elements is an integer in the range [0... r] r=O(n)

23 Analysis of Counting Sort Alg.: COUNTING-SORT(A, B, n, k) 1.for i ← 0 to r 2. do C[ i ] ← 0 3.for j ← 1 to n 4. do C[A[ j ]] ← C[A[ j ]] C[i] contains the number of elements equal to i 6.for i ← 1 to r 7. do C[ i ] ← C[ i ] + C[i -1] 8. C[i] contains the number of elements ≤ i 9.for j ← n downto do B[C[A[ j ]]] ← A[ j ] 11. C[A[ j ]] ← C[A[ j ]] - 1  (r)  (n)  (r)  (n) Overall time:  (n + r)

24 RADIX-SORT Alg.: RADIX-SORT (A, d) for i ← 1 to d do use a stable sort to sort array A on digit i 1 is the lowest order digit, d is the highest-order digit  (d(n+k))

25 Analysis of Bucket Sort Alg.: BUCKET-SORT(A, n) for i ← 1 to n do insert A[i] into list B[  nA[i]  ] for i ← 0 to n - 1 do sort list B[i] with quicksort sort concatenate lists B[0], B[1],..., B[n -1] together in order return the concatenated lists O(n)  (n) O(n)  (n)