1. CS 3343: Analysis of Algorithms
Lecture 4: Sum of series, Analyzing recursive algorithms
6/27/2018

2. Outline Review of last lecture Sum of series
Analyzing recursive algorithms
6/27/2018

3. L’ Hopital’s rule lim f(n) / g(n) = lim f(n)’ / g(n)’ n→∞ n→∞
Condition:
If both lim f(n) and lim g(n) =  or 0
n→∞
n→∞
6/27/2018

4. Stirling’s formula or
(constant)
6/27/2018

5. Properties of asymptotic notations
Textbook page 51
Transitivity
f(n) = (g(n)) and g(n) = (h(n))
=> f(n) = (h(n))
(holds true for o, O, , and  as well).
Symmetry
f(n) = (g(n)) if and only if g(n) = (f(n))
Transpose symmetry
f(n) = O(g(n)) if and only if g(n) = (f(n))
f(n) = o(g(n)) if and only if g(n) = (f(n))
6/27/2018

6. logarithms lg n = log2 n ln n = loge n, e ≈ 2.718 lgkn = (lg n)k
lg lg n = lg (lg n) = lg(2)n
lg(k) n = lg lg lg … lg n
lg24 = ?
lg(2)4 = ?
Compare lgkn vs lg(k)n?
6/27/2018

7. Useful rules for logarithms
For all a > 0, b > 0, c > 0, the following rules hold
logba = logca / logcb = lg a / lg b
logban = n logba
blogba = a
log (ab) = log a + log b
lg (2n) = ?
log (a/b) = log (a) – log(b)
lg (n/2) = ?
lg (1/n) = ?
logba = 1 / logab
6/27/2018

8. Useful rules for exponentials
For all a > 0, b > 0, c > 0, the following rules hold
a0 = 1 (00 = ?)
a1 = a
a-1 = 1/a
(am)n = amn
(am)n = (an)m
aman = am+n
6/27/2018

9. More advanced dominance ranking
6/27/2018

10. General plan for analyzing time efficiency of a non-recursive algorithm
Decide parameter (input size)
Identify most executed line (basic operation)
worst-case = average-case?
T(n) = i ti
T(n) = Θ (f(n))
6/27/2018

11. Find the order of growth for sums
T(n) = i=1..n i = Θ (n2)
T(n) = i=1..n log (i) = ?
T(n) = i=1..n n / 2i = ?
T(n) = i=1..n 2i = ? …
How to find out the actual order of growth?
Math…
Textbook Appendix A.1 (page )
6/27/2018

12. Closed form, or explicit formula
Arithmetic series
An arithmetic series is a sequence of numbers such that the difference of any two successive members of the sequence is a constant.
e.g.: 1, 2, 3, 4, 5
or 10, 12, 14, 16, 18, 20
In general:
Recursive definition
Closed form, or explicit formula
Or:
6/27/2018

13. Sum of arithmetic series
If a1, a2, …, an is an arithmetic series, then
e.g … + 99 = ?
(series definition: ai = 2i-1)
This is ∑ i = 1 to 50 (ai)
6/27/2018

14. Closed form, or explicit formula
Geometric series
A geometric series is a sequence of numbers such that the ratio between any two successive members of the sequence is a constant.
e.g.: 1, 2, 4, 8, 16, 32
or 10, 20, 40, 80, 160
or 1, ½, ¼, 1/8, 1/16
In general:
Recursive definition
Closed form, or explicit formula
Or:
6/27/2018

15. Sum of geometric series
if r < 1
if r > 1
if r = 1
6/27/2018

16. Sum of geometric series
if r < 1
if r > 1
if r = 1
6/27/2018

17. Important formulas
6/27/2018

18. Sum manipulation rules
Example:
6/27/2018

19. Sum manipulation rules
Example:
6/27/2018

20. Examples i=1..n n / 2i = n * i=1..n (½)i = ?
using the formula for geometric series:
i=0..n (½)i = 1 + ½ + ¼ + … (½)n = 2
Application: algorithm for allocating dynamic memories
6/27/2018

21. Examples
i=1..n log (i) = log 1 + log 2 + … + log n = log 1 x 2 x 3 x … x n = log n!
=  (n log n)
Application: algorithm for selection sort using priority queue
6/27/2018

22. Problem of the day
How do you find a coffee shop if you don’t know on which direction it might be?
6/27/2018

23. Recursive definition of sum of series
T (n) = i=0..n i is equivalent to:
T(n) = T(n-1) + n
T(0) = 0
T(n) = i=0..n ai is equivalent to:
T(n) = T(n-1) + an
T(0) = 1
Recurrence relation
Boundary condition
Recursive definition is often intuitive and easy to obtain. It is very useful in analyzing recursive algorithms, and some non-recursive algorithms too.
6/27/2018

24. Analyzing recursive algorithms
6/27/2018

25. Recursive algorithms General idea: Divide and Conquer
Divide a large problem into smaller ones
By a constant ratio
By a constant or some variable
Solve each smaller one recursively or explicitly
Combine the solutions of smaller ones to form a solution for the original problem
Divide and Conquer
6/27/2018

26. Merge sort MERGE-SORT A[1 . . n] If n = 1, done.
Recursively sort A[ n/2 ] and A[ n/2 n ] .
“Merge” the 2 sorted lists.
Key subroutine: MERGE
6/27/2018

27. Merging two sorted arrays
Subarray 1
Subarray 2 20 13 7 2 12 11 9 1
6/27/2018

28. Merging two sorted arrays
Subarray 1
Subarray 2 20 13 7 2 12 11 9 1
6/27/2018

29. Merging two sorted arrays20 13 7 2 12 11 9 1
6/27/2018

30. Merging two sorted arrays20 13 7 2 12 11 9 1
6/27/2018

31. Merging two sorted arrays20 13 7 2 12 11 9 1 1
6/27/2018

32. Merging two sorted arrays20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 1
6/27/2018

33. Merging two sorted arrays20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 1 2
6/27/2018

34. Merging two sorted arrays20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 1 2
6/27/2018

35. Merging two sorted arrays20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 1 2 7
6/27/2018

36. Merging two sorted arrays20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 20 13 12 11 9 1 2 7
6/27/2018

37. Merging two sorted arrays20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 20 13 12 11 9 1 2 7 9
6/27/2018

38. Merging two sorted arrays20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 20 13 12 11 9 20 13 12 11 1 2 7 9
6/27/2018

39. Merging two sorted arrays20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 20 13 12 11 9 20 13 12 11 1 2 7 9 11
6/27/2018

40. Merging two sorted arrays20 13 7 2 12 11 9 1 20 13 7 2 12 11 9 20 13 7 12 11 9 20 13 12 11 9 20 13 12 11 20 13 12 1 2 7 9 11
6/27/2018

41. Merging two sorted arrays20 13 7 2 12 11 9 1
6/27/2018

42. How to show the correctness of a recursive algorithm?
By induction:
Base case: prove it works for small examples
Inductive hypothesis: assume the solution is correct for all sub-problems
Step: show that, if the inductive hypothesis is correct, then the algorithm is correct for the original problem.
6/27/2018

43. Correctness of merge sort
MERGE-SORT A[1 . . n]
If n = 1, done.
Recursively sort A[ n/2 ] and A[ n/2 n ] .
“Merge” the 2 sorted lists.
Proof:
Base case: if n = 1, the algorithm will return the correct answer because A[1..1] is already sorted.
Inductive hypothesis: assume that the algorithm correctly sorts smaller suarrays, i.e., A[1.. n/2 ] and A[n/2+1..n].
Step: if A[1.. n/2 ] and A[n/2+1..n] are both correctly sorted, the whole array A[1.. n] is sorted after merging.
Therefore, the algorithm is correct.
6/27/2018

44. How to analyze the time-efficiency of a recursive algorithm?
Express the running time on input of size n as a function of the running time on smaller problems
6/27/2018

45. Analyzing merge sort T(n) MERGE-SORT A[1 . . n] Θ(1) 2T(n/2)
f(n)
MERGE-SORT A[1 . . n]
If n = 1, done.
Recursively sort A[ n/2 ] and A[ n/2 n ] .
“Merge” the 2 sorted lists
Sloppiness: Should be T( n/2 ) + T( n/2 ) , but it turns out not to matter asymptotically.
6/27/2018

46. Analyzing merge sort T(n) = 2 T(n/2) + f(n) +Θ(1) Divide: Trivial.
Conquer: Recursively sort 2 subarrays.
Combine: Merge two sorted subarrays
T(n) = 2 T(n/2) + f(n) +Θ(1)
# subproblems
Dividing and
Combining
subproblem size
What is the time for the base case?
What is f(n)?
What is the growth order of T(n)?
Constant
6/27/2018

47. Merging two sorted arrays20 13 7 2 12 11 9 1
Θ(n) time to merge a total of n elements (linear time).
6/27/2018

48. Recurrence for merge sort
T(n) =
Θ(1) if n = 1;
2T(n/2) + Θ(n) if n > 1.
Later we shall often omit stating the base case when T(n) = Q(1) for sufficiently small n, but only when it has no effect on the asymptotic solution to the recurrence.
But what does T(n) solve to? I.e., is it O(n) or O(n2) or O(n3) or …?
6/27/2018

49. Binary Search To find an element in a sorted array, we
Check the middle element
If ==, we’ve found it
else if less than wanted, search right half
else search left half
Example: Find 9 3 5 7 8 9 12 15
6/27/2018

50. Binary Search To find an element in a sorted array, we
Check the middle element
If ==, we’ve found it
else if less than wanted, search right half
else search left half
Example: Find 9 3 5 7 8 9 12 15
6/27/2018

51. Binary Search To find an element in a sorted array, we
Check the middle element
If ==, we’ve found it
else if less than wanted, search right half
else search left half
Example: Find 9 3 5 7 8 9 12 15
6/27/2018

52. Binary Search To find an element in a sorted array, we
Check the middle element
If ==, we’ve found it
else if less than wanted, search right half
else search left half
Example: Find 9 3 5 7 8 9 12 15
6/27/2018

53. Binary Search To find an element in a sorted array, we
Check the middle element
If ==, we’ve found it
else if less than wanted, search right half
else search left half
Example: Find 9 3 5 7 8 9 12 15
6/27/2018

54. Binary Search To find an element in a sorted array, we
Check the middle element
If ==, we’ve found it
else if less than wanted, search right half
else search left half
Example: Find 9 3 5 7 8 9 12 15
6/27/2018

55. What’s the recurrence relation for its running time?
Binary Search
BinarySearch (A[1..N], value) {
if (N == 0)
return -1; // not found
mid = (1+N)/2;
if (A[mid] == value)
return mid; // found
else if (A[mid] < value) return BinarySearch (A[mid+1, N], value) else
return BinarySearch (A[1..mid-1], value); }
What’s the recurrence relation for its running time?
6/27/2018

56. Recurrence for binary search
6/27/2018

57. Recursive Insertion Sort
RecursiveInsertionSort(A[1..n])
1. if (n == 1) do nothing;
2. RecursiveInsertionSort(A[1..n-1]);
3. Find index i in A such that A[i] <= A[n] < A[i+1];
4. Insert A[n] after A[i];
6/27/2018

58. Recurrence for insertion sort
6/27/2018

59. Compute factorial Factorial (n) if (n == 1) return 1;
return n * Factorial (n-1);
Note: here we use n as the size of the input. However, usually for such algorithms we would use log(n), i.e., the bits needed to represent n, as the input size.
6/27/2018

60. Recurrence for computing factorial
Note: here we use n as the size of the input. However, usually for such algorithms we would use log(n), i.e., the bits needed to represent n, as the input size.
6/27/2018

61. What do these mean?
Challenge: how to solve the recurrence to get a closed form, e.g. T(n) = Θ (n2) or T(n) = Θ(nlgn), or at least some bound such as T(n) = O(n2)?
6/27/2018

62. Solving recurrence
Running time of many algorithms can be expressed in one of the following two recursive forms or
Both can be very hard to solve. We focus on relatively easy ones, which you will encounter frequently in many real algorithms (and exams…)
6/27/2018

63. Solving recurrence Recursion tree / iteration method
Substitution method
Master method
6/27/2018