1. Analysis of Algorithms
Recitation 9
This week we’re doing analysis of algorithms. We’re touching on some proof techniques from class, as well as runtime (think O notation).

2. Review of Big-O

3. "Big O" Notation
Formal definition: f(n) is O(g(n)) if there exist constants c > 0 and N ≥ 0 such that for all n ≥ N, f(n) ≤ c·g(n)
c·g(n)
f(n) N
Get out far enough (for n ≥ N)
f(n) is at most c·g(n)
Intuitively, f(n) is O(g(n))
means that f(n) grows
like g(n) or slower

4. Prove that (n2 + n) is O(n2)
Formal definition: f(n) is O(g(n)) if there exist constants c > 0 and N ≥ 0 such that for all n ≥ N, f(n) ≤ c·g(n)
Let f(n) and g(n) be two functions.
f(n) >= 0 and g(n) >= 0.
n this is f(n)
<= <if 6 <= n, write as>
n + n
= <arith>
2*n
<choose c = 2>
= c*n this is c * g(n)
So choose c = 2 and N = 6
Example:
f(n) = n + 6
g(n) = n
We show that n+6 is O(n)

5. Prove that (n2 + n) is O(n2)
Formal definition: f(n) is O(g(n)) if there exist constants c > 0 and N ≥ 0 such that for all n ≥ N, f(n) ≤ c·g(n)
Let f(n) and g(n) be two functions.
f(n) >= 0 and g(n) >= 0.
It means that as n gets larger and larger, any constant c that you use becomes meaningless in relation to n, so throw it away.
The difference between executing 1,000,000 steps and 1,000,0006 is insignificant
We showed that n+6 is O(n).
In fact, you can change the 6 to any constant c you want and show that n+c is O(n)
Here, we try to explain that all this means is that, when you have a formula, to find its O(n) thingy, delete all but the most significant terms.
More examples are given later.
An algorithm that executes O(n) steps on input of size n is called a linear algorithm

6. Oft-used execution orders
In the same way, we can prove these kinds of things:
log(n) is O(log(n)) (logarithmic)
n + log(n) is O(n) (linear)
n/2 and 3*n are O(n)
n * log(n) + n is n * log(n)
n2 + 2*n is O(n2) (quadratic)
n3 + n is O(n3) (cubic)
2n + 5n is O(2n) (exponential)
Some useful facts about O notation, which may not always be obvious to students.

7. Understand? Then use informally
log(n) is O(log(n)) (logarithmic)
n + log(n) is O(n) (linear)
n/2 and 3*n are O(n)
n * log(n) + n is n * log(n)
n2 + 2*n is O(n2) (quadratic)
n3 + n is O(n3) (cubic)
2n + 5n is O(2n) (exponential)
Once you fully understand the concept, you can use it informally. Example:
An algorithm executes (7*n + 6) / log(n) steps.
It’s obviously linear, i.e. O(n)
Now we show them how to see that a messy formula is O(n), but just deleting the insignificant parts.
Here: (1) log(n), (2) division by 3, (3) addition of 6, (4) multiplication by 7.

8. Some Notes on O() Why don’t logarithm bases matter?
For constants x, y: O(logx n) = O((logx y)(logy n))
Since (logx y) is a constant, O(logx n) = O(logy n)
Usually: O(f(n)) × O(g(n)) = O(f(n) × g(n))
Such as if something that takes g(n) time for each of f(n) repetitions (loop within a loop)
Usually: O(f(n)) + O(g(n)) = O(max(f(n), g(n)))
“max” is whatever’s dominant as n approaches infinity
Example: O((n2-n)/2) = O((1/2)n2 + (-1/2)n) = O((1/2)n2) = O(n2)
Some useful facts about O notation, which may not always be obvious to students.

9. Analyzing an Algorithm

10. runtimeof MergeSort /** Sort b[h..k]. */
public static void mS(Comparable[] b, int h, int k) {
if (h >= k) return;
int e= (h+k)/2;
mS(b, h, e);
mS(b, e+1, k);
merge(b, h, e, k); }
Throughout, we use mS for mergeSort, to make slides easier to read
This is an implementation of mergesort. Sort the elements of the array b, meaning that nothing is returned but b is changed. Go through this code step by step with your group. Method mS sorts the elements of b between indexes h and k in place, so if h is at least k, then they’re nothing left to sort, so there’s nothing to do; otherwise, create e midway between h and k, sort b[h..e] and b[e+1..k] recursively, then we merge those two sorted parts. Merge here means we take two sorted parts of the array and turn them into one sorted part of the array. We’ll look at merge’s code later.
We will count the number of comparisons mS makes
Use T(n) for the number of array element comparisons that mS makes on an array of size n

11. Runtime public static void mS(Comparable[] b, int h, int k) {
if (h >= k) return;
int e= (h+k)/2;
mS(b, h, e);
mS(b, e+1, k);
merge(b, h, e, k); }
T(0) = 0
T(1) = 0
To calculate the number of steps the algorithms takes is our next task. Because of the recursion, we get a
(recurrence relation).
Clearly, when the number of elements to be sorted (the distance between h and k) is 0 or 1, we’re just done. It takes effectively 0 time.
Here, the only operations we are considering to have any significant cost are comparisons (comparing the values of things to be sorted). We’ll watch to make sure we’re not performing any other overly costly-looking operations.
Use T(n) for the number of array element comparisons that mS makes on an array of size n

12. Runtime public static void mS(Comparable[] b, int h, int k) {
if (h >= k) return;
int e= (h+k)/2;
mS(b, h, e);
mS(b, e+1, k);
merge(b, h, e, k); }
Recursion: T(n) = 2 * T(n/2) + comparisons made in merge
We are going to assume that we’re sorting a number of elements n that is a power of 2. This makes the analysis easier, and it has another property as well.
A non-rigorous justification for this: It’s fairly clear (when we see the code in full) that increasing the number of elements to sort will not make this algorithm run faster. Therefore, for any value n that is not a power of two, we know it runs faster than running our algorithm with the next power of 2 up elements. This is mostly sufficient for O() runtime analysis.
Simplify calculations: assume n is a power of 2

13. Pre: b[h..e] and b[e+1..k] are sorted.*/
/** Sort b[h..k].
Pre: b[h..e] and b[e+1..k] are sorted.*/
public static void merge (Comparable b[], int h, int e, int k) {
Comparable[] c= copy(b, h, e);
int i= h; int j= e+1; int m= 0;
/* inv: b[h..i-1] contains its final, sorted values
b[j..k] remains to be transferred
c[m..e-h] remains to be transferred */
for (i= h; i != k+1; i++) {
if (j <= k && (m > e-h || b[j].compareTo(c[m]) <= 0)) {
b[i]= b[j]; j++; }
else {
b[i]= c[m]; m++;
c free to be moved
m e-h
This is merge. You’re going to have to go over this very slowly.
An array c is created that is a copy of the first array segment we’re merging. The diagrams constitute the loop invariant. They show c, with index m, as well as b, with indices i and j. Go over the diagrams first! Then show (1) how the initalization makes the invariant true, (2) when the loop terminates, the desired answer holds, and (3) each iteration keeps the diagrams true and makes progress toward termination by increasing i.
The idea is this: we walk through the part of b that we’re merging (this is index i, which goes through the whole part of b we’re merging). At each element, we set it to be either the next element from c or the as yet unsorted part of b, depending upon which is less.
If m>e-h, that means that we’ve already merged in all of c, so we just always add stuff from the as yet unmerged part of b.
If j>k, then that means we’ve alredy merged all of the to-be-merged part of b, so we just add stuff from c.
b final, sorted free to be moved
h i j k

14. /** Sort b[h..k]. Pre: b[h..e] and b[e+1..k] are already sorted.*/
public static void merge (Comparable b[], int h, int e, int k) {
Comparable[] c= copy(b, h, e);
int i= h; int j= e+1; int m= 0;
for (i= h; i != k+1; i= i+1) {
if (j <= k && (m > e-h || b[j].compareTo(c[m]) <= 0)) {
b[i]= b[j]; j= j+1; }
else {
b[i]= c[m]; m= m+1;
O(e+1-h)
Loop body: O(1).
Executed k+1-h times.
Two things in merge take “significant” time to execute. Executing the body of this loop involves at most one comparison each time. The loop is iterated at most (k+1-h) times, so we can be sure that the number of comparisons involved is no more than the size of the array segment we’re sorting.
One might worry that creating copy c takes too much time, but it involves only copying e+1-h elements, so ultimately the time it takes is within O(k+1-h), which is what the loop part takes anyway.
We can therefore safely say that this method takes O(k-h) time.
Number of array element comparisons is the size of the array segment – 1.
Simplify: use the size of the array segment
O(k-h) time

15. Runtime
We show how to do an analysis, assuming n is a power of 2 (just to simplify the calculations)
Use T(n) for number of array element comparisons to mergesort an array segment of size n
public static void mS(Comparable[] b, int h, int k) {
if (h >= k) return;
int e= (h+k)/2;
mS(b, h, e); T(e+1-h) comparisons
mS(b, e+1, k); T(k-e) comparisons
merge(b, h, e, k); (k+1-h) comparisons }
Back to mergeSort:
We now know the number of comparisons merge takes, so to prove the number of comparisons the algorithm takes as a whole, we must figure out how to deal with this recursion.
To this end, we create T().
Thus: T(n) < 2 T(n/2) + n, with T(0) = 0, T(1) = 0

16. Runtime Thus, for any n a power of 2, we have T(1) = 0
T(n) = 2*T(n/2) + n for n > 1
We can prove that
T(n) = n lg n
lg n means log2 n
Here we have a base case (time to sort 1 element is 0), and a recursive case (time to sort n elements is twice the time to sort n/2, plus n more time).
Remember that we’re only using n = powers of 2, for simplicity.

17. Proof by recursion tree of T(n) = n lg n
T(n) = 2*T(n/2) + n, for n > 1, a power of 2, and T(1) = 0
T(n) merge time at level
lg n levels
T(n/2) T(n/2) n = n
T(n/4) T(n/4) T(n/4) T(n/4) (n/2) = n . . .
This is the “recursion tree” for mergeSort. We are going to calculate the time it takes to perform the merges at each “level of recursion.” That is to say, at the highest level, the merge of n elements takes n time. Before that can happen, the algorithm must make two merges of n/2 elements each, which take n/2 comparisons each, and so that’s a total of n time. Before those can happen, they each call a pair of merges of n/4 elements each. That’s 4 merges of n/4 elements each, for a total of n time.
There are lg n levels to this tree, which is to say that the total recursion depth of mergeSort is lg n. Therefore, there are ln n levels, with n time spent on each, or a total of n lg n time spent to execute this algorithm.
T(2) T(2) T(2) T(2) T(2) T(2) T(2) T(2) (n/2)2 = n
Each level requires n comparisons to merge. lg n levels.
Therefore T(n) = n lg n mergeSort has time O(n lg n)

18. MergeSort vs QuickSort
Covered QuickSort in Lecture
MergeSort requires extra space in memory
The way we’ve coded it, it needs that extra array
QuickSort is an “in place” or “in situ” algorithm. No extra
array. But it does require space for stack frame for re- cursive calls. Naïve algorithm: O(n), but can make O(log n)
Both have “average case” O(n lg n) runtime
MergeSort always has O(n lg n) runtime
Quicksort has “worst case” O(n2) runtime
Let’s prove it!
In lecture, we covered a number of sorting algorithms, most notably mergesort and quicksort
Mergesort, as we’ve written it, requires making this extra array ( we called it c) at each step, which eats extra memory
Whereas quicksort, as we covered it in class, required no extra memory. We did it all in place. This is a huge advantage if you’re sorting, say, gigabytes of stuff.
Both algorithms have good runtimes on average, which is dependent upon the data you gave them to sort. However, quicksort has a much worse worst case runtime.

19. Quicksort h j k <= x x >= x Pick some “pivot” value in the array
Partition the array:
Finish with the pivot value at some index j
everything to the left of j ≤ the pivot
everything to the right of j ≥ the pivot
Run QuickSort on b[h..j-1] and b[j+1..k]
This is an outline of how quicksort works. Note that it really depends how you pick your pivot: this is going to determine when we get much worse runtime.

20. Runtime of Quicksort
Base case: array segment of 0 or 1 elements takes no comparisons T(0) = T(1) = 0
Recursion:
partitioning an array segment of n elements takes n comparisons to some pivot
Partition creates length m and r segments (where m + r = n-1)
T(n) = n + T(m) + T(r)
/** Sort b[h..k] */
public static void QS
(int[] b, int h, int k) {
if (k – h < 1) return;
int j= partition(b, h, k);
QS(b, h, j-1);
QS(b, j+1, k); }
Basic runtime analysis of quicksort as coded in lecture. We’re counting comparisons made, again, as the “costly” operation.
Note that the resulting formula we get for T is very similar to the one we saw for mergesort.

21. Runtime of Quicksort T(n) = n + T(m) + T(r) /** Sort b[h..k] */
Look familiar?
If m and r are balanced (m ≈ r ≈ (n-1)/2), we know T(n) = n lg n.
Other extreme:
m=n-1, r=0
T(n) = n + T(n-1) + T(0)
/** Sort b[h..k] */
public static void QS
(int[] b, int h, int k) {
if (k – h < 1) return;
int j= partition(b, h, k);
QS(b, h, j-1);
QS(b, j+1, k); }
Here is where we see why how you pick your pivot matters. We want to make the two array segments that the pivot determines as balanced as possible.

22. Worst Case Runtime of Quicksort
When T(n) = n + T(n-1) + T(0)
Hypothesis: T(n) = (n2 – n)/2
Base Case: T(1) = (12 –1)/2=0
Inductive Hypothesis: assume T(k)=(k2-k)/2 T(k+1) = k + (k2-k)/ = (k2+k)/ = ((k+1)2 –(k+1))/2
Therefore, for all n ≥ 1: T(n) = (n2 – n)/2 = O(n2)
/** Sort b[h..k] */
public static void QS
(int[] b, int h, int k) {
if (k – h < 1) return;
int j= partition(b, h, k);
QS(b, h, j-1);
QS(b, j+1, k); }
When the two pivot segments are really imbalanced, we get the n^2 runtime. This is a basic inductive proof of this. We don’t even need to assume here that n is anything but a positive integer. Go through this proof slowly. The form of this style of proof is a large part of what we’re trying to teach here.

23. Worst Case Space of Quicksort
You can see that in the worst case, the depth of recursion is O(n). Since each recursive call involves creating a new stack frame, which takes space, in the worst case, Quicksort takes space O(n). That is not good! To get around this, rewrite QuickSort so that it is iterative but it sorts the smaller of two segments recursively. It is easy to do. The implementation in the java class that is on the website shows this.
/** Sort b[h..k] */
public static void QS
(int[] b, int h, int k) {
if (k – h < 1) return;
int j= partition(b, h, k);
QS(b, h, j-1);
QS(b, j+1, k); }