CS421 - Course Information Website Syllabus Schedule The Book:

What is an algorithm? Informally, an algorithm is any well-defined computational procedure that takes some value(s) as input and produces some value(s) as output. Goals for an algorithm: 1.Correct 2.Terminates 3.Efficient

What is CS421 about? We will engage in the theoretical study of the design and analysis of computer algorithms. –Analysis: predict the cost of an algorithm in terms of performance and resources used –Design: design algorithms which minimize such costs

Machine Model Assumptions.. Random Access Machine (RAM) Model: 1.Any memory cell can be accessed in 1 step. 2.Memory is not limited (unbounded). 3.Arbitrarily large integers can be stored in each memory cell. 4.Operations are executed sequentially. 5.Operators include: primitive arithmetic (+, -, *, /, modulo, etc..) logic (if..then, and, or, etc..) comparators (, =, etc..) and function calls. 6.Each operation has a unit cost of 1.

An Example.. Sorting Input: A sequence of n numbers,  a 1, a 2, …, a n  Output: A permutation (reordering),  a' 1, a' 2, …, a' n , of the input sequence such that a' 1  a' 2  …  a' n. Example: Input: Output:

Insertion Sort Pseudocode: I NSERTION -S ORT (A) 1for j ← 2 to length[A] 2dokey ← A[ j] 3i ← j – 1 4while i > 0 and A[i] > key 5doA[i+1] ← A[i] 6i ← i – 1 7A[i+1] = key

Example of insertion sort

824936

end

How “Good” is Insertion Sort? Recall our goals for an algorithm: 1.Correct 2.Terminates 3.Efficient

Correctness Informally: At each step the “current card”, j, is inserted into an already sorted subarray A[1.. j-1]. More formally: The loop invariant (a condition that does not change if correct) is that at the start of each for-loop, the subarray A[1.. j-1] consists of the elements in A[1.. j-1] but in sorted order.

Correctness These properties must hold for a loop invariant: Initialization: It is true prior to the first iteration. Maintenance: It is true before an iteration of the loop and remains true before the next iteration. Termination: When the loop terminates, the invariant yields a useful property which helps show the algorithm is correct.

Correctness In the case of insertion sort: Initialization: If j=2 at initialization, A[1..j-1] consists of a single element which is by definition sorted. Maintenance: Informally, the for-loop works by moving A[j-1], A[j-2], etc. one position to the right until the correct position for A[j] is found. Since A[1..j] is now sorted, when j is incremented, A[1..j-1] is sorted. Termination: The loop terminates when j=n+1. From the Maintenance property we know A[1..j-1] is now sorted. That is A[1..n] is now sorted. Hence the algorithm is correct.

Termination As we will see later in the semester determining if an arbitrary program terminates or halts is undecidable for Turing machines. In the specific instance of Insertion-Sort, it is easy to see that the two loops iterate at most n times each and thus the algorithm does not run indefinitely and does terminate.

Efficiency Running time is a measure of how many steps or primitive operations were performed. We have already stated that in the RAM machine model each operator has cost 1, but lets assume instead each operator has cost c i, where i is a line in our algorithm.

Kinds of Analysis Worst-case: T(n) = maximum time of algorithm on any input of size n. Average-case: T(n) = expected time of algorithm over all inputs of size n. –Requires assumption of statistical distribution of inputs. Best-case: T(n) = minimum time of algorithm on any input of size n. –Problematic because a generally slow algorithm may works fast on some input.

Running Time Lets analyze our algorithm once more.. I NSERTION -S ORT (A)costtimes 1for j ← 2 to length[A]c 1 n 2dokey ← A[ j]c 2 n-1 3i ← j – 1c 3 n-1 4while i > 0 and A[i] > keyc 4 ∑ j=2..n t j 5doA[i+1] ← A[i]c 5 ∑ j=2..n (t j -1) 6i ← i – 1c 6 ∑ j=2..n (t j -1) 7A[i+1] = keyc 7 n-1

Best Case T(n) = (c1+c2+c3+c4+c7)n – (c2+c3+c4+c7) I NSERTION -S ORT (A)costtimes 1for j ← 2 to length[A]c 1 n 2dokey ← A[ j]c 2 n-1 3i ← j – 1c 3 n-1 4while i > 0 and A[i] > keyc 4 ∑ j=2..n t j 5doA[i+1] ← A[i]c 5 ∑ j=2..n (t j -1) 6i ← i – 1c 6 ∑ j=2..n (t j -1) 7A[i+1] = keyc 7 n-1

Worst Case T(n) = (c 4 +c 5 +c 6 )n 2 /2 + (c 1 +c 2 +c 3 +c 4 /2-c 5 /2- c 6 /2c 7 )n – (c 2 + c 3 + c 4 + c 7 ) I NSERTION -S ORT (A)costtimes 1for j ← 2 to length[A]c 1 n 2dokey ← A[ j]c 2 n-1 3i ← j – 1c 3 n-1 4while i > 0 and A[i] > keyc 4 ∑ j=2..n t j 5doA[i+1] ← A[i]c 5 ∑ j=2..n (t j -1) 6i ← i – 1c 6 ∑ j=2..n (t j -1) 7A[i+1] = keyc 7 n-1

Average Case t j = j/2.. Only out of order half the time.. T(n) = (c 4 +c 5 +c 6 )n 2 /4 + (c 1 +c 2 +c 3 +c 4 /4-c 5 /4- c 6 /4c 7 )n – (c 2 + c 3 + c 4 + c 7 ) I NSERTION -S ORT (A)costtimes 1for j ← 2 to length[A]c 1 n 2dokey ← A[ j]c 2 n-1 3i ← j – 1c 3 n-1 4while i > 0 and A[i] > keyc 4 ∑ j=2..n t j 5doA[i+1] ← A[i]c 5 ∑ j=2..n (t j -1) 6i ← i – 1c 6 ∑ j=2..n (t j -1) 7A[i+1] = keyc 7 n-1

Machine Independent Analysis As the size of the input becomes large, the constants c i don’t matter as much as the exponents and log factors. The constants also make machine independent analysis impossible. –Ignore the constants. –Examine growth of T(n) as n → ∞. –Asymptotic Analysis

Order of Growth The rate of growth is of primary interest, so we consider only the leading term and ignore all constants (e.g. n^2) Thus, the worst case running time of Insertion Sort is Θ(n 2 ). Quadratic time. We will define this more precisely later.

Design Approach: Divide and Conquer Divide the problem into a number of subproblems. Conquer the subproblems recursively. Combine the subproblem solutions into the solution for the original problem. Recursion: when an algorithm calls itself.

Merge Sort Divide: Divide an n-element array into two subsequences of n/2 elements each. Conquer: Sort the two subsequences recursively with merge sort. Combine: Merge the two sorted arrays to produce the sorted sequence. Special Case: If the sequence has only one element the recursion “bottoms out” as the sequence is sorted by definition.

Merge Sort M ERGE -S ORT ( A[1.. n] ) 1.If n = 1, return A. 2.L = A[ 1..  n/2  ] 3.R = A[  n/2  +1.. n ] 4.L = Merge-Sort(L) 5.R = Merge-Sort(R) 6.Return Merge(L, R)

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays Time =  (n) to merge a total of n elements (linear time).

Analyzing merge sort M ERGE -S ORT A[1.. n] 1.If n = 1, done. 2.Recursively sort A[ 1..  n/2  ] and A[  n/2  +1.. n ]. 3.“Merge” the 2 sorted lists T(n)  (1) 2T(n/2)  (n) Sloppiness: Should be T(  n/2  ) + T(  n/2  ), but it turns out not to matter asymptotically.

Recurrence for merge sort T(n) =  (1) if n = 1; 2T(n/2) +  (n) if n > 1. We shall usually omit stating the base case when T(n) =  (1) for sufficiently small n, but only when it has no effect on the asymptotic solution to the recurrence.

Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.

Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. T(n)T(n)

Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. T(n/2) cn

Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn T(n/4) cn/2

Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/4 cn/2  (1) …

Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/4 cn/2  (1) … h = lg n

Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/4 cn/2  (1) … h = lg n cn

Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/4 cn/2  (1) … h = lg n cn

Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/4 cn/2  (1) … h = lg n cn …

Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/4 cn/2  (1) … h = lg n cn #leaves = n (n)(n) …

Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/4 cn/2  (1) … h = lg n cn #leaves = n (n)(n) Total  (n lg n) …

Conclusions  (n lg n) grows more slowly than  (n 2 ). Therefore, merge sort asymptotically beats insertion sort in the worst case.