Algorithms: the big picture Algorithm is sequence of operations that transforms input to output Example: Sort N items input a sequence (a1, a2, … aN) output a permutation of input sequence that is sorted Most often items to be sorted do not have a numerical value example: medical records define keys that point to satellite data sort keys

Algorithms begin with an idea Example: Insertion sort In partially sorted array, find where next item belongs Create storage allocation and insert the item Repeat until all items are sorted Important questions about this idea: will it work? how much storage is required? how fast does it run? To get answers to these questions, we need a pseudocode.

To write a pseudocode, a picture may be helpful: Insertion sort on 6 items. Sorts in place beginning at left end. 1st item is sorted by default. Next item to right of partially sorted array is placed in temporary storage. Leaves room to shift sorted items to right to open up the correct place to insert the item being sorted

Note: indents show the structure of pseudocode Worst case tj = j

Line-by-line accounting of operations Note: loop failure is counted as operation i.e. # of times an item in the sorted part has to be moved to open the correct place for the item being sorted

Worst case analysis: tj = j In worst-case, every element in A[1…j-1] sorted part of the array must be moved to the right With tj = j sums can be evaluated because they are related to the “arithmetic” sum

Order of growth Our primary interest is “How does runtime change with increasing input size?” Linear: T(N) = a + bN Quadratic : T(N) = a + bN + cN2 Logarithmic: T(N) = aN(log(bN)) The coefficients in these expressions are hard to calculate and probably depend on properties of the input other than size. To avoid this difficulty, we use “order of growth” Linear: T(N) = order(N) Quadratic : T(N) = order(N2) Logarithmic: T(N) = order(N(log(N))) Order of growth concept is expanded in Chapter 3

Prove by induction on integers that Structured proof: Base case: for some value of n (n=1 in case above) what is the RHS equal to? what is the LHS equal to? are they the same? Setup equation that reflects the strategy for proof: In this class we will use “if S(n-1) then S(n)” Inductive hypothesis (IH): an assumption that reflects the strategy and can be used on RHS of your setup equation. Application of IH: equations with S(n) on the LHS and all algebraic manipulations on the RHS.

I.H. Proof that Setup to use “if S(n-1) then S(n)” example of induction on integers using if S(n-1) then S(n) I.H.

Assignment 1: due 1/9/19 Geometric sum: Prove by induction on integers that Give a structured proof using the technique if S(n-1) then S(n). Include the following: base case setup inductive hypothesis application of inductive hypothesis with all algebra on RHS of equals

Review: Worst case analysis of insertion sort: tj = j including the test that fails With tj = j sums can be evaluated by relating them to the arithmetic sum

Use the arithmetic sum to evaluate the sums in the analysis of insertion sort runtime (n-1)n/2

We find the polynomial dependence of runtime on input size in worst case

Line-by-line analysis of insertion sort: worst case Collect terms: T(n) = a + bn + cn2 This is an upper bound that has the possibility of being equal to the runtime In notation of Chapter 3, T(n) = O(n2)

Worst case: T(n) = a + bn + cn2 = order of growth = n2 In asymptotic notation T(n) = O(n2) All quadratic terms come from analysis of “while” statement Was algebra really necessary? Worst case tj = j

Loop invariant and correctness Loop invariant = statement about iterative pseudo-codes Prove that the statement is true by induction on integers Initialization: true on 1st iteration (base case) Maintenance: if true on ith iteration, also true on i+1 (if S(n) then S(n+1)) At termination: truth of loop invariant shows that the algorithm is correct

Application of loop invariant to insertion sort Loop invariant: At the start of the jth iteration of for-loop 1-8, A(1…j-1) is sorted Initialization: On the 1st iteration j=2. A(1…j-1) = A(1) is sorted by default. Maintenance: By I.H. at start of jth iteration A(1…j-1) is sorted. On the jth iteration A(j) is inserted into A(1…j-1) at its proper order; hence at the start of j+1 iteration A(1…j) is sorted At termination: j = n+1; therefore A(1…j-1) = A(1…n). Whole array is sorted.

Merge sort: A new idea about sorting

Recursive sorting algorithm Divide-and-conquer phase: Recursively divide the problem into smaller pieces until you can get a solution by default. Push merge instructions onto stack Execution phase: Given default solution, assemble the full solution from successively larger pieces using merge instructions on the stack. On 1st call to Merge-Sort p = 1, r = n On recursive calls, the these dummy variables have different values Eventually, the test if p < r will fail and stacked instructions to merge sorted subarrays will execute. Example of execution phase for input (5, 2, 4, 7, 1, 3, 2, 6)

For recursive algorithms, analysis of runtime begins with a recurrence relation that describes divide and conquer T(n) = mT(n/k) + “overhead” Overhead includes everything not involved in solving m problems of size n/k For Merge-Sort, m = k = 2 and “overhead” is cost of merging

With an even number of items, Merge-Sort is described by the recurrence relation T(2k) = 2T(2k-1) + cost of merging To evaluate “cost of merging” we need a pseudocode for Merge(A,p,q,r), where sorted subarrays A(p…q) and A(q+1…r) are merged in sorted order

Cost of merging 2 sorted arrays containing a total of n elements Heuristic analysis: Consider 2 card decks sorted in increasing order and turned face-up decks not necessarily of equal size Compare the value of cards showing Place card of smaller value face down on the 3rd stack If one deck is empty, place all of the other deck on the 3rd stack How many comparison must be made? How many comparison must be made, worst case? Why is it important that the decks to be merged are sorted?

Use of “sentinel” cards Merge(A,p,q,r) uses the fact that subarrays A(p…q) and A(q+1…r) are sorted. All of the elements of A(p…q) and A(q+1…r) that have not been merged are larger than those in the merged array. The possibility exists that all of the elements of A(p…q) have been merged while some elements remain in A(q+1…r), or vice versa. Sentinel cards, with infinite value, are a way to ensure that this does not happen.

Pseudocode No nested loops. Cost is linear in total number of items to be merged = r – p +1

Sentinel cards make algorithm simpler but maybe slower

A linear Merge algoarithm is sufficient to make Merge-Sort asymptotically optimal. Recurrence for runtime becomes T(2k) = 2T(2k-1) + c2k for an even number of items. In general, T(n)=T(floor(n/2)) + T(ceiling(n/2)) + cn In chapter 4 we show that this recurrence has order of growth nlog(n) In chapter 8 we show that order(nlog(n)) is an asymptotic lower bound on runtime of sorting algorithm based on comparison of value.

 Assignment 2: due 1/11/19 Prove by induction on integers that recurrence T(2k) = 2 if k=1 T(2k) = 2T(2k-1) + 2k if k>1 has solution T(2k) = k2k Is this solution T(n)=nlg(n)? Why does the recursion have 2 parts? What are the base cases? What is the “setup” for if S(n-1) then S(n)? What is the inductive hypothesis? What is the application of the inductive hypothesis?

Prove by iteration that recurrence T(2k) = 2 if k=1 T(2k) = 2T(2k-1) + 2k if k>1 has solution T(2k) = k2k