CMPT 438 Algorithms

Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data, do something that would otherwise be impossible ▫Solve problems of significantly large size ▫Technology only improves things by a constant factor Compare algorithms Algorithms as a field of study ▫Learn about a standard set of algorithms ▫New discoveries arise ▫Numerous application areas Learn techniques of algorithm design and analysis

What are Algorithms? An algorithm is a sequence of computational steps that transform the input into the output. An algorithm is also a tool for solving a well- specified computational problem. ▫E.g., sorting problem: ▫ is an instance of the sorting problem.

An algorithm is correct if, for every input instance, it halts with the correct output.

Analyzing Algorithms Predict the amount of resources required: ▫memory: how much space is needed? ▫computational time: how fast the algorithm runs? FACT: running time grows with the size of the input Input size (number of elements in the input) ▫Size of an array, # of elements in a matrix, # of bits in the binary representation of the input, vertices and edges in a graph Def: Running time = the number of primitive operations (steps) executed before termination ▫Arithmetic operations (+, -, *), data movement, control, decision making (if, while), comparison

Algorithm Efficiency vs. Speed E.g.: sorting n numbers (n = 10 6 ) ▫Friend’s computer = 10 9 instructions/second ▫Friend’s algorithm = 2n 2 instructions (insertion sort) ▫Your computer = 10 7 instructions/second ▫Your algorithm = 50nlgn instructions (merge sort)

Algorithm Efficiency vs. Speed To sort 100 million numbers: Insertion sort takes more than 23 days Merge sort takes under 4 hours

Typical Running Time Functions 1 (constant running time): ▫Instructions are executed once or a few times logN (logarithmic) ▫A big problem is solved by cutting the original problem in smaller sizes, by a constant fraction at each step N (linear) ▫A small amount of processing is done on each input element N logN ▫A problem is solved by dividing it into smaller problems, solving them independently and combining the solution

Typical Running Time Functions N 2 (quadratic) ▫Typical for algorithms that process all pairs of data items (double nested loops) N 3 (cubic) ▫Processing of triples of data (triple nested loops) N K (polynomial) 2 N (exponential) ▫Few exponential algorithms are appropriate for practical use

Why Faster Algorithms?

Insertion Sort Idea: like sorting a hand of playing cards ▫Remove one card at a time from the table, and insert it into the correct position in the left hand  compare it with each of the cards already in the hand, from right to left

Example of insertion sort

INSERTION-SORT I NSERTION -S ORT (A, n) ⊳ A[1.. n] for j ← 2 to n do key ← A[ j] i ← j –1 while i > 0 and A[i] > key do A[i+1] ← A[i] i ← i –1 A[i+1] = key Insertion sort sorts the elements in place. i A: sorted key nj 1

Analysis of Insertion Sort

Running time Parameterize the running time by the size of the input, since short sequences are easier to sort than long ones.

Kinds of analyses Worst-case: T(n) =maximum time of algorithm on any input of size n. Average-case: T(n) =expected time of algorithm over all inputs of size n. Need assumption of statistical distribution of inputs. Best-case: Cheat with a slow algorithm that works fast on some input.

“Asymptotic Analysis” Machine-independent time What is insertion sort’s worst-case time? It depends on the speed of our computer BIG IDEA: Ignore machine-dependent constants. Look at growth of T(n) as n → ∞.

Θ-notation Math: Θ(g(n)) = { f (n): there exist positive constants c 1, c 2, and n 0 such that 0 ≤c 1 g(n) ≤f (n) ≤c 2 g(n) for all n≥n 0 } Engineering: Drop low-order terms; ignore leading constants. Example: 3n n 2 –5n = Θ(n 3 )

Best Case Analysis The array is already sorted ▫A[i] ≤ key upon the first time the while loop test is run (when i = j -1) ▫tj = 1

Worst Case Analysis The array is in reverse sorted order ▫Always A[i] > key in while loop test ▫Have to compare key with all elements to the left of the j-th position ▫compare with j-1 elements ▫tj = j

Average Case? All permutations equally likely.

Insertion Sort Summary Advantages ▫Good running time for “almost sorted” arrays θ(n) Disadvantages ▫θ(n 2 ) running time in worst and average case Is insertion sort a fast sorting algorithm? Moderately so, for small n. Not at all, for large n.

Worst-Case and Average-Case We usually concentrate on finding only the worst-case running time ▫an upper bound on the running time ▫For some algorithms, the worst case occurs often.  E.g., searching when information is not present in the DB ▫The average case is often as bad as the worst case.

Merge Sort

MERGE-SORT A[1.. n] 1.If n= 1, done. 2.Recursively sort A[ 1...n/2]and A[ [n/2]+1.. n ]. 3.“Merge” the 2 sorted lists.

Example

Divide-and-Conquer Divide the problem into a number of subproblems ▫Similar sub-problems of smaller size Conquer the sub-problems ▫Solve the sub-problems recursively ▫Sub-problem size small enough to solve the problems in straightforward manner Combine the solutions to the sub-problems ▫Obtain the solution for the original problem

Merge Sort Approach To sort an array A[p.. r]: Divide ▫Divide the n-element sequence to be sorted into two subsequences of n/2 elements each Conquer ▫Sort the subsequences recursively using merge sort ▫When the size of the sequences is 1 there is nothing more to do Combine ▫Merge the two sorted subsequences

Merge sort

Analyzing merge sort MERGE-SORT A[1.. n] 1.If n= 1, done. 2.Recursively sort A[ 1.. 「 n/2 」 ] and A[ 「 n/2 」 +1.. n ]. 3.“Merge”the 2sorted lists Sloppiness: Should be T( 「 n/2 」 ) + T( 「 n/2 」 ), but it turns out not to matter asymptotically. T(n) Θ(1) 2T(n/2) ?

Merging two sorted arrays

Merging two sorted arrays Time = Θ(n) to merge a total of n elements (linear time).

In place sort? Run time?

Analyzing Divide and Conquer Algorithms The recurrence is based on the three steps of the paradigm: ▫T(n) – running time on a problem of size n ▫Divide the problem into a subproblems, each of size n/b: takes D(n) ▫Conquer (solve) the subproblems: takes aT(n/b) ▫Combine the solutions: takes C(n) T(n) = aT(n/b) + D(n) + C(n) otherwise

MERGE – SORT Running Time Divide: ▫compute q as the average of p and r: D(n) = θ(1) Conquer: ▫recursively solve 2 subproblems, each of size n/2 -> 2T (n/2) Combine: ▫MERGE on an n-element subarray takes θ(n) time C(n) = θ(n) T(n) = 2T(n/2) + θ(n) if n > 1

Recurrence for merge sort We shall usually omit stating the base case when T(n) = Θ(1) for sufficiently small n, but only when it has no effect on the asymptotic solution to the recurrence. Θ(1) if n= 1; 2T(n/2)+ Θ(n) if n> 1. T(n) =

Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.

Recursion tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant. cn cn/2 cn/2 cn/4 cn/4 cn/4 cn/4 Θ(1) ． ． ．． ． ． h= lgn cn ． ． ．． ． ． #leaves = n Θ(n) Total= Θ( n lg n)

Conclusions Θ(n lg n) grows more slowly than Θ(n 2 ). Therefore, merge sort asymptotically beats insertion sort in the worst case. Disadvantage ▫Requires extra space Θ (n) In practice, merge sort beats insertion sort for n> 30 or so.

Divide-and-Conquer Example: Binary Search Find an element in a sorted array: 1. Divide: Check middle element. 2. Conquer: Recursively search 1 subarray. 3. Combine: Trivial. A[8] = {1, 2, 3, 4, 5, 7, 9, 11} Find 7

Divide-and-Conquer Example: Binary Search For an ordered array A, finds if x is in the array A[lo…hi]

Example A[8] = {1, 2, 3, 4, 5, 7, 9, 11} lo = 1 hi = 8 x = 6

Analysis of Binary Search ?

Divide-and-Conquer Example: Powering a Number ? ? ?

Readings Chapter 3 Appendix A Homework 1 Quiz