Lecture 4. Paradigm #2 Recursion Last time we discussed Fibonacci numbers F(n), and Alg fib(n) if (n <= 1) then return(n) else return(fib(n-1)+fib(n-2)) The problem with this algorithm is that it is woefully inefficient. Let T(n) denote the number of steps needed by fib(n). Then T(0) = 1, T(1) = 1, and T(n) = T(n-1) + T(n-2) + 1. It is now easy to guess the solution T(n) = 2 F(n+1) - 1 Proof by induction: Clearly the claim is true for n = 0, 1. Now assume it is true for all n < N; we prove it for n = N: T(N) = T(N-1) + T(N-2) + 1 = (2 F(N) - 1) + (2 F(N-1) - 1) + 1 = 2(F(N)+F(N-1)) - 1 = 2 F(N+1) - 1 We know F(n) = Θ(a n ), a=(1+√5)/2

Memoization A trick called "memoization“ can help recursion, store function values as they are computed. When the function is invoked, check the argument to see if the function value is already known; that is, don't recompute. Some programming languages even offer memoization as a built-in option. Here are two Maple programs for computing Fibonacci numbers; the first just uses the ordinary recursive method, and the second uses memoization (through the command "option remember"). A recursive one: f := proc(n) if n ≤ 1 then n else f(n - 1) + f(n - 2) end if end proc; One that looks recursive, but that uses memoization: g := proc(n) option remember; if n ≤ 1 then n else g(n - 1) + g(n - 2) end if end proc; When you run these and time them, you'll see the amazing difference in the running times. The "memoized" version runs in linear time.

Printing permutations Sometimes, recursion does provide simple & efficient solutions. Consider the problem of printing out all permutations of {1,2,...,n} in lexicographic order. Why might you want to do this? Well, some combinatorial problems have no known efficient solution (such as traveling salesman), and if you want to be absolutely sure you've covered all the possibilities for some small case. How can we do this? If we output a 1 followed by all possible permutations of the elements other than 1; then a 2 followed by all possible permutations of the elements other than 2; then a 3 followed by.... etc., we'll have covered all the cases exactly once.

Printing all permutations … So we might try a recursive algorithm. What should the input parameter be? If we just say n, with the intention that this gives all permutations of {1,2,..., n} that's not going to be good enough, since later we will be permuting some arbitrary subset of this. So you might think that the input parameters should be an arbitrary set S. But even this is not quite enough, since we will have to choose an arbitary element i out of S, and then print i followed by all the permutations of S - { i }. But if we don't want to store all the permutations of S - { i } before we output them we need some way to tell the program that when it goes and prints all the permutations of S - { i }, it should print i first, preceding each one. This suggests making a program with two parameters: one will be the fixed "prefix" of numbers that is printed out, and the second the set of remaining numbers to be permuted.

Printing permutations printperm(P,S) /* P is a prefix, S is a nonempty set */ if S={x} then print Px; else for each element i of S do printperm( (P,i), S - { i }); There are n! permutations. So any program must spend O(n*n!) time to print all the permutations, each taking n steps. Let me give a simple amortizing counting argument. We will be printing n*n! symbols. Each time the "else" statement is executed, we charge O(1) to that "i". This particular "i" in the n*n! symbols (note i appears in different permutations many times, but we are just referring to one instance of such "i") gets charged only once. Summing up, the total time is O(n · n!).

Paradigm #3: Divide-and-conquer Divide et impera [Divide and rule] -- Ancient political maxim cited by Machiavelli -- Julius Caesar (102-44BC) The paradigm of divide-and-conquer: -- DIVIDE problem up into smaller problems -- CONQUER by solving each subproblem -- COMBINE results together to solve the original problem

Divide & Conquer: MergeSort Example: merge sort (an O(n log n) algorithm for sorting) (See CLR, pp ) MERGE-SORT(A, p, r) /* A is an array to be sorted. This algorithm sorts the elements in the subarray A[p..r] */ if p < r then q := floor( (p+r)/2 ) MERGE-SORT(A, p, q) MERGE-SORT(A, q+1, r) MERGE(A, p, q, r)

MergeSort continues.. Let T(n) denote the number of comparisons performed by algorithm MERGE-SORT on an input of size n. Then we have T(n) = 2T(n/2) + n expanding … = 2 k T(n/2 k ) + kn …. = O(nlogn) --- when k=logn.

Another way: Prove T(2 k ) = (k+1)2 k by induction: It is true for k = 0. Now assume it is true for k; we will prove it for k+1. We have T(2 k+1 ) = 2T(2 k ) + 2 k+1 (by recursion) = 2(k+1)2 k + 2 k+1 (by induction) = (k+2) 2 k+1, and this proves the result by induction.

Divide & conquer: multiply 2 numbers Direct multiplication of 2 n-bit numbers takes n 2 steps. Note, we assume n is very large, and each register can hold only O(1) bits. When do we need to multiply two very large numbers? In Cryptography and Network Security message as numbers encryption and decryption need to multiply numbers My comment: but really: none of above seems to be a good enough reason. Even you wish to multiply a number of 1000 digits, an O(n 2 ) alg. is good enough!

How to multiply 2 n-bit numbers ************ ************************

History: 1960, AN Kolmogorov (we will meet him again later in this class) organized a seminar on mathematical problems in cybernetics at MSU. He conjectured Ω(n 2 ) lower bound for multiplication and other problems. Karatsuba, then 25 year old student, proposed the n log3 solution in a week, by divide and conquer. Kolmogorov was upset, he discussed this result in the next seminar, which was then terminated. The paper was written up by Kolmogorov, but authored by Karatsuba (who did not know before he received reprints) and was published in Sov Phys. Dol. AN Kolmogorov

Can we multiply 2 numbers faster? Karatsuba's 1962 algorithm for multiplying two n bit numbers in O(n 1.59 ) steps. Suppose we wish to multiply two n-bit numbers X Y. Let X = ab, Y = cd where a, b, c, d are n/2 bit numbers. Then XY = (a · 2 n/2 + b)(c · 2 n/2 + d) = (ac)2 n + (ad + bc)2 n/2 + bd cd ab X = Y =

Multiplying 2 numbers So we have broken the problem up to 4 subproblems each of size n/2. Thus, T(2 k ) = 4T(2 k-1 ) + c 2 k 4T(2 k-1 ) = 16 T(2 k-2 ) + 4c2 k-1 = 4 2 T(2 k-2 )+c2 k k-1 T(2) = 4 k T(1) + 4 k-1 · c · 2 Now T(1) = 1, so T(2 k ) = 4 k + c(2 k + 2 k k-1 ) ≤ 4 k + c 4 k = (4 k )(c+1). This gives T(n) = O(n 2 )! No improvement!

Multiplying 2 numbers But Karatsuba did not give up. He observed: XY = (2 n + 2 n/2 )· ac + 2 n/2 · (a-b) · (d-c) + (2 n/2 + 1)· bd Now, we have broken the problem up into only 3 subproblems, each of size n/2, plus some linear work. This time it should work! K(n) ≤ 3K(n/2) + cn ≤ c(3 k k+1 ) Putting n = 2k, we see that for n a power of 2, we get K(n) ≤ c(3 lg n + 1 ) – 2 lg n + 1 ) = c(3 n lg n) Here we have used the fact that a log(b) = b log(a). Since lg 3 is about , this gives us a O(n 1.59 ) algorithm Note: Using FFT. Schonhage and Strassen: O(nlogn loglogn) in In 2007, this was slightly improved by Martin Furer

Divide & conquer: finding max-min Problem: finding both the maximum and minimum of a set of n numbers. Obvious method: first compute the maximum, using n-1 comparisons; then discard this maximum, and compute the minimum of the remaining numbers, using n-2 comparisons. Total work: 2n-3 comparisons.

Maxmin by divide & conquer MAXMIN(S) /* find both the maximum and minimum elements of a set S */ if S = {a} then min=a, max=a; else if S = {a < b} min=a, max=b; else /* |S| > 2 */ divide S into 2 subsets, S1 and S2, such that S1 has floor(n/2) elements and S2 has ceil(n/2) elements (min1, max1) := MAXMIN(S1); (min2, max2) := MAXMIN(S2); min = min(min1, min2); max = max(max1, max2); return (max,min);

Time complexity T(n) = 1 when n =2 T(n) = 2T(n/2) + 2, otherwise. This gives T(n) = 3n/2 – 2, when n is a power of 2.