Grade School Revisited: How To Multiply Two Numbers CS Lecture 4 2 X 2 = 5

At the end of last lecture: Remember how to multiply 2 complex numbers a + bi and c + di? (a+bi)(c+di) = [ac –bd] + [ad + bc] i Input: a,b,c,d Output: ac-bd, ad+bc If a real multiplication costs $1 and an addition cost a penny. What is the cheapest way to obtain the output from the input? Can you do better than $4.02?

Gauss’ $3.05 Method: Input: a,b,c,d Output: ac-bd, bc+ad X 1 = a + b X 2 = c + d X 3 = X 1 X 2 = ac + ad + bc + bd X 4 = ac X 5 = bd X 6 = X 4 – X 5 = ac-bd X 7 = X 3 – X 4 – X 5 = bc + ad

Question: The Gauss “hack” saves one multiplication out of four. It requires 25% less work. Could there be a context where performing 3 multiplications for every 4 provides a more dramatic savings?

Odette Bonzo

How to add 2 n-bit numbers. ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** +

*** *** ** ** ** ** ** ** ** ** ** ** *** *** ** ** ** ** ** ** ** ** +

*** *** ** ** ** ** ** ** ** ** *** *** **** **** ** ** ** ** ** ** ** ** +

*** *** ** ** ** ** ** ** *** *** **** **** **** **** ** ** ** ** ** ** ** ** +

*** *** ** ** ** ** *** *** **** **** **** **** **** **** ** ** ** ** ** ** ** ** +

*** *** **** **** **** **** **** **** **** **** **** **** **** **** **** **** **** **** **** **** * * * * + * ** *

Time complexity of grade school addition *** *** **** **** **** **** **** **** **** **** **** **** **** **** **** **** **** **** **** **** + * ** * **** **** T(n) = The amount of time grade school addition uses to add two n-bit numbers What do you mean by “time”?

Time can depend on specifics A given algorithm will take different amounts of time on the same inputs depending on such factors as: –Processor speed –Instruction set –Disk speed –Brand of compiler

Our Goal We want to define TIME in a sense that transcends implementation details and allows us to make assertions about grade school addition in a very general way.

Hold on! You just admitted that it makes no sense to measure the time, T(n), taken by the method of grade school addition since the time depends on the implementation details. We will have to speak of the time taken by a particular implementation, as opposed to the time taken by the method in the abstract.

Don’t jump to conclusions! Your objections are serious, but not insurmountable. There is a very nice sense in which we can analyze grade school addition without ever having to worry about implementation details. Here is how it works...

On any reasonable computer adding 3 bits and writing down the two bit answer can be done in constant time. Pick any particular computer A and define c to be the time it takes to perform on that computer. Total time to add two n-bit numbers using grade school addition: cn [c time for each of n columns]

Implemented on another computer B the running time will be c’n where c’ is the time it takes to perform on that computer. Total time to add two n-bit numbers using grade school addition: c’n [c’ time for each of n columns]

The fact that we get a line is invariant under changes of implementations. Different machines result in different slopes, but time grows linearly as input size increases. # of bits in numbers timetime Machine A: cn Machine B: c’n

Thus we arrive at an implementation independent insight: Grade School Addition is a linear time algorithm. Determining the growth rate of the resource curve as the problem size increases is one of the fundamental ideas of computer science.

Is there a faster way to add? QUESTION: Is there an algorithm to add two n-bit numbers whose time grows sub-linearly in n?

Any algorithm for addition must read all of the input bits Suppose there is a mystery algorithm that does not examine each bit Give the algorithm a pair of numbers. There must be some unexamined bit position i in one of the numbers If the algorithm is not correct on the numbers, we found a bug If the algorithm is correct, flip the bit at position i and give the algorithm the new pair of numbers. It give the same answer as before so it must be wrong since the sum has changed

So any algorithm for addition must use time at least linear in the size of the numbers. Grade school addition is essentially as good as it can be.

How to multiply 2 n-bit numbers. X * * * * * * * * * * * * * * * * * * * * * * * * * * * * n2n2

I get it! The total time is bounded by cn 2.

Grade School Addition: Linear time Grade School Multiplication: Quadratic time No matter how dramatic the difference in the constants the quadratic curve will eventually dominate the linear curve # of bits in numbers timetime

Neat! We have demonstrated that as things scale multiplication is a harder problem than addition. Mathematical confirmation of our common sense.

Don’t jump to conclusions! We have argued that grade school multiplication uses more time than grade school addition. This is a comparison of the complexity of two algorithms. To argue that multiplication is an inherently harder problem than addition we would have to show that no possible multiplication algorithm runs in linear time.

Useful notation to discuss growth rates For any monotonic function f from the positive integers to the positive integers, we say “f = O(n)” or “f is O(n)” if: Some constant times n eventually dominates f [There exists a constant c such that for all sufficiently large n: f(n) <= cn ]

# of bits in numbers timetime f = O(n) means that there is a line that can be drawn that stays above f from some point on

Useful notation to discuss growth rates For any monotonic function f from the positive integers to the positive integers, we say “f =  (n)” or “f is  (n)” if: f eventually dominates some constant times n [There exists a constant c such that for all sufficiently large n: f(n) >= cn ]

# of bits in numbers timetime f =  (n) means that there is a line that can be drawn that stays below f from some point on

Useful notation to discuss growth rates For any monotonic function f from the positive integers to the positive integers, we say “f =  (n)” or “f is  (n)” if: f = O(n) and f =  (n)

# of bits in numbers timetime f =  (n) means that f can be sandwiched between two lines

Useful notation to discuss growth rates For any monotonic functions f and g from the positive integers to the positive integers, we say “f = O(g)” or “f is O(g)” if: Some constant times g eventually dominates f [There exists a constant c such that for all sufficiently large n: f(n) <= cg(n) ]

Useful notation to discuss growth rates For any monotonic functions f and g from the positive integers to the positive integers, we say “f =  (g)” or “f is  (g)” if: f eventually dominates some constant times g [There exists a constant c such that for all sufficiently large n: f(n) >= cg(n) ]

Useful notation to discuss growth rates For any monotonic functions f and g from the positive integers to the positive integers, we say “f =  (g)” or “f is  (g)” if: f = O(g) and f =  (g)

Quickies n = O(n 2 ) ? –YES n = O(sqrt(n)) ? –NO 3n 2 + 4n +  = O(n 2 ) ? –YES 3n 2 + 4n +  =  (n 2 ) ? –YES n 2 =  (nlogn) ? –YES n 2 logn =  (n 2 ) –NO

Grade School Addition:  (n) time Grade School Multiplication:  (n 2 ) time Is there a clever algorithm to multiply two numbers in linear time?

Despite years of research, no one knows! If you resolve this question, Carnegie Mellon will give you a PhD!

Is there a faster way to multiply two numbers than the way you learned in grade school?

Divide And Conquer (an approach to faster algorithms) DIVIDE a problem into smaller subproblems CONQUER them recursively GLUE the answers together so as to obtain the answer to the larger problem

Multiplication of 2 n-bit numbers X = Y = X = a 2 n/2 + b Y = c 2 n/2 + d XY = ac 2 n + (ad+bc) 2 n/2 + bd ab cd

Multiplication of 2 n-bit numbers X = Y = XY = ac 2 n + (ad+bc) 2 n/2 + bd ab cd MULT(X,Y): If |X| = |Y| = 1 then RETURN XY Break X into a;b and Y into c;d RETURN MULT(a,c) 2 n + (MULT(a,d) + MULT(b,c)) 2 n/2 + MULT(b,d)

Time required by MULT T(n) = time taken by MULT on two n-bit numbers What is T(n)? What is its growth rate? Is it  (n 2 )?

Recurrence Relation T(1) = k for some constant k T(n) = 4 T(n/2) + k’ n + k’’ for some constants k’ and k’’ MULT(X,Y): If |X| = |Y| = 1 then RETURN XY Break X into a;b and Y into c;d RETURN MULT(a,c) 2 n + (MULT(a,d) + MULT(b,c)) 2 n/2 + MULT(b,d)

Let’s be concrete T(1) = 1 T(n) = 4 T(n/2) + n Notice that T(n) is inductively defined for positive powers of 2 How do we unravel T(n) so that we can determine its growth rate?

Technique 1 Guess and Verify Guess: G(n) = 2n 2 – n Verify: G(1) = 1 and G(n) = 4 G(n/2) + n 4 [2(n/2) 2 – n/2] + n =2n 2 – 2n + n =2n 2 – n =G(n)

Technique 1 Guess and Verify Guess: G(n) = 2n 2 – n Verify: G(1) = 1 and G(n) = 4 G(n/2) + n Similarly: T(1) = 1 and T(n) = 4 T(n/2) + n So T(n) = G(n) =  (n 2 )

Technique 2 Guess Form and Calculate Coefficients Guess: T(n) = an 2 + bn + c for some a,b,c Calculate: T(1) = 1  a + b + c = 1 T(n) = 4 T(n/2) + n  an 2 + bn + c = 4 [a(n/2) 2 + b(n/2) + c] + n = an 2 + 2bn + 4c + n  (b+1)n + 3c = 0 Therefore: b=-1 c=0 a=2

Technique 3: Decorate The Tree T(n) = n + 4 T(n/2) n T(n/2) T(n) = T(1) T(1) = 1 1 =

n T(n/2) T(n) =

n T(n/2) T(n) = n/2 T(n/4)

n T(n) = n/2 T(n/4) n/2 T(n/4) n/2 T(n/4) n/2 T(n/4)

n T(n) = n/ n/4

n n/2 + n/2 + n/2 + n/2 Level i is the sum of 4 i copies of n/2 i n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/ i

= 1n = 2n = 4n = 2 i n = (n)n n( n) = n(2n-1) = 2n 2 -n

Divide and Conquer MULT:  (n 2 ) time Grade School Multiplication:  (n 2 ) time All that work for nothing!

MULT revisited MULT calls itself 4 times. Can you see a way to reduce the number of calls? MULT(X,Y): If |X| = |Y| = 1 then RETURN XY Break X into a;b and Y into c;d RETURN MULT(a,c) 2 n + (MULT(a,d) + MULT(b,c)) 2 n/2 + MULT(b,d)

Gauss’ Hack: Input: a,b,c,d Output: ac, ad+bc, bd X 1 = a + b X 2 = c + d X 3 = X 1 X 2 = ac + ad + bc + bd X 4 = ac X 5 = bd X 6 = X 3 – X 4 - X 5 = ad + bc

Gaussified MULT (Karatsuba 1962) T(n) = 3 T(n/2) + n Actually: T(n) = 2 T(n/2) + T(n/2 + 1) + kn MULT(X,Y): If |X| = |Y| = 1 then RETURN XY Break X into a;b and Y into c;d e = MULT(a,c) and f =MULT(b,d) RETURN e2 n + (MULT(a+b, c+d) – e - f) 2 n/2 + f

n T(n/2) T(n) =

n T(n/2) T(n) = n/2 T(n/4)

n T(n) = n/2 T(n/4) n/2 T(n/4) n/2 T(n/4)

n n/2 + n/2 + n/2 Level i is the sum of 3 i copies of n/2 i n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/ i

= 1n = 3/2n = 9/4n = (3/2) i n n(1+3/2+(3/2) (3/2) ) =

A useful sum

Substituting into our formula….

= 1n = 3/2n = 9/4n = (3/2) i n

Dramatic improvement for large n A huge savings over  (n 2 ) when n gets large. How did the Gauss Hack save more than 25% of the work?

Multiplication Algorithms Kindergartenn2 n Grade Schooln2n2 Karatsuban 1.58… Fastest Knownn logn loglogn

You’re cool! Are you free sometime this weekend? Not interested, Bonzo. I took the initiative and asked out a guy in my class.