Great Theoretical Ideas in Computer Science

Slides:



Advertisements
Similar presentations
Grade School Revisited: How To Multiply Two Numbers Great Theoretical Ideas In Computer Science Anupam Gupta Danny Sleator CS Fall 2010 Lecture.
Advertisements

CS Divide and Conquer/Recurrence Relations1 Divide and Conquer.
Lecture 3: Parallel Algorithm Design
Design & Analysis of Algorithms COMP 482 / ELEC 420 John Greiner.
Grade School Again: A Parallel Perspective Great Theoretical Ideas In Computer Science Steven RudichCS Spring 2004 Lecture 17Mar 16, 2004Carnegie.
Lecture 4. Paradigm #2 Recursion Last time we discussed Fibonacci numbers F(n), and Alg fib(n) if (n
Grade School Revisited: How To Multiply Two Numbers Great Theoretical Ideas In Computer Science Victor Adamchik Danny Sleator CS Spring 2010 Lecture.
Divide and Conquer. Recall Complexity Analysis – Comparison of algorithm – Big O Simplification From source code – Recursive.
CS Section 600 CS Section 002 Dr. Angela Guercio Spring 2010.
Notes on the analysis of multiplication algorithms.. Dr. M. Sakalli, Marmara University.
Lecture 4: Divide and Conquer III: Other Applications and Examples Shang-Hua Teng.
Introduction Credits: Steve Rudich, Jeff Edmond, Ping Xuan.
Grade School Revisited: How To Multiply Two Numbers Great Theoretical Ideas In Computer Science Steven RudichCS Spring 2004 Lecture 16March 4,
Recursion Credits: Jeff Edmonds, Ping Xuan. MULT(X,Y): If |X| = |Y| = 1 then RETURN XY Break X into a;b and Y into c;d e = MULT(a,c) and f =MULT(b,d)
On Time Versus Input Size Great Theoretical Ideas In Computer Science Steven RudichCS Spring 2004 Lecture 15March 2, 2004Carnegie Mellon University.
Introduction By Jeff Edmonds York University COSC 3101 Lecture 1 So you want to be a computer scientist? Grade School Revisited: How To Multiply Two Numbers.
Dynamic Programming Introduction to Algorithms Dynamic Programming CSE 680 Prof. Roger Crawfis.
Great Theoretical Ideas in Computer Science.
Grade School Revisited: How To Multiply Two Numbers Great Theoretical Ideas In Computer Science S. Rudich V. Adamchik CS Spring 2006 Lecture 18March.
Arithmetic.
COMPSCI 102 Introduction to Discrete Mathematics.
Complex Numbers Introduction.
MA/CSSE 473 Day 03 Asymptotics A Closer Look at Arithmetic With another student, try to write a precise, formal definition of “t(n) is in O(g(n))”
Divide-and-Conquer 7 2  9 4   2   4   7
Mathematics Review and Asymptotic Notation
Computer Algorithms Ch. 2
Great Theoretical Ideas in Computer Science.
Recursive Algorithms Introduction Applications to Numeric Computation.
Recursion Jeff Edmonds York University COSC 6111 Lecture 3 Friends & Steps for Recursion Derivatives Recursive Images Multiplying Parsing Ackermann.
Program Efficiency & Complexity Analysis. Algorithm Review An algorithm is a definite procedure for solving a problem in finite number of steps Algorithm.
Complexity 20-1 Complexity Andrei Bulatov Parallel Arithmetic.
COMPSCI 102 Introduction to Discrete Mathematics.
J. Elder COSC 3101N Thinking about Algorithms Abstractly Lecture 1. Introduction.
Introduction Pizzanu Kanongchaiyos, Ph.D.
Recurrences David Kauchak cs161 Summer Administrative Algorithms graded on efficiency! Be specific about the run times (e.g. log bases) Reminder:
4/3/2003CSE More Math CSE Algorithms Euclidean Algorithm Divide and Conquer.
COMPSCI 102 Discrete Mathematics for Computer Science.
COSC 3101Winter, 2004 COSC 3101 Design and Analysis of Algorithms Lecture 2. Relevant Mathematics: –The time complexity of an algorithm –Adding made easy.
Lecture 5 Today, how to solve recurrences We learned “guess and proved by induction” We also learned “substitution” method Today, we learn the “master.
Divide and Conquer. Recall Divide the problem into a number of sub-problems that are smaller instances of the same problem. Conquer the sub-problems by.
1 How to Multiply Slides by Kevin Wayne. Copyright © 2005 Pearson-Addison Wesley. All rights reserved. integers, matrices, and polynomials.
Divide and Conquer Faculty Name: Ruhi Fatima Topics Covered Divide and Conquer Matrix multiplication Recurrence.
Grade School Revisited: How To Multiply Two Numbers CS Lecture 4 2 X 2 = 5.
Unary, Binary, and Beyond Great Theoretical Ideas In Computer Science Steven RudichCS Spring 2003 Lecture 2Jan 16, 2003Carnegie Mellon University.
On Time Versus Input Size Great Theoretical Ideas In Computer Science S. Rudich V. Adamchik CS Spring 2006 Lecture 17March 21, 2006Carnegie Mellon.
CMPT 438 Algorithms.
Introduction to Analysis of Algorithms
Lecture 3: Parallel Algorithm Design
Jeff Edmonds York University
CS 3343: Analysis of Algorithms
Great Theoretical Ideas in Computer Science
Randomized Algorithms
Great Theoretical Ideas in Computer Science
Chapter 4 Divide-and-Conquer
Algorithms with numbers (1) CISC4080, Computer Algorithms
Lecture 3: Divide-and-conquer
How To Multiply Two Numbers
CS 3343: Analysis of Algorithms
Grade School Revisited: How To Multiply Two Numbers
CS 3343: Analysis of Algorithms
By Jeff Edmonds York University
CS 3343: Analysis of Algorithms
Real-time 1-input 1-output DSP systems
Data Structures Review Session
Grade School Revisited: How To Multiply Two Numbers
Introduction to Discrete Mathematics
CSCI 256 Data Structures and Algorithm Analysis Lecture 12
Topic: Divide and Conquer
Divide-and-Conquer 7 2  9 4   2   4   7
Introduction to Discrete Mathematics
Presentation transcript:

Great Theoretical Ideas in Computer Science 15-251 Great Theoretical Ideas in Computer Science

Grade School Revisited: How To Multiply Two Numbers Lecture 23 (November 13, 2007)

Gauss (a+bi)

(a+bi)(c+di) = [ac –bd] + [ad + bc] i Gauss’ Complex Puzzle Remember how to multiply two complex numbers a + bi and c + di? (a+bi)(c+di) = [ac –bd] + [ad + bc] i Input: a,b,c,d Output: ac-bd, ad+bc If multiplying two real numbers costs $1 and adding them costs a penny, what is the cheapest way to obtain the output from the input? Can you do better than $4.02?

Gauss’ $3.05 Method Input: a,b,c,d Output: ac-bd, ad+bc c X1 = a + b c X2 = c + d $ X3 = X1 X2 = ac + ad + bc + bd $ X4 = ac $ X5 = bd c X6 = X4 – X5 = ac - bd cc X7 = X3 – X4 – X5 = bc + ad

The Gauss optimization saves one multiplication out of four. It requires 25% less work.

+ Time complexity of grade school addition We saw that T(n) was linear * * * * * * * * * * T(n) = amount of time grade school addition uses to add two n-bit numbers + * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * We saw that T(n) was linear T(n) = Θ(n)

Time complexity of grade school multiplication * * * * * * * * * * * * * * * * * * * * * * * * n2 T(n) = The amount of time grade school multiplication uses to add two n-bit numbers We saw that T(n) was quadratic T(n) = Θ(n2)

Grade School Addition: Linear time Grade School Multiplication: Quadratic time # of bits in the numbers No matter how dramatic the difference in the constants, the quadratic curve will eventually dominate the linear curve

Is there a sub-linear time method for addition?

Any addition algorithm takes Ω(n) time Claim: Any algorithm for addition must read all of the input bits Proof: Suppose there is a mystery algorithm A that does not examine each bit Give A a pair of numbers. There must be some unexamined bit position i in one of the numbers

Any addition algorithm takes Ω(n) time * * * * * * * * * A did not read this bit at position i * * * * * * * * * * * * * * * * * * * If A is not correct on the inputs, we found a bug If A is correct, flip the bit at position i and give A the new pair of numbers. A gives the same answer as before, which is now wrong.

Grade school addition can’t be improved upon by more than a constant factor

Grade School Addition: Θ(n) time. Furthermore, it is optimal Grade School Multiplication: Θ(n2) time Is there a clever algorithm to multiply two numbers in linear time? Despite years of research, no one knows! If you resolve this question, Carnegie Mellon will give you a PhD!

Can we even break the quadratic time barrier? In other words, can we do something very different than grade school multiplication?

Divide And Conquer An approach to faster algorithms: DIVIDE a problem into smaller subproblems CONQUER them recursively GLUE the answers together so as to obtain the answer to the larger problem

Multiplication of 2 n-bit numbers n bits X = Y = a X b Y c d n/2 bits n/2 bits X = a 2n/2 + b Y = c 2n/2 + d X × Y = ac 2n + (ad + bc) 2n/2 + bd

Multiplication of 2 n-bit numbers X = Y = a b c d n/2 bits n/2 bits X × Y = ac 2n + (ad + bc) 2n/2 + bd MULT(X,Y): If |X| = |Y| = 1 then return XY else break X into a;b and Y into c;d return MULT(a,c) 2n + (MULT(a,d) + MULT(b,c)) 2n/2 + MULT(b,d)

Same thing for numbers in decimal! n digits X = Y = a b c d n/2 digits n/2 digits X = a 10n/2 + b Y = c 10n/2 + d X × Y = ac 10n + (ad + bc) 10n/2 + bd

Multiplying (Divide & Conquer style) 12345678 * 21394276 1234*2139 1234*4276 5678*2139 5678*4276 12*21 12*39 34*21 34*39 1*2 1*1 2*2 2*1 2 1 4 2 Hence: 12*21 = 2*102 + (1 + 4)101 + 2 = 252 X = Y = X × Y = ac 10n + (ad + bc) 10n/2 + bd a b c d

Multiplying (Divide & Conquer style) 12345678 * 21394276 1234*2139 1234*4276 5678*2139 5678*4276 12*21 12*39 34*21 34*39 252 468 714 1326 *104 + *102 + *102 + *1 = 2639526 X = Y = X × Y = ac 10n + (ad + bc) 10n/2 + bd a b c d

Multiplying (Divide & Conquer style) 12345678 * 21394276 1234*2139 1234*4276 5678*2139 5678*4276 2639526 5276584 12145242 24279128 *108 + *104 + *104 + *1 = 264126842539128 X = Y = X × Y = ac 10n + (ad + bc) 10n/2 + bd a b c d

Divide, Conquer, and Glue MULT(X,Y)

Divide, Conquer, and Glue MULT(X,Y): if |X| = |Y| = 1 then return XY, else…

Divide, Conquer, and Glue X=a;b Y=c;d MULT(X,Y): Mult(a,c) Mult(b,d) Mult(a,d) Mult(b,c)

Divide, Conquer, and Glue X=a;b Y=c;d MULT(X,Y): Mult(b,d) Mult(a,d) Mult(b,c) Mult(a,c)

Divide, Conquer, and Glue X=a;b Y=c;d MULT(X,Y): ac Mult(b,d) Mult(a,d) Mult(b,c)

Divide, Conquer, and Glue X=a;b Y=c;d MULT(X,Y): ac Mult(b,d) Mult(b,c) Mult(a,d)

Divide, Conquer, and Glue X=a;b Y=c;d MULT(X,Y): ac Mult(b,d) ad Mult(b,c)

Divide, Conquer, and Glue X=a;b Y=c;d MULT(X,Y): ac Mult(b,d) ad Mult(b,c)

Divide, Conquer, and Glue X=a;b Y=c;d MULT(X,Y): ac Mult(b,d) ad bc

Divide, Conquer, and Glue X=a;b Y=c;d MULT(X,Y): ac ad bc Mult(b,d)

Divide, Conquer, and Glue XY = ac2n +(ad+bc)2n/2 + bd X=a;b Y=c;d MULT(X,Y): ac ad bc bd

Time required by MULT T(n) = time taken by MULT on two n-bit numbers What is T(n)? What is its growth rate? Big Question: Is it Θ(n2)? T(n) = 4 T(n/2) + (k’n + k’’) conquering time divide and glue

Recurrence Relation T(1) = k for some constant k T(n) = 4 T(n/2) + k’n + k’’ for constants k’ and k’’ MULT(X,Y): If |X| = |Y| = 1 then return XY else break X into a;b and Y into c;d return MULT(a,c) 2n + (MULT(a,d) + MULT(b,c)) 2n/2 + MULT(b,d)

Recurrence Relation T(1) = 1 T(n) = 4 T(n/2) + n MULT(X,Y): If |X| = |Y| = 1 then return XY else break X into a;b and Y into c;d return MULT(a,c) 2n + (MULT(a,d) + MULT(b,c)) 2n/2 + MULT(b,d)

Technique: Labeled Tree Representation T(n) = n + 4 T(n/2) n T(n) = T(n/2) T(n/2) T(n/2) T(n/2) T(1) = 1 T(n) = 1

T(n) = 4 T(n/2) + (k’n + k’’) conquering time divide and glue ac ad bc X=a;b Y=c;d XY = ac2n + (ad+bc)2n/2 + bd ac ad bc bd

T(n) = n T(n/2) T(n/2) T(n/2) T(n/2)

T(n) = n n/2 T(n/2) T(n/2) T(n/2) T (n/4) T (n/4) T (n/4) T (n/4)

T(n) = n n/2 n/2 T(n/2) T(n/2) T (n/4) T (n/4) T (n/4) T (n/4) T (n/4)

1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1 n n/2 + n/2 + n/2 + n/2 . . . . . . . . . . . . . . . . . . . . . . . . . . 1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1 1 2 Level i is the sum of 4i copies of n/2i i log2(n)

Level i is the sum of 4i copies of n/2i n/2 + n/2 + n/2 + n/2 Level i is the sum of 4i copies of n/2i . . . . . . . . . . . . . . . . . . . . . . . . . . 1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1 1n = 2n = 4n = 2in = (n)n = n(1+2+4+8+ . . . +n) = n(2n-1) = 2n2-n

Divide and Conquer MULT: Θ(n2) time Grade School Multiplication: Θ(n2) time

MULT revisited MULT(X,Y): If |X| = |Y| = 1 then return XY else break X into a;b and Y into c;d return MULT(a,c) 2n + (MULT(a,d) + MULT(b,c)) 2n/2 + MULT(b,d) MULT calls itself 4 times. Can you see a way to reduce the number of calls?

Gauss’ optimization Input: a,b,c,d Output: ac-bd, ad+bc c X1 = a + b c X2 = c + d $ X3 = X1 X2 = ac + ad + bc + bd $ X4 = ac $ X5 = bd c X6 = X4 – X5 = ac - bd cc X7 = X3 – X4 – X5 = bc + ad

Karatsuba, Anatolii Alexeevich (1937-) Sometime in the late 1950’s Karatsuba had formulated the first algorithm to break the n2 barrier!

Gaussified MULT (Karatsuba 1962) MULT(X,Y): If |X| = |Y| = 1 then return XY else break X into a;b and Y into c;d e : = MULT(a,c) f := MULT(b,d) return e 2n + (MULT(a+b,c+d) – e - f) 2n/2 + f T(n) = 3 T(n/2) + n Actually: T(n) = 2 T(n/2) + T(n/2 + 1) + kn

T(n) = n T(n/2) T(n/2) T(n/2)

T(n) = n n/2 T(n/2) T(n/2) T (n/4) T (n/4) T (n/4)

1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1 n n/2 + n/2 + n/2 . . . . . . . . . . . . . . . . . . . . . . . . . . 1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1 1 2 Level i is the sum of 3i copies of n/2i i log2(n)

Level i is the sum of 3i copies of n/2i n/2 + n/2 + n/2 Level i is the sum of 3i copies of n/2i . . . . . . . . . . . . . . . . . . . . . . . . . . 1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1 1n = 3/2n = 9/4n = (3/2)in = (3/2)log nn = n(1+3/2+(3/2)2+ . . . + (3/2)log2 n) = 3n1.58… – 2n

Dramatic Improvement for Large n T(n) = 3nlog2 3 – 2n = Θ(nlog2 3) = Θ(n1.58…) A huge savings over Θ(n2) when n gets large.

n2 n 1.584

Multiplication Algorithms Kindergarten n2n Grade School n2 Karatsuba n1.58… Fastest Known n logn loglogn

n2 n 1.584 n log(n) loglog(n) n 1.584

A short digression on parallel algorithms

Adding n numbers For the next two slides, assume that the CPU can access any number, and add/mult/subtract any two numbers in unit time. Given n numbers a1, a2, …, an How much time to add them all up using 1 CPU? (n) The CPU must at least look at all the numbers.

Adding n numbers (in parallel) Given n numbers a1, a2, …, an How much time to add them all up using as many CPUs as you want? Think of this as getting a group of people together to add the n numbers. Not clear if any one CPU must look at all numbers so (n) lower does not hold any more. In fact, we can do it in O(log n) time.

Addition in the old model? * * * * * * * * * * How do CPUs add n-bit numbers? The k-th carry bit depends on the partial sum to the right of it If we had all the carry bits, we could compute the sum fast. How do we compute all the carry bits? * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

Here’s What You Need to Know… Gauss’s Multiplication Trick Proof of Lower bound for addition Divide and Conquer Solving Recurrences Karatsuba Multiplication Here’s What You Need to Know…