1. Space-Efficient Algorithms for Streaming Data Matthew Todd Adereth Yuval Ishai & Mahesh Viswanathan

2. Inspiration: n Problems with large amounts of data that is read in an arbitrary order n Any problem where the input data can only be read once AT&T Call Data n Problems where real-time reactions are needed and finite memory is available Network Routers Embedded Systems

3. Question: n How to compute f(X) efficiently if X is arbitrarily ordered? n Restrictions: –Input is read-once –Small space x 1 x 2 x 3 x 4 x 5 (3,x 3 ) (1,x 1 ) (4,x 4 ) (5,x 5 ) (2,x 2 )

4. Example: n Setting: Hospital n We have a list of organ donors and recipients and their blood types. We want to verify that all of our donor/recipient pairs are compatible Donors and Recipients are either “Type A” or “Type B” Only pairs of the same type are valid

5. The Ordered Case: n The input stream will be in the following order: Donor 1 A Recip. 1 A Donor 2 B Recip. 2 B Donor 3 B Recip. 3 B Donor 4 A Recip. 4 A

6. n How much memory do we need to verify this stream satisfies the property that the type of each donor is equal to the type of the corresponding recipient? Question: Donor 1 A Recip. 1 A Donor 2 B Recip. 2 B Donor 3 B Recip. 3 B Donor 4 A Recip. 4 A

7. n Only 2 bits (4 states) are required, no matter how many Donor/Recipient pairs there are: Solution: 00 01 10 11 A A AA,B B B B

8. The Unordered Case: n What if our data stream isn’t so nicely organized for us? What if the order of the input bits isn’t known in advance? Donor 5 A Donor 2 B Donor 9 B Recip. 2 B Donor 1 A Recip. 3 B Recip. 4 A Donor 4 A Recip. 9 B Donor 1 A Donor 5 A Recip. 4 A Donor 4 A Donor 2 B Recip. 2 B Donor 4 A

9. n How much memory do we need to verify this unordered stream of bits satisfies the property that the type of each donor is equal to the type of the corresponding recipient? –I will now show that at least n bits are required if there are n Donor/Recipient pairs The Same Question:

10. Communication Complexity n Two doctors, Dr. Alice and Dr. Bob, at different hospitals n Alice has a list of each donor’s type n Bob has a list of each recipient’s type n What is the least number of bits Alice must communicate to Bob in order for Bob to verify that all the Donor/Recipient pairs match?

11. Communication Complexity n Clearly, Alice can communicate n bits to Bob in order for him to decide if each pair is valid, but could it be done in less?

12. Communication Complexity –No! There are 2 n possible lists. If Alice and Bob had a technique to pass the message in n-1 bits there would be less than 2 n possible messages. –This would mean that there would be two lists that correspond to the same message. –Since each Donor list requires a unique Recipient list to be valid, each list must be communicated with a unique message

13. Communication Complexity AA…AA AA…AB AA…BA AA…BB BB…AA BB…AB BB…BA BB…BB Alice’s ListBob’s List 0…000 0…001 0…010 1…100 1…101 1…110 1…111 Message ? AA…AA AA…AB AA…BA AA…BB BB…AA BB…AB BB…BA BB…BB

14. n How much memory do we need to verify the unordered stream of bits satisfies the property that the type of each donor is equal to the type of the corresponding recipient? –I will now show that at least n bits are required if there are n Donor/Recipient pairs The Same Question:

15. Proof by Contradiction: n Assume there exists a method to verify the unordered stream of bits using less than n bits. n Alice now feeds her list into the algorithm as a partial stream. The memory will be in a particular configuration. n Alice then tells Bob the bit configuration of the memory.

16. Proof by Contradiction: n Bob, using his machine, sets the memory in his computer to the bit configuration Alice told him. n Bob now feeds his list into the algorithm. If the algorithm ends in a valid state, Bob’s list matches Alice’s.

17. Proof by Contradiction: n But we have already shown that Alice can’t communicate her message in less than n bits! n If an algorithm exists that uses less than n bits of memory we have a contradiction, therefore no such algorithm exists!

18. The Point: n This type of argument is one of the few tools we have to prove lower bounds on the amount of memory an algorithm must use to decide whether or not an unordered stream of bits has a given property. n I have been working this summer on developing techniques for proving tighter bounds on Streaming Complexity.