Mining Data Streams (Part 1)

Mining Data Streams (Part 1)
Note to other teachers and users of these slides: We would be delighted if you found this our material useful in giving your own lectures. Feel free to use these slides verbatim, or to modify them to fit your own needs. If you make use of a significant portion of these slides in your own lecture, please include this message, or a link to our web site: Mining Data Streams (Part 1)

New Topic: Infinite Data
High dim. data Locality sensitive hashing Clustering Dimensionality reduction Graph data PageRank, SimRank Community Detection Spam Detection Infinite data Filtering data streams Queries on streams Web advertising Machine learning SVM Decision Trees Perceptron, kNN Apps Recommender systems Association Rules Duplicate document detection

Data Streams In many data mining situations, we do not know the entire data set in advance Stream Management is important when the input rate is controlled externally: Google queries Twitter or Facebook status updates We can think of the data as infinite and non-stationary (the distribution changes over time)

Data Management Vs. Stream Management
In a DBMS, input is under the control of the owner. SQL INSERT commands or bulk loaders. Stream management is important when the input rate is controlled externally. Example: Google search queries. Example: Amazon loves to get orders for products, but does not control when they come in. Example: Satellites send down images, often petabytes/day. More predictable, but too much to deal with efficiently. The fundamental difference between a data stream and a database is who controls how data enters the system. Click 1 In a database system, the staff associated with management of the database generally insert data into the system, using a bulk loader, or even explicit SQL INSERT commands. The staff can decide how much data to load into the system, when, and how fast. Click 2 In a streaming environment, the management cannot control the rate of input. For example, the search queries that arrive at Google are generated by random people around the world, at their pace. Google staff have no control over the arrival of queries; they have to architect their system to deal with whatever data rate there is. You might think a transaction-processing system, like Walmart recording all the purchases at all its cash registers everywhere as a stream, and in a sense it is. But Walmart has a large but fixed number of registers, and checkout clerks can press the keys just so fast. So there is actually a pretty well defined limit on how fast data arrives in such a system.

The Stream Model Input elements enter at a rapid rate, at one or more input ports (i.e., streams) We call elements of the stream tuples The system cannot store the entire stream accessibly Q: How do you make critical calculations about the stream using a limited amount of (secondary) memory?

Two Forms of Query Ad-hoc queries: Normal queries asked one time about streams. Example: What is the maximum value seen so far in stream S? Standing queries: Queries that are, in principle, asked about the stream at all times. Example: Report each new maximum value ever seen in stream S. Streams can be queried in two modes. Click 1 The first is similar to the way we query a database system. You ask a query once and expect an answer about the state of the system at the time you asked the query. Click 2 For example, what is the maximum value seen in the stream from its beginning to the exact time the query is asked. This question can be answered by keeping a single value, the maximum, and updating it if necessary, each time a new stream element arrives. Click 3 The other kind of query is called a standing query. You write the query once, and you expect the system to make the answer available at all times, perhaps outputting a new value each time the answer changes. Click 4 For instance, a standing query might ask for a report of each stream element that is larger than any element seen so far on the stream. We can answer this one by keeping one value – the maximum. Each new element is compared with the max, and if it is larger, we do two things: output the value and update the max to be that value.

General Stream Processing Model
Ad-Hoc Queries Processor Standing Queries , 5, 2, 7, 0, 9, 3 a, r, v, t, y, h, b , 0, 1, 0, 1, 1, 0 time Streams Entering. Each is stream is composed of elements/tuples Output Limited Working Storage Archival Storage

Problems on Data Streams
Types of queries one wants on answer on a data stream: (we’ll do these today) Sampling data from a stream Construct a random sample Queries over sliding windows Number of items of type x in the last k elements of the stream

Problems on Data Streams
Types of queries one wants on answer on a data stream: (we’ll do these next time) Filtering a data stream Select elements with property x from the stream Counting distinct elements Number of distinct elements in the last k elements of the stream Estimating moments Estimate avg./std. dev. of last k elements Finding frequent elements

Applications (1) Mining query streams Mining click streams
Google wants to know what queries are more frequent today than yesterday Mining click streams Yahoo wants to know which of its pages are getting an unusual number of hits in the past hour Mining social network news feeds E.g., look for trending topics on Twitter, Facebook

Applications (2) Sensor Networks Telephone call records
Many sensors feeding into a central controller Telephone call records Data feeds into customer bills as well as settlements between telephone companies IP packets monitored at a switch Gather information for optimal routing Detect denial-of-service attacks

Sliding Windows A useful model of stream processing is that queries are about a window of length N – the N most recent elements received. Interesting case: N is so large it cannot be stored in main memory. Or, there are so many streams that windows for all do not fit in main memory. The concept of the sliding window is essential for many of the algorithms we are going to discuss. Click 1 The simplest form of window is defined by a fixed length N and consists of the most recent N elements received on a stream. Notice that each time a new element is received, the oldest element falls out of the window. Click 2 A variation is to define the window as all the elements that have arrived within some time interval T extending into the past – say the last hour. This sort of window has a storage requirement that is not fixed, since the number of arrivals within a time T can vary. In comparison, defining the window to hold a fixed number of elements lets us rely on needing storage space only up to a certain limit. Click 3 The interesting case is when we are using a window consisting of the last N stream elements, but N is so large, we cannot store N elements in main memory. Click 4 While we have options to increase the size of main memory, use many compute nodes to handle one window, or use disk in some cases, we also need to consider the case where there are many streams, perhaps millions, arriving at the same stream processor. In that case, N does not have to be too large before we cannot store all the windows in a way that allows us to get exact answers to queries about the contents of the windows.

q w e r t y u i o p a s d f g h j k l z x c v b n m
Here’s a little picture of a stream and a window of length 6. Initially, the stream has arrived up to this point j (POINT). The elements k, l, and so on, will arrive in the future. Click 1 Now, k arrives. The oldest element, s, is no longer part of the window, which continues to hold exactly 6 elements, as it always will. Click 2 Now, “ell” arrives, and d falls out of the window. Click 3 And z arrives, causing f to be dropped from the window. Past Future

Example: Averages Stream of integers, window of size N.
Standing query: what is the average of the integers in the window? For the first N inputs, sum and count to get the average. Afterward, when a new input i arrives, change the average by adding (i - j)/N, where j is the oldest integer in the window before i arrived. Good: O(1) time per input. Bad: Requires the entire window in main memory. Let’s take a really simple example. Click 1 We have a stream of integers. Click 2 The window is of size N, that is, the window will hold the N most recent integers in the stream. Click 3 We want the system to be able to answer one standing query: what is the average of the elements in the window. Click 4 Often, we imagine that a stream extends infinitely into the past, so we don’t worry about what happens before there have been enough arrivals to fill the window. However, realistically, we have to get started somehow. So let’s store the first N inputs as they arrive, and maintain the sum and count of elements seen so far, until the count reaches N. The average is the sum divided by the count at any point. Click 5 Now, suppose we have our window full, and it consists of the most recent N elements. We also store the average of these elements; that average is in the local storage but it not part of the window. Suppose a new element i arrives. The oldest element j in the window will fall out of the window. Thus, the change in the average is (i-j)/N (POINT). i/N accounts for the contribution i makes to the average, and –j/N accounts for the fact that j no longer contributes to the average. The important point is that in this manner, we can answer the query “what is the average of the elements in the window” doing only a small, fixed number of arithmetic steps with each arrival on the stream. That is far far better than having to compute the sum and average of all N elements in the window each time a new element arrives. But not every query about the current value of the window can be answered in equally convenient way.

Sampling from a Data Stream: Sampling a fixed-size sample
As the stream grows, the sample is of fixed size

Uniform Random Sampling
Sampling without replacement Randomly draw an element Don’t put it back Repeat s times Sampling with replacement Put it back Trivial in the RAM model The statistical difference is very small, for 𝑛≫𝑠

Weighted Random Sampling
Each element i is associated with a weight wi W=w1+…+wn Sampling without replacement Randomly draw an element according to probability distribution wi/W Don’t put it back Repeat s times Sampling with replacement?

Maintaining a fixed-size sample
Problem: Fixed-size sample Suppose we need to maintain a random sample S of size exactly s tuples E.g., main memory size constraint Why? Don’t know length of stream in advance Suppose at time n we have seen n items Each item is in the sample S with equal prob. s/n How to think about the problem: say s = 2 Stream: a x c y z k c d e g… At n= 5, each of the first 5 tuples is included in the sample S with equal prob. At n= 7, each of the first 7 tuples is included in the sample S with equal prob. Impractical solution would be to store all the n tuples seen so far and out of them pick s at random

Solution: Fixed Size Sample
Algorithm (a.k.a. Reservoir Sampling) Store all the first s elements of the stream to S Suppose we have seen n-1 elements, and now the nth element arrives (n > s) With probability s/n, keep the nth element, else discard it If we picked the nth element, then it replaces one of the s elements in the sample S, picked uniformly at random Claim: This algorithm maintains a sample S with the desired property: After n elements, the sample contains each element seen so far with probability s/n

Proof: By Induction We prove this by induction: Base case:
Assume that after n elements, the sample contains each element seen so far with probability s/n We need to show that after seeing element n+1 the sample maintains the property Sample contains each element seen so far with probability s/(n+1) Base case: After we see n=s elements the sample S has the desired property Each out of n=s elements is in the sample with probability s/s = 1

Proof: By Induction Inductive hypothesis: After n elements, the sample S contains each element seen so far with prob. s/n Now element n+1 arrives Inductive step: For elements already in S, probability that the algorithm keeps it in S is: So, at time n, tuples in S were there with prob. s/n Time nn+1, tuple stayed in S with prob. n/(n+1) So prob. tuple is in S at time n+1 = 𝒔 𝒏 ⋅ 𝒏 𝒏+𝟏 = 𝒔 𝒏+𝟏 Element n+1 not discarded Element in the sample not picked Element n+1 discarded

Reservoir Sampling Correctness Proof
Many “proofs” found online are actually wrong They only show that each item is sampled with probability 𝑠/𝑛 Need to show that every subset of size 𝑠 has the same probability to be the sample Correct proof relates with the Fisher-Yates shuffle s = 2 To initialize an array a of n elements to a randomly shuffled copy of S a[0] ← S[0] for i from 1 to n - 1 do r ← random (0 .. i) a[i] ← a[r] a[r] ← S[i] a b c d a b c d b a c d b c a d b d a c

Weighted Reservoir Sampling
WeightedReservoir-Chao(S[1..n], R[1..k]) WSum = 0 // fill the reservoir array for i = 1 to k R[i] := S[i] WSum = WSum + S[i].Weight/k for i = k+1 to n p = S[i].Weight / WSum // probability for this item j := random(0, 1); // important: inclusive range if j <= p // select item according to probability R[random(1,k)] := S[i] //uniform selection in reservoir for replacement

Priority Sampling For each element ai, generate a random number ri from [0, 1] as its priority. Keep the s elements with largest s priorities Correctness: Each s-subset has a chance of 1/{n \choose s} to be selected Priority for weighted sampling Each elements ai is associated with a weight wi Sample one element from a1,…,an, such that ai is sampled with probability wi/W, W = w1+…+wn N. G. Duffield, C. Lund, and M. Thorup, “Priority sampling for estimation of arbitrary subset sums.,” J. ACM (JACM) 54(6), vol. 54, no. 6, pp. 32–es, 2007.

Priority Sampling Priority for weighted sampling Algorithm
Each elements ai is associated with a weight wi Sample one element from a1,…,an, such that ai is sampled with probability wi/W, W = w1+…+wn Algorithm For each element ai, generate random number ri from [0,1] Associate priority rhoi = ri1/wi Select ai with max rhoi

Merging random samples
𝑆2 𝑆1 + 𝑁1 = 10 𝑁2 = 15

𝑆2 𝑆1 + 𝑁1 = 10 𝑁2 = 15 With prob. 𝑁1/(𝑁1+𝑁2), take a sample from 𝑆1

𝑆2 𝑆1 + 𝑁1 = 8 𝑁2 = 12 With prob. 𝑁2/(𝑁1+𝑁2), take a sample from 𝑆2 𝑁3 = 25 P. K. Agarwal, G. Cormode, Z. Huang, J. M. Phillips, Z. Wei, and K. Yi, “Mergeable Summaries,” presented at the Proc. ACM Symposium on Principles of Database Systems, 2012.

Alias Method Sampling as a data structure Uniform sampling: easy
Given N elements, build a data structure Query: return a sample from the N elements Uniform sampling: easy Weighted Sampling: Each element is associated with a probability pi, p1+…+pn = 1 Query: return a sample according to distribution p1,…, pn Walker, Alastair J “An Efficient Method for Generating Discrete Random Variables With General Distributions,”. ACM Transactions on Mathematical Software, 3: 253–256.

Alias Method Approach 1 Approach 2 Alias Method
For each query, generate random number r from [0,1] Find pi such that p1+…+pi-1 < r <= p1+…+pi O(n) space, O(n) query cost Approach 2 Store p1+…+pi for i = 1,…, n Binary search O(n) space, O(log n) query cost Alias Method O(n) space, O(1) query cost

Alias Method Sort p1 <= … <= pn Preprocessing Query
If p1 = 1/n, done. Else p1 < 1/n, pour 1/n – p1 portion of pn to form a probability of 1/n and remove p1 Recursively do above until all probability =1/n Each 1/n probability is associated with two old probabilities pi and pj Query Generate a random number from [1, n] to find a 1/n probability Generate a random number from [0, 1/n] to identify pi or pj A. Q. Li, A. Ahmed, S. Ravi, and A. J. Smola, “Reducing the sampling complexity of topic models.,” KDD, pp. 891–900, 2014.

Queries over a (long) Sliding Window

Sliding Windows A useful model of stream processing is that queries are about a window of length N – the N most recent elements received Interesting case: N is so large that the data cannot be stored in memory, or even on disk Or, there are so many streams that windows for all cannot be stored Amazon example: For every product X we keep 0/1 stream of whether that product was sold in the n-th transaction We want answer queries, how many times have we sold X in the last k sales

Sliding Window: 1 Stream
Sliding window on a single stream: N = 6 q w e r t y u i o p a s d f g h j k l z x c v b n m q w e r t y u i o p a s d f g h j k l z x c v b n m q w e r t y u i o p a s d f g h j k l z x c v b n m q w e r t y u i o p a s d f g h j k l z x c v b n m Past Future

Counting Bits (1) Problem:
Given a stream of 0s and 1s Be prepared to answer queries of the form How many 1s are in the last k bits? where k ≤ N Obvious solution: Store the most recent N bits When new bit comes in, discard the N+1st bit Suppose N=6 Past Future

Counting Bits (2) You can not get an exact answer without storing the entire window Real Problem: What if we cannot afford to store N bits? E.g., we’re processing 1 billion streams and N = 1 billion But we are happy with an approximate answer Past Future

An attempt: Simple solution
Q: How many 1s are in the last N bits? A simple solution that does not really solve our problem: Uniformity assumption Maintain 2 counters: S: number of 1s from the beginning of the stream Z: number of 0s from the beginning of the stream How many 1s are in the last N bits? 𝑵∙ 𝑺 𝑺+𝒁 But, what if stream is non-uniform? What if distribution changes over time? N Past Future

EH Method (Exponential Histogram)
[Datar, Gionis, Indyk, Motwani] EH Method (Exponential Histogram) EH solution that does not assume uniformity We store 𝑶(log𝟐𝑵) bits per stream Solution gives approximate answer, never off by more than 50% Error factor can be reduced to any fraction > 0, with more complicated algorithm and proportionally more stored bits

Idea: Exponential Windows
Solution that doesn’t (quite) work: Summarize exponentially increasing regions of the stream, looking backward Drop small regions if they begin at the same point as a larger region Window of width 16 has 6 1s 1 2 3 4 10 6 ? remember -- this is a strawman algorithm that doesn't really work, so I never spent much time worrying about it, and you shouldn't either. However, you don't have to worry about where the buckets begin or end in this algorithm, since that is determined completely from the count of bits received so far. The rule for updating is as follows. 1. when a bit comes in, create a bucket of length 1 with the proper count (0 or 1). 2. If any level has 3 buckets: a) add the rightmost two and create a bucket at the next higher level (twice the length) with that sum. b) delete the leftmost two buckets, keeping only the rightmost of the three.. 3. Repeat (2) recursively for progressively higher levels. I hope this helps. I would really invite students to figure it out if they care (they won't). N We can reconstruct the count of the last N bits, except we are not sure how many of the last 6 1s are included in the N

What’s Good? Stores only O(log2N ) bits Easy update as more bits enter
𝑶(log⁡𝑵) counts of log 𝟐 𝑵 bits each Easy update as more bits enter Error in count no greater than the number of 1s in the “unknown” area

What’s Not So Good? As long as the 1s are fairly evenly distributed, the error due to the unknown region is small – no more than 50% But it could be that all the 1s are in the unknown area at the end In that case, the error is unbounded! 1 2 3 4 10 6 N ?

[Datar, Gionis, Indyk, Motwani]
Fixup: EH method Idea: Instead of summarizing fixed-length blocks, summarize blocks with specific number of 1s: Let the block sizes (number of 1s) increase exponentially When there are few 1s in the window, block sizes stay small, so errors are small N

EH: Timestamps Each bit in the stream has a timestamp, starting 1, 2, … Record timestamps modulo N (the window size), so we can represent any relevant timestamp in 𝑶(𝒍𝒐 𝒈 𝟐 𝑵) bits

EH: Buckets A bucket in the EH method is a record consisting of:
(A) The timestamp of its end [O(log N) bits] (B) The number of 1s between its beginning and end [O(log log N) bits] Constraint on buckets: Number of 1s must be a power of 2 That explains the O(log log N) in (B) above N

Representing a Stream by Buckets
Either one or two buckets with the same power-of-2 number of 1s Buckets do not overlap in timestamps Buckets are sorted by size Earlier buckets are not smaller than later buckets Buckets disappear when their end-time is > N time units in the past

Example: Bucketized Stream
At least 1 of size 16. Partially beyond window. 2 of size 8 2 of size 4 1 of size 2 2 of size 1 N Three properties of buckets that are maintained: - Either one or two buckets with the same power-of-2 number of 1s - Buckets do not overlap in timestamps - Buckets are sorted by size

Updating Buckets (1) When a new bit comes in, drop the last (oldest) bucket if its end-time is prior to N time units before the current time 2 cases: Current bit is 0 or 1 If the current bit is 0: no other changes are needed

Updating Buckets (2) If the current bit is 1:
(1) Create a new bucket of size 1, for just this bit End timestamp = current time (2) If there are now three buckets of size 1, combine the oldest two into a bucket of size 2 (3) If there are now three buckets of size 2, combine the oldest two into a bucket of size 4 (4) And so on …

Example: Updating Buckets
Current state of the stream: Bit of value 1 arrives Two orange buckets get merged into a yellow bucket Next bit 1 arrives, new orange bucket is created, then 0 comes, then 1: Buckets get merged… State of the buckets after merging

How to Query? To estimate the number of 1s in the most recent N bits:
Sum the sizes of all buckets but the last (note “size” means the number of 1s in the bucket) Add half the size of the last bucket Remember: We do not know how many 1s of the last bucket are still within the wanted window

Example: Bucketized Stream
At least 1 of size 16. Partially beyond window. 2 of size 8 2 of size 4 1 of size 2 2 of size 1 N

Error Bound: Proof Why is error 50%? Let’s prove it!
Suppose the last bucket has size 2r Then by assuming 2r-1 (i.e., half) of its 1s are still within the window, we make an error of at most 2r-1 Since there is at least one bucket of each of the sizes less than 2r, the true sum is at least r-1 = 2r -1 Thus, error at most 50% At least 16 1s N

Further Reducing the Error
Instead of maintaining 1 or 2 of each size bucket, we allow either r-1 or r buckets (r > 2) Except for the largest size buckets; we can have any number between 1 and r of those Error is at most O(1/r) By picking r appropriately, we can tradeoff between number of bits we store and the error

Extensions Can we use the same trick to answer queries How many 1’s in the last k? where k < N? A: Find earliest bucket B that at overlaps with k. Number of 1s is the sum of sizes of more recent buckets + ½ size of B Can we handle the case where the stream is not bits, but integers, and we want the sum of the last k elements? k Yes, instead of bit counts keep partial sums

Extensions Stream of positive integers
We want the sum of the last k elements Amazon: Avg. price of last k sales Solution: (1) If you know all have at most m bits Treat m bits of each integer as a separate stream Use EH to count 1s in each integer The sum is = 𝑖=0 𝑚−1 𝑐 𝑖 2 𝑖 (2) Use buckets to keep partial sums Sum of elements in size b bucket is at most 2b ci …estimated count for i-th bit Idea: Sum in each bucket is at most 2b (unless bucket has only 1 integer) Bucket sizes: 16 8 4 2 1

Summary Sampling a fixed proportion of a stream
Sample size grows as the stream grows Sampling a fixed-size sample Reservoir sampling Counting the number of 1s in the last N elements Exponentially increasing windows Extensions: Number of 1s in any last k (k < N) elements Sums of integers in the last N elements

Homework 4.6.1, 4.6.2

Counter Based Summaries

Heavy hitters [MG82] Misra-Gries (MG) algorithm finds up to 𝑘 items that occur more than 1/𝑘 fraction of the time in a stream Estimate their frequencies with additive error ≤𝑁/𝑘 Equivalently, achieves 𝜀𝑁 error with 𝑂(1/𝜀) space Better than Count-Min but doesn’t support deletions Keep 𝑘 different candidates in hand. To add a new item: If item is monitored, increase its counter Else, if <𝑘 items monitored, add new item with count 1 Else, decrease all counts by 1 𝑘=5

Heavy hitters Misra-Gries (MG) algorithm finds up to 𝑘 items that occur more than 1/𝑘 fraction of the time in a stream [MG’82] Estimate their frequencies with additive error ≤𝑁/𝑘 Keep 𝑘 different candidates in hand. To add a new item: If item is monitored, increase its counter Else, if <𝑘 items monitored, add new item with count 1 Else, decrease all counts by 1 𝑘=5

MG analysis 𝑁 = total input size 𝑀 = sum of counters in data structure
Error in any estimated count at most (𝑁−𝑀)/(𝑘+1) Estimated count a lower bound on true count Each decrement spread over (𝑘+1) items: 1 new one and 𝑘 in MG Equivalent to deleting (𝑘+1) distinct items from stream At most (𝑁−𝑀)/(𝑘+1) decrement operations Hence, can have “deleted” (𝑁−𝑀)/(𝑘+1) copies of any item So estimated counts have at most this much error

Merging two MG summaries [ACHPZY12]
Merging algorithm: Merge two sets of 𝑘 counters in the obvious way Take the (𝑘+1)th largest counter = 𝐶 𝑘+1 , and subtract from all Delete non-positive counters 𝑘=5

Merging two MG summaries
This algorithm gives mergeability: Merge subtracts at least 𝑘+1 𝐶 𝑘+1 from counter sums So 𝑘+1 𝐶 𝑘+1  (𝑀1 + 𝑀2 – 𝑀12) Sum of remaining (at most 𝑘) counters is 𝑀12 By induction, error is ((𝑁1−𝑀1)+(𝑁2−𝑀2)+(𝑀1+𝑀2–𝑀12))/(𝑘+1) =((𝑁1+𝑁2) –𝑀12)/(𝑘+1) (prior error) (from merge) 𝑘=5 𝐶 𝑘+1

Mining Data Streams (Part 1)

Similar presentations

Presentation on theme: "Mining Data Streams (Part 1)"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Mining Data Streams (Part 1)

Similar presentations

Presentation on theme: "Mining Data Streams (Part 1)"— Presentation transcript:

Similar presentations

About project

Feedback