COMP53311 Data Stream Prepared by Raymond Wong Presented by Raymond Wong

COMP53312 Data Mining over Static Data 1.Association 2.Clustering 3.Classification Static Data Output (Data Mining Results)

COMP53313 Data Mining over Data Streams 1.Association 2.Clustering 3.Classification Output (Data Mining Results) … Unbounded Data Real-time Processing

COMP53314 Data Streams 12 … Less recentMore recent Each point: a transaction

COMP53315 Data Streams Traditional Data Mining Data Stream Mining Data TypeStatic Data of Limited Size Dynamic Data of Unlimited Size (which arrives at high speed) MemoryLimitedLimited  More challenging EfficiencyTime-ConsumingEfficient OutputExact AnswerApproximate (or Exact) Answer

COMP53316 Entire Data Streams 12 … Less recentMore recent Each point: a transaction Obtain the data mining results from all data points read so far

COMP53317 Entire Data Streams 12 … Less recentMore recent Each point: a transaction Obtain the data mining results over a sliding window

COMP53318 Data Streams Entire Data Streams Data Streams with Sliding Window

COMP53319 Entire Data Streams Association Clustering Classification Frequent pattern/item

COMP Frequent Item over Data Streams Let N be the length of the data streams Let s be the support threshold (in fraction) (e.g., 20%) Problem: We want to find all items with frequency >= sN 12 … Less recentMore recent Each point: a transaction

COMP Data Streams Traditional Data Mining Data Stream Mining Data TypeStatic Data of Limited Size Dynamic Data of Unlimited Size (which arrives at high speed) MemoryLimitedLimited  More challenging EfficiencyTime-ConsumingEfficient OutputExact AnswerApproximate (or Exact) Answer

COMP Data Streams Static Data Output (Data Mining Results) Output (Data Mining Results) … Unbounded Data Frequent item I 1 Infrequent item I 2 I 3 Frequent item I 1 I 3 Infrequent item I 2

COMP False Positive/Negative E.g. Expected Output Frequent item I 1 Infrequent item I 2 I 3 Algorithm Output Frequent item I 1 I 3 Infrequent item I 2 False Positive -The item is classified as frequent item -In fact, the item is infrequent Which item is one of the false positives?I3I3 More?No. No. of false positives = 1 If we say: The algorithm has no false positives. All true infrequent items are classified as infrequent items in the algorithm output.

COMP False Positive/Negative E.g. Expected Output Frequent item I 1 I 3 Infrequent item I 2 Algorithm Output Frequent item I 1 Infrequent item I 2 I 3 False Negative -The item is classified as infrequent item -In fact, the item is frequent Which item is one of the false negatives?I3I3 More?No. No. of false negatives = 1 No. of false positives =0 If we say: The algorithm has no false negatives. All true frequent items are classified as frequent items in the algorithm output.

COMP Data Streams Traditional Data Mining Data Stream Mining Data TypeStatic Data of Limited Size Dynamic Data of Unlimited Size (which arrives at high speed) MemoryLimitedLimited  More challenging EfficiencyTime-ConsumingEfficient OutputExact AnswerApproximate (or Exact) Answer We need to introduce an input error parameter 

COMP Data Streams Static Data Output (Data Mining Results) Output (Data Mining Results) … Unbounded Data Frequent item I 1 Infrequent item I 2 I 3 Frequent item I 1 I 3 Infrequent item I 2

COMP Data Streams Static Data Output (Data Mining Results) Output (Data Mining Results) … Unbounded Data Store the statistics of all items I 1 : 10 I 2 : 8 I 3 : 12 Estimate the statistics of all items I 1 : 10 I 2 : 4 I 3 : 10 ItemTrue FrequencyEstimated Frequency I1I1 10 I2I2 84 I3I N: total no. of occurrences of items N = 20  = 0.2  N = 4 Diff. D D <=  N ? Yes

COMP  -deficient synopsis Let N be the current length of the stream (or total no. of occurrences of items) Let  be an input parameter (a real number from 0 to 1) All true frequent items are classified as frequent items in the algorithm output. An algorithm maintains an  -deficient synopsis if its output satisfies the following properties Condition 2: The difference between the estimated frequency and the true frequency is at most  N. Condition 1: There is no false negative. Condition 3: All items whose true frequencies less than (s-  )N are classified as infrequent items in the algorithm output

COMP Frequent Pattern Mining over Entire Data Streams Algorithm Sticky Sampling Algorithm Lossy Counting Algorithm Space-Saving Algorithm

COMP Sticky Sampling Algorithm Sticky Sampling s   … Unbounded Data Statistics of items Output Stored in the memory Frequent items Infrequent items Support threshold Error parameter Confidence parameter

COMP Sticky Sampling Algorithm The sampling rate r varies over the lifetime of a stream Confidence parameter  (a small real number) Let t =  1/  ln(s -1  -1 )  Data No.r (sampling rate) 1 ~ 2t1 2t+1 ~ 4t2 4t+1 ~ 8t4 ……

COMP Sticky Sampling Algorithm The sampling rate r varies over the lifetime of a stream Confidence parameter  (a small real number) Let t =  1/  ln(s -1  -1 )  Data No.r (sampling rate) 1 ~ 2t1 2t+1 ~ 4t2 4t+1 ~ 8t4 …… e.g. s = 0.02  = 0.01  = 0.1 t = ~ 2*622 2*622+1 ~ 4*622 4*622+1 ~ 8*622 1~ ~ ~4976

COMP Sticky Sampling Algorithm The sampling rate r varies over the lifetime of a stream Confidence parameter  (a small real number) Let t =  1/  ln(s -1  -1 )  Data No.r (sampling rate) 1 ~ 2t1 2t+1 ~ 4t2 4t+1 ~ 8t4 …… e.g. s = 0.5  = 0.35  = 0.5 t = 4 1 ~ 2*4 2*4+1 ~ 4*4 4*4+1 ~ 8*4 1~8 9~16 17~32

COMP Sticky Sampling Algorithm 1.S: empty list  will contain (e, f) element Estimated frequency 2.When data e arrives,  if e exists in S, increment f in (e, f)  if e does not exist in S, add entry (e, 1) with prob. 1/r (where r: sampling rate) 3.Just after r changes, For each entry (e, f), Repeatedly toss a coin with P(head) = 1/r until the outcome of the coin toss is head If the outcome of the toss is tail, Decrement f in (e, f) If f = 0, delete the entry (e, f) 4. [Output] Get a list of items where f  +  N >= sN

COMP Analysis  -deficient synopsis Sticky Sampling computes an  -deficient synopsis with probability at least 1-  Memory Consumption Sticky Sampling occupies at most  2/  ln(s -1  -1 )  entries on average

COMP Frequent Pattern Mining over Entire Data Streams Algorithm Sticky Sampling Algorithm Lossy Counting Algorithm Space-Saving Algorithm

COMP Lossy Counting Algorithm Lossy Counting s  … Unbounded Data Statistics of items Output Stored in the memory Frequent items Infrequent items Support threshold Error parameter

COMP Lossy Counting Algorithm 12 … Less recentMore recent Each point: a transaction Bucket 1 Bucket 2 Bucket 3 … Bucket b current N: current length of stream b current = Width w =

COMP Lossy Counting Algorithm 1.D: Empty set Will contain (e, f,  ) element Frequency of element since this entry was inserted into D Max. possible error in f 2.When data e arrives, If e exists in D, Increment f in (e, f,  ) If e does not exist in D, Add entry (e, 1, b current -1) 3.Remove some entries in D whenever N  0 mod w (i.e., whenever it reaches the bucket boundary) The rule of deletion is: (e, f,  ) is deleted if f +  <= b current 4.[Output] Get a list of items where f  +  N >= sN

COMP Lossy Counting Algorithm  -deficient synopsis Lossy Counting computes an  -deficient synopsis Memory Consumption Lossy Counting occupies at most   1/  log(  N)  entries.

COMP Comparison  -deficient synopsis Memory Consumption Sticky Sampling 1-  confidence  2/  ln(s -1  -1 )  Lossy Counting100% confidence  1/  log(  N)  Memory = 1243 e.g. s = 0.02  = 0.01  = 0.1 N = 1000 Memory = 231

COMP Comparison  -deficient synopsis Memory Consumption Sticky Sampling 1-  confidence  2/  ln(s -1  -1 )  Lossy Counting100% confidence  1/  log(  N)  Memory = 1243 e.g. s = 0.02  = 0.01  = 0.1 N = 1,000,000 Memory = 922

COMP Comparison  -deficient synopsis Memory Consumption Sticky Sampling 1-  confidence  2/  ln(s -1  -1 )  Lossy Counting100% confidence  1/  log(  N)  Memory = 1243 e.g. s = 0.02  = 0.01  = 0.1 N = 1,000,000,000 Memory = 1612

COMP Frequent Pattern Mining over Entire Data Streams Algorithm Sticky Sampling Algorithm Lossy Counting Algorithm Space-Saving Algorithm

COMP Sticky Sampling Algorithm Sticky Sampling s   … Unbounded Data Statistics of items Output Stored in the memory Frequent items Infrequent items Support threshold Error parameter Confidence parameter

COMP Lossy Counting Algorithm Lossy Counting s  … Unbounded Data Statistics of items Output Stored in the memory Frequent items Infrequent items Support threshold Error parameter

COMP Space-Saving Algorithm Space-Saving s M … Unbounded Data Statistics of items Output Stored in the memory Frequent items Infrequent items Support threshold Memory parameter

COMP Space-Saving M: the greatest number of possible entries stored in the memory

COMP Space-Saving 1.D: Empty set Will contain (e, f,  ) element Frequency of element since this entry was inserted into D Max. possible error in f 2.p e = 0 3.When data e arrives, If e exists in D, Increment f in (e, f,  ) If e does not exist in D, If the size of D = M p e  min e  D {f +  } Remove all entries e where f +   p e Add entry (e, 1, p e ) 4.[Output] Get a list of items where f +  >= sN

COMP Space-Saving Greatest Error Let E be the greatest error in any estimated frequency. E  1/M  -deficient synopsis Space-Saving computes an  -deficient synopsis if E  

COMP Comparison  -deficient synopsis Memory Consumption Sticky Sampling 1-  confidence  2/  ln(s -1  -1 )  Lossy Counting100% confidence  1/  log(  N)  Space-Saving 100% confidence where E <=  M Memory = 1243 e.g. s = 0.02  = 0.01  = 0.1 N = 1,000,000,000 Memory = 1612 Memory can be very large (e.g., 4,000,000) Since E <= 1/M  the error is very small

COMP Data Streams Entire Data Streams Data Streams with Sliding Window

COMP Data Streams with Sliding Window Association Clustering Classification Frequent pattern/itemset

COMP Sliding Window Mining Frequent Itemsets in a sliding window E.g. t 1 : I 1 I 2 t 2 : I 1 I 3 I 4 … To find frequent itemsets in a sliding window t1t1 t2t2 … Sliding window

COMP B1B1 B2B2 B3B3 B4B4 Sliding Window Sliding window Last 4 batches Storage

COMP Sliding Window Sliding window B1B1 B2B2 B3B3 B4B4 Last 4 batches B5B5 Remove the whole batch Storage